In this blog, I wish to discuss on step by step guide to Performance Testing of a Web application, particularly that has a high amount of traffic on it.
Have you ever wonder how would you perform performance and load testing of a web application with say 10K or 30K users? Also, those virtual users should behave like regular users rather than robots. If you are looking for an answer to the above questions, then you are in the right place. If not, you would still be surprised by reading how we did it very easily!
Recently we had a customer who wanted to carry out web application performance testing with varying user load. Starting from 10K to 30K, uploading more than 100K videos (or ~16 TB) in an hour! This blog is all about how we did that.
Before we begin, I’m hoping that you have enough knowledge about Visual Studio Test Rig, if not, I encourage you to visit a blog written by my colleague – “Performance & Load Testing Using Visual Studio Test Rig.”
Now that you know “what is Test Rig” (Remember I’m counting on you ), let us first understand how test rig works-
Above image shows the overall architecture of a test environment:
- The left part – Consists of “Test Controller” & SQL Database, is primarily responsible for governing your test. Microsoft Visual Studio will be used for executing a test or rather an application interface. You may install visual studio on the same machine where you have a different controller or on a different machine.
- The middle part – consist of “Agents” are responsible for generating load on your system. So, a ‘Controller’ behaves as a master and sends commands to an ‘Agent(s)’ who behaves as a slave(s) and generates load on behalf.
- The right part – the web application which will be tested.
The left & middle part together are called “The Test Rig”.
Coming to the main part of our discussion,
“How did we achieve such high load requirement?”
Well, remember this thumb rule in the test rig, the more load you want to generate, the more resources (CPU/Memory/IO) a client (Agent) requires. Scaling a machine vertically has a limit but horizontal doesn’t. So, with an increase in load requirement, we can add more agents to our rig network i.e. more machines.
“How many agents do you need?”
Honestly, this depends on different requirements or rather a type of user personas and scenario you are executing:
For example, in my case there were 3 different scenario(s):
- Test the web application – This requires quite a lot of RAM because the agent renders the HTML in memory and then also runs extraction rules (code).
- Test the APIs consumed by Mobile App – This may not require that much of RAM as the Web application scenario do.
- Test the Storage services (Video upload services) – This will require more IO and throughput at the network level.
Now if you try to apply similar math to these different scenarios then it will fail!
We were using Azure Standard D4 VM for #1 & #2 whereas Azure Standard D15 VM for #3.
Ideally speaking, if you have a very normal (not complex) script to execute then 1 VM with 4 cores, 8GB RAM can generate 1K users.
From there you are on simple math. Say a VM of specific size handles 450 users and then if you want to go for 5K user load then you will need 11-12 VMs.
Do you do this on-premise?
Yes, if it is a very small test and won’t require many resources.
No, if this is large user load test then it’s better to go with some cloud provider (we went with Microsoft Azure, I really liked the idea of cloud which is scale as you need, pay as you use!).
Since we specifically used the cloud for this purpose, I’ll be talking about it more from here forward.
Remember I said earlier that with high load requirement, you do horizontal scaling of agents by adding more VMs…? But the louder question is what if this test will run for multiple time over the period? (In fact, we were doing such web application performance testing for almost 6 months for different part of an application with each release).
So that leads to next question, “how one maintains those large number of VMs and save cost also?”. The obvious answer is to delete the resources when they are not in use. The next challenge is, if we delete resources then it takes time to spin up the entire environment… (almost 10-15 minutes for 1 VM) and that too manually! That’s a waste of time… right? So in the next section, I’ll explain, how we did that with automation scripts.
- A bit prior knowledge around VM & Azure Cloud (because I’m going to do it for Azure).
- Bit of knowledge about PowerShell
- Azure Command Line tools
How is it done:
- Setup the rig environment with one Controller VM and one Agent VM: While you set up these two VMs,
- Make sure they both have same Username & Password. Remember, this is top most important and necessary. Otherwise, the controller won’t be able to communicate with the agent. The trust between a controller and an agent can be set up by adding both VM in the same workgroup and that’s harder to set up in the cloud! A simple approach is to keep the same Username & Password for both VMs, isn’t it?
- Make sure that you add these VMs into a Virtual Network. This also means that you should create a Virtual Network first prior to creating any VM.
- Make sure that they both are connected, ideal they should be. But I suggest that you verify and make 100% sure that they are indeed connected and communicate with each other. This you can verify from VS.
- Now that you know, Controller can communicate with Agent, we can convert Agent VM into a VM image. Later this image can be used for generating new VMs automatically. I suggest that if you have any static test data, keep them copied in VM before moving ahead. Specifically, in our case, it was Video files (Large video files of around ~7GBs). If you don’t do this, your test execution will take time for copying these files onto each agent and that’s again waste of time.
Follow these steps explained at “Create unmanaged VM image from Azure VM” for agent VM. Remember, once you mark your agent VM as generalized, they won’t be accessible, i.e. you cannot do RDP into that VM and will become useless.
- Hopefully, by now, you should have converted Agent VM into VM image (I’m sure you did) if so, let’s begin writing automation scripts for VM Image. See script in below image:
The script has:
- Configuration parameters
- A loop that decides how many VMs should be created
- A line of code in PS that creates VM
- A line of code that restarts VM. (We had to add this code because in our case, Controller was not able to connect with Agent right after creating VM).
And that’s it. You run that PS script and that will create 10, 20 or whatever number of VMs you want.
Test Controller and Agent installers are available at https://docs.microsoft.com/en-us/visualstudio/test/lab-management/install-configure-test-agents?view=vs-2017
Management of Controller and Agent from VS is available at https://msdn.microsoft.com/en-us/library/dd695837.aspx