I am evaluating a performance testing tool for one of the project. The AUT is built a web based application.
- Has Telerik controls
- Using HTTP/S protocols
- Load balanced
- SQL Server
- Measure the page load time for each page
- Measure the time required to complete the Acquire process
- Measure the time required to complete the Extract Process
- Measure the time required to complete the Transform Process
- Measure the time required to render data on page at client side after server side operations are completed
- Monitor the server performance on resource utilization
- Vary the user load from 10 to 50 concurrent users.
Can you please suggest any open source or licensed tools which I can evaluate.
It sounds like this may be your first foray into performance testing. There are some architectural questions to answer which can then lead you to a tool. Assuming this is your first effort, do not take the path of attempting to deliver value without training or mentoring on your first project. This will be impossible unless you are bringing two decades of refined foundations skills to the table. And, were that the case you probably would not be asking the questions you have asked.
Answer these questions
1. What is the communications architecture between the client and the next upstream architectural component? Once you know then then you can simply look for tools which exercise that interface.
Once you have a subset of tools to examine then there are tougher questions
2. Does it work in my environment? This means OS, software configuration, security policies, etc....
3. Does it capture performance counter/metrics in my environment? CPU, Disk, Memory and Network are just the beginning. Ultimately to understand why somethings is slow or has failed, which works for one, you need to understand what finite resource pool element is being impinged, oversubscribed, etc... at the time of the event. Tools which collect monitoring data allow you to line up on a common timescale the resource use with the time of slow performance or failure to understand where things went wonky. Knowing what to monitor leverages your foundation skills in architecture and systems analysis
4. Does this tool facilitate my analysis and reporting? You have requirements related to load and performance. There are tools all over the capability map. Some of ones which throw load and leave anything else up to your programming skills to figure out. Some will auto mark some candidates for examination. Some will report against an SAL elegantly. Others could care less. All of this leads to the concept of each tool having a labor quotient and a tool quotient to hit a given mark. Knowing how long and how much labor pins itself to the foundation skill of project management.
5. Do I have the skills to use the tool? This could be the best tool in the world. This could be the pagani of performance tools, but if everytime you get behind the wheel there is a loud crash as you hit a proverbial tree, then the tool has zero practical value.
Hire experts on each of your remaining tools. Not people that have a checkmark on their resume, but ones which are demonstrably experts in their field. Provide them all with the same requirements for exercise and reporting of results. Bring them in for two days. Time how long it takes to complete each task to the same defined point so you understand any labor differences. Keep in mind you will not be as productive for likely nine months to a year of daily use and any differences are likely to be exaggerated.
Or, I could just say "Use Brand X. It's the best." In a real world that would be nice but it just doesn't work.
Originally Posted by akshayj
Totally agreed with jpulley3.
Originally Posted by jpulley3
Given this requirement:
I believe the only choice is using Selenium browser automation framework and running your tests using Grid.
Measure the time required to render data on page at client side after server side operations are completed
If this requirement can be omitted or substituted by running single-threaded Selenium test in parallel or manual client-side performance assessment using i.e. YSlow, in that case any tool will fit.
If you don't plan to simulate more than 50 concurrent users you can consider HP LoadRunner, it will be free for that amount.
There is also a number of free and open source load testing tools, the most popular of them seems to be Apache JMeter
Actually you can measure the cost of rendering with the built in developer tools in every browser with a single user. Chrome is particularly adept at this where the total cost can be illustrated
In short, all you need is a process change, to have all of your developers and functional testers just look at the built in tools and compare to a reference standard
Originally Posted by email@example.com
It could be used more like an automated warning system. (If a page performs slower than X raise a warning), but I would not equate performance under selenium to what the user experiences.
I did this comparison for Open Source tools with Pros and Cons of each tool sometime back,