Alternatives to LoadRunner? Specifications detailed in post
After seeing the latest pricing quote from LoadRunner our budgetary dividion had a stroke and requested a look into alternatives
Can anyone recommend a toolset that allows the following:
- HTTP / HTML
- Services Interface (SOAP Protocol)
- Ability to run behind an internal firewall
- Ability to report basic metrics (Response Time, Throughput, Errors Encountered)
- Ability to use parameter files
- Ability to populate variables from HTML page body
- Ability to leverage vuser pacing (i.e. - vuser can be set to run only every XX seconds, even if it completes in YY timeframe where YY < XX)
- Ability to support Mobile / IE / Safari browser types
Trying to stay in the < $10K / month range (the cheaper the better!)
Most tools, including open source ones will support all your requirements except the "parameter files" (but there may be different mechanisms for data driven)
Originally Posted by kjackey
The firewall thing is mostly a configuration on yourside. Either open a tunnel for the load testing tool, have the load testing tool running behind the firewall, or make an short term exception during load testing.
Most tools will have metrics for response time, throughput, errors.
Almost all load testing tool have parameterization.
The vuser pacing you'll find most tools call this "ramp up" and "ramp down"
Browsers supported for the most part is irrelavent, as long as you can proxy the connection to the recording agent. But some tools have browser plugins that make this process much easier.
Thanks for the reply, but in order to meet our page transaction per second targets we need full-run-duration pacing outside of the ramp up and ramp down phases
I would recommend going for Apache JMeter which:
- What are your reporting requirements?
- What are your analytical requirements?
- What are your environment requirements for monitoring?
- What are the environment dependencies for what your tool needs to run on (i.e. Must run on the Go operating system, build X.y.z.122, etc....)
- What are the language proficiency skills of your performance testing team?
- Every tool has a labor quotient and a tool quotient in the delivery, what is considered the appropriate total and blend for each project in your company? What if the total labor for tool X exceeds the total labor + licence for tool Y, is this acceptable?
- Would it be suffient to run one page over and over again to hit your page load transaction rate? You appear to have a technical metric and not a business metric.
- Have you considered a performance engineering path instead of a performance test path? A moderately funded performance engineering path will have greater benefits than a half hearted, low funded performance test path?
Your license cost should be the last item you consider or you may wind up putting the roof on a house with the butt end of a screwdriver because it is the cheapest tool available when renting a pneumatic nailer would be a far more efficient path
Last edited by jpulley3; 06-14-2015 at 11:53 AM.
Finally getting back to this effort and I really appreciate your guiding thoughts James.
Here's a more robust requirements outlay:
Load Testing Tool Specifications
• Be able to run from local (in-house) generators (Windows Server 2008 R2)
• Able to execute ~1000 concurrent virtual users using maximum of 5 load generator machines
• Data collection completely self-contained within the local firewall during the run (cannot have significant impact on externally available bandwidth at any time!)
Metrics Required (all metrics sortable and reportable during / after run)
• Page Response Time Elements
o Server Response
o Render Start
o Customer page complete (DOM Ready)
o Full page load (DOM Complete)
• Communication Metrics
o Transactions / Second
Total Transactions / Second
Individual Transactions / Second
o Connections / Second
Total Connection / Second
Hits per second
Total Bytes / Second
Individual Transaction Bytes / Second (if possible)
• Error Reporting - Passed / Failed transactions
o Total Passed / Failed Transactions
o Pass / Fail rates for each individual transaction (percentage calculation very helpful)
o Detailed error messages on a per – transaction basis
• Generator Metrics (Per Generator)
o Running Virtual User Counts
o CPU Utilized
o Memory Utilized
o Drive Space Available
• Interfaces Required
o HTTP / HTML
o Services (SOAP)
• Coding requirements
o Common coding language
o Ability to leverage data in parameter files
o Ability to store and utilize variables collected from HTML / server responses
o Ability to navigate full user path on an e-commerce site (needs SSL login capability)
Execute Product Searches using page search engine
Browse products across multiple page types (category, individual product page, etc.)
HTTPS / SSL Account Login Capability (Guest / Registered User Accounts)
User Account Information Page Access / Updating
Add items to shopping cart
Complete Shopping Cart
Validate Order Completion Page
• Ability to leverage vuser pacing (i.e. - vuser can be set to run only every XX seconds, even if it completes in YY timeframe where YY < XX)
Does this help with any toolset recommendations?
With the above open source tools based upon HTTP direct exercise are off the table. You might consider answering these questions during development and functional testing with the tools available inside of the browser, otherwise you enter performance testing never understanding/knowing whether these are working on an acceptable basis for one user
Originally Posted by kjackey
You might also consider the value of a commercial tool which is able to run a full browser, known as a GUI virtual user in the LoadRunner parlance, to be able to take two machines at the bottom and the top end of the resource spectrum for what is expected to be used. You can then collect the majority of your stats at the HTTP layer then contrast a named timing record with it's GUI complement, such as "LOGIN" along with "LOGIN_LOW_HARDWARE" and "LOGIN_HIGH_HARDWARE." Of course, you can do the same with any network which is being impaired as well.
Your callout of Selenium will take you to one of two vendors for the support of performance testing, Browsermob and Saucelabs. They run performance tests as all browsers. This is considered state of the art as of 1995. The resource costs are so high, and the actual server performance so obscured by the weight of the client is what drove the development of API virtual users which could be run by the hundreds/thousands per host depending upon the resource costs of the virtual user and the resource availability of the load generator.
I notice you have no monitioring callouts or need to integrate monitoring data for your system under test. This will place enormous barriers in front of you as you attempt to diagnose performance issues without an understanding of what resources are under stress on the server at the time of a long response time. This observation is tool independent.
Thanks for the input!
Have been looking into the LoadRunner TruClient as a possible solution, but as you mention the resource overhead is quite significant thus why I am looking into what other vendors might be able to provide.
As far as the server monitoring is concerned we have other tools that provide that information, so not completely flying blind