Which product is better for load testing?
Radview's WebLoad or Mercury's LoadRunner.
Should a startup invest in building a lab and purchasing this $$$software or should we go w/ a consulting firm
Re: Load Testing
As far as which tool, it depends on your testing methodology for "load testing." For example, load testing should be considered part of performance testing. In that case, are you also going to be doing stress testing? What about volume testing? Are you going to do any capacity planning?
Secondly, what is your environment? Do you have any application or database servers?
At a base level, both tools are very good but they, like all tools, have their own strengths and weaknesses and those strengths and weaknesses should be measured relative to the strengths and weaknesses of your test organization in which those tools will be used.
As far as who does the testing, I would not recommend going with a consulting firm (and I have worked for outsourcers and for those doing the outsourcing) unless you absolutely cannot get anyone to run the tests. Performance testing is most definitely a science and an art because there is a blurry line between what is and what is not adequate performance in many cases. If there is no experience in-house for this type of testing, perhaps it would be good to have a consultant get you started.
Keep in mind, however, that you are not just going to be testing but monitoring as well. You also need someone who can interpret the results of the tests to determine likely bottleneck areas and what can be done to alleviate these. If you do not have these skills in-house, you should. So I would spend the time to hire the right personnel first before going to a consulting firm.
Both of the tools you mention are very easy to learn so that will not be your problem. The big effort comes in interpreting the results they are giving you correctly and making sure you have an accurate picture of what the system is doing. So rather than have consultants come in and tell you what might or might not be obvious, you can at least check out the price for hiring the consultant and sending someone to training for one of the tools.
Alternatively, you can look at some of the excellent books on the subject. One of the best books I have found on this is "Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning" by Daniel A. Menasce and Virgilio A.F. Almedia. Very readable and full of good information.
Re: Load Testing
I heared that Mercury interactive have also Astra Load Test which is less expensive.
Re: Load Testing
For web load/performance/stress testing, #1 rated tool is RSW e-Test Suite. Very robust and USER Friendly. You can download a 30 day demo copy from our website.
I disagree with the answer not to use a consulting company for testing, unless you want to purchase high-end hardware and very expensive software, depending on the amount of VU's you need to emulate. Unless you plan on using the software frequently, do not want to emulate 1000's of VU's, and have the time to learn the tool and analyse the results, I suggest you outsource to a test lab. I am not marketing our services, just giving justifying when you would or would not outsource this type of testing. IF you choose to use a test lab be sure to have well defined requirements and have them build a test plan, very important. Also ask about the analysis reports they offer.
Re: Load Testing
A decent *summary* review of many commerical load tools was recently done by ZDNET. You can review this article at the follow URL:
Note that to see the complete review you must follow the "next" link at the bottom of the page that the above URL brings up.
But to truly select the right tool for your test envrironment, team skill set, and web applications to test [and their underlying technology] you must do a hands-on evaluation. This ZDNET article though should help you narrow you hands-on review down to two or possibly three tools.
-Enjoy, Terry Horwath
[This message has been edited by Terry Horwath (edited 09-27-2000).]
Re: Load Testing
I could not agree with you more...I have used the RSW E-Suite for nearly 2 years now. Seems that there are quite a few postings about it now on these boards. I know tools like those mentioned above do hold the "popular" vote, but anyone 5 years out of high school knows where the popular kids all end up. HA.
My point here is that RSW is indeed quite user friendly, and don't simply fall pray to marketing and go with a tool before you at least use free evaluations of others.
Do you work for RSW?
Re: Load Testing
Why do you say "For web load/performance/stress testing, #1 rated tool is RSW e-Test Suite" Who rates it
that and why?
Also, does it handle any other protocols besides HTTP (like ODBC?)?
Re: Load Testing
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by scooter:
I disagree with the answer not to use a consulting company for testing, unless you want to purchase high-end hardware and very expensive software, depending on the amount of VU's you need to emulate.<HR></BLOCKQUOTE>
I have to disagree here. There are many places that I have been where home-grown solutions were used which, of course, do cost time and money but are still servicable. Or Microsoft's free Web Application Stress Tool can be used to simulate thousands of users with no payment required and easy learning curves. Even SoftLight's $5,000 tool, which, granted, does not contain a lot of frills, is pretty good and allows unlimited virtual users. Also, if a company is using certain application server tools, like Dynamo, it is very possible to use tools that comes with this, such as URL Hammer - again unlimited session virual users. If an organization does not want to do internal performance testing, I ask them why. It is usually as you state: money and resources. Then I point out alternative ways to handle this.
I am not so much against using a third-party to host hardware, which is one of the things you bring up in your post. You are correct that it can be difficult to always match your production system. But outsouring the hardware is different from outsourcing the performance testing. For example, many companies use a group called Exodus and do their performance testing on their equipment. Others sign up with companies like IBM or Intel who run special labs for this. But, again, this is the hardware - not the testing itself.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Unless you plan on using the software frequently, do not want to emulate 1000's of VU's, and have the time to learn the tool and analyse the results, I suggest you outsource to a test lab.<HR></BLOCKQUOTE>
I agree there are times when this is valid but I think that organization's can do this internally and, in fact, should do so since performance testing and analysis is one of the most important things that an organization can do.
I also agree with you that must check the test lab out thoroughly for how they test and how they analyze. Many of the test labs I have talked to (and, admittedly, this is not all that many) could not, many times, tell me what queueing model they use. Or how they simulate burst traffic. Or how they characterize the workload on a load balanced system. One time I had to explain to one group that access method simulation meant different connection speeds. You also have to make sure they can handle any application server issues that come up, such as with WebLogic, FutureTense, or Dynamo. Make sure they understand what it means to run multiple instances and what effects that have.
Re: Load Testing
Careful with tools that simply play back log files gang ( Msoft's product that Jeff made mention of ) - these are NOT true measures as to how your audience "acts".
What is "act"?
Look at a visitor to your site..
He/she clicks, dwells, clicks, dwells etc.
Why is this important?
You wish to "stress" your server(s) as an audience would, -testing for example, IIS timeouts ( important for shopping cart functionality & those ever so elusive "dropped baskets" ).
Simply barraging it with IP requests, will not replicate user input....& will most likely create a false sense of security.
RSW & other offerings make use of "Emulation" or "Virtual User Pacing" - essentially upon playback of your script, you have the option of:
- "as recorded" ( mouse click times including dwells )
- "without delay" ( like IP hammering via log files )
- "Random recording"
Additionally, there are several other factors that MUST be employed if you wish to "emulate" just what your audience "is" when considering a load of your server(s):
These are: "Browser Emulation" & "Line speed"
Let's face it, your audience is pinging you from Macs, WebTV, Linux boxes, AOL accounts, NT servers, Win 98 & 95 machines & more... all with different browser flavors ( Netscape 3.0 remains huge in large corporations as an example & must be accounted for ) & get this: - Their resides in this ever growing matrix, another factor: ( drum roll ) it's called "line speed".
You must (again were talking about user replication here ) consider 28.8, 56k, T1, DSL speeds when assigning virtual user their identities...Again, consider emulation of a percentage of virtual users off an AOL 4.0 browser & 56K line, a percentage of AOL 5.0 users thru a T1 line, a percentage of Netscape 4.5 & 3.0 browser users with 28.8, 56K & T1.... all as part of your overall architecture.....
Did we discuss "caching" of these users ( first time or repeat ) as yet? - that's also (ug) got to be considered, - next time..
Again, lot's of opinions out there - just be aware of your org's business model. Are they talking "E-comm" ? Are you expecting a varied group of world wide users? if so, PLEASE do your homework...
Look at the various automation tool vendors such as RSW & MI, their trial copies & ask their tech staff questions....
Director QA MotherNature.com
Re: Load Testing
I agree. By "act" you are basically referring to workload characterization. Most tools on the market do not do this right now but you can employ your own sort of modeling and stastistics gathering via a program like Excel. But also remember that how the workload characterization is just one part of the "acting." How the system "acts" is another.
I agree that Microsoft's solution is not good for all things but it is great for generating session loads or background noise or if you want to drill down into ASP components that might be a bottleneck. In the context in which I was referring to it, I simply meant it as a tool that could be used to simulate thousands of users without requiring a consulting firm to handle that virtual user number. (Of course, even that depends on if you have the PC resources to run those thousands of users.)
You basically want a tool that will allow you to consider service demand and all that this entails, such as queue time and service time as well as being able to distinguish between wait and idle time, which are different things especially from the statistical modeling aspect of it.
Also it is important to realize that this is not stressing, although I might be missing your point or you are just using the term in a general sense. Stressing is generally where you do not consider such things as think time or wait time; thus persona testing is not as much of an issue and you basically pile it up. What you are considering is more transaction, load, or volume testing which does concern itself more with persona testing and leads us right back to characterizing the workload.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>RSW & other offerings make use of "Emulation" or "Virtual User Pacing" - <HR></BLOCKQUOTE>
RSW's eLoad is an excellent tool but one thing that it, and many others, do not handle is a fundamental part of the performance testing: burst traffic and heavy-tailed distributions. Burst traffic, contrary to what many think, is not simply "bunches of users at once" or paced iterations of staged users.
As far as the browser emulation, that is not always the biggest deal in the world, at least in my experience, unless you are using something that delivers content based on the browser string passed in, which many application servers or content management tools will do. However I totally agree with you about access method simulation and it is extremely important to have a tool that allows you to mix different simulated users running different access method simulations within the same scenario. (In fact, that is part of bursty traffic modeling.)
I think one of the best things people can do is check which browsers are hitting their sites and which OS's. You seem to have different experiences in this regard because the last six places I have consulted with, there has been little to no WebTV or Linux hits. And even Macintosh was barely on the scale. There was a lot of AOL, however, and most of the PCs that were hitting us were Windows 98 and Windows NT. And most of the places I saw (referring just to corporations) were not using Netscape 3.0 but were using IE 5.x or Netscape Navigator 4.08 (which is the browser behind many of the current Communicator releases), although I do remember that about three years ago it was the case that most logs showed you Netscape 3.0. But that just goes to show you that, as you state, you have to apply it to your organization and look at the tool within that context.