| || |
I'm currently working on a standard practice for scalability testing a web server.
Now assuming there are 10 different transactions, and that the testing is being defined against a performance metric such as response time must remain below 10 seconds.
My question is are there any benefit from running scalability tests against each transaction separately, as opposed to running a scalability on all 10 transaction types as a whole.
My thought is by running the test against each transaction, I can determine the scalability potential for the individual transactions. Then when I run the test against all transaction types and determine the optimum user concurrency level that keeps the average response time for all transaction types below 10 seconds, I could then compare the two results. This may bring out any potential data, functional, database conflicts, when certain transaction types are requested simultaneously.
All said and done my question is "Is there any benifit in scalibility testing the individual transaction types". If so "Can a usefull comparison be made between those results and the results gained from the full scalibility test (All 10 transactions types running simultainous)". If so "What can these comparisons show?"
Any help on this would be apprciated.
Re: Scalibility Testing
I believe there are merits to both sets of tests. Running your Scalability tests individually is analogous to "Unit Testing" while running a single integrated test is more like "Integration or System Testing".
"Is there any benefit in scalability testing the individual transaction types" - Yes, it's often a lot easier to construct and analyze a test that focuses on a single transaction, in addition it may be possible to execute such a test earlier in the products life cycle and thereby catch an "ugly" transaction earlier. Resource constraints aside, it may make sense for you to individually test each new transaction separately (note, each transaction may merit a different performance target) before investing the time in constructing a larger multi-transaction test.
Unfortunately, good unit tests can't replace system testing, as the interactions of individual transaction may impact others, they can however reduce the number of performance issues that are not discovered until late in the products life cycle.
Lead Author "The Web Testing Handbook".