I have come across a situation that through the back door we have discovered that a vendor is testing with only 100 users.
Our system is supposed to be designed to support 40,000 users and currently supports 8,000.
Let me qualify this with the fact that with this particular vendors system we have had lots of trouble under stress conditions. They are trying to convince us to buy (full purchase) the next generation of their system.
My question is -- before I jump to conclusions -- is there any way that 100 users stress testing is enough?
the first questions is what do they certify their system to?
if they are a vendor and have an open culture you can always ask for benchmarking reports for the system under certain loads.
Also what do they mean by 100 users, if it was a backdoor discovery then it may not be correct information...
normally it would seem odd, but if the 100 users amounted to exercising 100 transactions a second - well thats 36000 transactions an hour - not knowing your test profile, i cannot say if it is good enough...but then your 40K users is only 8K users [img]images/icons/wink.gif[/img]
It might be the case that they are taking the approach of claiming that the number of Transactions/Sec that 8000 users will represent can be simulated with 100 users without using any sleep time. Although this is a cheaper way to test and can give you a good deal of confidence of what a system can do, it doesn't address problems that might occur when you have the actual 8000 users on the system using sleep times to realistically simulate those users. Issues of session states, caching and other things could be affected that the 100 user test wouldn't find due to the nature of the simulation.
The only other problem I would see in this is finding potential choke points with "X" number of users trying to access the site. If you are testing to see transaction per hour like Neil and James are suggesting that would be fine. If on the other hand you are trying to test how the system will handle the "X" number of users, it will fall far short.
In my current position I find it is important to test both and to do so in as realistic manner as possible.