I am currently involved in a load testing project just starting up for testing the website of a financial services company. We have just completed gathering all our business analysis and are now at the stage of identifying what actions we will need to script for our load test.
One issue that has arisen is the strategy for how we will have our vusers hitting the site, and there has been a lot of differing opinions in this area. For example, should we focus on creating scripts with thinktime equal to real users (which could be quite large given how the site is used), or do we focus more on setting up script with little thinktime so that we can focus more on concurrent requests.
I am personally of the opinion that it is a better approach to look at the test in terms of concurrent users & throughput on the site, however this then brings me onto the reason for me posting this, which is that if we do this, how do we then work out the relationship between concurrent vusers and real users (i.e. if we have a particular action that runs acceptably at 20 concurrent users, how do we go about relating this back to the number of real users that would be supported).
I have tried looking through various source material, but have thus far been unable to find anything that has been of much use, so any opinions offered on here would be greatly appreciated.
When you design a test, you should always make it close to production, or you may test the wrong things and the conclusion will be a poor indication of what will happen in production. It is risky to assume that the differences between a test and production do not affect the conclusion.
One thing comes to my mind is that by shortening the "think time", you shorten lifetime of the sessions, which could significantly change the test results.
First you need to define the goals of your tests. And I think both scenarios mentioned can be useful.
When a system is first delivered in a state that is ready to be load tested, my first goal is to put a lot of stress on the system. At this point, it is irrelevant how the load is generated and how realitic it is... the idea is to just push heavy load through the system and see what breaks. Usually there are many configurations and OS/Server/Code confiuration tweaks that need changed.
Once the system is somewhat tuned, you can look into capacity planning. This is where modelling realitic user scenarios gets more important. Usually you can get by with the method of shrinking your user base to a concentrated set of virtual users (short or nonexistant thinktimes or pauses between transactions). Just be aware that this scenario may mask some real issues. For example.. each session created on the server uses a small chunk of memory. If you are not destroying sessions on the server (timing out properly or whatever), you could run into memory shortage problems when you have a large set of users logged in. You would never find this out using a small set of hyper users.