We have recently come across a little glitch in reporting our results, let me give you a little background.
We have some business case or use case that we are testing.
Test 1 we ramped up users 1 every 10 seconds. The transaction response times where bad and the SQL server processor was maxing out with only 10 Vusers.
The developers made some changes and then Test 2 we ramped up users exactly the same way with 1 every 10 seconds. The transactions response times decreased dramatically, but the SQL server processor
maxed out sooner with only a few Vusers. This was due to the fact that the Vusers where imposing a higher load be the sheer fact that they where completing and re-running the scenario faster. Which showed through as a fact by the Hits per second increasing by 50%.
The problem arises now determining if the use case is passing or failing. This also leads to a real dilemma when reporting any results. It is really not accurate to report using the number of Vusers, but that is the only thing that an end users can understand.
Is there something we could do to our tests to more accurately model a users?
How do you report pass/fail by number of users?
I am not really sure of what you are trying to prove with your testing. So you are simulating a daily load of 8640 users. Are you inserting your own transaction points in the script? Do you do checks while the script is running? Are you running 6.5 analysis tool? I have more questions and maybe some answers. You can contact me via email or just keep posting. It will be slower but there will be a public record.
-- Mike --
There are 2 ways to estimate your load:
1) Amount of VUs
These measurables _must_ be used together
when estimating the load. Why? I leave you
find out the answer
Measuring end- to- end response time:
You can measure the response time that a typical user experiences by
running a GUI Vuser or RTE Vuser at the front end. GUI Vusers
emulate real users by submitting input to and receiving output from
the client application; RTE Vusers emulate real users submitting input
to and receiving output from a character- based application.
You can run GUI or RTE Vusers at the front end to measure the
response time across the entire network, including a terminal
emulator or GUI front end, network and server.
Measuring network and server response times:
You can measure network and server response time, excluding
response time of the GUI front end, by running DB Vusers on the client
machine. DB Vusers emulate client calls to the server without the user
interface. When you run many DB Vusers from the client machine, you
can measure how the load affects network and server response time.
In Measuring GUI response time:
You can determine how the client application interface affects
response time by subtracting the previous two measurements:
Measuring server response time:
You can measure the time it takes for the server to respond to a
request without going across the network. When you run DB Vusers
on a machine directly connected to the server, you can measure
In Measuring middleware- to- server response time:
You can measure response time from the server to middleware if you
have access to the middleware and its API. You can create DB Vusers
with the middleware API and measure the middleware- server
Hope this little bit helps. I also emailed you some documents for loadtesting.
-- Mike --
[This message has been edited by mracicot (edited 01-30-2001).]
[ QUOTE ]
Is there something we could do to our tests to more accurately model a users?<BR>How do you report pass/fail by number of users?<P><BR>
[/ QUOTE ]
Make your VUs mimic real user behavior. Think times, pacing, etc.
You pass/fail by getting good, measurable, requirements.