| || |
Real Load with Vitual Users?
In all Load Testing tools (I am using WebLoad), there is the concept of Vitual User. The number of vitual users that are 'hitting' the system at any given point is the load.
In order to achive realeam, one adds in Think Times (or wait states). By adjusting the Load and the Think Times, you can estimate the throughput for a given parameter before you even run the test. Both the load and the think times can be calculated mathimatically with the right model if the throughput is known.
So, here is my question: Since the think times can be adjust (you can have no think time, going all the way up to ??? days|weeks|years ...) With a faster think Time, if you put on a load of 100, it may really be a much higher load. How can one be assured that they are testing with the proper load and not over stressing the system?
Let me try explaining this with an example:
All of the think times are set based on what it would take a person setting down and doing each step would take (in seconds). The values are then divided by 1000 and that is the hard-coded think time. We then have a timeBase value (default=1000) to adjust the Think Time on a global scale. With the base sleep times, a baseline is obtained on how long one full round takes. based on that, a mathamical model was made that will find either the Requitions Per Year (our thoughput measurement), the load, or the value needed for the timeBase. This model has been shown to be acurate by comparing it with real tests.
Now, if I wanted to put though 90,000 requistions per year (RPY) using 1,000 vitual users, my model tells me I need a timeBase of 5,741. Now, I can achive the same throughput of 90,000 RPY using 100 vitual users by setting the timebase to 574.
What I am being told be development now is that 1 vitual users = ~100 real users due to the speed at which a vitual user does the same task. This doesn't seem right, but I am unsure on how to argue against it (being new to Performance Testing).
How do vitual users compare with real users?
555 North Mathilda Ave
Sunnyvale, CA 94086
Re: Real Load with Vitual Users?
As far as one virtual user equaling one hundred real users, that sounds like a problem in the virtual user setup or a problem in how development is looking at things. (I would guess the latter.) There should be a one-to-one correspondence at least in the degree to which it appears virtual users are hitting the system (unless you are considering iterations, of course, which can skew things).
Think time is just one aspect of simulating users and you can randomize this as well. You also have idle time and wait time, which are, technically speaking, different even if the tools do not state that.
But I think what you are really getting at is workload characterization, which is a fundamental part of performance testing. You basically have to look at a specific time slice (or foliation) and take representative user load during that time - meaning how many users are hitting your site. Not only that, but you have to consider what they are doing on the site.
You talk a lot about the mathematics you are using in general terms, but what you are not considering is that throughput is a variable measure of forced flow or utilization. The "Load" that you seem to be referring to is the "average load" on the system, which is a transaction performance measure. You do not state the model you use but it sounds like you might have some assumptions that are not in your benefit. If you are mathematically modeling the system, I would recommend looking at service demands and applying queuing theory - which accounts for service time and queue (wait) time. I think your discrepancies may be coming in because of the model you are using. To better understand that, though, I need to know if you are considering an open or closed model. Also is your model taking into account burstiness and heavy tailing? Do you assume an infinite population and how do you normalize it? Also, what is the throughput equation you are using and is it based on single or multi-class models?
Bottom line, a virtual user is going to be a recording of whatever you did on the site when setting up that script. So I guess what you could do is have one of these developers monitor the site while you are doing the tasks manually, then again when you are recording the virtual user, and then again when the virtual user is running alone. Personally, I think your developers are giving you a little wrong information or just honestly not understanding what they are looking at. But without knowing more about your setup and the nature of how you are running your virtual users, it is hard to say.
[This message has been edited by JeffNyman (edited 09-19-2000).]