I am using Ajax TruClient scripts for performance testing. The Test environment has 4 app server nodes and 4 Load Generators. So its one to one mapping.
In this cluster environment a load balancer sits between load generators and app servers.
To get more hits per second on app servers i increase the number of vusers. But increasing the number of Vusers is not resulting in higher peak hits per second. Load balancer is doing good job by evenly distributing the load. I am using 200 Vusers on 4 Load Generators. I have also tried using 8 Load Generators with 200 Vusers but peak hits per second is not increasing lineraly.
Have you faced a similar situation? What do you suggest in such a scenario? [img]/images/graemlins/confused.gif[/img]
If you like to check four app servers and are not worried about LB, then why not single out each of the four apps individually and put on the load? You can achieve this by turning off 3 out of 4 App servers and let LB do its job. It has to direct all requests to the single running App server and then you can put the load from 3 or 4 or 8 LGs that doesn't matter as the load is getting on your App server. Follow the same procedure with remaining three app servers.
Also try increasing think times to unrealistic values in your script so as to keep vusers alive in the system for longer duration and keep on putting more load. You should achieve a state wherein all users would be in system (as per LB distribution) keeping the optimum stress.
I am confused. If I understand the problem statement correctly, we have 4 app servers and 4 load generators. Even after increasing the "number of users" i.e., rate of load generation we are not seeing the increase in the number of hits per second overall.
Let us say you ran a test with 100 users and achieved X hits per second overall. does this mean even after increasing it to 200 vusers, you still get X hits per second. That's weird. If there are no bottlenecks then the hits per second has to increase with the increase in the number of virtual users (provided no other test conditions have changed). And what is it that the LB is doing a "good job". Sorry I don't understand. LB will distribute the load as they arrive into the system evenly. If hits per second is not increasing does this mean you have a slow network in the middle or the LG are stressed that they are pushing the load at a slower rate. given that you have increased the LGs to 8 as per the load balancing and load generation principle, you must get the required number of hits unless there is a major bottleneck in the system.
Is it always happening with anything beyond the established number of users? Do you see the hits per second increasing gradually as per your ramp up pattern? In that case, LR is doing fine but you have reached a point beyond which your system is not performing as expected.
Unless you have VERY powerful generators (eg. i7, 16GB Ram, Windows 7-64bit) you are probably maxing out the CPU on the generators.
Monitor your generators and do not allow cpu to stay above about 85%... if it does you have saturated your generator and adding more users will do nothing but make each vuser wait longer. Add more generators to spread the load.
Load Generators are very powerful. CPU usage on generators did not go beyond 50%.
The root cause of the problem was high CPU usage on app servers. On all 4 app servers CPU usage was above 90%. So when i started to increase number of Vusers only the job queue was getting bigger on app servers because they can't process more requests and hence Hits Per Second was not scaling up.
Load Generator, Load Balancer all were working good. Its just that app servers were peaked.
Thanks a lot to all of you for your guidance and help.