Question for Stress Testing...please help.
What kind of pass-fail criteria you for stress testing?
Re: Question for Stress Testing...please help.
I am guessing that you are talking about stress testing a web server so that is what I will give you my "thoughts" on. Some of these items are also just things to keep in mind when doing a load test against a web server.
For the web (ie. HTTP and HTTPS) a Pass/Fail can mean many things:
1. "If I get all '200 OK' HTTP replies back then it should have worked" is something that a lot of people think.. THIS IS NOT TRUE!!! Keep in mind that the HTTP layer is only the transport layer - in otherwords it only tells you "Yea.. The web server was able to fullfill the request." This should NEVER be considered a valid Pass/Fail without looking at the other items I have listed here.
2. Examine the HTTP Reply and it's body/content itself. Do the Title tags match what was originally captured/recorded/logged? Did I actually get the image back that I requested, or a HTML page stating it could not be found? Does the HTML page I get back contain a specific Anchor so I can be sure it is the same HTML page? ......
3. How long did it take to get the reply back from the server? Did it take 10 minutes and is that what I expected?? If you run a load/stress test and have a SLA (Service Level Agreement) with your customers of 8 seconds (which is the amount of time a normal user waits for a page to be displayed before moving on to Company #2 according to a study) and it is taking 10 minutes to get your page downloaded then I would consider that a failure!
4. Does a "Failure" occur only when you hit the server with a ton of requests all at once (such as a Login) that actually hit another server (such as an Oracle database that does the authentication process?) If so then it may not be a Failed web test but rather a sign that the Oracle server is having a problem. Try changing your thinking process and do some sort of "ramp up" test so all the users are not loggin into the server at the same time and the Operating System does not get hammered with Socket connection requests all at once. Many servers can handle thousands of users but fail with 100 if they all start hitting it at the same time.
5. Non-Persistant Connections can also cause a "Failure" to occur. Keep in mind that when you do not use Persistant Connections a web server has to do the following things for EVERY SINGLE HTTP REQUEST:
a. Accept() a new socket request.
b. Launch a new thread/process to handle the HTTP request (this is the web server stuff).
c. Read the Request and Process it.
d. Send back the Reply.
e. Close() the socket connection.
f. End the thread/process that was handling the HTTP request.
This could cause you to fail even though it would probably pass if the test was done the way most web browsers do HTTP calls today.
6. Multiple HTTP requests from one simulated user. This is how most web browsers also opperate (it is rumored that IE and Netscape can open as many as 32 concurrent socket connections to a web server!! The norm however is between 2 and 6.) If you are simulating 1000 users and each one is setup to use 4 concurrent connections to the web server then you are actually testing 4000 users instead of 1000 (at least that is how the web server sees it because it will start a HTTP thread/process for each one of those connections.) Both IE and Netscape ( I believe ) use only 1 connection whe they are setup to go through a Proxy. So if you "users" normally use a proxy then you will probably not want to do concurrent connections, but if they do not use a proxy then you will!!
Finally... Just keep in mind these two things and you should be good:
1. Who are my users? Are they using a Proxy? What version of HTTP are they using? Will they keep persistant connections with my server?
2. What are my users expectations? Will they wait the "8 seconds norm" or will they wait 10 minutes?
Hope this helps you out some!! If not then let me know and I will try to give you some more "thoughts"!!
Have a good day!
Re: Question for Stress Testing...please help.
The below questions were asked so I thought I would include them here and my "thoughts" on them.
1. What is the specific Anchor that can be found in the HTML reply?
Well, there really isn't one specific anchor to look for. See if the below sample situation helps you out some:
Say I record the following steps:
Open up www.yahoo.com and perform a search for say "OpenSTA" . I then click on one of those links and it takes me to another page and then I end the recording process.
When I replay the above I may get a HTML page back when the load testing tool does the search request but it may be an error page or it may not contain any results - I might have gotten a "server not available at this time" page instead!! If I check to see if the anchor I clicked on during the record process is in the HTML reply I get back then I can be sure that I am on the right track and that the web server returned the correct page to me.
You may want to check the OpenSTA web site or tech support department and find out if there is a way to get access to the HTML reply or if there is a way to search the reply for some specific text (say the text that you clicked on or the title of the page.)
2. What is the most important result I should look at to evaluate the test?
Ahh. The "what is important" question.. If I had to give a list of things that should be important it would have to be as follows (but keep in mind that this list can be different depending on what you want to test.)
a. Response Time - This could be a make or brake problem area with any web site. If a user has to wait a long time for the page to download then he/she will probably move on to another site and company!
b. Errors - Both HTTP and HTML errors are important. If you can run say 1000 users and they all get good response times, but only 700 of them finish with no errors, then around 30% of your visitors will have something wrong with the page when they visit it (maybe an image is missing, or some of the HTML gets messed up.) This could cause users to see your site as a "non-professional" type of site - they may consider the site to be a representation of the company itself and that could cost you business.
c. Page sizes or amount of data returned - Keep in mind that a page that contains 50,000 bytes of HTML can not only slow down your server but also slow down a network if it is an internal site (this could not only cause a bottle-neck at the network layer but also at the server layer since it has to process this large file on every request.)
Overall I would take into consideration the way that you serf the net. Do you like to wait more then 8-10 seconds for one page to come up in your browser? Do you like to see images that are missing or corrupted pages?
Also keep in mind the future! If you can currently run 1000 users with no problems and reasonable response times then great! But what about when your user base reaches 5000, 10000, 100000?? Will it be able to handle this amount of load? Since you are doing a load test you might as well find out!! The best thing to do when performing a load test is to go at it with a "worst case possible" point of view - what I mean by this is to not rely on caching by the browser or proxies in use and simulate the users with the "best" browser in mind (this goes back to the multiple requests at one time per simulated user.)
Have a good day!