Things to consider when Performance/Load testing.
Most people think of performance testing in once aspect. How do I create a load. Here are few things to think about...
- Load testing is no different they functional testing. There are various aspects of load that one must test and if you can narrow them down and focus on them you will have easier time testing, and also pinpointing issues. Mix them up in different patterns to isolate problems.
2. More load, or less capability.
- Don't just look at the load. Say if were you a QA person at MS and testing virtual memory. Don't use a system with 512megs of ram. How about if you use a box with 64megs? Wouldn't that make things easier? Think of things the same way, start with a simplest model and expand.
3. Time Compression.
- You don't have to run a test for 3 years to prove that it will run for 3 years. You can use 'compression' logic. Say if you expect load of 1000 transactions per day, if you can generate 200000 transactions, you can feel modestly comfortable that system will be up for say 2 years.
4. Don't forget the 'environment'
- Remember that no environment is pure and accurate. If a marathon runner trained to run on a smooth flat surface, do you think s/he will succeed in all marathons? Remember to do 'dirty' loads that include hung connection, lost connection, dropped packets, corrupt packets, de-synced packets and so forth.
5. Numbers sometimes mean jack.
- Contrary to the above statement, one must realize that no matter how much the number shows, there are cases that just can't be tested. Sometimes the best way to find a bug is to hire a contractor who specialized in the area and review the code.
6. Round the numbers up.
- Shown in example 3, I doubled the number to be on the safe side. Do this. Never round the number down, or cut corners.
7. Try different tools.
- Remember in the early 90's how the video vendors wrote drivers to do well in tests, while the video cards were horrible in a real life. Try different tools for kicks and see how the software handles it. You may find interesting things here.
8. CLEAN, CLEAN, CLEAN
- Clean environment isn't that expensive now a days. One of the best investment is a $200 spent on a 24 port switch and a cheap dsl router.
9. Hybrid testing.
- Take a system under a fixed load, and run funcitonal automation test against it. Again, you may find interest things here. Try other way around.
10. Incremental load testing.
- Remember that load testing takes time, so create check point where you take a code, and run a load and walk aways for a say a week or even a month. Doing this at various points may give you enough time to show you some problems that may not appear for a long time. Remember to use tools like performance monitor and plot a data once every 10 minutes. (don't let the log eat up your hard drive)
11. Don't discard any results.
- Sometimes it's difficult to see a problem. Don't discard any tests. Take all the information and enter them into defect tracking. Developer may see something in the log you may not.
12. Consider impacts to the system.
- Debug code, as well as extensive logging will bog your system down enough to mask certain bugs. Try to run as lean as possible.