Software Quality Assurance Forums: require to do test on production server

This is a great question! Unfortunately, it also has a very complex answer. A$H is right, you need to look at the risks and rewards of doing so.

My company provides a self-service external load testing service/tool, and so this question comes up often with our customers.

As such, I attempted to answer some of those questions in a new FAQ entry for our BrowserMob blog: Should I test on production?

To make things easier, I included my full response here:

--------------------------

There is no right/wrong answer to whether you should run a test on a production environment. In an ideal world, you'd have a staging environment that is exactly like your production one. But that is rarely actually the case.

Definitely do your initial performance testing internally first. Keep the tests simple and focussed, pinpointing specific parts of the app (certain parts of the UI, the database, etc). Sometimes these tests can be as simple as attaching a profiler (ie: YourKit for Java) to a developer's desktop and just navigating through the app by hand a few times or playing back a Selenium IDE test 10 times in a row.

Often times you'll find 80% of your performance problems with really simple, low-volume tests, since poor code usually stands out like a sore thumb even under low volumes.

Then it's time to ensure the whole environment will work together. This is where a full scale load test comes in. If you have a test/stage environment that you can use, definitely start there.

If things look good, then the next question is: is my production environment different enough from my test/stage environment to warrant additional testing there?

It's impossible to say for sure in your case, but in my experience, my test environments wouldn't have a load balancer or redundant databases and web servers. In those cases, I considered that a large enough difference that I wanted to do a real world, production environment test. Otherwise, my load balancers could be broken and I wouldn't necessarily know it until it's too late. The same could be for my database clustering configuration or any number of other settings.

If you do decide to test on production, then you'll need to analyze the risks:

If users are allowed to access the site, they could experience major slowdown and/or errors. Is this acceptable?
If users are allowed to access the site, it may be difficult for you to pinpoint the root cause of performance issues, especially if there are a lot of users doing different things.
If users are not allowed to access the site, is the business prepared to allow the site to be unavailable for some time.
If the load test generates a lot of data on the production database, it could cause slowdown that will continue even after the test is complete.

These are just some of the risks, but I hope it gives you an idea. In general, I tend to lean towards doing some sort of "real world" test. Afterall, that's why we created BrowserMob in the first place!

Whether that test is your actual production environment or a close approximation of that environment, the key thing is that you analyze the risks and understand what you're testing for. You're not just testing the software now, but you're testing the whole environment (load balancers, web servers, routers, etc).

--------------------
Patrick Lightbody
Founder, OpenQA and Selenium Remote Control
Founder, BrowserMob: On-demand, low-cost load testing with real browsers

On-demand: Control thousands of browsers in minutes.
Uses real browsers: True no-hassle scripting and automatic AJAX/Flash support.
Great reporting: MySQL data dump, full object-detail, real screenshots of browsers.
Low cost: Run for as little as $1 per virtual user per hour.