running regression on production
The developer manager of the company where I work at thinks that we need to run automated regression suite (Selenium) in production. I think that it is a terrible idea and that we should create a staging environment exactly like production and run our regression there and then just run a smoke test in production. I been looking for material to prove my point but I cannot find anything that will spell it out. I would like first if I am wrong and if I am not then please send me the link to any article that will help me prove my point.
I don't have a article, but I can tell you some of the reason why it is a bad idea.
Originally Posted by Augus
* Tests generally do not go through as much security scrutiny as production code. There was a study done by grid-tools.com that mentioned 50% of data breaches were through insiders through testing resources. (Take this with a grain of salt as they are selling a test data management product). Since tests generally do not run on authenticated hardware based HMACs, you have a lot of sensitive information that's in plaintext in memory while the test is running.
* Exhaustive regression test will not give you any more coverage on production vs. another environment. If tests are passing on a testing environment. The only big difference (provided a deployment procedure that uses packages instead of copying over files manually), would be just hardware and configuration. Neither of these would benefit from regression tests.
* Extra cost of maintaining tests for a live production environment is expensive. The extra time it takes for you to maintain tests on production, is it really more benefit than a few manual spot checks specifically around the feature released?
I am going to use your points while making my case!.
You might also want to point out that regression testing has a specific function - to determine whether or not there are any issues in existing functionality as a result of introducing new code.
If you're trying to find any problems while regression testing, then by running it in Production you're already too late - the code is live, so why bother?
(It is, though, an idea to run a light test suite on the Production system daily, to make sure that everything is still running smoothly. I've heard a story of a tester that set their homepage to their website so that they would be the first to know if the server went down...)
100% agree with above replies.
Other points that I would add are:-
1. Regression testing on Prod environment means it will impact the other users using the system at the time of testing (which will directly impact business)
2. To drive regression tests, you would possibly go for random values to check. For example, you take "0" or "null" as parameter or some field value. In case, the exception/error is not handled, there is a chance that it will hang the system or even destroy it
3. After regression, suppose you found some bugs. If somebody is working on a Prod system only, I'm not sure if they want to take the risk of implementing defect-fix directly to the Prod instance before checking it on a Test instance. That would be a huge risk, IMO. :O
Multiple linear regression attempts to model the relationship b/w two or more ,,and we need run regression any thing productions ,,,,,,,,,,,,,,
Another point would be in testing we not only test on data that is currently on system but also for the test cases that can happen in future which cannot be tested in production on current data.If we are missing that Risk area to test then what is the use to regression test in production.
Tags for this Thread