| || |
Daily automated UI regression on various browsers?
We have automated regression suit, which needs to be run on AUT on daily basis since there is almost a new build deployed everyday. This poses us with a challenge of completing entire regression in a day due to number of scripts to be run, browser and network slowness etc. We have multiple VMs available from where we initiate simultaneous execution of different scripts.
In general, I wanted to know if anyone is running automated regression on daily basis from different browsers and any strategy around it.
I've setup systems that have done this on every code check in. And also have done systems where this was continuously being done on changed code as the developer is working (but that's another story)
Originally Posted by Prajakt
1. Prioritize your tests.. smoke, features affected, regression. Then used a staggered build and test pipeline.
2. Parallelize the heck out if it. And if you have the budget, use a 3rd party service like SauceLabs, Browserstack, Testbot, etc... to allocate your browsers on demand. Easier to scale than maintaining your own VM farm.
3. Use a CI tool to schedule builds around check in. My favorite at the moment is bamboo and travis. forking builds with automatic merges makes life a lot easier.
4. Make your tests fail faster. For example, I avoid any sort of non hard coded waits.
5. Make your tests all independent. I use separate builders to build the test data needed for each test, which then cleans up after itself. Having tests share data/accounts is a good way of making your tests too coupled, so avoid that.
6. Use headless tests as the first screen when possible.
7. Think about the strategy for retiring tests. Most stops think in terms of requirement coverage when it comes to end to end automated tests. But that leads to creating more tests that take up time and harder to maintain. Think of tests are developer productivity, start retiring tests when you see overlaps in coverage and reducing the frequence they run in areas that do not break as often (for example, offload stable functionality of low importance to a weekend or nightly build, while keeping your important tests in CI).