How can I distribute tests for environments with different configurations?
I'm currently working in a software house and we need to assure quality in different configurations. Some example of these configurations are:
- Browser - B1, B2, B3 or B4
- Operative System - OS1, OS2 or OS3
- Database - DB1, DB2 or DB3
That is a simplification of the current number of combinations we have to test, but in this simplification we have 4 * 3 * 3 = 36 possible combinations, for each we need one different test environment. The current approach is to run every test in every combination, so each test will run in all different test environments.
We have thousands of tests that need to run continuously and running each test in so many environments takes too much time and most of the tests don't add value to run in all configurations. If I'm running a browser test, I don't care what database the server has installed, itís enough to run the test in all different browsers (4 in this example instead of 36).
I'm trying to distribute the tests by all the different test environments, so that a tests is only selected to run in more than one environment if strictly necessary to ensure quality. Can you please suggest any possible solution to solve this problem? Or any information about similar problems?
What frameworks and technology are you using for your tests?
Typically what I do for my clients is setup a Selenium Grid, either in house or subscribe to SauceLabs or Browserstack. That handles the cross browser and operating system part. What they provide you is a simple way of specifying a "DesiredCapabilities" where you can specify which browser and platform, and the Selenium Grid will find the closest matching machine in the grid and serve that up to you. The 2nd part of getting this to work is implementing configuration management so you can feed it different config files to run the same tests across DesiredCapabilities.
Databases you have no choice but to setup different test environments. Ideally you'll want to have CI deploy on those environments to ensure consistency.
As to get all those runs coordinated. You'll want to use a Build Management (sometimes referred to as a CI (continuous integration) or CD (continuous delivery) tool). Using those tools, you can setup different tasks with different build agents. Then using a parallel Jobs, you can run your tests on demand through a web interface. The most popular of those systems is Jenkins. My preference is towards bamboo. I've also used TeamCity, that is a pretty good one too.