What are you using to run your Selenium tests ?
I am looking for a test UI to run automated tests.
I am NOT looking for a test framework, a bug tracking tool, a requirements repository, or a testcase management tool.
Just a test runner.
1. Need a test UI that we can use to run automated selenium tests on multiple VM's (any flavor of selenium --- C#, Java, NodeJS, Python, etc)
2. The test UI needs to be able to run and rerun specific tests on-demand (If my test suite is 100 tests and only 10 fail, then I only want to rerun 10 tests)
3. Need to be able to retrieve historical results through test UI
4. Need to be able to schedule test runs (Jenkins can be used for this)
5. For an example of a test UI, please see the QTP test runner in Quality Center (TestLab)
Is there something out there reasonably priced or open source, or should we just write our own tool ?
1) For multiple platforms/browser, most people will use Selenium Grid. YOu have a choice of using a 3rd party grid like Sauce or Browserstack (there are several others out there, those 2 are the biggest at the moment), or you can build your own. Main reason you'll want to build a grid or leverage 3rd party gird vs running browsers directly on the build agents is you'll find it's a pain to get browsers running outside of a user context and running as a service. Also, it's alot easier to maintain VMs that just have browsers vs. VMs where you need to setup the build agents, etc... as well.
2) Most unit test framework runners do have rerun flags you can do via command line. However most don't expose them in their jenkins plugins because its generally bad form to even have flaky tests as part of CI
3) Jenkins and most other build tools, you can specify an "artifact", and it'll archive artifacts so you can look at them later.
4) My fav tool at the moment is AppVeyor. But Jenkins or other build tools work fine.
5) For the most part, i think in terms of a good UI/launcher, Quality center is good if you have the budget. Most TCMs that support test launching are quite pricey. My preference is the write a test reporter that makes API calls after the tests have run to report the tests results to TCM. That allows me to use cheaper TCM systems.
i have created an excel based runner. It has the capability to
a) display all tests in excel table
b) selection of tests to be run
c) ability to provide run time parameter (e.g. environment)
d) ability to run a single or multiple tests in a batch
e) hyperlink to view individual test result or batch result.
f) any many more...
High level Technical information:
It uses VBA to interact with ant & testng using a batch file
-- after the tests are selected, custom testng.xml file is generated
-- this testng.xml file is called in an ant task
-- ant build file is called using shell command of vba
I will post the entire utility sometime in future.
thanks for the feedback. the solution we choose will most likely be a combination of Jenkins/SeleniumGrid/excel
we are currently using QC/ALM but want to phase it out eventually. at my old job they used a web interface written in java which was awesome. if i could find something open source or cheap like that i would be psyched
A very hacky way of doing this I've done in the past is...
Originally Posted by Pastor of Muppets
1) create a parameterized build in Jenkins that can take in a test case ID, the do a deep text search for the unittest fixture that contains that test ID.
2) In the unittest fixture, add TestCase ID as either a decorator or a comment.
3) In the post build task of the parameterized Jenkins build, add a task that reports the test result to your TCM system (test case management)
4) Create a simple grease monkey script that inserts a Run test button, any time it sees a Test Case ID. So for example, when you login when you login to Jira and you see a Bug that refers to a testcase ID, a button will appear next to it and you can simply click that button to run the test. Of if you login to your TCM system and view a test plan, you'll see each test case with a run test button next to it.
5) With the grease monkey script, when you click on the button, have it generate a redirect to a URL that calls Jenkins API to execute the parameterized build with the test case ID parameter.
You get this really cool hodgepoge system that let's your run the corresponding automated test at a click of a button any time you see an associated test case.
For me - I had created my own framework with Maven, TestNG and driven through excel.
To run it - just need to run it as a maven proj and get this integrated with Jenkins too.
We can use other other build tools like Ant - but need to build your dependencies physically.