Need advice on automation/test management tools
I oversee a small group of testers for a small to mid size company. We produce both software products and hardware B2C products so we have development houses in multiple parts of the world, all being managed by independent groups within our organization. Each development house uses its own SD methodology as well as their own tools for ALM, such as TFS and JIRA.
My goal is to allow for this decentralized development structure, but to create a centralized Quality Center of Excellence that performs testing for all products. This means that I need a fairly product and platform agnostic set of tools that can synchronize with TFS, JIRA, or any other products that our development teams choose to use. My main goal is to find the best product or products to solve the following problems:
- Lack of test automation
- Lack of centralized test case repository and test case management
- Lack of centralized requirements management
- Lack of centralized defect tracking
I have some limited familiarity with Eggplant and Test Complete/QA Complete, but was wondering if anyone had any recommendations for software that will help solve these problems, is relatively easy and straightforward to use, scale-able to another 5-10 additional testers in the future, and wont be ridiculously expensive. Thanks in advance for the help.
In my opinion, there are basically two routes you can take:
1. Go for an integrated solution from a big tool vendor
The stock example for this is HP ALM (Quality Center) + UFT (QuickTest Pro), but other tool vendors such as IBM sell similar solutions. The downside to these solutions are the costs. Especially for a rather small organization such as yours, it might be rather expensive (I don't know the prices, but I know they don't exactly give their software away).
2. Create your own integrated solution.
An example I would suggest is:
* Using Jira for your defect tracking as you already have this within your organization, plus it's pretty much the de facto defect tracking standard.
* Using the Jira plugin Zephyr (https://marketplace.atlassian.com/pl...thed.zephyr.je) for your test and requirements management needs (remember that requirements can be registered in Jira just as well as bugs can)
For your test automation needs, it depends on the types of applications you develop. If they're all web based, Selenium Webdriver is free, open source and has a big community for support (on this board as well). Extensions are available for testing mobile (Android and iOS) applications as well. If you need to test applications outside of a browser, your tool selection would depend on the technology used.
Try and find a tool that can create reports in JUnit format. This will allow you to use a Continuous Integration (CI) solution such as Jenkins (Welcome to Jenkins CI! | Jenkins CI) to run your tests and have Jenkins report your test results. You could then use a Jira plugin such as JiraTestResultReporter (https://wiki.jenkins-ci.org/display/...eporter-plugin) to upload these test results back to Jira to have everything administered into one place.
This solution will take a little more work to set up, but you won't need to spend lots of money on licenses. Atlassian products are very fairly priced, and the rest is open source. Don't forget to take into account the amount of extra work needed to get everything up and running though
Good luck and if you have additional questions please do not hesitate to post them here or contact me directly.
Thank you for your feedback, this is some good insight into some things I did not originally consider. But I was wondering if the JIRA solution would integrate into Team Foundation Server. Although some of our teams use JIRA for defect tracking, the majority of our teams are using TFS. I don't see that changing anytime soon either. That was why I needed a central repository of test suites and requirements that can connect to any test team regardless of what ALM tool they are using. Hopefully that makes sense, and thanks again for your feedback.
Originally Posted by basd
The tool set you choose is dependent on the underlying tech you are testing and the skill set of the team doing it, if they are happy or not to code away at the open source tools or you want to remove that dependency on a highly technical resource.
For example, Selenium and QTP have been totally unreliable on our UI but work like a charm for other people. T-Plan and Eggplant worked much better. We settled for Eggplant but it is pricey.
Both Eggplant and T-Plan have their own test management tools. I have seen a product from IBM (RQM) that is really nice in terms of managing work from requirements to tests but we did not end up getting it. It would put JIRA to shame but you have to watch your budget.
Zephyr for JIRA is OK but a bit infantile from a QA reporting perspective.
Good Luck to you - let me know what you chose.
Originally Posted by QADirector1
Zephyr for Jira IMO is pretty horrible, you're better off using Zephr standalone with 2 way integration. As horrible of an experience like using Jira for project management. It feels very hackish and not well thought out. It's hard to manage large test suites in Zephyr and the reporting API isn't as robust as many of the stand alone TCM systems. For example, say some functionality grows and you want to split it into 2 separate suites and move test cases between them and manage them in a hierarchy. It's very hard to do it in Zephr for Jira.
I prefer a TCM with either a better API or 2 way integration with the bug tracker. It's not that hard to write adapters to facilitate the integration. So I'd rather not compromise on managing test cases/plans.
Hi QADirector1 ,
I would like to suggest another tool not yet mentioned here: PractiTest. SaaS Test Management Tool and QA Management Tools - PractiTest
As it's main "attraction" it is an ideal tool for creating a centralized repository ot test cases and is great for distibuted team.
Part of this is due to it's many integration possibilities with some of the tools you mentiones your variouse teams use. So, they can keep using them and the integration allows the information to be gathered to the test library. Plus, it also allows you to create your own integration via API so you can create your customized solution.
You also mentioned you wanted some flexibilty to increase and perhaps then decrease the amount of testers, which is compatible with PractiTest's scalable pay-as-you go pricing on even a daily basis.
Not only is it realy simple to use a start working with immediatly but the reporting is great to help you stay on top of everything every team has done. Most of PractiTest clients especially like the personal support offered not only during "setting up" but throughout usage of the tool.
No strings attached you can sign up for their demo and get a free trial period via their home page.
I you have any further questions please write me at: email@example.com