Building a testing department from scratch
As preface, I'd have posted this to the newbie section, but that seems to be restricted (!). If this is the wrong forum, please move the thread accordingly.
I am in the process of putting together a formalized testing department. I need a little help (to start) with identifying tools. We are using JIRA already and will be using Selenium, and I'm looking for something to manage test cases and help with the department-side management of the testing process.
I've been looking a bit at QAComplete, and would welcome any feedback on if it is good. We're open to using either commercial or open source. It needs to integrate with JIRA, and I need to integrate Selenium as well in some fashion. Any ideas? Is this the way to go, or should I be looking at something else?
Same question i have
Same question i have
Originally Posted by tabj
Don't worry too much about Selenium integration.
What's more important is the Test Manager has an API (preferably REST) that allows you to report your executions. This will allow you to add a TestWatcher rule in your unit test's that's driving Selenium to report back to the Test Manager. There are very few test managers that offer selenium integration out of the box, and the ones that do, usually is Selenium IDE (which is outdated, most people are using Selenium 2, aka Webdriver). As for Jira integration, it's good you find one with Good Push integration (pushing bugs from a failed test case to Jira, using the test manager to directly submit a bug). Some may support pull (taking new bugs submitted, and allowing you to associate them with a test case, so basically implementing a bug search within the test manager for the purpose of linking test cases.), but most that support pull integration do so poorly.
It's hard to say which is the best to use. I have opinions, even though I've evaluated nearly a dozen of them, I've only used like 3 of them in a long term evaluation to get a clear idea of their strengths and weaknesses.
* TestLink - free an open source. However it uses PHP with a poorly designed schema. Performance will slow down exponentially the more folder levels you add. If you decide to go with this, avoid subfolders. In terms of usability on a small scale, it's fairly usable. It doesn't have a very good API, but it's easy enough to do some things, and for other things, you can write directly into the DB because of it's fairly simple schema (which is also it's weakness). There are hacks to allow you to do Push integration with Jira. But I'd recommend avoiding it and just writing a small bit of PHP code to insert into the submit bug call back. This is because your setup may not necessarily be the same as the person who wrote the hack, and this is not a supported integration. It's better to do your own custom hack in this case.
* Test Rail - I find this one pretty good at adding and building a test library. I find it weak when trying to reorder and reorganize test cases. It does support 2 way integration with Jira. The push part (pushing new bugs to jira) works very well, the pull part (getting bugs filed via jira back and associating it with the test case) not as well. For the most part, if QA files the bugs through TestRail, you'll have the bugs associated with the test cases. It has a fairly good API. I think it's a bit annoying to work with because it was designed in the perspective of the person designing TestRail. For example, to report a result, you need to have a test execution ID first, which is cool if you were running a manual test. But in automated unsupervised test, how do you know what the test execution ID to use without a extra call to find it?
* Silk Central - Love / Hate. I love it has very good integration and lots of features. I hate how it's so hard to use. It's UI is littered with icons which you have no clue what they do. It's hard to figure out what you need without reading through the manual each time. It has pretty good push/pull integrations. It is however pretty expensive compared
QAComplete I only installed it and played around it a couple hours. My initial impressions is it's pretty good at managing test cases and requirements. I didn't get far enough to looking into their Jira integration, but I think they support a simple push integration. I didn't get a chance to look at their APIs.
Check out Inflectra's SpiraTest. It what I chose after evaluating a bunch of tools as a QC replacement. It does have selenium integration as well as Jira. Don't really use the selenium integration though as our tests are really just a command line call to startup the framework, then use the API to report results back.
Well. First I would check if there are real needs for on-premises or SaaS tool. I would use a SaaS tool if possible to minimize the maintenance needs. Next point to consider would be to check if I could get organization to write down user stories / requirements. I would really recommend to do it, makes test planning and maintaining the understanding what software easier. Saves time in the end. Most, but not all of the tools have possibility to manage requirements along with test cases. Then, yet another thing is the possibility for testers to use test test tool to enter defects, their two-way linking to developers defect management tool ( Jira in your case ) and it's link back to re-running the failed cases again. I would rate easiness of use a lot as it saves time and thus money.
As a tool suggestion to those mentionen above I would add Meliora Testlab that I have used lately in my project. It has a good combination of features along with decent interface.
I have experience using and administering ALMComplete from SmartBear over the last few months and I have found it to be a solid tool. We are using the on premises version but they do offer a SaaS version that seems to be popular. I do know that they are releasing much more frequent updates (monthly I believe) to the SaaS version and then rolling them up into larger releases every quarter or so for on premises clients. You can get an account on their SaaS version and try it out. That is how I keep up to date on what new features are coming down the pipe.
You mentioned Jira and ALMComplete does integrate with it. We are going to be integrating our ALMComplete and Jira systems later this year.
Do or do not... there is no try. -Yoda