Looking for JIRA test case add-on recommendations
I just wanted to see if anyone has had experience with any of the plethora of test case add-ons available for JIRA. I am looking for some experiences or recommendations.
Some brief background on my situation:
I work with a software company that is more along the lines of a research and development operation. Their software process is very loose, agile/scrum based and virtually no requirements at all aside from what is kept in Jira (the company is made up of mostly research scientists and the software team, and the software team makes most of the software decisions since no project or product managers are present). I am the only QA person and cycles are either hard and fast or slow to non-existent (the time I would be writing most test cases). Right now all test cases are kept in huge excel documents. I was hoping to transfer these to something that can be associated with issues in Jira and make the test cases easier to maintain, modular, re-usable and easy to mix and match to cobble together a test suite based on the current needs.
Some add-ons that I have come across are Zephyr, Practitest, Behave pro, Test Rail and Xray. I do not have a ton or time to trial more than two, maybe three, before making a decision and moving forward, thus why I am looking here for any suggestions, experiences etc. to help guide me a little bit.
Just to contribute to this some more, I have been reviewing the options (and there are tons...) and it appears so far that Zephyr is by far the most integrated in to Jira (they offer a specially created version for Jira 5 and 6). It adds an issues link in the navigation of Jira, bugs and tests can be linked directly, the dashboard can contain widgets with testing information, updated are included in the activity stream, no switching between browser windows, tabs or software in general. All in all it appears that you are just entering another Jira issue with the added Test Steps and Test Execution fields when the type of Test is selected.
However, they do not go in to much detail of how these are sorted in general such as are there projects under the Tests tab? doesnt appear so from the screenshots, although projects are specified when adding a test. Also, will the issues of type Test also show up within the other issues (bug, story, task etc.) for any given project, muddying the pool with the possible hundreds of test cases? (not a huge deal with filters I know but still curious, and would a test entered for a project count against the issue metrics for that project for a given time period?)
Unfortunately as well to try Zephyr for Jira you need to download the add on and install in with a trial key. I am not the Jira admin and would prefer to not mess with a currently live Jira system to trial an add-on until I am fairly certain it is my best choice so I have not had a chance to trial this yet.
The only other option I have looked at seriously is PractiTest, which is a more standalone option that integrates to Jira by linking back to issues numbers. It is pretty straight forward and a free trial is available (though obviously not linked to Jira) and I dont mind it but I also dont love the layout, it just seems tedious to me.
So aside from my original comment for recommendations I would appreciate any feedback or experiences with these two options specifically.
As far as add-ons, most I've seen are pretty horrible.
For Zepher, their Jira add on IMO is just as bad as the Jira TCM hack. From what I'm seen, it's better to get a dedicated TCM that has 2 way integration with Jira. You manage your test cases in a separate app. But the workflow you get will be more condusive to testing vs request tickets.
One of the biggest issues I saw when evaluating the different Jira add ons is they are not good at reorganizing large amounts of test cases. Where in some of the many TCM solutions out there you can drag and drop and reorder suites. You simply can't do that inside many of these Jira add-ons easily. This makes it very time consuming to create and reorganize tests quickly.
Thank you for the feedback. I will be having a lot of test cases (eventually) to sort through and order, so if that is something not handled well by direct Jira add-ons then I will keep that in mind.
Originally Posted by dlai
In looking for a dedicated TCM solution that integrates with Jira, what kind of integration should I look for that would be useful? One benefit I have liked so far of the Zephyr direct add-on is the charts, the dashboard integration, version association, sprint association etc that it has. It seems like most outside products that intergrate at most only associate test case number with bug/issue number. Is there software out there that perhaps does more than this? Do you have any specific recommendations?
I mention 2-way integration. Of course all the vendors advertise just "integration" and try to hide all the details. When I say 2-way integration, I mean both
Originally Posted by Resputan
1) Push - Being able to post new bugs directly from the TCM. So when you fail a test, there's a button to automatically file a bug that's prepopulated with the test steps to quickly, along with a link back to the TestCase.
2) Pull - The TCM is able to pull information from the bug database. So when you view a test case or test result. You can see at a glance all the bugs associated with that test case and result.
Different tools will have varying levels of push and pull, so hard to compare by descriptions alone.
I think Zephyr stand alone is much better than the Zephyr Jira add-on. The stand alone has 2 way integration and is much better organization and reporting features.
Currently I'm using test rail. Main reason for this is it's one of the cheaper SAAS services out there and we don't have an IT staff to maintain all our internal systems. We're also using the SAAS version of Jira that doesn't allow us to install custom add-ons. It's push integration is pretty good, however the pull integration is pretty bad. But I think it has just enough we can at least see the defects from the test result instance that it was pushed from.
Zephyr Enterprise - Real-Time Test Management
Originally Posted by dlai
First and foremost, off the bat, I do want to mention that I work for Zephyr. In terms of what dlai stated and attributing to the entire conversation that I have read here, Zephyr does have a couple of products - one, as mentioned earlier in the thread is Zephyr for JIRA, which is the add-on plugin to bring testing capabilities to be had directly within the JIRA instance. Our other product, Zephyr Enterprise Edition is our stand-alone test management solution that offers a more comprehensive testing platform with a strong, two-way integration with JIRA. From Zephyr Enterprise, you are able to pull in requirements from JIRA using JQL or saved filters on the JIRA side, map these to the test cases that you have built out on the Zephyr side, and then be able to push bugs out from the TCM. Alternatively, when filing a bug, you can also search through existing defects on the JIRA side and link them with test cases.
That push-and-pull mechanism works kicks in with our Defect Tracking tool in Zephyr Enterprise/Community that allows you to filter through JIRA and be able to see JIRA projects - filtering by status, component, and so forth, to view issues on the JIRA side directly from within Zephyr. You are able to see which defects are associated with which defects - and furthermore, modify or resolve those JIRA issues directly from within Zephyr. It is, without a doubt, we have the tightest two-way integration with JIRA out there.
For those that do want to see how Zephyr Enterprise works and performs before purchasing, we do have "Zephyr Community Edition" which is our free ten-user platform that can be had. The Community Edition serves as a great test management tool for smaller teams that only have one project and under ten users as well as for those that want to use the tool to evaluate the Zephyr Enterprise solution, which has no such project, user, customization, or reporting limitations.
Please view our website for additional information in regards to the product:
Flexible Test Management Software | Zephyr
Also, we do have videos in regards to specific feature sets and how they function:
Watch, learn and enjoy test management videos
Thank you David for that insight. I have since been trialing test rail and am really enjoying. After spending some more time on Zephyr for Jira I did realize what you said about it being limited in its test case organization and structure, especially for more complicated or larger projects. It would be great for a smaller or simple project but would quickly get unwieldy. Zephyr support themselves admitted this during a call I had with them and suggested their enterprise edition (was likely half upselling and half serious) for a more complete test case tracking and managing solution. However, that is much more expensive.
Since then while trying out test rail I have really enjoyed where they have taken it recently (I reviewed it a few years ago at another job and it was a little rough around the edges from what I recall). Everything is very customize-able and organization is intuitive and robust. Cost, as far as stand along solutions go, is very reasonable. It appears so far that this would be a great long term solution to grow in to that interacts, as you said, with JIRA just enough to be useful but doesnt force it so much as to cause hindrance to itself.
That all said, I am still in the trial phase of this so if anyone has any suggestions similar to Zephyr and Test Rail that integrate with Jira in some capacity I would love to hear them so I can try them and will provide more feedback here accordingly as well.
Just to add our experience for the discussion - we've been using PractiTest for a while with the JIRA integration:
The JIRA integration can be better, but it's good enough for what we need (and they do add stuff to it all the time) - they do integrate bugs from PractiTest to JIRA, and keep the status updated in the PractiTest reports.
We use their custom fields and customise their dashboard to have all the charts with the version and sprints.
Regarding the organisation and structure, PractiTest has filters to both test cases and Test suites (they call it TestSets) which really makes it easy to organise our tests (we have couple of thousands) and reuse them. Other than that it's pretty plain and simple, which we kind of like.
They don't support time tracking, which is a minus for us, but the overall experience (adding to that their fast support replies) is more than good.
This is probably too late, but I highly recommend PractiTest - SaaS Test Management Tool and QA Management Tools - PractiTest
Originally Posted by Resputan
In my own search for a Test Management tool that works well with JIRA (my favorite bug tracking system) I have found PractiTest to be above the rest in terms of ease of use and although not the one with the most functionality it ticks all the right boxes... Their support is also top notch, and pricing is very affordable. Their pricing model actually makes it a very scalable solution where you can easily add or remove active users depending on your testing needs and pay each month accordingly.
Let me know if there is anything more you (or anyone else) would like to know.
I am very eager to know which tool you have selected for your project finally. I still could not decide on practitest/zephyr/testrail. Your answer will help me a lot.