| || |
Test case tracking.
I see a lot of posts regarding bug databases and defect tracking software. I am looking for something along a slightly different grain.
What we want to do is to formulate "mandatory" test cases, and have them be "opened" upon each new release candidate of a codebase. When a test-case is open, it is up to a developer to run each test case (either using an automated method, which is developed along with the test case, or manually), report any problems, and if no problems occur, "close" the test case. The software is considered not to be in a releasable state until all "mandatory" test cases for that version are closed (open test cases indicate regressions) or invalidated (if they no longer apply). Attached to each test case is a versioned history, so that regressions can be easily evaluated.
We would also like to be able to add "supplementary" test cases to the "mandatory" test-cases. A supplementary test case could be performed by anyone at any time upon any version of the code. The difference is that repeated performances of these "supplementary" cases are not required in order to complete a release cycle. They are simply extra information that helps the developers gauge how well their software holds up in the real world where they haven't predicted all possible conditions. At a later time, a supplementary test case could be made mandatory.
As you see, this is a different idea from a traditional bug tracking database. We also would like to track "success" reports as much as "problem" reports. Tracking success reports, especially "supplementary" ones, will help us with users who are using the software in an uncommon configuration and are willing to enter a report even when things are working fine.
I envision the "supplementary" type of test cases tracking problem reports more than success reports, while the "mandatory" test cases should be tracking mostly successes so that the software is usually in a releasable state.
We are a small software development group, not in need of any fancy commercial tools, and without a separate QA group, so the tools will be used directly by developers. If there are open-source projects that do something like this, it would be perfect. Bugzilla doesn't quite fit the bill.
I saw another question in the archives sort of along this line:
test management tool
Is a "test management tool" the right term to describe what we are looking for?
Any suggestions appreciated. thanks!
Re: Test case tracking.
You are correct, what you are looking for is more a test management tool.
The mandatory test cases are typically called regression scenarios. It are the scenarios that must be executed each time to see if changes in the system have not affected the unchanged parts of the system.
As for a tool, you can evaluate some commercial tools like TestDirector of Mercury.
On the other hand, Excel sheets might do the trick. Creating a column for each new release and indicating the test cases as open. Having a column for each new release will allow you to track the history of the test results.