I have a question as to how to update the test bed of how many test cases passed and how many failed:
Lets say my test bed has 1000 test cases and I finished execution of 500 test cases out of which 400 passed and 100 failed. If I opened 10 defects for the 100 failed test cases and I am continuing execution. If the next day, the development team fixes 5 of the defects which I confirm to be closed, should I go back and update the status of the failed test cases (associated with the 5 defects that are closed) and mark them as passed? Or do we leave them as failed since they did fail and just close the defects?
Can you please provide feedback as to how you all do it?
Do you track the composite results or do you then rerun the entire bed of tests to confirm that they all pass together? We run all of our tests at one time and end up with a composite of results, so the last result of the test is what is shown - it may have failed before but if the last run says Pass its Passed.
I don't claim its the best system, but we keep a careful eye on it and I do record the history of the results so I can go back and check. I'm working on a Test Results database to keep a record of all the tests we run and their results so I can historically track trends in our tests and their success rates overall.
Nothing learns better than experience.
"So as I struggle with this issue I am confronted with the reality that noting is perfect."
We keep track of the test cases we do have Current and intial status(pass, fail, block) where we will be failing the test case if it is failed and log the defect, after the fix is done we mark it as pass if it as pass and in comments column we do metion the defect id and when it was retesee, we also have another column intial status, where we track of the previous history. say
Build 1.0 is released, and up on executing the test case if the test case is fail we mark it as fail in both intial and current status, again when build 1.1 is released then we test the fix and mark intail status as fail and current status as pass
All the metrics will be reflected in another worksheet reagrding the statuses