In general system testing is regarded by many to be at best 80odd % effective (many thinks its as low as 70%). This does not mean 20% of the bugs will be found after system testing, its more a statistical guess. The harder a bug is to find the less likely that it will be found at all.
There is no clear answer in my mind to your question, but I regard a happy user(s)/customer(s) sucessfully using the application as a sucess. Testing is merely a part of that products development.
I guess one approach would be setting (or quantifying) out the level of quality you want to achieve (i.e. up front) and then maybe you could state whether or not testing met that objective.
Statistics show that even very good testing generally catches only 90% of all bugs. This means that if your error rate in production is under 10%, your testing is effective. Please note that this is a generic industry number, and is not applicable to fields that involve human life...
The best testing organizations I've worked with, however, analyze every bug found in production to see what they missed and how they can avoid similar errors in the future.
Originally posted by qasleuth: Well let us not forget, bugs found in production is a failure of the entire team (if you want to term it as a failure). Team includes Dev and Test.
<font size="2" face="Verdana, Arial, Helvetica">Yes yes yes yes yes!
The team defines quality and defines the processes to try to achieve that quality - testing is just one of those processes.
If you leave quality up to test and test alone, it doesn't matter whether or not bugs slip through - I think you are failing as a project!
I guess the best summary would be to say that testing is at best flawed. Organisations must anticipate bugs being found after release and have systems in place to deal with it to minimise the possibility of it happening again in subsequant releases. I guess the only issue would be getting users/customers understanding this. People do seem very tolerant of Microsoft bugs BTW!
Thanks for reply,
Now we accept that 10 % bugs do slip in production but in our core Testing definition we state that all bugs will not be covered ,than why not effort of testing is Sucess if bugs encountered in production level is to accepted level considering Severity of bug???
There is no ultimate answer to your question. If you define the number and severity of post release bugs as some sort or pass fail criteria (acceptable level) "up front" (before development starts), then I guess you can gauge sucess.
BTW 10% bugs into production does not necessarily equate to 10% bugs found, its a statistical guess.
What this means is that if you found 900 bugs in system test, there could potentially be 100 undiscovered bugs after release. In practice most of these should be near impossible to detect and may remain so through the life of the product.
Well let us not forget, bugs found in production is a failure of the entire team (if you want to term it as a failure). Team includes Dev and Test.
And would also never forget Glenford Myers (not to quote exactly) - You wasted your time in test if you did not find issues (that i would term as failure)