The online community for software testing & quality assurance professionals
 
 
Calendar   Today's Topics
Sponsors:
Lost Password?

Home
BetaSoft
Blogs
Jobs
Training
News
Links
Downloads



Quality Engineering >> Defect Tracking

Pages: 1
SteveKay
Active Member


Reged: 01/28/05
Posts: 711
Loc: England
Using s-curve to predict defect numbers
      #344301 - 11/30/06 05:54 AM Attachment (303 downloads)

Does anyone have a view/experience on how realistic using the s-curve to predict defect numbers is?

I've just started a new 13 week phase of a project and decided to see if the s-curve graph can predict how many defects we're going to end up with by the end of the phase.

Our defect discovery rate didn't fit too well on the S-Curve in the last phase, but it was a bit of a slapdash analysis effort. (I've attached it for the curious) So I thought I'd try it again.

Has anyone had any success analysing defect discovery using s-curves? At the moment it's predicting we'll raise 14,300 defects in this phase, which seems a bit high given we only raised 164 in the last phase.

--------------------
Everywhere's within walking distance if you have enough time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
PeterNairn
Advanced Member


Reged: 06/22/01
Posts: 551
Loc: United Kingdom
Re: Using s-curve to predict defect numbers [Re: SteveKay]
      #345283 - 12/06/06 01:13 AM

Steve,
I do not use the S-curve so much as a predictor of what will happen, more as an input to prediction. It also, obviously, tells me what has happened. The simplistic view of the s-curve is that when you get to the top of the curve you can ship/stop testing and whilst this is far too simplistic, the overriding principle is correct.

At the risk of telling grandmothers how to suck eggs, testing goes in cycles, broadly, test preparation, testing of new and/or changed code, retesting of bug fixes, regression testing, stop testing. From my own experience, each of these cycles has their own “model” for bugs.

Test preparation raises a few bugs, usually not too many. This is usually a straight line on a graph.

Testing of new code raises lots of bugs, but not usually in a straight line. At the start of the cycle you will find all the show stoppers. Once these are cleared, you then have a rush of bugs as you hit the areas that you expect to find bugs and the rate then falls off as you hit the other areas. This is not necessarily the top of the s-curve, however, and that is where you can get false information. As bug fixes come in from the earlier testing, you then find the bugs that were masked by the earlier bugs and you get another rush of bugs. Also, what generally happens is that the areas that you expect to have a high number of bugs will (should) get tested first, so you will have a higher number of bugs raised here and then it will drop off as more stable parts of the system are tested. This can give you the false impression that you have neared the top of the s-curve.

Retesting of bug fixes raises fewer bugs and the picture to watch here is the number of bugs that are declared as fixed but are not as well as new bugs introduced by the fix. On your s-curve this can like a downturn in raise rate.

Regression Testing again should be fewer bugs unless something is really broken.

This means that one s-curve will not tell you anything. It has to be used in conjunction with other information, such as what part of the testing cycle are you in.

The only time I have found the S-curve useful on its own is when presenting to higher management, because they like graphs.

--------------------
See my blog http://www.sqablogs.com/PeteNairn


Post Extras: Print Post   Remind Me!   Notify Moderator  
lynneM - RIP
Moderator


Reged: 02/02/03
Posts: 3101
Loc: FL, USA
Re: Using s-curve to predict defect numbers [Re: PeterNairn]
      #345361 - 12/06/06 05:39 AM

I was interested in what you are trying to do and started to try and get some information on the "S" Curve and how it could be used. I only found sites referencing engineering etc. nothing for software. Can you provide me with a link or reference that I can use to do more research?

--------------------
Lynne

I have not failed. I've just found 10,000 ways that won't work" --Thomas Edison


Post Extras: Print Post   Remind Me!   Notify Moderator  
PeterNairn
Advanced Member


Reged: 06/22/01
Posts: 551
Loc: United Kingdom
Re: Using s-curve to predict defect numbers [Re: lynneM - RIP]
      #345462 - 12/06/06 09:32 AM

Lynne,
Try these links.

http://www.sqaforums.com/showflat.php?Cat=0&Number=301388&an=0&page=5

http://spinroot.com/gerard/pdf/paste01.pdf

Pete

--------------------
See my blog http://www.sqablogs.com/PeteNairn


Post Extras: Print Post   Remind Me!   Notify Moderator  
lynneM - RIP
Moderator


Reged: 02/02/03
Posts: 3101
Loc: FL, USA
Re: Using s-curve to predict defect numbers [Re: PeterNairn]
      #345463 - 12/06/06 09:33 AM

Thank you

--------------------
Lynne

I have not failed. I've just found 10,000 ways that won't work" --Thomas Edison


Post Extras: Print Post   Remind Me!   Notify Moderator  
SteveKay
Active Member


Reged: 01/28/05
Posts: 711
Loc: England
Re: Using s-curve to predict defect numbers [Re: lynneM - RIP]
      #345597 - 12/06/06 11:47 PM

Hi Lynne,
I don't remember where I first read about this but I found this pretty interesting as it relates to test management and gives some examples of interpreting the graphs: http://www.tisqa.org/speakers/ZBB-TISQA.ppt

It inspired me to do some quick graphs of our last phase and our current phase of the project to see whether it would tell me anything.

Our previous phase shows a faily linear pattern of finding defects, which doesn't match the S-curve but I'm not sure what that means or if it's relevant. I suspect not.

Thanks for the reply Pete, and the links.

The graph I posted contains all the defects raised against the previous 'phase'. Each phase involves some analysis (although not much), unit testing (but they don't report defects in the defect tracking system) system testing, re-testing and regression testing and finally 2 weeks of UAT. So each phase goes through a pretty typical lifecycle.

I think I might get more value out of the 'zero bug bounce' graph, which makes sense to me, have you heard of this too?

--------------------
Everywhere's within walking distance if you have enough time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
PeterNairn
Advanced Member


Reged: 06/22/01
Posts: 551
Loc: United Kingdom
Re: Using s-curve to predict defect numbers [Re: SteveKay]
      #345599 - 12/06/06 11:54 PM

Quote:


I think I might get more value out of the 'zero bug bounce' graph, which makes sense to me, have you heard of this too?




I have heard of it, but haven't used it and I am afraid I don't know anything about it.

--------------------
See my blog http://www.sqablogs.com/PeteNairn


Post Extras: Print Post   Remind Me!   Notify Moderator  
lynneM - RIP
Moderator


Reged: 02/02/03
Posts: 3101
Loc: FL, USA
Re: Using s-curve to predict defect numbers [Re: PeterNairn]
      #345720 - 12/07/06 06:11 AM

I have not heard of the Zero Bug bounce graph. I will look for info on it - if you have anything could you share where to find it also.

Thanks

--------------------
Lynne

I have not failed. I've just found 10,000 ways that won't work" --Thomas Edison


Post Extras: Print Post   Remind Me!   Notify Moderator  
SteveKay
Active Member


Reged: 01/28/05
Posts: 711
Loc: England
Re: Using s-curve to predict defect numbers [Re: lynneM - RIP]
      #345964 - 12/08/06 03:22 AM

Hi Lynne,
The first I heard about it was in the PPT in my last post.

It looks like it might be a term coined by Microsoft during VS2005 development. http://weblogs.asp.net/scottgu/archive/2004/10/23/246709.aspx

--------------------
Everywhere's within walking distance if you have enough time.


Post Extras: Print Post   Remind Me!   Notify Moderator  
PeterNairn
Advanced Member


Reged: 06/22/01
Posts: 551
Loc: United Kingdom
Re: Using s-curve to predict defect numbers [Re: SteveKay]
      #345984 - 12/08/06 04:52 AM Attachment (361 downloads)

I have done a bit of reading now on ZBB and fully understand what it is saying as I do this on a weekly basis, just didn't call it ZBB! I use this principle to predict when we will be at a zero bug bounce.

See attachment as this is my current "predictive curve" (which is what I call it!).

Notes:
1) The shape of the curve changes on a weekly basis as you plug in the actuals.
2) "IRP" is our term for "bugs"
3) The prediction has, so far, been quite accurate.
4) The algorithm used is based on predicted bug raise rate (from S-curve), predicted fix rate by development, based on an average bugs/day/developer and an average close rate bugs/day/tester. If you know how many developers and testers are working on the project, then you come up with the prediction.
5) As you can see, we aren't ready to ship yet!

--------------------
See my blog http://www.sqablogs.com/PeteNairn

Edited by peternairn (12/08/06 04:53 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1



Extra information
0 registered and 3 anonymous users are browsing this forum.

Moderator:  AJ, Daniel_S, swt88 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 9044

Rate this topic

Jump to

Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5