Merits/Demerits of Bebugging /Error Seeding
Just curious about this Error Seeding / Bebugging [Boris Beizer's book] technique.
If you had to assess the effectiveness of testing, would you:
1. Try error seeding ?
2. Rather rely on phase-to-phase defect migration as a statistic ?
Has anyone tried error seeding ? If you did, I'd like to know what your environment was - how did you run it. What were the people issues that came up ?
[Speed, quality, price - pick any two !]
Re: Merits/Demerits of Bebugging /Error Seeding
I did try it, once, before reading Boris' book.
The developers thought this was good fun, they saw it as a game where they could catch the testers out - which was OK with me. There were a number of problems which stopped me ever doing it again (or wanting to).
- The developers were not actually very good at putting bugs in (intentionally, they were very good at doing it unintentionally!). Most were very simplistic and easily found. When getting them to think a bit harder about the bugs, they put in bugs that System Test would not expect to find a high proportion of them, i.e. bugs that should/would have been found in Unit test, e.g. off by one in internal loops, non-initialisation of variables, etc. So the problem was that the developers were thinking of bugs that would catch out a fellow developer, rather than bugs that would catch out a Systems Tester.
- My testers were not too pleased with the idea and, I admit, I had to do the management act of saying "Trust me, you'll see the benefit" (Oh, how those words came back to haunt me ). Their attitude was that there were enough bugs already without having more put in, despite the fact that we took a stable release to do the bug seeding. They also saw it as a test of their performance, no matter how hard I tried to persuade them it was not, that I was trying to measure the effectiveness of our techniques.
- What happened was that the testers, of course, knew that there were a number of bugs that had been put in (although not how many), guessed, correctly, the type of bugs that the developers would put in and tested to find them. Yes, they missed a few, but not many and they didn't find a single unintentional bug.
Overall conclusion. It didn't tell me anything, except that developers were bad at designing bugs and my testers were good at second guessing, but I knew the latter anyway. It cost a bunch of time that would have been better spent on bug avoidance rather than bug seeding. It cost me some money in rebuilding the bridges with my testers down the pub (and every new idea I had for some time after that was met with "Oh, yeah, remember the bug seeding idea?"). So, for me - never again.
One thing we did avoid was the possibility of some seeded bugs getting into live, which was by taking a separate copy of the code and throwing it away afterwards.
The idea sounds good, but after reading Boris' book, looking at my own experiencea and sitting down and rationalising myself, I can't see how it would ever work, practically.
Re: Merits/Demerits of Bebugging /Error Seeding
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Has anyone tried error seeding ?<HR></BLOCKQUOTE>
I have done various types of fault injection aspects of testing. In general, I have found it be more or less useful depending upon how it was utilized and to what degree.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>...how did you run it.<HR></BLOCKQUOTE>
Well, first off, I do not think random error seeding is good, so I try to shy away from that. I think it has to be directed error seeding because the former introduces too much uncertainty. The problem is that random seeding is generally done to remove human bias and, obviously, directed seeding can bring that bias right back in.
You can, however, use directed seeding to gather various types of metrics. For example, assume TSD is Total Seed Detection (which is your seeded and natural faults). Then you take the number of faults you actually seeded (SN), the number you actually detected (SD) and then you can calculate your total unseeded defects.
Consider a case where TSD = 100, SN = 25, and SD = 10. Then you have:
((100-10)*25)/10 = 225
This is a pretty common metric but you get the idea. But I think you have to go beyond that widely-used metric.
To that end, I also set up confidence intervals within the context of the error seeding. This can get a little subjective unless you define the metrics by which seeded errors are going to be determined. The most common metric is obviously:
(Detected Seeded Errors)/(Total Seeded Errors) = (Detected NonSeeded Errors)/(Total NonSeeded Errors)
Also, if you have two or more people injecting the faults and depending on how you have others find them, I found it is best to introduce an overlap variable as part of the two systems of fault injection.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>What were the people issues that came up ?<HR></BLOCKQUOTE>
Well, one problem was that bias that I mentioned earlier. You run into a dilemma sometimes because the person most suited to inject the faults is also the person most likely to be biased in looking at where to put faults. In other words, you lose some of the serendipitous aspects (if you will) of the process.
Another problem I had was that the people working the process kept falling into the trap of thinking that error seeding is the exact same thing as mutation analysis (as even some authors of books have done). They are, technically, not the same thing at all.
All in all, you will probably find a lot of places that tell you how good the process is and it certainly can be. For a different take on this, you might consider the following paper: