Understanding and grouping defects discovered by customers
I've recently been promoted to Test Lead and have just found out that one of the measurements used by the Test Manager is a measurement of how many bugs customers find vs how many bugs testers find.
I'd like to start gathering information on the defects that customers find, to be able to spot the areas where the test team can improve and ultimately lower the defects found by customers. I realise that customers will always find bugs, but the purpose of this exercise would be to filter through the defects found by customers and try to find a way of grouping these, so that the testing effort can be more focused in a specific direction.
From what I've heard so far, they're only looking at the number of bugs found by customers, what I'm interested in finding is if there is an area that is consistently proving problematic and how we can go about fixing that. I'm not just interested in identifying an area of the product that is problematic, but also identifying if there are flaws in our testing approach.
Can anyone please recommend some reading on how to go about this? I've read through a lot of the general stuff, I'm now looking for something more specific on how to turn the raw data (customer logged defects) into something more useful for the team. So far I haven't been able to find what I'm looking for here.
A lot depends on how your customer-logged defects are stored - but you should be able to transform this in any case.
Your first priority would be to group into modules, then sub-modules. That gives you an insight into areas of the application that have problems.
After that, I'd look at grouping the customer-logged defects by how close they are to core function: is this defect something that really should have been caught earlier, or is it an edge case that was picked up by the only person in the known universe who uses that particular feature? Or is it something that only happens if the customer's machine crashes halfway through an extensive database operation? (Yes, I've seen these reported as bugs by customers. I've also had customers report data corruption bugs caused by them killing processes because they didn't want to wait for the process to complete.)
If your team has a reasonably comprehensive automated regression and a good testing approach, you'll probably find that customer-reported issues cluster around the most recent features that haven't had time to build up comprehensive automated regression. The more mature your automation is, the less likely it is that it will be feasible to add a test to handle a customer-reported issue (the more bizarre edge cases are often not viable to automate).
Do a Pareto analysis taking the defects for previous two quarters and segregate these into respective groups like, 'Coding defects', design, requirements, environment, integration etc. Like this, one can easily get the ratio where most defects are originating and take the steps accordingly.
One practice that I use is to perform a Root Cause Analysis on the bugs found by customers. This is the RCA template that I use, which examines the actual root cause, plus how it escaped. After doing a few of these, you can take a look at the reasons that the problem escaped and see if you have any trends or common causes.
Doing this level of analysis does take some time investment. You can start with the most severe bugs first. For example, on one project our customers (telco) assigned a severity. We examined all of the "critical" defects first, then analyzed the "major" bugs. Another, I started with the bugs that drove a patch.
Yes, RCA techniques are good.