Last year I was working on a system that collects and manipulates data (details irrellevant). This system was run on agents laptops and they synchronised data with headoffice by running an 'end of day' upload/download program.
Anyway, over the course of a few weeks clients started to report strange data corruptions occurring. This defect was subsequently investigated thoroughly over the course of about two weeks by half a dozen testers and programmers alike. No success in reproducing the problem.
I was called in to have a go (having a reputation as a bit of a bughound!)
Anyway, after 10 mins of playing around I asked one of the programmers :
'Could it be that they are running the end of day program without first closing down front office?'
His reply was:
'No, they have been told not to do that.'
So what did I do?
Wait for it...
I ran the end of day without first closing the front office system.
Guess what happenned!
So anyway, although never formally established, the total cost of the defect for us and our customers and their customers ran into 10's of 000's of £.
Our solution was to make the end of day program force the front office to close.
This was ruled out for the initial implementation of the system because some day dream believers really thought that people wouldn't press the big red 'crash' button if you ask them nicely not too!