Where I work the majority of the testing is done manually. After spending some time working with a list of conditions and trying to get a good combination we started using a test matrix.
In case my idea of a test matrix is different from the norm, this was a spreadsheet with a list of conditions to be tested on the left and a list of tests at the top. For each condition tested there would be an 'x' and the second row would count up all the 'x' marks and show how many times that condition had been tested.
Just to illustrate, here's a rough example:
I noticed some interesting things doing this:
1. As I'd already started writing the manual tests I first put in what I'd already written. Once I'd finished I was quite surprised to see how many conditions weren't covered even once. These conditions were mostly where several conditions needed to be combined together.
2. The number of tests needed to be written was around 100 and the conditions about 80. So this was 8,000 potential options for testing or not testing. I found it was very easy to make a mistake and not be able to spot it. To combat this I made some extra rows on the spreadsheet which basically checked to see if I'd made errors.
As an example if the testing was for a school enrollment system (this isn't what I'm actually working on). I may put an 'x' in the condition to enroll a pupil in a class that's full but have an outcome where enrollment is successful. Obviously this is NOT what should happen. Having an extra row at the bottom to see if an 'x' is found in both of these condition/outcome rows allowed me to quickly see whether I'd made this error in any of the 100 tests for this condition.
This is an idea I thought of myself and I found it quite useful. For more experienced testers, do you have a way of dealing with this type of problem?
3. The outcomes of the tests were mixed in with the conditions. It might have been better to separate them but the outcomes actually became input data for the same test. (e.g. enrolling a student in a certain school might involve some further steps of tuition fees being paid but not for other schools).
4. Another area where errors were more likely were the outcomes, see the above example. I eventually decided to reduce the errors to zero by instead of manually putting an 'x' in the relevant row, to have some kind of formula work out what the outcome should be and do it for me. As I pasted this formula across I was quite surprised to see quite a few blank cells change to 'x' and visa versa where I'd made a mistake. I feel quite confident that errors have been greatly reduced. Doing this obviously improved the amount of conditions/outcomes that were being covered where they otherwise wouldn't have been covered because of my error.
5. After finishing the test matrix I was left with the task of converting this into 100 separate manual test plans. I've found it quite a long winded task to do this even for a small number of scripts. I had the idea of writing a small VBA script which would basically work out what the tests should be based on which cells have an 'x' in them. It seems such an obvious idea that I'm wondering if someone has already thought of this and there's a good way of doing this?
I'd appreciate any comments on the above from people who have already been where I am now.