Beta Testing Processes
We are looking for ways to improve our beta test process. We define beta testing as "A customer validating the functional and non-functional requirements of a system. This includes techniques for using the contract, the statement of work, the software requirements specification, and/or the request for proposal to ensure that the delivered system meets all of the requirements. Formal testing conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system."
Our software is distributed to a specific industry (e.g. healthcare, etc.)and could not be used by the general public. I am interested in knowing what other organizations do for beta testing on new products and updates.
For Example: What is your process and policies for beta site selection and test management?
How do you incent individuals to be beta sites?
What are important measures for beta success?
What functional area is responsible for beta site selection and test management?
What functional areas are involved in beta testing and what role do they play?
Re: Beta Testing Processes
I run the beta testing programme at the company I work for. I am responsible for automated testing and alpha and beta testing, and anything else in the 'quality' and testing area the company wants to throw at me. I'm actually part of the 'Product Group' who develop, test and document the application.
Our product is also industry specific (student records for universities and colleges) so we can only get beta testers from the customer base. We have a release every 6 months so beta testing runs every 6 months. About 2-3 months into the development cycle we invite the customers to volunteer for beta testing.
We have tried invites to specific customers, a general invitation and the occasional bit of 'arm twisting'! We have a customer base of over 100 institutions but it can still be hard to find volunteers. They are offered free consultancy days in return for doing the testing, and of course they get to see new and enhanced features early. Beta testing is a big 'ask' as it can eat resources and can come at a difficult time of year when other work has to take priority.
The beta testers send in reports directly to me and I measure success, somewhat subjectively, by saying if I have several reports from an institution then they are testing the product. This may sound obvious, but I did have one tester who didn't start testing until the deadline date that we had given them. It was most frustrating! It can be tricky to measure 'success' because you not only want to hear about bugs but also about successful tests where everything went smoothly. Of course, the tendency is for the testers not to report back if everything is going OK!
To help we do send out some test cases and we ask them to return the results. But this is not always helpful as this means we are imposing a process on the testers when they might normally do it a different way. I do encourage testers to base their tests on their own processes.
We don't ask the institutions to test 'everything'. The product is massive and each customer uses different areas. We ask them to volunteer for areas they are interested in, and try to get coverage by having about 5 different institutions in each programme. This can be a bit of a problem as sometimes institutions will drop out of the programme unexpectedly.
I filter all the reports from the testers and any with problems are taken to the different product teams. Some product areas I know about myself so I will have a look at some of the problems because sometimes they are just misunderstandings. Testing is about the documentation as well, though, so if they don't understand what the manual says then that can be a 'bug' too! Otherwise I leave it to the product teams to interpret the results. If the process with the bug is very involved I leave it to the product team to talk to the tester but if it is a small thing I will do this, so that the teams can get on with putting the final touches to the release. If fixes are needed we try to supply them before the beta testing period ends (it is only about 3 weeks) so that there can be some retesting.
Each problem found is listed as a known issue and, if not resolved, will be published (with any other issues found internally) when the product is released. If there is a 'big' problem that would impact on the release it would be escalated to the directors to see if the release should be delayed (this has not happened in the 2 years I've been doing this).
I hope this helps