How to: Proper Testing vs Continuous (messy!) Delivery
I am currently working with an online game employer as the role as QA Manager, since 2 months now. I have some questions regarding the development/deployment processes here. First brief about the company:
- Company existed for 2 years.
- Working with online gaming, only one product.
- About 10 developers at the office.
- No project managers.
- 1 week iterations (planning meeting once a week).
- Continous Integration.
- All code must be reviewed by two developers and sent to Github.
- Developers code and dev test on own machines.
- Code is sent to staging environment (copy of production env) for testing (most of the times). Sometimes the code is deployed directly to producton environment without testing.
- Noone writes release notes. Noone has clear control of who's deployed what. A request is that they want me to write release notes after testing.
- Old bugs which are corrected and re-tested without any issues, often appear again.
- Areas which before have been tested and passed testing, starts showing bugs.
- Several integrations are not working on staging environment. F.eg. some payment and game functionality from suppliers, can only be tested on production. This have caused problems so that users/players cannot deposit money f.eg..
I have been working as a test manager for ~8 years in total but I have only been working in projects with a clear start and end date. I have always been working in waterfall or scrum with code freeze dates, separate isolated days only for testing and so on. Therefore this situation where developers are working continously without hold-up for testing, is totally new to me.
So, they have asked me to QA the development process itself and to bring tests into development process. I have made some changes in our ticket/work task handling system (Trello) and inserted the action "system testing" to the development process. Sometimes tickets are placed in that list, for me to test, sometimes not. Most of the times functionality are deployed directly to production without testing, which often causes problems for the users/players. Often they have to revert to last version but noone has written release notes so they really dont know whats to be expected and whats included in the previous release. I was asked to improve the release process and to write release notes.
My questions here are:
1. Is there any good release process for developers who work with fast iterations and continous integration? How can I control release management without slowing down the process too much?
2. Since the developers deploy new code several times a day, I cannot have my "code freeze" so I can test everything isolated. How is the best way for me here to test properly, to be able to test functionality properly without blocking the development process and keep the developers on the chair rolling thumbs? My suggestion was to test continously together when the developers release new functionality and then have "my own regression testing" like once a week, where I test all functionality since last week, together, and with the most important other functionality already deployed (f.eg. checking it's still possible to register, login, etc.). Not sure if that works well.
3. Why are old bugs showing up again? This happens a lot. What could be the cause? I have asked the developers "nicely" but no reply other that they can overwrite each other's code easily. So if someone is working on something and checking out code, and another developer is doing the same, they might check in code at the same time and overwrite (sort of). Is there any way to prevent that?
4. Also, why are new bugs discovered in areas which are just marked as passed in testing? Maybe a very hard question to answer, and of course when you check in code, the other functions could be affected as well as not covering everything with tests and such. But do you have any ideas and/or suggestions?
5. Today many of the developers deploy directly to production environment. Is there any way I can stop this? Also as mentioned, some functionality can only be tested on production environment. Is there any way I can suggest something for the developers/managers to not test important payment functionality on other environment instead?
6. Which environments do you suggest? I use to speak for DTAP- development, test, acceptance testing (staging), production. Is that really needed?
Any other constructive tips and tricks are welcome.
Thank you in advance and best regards.
Originally Posted by AwesomeUserName