weekly merge process - good or bad?
I need some advice. I will give you a little history first before jumping right into our current process and where I see faults. First our business model is an online service provider. One code base, very complex system.. We must be available 24/7.
We went through some downsizing over the past couple of years and now are left with a very small QA department while development is about 10 people. When QA had more people we did quarterly releases and since the layoffs we are doing weekly productions merges; we couldn't manage a full set of regression tests with just 1 tester. The merges include enhancements, projects, bug fixes, etc. The process has worked quite well except I feel that sometimes we don't get the coverage/support we need on projects. The QA department is now 2 people and contract when necessary. We have been doing things this way for well over a year, and burn out is near. I feel managing the process is chaos….I oversee the entire process from scheduling code for the week, to getting the developers to give us code, to testing, filing change control request, getting approval from committee, to coordinating with production support who will be performing the merge, etc. The entire process is very formal and several checks and balances in place due to auditors, etc. This occurs weekly….
- weekly merge schedule is constantly changing and new priorities being added daily, usually consists of 1-2 enhancements and 4-6 bugs.
- cvs challenges. We have 2 branches we currently work off of. 1 being the production branch and 1 being the development branch. Development branch has moved further and further away from the production branch that development has to create patches off the prod branch then patch the development branch and correct any discrepancies. Major headache for dev and us since we can’t test off the development branch because it is so different. We could but would be comparing apples to oranges.
- managing code dependencies, this bug fix touches a file that another bug fix touches so they both must go in together...same in the reverse order.
- no time for planning. review the bugs/enhancements and their documentation, test, approve and merge – then the week starts all over again.
- development has moving deadlines unlike QA
- director of dev likes the weekly merge process. Allows for flexibility…(no hard dates)
- product management also likes this process since they can change priorities quickly.
- managing customer expectations – customers can requests changes or complain enough about a non critical bug that it gets priority and quicker turn around time.
- poor quality, not necessary with the code but with other areas of the process – documentation, requirements, change management request, production code branch is constantly changing causes instability
- larger projects require more time but unable to take that time since the merge schedule will be affected and ultimately projects in line will be pushed
- last minute bug fixes before merge.
- project completion – majority of the time projects will go in to production in pieces, this creates more work for both Dev and QA
What are your thoughts on the process? Good or Bad?
How do I sell the Director of Development and my manager that this is not a good process and the steps to change it will be ?, while keeping bug fixes under a reasonable response time? The migration to another process will have to be slowly implemented…
I am desperate...thanks in advance.
Re: weekly merge process - good or bad?
You reaised many issues in your post. Assuming it will be hard to convince management that the entire process should be revisited (although it sure seems that way), I would start with identifying the core of the problem: what of all the issues you raised (or some underlying issue) do you feel is the most essential one? If only one thing could change - what would you want it to be?
By focusing on one issue, there is more chance that management will support a change.
I see many problems in what you describe. For example:
Bug dependencies (are components tested in isolation one from another?), project planning (ever changing priorities and tasks), under stuffed QC, proper development env etc...