At my company, we are actively exploring ways to expedite our functional upgrade testing.
Currently, my company supports close to 30 different build releases spread out over five major versions, all with varying degrees of differences within each
Data mapping tables
When customers upgrade, they normally go straight from one version to the next (example: they can go from version 2.04 to version 5.06 directly)
To test these upgrades, we currently, use a two pronged approach to validate upgrades:
Use an automated Tool to validate the db schema and the contents of specific mapping tables
Employ a series of manual tests to validate functional, performance, and conversion issues were not introduced (based upon each client’s current application settings).
This allows us to test in a manner consistent with the client environment to limit potential issues, but this has proven to be a time and labor intensive process; typically taking an average of five days to complete.
To expedite the process, we’re talking about forcing users to come in through common migration points. This way, I maintain static scripts from latest/greatest Version 2 to latest/greatest Version 3; latest/greatest V3 to V4, and so on so that the only variable would be my starting point (i.e. going from V2.04 to latest/greatest V 2; from there, the user can go to any version s/he would like).
However, I get the feeling that there are some serious “gotchas” out there that we haven’t considered, especially as it relates to data conversion issues (given how highly configurable our application is).
What pitfalls do you see in this model; and do you have any suggestions I could implement to limit these pitfalls?
We have the same issues as you have and our approach was exactly as you describe your "old" one. The only difference is that we see it as the best possible choice because most of the issues that appear in product are related to how the specific customer are using the DB schema for custom needs. The approach change your talk about will not help us anyhow.
Perhaps your application (and it’s usage) is somehow different to our so that your suggestion my work better for you. However you should also think about your customer reaction on this – I’m afraid our customer would blame as for lack of professionalism should we suppose such and upgrade mechanism.
?:the art of a constructive conflict perceived as a destructive diagnose.