| || |
I work for a department where we conduct Load and Stress Testing on a number of applications. Given that our Pre Production does'nt match up to what is in Production and the figures that we do have are somewhat not real. Is there somewhere where I can obtain some sort of a scaling factor. there is no way we can replicate production, there's no money this year. Help!!!!
Re: Scaling Factor
This is a widespread problem. Unfortunately there is not an easy answer to the general problem except possibly through voodoo and superstition. Within narrow contexts there tend to be better answers. I suggest you look at Weyuker's PNL vector.
The scalability ratio or factor is the crude ratio of the equipment and facilities available in the test lab vs. the live environment. Conventional wisdom (which may be wrong) says that if this ratio is very low (say 1 to 2 or 1 to 3), then extrapolation is possible -- extend trend lines beyond what is measurable in the lab, using curve-fitting. Often though because of budget limitations the ratio is way higher -- 1 to 10, 1 to 100 or worse.
Re: Scaling Factor
I wish I had something to add, but all I can really do for you is confirm what Ross said. I like to call that type of prediction "Black Magic" (Is that somehow related to voodoo?) [img]images/icons/wink.gif[/img]
If you have very accurate mathematical models (SPE type models) that you have verified under real-world load and you make certain assumptions about production being configured correctly, isolated environments, phases of the moon, etc - it CAN be done. Though I have only ever met 2 people who were smart enough to really pull it off with accuracy - and they started their models before the platform was even decided, no code had been written, no architecture planned, etc - not at the end of the project.
The fact of the matter is, if you're not a Computer Engineer with a PhD in math, an experienced Performance Engineer (who is a bit of a skeptic) will give you AT LEAST as accurate of a prediction as any published scaling factor will.
Trust your gut, document the risk, publicize the assumptions, cross your fingers and monitor EVERYTHING for the first month in production.
At least that is my recommendation - particularly if you are out of $.