| || |
Testing in Non-Production like Environment
Here is my situation. I am being tasked with performance testing an application. The business is wanting me to tell them how their application will perform when it reaches production. The business has also recently decided that although production web servers have 8 processors, we only need 4 and in some cases 2 processors on our servers. When we asked how to get production like numbers, we were told that we could mathmatically extrapolate what the performance would be with more processors.
I know that extrapolation like this is not going to be possible because every application performs differently so there will not be a magic number. (the business said to use 1.6 when going from 4 to 8 processors). Currently, we are planning on running our tests and showing that the results do not match up to a single "magic" number.
What I need help in finding is White Papers and Industry articles that address this issue. Any suggestions would also be helpful.
Re: Testing in Non-Production like Environment
Sounds like you have optimistic people who will not be doing the testing telling you how to do your job and then probably blaming you when things don't work well.
I for one have not heard of a single "magic number" that will allow you to scale from one set of machines to another. Not only do the applications under test behave differently when the number of processors are changed, what else can change? Behavior of the OS? Behavior of web server (number of clients)? Behavior of disk subsystem? So many variables. Also, What about the hardware itself? would imagine that a 4 proc Dell server might behave just a little differently than a 4 proc Compaq server which would be different that a 4 proc NoName server.
I would also imagine that the "extrapolation" from a 2 proc in any of the above lines, to a 4 proc in the same line would be a non-linear function. Heck, I would challenge your bosses to even define the function that correlates 2 proc vs 4 proc in the first place.
If your bosses want true performance numbers, you really have to be working on production class hardware. If they are too cheap, one alternative is to do the testing on the prod h/w before it goes live. This is one of the things that we do at my company.
Additionally, you may want to invest in some of the same prod h/w for your labs anyways, maybe not the same number of machines, but at least one of each part of your system, to enable some type of baseline performance numbers.
Extrapolation is not an exact science, but when its based on actual numbers, these guesses will at least be based on some tiny shread of reality.