goals-base testing method
I am a big fan in goals-base testing, the problem is in my all qa experience (almost 7 yrs) I couldn’t so far put this method in practical, the reason is I never had a test machine simulate the actual production environment. Hence, I used my performance result (system resource utilization) to gain a sense of the actual performance, for example, if in my testing machine my application can handle 20 users without crashing, I can assume with production environment my application can handle around 50 users without crashing (if the testing environment three times bigger than my machine) and so on
There is no standard method I used to come up with this number, but it seems always work,
But to all goals-base testing fans, have you ever simulate your production environment and reached your targets through your testing? I am just wondering
Re: goals-base testing method
I was lucky enough to work for a major UK bank that took performance very seriously.
The production system consisted of 2 seperate sites each running 32 webservers, 4 authentication servers (Site server), 6 UNIX boxes (and Orcale DB's) and 3 Mainframe LPARS. Each of these legs was designed to run at 50% of full capacity, thus a major outage at one site meant the other site could cope with full volume by running at 100%.
The performance enviroment was a full replica of 1 leg plus a scaled down version of the alternate leg (used for failover testing at low volumes).
The scenarios run by us were full current production based models for existing applications, and performance model volumes for new applications. As you can imagine the expense involved meant nothing less than full production volumes were considered adequate for formal signoff to go live.
My biggest worry about using scaled-down environments and extrapolating results based thereon (I know this is the only way for a lot of companies) is that consider the following:
Production has 8 webservers and 1 DB server. Perf has 2 webservers and 1 DB server. Simple maths, if 1 DB server can support 8 webservers and runs at 80% capacity in prod, 2 web servers under 25% load of prod means DB server should run at 20% capacity.
In reality it doesn't work like that (based on my experience, others may find it does).
Lokk forward to peoples thoughts on this, Mark.