Open Topic: Perf Test & Control Factors
This topic is designed to provoke curiosity.
The process for building an effective test is a very old one. The elements are well known: A Hypothesis, Steps to Reproduce, Expected Results, initial conditions and a control factor. The challenge we have in performance testing is that last element is often forgotten about. The nature of a control factor is to allow us to understand the quality of the test/sample and discard those which are problematic. It could be control data which under all conditions responds in a certain way, a control application run alongside the application under test to determine if our core app is having issues, or even a control generator to measure the effectiveness of our load generation. Ultimately this boils down to what is uncontrolled in your test environment: The network, the load generators (as in the case of VM based generators), the test servers (in use by others), etc… ideally we want our control factors to run parallel to the items which are uncontrolled in our test environment to provide a measure of how much influence these uncontrolled elements are having on our tests.
In my case I very commonly make use of a control generator, hardware matched to the rest of the generator population and lightly loaded with only a single virtual user of each type. Consider the graph located at
This graph contains two control elements, one a network sample, generated using a third party tool to reproduce the network data flows of a business process, and a control generator which is producing the sample set beginning with “Control_”. This example clearly illustrates an overloaded generator component influencing the response times collected for if the application was at fault both the control group and the global group would be degrading at a similar rate. The other graphs in this same location illustrate other conditions involving the dual control elements.
The ultimate goal of adding a control factor is to improve the integrity of test results & to make your tests more defensible so you can answer the question from a developer where you have just called his baby ugly, “You know, that’s a great question, let me tell you how we measured the impact of that factor on our test results….” As a process based profession we have ‘challenges’ with many of our participants being poor testers before they even pick up a performance test tool. More often than not this leads to low integrity results which are poor predictors in production. A control factor is but one step in improving your test integrity.
Replace ineffective offshore contracts, LoadRunnerByTheHour
. Starting @ $19.95/hr USD.
Put us to the test, skilled expertise is less expensive than you might imagine.
Twitter: @LoadRunnerBTH @PerfBytes