Analysis - Beyond Descriptive Statistics
Admittedly, I'm a relative newbee to performance engineering - 2 years.
As I began to conduct performance tests, I became uncomfortable with relying so heavily on the descriptive statistics (min, max, avg, std dev, and 90 percentile)provided by LoadRunner to provide the basis for assessing performance. Just recently, I formulated sample data sets that showed how similar descriptive statistics could be based on vastly different test data. It demonstrated to me that the descriptive statistics simply aren't enough.
So, in our performance analyzes, we have added the use of histograms. Initially, the data was copied into Excel, but it was a very laborious task. Eventually, I wrote SQL against the MDB file created by the LR analysis, I can very quickly and easily produce a histogram that provides 100 intervals in 1 second, 1/10th second, or 1/100th second. It also creates 20 intervals in 5 second, 1/2 second, and 1/20th second.
This has proven to be a very valuable analysis tool. For example, in one transaction, we uncovered that the average response time of 4 seconds was based on 90 percent of test cases with 1 second response time being averaged with 10 percent of test cases with 25 second response time. The histogram showed this 25 second "outlier" and we were able to split the results and more succinctly quantify behavior.
The deeper effort in the area of analysis began with my reading "Understanding Variation. The Key to Managing Chaos" by Donald J. Wheeler. I've been researching the use of Process Behavior Charts in our performance analysis. In particular, I'm trying to understand the validity of using these techniques and whether within-run and run-to-run variation analysis techniques has any merit in performance testing.
An area of interest for my organization is more than running a test and reporting the performance results. The organization is looking for answers, specifically, "based on performance test in the non-production environment, what is performance in the production environment going to be?". In other words, they are looking for me to predict the performance of an application. In order to do this, it is my belief that the within-run and run-to-run variation must be "in-control" as described in Wheeler's book. So, my resaearch in this area continues.
At the HP conference in Las Vegas last year, I raised this issue to various attendees and speakers. But, to no avail, I didn't get too much support for the notion, or that it was not a current area of concern.
Am I out in the weeds with this approach? Are there others like myself exploring/experimenting? Are the performance testing software suites lacking in the area of analysis? Are there other, more useful supplemental analysis tools? Is there information out there that I simply have not found? From the web searches I have performed, much of the work in predictability of application performance seems to be in the academic area, where testing assumptions are based on non-realistic assumptions.