automation tool during perf analysis
provide your thoughts how can automation tool help in performance analysis phase apart from iterating the scenarios. agreed that some tools do provide tuning console products, but like to have more thoughts and informatiuon about how automation tool help in analysis phase.
Re: automation tool during perf analysis
During our automated regression tests we also collect timing and resource usage statistics and compare these against base line values to see if any performance bug has creapt into the software. I consider this a necessary part of our regression testing, as we have had some critical performance bugs in the past.
The way this works is that at the start of the test we record the time, process memory usage, GDI and USER resource usage, and handle count. At the end of this we capture the same information and subtract to get the differences, which we compare against stored differences in the baseline. We typically expect execution times to be plus or minus up to 10%, with much smaller changes on resources. If we hit an issue we repeat the tests under either a performance profiler or execution profiler.
Using the profilers is pretty time consuming, so I only usually drag them out when I get an indication of a problem.
I guess we could put in a lot more resource and timing checkpoints, but at the moment its one per testcase.