Application Under Test Runs Faster with RFT
We run performance tests on an application; a certain operation takes 25 seconds when executed manually. If however, Rational Functional Tester (RFT) is used to perform the very same steps the application under test (AUT) runs faster and completes the operation in 19 seconds!
Now, I have already verified a few "explanations" to this situation and I can say with strong confidence that the above observation is correct:
- The same steps are executed both manually and with RFT
- I made sure that the timings provided by RFT are correct by following them by my stopwatch
- I want to emphasize that <u>with</u> RFT the AUT is <u>faster</u>--so it's definitely not the case of RFT taking away resources
I set up breakpoints in the code and managed to narrow the change in the behavior of the AUT down to one code line:
1. If RFT runs up to a point where it has not interacted with the AUT--just opens it and waits for a dialog to appear then if I take over and manually continue the steps I get results identical to the manual testing, i.e. 25 seconds.
2. If RFT runs one code line further and clicks on a dialog button and then I carry on with the steps manually the result will be 19 seconds.
Anybody has any idea why the performance changes and how to eliminate this? I mean this renders performance testing useless with RFT at least as to absolute values.