start PerfTest if the AUT will change?
I'm just project managing that perftest project and still learning so I'm looking forward to get your expertise on this matter.
We are about to start doing some performance testing on an application that will be new on our app landscape. That application feeds heavily into our existing ERP and vice versa. Both that app, as well as the ERP are under test.
Performance objectives are as follows:
- protect the current system performance for the business units that already use the existing ERP.
- ensure that the business unit that comes on board with that new application will not suffer of performance issues.
- if still possible, do some more testing to check scaleability of the new application; as well as the existing ERP.
We will start doing some performance tests on the ERP only, no issue there.
The new application is still under intense functional test (FT is delaying in fact...) and the functional team has told us that some UI fields will change.
Those fields themselves are not used in our performance test scripts and thus from a pure script perspective, we are fine.
However, our performance lead pointed out that the underlying code might have impact on the actual sytsem performance results. Fair point.
In the perfect world, PT should not start until the code is perfectly fine. My project is not happening in the perfect world and we have 2 choices:
- wait until the code of the AUT is deemed as stable. Preferred option of my lead. That will cause a delay of at least 3 weeks; and we can't afford that. We already have to be inventive to stick to the overall implementation project timelines; and to stay within budget as far as PT is concerned.
- start testing with the code that we have at hand so that we can at least filter out (and fix) major performance issues already as they would surface irregardless of relatively minor code changes. We'd have to live with some more re-work (retrofit perftest scripts (again) to reflect latest AUT functionality and re-execute some performance tests once done.
Can you confirm that the second option is the better option to go? The amount of rework and test re-execution is currently assessed to be a week; so still less then a full delay of 3 weeks. Also, having executed some perf tests already, we will already have a feeling on how well the application is performing instead of staying in the dark.
But again... In this case, I'm just a PM who is new to the PT world...
Thanks a lot in advance!
Re: start PerfTest if the AUT will change?
This is going to depend a lot on the priorities of the organization and the project.
If this is a critical, high-visibility project and minimization of risk is important, then you cannot test too early. Performance testing should start as soon as there is something (anything) to test. A great many performance problems are related to the architecture and configuration (as opposed to the code) - these will frequently turn up in early testing. Assuming, of course, that the test environment is identical to the target production environment.
Early in the project, the performance testing might be "light" - by which I mean it may not try to cover as much of the functionality as the performance testing should cover in later stages. Some of our clients call this "sanity" testing - as in: "We'd have to be insane to not test this system at least a little!"
If the project is low-priority and can live with the risk of delays late in the project, then waiting to test performance until the system is "done" is the norm - though still not considered best practice.