Should you check the response you get in a performance test
I'm having a discussion with a guy at work who said "ask any performance testing expert you like" so I thought I'd ask here.
Basically we have a web based application where the user is presented with a list of datasets, they click on one and the next page shows the data table for that dataset.
My approach has always been that in the scripts I write I check that the data I get in the response is the one I requested. So if my script requests datasets "Steve 1" then I check I get "Steve 1" in the response. I may not check the whole page, but I check enough to have a level of comfort that I'm getting the right data in response.
My argument is that under load, thread safety issues could mean that one user gets the other user's data, not the one they requested. This isn't the sort of thing that would be easily found in functional testing.
The performance testing guy here disagrees - he said it's enough to check that he get's the page - he doesn't care if it's got the data you requested or not. He said that this is a functional issue, so not within his remit. He's just recording the time it takes to do it.
I'm struggling to reason with him because it just seems fundamental to me. What's the point of measuring the time it takes to do something, if you're not sure it's doing the right thing.
So, what are people's thoughts about this?
Everywhere's within walking distance if you have enough time.
If I understand your scenario correctly, I fundamentally disagree with your performance testing guy.
If load causes functionality to go awry, that's a bug. A bug that almost certainly needs to be reported, diagnosed, and fixed. And a bug that will likely involve more than just the functional testing team to reproduce and help get solved. And conversely, I can make my application return virtually instantaneous responses, as long as nobody cares if the response is correct or not.
Now there is a squishy line here. How deep should the performance tests go to ensure functionality works correctly? Every shop draws the line somewhere. But in my experience, few shops don't care about functionality at all during their performance tests. I've written in the past about some mixed functional-and-load tests which also combined some manual tests under load: All Things Quality: Automation-Assisted Test Fest.
I've seen shops that get themselves tied up in knots with "it's not my problem" issues like this, and worry about slight overlaps of functional tests by the performance testing team, slight overlaps of performance tests by the functional testing team. I try hard not to let my Test Team fall into that trap.
I also disagree with your friend. If a performance test uncovers an issue, then how is that a bad thing?
I had the same thing happen to me during a load test, one user was getting another users data. And the only reason I found it was because I set up the content checks to check the data. Once I saw the functional issue arise, I showed it to the project team and they found that one of their server parameters was incorrectly set so they fixed it. What would've happened if I only checked the page title and not the data?
By not being thorough with content checks you're basically choosing to be a less effective tester. So I don't know why your friend is suggesting to remove code that enables you to do more effective testing. Some functional issues are missed during functional testing but are later uncovered during performance testing when the load is applied. Leaving out content checks is a risk so why risk it?
Ok, well check the frickin' page then. And that includes the data. If the data ain't right then the page ain't right.
"he said it's enough to check that he get's the page"
Last edited by LoadRunner421; 04-23-2013 at 08:36 AM.
I'm pretty much w/ Joe S. on this. I am for the most part assuming that the app is functionally correct when I'm doing performance testing. That said, I still often do rudimentary checking at various points just for sanity. :-)
I can argue either way. But I do think generally measuring response times and resources used should be the primary focus. For example, I can argue that returning the right data would be say an issue with the database itself and not the backend application under test, and the extra cpu cycles and resources should be channeled to creating more stress per VM used.
But I also do think it's good to have some light validation at key check points to make sure you're getting a consistent apples to apples comparison. So I'd probably say not validate the entire list or every single list request, but maybe every first request of the list, validate an expected 1 or 2 expected items in the list, and validate items along key state changes.
For issues such as concurrency, it's very hard to test isolate and reproduce at a system level. Let's say you find hit a functional error at the load test. There's no guarantee you'll get the same result again if you re-run that load test. IMO, these are best tested at the unit level using an event driven test where the exact conditions could be reproduced at a controlled environment at a small scale.
FURPS requirements are tied
The issue is that load impacts functionality, not just performance and usability metrics such as response time. One example would be a lagging cache (only discovered under load) where the data displayed did not represent the latest state. This is a functional issue as the requirements would simply state that the latest data entry be displayed in a concurrent user environment. That said the requirements should (and often donít) contain the performance requirements. In this way functionality is Ďtiedí to a given performance (workload) so the functional testing cannot be completed until the performance testing is completed. Sadly many performance engineers think that functionality testing ends and then performance testing begins.
Yup. One person being the OP's friend who said "He said that this is a functional issue, so not within his remit." He is ignoring the overlap. Well said.
Originally Posted by Ian
Your friend is wrong. If you are not checking for expected results then you are not testing. It's an implicit part of the testing process.
For my firm this would be something that would either lead to a "No Hire" or "No Retain" situation.
Can we all please stop referring to him as my 'friend'?
He's a nice enough guy, but I suspect he thinks I'm sticking my nose in his business, which may have been part of it.
Anyway it all ended nicely. He had a chat to a manager here about me, she got us all in a room, we had a discussion and when she asked why we wouldn't want to do a simple check to know we get the right page back - she's not a performance testing expert, but has common sense - he agreed to add the checks.
Everywhere's within walking distance if you have enough time.
Haha. Ended nicely for you, but not so much the other guy. Your manager is likely very unimpressed with him.