I have results for Release1 and Release2.
<font class="small">Code:</font><hr /><pre>
Response Time Transaction Per Second
Release1 33.66 100
Release2 53.02 35
Release2 should be better than Release1.
The Response Time is higher but Transactions per Second are lower.
Do you think Release2 is improved?
If I consider Response Time than Release2 did not improve but if I consider Transaction Per Second than Release2 did improve.
I don't quite understand your logic saying which release is better without knowing 'EXACTLY' what you are taking into account. Are you taking into account the response time? (is this the min response time or max or median?) vs the amount of transactions. Or are you assuming if a response time is lower - this is bad. I don't quite follow but here is my opinion of how I would give my analysis.
Isn't the "Transaction Per Second" means the number of hits you sent to the server? And the "Response time" is the amount of time that the server responded to the Transaction total.
You should read the 90% response of the analysis rather than "response time" Meaning, in those 35 transaction that you performed on the requested to the server, 90% of the time the server responded within 53.02 seconds or better
This isn't better than release 1. Release 1 is better due to the 100 transaction that were requested to the server, the server responded 33.66 sec or better
Second I do not think below comment is an appropriate from a Team Lead
[ QUOTE ]
"and you identify yourself as a Silk tester anyways"
[/ QUOTE ]
Anyways, we ONLY compare the RESPONSE time between Releases. If Response Time is LOWER then better the Release.
Do you think is this the right approach?
Our test run as below:
All tests were run for 30 minutes each. Each test started with 5 users, increased by 4 users after 3 minutes to a maximum of 10 users for 7 reports, running with 2 players, totaling 70 users.