I would like to get people's opinion on what is a better measurement when performance testing services that will reside on the Integration server(Mid tier) and access the DB server(generally DB2 on a Mainframe).
Is it better to set the goal of 4 TPS or Response times of .25 seconds OR is this the same thing? Just trying to set some performance objectives for my project teams and not sure which metric to use.
They are not the same thing at all. Transactions can take a short time or a long time to complete and it doens't necessarily have a bearing on response time. Measure TPS to measure the nitty-gritty performance as tuning and (new builds) occur. Measure response time to see when the configuration/application/technology components are not giving acceptable results as viewed by the end user and more tuning or development enhancements are needed.
One thing to remember about response time is the response time must also have a paired variable of the about of virual users running against the same box. E.g. if you run one virtual user doing say 'LoginTransaction' then the response time will be a lot faster than if you measure the same transaction with 500 virtual users. So you need to say the response time for 'thisTransaction' should be such and such with 'x' amount of users carry out normal activities.
The "classic" performance curve is Response Time (secs) (y-axis) vs Load (TPH) (x-axis) and to look for the "kink" when the response time suddenly increases with a small increase in load. Beyond that point your system is under a stressful load.
I think the measurement you use is highly dependent on what your goals are. If you want to be able to serve up many pages quickly, I would say that the response time number would be more valuable. COnversely if you have an app, where people are able to wait a few secs for a response from the server, then maybe the # of trans/sec might be the right number to use.
Bottom line: Which one are you (your company) more interested in taking a look at? Which one is more important from your customer's standpoint?
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by perfcoord: I would like to get people's opinion on what is a better measurement when performance testing services that will reside on the Integration server(Mid tier) and access the DB server(generally DB2 on a Mainframe).<HR></BLOCKQUOTE>How concurrent requests are handled by your component? If it uses the only instance and puts extra requests in a queue or launches extra threads/processes? In this case you have to measure throughput/response time as well as the number of concurrent connections (threads/instances).