SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Results 1 to 8 of 8
  1. #1
    Junior Member
    Join Date
    Jul 2003
    Location
    California
    Posts
    1
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    pinpointing bottleneck from performance test results.

    Hi,
    I have a very generic question.
    I perform a load test with increasing number of virtual users which gives me the end to end response time of the system.

    Now the response is very bad (it is giving me a very high response time even for less number of users). Now I know that something is wrong, but what should I start looking at? The backend database, the web server, the app server etc. Is there any method or framework which can help me pinpoint the cause of the bottleneck? Basically where should I start tuning the application so that it takes minimum time and effort and I dont end up wasting time on tuning the parts of the system which are not responsible for the bottleneck.

    How do performance analysts map a bottleneck to its cause?

    Thanks,
    Milap Shah

  2. #2
    Senior Member
    Join Date
    Apr 2001
    Location
    N. Mankato, MN
    Posts
    319
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    Well, to start you should monitor each part of the application when running your performance test. So you may want to run a query analyzer of some sort, monitor web/application server activity (i.e. on a Windows Box maybe using performance monitor)...if you are running client-side processing you may even need to monitor the client processing. Also I am not sure if the performance testing suite that your using breaks down the response time per page, but if it does then you can look at the specific page(s) that are involved.

    Also there are a lot of code/query/hardware performance tuning tips/tricks that can help...try doing a search on google. Also make sure that you only changing one thing at time, so that you can see where the improvement is coming from. (i.e. one item might improve performance by 30% but another item might decrease performance by 30%...so if these were done together then you would see no performance gain, even though there was one by making the one change but not the other)

  3. #3
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    Milap - you sent me an email on a related topic - I tried to reply and it bounced. Please follow up.

    **This has been a public service announcement** [img]images/icons/wink.gif[/img]
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  4. #4
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    Short answer. Look for patterns, which pages are slow? Which ones get slow first? What is going on with those pages? Get developers involved, show them results, ask them questions. Create tests that isolate one issue (suspect) at a time. Exploit it, analyze it, disect it.

    I'm in the middle of writing a 4 part theme as part of the Beyond Performance Testing series about this question exactly. It'll be out in a few months.
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  5. #5
    Senior Member
    Join Date
    Dec 2002
    Location
    New York, NY
    Posts
    157
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    A few suggestions --

    First, think up a “divide and conquer” strategy – how can you isolate and pinpoint the problem? As in software debugging, 90% of the time is spent just trying to localize the problem through intelligent trial-and-error. For example, you mention that end-to-end response time is too slow. Is there any way to determine where most of the delay is happening? It is useful to have a good model of the system architecture or topology to help guide your thinking. What probes reporting data or information sources do we already have available? What information sources would you ideally like to have, and where should they be placed in the system? How can you implement or attain these?

    Second, look for discontinuities, cusps or other sudden changes of behavior – things like the kink in the J-shaped hockey-stick curve. These often occur at or around bottlenecks.

    Third, ask around. Stanford has some pretty savvy people not to mention geniuses, and their networks have come a long way since my student days there.

    I have sent you directly some notes on bottleneck identification. If you wish to privately send me your test results (preferably graphed or charted), I will come back with some dumb questions which may just help.

  6. #6
    Junior Member
    Join Date
    May 2002
    Location
    Goodyear, AZ
    Posts
    12
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    When you run a typical suite of tests, you run a series of user levels such as 100, 200, 300, 400, etc users until something breaks (CPU is 100%, errors returned, etc). Many times when you look at response times and system performance statistics you'll see them increasing (or decreasing in the case of memory) from test run to test run while increasing load by increasing the number of users. At the point when the test does not run well (fails) you may need to run another test and back off the load or number of users a little from the failed test. This is because many times when something goes wrong, it causes a cascade that hides the true issue. Everyone here agrees to look for patterns and changes in behavior. At that point you start with the first thing that changed such as a particular response time increased in a manner not consistent with prior tests, or possibly the test system took a moment to take care of itself rather than user applications (system vs. user CPU time). There are so many possibilities of things to consider in the hardware, operating system, software, the network or the database. If any one of these starts to fail you get the appearance of poor performance. Which one is it and where within that component is the issue is what makes this fun.

  7. #7
    Moderator
    Join Date
    Aug 2000
    Location
    Vancouver, BC, Canada
    Posts
    1,189
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    The following is a piece of prose that I wrote about maximum throughput calculations. In this calculation you look at the system resources that are used by the different systems in your test. You then find the one that uses the most resources and that is your bottleneck. That will be the server that will determine how large a load you can run on your servers.

    Maximum Throughput Calculation
    Based on the measurements from the test tool and from the monitoring tools, you can calculate the maximum throughput that the system (used in the test) will support. This is essential information that will be used in the capacity planning.

    Throughput
    Given the following situation:
    Your system processed 23,000 queries in 1500 seconds. Your monitoring reveals the following utilization figures for your components.

    Utilization

    Server
    % Processor
    % Disk

    Web
    60
    0

    Database 1
    40
    23

    Database 2
    30
    20

    Database 3
    66
    33

    The maximum throughput is calculated as follows:
    Calculate your Average throughput. For instance if you measured 23,000 queries in 1500 seconds, so you have an Average Throughput of 23000/1500 = 15.3 Queries/Second.

    Then calculate you average service demand for all your measured components. For this you take the utilization of a component and divide that by your Average Throughput. The Web processor calculation gives therefore 0.6/15.3 = 0.03.

    Average Service Demand

    Server
    Processor
    Disk

    Web
    0.0392
    0

    Database 1
    0.0261
    0.015

    Database 2
    0.0196
    0.0131

    Database 3
    0.0431
    0.0261

    The component with the highest service demand is the processor in database server 3. This is the weakest link in out chain and this link will determine the Maximum Throughput we can achieve. We can calculate the maximum throughput by 1/0,0431 = 23.2 Queries/second.

    This method will not be of great use during debugging but I still hope this is of help.
    Roland Stens

  8. #8
    Junior Member
    Join Date
    Nov 2000
    Posts
    20
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: pinpointing bottleneck from performance test results.

    "Now I know that something is wrong, but what should I start looking at? The backend database, the web server, the app server etc."
    ---------------------------------------------

    Its is lot more easier if we go in a planned manner.

    Here is four point drill down strategy that may help

    1- Find clue from Hardware/OS Monitoring

    Processor Memory, HDD. Network(request times between servers): will lead to the culprit tier DB/Web/App server etc

    2- Find a clue from counters for Software running your application on Bottleneck Tier

    Set the counters for Softwares on the culprit tier e.g. For DB tier you would monitor the counters for Oracle/SQL server or If its App Server you would set counters for Weblogic/Webshpere (in j2ee). 2a) Tune and adjust parameters for the DB/App/Web Server running you application. 2b) Recheck the results. If not go to step 3.

    3-Find a Clue from Detailed Counters About your application.

    Set the appropriate counters of the DB/App/Web Server software running your application to identify bottleneck in YOUR application. 3a) Get it changed from developers and confirm results

    4- Go around these steps until you reach your performance testing goals

    Caution: This assumes that nothing is wrong is with Script or scenario design.

    However Identification of the problem is directly propotional to the understanding of application, softwares used in the application, its deployment mechanisms, hardwares involved and OSes used.


    Hope this helps
    Shahzad

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 8.33%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 11:57 PM.

Copyright BetaSoft Inc.