SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Page 1 of 2 12 LastLast
Results 1 to 10 of 16
  1. #1
    Junior Member
    Join Date
    Mar 2008
    Posts
    23
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    best way to represent my tests results

    Hi
    I;m doing performance tests on our product,on each test i calculate the average response time,(our response time don;t reach to more then 500ms).
    The problem is that wehn i compare this results with another run with the same configuration the average response time is not equal, What is the best way to calculate the results.(90%,or STDEV in excel?)
    Thanks
    Dan

  2. #2
    Member
    Join Date
    Jul 2006
    Location
    Melbourne, VIC
    Posts
    99
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    90th percentile is the more accurate to measure response time.
    How big the difference is?
    Regards,
    Zubair Syed

  3. #3
    Senior Member
    Join Date
    Mar 2003
    Location
    Uk
    Posts
    214
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    There are many ways to represent. The Idea with any report is to make sure you represent the complete picture, don't just mention the best happenings during the test. Also mention the bad ones, those are the ones of interest. Do not assume that the people who read the report want to see only the best part.If you are representing Percentile report then also make sure you give the complete data for the tests somewhere in appendix, so the ones who are really interested in understanding how the application behaved would get to see some detailed data.
    Cheers,
    Suresh


    http://cognitive-content.blogspot.com

    It is not necessary to change. Survival is not mandatory -W Edwards Deming

  4. #4
    Senior Member
    Join Date
    Apr 2003
    Location
    Wisconsin, USA
    Posts
    5,338
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    [ QUOTE ]
    90th percentile is the more accurate to measure response time.
    How big the difference is?

    [/ QUOTE ]

    Not necessarily more accurate. Just more accepted and probably closer to real life.

  5. #5
    Moderator
    Join Date
    Sep 2001
    Location
    Boston, MA
    Posts
    4,348
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    I like to report various stats for the results. normally i like to look at: avg, stdev, 80%, 90%, 95%, 99%
    Corey Goldberg
    Homepage: goldb.org
    Twitter: twitter.com/cgoldberg
    Google+: gplus.to/cgoldberg

  6. #6
    Member
    Join Date
    Jul 2006
    Location
    Melbourne, VIC
    Posts
    99
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    DSquared,
    True, more accpeted is the appropriate term.

    Regards,
    Regards,
    Zubair Syed

  7. #7
    Junior Member
    Join Date
    Mar 2008
    Posts
    23
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    The STDEV that i got is between 50 -80
    when the average of each call type is between 200-60 and the response time is in millisecond
    So if i get a response time of 200 and the STDEV is 80 its ok?
    and also i notice that when i do STDEV on the 90% the STDEV is smaller, is this a good idea?
    Thanks

  8. #8
    Junior Member
    Join Date
    Mar 2008
    Posts
    23
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    Another question...
    Why 90% is closer to real life?

  9. #9
    Senior Member
    Join Date
    Nov 2000
    Location
    Michigan, USA
    Posts
    840
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    [ QUOTE ]
    The STDEV that i got is between 50 -80
    when the average of each call type is between 200-60 and the response time is in millisecond
    So if i get a response time of 200 and the STDEV is 80 its ok?
    and also i notice that when i do STDEV on the 90% the STDEV is smaller, is this a good idea?
    Thanks

    [/ QUOTE ]

    A STDEV of 50 to 80 is quite high and casts doubt on the statistical significance of the results. If possible try to collect more data points by running longer tests and/or more iterations to identify trends in the percentiles, mean, and max.

    Naturally if you exclude the extreme 10% of the data (from a normal distribution) the STDEV is going to be smaller. But this is not statistically accurate. Though if you have a few "outliers" that fall outside the 99%, it may be reasonable to exclude them.

    I refer you to Scott Barber's latest papers at NobleStar, specifically "The User experience not Metrics" parts 6-9.
    A problem is a difference between what is perceived and what is desired, that
    we want to reduce
    (Dewey 1933)

  10. #10
    Moderator
    Join Date
    Sep 2001
    Location
    Boston, MA
    Posts
    4,348
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: best way to represent my tests results

    [ QUOTE ]
    mean (not avg)

    [/ QUOTE ]

    huh? mean and avg are the same thing.
    Corey Goldberg
    Homepage: goldb.org
    Twitter: twitter.com/cgoldberg
    Google+: gplus.to/cgoldberg

 

 
Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 8.33%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 11:49 AM.

Copyright BetaSoft Inc.