SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Results 1 to 6 of 6
  1. #1
    Junior Member
    Join Date
    Jan 2002
    Posts
    5
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Performance Testing Query

    Hi all,

    I'm a newbie here, so apologies if this is dead obvious question or has been asked often before (a quick search didn't show me any likely hits). I should also say that I'm a relative newbie to Web Testing as well.

    Basically the company I'm working for are bidding to develop a web front end for a medical records application. Obviously it's important that:
    o We present correct information
    o We only present information to properly authorised people.

    Our customer has done some safety analysis work and has placed a requirement on us to demonstrate that our system will not present incorrect information worse than once every 100,000 queries (obviously we're trying to develop it so that it never presents incorrect information, but that's hard to prove).

    Does anybody know of any resources or have any experience with demonstrating compliance with this kind of requirement?

    I'm vaguely aware of statistical testing, but am unsure of how it can be implemented in a web environment.

    Thanks for any help in advance.

    Regards

    Hambut

    ------------------

  2. #2
    Moderator Joe Strazzere's Avatar
    Join Date
    May 2000
    Location
    USA
    Posts
    13,170
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing Query

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by hambut:
    Our customer has done some safety analysis work and has placed a requirement on us to demonstrate that our system will not present incorrect information worse than once every 100,000 queries (obviously we're trying to develop it so that it never presents incorrect information, but that's hard to prove).
    <HR></BLOCKQUOTE>

    What do you mean by "incorrect information" here?

    How much easier will it be to demonstrate that you present incorrect information less than once in 100,000 queries than to prove you never present incorrect information?

    Or, is the requirment really to run 100,000 queries (one time) and prove that all 100,000 presented "correct information"?

    And is this really a performance issue? Or, is it a functional issue?

    ------------------
    - Joe (strazzerj@aol.com)

    [This message has been edited by jstrazzere (edited 01-04-2002).]
    Joe Strazzere
    Visit my website: AllThingsQuality.com to learn more about quality, testing, and QA!

  3. #3
    Junior Member
    Join Date
    Jan 2002
    Posts
    5
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing Query

    -- What do you mean by "incorrect
    -- information" here?

    Hmm - the system we're bidding for acts as an intermediary between an existing database and the users. So the system should present the same information to the user as that derived from the database.

    -- How much easier will it be to demonstrate
    -- that you present incorrect information
    -- less than once in 100,000 queries than to
    -- prove you never present incorrect
    -- information?

    It's a problem of testing - testing only shows the presence of errors, and can't show the absence of errors (although it can give more confidence in the absence of errors). Given this we can't just test it to show that it never presents incorrect information, but I think we can test it (a lot) to give sufficient confidence that it will fail so many times per transaction.

    -- Or, is the requirment really to run
    -- 100,000 queries (one time) and prove that
    -- all 100,000 presented "correct
    -- information"?

    The requirement is to provide evidence that the system will present incorrect information at most 1 in 100,000 transactions.

    -- And is this really a performance issue?
    -- Or, is it a functional issue?

    I think it's a combination of the two - we need to show a certain performance (i.e. no worse than x in x transactions) of certain functional behaviour (i.e. correct presentation of information).

    Cheers,

    Hambut

    ------------------

  4. #4
    Moderator Joe Strazzere's Avatar
    Join Date
    May 2000
    Location
    USA
    Posts
    13,170
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing Query

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by hambut:
    -- Or, is the requirment really to run
    -- 100,000 queries (one time) and prove that
    -- all 100,000 presented "correct
    -- information"?

    The requirement is to provide evidence that the system will present incorrect information at most 1 in 100,000 transactions.
    <HR></BLOCKQUOTE>

    This is no easier to prove than to try and prove that it will never give 1 incorrect answer.

    For example, suppose I run 100,000 transactions, and get 1 wrong answer. Does this meet your requirement? Probably not, since I might get a wrong answer the next time, or 2 times in the next 100,000, or never again.

    At best, you can only show statistically how often errors have occurred, and try to use this as an predictor of future occurrences.

    Come up with a number of test runs that makes the client happy, and measure the errors in that. 100,000, 200,000, 1,000,000 whatever - at least that way you'll be able to plan the bounded effort.

    ------------------
    - Joe (strazzerj@aol.com)
    Joe Strazzere
    Visit my website: AllThingsQuality.com to learn more about quality, testing, and QA!

  5. #5
    Member
    Join Date
    Sep 2001
    Location
    Sunnyvale, CA, Santa Clara
    Posts
    394
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing Query

    The performance issues will occur when the system is under load, ie TCP connections lost or web server overloaded. For these conditions you should have seperate requirements (ie 200 concurrent sessions etc). I think the data access requirement (including "who can access what") is functional and the customer should accept the results of an automated tool, given the numbers involved. I would look at a tool like OpenSTA http://www.opensta.org which would allow you to record and playback web scripts (for all test cases). The output should then be parsed to check for expected results. I would do this for a single user, so as not to overlap with any performance requirements. OpenSTA is free so you can experiment with it and show a demo to the customer, before you invest a lot of time.

    Hope this helps.

    Regards

    Ian

    ------------------

  6. #6
    Junior Member
    Join Date
    Jan 2002
    Posts
    5
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Performance Testing Query

    Thanks for the replies - they've been very helpful.

    The immediate action we're doing is to try and renegotiate this 1 in 100,000 requirement to something more obviously tested.

    Thanks for the help,

    Cheers,

    Hambut

    ------------------

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 11.54%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 05:07 PM.

Copyright BetaSoft Inc.