SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Results 1 to 9 of 9
  1. #1
    Member
    Join Date
    Dec 1999
    Location
    Portland, OR, USA
    Posts
    91
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Speed of coverage tools ?


    Most coverage tools cause the application under test to run much
    slower than normal. Estimates range from a factor of 2X to 10 X.

    What are the factors which affect this ?

    Which Code Coverage tool (at least for C or C++) gives the best
    performance, i.e. the least speed degredation ?



    ------------------
    Charles F. Radley - CSQE
    Oregon, USA.
    Quality Control Analyst at Syntel Inc
    Project Test Lead for client Daimler Trucks - North America.
    Interested in testing dot net web services and SOA systems.
    Charles F. Radley
    Oregon, USA.

  2. #2
    Senior Member
    Join Date
    Mar 2000
    Location
    Sweden
    Posts
    252
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    I have used Pure Coverage for several years, and the speed is almost as usual.
    This tool covers only statement coverage, i.e it is only the object code that is affected.
    If you use tools with bransch coverage, then also the source code is affected which might giove an additional overhead (I think).

    However, if you are using code coverage tool for unit tests, you shouldn't see too much difference. But if you try to test a big application (system test)... yes, I guess you will have problem.

    Best regards Björn


    ------------------

  3. #3
    Member
    Join Date
    Dec 1999
    Location
    Portland, OR, USA
    Posts
    91
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by Bjorn:
    I have used Pure Coverage for several years, and the speed is almost as usual.
    This tool covers only statement coverage, i.e it is only the object code that is affected.
    If you use tools with bransch coverage, then also the source code is affected which might giove an additional overhead (I think).
    >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    Bjorn, thanks for your reply:-

    Regarding purceover:

    My anecdotal information arises from four independent sources for a single tool, pureCover.

    Four independent organizations informally reported to me their
    experience in factor of speed degredation with pureCover as follows:

    organization 1 - no degredation
    organization 2 - 2X
    organization 3 - 5X - 10X
    organization 4 - 5x - 10X

    I do not know how purcecover works, some libraries must be built in, which seems like inline code.

    >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    However, if you are using code coverage tool for unit tests, you shouldn't see too much difference. But if you try to test a big application (system test)... yes, I guess you will have problem.
    <HR></BLOCKQUOTE>
    >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    We would like to test complete products.



    ------------------
    Charles F. Radley - CSQE
    Oregon, USA.
    Quality Control Analyst at Syntel Inc
    Project Test Lead for client Daimler Trucks - North America.
    Interested in testing dot net web services and SOA systems.
    Charles F. Radley
    Oregon, USA.

  4. #4
    Junior Member
    Join Date
    Feb 2001
    Location
    austin, tx, usa
    Posts
    1
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    Where can I get a copy of Pure Coverage?

  5. #5
    Senior Member
    Join Date
    Mar 2000
    Location
    Sweden
    Posts
    252
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by mflynn:
    Where can I get a copy of Pure Coverage?<HR></BLOCKQUOTE>

    You can visit their home site and check for an evaluation copy....
    www.rational.com


    Regards Björn

    ------------------

  6. #6
    Senior Member
    Join Date
    Mar 2000
    Location
    Sweden
    Posts
    252
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by cradley:
    [QUOTE]

    We would like to test complete products.


    <HR></BLOCKQUOTE>

    Testing a complete product (i.e system test) is usually done by black box methods. Test cases are built on requirements and design specs.
    During this kind of test it is usually not of any interest to measure code coverage.

    Code coverage measurement is typically an action taken during unit tests where white box and black box methods should be combined for best result.

    Have you considered a change in your way of working?

    Best regards Björn

    ------------------

  7. #7
    Member
    Join Date
    Dec 1999
    Location
    Portland, OR, USA
    Posts
    91
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by Bjorn:
    Testing a complete product (i.e system test) is usually done by black box methods. Test cases are built on requirements and design specs.
    During this kind of test it is usually not of any interest to measure code coverage.

    Code coverage measurement is typically an action taken during unit tests where white box and black box methods should be combined for best result.

    Have you considered a change in your way of working?

    Best regards Björn
    <HR></BLOCKQUOTE>

    We are interested to use code coverage tools to measure the effectivenesss of our black box testing.

    This does not necessarily mean statement coverage, at a black box level, sometimes function coverage is enough to give useful data for finding significant holes in the regression suite.

    This is a fairly common practice in black box test organizations.

    I am not involved in unit testing, that is a different department.


    ------------------
    Charles F. Radley - CSQE
    Oregon, USA.
    Quality Control Analyst at Syntel Inc
    Project Test Lead for client Daimler Trucks - North America.
    Interested in testing dot net web services and SOA systems.
    Charles F. Radley
    Oregon, USA.

  8. #8
    Senior Member
    Join Date
    Mar 2000
    Location
    Sweden
    Posts
    252
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    I understand your point and I agree with you as well.
    The difference is that we are using coverage tools to measure the effectiveness of our black box tests on a lower level.

    At the higher level (system test), our concern is almost only requirement coverage. We will make sure that we have test cases that covers all requirements. How this equals to code coverage is not of any interest, since we know (during our unit tests) that the code is executed with expected result up to a certain degree.

    Best regards Björn


    ------------------

  9. #9
    Junior Member
    Join Date
    Mar 2001
    Location
    Austin, Texas, USA
    Posts
    13
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Speed of coverage tools ?

    Tools that implement branch coverage by inserting probes in the source text should have only a small impact on application runtime. Such tools convert code such as:
    if (...)
    { \block1 }
    else { \block2 }
    into:
    if (...)
    { Visited[295]=true;
    \block1 }
    else { Visited[296]=true;
    \block2 }
    for the purposes of testing. Any decent
    compiler will generate very efficient
    code for array[constant_index]=constant;
    i.e., a load constant followed by a store instruction. So such overhead should be a few machine instructions per basic block.
    If basic blocks are very small (i.e,),
    (e.g, do a simple assignment), this could
    be as much as 50%, but in practice it is a lot less.

    Ira. D. Baxter
    Ira D. Baxter, CTO
    Semantic Designs, Inc.
    http://www.semdesigns.com

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 10.00%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 04:45 PM.

Copyright BetaSoft Inc.