SPONSORS:






User Tag List

Results 1 to 7 of 7
  1. #1
    Senior Member
    Join Date
    Sep 2000
    Posts
    159
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Time required to write automation?

    Once one has basic proficiency in SilkTest,
    is there a general rule of thumb for time required to automate a manual task?

    For example, 3 hours to program a SilkTest script to automate a 15 minute manual task.

    Right now, our department does 100% manual testing and is considering automation tools.

    Of course , it depends on the application and skill set.

    Let's say our typical user would be :

    QA Engineer with 1 yr programming background
    ( Java, C,)
    Testing a Java Application on NT and Solaris.


    My evaluation experience with Silk tells me that a lot of investment up front is required .

    Automation may not be the magic solution that managers are hoping for.

  2. #2
    Senior Member
    Join Date
    Sep 2000
    Posts
    159
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    To add further,

    I have basically completed the build acceptance test suite for our product.

    It takes about 15 minutes to run versus the
    45 minutes it took to test manually.

    It took me about 3 weeks to do this, but this includes the time spent on learning the
    Silk Tool .

    Another very time consuming part of automation seems to be the error handling
    and recovery.

  3. #3
    Senior Member
    Join Date
    Aug 1999
    Location
    MA
    Posts
    129
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    Yes, automation takes quite a bit more time up front, but you get the savings each test cycle, and it beats manual testing.

    Error recovery is not too hard, once you get the hang of thinking like an automated script. Automation that does not trap, log, and handle errors is not very worthwhile in my opinion. I've inherited several projects were sure, it worked when the product worked, but it should also work when the product fails!


    [This message has been edited by styler (edited 02-11-2001).]
    Steven Tyler
    Manager - Performance Engineering

    Kronos Incorporated
    tel: +1 978 947 4219

  4. #4
    Senior Member
    Join Date
    Sep 2000
    Posts
    159
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    I'm finding that I'm spending as much time writing exception handlers as I am writing
    the script that handles the normal behavior.

    problems I'm encountering with exception handling:

    1. how to force an error. I think I'll have to ask the developers about this.


    2. recover and go on to the next case

    All the steps in my script are dependent on each other. It's not like the Find text example described in the manual, where if the Search Backward fails, you can execute the Search Forward test.


    I go thru several setup dialogs in order in order to complete a task. If any error message occurs during these steps, then it's impossible to complete the task.

    So, I really don't have a "next testcase" to go to. My recovery system currently exits the application, re-launches and starts the steps again.

    I hope this is an aceptable recovery plan.

  5. #5
    Senior Member
    Join Date
    Jul 2000
    Posts
    186
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    I would concentrate on making the test cases independant of each other. It may seem like a lot of work now, but it will be worth it in the end.
    Tom

  6. #6
    Senior Member
    Join Date
    Jul 1999
    Location
    New York, NY, USA
    Posts
    137
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    Here is an Excerpt from an analysis I had to do for a proposal. Please keep in mind that these estimates are based on the methodology I use to deploy automation efforts. I am very heavy on preperation of rock solid include files and then on simplification of test scripts.

    The majority of test scripts we create can stand alone and run in any order. This can only be achieved by creating good logical appstates and creating a useful global functions library.


    4.1 Overhead of Automation
    An Automation effort will run about 15 – 25% of the total development effort for a project. The effort will vary due to the complexity of the AUT (application under test). This effort is not entirely in addition to standard QA efforts because much of the effort to deploy automation also produces results that would have been achieved even if only manual testing were used.
    4.1.1 Additional Effort for Automation
    Again looking at the analysis of the XXXXXXXXXXXXXXX web site, we can demonstrate that Silktest Automation would have added at most 18% to the Build and Test phases of the project. Manual QA for Tech Web totaled 13% of the Build and Test phases. (see Figure 2).

    XXXX WEB SITE (original effort)
    XXXX Web Site total Development Hours 1,330 hrs
    XXXX Web Site total QA Hours 205 hrs
    Total for Build and Manual Testing 1,535 hrs
    Automated regression test & framework. 271 hrs
    Total for Build, Manual and Automated testing 1,806 hrs

    Silktest automation as a % of XXXX Web Site’s original build & test effort. 17.65%
    Figure 2.

    4.1.2 Time Saved from Manual QA
    Although it is difficult to estimate exactly how much additional QA effort automation will take, it is important to note that it will also reduce effort in other areas of the process.
    Experiences have shown that about 1/3 of the early effort (creating an include file, application states and smoke tests) will directly uncover defects in the integration testing phase. The Golden Rule of Software QA is that the cost of repairing defects increases exponentially throughout the life cycle of a project. Early automation uncovers many of these defects before the development effort has been completed, thus saving more time than if they had been uncovered during full SIT (System Integration Testing).
    Looking again at the XXXX Web Site project, we can estimate that the portion of testing attributed to manual testing would be reduced from 205 to 137 hrs. This would reduce the additional effort for automation to approximately 13%.
    4.2 Efficiency of Automation
    Automation also greatly affects the efficiency of the QA process. Each revision of the automated test cases for a major release would probably be about 10-15% of the initial effort and the effort saved from manual testing will be increased.
    As automated tests mature, they become more thorough and free up the manual tester from many of the basic tests that are occur for each revision. An application that may have taken days to certify can be now be tested in hours. Thus, defects uncovered early can be repaired and retested before delivery, greatly reducing the risk associated with late fixes.
    Many of these defects are of the types that are difficult to uncover manually until a system is in full SIT (System Integration Testing). Automated test cases can be scripted to work around these defects, and report back when they are repaired. These tests will also ensure that if the defect later is re-introduced, it will be reported. When manually testing it is often easy to overlook a re-introduced defect. This is because a testers attention is focused on new areas of functionality.
    In order for automation to be successful, it must begin in the later stages of Integration testing, and continue through the end of User Acceptance Testing. The more defects that can be identified and isolated early, the easier it is to fix them. If automation is deployed too late, it can become a burden on the QA process


  7. #7
    Junior Member
    Join Date
    Feb 2001
    Location
    ottawa
    Posts
    21
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Time required to write automation?

    Raul, expect lots of time in the beginning to create your automation. but once it is created and firing on all cylinders it works very effeiciently! The best thing you can do is find out how it s going to be used in the QA process. I hope there is a QA proces in place where you work because you can build your automation around that process, automation is a compliment to manual testing not a replacment. Look at what is being done manually, automate the tedious stuff, stuff that is essential to run but time consuming usually acceptance tests and regression testing. But don't create complicated automation! keep your automation simple and to the point. believe me it will become way more effective this way! if you create huge testcases You will end up in a maintannece nightmare!
    You will need some type of strategy, What are the areas that are going to be tested, keep exepctations of managment and development managers in check.

    let me just give you an example and explain a little what we did and see how it relates to you(or anyone else for that matter)

    What we did here is create an centralized automation engine where testcases are thrown into the the engine(each testcase has a set of command lines to run) these testcase then are ran on the application. Each command line is executed one at a time
    eg.
    (ini file was used easy to get section names that contain all the commands)
    [scenario 1]
    menu=file\open, Open
    TextField=SetText, Filename:, c:\test.txt, open
    button=click, Open, test.txt - Notepad

    you will notice that we just used the silk methods as are actions. the last string is the expected window. Well, basically the whole testcase is an expected result right. expected vs. actual. if anywhere during the execution the testcase fails we will know where and why because each command is carried out individually. if the expected control is not there we will know, if the expected control is there but the dialog is not we will know, makes for great logging and error trapping!

    All the window decs are found out dynamically so we do not have to worry about any new windows or controls on a new build. of course if a window does change(the expected window) no problem because the engine will let us know what dialog and what the new name is.
    During this execution each step is logged and any error that occures will be flaged in the log file. Of course if a pass happens, well who cares we are looking for errors right.

    so in the testcase just do a find and replace of the old dialog with the new expected dialog or control name for that matter and re-reun the testcase...very re-usable and measurable! (or one can use constants in the testcase, something a little mor detailed for here right now)

    rerunning lots of testcases no problem because we are reading them from a directory (sys_getdircontents)not in the actual silk code. all testcases(sub-folders to) are saved to a temp file. So we just loop until we are out of testcases. If an error happens on a testcase, then that one is removed and a new testcase is ran. If a reboot happens no problem because the testcase has been removed the engine will just start in the next one.
    oh, and yes this engine can run on web application as well and on any windows platform...all we have to do is put conditional statements in the engine depending on what OS we are running on.

    I hope this helps a bit raul.
    I hope this gives you an example of how I used automation for us here. I find the best way to learn is thourgh trial and error and how other people failed and what was learned.
    let me know if you need anymore explaination on what I did or how I did things. There is no right or wrong way for automation to work just a way for it to work for you.

    What I did and what you need are common but still you will need to confgure it to your environement so the engine might not be what you need.
    I find this site very useful and full of great ideas and information sharing is awesome on here!!!

    if anybody see's anything wrong with my logic or wants to add a comment please I am always open to new ways of doing things and ideas.

    cheers to all,
    james



    [This message has been edited by jamesa00 (edited 04-25-2001).]

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 09:21 PM.

Copyright BetaSoft Inc.