SPONSORS:






User Tag List

Results 1 to 8 of 8
  1. #1
    Member
    Join Date
    Jan 2001
    Posts
    33
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Running SilkTest 24 hours a day...

    I have written an automated basic acceptance test that is run on our web site for each build. Every time the new build is pushed, I can fire the script off and it will run through basic functionality such as searching and downloading content.

    Management had the idea to run the silktest BAT against our live web site all the time -- to keep the BAT running and logging results so IT would know if there was an error with the site.

    This sounds good, but I have a couple of questions:
    1. Is SilkTest robust enough to run all the time on a machine? I have had some serious performance issues in the past with running it over a weekend, but haven't determined whether it's because of our application (slowing the machine down) or if it's Silk.
    2. Logging has to be done to the Windows System Log, which is monitered by Big Brother. Does anybody have experience with this sort of set up?

    Any help or advice here would be great. Thanks!

    Andrew

    ------------------

  2. #2
    Senior Member
    Join Date
    Jan 2001
    Posts
    750
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    Not recommended to run silktest continuously using silk script.

    You can try to run silk continuously through
    external programs like perl.

    I don't see what's the use of running all tests all the time if the code didn't change.

    Running a couple of times after a new build make sense but running it continuously is insane.

    ------------------

  3. #3
    Member
    Join Date
    Jan 2001
    Posts
    33
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by GodKnows:
    I don't see what's the use of running all tests all the time if the code didn't change.

    Running a couple of times after a new build make sense but running it continuously is insane.
    <HR></BLOCKQUOTE>

    Management thinks that this will be a good high-level (i.e. user-perspective) monitering tool against our live web site. If the script fails at some point in time, then we know there are issues with the user experience. IT has all the tools in the world to moniter load, usage, and an uncountable number of other low-level parameters with the system. But when it comes down to the bottom line, all management really wants to ensure is a functional and proper end-user experience.

    My automated basic acceptance test was designed for single or small iterative cycles, not continueous running, and from what I've seen, running Silktest for anymore than a day hoses the system it's running on pretty regularly. Like suggested previously in this thread, maybe running the Silk script via Perl every, say, 30 minutes, would give the same general results, but would spare the system from having to continuously run Silk.

    My other problem is reliability. So the script fails at some point -- how can I be sure there's an actual problem instead of some window being out of focus or something trivial like that? The issue has been raised because management see's this as a tool to plug right into the existing monitering tools and even paging IT if the script fails! I think this is rediculous because in my experience, Silk just doesn't run that reliably -- it will fail for any NUMBER of reasons, and part of the frustrations with using it are determining whether the fail was legit or not.

    Has anybody actually written bullet-proof Silk scripts?!

    Thanks for all the advice here...

    Andrew

    ------------------

  4. #4
    AJ
    AJ is offline
    Moderator AJ's Avatar
    Join Date
    Jun 1999
    Location
    San Jose, CA
    Posts
    1,691
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    What you are looking for is something like a hosted monitoring solution Which Segue now offers I believe SilkVision is the name.

    Many companies have exactly that solution and it's a great one for monitoring health of your servers.

    Now as to running SilkTest scripts over and over for ever, I generally do not recommend that. You can right some bullet proof scripts, but the sacrifice is you should make them small and simple. Also make sure to restart the browser with every run. So in your ScriptExit () function close your browser.

    This way the browser won't be give resources/memory problems.


    ------------------
    AJ Alhait
    BetaSoft Inc.
    AJ Alhait
    BetaSoft Inc.

  5. #5
    Junior Member
    Join Date
    Apr 2001
    Location
    Scotts Valley, CA
    Posts
    16
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    Maybe you *can* get SilkTest to do continuous testing, but I'm not convinced it's a good tool for that kind of thing. My suggestion to you is to use a load/stess testing tool to simulate a single virtual user carrying out the key transactions of your site, and fire off this user every 5 minutes or so. The CPU and memory overhead on this is going to be much less than would be required by SilkTest. You stand more of a chance keeping something like this running over an extended period of time. Plus, since your management are obviously interested in keeping the site running, if you don't already have such a tool, now would be a good time to suggest buying one! (BTW I have used SilkPerformer and I think it's a great product but simple things can be hard to do due to the overwhelmingly complex nature of the product).
    However, for true 24/7, I agree with previous respondents who suggested using a scripting language such as PERL.

    ------------------

  6. #6
    Super Member
    Join Date
    Jul 1999
    Location
    Rancho Santa Margarita, CA
    Posts
    1,439
    Post Thanks / Like
    Mentioned
    4 Post(s)
    Tagged
    1 Thread(s)

    Re: Running SilkTest 24 hours a day...

    Andrew,

    From a previous company, we used to run QA Partner scripts for 3 or more days without much difficulty. AUT is Windows App (not Web) and environment is pretty much controlled. As AJ suggested, our scripts do a lot of cleanups and unfortunately, each script has to close every single dialog/window that is open and it has to RESTART the AUT every single time. I said unfortunately since the real world USERS out there won't be doing the same thing. They will execute one feature, then another, and another without exiting the application in-between each task being verified by each testcase. If you start simulating what the users really do (in your scripts), then these memory/resources problems and stuffs...

    Is your so-called basic acceptance scripts simulating what the users (from beginners to advanced) do out there to your application?
    Are you sure your scripts were written to work regardless of what the live users are doing to your application? If so, how much do you think are your scripts simulating? (wow, I can't even guess if USER1 will press Ctrl-Alt-F10 or if USER2 will hit Alt-$ or something)

    Sounds like you are very frustrated with using SilkTest that can't produce your bullet-proof scripts. Have you ever used another test tool were you were able to produce bullet-proof scripts that runs 24/7 reliably via the Internet? Which test tool is IT? (i'm sure many of us would like to know)

    So your management wants this and wants that fro

  7. #7
    Senior Member
    Join Date
    Aug 1999
    Location
    Cambridge, UK
    Posts
    470
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    My experience of running silktest for long periods has shown up two significant problems:
    - extreme sensitivity to resource leaks in the aut, or anything else silktest uses to run it's tests (e.g. calling functions in external dlls). This can slow down and cripple the testrun, requiring user intervention on the target machine to get it back to a usable state.
    - results log getting too big. You may be able to get away with removing the results file every time a run finishes, then restarting silktest again with the same testplan. But if you just have one very big testplan that will take silktest hours (or days) to fully execute, then you need to be very careful about what gets logged to the results file; a .res (or .jou) file too big might not be openable, meaning you've just lost all your results data.

    ------------------

  8. #8
    AJ
    AJ is offline
    Moderator AJ's Avatar
    Join Date
    Jun 1999
    Location
    San Jose, CA
    Posts
    1,691
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Running SilkTest 24 hours a day...

    If you plan on doing that, create a Reset function that closes yuor appliaction/browser/aut and erstarts it every few runs.

    This will give you a better chance in succeeding

    ------------------
    AJ Alhait
    BetaSoft Inc.
    AJ Alhait
    BetaSoft Inc.

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 12:38 PM.

Copyright BetaSoft Inc.