SPONSORS:






User Tag List

Results 1 to 5 of 5
  1. #1
    Senior Member
    Join Date
    Oct 2000
    Location
    Laurel, Maryland, USA
    Posts
    204
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Silk and standalone Java application - very unreliable


    I have been automating procedures for a standalone Java application for about 6 months now, and I am not at all satisfied with the stability of SilkTest. I am interested to know if anyone else in a similar situation has experienced similar problems, or if Silk has proved to be a reliable tool.

    I am running Silk 5.5 with a Jre 1.2 built Network Management application running on 95, 98, NT, and 2000. Most of the test cases we have automated already are fairly involved - they take around 40-50 minutes each to execute. My original test cases were longer - each taking 2-3 hours to execute. I could never get these to complete successfully, so I had to break them down as much as possible.

    I know there are some problems with the clients - occasionally the will stop responding and the test fails. The problems that are really frustrating are the failures in Silk that seem to occur so often. I get various internal errors that cause tests to abort - usually when it is doing an Exists() check on a window. These error messages are meaningless to me since they are not script errors but errors from deeper in the SilkTest code.

    I have talked with Customer Support about a couple of them, but they say they can't debug the problem without using my application. My application is a client/server app and it is proprietary, so I can't send it to them.

    I am evaluated WinRunner before choosing Silk, and I'm thinking of doing a re-evaluation to see if WinRunner is more stable with my application.

    Any thoughts or suggestions?

    Thanks,

    Atlee

    ------------------
    Atlee
    Atlee

  2. #2
    Senior Member
    Join Date
    Jul 1999
    Location
    Bellingham, WA USA
    Posts
    1,323
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk and standalone Java application - very unreliable

    Atlee,

    You're not unique with these problems. Our company is moving in the direction of producing products that are 100% Java, and we are experiencing the same sort of behavior.

    SilkTest/QAPartner has a reputation for being quite reliable prior to the days of Java. The Java testing, however, requires a tremendous foresight on the part of the individuals writing the scripts. You almost have to anticipate where the difficulties are going to be. Timing issues have proven to be difficult. I find myself using far more error checking than I used to, and while loops that check for the existence of certain objects. In some instances, I've had to put code in place to abort an attempt to call a certain object after a timeout and instigate the call again.

    The sad part about this is, I didn't get any better results with the Mercury product when evaluating.

    Java is a bit of a moving target, and tool makers have a difficult task of keeping up with that. It is the same with browser releases.

    I don't have an easy answer for you. It is just a call for us as testers to write amazingly robust scripts!

    One suggestion for your calls to tech support is to ask them to do a WebEx session with you so they can see exactly what is happening as you run the script from your machine.

    ------------------


    [This message has been edited by DJGray (edited 10-17-2001).]

  3. #3
    Member
    Join Date
    Feb 2001
    Location
    New York, NY
    Posts
    41
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk and standalone Java application - very unreliable

    I agree that a lot of people are in the same situation. I been trying to automate stand a lone Java app for few month with partial success. Among various problems, the two main ones are:
    1. Silk 5.5 was loosing the frame definitions in the middle of the script. It happens 75% of the time. My debug statements are half of the code, but still no solution or insight why it is happening.
    2. The second is the issue with extended controls. We were using Borland controls and Silk was not able to work with them. The final solution to this problem was to remove the controls from application and replace with Swing. Very ugly solution.

    Have you tried JTest or CompuWare? They might have an alternative solution to Silk and WinRunner.

    ------------------
    -Lev Aks
    lev_aks@in-nyc.com

    [This message has been edited by levaks (edited 10-17-2001).]

  4. #4
    Senior Member
    Join Date
    Oct 2000
    Location
    Laurel, Maryland, USA
    Posts
    204
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk and standalone Java application - very unreliable

    Thanks for the responses. Somehow knowing I'm not the only one makes it a little easier to buckle down and work the problems out.

    I didn't think about the WebEx session - that may be a good way to handle it. It's hard to predict when the failures will occur, but I have a couple of scenario's that fail more often than not.

    I have also experienced problems with recognizing known windows in the middle of a script. Sometimes a pop-up window is displayed and Silk doesn't recognize it for some reason. It's the same window that it has recognized many times in the past, so I know it's not the environment setup. I can't figure out if it is my application causing the problem (for some reason the classpath is corrupted and Silk can't use its jar file) or if it is Silk that is confused. I have code in place to use Task Manager to kill the application, but this only works on NT's and 2000 machines.

    I haven't tried JTest or Compuware. I brought Compuware in for a presentation, but I wasn't that impressed with it. It wouldn't handle Unix either, so that ruled them out. I never looked into JTest, but maybe I will now.

    Thanks

    ------------------
    Atlee

    [This message has been edited by awhiteleather (edited 10-18-2001).]
    Atlee

  5. #5
    Senior Member
    Join Date
    Jun 2000
    Location
    Chicago, IL USA
    Posts
    383
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk and standalone Java application - very unreliable

    Atlee.

    We have a 99% Java client/server system application written with JDK 1.2.2. For GUI testing we are using SilkTest 5.0.3.2. For API testing we use test programs written in Java.

    Initially, we had a number of problems with the SilkTest<->Java interface. Some of the instabilities were eventually traced to memory and pointer bugs in the AUT, others have been fixed with custom methods incorporating special workarounds. With these fixes, our regression tests can run consistently and stably for several hours before I get a "stack overflow" that is probably caused by the AUT. It might be that SilkTest 5.0.3.2 is more stable than 5.5.

    Unfortunately, we still have some critical issues that do not have acceptable workarounds. Segue support used the same lame excuse with us -- "we can't help you without a copy of your application". Assuming that we trust them to keep our security protocols confidential, even our developers would need at week to set up the server and client for them to try. Why should we pay those ridiculous "maintenance fees" every year, if they flatly refuse to help us ? It would be much easier for them to give us the source code to "SilkTest_Java2.jar". Then, at least we could trace back the problems in their code and help them to fix it.

    In any case -- we have been preparing to get rid of SilkTest. Rational Robot is designed for Java testing. I have heard that it is still rather crude, but I expect it to mature. Until we decide on a replacement, we have not been using any of SilkTest's proprietary features, such as "plan" files. We only use 4Test code and we have been maintaining a strict separation of our testcases and test data from our test harness for the GUI. Our test harness allows all of our testcases to be expressed in terms of business objects and methods with parameters based on business rules. In this way, testcases which use objects and methods that are now written in 4Test can be easily adapted for a test harness written with another test tool.

    ------------------
    Rob Kapteyn
    Rob Kapteyn, CSTE
    RobKapteyn@gmail.com

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 10:45 PM.

Copyright BetaSoft Inc.