SPONSORS:






User Tag List

Results 1 to 8 of 8

Thread: Silk Survey

  1. #1
    Junior Member
    Join Date
    Jan 2001
    Location
    Charlottesvile, VA
    Posts
    3
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Silk Survey

    I'm currently conducting an evaluation of various automated test tools for the purpose of determining which tool is most compatible with my organizations needs. All responses to the following questions and any additional comments would be very appreciated by me and I'm sure others would also find your answers and comments very useful. TIA

    1.How long have you been using Silktest?
    2.What do you like about it?
    3.What do you not like about it?
    4.How well does Silktest perform?
    5.Does Silktest perform your entire set of tests?
    6.What kind of tests do you use Silktest for?
    7.How easy is it to interpret the results of the automated tests?
    8.Is Silktest now integrated into your work processes and standard procedures? If so, how much effort and how long did this take?
    9.Were there any benefits or problems in using the tool, which were not anticipated?
    10.How long did it take you to achieve real benefits?
    11.What are the critical factors for achieving payback?
    12.If you were doing it over again now, would you still purchase this tool? What would you do differently?
    13.If you have used both Silktestand
    Rational Robot, how would you compare the two products? (functionality, similarities, differences, overall feeling about each product)

  2. #2
    Senior Member
    Join Date
    Aug 1999
    Location
    MA
    Posts
    129
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk Survey

    <UL TYPE=SQUARE>
    1. 3 years
    2. Object oriented, very robust
    3. SYS_Execute is buggy, SilkTest in general is buggy on Windows 9x
    4. Very well
    5. No. Some Y2K tests done in DOS BATCH
    6. Mostly regression testing, and white box testing. Goal is to automate all manual test plans to some degree, and reduce manual test time hours. Silk is never tired, and for the most part is 100% repeatable.
    7. Very easy, up to the coder to properly use LogError, LogWarning, and print.
    8. Yes, about a year. I started with SilkTest coding standards for all SilkTest developers, and have buy in at VP level.
    9. Learning curve, most coworkers don't even have one college course in a programming language. We solved that by having a university come onsite to teach OOP with C++, paced by level of atendees.
    10. Immediately
    11. Buy in from management, time in schedules to code. Takes longer up front, savings come on tail end with coverage, execution time, and accuracy.
    12. Yes.
    13. N/A.[/list]

    [This message has been edited by styler (edited 01-24-2001).]
    Steven Tyler
    Manager - Performance Engineering

    Kronos Incorporated
    tel: +1 978 947 4219

  3. #3
    Guest

    Re: Silk Survey

    1.How long have you been using Silktest?

    6+ years (QA Partner/Silk)

    2.What do you like about it?

    4Test OO language.

    3.What do you not like about it?

    Because of the nature of our test environment and product under test (home networking products), we bind our Silk Agents to NetBIOS, not TCP/IP. Segue apparently does not test this configuration as each new release is buggy and tends to break previously stable functionality, or functionality that they claim works correctly when bound to TCP/IP.

    4.How well does Silktest perform?

    Issues with reliability and stability of Silk host and agents. We estimate 90% reliability in long duration unmanned automated test sessions; i.e. a Silk host or agent will crash at some point in a test session 10% of the time.

    5.Does Silktest perform your entire set of tests?

    No. Cannot test driver installation where PnP events may occur. Cannot test Asian language products due to lack of support for multi-byte character sets. Cannot test with WinME. Problems with Win2000.

    6.What kind of tests do you use Silktest for?

    Silk is at the core of our Acceptance and Regression level testing of every transmittal of our AnyPoint™ Connectivity Suite home networking applications. We rely on automation to test the Install and Uninstaller applications, an Internet sharing application, a drive/printer sharing/mapping application, an electronic registration program, a technical support utility, and other components of our product.

    7.How easy is it to interpret the results of the automated tests?

    Dependent on how well the developer has designed and coded each test case. We do not use any recorded test cases, so we do not rely on "built-in" results logging.

    8.Is Silktest now integrated into your work processes and standard procedures? If so, how much effort and how long did this take?

    Automated test suites have been designed to support Acceptance and Regression level testing of the retail and multiple OEM versions of our core software with multiple Win32 operating systems, in multiple languages (English, French, German), and with multiple types of AnyPoint network adapters (1 Mbps/10 Mbps, PCI/USB, HomePNA/HomeRF).

    We have successfully integrated code coverage analysis and memory/code level error checking with test automation using NuMega TrueCoverage and BoundsChecker. We have implemented fully automated management of "clean" base operating system image distribution to multiple target test PCs using ImageCast IC3 under program control from Silk based automated test suites.

    By following strict coding and design practices, and building an extensive library of reusable, robust, and maintainable test functions, it has taken 2.5 years of dedicated effort to reach this point.

    9.Were there any benefits or problems in using the tool, which were not anticipated?

    Poor on-time support for new Win32 operating systems (Win98, WinME) and new browsers (IE 5.5). Bugs that occur when using NetBIOS instead of TCP/IP with Silk Agents.

    Very bad technical support until Segue formed the Solutions Support Group, which has since provided exceptional support and patches to some of the defects that we have identified. Previously, Segue Tech Support had been very quick to “blame” advanced and expert end users for problems encountered with Silk. They were just as quick to try to sell us expensive training classes (which has been of little benefit to the test automation engineers I know who have attended).

    10.How long did it take you to achieve real benefits?

    Though development/deployment of new automated test strategies lags by 2 – 3 weeks, once deployed, allocated test resources (human testers) and test times have been significantly reduced, especially when considering that the same automated tests will be executed on multiple retail and OEM builds, under multiple o/s, and multiple languages.


    ------------------
    Tony Mrozinski
    Sr. Software Test Automation Engineer
    Home Networking Operations
    Intel Corporation

    [This message has been edited by Tony Mrozinski (edited 01-24-2001).]

  4. #4
    Senior Member
    Join Date
    Mar 2000
    Location
    pisctaway, NJ USA
    Posts
    188
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk Survey

    1. Two Years
    3. Support is not good. When you find a significant problem, you have to practaclly jump on board the Segue team to just demonstrate it.
    4. It does a good job considering the task at hand.
    5. No, just regression.
    6. Requirment driven.
    7. It's easy, but easier if you code your own "error messages" with your verfies.
    8. Yes, three months.
    9. You need someone CLOSE to full time on the tool. You cannot expect to work with record and playback 100%. I was recently told think of it as CUT and PASTE, and that is what most experience people do, use it as a cut and past tool.
    10.Three months
    11. Trained operator
    12. Nope, just hire the right person.
    13. Only winrunner, and nothing compares to the power of SilkTest, especially if you have long term plans for your testing.

    Anthony


  5. #5
    Senior Member
    Join Date
    Jun 2000
    Location
    Chicago, IL USA
    Posts
    383
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk Survey

    1.How long have you been using Silktest?
    3 years

    2.What do you like about it?
    The 4Test programming language is very flexible and powerful when you learn how to use it.
    No other tool allows me to simultaneously control multiple remote computers for Client/Server testing.
    I can make an automated test for almost anything if I want to spend the time to do it.
    With some planning, tests can be organized in an Object-Oriented framework which allows future changes to
    the application to be easily accomodated.
    The Object-Oriented capabilities allow me to test a very dynamic GUI with lots of "frames" which appear and
    dissappear inside of other windows.

    3.What do you not like about it?
    Technical "support" from Segue is really pathetic.
    Poor and misleading Segue documentation.
    Expensive and useless training classes.
    You really must hire an experienced SilkTest expert to get started with this tool.

    I also dislike the "free" QAOrganizer that Segue pushes and others in this forum seem to like very much.
    The main reason I dislike it is just that it uses a different syntax and much more rigid format than SilkTest itself.


    4.How well does Silktest perform?
    C++ support is rock solid and incredibly fast.
    VB support is not quite as stable, but still very good.
    Early Java support was buggy and slow -- the latest version is twice as fast and much more stable.
    I am often impressed by the performance of the 4Test language core -- whether is is manipulating huge arrays
    of test data or synchronizing multiple threads with remote Agents --
    SilkTest is amasingly fast and stable for a testing tool.

    5.Does Silktest perform your entire set of tests?
    Automation is not practical for many tests -- especially when the GUI design is always changing.
    I believe that SilkTest is CAPABLE of testing almost everything -- given lots of time and money.
    SilkTest is also not designed for heavy-duty load testing. (I do find plenty of load bugs with it though)

    6.What kind of tests do you use Silktest for?
    Regression Testing.
    Functional testing of Client/Server scenarios.
    Multiple user simulations -- sometimes lasting several days.

    7.How easy is it to interpret the results of the automated tests?
    If the test is designed properly - very easy.

    8.Is Silktest now integrated into your work processes and standard procedures? If so, how much effort and how long did this take?
    Standard Procedures??? -- we work with Java developers :-)

    9.Were there any benefits or problems in using the tool, which were not anticipated?
    Problem: The early Java support was buggy and unstable.
    Problem: Segue "support".
    Benefit: Once we had a clean SilkTest application harness for basic regression testing, we found that we could use this
    same harness to set up a variety of complex scenarios and simulations.
    These tests could not have been done manually, and have revealed important bugs.

    10.How long did it take you to achieve real benefits?
    2-3 months

    11.What are the critical factors for achieving payback?
    Test developers who can use the tool at an advanced level.

    12.If you were doing it over again now, would you still purchase this tool? What would you do differently?
    Yes.
    I would also get SilkPilot for Java API testing.

    13.If you have used both Silktest and Rational Robot, how would you compare the two products?
    (functionality, similarities, differences, overall feeling about each product)

    I have not used Robot.
    Our team does use Rational ClearCase and Rational DDTS (with SilkTest).
    The Rational products have a great design which allows developers to work quickly and intelligently,
    but they lack the stability required for production (and testing).
    I no longer accept developer builds designed to run under ClearCase, because ClearCase is not stable enough.
    I suspect Robot suffers similarly.

    [This message has been edited by RobKapteyn (edited 01-29-2001).]
    Rob Kapteyn, CSTE
    RobKapteyn@gmail.com

  6. #6
    Senior Member
    Join Date
    Sep 2000
    Posts
    159
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk Survey

    [QUOTE]Originally posted by Tony Mrozinski:
    [B][b]
    4.How well does Silktest perform?

    Issues with reliability and stability of Silk host and agents. We estimate 90% reliability in long duration unmanned automated test sessions; i.e. a Silk host or agent will crash at some point in a test session 10% of the time.

    Holy Moly. IMO, That type of failure rate is NOT acceptable for such a high-priced tool!

  7. #7
    Guest

    Re: Silk Survey

    Raul:

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Holy Moly. IMO, That type of failure rate is NOT acceptable for such a high-priced tool!<HR></BLOCKQUOTE>

    My point exactly. Segue's typical response is that these problems would not occur if we bound the Silk Agents to TCP/IP instead of NetBIOS. However, as mentioned elsewhere in my survey input, TCP/IP is not an option for our particular test environment.

    Some of the problems that we have found in Silk have been solved with patches provided by Segue. However, it took a lot of threats to get Segue's attention (one of the advantages of working for an "800 lb. gorilla" is the threat of potential lost sales to other software QA organizations within the same company who may be interested in purchasing Silk for their testing efforts).

    You'll notice that I didn't answer question #12 of the survey. I was not involved in the test tool selection phase for this particular project, but knowing what I know now, I would not have chosen Silk.

    ------------------
    Tony Mrozinski
    Sr. Software Test Automation Engineer
    Home Networking Operations
    Intel Corporation

  8. #8
    Senior Member
    Join Date
    Apr 2000
    Posts
    401
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Silk Survey

    Hi Just answering the 13th question. See the Robot Forum for my Robot survey.

    13) I have used SilkTest three times (total about a year upto 5.0.3 I think) and Robot. My feeling is this if you have Robot and Silk in a room and they both recognise the objects in your application. Go with Robot (shock horror) why? Robot is easier to learn. Two, ok I mentioned that Silk is more functional it certainly is out the box but if you know VB just create your own classes with all the methods you want, save to an ActiveX dll and hey presto you can add almost limitless functionality by using the CreateObject function. If you can get TestStudio then it will have tools to keep you happy for a long time.

    Hope this helps

    "Come with me if you want to live" Terminator, T2



    The Test Force is strong in this one

 

 

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 12:52 AM.

Copyright BetaSoft Inc.