SPONSORS:






User Tag List

Page 1 of 2 12 LastLast
Results 1 to 10 of 13
  1. #1
    Member
    Join Date
    Jun 2000
    Posts
    50
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Conclusion: SilkTest not worth the money

    Hey there !

    This is not meant as an inflammatory thread, simply as my conclusions regarding SilkTest. I hope that they might be useful to anyone considering spending time and money on this product.

    SilkTest is not reliable. I have trouble understanding companies endorsing SilkTest as the most reliable, scalable, insert your favorite adjective here, testing tool out there. Relative to other products it might be the best, but SilkTest's best is well below an acceptable level. I've been using SilkTest to "test" a financial web application. Our application is not simple, and uses many new web technologies. Granted.

    Over time I found myself trying to fix SilkTest performance instead of focusing on the application being tested. I won't go into details, but I will only say that I have had enough. I recommend that you seriously reconsider purchasing SilkTest in favour of more traditional QA approaches, such as hiring humans. They cost less, offer much better reliability, and they won't need your attention 24/7 after initial training.
    Welcome back the days of manual testing !!

    As a side-note, the testing tools offered by SilkTest on an out-of-the-box basis purely, are basically useless to a company running a serious web application. Squeezing some use out of SilkTest requires adept and extremely patient script programmers that are constantly able to revise their code and program around SilkTest defiencies.

    I repeat my pledge that SilkTest, at the current price at which it is offered by Segue, is not cost efficient. Unless some serious improvements are made in the following version or two, I would seriously consider abandoning SilkTest in favour of anything else available, even if the alternative is to test the web site manually.

    I also would like to mention very, very poor technical support on the side of Segue. They don't deserve your money, if only for that reason.

    Good luck to all SilkTest users,
    Nir Gilboa.

  2. #2
    Senior Member
    Join Date
    Jul 1999
    Location
    New York, NY, USA
    Posts
    137
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    WOW! I can't wait to see your reply after 3 months of Rational or Mercury. Silktest is the most complex of the testing tools out there and requires sophisticated testers who know scripting and the product. It is by far the most useful automation tool out available for most applications (Client/Server & Web). The other major tools also have their places so please don't think I am panning them.

    As for cost, in the end run they all cost about the same and are worth about 1-2 weeks of a good QA persons salary. I don't think that is a very high cost for most real companies.

    As for automation and web applications. If QA is properly involved in a project from the design phase on, it is possible to use automation to their advantage, but they must have clear goals of where it fits in and where it is a waste of time.

    Most web applications are being forced onto the market with little more than Unit testing being done. There is no forethought as to the impact this will have on the user community and things tend to get fixed on the fly. This is because of the misconception of 'internet time' an idea who's time is nearly up. True QA is more of a risk management process than purely a testing process. All QA persons need to determine if their efforts are improving quality and reducing risk, the Internet community wants to release and the risk be damned because of fear of being late or last. The funny thing is now watching all the first to markets fail, close up shop or see their stocks become truly market valued (see Priceline for a great example).

    If you are well versed in the financial markets (you should be if you are responsible for complex financial software) you must know that there are too many complex factors to be traced that can be easily overlooked by non-sophisticated testers. Automation is the most efficient way to test for correct results when there is no tolerance for error.

    As for your quest for a different tool, they all end up costing the same, but there may be a more appropriate tool for your testing purposes and style. However, if you take the same approach you have with Silktest (that is expect it to do all the work for you) you will fail again.

    As for Silktest's customer service, it's bad but it's getting better. Most of the experienced Silktest users I know help each other to a great extent. Places like this forum are a godsend for an automation users community and the people here can be very helpful. Maybe if you had expressed more of the difficulties you were encountering you may have found simple solutions to your problems?

    Good Day and Good Bye Nir.

    TAW

    <BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Programming is the ultimate way to get even with a computer, you are in charge. You tell the beast what to do, and it will obey you, even when you tell it to do something stupid. Computers are fast and obedient, not smart. Dan Gookin<HR></BLOCKQUOTE>

  3. #3
    Super Member
    Join Date
    Jul 1999
    Location
    Rancho Santa Margarita, CA
    Posts
    1,439
    Post Thanks / Like
    Mentioned
    4 Post(s)
    Tagged
    1 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    nirg,

    If you ONLY have 100 or less Test Cases or if you don't mind going thru your hundreds or thousands of tests each time you get a new release of your AUT(s), then you should not hesitate to go back to the way testing was being done in the 70s and earlier.
    In short, if you or your employer can afford to spend weeks or months to complete your Testing Cycle instead of hours or days (once you have your automated scripts done), then you should go back and do your test Manually and enjoy the extra money you save from not using any Test Tool.

    [This message has been edited by Gilbert (edited 10-13-2000).]

  4. #4
    Member
    Join Date
    Jun 2000
    Posts
    50
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    Thanks Gilbert.
    Unfortunately the damage if done, if I may call it that I don't think my comments are overly harsh towards SilkTest and Segue.

    There is no doubt in my mind that test automation is much more efficient than manual testing. However, I think you would agree that the frustration of running trough thousands of tests does not even come remotely close to that experienced when using a defective tool.

  5. #5
    Senior Member
    Join Date
    Sep 1999
    Location
    San Jose,CA,USA
    Posts
    340
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    nirg

    Sorry pal but I guess some will be inflamed. I feel like you have a few valid points. Yes these tools are tremendously overpriced. I totally agree that a software package of this complexity should not cost 4 to 5 times more than the Intel based system that runs it.

    Additionally it is TRUE that when you catch Segue on a major bug they may pretend it does not exist (like Firestone). Even worse they may tell you that a new release is comming that will fix the bug :-) (for anyone who does not know the inside joke .. don't hold your breath)

    Now for the flip side of the coin. Silk is the best in the market. I sincerely believe this is so, most especially after useing Mercury Interactive's Winrunner for the past year. Silk also supports A LOT of different and very dynamic environments and that AINT EASY MAN!

    Also they do it for one of the most demanding users in the Universe namely QA. Let's face it we are code cops, and we are never invited to code parties :-). Additionally if you check out the market for functional test tools(as opposed to load test tools) the market is not huge. Microsoft and Sun essentially looked into it, and decided the ROI (return on investment) was "chump change".


    So yes Silk is horrendously overpriced, yes it has some horrible flaws in it. But it is the best. As for price this is the reality of the situation .. additionally Segue isn't really making a ton of money after that E business solution marketing fiasco that almost bankrupted the company!!! I wonder how many marketing sales execs from Segue wandered the streets of boston with plackards "will work for food"??


    Finally from my perspective learning how to code tests is by far 1,000 times more rewarding than manually testing! In fact I never really did that, and tell perspective employes so. So I guess all I can say is give it some more time and effort. Also get ready for some frustration. Nothing worth doing in Software Today is well documented, or without frustration. Another harsh reality of our environment.

    Use betasoft as a platform to ask questions for learning, and work arounds, and see if you feel the same in 3 months.

    By the way for any one who has the patience and skill to manually or add hoc test reliably, and consistently God Bless you!
    You are a different breed of person than I am, and the industry will always need you.
    I could never do that and so I have to rely on these tool sets. By the way Woodman you were a bit rough on our pal (even though he kind of had it comming). I especially agree with the "Woodman's" last paragraph.

    BTW Gibert very pithy, I thought you had a really good point. I especially dig the humor. I keep seeing these 70's QA dudes in thier brown shoes, white shirts, and dress slacks pouring over these insanely large stacks of printouts. I bet they felt they were on the raw edge of it all!!

    Also nirg don't feel too beat up on I never learned anything from these forums by "playing it safe". You just keep it real man! Honestly after the smoked cleared you do have two very good points, and this community does feel your pain oh my Brother in QA. But if you are looking for a frustration free existance you picked a very strange career!!

    [This message has been edited by rick_weth (edited 10-15-2000).]

    [This message has been edited by rick_weth (edited 10-15-2000).]

    [This message has been edited by rick_weth (edited 10-16-2000).]
    Smoke me a kipper, I'll be back for breakfast!
    Sincerely
    Richard Weth
    Sr. QA Engineer

  6. #6
    Senior Member
    Join Date
    Jul 1999
    Location
    New York, NY, USA
    Posts
    137
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    Thanks Rick,

    BTW, My diatribe wasn't really meant as a flame, just an observation from doing this for the last 7 years. I always try to assist people on this forum (when I have the time) now matter how 'obvious' or repetitive the questions may be. I have also found a great deal of help in the responses of members here.

    What does sometimes gall me is when you give a reasoned response to someone about their problems and do not hear back on the results, some of us have to take our best guess when answering because every application seems to bring it's own challenges. It is nice to know if you helped someone or not, so that if the same thing happens to you, you can make a decision on whether or not to try a particular approach.

    Like what I was trying to say above, QA (to me anyway) is much more of a Risk Management process than just testing. Tools are a good way to reduce risk if applied correctly, but they will not do everything. QA persons need to understand where their tools are useful, and where they are not.

    In closing, Rick thanks for the help you have given in the past.

    Tom

  7. #7
    Member
    Join Date
    Mar 2000
    Location
    Los Angeles CA USA
    Posts
    41
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    I think it illustrates the sad state of affairs when the case for Silk Test comes down to "it sucks less than the competition".

    In all honesty, if my manager or I had been involved in the purchase decision, my company would not have purchased testing tools from Segue.

    My main beef with Silk Test is the amount of work it takes to keep test files (.inc & .t) current.

    I am testing against an ecommerce site that changes twice a week. Granted, not every upload is a major change in functionality.

    But changes do occur frequently enough that manual testing is faster than updating/rewriting scripts in Silk Test.


  8. #8
    Senior Member
    Join Date
    Sep 1999
    Location
    San Jose,CA,USA
    Posts
    340
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    This is to emoon

    You know I have been involved in an environment "like" yours. Where the code and functionality change so rapidly that the frustration level becomes too high.

    But I am curious as to precisely how Segue let you down? I mean if underlying functionality drastically changes, or there are massive Gui changes like two seperate windows become one (or visa versa), how is it that you get to lay the blame on Silk?!!!

    What do you propose as a solution in your critique especially when you and nirg label this generically as "SilkTest not worth the money" .. with the underlying message that programtic QA (or meta programming) can't or should not be done??

    Really I am at a loss. Perhaps YOUR environment is so dynamic and unstable that initially you should test it manually .. ok I can see that. But what of those poor customers?

    How are they supposed to use this companies ecommerce site if it changes as much as you lament? I see much head scratching and frustration from them as well. Do THEY have to read "whats new notes" every time as well??


    Also perhaps some of your design elements could be more dynamic. Additionally perhaps there are techniques you may not be aware of to data drive your tests.

    Such that if you change data in a few data files you can dynamically pick up new links, as well as certain control elements (yes pick list values, check boxes,.. even push buttons).

    I mean this language is so rich that within reason you can write code with a good deal more dynamics than you may be aware of. On the other hand if the site you are testing does change to such a fundemental level, that borders on chaos perhaps you might be reduced to manual test. Hopefully this is a small piece of what you do (though from what you write I doubt it).

    But again let me re-iterate how is it that Segue let you down? No matter how clever the code under these tools I keep seeing a few fundementals, based on where the industry is at with programming languages.

    I have found that with training and persistance you can do A LOT with Silk. So for me as long as I stay in QA it's either the programatic way, or the highway! Thus far my milage has been best with Silk, and I have yet to be "completely boxed in" with any environment for more than a few months.

    Perhaps I have been lucky .. and perhaps you have not been. But I have found that it is possible to change your luck, it all depends on how bad you want to.



    [This message has been edited by rick_weth (edited 10-17-2000).]
    Smoke me a kipper, I'll be back for breakfast!
    Sincerely
    Richard Weth
    Sr. QA Engineer

  9. #9
    Member
    Join Date
    Jun 2000
    Posts
    50
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    Rick,

    As I mentioned before, I'm responsible for the QA of a financial web site, www.assetplanner.com (which you should check out, btw). Since our site basically demands functionality testing, and by the way, I think it always comes down to that unless you are running a purely content-based site, we decide to write tests that would run using external data. Essentially we are free to run our functionality tests with any set of data taken from an Excel spreadsheet.

    Overall I am happy with our approach and I think in the past 3 months we have radically improved the quality of our offering, resolving both technical and so-called logical bugs. Perhaps there is a better strategy, and I would love to hear one, but this is the path we chose.

    Now, I mentioned reliability problems with SilkTest. Let me give you some examples: at one point we have multiple "dynamic" popup windows open. This might not be the best from a design perspective, as I am fully aware, but it's what we needed to deal with. Getting SilkTest to live peacefully with multiple popup windows that would update with new items required an almost insane amount of tweaking.

    Also the test script needs to restart the browser several times otherwise SilkTest would just deny the existence of windows or images, sometimes even links. Things that work when executed alone don't work 10 minutes into the test. Now I am not exaggerating, and yes I've got the understanding of SilkTest and 4Test needed. Just do a search for my posts and you will some of the problems in detail. Some workarounds, like the BrowserPage.GetChildren() trick were suggested by people like John Green, which shows you even the best fall back on a bag of tweaks. (No pun intented)

    Basically the amount of time and effort needed to get the tests we had in mind to work with SilkTest was just way out of proportion. And imagine that next week the interface changes, all your hard work and clever tweaks are obsolete, and instead of using your test tool, which is what you should be doing, you find yourself getting frustrated because SilkTest won't set some window active.

    Now imagine the further frustration when you resolve to call tech support after hours of teeth-grinding and they don't even bother to answer.

    So maybe I was overly critical in my first post, but in my mind getting robust testing scripts out of SilkTest that can really be used to improve the quality of the product at hand entails way too much overhead to be justified.

    Nir.

    [This message has been edited by nirg (edited 10-18-2000).]

  10. #10
    Senior Member
    Join Date
    Sep 1999
    Location
    San Jose,CA,USA
    Posts
    340
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Conclusion: SilkTest not worth the money

    nirg

    I want you to know one very important thing before I answer your post (I am really thinking about the response). The post to emoon should in now way but one be picked up by you. The only thing that bothers me about your post is the Topic label you selected.

    I found that unlike emoon you had some solid concrete technical issues. I want you to know I respect that. I also want you to know that everyone in this forum who sticks with automation, has felt the way you do.

    Believe me I have yelled at the walls in my cube many times cursing Segue tech support, Boston, and the Software Industry. All I can tell you is I'm still here, and I am one of those twisted guys who enjoy "tweaking". It's a challange and it drives me. Now I gotta go to a meeting for my Day Job.

    The most important thing I want you to get out of this is, I respect your position. Also Segue Tech support as I have stated on more that one occasion will not get you from where you are, to where you need to go!

    Smoke me a kipper, I'll be back for breakfast!
    Sincerely
    Richard Weth
    Sr. QA Engineer

 

 
Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.0.9 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Questions / Answers Form provided by vBAnswers (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominatevBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 11:00 PM.

Copyright BetaSoft Inc.