I'm currently conducting an evaluation of various automated test tools for the purpose of determining which tool is most compatible with my organizations needs. All responses to the following questions and any additional comments would be very appreciated by me and I'm sure others would also find your answers and comments very useful. TIA
1.How long have you been using Robot?
2.What do you like about it?
3.What do you not like about it?
4.How well does Robot perform?
5.Does Robot perform your entire set of tests?
6.What kind of tests do you use Robot for?
7.How easy is it to interpret the results of the automated tests?
8.Is Robot now integrated into your work processes and standard procedures? If so, how much effort and how long did this take?
9.Were there any benefits or problems in using the tool, which were not anticipated?
10.How long did it take you to achieve real benefits?
11.What are the critical factors for achieving payback?
12.If you were doing it over again now, would you still purchase this tool? What would you do differently?
13.If you have used both Robot and Silktest, how would you compare the two products? (functionality, similarities, differences, overall feeling about each product)
1) 2 half years (Cyrano previously)
2) Easy to learn from the box and the ability to create your own objects from classes you have created in VB, the datapool function for generating test data.
3) Not very good with the MFC applications I have tested like Roguewave stuff.
4) This depends what you mean but it replays everytime (if you coded correct in the first place!).
5) Nope there are some things I have to view visually and if it can't recognise certain objects I tend to test those manually.
6) Functional web, client/server and database validation tests.
7) Very easy but again this depends on what you write to log.
8) Yes but I am a consultant so I tend to create then hand over but if you want say a full robust framework that allows tests to be run from a UI say through access, web front end and to access business object functions, database and to be driven from a spreadsheet then in excess of 6 months easy but for say a regression suite of a basic application then 1 - 2 months. Again it is all relative.
9) With most automated tools the problem lies with whether you can recognise the majority of the applications objects from the box. If you can then you are there it will only be time allowed to automate that will hinder you. My problems have been with MFC support.
10) Usually around 2 months before the benefits show (for me).
11) The number one is make sure the tool can recognise the majority of the objects in your application (trust me you don't want to base your automation on screen co-ordinates it creates many a shelfware). Number 2 ensure sufficient time is given, automation is a full time job. Number three create frameworks or at least data-driven test suites integrated with products like Access or Excel.
12) I like Robot in fact I like all the tools. I would purchase Robot again if as stated above it recognised the objects. Some advice. The big three are Compuware, Rational and Mercury. Segue offer a good product and should be considered in evaluations but who will be surviving in a few years time (who knows) but Compuware turn over in excess of $2billin a year have a large professional services body covering every platform there tools range can only be matched by Rational. Rational are very good (the best some would argue I do) for Object based development using techniques like UML. They have a wealth of tools they should survive. Mercury are very strong and in the UK are way the market leaders but to maintain this position they will need to diversify/acquire or increase the level of functionality to go way beyond the other 3. There share price has been strong but when tools become more common they will need to add more value added services and products to maintain this.
Segue. Segue's product is very innovative and the most functional of the tools but as a company they betted big time on e-commerce and as we can see the market has not lived up to the hype (in UK). To me the best thing is for them to be brought by a bigger company that is missing test tools in it's tool set. However I can't see Rational or Compuware doing that as they have their own sets to maintain! Nor Mercury who could capture their customers in the states. The support in the UK could be alot better.
13) I have used SilkTest three times (total about a year) and Robot. My feeling is this if you have Robot and Silk in a room and they both recognise the objects in your application. Go with Robot (shock horror) why? Robot is easier to learn. Two, ok I mentioned that Silk is more functional it certainly is out the box but if you know VB just create your own classes with all the methods you want, save to an ActiveX dll and hey presto you can add almost limitless functionality by using the CreateObject function. If you can get TestStudio then it will have tools to keep you happy for a long time.
Was this a web screen? If yes ensure that you launch the browser by using the Robot web browser wizard (to launch web apps) or it will not recognise.
If not explain more about the application you are testing.
"Every time I try to talk to someone it's "sorry this" and "forgive me that" and "I'm not worthy"..." God, The Holy Grail