Record & Playback Advice
I work on a fairly non-technical QA team, and I've been given the task of locating a Record&Playback tool that may (or may not) provide a positive ROI for automated testing of our web application. I'm certainly aware of the limitations of relying on R&P for test automation, but I'm looking at this as a kind of proof of failure.
The only requirements for the tool are: (1) R&P feature, and (2) must be simple enough that non-coders could operate. I'm using a Mac, but I'm willing to hear arguments in favor of the Windows-based tools below.
The tools I've looked into so far:
- Sikuli: Didn't suit my purpose as it's not a true "record and playback" tool; also verification seems tricky.
- Squish GUI Tester: A little tricky for non-coders not used to an Eclipse-like environment - i.e., it was difficult to just "record and playback".
- Selenium IDE: I like this the most so far. Easy to get up and running, and the integration with more robust automation in Ruby is a bonus. The downside is it's Firefox-only.
Ones I haven't tried because they are Windows-only:
- IBM Rational Functional Tester
- SmartBear TestComplete
- HP Unified Functional Testing (Formerly QTP)
- Borland/MicroFocus SilkTest
Another downside to Selenium IDE is that I can only work with web-based elements. For example, if I want to automate "changing a profile picture in Facebook", I can't really choose an external file to upload and verify that it got uploaded. This is a serious limitation.
What do you guys think? Am I right in thinking Selenium IDE is kind of "as good as it gets" as far as Record & Playback? Or is there something I'm missing in the tools I've tried, or those that I haven't?
I know these type of threads can get annoying, so feel free to offer just a few words, anything that might help make a decision.
I would pick selenium builder Selenium Builder - The next evolution, over selenium IDE. IDE is an old product the creators want to kill. They just keep it on life support since too many people are still using it. Selenium builder is built to generate modern webdriver standards code, which you can export to different programming languages and use in what ever test framework you decide to run it in.
Thanks for responding David. I did see people pushing Selenium Builder over IDE, although it looks like it doesn't have support for basic flow control (as IDE does via plugins). I could be wrong.
Our web application is written in Ruby/Rails and operates with HTML, CSS, and JS on the front end. As you said, "selenium may not be the best" if our application uses embedded objects - does this mean that there is a different R&P tool that does offer support for this type of functionality?
On the question on flow control:
Originally Posted by rsh
In theory, tests should not need flow control because you are dealing with fixed inputs and expected outputs. Usually the only thing that switches is configurations (like switching enviornments, browsers, and user accounts), which you would then just export to code and replace the values with config values pulled form some framework. For example, I wrote WTF to do such a thing, https://github.com/wiredrive/wtframework
Of course if you still need flow control beyond that, you're still able to use if/else statements in the exported code.
On the question on embedded native control:
On the Question on Tools:
If you have a heavy amount of Native controls, like a Flash or SilverLight app that's embedded in the web page. Then you may consider tools more targeted at Native applications such as QTP, TestComplete, and the like..
If you have problematic controls, such as custom graphics and widgets like you see on games. You may opt for an image recognition tool such as Sikuli.
Last edited by dlai; 09-25-2014 at 10:01 AM.
Agreed on the concept of flow control. I was using it for initial setup and cleaning up between tests, which typically would be outsourced to some kind of test harness. With this particular experiment, I'd like all the testing (or as much as possible) to be accomplished through Record & Playback.
As far as native controls go, the AUT is an image manipulation application. So there are embedded JPG/PDF and the ability to upload/import new JPG/PDF (through put requests to AWS). Everything else is done in Ruby, HTML, CSS, and JS. In your opinion, could Selenium handle these cases? If it can, it doesn't seem obvious to me. For example, if a user imports an image and I want to look at the PDF and verify it's correct.
I don't think you'll find the perfect fit between ease of creation, and accuracy of validation, and range of capabilities.
There is a trick to file uploads in selenium, it requires getting your hands a bit dirty with code, https://saucelabs.com/resources/selenium-file-upload,
Some who prefer less coding may use an external script recorded in their tool of choice. Personally I don't like that approach because it's hard to deploy tests using native automation tools on the cloud effectively without involving VM's and licensing servers.
I think with the image verification, you'll most likely will have to write your own validation no matter which tool you use, except the image recognition based one. Those however have a serious weakness when it comes to validating numeric and generated text data, since you then have to start involving OCR (optical character recognition) into the verification which is less reliable than straight property compares.
Thanks for all the advice, David. What you've said makes a lot of sense. In the end, I think the task of enabling non-coders to efficiently automate a system is a bit of wishful thinking. For the amount of work that goes into learning a tool and all the workarounds required for the validations, why not just learn to code? I suppose at least with a tool like Builder, the non-coding QA could use the R&P feature to generate some basic webdriver scripts to give the coders something to work with. Then they are still contributing to the testing process.