Selecting Automation Tool
15 years ago I would go on a new project and see what tool would recognize the objects in the applications that i'm trying to test. The tools seem to have improved so now I would have a larger choice of candidates.
How does one now decide on what product to use? I'm most comfortable with QTP because I have been using it for a long time, but i don't think it may be a good enough reason to stay with it.
Last edited by bklabel1; 04-23-2013 at 07:17 AM.
Reason: Changingthe Title of the post
Give Ranorex a try.
I used QTP for years and then moved a across to Ranorex.
I appreciate the informaiton. Do you know if it integrates with Quality Center of I would need to?
I apologize that the title of the post thread does not match the topic. I edited the body of the post without knowing that I could not change the title.
You determine what is important to you and your team. Then you look for tools which best match your needs.
How does one now decide on what product to use?
Here's what I look for...
Run in my environment
Usable by my test team
Be generally more efficient than strictly manual testing
Detect changes in the System-Under-Test
Create Smoke Tests which run after every build
Automate the boring, repetitive stuff
Run predictably and repeatedly
Run some load, stress, and volume tests
Isolate failures easily
Run many tests, in spite of unexpected failures along the way
Start wide, build depth later
Automate what users do first (Getting Started Manual?)
Isolate the maintenance effort
Produce "readable" scripts
Ability to reset the environment as needed
Avoid false failures
Extensible - since we cannot predict all uses
Survive trivial changes to the System Under Test
Validate during tests, and at the end as appropriate
Ability to select and run subsets of the entire test suite
Ability to select and skip particular tests
Variable log levels (Verbose, Normal, Minimal)
Minimize dependencies between scripts
Minimize learning curve
Minimize maintenance time
Minimize post-run analysis time
Minimize dependence on golden machines
Record and Playback capability
see: All Things Quality: Things I Like to Have in my Test Automation Suites
From a test developer's perspective, when I approach tool evaluation, I look at the following in this order.
1) Can it recognize and interact with the objects in my AUT. (The reason I put this on top is this will be an automatic deal breaker. If the tool can't work with your AUT, then you might as well just build a bunch a test hooks and forget the tool all together) I think QTP and Rational is good in this area. They both try to support a wide range of windows objects.
2) How hard will it be to integrate this with CI. (automatic deployment and running it from the command line) For example, a solution that requires you to enter a license key does not make it easy to just check in your entire setup and deploy and test on a blank machine or VM image in one go. (I'm a strong believer in CI, if the tests are only run in QA, you are catching the bugs late and losing about 80% of the automated test's effectiveness. Ideally you want to catch bugs in the integration cycle where they most likely occur and save the devs time debugging the issue.) Is the tool easy to deploy cross platform? I think Selenium excels at this using Mavenized builds.
3) How hard will it be to scale the solution. (This is where I think QTP is weak) Let's say you're running it in CI with many parallel instances, how much does it cost to license each additional machine or VM image? For example Test Complete offers a cheaper runtime license (they separate running the tests via command line and their development license that has full access to the IDE) Selenium being a bare bones open API can be massively scaled since licenses are free. It really sucks when you have 1000 or so tests, but can only afford 3 or 4 machines to run them in parallel. The ideal goal of any CI setup is to get as many tests to run in under 5 minutes. When a solution is not cost scaleable, then it becomes cost prohibitive to throw more hardware at the issue (sad thing is, hardware is cheaper than the license of most automation tools over a 2 yr period).
4) Support a layered framework for keeping tests maintainable. Does it have features like object mapping, data driven, flow level abstractions (like keyword or behavior driven testing) that allow you to separate the different levels of logic to make your tests easy to maintain. (The reason this is this far down is because this can be handled by creating a framework on top of the tool, although creating a framework will either be timely or costly depending on if you use a 3rd party framework or waste dev cycles building your own)
5) Easy to use. (it's strange I put this so far down, but most testers are very tech savvy) I think most good testers are pretty quick learners and can adapt. Although that said, time is money, and if a tool isn't saving you time, then it's not a good tool.
6) Cost of license. (Reason I put this last is even some of the most expensive tools it's still not that big of a deal). If you think about the cost of a tool compared to the cost of your time as the salary ranges of the people using the tool, it really doesn't make sense to penny pinch in the case where 1 tool is superior to another.
Last edited by dlai; 04-15-2013 at 05:56 AM.
Ranorex have a blog that covers intergartion with a number of Test management systems.
There are certainly some frustrations with Ranorex when you compare to some of the "Premium" (expensive tools)
There are also some huge upsides the IDE and development environment is very good with lots of power and potential VBScript in QTP cannot compete with this. Object recognition is more complex in Ranorex it is also more powerful and once you understand it it is very flexable.
I have new hire here who was also a QTP Automator for many years and he loves the power and flexability of Ranorex. I know of others who have come across to Ranorex as well, none of them are keen to return to QTP.
Download a Demo version do the tutorial so you can understand how it works and I doubt you will go back to QTP.
I want to understand your post, but I am not familiar with the CI acronym.
CI = Continuous Integration. It's a common practice to have upon every developer checkin for the code to compile, unit tests run against it, packaged, then kick it off to automated deployment to a test system, then have automated tests executed against it in a completely automated fashion.