| || |
Automation Done Right
I am a recent member of QA Forums and trying to get my footing on how/where to post, so please excuse my transgressions. Following is the problem that I am trying to solve:
I am currently testing a Java Swing Application, and trying to identify the proper automation framework. I have reviewed several commercial applications such as QTP, WinRunner, Test Complete and Functional Tester, and found them to be mostly satisfactory, but not optimal, as they are:
2. Lock the user into predefined scripting languages.
3. Expensive (Either in upfront cost, or in maintenance).
I have read a lot about testing frameworks, keyword driven testing, and data driven testing, but have not found a solution that fills the following desired criteria for a testing tool:
1. The tool should be open source so it can be extended.
2. The tool should be platform neutral.
3. The tool should support automation using any/majority of languages.
4. The tool should be capable of automating a wide range of application types such as:
a. Web Based Applications
b. OS Based application (e.g. Windows, GTK, QT)
c. Hardware level drivers
5. The tool should be capable of automatically discovering the application objects.
6. The tool should be capable of:
a. code coverage analysis.
b. Application performance/profiling analysis.
c. Application security analysis.
I am aware of test projects that strive for most or all of the criteria above (e.g. TPTP or STAF), but am curious as to the collective experiences of the QA Forums members in achieving such a utopia. For instance with TPTP and STAF, it appears that these frameworks are being used to produce high quality applications, but are mostly being used by IBM which is the sponsor of the projects.
The documentation for these projects is sub-standard for a community at large (though I suspect that it suffices for their internal needs). It does a poor job of outlining how the applications are to be used, and how to extend them.
It seems to me that the Software Test Community should be striving for such a platform/language independence, and if something does not already exist, the community should be pushing (Nay, I say Shouting!) for projects and applications that meet these needs.
Re: Automation Done Right
I've not seen a single tool that can do everthing on your criteria list.
On the other hand, I've never needed a single tool to do all that.
Since every shop has different needs, it would be hard to rally a community to "Shout" for these specific needs.
Re: Automation Done Right
The only way I know of doing all the stuff you want to do is to use Perl and/or Python. There are lots and lots of modules to plug in that will give you the benefits that you want.
[disclaimer, this is windows nt based OS centric (nt, 2k, 2k3, xp, etc – linux is different)
But as Joe says, I cannot think of a single shrink wrapped product that does all this stuff. (and tbh if there was one, it would be pretty expensive and probably more work than developing custom throw away scripts [my personal favourite mode of testing])
Also, doing driver related stuff is ring-0 (kernel space) you would really need a tester that knows about these things to get this done. most test tools I have seen operate ring-2, user-space – in fact I cannot think of a single application that could do this. however the GUI::Test module for perl will allow you to sorta spoof this by directly manipulating the windows messages and signals (although I have never tried this and generally stay as far away from ring-0 as I can)
And I have heard of, yet never used things like this
So even if you had this tool, you would need someone of pretty exceptional ability to use it, so just get them to write their own [img]/images/graemlins/smile.gif[/img]
probably not the answer you are looking for, but I hope this helps.