| || |
evaluation comments of QuickTest 5.5
These are my Astra QuickTest (AQT) evaluation comments.
I evaluated the downloadable AQT [not the Professional
variant], version 5.5, during the first week of
November, 2001. I spent 25-30 hours in this evaluation,
so I would not consider it extensive. I would have
spent more time, but I really was not impressed with
this products features, as well as its pricing relative
to its limited capabilties.
AQT is a very good product for rapidly producing web
application testcases developed through the Microsoft
IE browser, if/when you only need the testcases for the
duration of a single software release, and you are
not attempting to do team development of those tests.
Rapid development is achieved through the traditional
capture/playback technique. The resulting testcase
is nicely presented in an iconic Tree View [by
default]. The "initial" recording is generally easy
to read and understand, as long as AQT selected a
sensible Logical Name for each Page and Object
interaction, which are recorded as Steps in the
Tree View. But as issue #9 below indicates things
get pretty messy in the Tree View when you attempt
to add more recording steps after the initial
recording has been completed.
While several versions of the Netscape browser are
also supported in addition to IE browsers [version
5.0 and above] it is not clear that you can record under
one browser and execute under another [and there are
very few programming things which you can do to
accomodate cross-browser differences at runtime].
The underlying scripting language [which you can easily
use by switching to the Expert View] is vbScript.
This is much more powerful than WinRunner's TSL
[roughly equivalent in power to Segue's 4Test]. But it
is difficult to exploit this scripting power due to
the AQT vbScript runtime environment [discussed in
greater detail below], as well as the difficulties of
dealing with objects that are [or are not!] in the
Tests are constructed in a hierarchy of Actions.
Actions include Steps. Steps can and will be nested
as the user navigates the application under test
during recording. The developer decides if and how
to use multiple Actions. Actions from one test can
be called/reused in another test[parameter passing
between Actions is generally done through the
Global DataSheet--which is very primative]. The
contents of an Action live in their own [vbScript]
namespace, so functions defined in one Action are
not reusable in another test's Action [code
reuse is discussed in greater detail below in
The Object Repository is roughly equivalent to WinRunner's
GUI Map, but much less flexible. You can't manually
add Pages/Objects to the Repository. And more importantly
you can NOT change the Logical Name of an object [which
is assigned by AQT when the object is learned during
a recording session].
Some specific issues, along with examples, are listed
[in no particular order] below:
1. Tests which include Actions from another test, reference that test's file structure with an embedded fully qualified pathname reference. Therefore multiple developers, each on their own PCs, must work in the same drive letter/dir structure in order to share actions. [Also refer to item #6 on issue related to source code control].
2. You can't define/modify the logical name of an object/step's property. The one selected by AQT is the one you have to live with. For example when recording against the Search Flight Results page on the Mercury demo web application clicking on a destination flights radio button recorded a "buttonX" logical name [where X is 1 - the max number of buttons in the list of flights], rather than the name of selected flight.
3. Error messages are not very informative. A example is the fact that if the Page property is incorrect [which can intentionally be messed up for eval purposes using the Object Property dialog] AQT consistently failed in the first object reference on that page [for example pushbutton click] indicating that the page's object failed [when in fact the Page could not be found].
4. The manuals fail to contain information about many things you might want to dig into. For example you can't find any description of the "miclass" Property, and therefore you can't figure out what are legitimate Value settings. [This is especially puzzling when the tool records references to all Pages as Property: miclass, Value = Page; it just seems to be saying that any loaded page will suffice for object recognition purposes].
In some situations you are left to ponder how you access data with some of the AQT objects, using their object model documentation. For example it is not clear [at least to me] from the documentation that you read a setting object's value with a default reference of setting("settingName"), while the documenatation provides descriptions on how to create an new value and delete an existing one.
5. If a series of Actions are created in a testcase, you can only have one expanded at a time. When the testcase is played back you can't see which steps are being executed as each action is played back [you only see the execution arrow pointing to the action]. This is very frustrating when waiting with timeout for something to fail.
This is even true when you have expanded an Action for visual display. If you then select the Test:Run Action From Step... command AQT colapses the Action and then starts execution.
6. Tests created by AQT create a signficant number of files and directories "behind" the scenes. WinRunner also does this but AQT creates many more. For example I create a very simple test with only 3 user interactions which simply logs into the Mercury demo website. The following dir structure and files were created under the tests name in the ..\Tests\loginTest dir:
This list grows rapidly as more pages and objects are recorded and new Actions created. This will make it very difficult to put tests under source code control.
7. A test's source code can only be viewed using the the AQT editor [i.e. the *.tsp files are stored in a proprietary binary format]. This may make it difficult for those who need to document tests. You should not need a licensed copy of AQT just to view code.
8. The AQT hosting environment [sometimes referred to as a vbScript "host platform" like Microsoft ASP and WSH] does not include an "include file" feature. This means you will have to perform non-intuitive workarounds to hack together reusable code libraries.
9. Maintenance of scripts: attempting to record additional Steps within a previously recorded Action leads to puzzling results. Many additional and unnecessary pages are placed in the Object Repository as well as several Steps in the Tree View.
For example record a script which simply logs into the Mercury demo web application that brings up the app, enters the login ID and password, and then clicks the LOGIN button. You initially have two pages recorded, the 1st Page ["Mercury Tours"] has 3 objects while the 2nd page ["Welcome to Mercury"] has just the Sync() method recorded. If you then highlight either the 2nd Page or the Sync() Steps and re-record a single SIGNOFF button click on the 2nd page, AQT [consistently] records the following Steps/objects:
a) another Browser Step [this makes no sense]
b) under which is a "Welcome to Mercury_2" Page [whichis identical to the original 2nd Page]
c) under which is the "Signoff Button" Click() step.
d) also under the new Browser Step is a new "Mercury Tours_2" page [which is identical to the original 1st Page] and it contains a Sync step.
Therefore by adding 1 additional button click Step [i.e. clicking on the 2nd Page's SIGNOFF button] resulted in 7 new items being added to the Object Repository!
The Tree View is also cluttered with these additional Steps/objects. And the test will not run if you just move the button click Step back into the original 2nd Page [where it belongs!] and delete all the other un-needed Steps--I think this might be feasible if you could directly edit the object definition into the Object Repository, but that is not permitted [you can only fiddle with existing object Property values using the Object Repository dialog].
The bottom line: design a testcase with lots of small Actions. Then if you need to re-record a portion of the test delete all of the Actions steps and re-record all of those interactions/Steps again. [From the NEWs groups it appears this is what Mercury recommends].
10. Pricing is absurd: as of 11/2/01 AQT is priced at $4400 a seat while AQT Professional is $6600 [which is the cost of WinRunner with WebTest]! I can't imagine why any customer would pay the same price for AQT Professional, rather than buying WinRunner/WebTest.
[This message has been edited by qadog (edited 11-05-2001).]
Re: evaluation comments of QuickTest 5.5
For another [more positive] slant on this tool checkout David Gilbert's impressions, based on 3 years experience, in the following post on the WinRunner board:
Re: evaluation comments of QuickTest 5.5
Included are supported by VBScript but it is tricky because there is no function include.
The include source code work well while debugged!
Try a search engine i am sure you will find how to do it...