| || |
Web Testing Tools Evaluation
Anybody evaluating ATT for Website Testing?
I have just had a telephone demo from RSW/Empirix.
Very easy to use. On our shortlist.
Anybody else evaluated this product, or even choose it, or not chose it. Why ?
Any examples of prices you may have would be useful.
Re: Web Testing Tools Evaluation
I am just finishing up an evaluation from Empirix and Segue at my current client. Empirix is definitely easy to use and is getting quite robust with their integration of VBA.
Why would I choose it? For the ease of use, the quick uptime, and the relatively quick ROI. Why would I not choose it? I would not choose it for environments where I have to build up robust test engines (although Empirix is fast approaching Segue in this arena, in my humble opinion).
Does e-Tester support a back-end test engine approach? This is important depending upon how the automation effort is going to be established. But, if a test-engine approach is sought (which is something I usually do) - I am not sure at this point how well e-Tester will do that. (This is mainly because the start/stop/execute portion of multiple scripts is mainly handled within the e-Monitor tool it seems like. SilkTest has something similar with QA Organizer - but the key is you can override the QA Organizer and simply use code to replicate what it does. WinRunner falls into the same problem, but not as much, because it uses TestDirector - but you can still use TSL code to make an event-driven test engine, just not very well.) One way I can see to do a test engine approach with Empirix is to use the VBA ScriptBegin, ScriptEnd, PageBegin, PageEnd logic in e-Monitor and the afterPlay and beforePlay VBA in e-Tester. But, again, this only matters if you care about test engines.
One thing I do not like is that the VBA integration does not extend across their tools. So, for example, e-Monitor and e-Tester have separate VBA implementations whereas with SilkTest, for example, I could do the same thing those two tools do via code.
The data bank wizard with Empirix has always seemed more trouble than it is worth to me (from a true automation perspective) although I have not made up my mind on this yet. I could write a three or four line piece of code in SilkTest or WinRunner that would do the same thing as the data bank wizard but in about half the time. (Also what about accessing other formats? Can the data bank wizard access anything other than comma delimited files, such as Access databases, Excel files, etc.? I am not sure on this yet.)
Also Empirix does not appear to allow a division of test cases (except in its own programmatic fashion via the Visual Script). Thus how do I match up my requirements in a test-case driven approach? To be fair, most tools do not do this. The best one that does is SilkTest which has a function called "testcase" that encapsulates the data and behavior of whatever you call a "test case" in the script. The QA Organizer that it comes with syncs up with this. The e-Monitor and e-Tester do not have that. They basically treat one whole test script as one giant test case. I suppose you could have code that would not run certain elements of the "test script" - but that means you have to change that for each and every script in e-Tester - not in the e-Manager. (WinRunner tries to emulate this with its tl_step function but does not really succeed. Visual Test, as another example, allows you to define a function in Visual Basic called "testcase.")
I am also still trying to work out how robust Empirix is for exception handling logic. I realize the tool will show me something that failed. But let us say the link on a page is not there. Now, I want the rest of the test script to execute (perhaps it is running unattended at night). So how does eTester get me to the rest of the script? (In other words, does the product support base states - a key component of automation. Base state does not, of course, mean baseline. A baseline is something that eTester appears to do very well.) The idea is I want my exception logic separate from my scripts - that way I can change my exception logic without necessarily modifying everything else. I talked to Empirix about this and it seems that for true exception handling I have to rely on two programs: e-Monitor and e-Tester. This is instead of relying on code in just one. It does not sound like the product has the concept of a base state.
But, having said all that, I find the tool very robust (particularly if you use Internet Explorer as your base for the automation) and the training time on it is absolutely negligble - it is very easy to get up and running immediately. And their code integration with VBA promises to get better and better.
Re: Web Testing Tools Evaluation
Can you explain a bit more about what you mean when you say:
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>One thing I do not like is that the VBA integration does not extend across their tools. So, for example, e-Monitor and e-Tester have separate VBA implementations whereas with SilkTest, for example, I could do the same thing those two tools do via code.<HR></BLOCKQUOTE>
I don't quite understand this, but would very much like to know more, as I am also currently evaluating these tools.
Also, how can a tool differ in its implementation of VBA? Either you're using/implementing VBA, or you're not; if they're using VBA then they've licensed it from Microsoft. From what I see, they're using something that is "like" VBA, but not really VBA (which explains the variance in their implementation.
[This message has been edited by TG (edited 03-13-2001).]
Re: Web Testing Tools Evaluation
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>Originally posted by TG:
Also, how can a tool differ in its implementation of VBA?<HR></BLOCKQUOTE>
You can offer different subsets of the functionality within different products. That is a different implementation. If the two VBA instances in the two products do not communicate those are separate implementations - even if they are the same product. I can have Windows installed on two different machines - just because they are both the same version of Windows does not mean they are the same implementation of it.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>From what I see, they're using something that is "like" VBA, but not really VBA (which explains the variance in their implementation).<HR></BLOCKQUOTE>
Well, it is a VBA license - so it is like VBA in the sense that it is VBA. It is, of course, modified to suit some of their own commands. When I referred to its implementation I was not referring to whether it actually was VBA or not, but rather how it integrates with each instance of VBA within the products.