| || |
Regression Testing Frustrations
I am in the process of putting into place a structured regression testing policy into a company that has a demanding release schedule and no prior automated testing… I have had plenty of experience with Empirix load testing but am finding purely regression testing a pain, because of what surely, should be basic things:
1) Version control of scripts – non existent within the suite (which is ludacris) – what do other people do here outside e-test (CVS etc.)?
2) Maintaining scripts – record playback is fine but when you have 50+ scripts and a rapidly changing website finding and replacing values across “manually scripted” tools seems a hell of a lot easier.
E-tester is great for quickly recording and playing back short dynamic scripts; and has enabled very quick and successful load testing cycles in the past… but in a more structured regression environment with more larger script... the GUI for everything approach becomes convoluted and time wasting.
3) Dynamically creating scripts from a template script – our site is multi lingual 8 languages so far and counting I am getting together a method of dynamically creating foreign language scripts from English “template” one… but gain, it is very painful – how do you guys manage this? Have you written a VBA app or just recorded multiple e-tester scripts for foreign languages?
4) Nightly regression tests – working on this at the moment… e-manager fills me with dread at the prospect of trying to regress 20 odd scripts against our new nightly build process with only a buggy html email sent with the results. Are any of you guys managing a large number of nightly regression tests with e-manager successfully?
5) Modular Scripts – only in the last release of the product have Empirix thought to include the ability to allow modular handling of regression testing scripts… but at the moment Jobs seem rubbish because for instance: you cannot iterate them; there is no concept of “prologue”, “run” and “epilogue” and you cannot maintain them because Jobs are “Read Only”
Do you guys use jobs? Or have modular scripting capabilities at all yet?
Lots of things I know in this post I know…
Please don’t think I am a Empirix hater either I think the suite is good and have used it successfully for testing large projects… but now my focus has totally changes to regression testing of releases.
Re: Regression Testing Frustrations
Well, I think I can offer at least a few suggestions, as we're also using the tool for a lot of our regression testing. In order:
1. I agree with your assessment of the lack of version control within scripts...it's WAY too easy to change a script and have no recourse to an earlier version. CVS all the way here; make sure you put up both *.mom and *.mom.vba files for each script (the rest is unnecessary, IMO).
2. There's a lot of trade-off to be made here. Some of our regression testing is very specific; i.e. I don't care about the images on page 1 of the script. Thus I shut off testing for page 1...it still plays, but I don't have to waste time trying to assess a bunch of "don't care" changes. If there are things that are going to be known regular changes, i.e. URLs, I build them right into my databanks. And finally, if there's a change across multiple scripts that's going to affect the same object throughout, I use a mass text editinbg tool to fix the problem in the *.mom files rather than go through the painstaking process of GUI updates.
3. Haven't tried this yet.
4. The first thing I shut off on e-Manager was the email reports. They don't get along with our mail server at all, and they are very limited in what they present. Everything gets written to an html log file, but more importantly, I add a short line of VBA to ensure that I can go back into any script that's part of a job that ran recently and look at the problems as though I just ran it (without this, nothing gets saved) Include the following:
Private Sub ThisJob_ScriptEnd()
5. "Jobs" are nothing more than the functionality of e-Manager, now available in the e-Tester interface itself. The only modularity is in what you design yourself; i.e. you have to create your own prologue script, etc and make sure that it sits where you want it in the overall job. Maintain them through e-Manager.