| || |
Payback w/ auto testing various Windows environments
I have been assigned to perform a preliminary assessment on UI automation testing within various Windows environments. This matrix will start with barebones Win98, Win2000, WinXP systems and work through all the service packs and versions of Internet Explorer (5.5 to present). I'm not sure if the Windows Updates will be addressed.
The application is a java applet within a web framework. However, my thought is that the amount of time & cost involved (configuring images, downtime of automation production machines, adding machines to domain after reimaging, more machines) the benefit would not be worth it. Currently, we do not perform any environmental testing as mentioned above. The OS is exclusively Windows (no linux or macintosh testing). If other OS's were supported than I would lean more towards supporting this effort.
Any thoughts/suggestions would be much appreciated
Re: Payback w/ auto testing various Windows environments
The first thing I'd do is see if there are combinations you can cut. Obvious ones would be WinXP + IE5.5sp2 and below (as XP comes with IE6.)
Less obvious ones would be based on the user base. If you know that all Win2k users, for example, have upgraded their browser to IE6 and are running at least one service pack on the OS, you can drop a few combinations.
Where I work, one of the major items we test is an image viewer plugin, which works for both IE and NS. For our testing, we normally use base systems. We only occasionally add OS service packs. As for the browsers, we usually separate IE into two categories: Those below IE5.5 sp2, and those at or above. The reason being that sp2 was when they quit supporting Netscape plugins (in favor of ActiveX controls.)
For the OS, you don't find too many serious differences between 2K and XP. Therefore, I wouldn't run the exact same configurations on both XP and 2K (we usually run WinXP + IE6, and Win2K + IE5.5 sp2.)
For the Windows Updates, I wouldn't bother with most as they are included in service packs later. You might want to run on a fully patched system, including all sp's, but I wouldn't do other combinations in between.