Evaluating Testing Tools
My company is in the process of evaluating an enterprise-wide testing suite. We've received RFP's but no demos yet. The suites we're considering are Copuware QATest, Segue Silk, Empirix e-Test, Mercury Astra, and Rational Robot.
Our company has a large AS/400 shop for processing and we on the front end are a web-based application. Our tool must support both platforms. The AS/400 group is driving our choice and we have not received RFP's from Rational or Mercury, however two of the analysts on our team have used both of them and like them. We're not even sure Rational supports the AS/400 environment.
Our situation is special in that we do not have one website that 1000's of users are hitting. We have 100's of websites that 100's of users are hitting on a farm with multiple servers. 1000 users would be our minimum right now for load testing. We double the number of users every year. This may make load testing tools unreasonable from a price standpoint (Silk in particular stands out here).
Our biggest issues are:
1) How quickly scripts can be updated. Two of our analysts saw an OhmSphere presentation (integrated with Winrunner) last year and were very impressed on how you can easily adapt in new pages, etc.
2) How easy the script language is to use. One quarter of our analysts have never used scripting or done any kind of programming.
3) Do they offer PVCS integration? We use both Tracker and VM.
4) Cross-browser support (Compuware doesn't have it I've read in one place and has it as a feature in another).
5) How good is the support? Would training be needed?
6) How quickly do these companies update their products for new browsers?
We have been told we can only demo two products, but we'd really like to do at least 3. Any information about our specific situation and your experience with the above products is much appreciated.