| || |
Poll - Collect stats on resources applied to large scale automation efforts
In a separate post, I asked "How big is your automation project(or something similar).
Got several responses, but Tony Davis gave me a reply I found particulary useful. Tony Davis said:
"We have around 15,000 lines of code in the base framework, and each script runs another 1,000 lines of code on average, sitting at 32 tests right now.
We have a goal to code a total of 280 different tests, so you are looking at close to 300,000 lines of code estimated to accomplish what we are set to do.
This isn't even considering the fringes of functionality, but only the base of it."
I thought this could be the beginning of another very 'real-world' resource poll about how many resources various companies that are serious about the test automation effort are actually applying to the effort, and some associated questions.
This could become something that could be printed out and shared with the managers that need to be involved in the financial signoffs and project approvals for test automation resources.
So here is what I believe is a pertinent questionaire that would yield some useful statistics when a new automation group is attempting to estimate and justify resources. Of course, every case is different - differnt projects have different levels of complexity, etc, but this poll would reveal some quantitative comparison regarding what other companies have actually been willing to invest in terms of dollars, resources, and 'lines of code'.
1.) For the largest most successful automation project in your company:
a.) How many resources - in terms of 'man-years' were applied to create automated regression suites for a particular product, broken down by:
i.) creating the documented testcases to be automated
ii.) the automation framework
iii.) the conversion of the automated testcases to the format required by your scripts (and what tool/input format did you use?)
b.) What technologies did you use (QTP, SilkTest, Perl) etc, and
i.) Approximately how many lines of code were written in each
ii.) How much resource is typically required to maintain both the documented testcases and test automation scritps ?
c.) What were the most difficult obstacles in your organization to completing the Automation project
i. were development groups involved, and perhaps required to alter the Application's Source code to add hooks, or expose methods
...if so, were they willing and enthusiastic , or was a lot of convincing required ?
d.) How are the test automation tasks and resources in your organization distributed
i.) do the test automation engineers write the testcases themselves
ii.) is there a separate group e.g. QA or Business Analysts, that creates the documented testcases ?
e.) What is the total number of testcases
i.) documented for the Product, vs
iii.) What are the average numbers of test steps in your testcases, and the average number of input parameters for each test step
( include a range, min, max, avg)
f.) Does the organization rely heavily on these cases, and how often are they run ?
g.) Were there unrealistic expectations associated with the Automation effort, and if so, how was this addressed ?
h.) What percentage of bugs are being found by your manual testers vs. the automation scripts ?
Feel free to answer only the subset of questions you feel are pertinent.
Thanks in advance.
Re: Poll - Collect stats on resources applied to large scale automation efforts
I left one pertinent question out:
i.) How much money has the organization spent on the Automation tools used for the project (initial purchase and including maintenance).