1. ## Productivity Metrics

Hi All,

I have a very simple question.
How does one calculate the Test creation productivity
The following details are being noted down
Number of testcases written
Time taken for the testcases in hours

Currently, we are dividing the hours by 8 for number of days
and then dividing the number of testcases written\number of days taken

1.is this correct???

Additionally, since the test design is an evolving process due to constant changes in the testcase, sometimes, we end up spending 2 hours to correct around 40-50 testcases, it would not be a new creation but an update to suit the updated requirement.
2.in which metrics should this be counted?
3.Can this be treated in Testcase creation.

Thanks,
Regards,
MS

2. ## Re: Productivity Metrics

Most companies that want to track such measures have a standard method for calculating them. You should ask your company.

Hours / testcases seems appropriate for what you are asking.

Some companies separate New Test Case Creation from Test Case Maintenance. But as I said, your company probably already has a standard method for this.

3. ## Re: Productivity Metrics

I have found that metric to sometimes be misleading. I found some testers had a tendency to write fewer, longer tests while others wrote more, shorter tests. Test steps per day seems to work OK as a general metric for me.

4. ## Re: Productivity Metrics

MS,

Calculating the number of test cases written by a tester per day as a productivity measure seems to make about as much sense as counting the number of bugs found as a measure of quality.

Without context it appears all test cases are equal; when in fact, all test cases are not equal.

5. ## Re: Productivity Metrics

The key to all these metrics questions is why do you want to measure them in the first place and how are the figures you collect going to be used?

There may not be a "right" answer, but any way you calculate productivity may give some value depending on context.

6. ## Re: Productivity Metrics

[ QUOTE ]

Currently, we are dividing the hours by 8 for number of days
and then dividing the number of testcases written\number of days taken

1.is this correct???

[/ QUOTE ]

No this isn't correct. Let me ask some questions here that might shed some light on what it's not correct.

1) How many words does each test case contain?
2) Why wouldn't you use a keylogger to determine the words/hour typed as a productivity measure?
3) Are there different standards for different employees?
4) Do you have any benchmarks in place for test case creation?
5) How do you judge the quality of a test case?

Let me expand on these a little more with a story. The story of the Frog and the Turtle. They were asked to carry boxes from one side of a chasm to another. The frog was faster than the turtle, but the turtle could carry more. After a couple passes over the bridge, which was far away, the frog and turtle had delivered just about as many boxes. That is until the frog found that he could jump the chasm.

"Ha ha! Look at me!" said the frog as he leapt over the chasm.

*Crash* the box hit the ground with a thud as the Frog landed on the other side with it.

"Whoops!" Huh, seems alright I guess.

Frog leapt over the chasm to grab another box, and then back over the chasm again, carrying box after box. That is until he slipped on takeoff and tumbled into the valley below.

"Ouch!" said the Frog. "Turtle! Help! Help me get out of here."

Being a good friend, Turtle slowly made his way down the cliff and carried frog back up to the top, along with his box.

"Phew!" said Frog. "Thanks a lot! I won't let that happen again."

Then Frog grabbed another box and, in his arrogance, leapt over the chasm again. Frog made it a couple more jumps, but once again fell it and yelled for Turtle to help and, being the good friend he is, Turtle helped him once again.

This happened numerous times more until Farmer John came to check on the two. He noticed a couple things. First, he noticed turtle had carried much fewer boxes over than frog. However, they were nice, organized, and in good condition. Frog's boxes were unorganized, damaged, and in serious need of repair. Since Turtle was obviously carrying fewer boxes over, Farmer John asked that he fix Frog's boxes instead. Hence, the two friends combined to do the work of one.

What's the moral of this story? If you judge productivity without knowing the whole story, chances are you will hurt the overall productivity.

[ QUOTE ]

Additionally, since the test design is an evolving process due to constant changes in the testcase, sometimes, we end up spending 2 hours to correct around 40-50 testcases, it would not be a new creation but an update to suit the updated requirement.
2.in which metrics should this be counted?
3.Can this be treated in Testcase creation.

[/ QUOTE ]

This is fixing boxes. I would first try to determine why I'm having to fix 40-50 test cases on a routine basis. Many times we try to implement solutions which address the symptom, rather than addressing the problem itself.

I hate productivity metrics, but if you really want to attempt to track this, then you need some serious benchmarking in place. Also, I don't think you can effectively track test cases based on the number of test cases created. You might be able to do this based on words/hour, but none of this will tell you the quality of the work being delivered. Actually, these measures will likely decrease the overall quality of the test cases being written.

For instance, I could write a test case for:
a) TC-0100-Calculator-Test numbers 0-9

or I could test

b)
TC-0100-Calculator-Test number 0
TC-0101-Calculator-Test number 1
TC-0102-Calculator-Test number 2
TC-0103-Calculator-Test number 3
TC-0104-Calculator-Test number 4
TC-0105-Calculator-Test number 5
TC-0106-Calculator-Test number 6
TC-0107-Calculator-Test number 7
TC-0108-Calculator-Test number 8
TC-0109-Calculator-Test number 9

Am I 10 times more productive using "b" than using "a"? No, I've actually decreased the overall productivity of the team not only because someone needs to review each of these test cases, but also because someone needs to read each of them on each test cycle. That means that, in this particular case, I have increased the time take to test the buttons on a calculator TTR(time to read test case)*10. I would say ten times, period, but that's not indicative because to test the buttons, it may take be half a second per button, so the first test case would take me, say, 25 seconds to execute (5 seconds execution, and 20 seconds to read the case). However, using "b" it would end up taking me 205 seconds.

So how do we create a metric that will track the efficiency of the test cases being written as well? This is why tracking productivity using subjective metrics is so dangerous. This is a real-world example, too. Well similar to one anyway.

A friend of mine told be about some Internationalization testing they set up with their team. He people write the test cases a while ago and his team was fairly familiar with them. They brought in a new kid and couldn't figure out why it was taking him like 4 days to test each language (there were 12 languages). Well each term in the app had it's own test case set up. All he had done before this was look at the screen, highlight all the test cases and mark them as passed if it all looked good. It would take them about a half a day to finish a language.

So I'm really sorry, but I can't tell you of any "good" way to track productivity of creation of test cases. Whatever makes your management feel warm and fuzzy is probably just as good as anything.

See? Look how much I typed here, and there is probably very little quality to what I wrote. Lots of filler with a couple points of genius thrown in. That doesn't mean it's a good post though [img]/images/graemlins/wink.gif[/img]

7. ## Re: Productivity Metrics

[ QUOTE ]

5) How do you judge the quality of a test case?

[/ QUOTE ]

this is key! One tends to identify productivity with number of test cases or hours in front of a computer, etc
but the key thing here is to know of you are properly covering the functionality of the SUT.

You can only measure productivity in a similar way you are proposing if you have a proper testing methodology in place where you really know inputs &amp; outputs for each of the testing stages

8. ## Re: Productivity Metrics

[ QUOTE ]
[ QUOTE ]

5) How do you judge the quality of a test case?

[/ QUOTE ]

this is key! One tends to identify productivity with number of test cases or hours in front of a computer, etc
but the key thing here is to know of you are properly covering the functionality of the SUT.

You can only measure productivity in a similar way you are proposing if you have a proper testing methodology in place where you really know inputs &amp; outputs for each of the testing stages

[/ QUOTE ]

And this is the QA Manager fixing boxes, in my opinion. I'll admit, some places are going to require that you give this to them, this sort of productivity metric. However, I will always ask, "Why do you want this?"

We all know what the answer will be. This is a retorical question in many cases, but it's something that needs to be asked. The reason I ask it is so I can introduce these other questions. I can also give a cost analysis. I don't come cheap, so when I tell someone that they are paying me, say, \$30,000 to generate this report on an annual basis, that's a pretty expensive report. There's not even any golden seal or anything. Just ink and paper.

Tell them, "You hired me for a reason. If that reason was to generate this report, then you're getting you money's worth. If you actually want me to manage these people, though, then this is a waste of my time and your resources." You can also throw in, "If you hired me to manage these people, then trust that they are being productive under my leadership. If we start missing deadlines because of poor productivity, then fire me."

I've said it before and I'll say it again, unless you are managing 300 employees, the only metric you need to hear, at the end of the day, is that people are making adequate progress or, when they aren't, they are getting help to make that progress. Even if you're managing 300 employees, I would assume that you impress on each of your team leads that this is how you want to operate and expect them to ensure their team is being productive. I find no trust in numbers.

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•

vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.