Measuring code coverage for automated UI testing
I have an idea to implement code coverage measurement for our automated UI test suite (which consists of about 450 tests).
More usually, code coverage tools are used with unit testing, but I think that outputting code coverage reports for our automated UI testing would be extremely useful.
I'm thinking something like running Visual Studio (with a code coverage add-in installed), and monitoring code coverage as the automated tests run. Of course, at the end of the run, we would get reports on code coverage, and the different types of coverage (statement, branch, etc). We can use these reports as an input to our decisions on what automated UI tests we should add.
Just wondering if anyone has experience in this area/what people think of this idea. Can it work? Would it add a lot of value (as I think it will), or should we just concentrate on getting code coverage reports from our unit testing?
Thank you all.
Just don't bother. In Acceptance level/ GUI level testing it's more important to have requirement coverage.
You know what'll happen if you do code coverage at acceptance level? First off, you will not hit the really hard to reach exception cases such as things to handle catastrophic failure, concurrency, etc..
And 2nd, you have the situation where you are testing a build the customer isn't getting. code coverage usually requires a special interpreter or instrumented build. Which is not the same build that gets shipped to customers. Their performance and reliability profiles are different, and you'll want to be testing as close to what'll be shipped to production as possible.
hmmmm...thanks for the advice; sounds like maybe I should push for reporting on code coverage of unit testing. But still be interested in any other views.
A very good answer from dlai!
You may want to look at what "value" you give at the end to the customer. Remember, to make quality count, the most important point is on requirements coverage. Hence, if you can uncover the functional impact for modules, that would have the greatest impact on covering the slated requirements.
Technically, you may probably end up showing a nice report (using indirect methods) for "code coverage", but that is not going to be helpful from a project / customer perspective.
A question back to you... What are you trying to achieve by doing a code coverage in the first place using automation tools? May be answering this would help you.
Definitely good for unit testing. Not only is it a good measure to make sure your unit tests cover the lines of code, it also helps in the design and architecture. Good code is easier to unittest, while bad code, those that are unnecessarily too complex is hard to unittest. In my current company, we have build triggers that measure code coverage and cyclomatic complexity. It's a fairly good way of keeping code quality very high.
Originally Posted by darkstar9999
Yeah, I will add my voice to David's. The return on investment on doing that at the functional test level is highly suspect. This would naturally lead to striving for a 100% code coverage metric(even dead/error handling code that would never actually be touched in production). Also double-down on the fact that doing code coverage can actually impact what you are testing. In prior experiments we would encounter timeout errors and such actually caused by the instrumentation.
thanks a lot all. So now I'm convinced that trying to measure code coverage of automated UI tests is NOT a good idea. So I'm going to see how I can push for code coverage reporting of our unit testing. We don't seem to do this at all currently. Andy79: In answer to your question: I'm very keen on white box testing, as I believe delving into how your application works 'under the hood' is extremely fruitful in finding issues/potential issues. The idea was that information on code not exercised by our automated UI tests could be used as intelligence to help decision-making on how we should extend the coverage of automation. I'm certainly not saying this is the only aspect that you would consider when deciding where to extend automation coverage (this is a multi-faceted question), but I was thinking it could help. But as it's become clear that this is not the way to go, the point is pretty moot.
It's a great idea, and something I'm not sure why more don't perform.
You're on the right track. Essentially you'd put it together as part of your build process. Trigger code-coverage compatible build, trigger automated GUI tests to execute against compiled build, after which reports and analysis are available for both the pass/fail of your automated GUI tests, as well as the test coverage of your GUI tests.
Many may think they know what code GUI tests are executing, but such analysis can reveal many unknowns or shed light on gaps where formerly assumptions were made regarding testing coverage.
Don't give up on the concept so quickly. There's a lot that can be gained from it.
Using froglogic's Squish GUI Tester & their Squish Code Coverage tools you can do exactly that
I have demo recordings of both tools, but not the two together (am happy to create one)
code coverage video: Squish Coco Webinar: How to Improve Testing Coverage with Automated Code Coverage Analysis - YouTube
gui automation video (this is against windows platform since you mentioned visual studio): Squish for Windows: Automated GUI Testing - YouTube
Happy to discuss this further
While you're validating requirements, don't you also want to ensure you're exercise all scenarios within those requirements, both from a GUI standpoint and from a logic standpoint?
Originally Posted by dlai
Each time you run your GUI automated tests you aren't running them using the code coverage build (some code coverage tools also don't require you change your source code, but as you mention simply instrument your build with their interpreter)
Originally Posted by dlai
There are times and places for this. For example, understanding how your current GUI test suite covers the code, where the gaps are, and how those gaps can be overcome. It can also illustrate redundant testing. From there, when planning for your next set of tests, you can also analyze how the tests exercise the given code, what you may need to add, and again, what may be redundant.
Then running your automated GUI tests, for example as part of your build process, can be done either with or without the code coverage analysis.
As features change, or fixes are made, once again, the coverge analysis can be brought back in. It's not a forever and always exercise that you need to perform with your automated GUI testing.
I ademendmently disagree with the statement code coverage is a good statistic for acceptance tests. These are the two main reasons I think why.
1) What is more important is requirement coverage. It doesn't matter if the developer wrote more of less code to get the job done. The bottom line is the Job needs to get done. While code coverage is measuring how many lines of the Job's implementation you're looked at, it doesn't measure how much of the said job is actually done. So code coverage from this stand point is a Vanity Metric. What would be better is to tie you tests to report back to a TCM or ALM like Quality Manager, etc.. so you have a nice report of what percentage of requirements were tested. Sure the developer did have a purpose of writing each line of code in the program. But the developer did not write the business requirements. The business requirements is what you need to be testing.
2) Acceptance tests from a complete engineering point of view only represents about 5-10% of the testing pyramid. » Inverting the Testing Pyramid » Managed Chaos by Naresh Jain
You will end up spending disproportional amount of effort increasing code coverage for something that only amounts to 5% of the total test effort to satisfy a Vanity Metric.
On the argument you run the same code. That is absolutely false. It's the same code in source code, but completely different in terms of run time. It's like saying that you have tested Windows 8 when you ran your tests on Windows XP. It's the same program, but underneath is a completely different platform. The same is true with acceptance testing. You want to run as close to the target environment as possible. Given you have good Unit test coverage, which is already testing the code itself. What you want to test at the acceptance level is the integration and platform risks. You simply cannot have that level of trust on an instrumented build.