How to improve robustness in my test automation framework
I've been working on a test automation framework for Android for a while now and I still feel that it lack robustness. I follow page object pattern, always start in a clean state, try to log everything and still feel that I debug more false fail than actually finding faults.
It have even gone so far that I wonder, what's the point? If I can't even trust my tests, how could any one else?
So anyone that have any good tips to increase robustness in your test automation? I've seen some crazy solutions like brute force by doing the test at least three times if it fails but I think that takes it a bit to far.
Or is the only way to keep running them locally till I find everything that can go wrong? At least that's my solution right now.
- Lots of wait wrapper utilities. It's very common among beginners to keep turning up the implicit waits until they have a badly written test.
- Vet your tests before they go into rotation. I tag my tests with [unstable] which run continuously in a separate suite for about a week before I move them into being a critical path acceptance test.
- Don't do hacky things to work around issues, and be courageous enough to insert test hooks. For example, say you do a hard coded wait for an ajax. The smarter thing would be to create a latch to the promise of the ajax request's return.
- Use a screen capture decorator (in java they're called @Annotations, in c# they're called [attributes]) on your tests to capture a screenshot upon failure.
- Always validate immediately after an action. Often times a failure is really a result of a previous action not completing correctly, 1 or 2 steps ahead. Usually by the time you do your failing action, it's too late to identify the action that's the true cause. A simple example is say the test logs in as a power users then navigates into an admin area, then performs an power user action. Say the test fails to login as a power user, but the failure will be at a point where it can't find the say, power user admin button. If you don't validate right after login that you are a power user, then you might wonder... Was the page load too slow and the power user button didn't render yet? Did I click on the power user button and nothing happened? maybe I clicked on the power user button and did the validation right before the action completed? You get alot of these strange situations if you don't validate often.
- Avoid testing UI. For example, say you have a complex hover over or drag and drop. I'd rather just add a test hook to short cut it because dealing with so many timing and positioning issue is time consuming and does not buy you much in the long term. Acceptance level tests should be more focused on system level integration, than any particular component.
Thanks for a nice answer!
Originally Posted by dlai
I've actually fallen in many of those holes before, but most of them are fixed by now. The only thing is that I may be to greedy with validation as I don't want to fill a test with too many asserts. But I guess I should start with that to make sure I really know cause of failure as you said.