| || |
- 1 Post By noumenon
- 2 Post By dlai
- 1 Post By neeshpal
Design a Test Automation Framework for System "XYZ"
I have been interviewing for a "Test Automation Lead" position. I've created test frameworks in Ruby/Rspec and Python/PyTest for API testing and end-to-end testing in my last two jobs. I was recently asked to design a test framework for a given system during the interview at two different companies. I didn't quite pull it off during the interviews, so here's what I was able to come up with so far. I'd appreciate any feedback or thoughts on the following.
Problem Statement: Consider a credit-card billing system that generates bills based on customer credit-card usage data and sends these bills out to the customers via email.
There's an admin website where an admin can (after logging in) choose the billing period (e.g. current month, last month, or any arbitrary from_date and to_date) and then click a button to generate bills. This form submit (button-click) makes a REST call (say an HTTP POST) with "from_date" and "to_date" as URL query parameters to a server. This REST call triggers the bill generation batch process. The batch process queries a MySQL DB and generates the bills.
The bills are formatted as simple HTML tables that are mailed to all the relevant customers in the DB for the given time period. Each bill contains details like total dues and minimum payment amount and the payment due date.
Give an outline of how you would design and structure your test automation framework. What classes and class-hierarchies do you foresee, what helper classes might you use? Your design should ideally be DRY and avoid code duplication. When you start writing performance tests, can your framework reuse your existing functional test code to some extent? How would you add UI (e.g. Selenium-based) tests to your existing functional-verification framework?
I started talking about how I would test components in isolation first (I guess I like to think bottom-up tackling the easier sub-problems first). I would have a BillingRestHelper class that would abstract the lower-level details of making the HTTP call into a method sort of like "def trigger_billing(from_date, to_date)" which would return the HTTP status (200/400/403 etc). I'd then run data-driven tests for various input/output combinations and assert accordingly for +ve and -ve test cases. At this purely-REST-level I am not concerned with the correctness of the bills or even if they actually generate. Just verifying the correctness of the API layer.
For just the UI-level, I would have some Selenium-based framework that would check element positioning, login/logout, ability to submit the biling form, catch any JS errors etc.
I assumed that the batch process could take hours or more for a real-world situation where a bank might have millions of customers. So to make things simple initially, I would have a well-controlled test environment consisting of credit-card transaction data for just 10 fake/sample customers in my test DB for the batch process. So after I trigger the billing generation, I don't have to sleep for hours (potentially) to wait for the emails to be sent out.
I'd set my test-DB to an initial state from some well-known SQL file that will generate fixed, pre-determined billing data (that I can assert on later). Then I'd start up the billing server against this test-DB before actually triggering any billing.
Once the 10 emails for 10 test-customers are sent to their test-email-accounts, I would programmatically login to email, open the bill HTML, assert on pre-known correct values of total amount due, due date etc. This would test the correctness of the actual bill generation+mailer component.
I would then work on end-to-end tests that start from Selenium, all the way to verifying correctness of the generated bills.
Then I would design and run perf-tests and tests against realistic data with lots of customers.
This seemed to be okay but didn't impress the interviewer, who was looking for a high-level class design. I think he wanted to see which class inherits from what, do I use interfaces or abstract classes, or something like that. I'm not from a Java background, and I haven't really built my Ruby+Rspec/Python+PyTest-based test frameworks in a top-down manner like this before. I consider too much OOP and design patterns to be overkill for most test frameworks where a few solid specs (Rspec style) can do the job easily and quickly (but I could be wrong). Comments on this are welcome.
So I muttered something about a SQLHelper class for setup that would load the well-known test data into MySQL. An EmailHelper class for handling all aspects of email. A BillVerifier class for doing the verification of the actual bill data from the email.
This didn't go too well and I want to learn how to solve this kind of problem in an interview situation :)
I've been trying to find this here and elsewhere: how to design a test automation framework for a given system. Could someone please point me to any well-known or well-designed test automation suites for any opensource project on Github? Are there any relevant resources elsewhere on the Web, or even better, could someone directly help with the given billing-system problem?
Thanks in advance.
Test Automation Framework is pretty vague in itself. What they're looking for will probably depend on how technical the person is asking you, and the level of test maturity they have in their automated process.
non-technical interviewer - He probably just want to know you have the experience to get things done, and citing high level of past setups you created will suffice.
technical interviewer - He probably has an idea what he wants build. So you want to go more into the design philosophy to your approach. For example, I'd probably explain my experiences of gutting other people's record and playback test, and why my layered approach is far easier to maintain and scale in the long term.
Immature testing pipeline - You'll want to talk more about how to create a testing workflow.
Mature testing pipeline - You'll want to talk more about how you can scale up the testing infrastructure.
It seemed pretty open ended question. May be they ask ,these kind of questions to see the vision of the candidate. But heres how I would go about it. Note that this is just a draft I came up with and can be further modified , think through and even be replaced with a better draft. Theres a lot of room for improvement when designing such things. May be while running these classes , we may find out that one or more class is useless and can be integrated with other classes or any such things. But I just wanted to give a shot at it. Any ideas to improve on are most welcome.
1. I would create a create a class seleniumGenerateBill with submit() method which would take to and from date.
2.second class would be this with FetchResponse and CheckResponse mehod that would check the rest response.
3. Next classDBCheck where we would check the db results.
4. Next a class to check HTML which would check the html generated by the billing process.
To run an end to end test , hopefully we would be able to initialize instances of these classes and run methods inside them.
This can also be further improved for data driving this end to end test.
Thanks for the post.
Originally Posted by neeshpal