good question, but no easy answer. It also depends a bit on the underlying technology of the application you want to test (web/sap/.net/java/...?), the functionality, the kind of tests that should be automated (smoke test, functional tests, module tests, ...). And of course it depends on the purpose. What should the framework do? Usually it takes a couple of days to do some analysis and design the framwork.
TestPartner in my opinion already has a framework in which to develop tests in. Unless you would want manual testers to execute tests from Excel or something of that sort, you do not need to create a framework.
The setting up of a test automation framework really depends on what you want to achieve with all the effort that you are putting in...
There is no rule-of-thumb and there is possibly no easy answer to your question but the following questions will possibly be of some help to you:
1. Do you want to get a fully customised test execution report? Should someone mandatorily have a TP license in order to be able to view the test execution results?
2. Would you like to be able to filter test execution results?
My framework provides us with the ability to filter test results by comments categorized as follows:
- Passed test cases
- Failed test cases
- General Information
- Comments relating to Exceptions
- Script execution summary
- Comments providing links to snapshots
3. Do you need to generate test metrics?
TOTAL No. of SCRIPTS EXECUTED
TOTAL No. of SUCCESSFUL SCRIPTS
TOTAL No. of FAILED SCRIPTS
TOTAL No. of SCRIPTS PARTLY EXECUTED DUE TO ERRORS
TOTAL No. of SUCCESSFUL TESTS CASES
TOTAL No. of FAILED TEST CASES
TOTAL No. of SCRIPTS SUSPENDED
TOTAL No. of SUCCESSFUL TEXT CHECKS
TOTAL No. of TEXT CHECK RETRIES ATTEMPTED
TOTAL No. of FAILED TEXT CHECKS
TOTAL No. of TEXT CHECKS PASSED ON SECOND ATTEMPT
TOTAL No. of DUMMY TEXT CHECKS USED
TOTAL No. of TIMES TP FAILED TO TAKE THE AUT TO THE TEST SITE
TOTAL No. of UNEXPECTED POPUPS ENCOUNTERED
TOTAL No. of WARNINGS GENERATED
TOTAL No. of EXCEPTIONS RAISED
TOTAL No. Of SNAPSHOTS GENERATED
4. Do you need to cater for automatic notifications (e.g. emailing, SMS if ever anything goes wrong during script execution?
5. Error Handling - How would you like to deal with errors in Scripts & Errors in the AUT?
6. Error reporting - Would you like to minimize 'cascaded errors' following a failure in the AUT?.
7. Test Environment - what kind of pre-test setup will be required?
8. Error recovery - What do you need to do achieve unattended test execution?
9. Monitoring of memory consumption (at specific points) by the target application. Do you need to know whether the application has failed to release the memory when a form is closed for example?
10. Do you want to develop your scripts based on a functional decomposition approach?
11. What are the various things you need to cater for in order to facilitate script maintenance?
12. Do you have a set of standard testpartner asset naming conventions for everybody to use within your team?
13. Do you have a set coding conventions and best practices?
14. Have you thought about Test data management?
15. Do you want to implement data-driven or keyword driven automation?
16. Where should your test data be stored?
Well, hope this helps you to think in terms of what you need...
In summary, my framework is what holds all of the above (and many more) altogether...
I am always confused by this question, perhaps because the questioner is also confused.
What exactly do you mean by an "automation framework" in testpartner? What do you want it to do that it doesn't already do?
As someone else pointed out, the basic automation framework is already present in TP. There is nothing to prepare, unless you can tell us what it is you want it to do in addition to what it already does.
I will go back to my original question. What exactly do you mean by "automation framework" and what do you want it to do?
To simply say you need an automation framework tells us exactly nothing. It is like saying that you want a tool to fix your car. You will have to get a lot more specific before we can help give you a direction to go.
We are Doing Sanity Testing on Webcentric application.Our Main Objective doing this is entering of all the fields and cing whether the screens are working fine or not.
Like this we have to check for 3500 screens in which some screens are pre requiste for others