Single or Multipule tools
Hi all, Many thanks in advance for your input.
I am in middle of evaluating and implementing automation framework in order to support our multiple products. We are in deadlock situation where some team members have disagreement on following
(Single Tool for multiple products will not work and therefore we require tool for each product to have efficient testing framework)
we have 4 main products using different technologies Adobe Flex, HTML5, Java but all the products integrates with each other in order to complete the entire process (for instance user will create a plan(A) on planning tool and submit, then user will use AM to approve and use the plan data for resource planning and submit. the other tools are reporting tools to generate the report)
In my opinion we should use a single tool to support our products if the selected tool is compatible with all the technologies. Now in my opinion, not to use multiple tools is based on the following
a) it is not cost effective for longer term as we will require multiple environments to support multiple testing tools
b) not feasible as we will require multiple testers with different competency level to ensure the quality of testing
c) not sure how we will be able to perform integration testing with have multiple tools, have not used such framework in the past so not completely sure but it dosn't buy me either as I believe there will be extra effort required to ensure a good level of integration is covered between all tools
d) Multiple training packages, license management etc
Please give your finding and how would you tackle this....
Never test the depth of the water with both feet.
Your test framework should be(actually must be in this scenario) automation tool independent. If you use, for example, QTP on one project but Selenium on another they should both interface with the same test flows and retrieve the same test data from a common source. The business logic must be separated from the scripts, and that logic should be what drives how scripts are executed. This allows you to take advantage of multiple tools that interface with different projects better. It would also allow you to cover your scenario c just fine. Your driverscript would execute test A in tool 1 and test B in tool 2 and not treat them any different other than the actual line of execution.
You certainly do still have the issue of requiring a diverse skillset when using multiple tools. But with the business logic removed from the scripts, not every tester needs to know how to script. You can just target experts in a particular tool to do the implementation while a non-scripting tester or even product owners are able to create and execute the test cases.
Actually, Selenium's wire protocol is a good way of doing this. If you implement the Selenium's wire protocol interface right into your product, you can use Selenium to automate your product just like you would any web page. However, implementing a xpath and css locators is very annoying, might be best to ask your developers to add ID's to everything and just implement the by ID locator.
A few thoughts from my experience with this kind of thing:
- no matter what the decision is, you're probably not going to please everyone on the team. Your best option is a compromise.
- if there's a central data store that all the applications pull from, you don't need to automate end-to-end multiple application tests (which is good, because these are resource heavy and tend to be somewhat fragile because of all the moving parts you're trying to keep coordinated).
- your best bet in the scenario you've described is something like this:
- automation for application A starts with resource database A, adds the plans needed, then closes the application and compares against baseline database A.
- resource database B is a copy of baseline database A. Automation for application B starts with resource database B, performs the operations needed, then closes and compares against baseline database B.
- resource database C is a copy of baseline database B. Automation for your reporting applications uses resource database C and validates the reports generated.
- if you can use a single tool, it makes the process much easier once everyone is trained. If not, look to use the minimum number of tools possible.
- if cost is not an issue, the large commercial tools can probably support all your applications.
- document from the start - it's way too painful to try to build documentation of what each test does later (speaking as someone who's had to do this).
- if possible (and not every test tool supports this, shame on them) you want a lot of small, granular test routines that can take parameters and be called from a larger harness. This does mean that your test routines have a lot of dependencies, but it also means you're coding each action precisely once - the maintainability boost you get more than makes up for the dependencies. An example is that you want to be able to pass arbitrary user credentials to a logon routine so you can log on as any user. You don't want a different logon routine for each kind of user you're testing with, and a test that just logs on and logs off as each user in turn is useless for your main needs.
- if possible, maintain your test data separately from your test code. How you maintain it is up to you (I've worked with what was effectively a normalized database in CSV files - which needed way more documentation than it got, but was effective). Test data can include the tests themselves (my preference) or just the data they use. The main thing here is that whatever method you choose needs to be able to be used by all your tools.
- avoid (I can't stress this one enough) tools that limit your ability to perform arbitrary queries against your test data and your application databases. No matter how good your tool is, there will be times when the simplest method of getting or setting data is to run a quick SQL query against the data source. Not all tools support building and running a SQL query this way (something like "UPDATE tablename SET fieldname = '" + variable.toString +"' WHERE idfield = " + idvariable - this is invaluable for quickly flipping a permission or modifying something that's a pain to access via GUI automation)