Shreejit
(Member)
04/05/12 11:04 PM
'My Beautiful Framework'

Hi All,

I started this thread so that people could share the 'one bestest part' about their framework.
Others could obviously take some tips out of it.
The best part about our framework (which is still in its infancy) is the strength of the keywords developed and robustness for eg. ClickOrValidate () - single keyword performs any click operation on the UI / Validates any data on the UI allowing us to reduce our overall list of keywords


ManishBansal
(Active Member)
04/06/12 02:59 AM
Re: 'My Beautiful Framework'

First of all... the title sounding me very funny..
Well a good thought, but you will find a lot of similar threads on the forum already.

Still lemme share some of my thoughts on this thread.

1) Framework should be very simple, small, fast, generic, should have very less files, i suggest only 1 file framework.
2) allow easy adoption by by other teams and other users.
3) Never reinvent wheel.
4) should make your life easy then complicating it.
5) if it is qtp based framework, it should leverage maximum QTP inbuilt functionalities.
6) maintenance should not be a nightmare.
7) everything should be planned in advance before developing a framework.
8) should be able to provide good crisp reporting drilling down to the actual issue.
9) always separate from business functionalities
10) and finally, just start using them then keep on building building and building.


Rastek
(Member)
04/09/12 06:04 AM
Re: 'My Beautiful Framework'

Shreejit Can you please provide more information about the framework. Is that a OTA-API based franework ?

What kind of a framework we are talking here ? Thanks.


lockdown
(Member)
04/10/12 02:12 AM
Re: 'My Beautiful Framework'

Hi Shreejit

I like this thread, it can give a bit of insight into what everyone's working on.

For me, it's the GUI layer on top of our keywords which is enabling our manual testers to automate the AUT without having to understand the QTP object repository structure.


mwsrosso
(Veteran)
04/10/12 04:44 AM
Re: 'My Beautiful Framework'

Quote:


1) Framework should be very simple, small, fast, generic, should have very less files, i suggest only 1 file framework.





Manish I have tried to think of how you could build a robust framework using only one file and I can't. Do you have any information on how this might be achieved?

mark Smith.


mwsrosso
(Veteran)
04/10/12 04:49 AM
Re: 'My Beautiful Framework'

Similar approach here Lockdown but using Excel as the user interface. One QTP resource maintaining the framework and testers 'creating' the automated tests which are converted into QTP executable code at runtime.

Mark Smith.


Shreejit
(Member)
04/10/12 10:04 AM
Re: 'My Beautiful Framework'

@lockdown - That was the idea..felt like a one stop shop to know what ppl have been doing
@lockdown - Care to provide a briefing on the GUI layer (any piece of code / snapshots would be of gr8 help)

Our framework is like what 3 months old now: Good code, easy to use, no code breaks
Problem areas:
1. Lot of keywords to achieve a lot of functionality (testers confused which keyword to call when and what parameters to pass to the functions)
Any suggestion what i can do here?
A chm file or something i am not sure what exactly should i be doing here
Something like a one stop
User Interface1 - List of keywords available
User Interface2 - List of keywords available
Generic keywords - SendEmail,ValidateDatabase,ValidateMainframeLogs,Comments
etc etc
basically ease the use of manual testers, so that the automation folks can focus on the more concrete problem solving areas


NoUse4aName
(Super Member)
04/10/12 10:23 AM
Re: 'My Beautiful Framework'

We don't use keywords per se. Go with more of a page object approach. Manual tester creates a reusable test of all possible interactions on a page, in english sentences, written like a manual test case but with data parameters in Quality Center. An actual test is comprised of calls to these reusable tests for each page in the application.

Automater comes in and codes up a reusable function/class for that page that takes in the parameters and executes all the steps(except a blank param means skip that step/leave field blank). Once that's done for all the needed reusable portions, manual testers can create thousands of tests and they'll be ready to run as automated without any action needed by an automation developer.

Test Processor calls the appropriate function/class based on what steps are in a particular test case. Using the OTA API can execute these tests from Quality Center test lab in QTP(retiring our usage of that) or Selenium and post results back to Quality Center.


mwsrosso
(Veteran)
04/10/12 11:03 AM
Attachment
Re: 'My Beautiful Framework'

Quote:

snapshots would be of gr8 help




What we do is when a keyword is selected from the keyword drop-down list is display a small overview of how to use the keyword but also populate the required fields with a prompt. See attached screenshot.

Mark Smith.


ManishBansal
(Active Member)
04/10/12 10:43 PM
Attachment
Re: 'My Beautiful Framework'

Hi Mark,
Let me try to elaborate.
1) Framework should be very simple, small, fast, generic, should have very less files, i suggest only 1 file framework.


1) very simple : for our end users (not just the users who are configuring & running it, but also for the other automation teams/developers to understand and reuse it 100%), other automation team should not spend much time referring to documents every time they are using.

2) small & single file : the framework we have developed is having only one file we call it Core.qfl
this is the back-born of the framework, it has very few say only 5 public functions and other are private)
it helps easy maintenance, and give portability, if anyone want to reuse this framework then he/she just have to add this file in his resources and that's all. (we keep this file in SVN)

Mark, we also have other files too, but we call them plugins to our framework ( CoreWeb.qfl, CoreNet.qfl, CoreJava.qfl, CoreInfragistics.qfl so and so forth) can be added on demand

We also have 2 more files 1- globalrecovery.qrs , 2 - RecoveryFunctions.qfl (Need not to attach them to resources)

Finally the best part of our framework is our COM dll, which needs to be register (framework automatically register if COM cannot able to create object) - Thanks to Tarun lalwani
we have all non-qtp utility functions in our dll. (screen-shot attached)

3) Fast - it is based on recovery , so we never check our control, just do the operation on that control, if the operation is successful, its good, otherwise recovery executes(No need to check Exist, disabled, ListItems, duplicateitems and duplicate objects explicitly )
One more thing, our recovery is doing nothing but calling a function and this reports (so no messy reporting due to recovery)

Its fast because, for all common operation we are using COM object (pre-compiled dll)

4)generic - this framework is really not having any business functionality, you just have to attach Core.qfl files, now record your test and then do some modification in OR, and your script is ready, that's all.

on the good node!, our framework is make our life & work simple efficient and not complex.


mwsrosso
(Veteran)
04/10/12 11:57 PM
Re: 'My Beautiful Framework'

You haven't mentioned a QTP driver script (Script.mts) do you still require the use of this in the framework manish?

Mark Smith.


ManishBansal
(Active Member)
04/11/12 02:48 AM
Re: 'My Beautiful Framework'

oh yes, thanks to point it out, their are two ways of doing it...


1) create an action (first in the flow), you can call it DriverAction (it will call 3 functions)

i) Initialize()
ii)DriverFunction()
iii) ExitTest() 'QTP inbuilt

or you can just call them from Action0 (Not visible in QTP), by modifying Script.mts

Mark,
adding these calls in Core.qfl create issues with QTP.




I wish i could share some document over the forum, but i am not permitted because they are part of our company assets.

but i am planning to update it on my blog/sqaforums, and share my strategy for discussion.


Shreejit
(Member)
04/11/12 10:20 AM
Re: 'My Beautiful Framework'

@Mark - Thats neat.
We are working on a chm file for ready reference and split it up into UI / Generic.UI's would be keywords available for specific UI level operations. Generic keywords would be available across all UIs (Prompts, Check exists, Sync etc)


Shreejit
(Member)
04/11/12 10:24 AM
Re: 'My Beautiful Framework'

Quote:

Once that's done for all the needed reusable portions, manual testers can create thousands of tests and they'll be ready to run as automated without any action needed by an automation developer.




This is our issue. Manual test scripts are completely independent of automation scripts. Whatever those guys dump into their sheets we have to interpret it and use it in our automation scripts

Also we have a suite of 80 scripts to be automated.
As of now a dedicated workbook exists for every script.
What should i do? Should i club them into all one excel?
Every workbook has on average 1000 lines in the excel.
80 workbooks would become huge eventually, right now we are automating just the 2nd script.


Shreejit
(Member)
04/11/12 10:32 AM
Re: 'My Beautiful Framework'

Also, I need help in one specific issue happens during navigation from Page A -> Page B
PageA (fill up fields)
Click on Continue button
Navigate to Page B

During this specific step i might run into hordes of integration issues.
1. Integration systems are down
2. Database is down
Is there a way i can trap these error messages inside my recovery function
(I get a script error on trying to perform the operation on an object on Page B since Page B was never displayed in the 1st place)
Function Recovery1(Object,Method,Arguments,Retval)
Is there a way to trap the innertext of the browser parent
I have the Object with me but not the parent
Something like Object.GetTOProperty("parent")
'Basically i wanted the browser parent of the object and find the innertext of the browser object and check for various integration issues
End Function

Also what does the "retval" contain?


GregEncke
(Member)
04/11/12 10:42 AM
Re: 'My Beautiful Framework'

I do a screen shot, exit the test and have a function that closes down resources.

kayd
(Member)
04/11/12 06:39 PM
Re: 'My Beautiful Framework'

can some one plz guide to me web testing framework. samples?

Shreejit
(Member)
04/11/12 08:41 PM
Re: 'My Beautiful Framework'

Quote:

I do a screen shot, exit the test and have a function that closes down resources.




Greg this is what my upper management also wants gracefully exit the test and clean up. But my point is unless its a worst case scneario why should we exit the test (this should be the last resort right?).
The best thing to do would be make qtp recover from the current error and try and re correct itself (as much as possible / proceed with a workaround)


mwsrosso
(Veteran)
04/11/12 11:25 PM
Re: 'My Beautiful Framework'

In the event of a test case failure our framework will report the issue, capture a screenshot and embeded a link to it in the HTML report, reset whatever is required then move on to the next test case.

This way we can run unattended test packs/suites while the testers create more tests and we (the automators) can work on requests for new functionality or enhancing existing functionality.

Mark Smith.


lockdown
(Member)
04/12/12 06:05 AM
Attachment
Re: 'My Beautiful Framework'

I've attached what we've got. It's pretty similar to most keyword approaches, only with an added GUI on top.

The GUI makes it really easy for the manual testers to know which objects they're working with and what the actions do, it does add a bit of an overhead when we add test objects - basically for each object added to the OR we have to "map" it into another Excel workbook and:
- Say where in the GUI tree structure it should be saved.
- Capture a screenshot.

This doesn't actually take too long as we have a screenshot capture tool we launch from the mapping workbook where we just drag around the object and it saves the screenshot for us.

This mapping worksheet also tells the framework which high level keywords are available to the GUI tree nodes, and which low level actions are available to the objects.

Timescale wise, it's evolved over the past 6 months. Fortunately I've been given time to develop this, I'm hoping to publish some stuff around it later in the year when I get some free time (and permission).

Regarding failure / recovery, we always call an initialise action at the start of each test - this determines if we need to shut down and clean up or if we can carry on from the open application.


GregEncke
(Member)
04/12/12 08:11 AM
Re: 'My Beautiful Framework'

Quote:

Quote:

I do a screen shot, exit the test and have a function that closes down resources.




Greg this is what my upper management also wants gracefully exit the test and clean up. But my point is unless its a worst case scneario why should we exit the test (this should be the last resort right?).
The best thing to do would be make qtp recover from the current error and try and re correct itself (as much as possible / proceed with a workaround)




Agreed, you have to know when to leave and when to try again.


Shreejit
(Member)
04/13/12 11:21 PM
Re: 'My Beautiful Framework'

@lockdown - Amazing stuff

vishimon
(Newbie)
04/15/12 04:56 AM
Re: 'My Beautiful Framework'

Good work @lockdown.

We have built a similar GUI based framework in my previous organization.. Likewise to yourself, we have several forms which display the business functions, utility functions, test cases etc. The tester can create new keywords using the GUI or simply reuse the existing ones in his test case.

The major difference is that this GUI is made from Classic ASP (built about 3.5 yrs back) and all the information (except OR) is fed into a SQL database i.e. all keywords, test cases are stored in an Sql database.

The advantages of using such a framework to us was specially
1) No code/script integration hassles i.e. in most of the keyword driven frameworks integrating the excel spreadsheets, OR maps and business functions of testers can be a pain at times especially if there are more than 3-4 of them.

2)Keywords immediately available i.e. Keyword created by Tester A is immediately available to Tester B as soon it is created. This has been one of the biggest gains.

3)The GUI + DB sits on a centralized server. We can actually schedule and execute automation batches on independent machines via the server machine itself. This feature helps us to ensure that the test execution is completed overnight i.e. if the batch is huge we can load this on multiple machines.

4) Using this framework improved our automation productivity as testers were forced to use the correct fn signatures, OR names etc. This was to address the issue we had observed that testers usually got their initial development runs wrong mostly due to spelling mistakes in excel; incorrect parameter given, etc.

The framework was developed in phases and it took us about 6 months to get to the stage we are with this framework.


No one is thinking if everyone is thinking alike !!


JonRLewis
(Member)
04/16/12 03:01 AM
Re: 'My Beautiful Framework'

I have read a lot of descriptions of Keyword Driven frameworks, and in almost all cases the focus has been on the clever technical aspects of their code - whether its using Data Dictionaries or integrating classes with VB - but almost no thought seems to be given to the interface and the non-technical person who will be using it. As a consequence the end-user is usually expected to remember the name of every keyword and the order and acceptable values of every function parameter. It has been really refreshing to read about some great solutions that have put at least as much effort into the interface as into the application.

My first attempt at a Keyword Driven framework (a couple of years ago) ended up with a VB.Net interface and a SQL Server database. This made it possible to create an interface with pre-populated dropdown lists for the screens, objects, actions and functions, and help text for what everything did and what values could be used, etc. (something very similar to what Vishimon was describing).

I have now moved to a consultancy which led to me looking for a more mobile, transferrable solution and this in turn led me to take another look at spreadsheets. The solution I have arrived at is to split the data from the scripting tool. There is a single Data spreadsheet with worksheets for each application with the following columns: Screen, Object, Object Class, Object ID. Object ID is the DP code for the object and thus eliminates the need for an OR. There is also a worksheet for the functions - columns: Application, Screen, Name, Description, Parameter1, Parameter2, ..., ParameterN. The Parameter columns hold a description of each parameter with permitted vales, etc. There is another worksheet for SubTests with Application, Screen, Name. When you open the Scripting spreadsheet, the first thing it does is use ADODB to query the data spreadsheet pulling out all the application, function and subtest data which is used to make Named Lists to use in Data Validation. Doing it this way means the technical people can update the data and changes are available whenever you open a script. For the scripter, what they get is a series of dropdown lists populated according to the other selections made, i.e. Column A is for Applications - once an application is chosen Column B (Screens) is populated with all the screens defined for the selected application, and so on. Functions can be generic (i.e. not linked to any specific application) e.g. CaptureScreenshot, or linked to a particular application (e.g. Login) or screen. When a function is selected help fields are populated with descriptions of the function and parameters, taken from the data spreadsheet. I have a button on the Scripting spreadsheet that will build a QTP script, and I'm part of the way towards allowing that script to be saved to QC.


Shreejit
(Member)
04/16/12 05:26 AM
Re: 'My Beautiful Framework'

@ Jon - Could you share any snapshot for better clarity?

lockdown
(Member)
04/16/12 05:44 AM
Attachment
Re: 'My Beautiful Framework'

Thanks guys

Database or Excel - I think it all comes down to what you're comfortable developing in. The things that I took into consideration were:

1) With excel we can cut and paste sets of keywords from other tests, use formulas in paramaters (e.g calculate dates for boundary tests), it's an environment people are comfortable using and it's allready installed on the test machines. Plus I know that I can easily extend it through VBA to add more bits in when I need to. Where I'm working, to use anything else would have been too much red tape and procurement of additional programming tools / developer resource.

2) I decided to use the QTP repository feature for the framework - there's a lot about it that I like, especially using the spy to grab all the objects on a gui which I can then tweak, or using the update from application feature. More importantly I knew that I would be training people up to add objects to the framework, so the OR tools in QTP are good for novices. We have to spend a little extra time mapping but that gives us our screenshots and logical structure that you can't always achieve through the OR or DP. Having said that, the way the framework works (execute() command) means I can use DP or even reference a function that returns an object if I wish to.

3) Team structure - With the GUI in place making it easy to use (and restricting input so incorrect values aren't entered) it means manual testers can write scripts, novice QTP engineers can work on mapping tasks, scripting too, and simple programming, and advanced engineers can work on more complicated functions and framework maintenance.

4) QC Integration - There's a 1:1 ratio between QTP test and test workbook which makes QC reporting and metrics easy. The downside to this is that you need a QTP test to represent each test case. I had to develop a tool takes the test workbooks, creates them in QC and then generates a test lab linking to the tests. It's pretty neat and gets a few smiles from people around me who see it as "automating the automation".

5) Maintance - What happens when the GUI/flows change? This is the biggest downside to having all the scripts in separate workbooks. To alleviate the pain, Ive written a search and replace tool that can go through all of the test workbooks, identify specific keywords and replace them with others.

6) How easy will it be to maintain the framework when I leave? All the excel stuff is essentially VBA & macros which alligns closely to the skills of QTP engineers.

7) Efficiency - I've looked at what overheads this approach introduces. The object mapping is made easier by letting users select the QTP objects from a treeview and launching a screenshot snagging tool saves us messing around with print screen and ms paint. So once you've added an object to the OR there's on average a 60 second overhead per object to then map it in the workbook and tag it up correctly. I've attached an example of how this looks if anyone wants some ideas.


Shreejit
(Member)
04/16/12 09:48 AM
Re: 'My Beautiful Framework'

@lockdown - Nice stuff buddy

mwsrosso
(Veteran)
04/16/12 10:32 AM
Re: 'My Beautiful Framework'

Quote:


1) With excel we can cut and paste sets of keywords from other tests




Lock we decided to have a Favourites sheet where the testers can put blocks of often used code that can be called from the Excel test script sheets.

So you can call Favourites lin x yo y and pass/receive any data required, sounds a bit messy but works good.

Mark Smith.


lockdown
(Member)
04/17/12 12:50 AM
Re: 'My Beautiful Framework'

@Mark: I spotted that on your screenshot, it's a neat idea. We could do with something like a "favorites" menu as searching for the relavent code in another workbook and then copying across is a bit cumbersome.

RPNarayana
(Newbie)
05/15/12 03:01 AM
Re: 'My Beautiful Framework'

Keywords, I follow the following methods
Maintain all the keywords in a Dictionary (giving the details of the keyword)

eg.

Name,
Description (purpose of the Keyword)
Actions (what the keyword does as much detail as possible)
Pre and Post keyword Names(where applicable)

Whoever wants to build the test using the keywords will now have a place to go to when they are unsure of anything.

Also keep the keyword names matching the actual functional or process names which the keyword is related to.

Hope it helps


vampire123
(Member)
05/15/12 02:06 PM
Re: 'My Beautiful Framework'

Waht do you mean by a subTest?
Is it possible to share the framework code?


JonRLewis
(Member)
05/16/12 01:17 AM
Re: 'My Beautiful Framework'

The user can save a series of steps as a SubTest, thus allowing them to create their own 'functions'. For example, they could create a script that launched their application, retrieved a set of credentials and logged on to the application with those credentials. They could save this as a SubTest (call it 'StartApp'), then in their next script they would select the function called RunSubTest and the StartApp SubTest would be available in a dropdown list in the Parameter1 column.

I can't share the code but I'm happy to answer any questions.


Mrinal_Nandi
(Member)
05/16/12 03:42 AM
Re: 'My Beautiful Framework'

All,
Good to see so many people sharing knowledge about frameworks. Got a very good idea about Keyword framework with some UI layers above it.

In my past project, we have developed Hybrid framework which have

1) 3 function file:-
1.a) Generic Function ex Reading the excel, Log file creation, Result updating.
1.b) Application related function ex: Login application, Log out application, search,
1.c)Environment Variable: Global environment variable declaring.

2) Input excels:
2.a) Test case wise data,
2.b) Generic data Ex: URL,USERID,PASSWORD.

3) Log File: Log file created for every execution.

4) Result File: Final result created in HTML format.

5) OR: We have around 100 test case for that we have used Shared OR.

We have used Environment variable like TestDir which can help to make the entire path relative.


mwsrosso
(Veteran)
05/16/12 04:47 AM
Re: 'My Beautiful Framework'

Something I haven't seen mentioned with an Excel based framework and I know we had to think long and hard about this before we implimented it, was allow recursive looping (For Next, While, Do Loop).

What we eventually implimented was a 'Do Until' Keyword which allowed you to loop until a given condition was True/False or loop a number of times.

has anyone else had to impliment looping in an Excel based framework and what method did you use.

Another thing we implimented was a report that highlighted objects in the OR that were not referenced in any test.

Mark Smith.


Shreejit
(Member)
05/17/12 09:09 AM
Re: 'My Beautiful Framework'

Hi Mark,

We have tried some luck with DOM for recursively trying to find objects next to objects in a complex web environment
Something like say i want to validate presence of text B next to A on the UI
A:<space>B
On different UI's we have different situations B (could be the nextsibling or B could be nextsibling.nextsibling)
A:<space><space><space>B
We try to loop across till the end of the row till we find B and break out of the loop once the next sibling object is found or object is Nothing
Single keyword called ValidateTextByText() for all UI's but keep it flexible to search across the entire row
Do Until obj.nextSibling.Innertext = "B" OR obj IsNothing
Set obj = obj.nextSibling
Loop
Not the exact code but similar logic


mwsrosso
(Veteran)
05/17/12 10:21 AM
Re: 'My Beautiful Framework'

Hi Shreejit, this is a general Loop that will work for all the scenarios we envisaged encountering, things like:

Loop while object exists/not exists.
For Next loop from 0 to UBound for an array.
Loop over the next x lines of code n times.
Loop while a condition is true/false

I try to avoid 'one-off' keyword functions where possible.

Mark Smith.


MJP
(Member)
05/17/12 01:57 PM
Re: 'My Beautiful Framework'

Hi Mark,
I have a simple loop command that is used in an Excel based framework. On a sheet with work flow commands there are usually some setup actions then a loop statement followed by the rest of the actions. On the first iteration the setup steps run. The following iterations start at the loop statement and skip the setup steps. The number iterations is the length of the data sheet(s).
Regards,
M.



Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5