SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Page 1 of 3 123 LastLast
Results 1 to 10 of 23
  1. #1
    Apprentice
    Join Date
    Feb 2011
    Posts
    29
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Configure recorded UI scripts as load tests?

    hi all,

    what I'm looking for is a tool that will:

    - enable me to record and play back test scenarios on a Windows app (note it IS a Windows app and NOT a web app)
    - enable these recorded scripts to be configured as load tests

    Tools that can record and playback don't seem to have any trouble working with our app, but it seems harder to find a tool that will enable the scripts to be configured as load tests.

    Basically, what I'm envisaging is:

    1. record some basis scenarios, including logging into the app
    2. configure these scripts as load tests - specifically:

    - 'tell' the tool to start say 200 instances of the client Windows app, and login to each instance as a different user
    - call scripts that perform say two basic test scenarios for some/most/all of those login sessions

    We don't currently have resources who can manually code a load testing solution, and that's why I'm looking for tools that can do the above.

    This may well be a tall order (or impossible), but this seems to be what my boss wants.

    So, in short, can it be done?

    thanks a lot in advance for any guidance

  2. #2
    Moderator Joe Strazzere's Avatar
    Join Date
    May 2000
    Location
    USA
    Posts
    13,170
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    [ QUOTE ]
    'tell' the tool to start say 200 instances of the client Windows app

    [/ QUOTE ]
    Are you sure that's what you really want to do? This is not a typical approach.

    How many machines are you going to use? If less than 200, do you think this would adequately mimic what will happen in production?

    What happens if you manually start 200 instances of your Windows client app on less than 200 machines?

    What is the intended target of your load test? That is, where are you measuring this load? The server? The Windows desktop? If the server, most tools would just record the traffic between the desktop and the server, then play it back using 200 virtual users.
    Joe Strazzere
    Visit my website: AllThingsQuality.com to learn more about quality, testing, and QA!

  3. #3
    Apprentice
    Join Date
    Feb 2011
    Posts
    29
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    thanks Joe

    yeah, I agree, this may not be the best approach, but it seems to be what my boss is after. Ie, he thinks a load testing solution can be developed without coding skills, by recording scripts and then configuring them as load tests.

    we want to be able to load test with about 5,000 to 6,000 concurrent users

    so the idea would be to have about 200 instances of the client app (each with a different logged-in user) running on each of about 25 virtual machines

    so in that way, we'd apply a load of about 5,000 concurrent users. I've no real doubt that this would adequately simulate a production server load of about 5,000 concurrent users.

    we simply don't have the hardware resources to support hundreds of virtual machines

    we're only interested in the load on the server, not the load on/performance of the client (of course, if we're running 200 instances of the client app on a machine, the response time of the client app on that machine is going to be completely unrealistic)

    I'm wondering whether recording the traffic between the desktop and the server, and playing it back with virtual users would be an adequate simulation of our system because:

    - user's permissions to access certain things are controlled by their Role, and a lot of server resources are used in determining what a user can and can't see according to their Role
    - the client app regularly sends various types of data to the server (at configurable intervals) - this places considerable load on the server

    Further, the client communicates with the server via a unique, custom protocol - and the data packets are binary, so I'm not sure we'd find any test tool that can deal with that.

    Any further advice/guidance very much appreciated!

    cheers

  4. #4
    Senior Member
    Join Date
    Jun 2011
    Location
    London, England
    Posts
    243
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    Your boss is wrong. Joe, on the other hand, is right.

    Your boss probably has little or no experience with load testing. Our man Joe has, well, I'm guessing a little more than your manager.

    Who do you trust?

    Seems to me that your approach is a train wreck waiting to happen.

    I very seriously doubt that you can run 200 client instances of your application concurrently on one machine and properly simulate a production load. You might be able to if your 'client' is feather light, has zero or very little interaction with the system and no GUI interface. Is that he case? If so, great, if not, then you need to concentrate on using a test tool that will simulate only the traffic flowing between your client and server - this is how performance testing can be done without using 1000s of machines.

    The actual manner in which this traffic will be simulated obviously varies but even the most arcane requests will pass over sockets and LoadRunner and friends will be able to pick this up and use it.

    But think before you go ahead, your company will have a budget assigned to this project, the best thing you can do is work out the best way to spend this money to get maximum benefit. You could go out and buy an enterprise level tool but can you use it? It's takes a lot of time to learn these things and even longer when the protocol is not standard.

    My feeling is you would be much better of looking at some sort of outsourced option. If it is a short term, one-off requirement this will cover your needs. Or, if you are looking to create a long term solution then you should consider hiring in someone with experience of what you want to do that can advise you on the right approach.

  5. #5
    Moderator Joe Strazzere's Avatar
    Join Date
    May 2000
    Location
    USA
    Posts
    13,170
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    [ QUOTE ]
    yeah, I agree, this may not be the best approach, but it seems to be what my boss is after. Ie, he thinks a load testing solution can be developed without coding skills, by recording scripts and then configuring them as load tests.

    we want to be able to load test with about 5,000 to 6,000 concurrent users

    so the idea would be to have about 200 instances of the client app (each with a different logged-in user) running on each of about 25 virtual machines

    [/ QUOTE ]
    Ok. I don't know what "configure them as load tests" means in this context.

    But let's assume that you only want to record and playback a script, on 200 instances of your client app, on one machine. I suppose if you can do that, you could do the same on 25 virtual machines and achieve something.

    First, attempt to manually run 200 instances of your client app on one machine. If that works, continue. If not, perhaps you could consider using 2000 VMs, each running 1 instance of your application?

    Next, find a tool that can automate your application. I use WinTask, but any other functional test automation tool may suffice. If money is a big concern you may wish to try something like AutoIT. Try to create a recording that automates your 200 instances using the roles and actions you desire.

    Finally, see if it generates the traffic you expect. If so, fire up 25 VMs and you are in business.

    Your boss may be right, but more likely, you'll find that 200 instances of your client app doesn't do what you hoped. I think you would be far better served with something like LoadRunner (and perhaps a consultant that understands how to use it).

    Good luck.
    Joe Strazzere
    Visit my website: AllThingsQuality.com to learn more about quality, testing, and QA!

  6. #6
    Member
    Join Date
    Aug 2006
    Posts
    344
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    We've run into similar issue at my workplace as well. And we had to do something similar not because of mandate by management or bad QA design, but because the Windows GUI app involved in the load testing could not be appropriately simulated with a back end API. So we had to launch multiple instances on a given machine, for X machines to generate a client load against the server being tested. And we couldn't affort a 1 instance to 1 machine ratio, so we try to fit as many as possible on a given machine/VM.

    What did help us in our case, and hopeful in the original poster's case is if the GUI app had some automation hooks into it that helps for driving load. Ours didn't offer much but it did at least offer command line parameters that would start up the client in certain configs (or user logins) as well as tell it to occassionally change views (which would cause some load activity). That way we could launch all these clients via a batch file and have it do some automation w/o involving any GUI automation/load test tools.

    For the original poster, it would be worthwhile to see if you can bring up issue and have your developers add some automation hooks/APIs into the client to do something similar as ours.

  7. #7
    Moderator
    Join Date
    Aug 2001
    Location
    NC
    Posts
    6,018
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    [ QUOTE ]

    ....but this seems to be what my boss wants....


    [/ QUOTE ]

    It is a known antipattern for success to have someone who does not understand a process well directing how something should be conducted.

    It does not matter if something as complex as performance testing is involved or if it is something as simple as changing a bike tire.
    James Pulley

    Replace ineffective offshore contracts, LoadRunnerByTheHour. Starting @ $19.95/hr USD.

    Put us to the test, skilled expertise is less expensive than you might imagine.

    Twitter: @LoadRunnerBTH @PerfBytes

  8. #8
    Moderator
    Join Date
    Aug 2001
    Location
    NC
    Posts
    6,018
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    1 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    daluu,

    What was the driving technical reason that prevented an API level test from beign conducted? These are such rare animals (Technically mandated GUI Tests) that I am wondering about the technical path that mandated the use of a GUI model.
    James Pulley

    Replace ineffective offshore contracts, LoadRunnerByTheHour. Starting @ $19.95/hr USD.

    Put us to the test, skilled expertise is less expensive than you might imagine.

    Twitter: @LoadRunnerBTH @PerfBytes

  9. #9
    Senior Member
    Join Date
    Jun 2011
    Location
    London, England
    Posts
    243
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    Perhaps there is a misconception being made here whereby it is perceived the 'better' option to use GUI scripts / multiple instances to run a load test because it is thought to be 'easier'...

    I think there are scenarios where this is true but I also think in most cases it turns out to actually be a lot more work, often very expensive and also a lot harder to go down the GUI / multi instance path than people first thought.

    Classic example (slightly off topic) is: Selenium Grid... "Selenium grid is not a load testing tool!" I might put that as my signature.

  10. #10
    Apprentice
    Join Date
    Feb 2011
    Posts
    29
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Configure recorded UI scripts as load tests?

    hi all, and thanks for the responses.

    apologies for being absent from this thread for a while: I've been somewhat indisposed (among other hindering factors).

    Apologies also for the length of this post, but it was hard to avoid...

    I'm very inclined to agree that the track I've been directed on to is the wrong one, but I suppose my task is to persuade the boss that this is so.

    we do currently run hundreds of instances of the client app on single machines (and execute basic test scenarios for those instances) in order to load test (using an in-house-developed load test tool), and this has been reasonably effective.

    However, this tool is lacking in features/sophistication (in particular, it provides no diagnostic tools) which is basically why we're looking to acquire a load test tool. This way of generating system load is adequate for our load testing purposes: I'm not sure why Oliver feels it wouldn't be? Note that we're only interested in server load, not client load. The client app is not feather light, it does have a lot of interaction with the server and yes, it has a GUI.

    I've looked at a few tools - but LoadRunner is ruled out because its cost is far in excess of our budget.

    Basically, we are looking for a load testing solution for the longer term (so outsourcing it is not favoured), and I've little doubt that to actually make this work, we'll need to hire a technical person to help us. My boss has indicated to me that he's completely agreeable to hiring
    such a person, but he wants me to recommend what tool we buy. My problem is that I feel you need a great deal of technical knowledge just to assess what tool/solution would best fit our needs - and I feel ill-equipped. However, I'll continue to try to do my best...

    I get the impression that there are a number of tools that will enable us to record the data packets going between the client and the server, however (and my understanding of this is a little vague) at some point we need to be able to interpret what's in the data packets to make the load test solution work. And the challenge there is of course that the app uses a custom protocol and binary data packets.

    Joe: the basic snag I've hit with trying to run automated UI scripts in multiple instances of the same app on the same machine is: how do you program the tool to differentiate between different instances of the same application? Eg:

    1. you have 3 instances of the AUT running
    2. you have a UI automation script that enters a search term in the AUT and returns results
    3. if you want the script to execute the scenario in each instance of the AUT, how do you get it do so?

    I'm thinking this may be one of the main reasons why load testing is done by recording and playing back network traffic - ie, due to the difficulty of playing back a UI script in multiple instances of the same app.

    daluu's problem domain sounds very similar to our's. And at this point I should say that I've been thinking about another potential solution. The client app does have APIs that can be used to execute logins as well as basic user scenarios. The APIs are used via XML data in HTTP packets - the current in-house-developed load test tool (that I mentioned above) does use these client APIs. I'm thinking what I may try to do is:

    1. start up and run our custom load test tool
    2. use a tool to record the data packets being sent from the tool to the client APIs
    3. try to play back the recording of the network traffic, to see if this may be a way of effectively generating load

    It does seem that using the client APIs may be a less difficult solution to implement as these use XML data wrapped in HTTP, rather than custom binary data packets.

    Alternatively, I guess a load test tool could be programmed to send XML data in HTTP to the client APIs (as opposed to trying to record and playback HTTP packets)

    daluu: I was interested to see that you used a batch file to start up multiple instances of your client app, because that's how we currently start multiple instances of our app on a single machine.

    James: I'm in furious agreement: basically, it's been obvious to me all along that this is a very complex, specialist area requiring skills and experience I don't have. And that in order to actually make any progress, we're going to have to get help (one way or another) from people that know what they're doing. I feel this is the way it's going to go...eventually.

    So anyway, basically it seems I have the following options:

    1. record UI scripts and somehow play them back in multiple instances of the application in order to generate load (seeming like the least favoured option)
    2. record and playback HTTP packets that can 'talk' to the client APIs to generate load, via a load test tool
    3. program a load test tool to send and receive XML data (in HTTP packets) to/from the client APIs to generate load

    Further advice and thoughts very much appreciated! [img]/images/graemlins/smile.gif[/img]

 

 
Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 10.00%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 05:41 AM.

Copyright BetaSoft Inc.