The online community for software testing & quality assurance professionals
 
 
Calendar   Today's Topics
Sponsors:




Lost Password?

Home
BetaSoft
Blogs
Jobs
Training
News
Links
Downloads



Software Testing >> Performance & Load Testing

Pages: 1
rhill84
Junior Member


Reged: 11/27/02
Posts: 7
Loc: England
Where to drive Load tests from?
      #45095 - 10/22/04 08:58 AM

Hi,
I have been given the task of writing a test plan to performance test our large multi tier web based system.

The testing will be conducted on the production environment to prove the actual hardware and infrastructure,firewalls etc... in the DMZ.

With this in mind, where should the performance testing be driven from?

If i drive the tests from the LAN, doesn't the LAN itself become a bottleneck?

Is the ideal setup to have the loadinjectors pointing directly at the system under test so other LAN traffic does not conflict the test metrics?

OR

Should the load traffic be routed via the internet?

Thanks for your help.

PS We are using Rational.?


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Where to drive Load tests from?
      #45096 - 10/22/04 11:14 AM

your LAN most likely has orders of magnitude more bandwidth than your WAN link. You will be able to drive much more load by staying off the internet.

Post Extras: Print Post   Remind Me!   Notify Moderator  
manojjain99
Advanced Member


Reged: 09/21/01
Posts: 495
Loc: Hyderabad, India
Re: Where to drive Load tests from?
      #45097 - 10/22/04 11:10 PM

Hi,

In this scenario, if response time for an action is 4 seconds @ 200 user load (LAN environment)then it would take more time when over internet. You have to take care about the difference between the bandwidth of WAN and LAN as this would be the only difference between your production and testing environment.

If you can route the traffic through internet then it would be more closer to production environment and results would be more accurate.


Post Extras: Print Post   Remind Me!   Notify Moderator  
olapidot
Member


Reged: 05/08/04
Posts: 85
Loc: Israel
Re: Where to drive Load tests from?
      #45098 - 10/23/04 04:34 AM

I think the best compromise would be to drive most (95%) of the traffic via LAN, with some Virtual Users using WAN. The "LAN Users'" job would be to load servers, while the "WAN Users" would be emplyed in order to feel the impact real users feel over the web when the servers are under load.

Post Extras: Print Post   Remind Me!   Notify Moderator  
olapidot
Member


Reged: 05/08/04
Posts: 85
Loc: Israel
Re: Where to drive Load tests from?
      #45099 - 10/23/04 04:39 AM

I think the best compromise would be to drive most (95%) of the traffic via LAN, with some Virtual Users using WAN. The "LAN Users'" job would be to load servers, while the "WAN Users" would be employed in order to feel the impact real users feel over the web when the servers are under load.

Post Extras: Print Post   Remind Me!   Notify Moderator  
richard.hill84
Member


Reged: 08/18/03
Posts: 34
Re: Where to drive Load tests from?
      #45100 - 10/25/04 03:36 AM

The internet is a variable that can constantly change in terms of speed.

Therefore if the internet is used to channel load traffic, the metrics I draw (response times gathered via Perf Studio) would differ from hour to hour.

Should the internet not always be excluded from from performance tests because of this reason?


Post Extras: Print Post   Remind Me!   Notify Moderator  
jbrusard
Member


Reged: 01/16/03
Posts: 47
Loc: Norwood
Re: Where to drive Load tests from?
      #45101 - 10/25/04 05:45 AM

Also, part of your load test is to identify if the LAN is in fact a bottleneck. In testing this, I would do a survey of runs during high usage and low usage ..ie( normal buisness hrs and non usage times) This will show certain latency issues for sure.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Where to drive Load tests from?
      #45102 - 10/25/04 06:08 AM

quote:

Should the internet not always be excluded from from performance tests because of this reason?

I tend to agree with this (assuming you are sure your WAN link has enough capacity and all network components it uses have been tested).

When testing application performance and load handling, I would rather have a set of results and present them with the caveat that the response times users will experience will include additional time due to Internet/WAN latency.

I don't see much value in giving hypothetical "over the internet" times when they vary so much.

Some of your users will be sharing a T1 with many users, some may be on dialup, some may have a dedicated T3 with a QoS SLA, some may have packets delivered by camels.

[ 10-25-2004, 07:17 AM: Message edited by: Corey_G ]


Post Extras: Print Post   Remind Me!   Notify Moderator  
bhomes
Member


Reged: 10/26/99
Posts: 88
Loc: France
Re: Where to drive Load tests from?
      #45103 - 10/25/04 06:49 AM

If the testing is on the production hardware, it might be better to test it thru the internet.

Although internet has variable loads, it is (in my opinion) better to test thru the internet as you will then go thru the standard incoming routes to the application (including firewall, etc.).

When showing the answers you should provide a range of response times, not a single value, thus reflecting the multiple executions at different time of the day (thus with different internet loads).

Performance testing is (should be?) about finding the bottlenecks in the response time. If you remove some of the bottlenecks from the start, the 'caveat' suggested by Corey_G is not measurable and could lead to unacceptable response time for the end-users.

You have to decide if it is better to have an answer with a range of values (and a small uncertainty) or a more precise value with an unknown value to be added (the latency and load on internet).

I usually start with a web spider providing me with data on the size and load time for the URLs of the website (tool : eValid at www.soft.com/evalid). From then you can use any tool (including Rational's to load test the application.

Hope this helps.

Regards
BH


Post Extras: Print Post   Remind Me!   Notify Moderator  
nathan.white
Member


Reged: 05/27/03
Posts: 222
Loc: St Louis, MO
Re: Where to drive Load tests from?
      #45104 - 10/25/04 07:07 AM

I agree with Corey and with BH to some extent. For our client-facing app, we have a set of severs that set outside of our firewall. However, they're still connected via a LAN. Like Corey, I don't see a lot of value in actually testing through the internet. If you know how your users connect to the internet (camels or T3), most tools will allow you to simulate connections speeds, though, from my point of view, I only care about our hardware/software. We run our tests at LAN-speed, which would cause worst-case performance on your servers. If you test at 28K, your servers may not even break a sweat. Remember, it's all about how your servers and infrastructure perform, not a single user.
Nate


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Where to drive Load tests from?
      #45105 - 10/25/04 07:36 AM

quote:
Although internet has variable loads, it is (in my opinion) better to test thru the internet as you will then go thru the standard incoming routes to the application (including firewall, etc.).

of course your network infrastructure needs to be thoroughly tested. But this does not need to be the same test effort.

quote:

When showing the answers you should provide a range of response times, not a single value, thus reflecting the multiple executions at different time of the day (thus with different internet loads).

Which Internet loads? your ISP? Your WAN link? your user's ISP? Which users? Which ISP? Which backone are they going over? How is the congestion on the network? Which network?
I just see way too many variables to get anything useful.

I think hearing "response time for transaction X is going to be approximately 1.5 seconds + network latency" is more valuable than hearing "response time will be between 2.0 seconds and 45 seconds, depending on the time of day and how the user is connected"

quote:

If you remove some of the bottlenecks from the start, the 'caveat' suggested by Corey_G is not measurable and could lead to unacceptable response time for the end-users.

Bottlenecks exist in your architecture/infrastructure/configuration/application code. Performance/load testing can have many different goals. It's extremely important to have your infrastructure tested (which is why my previous post stated "assuming you are sure your WAN link has enough capacity and all network components it uses have been tested").

I just don't see the value of wrapping all of that into your application performance testing, and I think it give you a confusing picture of system performance/scalability.


Post Extras: Print Post   Remind Me!   Notify Moderator  
rhill84
Junior Member


Reged: 11/27/02
Posts: 7
Loc: England
Re: Where to drive Load tests from?
      #45106 - 10/26/04 12:37 AM

Thanks for your constructive comments...
Seems like I have started a bit of a debate here.

All our Internet traffic is routed via another office in London. It is only a 1mb pipe between the offices and this seems to be throttled quite often by general Internet usage. This even makes browsing slow! UH.

This is my other concern about routing load traffic via the Internet because in any set-up the traffic will surely be bottlenecked before leaving the building where the load injectors sit.

However, the performance testing must go through the externally facing firewall as this is a key piece of hardware in the system that may cause problems.

Has anybody tried loading up an externally facing system through a firewall but not going via the Internet?

I think I should speak to our network guys to see what they say possible.

Any comments you have would be great.


Post Extras: Print Post   Remind Me!   Notify Moderator  
nathan.white
Member


Reged: 05/27/03
Posts: 222
Loc: St Louis, MO
Re: Where to drive Load tests from?
      #45107 - 10/25/04 01:13 PM

As I mentioned, we do that with our client-facing app. We place a few of our load servers outside the firewall and drive the test. They are connected to firewall via LAN-like speeds.

Post Extras: Print Post   Remind Me!   Notify Moderator  
bhomes
Member


Reged: 10/26/99
Posts: 88
Loc: France
Re: Where to drive Load tests from?
      #45108 - 10/27/04 09:19 AM

Corey
quote:

I think hearing "response time for transaction X is going to be approximately 1.5 seconds + network latency" is more valuable than hearing "response time will be between 2.0 seconds and 45 seconds, depending on the time of day and how the user is connected"

In the end the result is the same, isn't it ?
The only difference is that you do not detail that "network latency" can go from 0.5 to 43.5 seconds, so the manager has either less information to make an informed decision, or needs to ask you the network latency. (by the way, I have not noticed such a large discrepancy when working over the internet, so the point might be moot)

quote:
Performance/load testing can have many different goals.

Agreed, but who is the final customer/end user ? The one who will go thru the internet to purchase an item or someone on your own internal network ?
And you should also add the results from the infrastructure tests to any performance/load tests results.

quote:
I just don't see the value of wrapping all of that into your application performance testing
Then we differ, I believe it's better to be as close to reality as possible, even if it adds complexities in your performances tests.

quote:
I think it give you a confusing picture of system performance/scalability.

Performance test is inherently complex and we make choices that have an impact on the response times. These choices include (but are not limited to) ramp-up times, think times, type of transactions, etc, and include things we can not easily influence such as server interaction between the different processes already running on the servers.
The Internet induced delay can be considered as one of these choices. It provides more realistic information but analysing the results is more complicated.

Regards
BH


Post Extras: Print Post   Remind Me!   Notify Moderator  
nathan.white
Member


Reged: 05/27/03
Posts: 222
Loc: St Louis, MO
Re: Where to drive Load tests from?
      #45109 - 10/27/04 09:38 AM

BH,
If you take your line of reasoning further, what sort of and how many different browsers are you going to emulate in your scripts? What about processor types/numbers, bus speeds, memory, etc. in the client PCs? There's thousands of combinations that you should be testing and reporting on as all of these combinations will affect the end user experience as well.

For us, the goal is to make sure the infrastructure will support the user load and our "virtual users" experience acceptable response times. We make it clear to our business users that these virtual users do not represent what a real user would experience.
Nate


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Where to drive Load tests from?
      #45110 - 10/27/04 11:09 AM

quote:
In the end the result is the same, isn't it ? The only difference is that you do not detail that "network latency" can go from 0.5 to 43.5 seconds, so the manager has either less information to make an informed decision, or needs to ask you the network latency.
my point is that the latency will basically be an arbitrary number and have nothing to do with your application or infrastructure. Do you consider your application slower if your user connects to a saturated switch through a dialup connection? You probably do. I do not. Your infrastructure performs the same. You have no control over his connection.


quote:
I believe it's better to be as close to reality as possible, even if it adds complexities in your performances tests.
I don't think it adds complexity at all.. it just decreases clarity of yor results.


anyways.. not much more to say on this thread.


Post Extras: Print Post   Remind Me!   Notify Moderator  
bhomes
Member


Reged: 10/26/99
Posts: 88
Loc: France
Re: Where to drive Load tests from?
      #45111 - 10/28/04 07:29 AM

Nate,

Yes, there are lots of combinations, and configuration testing can cover the functional aspects of these different configurations. If you have the opportunity to create load tests with different combinations, so much the better

For me the goal of a test is to load an infrastructure in order to obtain response times reflecting reality (ie user experience) as much as possible.

Let me provide an example : if the link between the company network and the internet is limited, and that one tests directly on the company network, the bottleneck of the link will not be noticed. However your users will experience delays and switch to other suppliers.
On the other hand if your tests are run from outside, this can be detected.

I stress this aspect of realism because I encountered it previously.

When a user buys a product at Amazon and has bad response times, the user will not check if the cause is a slow link or a slow server response, the blame will be placed on Amazon. And this will happen even if the server at Amazon can support the load on their internal network.

Regards
BH


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Where to drive Load tests from?
      #45112 - 10/28/04 07:39 PM

BH,

I think you completely missed the point we were making.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1



Extra information
0 registered and 14 anonymous users are browsing this forum.

Moderator:  AJ, RSBarber, rstens, Corey_G 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 1931

Rate this topic

Jump to

Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5