SPONSORS:






User Tag List

Thanks Thanks:  0
Likes Likes:  0
Dislikes Dislikes:  0
Page 1 of 9 123456789 LastLast
Results 1 to 10 of 86
  1. #1
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Share your Performance Testing/Engineering War Stories...

    This thread is actually a tangent taken from another thread, but it was fun and interesting so I thought I'd give it a place of it's own.

    Share with us your best Performance Testing/Engineering War story. (ya'know the ones, that time when you got blamed because YOU found the performance problem, or that time when you warned them what was going to happen and they didn't believe you, etc)

    Here are two to start us off.

    Posted by RSBarber:
    A very large financial related company had me in to performance test their new web app. I tested and tested and told them there was simply NO WAY their infrastructure could handle the load. They told me I was wrong, said it was because we were accessing the www from inside the network - had to test remotely. So I went back to my home office and ran the scripts remotely at a low load. Told them again that there was NO WAY the app would hold - infrastructure was insufficient. They told me I was wrong and to run the scripts at full load (I got that in writing). So I did. It ended up taking down their entire external network costing them litterlally millions in down time, and fried some rather expensive equipment. Needless to say, they weren't happy with me stopped work and didn't want to pay. (We eventually won that arguement and got our $).

    I would argue that I did my job, did it well, documented it, and found the performance problem. They would argue that I cost them $ millions.

    Posted by Steve Jones:
    I was at a client site to performance test their internal web application. Told them they had a memory leak (sustained 200 users, and response times kept increasing). Because the tool I was using initially had some issues handling the session id's and I had to submit support tickets to get patches made, they blamed the tool. So, I showed them the logs that the requests we being sent properly, then they blamed me saying that I wasn't writing the scripts correctly. So I told them to write one correctly Then I stepped them through my script to show them I was doing everything right.

    After that, they went to the development manager and said that they have a memory leak. Again, he tried to point the finger at me, then they did find a tiny problem, fixed it and let me hit it again. Same thing, 200 users increasing response times. They didn't believe me, so they all stood over my shoulder watching the test run. And let me tell you, when you are watching paint dry you understand that the paint is drying even though it may not look like it. Well, for these managers, this was a concept that was a little hard to grasp. The 200 users had to be sustained for about an hour and 15 minutes before the response times increased.

    Once they saw what has happening, they pulled in an IBM person to look at the AIX boxes to see what was wrong. Long story short, and 3 more IBM people later. One of IBM's WebSphere experts, within 5 minutes of watching the server logs, knew it was a problem with their WebSpere and MQ architecture. MQ was filling up, and once full was dropping requests. This explained why it took so long for the 200 users to see their times increase.

    Even though I saved them money in the long run, the architects felt that I was EXPOSING them, so they got me phased out of the project.



    ------------------
    Scott Barber, Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    http://www.perftestplus.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  2. #2
    Moderator
    Join Date
    Aug 2000
    Location
    Vancouver, BC, Canada
    Posts
    1,189
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    When we started bringing in a Performance Test tool at my client's we were looking for little pilot projects to develop our procedures and gain experience with the tool and fine tune our methodology.
    At that time, the first Internet application (for external deployment) was developed and the project team agreed that it would be interesting to test their application.

    Our first test showed that the application could not sustain more than 1 user at the time because of database deadlocks (requirement was 20 concurrent users). The architect was adamant that it was our tool that distorted the truth because it pushed the application harder than necessary. We politely disagreed and ran several realistic simulations (wait times and all) to prove our point. Sadly this did not convince the belligerent architect.

    The project team ended up organizing a manual load test with 20 users (including client representatives) walking through some of the user scenarios. This painfully (remember the client was there) showed the same problems we already found, but now the team faced significant embarrassment.

    A quick fix was identified and we were asked to test this again. We tested and reported that the system could now support two users but that after that the same deadlock errors were occurring.
    Again our results were dismissed (same arguments, same architect) and yet another manual load test was performed. This load test resulted in the same observations as we had made. Then it finally dawned on the team that there was some serious trouble in their application. Management jumped in after the client raised a bit of a stink and a $250,000 re-architecting, re-development exercise was performed.

    Our test results were never doubted again, but the application never became a stellar performer either. The good news is that there are no deadlocks anymore when the system runs with loads under 40 users concurrent.


    ------------------
    Roland
    Roland Stens

  3. #3
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    LOL - Excellent! Thanks for sharing, Roland. (I don't know why, but I love hearing these stories from other people. I guess it's just nice to know that other people go through the same things.

    ***Not to all performance testers/engineers***

    Sometimes the "hire a bunch of interns" method of performance testing really is useful!

    ------------------
    Scott Barber, Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    http://www.perftestplus.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  4. #4
    Senior Member
    Join Date
    Nov 1999
    Posts
    164
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    Shooting the messenger seems to be standard practice. First stop in resolving those HTTP 500 server errors is commonly to blame the load generating tool. In a strict sense this is right of course - if it wasn't for the load, there wouldn't be any errors - but it kind of misses the point.

    Other people have other ways of handling problems, and it's also commmon to have people want to know how they can ignore a wide range of system errors - all those 404s on the home page do spoil the look of a nice management report after all.

    One thing that I sometimes find useful for a hard-core of people who simply will not believe results, is to ask them to manually exercise the application (eg browse the site) when the load test is being run. Doing this, they will often encounter the errors themselves which can help prove the point. It can however easily backfire:- if they don't encounter problems themselves, then they end up doubly convinced that the issue is with the tool. In the early stages of performance degradation that's very possible since only a small proportion of users may be experiencing problems.


    ------------------

  5. #5
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    Stronzo,

    I find that it is a tremendous help to really educate the developers on what the tool does and how it works before reporting the first issue. Catch one litte 404 or something during script development - have them verify it in the log, etc. Tends to get them on your side fast.

    Of course, I also find that going to the developers with issues FIRST before logging defects tends to help too, but depending on your organizations procedures that is not always possible.

    If we're gonna tell war stories, we should probably throw in a little advice about how to avoid similar problems in the future - when we have some.

    ------------------
    Scott Barber, Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    http://www.perftestplus.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

  6. #6
    Junior Member
    Join Date
    Aug 2002
    Location
    MENLO PARK, CA, USA
    Posts
    24
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    This discussion is great fun. Ok let me write down why some problems happen at load and not at normal times.
    a. Lack of synchronization in multi-threaded environment.
    b. using final static Java variables instead of instance variables.
    c. database row locking
    d. Abnormal webserver-servlet container bridge behaviour, GC, system maintenance phase during the run etc. This part is totally murky. I suspect clustering may be exposing some faults as well.
    Anyway my story on how e can do errors as performance testers-
    a. Incorrect data/iteration
    b. Lack of sufficient data with app.
    c. Lack of suficient knowledge of app. depending on data and configuration the flow of events in app may change. Cure- do a slow tango, run the scripts covering the entire data at slow clip, trying to cover as much data as you can.

    Now about performance project management. Chaos rules supreme. results are incomprabale due to ever changing network speed, database load, no of vusers, the rate of inputting the users. Every change needs to be applied in isolation to study the effect. But where is the time?

    I did have my share of doubting Thomas. The DBA repeatedly tried to run me down. Warned me I will lose my job after I tried to monitor database. They suspect tool, script and my ability. I use to open ten windows(lets say out of 50 Vusers) and show them what is appears on one of the screen. Another help is LR snapshot on error. Finally you look into the logs and the code yourself. However memory leak is the toughest thing to handle owing to the time it takes.

    Well after I was laid off my manager explained me why I was seeing the errors. we both learnt a few things.

    Currently I am working on project there are replacing a maniframe system with web system and giving us 1.25 Mbps line to let upto 200 users to run. This line is also shared for emails and other apps. We have also implemented data compression to handle the issue. This is almost first performance test exercise for my company. First the onsite team changed to data compression without informing me. Now when recording an action I went into tizzy after seeing binary data. After two days of research I discovered the source and the fix. LR will not take data compression. Now the developer onsite writes who is this jack and why I need to inform him for something a tester does not need to know. The client totally controls the machine, app server and database and allows no tuning( wrt outsourcing this should tell- whipping Vendor or being authoratarian is not the answer, even if the vendor is Indian). I made a couple of performance points to developer but there was almost incense in his replies (wrt developer vs QA reflects his perception of the skills of QA/testers).
    My recent found principle of working is do your job but there is no point in raising a ruckus on what is beyond the capability of the developer. WRT importance of testers, , testers are important in >0.5% defect companies. Where even a dodo would find bug obviously developer is more important. One would believe developer knows the performance but that is a myth. A developer know EJB well, servlets well but not solaris tunables, oracle buffer size importance, webserver design, webserver-app server bridge constraints, LDAP configuration. No DBA knows hows weblogic frames queries or how Websphere does that and how we can pass suitable hints to them. Most of the performance tunings are accidental and usually the tricks are empirical- make full use of resources-increase the threads, one BEA instance for two CPUs, index well.
    Back to the main theme of war stories- I am itching to fit in 'Beautiful mind' example. There is something going on out there, that only you can see. I do not believe in fighting too hard to make them see it. I would rather spend this time looking for personal advancement.
    Steve, so what was the resolution for your MQ Series problem?
    regards,
    Vik
    nb a.WRT: with respect to
    b.I referred to some other posts in the forum



    [This message has been edited by cvik (edited 01-05-2003).]

  7. #7
    Senior Member
    Join Date
    Nov 2002
    Location
    Columbus, OH, USA
    Posts
    245
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    cvik,

    Not sure what the actual resolution was to the problem. I know that they launched an emergency project to re-architect the application from the server side. Due to some hostilities from a competing consulting firm who wanted nothing more than for me to stop testing their app, and the fact that I was on a week by week contract where they ran out of money, I left the project before they got it finished. However, I did come back a couple months later as part of a larger team to help suppliment their QA staff and was able to sneak back into doing the performance testing and everything was working well by that time. I never did get to find out how much was actually changed.

    ------------------
    Steve_Jones@SoftHome.net
    Always remember that you are unique. Just like everybody else.

  8. #8
    Moderator
    Join Date
    Aug 2000
    Location
    Vancouver, BC, Canada
    Posts
    1,189
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    What still happens to me on a regular basis is that well-defined and thorough performance requirements suddenly become not important anymore or are even considered seriously flawed and not worth keeping.

    Example: We worked with the client to determine the maximum concurrent user level under certain scenarios. We went through a fairly extensive brainstorming and estimating exercise all based on actual historical information. At this point in time everybody is happy and in total agreement.

    Then we start to test and find out that there are serious problems reaching the maximum amount of concurrent users. Now suddenly, our (not the client's although they had the lead on this!) estimating sucks, our methodology is flawed, we are overly optimistic/pessimistic, we are confused and "no reasonable person would ever write a requirement like this".

    For us, this is then back into review mode and revisit the requirements. But more often than not, the requirements are changed to fit our results. These changes are basically based on nothing but on the incapability of the software to deliver the original requirement. But time runs out and the project team needs to move forward. yadayada

    Drives me nuts, but if the client signs off on it, we're fine.
    Actually, we're not. Typically a few months later somebody gives us a call about the problems there are having with app. so-and-so.

    And then we just start all over again.



    ------------------
    Roland
    Roland Stens

  9. #9
    Senior Member
    Join Date
    Aug 2001
    Posts
    420
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    my story is different, we supposed to deliver a project to the client next week, and I supposed to run a performance testing this week to pinpoint any bottle neck, the problem is, I cannot find any? I run test with 10 to 50 users (our max license is 50 vu) and I couldn’t crash sever or find any issue…the problem is I cannot tell my manager that there is no performance issue since this project has never been performance tested before…

    ------------------

  10. #10
    Senior Member
    Join Date
    Jul 2002
    Location
    Palm Bay, FL USA
    Posts
    2,346
    Post Thanks / Like
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Total Downloaded
    0

    Re: Share your Performance Testing/Engineering War Stories...

    This is where stress testing applies. I bet with 50 VU's doing creative, performance intensive activities with small to no delays you can find SOMETHING that could be better!

    Good Luck!

    Truth is, though, that clients are equally as likely to say your tests are inaccurate when you find that everything meets the requirements as when you find defects. Welcome to the life of a Performance Tester/Engineer!

    ------------------
    Scott Barber, Sr. Performance Engineer
    sbarber@noblestar.com
    http://www.noblestar.com
    http://www.perftestplus.com
    Scott Barber
    Chief Technologist, PerfTestPlus
    Executive Director, Association for Software Testing
    Co-Author, Performance Testing Guidance for Web Applications
    sbarber@perftestplus.com

    If you can see it in your mind...
    you will find it in your life.

 

 
Page 1 of 9 123456789 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Search Engine Optimisation provided by DragonByte SEO v2.0.36 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 9.38%
vBulletin Optimisation provided by vB Optimise v2.6.4 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
User Alert System provided by Advanced User Tagging v3.2.8 (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
vBNominate (Lite) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Feedback Buttons provided by Advanced Post Thanks / Like (Pro) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
Username Changing provided by Username Change (Free) - vBulletin Mods & Addons Copyright © 2016 DragonByte Technologies Ltd.
BetaSoft Inc.
Digital Point modules: Sphinx-based search
All times are GMT -8. The time now is 10:28 AM.

Copyright BetaSoft Inc.