The online community for software testing & quality assurance professionals
 
 
Calendar   Today's Topics
Sponsors:




Lost Password?

Home
BetaSoft
Blogs
Jobs
Training
News
Links
Downloads



Software Testing >> Performance & Load Testing

Pages: 1 | 2 | 3 | >> (show all)
QAStudy
Junior Member


Reged: 03/15/06
Posts: 5
Loc: Minnesota
Need opinion from exp. load testers- skillset
      #47176 - 04/11/06 01:31 PM

I am a QA manager in the process of interviewing consultants for a number of positions I have open.
I am surprised at the number of folks I am seeing who claim to have experience with multiple load testing tools as well as things like QTP and WinRunner. Frankly, I am skeptical.
So my question, for those of you who have been load testing for a number of years: Is this the kind of skill you can pick up without formal training? How long did it take you to develop finess with your particular load testing application. Do you have any advice for me for sorting out the good from the over-inflated?


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Need opinion from exp. load testers- skillset
      #47177 - 04/11/06 01:48 PM

I'd say about 50% of the QA resumes I have seen list "LoadRunner" (or comparable tool, but it always seems to be LoadRunner) as a skill. However, out of those that list it, only a VERY VERY VERY small percentage of those have even the slightest clue about load testing and performance.

>Is this the kind of skill you can
>pick up without formal training?

Yes absolutely. "Formal" training tends to take the form of a few days training from a tool vendor. After having completed that much training, you would still be basically clueless how to do proper load testing. Being self trained, or better yet; being mentored by someone with experience would be sufficient.


>How long did it take you to
>develop finess with your particular
>load testing application

knowing the tool well accounts for a small percentage of the overall knowledge you need to be succesfull. Learning to use a tool from Rational, Mercury, or Segue, is pretty trivial. You can be competent using the tools in a matter of hours, and master them in a matter of months. However, learning proper load testing and performance tuning skills takes years.


The difficulty is if you are not familiar with load/performance testing yourself, and you have to interview someone for this. Ideally you want someone experienced in this area to help interview so they can ask relevant questions. Its pretty simple to screen out the clueless.


Post Extras: Print Post   Remind Me!   Notify Moderator  
SteveO44
Active Member


Reged: 07/08/04
Posts: 1157
Loc: St. Louis, MO, USA
Re: Need opinion from exp. load testers- skillset
      #47178 - 04/11/06 01:49 PM

I'd say experience is the best "formal training" anyone can get and good knowledge of performance testing and their concepts should be tool independent. You either have it or you don't.

I can use a hammer, but I sure as heck can't build a house with it.


Post Extras: Print Post   Remind Me!   Notify Moderator  
jason_m
Advanced Member


Reged: 03/28/05
Posts: 627
Loc: St. Louis
Re: Need opinion from exp. load testers- skillset
      #47179 - 04/11/06 02:21 PM

As direct experience - we hired someone with five years Loadrunner experience, and fired them after six weeks. He couldn't tell us what an HTTP 200 was.

Ignore the tools, and interview on the protocols. Present the candidate with a problem (the server is responding with 503 errors) and ask them how they would diagnose it. This will tell you far more than the resume.


Post Extras: Print Post   Remind Me!   Notify Moderator  
JakeBrake
Moderator


Reged: 12/19/00
Posts: 15290
Loc: St. Louis - Year 2025
Re: Need opinion from exp. load testers- skillset
      #47180 - 04/11/06 02:47 PM

Put the candidate on a workstation with Vugen. Have the candidate record, parameterize, and correlate a script. This is the acid test for scripting skills.

[ 04-11-2006, 04:38 PM: Message edited by: JakeBrake ]


Post Extras: Print Post   Remind Me!   Notify Moderator  
RSBarberModerator
Moderator


Reged: 07/18/02
Posts: 1949
Loc: Palm Bay, FL USA
Re: Need opinion from exp. load testers- skillset
      #47181 - 04/11/06 08:29 PM

You all covered it pretty well, but let me put a different spin on it.

I've been considering taking my list of tools OFF my resume because it's nothing more than buzzword bingo at this point. The truth is that if I can't spend a weekend learning my way around a particular load generation tool, my client and I would be better off just picking a different tool (probably a free one).

That said, when I go into an interview, the first thing I tell someone if they start asking me about a tool is...

"If you want to test me on the tool, bring it in, but to me, it's a hammer. Stanley, DeWalt, K-Mart, it really doesn't matter to me - just tell me that we'll have a hammer and that you are willing to replace it if it breaks."

Basically, my point is that if the person you are interviewing is stressing their expertise with one (or more) tools, try to get the person to talk to you about performance testing without mentioning tools and see what you think. Someone who is worth hiring should be able to tell you a compelling story without needing to show-off their tool expertise.

My son is almost 7. I can teach him how to use a hammer, but I'm still the one building the shelves.


Post Extras: Print Post   Remind Me!   Notify Moderator  
JakeBrake
Moderator


Reged: 12/19/00
Posts: 15290
Loc: St. Louis - Year 2025
Re: Need opinion from exp. load testers- skillset
      #47182 - 04/12/06 04:54 AM

QAStudy,

Based upon the above responses, you might want to elaborate a bit more. If you are looking for a scripter/scenario developer / test executioner as opposed to a consultant who can lead the entire process to make sure the test assets serve to answer established questions about performance; then the skill sets need to encompass more.

My own recent experiences show that only one in forty resumes' touting this discipline have significant amounts of truth. Your concerns are correct based upon my experience. If you ever cruise the LoadRunner forum, you will see a surprising amount of evidence to support your concerns.


Post Extras: Print Post   Remind Me!   Notify Moderator  
swt88
Active Member


Reged: 05/13/03
Posts: 1193
Loc: Massachusetts
Re: Need opinion from exp. load testers- skillset
      #47183 - 04/12/06 06:31 AM

I was hired as a manual tester, but I'm currently working in a 'performance' team because I had "LoadRunner Experience". NOTE I had listed it at the bottom of my resume as 'tools used' - in no way did I claim any expertise.

When they were discussing this switch with me, I was given the impression that other team members were all "very skilled" in performance testing using LoadRunner. I kept emphasizing that my experience was small: I had used it less than 10 months and had only created scripts and executed small scenarios, not done any of the anaysis of the results.

3 months into it now and I find that I am the most skilled person in the group, both as a LR scripter and in basic analysis.

There is one engineer who is very skilled at the analysis of the data - both from LoadRunner results, the application logs AND the OS System Logs. He has been doing this for many years. Fortunately, I am learning quite a bit about this process from him.

So agree with what everyone has said so far - especially:
quote:

From Corey_G: ...knowing the tool well accounts for a small percentage of the overall knowledge you need to be successful. Learning to use a tool from Rational, Mercury, or Segue, is pretty trivial. You can be competent using the tools in a matter of hours, and master them in a matter of months. However, learning proper load testing and performance tuning skills takes years.

From jason_m: Ignore the tools, and interview on the protocols.



Post Extras: Print Post   Remind Me!   Notify Moderator  
James Pulley
Moderator


Reged: 08/01/01
Posts: 5551
Loc: NC
Re: Need opinion from exp. load testers- skillset
      #47184 - 04/12/06 07:27 AM

Mary,

Your experience illustrates why one has to go through 30++ or more resumes before one finds a "qualified" resource in LoadRunner.

Pulley


Post Extras: Print Post   Remind Me!   Notify Moderator  
tusharpandya
Member


Reged: 03/16/03
Posts: 122
Loc: Frisco, TX USA
Re: Need opinion from exp. load testers- skillset
      #47185 - 04/12/06 02:49 PM

I have come across this situation multiple times. I am a "Performance Tester" and often I look for people in support roles. Most will claim that they DID NOT prepare the scripts but only ran the load tests (clicked on the go button), when exposed. Also, some people who have used automated testing tools for regression testing think load is another record and playback tool just like win runner without realizing that it works at protocol level.

I would start with questions like what is http status code 503 or what is winsock error code 10054 to filter out.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Logi
Member


Reged: 03/12/01
Posts: 200
Loc: Moutain view/US
Re: Need opinion from exp. load testers- skillset
      #47186 - 05/10/06 12:24 AM

Throw the numbers ..... I often give them my benchmark report ask them whether they see any scalability issue with the data i provided .

Post Extras: Print Post   Remind Me!   Notify Moderator  
JakeBrake
Moderator


Reged: 12/19/00
Posts: 15290
Loc: St. Louis - Year 2025
Re: Need opinion from exp. load testers- skillset
      #47187 - 05/10/06 04:37 AM

quote:
Originally posted by Logi:
Throw the numbers ..... I often give them my benchmark report ask them whether they see any scalability issue with the data i provided .

Good idea, but it doesn't expose tool skills-level.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Logi
Member


Reged: 03/12/01
Posts: 200
Loc: Moutain view/US
Re: Need opinion from exp. load testers- skillset
      #47188 - 05/11/06 01:30 AM

I can live with a person who is not Loadrunner/SilkPerformer expert , but has got good understanding in Analysis part .

In my opinion it is easy learn the tool.


Post Extras: Print Post   Remind Me!   Notify Moderator  
JakeBrake
Moderator


Reged: 12/19/00
Posts: 15290
Loc: St. Louis - Year 2025
Re: Need opinion from exp. load testers- skillset
      #47189 - 05/11/06 04:22 AM

quote:
Originally posted by Logi:
I can live with a person who is not Loadrunner/SilkPerformer expert , but has got good understanding in Analysis part .

I can't! [Smile]

The original poster - QAstudy was not talking about "experts", nor am I. It takes months to become fluent in a few protocols and years to become expert at a tool like LoadRunner. If I hire a person for a LoadRunner job, I need someone who knows LoadRunner. To me it is less than immediately important if the person has the vaguely "good" understanding of the "Analysis part". (I expect they will improve on that with time). For one, if a person claiming LoadRunner experience can't do the work required to get to the "Analysis part", 1) they are of no use to me and, 2) they do not then have LoadRunner experience and nor is the person a LoadRunner person! If the person is weak on the "Analysis part", I can bring other skillsets (dbas, network & sys admins, etc.) to bear on that part and usually do. I rely on many sets of eyes to look over the whole landscape of the analytical data. I expect the LoadRunner person to listen and learn from the other eyes, with an ultimate goal of performance testing, tuning, and engineering - along the lines of Corey's suggestion.

My previously posted metric - one in forty is now one in forty one. Deception is on the increase!

http://www.qaforums.com/ultimatebb.php?ubb=get_topic;f=33;t=001866

Trying to stay within the confines of this part:
quote:
Originally posted by QAStudy:
... Do you have any advice for me for sorting out the good from the over-inflated?

... restating my method: sit the candidate down with the tool and then show me! And that is only a small part of my filtering process for the tool itself.

[ 05-11-2006, 04:53 AM: Message edited by: JakeBrake ]


Post Extras: Print Post   Remind Me!   Notify Moderator  
Logi
Member


Reged: 03/12/01
Posts: 200
Loc: Moutain view/US
Re: Need opinion from exp. load testers- skillset
      #47190 - 05/15/06 12:55 AM

Jake , My opinion was influenced by my day to day work , In my case we have been testing the sampe product for years and scripting is one time activity. In our case we spend so much time tweaking the DB , Application server rather than working with the tool .

So if you are from a services company which requires people to work on multiple projects then whatever you suggest is right , But on product development company i dont see people spending too much of time with the tool .

Again i might be wrong as i speak from my experience and not sure about other product companies.

cheers.

[ 05-15-2006, 01:01 AM: Message edited by: Logi ]


Post Extras: Print Post   Remind Me!   Notify Moderator  
tusharpandya
Member


Reged: 03/16/03
Posts: 122
Loc: Frisco, TX USA
Re: Need opinion from exp. load testers- skillset
      #47191 - 05/30/06 02:39 PM

Are you using the same script build after build? I recommend that you re-record and clean up scripts every time you get a new build. The script recorded against previous build will certainly be "out of sync" with the current build.

TP

quote:
Originally posted by Logi:
Jake , My opinion was influenced by my day to day work , In my case we have been testing the same product for years and scripting is one time activity. In our case we spend so much time tweaking the DB , Application server rather than working with the tool .

So if you are from a services company which requires people to work on multiple projects then whatever you suggest is right , But on product development company i don't see people spending too much of time with the tool .

Again i might be wrong as i speak from my experience and not sure about other product companies.

cheers.



Post Extras: Print Post   Remind Me!   Notify Moderator  
DSquared
Moderator


Reged: 04/02/03
Posts: 4546
Loc: Wisconsin, USA
Re: Need opinion from exp. load testers- skillset
      #47192 - 05/31/06 04:12 AM

Another spin. Jake and Tushar sort of hit on it.

The skillset required to determine what to build, to build, to maintain, and to execute a load test is quite different from the skillset required to diagnose and repair the problems diagnosed by a load test.

An example from my past life. I was head of a load testing team. The team consisted of myself serving in the roles of coordinator and lead load tester. I had another person who was an "assistant" load tester. Then we had a DBA. And an application specialist. And a network guy. All of us had different pieces of the pie. I worked with the other load tester to specify the what the mix of functionalities and the percentages were for the test. We then gave the results over to the team to diagnose. The DBAs did some table and index tuning. The app guy fixed things like select statements inside of loops instead of outside. The network guy worked on tuning routing tables on the network.

I guess my point is that you need to decide what exact skillset you are looking for in your "load test" person. I don't think you are going to get a one-size-fits-all universal person who can do the whole thing independent of other players.


Post Extras: Print Post   Remind Me!   Notify Moderator  
tzeng
Member


Reged: 02/07/00
Posts: 67
Loc: san jose,ca,usa
Re: Need opinion from exp. load testers- skillset
      #47193 - 06/10/06 12:45 AM

My opinion is:
Tools are easy to learn. But it is good to have some experience.

How to analyze a system is hard. Very hard.
Unless you already know how to analyze a queuing
network.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Need opinion from exp. load testers- skillset
      #47194 - 06/10/06 07:02 AM

quote:
I recommend that you re-record and clean up scripts every time you get a new build.
That is a very narrow outlook, tusharpandya, and load testing is a VERY wide subject.

In a past life, I was running scenarios against Oracle Apps. We had about 15 "live/current" scripts. Those scripts had taken anywhere from several days to several months to build and perfect. Re-recording them from scratch when Oracle came out with the next minor release (or build) would have been impossible.

Quote from your experience, certainly, but with the statement that "this is, of course, only true in the environment I'm accustomed to."


Post Extras: Print Post   Remind Me!   Notify Moderator  
Corey_GModerator
Veteran


Reged: 09/14/01
Posts: 4281
Loc: Boston, MA
Re: Need opinion from exp. load testers- skillset
      #47195 - 06/10/06 08:08 AM

quote:
Originally posted by tusharpandya:
Are you using the same script build after build? I recommend that you re-record and clean up scripts every time you get a new build. The script recorded against previous build will certainly be "out of sync" with the current build.

is your API really that unstable? ive never worked on a project that resembles what you describe.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | 3 | >> (show all)



Extra information
0 registered and 15 anonymous users are browsing this forum.

Moderator:  AJ, RSBarber, rstens, Corey_G 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating: *****
Topic views: 18015

Rate this topic

Jump to

Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5