| || |
Trouble with hiring SQA folk...
Iím having some trouble putting together a SQA group. The development department is growing rapidly as are the products in the RAD environment. I have 2 core products (web based) and some custom web sites to work on.
What I would like to do is pair up a qualified SQA Lead (have 0 right now) to each product / project and have a floating pool of Jr. testers work with the leads as well as occasionally assign them to a web site for which they will learn the trade of QA. These SQA Leads I would expect to have QA experience and be able represent my SQA group with professionalism. Their skills should be peer level to the programmers working on the product. But my main focus is to have experienced QA engineers that know the methods required and how to balance / implement them.
So as the testers / analysts learn more and the company grows, they will be ready to take on these lead projects and represent QA in the project groups. These candidates should at least be technically inclined, motivated to follow a career path in QA, and have a drive for learning.
My problem is this: lack of candidates in the area and the company (upper management) is expecting less of the candidates and is yielding a ďlets just get some people (warm bodies) in hereĒ approach. With the market the way that it is, Iím getting very little qualified candidates and a lot of low level knowledge candidates. My argument to them is the question of whether or not they would hire that caliber of experience to develop their products. But even with the answer of no, I still struggle with this issue when Iím setup with interviews with non-technical people that have absolutely no experience or education in technology and / or QA.
So am I wrong? Should I just accept this? I understand that a lack of qualified candidates may force me to being more flexible, but how flexible? I want my SQA group to be represented by firm skills that can profit the company by implementing QA the right way.
The other part of this is that we have developers that may push tasks to a lead member that may be developments job. I think that this may be an influence to management from development that we just need people in here to test (not caring of the skill set). There is little to no resistance from an inexperienced person that does not understand the ownership of tasks and what is expected both ways in a process. This would be a bad thing and difficult to mentor and guide this person and may confuse and deter them from a career in QA.
Maybe Iím just venting but I would like to hear of any similar experiences and what others have done or may do in my situation.
Re: Trouble with hiring SQA folk...
Brian...Boy did you ever pull my heart strings!
All I can say is "been there, done that".
And that "you are absolutely right"!
I've been in QA for 15 years and nothing can take the place of an experienced tester. You need to tell your management that you should have at least a supporting cast of senior testers to guide your team of junior testers (unless, for some reason, they think that you can do it by yourself).
Your post cannot be more accurate. I commend you on your forsight.
My horror story is this:
I was the only tester of a startup. I eventually grew the group with junior testers. I was their mentor and "taught them everything they know". As the company got busier, I justified the need for more testers. Unfortunately, "upper management" insisted that "anyone can test". As such they decided to give me developers that were not entirely busy. I figured "why not...they designed the product?"
What happened is that they gave me the most junior developers that only knew a little unit testing. They knew nothing about integration testing, product testing, second and third order testing, and what I like to call "putting on the customer's hat" (i.e. customer representation). My, then, senior testers spent so much time ramping them up and reviewing their work that we essentially did the same amount of work with twice as many people. My testers begged me never to let this happen again so I created financial and budget reports outlining how ineffective and expensive this type of "resource allocation" ended up. Suffice to say, this didn't happen anymore.
Do what you can to argue your point. My scenario isn't the same as yours but the issue is. You need a well established and EXPERIENCED test team. In fact, your accusations of development influencing management could be accurate. Like I said, "been there, done that". I don't know why anyone has this attitude that "anyone can test"? It must be from the old days when all tester had to do was to see if the red light or green light turned on. Products are way to complex today and requirements are more complicated. Putting it all together and testing it as the customer would, takes a special skill. Like you said, they have to want a career in QA. I've fired a few people that became testers because they couldn't find a development job (yet). Their heart wasn't in it and this was something to fill in their time until then. These people were the most useless and really dampened the morale in the test team.
Another thing you mentioned was the fact that THEY set up the interviews. Don't you review the resumes first and forward the good ones to call? That was the authority I had with the startup. And as mentioned...all my juniors eventually became seniors. Insist that you at least look at the resumes first and key in on anything that resembles the phrase "Looking for a career in QA". Be careful though, it could be in the cover letter.
HOWEVER, to slip in an arguement for your management team, perhaps they really do feel that you can build a test team alone using zero skilled personnel? That's how I started one of my teams, but I timed it so that I didn't end up in rushed projects with unskilled people.
Anyways, you vented, so I vented. I wish you luck brother! If you want, feel free to retell my story to your managers. If you want to take it one step further, drop me an email...cuz I can really relate!
PS...between you and me and this forum, we can vent, but please do not take this approach with your managers and collegues. Animosity can develop between QA and Development which will effect productivity. Try your best to "educate" them.
Re: Trouble with hiring SQA folk...
A lot of this is, as kjee stated, education. You have to show why you need what you need and, since you are dealing with management it only makes sense to show what you need in relation to the bottom line of the business - revenue. It sounds like you need some quality assurance processes in place. What you have been describing is the quality control (testing) aspects. If you have procedures that are being circumvented, there needs to be a stop-gap on that. This requires an escalation path but that will only work if you have buy-in for the procedures in question in the first place. Quality assurance should be the one providing that stop-gap - not the testers. These are cultural issues and it sounds like one thing that was not done initially was a quality implemention plan. This is something I try to do right from the moment I walk in the door - even if they claim they already did one.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>My argument to them is the question of whether or not they would hire that caliber of experience to develop their products.<HR></BLOCKQUOTE>
This is somewhat specious because there are two different mindsets that you are arguing for. Testers should not have the same mindsets as developers and vice versa. This is not to say that they should not be able to be able to step into the others' shoes at an overall level but they are different so that argument would probably not win your management over.
What I look for is not always the technical, but the mindset. Find people that balance curiosity with being obsessive. Find people that are pragmatic without being dogmatic. Anyone can be trained on a tool or a process. But it is harder to train in an attitude. A lot of times if you get people with the right mindset you can start them off simply by instilling a "test to break" attitude.
My answer: you cannot. At least not in any good amount of time given that you have other tasks. That is not fair to them or you. If you are dealing with non-intuitive people or people that just cannot grasp what needs to be done even from the standpoint of using the GUI, you have a problem. And this needs to be brought to management. If someone comes into a technology company today without even a basic computing skill set I demand to know who interviewed this person and by what criterion were they hired.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>So am I wrong? Should I just accept this? I understand that a lack of qualified candidates may force me to being more flexible, but how flexible?<HR></BLOCKQUOTE>
Put that to the business. You might want to relate your flexibility to their bottom line quality results. Factor in training time and anything else into a little spreadsheet and show them, visually, how this addition of having to train people essentially from the ground up will seriously add to your person-hours and completion dates. Since it is unlikely that completion dates will be moved back on your behalf, you can then point out that this means product will go out the door untested. Personally, what I like to do is show my possible testing coverage in relation to the business critical areas of the software/Web site/etc. and show, graphically, how much will not get tested and what this might mean. If you graph your person-hours with your test scenarios this should be easy to throw together. I use this to show my coverage problems when we have testers of different skill sets but also when there are a limited number that are being shuffled around.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>The other part of this is that we have developers that may push tasks to a lead member that may be developments job.<HR></BLOCKQUOTE>
Again, here it sounds like a lack of established quality assurance (not quality control) standards. Quality assurance should provide your escalation path and your standards that are agreed upon by the required members of management. There is also the matter of educating your testers about what their roles and responsibilities are. That is part of what makes up a good process. You can even say that all such requests have to come through you and make that point known to the development leads as well.
Your points are all valid and I think we can all feel your pain and no doubt everyone will offer different solutions based on their experiences. But what I think it boils down to is established processes that are followed or not followed. Part of that is making management understand why those processes are in place and why they should be followed. There should be a system of quality in place such that this can be referred to back to for disagreements about who has what role and who can pass the buck on to whom. Also remember to show management visually, via person-hours or test-hours, what will hinder you in your release cycle when you have to spend time training or looking over everyone's shoulder.
Re: Trouble with hiring SQA folk...
Thought you might find this email interesting. This email came across the Winrunner user group today as a response to a post, where someone requested info on "how to best prepare for (or how to best answer) Winrunner interview questions."
I completely agree with Jena (see below)..
In a message dated Thu, 5 Oct 2000 9:40:18 AM Eastern Daylight Time, "Jena Buttimer." <firstname.lastname@example.org> writes:
<< I've done many technical interviews for job candidates. What I have found is that there are many people who have WinRunner experience on their resume's, however during the tech interview, it shows that they have no idea at all. Alot of candidates were able to "fake it" because the person giving the interview knows even less about test automation, and the candidate knew all the buzz words.
My advice is, if you don't know it, or only know record/playback, be HONEST!!! Many a red-faced candidate who passed all the other interviews has left the technical interview with no job. If you tell me you know it, I expect you to know it! If you are honest about your experience, I will gear my questions toward basic knowledge and try to judge if you have learning potential.
I found so many people thought they had more experience than they really had, that in order not to waste time, I developed a little test for the candidates to take before the interview. Basicly, I gave them a set of instructions that would test their knowledge of WinRunner, simple programming techniques, and best practices. "
Re: Trouble with hiring SQA folk...
Elfriede, I agree with what that person is saying. In fact, I have similar questions and tests that I place on the candidate to see what level they are at. And yes, honesty is the best approach. I'm would hire someone who does not know and is honest before I would hire someone who is just trying to get a job. I can see right through someone that is not being honest. I am very well versed with all areas of QA and the tools used. I also know of the buzzwords and fish through them for what the person really knows. Some of my questions are subjective, however, that allows me to get a feel for what they would do in given situations and how they would fit in. Some may not be too good with the buzzwords but understand the details. Screening is important. I also inserted a bug in an application that was somewhat obvious to a QA engineer and gave them a chance to find it. Some did, some did not. Not the primary deciding factor but a good way to learn of the personís observation skills.
I believe in doing this because I once worked as a QA Eng. for a company that made these mistakes. The candidate was prepped well but deep down, did not have a clue. I saw this but the individual was convincing enough to the QA Manager. He got the job and I had to work with that incompetent fool. It was not because he lacked knowledge, but because he faked his was through everything that he did. So when push came to shove, he proved his true colors and did not last too long after that.
Some of the questions are:
What is quality assurance?
Whatís the difference between white-box and black box testing?
Whatís a software error?
Who is the link between support, engineering, and QA?
A qualified person should not struggle with the above although you may get different answers, all may be valid, but at least you can wean out those who are faking it right about here.
I have more depending on the candidate and they can get very detailed. Also, itís good to give them a scenario, whether it is a process issue, software defect, or documentation issue.
I know who I'm looking for. Just need to find them.
Re: Trouble with hiring SQA folk...
I appreciate you comments and suggestions. In response to Jeff, I do have processes but need people to execute them. I have been a stickler for them and I believe that it is a large part of QAs job to ensure that these processes exist and are used. If you throw in a Jr. Level tester into a project to represent QA, they will easily get the hat pulled over there eyes. A Sr. Tester / QA Lead, would understand when the hat is being tugged and know how to stop it without starting a war.
I had a Jr. guy working with me earlier this year and he had worked off an on for the company performing many tasks (updating user manuals, doing customer support, assisting with troubleshooting that could have been done by development). When he came into QA I had a serious talk with him about what QA was for and better defined it to him. Not micromanaging him, he sought out to learn more about it and did just that. He was easily persuaded to wearing whatever hat was thrown at him and would vary his tasks based on immediate requests. I almost cried when I saw him isolate a very complex issue for a developer and declined assistance to the developer to troubleshoot that issue. It was fantastic to see this happen with this individual.
I would like to clarify the following:
-Quote from JeffNyman-
"This is somewhat specious because there are two different mindsets that you are arguing for. Testers should not have the same mindsets as developers and vice versa. This is not to say that they should not be able to be able to step into the others' shoes at an overall level but they are different so that argument would probably not win your management over."
I don't mean their specific abilities, rather the caliber of experience that they possess. A 3-5 year seasoned developer that has done well in his or her career should be matched with the same. And this also applies to salary.
The company culture is pretty open and accepts processes and QA as I have presented it to them. I'm not shut out of a vote to what happens even with development processes. My only concern is that I need people that will represent QA that will carry enough expertise to make this useful.
I have many processes in place and some may need to change as we go. What I may need to do is provide upper management with # like Jeff said. It has been kind of sudden that I've been feeling the pressure to hire. But yes, I agree with you both that I need to provide them with a documented graphical view of the outputs based on the hires.
Re: Trouble with hiring SQA folk...
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>In response to Jeff, I do have processes but need people to execute them.<HR></BLOCKQUOTE>
Just as a quick note, if people do not execute them, they are, technically, not processes. Part of something being a process means it is executed. If there is no execution, there is no process - there are just good ideas. Having said that, however, I know what you are saying. It is sometimes very hard to get people to do what needs to be done. Sometimes relating this to your global vision and showing how that is practical at a day-to-day level is a good route to go.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>If you throw in a Jr. Level tester into a project to represent QA, they will easily get the hat pulled over there eyes.<HR></BLOCKQUOTE>
I agree but, then again, a junior level tester should not represent QA (Quality Assurance). They should represent QC (Quality Control). I know people balk at hearing quality control because it sounds like I am talking about an assembly line but I just refer to that as the testing branch of QA.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>A Sr. Tester / QA Lead, would understand when the hat is being tugged and know how to stop it without starting a war.<HR></BLOCKQUOTE>
Quite possibly. Also consider someone who is more oriented towards project management or process improvement. A senior level testers does not necessarily have process improvement skills for QA.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>The company culture is pretty open and accepts processes and QA as I have presented it to them.<HR></BLOCKQUOTE>
I guess it is hard to say without knowing your structure because it sounds like QA and QC are lumped together. Many times I know management looks at QA as "just testers" and, of course, this should not be the case. But since they oftentimes do that this means that they will not look at these "just testers" for process improvement. If, however, they are made to realize that quality assurance is responsible for the process and the control (testing) group is responsible for the product, this can sometimes help. It can still be a hard sell, of course, but at least the distinction is made and everyone is using the same terms.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>I'm not shut out of a vote to what happens even with development processes. My only concern is that I need people that will represent QA that will carry enough expertise to make this useful.<HR></BLOCKQUOTE>
Agreed. As I said earlier, consider knowledge engineers, process improvement individuals, project mangagers with a quality background, etc. I have had very good luck with people like that in this arena. Basically you can get high-level strategic people together with your low-level tactical people and you can make some good things happen.
<BLOCKQUOTE><font size="1" face="Verdana, Arial, Helvetica">quote:</font><HR>What I may need to do is provide upper management with # like Jeff said. It has been kind of sudden that I've been feeling the pressure to hire. But yes, I agree with you both that I need to provide them with a documented graphical view of the outputs based on the hires.<HR></BLOCKQUOTE>
Agreed. Show how the schedules, training time, return on investment, person-hours, test-hours, etc. all interrelate and how they, in the long term, affect the bottom line of the business as regards distributing a quality product. I can write all the quality audits and quality initiative documents in the world but if I do not back that up with some numbers that make the theory practical I am literally whistling in a very large windstorm.
Re: Trouble with hiring SQA folk...
Elfriede, regarding your post, I basically agree with what is said there. It is easy sometimes for candidates to snowball someone because of apparent knowledge. It is very easy to look knowledgable when you are dealing with someone else who is not. There are many times I went into interviews that my colleagues had said, "You are just going to love this person! They just know everything!" Well, I step in there and ask a few pointed, direct questions at a detail level and all of the sudden the "everything" that they knew becomes slightly more restricted.
Responding to bman on this, I too like to ask general questions and, many times, I am not so much concerned with the exact answer but with the process they use to formulate the answer so that I can get a handle on how they approach problems and how much cognitive friction they display. For example, I look to hear from a candidate (in some way, shape, or form) that testers cannot test quality into a product. I like to hear them state that they realize they cannot test everything. I like when they realize there must be a balance between complete domain ignorance and complete domain expertise.
I also look for basic responses regarding "psychological" questions, such as the "how would you deal with such-and-such" variety. I look to see if this person appears to be dogmatic in their assertions, rather than pragmatic. I look to see if they realize that testers are there to pursue defects and issues in the product, not the people who made the defects and issues.
Finally, I also look for broad knowledge. This more applies to senior level individuals but I like to see how current they are on topics relating to the field. For example, if I am interviewing a performance testing candidate I ask them some recent examples of what they imagine were major performance issues at various places. (Good answers would be AOL's site for the "Big Brother" show or Ion Storm's website when the games Daikatana or Deus Ex were released. Another would be George Bush's tax allocation and display pages.) I also like when other areas of quality have been studied by the person (even if never applied), such as usability, accessibility, etc. This shows that there is a general curiosity and interest in the subject.
Re: Trouble with hiring SQA folk...
For those of you trying to hire Winrunner experts, here is a quiz (posted to the Winrunner user group) which I think could be useful to weed out the "pretend-to-be's"
By Jena Buttimer:
"Have a PC ready with WR and the flight1a app available.
Give the candidate a FLOPPY disk labeled with their name, date, and WR exam.
Instruct them to save their work to the disk.
Instruct them that they can use any online help, manuals, or any other
written help to do the test.
Verify that using a successful login, three new orders can be added to the
Report findings to the output report.
1. Use "Flight 1A" application to record your test.
2. Create 3 scripts as follows:
Flight1ADriver - Create a batch script to run the other scripts.
Login - Create a script to log into the application
AddNew - Add a new order
3. Create a function library containing AT LEAST one function -
Write an external function that will get the sytem date, convert it to
today + 1 week, and use that date as the depart date in the script.
3. Make the test data driven.
4. Save all of your work on the floppy. I should be able to take your
floppy and run your scripts from ANY pc with WinRunner installed.
**NOTE: Feel free to document your work, use "best practice" techniques, and
include any error checking, or anything else you feel will enhance the
YOU HAVE ONE HOUR TO COMPLETE THE TEST!
Now, for those of you giving the test, here are the things I'm looking for:
First of all, this looks deceptively simple and rather vague. It's that way
for a reason. I want the candidate to create a script the way they would at
work. I want them to document (or not) the script, use best practices, etc.
In other words, I want to see their own style of scripting.
Next, I have them save it to a floppy for a reason. 8 out of 10 WILL NOT
SAVE THE GUI MAP WITH THEIR SCRIPTS!!! Instant failure!! This should be so
ingrained into an experienced wr user, they should do this without even
thinking about it!
I ask for a driver script just to see if they understand the concept of
calling one script from another and I also look to see if they use treturn to
return a return code. This in turn should cause them to use tl_step
statements. A script without tl_step's is a script that has no validation.
Just because a script passes, does not mean you have verified anything!
I ask them to write a simple date function, to show they understand the
I ask them to make it data driven to make sure they understand the concept.
Also to see if they will save the data file with the script!!
It took me about 15 minutes to complete the test. I think an hour is long
enough, but you decide. It's a simple test that will show alot about what
they know and what they don't know. I try to emphasize that this is not to
show who is the best scripter, just to show the level of their knowledge and
what areas they are weak. It pretty much shows you if they know absolutly
nothing, they can record and playback, they have good documentation skills,
and understand some advanced concepts. I would seriously consider anyone who
can pass the test for employment.
Let me know your ideas on how to make this better!
Re: Trouble with hiring SQA folk...
That is a pretty good quiz but a lot of time during interviews (at least the ones I have done) you may not have the time to do something like that. What I like to do, personally, is just ask pointed questions. For example, "What was the hardest thing you ever had to script in WinRunner TSL?" Or you can say "What commands do you dislike using the most in WinRunner's language?" You can also even make up a situation that you are trying to solve with the tool. For example, I sometimes ask: "You know, I am trying to get WinRunner go go through each Web page and generate all the elements that are on the page and save them to the GUI Map. How would you go about that?" If they are billing themselves as a WinRunner expert of any sort, they should be able to answer this - at least to some degree.
Unless the interviewer is really not on the ball, it is easy to see if someone stumbles through answers to these questions or answers so noncommitally and obtusely that they are obviously not as up to speed as they may be indicating. Granted, they could try to lie about it but you can ask them specific details of the language. The danger, of course, is that they may not have used certain aspects of the language and yet not be lying about other things.
For me, sitting someone down at the tool and having them spend time working with it on scenarios would only be resorted to if we needed someone to come in and do some heavy-duty WinRunner implementation right away.
Unless this person is being brought in as a WinRunner (or other tool) expert, it is always possible to train someone on a tool and that is why I tend to look more for the testing attitude rather than extreme tool smarts. I can train anyone on a tool and any reasonably intelligent person can figure one out. But what is harder to train is a testing attitude and an eye for what to look for. I have met many people who can code very well in TSL or 4Test but what they coded was not actually testing the parts that needed to be tested.