The online community for software testing & quality assurance professionals
 
 
Calendar   Today's Topics
Sponsors:




Lost Password?

Home
BetaSoft
Blogs
Jobs
Training
News
Links
Downloads



Miscellaneous Forums >> General Discussion

Pages: 1 | 2 | >> (show all)
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Recruitment - Testing the testers
      #667276 - 04/07/11 07:21 AM

Hi,

I'm doing some recruitment and I want testers who can find defects. I've been given some IKM test scores by our internal recruitment team that covers:

Test reporting and metrics
Test life cycle
Automation techniques
UAT
Env. management
Test case design
Detecting s/w errors
Issue tracking
Web testing
Test planning#
Test tool types
Reviews and inspection

But I don't really mind about the theory, I just want bug-hunters. And because it's a wide test there are only three questions that actually involve testing (I know I just took it). So, I can knock up some small web-app and get candidates to test it... Or is there someone that provides such an on-line testing? Something where people have an hour to find bugs and report the defects? I don't care if they know their equivalence partition theory from their elbow, I care about how good and practical they are at getting to the bug

Thoughts please people!


Post Extras: Print Post   Remind Me!   Notify Moderator  
michaeljf
Veteran


Reged: 09/17/01
Posts: 3979
Loc: Yankee Land
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667304 - 04/07/11 08:35 AM

Never done this formally so I wouldn't know how to begin, if you are really interested in how they do things I like your practical idea. Know how many defects there are, see who can find what and describe them.

I don't really trust outside tests to tell me more than a score, I find out more about the candidate by challenging them and seeing how they answer and respond.

--------------------
- M

Nothing learns better than experience.

"So as I struggle with this issue I am confronted with the reality that noting is perfect."
- Unknown

Now wasting blog space at QAForums Blogs - The Lookout


Post Extras: Print Post   Remind Me!   Notify Moderator  
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Re: Recruitment - Testing the testers [Re: michaeljf]
      #667309 - 04/07/11 08:44 AM

Quote:

I don't really trust outside tests to tell me more than a score, I find out more about the candidate by challenging them and seeing how they answer and respond.




I agree! However group recruitement always put testers through this IKM test which is 5% practical. And if I'd not seen those test results I'd be happy with the F2F answer and respond I've done So having decided the best way is just get them to do some testing, I assumed it's not unique and someone would have would have something on the market...


Post Extras: Print Post   Remind Me!   Notify Moderator  
Joe Strazzere
Moderator


Reged: 05/15/00
Posts: 12344
Loc: Massachusetts, USA
Re: Recruitment - Testing the testers [Re: michaeljf]
      #667312 - 04/07/11 08:48 AM

Are these testers going to be expected to just jump in and test an application? Or are they going to be expected to read Requirements and/or Specs, plan their testing, then execute their tests?

If more the latter, then an approach I've used is to bring a small extract from the Requirements to the interview, and have the candidate read it and ask a few clarifying questions. Then I expect the candidate to walk me through their thought process about what they might want to test in that area, why, and how.

Seems reasonably effective for the contractors I have hired recently.

--------------------
- Joe
Visit AllThingsQuality.com to learn more about quality, testing, and QA!

I speak only for me. I do not speak for my employer, nor for anyone else.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Recruitment - Testing the testers [Re: Joe Strazzere]
      #667326 - 04/07/11 09:59 AM

But he's really looking for an online "test," Joe. I'm not sure "... I expect the candidate to walk me through their thought process..." fits that - uh - requirement

Post Extras: Print Post   Remind Me!   Notify Moderator  
Joe Strazzere
Moderator


Reged: 05/15/00
Posts: 12344
Loc: Massachusetts, USA
Re: Recruitment - Testing the testers [Re: Peter Ruscoe]
      #667335 - 04/07/11 11:51 AM

Yup - I was just relating the approach I use. Sorry if that wasn't clear. Others' mileage may vary.

I was going to ask SimonFromLeeds how much he would be willing to pay me to do this for him. For the right price, I'd "certify" candidates. I could even do it online.



--------------------
- Joe
Visit AllThingsQuality.com to learn more about quality, testing, and QA!

I speak only for me. I do not speak for my employer, nor for anyone else.


Post Extras: Print Post   Remind Me!   Notify Moderator  
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Re: Recruitment - Testing the testers [Re: Joe Strazzere]
      #667381 - 04/08/11 12:58 AM

Hey Joe, thanks but I don't want to add another certification onto another!

For me, the area of certification is still a bit wild-west for testing. The introduction of the ISEB/ISTQB has certainly standardised the arena, but I know when I've taken ISTBQ based tests (like the IKM) I've scored well and I've never done a days testing in my life*. It's created a nice industry for itself.

The point is, surely there's a market need to be able to check how good testers are by getting them testing. In the same way checking ingredient quality, oven capabilities and chefs cooking knowledge would guide me to how good a cake someone could produce. I'd rather have a slice of cake and judge how good it was. (seeing their processes would be a bonus).

Anyway I'll stop posting/ranting and review my interview notes.

* not actually true, but not far off.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Joe Strazzere
Moderator


Reged: 05/15/00
Posts: 12344
Loc: Massachusetts, USA
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667406 - 04/08/11 04:10 AM

Quote:

The point is, surely there's a market need to be able to check how good testers are by getting them testing.



I'm sure there is a market need. And for the right price, you can find a company willing to fill that need. So, I'd like to explore the concept further through a thought experiment.

How much would you be willing to pay me per candidate to have them test my application and then have me tell you how many bugs they found in one hour?

Since you don't care about theory, and you only want to find bug-hunters, would the number of bugs found in one hour be all you need to receive from me? Or, is there more information necessary?

--------------------
- Joe
Visit AllThingsQuality.com to learn more about quality, testing, and QA!

I speak only for me. I do not speak for my employer, nor for anyone else.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Joe Strazzere
Moderator


Reged: 05/15/00
Posts: 12344
Loc: Massachusetts, USA
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667407 - 04/08/11 04:24 AM

Quote:

Hey Joe, thanks but I don't want to add another certification onto another!



I was trying to be humorous by using the word "certify" - obviously I missed the mark.

What I meant was to probe around the thought that you want someone else, or at least some website, to check out potential candidates for you, and report back how many bugs they were able to find.

I'm sure this is doable - for a price.

I know for a fact that there are companies in my part of the world would be willing to administer any competency test you would choose. I don't know for a fact that they do this online, but I have to think they could be convinced to do so for a fee.

I'm just wondering how serious you are, or if you were just looking for a free/extremely cheap way to accomplish this, without any of the safeguards that a serious effort would provide?

--------------------
- Joe
Visit AllThingsQuality.com to learn more about quality, testing, and QA!

I speak only for me. I do not speak for my employer, nor for anyone else.


Post Extras: Print Post   Remind Me!   Notify Moderator  
michaeljf
Veteran


Reged: 09/17/01
Posts: 3979
Loc: Yankee Land
Re: Recruitment - Testing the testers [Re: Joe Strazzere]
      #667421 - 04/08/11 05:13 AM

Quote:

For me, the area of certification is still a bit wild-west for testing. The introduction of the ISEB/ISTQB has certainly standardised the arena, but I know when I've taken ISTBQ based tests (like the IKM)



Somehow I can't quite equate these too. You are saying that a certification is too "wild west" yet you are comfortable with a test and score. To me, its the same thing. Maybe not to you, but I get the feeling that you are more a convert to IKM and feel that is your solution, but certifications are not. Yet both have some sort of testing of the tester and study involved, or at least some form of questioning.

I also don't really see what it is you are trying to get from this, is your comfort really that you find a layer between you and the candidate to do a pre-screen is better for your company than doing it yourself? While the allegory of baking is one thing I don't equate it with testing, there are far more variables in testing. Baking does have its own set of variables, not every baker will make the same cake with the same ingredients, just as not every tester will approach the same problem under the same conditions. To me it seems more like your interviewing process is somehow broken and you are using this IKM test to resolve that problem.

I may be wrong but that's my view on what I have been reading.

--------------------
- M

Nothing learns better than experience.

"So as I struggle with this issue I am confronted with the reality that noting is perfect."
- Unknown

Now wasting blog space at QAForums Blogs - The Lookout


Post Extras: Print Post   Remind Me!   Notify Moderator  
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Re: Recruitment - Testing the testers [Re: michaeljf]
      #667467 - 04/08/11 09:32 AM

Quote:

michaeljf: You are saying that a certification is too "wild west" yet you are comfortable with a test and score.




I didn't mean to give that impression!

My opinion is that ISTQB is still too immature ("wild west") to really add value in the testing yet. It's an appearance of useful standards, rather than useful standards. I was trying to use the random nature of IKM score results to rationalise/evidence my opinion.

I am uncomfortable with a test and score systems I've experienced (most recently IKM)

The allegory of baking I could simplify: Why test inputs only , when you could test outputs? And then to complicate again: I wouldn't only code-inspect 1% of an app and then present the results as likely indicator of total quality. But that's what I think IKM style tests do.

Quote:

michaeljf: To me it seems more like your interviewing process is somehow broken...




Maybes, but I hope not!

Quote:

michaeljf: ..and you are using this IKM test to resolve that problem.




Definitely not. I'm comfortable with all the F2F interview steps I've taken. I'm uncomfortable with the IKM testing and was wondering if there was something testing outputs that I could use instead, or whether I should just ignore them. The chat on this thread makes me think there's nothing out there, so I'll ignore the IKM.

And Joe, to answer your points: Soz for missing the humour. Am I serious, yeah, but I was looking for an existing service, not something bespoke. I was expecting to see a menu and determine if that's worth it to me, in this situation, not set a price. And I didn't just want to know how many bugs they can find, I want to see the defects they log.

Maybe when I get sacked for t'internet chatting rather than doing stuff, I'll create a site that matches my vision. And then set a price .

Cheers for the thoughts


Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667490 - 04/08/11 02:45 PM

Quote:

I was expecting to see a menu



Uh - well, that's not what this forum is about, Simon. We don't have a bunch of people with a list of web sites (or, you could say tools, or whatever), anxious to plop it out whenever someone asks. This is essentially a discussion forum.

And that's what you got.


Post Extras: Print Post   Remind Me!   Notify Moderator  
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Re: Recruitment - Testing the testers [Re: Peter Ruscoe]
      #667494 - 04/08/11 04:40 PM

Quote:

Peter Ruscoe: Uh - well, that's not what this forum is about, Simon. We don't have a bunch of people with a list of web sites (or, you could say tools, or whatever), anxious to plop it out whenever someone asks. This is essentially a discussion forum.

And that's what you got.


Fair enough. What's your opinions on multiple-choice/theory tests and hands-on s/w testing to assess potential recruits?

Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667497 - 04/08/11 06:50 PM

IMO every approach has value. But every recruiter has goals. Is a test (written or on-line) valuable - it depends! How critical is the position? Is is entry-level or experienced? Does the position require specialized knowledge? And so on and so forth.

When it comes to - specifically - online tests, what is to prevent a "ringer" (someone other than the applicant) completing the test? And if this is a significant risk (and I would say that it might be) what is the point of the online test in the first place?

I guess what I am getting to is this: If someone can submit a resume (which could be packed with lies), and you have an effective interview process, why waste time trying to find the "magic" online test, which is not even reliable?

Or, if it is administered in-house (to avoid impostors), why would you think that a computer-administered test (even backed up with pretty good AI) would be better than a human interviewer?

I guess I'm finding it hard to see why you are going down this road. It's been trodden many times before, and almost always ended up in a swamp.


Post Extras: Print Post   Remind Me!   Notify Moderator  
yagsy
Active Member


Reged: 11/26/01
Posts: 917
Loc: Greater Boston Area finally
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667539 - 04/10/11 01:49 PM

I've interviewed a few canddiates to fill positions and the approach I've taken is to give them practical questions, "what types of tests have you done", "give me an example of each", "what types of bugs did you find and what kind of approach did you take to find that bug", "was the bug found because it was steps relating to a test case previously written" and then I'll ask some domain specific questions if there is a need. These are just a few questions and I've been told by my managers that they like me to do the pre-screening because I can tell very quickly if the candidate is representing themselves with what is on their resume and what they have stated they can do. If the person can pass my interview with satisfactory results, then the manager will interview. And I learned so much from one person who has offered to help you, Joe Strazzere. He is very practical, efficient and knowledgeable.

All of answers to these questions and others tell me more about whether a candidate can perform the tasks to my satisfaction. Now, my questions vary based on the level experience needed, the type of testing needed and I absolutely will not focus on someone "who can find bugs". You focus on this, what you will get is someone who is very good at logging bug reports but are they really bugs? No matter how many times I would ask certain couple of my team members to stop reporting SYMPTOMS like "exception error" and no other explanation, they continued reporting symptoms. These people were taught filling out a high number of bug reports was more important than reporting legitimate bugs.

So perhaps, in the end, you might want people to know a little bit of theory so they can conceptualize the difference between a symptom of a bug and an actual bug. Years ago, I learned a very tough lesson, I reported a symptom of a bug and in the end wasted a developer's time of 2 days only to learn that the bug was not in the software the message was displayed but in the low level operating system. If I had tried some of the techniques I knew from my research and studies in class work, I would have reported a legitimate bug rather than the symptom.

--------------------
Going out of your comfort zone requires failure. True genius is measured by your recovery.

...Jean Ann
www.perfectpitchmarketinginc.com
http://on.fb.me/PPM100
www.projectrealms.com/

Edited by yagsy (04/10/11 01:53 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
SimonFromLeeds
Member


Reged: 08/12/04
Posts: 139
Loc: Leeds
Re: Recruitment - Testing the testers [Re: yagsy]
      #667584 - 04/11/11 03:10 AM

I didnt mean to come across as looking for a sliver bullet. In my criticism of IKM and ISTQB I was arguing against reducing the breadth and subtleties of testing to too rigid techniques and multiple choice questions. What I came looking for was something more practical, qualitative and trusted. And I was looking for something to compliment the F2F part of the selection process (not replace it), with something better than the IKM test. Apologies for not making myself clearer in my first post.

Quote:

yagsy you might want people to know a little bit of theory so they can conceptualize



Absolutely. What Im keen to avoid is people who can talk concepts but not apply them. I was hoping Id get that from seeing bugs theyd logged

Peter, Id not specifically considered how to avoid the ringer issue.

Quote:

Peter Why waste time trying to find the "magic" online test, which is not even reliable?



I think getting candidate to test s/w as part of the recruitment process is a good idea, and wondered if there was a shared resource. Im still glad I asked if such a service existed, even if it doesnt!


Post Extras: Print Post   Remind Me!   Notify Moderator  
michaeljf
Veteran


Reged: 09/17/01
Posts: 3979
Loc: Yankee Land
Re: Recruitment - Testing the testers [Re: SimonFromLeeds]
      #667605 - 04/11/11 04:57 AM

Quote:

I was looking for something to compliment the F2F part of the selection process (not replace it), with something better than the IKM test. Apologies for not making myself clearer in my first post.



Ok, I never got that from your earlier posts. Although I think alternatives have been presented here on how to do that, and I think you offered one as well, give the candidates an application to test and see how they do. If they find a defect have them explain it to you, or write it up and see how they communicate the issue to you. Nothing more practical than that.

Personally I would rather do this myself, so I can ask questions about how the candidate does things and see if its a fit for my company culture than have someone else who doesn't know my company manage the results. To each their own though.

--------------------
- M

Nothing learns better than experience.

"So as I struggle with this issue I am confronted with the reality that noting is perfect."
- Unknown

Now wasting blog space at QAForums Blogs - The Lookout


Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Recruitment - Testing the testers [Re: michaeljf]
      #667654 - 04/11/11 08:15 AM

Quote:

Ok, I never got that from your earlier posts.



<grin> That's probably because you wrote the idea off in your first response, Michael.
Quote:

I don't really trust outside tests to tell me more than a score, I find out more about the candidate by challenging them and seeing how they answer and respond.





He was, actually, asking for some sort of online test at the very outset, but this thread got off that and onto interview techniques almost right away.


Post Extras: Print Post   Remind Me!   Notify Moderator  
michaeljf
Veteran


Reged: 09/17/01
Posts: 3979
Loc: Yankee Land
Re: Recruitment - Testing the testers [Re: Peter Ruscoe]
      #667665 - 04/11/11 09:34 AM

Peter, I disagree. Even going back and reading over the initial post I still didn't see it. Grin as you need though.

Actually the point I thought he had which was valid was the test app, if you really want a practical to test people I think that's the best way. Your mileage may vary.

--------------------
- M

Nothing learns better than experience.

"So as I struggle with this issue I am confronted with the reality that noting is perfect."
- Unknown

Now wasting blog space at QAForums Blogs - The Lookout


Post Extras: Print Post   Remind Me!   Notify Moderator  
Peter Ruscoe
Veteran


Reged: 03/18/02
Posts: 7686
Loc: Tampa Bay
Re: Recruitment - Testing the testers [Re: michaeljf]
      #667711 - 04/11/11 01:59 PM

Yeah - OK. I guess his OP didn't indicate whether such a test (NOT an interview) would have to take place in his office or could be somewhere else.

This:
Quote:

So, I can knock up some small web-app and get candidates to test it... Or is there someone that provides such an on-line testing? Something where people have an hour to find bugs and report the defects?



is what I was going from.


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | >> (show all)



Extra information
0 registered and 30 anonymous users are browsing this forum.

Moderator:  Rich W., AJ, blueinatl 

Print Topic

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled

Rating:
Topic views: 17455

Rate this topic

Jump to

Contact Us | Privacy statement SQAForums

Powered by UBB.threads™ 6.5.5