Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 9988
Loc: Massachusetts, USA
|
|
In
http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future-of-software-testing-part-1.aspx
James Whittaker (currently of Microsoft, formerly of Florida Institute of Technology) points to uTest and argues that this is the beginning of a new era - Crowdsourcing:
"The next logical step in the evolution of testing is for vendors to provide testers and this is exactly the era we’ve entered with crowdsourcing. Yesterday’s announcement by Utest marks the beginning of this era and it is going to be very interesting to see it unfold."
I'm wondering what others think about uTest, and about the concept of Crowdsourcing.
I signed up when uTest first started in order to keep an eye on what they were attempting. Now that they are "open for business", I'd be interested to hear what others think, before I state my feelings.
-------------------- - Joe
I speak only for me. I do not speak for my company, or for anyone else.
Visit my NEW blog All Things Quality now at http://strazzere.blogspot.com
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
I do not approve of a 'pay-per-bug' system, especially if you want testers to follow a test plan/test scripts that you provide. Seems like it would be pretty easy to 'play the system' and get this service as a supplimental supply of cheap ($2000 is pretty cheap) testers to plough through well-trodden paths, freeing up your internal testers to go exploratory. Additionally, as a tester it could be soul-destroying to be on the other end of this - running scripts 'for free'.
I have deep seated reservations about the security risk in providing pre-release code to many and varied geographical locations, perhaps this is most useful for web testing.
From a management perpective - their demo video did not demonstrate to me how I could have any visibility on the actual coverage achieved.
|
philk10
Advanced Member
Reged: 01/20/05
Posts: 578
Loc: UK
|
|
One problem I have with this idea is that it leaves testing too late and is inefficient, what happened to the idea of testers finding bugs in the specs etc ?
will the U-test testers be doing load testing, security testing, usability testing ? how about testing the maintainability of the code ? if you found a security flaw might it be tempting to keep it quiet and hope it gets released and exploit it ?
-------------------- and My Blog
|
brentpaine
Veteran
Reged: 03/09/07
Posts: 3749
Loc: Waterloo, Ontario, Canada
|
|
I just tried to sign up and check it out, but there were some broken check boxes in the signup process which ask for the type of testing you've done. It just keeps refreshing and telling me I must select at least one. I JUST DID! I SELECTED EVERY ONE!!!
Plus, they only let you select either desktop or web experience, not both (cause they have that in a drop down).
Let the hounds go on your website first boys.
-------------------- Brent
--------------------
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
--------------------
What is Holistic Testing?
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
...also on the 'pay per bug' scheme...
i: Do they state anywhere what they pay?
ii: Who would want to be a tester at the tail end of the SDLC? Get me on the pre-alpha, boys, thare be gold!
|
jamboarmy1
Member
Reged: 10/19/07
Posts: 47
|
|
I don't see how this will work. What's to stop a developer rejecting every defect found (and hence not paying the tester)?. The developer could then fix the defects for minimal cost, and the testers lose out.
There's also the fact that any professional tester worth his salt is already employed full-time. Will these testers really be professionals, or just cowboys pretending they are qualified?
This strategy is presumably focused on using cheap testers, in developing counties like India for example, western testers couldn't compete in terms of cost. However, there are already companies who specialise in offshore testing, presumably at a cost benefit to onshore testing. These companies at least have the ability to claim that their employees are testing professionals, and have a verifiable track record in the field.
Edited by jamboarmy1 (08/21/08 11:16 AM)
|
jimhazen
Super Member
Reged: 01/31/02
Posts: 1791
Loc: Highlands Ranch, CO, USA
|
|
Time to redo my Scuba Instructor certification and move the family to Bonaire. At least I'll enjoy myself while getting crappy pay and long hours.
I feel sorry for the testers and clients of UTest. You know some CEO / CFO will eat this up. Another way to get testing for free and commoditize & cheapen our work.
Great... just fraking great!
-------------------- Jim
-------------------------------------------
For all the general stuff to know about QA/Test go here http://www.softwareqatest.com/
Edited by jimhazen (08/21/08 02:02 PM)
|
MarkCrowther
Member
Reged: 06/06/08
Posts: 60
Loc: Hampshire, UK
|
|
What if the budget allocation means the Test Manager can't accept valid bugs? Do they just ignore them, reject them or do they just cherry pick and reject the rest.. hmm, I just said that.
Signed up but can't get any details of pay grades, enter my profile data, etc. so we'll have to see how this goes.
It's interesting, but I think it's the same as other sites such as www.elance.com or www.guru.com just more focused on testers of course. It wont destroy the market for in-house or outsourced testing teams.
I better get some high value projects so my bugs aren't rejected and make sure I start early to find all the cool bugs or I'll not be eating this month ;p
Mark
-------------------- www.cyreath.co.uk | www.nmqa.com
|
Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 9988
Loc: Massachusetts, USA
|
|
My sense is that eTest will fill a "Beta Test" role.
As far as I can tell, this is a Web 2.0-ified way of getting minimally-paid Beta testing done. - take what would be a public Beta and make it semi-private - instead of offering t-shirts, iPods or the like, pay a few bucks per bug - let uTest log the bug reports for you
I suspect uTest's competition will come from some company that decides to go an even cheaper route. They will make a site more like a game show, and pay out "prizes" to the testers, rather than real money. The site will be advertiser-funded, so the companies who sign up for these sort-of-Beta-testers could do it for free, or a very minimal fee.
*** Note to AJ: This is what I was suggesting as a future direction for SQAForums... ***
I don't imagine this taking the place of any professional testing (except for shops so small they couldn't afford real testers anyway).
philk: "will the U-test testers be doing load testing, security testing, usability testing ? how about testing the maintainability of the code ?"
The uTest web site indicates that testers will be doing GUI, Functional, and Technical testing. The first two are fairly standard, and they define Technical as "Error produced or malfunction of the application. examples: error when presenting a page, JavaScript error , application crashes etc".
nm_stuartb: "Do they state anywhere what they pay?"
The pay depends on the type of bug found. Here are the per-accepted-bug pay rates posted for the few Projects that are displayed today. GUI: $2.00 - $2.67 Technical: $10.00 - $13.31 Functional: $5.00 - $6.66 Valuable Feedback: $2.50 - $3.34 Exceptional Feedback: $5.00 - $6.66
Here's one example I see. One company has been allowing 24 testers to test since late June. In that time, they have accepted 29 bugs, for a total payout (assuming the Max rate for each type) of $167.82. By my calculations that's an average of less than $7.00 per tester for the length of the project so far. (Me, I'd prefer a t-shirt.)
jamboarmy1: "Will these testers really be professionals, or just cowboys pretending they are qualified?"
When I signed up, I was never interviewed, or questioned in any way. As far as I know, there's not much real need to determine if the members are professionals or pretending, since they are only on board to report bugs, and are only paid if they do so.
MarkCrowther: "I better get some high value projects so my bugs aren't rejected and make sure I start early to find all the cool bugs or I'll not be eating this month."
Based on what I see so far, there won't be any "high value projects" here. And starting early won't matter much. If your eating depends on your uTest earnings, you'll have to learn to enjoy about one meal per month.
-------------------- - Joe
I speak only for me. I do not speak for my company, or for anyone else.
Visit my NEW blog All Things Quality now at http://strazzere.blogspot.com
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
Oh deary me; I was going to sign up out of curiosity but, if that's the pay scale, I'd rather walk the streets picking up loose change.
|
Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 9988
Loc: Massachusetts, USA
|
|
Quote:
Oh deary me; I was going to sign up out of curiosity but, if that's the pay scale, I'd rather walk the streets picking up loose change.
If you are looking to feed your curiosity, then the pay isn't as significant.
If you are looking to feed your family, then I suspect loose change will give you a better chance of success.
-------------------- - Joe
I speak only for me. I do not speak for my company, or for anyone else.
Visit my NEW blog All Things Quality now at http://strazzere.blogspot.com
|
drazle
Active Member
Reged: 11/30/00
Posts: 721
Loc: Michigan, USA
|
|
Not to mention that the pay goes to a "debit card" that charges $3 per month unless you maintain a balance - and it is the only way to get paid.
|
Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 9988
Loc: Massachusetts, USA
|
|
I think you can avoid the debit card by using Paypal?
-------------------- - Joe
I speak only for me. I do not speak for my company, or for anyone else.
Visit my NEW blog All Things Quality now at http://strazzere.blogspot.com
|
Peter Ruscoe
Veteran
Reged: 03/18/02
Posts: 6860
Loc: Tampa Bay
|
|
And then, as the recipient, you pay fees to PayPal!
|
JakeBrake
Moderator
Reged: 12/19/00
Posts: 15264
Loc: St. Louis - Year 2025
|
|
... in short then, with the payout one should plan on budgeting for a package of Ramen Noodles per month - when they are on sale @ 6 for a US Dollar?
|
Joe Strazzere
Moderator
Reged: 05/15/00
Posts: 9988
Loc: Massachusetts, USA
|
|
Quote:
... in short then, with the payout one should plan on budgeting for a package of Ramen Noodles per month - when they are on sale @ 6 for a US Dollar?
Probably depends on where you live. In some parts of the world, they may not have access to Ramen!
-------------------- - Joe
I speak only for me. I do not speak for my company, or for anyone else.
Visit my NEW blog All Things Quality now at http://strazzere.blogspot.com
|
JakeBrake
Moderator
Reged: 12/19/00
Posts: 15264
Loc: St. Louis - Year 2025
|
|
Quote:
...Probably depends on where you live. In some parts of the world, they may not have access to Ramen!
If they are doing crowdsourcing via the www, then they might be okay: 
http://www.ramendepot.com/r/ramendepot/default.asp?S=AC3&Document=FAQ
http://www.shopfoodex.com/maruchan-ramen-noodles-chicken-flavor-p-28550.html
http://www.koamart.com/shop/1-Ramen_Bags.asp
|
Roy Solomon
Junior Member
Reged: 11/28/04
Posts: 10
Loc: USA
|
|
Hi All,
The uTest "community testing" method is running in the last few months under a limited availability mode. During this time we opened our doors to only selected testing projects in order to make sure this new and exciting concept is working smoothly. We will have very soon much higher working volume since we opened our doors last week to every software company.
The feedback we got from all the companies that run a testing cycle is that a community of professional testers can provide enormous value. We thrilled to see that even James W believes like we are that this is the future of software testing.
Even more important is the feedback we are getting from the Tester Community. We have quite a lot of professional testers that perform various testing assignments, Work at their leisure, view latest applications and earn income.
We continue to work hard and improve our services and testing platform to make sure both the tester community and our customers will benefit from it.
Thank you for your feedback and suggestions.
Happy Testing,
Roy Solomon The uTest Team
Edited by Roy Solomon (08/24/08 07:12 AM)
|
stoofer
Member
Reged: 08/11/08
Posts: 250
|
|
Hi Roy,
Good to see your presence on the forums. Are there any plans to address the inherent issue in wanting a global testing community versus disparate costs of living? The $ is worth vastly different amounts across the globe - for example the rates I can see from having signed up, it would be difficult - if not impossible - to scratch out the UK minimum wage (currently standing at £5.52 GBP or $10.14 USD) after tax. This issue severely impacts your ability to attract professional testers from this region of the world.
Also, as someone who will be looking for some outsourcing support, how do you ensure security of the software under testing? As far as I can tell, my identity as a tester has not been validated - yet I can browse open projects and see what potential competitors are up to... If I approached you with a windows application to test - how could/would you ensure it was not available on BitTorrent the following day?
|
brentpaine
Veteran
Reged: 03/09/07
Posts: 3749
Loc: Waterloo, Ontario, Canada
|
|
Hey Roy,
Glad to see you at the forums. I have a question as well.
You're business model appears to be a good one. Unfortunately it would appear that your execution is lacking. I attempted to sign up and got the message as outlined in my message from the 21st.
I sent an email to your support team, but never got a response. Is this the service your team is dedicated to providing to the testing community?
I think it's actually unfortunate for you, because there are a couple of factors that will ultimately lead to the demise of a website. 1) Someone does it better than you; 2) You don't communicate with your public; 3) You gain a bad reputation.
So in the first case, because you have a strong business model, it goes without saying that we'll probably see more of these types of sites pop up around the web (well there kinda already is with eLance, etc., but we'll say there isn't). Testers are fairly resilient to errors, but we're also probably the most harsh when it comes to technical screwups. So we might give you the benefit of the doubt in most cases, but when you leave something go long enough, after we've told you about it, then we'll probably just leave you in our dust and never look back. Are you concerned about competition at this point? Do you have a plan to stifle competition enough to continuously increase your partner-base?
Secondly, Communication. Again, testers will probably let you off with more than the regular Joe, so you're lucky there. However, if someone can't even get an email response within a week, then you're doomed to fail. Also, if I can't get an email response then what do you think my expectations are for actually getting paid for the work that I do? Probably not very high. So then I must ask myself, why would I work for free? I have no reason to work for free. So why would I sign up? Continue? I probably wouldn't.
Finally, well, I mean it's obviously already happening. Unfortunately I don't see a lot about how you've managed to succeed or plan to succeed in the future. Based on the comments I'm seeing, there would be no professional who would sign up (even if they could) at the rate you're looking at.
Here's my wild stab at things. You're probably a younger entrepreneur, set up this as a school project, but it seemed like a good busienss model, so you shopped it around to venture capitalists in your area. You managed to find someone who saw the promise of the site and invested, maybe, a couple million bucks. You're greedy, though, (or a miser) and don't use the money properly, to grow your product. You try to stash as much of the money away as possible and do as much of the work on the site as you can, by yourself, because you want to stretch every dollar.
There was probably an agreement to allow the VC to work with you as an advisor (because that's what always happens when they give you tons of cash). However, he doesn't know squat about the software side of things, he's more intereted in the business end.
He helps you with everything from getting partners to promotions. He (and now your partners who have products waiting) rushes you to push out a product so he can get some sort of ROI. You give in like a virgin on prom night and release the product when it's, obviously, not ready.
Media picks up on the concept and runs with it. Unfortunately, there will be a couple of downfalls to the model. First, Partners won't give you untested code to really allow testers to make any REAL money. So testers won't test it and Partners won't see a real value to it.
Secondly, as a result of your frugal spending, you don't have the staff to manage your business. Turn-around time on critical things like emails and site-related issues will cause testers to leave because it's just not worth their time to wait anymore.
Finally, when partners realize that their software is getting tested by a bunch of monkeys who just happened to punch the correct keys on the signup form to fit their requirements, the partners will leave.
Anyway, in my opinon, I don't think this is at ALL the future of testing. While it seems like a great concept, in the end, I think it will end up like those "Get Paid To Read Emails" models where it just isn't worth your time to do it. Yes, at first it might seem like, "Hey, well at 10 cents an email, if I read 200 emails a day, that's like $20 a day!" even though that breaks down to like $2 an hour and they won't even send you more than 4 a day.
-------------------- Brent
--------------------
9 out of 10 people I prove wrong agree that I'm right. The other person is my wife.
--------------------
What is Holistic Testing?
|