50 Beta Testers. Not enough, or better than most?
Two software testers talked about the Joe Stump story this week. One was Elizabeth Hendrickson and the other UTest. The two attitudes to the same story are very different. Let me compare two of the views:
First Elizabeth Hendrickson "Problem #2: Thinking that “50 Beta Testers” and “200 Unit Tests” Constitutes Exhaustive Testing
Having beta testers and unit tests is a good and groovy thing. But it’s not sufficient, as this story shows. What appears to be missing is any kind of rigorous end-to-end testing."
"So even the most cursory Exploratory Testing by someone with testing skill would have been likely to reveal the problem."
"Not to lay blame at Stemp’s feet. We, of all people, know that bugs happen. Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many. But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version."
Whilst both blogs encourage more testing, I think I prefer Elizabeth's attitude to the whole thing. It's hard to know if 50 beta testers is sufficient or not without the context (last phrase included to avoid references to Unicorn Question in any comments..you know who you are...). Regardless, the attitude that if you 'do more testing than many' somehow vindicates the fact that showstopper bugs were found is I think a misguided one.
As Elizabeth rightly pointed out in her post, good exploratory testing would have picked up this bug and any good tester knows to test for these things.
I hark back to my post on other approaches to testing for Startups. There are better ways to test! Lets not fall into the pitfall of accepting a lower level of testing as sufficient. Just because Startups use Beta Testing as the preferred modus operandi , doesn't necessarily make it a good approach.