Posted on August 8, 2009 at 1:28pm EDT. More.

App Store Reviewer Incentives

I think Daniel Jalkut has it backwards:

Many of the mercenary testers I encountered were motivated to scrape the system for bugs, as ridiculous as they may be. They logged them into the bug system and then defended them at all costs, as if their lives depended on it. And it turned out, they did. At least, their paychecks did.

I would not be surprised to learn that App Store reviewers are working under a similar structure. A system that rewards “unique, valid rejections” would certainly explain the behavior we have seen coming to light in the past year.

QA teams, like the ones Daniel writes about, reward “unique, valid bug reports” because all the testers are testing the same piece of software. If a group of testers repeatedly file duplicate bug reports, clearly you have an inefficient team, and can afford to let some of them go. The team has an incentive to reward unique reports.

The App Store review team, on the other hand, has one reviewer per app, and each app is different. Even if the team was perversely trying to justify its existence by beefing up its rejection numbers, there would be no need to come up with “unique, valid rejections.” In that case it would be more efficient for them to reject apps with increasingly vague reasons, or no reason at all.

It is more likely that reviewers are severely punished (i.e., fired) for approving an app that later causes a problem. That’s the kind of situation that leads to overzealous reviewers without assuming that management is dumb.

And there are problems: all the writing on the Ninjawords rejection pits Apple against enlightened liberal people. Not pictured are the parents who have purchased an iPod touch for their young child and want to let them download games from the App Store, but maybe don’t want to let them downloading Motorboat, which is not actually about boats, but involves rubbing your nose on the touchscreen.

When Apple added ratings to the App Store, it did not do so in a vacuum. It has quite a few customers with children, who I suspect express themselves by making phone calls instead of posting on blogs. So while we’ve read a lot about the helpless developers who are being persecuted by puritannical Apple, we haven’t read the testimony of parents who are trying to give their child a little bit of freedom without having to worry that they will get the idea that sticking your nose in a woman’s cleavage is an acceptable thing to talk about during Thanksgiving dinner.