Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Venue
WWW (2014) (to appear)
Publication Year
2014
Authors
Panos Ipeirotis, Evgeniy Gabrilovich
BibTeX
Abstract
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the
knowledge of users and acquires new knowledge from them. Quizz operates by asking
users to complete short quizzes on specific topics; as a user answers the quiz
questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz
also incorporates questions for which we do not have a known answer; the answers
given by competent users provide useful signals for selecting the correct answers
for these questions. Quizz actively tries to identify knowledgeable users on the
Internet by running advertising campaigns, effectively leveraging the targeting
capabilities of existing, publicly available, ad placement services. Quizz
quantifies the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback allows the ad
targeting mechanism to further optimize ad placement. Our experiments, which
involve over ten thousand users, confirm that we can crowdsource knowledge curation
for niche and specialized topics, as the advertising network can automatically
identify users with the desired expertise and interest in the given topic. We
present controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as goals, which
incentivize the users to contribute. Finally, our cost-quality analysis indicates
that the cost of our approach is below that of hiring workers through
paid-crowdsourcing platforms, while offering the additional advantage of giving
access to billions of potential users all over the planet, and being able to reach
users with specialized expertise that is not typically available through existing
labor marketplaces.
