Performance tournaments with crowdsourced judges
Venue
Proceedings of the American Statistical Association, section on marketing statistics, American Statistical Association, 732 North Washtington Street, Alexandria, VA 22314-1943 (2013)
Publication Year
2013
Authors
Daryl Pregibon, Williiam D Heavlin
BibTeX
Abstract
A performance slam is a competition among a fixed set of performances whereby pairs
of performances are judged by audience participants. When performances are recorded
on electronic media, performance slams become amenable to audiences that watch
online and judge asynchronously (“crowdsourced”). In order to better entertain the
audience, we want to show the better performances (“exploitation”). In order to
identify the good videos, we want to glean a least some information about all
videos (“exploration”). Our approach has three elements: (1) We take our preference
model from Bradley and Terry (1952). (2) Its parameters we calculate by rewriting
the likelihood gradient into a fixed point estimate, one which mimics the estimate
of Mantel and Haenszel (1959). (3) Each pair of performances is chosen
sequentially, always chosen to minimize the weighted variance of (the logarithms
of) the Bradley-Terry parameter estimates. Our preferred weights consist of the
logrank weights proposed by Savage (1956).
