Human Computation Must Be Reproducible
Abstract
Human computation is the technique of performing a computational process by
outsourcing some of the difficult-to-automate steps to humans. In the social and
behavioral sciences, when using humans as measuring instruments, reproducibility
guides the design and evaluation of experiments. We argue that human computation
has similar properties, and that the results of human computation must be
reproducible, in the least, in order to be informative. We might additionally
require the results of human computation to have high validity or high utility, but
the results must be reproducible in order to measure the validity or utility to a
degree better than chance. Additionally, a focus on reproducibility has
implications for design of task and instructions, as well as for the communication
of the results. It is humbling how often the initial understanding of the task and
guidelines turns out to lack reproducibility. We suggest ensuring, measuring and
communicating reproducibility of human computation tasks.
