Jump to Content

How the Order of Response Options in a Running Tally Can Affect Online Survey Estimates

Tom Wells
Charles DiSogra
Papers Presented at the 64th Annual Conference of the American Association for Public Opinion Research (AAPOR), AMSTAT (2011), pp. 5582-5585

Abstract

In the design of online surveys, running tallies or constant sums are used to help respondents sum up the allocation of amounts so that the total sums to 100%. We hypothesized that for time allocation, the order of the presentation of the time categories could make a difference in the distribution of reported time spent. We expected primacy effects, with the first-presented time category having a higher allocation of time than the later-presented options. An experiment was conducted with a general population adult sample from KnowledgePanel®. In the experiment, respondents were asked to provide running tallies of the percentage of television they typically watch during the morning, afternoon, and evening (separately for weekdays and weekends). The order of the categories was rotated. Primacy effects were detected, however differences by position were small and not statistically significant. Because time spent watching TV is a regular activity, viewing patterns are more likely to be encoded or ingrained in memory, and more likely to be reported reliably, with responses less susceptible to order effects.