A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples
Venue
Online Panel Research: A Data Quality Perspective, Wiley (2014), pp. 23-53
Publication Year
2014
Authors
Mario Callegaro, Ana Villar, David S. Yeager, Jon A. Krosnick
BibTeX
Abstract
his chapter provides an overview of studies comparing the quality of data collected
by online survey panels by looking at three criteria: (1) comparisons of point
estimates from online panels to high-quality, established population benchmarks;
(2) comparisons of the relationship among variables; and (3) the reproducibility of
results for online survey panels conducted on probability samples to panels
conducted on nonprobability samples. When looking at point estimates, all online
survey panels differed to some extent from the population benchmarks. However, the
largest comparison studies suggest that point estimates from online panels of
nonprobability samples have higher differences as compared to benchmarks than
online panels of probability samples. This finding is consistent across time and
across studies conducted in different countries. Moreover, post-stratification
weighting strategies helped little and in an inconsistent way to reduce such
differences for data coming from online panels of nonprobability samples, whereas
these strategies did bring estimates from online panels of probability samples
consistently closer to the benchmarks. When comparing relationships among
variables, it was found that researchers would reach different conclusions when
using online panels of nonprobability samples versus panels of probability samples.
When looking at reproducibility of results, the limited evidence found suggests
that there are no substantial differences in replication and effect size across
probability and nonprobability samples for question wording experiments and when
comparing students samples to other samples. It is worth noting that in
pre-election polls, an area where abundant prior knowledge exists, online panels of
nonprobability samples have consistently performed as well and in some cases better
than polls based on probability samples in predicting election winners.
