ESM Versus Logs: Filling in the Gaps
Venue
American Anthropological Association 2014 Annual Conference
Publication Year
2014
Authors
BibTeX
Abstract
In the last few years we’ve all heard amazing stories of how “big data” can make
uncanny predictions about users (e.g., Target knowing when someone is pregnant
based on their purchases). However, there are just as many stories of when
analytics gets it wrong (e.g., Google Flu Trends overestimating flu cases in 2013).
It shouldn’t be at all surprising when predictions based strictly on click analysis
(sometimes with demographic data thrown in) gets it wrong because analytics tells
us only the WHAT, not the WHY. In the case of Google Flu Trends, the massive media
coverage influenced people to search on the keywords that previously indicated one
was coming down with the flu but now simply meant people were curious about it. Our
logs were missing the WHY behind the keywords. At Google, we conduct an annual
Experience Sampling Methodology (ESM) study with a large sample of our users
recording their needs, context, and experience to capture the WHY we cannot see in
our query stream. Over a five day period, participants tell us about their
experiences throughout their day and submit photos to explain things words alone
cannot capture. Doing this over a several month period every year allows us to
monitor changes in user’s subjective and objective behavior for a clearer picture
than analytics alone.
