Towards better measurement of attention and satisfaction in mobile search
Venue
SIGIR '14 Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval (2014), pp. 113-122
Publication Year
2014
Authors
Dmitry Lagun, Dale Webster, Chih-Hung Hsieh, Vidhya Navalpakkam
BibTeX
Abstract
Web Search has seen two big changes recently: rapid growth in mobile search
traffic, and an increasing trend towards providing answer-like results for
relatively simple information needs (e.g., [weather today]). Such results display
the answer or relevant information on the search page itself without requiring a
user to click. While clicks on organic search results have been used extensively to
infer result relevance and search satisfaction, clicks on answer-like results are
often rare (or meaningless), making it challenging to evaluate answer quality.
Together, these call for better measurement and understanding of search
satisfaction on mobile devices. In this paper, we studied whether tracking the
browser viewport (visible portion of a web page) on mobile phones could enable
accurate measurement of user attention at scale, and provide good measurement of
search satisfaction in the absence of clicks. Focusing on answer-like results in
web search, we designed a lab study to systematically vary answer presence and
relevance (to the user's information need), obtained satisfaction ratings from
users, and simultaneously recorded eye gaze and viewport data as users performed
search tasks. Using this ground truth, we identified increased scrolling past
answer and increased time below answer as clear, measurable signals of user
dissatisfaction with answers. While the viewport may contain three to four results
at any given time, we found strong correlations between gaze duration and viewport
duration on a per result basis, and that the average user attention is focused on
the top half of the phone screen, suggesting that we may be able to scalably and
reliably identify which specific result the user is looking at, from viewport data
alone.
