EYEORG: A Platform For Crowdsourcing Web Quality Of Experience Measurements
Abstract
Tremendous effort has gone into the ongoing battle to make
webpages load faster. This effort has culminated in new protocols
(QUIC, SPDY, and HTTP/2) as well as novel content
delivery mechanisms. In addition, companies like Google
and SpeedCurve investigated how to measure “page load
time” (PLT) in a way that captures human perception. This is
challenging since even estimating when a web page is loaded
has proven to be difficult in modern web pages. In this paper
we present Eyeorg [13], a platform for crowdsourcing web
quality of experience measurements. Eyeorg overcomes the
scaling and automation challenges of recruiting users and
collecting consistent user-perceived quality measurements.
We validate Eyeorg’s capabilities via a set of 100 trusted
participants. Next, we showcase its functionalities via three
measurement campaigns, each involving 1,000 paid participants,
to 1) study the quality of several PLT metrics, 2) compare
HTTP/1.1 and HTTP/2 performance, and 3) assess the
impact of online advertisements and ad blockers on user experience.
We find that commonly used, and even novel and
sophisticated PLT metrics fail to represent actual human perception
of PLT, that the performance gains from HTTP/2
are imperceivable in some circumstances, and that not all
ad blockers are created equal.