Jon Vaver
Jon Vaver is a Senior Quantitative Analyst at Google. His work focuses on experimental and observational methods for measuring advertising effectiveness. He received a B.A. in Mathematics and Physics from Cornell College and a Ph.D. in Applied Mathematics from the University of Virginia. Prior to joining Google, he worked at the Department of Defense and on an operations research team at USWest/Qwest Communications.
Research Areas
Authored Publications
Google Publications
Other Publications
Sort By
Preview abstract
Many digital advertisers continue to rely on attribution models to estimate the effectiveness of their marketing spend, allocate budget, and guide bidding decisions for real time auctions. The work described in this paper builds on previous efforts to better understand the capabilities and limitations of attribution models using simulated path data with experiment-based ground truth. While previous efforts were based on a generic specification of user path characteristics (e.g., ad channels considered, observed events included, and the transition rates between observed events), here we generalize the process to include a pre-analysis optimization step that matches the characteristics of the simulated path data with a set of reference path data from a particular advertiser. An attribution model analysis conducted with path-matched data is more relevant and applicable to an advertiser than generic path data. We demonstrate this path-fitting process using data from Booking.com. The simulated matched paths are used to demonstrate a few key capabilities and limitations for several position-based attribution models.
View details
Preview abstract
Co-viewing refers to the situation in which multiple people share the experience of watching video content and ads in the same room and at the same time. In this paper, we use online surveys to measure the co-viewing rate for YouTube videos that are watched on a TV screen. These simple one-question surveys are designed to optimize response accuracy. Our analysis of survey results identifies variations in co-viewing rate with respect to important factors that include the demographic group (age/gender) of the primary viewer, time of day, and the genre of the video content. Additionally, we fit a model based on these covariates to predict the co-viewing rate for ad impressions that are not directly informed by a survey response. We present results from a case study and show how co-viewing changes across these covariates.
View details
Preview abstract
Many advertisers rely on attribution to make a variety of tactical and strategic marketing decisions, and there is no shortage of attribution models for advertisers to consider. In the end, most advertisers choose an attribution model based on their preconceived notions about how attribution credit should be allocated. A misguided selection can lead an advertiser to use erroneous information in making marketing decisions. In this paper, we address this issue by identifying a well-defined objective for attribution modeling and proposing a systematic approach for evaluating and comparing attribution model performance using simulation. Following this process also leads to a better understanding of the conditions under which attribution models are able to provide useful and reliable information for advertisers.
View details
Preview abstract
In this paper we tackle the marketing problem of assigning credit for a successful outcome to events that occur prior to the success, otherwise known as the attribution problem. In the world of digital advertising, attribution is widely used to formulate and evaluate marketing but often without a clear specification of the measurement objective and the decision-making needs. We formalize the problem of attribution under a causal framework, note its shortcomings, and suggest an attribution algorithm evaluated via simulation.
View details
Preview abstract
Advertisers often estimate the performance of their online advertising by either running randomized experiments, or applying models to observational data. While randomized experiments are the gold standard of measurement, their cost and complexity often lead advertisers to rely instead on observational methods, such as attribution models. A previous paper demonstrated the limitations of attribution models, as well as information issues that limit their performance. This paper introduces "near impressions", an additional source of observational data that can be used to estimate causal ad impact without experiments. We use both simulated and real experiments to demonstrate that near impressions greatly improve our ability to accurately measure the true value generated by ads.
View details
Preview abstract
Advertising is becoming more and more complex, and there is a strong demand for measurement tools that are capable of keeping up. In tandem with new measurement problems and solutions, new capabilities for evaluating measurement methodologies are needed. Given the complex marketing environment and the multitude of analytical methods that are available, simulation has become an essential tool for evaluating and comparing analysis options.
This paper describes the Aggregate Marketing System Simulator (AMASS), a sim- ulation tool capable of generating aggregate-level time series data related to marketing measurement (e.g., channel-level marketing spend, website visits, competitor spend, pricing, sales volume, etc.). It is flexible enough to model a wide variety of marketing situations that include different mixes of advertising spend, levels of ad effectiveness, types of ad targeting, sales seasonality, competitor activity, and much more. A key feature of AMASS is that it generates ground truth for marketing performance met- rics, including return on ad spend and marginal return on ad spend. The capabilities provided by AMASS create a foundation for evaluating and improving measurement methods, including media mix models (MMMs), campaign optimization (Scott, 2015), and geo experiments (Vaver and Koehler, 2011), across complex modeling scenarios.
View details
Preview abstract
Two previously published papers (Vaver and Koehler, 2011, 2012) describe
a model for analyzing geo experiments. This model was designed to measure
advertising effectiveness using the rigor of a randomized experiment with replication
across geographic units providing confidence interval estimates. While effective, this
geo-based regression (GBR) approach is less applicable, or not applicable at all,
for situations in which few geographic units are available for testing (e.g. smaller
countries, or subregions of larger countries) These situations also include the so-called
matched market tests, which may compare the behavior of users in a single
control region with the behavior of users in a single test region. To fill this gap, we
have developed an analogous time-based regression (TBR) approach for analyzing
geo experiments. This methodology predicts the time series of the counterfactual
market response, allowing for direct estimation of the cumulative causal effect at
the end of the experiment. In this paper we describe this model and evaluate its
performance using simulation.
View details
Preview abstract
The accuracy of an attribution model is limited by the assumptions of the model, and the quality and completeness of the data available to the model. Common digital attribution models on the market make a critical, yet hidden, assumption that ads only affect users by directly changing their propensity to convert. These models assume that ad exposure does not change user behavior in other ways, such as driving additional website visits, generating branded searches, or creating awareness and interest in the advertiser. In a previous paper, we described a Digital Advertising System Simulation (DASS) for modeling advertising and its impact on user behavior. In this paper, we use this simulation to demonstrate that current models fail to accurately capture the true number of incremental conversions generated by ads that impact user behavior, and introduce an Upstream Data-Driven Attribution (UDDA) model to address this shortcoming. We also demonstrate that development beyond UDDA is still required to address a lack of data completeness, and situations that include highly targeting advertising.
View details
Preview abstract
We describe a Digital Advertising System Simulation (DASS) for modeling advertising and its impact on user behavior. DASS is both flexible and general, and can be applied to research on a wide range of topics, such as digital attribution, ad fatigue, campaign optimization, and marketing mix modeling. This paper introduces the basic DASS simulation framework and illustrates its application to digital attribution. We show that common position-based attribution models fail to capture the true causal effects of advertising across several simple scenarios. These results lay a groundwork for the evaluation of more complex attribution models, and the development of improved models.
View details
Preview abstract
In a previous paper [6] we described the application of geo experiments to the measurement of advertising effectiveness. One reason this method of measurement is attractive is that it provides the rigor of a randomized experiment. However, related decisions, such as where and how to spend advertising budget, are not static. To address this issue, we extend this methodology to provide periodic (ongoing) measurement of ad effectiveness. In this approach, the test and control assignments of each geographic region rotate across multiple test periods, and these rotations provide the opportunity to generate a sequence of measurements of campaign effectiveness. The data across test periods can also be pooled to create a single aggregate measurement of campaign effectiveness. These sequential and pooled measurements have smaller confidence intervals than measurements from a series of geo experiments with a single test period. Alternatively, the same confidence interval can be achieved with a reduced magnitude and/or duration of ad spend change, thereby lowering the cost of measurement. The net result is a better method for periodic and isolated measurement of ad effectiveness.
View details
Catching a viral video
Tom Broxton
Yannet Interian
Journal of Intelligent Information Systems (2011), pp. 1-19
Preview abstract
The sharing and re-sharing of videos on social sites, blogs e-mail, and other means has given rise to the phenomenon of viral videos—videos that become popular through internet sharing. In this paper we seek to better understand viral videos on YouTube by analyzing sharing and its relationship to video popularity using millions of YouTube videos. The socialness of a video is quantified by classifying the referrer sources for video views as social (e.g. an emailed link, Facebook referral) or non-social (e.g. a link from related videos). We find that viewership patterns of highly social videos are very different from less social videos. For example, the highly social videos rise to, and fall from, their peak popularity more quickly than less social videos. We also find that not all highly social videos become popular, and not all popular videos are highly social. By using our insights on viral videos we are able develop a method for ranking blogs and websites on their ability to spread viral videos.
View details
Preview abstract
Advertisers have a fundamental need to quantify the effectiveness of their advertising. For search ad spend, this information provides a basis for formulating strategies related to bidding, budgeting, and campaign design. One approach that Google has successfully employed to measure advertising effectiveness is geo experiments. In these experiments, non-overlapping geographic regions are randomly assigned to a control or treatment condition, and each region realizes its assigned condition through the use of geo-targeted advertising. This paper describes the application of geo experiments and demonstrates that they are conceptually simple, have a systematic and effective design process, and provide results that are easy to interpret.
View details
Preview abstract
The sharing and re-sharing of videos on social sites, blogs e-mail, and other means has given rise to the phenomenon of viral videos – videos that become popular through internet sharing. In this paper we seek to better understand viral videos on YouTube by analyzing sharing and its relationship to video popularity using 1.5 million YouTube videos. The socialness of a video is quantified by classifying the
referrer sources for video views as social (e.g. an emailed link) or non-social (e.g. a link from related videos). By segmenting videos according to their fraction of social views, we find that viewership patterns of highly social videos is very different than less social videos. For example, the highly social videos rise to, and fall from, their peak popularity more quickly than less social videos. We also find that not all highly social videos become popular, and not all popular videos are highly social. And, despite their ability to generate large volumes of views over a short period of time, only 21% of the most popular videos (in terms of 30-day views) can be classified as viral. The observations made here lay the ground work for future work related to the creation of classification and predictive models for online videos.
View details
No Results Found
