More Bang for Their Bucks: Assessing New Features for Online Advertisers
AdKDD07 (in the ACM digital library) (2007)
Diane Lambert, Daryl Pregibon
This paper addresses the problem of drawing valid inferences from white-list trials about the effects of new features on advertiser happiness. We are guided by three principles. First, statistical procedures for white-list trials are likely to be applied in an automated way, so they should be robust to violations of modeling assumptions. Second, standard analysis tools should be preferred over custom-built ones, both for clarity and for robustness. Standard tools have withstood the test of time and have been thoroughly debugged. Finally, it should be easy to compute reliable confidence intervals for the estimator. We review an estimator that has all these attributes, allowing us to make valid inferences about the effects of a new feature on advertiser happiness. In the example in this paper, the new feature was introduced during the holiday shopping season, thereby further complicating the analysis.