Continuous Pipelines at Google
Venue
SRECon Europe 2015, USENIX, Dublin, Ireland, pp. 12
Publication Year
2015
Authors
BibTeX
Abstract
This article focuses on the real life challenges of managing data processing
pipelines of depth and complexity. It considers the frequency continuum between
periodic pipelines that run very infrequently through continuous pipelines that
never stop running, and discusses the discontinuities that can produce significant
operational problems. A fresh take on the masterĀslave model is presented as a more
reliable and better scaling alternative to the periodic pipeline for processing Big
Data.
