Google’s Ads Data Infrastructure systems run the multi- billion dollar ads business
at Google. High availability and strong consistency are critical for these systems.
While most distributed systems handle machine-level failures well, handling
datacenter-level failures is less common. In our experience, handling
datacenter-level failures is critical for running true high availability systems.
Most of our systems (e.g. Photon, F1, Mesa) now support multi-homing as a
fundamental design property. Multi-homed systems run live in multiple datacenters
all the time, adaptively moving load between datacenters, with the ability to
handle outages of any scale completely transparently. This paper focuses primarily
on stream processing systems, and describes our general approaches for building
high availability multi-homed systems, discusses common challenges and solutions,
and shares what we have learned in building and running these large-scale systems
for over ten years.