The talk presents key aspects faced when building language models (LM) for the
google.com query stream, and their use for automatic speech recognition (ASR).
Distributed LM tools enable us to handle a huge amount of data, and experiment with
LMs that are two orders of magnitude larger than usual. An empirical exploration of
the problem led us to re-discovering a less known interaction between Kneser-Ney
smoothing and entropy pruning, possible non-stationarity of the query stream, as
well as strong dependence on various English locales---USA, Britain and Australia.
LM compression techniques allowed us to use one billion n-gram LMs in the first
pass of an ASR system built on FST technology, and evaluate empirically whether a
two-pass system architecture has any losses over one pass.