Asynchronous Stochastic Optimization for Sequence Training of Deep Neural Networks
Venue
Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), IEEE, Firenze, Italy (2014)
Publication Year
2014
Authors
Georg Heigold, Erik McDermott, Vincent Vanhoucke, Andrew Senior, Michiel Bacchiani
BibTeX
Abstract
This paper explores asynchronous stochastic optimization for sequence training of
deep neural networks. Sequence training requires more computation than frame-level
training using pre-computed frame data. This leads to several complications for
stochastic optimization, arising from significant asynchrony in model updates under
massive parallelization, and limited data shuffling due to utterance-chunked
processing. We analyze the impact of these two issues on the efficiency and
performance of sequence training. In particular, we suggest a framework to
formalize the reasoning about the asynchrony and present experimental results on
both small and large scale Voice Search tasks to validate the effectiveness and
efficiency of asynchronous stochastic optimization.
