Large-Scale Learning with Less RAM via Randomization
Venue
Proceedings of the 30 International Conference on Machine Learning (ICML) (2013), pp. 10
Publication Year
2013
Authors
Daniel Golovin, D. Sculley, H. Brendan McMahan, Michael Young
BibTeX
Abstract
We reduce the memory footprint of popular large-scale online learning methods by
projecting our weight vector onto a coarse discrete set using randomized rounding.
Compared to standard 32-bit float encodings, this reduces RAM usage by more than
50% during training and by up to 95% when making predictions from a fixed model,
with almost no loss in accuracy. We also show that randomized counting can be used
to implement per-coordinate learning rates, improving model quality with little
additional RAM. We prove these memory-saving methods achieve regret guarantees
similar to their exact variants. Empirical evaluation confirms excellent
performance, dominating standard approaches across memory versus accuracy
tradeoffs.
