Machine Learning Algorithms and Techniques
The Google Brain team’s mission is "Make machines intelligent. Improve people’s lives." We combine open-ended machine learning research with world-class system engineering and Google-scale computing resources to realize this mission.
Our research started with the development of DistBelief, as a common platform to experiment with various unsupervised and supervised learning algorithms for computer vision, speech recognition and other areas. In computer vision, our team members have played key roles in developing award-winning AlexNet and InceptionNet models, and DeepDream. In Speech Recognition, our team members have pioneered the use of deep Learning for acoustic modeling. In natural language understanding, our team members have advanced word vectors, neural language modeling and pioneered sequence to sequence learning.
We currently conduct fundamental research to further advance key areas in machine intelligence and to create a better theoretical understanding of deep learning such as in Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity. Our recent research achievements also include: unsupervised learning, adversarial training, structured learning, long-term dependencies, knowledge distillation, general learning algorithms, understanding of learning algorithms, reinforcement learning, AI safety and TensorFlow.
Some of Our Publications
- Building high-level features using large scale unsupervised learning Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg Corrado, Jeff Dean, Andrew Y. Ng. 2012 (1,150 citations)
- Intriguing properties of neural networks Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. ICLR 2014 (388 citations)
- Distilling the Knowledge in a Neural Network Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean. NIPS Deep Learning and Representation Learning Workshop, 2014 (282 citations)
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam M. Shazeer. NIPS, 2015 (109 citations)
- Concrete Problems in AI Safety Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. ArXiv, 2016 (28 citations)
- Train faster, generalize better: Stability of stochastic gradient descent Moritz Hardt, Benjamin Recht, Yoram Singer. JMLR, 2016 (63 citations)
Publications by Year
- 2016
- A Neural Transducer Navdeep Jaitly, David Sussillo, Quoc V. Le, Oriol Vinyals, Ilya Sutskever, Samy Bengio. NIPS, 2016 (5 citations)
- Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow. ICLR Workshop, 2016 (59 citations)
- Adversarial examples in the physical world Alexey Kurakin, Ian Goodfellow, Samy Bengio. ArXiv, 2016 (24 citations)
- An Online Sequence-to-Sequence Model Using Partial Conditioning Navdeep Jaitly, Quoc V. Le, Oriol Vinyals, Ilya Sutskever, Samy Bengio. ArXiv, 2016 (6 citations)
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, Geoffrey E. Hinton. NIPS, 2016 (30 citations)
- Can Active Memory Replace Attention? Lukasz Kaiser and Samy Bengio. NIPS, 2016 (4 citations)
- Concrete Problems in AI Safety Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. 2016 (28 citations)
- Deep Learning Games Dale Schuurmans, Martin Zinkevich. NIPS, 2016 (1 citation)
- Deep Learning with Differential Privacy Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. ArXiv, 2016. An updated version of this work will appear in ACM CCS 2016 (35 citations)
- DeepMath - Deep Sequence Models for Premise Selection Alex Alemi, Francois Chollet, Geoffrey Irving, Christian Szegedy, and Josef Urban. NIPS, 2016 (6 citations)
- Density estimation using Real NVP Viet Hanh Laurent Dinh, Jascha Sohl-dickstein, Samy Bengio. ArXiv, 2016 (26 citations)
- Domain Separation Networks Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, Dumitru Erhan. NIPS, 2016 (12 citations)
- Equality of Opportunity in Supervised Learning M. Hardt, E. Price, N. Srebro. NIPS, 2016 (14 citations)
- Exponential expressivity in deep neural networks through transient chaos Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli. NIPS, 2016 (7 citations)
- Layer Normalization Jimmy Ba, Ryan Kiros, Geoffrey E. Hinton. ArXiv, 2016 (46 citations)
- Mastering the game of Go with deep neural networks and tree search David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis. Nature, 2016. (848 citations)
- Net2Net: Accelerating Learning via Knowledge Transfer Tianqi Chen, Ian Goodfellow, Jonathon Shlens. ICLR, 2016 (30 citations)
- Neural GPUs Learn Algorithms Lukasz Kaiser, Ilya Sutskever. ICLR, 2016 (57 citations)
- Neural Programmer: Inducing Latent Programs with Gradient Descent Arvind Neelakantan, Quoc V. Le, Ilya Sutskever. ICLR, 2016 (58 citations)
- Neural Random-Access Machines Karol Kurach, Marcin Andrychowicz, Ilya Sutskever. ICR, 2016 (42 citations)
- On the expressive power of deep neural networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein. ArXiv, 2016 (11 citations)
- Order matters: Sequence to sequence for sets Oriol Vinyals, Samy Bengio, Manjunath Kudlur. ICLR, 2016 (32 citations)
- Reward Augmented Maximum Likelihood for Neural Structured Prediction Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans. NIPS, 2016 (10 citations)
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity Amit Daniely, Roy Frostig, Yoram Singer. NIPS, 2016 (13 citations)
- Towards Principled Unsupervised Learning Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, Danilo Rezende, Tim Lillicrap, Oriol Vinyals. ICLR, 2016 (5 citations)
- Train faster, generalize better: Stability of stochastic gradient descent Moritz Hardt, Benjamin Recht, Yoram Singer. JMLR, 2016 (62 citations)
- Unsupervised Learning for Physical Interaction via Video Prediction Chelsea Finn, Ian Goodfellow, Sergey Levine. NIPS, 2016 (27 citations)
- Using Fast Weights to Attend to the Recent Past Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Leibo, Catalin Ionescu. NIPS, 2016 (5 citations)
- Virtual Adversarial Training on Semi-Supervised Text Classification Takeru Miyato, Andrew Dai and Ian Goodfellow. ArXiv, 2016
- 2015
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. ArXiv, 2015 (110 citations)
- A Unified Approach to Boundedness Properties in MSO Lukasz Kaiser, Martin Lang, Simon Leßenich, Christof Löding. CSL, 2015 (4 citations)
- Adding Gradient Noise Improves Learning for Very Deep Networks Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, James Martens. ArXiv, 2015 (39 citations)
- An empirical exploration of recurrent network architectures Rafal Jozefowicz, Wojciech Zaremba, Ilya Sutskever. ICML, 2015 (186 citations)
- Deep convolutional neural networks for large-scale speech tasks Tara N Sainath, Brian Kingsbury, George Saon, Hagen Soltau, Abdel-rahman Mohamed, George Dahl, Bhuvana Ramabhadran. Neural Networks, 2015 (88 citations)
- Deep learning Y LeCun, Y Bengio, G Hinton. Nature, 2015 (2,264 citations)
- Deep Networks With Large Output Spaces Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, Jay Yagnik. ICLR, 2015 (12 citations)
- Explaining and Harnessing Adversarial Examples Ian Goodfellow, Jonathon Shlens, Christian Szegedy. ICLR, 2015 (227 citations)
- Going Deeper with Convolutions Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan,Vincent Vanhoucke, Andrew Rabinovich. CVPR, 2015 (2,899 citations)
- Guest Editorial: Deep Learning Marc'Aurelio Ranzato, Geoffrey E. Hinton, Yann LeCun. International Journal of Computer Vision, vol. 113 (2015), pp. 1-2 (7 citations)
- Pointer Networks Oriol Vinyals, Meire Fortunato, Navdeep Jaitly. NIPS, 2015 (92 citations)
- Qualitatively Characterizing Neural Network Optimization Problems Ian Goodfellow, Oriol Vinyals, Andrew Saxe. ICLR, 2015 (29 citations)
- Rethinking the inception architecture for computer vision Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna. ArXiv, 2015 (227 citations)
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam M. Shazeer. NIPS, 2015 (107 citations)
- Semi-supervised sequence learning Andrew M. Dai, Quoc V. Le. NIPS, 2015 (66 citations)
- Show and Tell: A Neural Image Caption Generator Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. CVPR, 2015 (766 citations)
- Training Deep Neural Networks on Noisy Labels with Bootstrapping Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich. ICLR, 2015 (41 citations)
- 2014
- Distilling the Knowledge in a Neural Network Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean. NIPS Deep Learning and Representation Learning Workshop, 2014 (281 citations)
- Intriguing properties of neural networks Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. ICLR 2014 (386 citations)
- Learning Factored Representations in a Deep Mixture of Experts David Eigen, Marc’Aurelio Ranzato, Ilya Sutskever. ArXiv, 2014 (5 citations)
- Learning to Execute Wojciech Zaremba, Ilya Sutskever. ArXiv, 2014 (131 citations)
- Local Collaborative Ranking Joonseok Lee, Samy Bengio, Seungyeon Kim, Guy Lebanon, Yoram Singer. Proceedings of the 23rd International World Wide Web Conference (WWW), ACM (2014) (50 citations)
- Move evaluation in Go using deep convolutional neural networks Chris J Maddison, Aja Huang, Ilya Sutskever, David Silver. ArXiv, 2014 (50 citations)
- On Learning Where To Look Marc'Aurelio Ranzato. ArXiv, 2014 (14 citations)
- Random Walk Initialization for Training Very Deep Feedforward Networks David Sussillo, L.F. Abbott. ArXiv, 2014 (9 citations)
- Random walks: Training very deep nonlinear feed-forward networks with smart initialization David Sussillo, Larry F. Abbott. ArXiv, 2014 (6 citations)
- Recurrent neural network regularization Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals. ArXiv, 2014 (318 citations)
- Reinforcement learning Neural Turing machines Wojciech Zaremba, Ilya Sutskever. ArXiv, 2014 (65 citations)
- Sequence to Sequence Learning with Neural Networks Ilya Sutskever, Oriol Vinyals, Quoc V. Le. NIPS, 2014 (1,465 citations)
- Training Highly Multi-class Linear Classifiers Maya Gupta, Samy Bengio, Jason Weston. JMLR, 2014 (22 citations)
- Zero-shot learning by convex combination of semantic embeddings Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, Jeffrey Dean. ICLR, 2014 (118 citations)
- 2013
- Devise: A deep visual-semantic embedding model Andreas Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, Tomas Mikolov. NIPS, 2013 (375 citations)
- Fastfood-approximating kernel expansions in loglinear time Quoc V. Le, Tamás Sarlós, Alex Smola. ICML, 2013 (147 citations)
- Fastfood-computing Hilbert Space Expansions in Loglinear Time Quoc V. Le, Tamas Sarlós, Alex Smola. ICML, 2013 (147 citations)
- On rectified linear units for speech processing Matthew D. Zeiler, M Ranzato, Rajat Monga, Min Mao, Kun Yang, Quoc V. Le, Patrick Nguyen, Alan Senior, Vincent Vanhoucke, Jeffrey Dean, Geoffrey E. Hinton. ICASSP, 2013 (206 citations)
- Scalable, high-quality object detection Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean. NIPS, 2013 (84 citations)
- 2012
- Building high-level features using large scale unsupervised learning Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg Corrado, Jeff Dean, Andrew Y. Ng. 2012 (1,147 citations)
Some of Our Team
- Martin Abadi
- Anelia Angelova
- Samy Bengio
- Greg Corrado
- Amit Daniely
- Jeffrey Dean
- George Dahl
- Andrew Dai
- Douglas Eck
- Dumitru Erhan
- Stephan Gouws
- Vineet Gupta
- Moritz Hardt
- Geoffrey Hinton
- Geoffrey Irving
- Navdeep Jaitly
- Lukasz Kaiser
- Tomer Koren
- Quoc V. Le
- Manjunath Kudlur
- Nevena Lazic
- Mohammad Norouzi
- Christopher Olah
- Pierre Sermanet
- Yoram Singer
- Jonathon Shlens
- Mike Schuster
- Jascha Sohl-Dickstein
- David Sussillo
- Kunal Talwar
- Vincent Vanhoucke