Machine Learning Algorithms and Techniques
The Google Brain team’s mission is "Make machines intelligent. Improve people’s lives." We combine open-ended machine learning research with world-class system engineering and Google-scale computing resources to realize this mission.
Our research started with the development of DistBelief, as a common platform to experiment with various unsupervised and supervised learning algorithms for computer vision, speech recognition and other areas. In computer vision, our team members have played key roles in developing award-winning AlexNet and InceptionNet models, and DeepDream. In Speech Recognition, our team members have pioneered the use of deep Learning for acoustic modeling. In natural language understanding, our team members have advanced word vectors, neural language modeling and pioneered sequence to sequence learning.
We currently conduct fundamental research to further advance key areas in machine intelligence. Our recent research achievements also include: unsupervised learning, adversarial training, structured learning, long-term dependencies, knowledge distillation, general learning algorithms, understanding of learning algorithms, reinforcement learning, AI safety and TensorFlow.
Representative publications by Google Brain team members
Selected publications
- Building high-level features using large scale unsupervised learning Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg Corrado, Jeff Dean, Andrew Y. Ng. 2012 (877 citations)
- Intriguing properties of neural networks Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. ICLR 2014 (205 citations)
- Distilling the Knowledge in a Neural Network Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean. NIPS Deep Learning and Representation Learning Workshop, 2014 (143 citations)
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam M. Shazeer. NIPS, 2015 (35 citations)
- Concrete Problems in AI Safety Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. ArXiv, 2016 (1 citation)
Other publications
- 2016
- Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow. ICLR Workshop, 2016 (6 citations)
- Adversarial examples in the physical world Alexey Kurakin, Ian Goodfellow, Samy Bengio. ArXiv, 2016
- An Online Sequence-to-Sequence Model Using Partial Conditioning Navdeep Jaitly, Quoc V. Le, Oriol Vinyals, Ilya Sutskever, Samy Bengio. ArXiv, 2016
- Concrete Problems in AI Safety Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. 2016 (1 citation)
- Deep Learning with Differential Privacy Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang. ArXiv, 2016. An updated version of this work will appear in ACM CCS 2016 (1 citation)
- DeepMath - Deep Sequence Models for Premise Selection Alex A. Alemi, Francois Chollet, Geoffrey Irving, Christian Szegedy, Josef Urban. ArXiv, 2016
- Density estimation using Real NVP Viet Hanh Laurent Dinh, Jascha Sohl-dickstein, Samy Bengio. ArXiv, 2016 (2 citations)
- Exponential expressivity in deep neural networks through transient chaos Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli. ArXiv, 2016
- Layer Normalization Jimmy Ba, Ryan Kiros, Geoffrey E. Hinton. ArXiv, 2016
- Mastering the game of Go with deep neural networks and tree search David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis. Nature, 2016. (176 citations)
- Net2Net: Accelerating Learning via Knowledge Transfer Tianqi Chen, Ian Goodfellow, Jonathon Shlens. ICLR, 2016 (13 citations)
- Neural GPUs Learn Algorithms Lukasz Kaiser, Ilya Sutskever. ICLR, 2016 (16citations)
- Neural Programmer: Inducing Latent Programs with Gradient Descent Arvind Neelakantan, Quoc V. Le, Ilya Sutskever. ICLR, 2016 (20 citations)
- Neural Random-Access Machines Karol Kurach, Marcin Andrychowicz, Ilya Sutskever. ICR, 2016 (12 citations)
- On the expressive power of deep neural networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl-Dickstein. ArXiv, 2016
- Order matters: Sequence to sequence for sets Oriol Vinyals, Samy Bengio, Manjunath Kudlur. ICLR, 2016 (8 citations)
- Towards Principled Unsupervised Learning Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, Danilo Rezende, Tim Lillicrap, Oriol Vinyals. ICLR, 2016 (3 citations)
- Virtual Adversarial Training on Semi-Supervised Text Classification Takeru Miyato, Andrew Dai and Ian Goodfellow. ArXiv, 2016
- 2015
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. ArXiv, 2015 (51 citations)
- A Unified Approach to Boundedness Properties in MSO Lukasz Kaiser, Martin Lang, Simon Leßenich, Christof Löding. CSL, 2015 (3 citations)
- Adding Gradient Noise Improves Learning for Very Deep Networks Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, James Martens. ArXiv, 2015 (16 citations)
- An empirical exploration of recurrent network architectures Rafal Jozefowicz, Wojciech Zaremba, Ilya Sutskever. ICML, 2015 (67 citations)
- Deep convolutional neural networks for large-scale speech tasks Tara N Sainath, Brian Kingsbury, George Saon, Hagen Soltau, Abdel-rahman Mohamed, George Dahl, Bhuvana Ramabhadran. Neural Networks, 2015 (44 citations)
- Deep learning Y LeCun, Y Bengio, G Hinton. Nature, 2015 (677 citations)
- Deep Networks With Large Output Spaces Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, Jay Yagnik. ICLR, 2015 (7 citations)
- Explaining and Harnessing Adversarial Examples Ian Goodfellow, Jonathon Shlens, Christian Szegedy. ICLR, 2015 (94 citations)
- Going Deeper with Convolutions Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan,Vincent Vanhoucke, Andrew Rabinovich. CVPR, 2015 (1327 citations)
- Guest Editorial: Deep Learning Marc'Aurelio Ranzato, Geoffrey E. Hinton, Yann LeCun. International Journal of Computer Vision, vol. 113 (2015), pp. 1-2 (2 citations)
- Pointer Networks Oriol Vinyals, Meire Fortunato, Navdeep Jaitly. NIPS, 2015 (32 citations)
- Qualitatively Characterizing Neural Network Optimization Problems Ian Goodfellow, Oriol Vinyals, Andrew Saxe. ICLR, 2015 (12 citations)
- Rethinking the inception architecture for computer vision Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna. ArXiv, 2015 (39 citations)
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam M. Shazeer. NIPS, 2015 (35 citations)
- Semi-supervised sequence learning Andrew M. Dai, Quoc V. Le. NIPS, 2015 (24 citations)
- Show and Tell: A Neural Image Caption Generator Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. CVPR, 2015 (399 citations)
- Training Deep Neural Networks on Noisy Labels with Bootstrapping Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich. ICLR, 2015 (21 citations)
- 2014
- Intriguing properties of neural networks Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. ICLR 2014 (205 citations)
- Learning Factored Representations in a Deep Mixture of Experts David Eigen, Marc’Aurelio Ranzato, Ilya Sutskever. ArXiv, 2014 (3 citations)
- Learning to Execute Wojciech Zaremba, Ilya Sutskever. ArXiv, 2014 (73 citations)
- Local Collaborative Ranking Joonseok Lee, Samy Bengio, Seungyeon Kim, Guy Lebanon, Yoram Singer. Proceedings of the 23rd International World Wide Web Conference (WWW), ACM (2014) (35 citations)
- Move evaluation in Go using deep convolutional neural networks Chris J Maddison, Aja Huang, Ilya Sutskever, David Silver. ArXiv, 2014 (30 citations)
- On Learning Where To Look Marc'Aurelio Ranzato. ArXiv, 2014 (9 citations)
- Random Walk Initialization for Training Very Deep Feedforward Networks David Sussillo, L.F. Abbott. ArXiv, 2014 (3 citations)
- Random walks: Training very deep nonlinear feed-forward networks with smart initialization David Sussillo, Larry F. Abbott. ArXiv, 2014 (4 citations)
- Recurrent neural network regularization Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals. ArXiv, 2014 (136 citations)
- Reinforcement learning Neural Turing machines Wojciech Zaremba, Ilya Sutskever. ArXiv, 2014 (37 citations)
- Sequence to Sequence Learning with Neural Networks Ilya Sutskever, Oriol Vinyals, Quoc V. Le. NIPS, 2014 (650 citations)
- Training Highly Multi-class Linear Classifiers Maya Gupta, Samy Bengio, Jason Weston. JMLR, 2014 (12 citations)
- Zero-shot learning by convex combination of semantic embeddings Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, Jeffrey Dean. ICLR, 2014 (67 citations)
- 2013
- Devise: A deep visual-semantic embedding model Andreas Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, Tomas Mikolov. NIPS, 2013 (236 citations)
- Fastfood-approximating kernel expansions in loglinear time Quoc V. Le, Tamás Sarlós, Alex Smola. ICML, 2013 (115 citations)
- Fastfood-computing Hilbert Space Expansions in Loglinear Time Quoc V. Le, Tamas Sarlós, Alex Smola. ICML, 2013 (115 citations)
- On rectified linear units for speech processing Matthew D. Zeiler, M Ranzato, Rajat Monga, Min Mao, Kun Yang, Quoc V. Le, Patrick Nguyen, Alan Senior, Vincent Vanhoucke, Jeffrey Dean, Geoffrey E. Hinton. ICASSP, 2013 (157 citations)
- Scalable, high-quality object detection Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean. NIPS, 2013 (55 citations)
Current Google Brain team members who work on this area
- Martin Abadi
- Anelia Angelova
- Samy Bengio
- Greg Corrado
- Jeffrey Dean
- George Dahl
- Andrew Dai
- Douglas Eck
- Dumitru Erhan
- Stephan Gouws
- Geoffrey Hinton
- Geoffrey Irving
- Navdeep Jaitly
- Lukasz Kaiser
- Quoc V. Le
- Manjunath Kudlur
- Mohammad Norouzi
- Christopher Olah
- Pierre Sermanet
- Jonathon Shlens
- Mike Schuster
- Jascha Sohl-Dickstein
- David Sussillo
- Vincent Vanhoucke
