We investigate a reduction of supervised learning to game playing that reveals new
connections and learning methods. For convex one-layer problems, we demonstrate an
equivalence between global minimizers of the training problem and Nash equilibria
in a simple game. We then show how the game can be extended to general acyclic
neural networks with differentiable convex gates, establishing a bijection between
the Nash equilibria and critical (or KKT) points of the deep learning problem.
Based on these connections we investigate alternative learning methods, and find
that regret matching can achieve competitive training performance while producing
sparser models than current deep learning approaches.