We present new algorithms for adaptively learning artificial neural networks. Our
algorithms (ADANET) adaptively learn both the structure of the network and its
weights. They are based on a solid theoretical analysis, including data-dependent
generalization guarantees that we prove and discuss in detail. We report the
results of large-scale experiments with one of our algorithms on several binary
classification tasks extracted from the CIFAR-10 dataset and on the Criteo dataset.
The results demonstrate that our algorithm can automatically learn network
structures with very competitive performance accuracies when compared with those
achieved by neural networks found by standard approaches.