Sequence-to-sequence models have shown success in end-to-end speech recognition.
However these models have only used shallow acoustic encoder networks. In our work,
we successively train very deep convolutional networks to add more expressive power
and better generalization for end-to-end ASR models. We apply network-in-network
principles, batch normalization, residual connections and convolutional LSTMs to
build very deep recurrent and convolutional structures. Our models exploit the
spectral structure in the feature space and add computational depth without
overfitting issues. We experiment with the WSJ ASR task and achieve 10.5% word
error rate without any dictionary or language using a 15 layer deep network.