An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Venue
International Conference on Computer Vision (ICCV) (2015)
Publication Year
2015
Authors
Yu Cheng, Felix X. Yu, Rogerio Feris, Sanjiv Kumar, Shih-Fu Chang
BibTeX
Abstract
We explore the redundancy of parameters in deep neural networks by replacing the
conventional linear projection in fully-connected layers with the circulant
projection. The circulant structure substantially reduces memory footprint and
enables the use of the Fast Fourier Transform to speed up the computation.
Considering a fully-connected neural network layer with d input nodes, and d output
nodes, this method improves the time complexity from O(d^2) to O(dlogd) and space
complexity from O(d^2) to O(d). The space savings are particularly important for
modern deep convolutional neural network architectures, where fully-connected
layers typically contain more than 90% of the network parameters. We further show
that the gradient computation and optimization of the circulant projections can be
performed very efficiently. Our experiments on three standard datasets show that
the proposed approach achieves this significant gain in storage and efficiency with
minimal increase in error rate compared to neural networks with unstructured
projections.
