Speeding up Convolutional Neural Networks with Low Rank Expansions

Abstract

The focus of this paper is speeding up the evaluation of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability. Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain. Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition, showing a possible 2.5x speedup with no loss in accuracy, and 4.5x speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.

Keywords

SpeedupComputer scienceConvolutional neural networkRedundancy (engineering)Deep learningComputer engineeringArtificial intelligenceParallel computingAlgorithm

Affiliated Institutions

University of Oxford GB

Related Publications

Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

Hanchuan Peng , Fuhui Long , Chen Ding

Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion base...

2005 IEEE Transactions on Pattern Analysis... 10050 citations

A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton , Simon Osindero , Yee‐Whye Teh

We show how to use “complementary priors” to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. U...

2006 Neural Computation 16027 citations

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Mingxing Tan , Quoc V. Le

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper...

2019 arXiv (Cornell University) 5008 citations

Learning Deconvolution Network for Semantic Segmentation

Hyeonwoo Noh , Seunghoon Hong , Bohyung Han

We propose a novel semantic segmentation algorithm by learning a deep deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer ne...

2015 3978 citations

GhostNet: More Features From Cheap Operations

Kai Han , Yunhe Wang , Qi Tian +3 more

Deploying convolutional neural networks (CNNs) on embedded devices is difficult due to the limited memory and computation resources. The redundancy in feature maps is an importa...

2020 3894 citations

Publication Info

Year: 2014
Type: preprint
Citations: 543
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Speeding up Convolutional Neural Networks with Low Rank Expansions

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

543

OpenAlex

Cite This

APA Style

                            
                                    Max Jaderberg, 
                                
                                    Andrea Vedaldi, 
                                
                                    Andrew Zisserman
                                
                            (2014). 
                            Speeding up Convolutional Neural Networks with Low Rank Expansions. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1405.3866

Identifiers

DOI: 10.48550/arxiv.1405.3866