Training Very Deep Networks | RDL Research Database

Abstract

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

Keywords

Computer scienceTraining (meteorology)Simple (philosophy)Deep neural networksArtificial intelligenceArtificial neural networkInformation flowArchitectureDeep learningGradient descentFlow (mathematics)Distributed computingGeography

Affiliated Institutions

Related Publications

Highway Networks

Rupesh K. Srivastava , Klaus Greff , Jürgen Schmidhuber

There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult w...

2015 arXiv (Cornell University) 301 citations

Going deeper with convolutions

Christian Szegedy , Wei Liu , Yangqing Jia +6 more

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Sca...

2015 45596 citations

Generalization of Back propagation to Recurrent and Higher Order Neural Networks

Fernando J. Pineda

A general method for deriving backpropagation algorithms for networks with recurrent and higher order networks is introduced. The propagation of activation in these networks is ...

1987 Neural Information Processing Systems 123 citations

Long Short-Term Memory

Sepp Hochreiter , Jürgen Schmidhuber

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We brief...

1997 Neural Computation 90535 citations

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe , Christian Szegedy

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. T...

2024 arXiv (Cornell University) 15635 citations

Publication Info

Year: 2015
Type: preprint
Citations: 1100
Access: Closed

External Links

Citation Metrics

1100

OpenAlex

Cite This

APA Style

                            
                                    Rupesh K. Srivastava, 
                                
                                    Klaus Greff, 
                                
                                    Jürgen Schmidhuber
                                
                            (2015). 
                            Training Very Deep Networks. 
                            arXiv (Cornell University)
                            
                            .