Massively Parallel Methods for Deep Reinforcement Learning

Abstract

We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.

Keywords

Reinforcement learningMassively parallelComputer scienceArchitectureHyperparameterArtificial neural networkFunction (biology)Distributed computingDeep learningArtificial intelligenceParallel computing

Affiliated Institutions

Google (United States) US

Related Publications

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

Stefan Elfwing , Eiji Uchibe , Kenji Doya

In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level hu...

2018 Neural Networks 1643 citations

Playing Atari with Deep Reinforcement Learning

Alex Graves , Ioannis Antonoglou , Daan Wierstra +4 more

We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolu...

2013 arXiv (Cornell University) 5109 citations

Deep Reinforcement Learning with Double Q-Learning

Hado van Hasselt , Arthur Guez , David Silver

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are comm...

2016 Proceedings of the AAAI Conference on... 3514 citations

Rainbow: Combining Improvements in Deep Reinforcement Learning

Matteo Hessel , Joseph Modayil , Hado van Hasselt +7 more

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and ...

2018 Proceedings of the AAAI Conference on... 1630 citations

Asynchronous Methods for Deep Reinforcement Learning

Volodymyr Mnih , Adrià Puigdomènech Badia , Mehdi Mirza +5 more

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network contro...

2016 arXiv (Cornell University) 1690 citations

Publication Info

Year: 2015
Type: preprint
Citations: 405
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Massively Parallel Methods for Deep Reinforcement Learning

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

405

OpenAlex

Cite This

APA Style

                            
                                    Arun Sukumaran Nair, 
                                
                                    P. Srinivasan, 
                                
                                    Sam Blackwell
                                
                                et al.
                            
                            (2015). 
                            Massively Parallel Methods for Deep Reinforcement Learning. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1507.04296

Identifiers

DOI: 10.48550/arxiv.1507.04296