Revisiting Distributed Synchronous SGD

Abstract

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.

Keywords

Computer scienceDistributed computing

Related Publications

Incremental, iterative data processing with timely dataflow

Derek G. Murray , Frank McSherry , Michael Isard +3 more

We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations....

2016 Communications of the ACM 36 citations

Large Scale Distributed Deep Networks

Jay B. Dean , Greg S. Corrado , Rajat Monga +9 more

Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider t...

2012 2906 citations

Project Adam: building an efficient and scalable deep learning training system

Trishul Chilimbi , Yutaka Suzue , Johnson Apacible +1 more

Large deep neural network models have recently demonstrated state-of-the-art accuracy on hard visual recognition tasks. Unfortunately such models are extremely time consuming to...

2014 Operating Systems Design and Implemen... 596 citations

Publication Info

Year: 2017
Type: preprint
Citations: 609
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Revisiting Distributed Synchronous SGD

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

609

OpenAlex

Cite This

APA Style

                            
                                    Jianmin Chen, 
                                
                                    Rajat Monga, 
                                
                                    Samy Bengio
                                
                                et al.
                            
                            (2017). 
                            Revisiting Distributed Synchronous SGD. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1702.05800

Identifiers

DOI: 10.48550/arxiv.1702.05800