Deep Mutual Learning | RDL Research Database

Abstract

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary - mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.

Keywords

Computer scienceDistillationArtificial intelligenceProcess (computing)Variety (cybernetics)Transfer of learningMachine learningTransfer (computing)Simple (philosophy)Parallel computingProgramming language

Affiliated Institutions

Related Publications

Learning Deconvolution Network for Semantic Segmentation

Hyeonwoo Noh , Seunghoon Hong , Bohyung Han

We propose a novel semantic segmentation algorithm by learning a deep deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer ne...

2015 3978 citations

Generative Adversarial Networks: An Overview

Antonia Creswell , Tom White , Vincent Dumoulin +3 more

Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this by deriving backpropagation sig...

2018 IEEE Signal Processing Magazine 4073 citations

Emerging Properties in Self-Supervised Vision Transformers

Mathilde Caron , Hugo Touvron , Ishan Misra +4 more

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond t...

2021 2021 IEEE/CVF International Conferenc... 4220 citations

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Anh‐Tu Nguyen , Jason Yosinski , Jeff Clune

Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Giv...

2015 3232 citations

Fully convolutional networks for semantic segmentation

Jonathan Long , Evan Shelhamer , Trevor Darrell

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, ex...

2015 35498 citations

Publication Info

Year: 2018
Type: article
Citations: 1668
Access: Closed

External Links

Download PDF (Free) View on DOI.org arXiv Semantic Scholar

Social Impact

Altmetric

Deep Mutual Learning

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1668

OpenAlex

287

Influential

1344

CrossRef

Cite This

APA Style

                            
                                    Ying Zhang, 
                                
                                    Tao Xiang, 
                                
                                    Timothy M. Hospedales
                                
                                et al.
                            
                            (2018). 
                            Deep Mutual Learning. 
                            2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
                            
                            .
                            https://doi.org/10.1109/cvpr.2018.00454

Identifiers

DOI: 10.1109/cvpr.2018.00454
arXiv: 1706.00384

Data Quality

Data completeness: 84%