Supervised Multimodal Bitransformers for Classifying Images and Text

Douwe Kiela; Suvrat Bhooshan; Hamed Firooz; Ethan Perez; Davide Testuggine

doi:10.48550/arxiv.1909.02950

Abstract

Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.

Keywords

Computer scienceBenchmark (surveying)EncoderArtificial intelligenceModalitiesTransformerMultimodal learningMachine learningVariety (cybernetics)Natural language processingPattern recognition (psychology)Engineering

Related Publications

TinyBERT: Distilling BERT for Natural Language Understanding

Xiaoqi Jiao , Yichun Yin , Lifeng Shang +5 more

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usuall...

2020 Findings of the Association for Compu... 1491 citations

BERT Rediscovers the Classical NLP Pipeline

Ian Tenney , Dipanjan Das , Ellie Pavlick

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is cap...

2019 1214 citations

Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey

Longlong Jing , Yingli Tian

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer...

2020 IEEE Transactions on Pattern Analysis... 1818 citations

Universal Sentence Encoder

Daniel Cer , Yinfei Yang , Sheng-yi Kong +10 more

We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate pe...

2018 arXiv (Cornell University) 1289 citations

ALBERT: A Lite BERT for Self-supervised Learning of Language\n Representations

Zhenzhong Lan , Mingda Chen , Sebastian Goodman +3 more

Increasing model size when pretraining natural language representations often\nresults in improved performance on downstream tasks. However, at some point\nfurther model increas...

2019 arXiv (Cornell University) 4051 citations

Publication Info

Year: 2019
Type: preprint
Citations: 163
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Supervised Multimodal Bitransformers for Classifying Images and Text

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

163

OpenAlex

Cite This

APA Style

                            
                                    Douwe Kiela, 
                                
                                    Suvrat Bhooshan, 
                                
                                    Hamed Firooz
                                
                                et al.
                            
                            (2019). 
                            Supervised Multimodal Bitransformers for Classifying Images and Text. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1909.02950

Identifiers

DOI: 10.48550/arxiv.1909.02950