Abstract

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

Keywords

Closed captioningComputer scienceLeverage (statistics)Joint (building)Artificial intelligenceVocabularyLanguage modelNatural language processingSpeech recognitionMachine learningLinguisticsImage (mathematics)

Affiliated Institutions

Related Publications

Publication Info

Year
2019
Type
article
Citations
1079
Access
Closed

External Links

View on DOI.org

Social Impact

Altmetric
PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1079
OpenAlex

Cite This

Chen Sun, Austin Myers, Carl Vondrick et al. (2019). VideoBERT: A Joint Model for Video and Language Representation Learning. . https://doi.org/10.1109/iccv.2019.00756

Identifiers

DOI
10.1109/iccv.2019.00756