Deep Contextualized Word Representations

Abstract

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

Keywords

PolysemyComputer scienceNatural language processingArtificial intelligenceWord (group theory)SyntaxTextual entailmentSemantics (computer science)Representation (politics)LinguisticsDeep learningLogical consequenceProgramming language

Related Publications

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Jinhyuk Lee , Wonjin Yoon , Sungdong Kim +4 more

Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processi...

2019 Bioinformatics 6148 citations

Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer

Colin Raffel , Noam Shazeer , Adam Roberts +6 more

Transfer learning, where a model is first pre-trained on a data-rich task\nbefore being fine-tuned on a downstream task, has emerged as a powerful\ntechnique in natural language...

2019 arXiv (Cornell University) 8299 citations

Chemical Markup, XML, and the World Wide Web. 6. CMLReact, an XML Vocabulary for Chemical Reactions

Gemma L. Holliday , Peter Murray‐Rust , Henry S. Rzepa

A set of components (CMLReact) for managing chemical and biochemical reactions has been added to CML. These can be combined to support most of the strategies for the formal repr...

2005 Journal of Chemical Information and M... 66 citations

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Richard Zhang , Phillip Isola , Alexei A. Efros +2 more

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, ...

2018 10763 citations

Representation and Structure in Connectionist Models

Jeffrey L. Elman

Abstract : This paper focuses on the nature of representations in connectionist models. It addresses two issues: Can connectionist models develop representations which possess i...

1989 179 citations

Publication Info

Year: 2018
Type: preprint
Pages: 2227-2237
Citations: 1786
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Deep Contextualized Word Representations

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1786

OpenAlex

Cite This

APA Style

                            
                                    Matthew E. Peters, 
                                
                                    Mark E Neumann, 
                                
                                    Mohit Iyyer
                                
                                et al.
                            
                            (2018). 
                            Deep Contextualized Word Representations. 
                            
                            , 2227-2237.
                            https://doi.org/10.18653/v1/n18-1202

Identifiers

DOI: 10.18653/v1/n18-1202