Exploring the Limits of Language Modeling

Rafał Józefowicz; Oriol Vinyals; Mike Schuster; Noam Shazeer; Yonghui Wu

doi:10.48550/arxiv.1602.02410

Abstract

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

Keywords

LinguisticsComputer sciencePhilosophy

Affiliated Institutions

Google (United States) US

Related Publications

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

裕二池谷 , Robert Tinn , Hao Cheng +6 more

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on g...

2021 ACM Transactions on Computing for Hea... 1737 citations

Publicly Available Clinical

Emily Alsentzer , John R. Murphy , William Boag +4 more

Contextual word embedding models such as ELMo and BERT have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these m...

2019 Proceedings of the 2nd Clinical Natur... 1422 citations

ERNIE: Enhanced Language Representation with Informative Entities

Zhengyan Zhang , Xu Han , Zhiyuan Liu +3 more

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently ...

2019 1349 citations

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Zihang Dai , Zhilin Yang , Yiming Yang +3 more

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural archit...

2019 3018 citations

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Jinhyuk Lee , Wonjin Yoon , Sungdong Kim +4 more

Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processi...

2019 Bioinformatics 6148 citations

Publication Info

Year: 2016
Type: preprint
Citations: 915
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Exploring the Limits of Language Modeling

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

915

OpenAlex

Cite This

APA Style

                            
                                    Rafał Józefowicz, 
                                
                                    Oriol Vinyals, 
                                
                                    Mike Schuster
                                
                                et al.
                            
                            (2016). 
                            Exploring the Limits of Language Modeling. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1602.02410

Identifiers

DOI: 10.48550/arxiv.1602.02410