Pre-trained models for natural language processing: A survey

Xipeng Qiu; Tianxiang Sun; Yige Xu; Yunfan Shao; Ning Dai; Xuanjing Huang

doi:10.1007/s11431-020-1647-3

Keywords

Computer scienceNatural (archaeology)Natural language processingGeographyArchaeology

Affiliated Institutions

Fudan University CN

Related Publications

Learning to Prompt for Vision-Language Models

Kaiyang Zhou , Jingkang Yang , Chen Change Loy +1 more

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Differen...

2022 International Journal of Computer Vision 2040 citations

TinyBERT: Distilling BERT for Natural Language Understanding

Xiaoqi Jiao , Yichun Yin , Lifeng Shang +5 more

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usuall...

2020 Findings of the Association for Compu... 1491 citations

Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer

Colin Raffel , Noam Shazeer , Adam Roberts +6 more

Transfer learning, where a model is first pre-trained on a data-rich task\nbefore being fine-tuned on a downstream task, has emerged as a powerful\ntechnique in natural language...

2019 arXiv (Cornell University) 8299 citations

Supervised Learning of Universal Sentence Representations from Natural\n Language Inference Data

Alexis Conneau , Douwe Kiela , Holger Schwenk +2 more

Many modern NLP systems rely on word embeddings, previously trained in an\nunsupervised manner on large corpora, as base features. Efforts to obtain\nembeddings for larger chunk...

2017 arXiv (Cornell University) 2038 citations

Survey of Hallucination in Natural Language Generation

Ziwei Ji , Nayeon Lee , Rita Frieske +7 more

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-bas...

2022 ACM Computing Surveys 2392 citations

Publication Info

Year: 2020
Type: article
Volume: 63
Issue: 10
Pages: 1872-1897
Citations: 1400
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Pre-trained models for natural language processing: A survey

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1400

OpenAlex

Cite This

APA Style

                            
                                
                                    Xipeng Qiu, 
                                
                                    Tianxiang Sun, 
                                
                                    Yige Xu
                                
                                et al.
                            
                            (2020). 
                            Pre-trained models for natural language processing: A survey. 
                            Science China Technological Sciences
                            , 63
                            (10)
                            , 1872-1897.
                            https://doi.org/10.1007/s11431-020-1647-3
                        

Identifiers

DOI: 10.1007/s11431-020-1647-3