Abstract

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations to help adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement) as of February 25, 2019 on the latest GLUE test set. We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. Our code and pre-trained models will be made publicly available.

Keywords

Computer scienceDomain adaptationTransformerArtificial intelligenceBenchmark (surveying)Natural language understandingTask (project management)Artificial neural networkRegularization (linguistics)Deep neural networksDeep learningLabeled dataTask analysisLanguage modelNatural language processingTest setMachine learningNatural language

Affiliated Institutions

Related Publications

Publication Info

Year
2019
Type
preprint
Citations
1027
Access
Closed

External Links

View on DOI.org

Social Impact

Altmetric
PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1027
OpenAlex

Cite This

Xiaodong Liu, Pengcheng He, Weizhu Chen et al. (2019). Multi-Task Deep Neural Networks for Natural Language Understanding. . https://doi.org/10.18653/v1/p19-1441

Identifiers

DOI
10.18653/v1/p19-1441