SQuAD: 100,000+ Questions for Machine Comprehension of Text

Pranav Rajpurkar; Jian Zhang; Konstantin Lopyrev; Percy Liang

doi:10.18653/v1/d16-1264

Abstract

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Keywords

Computer scienceComprehensionNatural language processingArtificial intelligenceInformation retrievalProgramming language

Related Publications

Know What You Don’t Know: Unanswerable Questions for SQuAD

Pranav Rajpurkar , Robin Jia , Percy Liang

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for ...

2018 Proceedings of the 56th Annual Meetin... 2087 citations

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski , Jennimaria Palomaki , Olivia Redfield +15 more

We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator...

2019 Transactions of the Association for C... 1830 citations

Large language models encode clinical knowledge

Karan Singhal , Shekoofeh Azizi , Tao Tu +29 more

Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of mode...

2023 Nature 2248 citations

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Mike Lewis , Yinhan Liu , Naman Goyal +5 more

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning...

2020 1222 citations

Deep Learning--based Text Classification

Shervin Minaee , Nal Kalchbrenner , Erik Cambria +3 more

Deep learning--based models have surpassed classical machine learning--based approaches in various text classification tasks, including sentiment analysis, news categorization, ...

2021 ACM Computing Surveys 1298 citations

Publication Info

Year: 2016
Type: article
Pages: 2383-2392
Citations: 6054
Access: Closed

External Links

Download PDF (Free) View on DOI.org arXiv Semantic Scholar

Social Impact

Altmetric

SQuAD: 100,000+ Questions for Machine Comprehension of Text

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

6054

OpenAlex

1705

Influential

2390

CrossRef

Cite This

APA Style

                            
                                    Pranav Rajpurkar, 
                                
                                    Jian Zhang, 
                                
                                    Konstantin Lopyrev
                                
                                et al.
                            
                            (2016). 
                            SQuAD: 100,000+ Questions for Machine Comprehension of Text. 
                            Proceedings of the 2016 Conference on Empirical Methods in Natural
          Language Processing
                            
                            , 2383-2392.
                            https://doi.org/10.18653/v1/d16-1264

Identifiers

DOI: 10.18653/v1/d16-1264
arXiv: 1606.05250

Data Quality

Data completeness: 84%