What is DREAM?
DREAM is a multiple-choice Dialogue-based REAding comprehension exaMination dataset. In contrast to existing reading comprehension datasets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding.
DREAM contains 10,197 multiple choice questions for 6,444 dialogues, collected from English-as-a-foreign-language examinations designed by human experts. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge.
Report Your Results
If you have new results, please send an email to dream@dataset.org with the link to your paper!
Leaderboard
| Report Time | Model | Accuracy |
|---|---|---|
|
Human Performance
Tencent & Cornell & UW & AI2 Sun et al., 2019 |
95.5 | |
| Apr 25, 2019 | BERT-Large (single model) https://github.com/nlpdata/mrc_bert_baseline | 66.8 |
| Apr 23, 2019 | BERT-Base (single model) https://github.com/nlpdata/mrc_bert_baseline | 63.2 |
| Feb 01, 2019 |
GBDT++ and FTLM++ (ensemble)
Tencent & Cornell & UW & AI2 Sun et al., 2019 |
59.5 |
| Feb 23, 2019 |
EER + FT (single model)
Tencent & TTIC & Cornell & UPenn Wang et al., 2019 |
57.7 |
| Feb 01, 2019 |
FTLM++ (single model)
Tencent & Cornell & UW & AI2 Sun et al., 2019 |
57.4 |
| Feb 01, 2019 |
Finetuned Transformer LM (single model) (*)
OpenAI Radford et al., 2018 |
55.5 |
| Feb 01, 2019 |
GBDT++ (single model)
Tencent & Cornell & UW & AI2 Sun et al., 2019 |
52.8 |
| Feb 01, 2019 |
DSW++ (single model)
Tencent & Cornell & UW & AI2 Sun et al., 2019 |
50.1 |
| Feb 01, 2019 |
Co-Matching (single model) (*)
Singapore Management University & IBM Research Wang et al., 2018 |
45.5 |
| Feb 01, 2019 |
Distance-Based Sliding Window (single model) (*)
Microsoft Research Richardson et al., 2013 |
44.6 |
| Feb 01, 2019 |
Sliding Window (single model) (*)
Microsoft Research Richardson et al., 2013 |
42.5 |
| Feb 01, 2019 |
Word Matching (single model) (*)
Microsoft Research Yih et al., 2013 |
42.0 |
| Feb 01, 2019 |
Gated-Attention Reader (single model) (*)
Carnegie Mellon University Dhingra et al., 2017 |
41.3 |
| Feb 01, 2019 |
Stanford Attentive Reader (single model) (*)
Stanford University Chen et al., 2016 |
39.8 |
*: Run and reported by Sun et al., 2019.