Text · dataset · 2022
NeQA - Can Large Language Models Understand Negation in Multi-choice Questions?
Listed in Hugging Face Datasets
NeQA: Can Large Language Models Understand Negation in Multi-choice Questions?
Description
(Zhengping Zhou and Yuhui Zhang) General description This task takes an existing multiple-choice dataset and negates a part of each question to see if language models are sensitive to negation. The authors find that smaller language models display approximately random performance whereas the performance of larger models become significantly worse than random.
Language models failing to follow… See the full description on the dataset page: huggingface.co/datasets/inverse-scaling/NeQA.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/inverse-scaling/NeQA ↗
landing page · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/inverse-scaling/NeQA ↗
metadata API · from Hugging Face
Topics
- Stated by source
- multiple choice · question answering · tabular · text · zero shot classification
- From keywords
- Computer Science & AI
- Inferred from text
- Artificial intelligence 71%
Provenance · 1 source records, 14 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | inverse-scaling/NeQA | 7 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].anzsrc:group:4602 | enrichment · Hugging Face | taxonomy-embedding@1.1.0 | title+keywords+description (71%) |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:tabular | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:text | source · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[task].hf_task:multiple-choice | source · Hugging Face | connector:huggingface@1.0.0 | /tags[task_categories:*] |
| concepts[task].hf_task:question-answering | source · Hugging Face | connector:huggingface@1.0.0 | /tags[task_categories:*] |
| concepts[task].hf_task:zero-shot-classification | source · Hugging Face | connector:huggingface@1.0.0 | /tags[task_categories:*] |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| license | source · Hugging Face | connector:huggingface@1.0.0 | /tags[license:*] |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |