KETI-NLP/kvqa

We hope that the KVQA dataset can simultaneously provide opportunities for the development of Korean VQA technology as well as creation of meaningful social value in Korean society. You can download KVQA dataset via this link. ## Evaluation We measure the model's accuracy by using answers collected from 10 different people for each question.

If the answer provided by a VQA model is equal to 3 or more answers from 10 annotators, it gets 100%; if less than 3, it gets a partial score proportionately. To be consistent with ‘human accuracies’, measured accuracies are averaged over all 10 choose 9 sets of human annotators. Please refer to VQA Evaluation which we follow. ## Usage ```python from datasets import load_dataset raw_datasets = load_dataset( "kvqa.py", "default", cache_dir="huggingface_datasets", data_dir="data", ignore_verifications=True, ) dataset_train = raw_datasets["train"] for item in dataset_train: print(item) exit() ``` ## Data statistics ### v1.0 (Jan. 2020) | | Overall (%) | Yes/no (%) | Number (%) | Etc (%) | Unanswerable (%) | |:------------|:---------------|:-------------|:-------------|:---------------|:-----------------| | # images | 100,445 (100) | 6,124 (6.10) | 9,332 (9.29) | 69,069 (68.76) | 15,920 (15.85) | | # questions | 100,445 (100) | 6,124 (6.10) | 9,332 (9.29) | 69,069 (68.76) | 15,920 (15.85) | | # answers | 1,004,450 (100)| 61,240 (6.10)| 93,320 (9.29)| 690,690 (68.76)| 159,200 (15.85) | ## Data ### Data field description | Name | Type | Description | |:---------------------------------|:---------|:---------------------------------------------------------------| | VQA | `[dict]` | `list` of `dict` holding VQA data | | +- image | `str` | filename of image | | +- source | `str` | data source `["kvqa" | "vizwiz"]` | | +- answers | `[dict]` | `list` of `dict` holding 10 answers | | +--- answer | `str` | answer in `string` | | +--- answer_confidence | `str` | `["yes" | "maybe" | "no"]` | | +- question | `str` | question about the image | | +- answerable | `int` | answerable? `[0 | 1]` | | +- answer_type | `str` | answer type `["number" | "yes/no" | "unanswerable" | "other"]` | ### Data example ```json [{ "image": "KVQA_190712_00143.jpg", "source": "kvqa", "answers": [{ "answer": "피아노", "answer_confidence": "yes" }, { "answer": "피아노", "answer_confidence": "yes" }, { "answer": "피아노 치고있다", "answer_confidence": "maybe" }, { "answer": "unanswerable", "answer_confidence": "maybe" }, { "answer": "게임", "answer_confidence": "maybe" }, { "answer": "피아노 앞에서 무언가를 보고 있음", "answer_confidence": "maybe" }, { "answer": "피아노치고있어", "answer_confidence": "maybe" }, { "answer": "피아노치고있어요", "answer_confidence": "maybe" }, { "answer": "피아노 연주", "answer_confidence": "maybe" }, { "answer": "피아노 치기", "answer_confidence": "yes" }], "question": "방에 있는 사람은 지금 뭘하고 있지?", "answerable": 1, "answer_type": "other" }, { "image": "VizWiz_train_000000008148.jpg", "source": "vizwiz", "answers": [{ "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "티비 리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "maybe" }, { "answer": "리모컨", "answer_confidence": "yes" }, { "answer": "리모컨", "answer_confidence": "yes" }], "question": "이것은 무엇인가요?", "answerable": 1, "answer_type": "other" } ] ```