Data · dataset · 2022
Modelling word learning and recognition using visually grounded speech
Listed in DANS Data Station Social Sciences and Humanities
A set of recorded isolated nouns, verbs and image annotations used for testing the word recognition performance of our speech2image model.
Description
We trained a word recognition model on a set of images and utterances. The model should learn to recognise words without ever having seen written transcripts.
The word recognition performance is measured as the number of retrieved images out of 10 displaying the correct visual referent. We furthermore collected new ground truth object and action annotations for the Flickr8k test images for this purposes. This consists of 1000 images, all annotated for the presence of the 50 actions and objects corresponding to the test verbs and nouns.
Read the rest (2 more)
In order to test the word recognition performance we took the 50 most common nouns and 50 most common verbs in the training data, confirmed that there were at least 10 images in our test image data that displayed these actions and objects. These nouns and verbs where recorded in singular and plural form (nouns) and in root, third person and progressive form (verbs). We furthermore annotated 1000 images from the Flickr8k test set for the presence of these nouns and verbs.
These annotations are included in .CSV format
Links
Where it is published
- Dataverse dataset page ssh.datastations.nl/dataset.xhtml?persistentId=doi%3A10.17026%2FDANS-22N-XH47 ↗
landing page · from ssh datastations nl
- DOI doi.org/10.17026/dans-22n-xh47 ↗
DOI / persistent id · from ssh datastations nl
Catalogue records · 1
- Dataverse API ssh.datastations.nl/api/datasets/:persistentId/?persistentId=doi%3A10.17026%2FDANS… ↗
metadata API · from ssh datastations nl
Topics
- Stated by source
- Arts and Humanities · Computer and Information Science
- From keywords
- Computer Science & AI · Humanities · Social Science · Speech recognition
Provenance · 1 source records, 14 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| DANS Data Station Social Sciences and Humanities | doi:10.17026/DANS-22N-XH47 | 4 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| concepts[field].anzsrc:field:460212 | mapping · ssh datastations nl | vocabulary-mapper@1.0.0 | keywords['speech recognition'] |
| concepts[field].dataverse_subject:arts-and-humanities | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /subjects |
| concepts[field].dataverse_subject:computer-and-information-science | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /subjects |
| concepts[field].local:field:computer-science-ai | mapping · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /subjects |
| concepts[field].local:field:humanities | mapping · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /subjects |
| concepts[field].local:field:social-science | mapping · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /subjects |
| concepts[modality].local:modality:audio | enrichment · ssh datastations nl | keyword-concept-rules@1.0.0 | title+description (65%) |
| concepts[modality].local:modality:image | enrichment · ssh datastations nl | keyword-concept-rules@1.0.0 | title+description (75%) |
| created_date | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | |
| description | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /description |
| publication_date | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | |
| title | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | /name |
| updated_date | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 | |
| version_label | source · ssh datastations nl | connector:ssh_datastations_nl@1.0.0 |