Data · dataset · 2023
SAM meets Gaze: Passive Eye Tracking for Prompt-based Instance Segmentation
Listed in datastore by Universität Münster
The annotation of large new datasets for machine learning is a very time-consuming and expensive process.
Description
This is particularly true for pixel-accurate labelling of e.g. segmentation masks. Prompt-based methods have been developed to accelerate this label generation process by allowing the model to incorporate additional clues from other sources such as humans.
The recently published Segment Anything foundation model (SAM) extends this approach by providing a flexible framework with a model that was trained on more than 1 billion segmentation masks, while also being able to exploit explicit user input. In this paper, we explore the usage of a passive eye tracking system to collect gaze data during unconstrained image inspections which we integrate as a novel prompt input for SAM.
Read the rest (1 more)
We evaluated our method on the original SAM model and finetuned the prompt encoder and mask decoder for different gaze-based inputs, namely fixation points, blurred gaze maps and multiple heatmap variants. Our results indicate that the acquisition of gaze data is faster than other prompt-based approaches while the segmentation performance stays comparable to the state-of-the-art performance of SAM.
Links
Where it is published
- datastore by Universität Münster record datastore.uni-muenster.de/records/r1p0d-dtn64 ↗
landing page · from datastore uni muenster de
- DOI doi.org/10.17879/08958601057 ↗
DOI / persistent id · from datastore uni muenster de
Catalogue records · 1
- InvenioRDM API datastore.uni-muenster.de/api/records/r1p0d-dtn64 ↗
metadata API · from datastore uni muenster de
Topics
- Stated by source
- Deep Learning · Eye tracking · Fixations · Gaze Data · Gaze Prompts · Image Annotations · Instance Segmentation · Segment Anything Model (SAM)
- From keywords
- Computer Science & AI · Deep learning
- Inferred from text
- Image 75%
Provenance · 1 source records, 17 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| datastore by Universität Münster | r1p0d-dtn64 | 2 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[field].anzsrc:field:461103 | mapping · datastore uni muenster de | vocabulary-mapper@1.0.0 | keywords['Deep Learning'] |
| concepts[field].local:field:computer-science-ai | mapping · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[modality].local:modality:image | enrichment · datastore uni muenster de | keyword-concept-rules@1.0.0 | title+description (75%) |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:deep-learning | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:eye-tracking | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:fixations | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:gaze-data | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:gaze-prompts | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:image-annotations | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:instance-segmentation | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| concepts[topic].invenio_subject:datastore_uni_muenster_de:segment-anything-model-sam | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| description | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | /metadata/description |
| license | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| publication_date | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | |
| title | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 | /metadata/title |
| updated_date | source · datastore uni muenster de | connector:datastore_uni_muenster_de@1.0.0 |