Supplementary file 1_The effect of syntactic and semantic information on word grounding through visual perception.pdf

This study quantitatively evaluates how syntactic and semantic information affects word-grounding performance.</p>Methods<p>We evaluated five configurations within a generative multimodal Bayesian framework. The baseline grounds words using perceptual cues, while the additional configurations independently incorporate word-position indices, part-of-speech tags, FastText embeddings, and transformer-based contextual embeddings.

The models were evaluated using a CLEVR-derived 3D scene dataset and synonym-substituted descriptions.</p>Results<p>Linguistic information improved grounding accuracy across the Color, Geometry, and Spatial Relation modalities. Contextual embeddings produced a 53.8% absolute improvement in object-grounding accuracy over the baseline. Only models incorporating semantic embeddings generalized effectively to synonym-substituted descriptions without retraining.</p>Discussion<p>These findings demonstrate the contribution of linguistic structure and semantic representations to perceptual alignment and provide empirical foundations for developing language-grounding systems.</p>