Data Sheet 1_AI-augmented ACT-R modeling in real interfaces: automating visual perception, knowledge acquisition, and motor misexecution.pdf

A spreadsheet knowledge-acquisition analysis showed that prompting the LLMs with the instruction and UIA grounding can produce the intended action and live visual-object names for every scannable target; the file-name input was the sole exception because it has no stable UIA label. A six-task Windows analysis then tested the feasibility of EVisiTor beyond Excel. Broad instructions often resulted in alternative valid methods: exact action agreement ranged from 42.5%–63.6% without UIA and 18.8%–70.6% with UIA, although 41 of 42 generated parses in each condition were functionally sufficient.

Detailed guides raised exact agreement to 70.0%–85.0% without UIA and 76.2%–90.0% with UIA, and validated chunks drove the task-independent model through all six tasks in live Windows applications.</p>Discussion<p>Together, the results demonstrate EVisiTor's potential for automating interface perception and instruction-based knowledge acquisition for a symbolic cognitive model, while identifying instruction specificity and individual variability as targets for future work.</p>