Data Sheet 1_Language-dependent variation in observed mechanistic performance of web-enabled large language models across myeloid–mucosal immune contexts: a blinded expert evaluation.pdf

Each standardized value represents performance relative to an individual reviewer’s own scoring distribution rather than an absolute cross-language MFS difference.</p>Results<p>All 480 responses received two ratings (960 evaluations). Inter-rater agreement was moderate-to-good (Krippendorff α=0.709; Lin CCC = 0.709). Overall model performance differed substantially (F(4,23)=95.80, P<.001).

Kimi K3 (mean MFS 20.65) and ChatGPT 5.6 Sol (20.43) outperformed DeepSeek V4 Flash 0731, ChatGPT 5.6 Luna, and Xiaomi V2.5 Pro Ultra after Holm correction (all P<.001), while differing nonsignificantly from each other (P = .182). Reviewer-standardized scores showed a strong model-by-language interaction in observed relative performance (F(12,191)=44.21, P<.001), whereas no statistically significant model-by-complexity interaction was detected in the present question set (P = .328).</p>Conclusion<p>Mechanistic fidelity in the evaluated web-enabled systems was strongly model dependent, while observed relative performance varied across language panels.

Because response language was not separable from panel composition and could also influence translation and web retrieval, these findings do not establish an intrinsic causal effect of language on underlying model reasoning. Multilingual biomedical evaluation should therefore test long-form mechanistic explanations directly in the language and deployment setting of intended use rather than infer equivalence from English performance alone.</p>