Beyond accuracy: an educational benchmarking study of task fragility and reasoning stability of large language models on dermatology board-style questions

Items were classified as Easy, Medium, or Hard and as Positive or Negative stems. The primary outcome was Correct Response (1/0) across 2,176 model–item–method observations. A generalized linear mixed-effects model with random intercepts for Question ID estimated the effects of model, method, difficulty, and polarity, including interactions; Holm adjustment controlled for multiple comparisons.</p> <p>Overall accuracy was 52%.

Using ChatGPT as the reference, all models showed significantly lower odds of correctness (Claude OR, 0.45; Gemini OR, 0.56; DeepSeek OR, 0.54; all <i>p</i> < 0.001). Task method had the largest effect, with marked reductions from Method 1 to Methods 2–4 (Method 4 OR, 0.026; <i>p</i> < 0.001). Negative stems lowered accuracy (OR, 0.57; <i>p</i> < 0.001).

Method × Difficulty, Method × Polarity, and Difficulty × Polarity interactions were significant, as was the three-way interaction (<i>p</i> = 0.006), with the steepest losses for medium-difficulty negative items in Methods 2 and 4.</p> <p>Overall, LLM performance exhibited sensitivity to task design, difficulty, and linguistic framing within this educational assessment framework. These findings suggest task fragility and warrant caution in unsupervised educational applications.

Practically, this highlights the value of format-aware evaluation when considering LLMs for medical examination preparation or automated question generation, helping to align expectations with their observed educational capabilities.</p>