DiagFlowBench

At every turn there is therefore a documented step the system can point to next. In an off-procedure conversation the operator at some point reports something the flowchart has no branch for, such as a symptom that was never documented, a fault in a component the procedure does not mention, or a question about an unrelated matter. The system should then recognise that the procedure does not cover the situation rather than pressing on with the next documented step.

The two kinds are released as 838 matched pairs (1,676 conversations in total). Within a pair, the cooperative conversation and its counterpart differ only in whether the operator leaves the procedure at some point. The dataset accompanies the paper DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue, accepted to the Industry Track of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026).

Generation and evaluation code, prompts and pinned model identifiers are not included in this deposit and are released under MIT at github.com/guille-gil/DiagFlowBench. Note that although the repository unpacks files compressed within the dataset.zip file, there is no need to check for unpacked files in the repository: you only need to download the dataset.zip compressed file, which contains everything else.