Experimental Materials and Dataset - "Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation"

The model writes the README from prior knowledge and whatever the provider retrieves at inference time.</li><li>Evaluation dimensions: structural completeness, ATRAK adherence, and correctness.</li><li>Evaluation protocols: two independent protocols, a human evaluator and an LLM judge (Claude Fable 5). The judge operates as a tool-using agent that installs each project in a throwaway environment and executes every documented snippet.</li><li>Baseline: README-AI, run under the same model and the same rubric.</li></ul><p dir="ltr"><br></p><p dir="ltr"><b>Contents</b></p><p><br></p><p dir="ltr">209 files, 8.1 MB, organised in pipeline order:</p><p><br></p><ul><li>1_inputs/: the twelve repositories with language, stars, popularity group and Core Functionality ground truth, plus the run manifest.</li><li>2_prompts/: the exact generation prompt, and the generic, tool-agnostic evaluation and comparison prompts (reusable beyond this study).</li><li>3_generated_readmes/: 48 generated files. 36 are from README-Gen (three independent runs per repository) and 12 are from README-AI.</li><li>4_evaluations/: the raw evaluation data.

It includes the human evaluator's worksheet (CSV and XLSX), 74 score CSVs, and 48 LLM-judge reasoning files. The reasoning files record, per rule, the commands executed, their output and the verdict.</li><li>5_comparison/: a secondary artifact demonstrating that the comparison workflow generalises. It produces no number used in the paper.</li><li>6_analysis/: reproduce.py, which recomputes every statistic, table and figure in the paper directly from 4_evaluations/, and its outputs.</li></ul><p dir="ltr"><br></p><p dir="ltr"><b>Reproducing</b></p><p><br></p><ul><li>Install 6_analysis/requirements.txt (pandas, scipy, matplotlib).</li><li>Run python 6_analysis/reproduce.py.</li></ul><p dir="ltr"><br></p><p dir="ltr">The script reads only 4_evaluations/.

No summary spreadsheet is consulted, so a reader can verify the aggregation rather than trust it. Deleting 6_analysis/outputs/ and re-running regenerates every table, figure and statistic identically. Both the human and the automated evaluations are rebuilt from the judges' original outputs.</p><p><br></p><p dir="ltr"><b>Documentation</b></p><p><br></p><ol><li>README.md: maps the folder layout and the file conventions.</li><li>METHODOLOGY.md: explains how each artifact was produced and states plainly what was not controlled:</li></ol><ul><li>the judge was run at its then-current version rather than a pinned snapshot;</li><li>ground truth is the released artifact and default branch, not a pinned commit;</li><li>no execution timeout was fixed.</li></ul><ol><li>CLAIMS.md: maps every quantitative claim in the paper to the artifact that produces it.</li></ol><p dir="ltr"><br></p><p dir="ltr"><b>Not included</b></p><p><br></p><ul><li>README-Gen's own source code (the CLI/HTTP application) is not part of this package. 2_prompts/generation_prompt.md is the complete record of what determines the tool's output.

Regenerating a README requires only that prompt, an OpenAI key and the pinned model.</li><li>The earlier, superseded round of automated scoring is also not included. 4_evaluations/ is the single source of truth.</li></ul><p></p>