Introspective Uncertainty Estimation for LLM-Based Code Generation

Static single-token probes perform best, reaching 0.90 AUROC and 0.96 F1 on LCB in the best settings, generally surpassing the thresholds of previously reported static probe baselines for IUE. More elaborate dynamic token-selection and sequence-modeling strategies yield no consistent gains. While generalization across tasks, domains, and token positions is feasible, setting-dependent degradation largely remains for real-world software projects.

At a fine granularity, line-level prediction in mixed-program settings is substantially harder than response-level estimation. However, in a conditional localization setup with known-incorrect programs, Top-K point-of-failure ranking remains effective, achieving a Top-3 hit rate of 81% in the best setting. Overall, the findings suggest that hidden states are a robust and informative resource for estimating functional code correctness and localizing faults, supporting a two-stage workflow that combines response-level risk screening with targeted line-level prioritization, and motivate further research on IUE for LLM-based code generation.</p><p dir="ltr">This archive contains multiple dataset contributions from the thesis:</p><p dir="ltr">• Comprehensive Code Dataset: We curate a dataset of code generations across two state-of-the-art code generation benchmarks, produced with four distinct open-weight LLMs, and assign response-level correctness labels using benchmark-provided test suites.</p><p dir="ltr">• High-Granularity Augmented Dataset: We augment one benchmark for two models with automatically repaired program versions of each incorrect program.

This augmented dataset provides fine-grained, token- and line-level labels, which are derived by computing diffs between the original and repaired code.</p>