Replication package for "Vibe Testing for REST APIs: A Context-Taxonomic Benchmark of LLM-Generated Test Suites"

Vibe Testing is an execution-guided workflow in which a large language model (LLM) synthesizes, runs, diagnoses, and repairs executable REST API test suites in a controlled feedback loop. The study quantifies how the type and amount of supplied context shape both the validity and the defect-detection power of the generated tests, using a five-level context taxonomy (L0: OpenAPI specification only; L1: + business documentation; L2: + source code; L3: + database schema; L4: + reference tests) on a purpose-built, contamination-controlled 50-endpoint Bookstore API.

Package contents (folders 01-10): - 01_framework: the Vibe Testing pipeline (context assembly, planning, generation, validation, metrics, repair) and all experiment configurations. - 02_target_api: the 50-endpoint Bookstore REST API under test (FastAPI + SQLite). - 03_context_artifacts: the five context-level input artifacts. - 04_context_axis: balanced experiment (5 cloud LLMs x 5 levels x 5 runs = 125 runs) with per-run metrics and all 125 generated suites. - 05_temperature_axis: temperature sweeps (7 models x 6 temperatures at L1). - 06_openweight_context_matrix: auxiliary open-weight model results. - 07_mutation_testing: 13 hand-curated crud.py mutants and per-suite kill results for all 125 cloud suites. - 08_instruction_following: instruction-following-compliance probe results. - 09_statistics: nonparametric analyses (Friedman, Kendall's W, Wilcoxon, Mann-Whitney, Cliff's delta, bootstrap CIs). - 10_figures: the eight published figures and their generator scripts.