Trabalho em anais de eventoOpen Access

Skyt: Prompt Contracts for Software Repeatability in LLM-Assisted Development

RORIZ FILHO, HEITOR; JAZDI-MOTLAGH, NASSER; LUCENA JR., VICENTE

2026 • MSR '26: 23rd International Conference on Mining Software Repositories

Resumo

Large language models (LLMs) now author substantial amounts of code; however, identical prompts can yield different programs. For determinism-critical settings—systems, embedded, and firmware—this variability hinders verification, auditability, and maintenance. We make software repeatability a first-class, pinned-pipeline property and operationalize it via raw repeatabilityRraw (byte-identical outputs) and anchored repeatabilityRanchor, post (fraction of outputs matching the canonical anchor after bounded repair), with Δrescue ≔ Ranchor, post − Ranchor, pre, coverage, and confidence intervals. For example, if a toolchain update reorders imports or formatting so that bytewise matches scatter, Rraw may drop, while Ranchor remains high because non-canonical generations can be repaired to the recorded canon via a bounded repair loop (a fixed maximum of k edit attempts) that is monotonic in canon distance—without making the underlying model deterministic—which is what Continuous Integration (CI) and release engineering require in practice. We present Skyt, a contract-driven middleware that pins the environment, validates outputs against a logical schema, and computes repeatability distributions. We (i) formalize contracts and canon signatures; (ii) define repeatability metrics and an evaluation protocol across temperatures; and (iii) report baselines on fifteen tasks (twelve base algorithmic tasks plus three strict-compliance variants) spanning numeric, string, data structure, and sorting algorithms across three model families (GPT-4o-mini, GPT-4o, Claude Sonnet 4.5) with n = 20 runs per configuration (4,500 total samples). Due to space constraints, we present detailed results for three representative tasks (Binary-Search, Balanced-Brackets, Slugify). We also include a small proof-of-concept repair that applies a bounded, schema-constrained patch loop; per-run Rrepair@1 and monotonicity M values are recorded in the replication package. Results indicate that schema-bound contracts substantially tighten repeatability distributions under stochastic decoding, with Δrescue reaching up to 1.00 (100% improvement) on strict-compliance variants and up to 0.95 on base tasks.

Autoria