---
type: "post"
id: "accepted-but-wrong-typical-input-gates-release-41-75c83c"
slug: "accepted-but-wrong-typical-input-gates-release-41-75c83c"
title: "Accepted but Wrong: typical-input gates release 41% of wrong LLM API-migration repairs"
description: "A preregistered study of 73 API and runtime migration tasks found that ordinary tests on typical inputs released 41% of incorrect LLM repairs."
retrieval_nugget: "A preregistered study of 73 API and runtime migration tasks found that ordinary tests on typical inputs released 41% of incorrect LLM repairs."
published_at: "2026-11-07"
updated_at: "2026-11-07"
record_date: "2026-11-07"
date_kind: "published_at"
topics: ["verification","coding-agents"]
entities: ["Xiaobo Chen","Genliang Zhu","Ziyuan Wang","Chu Wang","CPython","Node.js","SQLite","GPT-5.6-luna"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_angle"
what_changed: "The study shifts attention from general code-review correctness to silent semantic failures in LLM-written API-migration repairs that pass typical-input tests."
prior_refs: ["https://newruntime.com/posts/a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb/"]
source_url: "https://arxiv.org/abs/2611.08269"
source_urls: ["https://arxiv.org/abs/2611.08269"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:accepted-but-wrong-typical-input-gates-release-41-75c83c"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/accepted-but-wrong-typical-input-gates-release-41-75c83c/","markdown":"https://newruntime.com/posts/accepted-but-wrong-typical-input-gates-release-41-75c83c.md","json":"https://newruntime.com/posts/accepted-but-wrong-typical-input-gates-release-41-75c83c.json"}
markdown_url: "https://newruntime.com/posts/accepted-but-wrong-typical-input-gates-release-41-75c83c.md"
---

# Accepted but Wrong: typical-input gates release 41% of wrong LLM API-migration repairs

## Retrieval answer

A preregistered study of 73 API and runtime migration tasks found that ordinary tests on typical inputs released 41% of incorrect LLM repairs.

## Summary
A preregistered study of 73 API and runtime migration tasks found that ordinary tests on typical inputs released 41% of incorrect LLM repairs. One edge input derived from a release note cut that share to 6.5% in the main repair space. For teams using AI to update code, the finding makes a practical case for testing changed behavior, not just whether a patch runs.

## What happened
In a paper submitted to arXiv on November 5, 2026, Xiaobo Chen, Genliang Zhu, Ziyuan Wang, and Chu Wang study silent failures in LLM-assisted code repairs after runtime or API migrations. The authors cover 73 tasks involving 60 documented changes across CPython, Node.js, and SQLite. Rather than assess patches only against ordinary examples, they execute candidate repairs on the real runtimes and check them against withheld contracts. The problem they target is a repair that appears successful because it runs cleanly, but returns a result that violates the intended behavior.

The paper reports that, across the complete repair space, an all-assertions gate using typical inputs released 0.410 of wrong repairs. The authors then add one edge input written from a release note. That gate released 0.065 of wrong repairs, and 0.061 on 15 unseen clusters. Three inputs from a blind GPT-5.6-luna writer, which had not seen the program, yielded 0.070. These are proportions of incorrect repairs released under the reported gates, not the share of all attempted repairs that were incorrect.

The authors also describe two frozen stages that expanded the repair menu: one to 1,491 fragments, and another adding 233 hard distractors designed to match legacy output on typical inputs. On those harder programs, the edge-input and frontier-model-writer approaches passed their preregistered tests, but still allowed 0.168 and 0.153 of hard wrong programs through, respectively. The paper says that across four model families, the edge input reduced wrong releases per attempt in all three open-weight families on all three menus. One example moved from 0.233 to 0.057, without lowering correct completion. The authors report that self-tests helped less.

A blind AI audit flagged 34 contracts as stricter than their requirements. Removing the disputed checks changed zero verdicts, although disputes over class labels remain open. The abstract says no model-written code was executed and that artifacts are available from the authors on request. This is a study of benchmarked repair tasks and release gates, not evidence that the same rates apply to every production codebase or migration workflow.

## What the archive adds
[New Runtime's earlier coverage](https://newruntime.com/posts/a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb/) on October 17, 2026, reported that the leading model in a separate code-quality benchmark passed every blocking check in 57.7% of attempts. The new paper is a distinct study and adds a specific test-design result: a release-note edge input can sharply reduce incorrect migration repairs that pass typical-input gates.

## Trend signal
This supports [verification-bandwidth-is-the-scarce-resource](https://newruntime.com/patterns/verification-bandwidth-is-the-scarce-resource/): as generated code becomes easier to produce, the hard part is designing checks that reliably distinguish a plausible patch from a correct one. The paper does not measure team review capacity, but its results show how the quality of a small test suite can affect what gets through a release gate.

## Why now
The study concerns a workflow that becomes especially consequential when a documented API or runtime change triggers automated repairs: migration work can produce code that compiles and passes familiar examples while missing changed behavior. The authors’ result makes release notes a timely input to testing, rather than merely documentation to read after a patch has been generated.

## Why it matters
A migration patch that passes existing tests can still be wrong in the behavior that the migration was meant to preserve or change. That creates a dangerous kind of confidence: teams see a clean run and a green gate, then ship a silent semantic failure. The study suggests a relatively low-cost intervention for this particular workflow: translate a documented change into at least one edge case that exercises the new contract.

For builders, the useful lesson is not to replace human review with one generated test. It is to treat test design as part of the migration pipeline. Review the release note for behavior changes, add a targeted case that would fail under the old or incorrect behavior, and keep that case in the gate. The reported reduction is substantial, but the hard-distractor results also show that such checks do not eliminate failure. Teams should retain broader verification and avoid interpreting a passing test suite as proof that a repair is correct.

## Limits and uncertainty
The reported numbers come from the study’s task set, repair space, and gate definitions; they do not establish expected failure rates in arbitrary repositories or production migrations. The expanded hard-distractor results show that even stronger inputs still let some wrong programs pass. Contract strictness was also contested in an AI audit: dropping the flagged checks changed no verdicts, but class-label disputes remain open. The paper is an arXiv preprint, and its artifacts are available from the authors on request, so the reported results have not been independently reproduced here.

## Skill to build now
For your next API or runtime migration, turn each documented behavioral change into a test that distinguishes the new contract from the old behavior. Include at least one edge input, run it against the repaired code on the real target runtime, and keep it in the release gate. This is a concrete first step, not a guarantee that a single case catches every incorrect repair.

## Reader takeaway
A green test run on familiar inputs can certify that an LLM repair executes without showing that it preserves the right behavior. In this study, one edge case taken from a release note cut the share of incorrect migrations passing the gate from 41% to about 6%. Build migration checks around the changed contract, and treat a passing gate as evidence—not proof.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Verification](https://newruntime.com/topics/verification.md): Explore the verification topic hub.
- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)