The flagged story is Vercel’s claim that, after four weeks in production, an in-house software factory was authoring a quarter to a third of merged AI SDK pull requests and closing most issues. That matters because AI SDK is foundational TypeScript infrastructure—provider adapters, UI bindings, sandboxes, and coding-agent harnesses—and because public, measured agent-SDLC results are still rare.
Vercel’s 12 August 2026 post says the vercel/ai repo had grown to more than 1,000 open issues and almost 800 pull requests by late June, after 100-plus new issues a month. Maintainers argued that more interactive coding agents would not close the gap, because every change still competed for the same human reviewer. They built ai-sdk-factory instead: GitHub webhooks feed task-specific agents, and a person merges every change. A social summary compressed the result into “up to 35% of merged PRs,” “70% of issues in July,” and “open bugs down 25%.” Those figures need the primary post and GitHub artifacts, not the tweet.
What we checked
- What is the canonical Vercel article URL and methodology behind the AI SDK software-factory metrics? partly confirmed. The canonical source is Vercel’s 12 August 2026 engineering post “Building a software factory for AI SDK,” by Lars Grammel and Eric Dodds. It publishes charted results for four weeks in production but does not disclose denominators, labeling rules, or how “closed by the factory” is counted. [1]
- Has New Runtime already covered this exact four-week Vercel AI SDK factory result? refuted. No dedicated New Runtime analysis of this result was found. Its software-factory posts cover Warp, Augment, and Factory/Comarch, and its AI SDK 7 note treats the SDK as an agent runtime, not the August factory metrics. [15][16][17]
- The factory authors 25–35% of merged PRs, closed about 70% of July issues, and reduced open bugs by 25%. partly confirmed. The tweet’s “up to 35%” is the high end of the blog’s 25–35% weekly share of merges to main; “70% of issues in July” is lower than the blog’s lede (70–80%) and results section (over 75% of closed issues). Public GitHub confirms a live factory bot, but this session could not reconstruct those percentages, and the roughly 25% open-bug drop appears only in Vercel’s post. [1][10]
- Each factory step is an agent, and humans merge every change. confirmed. The production factory uses dedicated agents for classification, reproduction, fixes, review, backports, docs, and features, with a human merge. The Foreman template omits merge from the agent tool surface, example PRs were merged by Lars Grammel, and an independent GitHub API study found all 294 September factory PRs merged by people in an approved state. [1][7][3][9]
What we found
- The canonical write-up is Vercel’s 12 August 2026 engineering post by Lars Grammel and Eric Dodds. After “just over four weeks” in production, factory agents authored 25–35% of weekly merges to main, more than 50% of weekly v6 (and similarly v5) backport merges, and over 75% of issues closed in July. Open issues fell from a peak of 1,022 in late June to 844 by early August, and open bugs were “down roughly 25%.” The post includes charts but no methodology appendix. [1]
- The blog’s worked example matches public GitHub. Community issue 17898 (blocked-domain support in OpenAI web search, opened 24 July 2026) was classified as a Feature, analyzed with a type probe, implemented in PR 18033 by ai-sdk-factory[bot], given a structured risk review, and merged by lgrammel the same morning. The bot then opened v6 and v5 backports (18035 clean, 18036 marked “fixed merge”), which lgrammel also merged. The change shipped in package releases including ai 7.0.41. [2][3][4][5][1]
- Architecture is one agent per task, not one mega-agent with skills. Agents run in isolated Vercel Sandboxes with task-scoped secrets and a network shield, because public issues, PRs, comments, and links are treated as attacker-controlled. The cloud path uses Vercel Functions, Queues, Blob, Sandbox, and Neon Postgres. Runs end as success, flawed, blocked, or manual; only success ships, and the rest feed prompts, evals, or environment fixes. [1]
- Vercel productized the pattern as eve software factories. Official docs describe intake, investigation, implementation, review, then a human decision. The Foreman template uses four stations (classifier, analyst, implementer, reviewer on a different model vendor), stops unattended runs at a draft PR, keeps merge off the tool surface, and injects git credentials at the sandbox firewall. Vercel Academy points to the AI SDK factory as the production-scale example. The live factory has more agent types than the four-station template. [6][7][8]
- Independent reporting supports the human-merge claim after the four-week window. ROBOCO’s 10 October 2026 GitHub API review found all 294 September factory PRs on vercel/ai merged by people in an APPROVED state, with a sample of 20 approvers all human and merge concentration of 63% of those 294. Factory-authored merged PRs were still landing in mid-to-late October 2026. [9][10][11]
- The factory changed maintainer workflow, not only throughput. On 14 July 2026, maintainers announced that the factory would take on issue review, triage, reproduction, and bugfix PRs, and they later denied co-authorship for agent-only “audit bugs” while asking contributors to submit reusable audit prompts. Open issues on 23 October 2026 still carried factory-active labels. The 1,022-to-844 drop is about 17% of all open issues; the 25% figure is specifically open bugs and was not independently counted here. The core ai npm package showed 18,352,661 weekly downloads on 22 October 2026; Vercel’s August “over 20 million a week” claim is not reproduced from that single-package number. [12][13][14][1]
What it means
Builders should copy the control plane, not the percentages. The durable design is a queue of narrow agents, each with its own prompt, sandbox, and evals; an evidence packet that makes review cheaper; isolation against untrusted public-issue input; an independent reviewer that sees the diff rather than the implementer’s story; and merge authority kept off the agent tool surface. Vercel’s own lesson is that improving the factory—feeding flawed, blocked, and manual runs back into prompts and evals—becomes the job.
Do not treat 25–35% authorship as a transferable productivity law. AI SDK is a high-volume, convention-heavy public SDK with existing tests, a dedicated maintainer, and backport lines that used to be skipped. The reusable Foreman template is a four-station subset of that production system. Track reviewer time, revert rate, and merge concentration alongside authorship share, and assume public-repo intake is hostile.
Still open
- What denominators and labeling rules sit behind 25–35% of weekly main merges and “closed by the factory” for July issues?
- Was the roughly 25% open-bug drop causal, or partly classification, duplicates, and deferred features?
- How did median review time, revert rate, and post-merge defect rate compare for factory-authored versus human-authored PRs?
- Does the 25–35% main-branch authorship share still hold in October 2026, when factory volume includes many backports?