What happened
Stripe’s announcement says its “Stripe evals” benchmark measures how coding agents build integrations from realistic developer goals. The scenarios cover one-time payments, subscriptions and Connect, Stripe’s marketplace and platform offering. Examples include hosted checkout, metered billing, per-seat subscriptions and splitting marketplace payments. A role-playing agent simulates a developer who begins with an incomplete request and follows up on unresolved needs, while avoiding coaching the coding agent on Stripe APIs.
Stripe tested 69 public cases—24 payments, 26 subscriptions and 19 Connect cases—across 14 model-and-harness pairs. The harnesses were Claude Code, Codex and OpenCode, with a mix of models. Each case ran twice with a fixed task and configuration, both with and without the Stripe agent plugin. That plugin combines Stripe-authored skills with tools served through an MCP server, including documentation search and an implementation planner. Runs took place headlessly in isolated workspaces, where agents could inspect repositories, write files, run commands and revise their work through multiple turns.
Across all 14 pairs, the plugin raised the mean criteria-success score by roughly 3 to 8 percentage points. The pooled score rose from 78.1% without the plugin to 82.6% with it. Stripe also reports a 96% reduction in web searches per evaluation, from 1.26 to 0.05, and a 63% reduction in page fetches. The plugin’s effect varied by model: for example, GPT-6.1 Sol with Codex moved from 85.6% to 89.6% on average, while Gemini 3.8 Flash with OpenCode rose from 64.8% to 72.7%. Stripe says outdated parameter references fell from 7% to 1.3% of API-call parameters.
The headline average does not mean that an integration met every important requirement. Stripe separately reports strict pass rates, requiring every MUST criterion in a case to pass. By this measure, the average fell from 78.1% without the plugin to 29.4%—the same values Stripe gives for the average score and strict pass rate in its summary, while its detailed table reports different pooled strict-pass figures for individual configurations. Stripe emphasizes that strict pass is sensitive to judging or data errors. A scenario-specific example illustrates the last-mile risk: an agent checks that a PaymentIntent succeeded but does not check for a later refund or dispute before fulfillment.
For grading, Stripe separates criteria into MUST requirements and SHOULD recommendations, and judges each criterion separately against both the session transcript and the resulting workspace. It used GPT-6.1 Sol as role-player and judge across evaluations, with a separate Opus judging probe that Stripe says moved scores in lockstep and did not show bias. Runs had a 45-minute timeout; timed-out errors were excluded from scoring. Stripe is publishing the benchmark and Forge evaluation harness and says it plans to add cases, models and harnesses over time.
What the archive adds
We have not covered this before; it opens a new thread.
Trend signal
This supports verification bandwidth as the scarce engineering resource: Stripe’s large gap between average criterion scores and all-MUST pass rates illustrates why a plausible implementation is not the same as a verified one. The benchmark does not establish that this gap generalizes beyond Stripe integrations, but it makes the verification distinction measurable.
Why it matters
Teams often report agent performance as a pass rate or average score, but those summaries can hide failures in requirements that are individually rare and collectively essential. A payment integration can look nearly correct while missing a refund, dispute, subscription edge case or marketplace-specific decision. Stripe’s split between MUST and SHOULD criteria offers a practical model: identify which failures invalidate the result, then report both partial progress and complete-case success.
The results also suggest that authoritative, current documentation can change how agents work, not just how well they score. Fewer web searches and outdated parameters indicate that a domain-specific plugin may reduce reliance on broad retrieval. But the score gains do not establish that the plugin is cost-effective in every environment: some model runs cost more with the plugin, and the benchmark measures Stripe scenarios rather than all software tasks. Builders should treat it as evidence for testing contextual tools against their own integration risks, not as a universal ranking of coding agents.
Limits and uncertainty
Stripe says the benchmark checks specification through observable behavior and artifacts, not runtime execution: a well-shaped but nonfunctional integration could receive full marks. It does not measure product recommendation quality or long-term maintenance as APIs change. OpenCode’s web searches could not be directly observed, and the source notes that strict pass rates are sensitive to data and judgment errors. Results reflect two repetitions per case and one role-player and judge model; they do not establish how stable the rankings are across more runs or other judging setups. Stripe plans future additions, so these are initial results rather than a settled comparison.
Skill to build now
For an agent workflow that generates API integrations, write a small scenario rubric before running the agent. Mark every requirement that would make the result unsafe or invalid as MUST; keep good-practice preferences as SHOULD. Then calculate both criterion-level success and the share of runs that pass every MUST. If a payment flow can trigger fulfillment, include a check for refund or dispute state—not just successful payment—and inspect the generated artifact as well as the conversation.
Reader takeaway
The useful lesson is not simply that Stripe’s plugin improved scores. It is that average correctness can conceal a much lower rate of complete integrations. When you evaluate an agent that writes production-facing integrations, measure the last mile explicitly: which runs satisfy every requirement that must not fail?