Field noteNov 20, 20266 min read1 source

CodeRabbit Verify: Test agent-written code in a real running app before it ships.

In one minute

CodeRabbit has opened Verify in public beta, adding tests of running applications to the review workflow for AI-generated code. It builds PR-specific scenarios, supports reusable regression journeys, and preserves evidence from failures. For teams shipping agent-written changes, the release offers a way to check user-visible behavior alongside code-level tests.

What changed: CodeRabbit has launched Verify in public beta, adding tests of running applications and user workflows to catch functional bugs in agent-written code.

What happened

CodeRabbit announced Verify on November 18, 2026. The company describes it as an application-testing product that checks behavior changed by a pull request or across a repository. For a PR, Verify builds test scenarios from its commits, relevant source code, tests and documentation. If CodeRabbit has reviewed the latest commit, that review context can also inform the test plan. The expected results are based on intended behavior, documentation and existing test expectations; the new implementation does not get to define its own correctness. Teams can inspect and refine scenarios before a run.

The announcement illustrates the difference between a visible success and a correct outcome. In its coupon example, a checkout flow shows a $100 order reduced to $80, but the application records a simulated $100 charge. Verify identifies the mismatch. CodeRabbit says results are posted in the PR with run summaries and links to evidence and recordings. When new commits arrive, Verify can update the scenario plan and rerun checks that need fresh validation; earlier results remain associated with the commit they tested. Findings can be handed to CodeRabbit Agent for a fix or exported with reproduction context for another coding agent.

For regression testing, Verify inspects a repository and proposes user journeys in plain English. Each describes a user, goal, steps and outcomes that should hold. Teams can discuss and revise proposals, then approve them into a reusable journey library. Selected journeys can run on demand, be grouped with tags, or be scheduled against a branch and test environment. Scheduled runs can create issues in connected Jira or Linear trackers for new bugs and testing blockers; a later run that verifies a fix can close the finding and update its linked issue.

Verify provides a live browser view and retains recordings, screenshots and logs for investigating failures. It distinguishes an application bug from a blocker such as an application that could not start or a required check that could not be completed. For environment setup, CodeRabbit Agent inspects project requirements, prepares a sandbox, installs dependencies, configures supporting services or emulators, starts the application and checks readiness. Testing happens in CodeRabbit's sandbox, not production. The product saves reusable setup and startup instructions and a snapshot for later runs. During public beta, PR test runs, full-repository journey runs and fixes with CodeRabbit Agent are free. The suggested starting point is commenting @coderabbitai verify on a PR; an unconfigured repository receives a setup link in CodeRabbit's reply.

What the archive adds

New Runtime's earlier coverage of ThruLine 0.3.0, dated November 7, 2026, described a Mac app checking coding agents' completion claims against local run logs and whether relevant code was retested. Verify is a new event on the same broader problem—checking agent work—but adds browser-based application behavior scenarios, reusable regression journeys and handoff of evidence for repair. The overlap makes this a new fact on a known thread, not a restatement.

Trend signal

This does not yet fit a pattern New Runtime tracks. The release nevertheless targets a familiar operational constraint: teams need a practical way to assess whether an agent-generated change works for users, rather than merely whether it compiles or passes a narrow test suite. CodeRabbit's proposed journeys and retained execution evidence could distribute some of that checking work, but the announcement does not measure reductions in QA time, defect rates or review load.

Why now

CodeRabbit frames the release around a growing volume of code from coding agents and the impracticality of manually testing every change. The announcement argues that teams either slow releases while QA works through a backlog or ship changes without adequate testing. Verify is positioned as a response to that tension, though the company provides no adoption or volume data quantifying it.

Connections to other events

New Runtime's earlier coverage of ThruLine on November 7, 2026, also examined checks on coding-agent completion claims. ThruLine used local run logs and retest status; Verify describes end-to-end journeys in a running application, with recordings and screenshots attached to findings. Both make verification an explicit step after an agent reports work as done, but they inspect different evidence and operate at different points in the workflow.

Why it matters

A passing build or a plausible code diff does not establish that a customer can complete a workflow. Verify's coupon example makes that distinction concrete: the displayed total was right while the simulated charge was wrong. For teams using coding agents, a way to exercise application behavior can expose failures that unit tests or a cursory review may not catch, while supplying an agent with observations and reproduction context instead of a vague bug report.

The reusable journey library also points to an operational choice: teams can treat critical user flows as maintained checks, rather than recreate test plans for each change. But this shifts work rather than removing it. Someone still needs to judge whether proposed journeys represent the product, approve expected outcomes, choose environments and review failures. The tool may make evidence cheaper to collect; it does not make correctness automatic. Teams should evaluate whether the journeys cover the behavior they actually promise and whether handoffs lead to fixes that pass independent checks.

Limits and uncertainty

CodeRabbit announced a public beta, but did not state a beta end date, post-beta pricing, supported application stacks, execution limits or service-level commitments. It describes the tests as running in its sandbox, but the announcement does not quantify environment setup time or explain the full security and data-handling model for repository code, test data and credentials. The company also gives no independent evaluation of detection accuracy, false positives, coverage or effect on release quality. The coupon scenario is illustrative and uses a simulated charge, not a real payment. Teams should establish how Verify behaves on their own applications before relying on it as a release gate.

Skill to build now

Pick one high-value user journey—checkout, account creation or another flow that must not break—and write down its expected outcome independently of the current implementation. Ask Verify to propose or run a scenario against a PR, inspect the plan before execution, and compare the retained evidence with the expectation. If the flow depends on a test service or emulator, check what setup instructions the sandbox saves so a later run can reproduce the environment.

Reader takeaway

Verify makes application-level checks and evidence part of the coding-agent PR loop: scenarios can be reviewed before running, regression journeys can be reused, and failures can travel to a repair agent with context. That is a useful capability to test, not proof that an agent-written change is safe to ship. Keep expected behavior and human review separate from the implementation being tested.

Evidence / sources

  1. 1coderabbit.ai/blog/introducing-coderabbit-verify
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicCode ReviewExplore the code-review topic hub.
  2. 02topicVerificationExplore the verification topic hub.
  3. 03topicCoding AgentsExplore the coding-agents topic hub.
  4. 04archiveField NotesOpen the latest editorial analysis.
  5. 05sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract