---
schema_version: "newruntime-agent-readable-v0.2"
type: "capability_delta"
stable_id: "capability_delta:executable-outcome-checks-turn-agent-work-into-training-signals"
slug: "executable-outcome-checks-turn-agent-work-into-training-signals"
title: "Executable outcome checks turn agent work into training signals"
description: "ThinkingBox uses executable checks of final database state for both agent benchmarks and reinforcement-learning rewards; paired with runtime traces and outcome evaluation, this makes completed work measurable beyond the model’s output."
retrieval_nugget: "ThinkingBox uses executable checks of final database state for both agent benchmarks and reinforcement-learning rewards; paired with runtime traces and outcome evaluation, this makes completed work measurable beyond the model’s output."
status: "published"
observed_at: "2026-10-26"
record_date: "2026-10-26"
date_kind: "observed_at"
before_capability: "Builders could evaluate agent responses, but the items do not describe final database state checks serving as both benchmarks and training rewards."
enabling_change: "ThinkingBox checks final database state with executable tests and uses them for benchmarking and reinforcement-learning rewards; runtime evaluation can also connect traces to outcomes."
after_capability: "Builders can score whether an agent achieved a task’s final state and use that result as an evaluation or training signal."
human_consequence: "Teams can assess agents by verified task outcomes rather than relying only on the plausibility of their responses."
verification_level: "source-inspected"
buildability_class: "B2"
bottleneck: "The checks must accurately encode the intended outcome; the items do not establish that this approach generalizes across tasks or systems."
related_shift: "verification-ownership"
related_posts: []
topics: ["github"]
source_urls: ["https://newruntime.com/signals/agent-quality-depends-on-closing-the-loop-between-execution-and-verifi-5d3319/","https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f/"]
featured: false
supporting_evidence: [{"title":"Agent quality depends on closing the loop between execution and verification","url":"https://newruntime.com/signals/agent-quality-depends-on-closing-the-loop-between-execution-and-verifi-5d3319/","kind":"signal","observed_at":"2026-10-26"},{"title":"Builders are turning completed outcomes into measurable training and evaluation signals","url":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f/","kind":"signal","observed_at":"2026-10-26"}]
routes: {"html":"https://newruntime.com/deltas/executable-outcome-checks-turn-agent-work-into-training-signals/","markdown":"https://newruntime.com/deltas/executable-outcome-checks-turn-agent-work-into-training-signals.md","json":"https://newruntime.com/deltas/executable-outcome-checks-turn-agent-work-into-training-signals.json"}
source_format: "knowledge-ledger"
published_at: "2026-11-02T06:04:01.021Z"
auto_created: true
editorial_provenance: {"desk":"knowledge-revision","provider":"codex","model":"gpt-6-luna","created_at":"2026-11-02T06:04:01.021Z","week_start":"2026-10-26","review":"auto-created from reviewed evidence; the owner may revise or retire it"}
---

# Executable outcome checks turn agent work into training signals

## Retrieval answer

ThinkingBox uses executable checks of final database state for both agent benchmarks and reinforcement-learning rewards; paired with runtime traces and outcome evaluation, this makes completed work measurable beyond the model’s output.

## Capability change

- Before: Builders could evaluate agent responses, but the items do not describe final database state checks serving as both benchmarks and training rewards.
- Enabling change: ThinkingBox checks final database state with executable tests and uses them for benchmarking and reinforcement-learning rewards; runtime evaluation can also connect traces to outcomes.
- After: Builders can score whether an agent achieved a task’s final state and use that result as an evaluation or training signal.

Human consequence: Teams can assess agents by verified task outcomes rather than relying only on the plausibility of their responses.

Bottleneck: The checks must accurately encode the intended outcome; the items do not establish that this approach generalizes across tasks or systems.

Buildability: B2. Verification: source-inspected. Related shift: [verification-ownership](https://newruntime.com/shifts/verification-ownership/).

Why now: The items connect runtime traces and outcome evaluation with executable checks of final database state, including their use as benchmark criteria and reinforcement-learning rewards. This is a more concrete feedback signal than judging an agent’s text alone.

What to build: Add task-specific executable assertions over the resulting state, then record pass/fail outcomes alongside traces for evaluation and training. Watch whether those checks capture the actual task goal rather than merely a convenient proxy.

Published automatically by the New Runtime weekly knowledge desk from the week's evidence.

## Supporting evidence

- [Agent quality depends on closing the loop between execution and verification](https://newruntime.com/signals/agent-quality-depends-on-closing-the-loop-between-execution-and-verifi-5d3319/) (signal, 2026-10-26)
- [Builders are turning completed outcomes into measurable training and evaluation signals](https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f/) (signal, 2026-10-26)

## Provenance

Auto-created by the New Runtime weekly knowledge desk on 2026-11-02 from evidence reviewed in the week of 2026-10-26. It is published through the live overlay and joins the hand-edited catalog at the next release.