---
id: "nr-builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f"
slug: "builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f"
type: "raw_signal"
title: "Builders are turning completed outcomes into measurable training and evaluation signals"
description: "ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes."
observed_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "observed_at"
novelty: "notable"
source_platform: "newruntime.com"
topics: ["github"]
entities: ["Factory","Agent Effectiveness","Factory Analytics","Droid","Local Signals","Jira","Linear","GitHub","Microsoft","Hugging Face"]
related_patterns: ["agent-economics-moves-to-completed-work","verification-bandwidth-is-the-scarce-resource","goal-scoped-loops-replace-manual-continuation"]
source_url: "https://factory.com/news/agent-effectiveness"
source_urls: ["https://factory.com/news/agent-effectiveness","https://huggingface.co/blog/microsoft/thinkingbox","https://huggingface.co/blog/microsoft/thinkingbox-training"]
why_it_matters: "ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes. These approaches evaluate whether work actually landed, rather than relying only on the agent’s account of what it did."
verification_level: "source-linked"
signal_type: "synthesis"
synthesis_of: [{"slug":"factory-says-agent-effectiveness-is-now-generally-available-e5d7b0","title":"Factory says Agent Effectiveness is now generally available in Factory Analytics for Enterprise organizations","url":"https://newruntime.com/posts/factory-says-agent-effectiveness-is-now-generally-available-e5d7b0/"},{"slug":"the-agent-said-it-was-done-the-database-3a02f5","title":"The Agent Said It Was Done. The Database Disagreed.","url":"https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5/"},{"slug":"turning-database-grading-into-rl-rewards-post-training-46f950","title":"Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox","url":"https://newruntime.com/posts/turning-database-grading-into-rl-rewards-post-training-46f950/"}]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "signal:builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f"
retrieval_nugget: "ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes."
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f/","markdown":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f.md","json":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f.json"}
---

# Builders are turning completed outcomes into measurable training and evaluation signals

## Retrieval answer

ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes.

ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes. These approaches evaluate whether work actually landed, rather than relying only on the agent’s account of what it did.

From the Field Notes:

- [Factory says Agent Effectiveness is now generally available in Factory Analytics for Enterprise organizations](https://newruntime.com/posts/factory-says-agent-effectiveness-is-now-generally-available-e5d7b0/) (2026-10-24)
- [The Agent Said It Was Done. The Database Disagreed.](https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5/) (2026-10-24)
- [Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox](https://newruntime.com/posts/turning-database-grading-into-rl-rewards-post-training-46f950/) (2026-10-23)

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub.
- collection: [Signals](https://newruntime.com/signals.json): Open the signals and the raw-signal archive.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)