{"id":"nr-builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f","slug":"builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f","type":"raw_signal","title":"Builders are turning completed outcomes into measurable training and evaluation signals","description":"ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes.","observed_at":"2026-10-24","record_date":"2026-10-24","date_kind":"observed_at","novelty":"notable","source_platform":"newruntime.com","topics":["github"],"entities":["Factory","Agent Effectiveness","Factory Analytics","Droid","Local Signals","Jira","Linear","GitHub","Microsoft","Hugging Face"],"related_patterns":["agent-economics-moves-to-completed-work","verification-bandwidth-is-the-scarce-resource","goal-scoped-loops-replace-manual-continuation"],"source_url":"https://factory.com/news/agent-effectiveness","source_urls":["https://factory.com/news/agent-effectiveness","https://huggingface.co/blog/microsoft/thinkingbox","https://huggingface.co/blog/microsoft/thinkingbox-training"],"why_it_matters":"ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes. These approaches evaluate whether work actually landed, rather than relying only on the agent’s account of what it did.","verification_level":"source-linked","signal_type":"synthesis","synthesis_of":[{"slug":"factory-says-agent-effectiveness-is-now-generally-available-e5d7b0","title":"Factory says Agent Effectiveness is now generally available in Factory Analytics for Enterprise organizations","url":"https://newruntime.com/posts/factory-says-agent-effectiveness-is-now-generally-available-e5d7b0/"},{"slug":"the-agent-said-it-was-done-the-database-3a02f5","title":"The Agent Said It Was Done. The Database Disagreed.","url":"https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5/"},{"slug":"turning-database-grading-into-rl-rewards-post-training-46f950","title":"Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox","url":"https://newruntime.com/posts/turning-database-grading-into-rl-rewards-post-training-46f950/"}],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f","retrieval_nugget":"ThinkingBox uses executable checks of final database state both to benchmark agents and to provide reinforcement-learning rewards, while Factory Analytics links agent sessions to delivery outcomes.","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f/","markdown":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f.md","json":"https://newruntime.com/signals/builders-are-turning-completed-outcomes-into-measurable-training-and-e-54937f.json"},"next_reads":[{"type":"topic","path":"/topics/github/","url":"https://newruntime.com/topics/github/","title":"Github","reason":"Explore the github topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/github.md","json_url":"https://newruntime.com/topics/github.json"},{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Signals","reason":"Open the signals and the raw-signal archive.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/sources/","url":"https://newruntime.com/sources/","title":"Sources","reason":"See the original sources this archive cites.","media_type":"text/html"}]}