---
type: "post"
id: "the-agent-said-it-was-done-the-database-3a02f5"
slug: "the-agent-said-it-was-done-the-database-3a02f5"
title: "The Agent Said It Was Done. The Database Disagreed."
description: "Microsoft and Hugging Face have introduced ThinkingBox, a benchmark for AI agents that work through MCP-connected, stateful business systems."
retrieval_nugget: "Microsoft and Hugging Face have introduced ThinkingBox, a benchmark for AI agents that work through MCP-connected, stateful business systems."
published_at: "2026-10-23"
updated_at: "2026-10-23"
record_date: "2026-10-23"
date_kind: "published_at"
topics: ["claude"]
entities: ["Microsoft","Hugging Face","ThinkingBox","OpenEnv","Kimi-K3","Claude Opus 5"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource","agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Adds ThinkingBox’s Hugging Face benchmark release, which scores agents’ terminal database state and tests whether they can repeat success across twenty runs."
prior_refs: ["memory:dc3f2a17-b00a-8534-aa02-4392142eef4e","memory:cc535902-70fa-8437-ae74-4676ff99bf2e"]
source_url: "https://huggingface.co/blog/microsoft/thinkingbox"
source_urls: ["https://huggingface.co/blog/microsoft/thinkingbox"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:the-agent-said-it-was-done-the-database-3a02f5"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5/","markdown":"https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5.md","json":"https://newruntime.com/posts/the-agent-said-it-was-done-the-database-3a02f5.json"}
---

# The Agent Said It Was Done. The Database Disagreed.

## Retrieval answer

Microsoft and Hugging Face have introduced ThinkingBox, a benchmark for AI agents that work through MCP-connected, stateful business systems.

Microsoft and Hugging Face have introduced ThinkingBox, a benchmark for AI agents that work through MCP-connected, stateful business systems. Rather than grading an agent’s prose or tool-call format, it checks terminal backend state and side effects after a task.

ThinkingBox contains 507 workflows across retail, auto insurance, travel, neobanking, and consulting. Each task is run 20 times from an identical clean backend. The release reports pass@1, whether a task succeeds at least once in 20 attempts, and the literal count of tasks that pass all 20 runs.

In a 121,680-trial common-set analysis across 12 models, 79,853 attempts failed executable checks. Of those failures, 67.24% ended cleanly, used a state-changing tool, and reported no final tool error. The authors also report model-specific consistency and estimated cost measures; for example, Kimi-K3 solved 476 of 507 tasks at least once but passed all 20 runs on 68 tasks.

The source says the benchmark can be run through OpenEnv. It does not establish how these results predict reliability in a particular production stack, nor does it provide a production-service price or guarantee of agent performance.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)