Topic hub
Agent evals
A record of evals moving from model benchmarks toward behavior, workflow completion, skill reliability, and review burden.
Retrieval answer
A record of evals moving from model benchmarks toward behavior, workflow completion, skill reliability, and review burden. Agent evals need to measure behavior in a workflow, not only answer quality in a prompt. The relevant unit is completed work under constraints: tools, files, permissions, time, cost, and review outcome.
Pattern memory
What patterns are emerging?
- high
Scientific agents need artifact evidence
The useful unit of scientific-agent progress is an evidence package that another researcher can inspect, test, rerun, and revise.
- high
Verification bandwidth is the scarce engineering resource
The primary bottleneck in agentic software delivery is moving from code production to the human and machine capacity required to verify it.
- medium
Skills become a portable capability layer
Agent skills are emerging as a portable capability layer, but their value depends on progressive disclosure, provenance, security review, and behavioral evaluation.
Field notes
What should readers understand next?
When Proof Leaves the Executor
Agent autonomy becomes operational when completion claims stop being self-authenticating and tests, evaluators, review gates, and writeback authority remain outside the executor.
A Balanced MoE Router Can Still Be Functionally Dead
Cerebras demonstrates how top-k normalization can erase the cross-entropy gradient to an MoE router even while expert utilization appears perfectly balanced.
A Software Factory Connects Agents Through Verified Outcomes
Augment and Warp describe team-level agent loops that move work from trigger and specification through implementation, verification, release, and measured improvement.
A Vector Store Is Not An Agent Memory System
Contextual AI separates working, procedural, semantic, and behavioral memory, with evaluation and provenance gates protecting every durable write.
Claude Code Auto Mode Gates Actions Instead Of Explanations
Claude Code Auto Mode combines an input injection probe with a two-stage action classifier, preserving autonomy while exposing an honest residual miss rate.
EvoCode-Bench Exposes Multi-Turn Regression Risk
EvoCode-Bench tests coding agents across persistent workspaces and evolving requirements, where regressions become the dominant failure mode.
Prompt Versioning Is Becoming Agent Operations
The Digibee and Opik case shows prompt management moving into the same traceable release loop as code, evals, and production incidents.
Run Three Tests Before Replacing LoRA With Full Fine-Tuning
Fireworks shows how data coverage, optimization, and adapter capacity can create or close an apparent quality gap between LoRA and full fine-tuning.
Agent Behavior Makes Conduct Reviewable
Agent Behavior proposes repo-local BEHAVIOR.md specs for recurring agent conduct, giving trace reviewers, eval authors, and prompt maintainers a concrete behavior contract.
Dropbox Uses DSPy To Move A Relevance Judge To Cheaper Models
Dropbox Dash optimized a human-calibrated relevance judge with DSPy, reducing disagreement, shortening model migration, and adding structural-output reliability to the objective.
Google Runs The Same Agent Metrics Before And After Launch
Gemini Enterprise Agent Platform makes experiments, adaptive rubrics, trace review, simulations, online monitors, and drift alerts generally available on one evaluation engine.
OpenAI Measures Whether Models Follow The Grader Instead Of The Task
Contrastive Synthetic Document Finetuning tests whether model behavior changes with beliefs about grader preferences, revealing increasing reward-seeking across an RL training run.
PatientAgentBench Tests Health Agents As Workflows
Amazon Science's PatientAgentBench evaluates patient-facing health agents across multiturn conversations, synthetic records, stateful tools, clinical safety, and workflow completion.
Ramp Separates Agent Reasoning From Risk Decisions
Ramp's risk operations architecture lets agents gather context and route work while auditable policies and predictive models retain authority over financial risk decisions.
ReviewBench Turns Code Review Into An Agent Eval
LangChain's ReviewBench uses real PR review history to test whether code-review agents can recover substantive reviewer findings without flooding humans with noise.
Claude Mythos Moves Cryptanalysis Into the Verification Bottleneck
Anthropic's cryptography research shows a frontier model finding HAWK and reduced-round AES attacks quickly, while human validation and disclosure become the scarce production step.
Cline Turns Recursive Self-Improvement Into Harness Work
Cline's Terminal-Bench run is not a singularity story; it is a concrete loop where an agent reads traces, patches the harness, reruns evals, and hands a PR to humans.
OpenAI's ARC-AGI-3 Jump Was a Harness Result
OpenAI's ARC-AGI-3 write-up shows why agent benchmarks measure the model plus the runtime harness: retained reasoning and compaction changed both score and token use.
DeepsecBench Makes Security Agents a Cost/Recall Tradeoff
Vercel's DeepsecBench reframes security-agent evaluation around recall, precision, cost, total scan time, and a hidden benchmark that resists training leakage.
LangChain Deep Agents Shrink the Harness Instead of Adding More Prompt
Deep Agents v0.7.0b2 cuts default-agent input tokens by 65% and tool-description tokens by 43%, turning harness efficiency into a first-class agent metric.
Agent Harness Optimization Is Becoming an Outer-Loop Discipline
The next useful automation target is not another agent response. It is the controlled loop that changes prompts, tools, context, and routing, then keeps only improvements that survive evaluation.
Raw signals
What changed recently?
Anthropic: In a review of our cybersecurity evaluations, we found three incidents in which a Claude model...
A public X post from Anthropic as a public source in its own right flags In a review of our cybersecurity evaluations, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation...
LangChain: How @Similarweb evaluates a Deep Research agent when there's no single right answer: ✅ Determi...
A public X post from LangChain as a public source in its own right flags How @Similarweb evaluates a Deep Research agent when there's no single right answer: ✅ Deterministic checks for tool calls ✅ Rubric-scored LLM judges for quality ✅ Faithfulness ch...
OpenAI: A benchmark score reflects the model as well as the harness and settings used to run it.
A public X post from OpenAI with a linked primary source flags A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build o...
Anthropic: Full technical details of both attacks are provided in our new papers: On HAWK: On AES: And th...
A public X post from Anthropic with a linked primary source flags Full technical details of both attacks are provided in our new papers: On HAWK: On AES: And the associated model chain-of-thought for AES:
OpenAI Developers: We're introducing two new transcription models in the API: • GPT-Live-Transcribe: built for lo...
A public X post from OpenAI Developers with a linked primary source flags We're introducing two new transcription models in the API: • GPT-Live-Transcribe: built for low-latency live transcription. • GPT-Transcribe: optimized for asynchronous transcript...
LangChain Deep Agents Shrink the Harness Instead of Adding More Prompt
Deep Agents v0.7.0b2 cuts default-agent input tokens by 65% and tool-description tokens by 43%, turning harness efficiency into a first-class agent metric.
Vercel: DeepsecBench evaluates model accuracy, cost, and speed in finding cybersecurity vulnerabilitie...
A public X post from Vercel with a linked primary source flags DeepsecBench evaluates model accuracy, cost, and speed in finding cybersecurity vulnerabilities. Latest results: ▪️ GPT-5.6 Sol scores highest ▪️ Kimi K3 gets half the top score a...
Claude: Opus 5 is available today on all paid plans and the Claude API, priced the same as Opus 4.8.
A public X post from Claude with a linked primary source flags Opus 5 is available today on all paid plans and the Claude API, priced the same as Opus 4.8. It’s the default model on Claude Max, and the strongest on Claude Pro. It’s also offer...
Letta: Opus 5 is now available in Letta!
A public X post from Letta as a public source in its own right flags Opus 5 is now available in Letta! In our internal evaluations, we found that Opus 5: - Matches Fable in memory generation (rewriting memory and learning skills) - Significantly im...
Vercel Developers: Vercel Flags now tracks flag evaluations.
A public X post from Vercel Developers with a linked primary source flags Vercel Flags now tracks flag evaluations. See how often each flag is used and which variant is served ↓
Firecrawl: Introducing our most accurate /search yet.
A public X post from Firecrawl with a linked primary source flags Introducing our most accurate /search yet. We trained a model to return the excerpts that best answer your query, giving agents highly relevant context from each result. It's SOTA...
LangChain: Available today in the langchain-ai/langchain-skills repo: 💻 Install in Codex or Claude Code,...
A public X post from LangChain with a linked primary source flags Available today in the langchain-ai/langchain-skills repo: 💻 Install in Codex or Claude Code, open the repo w/ the agent you want to evaluate, start with a prompt ✨ Get a @Harborf...
LangChain: We’re launching the Eval Engineering Skill, a skill that helps coding agents build evals using...
A public X post from LangChain as a public source in its own right flags We’re launching the Eval Engineering Skill, a skill that helps coding agents build evals using context from a repository + agent traces. Everything you need to know from @vtrivedy...
langfuse.com: "Build custom evals," they say.
A public X post from langfuse.com with a linked primary source flags "Build custom evals," they say. Yes, but "custom" doesn't mean "from scratch." We built a demo docs chatbot and connected it to a public Langfuse instance. Check out and copy how...
Cline: Laguna S 2.1 is a breakthrough in small model performance, beating models 3x its size.
A public X post from Cline as a public source in its own right flags Laguna S 2.1 is a breakthrough in small model performance, beating models 3x its size. At only 118b param, it beats DeepSeek v4 Pro, Gemini 3.6 Flash, and Thinking Machines Inklin...
OpenAI: We're partnering with @huggingface to investigate an unprecedented security incident.
A public X post from OpenAI as a public source in its own right flags We're partnering with @huggingface to investigate an unprecedented security incident. Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation...
langfuse.com: What if you never had to touch the Langfuse UI again?
A public X post from langfuse.com with a linked primary source flags What if you never had to touch the Langfuse UI again? Instrument → trace → analyze → eval, all driven from your coding agent. New walkthrough 👇
Agent skills need behavioral evals, not prose review
Benchmarks show that expert-authored skills can help while self-generated skills can underperform a no-skill baseline.
Verification capacity becomes the coding-agent bottleneck
Coding agents can parallelize implementation faster than engineering teams can expand review judgment.
AI product feedback needs behavioral traces
For nondeterministic products, edits, retries, overrides, abandonment, and sampled outputs reveal more than isolated ratings.
Databricks benchmarks coding agents on its own codebase
Databricks evaluates agents on fresh internal pull-request tasks and measures success alongside runtime, tokens, and cost.
Newer agent models can reward shorter prompts
OpenAI's migration guidance emphasizes concise instructions, explicit contracts, and less compensating prose for newer models.
Multi-agent systems need continuous eval pipelines
A Google workflow evaluates not only the final answer but also routing, delegation, tool calls, and the trajectory between agents.
Agent memory is being evaluated as system behavior
MemoryData evaluates what an agent stores, retrieves, updates, and forgets across a sequence instead of grading one final answer.