NEW RUNTIME

Topic hub

Agent evals

A record of evals moving from model benchmarks toward behavior, workflow completion, skill reliability, and review burden.

Short answer

  • Agent evals need to measure behavior in a workflow, not only answer quality in a prompt.
  • The relevant unit is completed work under constraints: tools, files, permissions, time, cost, and review outcome.
  • A good eval makes regressions visible before the agent is trusted with larger work.

Pattern memory

Current hypotheses

2 patterns
  1. medium

    Skills become a portable capability layer

    Agent skills are emerging as a portable capability layer, but their value depends on progressive disclosure, provenance, security review, and behavioral evaluation.

  2. high

    Verification bandwidth is the scarce engineering resource

    The primary bottleneck in agentic software delivery is moving from code production to the human and machine capacity required to verify it.

Raw signals

Recent observations

11 signals

  1. Agent skills need behavioral evals, not prose review

    Benchmarks show that expert-authored skills can help while self-generated skills can underperform a no-skill baseline.

  2. Verification capacity becomes the coding-agent bottleneck

    Coding agents can parallelize implementation faster than engineering teams can expand review judgment.

  3. AI product feedback needs behavioral traces

    For nondeterministic products, edits, retries, overrides, abandonment, and sampled outputs reveal more than isolated ratings.

  4. Databricks benchmarks coding agents on its own codebase

    Databricks evaluates agents on fresh internal pull-request tasks and measures success alongside runtime, tokens, and cost.

  5. Newer agent models can reward shorter prompts

    OpenAI's migration guidance emphasizes concise instructions, explicit contracts, and less compensating prose for newer models.

  6. Multi-agent systems need continuous eval pipelines

    A Google workflow evaluates not only the final answer but also routing, delegation, tool calls, and the trajectory between agents.

  7. Agent memory is being evaluated as system behavior

    MemoryData evaluates what an agent stores, retrieves, updates, and forgets across a sequence instead of grading one final answer.

  8. Harvey trained a legal agent with an applied compute loop

    Harvey describes domain experts, task environments, evaluation, and iterative training as one system for legal agent performance.

  9. Verifiability is an AI product feature

    Hamel Husain's eval-smell framework treats missing traces, weak rubrics, and unverifiable outputs as product defects.

  10. Prompt debt turns natural language into legacy code

    System prompts accumulate patches for edge cases until model changes and local fixes make the behavior fragile and opaque.

  11. AI compresses coding, not the whole software lifecycle

    Implementation time can collapse while requirements, architecture, validation, review, and maintenance stay constrained by human judgment.