---
type: "post"
id: "what-is-eval-driven-development-how-to-improve-52927d"
slug: "what-is-eval-driven-development-how-to-improve-52927d"
title: "What Is Eval-Driven Development? How to Improve AI Agents"
description: "LangChain’s guide presents eval-driven development (EDD) as a three-step loop for improving agents: define the behavior, write an evaluation before building, and compare each change against a baseline."
retrieval_nugget: "LangChain’s guide presents eval-driven development (EDD) as a three-step loop for improving agents: define the behavior, write an evaluation before building, and compare each change against a baseline. The practical point for builders is that agent changes must be tested for regressions as well as gains."
published_at: "2026-11-09"
updated_at: "2026-11-09"
record_date: "2026-11-09"
date_kind: "published_at"
topics: ["evals","agents","agent-loops"]
entities: ["LangChain","LangSmith","Harbor","Terminal-Bench"]
related_patterns: ["ai-native-orgs-move-to-review-and-orchestration"]
novelty_label: "new_angle"
what_changed: "It frames evaluation as a continuous development loop for AI agents, including defining success criteria, comparing iterations, and feeding CI and production failures back into eval datasets."
prior_refs: []
source_url: "https://www.langchain.com/blog/what-is-eval-driven-development"
source_urls: ["https://www.langchain.com/blog/what-is-eval-driven-development"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:what-is-eval-driven-development-how-to-improve-52927d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/what-is-eval-driven-development-how-to-improve-52927d/","markdown":"https://newruntime.com/posts/what-is-eval-driven-development-how-to-improve-52927d.md","json":"https://newruntime.com/posts/what-is-eval-driven-development-how-to-improve-52927d.json"}
markdown_url: "https://newruntime.com/posts/what-is-eval-driven-development-how-to-improve-52927d.md"
---

# What Is Eval-Driven Development? How to Improve AI Agents

## Retrieval answer

LangChain’s guide presents eval-driven development (EDD) as a three-step loop for improving agents: define the behavior, write an evaluation before building, and compare each change against a baseline. The practical point for builders is that agent changes must be tested for regressions as well as gains.

## Summary
LangChain’s guide presents eval-driven development (EDD) as a three-step loop for improving agents: define the behavior, write an evaluation before building, and compare each change against a baseline. The practical point for builders is that agent changes must be tested for regressions as well as gains.

## What happened
LangChain’s November 9, 2026 article describes EDD as a response to a familiar development problem: prompt-driven manual testing can reveal early mistakes, but becomes a weak guide as an agent grows. Agents are non-deterministic, may reach a correct result through different wording or tool sequences, and can call unnecessary tools or skip useful ones. The article says an evaluation scores runs across a set of examples, allowing teams to assess more than a single final answer, including tool use or other parts of a run.

The first step is to define what good behavior means for one target behavior at a time. LangChain recommends a short specification that says when the behavior should occur, what the agent should and should not do, and which existing behaviors must remain intact. Its example is an internal documentation agent with `search_docs` and `web_search` tools. The team wants to add `create_ticket` for reports of broken systems, while keeping how-to questions in the existing search flow.

The second step is to create a dataset and evaluator before implementing the change. The guide recommends using real agent runs as a source of examples when possible, because real user phrasing can expose cases developers might not anticipate. In the example, the dataset includes both ticket requests and questions that should continue to use the existing search tools. The evaluator can be code when the desired check is deterministic, such as whether the right tool was called, or an LLM judge when it is not. LangChain also discusses agents that change an environment: those need isolated runs and checks of the resulting state, and the article points to Harbor and Terminal-Bench as examples of that style of evaluation.

The team then runs a baseline, implements `create_ticket`, and runs the same evaluation again. LangChain’s example shows why looking only at an average score can mislead: ticket requests begin passing, but a VPN how-to question is incorrectly sent to the ticket tool. Reviewing the trace reveals that the tool description says to use it for “VPN and laptop issues,” a scope broad enough to catch questions that need documentation rather than IT intervention. Narrowing the description to requests that require IT action makes the examples pass. The guide cautions that tuning against a small fixed dataset can overfit; a perfect score there is not proof of production quality.

## What the archive adds
New Runtime’s earlier coverage has treated evaluation as part of agent building: [an October 22 note on Verifiers v0.4.0](https://newruntime.com/posts/unified-uv-run-eval-cli-replaces-separate-eval-573358/) covered a unified evaluation CLI, while [a November 3 note on chat-agent evaluation](https://newruntime.com/posts/steal-our-eval-setup-for-chat-agents-c59267/) described converting production signals into repeatable cases. LangChain’s guide is a new practical framing of the same broader development discipline, not a release announcement.

## Trend signal
This supports [AI-native organizations moving toward review and orchestration](https://newruntime.com/patterns/ai-native-orgs-move-to-review-and-orchestration/): the workflow makes specifying behavior, comparing outcomes, and inspecting regressions explicit parts of agent development. The guide focuses on evaluation mechanics, not organizational adoption, so it is supporting evidence rather than a broad demonstration of the pattern.

## Why it matters
An agent can appear better on its headline task while quietly breaking a behavior users already depend on. A baseline and a fixed set of representative cases give a team a way to see that tradeoff before shipping, while traces help locate the cause instead of treating the score as a verdict. This matters most when tools can take actions or alter an environment: success cannot be judged solely from fluent output. Teams still need to choose what counts as success and maintain examples that reflect real use. EDD does not remove human judgment, but it makes changes and regressions easier to inspect and discuss.

## What happens next
- **Teams adopting this workflow will expand evaluation datasets with production failures and edge cases** (6 months, 65%). Watch for: new examples being added after regressions found in deployed traces.

## Limits and uncertainty
LangChain presents a development method and an illustrative internal-docs-agent scenario, not a controlled study of EDD’s effect on reliability or development time. The article does not establish that a particular dataset size, score threshold, or evaluator design generalizes across agent types. Its own overfitting warning is material: developers can optimize for known examples while missing real user behavior. The value of the process therefore depends on the quality and coverage of the specification, dataset, evaluator, and review of failures.

## Skill to build now
Choose one agent behavior that is about to change. Write down when it should happen, what action should and should not occur, and which existing behavior must remain intact. Turn those requirements into a small set of realistic examples and an evaluator before changing the agent. Save a baseline, compare the new run example by example, and inspect traces for regressions instead of relying only on the average score.

## Reader takeaway
Treat every meaningful agent change as a comparison against an explicit behavioral contract. Evaluate both the capability you are adding and the behavior you cannot afford to break; a clean score on a small suite is a starting signal, not a production guarantee.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Agent Loops](https://newruntime.com/topics/agent-loops.md): Explore the agent-loops topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)