EvoCode-Bench is a useful correction to how coding agents are normally evaluated. A single prompt with a final pass/fail result misses the real failure mode of agentic coding: the agent has to keep working in the same codebase as requirements evolve.
The benchmark uses 26 tasks across 227 sequential rounds. A persistent container carries code, dependencies, and architectural decisions forward. Each new round extends, corrects, or conflicts with earlier requirements, and the test suite checks cumulative behavior rather than only the latest change.
That design exposes the bottleneck. Agents are often capable of building the new feature, but they break something that previously worked. The source reports that more than half of multi-turn failures happen on rounds that models solve easily from a clean state, and that pass rates drop sharply as rounds accumulate.
The important product lesson is that memory and verification are inseparable. If an agent does not track requirements and rerun old checks, it can look competent turn by turn while the system gets worse. Persistent planning documents materially improve results because they preserve the contract across edits.
New Runtime synthesis: coding-agent evaluation has to test sustained work, not isolated brilliance. The artifact that matters is not one solved task; it is a codebase that survives changing instructions without silent regressions.