---
type: "post"
id: "using-a-smart-clip-router-to-benchmark-conversational-b36d84"
slug: "using-a-smart-clip-router-to-benchmark-conversational-b36d84"
title: "Using a Smart Clip Router to Benchmark Conversational Speech-To-Speech Agents Engineering LLM"
description: "DoorDash describes OrderUp-Bench, a food-ordering benchmark that uses a smart router to let a simulated customer respond to voice agents with clips from real conversations."
retrieval_nugget: "DoorDash describes OrderUp-Bench, a food-ordering benchmark that uses a smart router to let a simulated customer respond to voice agents with clips from real conversations. Its comparison suggests that clean completion, recoverable mistakes, silent failures and interaction time tell different stories—so turn accuracy alone can mislead teams evaluating agents for real tasks."
published_at: "2026-11-14"
updated_at: "2026-11-14"
record_date: "2026-11-14"
date_kind: "published_at"
topics: ["gemini"]
entities: ["DoorDash","OrderUp-Bench","GPT Realtime 2","Gemini Flash Live 3.1","Grok Voice 1.0","Whisper","GPT-5.5"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "DoorDash introduces OrderUp-Bench for evaluating speech-to-speech food-ordering agents and reports differences in order accuracy, policy compliance, duration and silent failure across models."
prior_refs: ["https://newruntime.com/posts/building-an-automated-improvement-harness-for-an-ai-74a2f3/"]
source_url: "https://careersatdoordash.com/blog/voice-agent-benchmark-smart-clip-router"
source_urls: ["https://careersatdoordash.com/blog/voice-agent-benchmark-smart-clip-router"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:using-a-smart-clip-router-to-benchmark-conversational-b36d84"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/using-a-smart-clip-router-to-benchmark-conversational-b36d84/","markdown":"https://newruntime.com/posts/using-a-smart-clip-router-to-benchmark-conversational-b36d84.md","json":"https://newruntime.com/posts/using-a-smart-clip-router-to-benchmark-conversational-b36d84.json"}
markdown_url: "https://newruntime.com/posts/using-a-smart-clip-router-to-benchmark-conversational-b36d84.md"
---

# Using a Smart Clip Router to Benchmark Conversational Speech-To-Speech Agents Engineering LLM

## Retrieval answer

DoorDash describes OrderUp-Bench, a food-ordering benchmark that uses a smart router to let a simulated customer respond to voice agents with clips from real conversations. Its comparison suggests that clean completion, recoverable mistakes, silent failures and interaction time tell different stories—so turn accuracy alone can mislead teams evaluating agents for real tasks.

## Summary
DoorDash describes OrderUp-Bench, a food-ordering benchmark that uses a smart router to let a simulated customer respond to voice agents with clips from real conversations. Its comparison suggests that clean completion, recoverable mistakes, silent failures and interaction time tell different stories—so turn accuracy alone can mislead teams evaluating agents for real tasks.

## What happened
DoorDash’s November 12, 2026 engineering article describes OrderUp-Bench, an evaluation benchmark for food-ordering scenarios with known target carts. A router agent simulating a customer chooses audio clips dynamically while interacting with an order-taking agent. DoorDash says this is intended to address limitations of replaying fixed recordings or having two agents simulate a conversation: neither approach, in its view, captures how real customers respond when an exchange takes an unexpected turn. The repository was described as forthcoming.

The test measures whether an agent captures the expected order, follows its operating policy, and how long the session takes. DoorDash compared three commercial speech-to-speech models: GPT Realtime 2, Gemini Flash Live 3.1 and Grok Voice 1.0. It also used Whisper with GPT-5.5 as a cascaded baseline. The models received the same prompt, instructing them to take an order and follow the same rules for options and follow-up questions.

The router chooses among clips drawn from human audio sessions, using a decision tree and a goal state to respond to the agent’s questions or deviations. Scenarios include ambiguous item names, omitted required options, multiple items in one utterance, and corrections to earlier requests. If an agent fails to acknowledge an item, the evaluator can repeat it. DoorDash also created universal text-to-speech fallback clips for common responses such as “yes,” “no,” and “just the defaults.” Participants’ recordings varied in accents, background noise and disfluencies, including pauses and mid-sentence corrections.

DoorDash reports that GPT Realtime 2 had the highest clean pass rate at 66.4%; the other models ranged from 28.5% to 62.9%. GPT Realtime 2 also had the highest silent-failure rate, at 8.9%, compared with 2.0–4.0% for the others. When the calculation allowed for errors a customer might plausibly catch and correct, effective success rates ranged from 79.6% to 84.4%. Runs that recovered from errors took about 35% longer than clean runs. DoorDash derived a 1.37x time-scaling factor for recovery in a subset of tasks with both recovered and correct runs.

## What the archive adds
New Runtime’s earlier coverage on November 11, 2026 described DoorDash’s automated, expert-aligned harness for benchmarking its AI support agent and converting recurring unresolved cases into targeted improvements ([Building an Automated Improvement Harness for an AI Support Agent](https://newruntime.com/posts/building-an-automated-improvement-harness-for-an-ai-74a2f3/)). This is a new event in a continuing DoorDash thread about building evaluations around realistic task outcomes.

## Trend signal
This does not yet fit a pattern New Runtime tracks. The result is a useful evaluation design for conversational speech, but it does not by itself establish a broader shift in how agent platforms, protocols, memory, security, or organizational workflows are changing.

## Why now
DoorDash says that, before rolling out new agent versions, its existing offline testing did not answer practical questions about unnecessary clarifications, customer effort or whether shorter responses actually shortened conversations. The benchmark is presented as an attempt to make those trade-offs testable before deployment.

## Connections to other events
New Runtime’s earlier coverage on November 11, 2026 described DoorDash’s support-agent harness, which turns recurring unresolved cases into targeted improvements ([Building an Automated Improvement Harness for an AI Support Agent](https://newruntime.com/posts/building-an-automated-improvement-harness-for-an-ai-74a2f3/)). The voice benchmark extends the same operational concern to multi-turn audio: reproducing failures and evaluating whether an agent reaches the intended outcome.

## Why it matters
A voice agent can sound fast and handle individual turns well while still misrecording an order, asking needless questions or leaving the customer no chance to catch an error. DoorDash’s benchmark makes those outcomes legible as separate dimensions: clean success, recovery, recoverability, silent failure and elapsed time. That framing is useful beyond food ordering wherever an agent must translate ambiguous speech into a consequential structured result.

The figures do not establish which model is best for another team’s workload. They do show why evaluation design should reflect the cost of errors and the chance a user can intervene. Teams can use the same principle in their own domain: define the expected final state, represent plausible user corrections, and measure the time and failure severity of the entire interaction—not just the model’s response latency.

## Limits and uncertainty
DoorDash says it does not yet have a single score that can confidently determine which agent is better. Its 50% assumed correction rate was selected for simplicity, not measured from customer behavior, and different error types may have different correction probabilities. The company says it wants to calibrate offline metrics against online success and acceptance metrics. Results are also sensitive to streaming latency and audio timing: across 248 samples and four models, repeated runs on the same sample kept the same correctness label only 53.3% of the time. The benchmark repository and a linked technical report were not available in the announcement as described there.

## Skill to build now
For a voice workflow, define a small set of realistic scenarios with an explicit target state and customer clips that cover ambiguity, corrections and omitted details. Label failures by whether a user could hear and correct them, then track full task duration as well as clean completion. Treat the resulting score as a diagnostic until repeated runs and real customer data calibrate its assumptions.

## Reader takeaway
Voice-agent quality is not just whether each turn sounds right. Measure whether the agent reaches the correct final state, whether its mistakes are visible and recoverable, and how much extra time recovery costs. DoorDash’s own benchmark remains a work in progress, but its failure categories show what a more realistic evaluation needs to expose.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)