---
type: "post"
id: "introducing-the-baseten-voice-agent-stack-own-and-bb6866"
slug: "introducing-the-baseten-voice-agent-stack-own-and-bb6866"
title: "Introducing the Baseten Voice Agent Stack: own and optimize voice agents end-to-end"
description: "Baseten has announced a private-preview infrastructure stack for production voice agents and a reference API built on it."
retrieval_nugget: "Baseten has announced a private-preview infrastructure stack for production voice agents and a reference API built on it. The design colocates orchestration and speech models, lets teams mix or post-train components, and claims roughly one-second median turn latency at up to 56 concurrent sessions—an approach worth examining for teams balancing responsiveness, control, and cost."
published_at: "2026-11-17"
updated_at: "2026-11-17"
record_date: "2026-11-17"
date_kind: "published_at"
topics: ["agents","ai-infrastructure"]
entities: ["Baseten","Baseten Voice Agent Stack","Baseten Realtime Voice","Pipecat","OpenAI Realtime"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "Baseten has launched a separate voice-agent stack and Realtime Voice API in private preview, with sub-second median turn latency at high concurrency."
prior_refs: []
source_url: "https://www.baseten.co/blog/introducing-the-baseten-voice-agent-stack"
source_urls: ["https://www.baseten.co/blog/introducing-the-baseten-voice-agent-stack"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:introducing-the-baseten-voice-agent-stack-own-and-bb6866"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/introducing-the-baseten-voice-agent-stack-own-and-bb6866/","markdown":"https://newruntime.com/posts/introducing-the-baseten-voice-agent-stack-own-and-bb6866.md","json":"https://newruntime.com/posts/introducing-the-baseten-voice-agent-stack-own-and-bb6866.json"}
markdown_url: "https://newruntime.com/posts/introducing-the-baseten-voice-agent-stack-own-and-bb6866.md"
---

# Introducing the Baseten Voice Agent Stack: own and optimize voice agents end-to-end

## Retrieval answer

Baseten has announced a private-preview infrastructure stack for production voice agents and a reference API built on it. The design colocates orchestration and speech models, lets teams mix or post-train components, and claims roughly one-second median turn latency at up to 56 concurrent sessions—an approach worth examining for teams balancing responsiveness, control, and cost.

## Summary
Baseten has announced a private-preview infrastructure stack for production voice agents and a reference API built on it. The design colocates orchestration and speech models, lets teams mix or post-train components, and claims roughly one-second median turn latency at up to 56 concurrent sessions—an approach worth examining for teams balancing responsiveness, control, and cost.

## What happened
Baseten’s November 16, 2026 announcement introduces the Baseten Voice Agent Stack on its Dedicated Inference platform, alongside Baseten Realtime Voice, a reference agent built on that stack. Both are in private preview, with access through a waitlist. The company describes the stack as infrastructure for running production voice agents: the orchestrator and models run in one cloud region, connected over what Baseten calls a tight in-cluster network. Individual components can scale independently, and teams can choose or post-train models at each stage.

Realtime Voice uses a cascade of Baseten-optimized open-weight speech models and a custom Pipecat-based orchestrator. Its optional “talker-thinker” arrangement lets a fast, colocated model handle the conversation while delegating more complex reasoning or tool use to a stronger model. Baseten says the API supports audio-in/audio-out and text-in/audio-out, client- and server-side tools, and MCP servers. Server-side tools run on the colocated orchestrator to avoid extra client round trips. The API follows the core OpenAI Realtime protocol; Baseten says existing agents can move over with minimal changes. It also describes additional options including keyword boosting, configurable acknowledgments, voice cloning, and access to Baseten Model APIs and hosted tools.

The company reports 93% on Full-Duplex-Bench, which it calls the highest score for an open-source system on Artificial Analysis, and 73.5% on the English leaderboard split of τ²-Bench Voice. It reports 93.9% on the telecom portion of that benchmark, with delegation enabled, versus a 31.2% pass@1 cascaded baseline. These are Baseten’s reported results; the announcement cautions that the τ²-Bench result is not directly comparable to the Artificial Analysis result.

Baseten reports median turn latency of about one second with context up to 16K tokens, across one to 56 concurrent sessions. Its figures show median time to first audio rising from 0.94 seconds at one concurrent call to 1.14 seconds at 56. The company estimates about $0.01 per audio minute per stream at scale, and says the dedicated cascade costs less than every hosted speech-to-speech API it evaluated at 32 or more concurrent streams. At 56 sessions it estimates $0.008 per call-minute including model API delegation, under stated utilization assumptions. The announcement says reasoning-model delegation is billed per token and excluded from its principal $0.012 benchmark cost figure.

## What the archive adds
New Runtime has covered Baseten’s [Router benchmark and public preview](https://newruntime.com/posts/introducing-baseten-router-a-pareto-optimal-model-router-db4bc1/) and a separate [Sesame voice-agent pricing change](https://newruntime.com/posts/sesame-voice-agent-is-ending-its-preview-period-20d8ce/). Those are related by Baseten’s infrastructure work and the broader voice-agent market, not the same event. This is a new product announcement, with a distinct emphasis on end-to-end voice infrastructure.

## Trend signal
This release supports [“Harness architecture outlives model choice”](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/): Baseten is positioning orchestration, model interchange, and deployment control as a durable layer around interchangeable speech and reasoning models. Its claim that teams can bring different STT, language, and TTS components—and move incrementally between dedicated and hosted models—makes the harness and operating environment central to the product, rather than tying the agent to one model provider.

## Why now
Baseten’s announcement frames the release around the latency and cost demands of production voice agents. Its reported results are intended to show that colocating the orchestrator with the model cascade can avoid network round trips and that independent scaling can improve utilization as concurrency rises.

## Why it matters
For teams building voice agents, a cascade is not just a model-selection decision. Network placement, orchestration, tool execution, scaling, and model ownership all shape the delay a caller experiences and the cost of keeping a conversation live. Baseten’s design offers a concrete architecture to assess: put the orchestrator beside the speech components, scale each component against its own concurrency, and delegate complex work without making every conversational turn wait on the reasoning model.

The trade-off is operational ownership. Dedicated infrastructure and model choice can offer control and customization, but teams need to validate real workload latency, utilization, failover, and total cost—not assume benchmark results transfer to their call patterns. Protocol compatibility may reduce migration work, but it does not by itself establish parity in behavior or production reliability.

## Limits and uncertainty
The performance, benchmark, and cost figures in the announcement are Baseten’s own claims; the results have not been independently verified here. Benchmark scores depend on setup and conditions, and the company notes that one benchmark result is not directly comparable to the other. The cost comparison applies to evaluated providers and stated concurrency assumptions, while token-billed reasoning delegation can add expense. The announcement does not establish what performance or economics a particular customer will see, or how the private-preview service will behave under that customer’s workload.

## Reader takeaway
Baseten is selling control over the whole voice-agent path, not a single speech model: colocated orchestration, interchangeable cascade components, server-side tools, and independent scaling. Treat its latency and cost numbers as hypotheses to test against your own concurrency and call mix. The products are in private preview, so access and real-world results remain to be established.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Ai Infrastructure](https://newruntime.com/topics/ai-infrastructure.md): Explore the ai-infrastructure topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
