What happened
AWS's announcement says SageMaker AI now supports a companion metadata stream for metering usage during bidirectional streaming inference. In the existing setup, clients send audio through an HTTP/2 event stream and SageMaker connects to the model server over WebSockets. A session can last 30 seconds or 30 minutes, so the billable total may accumulate after the session begins; the response-header method used for request-response inference cannot report periodic consumption during the live session.
A model server opts in by returning X-Amzn-SageMaker-Metadata-Stream-Supported: true in the data WebSocket upgrade response. SageMaker then opens a second WebSocket, by default at /invocations-bidirectional-stream-metadata. The data and metadata handshakes share an X-Amzn-SageMaker-Request-Id, which the server uses to associate metering records with the inference session. The server sends JSON text frames, each containing exactly one record with the inference.count dimension and a nonnegative ConsumedUnits value. A ClientToken can make messages idempotent on that connection.
The metadata stream permits one message per second per connection and limits each message to 512 bytes. AWS recommends reporting incrementally as usage accrues, then sending a final record when a session ends. A product charging by audio second could report 720 units for 12 minutes of metered audio. The provider defines what audio counts; the open connection duration does not itself define the billable quantity. The Marketplace offer maps the unit to its price, shown as one per-unit price for real-time inference, with up to three decimal places.
If the metadata connection fails, inference continues by default and metering stops; providers can set X-Amzn-SageMaker-Metadata-Stream-Required: true to stop inference when that stream is unavailable. This favors metering continuity over availability, but does not ensure malformed or dropped records are billed. AWS says providers should reconcile records against Marketplace usage reports and buyer charges, and test normal completion, interrupted sessions, stream loss, and malformed messages before release.
Deepgram is the launch partner, offering usage-based STT and TTS billing through AWS Marketplace. Customers subscribe to a Deepgram streaming model, deploy a SageMaker real-time endpoint, and connect through the bidirectional streaming API; AWS says no client-side configuration is needed for Deepgram's automatic metering. Endpoint infrastructure charges still apply. Model providers can integrate the metadata stream themselves, validate through a limited listing and allowlisted buyer, and offer negotiated per-unit rates or fixed-price private offers.
What the archive adds
New Runtime's earlier coverage noted that Google API Gateway had added public-preview support for streaming APIs, including bidirectional traffic, on October 22, 2026. That was a gateway capability; this is a new AWS Marketplace billing mechanism for streaming inference, not a restatement.
Trend signal
This supports agent-ready software exposes capabilities only weakly: AWS is exposing a defined metering interface to model servers, but the announcement is about billing infrastructure, not agent composition or permissioned capabilities. This does not yet fit a pattern New Runtime tracks.
Why it matters
For teams building real-time speech products, billing is part of the production interface. A session-based service needs a way to count actual work without treating an idle or long-lived connection as equivalent to audio processed. The companion stream gives providers a specific mechanism to send that count while inference remains live, and Marketplace can carry the resulting software charge on the AWS bill.
The key operational trade-off is between inference availability and metering continuity. Defaulting to continued inference means a failed metering connection can leave usage unreported; making it required can interrupt inference when the connection fails. Teams therefore need to decide which risk their pricing model can tolerate, report usage incrementally, and test reconciliation rather than assuming transport delivery equals billing. Infrastructure charges remain a separate cost, so per-unit software pricing does not make endpoint capacity free during quiet periods.
Limits and uncertainty
AWS describes the protocol and recommends reconciliation, but the announcement does not quantify adoption beyond naming Deepgram as launch partner or provide comparative customer cost outcomes. The billable unit depends on each provider's definition and Marketplace offer; the example of 720 units applies only to a product charging per second of metered audio. A reliable WebSocket transport does not guarantee a record is accepted and billed: malformed or dropped messages may not be charged. Endpoint infrastructure charges remain in addition to the software charge.
Skill to build now
If you operate a Marketplace-listed streaming model, implement a known-quantity test before public release: generate a controlled audio duration, report incremental ConsumedUnits, and compare server records with both seller usage reports and buyer charges. Include interruption, metadata-stream loss, and malformed-message cases. AWS recommends deleting test endpoints afterward to stop ongoing infrastructure charges.
Reader takeaway
SageMaker's new companion WebSocket lets model providers meter software usage during live bidirectional inference, which is useful for streaming speech products whose charge should follow processed audio. It does not replace endpoint pricing or eliminate reconciliation work: providers define the unit, and they must test that reported usage becomes billed usage.