Observed facts: AWS says Deepgram ported its Nova-3 speech-to-text model from standard PyTorch to Trainium2 using the Neuron stack, then tuned kernels and a data-parallel Rust runtime. The reported configuration uses eight model replicas on one EC2 trn2.3xlarge instance. AWS reports 11–70% more audio per dollar than the compute options Deepgram had been using.
Mechanism: Compilation to the target accelerator, fused-kernel tuning, profiling and batch-size selection, combined with a data-parallel Rust runtime, shape the reported system-level throughput per dollar.
Why now: As speech becomes a high-volume AI workload, serving cost determines how much audio teams can process within a fixed infrastructure budget; the case provides an example of optimization beyond model selection.
First implication: Teams should benchmark the complete serving path, including batch sizes, runtime, and quality constraints, rather than extrapolating from an accelerator headline.
Evidence boundary: The figures come from AWS's vendor case study, concern Deepgram's offline throughput-oriented batch workload, and use instance prices at the time of evaluation; the supplied account does not give absolute throughput or independent replication.
First watch condition: Look for independently reproducible measurements on comparable workloads that disclose absolute throughput, accuracy, and total serving costs.
The account is useful as an implementation pattern, but not as proof that the same savings transfer to real-time voice agents, other speech models, or another team's workload. Its reported metric joins hardware economics to engineering choices and a specific batch operating point. That makes replication details—not the percentage range alone—the important evidence to follow.