---
type: analysis
slug: research-google-s-tpus-are-becoming-a-serving-layer-outside-google-but-the-proof-so-far
title: "Google's TPUs are becoming a serving layer outside Google, but the proof so far is benchmarks and vendor claims, not audited production data"
description: "SemiAnalysis's InferenceX preview puts Ironwood ahead of B200/B300 on FP8 cost per token. Google has open-sourced native PyTorch, vLLM and SGLang backends, and Meta and Anthropic are external users. Still missing: independent production latency, availability and FP4-era comparisons."
published_at: 2026-10-23T02:00:00.000Z
topics: ["google","anthropic","benchmarks","research"]
source_urls: ["https://inferencex.semianalysis.com/blog/tpu-inferencex-full-steam","https://inferencex.semianalysis.com/about","https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus","https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu/","https://anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services","https://finance.yahoo.com/news/google-signs-multibillion-dollar-ai-232844608.html","https://www.networkworld.com/article/4173075/google-opens-tpus-to-enterprises-beyond-its-own-cloud-via-blackstone-jv.html"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
The story's label, 'TPU InferenceX', blends two separate things. InferenceX is SemiAnalysis's open-source continuous inference benchmark, not a serving product. On September 7, 2026, SemiAnalysis published the first third-party Ironwood (TPUv7) results on an 'InferenceX Official Preview' under the headline 'TPU Inference Externalization Full Steam Ahead'. The real question is whether Google's TPUs, long a closed internal edge behind Search, Ads, YouTube and Gemini, are turning into a general inference platform that outside teams can buy, rent and program with standard tools.

That matters for builders because accelerator choice has been tied to software lock-in. If vLLM and SGLang run natively on TPUs through PyTorch, and capacity is sold outside Google Cloud, TPUs become a price-competing serving option rather than a Google-only option.

## What we checked

- **External TPU inference turns TPUs from Google's internal advantage into a market serving layer only with independent production workloads and a portable API.** partly confirmed. Both conditions are starting to appear. The October 20, 2026 open-source releases (TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU) keep the standard vLLM/SGLang schedulers and API servers. Meta is quoted as running SGLang-TorchTPU in production, and Anthropic ties its TPU expansion to serving demand. However, the engines are labeled preview, the production evidence is vendor-supplied quotes, and no third party has published production latency or availability data. [4][3][5]
- **(Added) Does Ironwood's InferenceX cost-per-token lead over Nvidia hold across configurations?** partly confirmed. Against aggregated FP8 B200/B300 it holds across much of the latency curve, but it comes with higher TTFT. SemiAnalysis itself says GB300 NVL72 with disaggregated serving keeps roughly a 30% perf-per-dollar edge in the middle latency range, and that Nvidia's FP4 has a quality-adjusted lead because TPUv7 has no FP4. [1]
- **(Added) Is TPU access genuinely leaving Google Cloud (sales, leasing, third-party clouds)?** confirmed. Anthropic committed to up to one million TPUs. Reuters, citing The Information, reported a multibillion-dollar Meta rental deal, which neither company confirmed. Google and Blackstone announced a TPU compute-as-a-service venture targeting about 500 MW in 2027. [5][6][7]

## What we found

- SemiAnalysis benchmarked Qwen3.5 397B in FP8 at 8k input / 1k output. At 100 tok/s/user it reports Ironwood at about $0.181 per million tokens, against $0.222 for B200 and $0.276 for B300. At 20 tok/s/user it reports 50.4% more tokens per dollar than B200 and 96.0% more than B300. Those figures rest on SemiAnalysis's own TCO assumptions, and mean TTFT at concurrency 256 was 5.41 s on Ironwood versus 3.75 s on B200 and 2.40 s on B300. [1]
- The same post spells out the limits. Disaggregated GB300 NVL72 holds roughly a 30% perf-per-dollar advantage in the middle latency range. TPUv7 has no native FP4. On TPU, speculative decoding (MTP), prefill-decode disaggregation, KV offload and agentic workloads were still gaps. It also says Ironwood is the first generation Google sells outright as well as rents. [1]
- InferenceX says every dashboard point comes from a public GitHub Actions run of a recipe committed to its repo, with artifacts kept for 90 days. That makes the numbers checkable in principle. The About page lists Nvidia, AMD, OpenAI, Microsoft and Meta as organizations that 'support and trust' the project but discloses no funding terms. 'Official preview' is not defined there. [2]
- On October 20, 2026, Google open-sourced TorchTPU, a native PyTorch backend co-developed with Meta's PyTorch team. Google positions TPUs as a PyTorch device next to GPUs and CPUs, with integrations for vLLM, SGLang, llm-d, Ray and Hugging Face. Google's figures, such as 52% MFU on Qwen 3 8B, are self-reported, and the post contains no independent benchmarks. [3]
- Google's Developers Blog announced vLLM-TorchTPU (with Inferact) and SGLang-TorchTPU (with RadixArk). Each keeps its framework's scheduler, KV cache manager and API server, and uses TPU Sync, an open-source C++ library, for KV transfer. Both are described as running in production with preview customers, and a Meta engineer is quoted saying SGLang-TorchTPU runs in production at Meta. Google cites a 2.01-cent per-million-token InferenceX figure for Qwen 3.5. It doesn't match the SemiAnalysis post's headline numbers, and the configuration behind it isn't given. [4]
- Demand outside Google is real but concentrated. Anthropic announced in October 2025 that it would use up to one million TPUs, bringing well over a gigawatt online in 2026, and tied the expansion to serving customer demand. Reuters, relaying The Information, reported a multi-year, multibillion-dollar Meta TPU rental; Meta and Google declined to comment. In May 2026 Google and Blackstone announced a TPU compute-as-a-service venture with $5B in Blackstone equity and about 500 MW targeted for 2027. [5][6][7]

## What it means

For builders, TPUs are now a realistic second source for serving open-weight models through vLLM or SGLang. The API layer your application calls stays the same, and the device and kernels change underneath. That is the 'portable API' half of the hypothesis, and it shipped in October 2026. The 'independent production' half is weaker. The named production users (Anthropic, Meta) are huge, strategically tied buyers with direct engineering access to Google. What matters for a typical team, namely latency SLOs, availability, quota and support outside that inner circle, has not been publicly measured.

In practice, treat the InferenceX TPU numbers as a hypothesis to test on your own workload, not a procurement answer. Rerun the public recipes. Compare at your target interactivity and TTFT, not at the most favorable point on the curve. Include Nvidia FP4 and disaggregated NVL72 setups, where SemiAnalysis concedes Nvidia leads. Check model coverage: architectures that don't fit the 256-wide MXU are harder to bring up. Because the inference engines are labeled preview and TorchTPU's code has only been public since October 20, budget for porting friction and keep a GPU fallback.

## Still open

- What latency, availability and p99 TTFT do external non-anchor customers get on Ironwood in production, compared with Google's internal serving? No public data was found.
- How do Google's '2.01 cents per million tokens' Qwen 3.5 figure and SemiAnalysis's $0.098–$0.181 figures relate: different latency targets, later runs or a different cost basis?
- Did the promised AgentX and accuracy results for TPU appear, and do they hold with MTP and disaggregation enabled on both sides?
- What are the pricing, quota and SLA terms for TPU capacity through the Blackstone venture and direct sales, and has Meta's outright purchase been confirmed?

## Sources

1. [TPU Inference Externalization Full Steam Ahead (SemiAnalysis InferenceX)](https://inferencex.semianalysis.com/blog/tpu-inferencex-full-steam) (primary)
2. [InferenceX About / methodology](https://inferencex.semianalysis.com/about) (primary)
3. [TorchTPU is now open source: Run native PyTorch on TPUs (Google Cloud Blog)](https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus) (primary)
4. [Announcing New vLLM and SGLang Inference Engines on TPU (Google Developers Blog)](https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu/) (primary)
5. [Expanding our use of Google Cloud TPUs and services (Anthropic)](https://anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services) (primary)
6. [Meta signs multi-billion-dollar deal to rent Google AI chips, The Information reports (Reuters via Yahoo Finance)](https://finance.yahoo.com/news/google-signs-multibillion-dollar-ai-232844608.html)
7. [Google opens TPUs to enterprises beyond its own cloud via Blackstone JV (Network World)](https://www.networkworld.com/article/4173075/google-opens-tpus-to-enterprises-beyond-its-own-cloud-via-blackstone-jv.html)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Google](https://newruntime.com/topics/google.md): Explore the google topic hub.
- topic hub: [Anthropic](https://newruntime.com/topics/anthropic.md): Explore the anthropic topic hub.
- topic hub: [Benchmarks](https://newruntime.com/topics/benchmarks.md): Explore the benchmarks topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)