The engines retain each framework’s worker model, scheduler, KV-cache manager and API server while adding TPU implementations beneath them. Google lists prefix caching, host and Managed Lustre KV-cache offload, disaggregated prefill/decode, multimodal support and long-context features. The company says the engines are running with preview customers and reports public benchmark results; these are company-reported claims, not independent validation. Source: https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu
Announcing New vLLM and SGLang Inference Engines on TPU
In one minute
Google announced native PyTorch backends for vLLM and SGLang on TPU, built on TorchTPU in collaboration with Inferact and RadixArk.
What changed: Google now announces the launch of native PyTorch vLLM and SGLang inference engines on TPU, extending the earlier partnership teaser about TPU kernels and TorchTPU.