Field noteNov 17, 20265 min read1 source

TPU Model Design Considerations and Optimization Guide

In one minute

Google has published an open guide to co-designing model architectures and workloads for its TPU hardware, with rules for dimensions, routing, parallelism and serving. For teams building on TPUs, the useful shift is from generic model choices to hardware-aware decisions tested in wall-clock ablations, not assumed from FLOPs alone.

What changed: Google has published an open-source TPU model-design guide with hardware-specific rules for shapes, architecture, and parallelism, including guidance for Ironwood and future TPU generations.

What happened

Google’s Developers Blog published “TPU Model Design Considerations and Optimization Guide” on November 16, 2026. Staff Software Engineer Gagik Amirkhanyan and Software Engineer Vaibhav Singh introduce the open-source AI-Hypercomputer/tpu-codesign guide. Its stated focus is training and inference for LLMs, mixture-of-experts (MoE) models and other hardware-intensive workloads, with designs for Ironwood, also called TPU v7x, and principles intended to carry forward to TPU 8t and TPU 8i. Google says the companion repository will grow model- and workload-specific sections over time.

The recommendations fall into three groups. First, shape rules: align reduction and output dimensions to multiples of 256 to match the TPU’s 256×256 systolic array, and use static tensor shapes. For dynamic requests, Google recommends a small set of precompiled sequence-length and batch-size buckets, masking, and fixed-shape buffers that represent variable data with metadata such as lengths and offsets. The guide highlights head dimension as a commonly missed case: it recommends 256, with proportionally fewer query and key-value heads to preserve total projection widths. For teams adapting an existing checkpoint, Google says its parallelism advice can be applied without changing the checkpoint.

Second, the guide presents architecture choices that should be validated with ablations because they can affect model quality. One is LatentMoE: project routed tokens to a narrower latent width before dispatching them to experts, then project back after combining them. Google recommends keeping the latent width divisible by 256 and describes FP8 dispatch with BF16 projections. In its Ironwood profiling, a 30B, 128-expert MoE with latent width 768 cut on-device permute time 2.8×, increased end-to-end training throughput 14% in tokens per second, and freed about 25 GB of activation HBM per device when rematerialization checkpoints were anchored on the compressed representation. The post says the throughput result was 322 versus 270 TFLOP/s per chip, or 19% on that measure, because the latent projections add roughly 5% FLOPs per token.

Third, Google covers expressing parallelism: map communication-heavy axes to physical links, choose tensor, expert, context and data parallelism for the target topology, and profile rather than assume a layout wins. It also discusses static-shape handling for ragged MoE routing and paged attention, speculative decoding, SparseCore offload, and keeping vector-unit work hidden behind matrix operations. The guide gives examples of generation-specific differences: TPU 8i’s Boardfly topology and Collectives Acceleration Engine, and TPU 8t’s native FP4. It cautions that thresholds measured on Ironwood, including a block-size threshold for quantization scaling, should be remeasured on new hardware.

Google’s overarching measurement advice is to compare wall-clock performance on a tuned software stack rather than rely on paper FLOPs. The post reports that systems tuning alone raised an open 30B MoE from 12% to 23% MFU in its Ironwood benchmarks, before any model change. It points readers to the repository’s diagnostics guide, reference configurations and profiling resources.

What the archive adds

New Runtime’s earlier coverage on TorchTitan and TPU model optimization (October 22, 2026) focused on compiler-visible training structure. This guide broadens that thread into a set of hardware-grounded model and workload design rules. It is a new publication on a known TPU optimization story, not a restatement of the earlier note.

Trend signal

This does not yet fit a pattern New Runtime tracks. The guide offers concrete hardware-specific optimization advice, but it does not establish a broader shift in agent systems, interfaces or organizational practice.

Why now

Google says the core principles carry forward to TPU 8t and TPU 8i, including the newly announced TPU 8tx and TPU 8i Max. It also says follow-up posts will cover eighth-generation designs in depth, positioning this guide as a foundation while those systems arrive.

Why it matters

For teams already committed to TPU, the guide turns hardware constraints into decisions they can test early: head dimensions, static-shape strategy, expert dispatch, and parallelism mapping. Those choices can affect utilization, memory pressure, training throughput and serving latency; some, such as latent routing, also change model structure and need quality checks. Google’s own results reinforce why profiling matters: software-stack tuning can move utilization substantially before an architecture change is attempted. The practical value is not that one recipe will work everywhere, but that teams have explicit hypotheses and measurements to use against their own model, concurrency and hardware generation.

Limits and uncertainty

The performance figures are Google’s measurements on Ironwood systems and specific open models, shapes and configurations; they are not independent benchmarks or guarantees for other workloads. The post explicitly says exact percentages are configuration-specific, even when trends may transfer. Architecture changes such as LatentMoE require quality ablations, while topology and kernel thresholds can change across TPU generations. The guide does not establish that its recommendations outperform alternatives on a reader’s own model or software stack; teams must profile and validate those outcomes.

Skill to build now

Build a small wall-clock ablation loop before changing model architecture: establish a tuned baseline, change one shape or parallelism choice at a time, and record throughput, memory and quality on the target workload. Start with shape alignment and static buckets; test structural options such as LatentMoE only with quality checks. Keep the hardware generation and concurrency fixed in each comparison.

Reader takeaway

Treat TPU model design as a hardware-and-software co-design problem, not a model architecture exercise in isolation. Apply the low-risk shape and static-buffer rules first; validate quality-sensitive choices and parallelism with measured wall-clock results on the actual target system.

Evidence / sources

  1. 1developers.googleblog.com/tpu-model-design-considerations-and-optimization-guide
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicGoogleExplore the google topic hub.
  2. 02archiveField NotesOpen the latest editorial analysis.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract