---
type: "post"
id: "launching-a-trillion-parameter-model-at-sakana-ai-70c04b"
slug: "launching-a-trillion-parameter-model-at-sakana-ai-70c04b"
title: "Launching a trillion-parameter model at Sakana AI"
description: "Modal published a production account of serving Sakana AI's roughly one-trillion-parameter Namazu model."
retrieval_nugget: "Modal published a production account of serving Sakana AI's roughly one-trillion-parameter Namazu model. The case matters because it frames efficient large-model inference as a stack problem: language-aware quantization, traffic-specific speculation, kernel work, cache-aware routing, and elastic capacity all affect the API a builder actually experiences."
published_at: "2026-10-28"
updated_at: "2026-10-28"
record_date: "2026-10-28"
date_kind: "published_at"
topics: ["harness-engineering","agent-runtime"]
entities: ["Modal","Sakana AI","Sakana Namazu","NVIDIA Blackwell","DFlash","DFlash2"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "Adds a Sakana AI deployment: Modal tuned NVFP4 quantization for Japanese-language retention in the ~1T-parameter Sakana Namazu model; priors concern coding-agent inference scale."
prior_refs: ["memory:8e511205-51d4-85cb-ad54-6ad39ce7843c","memory:dd102e3a-6227-864d-a629-ee188fdc49d2"]
source_url: "https://modal.com/blog/sakana-case-study"
source_urls: ["https://modal.com/blog/sakana-case-study"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:launching-a-trillion-parameter-model-at-sakana-ai-70c04b"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/launching-a-trillion-parameter-model-at-sakana-ai-70c04b/","markdown":"https://newruntime.com/posts/launching-a-trillion-parameter-model-at-sakana-ai-70c04b.md","json":"https://newruntime.com/posts/launching-a-trillion-parameter-model-at-sakana-ai-70c04b.json"}
---

# Launching a trillion-parameter model at Sakana AI

## Retrieval answer

Modal published a production account of serving Sakana AI's roughly one-trillion-parameter Namazu model. The case matters because it frames efficient large-model inference as a stack problem: language-aware quantization, traffic-specific speculation, kernel work, cache-aware routing, and elastic capacity all affect the API a builder actually experiences.

## Summary
Modal published a production account of serving Sakana AI's roughly one-trillion-parameter Namazu model. The case matters because it frames efficient large-model inference as a stack problem: language-aware quantization, traffic-specific speculation, kernel work, cache-aware routing, and elastic capacity all affect the API a builder actually experiences.

## What happened
On October 23, 2026, Modal described work with Sakana AI to put Sakana Namazu into production. Namazu is a roughly one-trillion-parameter mixture-of-experts model intended for Japanese business context, live web search, and code execution. Modal says the engagement began before the public API launch, when Sakana needed to serve the model without committing in advance to reserved GPU capacity. The article is a vendor customer story, but it provides specific implementation choices and measurements rather than only a deployment announcement.

The first problem was memory. A model of this size at FP16 or BF16 requires a very large GPU cluster, so Modal used NVFP4 on NVIDIA Blackwell GPUs. It did not quantize every component. Routed experts, which Modal says contain most of the parameters, were quantized, while attention, the router, and shared experts remained in BF16. Calibration used conversations in Namazu's style and was weighted toward Japanese. Modal says this ensured representative traffic for each of the model's 23,040 experts before scales were fixed, and that Japanese instruction-following performance matched Sakana's production checkpoint within the noise floor of Sakana's published benchmarks.

Modal then replaced a generic speculative-decoding draft model with a custom DFlash model trained on representative Namazu traffic. Its key claim is that training data retained the exact token IDs emitted by the model. Decoding text and tokenizing it again can alter token boundaries in tool calls, JSON schemas, and reasoning markers, reducing the likelihood that the main model accepts drafted tokens. Modal built a capture path from production token IDs to a corpus exceeding 30 billion tokens. Compared with a generic drafter whose acceptance was below 40%, the custom version reportedly raised speculative token acceptance by 25%, cut per-token decode latency by about 35%, and allowed about 40% more traffic per replica before queue latency rose.

After public traffic arrived, the teams trained a second drafter, DFlash2, from roughly one million real requests. In a replay of 1,000 held-out requests using customer sampling settings, average accepted draft length rose from 3.51 to 5.01 and p99 latency fell from 68 seconds to 52 seconds. Modal also reports removing a CPU-to-GPU synchronization stall, moving NVFP4 expert operations to faster kernels, caching repeated prefill setup, and making speculative-token acceptance one shared decision across four tensor-parallel GPUs. Finally, its KV-aware router directed requests toward replicas likely to hold matching prompt prefixes in cache. Modal says multi-minute time-to-first-token spikes disappeared and queues approached zero, while decode speed was unchanged.

## What the archive adds
This is a new operational fact on a known trillion-model serving thread. [How to serve trillions of tokens for trillion](https://newruntime.com/posts/how-to-serve-trillions-of-tokens-for-trillion-6b3dbd/) argued that reuse of overlapping long context can dominate coding-agent inference economics. [How Cerebras Deploys Trillion-Parameter AI Models](https://newruntime.com/posts/how-cerebras-deploys-trillion-parameter-ai-models-06db8d/) covered a different deployment approach for Kimi K3. The Modal account adds a Japanese MoE API case where quantization, speculative decoding, and routing are co-optimized together.

## Trend signal
This supports [Harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/). Namazu's model weights are only one part of the delivered service. Token capture, drafter training, kernel selection, cross-GPU correctness, grammar-constrained output, routing, and autoscaling determine latency and usable capacity. The source does not establish that this architecture transfers unchanged to other models, but it gives a concrete example of the serving harness becoming a consequential product boundary.

## Why now
The timing follows an API launch. Before Sakana Namazu had public requests, Modal trained a first drafter on representative traffic; after launch, it could retrain on roughly one million real requests and test the newer drafter against held-out production-like traffic. The account therefore captures the transition from pre-launch approximation to feedback from actual request distributions.

## Why it matters
Teams evaluating a large-model API should not treat quantization or speculative decoding as isolated switches. The source suggests that their gains depend on the target distribution: language register, structured tool calls, sampling behavior, and cached prefixes all matter. For builders exposing JSON-schema tool interfaces, the detail about preserving raw token IDs is especially useful. A text-only training pipeline can silently change token boundaries that affect constrained generation and draft acceptance.

The capacity lesson is also practical. Modal presents elastic GPUs as an alternative to reserving a launch-sized cluster before demand is known. That may reduce initial capacity risk, although the article does not disclose pricing, availability guarantees, or the cost of on-demand capacity. The relevant engineering question is not only tokens per second, but whether the complete system holds tail latency and correctness under the request mix customers send.

## What happens next
- **Traffic-trained speculative drafters will become a standard iteration after major model API launches.** (6 months, 65%). Watch for: providers publishing acceptance and tail-latency results from real request traces.
- **Structured-output paths will receive the same serving optimizations as unconstrained generation.** (12 months, 60%). Watch for: explicit grammar-constrained speculative-decoding support in production inference stacks.

## Limits and uncertainty
The evidence is Modal's own customer case study and the reported outcomes are not independently audited. It does not provide absolute throughput, GPU counts, cost per token, model-quality tables, availability results, or a comparison with other serving platforms. Several figures are directional or approximate, and the p99 replay result covers 1,000 held-out requests rather than a disclosed long-run production sample. The article also does not establish whether its NVFP4 calibration recipe, DFlash training method, or routing behavior generalizes beyond Namazu, Japanese traffic, four-GPU tensor parallelism, and Modal's infrastructure.

## Skill to build now
Instrument an inference service so it stores privacy-reviewed, access-controlled token-level traces alongside request class, sampling settings, schema use, cache-hit signals, and latency. Then split evaluation by those conditions rather than relying on one aggregate benchmark. This week, test whether decoding and re-tokenizing your own structured outputs changes token sequences, and measure whether that change affects speculative acceptance or schema-constrained latency.

## Reader takeaway
A trillion-parameter model becomes a usable API through serving details that are easy to overlook: selective precision, traffic-specific draft models, exact token alignment, cross-device verification, cache-aware routing, and capacity policy. For teams building with AI, production behavior is a property of that whole system, not just the checkpoint.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Harness Engineering](https://newruntime.com/topics/harness-engineering.md): Explore the harness-engineering topic hub.
- topic hub: [Agent Runtime](https://newruntime.com/topics/agent-runtime.md): Explore the agent-runtime topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)