---
type: "post"
id: "clear-compared-tokenization-across-roughly-200-languages-and-e64ac3"
slug: "clear-compared-tokenization-across-roughly-200-languages-and-e64ac3"
title: "CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume"
description: "CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families."
retrieval_nugget: "CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families. Because many providers charge by token, the gap can affect inference cost, latency, and usable context for teams serving multilingual users."
published_at: "2026-11-03"
updated_at: "2026-11-03"
record_date: "2026-11-03"
date_kind: "published_at"
topics: ["claude","gemini","anthropic"]
entities: ["Coalition for Language Equity in AI Research","FLORES-200","GPT-5","NLLB-200","Gemini 3.8 Flash","Claude Fable 5","Qwen 3","DeepSeek V4.1 Flash"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "CLEAR’s research quantifies how tokenizer inefficiency makes equivalent text costlier across languages, including examples where Malayalam uses twice and Amharic six times the English tokens."
prior_refs: ["memory:34ab0d0b-4f37-8573-a113-0588c51367eb","memory:531e8012-0674-8e8d-afd1-6394fe02b7db"]
source_url: "https://clear-languages.org/blog/the-tokenizer-tax"
source_urls: ["https://clear-languages.org/blog/the-tokenizer-tax"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:clear-compared-tokenization-across-roughly-200-languages-and-e64ac3"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3/","markdown":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3.md","json":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3.json"}
---

# CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume

## Retrieval answer

CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families. Because many providers charge by token, the gap can affect inference cost, latency, and usable context for teams serving multilingual users.

## Summary
CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families. Because many providers charge by token, the gap can affect inference cost, latency, and usable context for teams serving multilingual users.

## What happened
In “The Tokenizer Tax,” published September 29, 2026, researchers from the Coalition for Language Equity in AI Research (CLEAR) examine how tokenizers turn text into the pieces a language model processes. A tokenizer that splits text into smaller fragments uses more tokens for equivalent content. Since providers commonly price usage by token, the authors argue that speakers of languages which tokenize inefficiently can face higher effective costs. They use Anthropic’s Fable 5 as an example of token-based pricing: $10 per million input tokens and $50 per million output tokens.

The article illustrates the issue with equivalent versions of the sentence “People should benefit from AI, regardless of the language they speak.” With GPT-5’s tokenizer, the English sentence takes 13 tokens, Malayalam takes 26, and Amharic takes 78. CLEAR says these versions have nearly the same number of characters. Malayalam words are split into more fragments; in the Amharic example, Ge’ez characters are split into two byte-level tokens each. At a fixed per-token rate, CLEAR frames the Amharic input as six times more expensive than the English input.

For a broader comparison, CLEAR used FLORES-200, a benchmark of professionally translated sentences in roughly 200 languages. It tokenized equivalent content with models from major labs. Its headline metric is the max/min token-count ratio: the difference between the highest and lowest token counts for the same content. It also reports a Gini coefficient as a distributional check. Its grading rubric assigns an A to ratios of 3x or less, B to up to 6x, C to up to 9x, D to up to 12x, and F above 12x.

The reported results differ materially by model family. CLEAR lists a 2.7x ratio for Meta’s NLLB-200 translation model and 5.1x for the listed Google Gemini and Gemma families. Its table reports 9.3x for Anthropic’s Claude Fable 5 and Opus 4.8, 9.5x for Meta’s Llama 4 Scout and Muse Glimmer, 10.5x for Qwen 3, 10.6x for DeepSeek V4.1 Flash, and 11.3x for Kimi K3. The table gives GPT-5 and gpt-oss a 13.8x result, while the highest result in the study reaches 17x. CLEAR says no general-purpose frontier model in its analysis earns better than a B, and most receive an F.

Among the 20 languages with the greatest web presence, CLEAR reports ratios between 1.5x and 3.1x, with one exception. Across FLORES-200, the range is 2.7x to 17x. The article says token count can affect token-billed cost, processing time, and how much original text fits in a fixed context window. CLEAR recommends that developers publish disparity measures in model cards, account for more languages in future tokenizer design, and consider character-normalized pricing in the interim. This is an analysis and a set of recommendations, not a provider pricing or product announcement.

## What the archive adds
We have not covered this before; it opens a new thread.

## Trend signal
No supplied SITE_PATTERNS pattern or FIELD_NOTE_MEMORY assumption directly addresses unequal tokenization cost across languages.

## Why it matters
For a team building a multilingual product, an English-only token estimate can misstate operating cost and the amount of source material that fits into a model context window. The practical response is to measure representative inputs in the languages a product actually serves, across the models under consideration. Tokenization efficiency becomes another model-selection variable alongside answer quality, listed price, throughput, and operational constraints.

The study also gives procurement and product teams a concrete question for vendors: whether they can provide language-specific tokenization or usage data rather than a single nominal rate. Its comparison does not mean every language or workload will encounter the worst reported ratio. It does show that a universal per-token price is not automatically a language-neutral price in use.

## Limits and uncertainty
CLEAR’s figures come from tokenizing translated sentences in FLORES-200. The supplied material does not establish that the same ratios apply to every prompt, domain, tokenizer version, or production workload. The max/min metric deliberately emphasizes the worst-served language, and the authors note that it can produce a different picture from the Gini coefficient.

Token count is not a measure of response quality, end-to-end latency, or total application cost. The article does not document any provider changing a tokenizer, adopting character-based pricing, or implementing CLEAR’s recommendations. Model labels and grades are the study’s reported comparisons, not an independent audit of every available configuration.

## Skill to build now
Add a tokenization check to multilingual evaluation. Select several real prompts, documents, and expected outputs for each supported language; run them through the candidate model tokenizers; then compare token counts, cost projections, and context consumption for equivalent tasks. Record tokenizer and model versions with the results. This produces a decision input that is more useful than extrapolating from English traffic alone.

## Reader takeaway
Per-token billing can conceal a language-dependent cost multiplier. Before committing to a model or setting multilingual pricing expectations, test representative text in every material user language and evaluate token counts alongside quality and latency.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- topic hub: [Anthropic](https://newruntime.com/topics/anthropic.md): Explore the anthropic topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)