{"type":"post","id":"clear-compared-tokenization-across-roughly-200-languages-and-e64ac3","slug":"clear-compared-tokenization-across-roughly-200-languages-and-e64ac3","title":"CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume","description":"CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families.","retrieval_nugget":"CLEAR compared tokenization across roughly 200 languages and found that equivalent content can consume sharply different numbers of tokens across model families. Because many providers charge by token, the gap can affect inference cost, latency, and usable context for teams serving multilingual users.","published_at":"2026-11-03","updated_at":"2026-11-03","record_date":"2026-11-03","date_kind":"published_at","topics":["claude","gemini","anthropic"],"entities":["Coalition for Language Equity in AI Research","FLORES-200","GPT-5","NLLB-200","Gemini 3.8 Flash","Claude Fable 5","Qwen 3","DeepSeek V4.1 Flash"],"related_patterns":["agent-economics-moves-to-completed-work"],"novelty_label":"new_fact","what_changed":"CLEAR’s research quantifies how tokenizer inefficiency makes equivalent text costlier across languages, including examples where Malayalam uses twice and Amharic six times the English tokens.","prior_refs":["memory:34ab0d0b-4f37-8573-a113-0588c51367eb","memory:531e8012-0674-8e8d-afd1-6394fe02b7db"],"source_url":"https://clear-languages.org/blog/the-tokenizer-tax","source_urls":["https://clear-languages.org/blog/the-tokenizer-tax"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:clear-compared-tokenization-across-roughly-200-languages-and-e64ac3","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3/","markdown":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3.md","json":"https://newruntime.com/posts/clear-compared-tokenization-across-roughly-200-languages-and-e64ac3.json"},"next_reads":[{"type":"topic","path":"/topics/claude/","url":"https://newruntime.com/topics/claude/","title":"Claude","reason":"Explore the claude topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/claude.md","json_url":"https://newruntime.com/topics/claude.json"},{"type":"topic","path":"/topics/gemini/","url":"https://newruntime.com/topics/gemini/","title":"Gemini","reason":"Explore the gemini topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/gemini.md","json_url":"https://newruntime.com/topics/gemini.json"},{"type":"topic","path":"/topics/anthropic/","url":"https://newruntime.com/topics/anthropic/","title":"Anthropic","reason":"Explore the anthropic topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/anthropic.md","json_url":"https://newruntime.com/topics/anthropic.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}