Vocab Break examines an apparent reversal in tokenizer design: Claude's current tokenizer may use roughly 15,000 entries while Qwen 3.8 uses about 250,000.
The proposed mechanism is a gradient bottleneck in the final language-model head. A very large vocabulary expands the space through which the training loss must pass, while the hidden dimension remains much smaller. A smaller vocabulary reduces that projection and can avoid rarely trained glitch tokens, but it produces more tokens for the same input.
The useful signal is a systems tradeoff, not a claim that small vocabularies always win. Tokenizer size changes training feedback, inference length, attention cost, memory, multilingual behavior, and price together. The next evidence is controlled model-quality and serving comparisons, not tokenizer reconstruction alone.