The paper examines low-precision storage for n-gram embedding tables: learned representations of short consecutive token sequences used in some open-weight language models.
Across nanochat-, Llama-, and GPT-2-family models trained from scratch, the authors report that four-bit tables reduce table memory by about 75% while retaining more than 99% of the validation-loss improvement attributed to the tables. At lower bit widths, quality falls further; a short fine-tuning phase recovered a significant part of that loss in their experiments.
For DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, the paper reports 47–55% table-memory savings, perplexity increases below 1.2%, and largely preserved downstream-task performance. It also notes that DeepSeek-V4.1-Flash allocates 196 billion parameters to Engram n-gram memory. The paper does not establish production latency, hardware costs, implementation availability, or results beyond the evaluated models and tasks. Source: https://arxiv.org/abs/2610.31646