---
type: "post"
id: "paper-4-bit-n-gram-embedding-tables-cut-1822b8"
slug: "paper-4-bit-n-gram-embedding-tables-cut-1822b8"
title: "Paper: 4-bit n-gram embedding tables cut table memory about 75% while keeping over 99% of their gains"
description: "Stephen Chung and Wenyu Du have posted “Low-Bit N-Gram Embeddings for Language Models” on arXiv."
retrieval_nugget: "Stephen Chung and Wenyu Du have posted “Low-Bit N-Gram Embeddings for Language Models” on arXiv."
published_at: "2026-10-24"
updated_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "published_at"
topics: ["models"]
entities: ["Low-Bit N-Gram Embeddings for Language Models","DeepSeek-V4.1-Flash","Qwen3.8-Flash-Next","Engram","Stephen Chung","Wenyu Du"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "This paper applies low-bit quantization specifically to n-gram embedding tables, reporting roughly 75% memory savings with over 99% of their validation-loss gains preserved."
prior_refs: ["memory:521fdd49-c9d4-85dd-a192-e77689c0e788","memory:701eb295-ddc9-89c3-a0d9-a63223e6ed03","memory:80dacccd-109c-8790-a5d7-e29633b6ccd1","memory:dbf688b1-4b4f-82e3-a6a0-4cd6b08e021c","memory:20682048-be1f-868d-a70b-60a37aff5b6c","memory:6cfedb1f-6c4e-86f3-af75-44dae943e030","memory:534cf4a8-4549-88c9-af9b-9c8dc69b2c4f","memory:f084c1be-ff40-8733-af8a-d94b52b26837"]
source_url: "https://arxiv.org/abs/2610.31646"
source_urls: ["https://arxiv.org/abs/2610.31646"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:paper-4-bit-n-gram-embedding-tables-cut-1822b8"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/paper-4-bit-n-gram-embedding-tables-cut-1822b8/","markdown":"https://newruntime.com/posts/paper-4-bit-n-gram-embedding-tables-cut-1822b8.md","json":"https://newruntime.com/posts/paper-4-bit-n-gram-embedding-tables-cut-1822b8.json"}
---

# Paper: 4-bit n-gram embedding tables cut table memory about 75% while keeping over 99% of their gains

## Retrieval answer

Stephen Chung and Wenyu Du have posted “Low-Bit N-Gram Embeddings for Language Models” on arXiv.

Stephen Chung and Wenyu Du have posted “Low-Bit N-Gram Embeddings for Language Models” on arXiv. The paper examines low-precision storage for n-gram embedding tables: learned representations of short consecutive token sequences used in some open-weight language models.

Across nanochat-, Llama-, and GPT-2-family models trained from scratch, the authors report that four-bit tables reduce table memory by about 75% while retaining more than 99% of the validation-loss improvement attributed to the tables. At lower bit widths, quality falls further; a short fine-tuning phase recovered a significant part of that loss in their experiments.

For DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, the paper reports 47–55% table-memory savings, perplexity increases below 1.2%, and largely preserved downstream-task performance. It also notes that DeepSeek-V4.1-Flash allocates 196 billion parameters to Engram n-gram memory. The paper does not establish production latency, hardware costs, implementation availability, or results beyond the evaluated models and tasks. Source: https://arxiv.org/abs/2610.31646

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)