---
type: "post"
id: "slq-statistically-lossless-quantization-of-large-language-models-df7e8d"
slug: "slq-statistically-lossless-quantization-of-large-language-models-df7e8d"
title: "SLQ: Statistically-Lossless Quantization of Large Language Models"
description: "IST-DASLab’s SLQ repository describes a quantization workflow that searches for low-bit LLM weights under explicit task-accuracy or next-token-distribution targets."
retrieval_nugget: "IST-DASLab’s SLQ repository describes a quantization workflow that searches for low-bit LLM weights under explicit task-accuracy or next-token-distribution targets. Its reported trade-off—less precision with measured fidelity and higher vLLM throughput—matters to teams deciding whether model-serving savings justify the evaluation and deployment work."
published_at: "2026-11-17"
updated_at: "2026-11-17"
record_date: "2026-11-17"
date_kind: "published_at"
topics: ["models"]
entities: ["IST-DASLab","SLQ","Michael Helcig","Eldar Kurtic","Dan Alistarh","vLLM","Humming"]
related_patterns: []
novelty_label: "new_angle"
what_changed: "The post sarcastically predicts a wave of papers that improve benchmark scores by tuning acceptance rates, task selection, and quantization."
prior_refs: []
source_url: "https://github.com/IST-DASLab/SLQ"
source_urls: ["https://github.com/IST-DASLab/SLQ"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:slq-statistically-lossless-quantization-of-large-language-models-df7e8d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/slq-statistically-lossless-quantization-of-large-language-models-df7e8d/","markdown":"https://newruntime.com/posts/slq-statistically-lossless-quantization-of-large-language-models-df7e8d.md","json":"https://newruntime.com/posts/slq-statistically-lossless-quantization-of-large-language-models-df7e8d.json"}
markdown_url: "https://newruntime.com/posts/slq-statistically-lossless-quantization-of-large-language-models-df7e8d.md"
---

# SLQ: Statistically-Lossless Quantization of Large Language Models

## Retrieval answer

IST-DASLab’s SLQ repository describes a quantization workflow that searches for low-bit LLM weights under explicit task-accuracy or next-token-distribution targets. Its reported trade-off—less precision with measured fidelity and higher vLLM throughput—matters to teams deciding whether model-serving savings justify the evaluation and deployment work.

## Summary
IST-DASLab’s SLQ repository describes a quantization workflow that searches for low-bit LLM weights under explicit task-accuracy or next-token-distribution targets. Its reported trade-off—less precision with measured fidelity and higher vLLM throughput—matters to teams deciding whether model-serving savings justify the evaluation and deployment work.

## What happened
IST-DASLab published SLQ, short for Statistically-Lossless Quantization of Large Language Models, under the Apache-2.0 license. The repository cites a COLM 2026 paper by Michael Helcig, Eldar Kurtic, and Dan Alistarh. SLQ is not presented as a single fixed-precision checkpoint. Instead, it searches for a per-layer weight-bitwidth allocation that reaches a fidelity target while minimizing the model’s average bitwidth.

The project defines two such targets. Task-Lossless, or TL, requires mean zero-shot benchmark recovery of at least 99% of the BF16 model. The repository characterizes that level as within the sampling variance of the BF16 model and reports achieving it at 3.3 to 4.7 bits per parameter. Distribution-Lossless, or DL, uses Expected Acceptance Rate, or EAR, to compare next-token distributions with BF16. Its stated target is EAR of at least 0.99; the repository notes that experiments in the paper use a model-dependent 0.985 to 0.99 range. The reported DL configurations use 5.0 to 6.6 bits per parameter.

The implementation begins by running layer-wise GPTQ at each bitwidth from 2 through 8. It uses asymmetric integer quantization with a group size of 128. It then builds a sensitivity database using linear and multi-bitwidth Shapley estimation. The documented default uses 512 sequences of 2,048 FineWeb-Edu calibration tokens and a top-K value of 10. An integer linear program allocates bitwidths across layers from the resulting cost table. A binary search then seeks the lowest average bitwidth satisfying the selected constraint.

For DL, the default search measures EAR with one forward pass on the calibration set at each binary-search step. An alternative can use EAR predicted from the cost table, avoiding model inference during the search. The TL route uses a measured anchor: the mean benchmark recovery of a uniform 4-bit model built from the same layer data and compared with BF16. According to the repository, the calibrated search needs two forward passes after that anchor and no further benchmark runs. It rejects a candidate if the ratio between measured and predicted KL divergence drifts by more than 2x. A separate benchmark-in-the-loop script evaluates the benchmark suite at every binary-search step for a more expensive validation path.

IST-DASLab reports that all SLQ models retained at least 99% of BF16 benchmark accuracy while serving 1.7 to 3.7 times faster in vLLM. The measurement is ShareGPT tokens per second per GPU on L40S hardware. The project requires Python 3.10 or later and CUDA GPUs. Its `humming_online` format needs a vLLM build with Humming kernels; a fake-quantized checkpoint format is also documented for stock vLLM or Transformers.

## What the archive adds
New Runtime’s earlier coverage, [“Measuring benchmark optimization in speech recognition”](https://newruntime.com/posts/measuring-benchmark-optimization-in-speech-recognition/) (September 17, 2026), described held-out evaluation as a way to distinguish capability from benchmark familiarity. SLQ is a new event, but it brings the same concern to serving optimization: a compression claim needs a defined fidelity test rather than a speed number alone.

## Trend signal
This supports the New Runtime pattern that verification bandwidth is the scarce engineering resource. Serving gains become useful only when a team can establish that a reduced-precision model remains acceptable for its intended work. SLQ makes verification an explicit part of the procedure through benchmark recovery, EAR, KL checks, and a guardrail for prediction drift. Those checks can reduce repeated evaluation, but they do not remove the need to validate a representative production workload.

## Why it matters
Quantization is often treated as a simple choice between smaller weights and worse outputs. SLQ offers a more operational framing: define what has to remain stable, then search for a layer-wise configuration that meets that boundary. That gives an infrastructure team something more actionable than comparing a uniform 4-bit checkpoint against BF16 and hoping aggregate scores reveal the relevant trade-off.

The practical benefit is potentially material where GPU memory, throughput, or serving cost constrains a deployment. The reported 1.7 to 3.7 times throughput range is not merely a weight-size claim; it is an end-to-end serving measurement on a named hardware and workload setup. The method also distinguishes two kinds of preservation. TL is oriented around benchmark recovery, while DL is oriented around the next-token distribution, which may be relevant when a team wants a tighter comparison to the base model’s behavior.

The cost is an engineering workflow rather than a checkpoint download. A team needs calibration data, an explicit threshold, an appropriate baseline, and a serving stack compatible with the chosen format. It must also decide whether the faster calibrated search is sufficient or whether the benchmark-in-the-loop mode is warranted. The repository’s value is therefore a repeatable decision process, not a general promise that low-bit inference will preserve every task a production system performs.

## Limits and uncertainty
The repository reports throughput on L40S GPUs under a ShareGPT workload, which does not establish the same performance on other accelerators, model architectures, batch profiles, context lengths, or request mixes. Its fidelity definitions are also tied to the selected zero-shot benchmarks and calibration data. A high recovery value or EAR does not independently demonstrate that a domain-specific workflow, tool call, or long-running agent task remains unchanged.

The reported Humming serving results depend on a vLLM build with Humming kernels. Teams using stock serving stacks should not assume identical throughput. The repository also does not establish how much compute, operator time, or repeated recalibration different model sizes and changing workloads require. Those costs matter when judging whether the optimization is worthwhile.

## Skill to build now
Create a quantization acceptance contract before producing a compressed checkpoint. Pick a BF16 baseline, identify a small set of critical tasks, and state a pass threshold such as benchmark recovery or an output-distribution measure. Use a calibration set that resembles the intended workload. If using a predicted search, add a measured check and reject a candidate when the measured quality diverges materially from the prediction. Only then compare throughput on the hardware and request pattern that will actually serve users.

## Reader takeaway
SLQ turns quantization from a generic precision setting into a constrained search with declared quality gates. For teams facing inference limits, its useful lesson is to set the fidelity boundary first, allocate precision where it matters, and treat a speedup as a production result only after verifying it against the workload that matters.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
