What happened
The IST-DASLab repository presents Disaggregated Quantization (DQ) as a way to specialize computation formats, weight encodings, and storage placement for the two phases of LLM inference: prefill, which processes the prompt, and decode, which generates output tokens. The authors' premise is that the phases have different bottlenecks. Hardware-native low-precision arithmetic can accelerate prefill, while compact weight encodings can reduce memory traffic during generation.
The codebase describes three arrangements. Format disaggregation shares weights while using native NVFP4 prefill and weight-only decode. Full disaggregation uses separate prefill and decode weights, adding a checkpoint but supporting stronger low-bit accuracy. Offloaded disaggregated prefill (ODP) streams the added prefill weights from SSD through reusable buffers, avoiding additional device weight residency while adding storage and loading latency. The training approach, quantization-aware distillation with disaggregation (QADD), uses the supervised fine-tuning token mask to select phase-specific pathways and response targets; gradients reach the prefill pathway through representations consumed by decode. The repository says the method can work with shared or separate weights, or with a frozen external decoder.
The repo reports released Qwen3.8-27B NVFP4 prefillers for use with existing Unsloth GGUF decoders. On that setup, IQ1_S accuracy rose from 29.04% to 61.54% on MMLU-Pro and from 24.39% to 59.65% on MMMU-Pro, with no change to the decoder. The repository also says text-only training transferred to visual reasoning, and that the strongest gains were at low bitwidths; some higher-precision formats lost a little MMMU-Pro accuracy. These evaluations use disaggregated vLLM serving with GGUF weights dequantized to BF16, while the size axis refers to encoded GGUF backbone weights rather than evaluation-backend allocations.
ODP's separate latency results use a custom llama.cpp integration on DGX Spark. For Qwen3.8-27B at 8K context, reported time to first token fell from 12.27 seconds to 6.90 seconds. Across measured 4K–32K contexts, the repo reports 1.38×–1.78× speedups over native llama.cpp inference with Unsloth IQ1_S. Shorter prompts remained slower because of SSD loading. The core experiments also cover Qwen 3 models from 0.6B to 8B and Gemma 3 models from 270M to 12B, using NVFP4, LUT3, and LUT2 formats. The authors report that full disaggregation improves low-bit accuracy across decode-heavy and prefill-heavy tasks, with larger gains on prefill-heavy tasks. Reproduction scripts target a SLURM/container cluster, not a standalone laptop.
What the archive adds
We have not covered this before; it opens a new thread. An earlier New Runtime note on local inference argued that fitting a model into hardware does not by itself make it usable, but that note is not a direct report on this method.
Trend signal
This supports agent economics moves to completed work only weakly: lower latency and better low-bit accuracy could change inference cost, but the repository does not measure cost per verified task or agent completion rates. The more immediate signal is a hardware- and workload-specific optimization, not proof of a new economic unit.
Why it matters
For teams testing small-memory or low-bit deployments, DQ challenges the assumption that a single quantization format must serve every inference phase. If prompt processing and generation stress different resources, specializing their representations may create a useful accuracy/latency trade-off without replacing an existing decoder. ODP's SSD streaming is particularly relevant where device memory is constrained, although it exchanges extra GPU-resident weights for disk storage and loading overhead.
The practical implication is to evaluate the whole serving path, rather than infer production benefit from a benchmark score alone. Accuracy was measured in one serving stack, while the highlighted 27B latency numbers came from a separate custom llama.cpp integration. Teams should test their own model, hardware, context lengths, batching, and workload mix before planning around these gains. The repository explicitly does not evaluate highly batched serving, multi-turn cache rebuilds, or agentic behavior, so its numbers do not establish throughput or end-to-end task improvement in those settings.
Limits and uncertainty
The headline figures are the authors' measurements, not independent replications. Accuracy and latency were measured in different serving setups, and the reported context-length and hardware conditions matter to interpreting the results. The repo says error bars for core results represent variation across five late QADD checkpoints, not independent training seeds. It also notes that ODP avoids extra device weight residency, not KV-cache or other runtime allocations, and still requires storing the prefill checkpoint on SSD. Performance under highly batched serving, multi-turn cache rebuilds, and agentic workloads has not been evaluated in the repository's stated experiments.
Skill to build now
A useful first experiment is a small, controlled comparison of a weight-only quantized decoder against the corresponding DQ setup on the same model, hardware, prompt set, and context lengths. Track quality and time to first token separately, and record the serving backend and memory allocations. The repository's scripts require adapting cluster, container, and filesystem settings, so treat reproduction as a cluster experiment rather than a laptop install.
Reader takeaway
Disaggregated quantization makes a concrete case for choosing formats by inference phase: it reports large low-bit accuracy gains and lower time to first token in particular setups. Those results are promising for constrained inference, but they are not yet evidence of general serving or agent-level gains. Reproduce them on your own workload before treating them as an operational improvement.