GSQ proposes learned scalar quantization for low-bit LLM deployment
An arXiv paper introduces Gumbel-Softmax Quantization (GSQ), a post-training scalar quantization method that jointly learns per-coordinate grid assignments and per-group scales. The authors report that it closes most of the accuracy gap to QTIP at 2 and 3 bits on Llama 3.1 8B and 70B Instruct, while using a symmetric scalar grid compatible with existing scalar inference kernels.
The paper also reports applying GSQ to public GGUF checkpoints and projecting results back into the same deployment format, as well as scaling the method to Kimi-K2.5, a trillion-scale mixture-of-experts model. These are reported experiments, not independent replication. The supplied source does not establish end-to-end latency, memory use, or quality in particular application workloads.