Google's TurboQuant Compresses LLM Key-Value Caches to 3 Bits With Zero Accuracy Loss

Google Research published TurboQuant, a new quantization algorithm that compresses LLM key-value (KV) caches to just 3 bits per value — without retraining, without fine-tuning, and without measurable accuracy loss. The work will be presented at ICLR 2026.

The KV cache is the memory a transformer model keeps of past tokens during inference. At long context lengths, it dominates memory usage. TurboQuant’s claim is aggressive: a 6x reduction in KV memory footprint, plus an 8x speedup in computing attention logits on H100 GPUs using 4-bit compression compared to 32-bit unquantized keys.

TurboQuant is built on two component algorithms, both published alongside it:

QJL (Quantized Johnson-Lindenstrauss) maps high-dimensional vectors to sign bits (+1 or -1) using the Johnson-Lindenstrauss transform. The resulting representation requires zero memory overhead. A precision-balanced estimator then reconstructs accurate attention scores from the compressed form.

PolarQuant takes a different angle — literally. It converts vectors from Cartesian to polar coordinates, replacing (X, Y, Z) distances with radius and angle. Because the angular distribution is predictable and concentrated, the normalization step becomes unnecessary, eliminating the memory overhead that conventional quantization methods carry.

TurboQuant combines both: QJL handles the bulk compression, PolarQuant handles the residual structure.

Google evaluated all three algorithms on Gemma and Mistral across LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval. On the needle-in-haystack benchmark — which tests whether a model can retrieve a single specific fact from a very long context — TurboQuant achieves perfect downstream results while reducing KV memory by at least 6x. PolarQuant is described as “nearly lossless” on the same task.

In vector search — a separate application from KV caching — TurboQuant outperforms state-of-the-art baselines PQ and RabbiQ on the 1@k recall ratio, despite those baselines using larger codebooks and dataset-specific tuning.

Why this matters: memory is the primary bottleneck for running long-context models efficiently. Serving a model with a 128K-token context requires enormous KV cache memory, which limits throughput and raises inference cost. A 6x compression ratio with no accuracy penalty directly translates to either serving more concurrent users at the same cost, or running the same load on cheaper hardware.

The “data-oblivious” property is equally important. TurboQuant doesn’t need to see training data to calibrate — it applies at inference time. That means it can be dropped into existing model deployments without modification.

No production release date was announced. The paper is available via Google Research, and the ICLR 2026 presentation will follow in April.