Vectors are the basic tools AI models use to process information. Simple vectors describe points while high-dimensional embeddings capture complex details, dash-like features of an embedding image, or meanings of words but use much more memory. This can slow down the key value cache, A Fast storage for frequently accessed data, so the computer has to search a slower database.
Vector quantization is a classic data compression method that reduces the size of multidimensional vectors. This helps AI in two ways. It speeds up vector search, which is the first technology behind large-scale AI and search engines, and it reduces key‑value cache slowdowns by making key‑value pairs smaller. This means faster searches and lower memory cost. However, traditional vector quantization often incurs a memory cost because most methods must compute and store precise quantization constants for each small block of data. This can add 1 or 2 extra bits per number, partly defeating the purpose of compressing the data.
Today, we are introducing TurboQuant, a new compression algorithm that addresses the memory load value. In vector quantization, TurboQuant uses two other methods: quantized Johnson Lindenstrauss (QJL) and polar quant to achieve its results. In our test, all three techniques help reduce key-value bottlenecks without lowering AI model effectiveness. This could have a big impact on any use case that relies on compression, especially in search and AI.
How TurboQuant Works
TurboQuant is a compression method that significantly reduces model size without sacrificing accuracy. It is suitable for both key value compression and vector search. Its approach involves two main steps. Each builds on concepts introduced by Polar Font and QJL.
- High-value compression (the polar quant method): Turboquant starts by randomly rotating data vectors. This clever step simplifies the data geometry, making it easy to apply a standard, high-quality quantizer (a tool that maps a large set of continuous variables, such as precise decimals, to a smaller discrete set of symbols or numbers, such as integers) to each part of the vector individually. This first step uses most of the compression power (the majority of the bits) to capture the main concept and strength of the source vector.
- Next, TurboQuant applies QJL to the remaining error using a single bit, thereby improving the accuracy of the attention score.
To better understand TurboQuant’s efficiency, let’s examine the specific roles that QJL (quantized Johnson Lindenstrauss) and Polar Quant (polar coordinate quantization) play in its two-step process.
QJL The Zero Overhead 1-Bit Trick
QJL uses the Johnson–Lindenstrauss transform, a mathematical method, to compress high-dimensional datasets while preserving important distance relationships between data points. It converts each vector element to a single sign bit, either +1 or −1. This creates a fast shorthand that requires no extra memory. To keep the results accurate, QJL uses a special estimator that bridges a high-precision query with a simpler, lower-precision dataset. This helps the model accurately calculate the attention score, which determines which parts of its input matter most and which can be ignored.
PolarQuant: A New Angle Of Compression
Polar Quant solves the memory cost problem in a different way. Instead of using standard coordinates like x, y, and z to represent distance along each axis, Polar Quant converts the vector to polar coordinates. This is like saying go five blocks at a 37-degree angle instead of going three blocks east and four blocks north. This gives two pieces of information. The radius shows the strength of the data, and the angle shows its direction and/or meaning because the angles follow a known pattern. The model does not need to perform a costly data normalization step. It maps data onto a fixed, predictable circular grid with predefined boundaries, rather than a square grid whose boundaries keep changing. This lets Polar Quant avoid the memory overhead of older models.
Experiments And Results
We tested all three algorithms on standard long-context benchmarks, including Long Bench, Needle in a Haystack, Zero Scrolls, RULER, and L-Eval, using open-source LLMs such as GAMA and Mistral. TurboQuant delivers top performance across both dot-product distortion and recall while using less key-value (KV) memory. The chart below summarizes how TurboQuant, PolarQuant, and KIVI baselines performed on tasks such as question answering, code generation, and summarization.
The chart below shows how the algorithms performed on long-context needled-in-hashtag tasks, which test a model’s ability to find information hidden in large text. TurboQuant achieved perfect results across all benchmarks and reduced the key-value memory footprint by at least 6x. PolarQuant performed almost as well for this task.
TurboQuant can reduce the key-value cache to just 3 bits without any training or fine-tuning, while maintaining model effectiveness. It also runs faster than the first LLMs (Gemma and Mistral). TurboQuant is easy to implement and adds almost no extra runtime. The plot below shows that 4-bit TurboQuant A can be up to 8× faster than 32-bit unquantized keys on H100 GPUs.
This makes it ideal for supporting use cases. This makes TurboPoint a great fit for jobs like vector search, where it can speed up index building. We tested TurboQuant with high-dimensional vector search against top methods like PQ and Rabbi Q using 1@K recall ratio. This ratio shows how often the algorithm correctly identifies the true top inner product result among its top K guesses. TurboQuant consistently achieved higher recall ratios than the baseline techniques, even though the baselines use large codebooks and require tuning for each dataset. This shows that TurboQuant is both strong and efficient for large-scale search shifts in high-dimensional search, creating a new benchmark for achievable skill. It delivers near-optimal distortion rates in a data-oblivious manner. This shows that our nearest-neighbor engines operate with the efficiency of a 3-bit system while maintaining the accuracy of much heavier models.
Peering Forward
TurboQuant, QJL, and PolarQuant are not simply practical engineering solutions. They are also important algorithmic advances supported by strong logical proofs. These methods work well in practice and are often proven efficient, operating close to the theoretical maximum. This solid foundation makes them reliable for large, critical systems.
One main use of these methods is to solve key‑value cache bottlenecks in models like Gemini, but efficient online vector quantization has an even wider impact. For example, modern search is moving beyond keywords to understand intent and meaning. This shift needs vector search, which finds the most relevant items in a database with billions of vectors. Techniques like TurboQuant are essential. They enable the construction and querying of large vector indexes with minimal memory usage. Almost no pre-processing, along with top accuracy, makes semantic search at scale faster and more efficient. This is important as AI is integrated into more products. Advances in vector quantization will become even more important.
Source: TurboQuant: Redefining AI efficiency with extreme compression
