E8 Lattice Quantization Outperforms Scalar Quantization for KV Caches
This article discusses how the E8 lattice quantization technique can significantly outperform scalar quantization for compressing key-value (KV) cache data, achieving 3x better compression compared to scalar quantization at the same distortion level.
Why it matters
This technique can significantly improve the compression of KV cache data, which is crucial for efficient storage and transmission of large-scale machine learning models and other data-intensive applications.
Key Points
- 1Scalar quantization treats each number independently, wasting bits
- 2E8 lattice quantization groups 8 numbers at a time, exploiting correlations
- 3E8 lattice quantization creates a highly compressible distribution due to the non-uniform assignment of lattice points
- 4Hadamard rotation is used to address the issue of outliers in raw KV vectors
Details
The article explains that most KV cache quantization methods use scalar quantization, which rounds each float value independently to the nearest 2-bit or 4-bit value. This approach wastes bits, as it does not exploit the correlations between dimensions. In contrast, the E8 lattice quantization technique groups 8 numbers at a time and snaps each group to the nearest E8 lattice point. This non-uniform assignment of lattice points creates a highly compressible distribution, leading to a 3x better compression under entropy coding compared to scalar quantization at the same distortion level. However, the raw KV vectors can have heavy-tailed distributions, which can inflate the quantization scale for all 8 dimensions in a group. To address this, the article suggests applying a Hadamard rotation to the vector before quantization, which spreads the energy uniformly across all dimensions, making the distribution near-isotropic. The combination of Hadamard rotation and E8 lattice quantization is the key to the performance of the NexusQuant library.
No comments yet
Be the first to comment