Compressing Vision-Language Models with TurboQuant

The author implemented TurboQuant, a technique from Google that compresses transformer KV caches, as a plugin for vision-language models. They tested it on a large video model and found significant memory savings with minimal quality loss.

Details

Google recently published TurboQuant, a technique that can compress transformer KV caches by 4-5x with no accuracy loss. The author wanted to test if this technique would work on vision-language models processing video, which can have 10x larger KV caches than text-only models. They developed a TurboQuant plugin for the vLLM framework that can be easily integrated without modifying the model code. The results show that TurboQuant compression works well on the Molmo2 vision-language model, reducing the 1.6GB KV cache to just 435MB on an RTX 4090 GPU. The output quality was nearly identical to the uncompressed baseline. The author also developed an incremental dequantization approach that reduced the decompression overhead from 3.36x to 1.78x. Additionally, the author found and fixed several bugs that were not present in the original TurboQuant paper, including issues with FP16 norms, QJL correction, and multi-layer precision drift in fused kernels. These discoveries highlight the importance of validating new compression techniques across a variety of model architectures and hardware.