TurboQuant MoE 0.3.0 Introduces Compression and Optimization Techniques

The article highlights the key features in version 0.3.0 of the TurboQuant MoE (Mixture of Experts) library, including techniques for efficient storage, acceleration, and security in large language models.

Details

The TurboQuant MoE 0.3.0 release introduces several key optimizations and techniques to improve the performance and efficiency of large language models. The True 3-bit PolarQuant feature achieves significant storage compression by physically bit-packing 8x3-bit values into 3 bytes, while maintaining less than 0.1% accuracy drop. The Cross-Layer KV Delta technique further compresses the intermediate layers by storing 3-bit anchor layers and 1-bit signed deltas, resulting in 14x compression. The Speculative KV Prefill accelerates the prefill phase by 2-3x using 1-bit sketches for fast draft KV generation and verification. The Temporal Expert Fusion leverages SVD-based merging of rarely-used experts to reclaim 20-30% of MoE weight VRAM without any quality loss. Additionally, the library introduces Cross-Request Prefix Sharing for global management of common KV blocks across concurrent requests, and a Fast Walsh-Hadamard Transform (FWHT) for faster quantization on power-of-2 dimensions. Finally, the Cryptographic KV Watermarking feature uses HMAC-seeded LSB watermarking of KV scales for attribution and auditing purposes.