NexusQuant Benchmarks: Honest Compression Results

The article presents the full set of benchmark results for NexusQuant, a training-free KV cache compression system for transformer models. It covers performance on Mistral-7B and Llama-3-8B models, including perplexity deltas and compression ratios at different eviction rates.

Details

The article presents a comprehensive look at the benchmarking results for NexusQuant, a system that compresses the KV cache of transformer models at inference time without retraining. It covers the performance on two different model architectures, Mistral-7B and Llama-3-8B, across a range of prefix lengths and eviction rates. The key findings include the discovery of a 'catastrophic' perplexity increase at high eviction rates (>60%) for long context (>2K tokens) on Mistral-7B, which the authors attribute to a fundamental capacity loss in the model. In contrast, the Llama-3-8B model surprisingly showed improved perplexity with compression, which the authors hypothesize is due to the compression acting as a beneficial regularizer for the model's grouped-query attention architecture. The article also explores the impact of text domain, finding that academic text compresses best while creative/narrative text is the most challenging. Finally, the authors present results from downstream task evaluations, showing that factual recall tasks are the most resilient to compression.