How TurboQuant Reduces RAM Usage for Large Language Models

This article explains how large language models (LLMs) store words as high-dimensional vectors, which leads to massive memory usage during inference. It introduces TurboQuant, a technique that compresses these vectors using a scale and code approach to significantly reduce RAM requirements.

Details

Large language models (LLMs) like GPT-3 and DALL-E do not store words as text, but rather as high-dimensional vectors of numbers. These vectors represent the semantic relationships between words, with similar words located close together in the vector space. However, as a token moves through the model, it gets transformed at each layer, resulting in the storage of thousands of numbers per token in the KV cache. For a conversation with 2,000 tokens, this can lead to over 500 million numbers, consuming around 1GB of RAM. Simply reducing the precision of these numbers (e.g., using 8-bit instead of 16-bit) is not enough, as it can distort the subtle numerical relationships that the model relies on for accurate attention calculations. TurboQuant addresses this by storing the numbers in a structured way using a scale and a small integer code, allowing for significant compression without sacrificing model accuracy. This approach reduces the memory footprint of the KV cache, enabling LLMs to handle longer conversations and serve more users simultaneously.