Leveraging KV Cache to Accelerate Text Generation in Large Language Models

This article explains how Large Language Models (LLMs) generate text one token at a time, and how the use of Key-Value (KV) Cache can help speed up this process by storing and reusing past computation results.

Details

Large Language Models (LLMs) are trained on massive amounts of text data and can understand and generate human language. They generate text one token at a time, where a token can be a word, part of a word, or a single character. To predict the next token, the model looks at all the previous tokens in the sequence. This is done using the attention layer, which converts each token into a Query (what the token is looking for), a Key (what information the token holds), and a Value (the actual information). The model then compares the Query of the current token to the Keys of all previous tokens to determine relevance, and uses the corresponding Values to make the prediction. However, this repeated computation of Keys, Values, and Queries for the entire sequence can be inefficient, especially as the sequence gets longer.