Optimizing llama.cpp Performance on 8GB GPUs

The article discusses how the performance of llama.cpp can change up to 5x on 8GB GPUs by optimizing key settings. It provides a guide on the optimal values for 5 critical options.

Details

The article focuses on optimizing the performance of the llama.cpp language model on 8GB GPUs. It covers 5 key options that can have a significant impact: 1. -ngl (number of GPU layers): This determines how many of the model's transformer layers are offloaded to the GPU. The optimal value maximizes GPU utilization without exceeding the 8GB VRAM limit. 2. -c (context length): This sets the maximum number of tokens the model can reference during inference. It directly impacts the VRAM consumption of the key-value cache, which can exceed 8GB for large context lengths. 3. --cache-type-k/-cache-type-v: These options enable quantization of the key-value cache, reducing VRAM usage by 50-75% with minimal quality degradation. 4. --flash-attn: This efficient attention calculation algorithm can improve performance by 10% on long contexts with no downsides. 5. -b (batch size) and -t (thread count): Optimal batch size is 512 to avoid VRAM spikes. Thread count should be set to the physical core count, not the logical thread count, to avoid memory bandwidth contention. The article provides specific recommended settings for 8GB GPUs and different model sizes, along with the rationale behind the optimal values.