Surgical Memory Alignment Enables Running Large Language Models on Low-End GPUs

The author developed a custom framework called 'QKV Core' that uses 'Surgical Alignment' to efficiently load and run large language models like Qwen-2.5-7B on a 4GB GTX 1050 GPU without CPU offloading.

Details

The author faced challenges running modern 7B language models like Llama-3 or Qwen-2.5 on their low-end 4GB GTX 1050 GPU. Standard quantization tools often add unnecessary padding to tensors to maintain uniform block sizes, which wastes precious VRAM on smaller GPUs. To address this, the author developed 'QKV Core', a custom framework that analyzes the entropy of each layer, switches between dictionary coding and raw storage, and precisely aligns memory blocks to adhere to the llama.cpp library's block boundaries without the usual padding waste. This 'Surgical Alignment' technique saved around 44MB of VRAM per model, allowing the entire Qwen-2.5-7B to run purely on the GPU without crashes. Additionally, the cache-aligned memory blocks resulted in a 34% improvement in I/O load times using Numba-accelerated kernels. The author is open-sourcing QKV Core as an early/experimental solution for users with 4GB/6GB GPUs struggling with OOM issues when running large language models.