Avoiding GPU Crashes When Loading Large AI Models

The article discusses how to properly estimate the VRAM required to load large AI models and avoid crashes due to insufficient memory. It introduces a CLI tool called 'gpu-memory-guard' to check if a model will fit on the available GPU memory before attempting to load it.

Details

The article describes how the author encountered repeated GPU crashes when trying to load a large 13B parameter AI model on a 24GB GPU. The issue was that the 'free VRAM' reported by nvidia-smi did not accurately reflect the memory required to load the model. There are three main factors that eat into the 'free VRAM': CUDA context overhead, memory usage by the display server and other processes, and the key-value cache allocated by the model during inference. The author provides the formula to calculate the actual VRAM required, which includes the model size, key-value cache, activation overhead, CUDA context, and a safety buffer. To simplify this process, the author created a CLI tool called 'gpu-memory-guard' that can check if a model will fit on the available GPU memory before attempting to load it. This helps avoid crashes and wasted time from trying to load models that exceed the GPU's memory capacity.