Optimizing AI Inference on a Laptop with C++ and Batching

The article describes how the author built a high-performance C++ inference engine that can achieve 2,240 TPS on a 2019 laptop with an AMD Ryzen 5 processor. The key techniques used include batching, threading, and system design optimizations.

Details

The author built an AI inference engine on a 2019 HP laptop with an AMD Ryzen 5 3500U processor, 8GB RAM, and Radeon Vega 8 graphics. The goal was to squeeze the best performance out of the limited hardware by relying on batching, threading, and system design optimizations. At the core, the author recognized that AI models are essentially large chains of linear algebra operations, which can be efficiently executed on CPUs through vectorization and parallel processing. To achieve this, the author used a C++ implementation with a gRPC-based communication layer, a thread pool for orchestration, and the ONNX Runtime library for inference. The key innovations include using a fixed thread pool to avoid the overhead of managing many threads, implementing a batching logic to leverage SIMD instructions, and keeping the entire model in RAM to avoid the performance impact of using the SSD. Through these optimizations, the author was able to achieve 2,240 transactions per second (TPS) on the laptop, a significant improvement over the typical performance of AI models on consumer hardware.