Calculating the True Cost of On-Prem AI Inference

The article discusses the problem of cost tracking tools not accurately accounting for the real costs of running AI inference on-premises. It introduces InferCost, an open-source tool that calculates the true cost by considering hardware amortization and electricity usage.

Details

The article highlights the problem that current cost tracking tools in the AI ecosystem do not accurately account for the real costs of running large language models (LLMs) on-premises. These tools often show a $0 cost, failing to consider the hardware and electricity expenses. The article introduces InferCost, an open-source Kubernetes operator that calculates the true cost of on-prem AI inference by factoring in hardware economics like amortization and actual GPU power draw. InferCost integrates with Prometheus and Grafana to provide per-model, per-team, and per-token cost visibility. The article shares real-world findings from deploying InferCost, showing that on-prem inference can be 84-94% cheaper than cloud APIs for most models, except for the smallest ones. The true cost includes both the fixed infrastructure cost and the marginal cost per token during active inference. The goal is to provide accurate cost data to help organizations make informed decisions about their AI deployment strategy.