The AI Model You Chose Was Picked by a Server, Not a Score

This article discusses how server configurations can significantly impact AI model benchmarks, undermining their reliability as a measure of true model capability.

Details

The article explains how the author runs tests on various AI models and scores them based on their own criteria, rather than relying on leaderboards. Anthropic published a paper showing that server configuration alone can swing benchmark scores by up to 6 points, which is larger than most leaderboard gaps. This is because resource limits can cause models to fail tasks not due to incapability, but due to server constraints. The paper also shows that above a certain resource level, extra power lets the model try approaches it couldn't attempt before, changing what the benchmark is actually measuring. The article argues this is an instance of Goodhart's Law, where a measure becomes a target and stops being a good measure. It provides examples of other benchmarks like ImageNet, GLUE, and SWE-Bench being gamed by models learning contextual clues rather than true understanding. The author concludes that this issue is hard to fix due to commercial pressures, as labs want their numbers to look good and are unlikely to coordinate on shared server standards.