The Benchmark Reality Gap: Where Are the Non-Thinking Model Benchmarks?

The article discusses the lack of transparent benchmarks for non-thinking AI models, which are the ones most users rely on daily, in contrast to the focus on reasoning-heavy 'thinking' models in AI benchmarks.

Details

The article argues that while it makes sense for AI benchmarks to focus on reasoning-heavy 'thinking' models that produce the best possible results, the reality is that over 90% of all AI answers people actually trust and use are instant responses generated without explicit thinking. This is especially true for free tiers or lower-cost plans, where requests are handled by fast, non-thinking models. However, the author has learned that even OpenAI has removed routing for Free and Go users, increasing the share of 'Thinking' responses from 1% to approximately 7%. Unfortunately, users are still accustomed to faster being better, and many are apparently unaware of how tricky this can be. The article highlights the gap between the focus on benchmarking 'thinking' models and the dominance of non-thinking models in real-world usage, and argues that providers should publish benchmarks for these instant models as well to better reflect everyday reality.