The AI Benchmark Where Simple Beats Smart

The latest ARC-AGI-3 benchmark results show that simple CNN and graph-search algorithms outperformed state-of-the-art large language models like GPT-5 and Claude by a significant margin, raising questions about the limits of the current AI paradigm.

Details

The ARC-AGI-3 benchmark tests novel visual reasoning in interactive environments, where each task is procedurally generated and cannot be solved by simply retrieving patterns from training data. While state-of-the-art large language models like GPT-5 and Claude scored below 1% on the benchmark, simple CNN and graph-search algorithms reached 12.58%. This suggests that the kind of generalization that frontier AI models excel at may not be the only type of generalization that matters for genuine intelligence. The ARC Prize Foundation has long argued that large language models achieve 'crystallized intelligence' (pattern retrieval) rather than 'fluid intelligence' (constructing solutions from first principles). ARC-AGI-3 provides a clear test of this distinction, and the results challenge the premise that scaling up transformers will lead to human-level general intelligence. The open-source toolkit released alongside the benchmark enables researchers to experiment with alternative architectures that combine symbolic reasoning, search, and perception, which may lead to more promising approaches than the current focus on ever-larger language models.