Cheap Models Beat Expensive AI Through Structured Debate

Three inexpensive AI models outperformed the more expensive Claude model on an educational assessment task by engaging in structured debate, rather than just voting on answers.

Details

The article describes an experiment where three relatively inexpensive AI models were able to outperform the more expensive Claude model on an educational assessment task. The key was that the models engaged in a structured debate process, rather than just voting on answers. The debate protocol, called ICE (Iterative Consensus Ensemble), involves three phases: 1) models answer independently, 2) each model critiques the other two answers, and 3) models revise their responses based on the critiques. This debate process led to 88% accuracy, compared to 76% for Claude alone. The article argues that debate is more effective than voting because it requires the models to engage with the substance of disagreements, not just aggregate answers mechanically. Genuine diversity in model training and architecture is critical for this to work - 'different personas' alone is not enough. The article concludes that the structure of interaction matters more than individual model capabilities, and that creating conditions for real disagreement and engagement is the key challenge.