How Karpathy's Autoresearch Unlocked a Breakthrough for a Non-Data Scientist

The article describes how the author, a non-data scientist, used Karpathy's autoresearch technique to solve a challenging machine learning problem and achieve a significant performance improvement, going from a ceiling of 0.581 AUC to 0.6747 AUC.

Details

The article describes the author's journey in trying to solve a machine learning problem involving CRM data and call recordings. Despite trying various techniques like XGBoost, feature engineering, and extracting features from transcripts, the author was unable to break past a 0.581 AUC ceiling. After learning about Karpathy's autoresearch approach, the author set up an automated experiment loop that allowed an AI agent to explore different model architectures and techniques. This agent-driven exploration, combined with the author's own guidance and 'rubber duck debugging' with the agent, led to the discovery of a new technique that jumped the AUC from 0.581 to 0.628 in a single step. Over the course of 165 experiments, the agent was able to push the AUC all the way up to 0.6747, a 15.6% gain from the original dataset. The article covers the specific stacking architecture that broke through the 0.58 ceiling, as well as what happened when the agent spawned its own research sub-agent mid-run.