An AI Agent Found 20 ML Improvements Karpathy Had Missed in 20 Years

Andrej Karpathy's 'autoresearch' framework runs an AI agent that can autonomously optimize machine learning models, finding improvements Karpathy missed over 20 years. The agent's simple design and focus on measurable objectives make it a powerful tool for iterative ML development.

Details

Karpathy's 'autoresearch' framework is a 630-line Python script that runs an AI agent in a loop: it reads a training script, forms a hypothesis, modifies the code, runs a short training job, evaluates the results against a scalar metric, and repeats. On Karpathy's own ML setup, the agent ran 700 experiments over two days and found 20 optimizations that led to an 11% training speedup. Shopify's CEO then ran the same approach on internal data, with the agent performing 37 overnight experiments and achieving a 19% performance gain, as well as a 53% faster rendering time and 61% fewer memory allocations for their Liquid templating engine. The key to the framework's success is its focus on measurable objectives - it works best for problems where quality can be evaluated with a single scalar value, rather than more subjective measures like natural language quality or product decisions. Karpathy acknowledges this constraint, noting that the framework 'works best on problems where you have a clear eval.' While the 'autonomous research' framing may be overstated, the framework's simple design and impressive results demonstrate its potential as a tool for iterative ML development and optimization.