Andrej Karpathy's Method for Building Effective AI Skills

Andrej Karpathy's autoresearch method provides a systematic approach to developing AI-powered skills that actually work, by defining test cases, measuring performance, and iterating until the skill meets the desired criteria.

Details

Karpathy's insight is that the same pattern used to optimize ML training runs can be applied to optimizing agent instructions. Instead of hoping a skill works, you can run it through controlled evaluations: 1) Define test cases with expected outputs, 2) Run the skill against the evaluations and capture the outputs, 3) Measure the pass/fail rates against the expected outputs, and 4) Iterate on the instructions until the skill meets the target accuracy. This approach helped Karpathy's fundraising and sales skills jump from 70% to 94% accuracy and 65% to 91% MEDDIC compliance, respectively. The key is to not just test the skill manually, but to systematically measure its performance against defined criteria. Teams should also benchmark the skill against the raw model to ensure it is actually improving performance, not constraining the model. Finally, optimizing the skill description to match how users actually ask for the functionality is critical to ensuring the skill is triggered appropriately.