#250 – Toby Ord on where AGI timelines go wrong

<p>Both Silicon Valley and the public can’t get enough of ‘AGI timelines.’ But Toby Ord, senior researcher at <a href="https://aigi.ox.ac.uk/">Oxford’s AI Governance Initiative</a> and author of <a href="https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity"><em>The Precipice</em></a>, believes we consistently make big mistakes when thinking about them. He lays out the 14 ways he most often sees people go wrong:</p><ol><li>Assuming AI research is just hill-climbing</li><li>Imagining AI research is just programming</li><li>Forecasting “could” instead of “will”</li><li>Believing the current benchmark is the last one</li><li>Extrapolating trends with no clear finish line</li><li>Assuming inputs keep scaling at the same rate</li><li>Conflating intelligence with capability</li><li>Consuming point estimates and discarding the error bars</li><li>Dismissing dissenting experts</li><li>Forecasting very different things while using the same words</li><li>Assuming capabilities arrive together</li><li>Treating “we don’t know” as permission to carry on as usual</li><li>Choosing a plan that minimises regret rather than maximises impact</li><li>Trusting surface model impressiveness</li></ol><p>In this extended conversation with Rob Wiblin, Toby also explains why he thinks:</p><ul><li>AI self-improvement is uniquely dangerous in four ways, but also might not even work</li><li>A ban on superintelligence is possible</li><li>A US-China treaty on superintelligence is also possible</li><li>The case for ‘broad timelines’</li><li>Transformative AI is likely a decade away</li><li>We should just ban unmonitorable chain-of-thought today.</li></ul><p><em>This episode was recorded on July 2, 2026.</em></p><p><a href="https://80k.info/to26"><strong>Links to learn more, video, and full transcript: https://80k.info/to26</strong></a></p><p><br><strong>Want to get up to speed on AI?</strong> <em>We’ve got a </em><a href="https://80000hours.org/podcast/on-artificial-intelligence/"><em>crash course of 10 of our podcast episodes</em></a><em> designed to help you get to grips with transformative AI — particularly if you’re new to the topic — and what you can do to help shape its trajectory.</em></p><p>Chapters:</p><ul><li>Toby Ord is back — for the 5th time! (00:00:00)</li><li>AI self-improvement might not matter (00:00:14)</li><li>4 ways AI self-improvement is dangerous (00:12:39)</li><li>A US-China treaty on superintelligence is possible (00:20:47)</li><li>Could we ban superintelligence? (00:37:07)</li><li>We should just ban unmonitorable chain of thought (00:57:46)</li><li>Why Toby thinks AGI is a decade away (01:09:28)</li><li>Even superintelligence needs work experience (01:17:50)</li><li>Is AI coming for mathematicians? (01:32:22)</li><li>The case for broad timelines (01:45:01)</li><li>How should broad timelines change what we do? (02:22:24)</li><li>Are current models all they’re cracked up to be? (02:31:03)</li><li>Coordinating careers for different timelines (02:43:36)</li></ul><p><em>Our production team includes:</em></p><ul><li><em>Video editors: Josh Alward, Dominic Armstrong, Ollie Bignell, Andrés Escobar, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour</em></li><li><em>Producers: Elizabeth Cox and Nick Stockton</em></li><li><em>Coordination and support: Katy Moore and Lou Moran</em></li><li><em>Camera operator: Jeremy Chevillotte</em></li><li><em>Music: </em><a href="https://open.spotify.com/artist/4lWobp6IUcSZ7w5mhnU1c9"><em>CORBIT</em></a></li></ul>