Why AI Evaluation Science Can't Keep Up (with Carina Prunkl)

<p>Carina Prunkl is a researcher at Inria. She joins the podcast to discuss how to assess the capabilities and risks of general-purpose AI. We examine why systems can solve hard coding and math problems yet still fail at simple tasks, why pre-deployment tests often miss real-world behavior, and how faster capability gains can increase misuse risks. The conversation also covers de-skilling, red teaming, layered safeguards, and warning signs that AIs might undermine oversight.</p><p></p><p><strong>LINKS:</strong></p><ul><li><a href="https://carina-prunkl.squarespace.com">Carina Prunkl personal website</a></li></ul><p></p><p><strong>CHAPTERS:</strong></p> <p>(00:00) Episode Preview</p> <p>(01:04) Introducing the report</p> <p>(02:10) Jagged frontier capabilities</p> <p>(05:29) Formal reasoning progress</p> <p>(12:36) Risks and evaluation science</p> <p>(19:00) Funding evaluation capacity</p> <p>(24:03) Autonomy and de-skilling</p> <p>(31:32) Authenticity and AI companions</p> <p>(41:00) Defense in depth methods</p> <p>(48:34) Loss of control risks</p> <p>(53:16) Where to read report</p> <p></p><p><strong>PRODUCED BY:</strong></p> <p><a href="https://aipodcast.ing">https://aipodcast.ing</a></p> <p></p><p><strong>SOCIAL LINKS:</strong></p> <p>Website: <a href="https://podcast.futureoflife.org">https://podcast.futureoflife.org</a></p> <p>Twitter (FLI): <a href="https://x.com/FLI_org">https://x.com/FLI_org</a></p> <p>Twitter (Gus): <a href="https://x.com/gusdocker">https://x.com/gusdocker</a></p> <p>LinkedIn: <a href="https://www.linkedin.com/company/future-of-life-institute/">https://www.linkedin.com/company/future-of-life-institute/</a></p> <p>YouTube: <a href="https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/">https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/</a></p> <p>Apple: <a href="https://geo.itunes.apple.com/us/podcast/id1170991978">https://geo.itunes.apple.com/us/podcast/id1170991978</a></p> <p>Spotify: <a href="https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP">https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP</a></p> <br>