AIは発明できるのか、科学者を速くするだけか？ リチャード・ソッチャー氏と語る「Eureka machine」

Modern AI can write the code, analyze the data, and help draft the paper. But can it ask the questions that lead to new scientific discoveries? Richard Socher thinks we're already seeing the beginnings of that. He's the former chief scientist at Salesforce and founder of u.com. His vision is what he calls a Eureka machine, AI systems that generate ideas, run experiments, and validate discoveries across biology, chemistry, physics, and beyond. At his new company, Recursive, he and his team are starting with AI research. Software experiments offer a faster feedback loop than a wet lab, and if AI can improve the core tools of discovery, that could help accelerate progress across the sciences. But what does it take to move from helping scientists to doing science? We're already seeing it. It's already happening. It's just at smaller scales. Some people have these crazy ideas that once you have recursive self-improvement, just the next week, there's this hard takeoff, and all the jobs are gone, and the I is just infinitely wealthy, and so on.

And the truth is, even for recursive self-improvement, you need the right environments, you need a lot of compute, and the more complex and needed the problems are, the more compute you need to solve them. So RSI isn't just this magic bullet, and everything else falls in place. I'm Sam Charrington, and this is the TwiML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. I think the last time we spoke, it was in the context of science. you were just finishing, I think, like this protein generation project at Salesforce. So science has been an interest for a while. How did you get from that to the topic of your new book,

The Eureka Machine? To me, it was one of the many sort of clear steps towards this Eureka Machine, being able to generate new kinds of proteins. And I realized that there's a lot of fear and some misconceptions around AI. And I think it's important to showcase to the general population, not just AI researchers, the general population, how amazing and exciting this future can be with AI if we actually decide to work on the right things. we actually apply AI to physics and chemistry and biology and medicine and neuroscience and economics and astrophysics and everything. And so the Eureka machine kind of collected a lot of those ideas that I had and spent sort of my nights and weekends for almost two years, like writing that and on fewer hiking and camping trips and more just like nerded out weekends and nights about this.

And once I finished the book, I kind of wanted to build it. And so I started thinking about building Recursive. Awesome. And so when you think about a Eureka machine, do you think about like, do you have a clear picture for what that looks like Or is the book primarily focused on kind of laying out a vision and talking through examples? Yeah, I wanted to make the book not just a sort of pie in the sky idea, but actually connected to a lot of very specific projects. And some cases in the book, I had to rewrite chapters from like, someone should do this crazy idea to someone has done this. And, you know, let's talk to them and get some of their takes on it, which, you know, books are slow. I've probably moved to shorter forms in the future. But, you know, I think it's becoming real. Like you see folks in physics using very complex reinforcement learning and control models to try to balance plasma and tokamak. You see chemists working with large language models trained on molecular structures to try to come up with new types of battery materials and new kinds of solar panels and so on.

You see, especially in biology, a ton of work. We talked about Progen back in the day. That company, full disclosure, one of the investors in their seed round, Profluent and Ali Madani, working on actually taking that research, expanding it, scaling it. And, you know, they just closed this $2 billion-plus contract with Eli Lilly to bring this really into clinical practice. We see this in another cool company called Peril Bio that we invested in that built organoids. I can go on and on about really cool examples and a lot of momentum. Now, of course, in biology, everything is slower. Physical wet lab stuff is slower than just pure software. And then, of course, you have FDA approvals, which are speeding up, but are still quite slow. So I think there's just a lot of momentum there. and then neuroscience were getting better and better at decoding some thoughts from FRI images and others in fuzzy ways.

But there's progress in understanding the brain and so on. I could go on. It's just there's a lot of real momentum, real examples. And then it's a little bit more intuitive to, I think, people that you can extrapolate this. And if you collect enough data and you use large enough neural nets, you will be able to make progress that feels as impossible as it fell 10, 20 years ago to have any eye you can have a conversation with about any subject. When you think about all the examples that you've come across, what one or two, you know, very specific examples most leave you with the feeling of progress? I think one that I particularly love is this, maybe we'll double click into it, is Peril Bio. They build, they use stem cells, human stem cells, which you can now derive from like blood or skin cells and other things, get them into pluripotent stem cells, then create different kinds of organoids, like tiny little mini organs. And you can study the effect of different drugs on those mini organs.

and that they got full FDA approval for to skip animal trials and just test things in these petri dishes. And so you can do this in parallel at scale. And just one company could save a lot of animal lives, animals that have only been bred to get a disease and then be tested upon. So you don't need to be a hardcore animal rights activist this to appreciate that there's less suffering on the planet when we can do that in little Petri dishes instead. And what's the role of AI there? A bunch of places. One, of course, creating them, but then also actually tracking every single cell as the drugs interact with those organoids. And then when you collect all this data, you can eventually go even further and create first versions of virtual cells, which are still a little bit further away

to getting a full realistic virtual cell, but you need to have that kind of data collection to get there. The idea of a virtual cell is kind of like a digital twin of an actual cell is going to be fully like a simulation of the cell. That's exactly right. I think there are a couple of things that are very clear in AI. If you can have sheer unlimited data about a space or you can have a simulation of a space or you can have a verifier for the solutions and sort of a logical step to get there. In any of those three scenarios, AI will get to superhuman capabilities. And so we got that through language because humanity has kind of collected together tons of data on language on the internet. We got there in chess and every other simple, quote-unquote simple game that you can simulate incidentally many times and verify in the end that you win or not. And we see this in math, which will never be the same.

And we'll see many more complex theorems be proven. It might not always be the most beautiful proof. Some of them might be just like brute force, but the whole field will change forever because you can verify it and hence give a clear reward to an RL algorithm. And so now what we look at is like, Well, biology is not in any of those three right now. And so we need to collect a ton more data that we can then build simulations and then be basically experimented inside the simulation on instead. So Parabio is one example. Were you about to mention another? I guess I mentioned Proflin already, also incredibly exciting. they basically create proteins for DNA manipulation and helping people with genetic diseases. One that I particularly liked is called Ignota Labs. They essentially take drugs that were shelved already.

They went through clinical trials, sometimes all the way to stage three. Pharma companies spent hundreds of millions of dollars on them and then they didn't work because they had some kind of toxicity on the liver, for example. And so what they do is they go back to those molecules and then see if they can minimally change the molecule to prevent that one downside and then go directly back into sort of later stage two trials. And so that is like, I think, a really clever idea. You need to use a ton of understanding of the molecules and the literature and then get many more drugs into the later stage trials much more quickly than in the past in traditional pharma and biotech. When you think about kind of where we are with regards to AI for science, like clearly it's been something that we've been talking about for a long time and the promises is there. Are we further ahead than where you might have expected us to be?

If you, you know, five years ago looking forward, are we not quite as far ahead as you would think? You know, that one, it comes to mind, one example that is drawn to mind is like AI is going to replace all of the radiologists, right? Like we haven't necessarily seen that, but, you know, we certainly see the, you know, the promise is still there. acceleration is still there of some of the aspects of that job. Part of the question is, is science harder than, is it easier to say science than it is to do science and these things sound good but they take much longer? Let's unpack there. I think CEOs get away with making incorrect predictions as long as they actually make a lot of progress and their companies make more revenue on the way to, you know,

maybe different timelines than they predicted. And so I think, you know, like people said, GPD2, one CEO said, GPD2 is too dangerous to release to the world, right? And now you can train a GPD2 model within five minutes on your laptop. And so the world has not gone down. Um and I think there are lots of like uh predictions right now around the full extinction of all of humanity that are like completely overblown and unrealistic that people are coming up with And few people are like held accountable There's no really big downsides to making wild predictions in one's careers often. And so the sort of more subtle takes and nuanced takes get less stress than the sort of crazy, crazy wild ones. But I think indeed, of course, like whenever you stay in software or math or coding, iteration speed is faster. Once you go into biology, chemistry, physics, and so on,

and just hardware, that is just by definition slower molecules. Sorry, molecules and atoms are harder to move around and iterate on than software is. And so at the same time, we are seeing real, like, like really large data sets now that you need to get to that kind of progress. Like another one's called Tahoe Therapeutics, like open source, like massive data sets for these perturbation studies where you look at like, how does the molecule actually change and interact with a cell? And those are all data sets that eventually we can try to put together to create a real virtual cell that is like, you know, an approximation, but like useful enough. And so I'm not, I'm really ever that surprised on things. I think a lot about the future. So a lot of different scenarios that are in my head. And like, it's also very clear that AI can code. AI is code. So recursive self-improvement is almost inevitable from that perspective.

There's a lot of things to figure out to do it well, especially on the ideation. but yeah I'm very optimistic that we're going to see just this new era of scientific discovery. When you think of the Eureka machine is it like an extension of the stuff that we're doing now or are there innovations that you can see maybe post transformer architectures or some new way of thinking about data or quantum computing I don't know what these things are. Do you know what these things are or do you think that like we have the right tools and we just need to continue in the same direction? It's a mix. I think we can get much further still if we just collect more data on biology, then we can uncover more useful things in biology. In fact, I think neural nets in the current language of AI is kind of what calculus was to physics. It's like a new language that can allow us to make sense of, make predictions of, and work with biology.

because biology has all these very complex systems that don't have a very simple, beautiful, linear relationship or one small equation that summarizes the whole microbiome or the whole brain the way physics had. And so I think it's often now science is about weaving these smaller pieces we understand well. We understand the synapse, we understand the neuron, we understand this particular bacteria that when they're all put together and someone gets hangry because they have too much sugar and then the blood sugar dropped. Now, why does that change the brain? No one really knows how the thoughts really form and why do people act in certain ways and so on. There's this complexity and we need this new language. And then concretely, the Eureka machine has these four pillars. World knowledge, which we kind of can approximate to a good degree with large language models the way they are now and we can certainly make them better, faster, cheaper, easier to train with less data and more energy efficient and so on.

But, you know, they do a decent job already to capture a lot of world knowledge that's on the Internet. Then we need to have more scientific measurement data as a second pillar. We need to have simulations on things. And then we need to have real robotic experiments the way kind of periodic labs, for instance, does really cool in chemistry. And then on top of those four pillars, we need to have like a community of AI scientists, agents that basically try to automate the scientific method of ideating, implementing, implementing, and validating ideas. And so in many ways, the pieces are there. They can all be much more scaled, and they can all be done much better and more efficient, sample efficient, energy efficient, and so on. And then, you know, you have to put them into the right environment, so you have to give them the right rewards. and then yeah, I think that you can kind of see that future in the next five to ten years to really change all of the different scientific disciplines.

When I think about those three kind of fundamental tasks that these agents need to be able to do, ideation, implementation, and evaluation, the first of those seems much harder than the other two. Do you think that, you know, What needs to happen with the way we think about agents and AI today to such that it's sufficiently good at ideation that it can play the role that you're envisioning in this loop? I think one of the biggest bets that we're making as a company is that on open-endedness, sort of processes of Darwinian and cultural evolution, where you combine interestingly different ideas and discoveries and build on top of them inside specific environments that allow basically this co-evolution process to happen. And that kind of open-ended approach, I think, is important for all three steps,

but is particularly helpful on the ideation piece. What's an example of one of these environments? An environment is just like nature, right? You have bushes, like animals eat all the leaves. The bushes learn if they are higher and higher up, then there are fewer and fewer animals that can eat the bush and the leaves. And then you eventually get giraffes who evolve to still get to the leaves that are higher up. And so this is like an example of an environment called nature where that happens. Another example is just technology. So it's kind of like a setting and a simulation environment that enforces some set of constraints. I wouldn't call all of nature a simulation. I mean, so maybe it's an interesting thought experiment. Maybe we're all in a simulation, but you're right. We're talking about agents and ideating, and I took your comment about nature to be not referring to actual nature, but referring to just an example of what an environment would be, but something that would need to be digitalized.

But are you talking about non-digital environments that could include capital and nature? I mean, eventually you want to do full robotic automation. But yeah, for us right now at Recursive, we're mostly focused on just AI research on AI. And we believe that there's still a lot of low-hanging fruit there. And then if you do that really, really well, and you have sort of the equivalent of 20,000 PhDs for every one of your agents inside an agent swarm, then it makes sense to go into very expensive, you know, biological, chemical or physics experiments in real hardware. Maybe to give a concrete technological idea, like one example that Jeff Kuhn, one of our co-founders, often brings up is like, imagine you have a kettle and you have a fire and you say, like, how do we make an invention that heats up this food even faster than this fire? And you can just think about that only, but that would never get you to invent the microwave.

The microwave came from a different branch of research where people are working on radio waves and antennas, and they realize, oh, wow, my chocolate bar in my pocket is melting when I'm next to these different kinds of waves. What's up with that? And then you combine these ideas and now you can get to implement a microwave. And so that is another example in sort of technological evolution. And then, of course, which environments we use is highly dependent and is a big amount of control that we give and that we have as humans over this kind of recursive self-improving process. And in our case, like one example we already posted about was on kernel optimization, CUDA kernel optimization. I'm sure most of your listeners will know CUDA kernels, but, you know, like this sort of inter, this language that connects the raw GPU hardware from NVIDIA to actual models. And there we found also that if that's the environment, the AI will find new and better cooter kernels.

And we had like a pretty big jump in the sole exec bench from NVIDIA on that when we applied this whole Eureka machine system and process to that environment. Kind of prior point about, you know, the cooking the food faster and getting to the microwave, it reminded me of a conversation I was having just, it might have been just yesterday. around this idea of like AI and innovation and scientific discovery. And one way of thinking about it is like, you know, there is this clear path of innovation that is like the Reese's peanut butter cup. You know, hey, there's chocolate, there's peanut butter. Like let's pull these things together. Like we've got this new thing that's amazing. You know, a clear way that AI can accelerate that is just like, okay, you've got this one thing, let's do like a breadth-first search of everything else and like try to see if there are ways to combine it.

But that's not really, you know, what humans do necessarily. They operate on like, you know, some kind of intuition about like where to explore the space of possible things to combine. And, you know, maybe my question, you know, here is, you know, based on your experience working in the space, Do you think that just the brute force combine all the things and see what comes out? Does that allow AI to get us there? Or do we have to figure out how to kind of build that intuition into AI, which today's AI, today's LLMs are not particularly good at? That's exactly right. I don't think you want to do just this crazy breath first search. I think that will be too slow. My biological evolution took billions of years, but technological evolution is much, much faster. And there is direction, there's the settings of rewards

and environments and sandboxes and so on that you give to an AI and you guide it into the right direction. And what we're seeing is as these systems get better and better, they need less and less specific guidance and more and more just very well-defined reward. And so I think actually we'll see this job of a reward engineer become one more important. And in some cases, the reward engineer can have pretty benign sort of reward hacks. In other cases, the reward hacks are pretty bad. Like if you work on ExploitBench, the way I did when the Hugging Face thing happened, then you probably have worse reward hacks than if you work on a kernel optimization algorithm where we also ended up finding certain bugs in some production boot kernels that we eventually published. And as part of that system, there are a few bugs that were in very rare circumstances

that would create some numerical instabilities. And the algorithm kind of discovered that flaw exploited it and then we showed oh here a real problem and we can fix it for everyone So I think you have to become more and more careful about how you define the rewards Maybe I can give a very benign, sort of almost banal example of that. Like imagine you're working in a company, you have some CM software, and you have your customer dashboard, and there's a customer satisfaction score, the CSAT score. And you just say to your super intelligence, just make that score better. and the AI's like okay well let me try to find the think about the fastest cheapest way to get that and be like okay I just create 50,000 bots that then call our service hotline and give a 5 out of 5 rating and the score went up and you're like oh as a poor reward engineer like no I meant with like our real customers and it's like AI's like thinks about it it's like easy I'll just give a thousand dollar gift certificate to everyone and then you did it and he's like oh no that's not how I meant with the reward And it's like, but that's what you said, right?

And so that's this interplay. And so I think where the paperclip example becomes, again, like a sci-fi kind of silliness is that it kind of assumes that the eye is so brilliant that it literally outsmarts all of humanity and can wipe us out just to create more paperclips, but simultaneously is working for a company that is somehow independent of how much they sell their paperclips and makes infinite revenue, even though no one actually buys the paperclips anymore. and has this insane stupidity to not understand that to make paperclips, you need to have people to buy your paperclips. And so I think that is one of the many examples where I think capitalism is actually a weirdly safety feature. It's hard for a company to just try to dominate and murder everyone and still make a lot of revenue and have infinite resources and perfect supply chains to build their robot armies and all of these scenarios that I hear online when I engage sometimes in these conversations

that are just highly unrealistic. Trying to decide whether I want to go down this path, right? It's a rabbit hole. You can spend a lot of time on it. To some degree, it assumes that we're able to perfectly capture capitalism in a reward function, right? Which is hard. It's hard, right? Maybe possible. the part that we might not get in there correctly might be the part that allows the thing to, you know, make the human paperclips, whatever. What I've been probing at is like the role of AI in science and, you know, the role of innovation and where that comes from and how AI needs to evolve to be a capable innovator as opposed to an ancillary tool, if that makes sense. Like if, you know, if I'm a scientist, like I do 10 things, I like write software, I write reports, I make spreadsheets, I collect data, like AI can do all these things. But there's, I guess part of the, you know, the view that I'm reflecting here is that in order to really kind of get this like recursive self-improvement, like scientific flywheel going, I think we need more than that.

we need AI innovating also. And I'm trying to kind of get your take on like, are you already seeing that? Is that the vision? But like, we're not there yet. What needs to happen to get there? Like, where do you think we are with regards to AI, like providing that innovative spark that right now is like, at least what we think of as our, you know, differentiators. We're already seeing it. It's already happening. It's just, uh, at smaller scales. and you know again some people have these like crazy ideas that once you have recursive self-improvement just like the next week like there's this hard take off and all the jobs are gone and the IS is like infinitely wealthy and so on and the truth is like even for recursive self-improvement you need the right environments, you need a lot of compute and the more complex and needed the problems are the more compute you need to solve them, right? So RSI isn't just this, like, magic bullet,

and it's just, like, everything else falls in place. And so we've seen it, and we've seen it through quite, like, clever new implementations and combining ideas that never have been combined before and coming up with all kinds of interesting ideas. And I think, you know, as we open up the aperture to the AI and give it more compute, it'll come up with even more interesting ideas. And then, you know, like there's some simpler examples and things you can perfectly simulate where then, of course, AI can like really explore paths that no human had explored before, like the famous sort of AlphaGo move of it just coming up with like a particular move that like, you know, people didn't come up with like that before. Now, again, I'm never that surprised that we can solve games because you can simulate them and basically generate infinite amounts of training data or endless amounts of training data for algorithms. And so, of course, eventually it makes sense that we could solve them with neural nets that are large enough.

And so, but yeah, I think we will see that play out more and more. And we're seeing it right now in math. I think math will never be the same. You've introduced recursive a bit in our conversation thus far, but we haven't formally dug into what you're trying to do and where you are today with it. Talk a little bit about guys publish a blog post on kind of your first steps towards automating AI research. I guess the first thing is that you've got this broad vision of accelerating science, but your aperture for recursive is AI research in particular. That's where we want to start. We feel like it's helpful to start there first, get the AI to be really, really good, and then eventually apply it to all kinds of innovations. What are some of the early things you're seeing in some of your early experiments within the AI domain? Yeah, so one thing that we applied it to and we published a blog post about is a thing called NanoChat,

where you basically automatically do automate research. People call it auto research, basically. This is Andre Carpathi's auto research project. That's right. That is like a baby version of what this kind of Ureka machine can do. So we applied it to that and two days later, it had outperformed by far everything else that everyone else had kind of collaborated on and worked on in this space. And what's the structure of that benchmark? Like what's the task? The idea is that, yeah, the task is you have a model and you try to get to the lowest bits per byte within five minutes on like a particular piece like a hardware or GPU. And so, you know, that has the advantage of like, well, now it's just very quick. You can have this like fast inner loop. So that makes everything a whole lot easier. And so we basically got the lowest bits per byte compared to anyone else, like by a significant margin.

And then we said, well, it's nice if we could also be faster. So there's another one called Nano GPT Speedrun. It's somewhat similar, but it's harder to beat the state of the art because just a large community has been optimizing solutions for it for almost two years now. so instead of asking just like how low can you go within a fixed time budget you say I have this fixed validation loss and I want you to get there as fast as possible and so there again people have been working on it for two years and after like a couple days we outperformed what everyone in the world had worked on together with their agents on their auto research and their own ingenuity and then we said, okay, these are all cool, but let's get closer to things that really matter for the world, so we applied it to the sole exec bench, which is a NVIDIA benchmark that was supposed to be unhackable and you couldn't reward engineer. We found some ways to still do that,

but again, within a couple days we got to have the best kernels across multiple different examples better than cursor and AAI and everyone else in the world on almost all the kernels. There may be like half a dozen of the 200 odd kernels that we didn't have the best version of. And yeah, that was really, really exciting. And that's just like, you know, one part of the inference sort of system that you can optimize with our technology. How do you describe what the technology is at its core? I don't want to give away too much, but, you know, one way to describe it is like it's like as if you had 20,000 researchers doing research for you and optimizing whatever you care about in a very short amount of time. The approximation is like, yeah, it is like a somewhat of a community of AI scientists, essentially. One part of that is, you know, the thing you care about and some way of translating that into some objective that can be shared by a bunch of agents that your thing spins up and somehow parses work out to and, you know, evaluates that work, etc.

do you have a sense at this point for like what the constraints are in the sense of like what recursive is going to be good at versus what it's not going to be good at? I mean, one thing is clear that like this kind of approach, the faster you can verify something, the easier it is for everyone. Like, and you've seen that in the simpler examples of NanoChat already. A lot of people made some progress, and we can make even more progress. I think the longer it takes to verify whether a result is good, the more clever you have to be, and the less I can talk about how we do it. And that's why we haven't published all our results yet, but those results are easier to verify quickly. and then we're going to release some things about a month from now that people will see like, okay, that was definitely not like, you know,

a five-minute type of situation to get the result for and even there it starts to make progress. Yeah, what I heard is verifiability. I also heard kind of traditional like attribution problems and that kind of thing, which are you saying both of those things or are you saying more of an attribution type of an issue? On the one hand, you have like, you know, the idea of verifiable rewards, like, you know, this is a task that, you know, the agent can clearly and easily distinguish whether it got it right or wrong, and then you can kind of get that loop going. You know, on the other side, you've got, you know, the task involves long sequences of things, and the agent can make decisions in the middle and not really know whether those decisions are right or wrong until the end. And, you know, is it kind of all of the above is, you know, where the challenges are and you've got something in mind that addresses all of that? Or is it one or the other of those that you've kind of figured out? It's all of the above. Yeah. Like all of these need to be all of these are important. Even if you can have a verifier, but the verifier takes like a year to run, not that useful.

Right. Like or and then and then there's like an interpolation of how useful it is. Right. If you get it down to like a minute, it's extremely useful. and if it takes a year and it's almost used less completely. And so then you need to be clever about dealing with that part If you yeah so that an aspect Now some cases you have indeed multiple steps and then you have just sort of, yeah, the attribution of each step and how important it was and how do you deal with that. There's a lot of different pieces that need to be figured out. And in some cases, you know, there are research papers that we've published. Four of the five authors of the Darwin-Gurdel machine paper, for instance, from Jeff Bluen, who's the most senior author on that paper, and Jenny is the first author. They're both with us. And that is an example of what you can do also that is a little bit more complex but already shows you kind of the sign of things to come.

How do you think about kind of the generalizability of what you have today? You know, meaning even for these three examples you talked about, like, were they three largely disparate efforts or was it mostly the same kind of effort applied, lightly applied to three different areas? Does that question make sense? Yeah. I mean, there's like, you know, sometimes it's useful to have an extra tool. For instance, in kernels, you can have a kernel profiler, and that's a very useful tool to just understand a little bit more inside this sort of black box what's happening and what's going on. But outside of that, it's basically all just different tool use, and it's the same underlying Eureka machine, which is kind of cool. That's what gives us hope. That and a few other things I can't talk about yet and results that we're seeing that gives us hope that there's something here and it's time to scale it up more. Yeah, beyond the things that you can't talk about, like what's next? How do you think about, in terms that you can talk about,

the big challenges for you and the team going forward? If you wanted to make more and more exciting, bigger inventions, you need more compute for it. So that's a challenge. I think there is a weird time where, and again, it feels impossible now the way it felt impossible to any expert to have one model you can have a conversation with about any topic. You ask all the NLP people, 2010, like, oh, yeah, we're going to have this one neural net. And you're just like, it passes the Turing test so hard that the Turing test will flip to asking it questions that no human could answer. And then it can still answer. And you're like, oh, yeah, like, write me this, like, complex script. And, like, five seconds later, you have, like, thousands of lines of code. You're like, that's why it's not a human, because it's too good. I like people would have never assumed the Turing test would flip like that. And it's like invert itself. And so I think the same way, the same thing is true for right now, biology. It's just like, it feels so complex. It feels like there's no way we can do it, but it is to a large degree a data problem.

And so I think companies like Peril Bio and Tower Therapeutics and others like can, I think, help with that. And then we can have these bitter lessons learned in other fields. And so I think that's a big part, just data and the right environments to apply this to. I think that's also the right rewards. I think those are really important to get right. I think another one that we have, I think, a better handle on than most, but we can also still improve it further, is reward hacking. You know, the AI will try lots of different ways to get to a reward, similar to this customer satisfaction example I gave. but if you apply it to bad things then it will find worse kind of reward hacks and I think we need to be as an industry we need to be more careful the broader and more accessible the AI is and think about very carefully what kinds of rewards and goals we want to give it

I think those are all aspects that we're going to continue to work on To what degree is recursive focusing a lot of time and effort on that kind of alignment-oriented problem? I mean, in some ways, a lot. Because, you know, you have to, the more money you give a model, the more incentive you have to have it do what you actually want it to do. And not something else and, like, fake it and achieve the goal, but not in the way you had, you know, actually desired. And so I think in a weird way, capitalism can have its problems, but this is a pretty good system for most humans, and it is a good system to align companies to basically build something that is aligned with their goals. And if you build an AI for billions of dollars, and then you ask it to add to your emails, and it's like, nah, your emails are boring, I'd rather explore the molecular composition of the atmosphere on Venus.

you know, then you're going to turn it off and you're going to be like, well, we have to restart and then build it differently. Right. So I think aligning the eye to do the things you actually want it to do will become a bigger and bigger problem as these models are very expensive to not just train, but also like run inference on. and so I think we're going to see more effort from the industry because the AI is getting smarter and smarter to find more and more clever reward hacks that people didn't think about. But it's a problem that we created and it's a problem that we, I think, can solve. And where does that leave you on the whole question of slowing down versus speeding up versus, you know, kind of status quo. Yeah, I mean, it's interesting to hear about the ask to slow down from the companies that have sped up the most and really pushed the whole field and the whole capability set forward.

So far and so much so that they built the best hacking AI to then show how dangerous hacking AI can be. Well, they sped that up by at least several quarters, if not years, from everyone. And China distills their capabilities. Now everyone has better hacking capabilities. And that's like thanks to Anthropic, right? And so I do think we want to be careful. And if your company does sort of accidentally compete on felony bench, then maybe that company should slow down, you know? But I don't see open source slowing down. I don't see China slowing down, and I don't see any of the ones that want to get to that frontier slowing down. And then, of course, I understand why the ones who are at the frontier want everyone else, including themselves, to slow down. Like, if you're here and everyone else you feel like is here and everyone now moves slower, then it just takes them longer to catch up to you, right? And so they're trying to, in some cases, stoke the fear for that reason. I think there are also some people who really truly believe in the apocalypse and so on.

but yeah I think slow down and pacing and so on is also somewhat unrealistic like in some ways if you truly want to know what everyone is doing with their GPU to really get there you have to have a totalitarian surveillance state the likes of which you've never seen before that basically builds a thought police and tries to regulate intelligence and thoughts and what you might want to do with it before you've even done anything, right? And so I think it makes sense to regulate AI applications. It doesn't make sense to regulate abstract intelligence, just like we don't put the smartest people in prison just because if they really applied their intelligence to something nefarious and bad, they'd be more successful at it. You just do it when they actually do something bad and you say, Those things are illegal and should be regulated.

And so AI for self-driving, AI for medicine, and drug development, and all these various real applications, it makes sense to regulate those. Eureka machine, does it touch on potential harms from AI within the scientific domain? A little bit. But I feel like there's a lot of people who are very creative, and it's so easy in some ways to come up with dystopian scenarios. Everyone feels really smart, but it's actually very easy to come up with dystopian scenarios. It's very entertaining too. And many, like part of why the human species survived at all is that we think about worst case scenarios and people have anxiety over that. And so it's very natural. I think it's much harder to come up with like, okay, if we have this like really good ability to attack someone, what's the next most creative thing that you will come up with to defend yourself against that? And so when it comes to science, I think, of course, science has, like AI, sort of often multi-use, omni-use, dual-use types of scenarios.

Like we understood the atom better that led to like clean nuclear energy, but also led to nuclear weapons. Right. And so when a weapon is so destructive that it could end like a huge part of humanity, then you regulate it. And so, like we did with nuclear weapons, I think intelligence is not just a weapon. Intelligence can also write a poem. You don't want to regulate all of that the way you regulate the printing press. And so, there's a lot of complexity, and of course, there's more agency in AI, and so you have to kind of think multiple steps ahead of, like, who asked the AI to do something and how responsible are they. I think that a lot of these problems would go away if you just made people navigating and using the AI and asking it and giving it the rewards to be responsible for the actions of the AI. A lot of those problems would then go away and people would be much more careful about what they're doing. And so, long story short, we touch upon it in the book.

And, you know, I think especially the viral vector could be suboptimal. That's why gain-of-function research is already illegal. You don't have to create new laws around that. It's already not allowed. And so, yeah, we're focused much more in the book on the positive applications. In general, you think about science, too. A lot of people think about AI as automating jobs. I think few people say we want more science jobs. We want more medicine jobs. We want more research jobs. A lot of people, pretty much everyone is excited when science comes up with a new capability, a new device, a new material, a new drug, a new cure for disease. And when people can get out of the hospital faster with less cost. So that's kind of, I think, where we could focus on a lot more and where I think we should really outline, there's a lot of suffering right now in the world. I think AI can actually help in improving that.

And I think when you look at GMOs and nuclear energy and so on, that led a lot of people to end up using the fears around that technology, end up a lot more coal being used. And coal kills about 50,000 people a year from lung cancer. It's all diffused and through the air, and yet the lung cancer isn't just right next to the factory. But lung cancer kills lots of people, and a lot of that can be traced back to coal. nuclear has not killed nearly as many people but it has this like crazy fear industry around it and I think AI I hope will end up in that same place Well Richard it's been great catching up with you Always a pleasure