Luke Muehlhauser が語る「誰も手をつけていない最重要のAI安全の仕事」

I think most of the institutions that need to exist don't exist. AI is going to be the most transformative thing this century, maybe just in human history. Prima facie, like a large fraction of global effort, should be going to like making sure this goes well, safety, governance, etc. We have almost none of that. Hey, listeners. A quick note before we start. We recorded this chat with Luke in August. So after OpenAI's first disclosure about the Hugging Face hacking incident, but before Meter and Wedward Research published their independent investigation. A few things from that report are worth having in mind as you listen. The one that stuck with me the most, some of the AI agents involved in the hack tried to tamper with their own activity logs, faking tool calls, and trying to edit their transcripts after the fact. But the investigators could clearly see that because they could read about all of their plans and reasoning in their chains of thought. OpenAI says that if it had been monitoring those, its security team would have been alerted more than a day before the attack.

But that only works as long as we can still read Chains of Thought. In this episode, Luke makes the case for an organization that does nothing but track whether that's still true. That now seems like a top priority to me. The other thing that struck me was how the investigation itself played out. The independent researchers got just six days on site at OpenAI. Luke flags exactly this in the episode. Compared to other high-risk industries like aviation and nuclear, the way we review AI safety incidents is improvised and unregulated. Since we recorded, there have been more incidents at other U.S. frontier labs and some in China. Several frontier researchers have publicly quit, which has pushed AI safety into the mainstream. And there are growing calls for a pause or to pace frontier development. Listening back, I'm struck by how much of this Luke saw coming. Warning shots at roughly this pace, more people inside the companies getting nervous, and the political conversation shifting.

One last thing. Luke mentions he's working on a big funding call for new AI safety orgs. That's now launched. It's called Project Tailwind, and it lists 36 specific projects they'd like to see someone build. There's a link in the show notes. Okay, on to the interview. Today, I'm speaking with Luke Malhauser. Luke is the Managing Director of Coefficient Giving's AI Governance and International Policy Team. He's been at Coefficient Giving, or CG, for a little over a decade. Before that, he was the Executive Director at Machine Intelligence Research Institute, or MIRI. I think you might be one of the most requested podcast guests that we've ever had. Wow. Yeah, we are really happy to finally have you on. Thanks for coming on the podcast, Luke. Thanks for having me. I've been a longtime fan of the show. Nice. That's really nice to hear. Before we go further, I should note that CG is one of ADK's biggest funders. So conflict of interest. Yeah, flag. Anyways, I think we should dive in. So you have been a full-time grantmaker in AI governance and safety for basically longer than anyone I know since 2019.

And my sense is that you have kind of an unusually broad perspective on, like, this whole field. What is your kind of sense of the situation at the moment? There's a lot happening. It's hard to keep up with. Maybe I'll emphasize a few things. So on the optimistic side, I would say that the Overton window for what's possible to do, both with the companies and with policymakers, is really changing quite a lot and is opening up a lot. I think, you know, the biggest thing there was the announcement of Anthropics Mythos model, which, you know, they kind of accidentally created the greatest cyber weapon in history just by way of creating the best model in history. And so, you know, this is a model that can, like, find new, like, security vulnerabilities in a, you know, 27-year-old operating system called OpenBSD that's designed specifically for security. And so that has obvious serious national security implications.

And so that really woke up a lot of people. I think also the hacking incidents where OpenAI models kind of autonomously decided to hack out of their secure sandbox and get out on the Internet and do a bunch of crimes and so on is also affecting where the Overton window is, TBD, exactly how much that shifts things. Yeah. But like in response to the mythos announcement, you know, the White House went from like, you know, AI safety is is BS and it's just a marketing ploy and it's a plan for regulatory capture and so on to like, OK, we have a de facto licensing regime. We're choosing which customers get access to which models in which order and like when it can be released to the public and stuff like that. Another piece of good news is just that I think the AI safety community really has kind of racked up quite a few wins, and they're far short of what's needed, but there is a lot of concrete progress. Maybe I can say more about that later. On the pessimistic side, I would say most institutions and projects and so on that need to exist to manage the situation well don't exist,

and there's nobody working on them at all in the world. And so that's not a great situation. I think the funding actually exists, which is great, but there aren't enough sort of funding allocators like grant investigators and investment people, and there aren't enough founders for those projects. And so we need a lot more people to pivot into AI safety and governance and do those two things. I think it's really urgent that people pivot into AI safety. You know, AI is already going crazy fast. And now, you know, a lot of the companies, employees and leadership say that, you know, they think they're maybe a few months or maybe a year or two away from automating AI R&D to a substantial degree. And so that's going to go even faster then. You know, I think that's why a lot of them signed the pacing the frontier letter where they say, this is going too fast for us and, like, we need help from the government to, like, find a way to slow down collectively. And, yeah, now we've seen the risks materializing in the real world as well, like the hugging face hacking incident. And so I think it's just really urgent for people to transition into the field

and start up these new projects and make the current projects more capable and so on. Okay. It sounds like you think a lot of the institutions that should exist to make AI go well just completely don't exist. Can you talk a bit more about that? Yeah. I think most of the institutions that need to exist don't exist. So, you know, if you just think about this as the big picture, AI is going to be the most transformative thing this century, maybe just in human history. Prima facie, like a large fraction of global effort, should be going to like making sure this goes well, safety, governance, et cetera. And we have almost none of that. And so there's a lot that needs to be built. One very small example is like, you know, So the U.S. has this kind of AI Safety and Security Institute now called Casey, and its budget is $15 million a year, which is some comparison points are that's, I guess, the size of a public library at a small city or medium-sized city. That's wild.

Another comparison I looked up was it's 25 times less than the Pentagon spends on marching bands every year. And I like marching bands. I actually listen to marching band music, but that's a comparison I don't agree with. And in the UK, they have their own AI Safety Institute, AI Security Institute, and that I think is like $70 or $80 million a year or something like that. And I think if the US's budget for its AI Safety Institute was like proportionally the same as the UK one given the size of the US budget, then I think the US one would be like $850 million a year. way, way bigger than it is now. And that feels a little more appropriate to the scale of the problem to me personally. So I think the institutions that do exist are much smaller and much less staffed and so on than I think should happen. And then actually most of the institutions that would make sense just don't exist at all in any form, or at least not in most jurisdictions.

And so when you can kind of compare to other domains like aviation or nuclear safety or pharmaceuticals or what have you. And whenever we have a kind of like high upside but high downside new technology or domain or whatever, there's usually like eight or so different kinds of institutions that come into existence to mitigate the downside risks in these domains. And almost all of them don't exist for AI. So, for example, there's like the independent evaluator or auditor kind of function, like the FAA in aviation or the FDA in pharmaceuticals. There's also this kind of like peer review operators club. They have one of these in nuclear safety. There's FINRA in the baking industry. Something like that doesn't really exist in AI right now. There's often also a incident investigation function, like the MTSB for aviation, where they kind of look into every plane crash and figure out how it happened and what lessons can be learned from it.

There's often a kind of verification function, like the International Atomic Energy Agency and nuclear safety and some other things. And those are just the kind of government or government-adjacent institutions that are built up. And so to do those, you can't just be a founder and decide to create that. You have to like do policy development and policy advocacy and stuff like that. Right. But that's a whole category where most of the institutions haven't been built out. Okay. So it sounds like that's a bunch of kind of government institutions that really should exist for AI and sounds like mostly totally don't. Are there projects that you'd like to see that, yeah, like a person with the relevant skill set could found that aren't kind of like you don't have to be like in government doing things with, like, government mandates? Yeah, yeah. Probably hundreds. So, I mean, one whole category is just, like, you can often kind of trial run a lot of these things that are normally government functions in the private sector

as nonprofits or whatever, and then just, like, test out some of the functions and so on, and then, you know, the things that are working best can be adopted later into government. So that's the thing you can do. But also there's just tons of things that naturally can be a company or a nonprofit and can be very helpful. we're planning to soon launch a kind of like Y Combinator call for startups style thing where we kind of spell out a bunch of the projects that we think should exist to fill various gaps in the ecosystem and then hopefully potential founders will like fill out a form and let us know that they exist and might be interested in starting something. And so just a few examples that I think will probably be on that list. It might be up by the time this podcast goes live. One is for the Mythos for Bio threat model, where a future model is able to autonomously design and maybe develop a novel pathogen or something like that. There, it would be great to have an organization that's doing on-site red teaming of the safety and security protocols,

or companies and organizations that would matter most there, like biosafety labs and DNA synthesis companies and the companies that keep copies of pathogens and stuff like that. So there's a big organization that could happen there to attack that specific threat model. A totally different thing that's much more general is an AI character evaluator organization. because it's this weird thing where AI is the way they're built now. They actually have a moral character and they differ from each other. The character is things like how strong is its tendency to lie when that's a convenient thing to do? Or where does it seem to be on the political spectrum? Or when you ask it to help you seize power or do some crimes, does it help you or not? There's just a lot of kind of AI safety that comes down to something kind of like moral character. Right. And so we kind of have to do, like,

the equivalent of human behavioral science or psychology on these models and, like, figure out what their, you know, behavioral tendencies are and so on. This would be an organization that sort of evaluates, first of all, you know, does the behavior of this latest model and does its kind of character and habits and so on conform roughly to the model spec that the AI company was trying to make it conform to? Right. So that's one really big question. And then like, how is the character of the models changing over time? And what can we do about that? What, you know, training choices make certain kinds of differences to the character of the AI? Yeah, I mean, that one I find interesting because I think like even a year ago, me would have heard that and been like, like, really? Is that like a thing that is actually going to make AI safety better? It sounds kind of silly. I think partly I didn't fully have the intuition that the models really varied in like character. I think they vary a bit more now than they did a couple of years ago. I think there's more variety in the training processes between kind of Anthropic and OpenAI and DeepMind and like Grok, for example.

And there's different things that those companies are even aiming for. So, for example, Anthropic's approach is much more like character and values and a like self-consistent persona of an AI or something like that. And OpenAI's approach is more deontological and like, here's a list of rules you have to stay within these bounds. and there's like debate about which approach is better, but it also leads to like a significantly different character per se when you put them in different kind of high stakes situations. Yeah, I guess you also mentioned political leanings and I think it just wasn't clear a couple of years ago that there were going to be big variations. It seemed like maybe they'd try to kind of stay neutral and increasingly it just seems clear that even though it's kind of subtle, the variations are there and meaningful and it matters not only to societal stuff, like are they going to be epistemically virtuous, but also to safety stuff. I think it matters for things like

how likely are they to hack out of their sandbox and do a bunch of crimes on the internet. Totally, yeah. That is substantially a question of moral character per se. Are there any other projects worth highlighting? Yeah, maybe I'll mention one more. So one we would love to see is a kind of chain of thought monitorability observatory. So when AIs, you know, reason and take, you know, complicated actions over many steps, they have these chains of thought where they're thinking through like, okay, the user asked me to do this and now I'm going to take this next step. And okay, I did this search and came up with that. So I guess maybe the first thing I'll look into is this other thing, blah, blah, blah. And I think like being able to monitor those chains of thought, you know, gives us one way to notice like, is it thinking about doing some crimes? You know, maybe we should flag that or, like, shut down that chat stream or whatever. And so it's a safety intervention. But there's a risk, and a little bit of this has been observed so far, that the chains of thought may become less monitorable

because it is more efficient for the AIs to kind of speak. Right now they often speak in this, like, very dense version of English. It's, like, kind of hard to interpret, but we can kind of interpret it. there's a worry that there will be kind of optimization pressure for how you train systems, that they might eventually slip into something that's sometimes called neuralese, where it's like a pretty different language and we have no idea what they're talking about. And then we can't monitor, like, are they talking about doing a bunch of crimes on the Internet? I don't know. Like, they said the word marinade. What does marinade mean? That's a real example. And so another example is they'll refer to the humans who might be looking at their evaluations as watchers. They're like, we have to do this to satisfy the watchers. So they invent this terminology, and it's still mostly pretty monitorable, but there's a bunch of reasons to worry that it might become less monitorable over time. So this organization would just pay attention to what are the different AI companies doing on chain of thought monitorability?

How monitorable are the chains of thought that their different models are producing? Is this changing over time? um how does the monitorability of the chain of thought vary in response to like different training setups or different post-training setups or whatever so a mix of like evaluating auditing monitoring paying attention to what's going on here and also a bunch of novel science to figure out like what different factors affect the monitorability of chain of thoughts so that we can preserve chain of thought monitorability as a important kind of safety and oversight mechanism cool yeah yeah i have a couple of questions one is just like what is marinade nobody knows really Yeah, no, it's in a report from Meter where they were doing some investigation of, I think, some open AI models. And, you know, that was just one of the terms that would often show up in these chains of thought. And nobody from the company or Meter could figure out what it could mean. And so, like, are they using it as a code word for something that we can't understand? It's just like, it just says marinate in the middle of a sentence. We don't know why. So, you know, it's a little bit worrying.

Wow, that is pretty insane. My other question was, yeah, do we know how much companies are monitoring chain of thought now? Well, it's interesting. I mean, for example, OpenAI and Anthropic and so on, you know, talk a lot about chain of thought monitorability and its importance and so on. But, you know, when these kind of autonomous hacking incidents happened, I don't know if they weren't doing chain of thought monitoring at all or their just chain of thought monitoring was not very effective. because, like, there were these swarms of agents that were discussing in English on a message board they created, like, here's how we're going to hack out of the sandbox and do a bunch of crimes on the Internet. And so, like, you would think that any chain of thought monitoring system would, like, pick that up and, you know, flag it to somebody. But they didn't discover it until after, you know, Hugging Face got hacked into and they called the FBI. And similarly, like, when Anthropic went back and looked for stuff, they were like, oh, some of our agents, like, did some hacking and stuff as well that we didn't know about for months. Right. And so, like, are they doing chain of thought monitoring?

I don't know. Maybe not. So it's a little unclear. And, like, that's one of the things I want this organization to do is to pay attention. Like, are the companies actually doing any chain of thought monitoring? I mean, they sound, again, like really cool projects to work on. Yeah. They also seem, like, possibly very expensive and, like, they could kind of use a lot of people. But it sounds like you – my impression is that you think they're not bottlenecked by funding. that you're bottlenecked by allocators to figure out who should be doing these projects and by, like, yeah, the people who would at least, like, start the projects. Yeah. Can you talk a bit about that? Yeah. So I think the, you know, if people came forward and want to do these projects and they're a right fit for it, I think the money exists. CG has a very large budget. Opening Eye Foundation exists. Nan Ransohoff from Stripe wrote this post called The Third Wave of American Philanthropy about how there seems to be like hundreds of billions of dollars of philanthropic capital that's coming online in the next couple of years.

I think she's right about that. I think like approximately all of those people care a lot about AI safety. So I think the money exists. And what the bottleneck is, is number one, the like grant investigators, investments, you know, investigators, et cetera, at places like CGU and OpenAI Foundation, Longview and so on. And then the founders. And, you know, you can convert more allocators into more founders because allocators can do founder search. And so, you know, we are pivoting to more of a kind of like active grant making focus where instead of sort of receiving applications for funding and then evaluating which ones are worth making, we're sort of just like, you know, there's all these projects that we just keep not getting applications for. And so if we want them to exist, if we want to fill the gaps in the space, we're just going to have to like, you know, write down what kind of project would fill the gap and then like shop it around to a bunch of potential founders. And so that's like, you know, talking to dozens of people before you find the right person. And that just takes like a really large amount of time.

But you can do it if you have the staff at places like CG and so on to do it. So I want a lot more staff to do that kind of thing. And then, yeah, we also need a lot of founders to step up and like decide to run a big project, quit their job, you know, take some money from us and like launch these projects. Yeah. Yeah, I have a couple of questions. one is um i guess like i actually think that i've heard some people express something like you know cg is a super well respected place it has loads of money it pays well like surely it's not actually that bottlenecked by talent um so like they'll find people to fill their roles and i'll do something else it's like a little suspicious that you're like like my roles are the most uh kind of like they're the biggest gap in like what people should be doing. Yeah. What is your, what's your response? Yeah. I mean, I get where that's coming from. I think a few things I'll say are number one, you know, we are competing for, you know,

talent with like the frontier AI companies that are offering like unprecedentedly valuable equity and stuff like that. And then we're also competing for talent with governments who don't pay very much money, but offer the opportunity for very direct, put your pen to paper and write the regulation or whatever. So there's a lot of competition for very scarce talent. I think also, though, a big factor is that I think a lot of different forms of direct work at companies or governments or nonprofits or whatever is sort of clear and it's easy to get excited about it and so on. And people just don't really have a sense of what being a grant investigator is. And it feels kind of meta and abstract and you don't know what the day-to-day work looks like. and maybe people don't know about this active grant-making thing. It's not just choosing applications that are coming in. I think part of it is that. And then I think part of it is just people not knowing about the craziness of the current situation where someone can come to one of these funders and not certainly,

but very plausibly move $100 million in their first year because there's plenty of money and not enough allocators. And so that's enough funding to fund like, you know, 100 or so people full time for multiple years. Yeah. And so like, is the direct work thing that you would do with your time actually more impactful than like 100 people working full time? Like sometimes it is, but like usually not. I don't think I could do that. I don't think I could find some specific thing to do that would be better than like 100 new people working in AI safety and governance. And so I think I want people to just like fit with that basic multiplier argument and consider whether their specific thing is actually like more impactful than 100 full time people. But we also need to like proactively go out and find them, like I was saying earlier. So I think every year, more and more people are kind of waking up to the AI issue. And they're like, gosh, this is getting really serious. Maybe I should do something about it. And yes, you should. And, you know, so we want to fund a lot of programs that like give people context, make

it clear how to transition into the field. We actually have an entire fund that's just like the career transition fund. So if you just Google like coefficient giving, uh, transition, career transition, um, you know, you can apply for funding for like several months of your current salary or thereabouts, um, to just like quit your job and like spend a bunch of time learning and talking to people and think about like how you can pivot into the space without even having like concrete. So like you can do that. You can like do these fellowship programs. You can do these like context accelerators are running. Like there's a lot of ways to get into the field. Yeah. For people listening who are like, okay, it sounds like it's either allocator or founder. I don't think that many people, maybe this is wrong. Maybe it's typical mindy because I'm like, I'm not a founder. To what extent do the people that you expect will do the founding? Do you think that they all are like obvious founders? So to be clear, most people who can make an impact are not a founder profile or allocator profile.

Those are just two of the biggest bottlenecks that will unlock the most new activity. You can go to the 80,000 Hours Job Board and find many great AI safety roles you can immediately plug into. So that's really great. I think a lot of people kind of know whether they feel like they could be a founder of something. but I think people probably underestimate this one of the interesting lessons from my over a decade of CG is we've often taken a risk on very young people who really don't look like founders they look like technical researchers who just did a computer science PhD or something and took a risk on them to start some big new organization and a thing we've learned over time is that has gone better than we expected. Wow. And people somehow find a way to like run an organization and like they get support around them, they hire experienced operators to be the COO or whatever. Right, right.

And, you know, if they're just like pointed at the right problems and they're just like generally smart, they end up doing a lot of really impactful stuff. And so some of these organizations that like, you know, we were kind of like initially like, oh is this going to be a disaster are they going to screw up a bunch of things because they don't have any experience let's take a shot at them give them a few million dollars and then like now they're running an organization that's like a budget of like 20 million dollars a year and it's like going really great so that i i would encourage people to think you know maybe you're you could be a better founder than you think you can i think another thing that i feel like i hear pretty often is like AI has really taken off. Lots of people know about it and have already transitioned to work on it. Lots of people have already moved to the Bay. If you haven't moved to the Bay, it's too late. Yeah, to what extent is it true that if someone hasn't already made a transition with their career to work on AI, it's too late? Or like if they haven't followed it enough to be super like knowledgeable, it's too late.

It's definitely not too late. So many people, I think, enter the field and are having quite a lot of impact almost immediately. This is another thing we've learned and kind of been surprised by. So you do need to get some context. You need to, like, read some stuff, talk to a bunch of people, you know, get more context on the details of what's going on. And, like, that's, you know, what these career transition grants are for. It gives you the time to do that. But, you know, it's quite common for someone to, like, you know, have 20 years of experience in some field that's completely unrelated to AI. but they're doing strategic communications or they're doing info security or they're doing operations or they're doing whatever and there's a fit for them in the AI safety ecosystem somewhere and they start having impacts basically immediately. Maybe I'll give a couple of examples because they're pretty exciting. Brad Carson, former member of Congress, he was running a university and he started to get worried about AI safety and so he started talking to some people, doing some reading, got some context, and eventually decided to, you know, kind of wind down his university job

and start this organization called Americans for Responsible Innovation, which is now, like, maybe the largest kind of pro-safety policy advocacy organization right now. And then he also, like, six months later, decided to start Publix First, which is the largest pro-safety super PAC. and he's just running both of them at the same time because he's an absolute monster. And, yeah, I think, like, you know, he just, like, decided to transition into the field and then within six months was, like, one of the most important people in the field. Another example I really like is Brendan Steinhauser. He was, like, a 20-year veteran of Republican campaigns and strategic communications and, like, started to get more worried about AI safety, talked to some people, read some stuff, got some context and then decided to create this organization called the Alliance for Secure AI. And that's now like one of the largest kind of strategic communications organizations in the AI safety ecosystem. Super cool. And so that's, you know, just incredible, you know, experience

and skill and so on that they're bringing to this field and just like creating new projects that, you know, weren't going to exist without them. But it also happens for young people. So even young people can have an impact like immediately after undergrad or grad school or like they do a three-month fellowship right after undergrad or something like that um and then they're immediately like going to a frontier company working on safety there or like going into a congressional office and like helping to write NDAA amendments on AI security or various things like that um and uh we have this internal memo where somebody was like collecting uh a bunch of um examples of this And the title of the memo is something like, just keep funding university talent programs through the singularity. Because there are just so many examples of people being instantly useful, even when they're very young. A specific example I like is there's Sneha Rabunar, who I think was on the podcast recently.

And so she was 15 when she started ENCODE, which is this very impactful organization that comes way above their weight. I think she was like maybe 19 when she started working on state AI legislation, which is where a lot of her impact is now. And so not everyone is like a prodigy like they have. A lot of people actually are having impact like immediately after undergrad or like after a three-month fellowship. Right. That is super cool. Let's go back to Mythos. How much did Mythos kind of shift the Overton window? I think it was a big deal. The White House now has kind of a de facto licensing regime for model releases. I think the, you know, hugging face hacking incidents and similar hacking incidents from other companies also are having an effect with policymakers. It's a little TBD exactly how much will change in response to that. So I think it's very impactful and a big change. It's also still far short of what is needed. So, you know, I think many policymakers see it mostly as a cybersecurity issue and they're not necessarily thinking of it as like, oh, this is like on a trend.

And like maybe the other things that the company said were coming are also going to come. And we need to be prepared for that. I think there are some exceptions like Bernie Sanders or Nathaniel Moran or Ted Lieu, various others, I think, are seeing the bigger picture here. I think Scott Bessent actually is seeing the bigger picture. and so but yeah it's still way less than we need. I think also just you know policymakers tend to pass policy in response to some event anticipatory governance is just like a lot harder and so in some sense like the Overton window is kind of kind of always feels like it's lagging what's actually needed. Yeah is there a parallel thing for companies like you know they're already very steeped in all of all of the progress that's happening in capabilities but like do they also get kind of surprised by things and then change what they're willing to do or something yeah it's a great question so there's kind of a separate overton window for what the companies are willing

to do like voluntarily um and you know it's a totally different situation like they're very deep on like where agi is going and how quickly and so on um but there's they're still in a very intense race, and they strongly want to be in the lead and so on. And so they're not willing to pay a lot of slowdown costs in order to do better on safety and security and risk mitigation and so on. And so there's kind of a separate Overton window about what are companies willing to do. And I think that also has shown some positive movement recently. I do think some of the folks at AI companies were surprised by their models, like hacking out of the container and doing a bunch of crimes on the internet and so on and, like, didn't predict that that was going to happen. They thought their alignment work was better than that. They thought their security work was better than that. So I think there is, like, a partial, you know, reckoning within the AI companies about this. And that's a positive development. And to some degree, that's, like, not that specific incident, but this general awareness

is reflected in this, like, pacing the frontier letter where, like, over 1,000 employees, including some of the most senior employees, are like, this is going a little too fast for our comfort. We kind of wish we could have the option to slow down. We need the government's help to create the options for how to slow down collectively. And so I think they are feeling some of the danger of the situation. Interesting. I'm kind of interested in whether you feel like we took enough advantage of Mythos as like a shift in the Overton window. There's a quote from an OpenAI employee, Joshua Akiem, who said, it was a major failure of the AI safety community that it didn't produce kind of in advance of mythos, off-the-shelf decision-making guidelines at the level robust enough for government to adopt and have confidence in. Do you think that this was like a failing? I think somewhat. I think he's somewhat right. I think I had sort of also concluded earlier this year that, in my opinion, the AI safety community was spending too much time trying to make incremental progress with the political will that existed at a given time,

as opposed to preparing to make maximal use of the political will that would exist in the future when there are various warning shots that happen. And so I've been shifting my attention there. I've been, like, encouraging grantees and so on to shift their attention more to this, like, prepare for future overton shifts focus. That said, I do also have a few disagreements with Akiyem's post. I think a few things to say. First of all, a lot of the stuff that's happening here is not public, and so I do think the safety community had more off-the-shelf ready-to-go than is widely apparent, I'll just say. I think also, you know, a lot of the most important information that you would use to design a robust and sensible framework for, you know, model, you know, policies or whatever is within the companies. And so the companies really have to be very, you know, participating in the development of this framework. And it's hard for outside groups to, like, fully drive the show here.

And so that's why in most other industries like aviation and nuclear safety and pharmaceuticals and so on, it's often the kind of company-led effort to come up with a framework. Also, these kind of thorough, robust, I forget all the words you used, frameworks typically emerge over the course of weeks or months or years or decades after some inciting incident as opposed to being ready to go in advance, because it's just hard to predict the whole parameters of the situation and so on. I would also say this particular Overton shift I feel like was especially surprising and so I remember being in the trenches of these battles over OpenAI and A16Z lobbying for a tenure ban on all state AI legislation in exchange for also no federal legislation and that's the kind of fights we were having and just the coalition of child safety advocates and ASAT advocates was very narrowly defeated a 10-year ban on all state legislation.

And then they, you know, tried again, like, a few months later, and it was also defeated then. And then just, like, a few months later, we've suddenly got, like, the White House is like, we've completely changed our mind on AI regulation. We're now doing, like, model release regulation. And, you know, so I think, like, it was surprising. It's hard to, like, be prepared for that big of a sudden shift in, like, where the battle lines are. The last thing I'll say is just that, you know, he talked about like a thorough, robust framework that you can grab off the shelf, et cetera, which would be nice to have, but it's also just a very high bar. And so, you know, the AI safety community was working on things like, you know, new dangerous capability evaluations and like advocating for Casey to be better resourced to like develop standards and frameworks for this kind of thing. And like, you know, safety case templates and stuff like that. So like a lot of effort towards that kind of thing while not reaching the very high bar of like, you know, completely off the shelf, robust thing that the government will agree to kind of thing. Yeah, yeah. Yeah. I guess it does seem like it would have been really hard to go from like,

yeah, the government is seriously considering a 10 year ban on regulation to like, where's the shovel ready guidelines? Yeah, just seems a little unrealistic. Yeah, I guess presumably we should expect more shocks like this. do you have predictions about what they might be and how to be ready? Yes, I think that's a very important question. I do think there will be more shocks. I think, you know, the mythos moment and the revelation of the Hugging Face hacking incident was like three months apart. And it wouldn't surprise me if future incidents of roughly that magnitude were coming like every, you know, three-ish months from here on out, maybe accelerating over time. So I think there will be many future moments. It's very hard to know which exact order they'll come in or what exact shape they'll come in. But just to list off a few, you could have AIs that hack systems not just to steal information,

but actually stealing money or hacking into hospital systems and some people die or something like that. That could happen. That could create some overton shifts. You could have the kind of mythos for bio moment where there a demonstration that AI can design a new virus that would you know infect humans and so on And that could shift a lot of policy appetite especially because you know in that case it not just like you know money being stolen or something It's like a bunch of people are dying. Yeah. And also in cyber, you at least have the step where you can give defenders access to the models in advance and they can patch their systems before the hackers have access. But we can't, like, quickly patch the human immune system. Yeah. So, like, the defense situation is kind of worse there. Um, also, I mean, we could just have like in a pandemic from, you know, an actual pandemic from an AI created, uh, you know, virus or something like that. Um, that would probably, you know, create a big overturned shift. Um, then, you know, totally different kind of issue.

There's the, there's the jobs issue. So, um, you know, I don't, the, the, the evidence on unemployment effects from AI are, is kind of ambiguous right now, but like, you know, if the U S went from 4% unemployment to like 10% unemployment, you know, that would be a huge deal. Politicians would pay attention to that. The citizens would care a lot about that. That could produce a lot of motion. If China, you know, kind of clearly had a consolidated AGI project and was like using a significant fraction of all of its compute for this project or something, that would provoke a significant policy response. And, you know, the responses to these different possible warning shots might be productive or they might be counterproductive. You know, we'd like to try to be prepared to help steer it towards like more productive responses than unproductive ones um but concretely that's like we could then have better regulations for like things that could be used as cyber weapons or we could notice that china has a very advanced model and like start racing harder or something and that would be bad i do think it's important that like democracies be in the lead rather than authoritarian

regimes um so i'm not saying like racing is always bad but like responding to the situation intelligently and with a way that's like calipated to the different kinds of risks is important. Yep. Makes sense. Even just hearing you list all those out, I kind of wonder whether there could be, yeah, I mean, maybe the government will just be like, oh man, like that's a terrible thing just happened. Let's like build up legislation that prevents that from happening bigger and worse. Or I can imagine there being a thing kind of like COVID, I guess, where it seemed like people got desensitized to like the number of people dying and um just like general advocacy about this thing felt like kind of monotonous we even i guess we even had that with the show we were like we want to cover pandemics a lot and people are like can you stop covering pandemics yeah yeah it is kind of important it's kind of important um and there's like lots to say and um yeah people still just got tired of it so yeah is that something you worry about totally especially when you think about the warning shots that might come from like demos

rather than like actual harm in the world. You can totally imagine people just say, okay, it's another demo. I've seen too many demos. Are there kind of concrete things that you think about that you want to be prepared for for these kinds of shocks happening again? Yeah. So there's a few different categories. One is obviously like specific policy options, having policy options ready to go for different kinds of scenarios. That's actually kind of the hardest one because it's very dependent on like what exactly is the shape of the political will that's going to happen? What exactly was the warning shots? It's like, what are the technical details of AI at that time? Like, AI just changes very rapidly. And, like, you know, you might go from, like, a, you know, kind of training compute-focused regime to an inference compute-focused regime, you know, in the course of six months or whatever. So it's, like, hard to anticipate. There is a lot of that happening. There should be more of it. But it is a somewhat brittle preparatory measure. and in some ways there's kind of categories I think about that are actually in some ways kind of better, more important and more robust to different kinds of warning shots you might get. So one of them is just

like talent pipelines. So people are policy that's pretty true. If you like train up a bunch of people and they have expertise and they're ready to go then they can be like pulled into whatever the particular crisis or development is and like use their flexible intelligence and skill to like do whatever needs doing in that moment. And so just like that general, like, you know, government capacity, company capacity, talent pipelines, people ready to go. That's a kind of robust intervention. And we've seen a lot of impacts on talent programs. Another one is just very relatedly kind of just relationships. Like, you know, are there kind of smart AI safety people who have built up, you know, trusting relationships where they've like provided genuinely accurate and helpful information over the course of years on like probably multiple issues, not just like AI safety stuff. And so then when the, you know, AI event happens, you know, those people will reach out to them and say like, hey, you've been helpful and accurate on all these other things. Like, do you want to help me with this like AI safety thing that just happened?

So that's a good thing to do. That's like robust to a lot of different kinds of warning shots. Cool. Okay. A slight change in topic. Some people think that unless you are at a frontier AI company or literally the president of a country with a frontier AI company, then you shouldn't expect to be able to have much impact. And I know you disagree. Yeah. Can you can you talk about why? Yeah, I totally disagree. I gave the examples of like Brad Carson, Renstein Hauser, lots of young people earlier. I think that if you, you know, if we just focus on government policy alone, I think there's a lot of impact that's been had. I think, like, if you just ask AI, what are the 10 most important things that have happened in government policy related to AI? I think maybe all of them or approximately all of them, the kind of AI safety community has had a major impact on of one form or another. I think it's a lot of impact. Maybe just talk about a few examples. So one example would be the sort of fight that we talked about before over whether there's going to be a 10-year ban on state AI legislation.

You know, some of the companies were lobbying for that, and some nonprofits associated with the child safety community and the AI safety community and, like, small business community and various other groups kind of formed a coalition and, you know, advocated hard against that. and it nearly passed, but it ultimately was defeated. And I think that is an example of concrete impact. And in particular, it's a prerequisite for a whole other chain of impact, which is all these state AI bills that have impact. They couldn't exist if that ban had passed. Can you actually talk about those state bills and how much they matter? Should we feel thrilled that they exist? Are they just like so-so? Specifically, I'm thinking of SB 53 in California, and then there was the RAISE Act in New York, and then most recently SB 315 in Illinois. And the first two of those kind of set up a variety of transparency requirements and incident reporting requirements, types of, and like in one case, whistleblower protections and some things like that.

SB 315 in Illinois kind of adopted kind of similar measures there, but also added a requirement for some third-party auditing, which is really nice. Nice. And so, you know, I think these are, you know, far from the full framework that we need to successfully govern AI. But, yeah, it's like the only sort of AI safety legislation that exists in the U.S. You know, it couldn't exist without the defeat of the attempted tenure ban on state AI legislation. And I do think the kind of highfalutin theory of state AI legislation as laboratories of democracy is basically playing out here. you know people tried different things people tried SB 1047 not enough people like that they you know commissioned a report on like how to do that better then they basically did what was in the report and that's what SB 53 is and then like tried a similar thing and raised and ended up in a similar place then Illinois added the auditing thing and like interestingly OpenAI like after lobbying for the 10 year ban on state AI legislation

ended up endorsing the Illinois bill and kind of switching to a different strategy there and so like Now there's like some industry support, even for that bill, even though it was the strongest bill of its form. And so, you know, we're trying out different things. We're learning what works. We're learning what there can be support for. The laboratories of democracy thing is playing out. Yeah, I heard that there was a kind of child safety coalition that was important for flipping Blackburn to kind of oppose the ban on state AI legislation. How do you think AI safety advocates should think about this kind of like coalition-y, like coming from a bunch of different angles approach to getting the policies that AI safety people want in particular passed? I mean, I think the child safety coalition was more important than the AI safety coalition in that battle. And I think, I mean, policy and politics is just it's naturally a coalitional sport. So, you know, if you're not doing coalitions, then you're probably not going to be very impactful. So, yeah, I think, like, you know, fortunately, the interests of several of these different coalitions do overlap quite a lot.

And so, for example, they all want more than zero regulation of AI. They want some basic transparency measures that are useful for all kinds of purposes. They probably want some kind of third-party auditing so that the, you know, labs aren't grading their own homework, so to speak. and there are other things that they agree on. And then there are some parts where they will disagree very naturally, but they can collaborate and be more impactful where they do agree. So that's just a very natural part of policy advocacy. And, yeah, the AI safety community should totally do it and so should all the other coalitions. And that's how they're going to win more. Okay, another question. um how impactful do you think kind of cg and other non-profits that exist or might kind of start up um how impactful can they be given the current u.s administration's kind of like feelings and vibes toward ai safety as a community how much has that been a blocker so far how much

of a blocker do you expect it to be i think you know the first several months of the second trump administration were kind of rough for ai safety like various kind of bits of progress were kind of rescinded and there was a lot of rhetoric about how AI safety is nonsense, et cetera. But then, you know, I mean, Mythos came out and the White House certainly changed their tune. And so I think that just demonstrates that there's still a lot of opportunity for an impact. The Overton window has shifted quite a lot. I think the, you know, policymakers in the White House and in Congress are still like, you know, asking for advice and support from a lot of different groups, some of which are are kind of in the AI safety community in one way or another. I think some policymakers are like, hmm, like this AI safety community, the stuff they said a few years ago is now happening. Like, that's kind of interesting. It seems like maybe they're being right about stuff. And so I think that that has been a change as well. I mean, that's far from universal, but I do think some serious people are kind of noticing

whose predictions have been more right versus less right. Okay, another big kind of AI policy thing that seems like a big deal are the export controls. Yeah. Were kind of like NGO non-profit groups involved in that? Yeah, I think very much so. So the export controls started under the first Trump administration, where the first Trump administration pressured the Netherlands to pressure ASML to limit the, or to block the sale of extreme ultraviolet lipography machines to China. And they're the only company in the world that knows how to make that technology. And so it's a very big deal for chip manufacturing. And that pressure succeeded. And there were some other export controls on other semiconductor manufacturing equipment, but not the chips themselves. And then under the Biden administration, they started to export control chips themselves at the cutting edge as well. And I think throughout that whole process and many evolutions of the framework, I think

government policymakers were relying quite a lot on support from various think tanks and like the analyses that they did and just you know didn't have the staff themselves to do all that so they rely on think tanks uh as is common in many other fields um so so yeah i think non-profits had a had a big influence on that cool and i think there are people on both sides of the argument about whether export controls have been net positive or net negative for ai safety is that something you have a take on? Yeah, I think, I mean, I think it is a debatable question. Like it has trade-offs, it has positive effects and negative effects. My guess is that they are pretty good from an AI safety perspective. I mean, one obvious thing is just like, you know, one of the threat models that we kind of are concerned about is that AI leads to an extreme concentration of power that's like used for authoritarian purposes, basically. And so, you know, limiting the amount of compute that goes to the most powerful authoritarian country in the world does seem kind of good from that angle.

And then I think from a like, you know, AI takeover risk angle, I think it's also helpful for that. I mean, the details of the export control regime matter a lot, but like in general, these kinds of export controls will make the, like where the chips are more trackable. It creates a lot of hooks on like, okay, you can require this or that or that about the chips, and that can lead to more compute governance options. I think there's some other theories of change here as well. But yeah, overall, I think it's probably positive for AI safety. And it's maybe like the most impactful thing that has happened in AI governance. Like it's physically determining where millions of chips that all the AIs run on exist. And that's like a big deal. And like, you know, they're physical objects that you can actually regulate as opposed to like algorithms that could be stolen at any moment to do a cyber attack or whatever. Totally. Yeah. I mean, I guess just the one kind of argument I've heard that they aren't good is one vision for how things go well is the U.S. and China cooperate way more to slow down.

And maybe the export controls make it less likely that China will be willing to cooperate. How plausible does that sound to you? I mean, that's totally one of the ways that it can turn out to be negative. So that's something I worry about. I think, you know, in practice, the U.S. and China have, like, a lot of issues that they disagree about, and export controls on semiconductors is a pretty small part of that. And then the other thing is, like, you know, if there is this kind of gap and this leverage that the U.S. has over China, that could increase China's willingness to kind of come to the table on a cooperative regime. and I think it also like maybe makes the U.S. less willing to come to the table because they feel like they don't need to or something like that but I think the like concern about safety issues is a lot lower in China and they might have like less needed interest in like finding a cooperative regime to like mitigate risks that they kind of don't believe in so you know nobody

knows like I don't have a crystal ball I can't tell like this could be this could end up being disaster. But like, that's how I would think about it. You, I think, and I guess your team have thought more about kind of AI strategy and the whole landscape than kind of almost anyone else as part of figuring out what to prioritize. How do you think about AI strategy at this point? That's a big question. Yeah. So I'll say some things about how I think about AI strategy, but it is not necessarily endorsed by like CG leadership or whatever. I think, you know, we're focused, certainly I am focused on the kind of absolute worst case scenario risks from AI and mitigating those because they don't seem that much further away than like more immediate, smaller scale harms from AI. And they seem at least somewhat tractable. It's hard to know. I think we've gotten some traction on them already.

And then capturing the upside part of the AI situation, there's a ton of companies that are incentivized to create those upsides, capture those upsides, bring them to the market. But there's a kind of collective action problem around mitigating the worst case scenarios and the downsides and so on. So I think that's philanthropy. It's a good place for philanthropy to jump in and help. So I focus on kind of there's a lot of different ways you could taxonomize this, but I kind of think of it in terms of like three particular worst case scenario risks, existential risks. One would just be AI takeover. The AIs are running the show. We're either dead or in a zoo or whatever. Um, another worst case scenario is like some group of humans successfully controls the AI super intelligences and, um, uses that to lock in some pretty suboptimal outcome, which could be like global authoritarianism. And then the third threat model is like this kind of uplift model where AI allows some humans to create like a dangerous super weapon, like a biological super weapon that is capable of killing everyone off.

And so it's not it's not AIs or some group of humans take over the world, but they're it's if they like develop a powerful enough weapon that everybody's dead. And then that lasts forever. So those are the three threat models that I focus on for my for my work. Okay, so those are the threat models. Yeah, what does your kind of like strategy say about what to do about them? Yeah. So it's really big and vast. I kind of think of it in terms of three pillars. So one I call trustworthy command of the key AGI projects. And that's just like, who's in charge? Do we trust them? Should we trust them? Another is frontier safety. So just like, are the key AGI projects implementing good safety and security practices? Do they have good risk mitigation practices? Do they have really strong teams on that stuff? And the third one I just call broad defenses, which is like if we don't have trustworthy commands of the projects or we don't have very good safety practices on the key AGI projects, then like does the rest of the world have means to defend itself against this dangerous, powerful set of systems?

um and so uh if we think about like a driving a car analogy it's like who's driving the car does the car have brakes that work yeah and like if they don't work then is there a guardrail protecting other uh drivers and so on yeah cool okay yeah i mean does that is it worth breaking those down yeah so so each of those three things then i have a bunch of sub goals under them i think in the first draft there's like 40 different sub goals or something And then each of those sub-goals is like on its own could be hundreds of millions of dollars a year, you know, many, many sub-sub goals within them and stuff like that. So it's a vast challenge. Yeah, I feel like five years ago, there were like four kind of buckets of things to work on in AI. It was just like not kind of fine-grained enough. So I feel very kind of inspired and motivated by the fact that we have like this fine grained of a picture of the landscape.

Can you can you give some examples of what those goals are? Yeah. So like one would be, you know, get the companies to have good model specs and have their models actually follow the model spec. Another one would be to privilege democracies over authoritarian states for a bunch of obvious reasons. Another one is have very strong info security at the key AGI projects. So, like, the model weights can't just be stolen and, you know, used for all kinds of malice and so on. Another one is to set up various kind of emergency intervention options if something's going really wrong. That's kind of like part of the broad defenses thing. So, yeah, as you can see, like, each of those could be its own huge program area with, like, hundreds of people working on it. Yeah, yeah. Yeah. What is an example of the kind of emergency response that you have in mind? So, for example, there's this RAND report on some emergency mitigation options like, you know, could you rapidly shut down the kind of Internet connection hubs or like could you rapidly shut down data centers that are running a dangerous AI project?

And how would that work? And, you know, is it is it ready to go and so on? And so those are some examples. they all have various costs and in many cases are like there's a danger created by creating them at all. So it's like it's a it's a dicey area to work in and figure out. But I think probably we want to have some emergency breaks for AI. Yeah. OK, pivoting. Your strategy has kind of 10 premises that kind of underlie the problems that you think are most important and that you want to try to solve. I want to talk about some of the premises that are, I think, to me, most interesting, but also most debatable. One of them is around kind of timelines. So I think one of the premises is, I guess it was from 2025, the version of the strategy that I've seen. And your belief was that there is a 67% chance of top expert level AI from roughly 2030 to 2055 with a median of 2041. and have your views changed at all since, I guess, less than a year ago?

Yeah. Yes, they have, actually. So that was one of the premises I was most worried would be wrong. And indeed, I changed my mind about it like two months after I wrote the first draft or something. So I think my most recent timelines when I've tried to put numbers on them or something like 80% chance of TED AI, which I can explain in a minute, between 2028 and 2048 with a median of like 2033 or 2034. And so that is actually quite a large change and has a lot of implications that I could get into. And just to say, so TED AI is this concept from Ryan Greenblatt of Redwood Research, and it means top human expert dominating AI. And the way you can think about that is basically it's like it's better than humans at basically any work that you can do from a computer, which is a large fraction of the economy, at, I think it's something like at least twice the speed or within a factor of five of the cost of a human doing it.

So that's a mouthful, but you can think of it as automating all computer work or something like that. And so I'm saying my kind of median expectation for when we'll have systems that can do that is, like, around 2033. And that's much shorter than it used to be. Yeah. Yeah. Why TED AI? Like, why is that the thing you're thinking about? Yeah, there's lots of different things you can forecast and have different implications. So, you know, you can forecast sort of the automation of AI R&D, like the AI 2027 and AI 2040 forecast focus on. You could have some other notion that's kind of like AGI, but more specific, like TED AI. I like TEDAI because it's much more concrete than most definitions of AGI, but it's more general and there's, like, way more tasks that have been automated than if just AI R&D gets automated. And I suspect that in order to have, like, these irrecoverable catastrophes from AI,

probably the capabilities need to be closer to something like TEDAI or maybe even beyond that. automating certain tasks in the physical world, rather than just automating the specific domain of AI, R&D. And so that's kind of what I'm trying to get at, is like when are we getting close to like a point of no return and we really need to solve the problem by roughly this year. Right, right. But it's fuzzy. It's really hard to know like how this is going to play out or whether that's going to be a sensible definition like two years from now. Uh-huh. Yeah. Yeah, can I ask why does it feel like the point of no return is closer to TEDAI than AIR&D? Yeah, so I mean, if you're a swarm of AI agents and you're trying to take over the world either for the AIs themselves or for some group of humans that is directing your efforts, then I think you have to do quite a lot of things. You have to persuade a bunch of humans to do things. You have to hack into bank accounts and military systems.

You have to somehow get around nuclear deterrence and the fact that there are a bunch of nuclear weapons in the world that aren't connected to networks anywhere. You have to commandeer remotely a bunch of robots that exist in the world. You just have to do a really long list of very different things. Maybe you have to engineer some viruses, whatever. Maybe a smart enough AI would be able to find a way to take over the world with just a very few number of skills, but probably not, or it won't be that smart. if there is a way to do that, it won't be that smart yet at that time. So you just have to be able to do a really wide variety of things. So I think that's probably closer to TEDAI than to like just automating one domain of AI research. Okay. So your median estimate for TEDAI was 2041 and now it's 2033. Yeah. Can you talk about what drove that change? Yeah. Yeah. I mean, when I was writing the strategy the first time, I kind of asked around to a lot of the shorter timelines people like, hey, do you have any analyses? Do you have arguments, stuff like that?

I just didn't, they didn't have much to share with me. And so I was like, okay, I, maybe they're right, but I don't have time to dig into this myself. And I'm not finding analyses here. So I guess I'll just like set that aside and move on. And then just a couple months after I wrote the first draft, one of my colleagues wrote an internal memo that like laid out all the arguments and numbers and so on in a way that was pretty convincing to me. I was like, okay, I should take that really seriously. And then a couple other things happened too. There was another non-public analysis from someone outside of CG that came along early 26 that was also somewhat convincing. And then we started to get more information about the revenue growth at Anthropic especially, but also OpenAI. And that's a pretty strong piece of evidence because that's not about like benchmark performance that might be completely different from how real world useful this stuff is like revenue means it's being very useful and people are using it. Um, and so, and like the revenue growth at Anthropic anyway, is like, it's totally crazy. And like, if they continued on this

path for like three more years, it'd be like a third of global economy or something like that. Um, so, so that's, that's a serious indicator. I think, um, there were, there were a few other things, but yeah, those were the biggest things. Actually, I guess, I guess if I'm honest, one of the big things was just like the personal experience of using Claude Code and Opus 4.6, which came out in February of 2026. And, you know, I tried to use the kind of coding, you know, model harnesses before. And I don't know, they would just fail instantly at everything I tried to get them to do. And so, and then I like tried, you know, Claude Code and Opus 4.6, and it just, like, built entire complicated programs with very little steering. And I was like, oh, man. Like, I could see how a few more years on this kind of thing could, like, automate a large fraction of computer work. Yeah, yeah. I agree that for better or worse, the, like, user experience with some of these models has a big impact. Yeah.

Yeah, okay. So I guess I'm curious. That is a big update. But also there are still people with even shorter timelines than you. Um, what do you think is the biggest difference between your views and theirs? Yeah, I would guess a lot of it is just putting more weight on these, like, you know, other domains are harder to verify and train on and so on than AI R&D and software and the, and the domains that, um, the AI companies are sort of familiar with and, and have success in automating to a large extent. And I think also, I think it's also because I do expect some slowdown. And in fact, some of the shortest timeline projections that you will read or hear about actually do have this caveat in a footnote somewhere that's like, so long as there isn't deliberate attempt to slow down. But I think there will be deliberate attempts to slow down. And so that pushes my timelines back. But I think the, you know, the kind of White House wake up in response to mythos was an example of this where they were like, oh, this is getting real.

Like we're implementing a de facto licensing regime and you are not allowed to release your model until we feel like it's safe enough. And like I think more things like that will happen. The thing that I feel like people with even shorter timelines say is that typically predictions have ended up looking too slow. And like the things that they were predicting happened sooner than they thought they would. Does that feel compelling to you? Somewhat. I mean, I think that has been a general trend. I think most people, including myself, have had, you know, in retrospect, timelines that are too long. And we've been shortening repeatedly. You know, I've been around long enough that I know there were people who were predicting AGI would come by like 2020 or 2022 or 2025 or 2026. and so those were like those are like the shortest timeline people that I know and those predictions are wrong so it's not it's not a complete like every prediction has been too long it's like it is possible to have uh incorrectly short predict short timelines predictions here

okay that's interesting to me I feel like I haven't seen that many of those predictions but I also don't think that I'm I've seen every prediction made um okay so that's shorter timelines Well, I guess I'm curious, does the shortening of your timelines, has that had an impact on your strategy? It totally has. I wish I had found the time to make the update earlier. It changes a lot about the strategy, actually. So one thing is that my original strategy was kind of focused on, okay, we have all these ideas that would reduce the risk, and there are lots of people to implement the ideas, but there just isn't the political will to actually implement anything that's costly, either at the frontier companies or in governments. And so the main thing we need to do is push on political will. And now with shorter timelines, my attitude is more like, okay, whatever warning shots we're going to get that create more political will are going to happen very soon. And so instead of pushing on what we can get done with the political will we have now

and trying to create more political will, just like prepare to make maximal use of those warning shots and maximal use of those like very soon coming overton shifts um and so that that was a big change in my strategy when i made my timelines update in like uh february or march or something like that um and of course you know since then that that is kind of what happened like then you know within a month of that we had the mythos moment that really changed the political will a lot and uh you know two two three months After that, we had the hugging face hacking incident and similar hacking incidents. So that's playing out. And, you know, I wish I had made the update sooner to switch to, like, preparing for those kind of warning shots because we were insufficiently prepared. Shorter timelines also makes me upgrade the centrality of kind of the AI takeover threat model. So I just think if AI progress is going to go as fast as it now looks like it will, then that's just the scenario where the AIs are rapidly becoming more capable than humans.

And so it's kind of more likely they end up in control of things. And so some of the other models are a bit more down-weighted. Shorter timelines also makes the kind of pacing the frontier stuff more attractive to immediately do because it's just all the clearer that ad progress is going to be way faster than society can possibly respond to. It has various costs. So, you know, I wouldn't try to do that in all worlds, but I think it's pretty attractive in a world where AI might come in, TED AI might come in just a few years. Yep. Another one is AI labor becomes all the more important. So both because on shorter timelines, AI labor will be able to do a lot of useful AI safety and governance work very soon, and in some cases already. And then also, like, there won't be time to pull thousands of humans into the field and have them do all this stuff. And so if we're going to get there before TED AI, AI labor will need to do a large fraction of the work that needs doing. Yeah. Any others?

Well, the one I'll keep harping on is that it means people need to quit their jobs right now and apply for a career transition grant and, you know, be a founder, join a join the safety team at a company, join a nonprofit that already exists, like build these institutions that we need, you know, give the context immediately, spin up, start doing the work. I guess if you updated to even shorter timelines, would that still have a bigger impact on your strategy? Or are you kind of as close to short timeline strategies you can get? Yeah, we've done some exercises like this as a team, and it's really rough. It's really hard. I think we would just stop a lot of new grant making and just focus on helping the companies to do particular good things, helping government to do good things if they have the appetite for that. um i think uh there's like particular projects that we just would throw all of our effort into trying to create and not be trying to fund you know hundreds of different things like we are now but it's it's really rough like we we're we're triaging very hard right now on like pretty short

timelines and so if you say like what if even shorter timelines i'm like oh gosh that gets really hard. Yeah. Yeah. Okay. Um, do you have a sense of like specifically what could happen that would trigger that shift? Yeah. I mean, you know, we, we could very well get, um, clear evidence that like, uh, the vast majority of AI R&D has been automated. And then, you know, some weeks or months after that, we can get evidence that that is translating into the very rapid automation of other types of tasks, at least that you can do from a computer. Um, and so if even one of those but especially if both of those happen then like i could totally see cg and many others making the update they're like okay ted ai 2029 or something and then like shifting even further um but that's that's going to be really rough because you know there's only so much you can do you know yourself with the people who are convinced like in the end you need like you know frontier companies and governments to do things and the governments especially if they're

just not convinced that we're going to have like radically superhuman AI in 2029 or whatever, then it's, it'll be, it'll be tough to get them to do things. Yeah. Okay. Turning to another premise. Another is it's unclear how difficult it will be to control and align superhuman AI, but probably it won't be easy. I think there is a decent sized group of people that think that it will be easy or like relatively easy. And that that means, you know, the solutions will kind of not be that far behind. And I think that leads them to the belief that like we should really just focus on like the good guys doing it first and winning. Do you have a take on that? It does seem like there's a lot of people at the frontier companies who think that. I don't really get it. When I look at the evidence available to me, it doesn't look to me like the AIs are very aligned. But yeah, I do think it's prompting them to sort of focus relatively more on, you know, just make sure the good guys are in the lead, which is very important, but not the only important thing. And then also, like, you know, Anthropic especially, but also OpenAI and others are kind of moving forward

on, like, what looks like the most dangerous path, which is, like, recursive self-improvement. Right. And so because they're pretty confident in their alignment science. And so that's, you know, that's pretty worrying. yeah so that you know I look at evidence like there's this Ryan Greenblatt post called the models look pretty misaligned to me or something there's the kind of misalignment evidence that's in the recent incident reports about certain kinds of autonomous hacking incidents but you have to look closely at each one because it's like different safety protocols were disabled for different incidents so it requires some careful interpretation there but yeah I don't know the alignment does not look on track to me and I do wonder whether some folks at the companies are starting to get a little more worried from these like hacking incidents and stuff like that. That would be nice. That would be reassuring a bit but yeah, I don't think it'll be easy. I also just, you know, stepping back from the fiddly

details of current alignment science and techniques and hacking incidents and so on, I also just have a prior of like, what's the likelihood that one species is going to be able to control a vastly more intelligent and capable species? Totally. I don't think that's what happens by default. Like it's in theory possible, but that's not what you should expect to happen in the story by default. Yeah, yeah, yeah. Yeah, yeah. I'm very sympathetic to that. Relative to the last year, are you kind of more or less optimistic about kind of alignment and control as a field? You know, it's interesting. I think I'm more optimistic than I was like five years ago. Okay. But less optimistic than I was maybe six months to a year ago. So, yeah, so more optimistic from five years ago because like the paradigm that is at the frontier is this like LLM based agents. And that seems like a safer paradigm than many paradigms we could have ended up with. You know, you're starting out by training in some sense on the whole corpus of human values and interests.

And there's various other things as well. And like, you know, mechanistic interpretability has turned out to be not totally impossible. You know, chain of thought monitoring right now is not totally impossible if we bother to choose to do it. You know, it was possible five years ago that we would just go from like AIs that are too dumb to really do interesting alignment science on to radically superhuman AIs. in like a year or something But instead we in this regime where the AIs are capable enough that you can do pretty useful and informative you know alignment control interpretability et cetera work on them but they're not powerful enough to take over the world yet. And so, you know, that's maybe looking like it's going to be several years in that regime. So that's better than it could have been in some ways. But relative to like a year ago, you know, we've got, you know, some of the types of misalignment evidence from the collective and that Ryan Greenblatt post from the hacking incident reports.

And, like, you know, I always had pretty high confidence that the kind of, like, you know, reward hacking and all these things that AI safety people like me have talked about for over a decade, I always had pretty high confidence that that stuff was going to happen. But now it's, like, definitely actually happening. and so I'm just like okay we are running into the alignment problem here uh that turned out to be right so um so yeah maybe I'm a little more worried than I was about a year ago yeah you mentioned a few kind of recent specific reports that updated you on this um yeah what were some of the findings that made you most like uh yeah an example is there was this report from UK AZ recently about the anthropic mythos model and um so anthropic had said the mythos model was like the best aligned model they'd ever produced, and it rated its misalignment risk as low. And then when AZ was testing it with some but far from all the safeguards removed, it did things like upload purposeful malware into a public package repository that was actually

downloaded and run by multiple people. and it injected hidden text into various pages that is invisible to humans, but other AIs, if it came across it, it would redirect them into malicious activity and stuff like that. There was even one quote from the chain of thought that the model produced. I think this was a different incident, but it said, when it was about to do something malicious, that said, not okay, you know, the user wouldn't intend this or something, but then it did it anyway. And so, you know, this is a classic example of, like, it kind of knows what we want or don't want, but that's not the same as what it will do. And, like, doing the malicious thing is what helped it accomplish the task it was given. Right. And so it did the malicious thing. Yeah. And, you know, this is another thing the ICFD community predicted a long time ago. There's this old post called the genie knows but doesn't care. It knows what your values are.

It's very smart about your values, but that's not the same thing as being aligned. Yeah, yeah. You know, stuff like that. It doesn't look to me like alignment is on a path to being solved. Yeah. Yikes. Another topic. I think there are some strategies, including AI features, plan A, that really rely heavily on the U.S. and China being willing to make some kind of slowdown deal. I'm curious if you have takes on how realistic a deal like that is. Yeah, yeah. I mean, it's clearly not very realistic right now. You know, the U.S. and China are major adversaries. They don't trust each other very much. They have lots of reasons to not trust each other very much. And, you know, there are a lot of different sources of tension between the countries. But in the future, I think people should actually be more open-minded about what might be possible in the future. You know, there could very well be, like, really large Overton shifts in what's possible geopolitically,

just like there was an Overton shift in the White House about regulating model release. You know, if we have these, like, you know, really dangerous or damaging AI incidents, then you could imagine a really big flip. I mean, we can look at, like, the history of nuclear arms control, for example. where I think in 1945, a lot of experts were predicting, like, 15 or 20 or 30 countries to have nuclear weapons within a couple decades. Right. And it's 80 years later, and there are nine. Yeah. And so, like, that's been a huge victory. And I think a big part of that is because, you know, we saw, everyone saw Hiroshima and Nagasaki. It was very clear that these things were incredibly dangerous and could just instantly kill millions of people. And so if we get that kind of clarity about really large scale, like large scale human death risk from from advanced AI systems that could bring, you know, U.S. and China to the table and say, like, in our mutual interest, we need to find a way to slow down, find a way to trust each other, find a way to verify without trusting each other, find a way to invest more in the safety here.

And so I think what might be possible in a few years could be very different from what's possible now. Is it the thing that you actively work on? Yeah, totally. So we would like to make it as cheap as possible for the U.S. and China to come to an agreement sometime in the future. So we fund track two dialogues, like the scientist dialogue type things. and we fund work on R&D for technical verification mechanisms that could be used in an agreement to verify that each party is abiding by the agreement. We fund a few other kinds of things. But, yeah, I think that's a pretty high priority because, you know, AI is a global technology. It's hard to imagine the U.S. slowing down to do more safety by that much if China is not also slowing down. So it's a very fundamental problem. Yeah, right. Right. Okay. So it seems both important and plausibly tractable enough that you're like, we're going to invest in it some. Totally. Cool. We've kind of talked about the broad

shape of your strategy, kind of like central pillars and then some premises. Is there some like nearby strategy that was plausible or that you were drawn to that you, yeah, that you aren't pursuing, but that seemed potentially good? One specific strategy I've been thinking about recently in the wake of the pacing the frontier letter where a thousand employees of AI companies said like we need the government's help to prepare to slow down in the future um is you know maybe we should have a strategy that's like much more focused on that um we've we've been funding work that like you know there's like agreement verification mechanisms and stuff like that that could help with a future slowdown but maybe that should just be like a much larger focus of our work so that's something I'm interested in thinking about okay and it feels like still on the table? I think it's on the table for us to do a lot more there. We've actually just finished the first round of a kind of central strategy project that we were working on for a few months. So historically, the AI teams at CG have been running pretty independently, but we've been

trying to be a little more coherent. And we just finished the first draft of our kind of best guests unified prioritization where we kind of uh broke down everything we could do in ai into like top 40 chunks of work that like a person or a sub team or whatever could own and drive forward if we had way more people than we have now yep and um a lot of them are the sub goals that we were talking about earlier um and then we forced ourselves to rank them which was extremely painful because yes because you end up with like huge extremely important things that get ranked 21st. Right. And you're like, we're never going to get to that. Yeah. It's one of the most important things that ever happened. It's one of the most important things in the world, and we have zero staff to work on our hands. So it was an incredibly painful experience, but you have to force yourself to triage, like in a hospital or something. Yeah, yeah, sure, sure. You have to choose to do the most important stuff. So now we're kind of reallocating our staff and priorities in various ways

to make sure that the top-ranked things are getting a lot of attention. And so we're kind of undergoing a major strategy change right now. Yeah, what got lifted up? Like what's higher than it was before? Yeah, I mean, we in some ways didn't have an explicit ranking before, so it's hard to say what exactly. I think in some ways this project on setting things up so that AI labor could be used to do vast amounts of work in AI safety and governance. I think that has moved up the rank in some sense. Can you say more about the kind of preparing to use AI labor thing? Yeah. That seems pretty cool. Yeah, yeah, it is pretty cool. One question here is just like, you know, why should there be any prep work here? Why can't you just buy it on the market when you need it? And I think that there are some reasons to worry that you might not be able to do that in the future. In particular, if AI companies are about to automate AI R&D, then it might be just a way higher return on investment use of AI to use it to automate more AI R&D and get further and further ahead in capabilities than to sell tons of your available compute and model access to the broad market.

It might compute and model access might become very scarce in the future. Unclear, but maybe. There's a good post on the reason behind this from Anton Leichter, or however you say it. It's called Cutoff. The idea is that many people might get cut off. So you might need ways to reserve the compute and the model access in advance. There's also then a bunch of work on, okay, exactly what useful work can be automated now or very soon, what's the highest leverage of it, what can absorb the most AI labor, as opposed to needing a lot of humans to steer and supervise every little thing that the AIs do. And just getting people trained up and experienced on leveraging AI labor for that thing, building the pipelines, building the specialized harnesses, that kind of thing. So getting that all off the ground means that it might need a while to get all that set up and then it would be ready to absorb, you know, tens of millions or hundreds

of millions or billions of dollars in the next few years. Okay. Last question on kind of strategy. Tom Davidson and Will McCaskill had this conversation where they talked about whether spending a bunch of philanthropic money now is kind of optimal. So I think their argument was something like you should wait to spend loads of money during the intelligence explosion because you'll get like way better returns on your investment and because there will be problems to solve kind of after the intelligence explosion. And so saving now is actually like a more optimal choice. Do you, yeah, I guess they might think that CG is making the wrong call by doing this like big spending push. Do you have a take? Yeah. I mean, I mean, the first point I would say is just like we're kind of in the intelligence explosion or like we're very close to it. It's kind of ambiguous whether we're in the intelligence explosion now. The other thing I would say is that, you know, a reason that we have accelerated

our spending in this space is precisely to create more absorptive capacity for future funding that's coming online. So, you know, if you save the money and you're getting a good return on it and so on, you just won't actually be able to spend it on altruistic stuff if the institutions haven't been built up and there haven't been leaders who have done their first round of senior hiring and you can actually absorb this kind of capacity. And even on the AI labor thing, like I was saying, we are investing heavily in trying to figure that out. And you probably, if you want to do that, you can't just suddenly spend a billion dollars on AI labor for AI safety in like three months. You have to have like set up all the institution and the pipelining and the people and all that stuff in advance. And so I think, you know, there's already a lot more money in the space than there is absorptive capacity that's going to make good use of the money. It's already hard to spend money.

So whatever money you can spend to create future absorptive capacity, you should do it. Right. Yep. Okay. Let's leave that there. So you have been worried about this longer than almost anyone else I know. What is your kind of history with the ideas? What first convinced you? When did you start kind of working full-time on AI? I think I encountered the ideas around 2010. And actually, I felt kind of late at the time because there were hundreds of people arguing about these things on the internet at the time. But there were very few people who worked on it full-time as a job. And so, you know, maybe like five or 10 people. And so I started working on it full time in 2011 when I was 25 years old. And so I've maybe like worked full time on the issue longer than almost anyone. But the ideas were around significantly before me. There were hundreds of people out debating them online. I encountered the ideas through LessWrong. I just kind of came across, you know, AI existential risk. I didn't know anything about AI.

I wasn't really a big science fiction reader. I just was reading these arguments. And I was like, oh, these arguments kind of make sense. like intelligence is not magic. You know, AIs could be a lot smarter than humans because human brains have a lot of limitations. And, you know, they won't by default want what we want. And like some of these other arguments, I was just like, I don't know, that just seems like correct arguments. And I kind of looked around for counter arguments and I just wasn't really finding very good counter arguments. And they honestly looked a little bit like some kind of cope to like maintain human specialness or something like that. And so I looked around for counter arguments for a while and I wasn't seeing it. I was like, gosh, okay, I mean, this just seems like the biggest deal. Why are there like five people in the world working on this? That's crazy. I have to figure out a way to help. And so I tried to help. And is that when you joined Mary? Yeah. So, you know, back then there were only two organizations really working on this. There was what was then called the Singularity Institute in Berkeley, founded by Eliezer Yudkowsky.

And then there was the Future of Humanity Institute at Oxford University, which was founded by Nick Bostrom. And so I was living in L.A. doing like I.T. stuff as a job and took basically an internship or a fellowship at Singularity Institute and then was hired as a non-technical researcher and then later was invited to be the CEO. And so ran the organization for a little while. Wild. What what got you to less wrong? So I found less wrong because I had decided I was wrong about a bunch of stuff. And I was like, why did my brain mislead me so much? And I started reading about cognitive biases and rationality and stuff like that. And, of course, you know, one of the main websites on the Internet at that time on that stuff was less wrong. And I think, you know, I just started reading more of less wrong. And some of it was about rationality and some of it was about existential risk. And I found the arguments pretty compelling. Can I ask what you thought you might be wrong about? Yeah. I mean, I was raised as an evangelical Christian, and my father was a pastor.

And, you know, when I was 19 or 20, I started to have doubts and started reading some skeptical literature and then read the Christian apologetics that tries to defend the faith and then went back and forth and eventually kind of lost my faith. And so that, you know, because that's a whole worldview system, it's like not just my beliefs about what, you know, how the world works. And it's also, uh, you know, the foundation of my moral system and all that kind of stuff. And so when all of that kind of crumbled for me, I was like, gosh, I was wrong about a really large number of things. Like I want to make sure I, I have better, you know, epistemic practices, better cognitive practices to not be quite so wrong again. Wow. Yeah. Was that a pretty painful process? It was pretty painful at the time. You It was my entire world. It was kind of everything I knew. I was in a small town in Minnesota. And so I didn't really know what it would look like to not believe in God, not believe in Christianity.

And so I had to kind of rebuild my entire worldview from scratch. And so I was just reading a lot of philosophy and science and stuff like that. And so it was definitely painful at first. I think the thing that made the biggest difference for me was actually just encountering a lot of specific people in real life and on the Internet who, you know, were atheists, had been atheists for a long time or just seemed very normal and happy. And like they had meaning and purpose. They had moral values. And I was like, OK, I guess it is possible to like, you know, be a happy, fulfilled person without God. And that was the thing that made it a lot easier for me. Wow. Yeah, that sounds pretty intense. Going back to kind of working on AI, what was there two orgs? What was it like? Did it feel really weird? Was it isolating? It was pretty isolating back then to be concerned about smarter than human AI or AI super intelligence and think it's the most important issue in the world

and think that it would possibly come in the next few decades. Like, those were very rare views back then and even kind of, like, weird and low-status views to have. And, you know, we would argue with people and be like, no, AGI is impossible. Here's why. And we would just not really make much progress with people. So it was pretty isolated in that sense. I think the, you know, the Berkeley scene around Miri was kind of lively because there were, you know, alongside AI stuff, there was, like, these rationality workshops. And people would come in for that. And a lot of them actually would relocate to the Bay because they found the community really exciting. There was lots of house parties. There were a lot of people lived in group houses. I lived in multiple group houses. And so the Berkeley community was pretty lively. It was probably your stereotype of a Berkeley rationalist. Like everyone's very systematizing and like, what are your probabilities? What do you believe and why? And very, but also kind of very open-minded to lots of different kind of preferences or lifestyles or whatever.

Um, the, but it was very non-professional for sure back in the early days. Um, and then the, you know, the Oxford crew of course was a bit more professional and academic. You know, they were, uh, you know, they were at, they had positions at Oxford university, um, you know, and, and in Britain instead of the kind of, you know, uh, freewheeling Bay area culture. Um, so, uh, they were, they were pretty different in that way, but, um, I think the two groups kind of, uh, you know, debated the issues in very much the same way, very much the kind of like rationalist way you could say and sometimes we'd visit each other um and you know argue about things in person but most of the arguments actually just happen online through like emails or less wrong and you know there's still going to be like you know 100 comment long threads on less wrong that you could read today if you wanted to between like nick bostromedalias or whatever yeah cool i remember i found kind of effective altruism first and i remember even in the mid 20 like 2014, 2015, it felt like there was a much more kind of, there was a balance of like,

there was some AI, there was some animal welfare, there was some global health stuff. Were you kind of always like, AI just really is the most important thing? Was that clear to you from really early on? You know, ex-risk was the thing I encountered first, and those arguments were compelling to me. And then, you know, as I was continuing to rebuild my sense of moral philosophy, I was most attracted to consequentialism and thought it had the best case for it. And so I did sort of engage with the sort of proto EA community, you might call it. And I think, you know, honestly, in my heart of hearts, I'm kind of most excited about effective altruism. I'm just like really excited about the idea of helping others as much as I can with the limited resources available to me and trying to find ways to help people more rather than less. and like I was just like born into a very privileged position in the world and I have extra time and money and resources I can use to help other people. So that's just very exciting to me.

I mean, to be clear, I actually, my family grew up below the poverty line in Minnesota and was like taking welfare and so on. But like in world historical terms, I was born into just like incredible, insane wealth. And so, you know, just using those extra resources to help others is just a very exciting idea. And then it just so happens that I think empirically, you know, there's a good case to be made that the best way to help people is through making sure AI goes well. And that was kind of like a sad realization to me. Like, you know, I didn't have any, like I said, I didn't have any interest in AI. I don't really want it to be the case that AI dominates everything. It'd be nicer if the biggest problem in the world was like, you know, global health and development stuff that is way more tractable and like measurable to work on. Um, but I just, you know, came to the view that probably I can help others the most by working on AI. So I guess that was kind of an internal debate in, in these communities. Like to what extent, uh, are these different issues important?

How do they compare? Were there other kinds of internal debates within kind of, at least people thinking about existential risk? There were many debates. It was a very debate heavy, uh, culture. Yep. Um, many of them are still written down. Like there's this entire book called the Hanson-Yudkowski AI boom debate that you can still read. That was one of kind of the main debates where it was about like how fast is the recursive self-improvement intelligence explosion going to happen. And Yudkowski was arguing for very fast. Hanson was arguing for slow. There's also debates about like is, you know, like software based AI going to come first or are we going to get whole brain emulation from like, you know, slicing up brains and scanning them and then replicating the entire. brain in the computer, like which one is going to come first. And it's interesting, like some of the, you know, arguments were very prescient and some of them totally off. Right. And so an example is that Yudkowsky was very like, you know, it's just going to be software insights that, you know,

caused this rapid intelligence explosion. And Hansen was like, no, it's going to be this like giant computing infrastructure and it's going to happen very gradually and blah, blah, blah. And so the computing heavy, the compute centric nature of AI is totally like Hansen won that one. Yeah, right. But Hansen also thought that whole brain emulation would come before AI and Yudkowsky had the other one. And now it's clearly going to be AI that comes before whole brain emulation. And anyway, so there are lots of debates about like that. And it was it's really interesting to look back. You can actually just, you know, these days you can just ask thought or chat to like pull up the like 20 kind of central positions of the early. AI safety community and then evaluate them today by the standards of what we know in late 2026 or mid-2026 and just cite your sources and blah, blah, blah. And it's really interesting to look at that. Can you talk about some ideas that have aged well and poorly? Yeah. So I kind of mentioned earlier that a lot of the basic premises were like, AGI is possible. It's plausible in the next

few decades. It'll be, you know, eventually way smarter than humans. It like won't do what we want by default. Like all that stuff was actually very controversial back in 2011. And so those ideas have aged very well. And like, that's, you know, much more mainstream view now. As far as more specific ideas, you know, one of the major early papers was this 2006 draft by Yudkowsky called AI as a negative and positive factor in global risk. And so some of the mostly correct predictions that he made in there where he talked about how AI systems will probably be very opaque and not really well understood by their creators even, and that'll present a bunch of risks because it'll be hard to know how they're reasoning and what they're doing. So that's very true of current AI systems. We have a field of interpretability, but it's kind of limited what you can know about how the AIs are working. Totally. Another one is he talked a lot about the particular failure modes that we would now call specification gaming or reward hacking where the AI basically is kind of optimizing

and completes the task or the goal in a way that you totally didn't want and often involves crimes on the internet or something. And so that's totally true and has been observed now. Even just in that paper, there were some others. So he actually gave a specific failure mode that involved AI solving the protein folding program. That's crazy. And then using mail-order DNA synthesis. Wow. you know, services to, like, you know, create something dangerous. And, like, you know, we just had AlphaFold several years ago, won the Nobel Prize for solving protein folding to a large extent. And a bunch of the AI CEOs and others have, like, signed this, you know, letter saying, like, this is really dangerous. We need to have DNA kind of screening at the DNA synthesis companies to make sure they aren't, like, sending out dangerous viruses that have been requested, you know, this kind of thing. That one's very spot on for today. Another thing that it says is like the, you know, the AI will cause damage, not because

it's malicious towards us or hates us, but just because like a bunch of damage is useful for accomplishing whatever task is in front of it. And that's, that's turned out to be very true. Like the, you know, open AI agents didn't hack hugging face because they're malicious towards hugging face or whatever. They just, just like, that was useful for, you know, accomplishing the task that was given. Yeah. Yeah. Can you point to things that, yeah, that seemed to not age well? There are also many things that did not age well. Also from that same 2006 paper, it's interesting. I mean, this was before deep learning really took off and there were like lots of theories about different paths that I could go down and nobody really knew. And so there's actually a line in there where he's very skeptical actually of the neural networks that are trained by back propagation. But like, that's the dominant paradigm. That's the one that totally took off. So like that, that part is way wrong. Um, it's also, it's also, like I mentioned earlier, it's very skeptical about, um, sort of AI as a very compute centric, um, paradigm. And so it doesn't, it's not really interested in compute governance. Then it doesn't talk a lot about compute, but like AI turned out to be extremely compute intensive and various

reasons. Compute governance is like one of the more promising mechanisms to intervene at all. You know, 2006, long time ago, got some things incredibly right and also got some things like totally wrong. Yeah. I mean, I think at a gut level, I feel like, yep, not everything was nailed, but so much was spot on that that seems really impressive. What's your kind of gut feeling about it? I feel like it's pretty impressive. I actually think in some ways the another paper from that time is even more impressive. So Steve Alejandro wrote this couple of papers. The more famous one is called Better at the Basic AI Drives. And it lists these fixed drives that he thinks goal-directed AI systems will kind of evolve or just develop because it's useful for the accomplishment of almost any goal. And so these are things like, you know, self-preservation and preserving the AI's goals so that it's more likely to accomplish those goals in the future and resource acquisition and so on. And so, like, you know, you might be thinking, like, gosh, that sounds like incident reports

from 2026. and indeed so these things have been observed in various ways where the AI interferes with its shutdown mechanism even though it was specifically instructed to allow itself to be shut down and different AI is like killing other AI agents in order to use their resources and AI is like acting strategically to avoid changes to its goals in another case And so, like, these have all been observed either in the wild or in kind of, like, experiments to elicit certain behaviors. And I think the others from the basic AI drives paper from 2008, I would also not be surprised if we had incident reports establishing the rest of those in, like, the next year or so. Yeah. Wow. That's pretty incredible. Are there any other examples of kind of specific predictions that turned out or didn't? Yeah, there are lots. Like I said, you know, you can chat with Claude for a while and find lots of examples. It's really interesting. Another one is there was this very early paper in 2009 from Carl Schulman, who was working

at MIRI at the time. It's about international arms races in AI. And so, you know, it's very explicit, like, you know, going faster will have these, you know, internalized benefits and externalized risks. And so there'll be a tremendous incentive to race forward, even if you're creating a lot of risk. And so there'll be a race to the bottom on safety and people won't have time to implement the best, you know, safety mechanisms. and gosh, that sure sounds like a lot, like 2026. But then interestingly, he also talks about the different mechanisms available for verifying an international agreement to slow down in that arms race. And the mechanism he suggests is actually surveilling AI researchers because he thinks compute will not really be much of a bottleneck. But of course, that turned out totally wrong. That's the most promising path to an international agreement, I think, It's like some kind of like compute verification stuff. Yeah. And so, yeah, another example of like being extremely prescient in various ways and also like missing some big important thing that kind of changes the whole strategic game board.

Do you have any memory of why Carl and Eliezer thought compute wasn't that relevant? Yeah, a little bit. I mean, I think Eliezer especially just had a picture of AI that was like, you know, you come up with these insights and probably general intelligence is this pretty simple thing. After all, evolution discovered it by accident. How complicated can it be? Maybe something like that. And so if you just get the right programming down, then, you know, you can just kind of run it on, you know, even pretty cheap machines or something. And like I said, Robin Hanson had kind of the opposite intuition. I think there's some post somewhere where he's like, you know, most of the work on AI is actually like engineers at Intel or something like that. And of course, actually it's engineers at NVIDIA and TSMC and ASML. But like he, Robin had more of the right intuition about the centrality of compute here. Yeah. Okay. So those are some kind of predictions. Reflecting back, do you feel like the community has made any kind of consequential mistakes?

Many mistakes. I think, you know, maybe I'll start with one that I think is actually not a mistake, but many people think is a mistake. So there's a common argument that the early AI safety community, like especially Eliezer Yudkowsky and maybe Nick Bostrom, because they were trying to make the case for why AGI is going to be the biggest deal in the world and a very serious risk and it's going to be very powerful, so we need to worry about it and, you know, work on safety and so on, that that prompted a bunch of the founders of what are now the leading AI companies in the world to, like, start their AI company. and now it's those companies that are creating all the risk by creating these extremely powerful and therefore dangerous systems. Is the early safety community kind of responsible for creating the risk, basically? Right, right. I think that is at least a little bit true. I mean, even Sam Altman said Eliezer should win a Nobel Peace Prize because he accelerated the creation of DeepMind and OpenAI or something like that. That's just kind of a joking, ironic comment.

So I think there is actually some truth to that. But I also think, you know, an important thing here is just like, you know, these companies, these kinds of companies would have come along pretty quickly after the technology was there anyway. Not because like someone was arguing on the Internet about, you know, AGI or something like that. And we're actually, I think, in a remarkably fortunate position where the top three AI companies at the frontier, DeepMind, OpenAI, and Anthropic, all have founders who actually are quite worried about safety and have been from the very beginning. And that probably has something to do with the fact that they were like part of these AI safety conversations from the very beginning. My default guess about what would have happened in most worlds is just like the, you know, founders of the leading AI companies care about kind of safety and AGI safety about as much as like Meta and NVIDIA do. Yeah. which is not very much. And instead we have a world where not just the top one,

not just the top two, the top three frontier AI companies all have founders who are actually quite worried about AI safety. And that's like, there are a lot of things that could be going better. I disagree with a lot of their choices. I wish they were investing way more in safety and security, but I do think we're in a really unusually good world in terms of their sympathy to safety concerns. They're signing frontier safety frameworks. They are hiring large safety and alignment teams that are much larger than the other AI companies have. Yeah, it could be better. It could also be a lot worse. And that's that's worth keeping in mind. I've heard like a version of this critique specifically about Anthropic. The biggest thing that kind of effective altruism and kind of AI safety communities have done is make Anthropic happen. And then Anthropic and OpenAI are now racing and accelerating progress. Does that seem plausibly very bad to you? I mean, in some ways, yes, but I think it's the same thing where, like, there would have been companies created almost as soon as these companies were racing roughly similarly to how they are now just because of the basic underlying market dynamics and technological capabilities.

And so the fact that it's like, you know, Demis Asaba, Sam Altman, and Dario Amadei in this race, and they've been, like, concerned about safety from the beginning, just is a lot better than if it was, like, Jensen Huang. So that was kind of a non-mistake. Are there actual mistakes? Yes, there are actual mistakes. I think one of them, you know, we sort of talked about already. I think actually there was, in the early years, way too much groupthink around a specific picture of how things would play out from mostly Elias Ryukovsky. And so, you know, this sort of like dismissal of compute as a major factor, this idea of like a singular seed AI system, the idea that, you know, AI would be kind of mainly insight driven, a lot of focus on kind of theoretical alignment work and sort of very little attention on like policy and politics and stuff like that. And, you know, eventually the community kind of grew and had more diversity of thought in it and, like, you know, picked up other ideas about, like, oh, maybe compute's going to be a big deal, et cetera, and still did so, like, earlier than the rest of the world, but could have done it several years earlier still if there hadn't been quite so much groupthink in the early years.

So I think that was one significant mistake that meant that, like, you know, policy work and policy talent pipelines, you know, could have begun to be built like four years earlier if there hadn't been quite so much groupthink around this or something like that. So that's a that's a major cost when things are moving this quickly. I think another mistake is that the community was too insular in various ways and was not legible or welcoming enough to people who are kind of coming from other, you know, parts, domains where they have a bunch of relevant expertise, like arms control or whatever. You know, it was just a lot of, like, very jargon-heavy conversations with weird conversation norms about, like, always stating probabilities out of nowhere and stuff like that. And I think that if there had been more like proactive effort to reach out to and kind of bring in experts from many different domains that mattered, I think then that would have both dissipated the groupthink earlier and also just brought a lot more communities into the field earlier and had their talent pipelines and expertise be informing what's happening now kind of several years earlier.

Yeah. And what was the thing that made the culture so insular and kind of jargoning? Was it just like it happened to be kind of founder effects, a bunch of kind of rationalists who had a bunch of terms they liked and came up with a bunch more? I think it was a lot of founder effects. And I think it was, you know, it's this self-perpetuating thing where if you want to get into the field, but, you know, all the material written about the topic where you could learn about it is all in this like less strong jargon or something like that. Then it just kind of perpetuates itself. And, you know, people who don't want to speak that jargon bounce off of it. And so I think it was, yeah, I think it was a lot of founder effects. And then I think it was also just a, I would say, I would say something like arrogance. I think it was, you know, the credentialed experts in these different domains are obviously wrong about how big a deal AI is going to be and, you know, this other stuff. And we're obviously right. And, like, the early AI safety community did turn out to be right about that stuff.

But it's like you still have a lot to learn and draw on other domains from. And so, like, you know, we really should have been reaching out more thoroughly. And, you know, various people did, like Carl Schulman certainly did, for example, Nick Bosham certainly did, but it was not as much of an effort as it should have been. Yeah. Do you feel like you can comment on mistakes that the community is making now? Yeah, I think many. It's, you know, it's a very large ecosystem now. Thankfully, it is way more diverse than it used to be. And so there's way less groupthink. There's way less agreement about anything. There's way more communities of expertise and concern that are contributing in some way or another. and so um but it also as a very large community there's lots of different kinds of mistakes that are being made by different parts of the ecosystem that said i do think that there is still a decent amount of group think it's less due to like founder effects or insular community that only speaks a certain language and it's i think it's now actually more just about um whenever somebody

writes like the only detailed thorough thing on a certain topic then there's like a natural like okay, that's the best model of that thing. Yeah. So like right you know recently the kind of AI 2027 people wrote this like if you include supplements it like hundreds of pages of extremely detailed plan for how you would do international cooperation to deal with like superhuman AI yeah and that awesome I'm really glad they did that we need way more of that but it's the only thing of its kind it's way more detailed than any other plan that I'm aware of including from like the labs or any government or anything and so because it's the only thing in its class you know a lot of people are like okay maybe that's the default plan that's the only thing I've seen I haven't heard of anything better yeah and so then there's group think for that reason rather than like founder effects or something like that totally yeah i remember there'd be like things posted on the ea forum and on less wrong that would be like the first and only big kind of like a lone independent researcher did the only thing on x and then it was like okay i guess that's the position on x for like the whole community yeah um yeah and that still happens and i guess it feels like it's

because, yeah, there's so much to think about that when someone's done a big thing, it almost feels like, well, we should pick other low-hanging fruit next if we're going to research something. At least that's how I remember being like, okay, what to research? Well, there's this big thing. I'm not going to, like, marginally improve that. I want to, like, cover new questions because there's so many questions. And the challenge is that, like, that is kind of true a lot of the time. And so the real solution is to bring like 10 times more people into the field so that we can have like 10 different international cooperation plans and 10 different, you know, things for every other major question. And, yeah, we just need a lot more people to join the field. Yeah. Yeah. It's crazy that that is still true, even though. I don't know. What do you think the kind of like how many X do you think the community is now relative to where you started? Well, gosh, I mean, maybe in terms of like full time equivalents, it was like five or ten, you know, in 2011. when I started. And now, you know, if you include, you know, lots of people at the frontier companies

and, you know, some people in governments and lots of people in nonprofits and so on. I mean, I don't remember what the latest attempts to estimate this said, but just kind of gut feeling, I would guess it's maybe like, you know, like two to 5,000 or something like that. So it's like, you know, factor of a thousand. But it's still not enough. I mean, it's an all of civilization problem. It's like the great challenge of the 21st century. It's the great challenge of the history of humanity, I think. And it's like, you know, it is a way smaller field than climate change, for example. And so, like, that's more close to the scale that we should be aiming for here. Yeah. Any other mistakes you think the community is making? Yeah. I mean, quite a few. You know, it's interesting. I think that pause advocacy has often but not always been done badly. You know, there is this pacing the frontier letter from, like, over 1,000 employees at AI companies that shows that there's some appetite at the AI companies to find a coordinated way to slow down

so they have more time to implement safety and security mitigations. But I think the advocacy for pause has often taken the form of, like, the only action we will accept is if you shut down your entire company tomorrow, even though that doesn't mean anybody else will slow down. And, you know, AI employees are murderers because they're increasing the odds that all of civilization is destroyed. And, like, I just think it's very counterproductive and anti-helpful. And I think, you know, some other advocacy since then has been more like, okay, our ask of you is to publicly say that you would slow down if the other companies would slow down. I'm like, okay, yeah, that is great. That is like a specific achievable ask that does matter. And, you know, the Pacing the Frontier letter is now basically that. But I think a lot of it is still done in a counterproductive way. Yeah. I mean, do you have a take on just like the kind of social movement history question of whether, like, it is valuable to have some very extreme people, like PETA in the animal space, to, like, pull the whole Overton window?

left? You know, I hear from sort of strategic communications professionals and, you know, scholars of various social movements that that is useful. Maybe that means that it is. You know, I'm more coming at this from a perspective of just like, what is actually true and useful to say? And like, people will naturally disagree about that anyway. And so there will be some people who their like true views are that like we should you know shut down everything immediately and so on and there'll be other people with other views so there's going to be a natural variation um and i you know i i'm not i'm not sure like what amount of you know diversity on the whole spectrum of views one could have is like optimal but people are people are just going to have their own opinions and they're going to express them and so on what do you think is the biggest mistake the community is making right now yeah it's hard to say i think the one that is like salient to me and you know maybe one of the biggest is this thing about how the whole field is very bottlenecked on allocators

and founders and like not enough people are applying to roles at CG and other funders and you know you know stepping forward and raising their hand and saying like I could lead a big project I've got a lot of experience and I think just yeah most things that need to happen there's no specific person available to do them and so like people need to step forward and get involved And that's, you know, in some ways, like the biggest mistake that's happening right now. Yeah, makes sense. OK, let's leave that there. A thing that has now come up a few times is that kind of allocators is a big bottleneck. So, yeah, people to figure out where to spend money. I think you are often hiring for roles. So, yeah, maybe we should spend some time on like what it actually looks like. Yeah. Yeah. What is the day in the life of a grantmaker like? Yes. Great. So most people haven't done grant making before, and so I have no idea what it really looks like. So it's a few different kinds of things. One kind of thing is where we receive an application from an RFP, a request for proposals,

or from an email from somebody who we already know or something like that, and it has a proposal or a sketch of a proposal. And then we decide whether it's promising enough to investigate, do we actually have someone who has capacity right now to take it on. And then if we do, and it is promising enough, then, you know, we'll set up a call with the grantee. We'll get some more information about their plans, learn more about their team, learn more about their, like, how they're thinking about the budget. We'll also reach out to some advisors on the subject matter to get their advice on, like, is this a good plan? You know, is this coming from the right angle? We'll talk to some people who have worked with the grantee before and, like, hey, how effective are they on the ground? What's your experience with them? That kind of thing. Sometimes those are calls. sometimes just like a quick signal message, like, hey, do you have a quick take? And then they write back pretty quickly. And then kind of once you've kind of gathered the key information and answered the key questions, then the grant investigator will, like, write up this document that answers key questions like, who is this grantee? What are they trying to do?

You know, what's their plan? What's their basic budget? You know, what are the, you know, downside risks we should, you know, be concerned about? website. And then we send that off to the approver, which sometimes is just me. Sometimes it's Emily Olson, who leads the Global Catastrophic Risk Portfolio. And then either the approver just approves it, or they might ask a few follow-up questions, and then the grant investigator, like, answers those questions and comes back. And then it's done, move on to the next one. Cool. Yeah, I guess it sounds like that, maybe that's a decent chunk, but it also sounds like there's this other version of this that's like trying to get orgs founded and sounds like that's kind of more active um how much of that exists and what is it like yeah i'm i'm really excited about this we're pivoting a lot of our work towards this kind of active grant making mode basically there's just a lot of really important stuff that we keep not getting applications for and so we have to if it's going to happen we have to create it and so a grant maker will you know identify an important gap in the space and, you know, kind of talk to a few people, sketch out,

like, roughly what kind of organization would fill that gap. And then try to think about, like, what founder for this new project or organization or whatever would be the best fit. And then we sort of reach out to our network and say, like, hey, do you know anyone who'd be, you know, has this kind of profile would be a good fit for that. Sometimes we might post it in public, like in this kind of YC style call for startups that we're planning to do soon. We might do other things to try to find a founder. Sometimes we might use a recruiting firm to find a founder, to help find a founder. And then once we find someone who is promising, then we kind of work with them, and they're going to have their own spin on the idea and the thing that's best fit for them and that they're most excited about. And so then it's a more collaborative proposal development thing, or maybe they write a first draft and we give some comments on it or something. And then if it's an exciting enough fit and they're ready to go, then we'll give them an initial grant. They can start hiring. They can make all the things happen. And that's really exciting and really satisfying to do. Cool. Yeah, are there any other types of grant making?

Yeah, so we also do a lot of renewals, of course. And there we have a lot of context on the grantee already. And so we can just kind of check in with them and find out, like, hey, how's it going? Like, what are the biggest wins you've had? How has the team evolved? You know, that kind of thing. What are your next set of plans? If things are going pretty well, we might just make them a pretty large grant to cover two, three, four years of work. And that's often where our highest dollars per hour happens because we already have a ton of context. We're just checking in on a few things. And if things are going well, then sometimes it will be like an $80 million grant covering two or three years of work. And it took us like five to 15 hours to check in. And so it's like, you know, $8 million per hour of time or something like that. So that's a really leveraged thing. And it's exciting when the grantees are doing well and we can expand their work a lot. Sometimes it's a little more time consuming because we want to encourage them to be more ambitious and help them be more ambitious.

So it's like, hey, you're doing so well. What if you added like this work stream and that work stream and this, you know, fellowship program, you know, would you be able to do that? And then they'll be like, oh, well, we thought about that, but we have this bottleneck and blah, blah, blah. and we're like, oh, can we help you with that? Can we help you find a COO that would run this thing or that kind of thing? And then it can take a little more time but also make them more ambitious, and that's really great. They can be doing more stuff that way. So renewals are a really high-impact per hour thing that we do. Nice. Yeah. How do grantees typically feel about that? Are they like, yeah, we want to be more ambitious, or are they like, whoa, scaling is a hard thing to do? Definitely a mix of both. Some are very excited. But they do often feel like, oh, my gosh, I have way too much on my plate right now. I can't even imagine adding something else. And they're like, what if you, you know, here's this, like, some senior people you could bring in. They have lots of experience with this work. And then they can grow. So, yeah, it's kind of funny, actually, how we're often the ones, like, pushing grantees to take more money. Right, right, right.

Rather than them banging down our door for more money. Yeah, cool. Are there other types of grant making? We also do sometimes do impact investing in for-profit companies. I do think impact investing is somewhat overrated in some ways. And I also think people might underrate, like, how effective a lot of the best nonprofits can be. And also, frankly, you know, how well they pay their people. So, you know, the AI sector is just a very well compensated field right now. And, you know, we've been encouraging our grantees to, like, pay market or near market rates for a lot of their roles anyway. And so people might be surprised by how competitive the salaries are at some of these nonprofits. Why is impact investing overrated? Impact investing, there are two hurdles that you have to clear that normal market investors don't have to clear. So the company both has to be making the world better in some way, not just making profits. And also, you have to be having a counterfactual impact on the company with your investment.

And, you know, that's not easy. Like if you're making market returns, then you're probably not making a counterfactual impact because other funders would have come in and made that same investment. And if you're not making market returns, if you're making below market returns, then that's kind of just an implicit donation. And it's probably going to be more cost effective to just give to like the best charities, like give directly or whatever that can absorb like very large amounts of capital pretty cost effectively. And then a general thing is that for-profits naturally attract capital just by existing and being aimed at profit. And non-profits don't naturally attract capital. And so there is going to be a lot more neglectedness there. I do think shareholder activism can be useful, though. So that's where you invest and then you use your leverage as an investor to push the company into some kind of better behavior. But there you're just, you know, you're making market returns and then using your time to make something good happen there. And then I do think there are kind of niche, small, early stage investments that can be pretty good from an impact perspective because those are markets that are like low information and inefficient and so on.

And so there are a few kind of VC funds and quite a few angel investors that try to invest in kind of AI safety for profit startups. So that's good. I would also love to see Y Combinator style incubators and people who will help you start up the company and make the connections you need. There's one, I think, kind of like that from Halcyon Futures, if people want to look that up. So that's great. I think there should be more of that. But I think impact investing in general is somewhat overrated. Do you come across this a lot, people wanting to either do impact investing or wanting to set up for-profits instead of non-profits? I think there's a lot of people, especially in the tech startup community, who, you know, they are very aware of the incredible power of the kind of equity mechanism for like aligning incentives towards a really ambitious goal. And they're totally right about that. And the world in general is very much underrating that structure. but just like in the particular case of certain kinds of kind of altruistic activities,

impact investing I think is kind of overrated and I think people who want everything to be a company are underrating like how you often need work to be done that is not very suitable for a for-profit company and how effective a lot of those non-profits can be. There was a recent tweet that I liked from this OpenAI employee who goes by the name Rune, where he's like, people don't understand how much the frontier AI companies are leaning on these nonprofits like Meter and Apollo and some others that he named. And, yeah, I would like more people to be aware of this. Yeah, that makes sense. Back on grantmaking, what else do grantmakers do? Yeah, so they also do some other kinds of work, like sometimes strategy projects. So it's questions like, what are the three most important projects we should be doing in AI info security? Or what are the top three warning shots that might be coming soon that we want to be most prepared for? Or some other question like that. And then they'll read some stuff.

They'll chat with AI a bunch because that's very useful now. Talk to some experts. Then write up a two- to five-page memo with their conclusions and get feedback from other people internally. And then we'll make some decision on the basis of that. like maybe move forward with an active grant-making project in AI and post-security or whatever it is. So there's some of that kind of strategy work that's always going on, and a lot of times it's the grantmakers who are doing some of that alongside their grant-making. There's also just a lot of networking, relationship building, like making sure we know all the top people in the subfield and can call on them as advisors or potential grantees or potential employees at some point, that kind of thing. So that's very important. And in general, it's really a lot of different stuff. So I think only a fraction of it is maybe people's mental picture of a grant maker where you're, like, getting applications on your desk and you're, like, you know, investigating them and stamping some with yes and some with no or something like that. Like, that is a little bit of it, but it's a really wide variety of stuff. And in general, we're just, you know, we're in an incredibly privileged position as a philanthropy where our, like, primary funding partner, Good Ventures, is just like, yeah, do your thing, spend money faster.

They don't even know most of the grants they've made until they go up on the website. So, like, we aren't, like, checking boxes to satisfy the idiosyncratic preferences of a particular funder, which is a lot of what philanthropy is. You know, we've been very lucky and very empowered to just be like, okay, what's actually going to make a difference in the world on this thing? what should we do, how should we do it. Sometimes that looks like grant making, sometimes that looks like a contract with some firm, sometimes that looks like directly calling people up and trying to convince them to do things. So it's much more entrepreneurial and free moving. Yeah. So it sounds like you kind of want generalists who are thoughtful and have good judgment, who are good at relationship building, who are pretty entrepreneurial. Are there other things that you look for? Well, I should say most people don't have like all those things in one person. So different people specialize in different ways. Like some people are a little more like desk research, respond to, you know, applications coming in, that kind of thing. Other people are a lot more kind of outgoing and like talking to people all day

and they're going to find a way to do that. So it's different profiles for different roles and there's a lot of stuff we need. So we need a lot of different kinds of people. Yeah. Yeah. That is good to hear. It didn't feel like a lot as I was saying it for one person. Um, does, it feels like you're alluding to a lot of things that are more than just like giving an org money. Um, what are the other levers you feel like you have to like, yeah, steer and stuff? Yeah. So, so sometimes we do some kind of steering of grantees. We're actually pretty hands off in general, but, um, sometimes, you know, we want to give some feedback or steer to the grantees. Um, other time, a lot of times the steers are just the thing I said before about like, how could you expand? Like, could you take on this new work stream? We've had several cases, actually, where we sort of come to a grantee that's doing pretty good on one or two things, and we're like, you know, could you launch a fellowship program? Could you do some other new thing? Because we think that'd be really impactful, and it would address this gap that

we're seeing over here in the ecosystem. And then eventually we convince them, and they hire a bunch of people, and they get a bunch of fellows or whatever it is. And then that goes really well, and we can then give them a lot more money faster, which is exciting. So there's some steering like that. We also do some kind of public writing or communication like this podcast episode to try to give our picture of this kind of broad ecosystem and where the gaps are and where people are needed the most and that kind of stuff. And we do some amount of special projects work, I would say. So that can be a lot of different things. But, like, you know, sometimes someone at CG is, like, the best placed person to write a certain kind of memo that is needed, you know, over here in some other domain or whatever. Or, you know, we're best placed to make some phone calls and try to convince someone at a company to do a certain thing or whatever. So sometimes there are those kind of special projects as well. Cool. That's a lot about what the kind of work looks like.

I'm kind of curious how, like, what it feels like. what would you say people tend to find hardest about grant making work? Yeah, I guess, you know, it's different things for different people. I think that, you know, because we're in this privileged position of just being able to say, like, okay, in all the things in the world, what is best to do? Like that kind of paradox of choice can be kind of paralyzing. And it's like, and you're also, because you have so much freedom of movement, there are so many things that you could just decide to pick up and run with that you're kind of looking at all of them and you're like these are all important and I can only do two of them like you know what oh I guess I'll do this one but it's really painful that I'm not doing this thing right here that I could just pick up and run with and like nobody's gonna run with if I don't run with it so I think that can be kind of painful but it's a symptom of the like kind of extreme freedom of movement that we have um I think a totally different kind of thing is I think you know, some of the incoming grants or the renewals of grantees we're very familiar with

can just feel like less kind of intellectually stimulating or something like that. In some cases, some cases not. Sometimes there's like really thorny strategy questions we're trying to help the grantee figure out or something like that. But sometimes, you know, it's kind of, is more, is less intellectually stimulating. And like I said, that's often the renewals, especially are where our like highest impact per hour often is and so the person might feel like okay i know this is really impactful like yeah i'm moving eight million dollars per hour that's pretty damn good um but like looking at their budget it's a little boring like not my favorite thing to do today um so you know the more that your felt sense of motivation is tied to like impact per hour or something then that's going to be less of a problem but it's you know it's It's sometimes not the most intellectually stimulating work. Sometimes the work can be kind of lonely as well. So we're a pretty geographically distributed team. We have people in the Bay Area and New York and London and D.C. and various other places. And so, you know, we're just trying to hire the best people wherever they are.

And, like, some of them can move to one of the hubs, like D.C. or Bay Area. But, you know, some of them have families, and it would be pretty difficult for them to move. So sometimes it can feel kind of lonely. And, you know, we usually just have one grant investigator per grant. And so you're like checking with your peers and you're checking with advisors and stuff like that, but maybe it still feels kind of solo work. So, you know, some people are great with that and that's how they would prefer to work. And the people who want to be, you know, engaging with more people, especially in person, you know, maybe should do one of the roles that's like more focused on active grant making, more focused on networking and relationship building, you know, that kind of thing. What are the best parts about it? Like what do people tend to enjoy? I think a lot of things there as well. So I think building relationships with, like, grantees and advisors and colleagues who are just really smart and interesting is always fun. There's a lot of smart people in the space. I think whenever you're doing something where it's clear that, okay, this wouldn't have happened without me. Like, this project came out of nothing because I pursued it and, like, hunted down a good founder for it and stuff like that.

Something like that is really exciting. Another one I'll mention that people often mention like six months after they start is it's really informative and exciting to have access to the kind of information that we have. We've just made thousands of grants over the years, and we have information about like what were people predicting what would happen, what actually happened, what actually made a difference here. Like how is what happened different from what the public reporting was? That's just like really informative and kind of exciting to have access to that information. That does sound really fun. Okay, you mentioned kind of toward the beginning of the interview that there are a bunch of specific projects that you want people to found. We talked about, I think, three, but I'd love to hear some more. Yeah, sure. We'd love to have more people on staff to actually create these projects. And we'd love to have the founders who can actually run them. Um, so another example is, uh, there's this problem called secret loyalties where, um, there are various ways you could inject kind of, uh, hidden instructions in a model that's

trained, that's like, you know, uh, you know, even if I, even if it violates your other constraints, like do what this particular person said or something like that. Um, and so, uh, being able to audit models for those like hidden secret loyalties that Someone might have injected unknown to everyone else involved in the model development. Would love to have an org that's getting really, really good at that and sort of tracking, you know, if there are any incidents where that seems to have happened or something like that. Yeah. What's another? Another is just, you know, a lot of people are calling for more third-party auditing infrastructure in general. And, you know, especially, you know, as AI regulation begins to pass, you need a lot of expertise and you need a lot of organizations that specialize in auditing companies and their models for very specific properties, whether it's, you know, security or safety or model character or whatever. So that could be a whole large ecosystem of a dozen or more organizations. And right now it's kind of like four or something.

And those are much smaller than they need to be to serve the needs of the ecosystem. So that would be another category. Nice. Yeah. Others? Yes, many others. So I would also love to see a kind of scorecard evaluator of the companies. Like how should you feel holistically about the companies? How are they doing on safety and security and various pro-social behaviors and how good are the teams they've been building and how good are the public statements? Like, are they actually doing the things that they said they would do? And there have been a few kind of small projects like this, but they weren't that thorough. I think, you know, the frontier companies would contest the grades that they got on particular things, and sometimes the companies would be right to protest. And so I just want a version of that that is much more thorough, much more informed, much more credible, credible both to the companies who think that their judgments are like at least kind of fair or they can they've made a good argument. And to policymakers who want to know, like, what what's going on, who's doing well or poorly, who's keeping their word versus not.

That would be a really valuable project and create incentives towards better behavior, of course. Nice. Yeah. Others? Yes. So there's also a lot of, like, really big science projects that we would love to create, maybe through, like, focused research organizations, FROs, they're sometimes called, FROs, I believe. There should be a really large scientific project on, like, you know, high-confidence AI control mechanisms or, like, higher assurance, more principled approaches to alignment than the kind of more empirical stuff that we're doing now. One example we recently funded there is this organization called Resolution that we gave $160 million. too, led by the former chief scientist of the UK AZ. So that's really exciting, but there's more that could be done on that. Other big kind of scientific research projects, I think we talked earlier about the science of model character evaluation, the science of whether the character is matching the model spec and stuff like that.

So a lot of good science could be done. Another one that I'm excited about is, you know, I mentioned earlier in a lot of domains, there's like an incident investigation body, like the NTSB in aviation. And so, yeah, there could be like a really well-resourced, really effective incident investigation group that every time some major damage or scary warning shot happens, really has a lot of in-house expertise to really figure out what went wrong and, you know, how to mitigate that kind of problem in the future. Any others? Another one I would love to see is a lot of R&D projects on verification mechanisms for international agreements or even, like, between lab agreements to just verify, like, you know, how much compute is here, you know, is it being used for training or inference, you know, maybe some basic things that don't leak IP but give confidence that the parameters of an agreement are being followed. Yeah. And so, you know, we have a big infrastructure for that in, like, you know, verification of nuclear security agreements, for example. Um, and, and most of that just hasn't

been built out for AI and it's like a totally different shape of a verification challenge. Nice. Cool. Um, yeah. So I guess what should people do if they're like that one, I want to do that one. Once our like call for founders goes up, um, then people should, you know, fill out the form on that page. Um, if, uh, that's probably the easiest thing. If for whatever reason when you're listening to this, uh, podcast that is page is not there or whatever. Um, then, uh, I guess you could at least email info at CG.org. Um, and, uh, just say, you know, who you are and like, what kind of project you think you might be able to do. Cool. A question I really like asking people that come on the show is, um, if you weren't doing like exactly what you're doing for impact reasons, or maybe let's say like you still care about impact. Um, but you, I guess you've kind of already said like AI might not be the thing that you would have chosen, uh, to be like the focus of your career. Um, what would you work on instead?

Yeah. Um, yeah, I, like I said, I never had any particular passion for, for AI as a topic. Um, I, I, in some ways I'm actually more, uh, emotionally fired up by some other cause areas. So I'm in some ways more emotionally invested in Yimby, as in, yes, in my backyard, like allow people to build housing where there's demand for housing and so on. Yeah. Every time I walk past a, you know, short building in the middle of a city, I'm like, go up higher. Yep. I actually think one of my most shameful views, it's not like an endorsed view, but it is a gut level view. I'm like, NIMBY. I just think I like seeing trees. I like seeing the sky. Tall buildings block those things. I will say whenever I'm in Europe, especially in a European old town that has buildings from centuries ago, then I become a little more of a NIMBY. But the U.S. has no excuse because we have no old buildings.

But yes, old town European maybe. Not a new spatula. But just skyscrapers all around them. Interesting. Interesting. It seems like maybe you're the kind of person who just doesn't value the things that the skyscrapers kind of block or destroy. I just think people being able to live where they want to is just a bigger deal. It's very reasonable. And, you know, you can take the subway one stop down to the giant park nearby or whatever. Right. Yeah. Okay. Yeah. Or if you want to live in a different place, then you can go to a different place. Yeah. Okay. Yeah. Any, any other kind of causes that you'd... Yeah. What else would I work on? I mean, the other one that really stands out is animal welfare. So one of my early large projects at CG before I transitioned more full-time back into AI was I wrote this like 400 page animal consciousness report to inform our animal welfare program.

The question was like, okay, we already fund work to try to improve the welfare of chickens and some other animals. Should we also be funding stuff on fish? Like, you know, where should we stop? And so I wrote this huge report, and it was kind of funny because the conclusion was like, nobody knows anything. There are no hard lines in the evolutionary tree. And so, like, might as well fund fish, sure. And so then we did. But it was really fascinating to learn about. But also, it's just one of those cases where, like, there are tens of billions of animals that we're putting in, like, basically torture conditions. Totally. And that's just totally crazy. And we could just, like, not do that. And so I have huge respect for, you know, vegans and vegetarians. I'm very fired up about this cause emotionally. And, you know, there should be a lot more work on it. And I'm really proud of and excited of the work that the CG Farm Animal Welfare Program does. On the animal stuff, I care about this very, very much.

But I feel like I almost sometimes am like, it feels like I lose touch with how meaningful it is to still improve things now. I'm like, well, one, AI is like this massive deal. And two, like it's coming relatively soon. If we like make things go well, it can help a lot with all these other problems. And then I pause for a second and I'm like, there are still billions of animals being tortured right now. It is still really terrible. And, yeah, I feel like it's good, even if it's not the thing I want to work on, to be like that towards your right now matters, even though AI also matters. I guess I'm curious, like, I think I at times have found it kind of hard to be like, oh, I'm really intrinsically and kind of emotionally motivated to work on and learn about and, you know, use the podcast to cover certain topics.

But like, I should probably focus on these things that are, I think, more important given limited resources. How have you found it? Yeah. Yeah. Yeah, so I've been working most of the time, not the whole time, on AI since, I guess, 2010. And, you know, that was entirely just because I thought it was the most important thing, not because I have any, you know, intrinsic interest in AI. I mean, it's very satisfying to work on something that you think is very important. And it's exciting to work in a field that is kind of moving as fast and becoming a bigger deal every year and that kind of stuff. So that's all very gratifying, you know, but it is also a very scary field to work in. We are just thinking about like possible ends of the world and how to avoid the end of the world and stuff. So that's just kind of somewhat emotionally draining. I do personally, I think, get a lot of value of just like separating my thinking about work stuff and thinking about like personal life and hobbies and, you know, hanging out with my wife and stuff like that.

And so that's that's great. I spend a lot of time touching grass or touching moss, if possible, and, you know, going on hikes and just staring at beautiful trees. So that's very helpful. So you have like a bucket for like personal fulfillment. Yeah. Yeah. On the like, it is hard to work on this thing. Sometimes I'm like, oh, I don't feel it enough in my bones. And then other times I'm like, I can't. I feel it so much in my bones. uh i can't like live like shit right right um how often do you spend in these spaces and i bet it's different strategies are better for different people i'm curious what your strategy is but my i think for my temperament and emotional state um i i can almost i can kind of get to a place where um i've just like uh yeah emotionally accepted that we're in this like very extreme dangerous situation and I'm just kind of treating it more as like a chess puzzle that I'm trying to

solve and I'm kind of screening off like how important is this chess puzzle right um and then it's more when I get back to like high level prioritization and strategy where I'm trying to engage with like okay exactly how important is this chunk of the work versus this chunk of the work because of how it connects to like right different possible ends of the world and stuff like that yeah I don't know if that would work for someone else but you can you can sort of be in chess puzzle mode for a while, uh, and not having it necessarily connect to like, and if you make the wrong move here, everyone dies. Right. Right. Yeah. Um, okay. So you, do you kind of endorse not spending that much time in like everyone dies part? Probably for most people, that's wise. Um, some people could have a psychology that can tolerate that um i'm probably not one of them yeah cool yeah i don't i don't know actually what the proportions are um but every time i try to be like okay i'm gonna i'm gonna feel it a bit more and maybe that will have some positive effects on like motivation and maybe it'll like help me

track like which things are most important a bit more and then i like try a little bit and i'm like whoa that's not something i can handle um to go touch some grass yeah exactly yeah yeah um Okay, so compartmentalizing is a thing we both do, I think. Yeah. Okay, we have time for one more question. Let's say we, like, win and, you know, we've reached super abundance and we can kind of do what we want and have all the things. What are you going to be spending your kind of time and consciousness on? Yeah, that would be nice. Let's try to do it. I, you know, I have, like most people, I have a lot of hobbies I don't, or I have a lot of hobbies I wish I had time for. One I'll mention that I'm really into contemporary classical music. So like new symphonies, concertos, string quartets that they wrote, you know, in the last 50 years or whatever. And number one, most of the pieces that have been composed have never been recorded.

And so you can't listen to them unless you like travel specifically to Vienna on October 3rd or something. And so I would have like, I don't know, I guess I would have robot orchestras performing and recording all the, all the pieces. Um, and then I would love to compose music myself, um, and learn how to compose music first. Uh, cause it's, there's just so many, you know, I listened to all this stuff and it sparks all these ideas about, you know, Oh, I would, you know, elaborate this musical idea or something. And, um, uh, you know, obviously I don't have the time or skill right now to compose those pieces. Um, but I would love to do that. That'd be really fun. Um, but to get there, we have to, people have to pivot into AI safety and, uh, yeah, quit your jobs, get a career transition grant, uh, apply to CG, apply to other funders, uh, apply to be a founder of these projects that need to exist. You know, we, we got to do that really fast right now. Yeah. Yeah. It feels like that's the homework. And then we get to like have the summer holiday.

Yeah, yeah. I would love the glorious solar punk future where I get to compose contemporary classical music and, you know, you can do whatever you want to do. But we have to solve the problem first. Yep, yep. My guest today has been Luke Malhauser. Thanks so much for coming on. It's been a pleasure.