エリゼア・ユドコウスキーへの異議 ― PauseAI US のホリー・エルモアと語る

Eliezer Yudkowsky. I know you blocked him. I'm just pissed at the way that he does things specifically to, like, alienate and, like, damage the reputation of the cause. Like, taking a picture that's an homage to, like, a gay picture with Ayla. And I just felt like he betrayed me by doing that. Like, making it, like, difficult for me to do AI safety work in the public eye. It always didn't sit right with you that you thought that he was trying to coup the world. He has never this entire time, even after he realized, like, alignment is going to be harder and harder and harder, reflected on, like, was it okay for me to try to just usurp the entire world by making the singularity? I claim that when Eliezer got farther along, he probably would have involved more humans. I think we have a lot of evidence that he wasn't going to do that. He wanted, like, his team at Miri to make an AI and then commit a pivotal act. So that would have been the coup. I mean, at what point would you start involving more people?

Welcome to Doom Debate. Today, I'm excited to have returning champion, friend of the show, and executive director of Posse IUS, Dr. Holly Elmore. That's right, she has a PhD from Harvard University in evolutionary biology. You may know that in 2023, she led the first Pause AI protest in the U.S. It was very early back then. Not that many people were calling to Pause AI. And then over the next few years, you started getting all kinds of prominent voices coming around to the same page. Most recently, Jacob Coxon and the famous Pacing the Frontier letter and Gary Marcus, another friend of the show, coming around and saying we should at the very least pause OpenAI, if not do a larger pause than that. Polly Elmore, welcome to Doom Debates. Hi, Leran. Thanks for having me. Okay, so let's start with Paws AI. You're the founder and executive director. I do feel like your mission, our mission, you know, I'm a volunteer for it as well, our mission to shift the Overton window, I would say has been a huge success, right?

If you just consider how much the Overton window has shifted. Do you feel that way? I would say I'm the founder of Paws AI US, and that's our full name. So nobody get confused. If you want to check out my organization, It's pauseai-us.org. Just to disambiguate, there are other Pause AIs in other countries. So, yes, I can't believe in some ways how vindicated we've been in such a short time. It's truly amazing. So, recently, with Pause AI US, we have these events, Pause Con DC, where we do what's called a Hill Day, where we bring people in to speak to their representatives, constituents. It was like four days after the Jacob Coxon tweet storm. And the difference in the receptivity from congressional offices was wild. They wanted a lot of detail suddenly from our constituents. It was just wild. It was a huge inflection point for the public, but even bigger it felt like for Congress.

This went from being a fringe thing they could just completely optionally even acknowledge to everybody's got to have a position on this. Yeah, I remember that. I mean, it was a surreal day to see dozens of people in U.S. Congress tweeting the exact kind of stuff that you would have been a total freak to be tweeting in 2023. And everybody was like laughing at us. You know, we were tweeting that stuff. But now it's like people in Congress. And so the character assassinations are really falling thin when it's this mainstream idea that the majority of Americans are now concerned about. Yeah, they're really ringing hollow. The Jacob Coxon tweet, for whatever reason, was just the straw that broke the camel's back. And suddenly everyone remembers that is like this is when we were told, which is wild because like I know you knew before. It's like really what happened is like the latent heat built up and then it boiled finally, like when the very last increment was added. But like the way the story goes down is like we didn't know. And then when we knew, we acted. And like I just saw that being like those bricks being laid in real time, which is very surreal.

You know, I personally got to experience the before and after response from Congress because a Posi-I-US volunteer that I've been working with, we had some calls with the high-level staff of members of Congress. And I still remember it was only a few weeks ago I heard a reaction like, listen, we're not doing the treaty, okay? That's a no for us. So what else can you tell us? You can tell us about the face huggers. Like didn't really understand what was going on with hugging face. And we're like, yeah, you know, just keep this issue in mind, okay? And they basically weren't hearing it. And then, you know, a couple weeks later is before and after the Coxon moment. They're like, OK, can you please let us know how much you think we should fund the USA safety institute? Give us the logic for a number here. And we're like, well, the UK is already doing 85 million. So you got to do more than that. So the discussion changed very rapidly. I'm hearing lots of interesting anecdotes from our volunteers about like the difference from like meetings they've had before to meetings after. And then there's demand now for different kinds of like support from us. Like, like offices want, like, they're like, okay, actually can pause AI, like help us understand, you know, what, what kinds of like legislation is out there or like what kinds of examples are out there for dealing with this.

And, like, we have been working on assembling those and having them ready for when people want them. But in, like, a span of two weeks, like, I realized, like, our content was going to have to change a lot from, like, basic education of elected officials to, like, dynamically responding and, like, doing, like, interesting, like, extensions of their staff work to, like, help them understand, like, much more complicated legislative questions, which is amazing progress. Really, really excited about that move. I want to like clarify for people who I don't know maybe they've gotten used to this debate or they like feel like oh pause is one side and then there's this other stuff. Pause is a simple policy and it's really what should proceed any next step. You don't make the problem worse before you figure out your plan and do it. That's all pause is. Pause isn't even saying we have to handle it a certain way in the end once we have the information once we have public buy in once like there's been a period of like societal deliberation in a democratic way. So pause is really like the simplest, most obvious solution.

We're in a wild situation, probably less wild than people think, because like I think a lot of people, especially like tech Twitter, they don't know how government works. And like they don't know that they do listen to constituents. Like, if you're an interested constituent and you're an expert in an issue or, like, you know, kind of like a citizen expert, you can be quite helpful because you're, like, the real – you're from the group that they represent. And there's very few groups that have, like, demonstrated, like, a straight-up, like, public interest in AI policy like our volunteer group has. So it's sad that it comes down to us, but also the opportunity is really, really real. Yeah. Yeah, I mean, that is also something I experienced too. It's like, that's right, you can meet with Congress. Like, it's not that hard. Sometimes you have to be a little bit persistent, but they're certainly open to it. And there's no cabal of insiders that they only meet with the insiders. Like, no, you know, we're all just people. We all show up. It's not that hard if you're persistent to, like, meet them at a certain conference

or you get on their phone schedule or go to an organization that's going to talk to them. Like, they're not that inaccessible. So what else about Paws AI? You've been running it for three years. one thought I had is pause AI versus this new thing pacing the frontier. Because pacing the frontier, that seems to be winning the branding war right now, right? It's the most viral, the most fit meme right now, pacing the frontier, probably. I don't perceive that. That's interesting. Well, it seems like it gives people the most cover your ass potential, right? Because it can mean anything, whereas pause is a little bit more clear. But it's funny, because I remember back in 2023, I remember being there with you at the manifest conference. I think that was the first time you and I met in person. And I think it was when you were still a little bit uncertain whether Paws AI was the ideal name. And at the time, I think you and I were thinking like, yeah, it's a good name because it just means Paws. It doesn't mean we have to stop it forever. It's like a nice, flexible name. So we thought it was going to be like really strong, universal branding. And in retrospect, I still think it's pretty good branding. But now we have this other meme, Pace the Frontier. Maybe that's a more fit meme. What do you think? I think that's only more fit

among, like, tech insiders, and, like, that's going to be increasingly, like, a very small part of the conversation, like, now that everyone is aware of the threat to everyone. So I don't see Pacing the Frontier come up that much in anywhere outside of tech or, like, fans of the AI companies. Like, I don't even hear that as something that, like, government officials are, like, really excited about. I think because government officials realize the serious problems with this idea, which is it sounds like it's a compromise. Like it sounds like, oh, like rather than having to pause all the way, like we'll just slow down a little. And that's kind of so like cognitively, it might feel like that would be easier. But it's actually way, way harder to just slow down and pace or whatever that means than to pause. It's very hard to monitor open AI and inanthropic pacing. you know it's but it's very easy to monitor if they have paused like if certain activities have ceased you know whether they ever resume again or not so i did like the original letter only in the

sense that it seemed it expressed like fear you know and and doubt about like what the ai companies had set up until now you know and so it was a concession definitely like it's not like we have to go as fast as possible. But it's a very calculated concession, which is like, we need to maintain, yes, yes, you're right. It is going too fast. Like, we weren't doing things perfectly. But, like, we still need to be in control. And, like, if you give us control and if you remove the barriers that the government put in place to us actually slowing down, then, like, we will do it. So, I mean, I'm pretty negative toward this proposal. Yeah, yeah. I'm pretty negative toward it, too. I mean, if it's an incremental improvement and it's going to get better soon. Great. But it's Weasley. And as you point out, it's like pacing. Okay, four minute mile. That's my pace. Wait, what? That's like a fast pace. Right? That doesn't count. That's the pace. Yeah. So where we net out is we're keeping our distance from the phrase pacing the frontier.

We're not adopting it. But if it ever started showing a lot of success, if it ever got tied to policy where the government was centralized and we were building the dial to slower pause AI, you could imagine embracing it, right? yeah I think there's very few situations where it would ever really make more sense as a policy but if that's what it took to get agreement as long as the control actually was like representative of the governed that would take care of like the biggest concern for me got it got okay yeah if you ever do embrace it you can just be like oh yeah pause AI that's just the American way to say pace AI, you know, it's just like Tesla, Tesla, you know, pause AI, pace AI, same thing. That's my accent, yeah. Yeah, well, they definitely wanted, it was kind of an honor, too, because they clearly wanted to pick up on the pause meme, like, and they consider it important. But, like, pause is the name of this idea now. Like, this is what Bernie Sanders calls it. So that's also a pretty big cultural victory. And then, like, pace is clearly trying to, like, swoop in and, like, take a little bit

of that space. Well, this is what I mean by Overton Windows Success, right? So you have spent the last three years, and me too, the show, and volunteering for Possei US, we've spent the last three years trying to make it okay to yell Possei, going so far as to literally yell in a megaphone at these companies and to quote them and call them names. Yeah, Lidl was on the front lines. He was the guy on the megaphone, like, keeping everybody kind of, like, entertained for the whole protest, like, and getting lots of great clips. Like, he would just, like, go for, like, an hour oratory. It was very impressive. Exactly. You know, that's what I do. I like to talk. So we were doing this for three years, and we were creating that space that when people say Pace the Frontier, it has a good ring to it because you're like, oh, okay, it's like pause, but it's more measured, right? Whereas if you'd yelled Pace the Frontier a couple years ago, it'd be like, whoa, whoa, whoa, Pace the Frontier. What is this? Where is this coming from, right? Because the idea hadn't been seeded in mind space. Yeah. And then I, you know, for just a little info about how the Overton window works, like you should start really with like what you want or even more as far as like negotiating.

And so starting weaker with like, oh, like maybe we should just slow down and more like wishy-washy. Like if we had done that, you know, then the debate now would be something like closer to like, do we have to stop at all? You know, so you really don't do yourself any favors by like weakening your opening position. And, like, now slow instead of being, like, people kind of trying to meet Paws AI and, like, suggesting, like, well, maybe we could slow down. Like, slow is now the offer of these companies, but they had to, you know, name it their own thing. Pace. So, also regarding Paws AI, there's two organizations, as you said. There's Paws AI US and there's Paws AI Global. And within Paws AI Global, I know there's people who are, you know, there's, like, Paws AI UK, Paws AI France, Paws AI Belgium. So there's regional chapters of it. But then PosiAi US is just a totally separate organization. And before, you know, I'm a member of both. I've just been online in both for a while. Yeah, I'm not like spending that much time, you know, doing executive operations or anything,

right, but I'm a volunteer. I'm helpful. And it just, it seemed like two really aligned organizations. I think both organizations want to PosiAi. But then a few weeks ago, there was like a less wrong post, right? there was an open letter where Posi-I Global came out as like, we disendorsed Posi-I-US, right? Because they have like, no, they disendorsed me. It was specifically me. Yeah. At the end of the day, we all want to see AI get paused. I think the two organizations are aligned, even if they're not operationally working together. We also want to talk about Eliezer Yudkowsky, the leader of the movement, somebody I've been following since 2007. I've said on this show last year when his book came out that I think he is arguably the most important person of our time, the most important thinker of our time. I stand by that. He's also the most important thinker in terms of how my own thoughts are shaped. He's had the most intellectual influence on me. I'm constantly pulling out Eliezer Yudkowsky stuff, and it's hard to imagine being as productive of a thinker as an adult

without all the rationality scaffolding that Eliezer Yudkowsky helps set up. Okay, so I'm an Eliezer Yudkowsky fanboy. This is a Yudkowskian show. But now you're coming here actually having a beef with Eliezer, and you actually think that he's, like, overrated, and he's actually, like, doing something that's harmful to the Possei movement. I don't want to say overrated. I think he is extremely important, like you said, but it's not, like, all a force for good. And I, so I kind of, I didn't know the full history of Eliezer, like, for much of my time in this world. I was never attracted to the world because of Eliezer, and I never, like, did a deep dive on, like, all of his writings. Like, I read the sequences because that was the thing. But I didn't learn a lot of the stuff about the history of Miri and the Singularity Institute until fairly recently, like, did I really understand the full story. and I the picture has like emerged of like Eliezer as someone who like wanted to take over the world and felt that it was his right if he could figure out

how to make the singularity and he's never really repented of that he has never this entire time even after he realized like alignment is going to be harder and harder and harder like reflected on like was it okay for me to try to just usurp the entire world by making the singularity and releasing it Like, he's never, ever said that that was not okay. He's basically, like, now what he does is, like, defend a position that, like, that would never, ever come up. It's completely moot, so we don't have to talk about it. You're in good company with this particular beef because Garrison Lovely was on the show last week promoting his new book. Everybody should check that out, Obsolete. He's in a similar position as you being like, man, Eliezer was trying to push unilaterally. I think he used the phrase, who the world. I think that's even in his book. so it's a popular beef to have my defense on the show when i talk with garrison is that actually coherent extrapolated volition which is the ideal that elio zirikowski was hoping to bring about with his superintelligence research program back in the early 2000s his coherent extrapolated

volition is not like imposing his will on the world it's actually just having the ai figure out the meta problem of how humanity can democratically pull their wills together at the points where they converge and direct the future that way. So I think describing it as a coup is very unfair to Eliezer. I, okay. So yeah, what you miss on this is like deciding to submit to that criterion is not, there's no vote for that in Eliezer's world. Like, and actually he took great pains to avoid it coming to the attention of people who wouldn't like it. And he's never really changed on that. Like he does not like, it really, like, hurts him to talk about these issues with people outside of his group. He complains about it constantly, that, like, you know, they don't know the way to talk. Like, a nerdy blogger will, like, be charitable and, like, understand your position in a way that, like, a journalist, boo, you know, hates journalists and always talks about that, like, will not understand it,

and they won't give you charity, and, like, he just doesn't like, like, taking things to, like, the actual set of, like, all people who would be affected. I mean, you can tell me that somebody would represent my will better than me, but if you impose them without asking me, then that is that you have overridden my will. Like, it's not respecting people's rights and self-determination. Also, like, I don't know, it seems like with the CEV stuff, you're asking people to imagine, like, a world where the principal agent problem is just, like, not a problem. And you're saying, like, that's part of the idea of CEV. But it's like, if that's true, then you're talking about an ideal that can't be realized or we have no idea like how it can be realized. So I don't really care that he's talking about some hypothetical thing that it wouldn't be a violation of people's rights. Like, but it would somehow just be an extension of them. Like, and that's that doesn't exist. It's like in the singularity will be a utopia. And so everybody will love it. And so retroactively, they'll say they'll wish that they had been in the singularity.

Like, OK, well, coherent extrapolation, you know, it's a very idealized vision. Okay, so it involves solving the principal agent problem, right? So with coherent extrapolated volition, because it's just an ideal that's, you know, without specifying how to solve the ideal, it's just kind of a direction in the distance that he's hoping the AI will help run toward. You get to just specify solve the principal agent problem. But there are just AIs up until then that control stuff? I mean, no, it's not. That stuff is not excused because he had the idea that nobody would be bothered by it. Like, okay, I think the ideal, maybe we can agree that if the AI was always faithful to CEV as written, which is a very ambitious ideal, it has a lot of placeholders that we don't know how to implement. But it's, you know, that's how theory often works is you start with like, barely being able to define where the ideal point in distance would be. And then you're like, hey, here's like an approximation algorithm, right? You like iterate from the ideal to like something you can do that's practical. And CEV really doesn't claim to do much in the way of operationalizing it. It's just already making progress, defining an ideal.

so you can criticize you can say that the ideal is impossible which i don't actually agree with i think it's a pretty well-defined ideal or you can do the other criticism which i think you're also doing which is like the way he was going to try to achieve the ideal isn't democratic like the bootstrapping phase where you release power to the ai even though the ai is supposed to give you back power it's kind of like risky like you're releasing power you're hoping to get power back but maybe you won't maybe something will fail and the decision to release it this is basically what garrison and a lot of the commenters on my last episode were criticizing they're like how can you have a process release it and my argument to that is i claim that when eliezer got farther along he probably would have involved more humans right i think it's impossible to say because he never got far along at all because he realized hey this project isn't the way to go so i just it's hard to do that he wanted like his team at miri to make an ai and then commit a pivotal act to make which is making ai to like ensure that others can't make ai and like so that would have been the coup i mean you're like at what point would he start involving more people right but you're

you're just making an assumption so you're saying because he talked about a pivotal act needing to happen at some point i think you're you're making an inference like therefore he wanted to like personally be in charge of the pivotal act and then it would be too late for any other humans to intervene but maybe he would have paused and gotten some people to review the pivotal act that he was about to do. I think he wanted the AI to be in charge of the pivotal act but he also felt that the AI would have like very more perfect judgment than any human which would mean it was very close to Eliezer's judgment and like you know Eliezer can he's not you know picky it's just that he sees how things should be. He sees where civilization is inadequate. Let me also take a step to define pivotal acts. So as I recall reading Eliezer's technical definition it's something like an act is pivotal if it permanently changes the course of the future in like a significant way, like it alters a big piece of the future. So if you just do something like a particular scientific breakthrough, like that could be cool, but not exactly a pivotal act. He's previously used the example of like, imagine an AI that burns all the GPUs

just as a purely hypothetical example. Like you never personally wanted to burn all the GPUs, but he's like, hey, if an AI did that, that could be pivotal because it could stop like the bad singularity. That would be an example of a pivotal act. Or if the AI became a singleton with power over everything that no other AI could stop, that would also be a pivotal act. I think that's the kind of thing he's talking about when doing a pivotal act. And I think you're saying pivotal act could easily be like the point of no return. So whoever gets to shape the pivotal act is granting themselves the power of the future. If the pivotal act is something that prevents others from making AI or that like seizes control of that possibility and they made an AI, like that AI will control the future, right? Like, unless something happens to it. Like, but that's, that was the, that's the idea. And, like, when people talk about pivotal acts and, like, the rest of that discourse, I actually didn't know it was that broad of a definition originally. Like, because people pretty much only mean, like, making superintelligence in order to, like, take control of the board,

if not, like, reach the singularity yet. That was my understanding of what a pivotal act was. Okay, well, so let me just ask you this. Maybe we can close out this half of the argument if you agree. Do you think Eliezer did like a legitimate work of philosophy, basically, when he just defined, yeah, coherent extrapolated volition. The AI has to basically model us really well, right, like a fine level of detail and ask itself the hypothetical question. Okay, if we all as a species had many years to deliberate and consider all these scenarios and find points of convergence, right, that's me doing CEV in a nutshell, coherent extrapolated volition. And the general concept as a philosophical idea, do you think that sounds right? And the only question then is the imperfection of executing it? Or do you think you even made a mistake in the philosophy? I'm not trying to be difficult. I really do think that this is where a lot of the important stuff is. So my background is in evolutionary biology. And one of my specialties is a pretty niche topic called genetic conflict. And it's about conflict between the genes in the genome of one organism.

and there's just a lot of extremely subtle stuff about alignment and misalignment that you learn for I have a blog post where I try to explain this for like a lay audience and some of the insights and there is a there's a very flawed understanding of misalignment among Eliezer's ilk and like you know so Eliezer's not the one who wrote the Moloch essay that was Scott Alexander but that concept is extremely, extremely flawed. And it has the same flaw as a lot of the thinking about alignment for AI. Like that, you know, oh, it's stupid that there's these little subtleness alignments that cause a race to the bottom. And therefore, we should just not have them. But like, there's not an understanding of like, okay, why not? In rationalism, the answer is you should just be smart. You should figure out the best way to do it. And everybody should have to do the best way. And one, there is no best way. Like, there's, you know, Even if you can solve mathematically, like, what's the most efficient way to do something, that doesn't mean that it's the way that suits every person best.

Like, there might just be genuinely misaligned incentives, and you can't force them to be aligned because the world is imperfect and it should be perfect. And so, like... Just to clarify, you're saying he messed up the philosophy, like, of CED, even hypothetically? I think, well, like, the part, the version of CED that's true that you described is just kind of like magic. Like, well, what if we did know what everyone wanted? It's really not right. But it is, I mean, the theoretical ideal, I mean, the answer to the question of what does everybody want is in principle stored in the neuron configurations, right? So an AI that's sufficiently good at decoding that, right, it is a coherent hypothetical here. I think there's not one answer for that describes all people. It's about, like, the system that is used, you know, to determine there's information about people's preferences. Yes, it's weird because this whole world is like anti-moral, anti-realists, but they are realists about this like theoretical perfect society. Like, and they think like if they were reaching that, if they were reaching some maximum, if they were optimizing, you know, like on some graph that, you know, they could draw from real data, like theoretically, that that would mean that it was a perfect society and therefore like everyone has to agree.

it's very in contrast to like a rights view you know which it's weird too because this world is also very pro rights but like I just trying to understand what you saying you saying his philosophy was flawed because this whole idea that CV you know the C stands for coherent You saying that it so incoherent that there's not going to be a coherent extrapolate. That's where you're coming from? No, it's more that there is some algorithm used to make it cohere and like the devil is in the details of how that would work. Right. I mean, that is true. The devil is in the details, right? I think all he did was just like he threw some guidelines and they're a high level and it was just like, look, imagine, so I think you have to refer to what you think is the alternative process, because what he tried to do is be like, yeah, imagine the ideal alternative process just happened in a big simulation, right? So you imagine, like, you know, it would be a great process. Give people so many years to just deliberate. Just let the messy deliberation happen. And he's like, right, yeah, we're doing that in the simulation, you know? I do believe that you can, like, take processes that aggregate preferences

and augment them, you know, and that that's what you're describing with CEV. I don't think you can augment them to the point that it skips the step of people agreeing to enter this, like, different world where, you know, it's not just that their governance will be different. It's that, like, all of their options will be radically different because of the singularity. And I've written some stuff, like, a couple blog posts about, like, the problems, sort of logical paradoxes and stuff created by abundance imaginations about the singularity. Like, it's not heaven. If, you know, if your friends decide to go into the singularity and you stay out, like, that takes away your world. Like, it's like, ravished by that. Like, it's, you know, you can't, you don't really have an option to just opt out. If the singularity places you in some matrix where your world never changed, you're being deceived. You know, like, you're being wronged at, you know, some point here. Well, I think you have an advantage in the argument here because it's easy to point out holes in somebody else's proposal, but what is the alternative? What is my proposal for perfect governance?

I mean, how do you imagine like the next hundred years playing out? I mean, you say you say I want to pause AI. So if we pause AI for a hundred years, then we'd certainly not get the maximum kind of scientific growth that we're hoping for, which is potentially it's something to consider. I mean, freezing the timeline, you know, in the kind of like the matrix, you know, like slowing down all of technology. I mean, that is certainly one option and it might even be the prudent option, right? I could even endorse it, but I have to admit that that doesn't seem to me like the ultimate ideal option. That seems like a compromise, kind of like a sad compromise option because we know we can't do better as a species. So we just have to like wait. But if you give us like a thousand years, 10,000 years, I don't think freeze 2026 forever is like a good 10,000 year plan. So I think when you start thinking about what's your ideal 10,000 year plan, I think you start realizing like okay go here and extrapolate abolition does kind of sound like that no there's like a teleological way that you're thinking here like we have to be reaching some greater world or like some like and then you're saying like that well then like what's

your solution like this is a great world how do you propose that we get to like a future greater world and it's like it actually doesn't have to work that way like if you know people have rights and self-determination about their lives, they don't have to agree. They don't have to get with the program of, like, greater, like, meta-structures, which is, like, a lot what Eliezer is really interested in happening. But you have to get other people to agree, and he has a lot of trouble with that, and so he comes up with this idea for, like, how he thinks, like, legally in his mind, like, in the way he sees the game, like, if you make better technology, then that just means you can do whatever you want because nobody can argue, you know, when somebody just, what, of their own free will with their own body makes technology, like, and not just changing the environment. Like, you're just allowed to change the environment that way. And I'm saying that that's, like, fundamentally utter-cutting the self-determination of others, and that is against, like, the spirit of our laws and the spirit of, like, humans' rights. Right, but I'm still not clear on what you

think is better, right? So you want something with more self-determination? Why does I have to provide something better? Like, we could go to heaven, like everything could be perfect so it could be actually out of the physical plane and so there wouldn't even be the contradictions and limitations of the singularity because that's still limited by the laws of physics like and actually if you're in heaven then all logical contradictions are resolved so like once we were there like all of your logical issues that you have with heaven like just be resolved okay the question i'm asking is like okay so cev is one proposal on a direction to go to right if we made a successful handoff to an ai that actually then turned around and worked on CEV, to me, that sounds good, especially because, you know, CEV would have plenty of checkpoints. Like, when you imagine, like, oh, my God, what if CEV is, like, the monkey's paw curls and it goes awry? You know, it's the definition of CEV that it's not going to go awry. It's not that I think CEV would go wrong, and I'm saying you're wrong that it would be, like, this good solution. I'm saying CEV is not what people have chosen, and you need to get their permission first.

you're saying there's this way in which Eliezer could like through technology get people's preferences without asking them and then that would actually be it would be the right thing to do to do what the you know the CEV in this machine said the CEV like encompasses like what it is to be that person and self-determine and then we say without asking them right I mean it would ascertain what they want even if it literally didn't ask in one X. Maybe, I mean, maybe in some cases they would ask them verbally, but that's not necessarily the most efficient way to ascertain what somebody wants, especially if you want to ask them a million questions. That's the only way to ask the person, for them to give an answer. Yeah, but if you scan somebody's brain, you already know what they're going to say if you're asking what they want, and you can do it faster. But it's not about just knowing what the answer is. It's about actually asking them. Like, they need to actually have... Okay, so you're wedded to the physical process of asking, even if I can predict with 100% fidelity what somebody's going to say, you think it's important for me to actually, like, say it at 1x speed? Because you're compromising the amount of questions I can ask them if you do that.

Why do we have to ask them more questions? So one of the observations of CUV is, like, hey, we want to make sure that people get a chance to consider a lot of stuff, right? Like, isn't it nice if things go nice and slow and you can deliberate about stuff? But the funny thing is that you can translate go nice and slow into just ask somebody a bunch of questions, and then you get the same advantages if you let a bunch of time pass? Yes. Okay, so you're treating people like the point is to figure out what they will say. And it's just one way to figure out by asking them. But it's not even the best way. And actually, like, what we should do to govern is some other process that, you know, like replicates their brain in order to ask them. and that is not the same as like respecting consent like that is just like fundamentally like they should be in charge of they should also be in charge of their brain getting replicated it comes down to principal agent problems like it might it can be tempting to say like oh i i just

basically did like i just have like another version of the agent and like that and so and or it's like increasing, reaching perfection, like how closely this agent would answer, you know, in the same way as the original. But they're like not the same. And you can, like, their incentives are different because of being like different people. I hear you. Just keep in mind, at the end of the day, something needs to get done. I actually take issue with that. Why does something need to get done? Let me just work with your example, right? You're saying that at least for some people, it's messed up to scan their brain without consent. I would guess that, you know, I'm somebody comfortable... It's messed up to make a version of them without consent. I'm personally comfortable having my brain scanned if it doesn't cause any pain or, you know, negative emotions. And, you know... What about the copy of you? Yeah, I mean... I'd have to feel it out, but I'm open to the general principle. You're out of control of what happens to that copy of you. And, like, in these simulations, like, presumably,

you're unhappy in a lot of the outcomes, and, like, that's how they know, you know? and like that's a copy of you don't think if it's creating a lot of versions of me that are unhappy then i might have a beef with it okay but if it's the general principle of scanning my brain i'm open to it but then i was running with your hypothetical like what if a bunch of people just never give consent to have their brain scan okay well you can imagine the ideal ceva is taking all that into account because its goal isn't to try to rob anybody of any agency or consent so it's like all right here's the the section of people who are really hardcore about never having their brain scanned and we're reducing the amount of questions that we're asking them we're taking the liberty to ask more questions to the people who let themselves be cloned or operated at a faster speed so we're getting more answers out of them you know that's just how it has to go you know there's compromises they're giving us a more detailed model of what they want because they're giving us more stimulated answers and we're just going to combine everything right i mean at the end of the end you could object you could be like no everybody you have to ask everybody the same amount of questions but it's like as you can see there's always going to be trade-offs and compromises right we just need like a general direction in terms of like how to guide the trade-offs that are being made. Why is, like, the one thing

that has to remain constant, that there has to be, like, a machine aggregating everybody's thoughts, like, and then only after it's done that process does it make it retroactively okay and, like, give consent to the whole process? Like, why do we have to do that? Like, why does something have to be done? Like, it's not that we have to, it's just I bring it back, it's just like, go back to Holly's ideal world. Imagine you get to, you're like, God, you just have to set the direction of the world. a world that sounds like more ideal on paper. Like, we're talking about the world that exists and what you're allowed to do with it. Okay, just give me a world that's better than CEV, right? Because you're saying CEV is not the best part. Okay, so give me one example of a world that's better than CEV. Well, I think because I think this is like a massive violation of consent and a coup of the world, this world is better than CEV. What about CEV where every individual gets to consent to everything that's done with their brain and their consciousness? Isn't that better than this world? if individuals said I want to enter that whole process then that would be much better

there might still be philosophical issues uncovered like what about the clones presumably if the individuals agree then the clones agree almost all of them if this is what people consented to do with their governance then that is the issue let me ask it to you this way let's say we're looking 10,000 years ahead So, and imagine we do pause the IT. We both want to pause the IT. We both agree it's going in a bad place today. It's going to be uncontrollable, right? We both have that issue. So let's say we pause it and we pace it and we, like, engineer smarter humans, right? We use assortative mating or we figure out just ways to raise human IQ and humans become better at safety and we're going so slow. It takes us, like, a thousand years to make GPT-7, right? Like, we're going so slow and careful. We're doing so many safety tests and it's going pretty well. But, like, eventually, are you on the same page as me that life should probably get transhumanist, right? Some people should have the option, which I would personally probably take of like discarding our bodies. I mean, granted, it happened with people's like meaningful consent and self-determination

along the way. I think that like, that yeah, there should be options probably for that. But I'm not really interested in like controlling what happens like in the future like that. I'm really just also okay with like, say this is about the level that like human cooperative like governance can sustain of technology and like these are our lives like i also think that that would be fine i don't have this like yearning that things have to be more perfect and it does seem like you're kind of like reasoning backwards from that and and leaser definitely is and like the the ultimately it's like reasoning backwards from not dying but we're in a point in time where the whole the whole galaxy is at stake here we're both concerned that AI is about to brick the galaxy, the observable universe, correct? Yes. Okay, so the universe is at stake. So it does make sense right now, given that there's going to be this runaway agent that a lot of smart people, like I just had some people from Center for AI Safety on, Richard and Adam Koja,

and they were talking about how, you know, they're super forecasters, and they're saying, yeah, our forecast is that in 2050, if AI wanted to make a Dyson swarm, they could do it in less than 10 years. So, I mean, there's some major technology and galaxy-level stuff at stake in the near future. So I think it's reasonable, as we stand here today, to be like, look, are you up for a transhumanist future? I think we should probably clarify that. I don't see why that's the question. Do you see, though, how you're kind of saying the thing that's really bad about AI being bricked is that we won't have this galactic-level technology that we could have? I don't really see that. I think the thing that's bad about AI usurping the world and killing people is usurping the world and killing people. like right now. Like, I don't think like we have to... I mean, I get that right. I get that freezing 2026 is better than letting the AI run wild, okay? But I think we need to let our imaginations go a little farther than freezing 2026. Like literally just, okay, the year is 2500 AD. Okay, so like, simple question. Like, do you think we should probably have a Dyson Swarm by then and get much more power or no?

Why? Like, I don't understand. Like, why do, why should I, like, why should... I mean, to me, it seems like an obvious yes, because why would we sift the sun through a star? We only have a billion years, okay, before the sun becomes uninhabitable, like the solar system becomes uninhabitable. Why? So that, like, human descendants can go to other stars and stuff? Or, like, why do we need to do that? Yeah, I mean, simple question, right? So in a billion years, the Earth is going to be uninhabitable. So isn't it an easy yes to be like, hey, we should definitely spread to other solar systems within a billion years? I don't think that's an easy yes. I think pursuing that could easily squander what we have, you know? Wait, so you don't think it's an easy guess that in a billion years from now, humanity should have a backup solar system? Should? I don't... At what cost? I don't know if that'll work. Okay, why would it be? Certainly at the cost. I mean, at some point we're going to have to take a small risk of developing safe AI. There's probably going to be non-zero risk, but we'll make it small. My ideal is we pause AI now, have smart people who actually can convince the species that they know what they're doing,

which unfortunately are not around today. I think that's a realistic problem to solve in the future. And then once we have that as a low-hanging fruit in terms of something that I think should be unobjectionable, when the sun burns out, I don't think we should just be here getting burned up, right? I think we should also be somewhere else. Well, it really depends on what you mean by should. I think it would be better, all things equal, to not burn up and have somewhere else to go. I think that would be really cool. I enjoy sci-fi, thinking about that sort of thing. The idea would be cool. I'm just like the idea that we should like I just like it's really getting ahead of ourselves like you know it just might like the pursuit of that or like letting yourself like think like this is how things should be like I think could be like quite wrong like we should do the best. Why are you saying ahead of ourselves I think by 2100 one way or the other I think the sun is going to be swarmed. I feel like that's a pretty robust prediction for 2100 from my perspective. I guess I, to answer what you're saying now, I think it's a very realistic possibility that, like, the best thing we can do is just enjoy the time that we had.

Like, and that trying, like, our attempts, like, if we were, like, really worried about the sun burning out, like, would possibly, like, lead us to do things like destructively clone ourselves and then, like, the clones have no interiority or something like that. I think, like, it's fun to hope, and it's good and, like, hopeful to think about how things could be better in the future. But, like, saying it has to kind of be this technology or, like, we know, like, the path that it's going to take. And, like, it pretty much has to be that we, like, build Dyson swarms. And then in order to do that, we have to make AI. So we have to take this risk at some point. Like, I just think that's really getting ahead of ourselves. I think there's going to be, like, lots of different pathways opened. Also, you know, I kind of misspoke because I think if we pause AI, we can very possibly pause AI for 100 years or not, you know, somewhat. So I'll say, hey, by 10,000 AD, yeah, I consider it a robust prediction that the sun is going to be swarmed. That seems robust, right? Yeah, maybe you can predict that. I think that's valid. I just, like, this idea that there's, like, a teleology that, like, we do need, like, certain technologies and that we need to, like, go to other planets and stuff and that we need to be, like, working on that now.

And, like, you know, we'll be sad if we don't get that progress in our lifetimes, like, if we just die normal lives. Let me try again, because I don't think you've given me a workable alternative here, okay? Because if we pause AI today, there's a bunch of people who are going to have strong preferences for doing big projects in the next decade or century or millennium. There's a bunch of people who are going to really want to do big projects. I personally want to get the project going of backing ourselves up to other planets, and I'm okay to wait 1,000 years, right? I'm not, like, super impatient about it, but I think we should get it going. I think we should run our civilization such that we're increasing our capabilities. I don't think we should stop at human level. I also think we should fight off the grabby aliens, right? Because I actually think Robin Hanson made a very credible prediction that there's aliens that are on their way to us at like 10 to 90% the speed of light, and they'll be here in about a billion years. I actually find that very convincing. That's a likely prediction. And I think we should defend our area of the galaxy. I think we shouldn't just automatically let the grabby aliens conquer us. I think we should also be grabby. So these are the kind of preferences.

and you know that's in a billion years but even in like 10 000 years right i think there are going to be like dangerous things on earth that we have to make sure to fight and we could tie our hands behind our back and be like okay yeah you have to fight all these different issues but you have to use 2026 era technology and you know maybe that's the best solution but i suspect that the ideal future i want to see is i do want to see some kind of slow paced frontier a slower pace frontier than anthropic ones but like some sort of frontier that's evolving and i do want to see some sort of governance and the idea of coherent extrapolation is like okay let's put all these criteria that we that you want and like if you were to turn the dial up on getting those things faster what would it look like hypothetically like i think it's like a very productive intellectual exercise to be like again where where are we trying to go i don't really understand the argument in that i mean it sounds like you're saying like the vision of the future that you would like and that like the idea of cev like feels like a pathway to that future and like if

it weren't implemented then we couldn't we couldn't have it as well because we couldn't have like super intelligence and we need super intelligence to do these things to plan for like our billion year future as a species like it feels to me like cev is kind of like load-bearing in like a sci-fi vision of the future and it's like a bridge that, like, makes it okay. And I'm just saying, I don't think you can retroactively get consent to, like, radically change everything for everybody else. All right, well, we've certainly gone back and forth a lot on the theory aspect of it, and then we can go back to what I think is stronger, which is, like, the execution aspect of it. Like, should he have been very explicit that there's going to be a human-only discussion, like, before he scanned anybody's brains, right? Before Eliezer's project has gone and scanned everybody's brains and let the AI recursively stop and prove. Before that point, did he want to have a checkpoint where some democratic process happens while we're still human, right? And he gets the go-ahead from humans.

Has there been any indication of that in any of his writings? You know more than I do. Off the top of my head, I don't know if he said this is the part where all the humans get together. Okay, I'm going to go back and reread CEV. Maybe I'll find that part. Maybe I won't. But he was so early on in his project, I don't think that we can blame him for being like, you didn't convene a bunch of people. It's like, yeah, because he was barely, like, he was writing the first line of code, and he didn't even know about, like, modern AI technology. It just seemed like we had decades or centuries before this would happen. Yeah, but that's not what he's, I mean, he's thinking, like, decades, centuries, billions of years in the future about, like, what humanity has to do. He didn't think at all about, like, he actually did. So in his, I think this document was released in 2000, one of his, like, planning the singularity documents, he talks about avoiding people finding out before this is released. And, like, he even has a line in there, like, you know, it's like these naysayers will say that we're, like, ending humanity whether we own up to it or not. Like, just kind of, like, jokes like that.

I think he was thinking about, he's like, people will not like it. He didn't think, like, this would go to a vote and be, like, the will of people. just to clarify the timeline here, right? I think you might be referring to like his writings from like 1999, 2000. I can tell you personally, I'm a relative OG Yudkowskian, right? I came on the scene in 2007 and started reading him and he had made it very clear. I distinctly remember my first exposure when I was browsing through the entire Eliezer catalog in 2007, 2008. He was very clear. Everything I wrote before 2005, I was on the wrong track. I disowned that. He still archived the writings and I would go back and read in 2000. He's like, you know, we're going to build to the singularity, reaching the singularity is like a great transhumanist achievement. And at the time I also remember myself in 2005 I wasn't reading Eliezer Yudkowsky but I was reading like Marshall Brain who was also ahead of his time being a singularitarian being like you are going to discard your body like robots are going to do a lot of our tasks and I was reading it and I was being like hell yeah robot revolution let's do it right because nobody had turned, flipped the switch in my mind to be like oh we're also going to lose control of crap right it just didn't occur to me because I'm not a genius. Like it takes you know it takes some novel

thinking to have that real realization. It's not just that humanity or that person would lose control it's that they in the first place the plan was to to voice this on normies without their consent and pass their objections you're you're beefing with the 2000 version of his writing correct the one that he disowned a long long long time he didn't really change the thing he changed was saying that you couldn't align ai like or you know it was really hard to align ai but he still felt like once it's aligned once we figure that out like in our little group, like, then the plan is on. He never has said, to my knowledge, he's never taken back anything he said about, like, overriding people's consent. He also ran the rationalist community exactly the way he had planned to run the program, the singularitarian community of, like, making the last program. He, by avoiding people who, you know, might not like it and, like, not, and keeping it kind of secret and having, like, the walled garden philosophy. Like, when I read that document, I guess it's from 2000, it's one of these, like, planning the singularity documents.

I, it gave me chills because it was exactly what he continued to do with stuff like HPMOR, like, with Harry Potter and the Methods of Rationality. That was, like, the plan, and the plan originally in the singularity was to do that so that technophobes didn't find out and stop him. Okay, I think, I mean, this is my steel man, right? This is the strongest version of what you're saying, which I think has some merit, okay? I'm not, like, super swayed by it, but I'd ask somewhere to be like, hey, couldn't you extrapolate Eliezer's trajectory where he was being pretty secretive, he wasn't telling anybody what he knew about how AI was likely to work, right? So you can imagine, like, he's the Sam Altman of the day, right? Like, a slightly alternate timeline, and you can imagine he never involves the government, or he never puts it up to, like, a vote of humanity. And there is a situation that you, even if he's going toward coherent extrapolated volition, pulling the trigger to risk handing control of the AI to do coherent extrapolation vision is like a non-democratic move that you could harshly criticize him for in that hypothetical reality. And you suspect that that's where he was trying to steer the world. And you've got a beef with that prediction of what he would have done.

One difference. I have a beef with him wanting to do it and advocating for it, even if he was not capable of doing it, even if it could never have happened. It's not if it happened, like, then that would be bad it's like he was planning a coup and i think people are morally responsible for this like and fomenting it i don't know if it was even realistic that he could have done this without other people finding out or having to get involved i mean what in fact did happen is a ton of people had to be involved to make ai you know he but he was irresponsible there were actual bad things that happened from like fomenting this and then i just think like morally he's responsible for just It didn't even, like, occur to him, it seems like. It occurred to him that people would try to stop him and not want it, and he just came up with ways to, like, get around them. And I just think that's bad. That's just on its own just bad. All right, imagine you get a time machine, you find yourself in the year 2000, and instead of Eliezer, it's you. You are Eliezer. You're the one who sees how powerful AI is ultimately going to become,

and you can see the alignment problem. So you get to do what Eliezer should have done, right? you get to be that shining light, that thought leader. But no thought leader was needed is the thing. And by the way, this also makes me think I would go seek out Congressman Brad Sherman. Did you see that clip of Brad Sherman? Yeah, back in 2004. That guy is a time traveler, okay? He clearly went back to the future. What are you doing there? I'd be like, find me the time machine. He's basically the Doc Brown of the time because he was on the floor of Congress talking about how humanity is going to lose control with some of the technology we build. Likewise, it is possible for us through silicon engineering, through computer engineering, that someday we will invent machines considerably smarter than us, who may or may not regard us as their appropriate peers or masters. I know this is science fiction, but wouldn't it be wise to spend a few years and the minds of a few people a lot smarter than I am trying to figure out what we would do if science begins to offer this as an alternative for humankind?

But anyway, so you're there with him, and how do you play it instead of how it is you play it? What do you do? I mean, I think, well, I'll use an Eliezer phrase, virtue of silence. Like, there was no need. Basically, this is all dual technology. Even safety questions are dual-use research. And there are a lot more dangerous uses than there are, like, aligned, safe uses. And alignment might not even be coherent. So a separate beef I have with Eliezer is that he portrayed alignment as if it must be a solvable problem. I don't think he just realistically assessed, like, actually as a philosophical problem, like, what are the outcomes here? And if you had done that honestly you might have determined like yeah it just actually really important that we not mess with higher intelligence than us and i will not therefore raise a lot of interest in the topic of super intelligence and like i will not therefore find like a kind of research that feels acceptable and moral to do that opens the door i will not therefore found a community that is obsessed and like really like in their minds like wedded to the idea of like never dying because they

enter the singularity, and they think that they need to make this happen. Like, that would definitely not do all of this. Well, isn't that ironic that you think not messing with technology is such a good solution? Because the very solution you endorsed of telling people not to mess with technology required you to have a time machine, so you just endorsed having a time machine. In all seriousness, okay, in that timeline, I still think there would be somebody like a Sam Allman who would still figure out that, you know, he would go a little slower, right? It would be delayed by five years, but they would still be like, ooh, this intelligence is powerful, right? Like, you don't need Eliezer Kowski to figure that out. No, no. Yeah, you don't need Eliezer to figure out any of this. But he's had a big impact, and he clearly inspired, like, the people in the current field to pursue AI. We don't know if it would have been, like, five years. See, I don't believe in this, like, tech tree stuff. I mean, it's similar to evolutionary biology. Like, there are times when it's clear, like, this environment suddenly, like, you know, a bunch of things convergently evolved the same, you know. Like, clearly there was something that allowed it, and then suddenly pretty much as soon as the opportunity arose, like, with, like, genetic variation and mutations, like, it happened.

But, like, I don't think technology is like that. I think, like, this narrative has been pushed because people who want that particular future are basically just asserting that it's inevitable and it's, like, the true religion. The world is just kind of, like, listening to this story that, well, technology is actually the force that, like, moves our society forward, and it's really just a force that, like, moves through us. And this is actually very similar to what the Nazis said. If you listen to the Nuremberg trial transcripts, what they said in their defense was like, well, if it weren't if it wasn't us, like it would just it was really just the forces of history moving through us. So somebody else would have risen to make the Aryan race, you know, like give them the elbow room that they needed and expel the impurities like that just would have happened. And so, like, it's just a way to, like, reduce personal responsibility and agency for having, like, made actions toward that future. Like, I don't think that there is. You have to look at the counterfactual. If somebody told me that at the Nuremberg tribe, if I were the leader of Germany, I would have not done a Holocaust at all. Like, that's an easy substitution.

I would have gone, I can see exactly how the future could have gone differently if there was no Nazism in Germany. So in your case, when you're saying, okay, you get to be a loser, you would just keep your mouth shut, but there still would be an AI race, in my opinion. Like, that counterfactual doesn't look very different. Well, I think it's possible it would be quite different, is, you know, what I'm saying. Like, there's clearly technologies that are not invented for, like, hundreds of years when they could have been invented. You know, we know that. We don't know what factors would have been involved. Yeah, but there's a lot of mundane AI progress, you know, like Deep Dream in 2015. You know, like, I don't know. I'm not seeing more than a fact. The very most 10-year delay. Like, I'm just, you know, technologies have a way of all being like, you know, this lattice of tech that can build another. And at the end of the day, AI, as Ilya says, you know, these models want to learn. And, like, there's a lot of ways that you can stumble on this, you know, Sutton's law, right? Like, let's throw more computer. Given that Moore's law was a thing independent of AI, right? Moore's law was kicking on the door of the AI scaling law.

Yeah, there's many pathways, but you shouldn't, like, make it happen. And actually, I don't think that we can assume that it just, you know, if these guys hadn't done it, like, there would have been just right behind, like, somebody else willing to discover it. That's just not always how things are. Like, there's a lot of contingency in these situations. yeah but realistically but it totally would have right i mean don't you think there's like a more than 50 chance that it would have in 10 years i don't know okay 30 years yeah probably there would have been more progress than neural nets but like the way that things came together in this case like because of eliezer you know there's the rationalist community and there's like the ea community gets involved and there's this huge like talent farming operation and ai alignment is the highest status cause for like a decade and it's the basis of that people love the like fun cult that eliezer has made around it like they like hpmor they like the way the sequences are like being initiated into like being a wizard or something and like and that makes it so that this is like the hot place to be

and there's all there's all of this talent ready to work at places like open eye and anthropic and anthropic is just ea now like this is the project of ea now like all of the high status people went there this is like the big thing going on without those talent bases like i don't think we would be in the situation we're in today like that does make a difference and so like would there still be like progress and neuralness yes maybe someone would figure out transformers around the same time who knows but like i don't think we would have people throwing like that amount of compute that amount of talent and trying things out different training like that he greatly greatly accelerated all of this. And he could have, I think he had the intellectual insight at the time to not do that. And he noticed problems, like maybe people wouldn't agree. And he was just like, let's just step over them, make sure they don't find out. It doesn't matter. Like, we don't have to ask them because they're stupid. Like, he did a lot of bad things here. And he's never repented for that. Okay, so you've made your position pretty clear. So I'll just give my position for contrast.

So I will agree that Eliezer accelerated progress towards superintelligence. I mean, Sam Altman himself has, in this very dickish way, come out and say, like, oh, thank you, Eliezer, for accelerating AI. Or I really like shivving Eliezer because that's obviously the opposite of what Eliezer wanted to do. Sorry, it's not that funny. It's just that I described it exactly that way to a journalist the other day. And so, yeah, that's like a dagger. Yeah, yeah, exactly. Right. So I'll admit, I'll say five years. You know, I think it could easily be more than five. It could easily be less than five. It's hard for me to guess exactly. So let's call it five years. I think he accelerated AI. I also think, like, when I just zoom out, I'm like, okay, the impact of Eliezer Yudkowsky on AI. I mean, there's two impacts. Number one, he's calling, like, hey, guys, the singularity is coming. It's coming in 100 years, 50 years, five years, whatever it is. The singularity is coming. I heard that loud and clear in 2007 when I started reading him, and I think he was correct. It looks now he's obviously correct, and it's impressive to call that out that early. Kurzweil was saying it, too, so he wasn't the only one. But then the other thing Eliezer said is, and it's going to be uncontrollable, and we can't survive it,

and this is reckless, humanity's not prepared to deal with it. And he sounded the alarm in the 2000s, which to me, that's pretty impressive, even saying that it's going to be dangerous. The alarm caused the problem. You could say that, but you can still point to good effects. Like, okay, OpenAI started too early, they're a net negative to exist, okay, sure, that's alarming, but at least they had a safety team, like, from day one or from the very early days, they had like the world safety talent yeah okay look and you could argue that consequentially maybe it was net negative but look the guy has been super intellectually honest and he pointed to the safety problem very impressively early so the fact that a bunch of other people use that to be like oh we're so cool we're just going to build capabilities and do a crappy job at safety okay that's a shame and maybe like a super genius consequentialist tell the other maybe he should have known that he shouldn't do it but by the standards of a human i think he's acted with a lot of integrity and he doesn't really deserve much blame well that's why he never brought it up to the government he would tell people that if you bring it up to the government they're just

going to think it's powerful and so you shouldn't and i really wish he had like equal opportunities spoken to you know the all sectors of society and not just like tech what became tech super villains really on the basis of like his tech suggestions the thing is i do have a lot of affection for eliezer like i think i i not a lot of people don't like his writing style i love his writing style i think it's very engaging i think like he's incredibly clever and like like i was just like wow like his charisma like i feel it like it's just totally like he's really like engaging to listen to i find the way he talks like incredibly pleasing a big you know word verbal person like and i think he is very sensitive in some ways like he is very compassionate for certain things like he's very compassionate for a feeling of for like him and people like him feeling like inhibited like capable children who couldn't do everything that they wanted or they felt that they were able to do or like people with sensory issues but he's extremely and you

can anybody can tell this even if you're reading him affectionately he's extremely arrogant he has a lot of contempt for normies and for like our society and how it doesn't work and how it doesn't suit him and he just let that run wild and it's very irresponsible and I feel like he does have like the ability and the responsibility to have done better than that and some humility would have helped a lot so you know even a lot of his like most charming stuff like his his his arrogance and his like willingness to just like say some bold things is like part of what makes him so like entertaining to listen to you know he's like bold enough to to try it he goes for some ambitious things like I'm going to list all of these you know like rules for rationality but like that's also clearly clearly his flaw and I think it's we've really paid a lot for that like he brought a vision that was really exciting that was also like really really dangerous and it really inspired bad people and then he also wouldn't do anything about the bad people because he saw those as like his people

you know people in the tech world and then now here we are. Alright I think we've heard your full case against Eliezer and all this last stuff you're saying I think it does count as psychoanalyzing him, you know, like arrogance. That is a psychoanalysis. And we talked about this last time, right? I do think you go out on a limb where sometimes you say your own mental model of what's going on inside people's heads to make them act the way they do. And I claim that it's not always accurate. I think you probably have to acknowledge that sometimes you have an error rate when you attempt this, right? I mean, surely I don't. Yeah, I don't know from them. Like Eliezer won't admit to any of this. I've talked to him about this, and he just won't even engage. But yeah, that's just based on my looking in. The thing about psychoanalyzing Eliezer is that he's obviously nearly atypical. He's a legit genius, in my opinion, and so is Elon Musk. I'm not afraid to say it. These are intellectual greats. I think Eliezer's a genius. I don't think that's a stretch. And he has the ability to think many steps ahead, which I think is demonstrated again and again.

I think he thought many steps ahead in AI safety. I think he also thinks many steps ahead about his own behavior and how he conducts himself. And I actually think he has this sense of humility where he's actually very careful not to be too arrogant. And you can argue that it's backfiring and he actually is very arrogant. But from my perspective, there's a lot to be said about how his arrogance level is actually pretty calibrated because, you know, he has all these achievements that people are coming around to after decades. So you don't want to not be arrogant enough when that's the position that you're in. I mean, he's had examples where he's like, look, these people from effective altruism from these EA organizations had all this research put out. I think Ajay Akatra had this paper saying, here's why I think AI is coming in 30 years because of biological anchors. And this was like in 2021 when it was like clearly not coming in 30 years to those of us who had more of an inside view sense, like 30 years seemed like way too much, but the paper was coming out. And as it was like shooting down that paper at the time, should he have been humble or should he have been arrogant and correct? You know what I mean? And so I don't have the same sense as you that Aliazhar is too arrogant.

He's definitely too arrogant. Like, I'm not talking about, like, taking credit for real accomplishments, but that's arrogance. Like, he just, he has very sweeping statements about, like, types of people and, like, people who don't get it. And then the way he talks about just our society in general is, like, really contemptful. Like, you know, he'll say, I'm a Dothalani, so I don't understand what you do on Earth. He's saying if humans were better at cooperating, solving coordination problems, then we would live in this alien world called Dothalon, and I feel like that's kind of where I would naturally live. So he's comparing Earth to Dothalon, which, okay, yeah, I agree. This is a lot of stuff like a prophet all the time. I do feel he has mannerisms of somebody on the spectrum, which I also think I do a little bit. I can kind of identify with part of that. And I feel like I also can. And, like, I don't feel like that's, like, an asymmetric weapon between them. Right, right, right. That's true, that's true. This show is pro-spectrum.

But with Eliezer, I think, you know, when you see him get really into, like, all these deep explanations, when you, like, connect, like, look, here's evolution, here's, like, Drexler's nanotechnology, right? He loves busting out that textbook and citing the source. And he's like, look, it's all deep and interconnected. You know, that's, like, very Aspie-grade explanation. Like, he's really done the work. he's built this mental edifice and like look I mean I respect it because oftentimes that's where that's how humanity moves forward you know is people like that really doing the work no he's I mean that's where he shines you know it's like he's just writing and he's making like deep connections and even if like I don't know his hypothesis isn't correct it's always like extremely like interesting and entertaining to read and he just writes very well I again love his style but that's really not the issue basically like on domains that he gets he's great and then he's very dismissive of domains that he does not understand he acts you know like disdainful and then the kind of like phobic of like mainstream society in a lot of ways like the way he acts

towards journalists is a great example like he'll just talk about them like it is okay for him to just openly vilify journalists and say like oh they never get anything right and he means because like they don't like publish his entire paragraph that he gave them. Like he's just, he's very contemptful of like other ways of conversing. He just thinks they're inferior. Like he's very quick to just say something's inferior that he does not understand. And he thinks that about like most of our society. And he just thinks like he knows better how to do it. And I think he's wrong that, that his solution would work. And he was like catastrophically wrong about advocating for the singularity. Okay. All right. All right. last thing I want to say kind of a meta comment about this whole thing so you were on the show a year ago in the context of Eliezer's book launch and our whole central claim of that show is like hey rationalists are a circular firing squad and here's this guy Eliezer who's like the leader of the movement and now he's out with the book why can't rationalists unite around them but in this conversation right it's like you're kind of distancing yourself from Eliezer be like you

know what Eliezer's gotta atone for his sins right now okay he's like such a flawed character and it's like I don't know I don't think he's not flawed I think he's he's pretty pretty legit Well, yeah, I mean, I've been on a long journey of just completely falling out with that whole circle. A lot of it is because they won't have me. I thought that, like, I would, you know, we would bring up these topics as they came up. Or I would come to people and be like, oh, my God, guys, we could actually just pause. You can talk to the public about it. This is great. You know, this is so much easier than what we were trying to do with alignment, right? you know and like I just thought they would come with me and be excited you know because I thought we were united by a goal of protecting people from AI danger and um and then stuff with um consent like oh my god guys like I had also read this stuff about alignment and it just like I guess because it didn't feel like it was going to happen anytime soon or like I wasn't taking it like super seriously like I didn't realize like how anti-democratic it is and you know and I expected people to be like yeah we should like work that in more as a solution or something and then they just didn't. And so I think there is a time to apostatize, you know, and I have done it.

You know, there's a time when like that's the right thing to do. What I meant by that critique then was like people who clearly were like huge Eliezer fans. Maybe it was more an accusation of hypocrisy than I realized. Like they said that they, you know, wanted to get the word out about this and they wanted to like Eliezer's words are like the words they trust. And then suddenly that wasn't the case. You know, when he comes out with something that is like a plain language thing like sharing this information with people outside of their group like they just you know they don't like it my sociological view on what's happening there is like matured since then like i think like the secret knowledge about ai danger and safety like was the glue binding together the inner like the very inner group like the cults of rationalism and like EA and that having that knowledge just like shared with anybody was like, I was seen as like giving it away and they didn't like that to normies. And, and then I was seen as more normie in the first place. Cause like,

I don't look autistic. I don't know. Right. And like, and that they thought he was doing that too. And those are, those are all related to why I'm out now. Okay. So for my perspective though, I just wish you didn't opt for framing yourself as having such a big rift with Eliezer. It feels like it didn't need to happen that way, right? Because I know you blocked him. Well, I think actually a reason for more of like a rift rift is like, I'm just pissed at like the way that he does things specifically to like alienate and like damage the reputation of the cause. And like the fact that he's like blasting that on me by like taking a picture that's an homage to, like, a gangbang picture with Ayla, like, and doing other, like, crazy stuff. It feels to me like he's trying to repulse normies, when, even though he's also, he's, like, holding his nose and, like, you know, talking to, like, the government and talking to the public. Like, I, it's, that's what I get from it.

Like, I just think, like, that is definitely the effect of it. And that just blew up, like, this week. And I've been taking, like, nonstop calls with journalists. like having to distance myself and pause the IUS and like have some faction of like the AI safety world giving this message that's not tainted by this extremely cultish behavior that's very off-putting to others. That's the thing that really like tore it. I blocked Eliezer after that photo came out and I just thought like I just can't. Like already I had like kind of delicately kind of been like, ha ha, yeah, he's got his personal life, you know, to like certain people. And I felt like I had done a lot of promotion of his work, and I just felt like he betrayed me by doing that, like making it difficult for me to do AI safety work in the public eye. Now I understand the order of events, right? It always didn't sit right with you that you thought that he was trying to coup the world, right? The beef that we talked about, you know, working on. I didn't really understand that until it kind of around the same time. yeah okay but yeah so when you saw that photo hit the twitter timeline when i think it was like after

a less online or a manifest conference in berkeley they got ayla and some people together and they took a photo that was i think representative was like doing like a copy or how do you describe the photo homage yeah like an homage to that photo from this is like it starts to make me uncomfortable like telling journalists about this but like yeah there's a famous photo of like i don't really want to describe it but it's anyway it looks like that there's like the one girl all right there's the five guys standing behind and like looking down and not only was eliezer in this photo eliezer like begged people to dare him to take this photo basically like he went around saying like because actually after this is on your show right that he did that debate where he wore the silly outfit yeah yeah he after he did that he was like defending himself he's saying like for 10k i would do tons of stuff i would you know have ayla on a sofa you know sitting cross-legged and five guys you know behind in fedoras like looking at like and i guess somebody gave him ten thousand

dollars to do it and then he like but he was like excited i think it might have been awkward or he just liked that yeah nobody even paid him 10k which which makes it i don't even know if somebody gave him 10k honestly but he clearly wanted to do it it was it originated from him so here's the thing right is this is like the the next topic we were gonna segue into which is just like this whole idea of like you know the sexual preferences of the rationalist community i personally happen to not be part of that right i'm just like married and i've never been poly i i certainly don't have any publicly stated untraditional sexual preferences i'm certainly not judging like i don't think it's bad to be sexually atypical but i think the problem that you're mentioning is like okay well we're now doing public relations right so we're trying to send a good message to the public and it does seem like the public isn't really on board with like polyamory or like with flaunting sexuality and so it really triggered you when you saw that photo you're like what are you doing you're trying to like hold yourself up as like being good to communicate with the public but you're like doing the these distasteful sexual ads that aren't going to play well it's not just being poly like i don't think that that maybe that

could have been a scandal but i don't really think it would have been the same way and i would have just felt like hey like back off like you know these are his civil rights like if people were just mad simply that you know those guys were poly it's that the photos are represented they're a reference to something that ayla is also quite open about which is hosting these consensual non-consent orgies where the women are called prey and the men are called predators and they the women in an orgy, and it's very overlapping with, like, the AI safety researchers and rationalist community. But, like, wow, it seems really easy for the rules not to be followed and for someone to get hurt there, and it kind of seems like a variation. I mean, I'm sure they were taking issue with that definition of, right, like, the idea of CMC is, like, there very much is consent, but they're role-playing as if they're not giving consent, but they're still able to, like, have a say for it and stuff, right? So, I would claim, I think it's fair to say nobody's, like, actually being... Well, that's the idea. It certainly would be an easy place for misunderstandings to occur, for boundaries to be pushed.

Like, it seems hard to have enough, like, safety in place in a thing like that, especially if you can get benefits at work in AI safety from being there. Like, there's, like, pressure to, you know, not – it's weird in the sense of, like, you know, boundary crossing enough to have people who work together and live together in group houses and stuff, like, also all being, like, like really densely connected in sexual relationships. Like that makes it hard enough for people to have good boundaries and to like, you know, really if they have a boundary, actually assert it rather than feeling a lot of pressure or just not express it or not feel it. It's just like one of the most risky kinks in that way. Like if you go to like a kink fair like Folsom, they don't allow it. Like no has to mean no. That's like a thing that is often, you know, like a really hard line in kink spaces because if you're blurring, you know, like, how do you communicate what's okay? The name of this event is Red Means No, and that's, like, they have a no word. Like, they have, like, rules, which I think, you know, if, like, two people really agree to them,

like, should cover it probably. But, like, it's just a dangerous thing to be, like, popular in a community, and it's just, like, so there's a lot of risk there. And it's also just not something that has to be, like, on main. and like that just doesn't need to be public information right i mean my take on the whole thing is i think he the society is asking a lot of eliezer because i often use the analogy of we're getting werner von braun out on the front lines you know werner von braun was the german scientist who recruited by the u.s or seized by the u.s after the nazi regime fell and he was you know he developed the the v2 rockets and and then he worked on the u.s space program but his job wasn't to go fight the war out on the front lines like a soldier, right? It's like he was very much in the back rooms, like, doing the calculations. That's how I see the role of Eliezer, is, like, he illuminated the idea, right? He was the intellectual pioneer. He's done us such a great service with all the theories that he developed, all the principles that he's developed and popularized. But now to ask him, like, okay, you're getting up on TV, like, you're going to be the one to communicate. Okay, but you've got atypical mannerisms, atypical preferences, right?

So, and, like, he can, I don't think he's that bad at it, right? He's not, like, horrible. He's, like okay in some interviews right sometimes he looks and sounds better than others but we're asking so much of him and it's like no judgment but as you point out like when he's like proactively tweeting it it's like he's not really accepting his role as this kind of mainstream communicator well i don't know it's like even more than that it's like don't you care what's probably gonna happen because of that like i just and i i don't think that he has to stop like living his lifestyle You know, it's just like his image being that like, don't you care? Like, what's probably going to happen because of you sharing this? Like, I know that like, you know, Democratic staffers are on Blue Sky where some of this stuff was recently shared. And like, they're getting worried like about like, you know, the fact that Bernie has this connection to Eliezer and like he talked to Eliezer and had a video with him and that that's like seems related to his bill. It's not just like kink shaming. Like, I think, like, there is a connection between the, like, the sexual practices of that community and AI safety.

Like, they believe that, like, somebody who, like, a normie who isn't rational enough, like, doesn't have the right, like, neurotype and type of insight, just they can't handle either. And so and like John Wentworth, who's in that photo, he confirms this that like to him. Yes, he sees it as a qualification for AI alignment work that somebody would go to these because it shows that you can like see through when society is being like mainstream society is being like stupid and triggered is what he said. and there is a belief about polyamory as well that like you know just yeah like maybe normal people can't handle it but like we're sort of like you know we're we are more rational and we like optimize our relationships and like we understand that like this is the way things can be or we don't think that like this stuff is weird like we lean into it and we like discover our debauchery that it kind of is all a similar attitude and it is like the central identity of that cult and so I really don't think it's wrong for people to I don't think people are getting the wrong idea from that image. Like when they, they look at that or when they see things that Aayla says about you know like their activities Like I don I think there is like a belief about themselves that like they are better and there like a

they're separate and kind of wanting to alienate people who can't get on their level. And that is really harmful to, you know, trying to get people to understand this message and be on board with it. And I'm only grateful that it seems like we're finally far enough that like the message won't entirely sink because of how Eliezer talks about it. Like, it does have enough of its own life, and people have seen enough of the issues, like, with the hugging face incident for themselves. And I have no intention of taking any damage for them. You know, like, we don't, Posse IUS does not represent that kind of mentality. Like, we are for everyone. Yeah, the John Wentworth case is interesting to touch on here because there's a few conflicting things about it in my analysis. So this happened in the last week. I think the timeline of events is first, like, over the last couple years. So John Wentworth is, like, an independent AI researcher. He's got high karma on less wrong. Personally, I've never gone super deep into the John Wentworth alignment research. It's been a little hard for me to get into the exact substance of his research.

I know he's talked about, like, natural abstractions, as I recall correctly. It seemed like there might be some meat there, but I've heard people that I respect being, like, I feel like he's off base, but I've heard other people I respect being, like, well, it's great that he's trying this direction. Okay, so my net impression of him is that he's adding value. You know, we need more people like him going off in independent research directions. Okay, that's my general impression. There were also some posts that I wasn't personally familiar with where he's being very candid on less wrong. Like, this is my perception of, like, courtship and dating. My perception is that females, you know, they love CNC. They love that idea of, like, feeling like they're not giving consent even if they're actually safe. But they love that aspect of it. Which, look, I think, you know, I've read some evolutionary psychology. I mean, you've studied evolutionary psychology. I think there's something to it, right? That, like, some subset of women in some circumstances where they feel like there's some danger to the sex, it can be a turn-on. Is that fair to say? Don't get me on the record about that. I mean, look, the fact that females show up to the CNC party, right? I'm going to go out on a limb and say there are some females who aren't being abused at those parties,

and they're getting turned on. That's certainly what some of them say, and I think that's no reason not to believe them. I do think there's a reason to question like if the setup is going to like eventually involve people who then like that environment pressures them against what they really want or their best interests. But yeah, there are people. Ayla is one of them who, you know, just just says over a very long time that this is a strong interest of hers. And it's it she does it safely. And I have every reason to believe that that's how she feels about it. okay so i i skimmed some of john wentworth's less wrong posts that are now stirring up controversies and he's basically saying like oh let me give you like an infield report kind of like pick up artist language like here's here's my report of like some sexual experiences i've had like everybody else is welcome to share their reports we can like compare notes so there's like these very candid sexual reports and from his perspective he's like yeah cnc it seems like everybody who's like a good thinker sees the value of cnc and i've had so many experiences with it but then some commenters are

like, well, don't you think you just have a biased sample where, like, you go to these parties, like, with these conventions where people just are into this, but maybe the average person isn't into this, right? So, like, I certainly respect that, like, he's found a niche where it does seem like people in that niche are happy to engage with him, right? And he's found success in his niche. Like, I don't want to judge him for that. Posting the account on Les Strong, if he's going to be a public figure, you know, he could draw some arrows from that because he's becoming an easy target because people like to make fun of people who are being, like, you know, sexually going out on a limb, like, not doing the most standard thing. But this all came to a head in this last week because, you know, survival and flourishing fund, right? They've got the money. They're funded by Jan Tollen and Dustin Moskowitz, and they get to lay down some standards. So we were just talking earlier in this conversation, like they made this big thing, like they don't like advocacy. They don't like, quote unquote, violence, even when it's just like kind of like a verbal hint of disagreeing with somebody, like the definition of violence is unclear. But basically, they got to lay down on these rules of who gets their money, which from my perspective, it's like, look, it's just rich guys. Like, it's their money, right? It's not the government's money. Like, they can make up whatever rules they want.

and people can apply to get their money or not apply, right? So, like, I think they have every right to make up these rules, even if they're doing things I don't agree with or I don't understand. So, anyway, in the case of John Wentworth, they'd already given him a grant that I think was hundreds of thousands of dollars, maybe $500,000, something like that. Yeah. And it was for his AI research, which, like I said, I think he has a pretty good reputation for his AI research. So they pulled his grant, and now there's some controversy of people being like, really, like, you know, he's a private person. He's not going on interviews, you know, going on Fox News, He's telling people, you're talking to Congress, like, he's just sitting there doing research. So now we're not letting AI researchers have, like, a sex life on the side, right? So there's some controversy whether that was very low. But what he said was that it was mixed with his professional opinion. And, like, that is just sexual harassment. Like, if this is kind of a workplace by extension. So, like, the line which makes sense is that this is completely separate. And it's not even mostly composed of this AI safety community. It's just people being drawn to this very narrow kink.

And so it should be people from kind of like all over the Berkeley community. But then John Wentworth goes and posts this comment and says, actually, it's 50-50 rationalists, and I respect people more as AI alignment researchers if they're there. And so, like that. Right, I agree. He really put his foot in the mouth of the cheek because you've got to be careful to be like, look, this is something I do for fun. I also do AI research. But he did kind of put his foot in his mouth and be like, well, I can't help respecting somebody more if they're part of this community. Like, this is the community. Like, if there's that kind of opportunity that's, like, allowed to advance your career, like, that's really not okay. Like, that is, like, sexual harassment at work. Right. So I have a relatively normal person's perspective on all this. I got a message from somebody. She'll remain anonymous, but I'll mention it as a female. It's a female perspective on all this, who's not, like, a super insider of the community, is relatively new to the whole AI stuff. And this is what she wrote. I thought it was an interesting, potentially representative opinion. She says,

If ethical and kept private, then whatever. I doubt it's all ethical from my point of view, though. That the other side has this particular one up, that they can point to these as a way to get people to question certain AI safety, leaderships, credibility, morals, and authority is so predictable and frustrating. The future of humanity is greater than publicly advertised orgies. Pretty simple, and I am very glad that Holly posted about this on X. Let other somewhat less important people be the public faces for the acceptance of these kinks if they truly feel that is where society should head. I take this kind of thing very seriously. in a high regard so as to ensure the dignity and respect for all is not in question and i flat out do not want to work under leadership that doesn't have their shit together in this way acknowledging that no one is perfect if you can tell me what orbs if any to avoid please let me know yeah so so that that is i think a reaction that a lot of people are going to have maybe i don't agree with like a hundred percent but like it's just like look at some point i mean pr is what we're doing now right so so maybe a survival and flourishing fund was right on this particular decision of like You know, sorry, guys, but we're really cleaning up our act here in terms of PR.

I think so. They can't allow people to just be like, yeah, like, I'm funded by SFF to do my work, and I preferentially hire people who I meet at this orgy. Like, because, because. Right, right. And to be fair, he wasn't hiring. He wasn't hiring. But okay, okay. All right. All right. So we beat the dead horse. Well, he said he wasn't hiring whatever. Like, but just, you know, he works with them. Like, he gives them a leg up. Like, I once interviewed with John to be his mentee. I think probably everything would have been fine. He seemed like perfectly nice. I was coming in probably with more expertise on like the, it was like genomics angle on the natural distractions thing. And so I felt like we were peers, but like, I mean, like if this, if these had existed at that time and he was just like a cool guy, he was like very open about his kinks and stuff. And he kind of like, I don't know, like other people like possibly could be favored. Like that could just be pressure that you don't need to, like, participate in, like, very intense, like, traumatic, probably for most people, sexual activity.

Right. Okay, so I noticed the pattern the same way that Survival and Flourishing Fund has gone this route where they're trying to be, like, very clean, but you're also trying to be really clean and, you know, unassailable in your own way, which is, like, I remember you were very strict about kicking out the soppy-eyed people just because they had, like, a whiff of violence, and then Sam Kirchner eventually, like, attacked his co-colleague or whatever as part of his own organization. so like I think you made a good call to just like draw a clear line like nope these people have like a small whiff of violence I'm going to disassociate with them it actually was even less than that because they wanted to do illegal stuff like they wanted to do civil disobedience but they I just don't break the law in Paseous right exactly they wanted to do civil right it wasn't it wasn't clear to me that they were violent at the time but they were like very passionate right and so it's like look you guys are passionate just and you say you're not going to break the law but like it It just feels like your passion might get the better. I remember that was, like, my sense at the time. I mean, like, they were planning to break the law as far as, like, chaining themselves to the door of opening up. Right, like civil disobedience. Yeah, like a nonviolent way of breaking the law. Yeah, so it wasn't, I didn't think they were violent at that time, but, yeah, but then, of course.

Right, you didn't even want to do the civil disobedience, basically, right? And I just, yeah, I think the only place to get off before he had that whatever happened, like, he snapped, like, was him wanting to break the law. Like, and I'm glad that I did. And I think SFF did the right thing to remove this funding from John Wentworth. And I'm also pretty tough with volunteers. Like, you know, people, I get a lot of blowback for this because people are like, oh, you're too. But, like, I don't, any kind of, like, liability that they create that's, like, against our rules. Like, I mean, we really can't afford that. And, like, I kick a lot of people out. And then Posi-i Global had the Molotov cocktail guy, like, in their Discord. I think they handled it, given that they admitted him in the first place, I think they handled it right. Like, they gave him a moderation warning. He didn't say anything violent. But he wouldn't have been in my good word. Just to clarify for the viewers, so the guy who ended up throwing the Molotov cocktail at Sam Altman's house before that, I don't know if it was a couple weeks before or whatever, he entered the posi-eye discord. Hey, posi-eye global. Right, posi-eye global. And he said something to indicate that he might be interested in, like, escalating the violence or something. And you're saying they were actually good about being like,

hey, you know, that's not what we're here to do. They were, but if he ever tried to get into our Discord, he was bounced because the name Butlerian Jihad would have had him removed. Like, because it had, like, a Jihad, like, violent. And I think, like, that works, you know? And now we don't even have a public Discord just because it's just not worth it. You become, you're considered responsible for whatever people say, even though it's just, like, social media. And that's what happened to Pauzei Global. And they, of course, blew back on us, me having the same name. But yeah, so like I'm super strict about that. And I feel that it has been vindicated. Like every big event that has occurred has vindicated those decisions. So like I think that's right. It is interesting, right? Because like I said, at the same time as some people consider you not good enough for their standards, you're so strict about applying your own standards and turning it back on a bunch of other people. And I think your standards are really good. You've taught me a lot on that front. So you kicked out all the people who are going to break the law. and you've got these signature forms, which at the time when you first rolled these out,

it still seemed potentially premature to me. Like, wow, you're going hardcore. But now in retrospect, it's like, wow, good call. You know, you really braced yourself for all the potential, you know, upswell and like accusations that are flying around. Like you definitely anticipated that. And another thing right now, another thing you're doing is you're saying, look, even Eliezer Yudkowsky and all the sex stuff, like you've distanced yourself on that like pretty nicely from my perspective, right? You're like, yep, none of this stuff is part of Pauze Ai. like we don't do any public sex of any kind. Yes, to say the least, yeah. Yeah, I'm going to put my foot on the ground, okay? Doom Debate does not have any kind of relationship or opinion on sexual events. Okay, I can join you on that. Another thing that comes to mind when you're doing such a good job enforcing standards in your organization is your take on humor and like sarcasm, right? So you were also on this very early, this idea of Poe's Law related to Poe's Law, where somebody will post something and in order to understand it, you have to assume it's sarcastic and that they mean the opposite.

But Poe's Law is saying, okay, there's always going to be some fraction of people who just read it and they don't do the inversion. Like they just read it and then they take it seriously. And it's to the point where Poe's Law, it's now common knowledge, to the point where just nobody should be tweeting anything where the serious interpretation is like horrible. Like you just can't have a horrible serious interpretation unless, you know, look, there's some accounts on Twitter that they're like anonymous accounts, or there's people who really don't have any, they don't represent any organization. They're willing to take like the full, you know, they're uncancellable. They're willing to take the full consequences, even if a bunch of people hate them or if their tweet has a bad influence. There's certain people who can take that risk. But even me, I'm like relatively uncancellable. You know, Doom Debates, we're just viewer funded right now, right? So I don't really have a boss. You know, you guys are my boss, the viewers, doomdebates.com slash Jenny. I'm relatively uncancellable. And even I have come around to following this law like you'll never see me tweet anything of consequence we're taking the tweet literally and not seeing any humor in it even if you want to do that it'll still have a good meeting that's that's the standard that i learned this year but you were actually very

early to that right you've been doing it for years yeah i guess what else is there to add that was pretty good summary i i think even further that like i just see like the way that i am tempted to use sarcasm now that i if i use sarcasm i mark it i say and a lot of people found that like unbearably cringe when I started doing that but like I mark it if there's no other way I can think to express this except saying something sarcastically but usually I try to just express it a different way and pretty much every time I'm tempted to use sarcasm it's because I don't want to fully think through the issue I just want to like dash off a comment or I want to like discharge some feeling about it but without like actually if I take the time to state what I mean seriously like it changes the meaning like for me a lot or it makes it feel like too aggressive and it's like why is this too aggressive like either I don't want to say it or I'm afraid like I'm kind of contributing to this like ambient like frog boiling of like haha yeah we know we know all these bad things but haha like we're allowed to just joke about them and so we don't really have

to take them seriously and like if you joke about it and that kind of releases the tension that I had from feeling this water heat up and so when we'll all just help each other so like when when like Sam Altman says something like horrible and chilling, like, and people are like, you know, the reply is like, you know, no problem here, huh? You know, like people's like QT on that is like, what are you saying? Like, you're just afraid to say like, I condemn this. This is actually terrible. You know, like this is, this is a horrible, like what that would feel, what? Too sad, too serious. You're afraid that your friends who work at opening eye would see it. And like, they would feel like you're putting them on blast. It's, it pretty much, it contributes to like a fake, you know, reality that everybody's agreeing to live in like when we lazily use like jokes and sarcasm to paper over like the real intent or to like release the tension i'm still in the business of trying to be funny sometimes don't get me wrong i like a good funny tweet but i'm adding a constraint you know and look people say constraints actually help with humor okay so it's just an additional constraint that i'm never going to tweet something that if you just take it literally it's horrible

like it has to at least be okay if you take it literally yeah or like i just i want it to be true, even if people don't understand, like, a humorous aspect of it. And if it's not, I think people need to, like, take accountability for that. Like, I think, like, just sarcasm without marking it is just a lie. And I'll forget sometimes. I'll say, I'll make a sarcastic comment without marking it because it's just, you know, natural. And, you know, if you're speaking, you mark it with the tone. And I would just do that. And I, it's amazing how often I immediately get caught by like some poor, like incredibly aspy person is like, well, why would you say this? You know, I guess they're used to me telling the truth, you know, and earnestly. And so I'm like, you're right. This is unmarked sarcasm. Like, I did not mean that, you know, like that way. Yeah, yeah, yeah. So a recent theme of your tweeting has been continuing on something that we brought up in the last show we did. You used to call it Holly's Basilisk, right? This idea that like, hey, we're going to retroactively try AI company employees as criminals.

same as how they did in the Nuremberg trial, even though during the war they couldn't do the trial. They're going to retroactively do the trial. And you're still of the mind that if somebody's working at an AI company today, they should eventually go to jail for what they're doing today, even if there's no case against them. There's no indictment or arrest of them yet. But you still think that they are setting themselves up to go to jail, correct? I think so. I mean, I think if you think about a scenario, you know, where we're not all dead, you know, but we have solved this problem. Like, I think it'll be because we recognize how incredibly serious the risk was. And at that point, it's just going to become indefensible that people were saying that they believe there was a, you know, 10 to however much, like 80, 90 percent chance of doom. But they were working on this technology anyway. And people are writing their own prosecution for this entire, you know, last, like, since 2016, you know, working at OpenAI. So if there's no repentance, then I think, like, that case is going to be pretty obvious that they should at least have, like, some way of repaying society for the risk they put them on.

We just have to take seriously, like, when just because you didn't think you would go to jail because it wasn't a crime then doesn't mean that you can't, that you can just get away with stuff, like, putting people at risk like that. And I think the risk, increasingly people understand how much what these employees saw themselves as doing was, like, a coup. And that is also, you know, that's already a crime, but, like, they also will see that morally blameworthy. right okay well you know i went back and forth on the jail thing with you in alvaro cuba a few weeks ago and i was saying like look you got to do this in order where first you got to make the law and announce that you're going to start prosecuting and then people have a chance to stop or quit and then if they don't then you gotta enforce it but then your guys's argument was like well couldn't you already interpret present law of putting people in danger and say that they're actually already committing a present crime and i'm like yeah i guess you could so now we're just arguing over the gray area, and I'm not fully convinced. I would still love to see them be very clear about the law,

just being like, hey, what AI companies are doing is illegal. Like, we're being very explicit. We're not relying on an interpretation of an old law. I feel like that would be the ideal. I mean, yeah, but, like, it's also ideal for the AI companies because they can argue. Now there's this whole process that has to be initiated that takes, on their timescales, like, forever. You know, there's so much they can do in the amount of time it takes like legislation to be made by Congress. Like we're having a hard enough time, like having legislation to put a stop to this, you know, huge impending disaster already. And the case law, just like interpreting things in a different way that requires a case to be brought. And so like there needs to be some willingness to prosecute or it's like private civil law, you know, cases that are beginning to be brought, but that takes a long time. And I think like the, the truth is that like the computer fraud and abuse act, like already covers what happened in the hugging face incident because a computer was authorized without permission. Like that's the standard in that law.

So I think there's a strong and I think when this goes to trial, this that is the interpretation that the judge should have. But because this is a new a new there's a new argument here that's never been tried before, that the agent did it without, you know, opening eye telling them to. so who's responsible, that'll have to get, like, litigated. I think probably that will be decided as opening and being responsible. But there's, you know, a lot of corruption in this process, too, and if there's, like, a ton of lobbying and pressure at, like, all levels of government, you know, for that decision to go another way, like, that would be highly influenceable. So this is why I want to create a strong public expectation and, like, anchor people in understanding, like, that I think the spirit of this law has already been substantially violated. And, like, the details will have to be worked out or, like, possibly eventually if we have enough time, you know, a new law that's more clear, like, applying specifically to AI agents, like, there'll be time for that.

But if we don't get lawsuits like this, then, like, we might not get them to stop in time. So I think it's really worth encouraging. And I think certainly like the morally they're culpable, no matter what, you know, like the details about the law and this like completely unforeseen situation, you know, are they matter. But like, you know, we know morally that they are culpable for this happening. okay yeah i want to highlight this one recent tweet i think this gives people the gist of what your recent tweeting has been looking like you know a lot of focus on jail which you know fair enough fair enough i mean it's like you're always out in front of the conversation and this idea that like look whether it's today or whether it's soon this idea that what you guys are doing shouldn't be legal right and it won't be hopefully if we are to survive it won't be so get the idea of jail on your minds okay i think i can get behind that i think i support what you're doing here all right so this is a tweet from september 12th this guy tomec korbach who worked in open ai at the time i don't want to spoil what happened but he says he says i'm quite unhappy with much of what open ai does i am very happy that i'm allowed to say i'm quite unhappy with much of

what open ai does but then you reply to him and you're like don't fucking work somewhere you're quote-unquote quite unhappy with ethically that's on you and you and you should go to jail for what you, not OpenAI, have done. And then later, a couple weeks later, he's one of the three people who got fired by OpenAI for allegedly leaking to an AI safety organization. That's all we know. OpenAI hasn't made a statement about it. They fired him on, I think, a Saturday, maybe a Friday, you know, like a Saturday night massacre type of thing. And then I was joking around like, oh, you know, it's tough for me to improve AI companies from the inside because of this whole issue that they're allowed to fire me. That was my joke about people who join AI companies who want to improve them from the inside so and then you quote tweeted the whole exchange after they got fired and you said this guy is lucky that opening i fired him it'll probably reduce his sentence so you're always bringing it around to jail okay you're on that beat like i said i respect that i think that's a reasonable choice but you're certainly committed to this angle all right let's let's get close to the wrap-up here because we've covered a lot of ground and i would love to keep

doing this every month any final thoughts final thoughts man everything just the pace of this is crazy i feel like every day every week i'm like wow that was a real sprint and then we just enter another sprint it's wild and especially i can't believe how much like damage control i feel like i'm doing for i really have gotten punished for my time with the rationalists and eas like i just i'm having to like like fight off all of their just like baggage and associations constantly including here but you know i do i'm a fan of and i tell my volunteers like but where you find the resistance like on the way to pausing ai like that's where the work is like you're not you know if you have the idea you're like hung up in the wrong place like like a lot of people will you know be trying to get meetings with their congress people and they're kind of getting the run around from the office and i tell them like this is where the work is then because like when you persistently want to make a meeting like that lets the office know and then you know being in the meeting is another way to let the office know and then like they're having a longer

relationship where you know the more stuff is another way to let the office know that people want to pause the eye but if this is where you're meeting the resistance like this is where it is right now where i'm meeting the resistance is just like the you just wild like insider dynamics of that community and like the world is really catching on it seems like since the coxswain tweet like just happened to be like the straw that broke the camel's back like now people are taking x-risk more seriously and so it's like making them realize like oh my god like you guys were in charge of this and it's yeah so it's it's been a lot of it's great to have people like really knowing and understanding and they're in they're engaging on a different level which is amazing but it's like also it just kind of exposes like all of this territory that a lot of it didn't seem so so bad or so risky but when you like reach a new level of of fame it just like kind of is reborn so yeah kind of a conclusion from like the last like two months has been like i used to sort of see like the programming of pause aius is pretty separate from having to do all this like

insider stuff and insider influence even just addressing the culture of ea and rationality like has become like a part of my job and it's bizarre it's just like part of the history of like the idea of the ai safety world and it's part of like understanding you know who you should trust and Why? So I don't know. So gossip has become a job. Right. All right, let's wrap on this. I think this quote is relevant right now, given the rapid progress. This is falsely attributed to Mahatma Gandhi, but it's actually a variation of a phrase from a labor organizer, Nicholas Klein. Anyway, the quote is, First they ignore you, then they laugh at you, then they fight you, then you win. Well, they're definitely fighting us now, right? They've gotten to the point of fighting. Yeah. All right, Holly Elmore, thanks so much for coming back to Dune Debates. Hope to see you again next month. Keep up the good fight. Thanks for having me.