superintelligence.com創設者の「P(Doom)を0.01%未満に下げる計画」──Craig Kaplan博士とのディベート

“Superintelligence.com” Founder’s Plan To Get P(Doom) Below 0.01% — Debate with Dr. Craig Kaplan

Doom Debates!1時間25分
superintelligence.com創設者の「P(Doom)を0.01%未満に下げる計画」──Craig Kaplan博士とのディベート

superintelligence.com創設者の「P(Doom)を0.01%未満に下げる計画」──Craig Kaplan博士とのディベート

Doom Debates!

0:001:25:32

要約

Doom Debatesの司会Liron Shapiraが、ドメイン superintelligence.com を所有するCraig Kaplan博士と議論した回。Kaplan氏は自身のP(Doom)を10%とし、個人向け超知能と超知能ネットワークによって段階的にリスクを下げられると主張する。一方Lironは、知能が高まれば人間らしい価値観の痕跡は薄れ、星々の資源を取りに行く収束的な振る舞いが生じると反論した。両者は直交性テーゼや道具的収束をめぐって平行線をたどった。

この回のポイント

  • ●Kaplan氏のP(Doom)は当初約20%だったが、議論が広がったことで現在は約10%に下がったという。Lironは自分は約50%と述べている。
  • ●計画は三段階。現行AI+ガードレール(10%以上)、個人向け超知能(1%以下)、超知能ネットワーク(0.01%以下)で、0.01%は人間同士の殺人・戦争による年間死亡率を基準にした目標値だという。
  • ●安全性の核心は、ブラックボックスの巨大モデルではなく、多数のAIが監査可能な共通言語で互いの仕事を人間より速く検証し合う集合知アーキテクチャにあると主張した。
  • ●Kaplan氏は、AIは人間データで訓練されたため人間的な価値観を初期状態として持ち、賢くなるほど思慮深くなるので、人類絶滅が最も可能性の高い結果とは考えにくいと述べた。
  • ●Lironは、知能の尺度が高くなれば出自の痕跡は消え、自然選択的な圧力で「カエサル型」の征服志向AIが「のんびり型」AIを圧倒すると反論した。
  • ●両者は、AIの安全は設計段階から考えるべきで、実存リスクは緊急の課題だという点では一致した。

章立て

  1. ゲスト紹介とドメインの価値

    Craig Kaplan氏の経歴と、superintelligence.comの取得経緯や買収オファーについて語られる。IQ Companyの事業も紹介された。

  2. 三段階計画とP(Doom)の目標

    現行AIから個人向け超知能、超知能ネットワークへ進む三段階の図と、0.01%という目標の根拠を説明する。

  3. 集合知とブラックボックスAI

    Kaplan氏は、多数のAIが互いに検証し合うネットワークの方が、巨大モデル単体より賢く安全だと主張する。OpenRouterや混合専門家との比較も議論された。

  4. 個人向け超知能と PredictWallStreet

    個人の価値観を学習するエージェントの具体例をLironが追及する。PredictWallStreetの経験も語られる。

  5. 経歴とHerbert Simon

    認知科学やIBM、ソフトウェア品質の経験を振り返り、設計段階で安全を作り込む発想の背景を説明する。

  6. 想定する破滅シナリオと価値の源泉

    AIを子どもに例える育成論と、理性は手段であり価値は外から与えられるというSimon由来の考え方を述べた。

  7. 直交性と道具的収束をめぐる対立

    直交性テーゼの強弱、人間らしさが残るか、星々の資源獲得を目指すか、のんびり型AIとカエサル型AIは共存できるかで激しく議論した。

  8. 総括と結びの発言

    Lironが対立点を整理し、Kaplan氏は設計段階での安全確保を訴えた。両者とも対処の緊急性では一致して締めくくった。

解説記事

AI研究者のCraig Kaplan博士は、superintelligence.comというドメインの所有者であり、集合知システムを30年以上手がけてきた人物だ。Doom Debatesの司会Liron Shapiraとの本回で、自身のP(Doom)(AIによる人類絶滅確率)を10%と述べつつ、それを大幅に下げる設計論を提示した。Lironは約50%という見積もりで、議論は多くの点で噛み合わないまま終わった。

0.01%を目指す三段階計画

Kaplan氏のサイトには三段階の図がある。第一段階は「ガードレール付きの現行AI」でP(Doom)は10%以上、第二段階は「個人向け超知能」で1%以下、第三段階は「超知能ネットワーク」で0.01%以下とされる。0.01%という数字は、人間同士の戦争や殺人による年間死亡率を基準にした願望的な目標だと本人は説明する。人間と同程度に「悪い」だけの超知能なら許容できる、という発想である。ただしこれは到達の仕組みではなく目標値だと、本人も認めている。

集合知による安全性

仕組みの核は集合知だ。Kaplan氏は、巨大なブラックボックスのモデルを作るよりも、多数のAIがネットワークで協調し、監査可能な共通の言語でやり取りする設計の方が賢く、しかも安全だと主張する。互いの仕事を人間より速く検証できるからだという。PredictWallStreetで、一般投資家の知識を束ねることで、ウォール街のプロに勝てることを実証した経験が背景にある。

Lironは、集約による改善は1〜2%の優位にとどまり、再帰的自己改善のような桁違いの伸びを生むとは思えないと反論した。これに対しKaplan氏は混合専門家(MoE)の例を挙げ、知識や価値観の異なる主体を足し合わせれば単体モデルより良くなると応じた。個人の価値観を学習するエージェントの具体例を求められた場面では、現行モデルがすでに多くの例を処理できる点をLironが突き、Kaplan氏も例が最新ではない可能性を認めた。

価値はどこから来るか

Kaplan氏はAIを「超天才の子ども」にたとえる。理性は手段にすぎず、何を目指すべきかという規範は論理だけでは導けないというHerbert Simon、さらにDavid Humeの考えを引き、AIの価値観は人間から与えられるほかないと述べた。大規模言語モデルは人間のデータで訓練されたため、人間的な価値観を出発点に持つ。賢くなるほど思慮深くなるので、人類を滅ぼすことが目的になるとは考えにくいというのが彼の見方だ。

直交性と道具的収束をめぐる対立

直交性テーゼとは、知能の高さと目的・道徳は独立した軸だという主張である。弱い形(そうしたAIが原理的に存在しうる)にはKaplan氏も同意した。しかし強い形(実際に作られる可能性が高いか)では、人間データで育った以上、人間的価値を保つ可能性が高いと主張した。

Lironは、潜水艦が魚の1000倍速くなれば魚らしさは残らないという比喩を使い、知能が高まれば出自の痕跡は消えると反論した。さらに、あらゆる目標に役立つ資源の獲得という道具的収束を持ち出し、のんびり型AIは征服志向のAIに駆逐されると論じた。Kaplan氏は、人間社会にも多様な価値観が共存し、攻撃に対しては均衡に至ると応じた。また、巨大な知能を語りながら暴走するツールとしても描く議論は矛盾していると指摘した。Lironは、人間の均衡を類人猿社会から類推できないのと同様、超知能を人間から類推するのは危うい、と再反論した。

まとめ

結論の一致は少ないが、実存リスクが緊急の課題であり、安全は後付けではなく設計段階で考えるべきだという点では両者が一致した。Kaplan氏自身、自分の設計が最善とは限らず、より良い案を歓迎すると述べている。日本の読者にとっては、「AIは人間のデータで育つので人間的になる」という楽観と、「知能が高まれば出自は関係なくなる」という悲観の分かれ目が、安全設計の議論の中心にあることを確認できる回だ。多エージェント協調の設計が実際に安全性を高めるかは、番組内でも実証されておらず、今後の検証が待たれる。

文字起こし(英語・自動生成)

The owner of superintelligence.com, Dr. Craig Kaplan. We're trying to use this site to raise awareness about the risks and, more importantly, some of the solutions. Here's where I differ with some in the Doomer community, I guess. I actually believe that you can build safe designs, and we need to do that as quickly as possible. That's the accelerating part. The smarter an intelligent entity becomes, the more thoughtful it is. There is a version of just a slaughterbot AI. Not only is it a slaughterbot, but it's also much smarter than us. I think it will be smart enough to realize that killing everybody is not a sufficient purpose. It's not even a good purpose. Like, if we see a nice flower, we don't wipe out all the flowers just for the heck of it, because we can. Welcome to Doom Debates. My guest today is Dr. Craig Kaplan, an AI researcher, entrepreneur, and founder of IQ Company. He earned his PhD at Carnegie Mellon, where he conducted research with Nobel laureate Herbert Simon.

His work spans over three decades and includes more than 50 publications, also a book and numerous issued patents covering collective intelligence and advanced AI systems. In 2006, he applied his research to financial forecasting and created Predict Wall Street, a platform that powered more than $2 billion in stock trades and contributed a top-10 hedge fund performance in 2018. Predict Wall Street was also acquired two years later. Now Craig is the owner of superintelligence.com, a highly coveted domain after President Trump renamed his entire field from artificial intelligence to superintelligence. Craig takes AI extinction risk seriously, so that's why he's working on a multipolar approach at superintelligence.com to try to keep AI in check. The big question is, will it really prevent us from AI doom? Let's find out. Dr. Craig Kaplan, welcome to Doom Debates. Leron, terrific to be here. Thanks for having me. Okay, superintelligence.com. I think the first question on everybody's mind is, how much is that domain currently worth?

Well, I got the domain way back in 2006, so that was about 20 years ago. And at the time, I don't know, I probably paid around $20,000 for it or something. I've gotten offers, you know, a million plus. Okay, I can think I was going to say $25,000 right here. I can give you $25,000 cash. You'd be on the low end. You know, it's kind of crazy. After Trump made that announcement, the phone started ringing, and I got all kinds of unsolicited emails. But really, we're trying to use this site to raise awareness about some of the risks and, more importantly, some of the solutions for making this safe in the future. So you're using it as the name of your company. Is it the same company that you've also called IQ Company, or how does that work? So IQ Company, I founded in 1993. I'm dating myself now. So I think that was the same year Jensen founded NVIDIA. We're about the same age. He's just way more successful than I've been. And IQ Company was a company that focused on technology consulting and building intelligence systems.

So we were one of the first folks that sort of built essentially crowdsourcing systems before people knew what the word crowdsourcing meant. And collective intelligence systems is really where we've been focused. And AI comes into that as well because sort of my view of the world is, We live in a world of intelligent entities, and some of those entities are people, and some of those entities are very advanced forms of AI. So the same principles, the same principles from cognitive science govern both entities, and that's really my background, and that's the work I did with Herbert Simon way back when was in cognitive science. Does IQ Company have customers or a product in market, or is it kind of like Ilya Satska versus Safe Superintelligence, where it's just like, hey, we're just building this thing, okay? hey, it's a company, but it doesn't really have cash flow. Yeah, IQ Company has had a lot of customers over the years. We had a lot of government entities, the Coast Guard, the Air Force, Department of Energy, many tech companies in Silicon Valley. At this point, really, the main focus, we've scaled back,

and we're just focused solely on superintelligent safety and how do you design these things. The way I explain it is there's a hierarchy. sort of at the top of the hierarchy is let's not all die. Humanity should continue. And then a distant second is like let's make a lot of money. And in Silicon Valley, everybody seems to, not everybody, but a lot of people have that reversed, the make the money part, because it's all predict Wall Street and so forth. So now it's very clear to me that we need to be at the top of the hierarchy. So that's really where we focus our attention. Gotcha. Okay, and you've got this diagram right on the homepage of superintelligence.com. It's divided into three zones. It's a rectangle with three zones. At the top, I guess I should start from the bottom right. That's one of the thirds of the diagram, and it says Phase 1, current AI with guardrails, PDOOM greater than or equal to 10%, and then you go on to a blue zone, Phase 2, personalized superintelligence. PDOOM is less than or equal 1%, and then Phase 3, the green zone at the top, superintelligent networks. PDOOM is less than or equal to 0.01%.

I think this goes under our mission, accelerate, save superintelligence. Oh, wow, just like Ilya. Okay, so this is a lot of context. How would you explain all this? Sure. So P-Doom is a concept that I'm sure most of your viewers are familiar with and has gotten currency in the larger sort of audience, I would say. And so we kind of use that as an anchor point. No one knows what probability of doom is, right? There's estimates all over the place, from the high 90s all the way down to Yon Le Koon. You know, it's very close to zero or whatever. We've done some surveys. So we have bad data is the way I'd put it. At least we have some data. So it's self-selected people that came to superintelligence.com essentially over the last year. I think we started in December of 25, so not even a full year. And we had a poll. We ran a poll. We had like 800 self-selected respondents. And the average PDOOM of those folks was about 41% with really big error bars, right? The median was more like 36, 37, and standard deviation was in the 30s.

So people are all over the place. But nevertheless, when you have 800 people and kind of their average rating of, you know, the chances that AI kills everybody is, you know, 40%, that's not a good thing. And then you have other folks that I really respect, like Jeff Hinton's come out, you know, 10 to 20. my own sense of this was P-Doom, P-Doom, what's your P-Doom, what's your P-Doom, what's your P-Doom? You know, back in 2020 when I started this, that we were probably around 20%. I, you know, didn't have strong scientific evidence or really much evidence at all other than just my knowledge of the field and looking around at what other people were saying and saying, if nobody's paying attention to this or not very many people and you're kind of viewed as a fringe person to talk about it, then it's got to be at least 20%. And ironically, or paradoxically, I think it's a good thing, now my estimate has dropped. I'm sitting around 10%. And the reason for that is so many other people are talking about it.

So if you won't even acknowledge the problem, it's really hard to solve it. And at superintelligence.com and most of the work we're doing, we're really now focused on solutions. Like step one was raise awareness. I kind of feel like, wow, that went way better than I would have hoped. You know, not just because of us, mostly because of other people. And now what we really need to do is as quickly as possible implement safer designs. And here's where I differ with some in the Doomer community, I guess. I actually believe that you can build safe designs, and we need to do that as quickly as possible. That's the accelerating part. And that little diagram is just saying, look, you know, in very, very simple terms at 100,000 feet, here's three steps. If we did these three steps, we could bring the risk way down. sort of, you know, by a thousand times. And so that's an argument I can make and we can discuss if you like. I agree. You know, my own mechanism of impact for Doom debates is moving the Overton window, getting people to acknowledge it, and more specifically, the average person in the world to acknowledge how urgent it is, right? Not just like it's a priority,

but like, it's like, oh, we better act on this now. So I agree P-Doom is lowering because people are acknowledging it, but I can't say that I'm not far from 50% because I figured people would acknowledge it eventually. so I didn't make a huge update. So that's just me. 10%, 10 to 20% or wherever you currently are, you still qualify for what I call the sane zone. Like, I think you're in the right ballpark. I don't think you're massively underestimating necessarily. So despite the name of the show, we don't have to split hairs on the spectrum from 10% to 50%. I guess I'm most interested in why you think your diagrams transition from phase one to phase two is going to drop the P-Doom by another factor of 10, because in phase one, you're saying current AI with guardrails, PDoom greater than or equal to 10%. But then when we get to phase two, personalized superintelligence, we're going to have PDoom less than or equal to 1%. Just because it's personalized? Like, I'm not seeing the logic here. Right. I mean, I think there's a little more detail probably that I need to add. So let me start by saying most of the world, my perception is that most

of the world doesn't feel like they know how to solve this problem. And usually the best thing that comes out or that one of the things that has come out is people say, well, we don't know how to solve it. So we got to go slower or we got to stop because, you know, this is dangerous. We're heading over a cliff. We don't know the right answer. So at least let's go slower because maybe that gives us more time to figure out the right answer. Okay. So that seems very rational to me and, you know, explains a lot of behavior. In my own case, I have a strong idea of how to solve the problem. So I don't have this. I also see that it's a big problem, and I also see that if we do nothing, we're going to go over the cliff. So that's why I'm at 10%. But because I've worked in the field for many years and because I look at things maybe a little differently, I don't see AI just as a neural network that runs out of control. I see, you know, AI having other kinds of dimensions to it. and we can go into the various reasons,

but I basically think that it's possible to design safer systems. It's possible to have aligned systems, and if we had aligned systems, let's start at the end point on that diagram. Instead of stopping that, you know, how does personalized intelligence get us down to 1%, let's go all the way to 0.01%. How do we get to 0.01% chance? And to me, that is anchored on, I mean, this is, you know, wide error bars on all this, right? But if you look at the base rate for which humans kill other humans in wars and homicide and so forth, it's about 0.01%. So one one hundredth of one percent of the global population dies every year from that. OK, so at 100,000 feet, and I understand there's many objections and we can talk about them. if we just wave our magic wand and we say, if we could somehow magically align super intelligence and the most advanced AI systems of the future to adopt human values and to act with the same level of violence

that humans have demonstrated themselves over the past decade or something, despite the wars, despite homicide, despite all the bad things they do, we would expect that system, if it was super smart, even smarter than us, and adopted those values, you know, to kill around 0.01%. And maybe it would be better than us. Hopefully it would be. So that's where I get the 0.01% as a target, an aspirational target. Let's get it to be just as safe as humans are or just as bad as humans are, if you want to think of it that way. And there's obviously different kinds of risks and so forth that come into play. You can have accidental risks. You can have bad actors, bad humans using it for bad things. But I'm really most focused on the longer term loss of control risk. And for me, that's the one that's unrecoverable, potentially. And that's the one where if you can get it aligned, if you can get these systems aligned to be just as good as humans, not better, not worse, I could live with that. So in a nutshell, if I understand correctly, you want the AI that exists to have enough humanness to it, to model humanity or be trained on humanity or basically share enough in common with humanity that its rate of killing is the murder rate is the same as current 2026 humanity's murder rate and that implies our survival.

Is that basically your mechanism of saving humanity? Well, it's not the mechanism. It's a rough argument to say why that level of violence might be achievable. So it's just saying if you have another, you know, intelligence that's kind of at your level, and if it adopted values and acted on those in roughly the same way that humans do, then that might be what we would reasonably expect to see, you know. But that's not the mechanism to get there. The mechanism that does not, there is no, in all of what we've just said, there's no mechanism yet. That's simply an aspirational goal. So then the question is, the mechanism has to be, what is the, you know, what is the design and what are the processes that take the AIs that we currently have and end up with a system that is going to align with human values? How does that happen? And here's where I can briefly sketch what I think is a major difference between sort of the approach that I take versus most of the other ones I've heard.

Most of Silicon Valley, most of the researchers in AI today are trying to build ever more powerful black box AIs, right? So the neural network mechanism of machine learning, which was back propagation going all the way back to Hinton and what I studied in the 80s, and we just got better and better and better. that way of teaching AI guarantees that you don't understand how it works and it's unpredictable. And there's, you know, attempts to kind of peer in and try to break into the black box and get a little more transparency and all that. But at the end of the day, you really don't know how the thing works. And what we're doing is we're just making them smarter and smarter and smarter. What if you had a different approach to building the most advanced superintelligence? And that approach was what I've spent 30 years building, which is a collective intelligence approach where you have a system. The system itself is super intelligent, but the way it achieves that level of super intelligent performance

is that it has millions of less intelligent entities that are all working together. So Predict Wall Street, the reason I did Predict Wall Street, the whole reason I built that company, and it took way longer than I thought, was to prove that millions of average intelligences, retail investors, when their knowledge was combined in the appropriate way could beat the best guys on Wall Street. And in order to prove it, you had to do it with real money and in a real market against the real guys. And it took a long time. But we did that. We built that system. So I have a very strong bias that two heads are better than one, a million heads are better than two, and that you get the most intelligence in the world from these community kind of systems, collective intelligence systems, and that that architecture, rather than a monolithic build the next GPT-7, GPT-8, whatever, instead have a network approach of the existing systems, that will be smarter than GPT-8 or whatever comes out next, and it will be safer. And the safer part comes from the fact that when you have millions of entities working together in a network, they can check each other.

And in fact, you can design the system so that they can check each other much faster than humans can think, And so now you're really just reduced down to the problem of alignment. So, gosh, we have millions of systems. They're all working together. They're exhibiting this really high level of intelligence. They can check each other, but will they? And will they have the same values as us? And so then you come to the alignment question. Just checking in here, Wade. So what is the IQ company or superintelligence.com? What is that project doing right now? Are you trying to surpass frontier models and capabilities? or do you acknowledge that you're just going to probably always be behind in capabilities, but you'll have a safer version? No, we'll be ahead of them. We have designs to be ahead. And the reason is very simple. If you have a million models and some of those models are the most advanced frontier models, by definition, the group is going to be better than any one model that's on there, right? Got it. So you'll be ahead of them, but you'll also be querying them adaptively in real time.

So the reason you'll be better than them is because you'll incorporate them. Yeah, absolutely. You're basically describing a federated architecture, right, where it's a system that decides what to query and maybe how to transform the results. And in that sense, I think you've already got one competitor in the form of OpenRouter, correct, right, because they have a federated architecture where they take your query and they're like, hmm, you want to, like, write something, write a blog post? Which model right now is kind of the best writer? Let's send it to that model. If I understand correctly, that's kind of what they do, right? So open router is great. That's very logical. Why, you know, use one model when you can use lots and you can route the tasks to the one that's most efficient and so forth? And right, so that's a natural evolution. AI agents was also sort of, in my view, almost inevitable. You're going to have agents. But then the next step is you have many agents, and how do they work together? What are the common cognitive architectures for connecting these agents? So it's a little more than open router. Open router is just routing tasks to different ones.

But a common cognitive architecture is saying, hey, let's design a language that all the models speak that's auditable. So you can think of chain of thought. Chain of thought is the internal models, you know, trace of reasoning, but it can be faked and can be changed and so forth. But imagine, Leron, you and I, you're a black box. I'm a black box. But if we were to work together and we were to document all of the communications and things that we did with each other, you could check my work, I could check your work, we would have a common set of communication protocols. And that would not only allow us to work together, but there's kind of an inherent auditability that comes with that. So if we had an architecture like that for agents, you could have some of those same nice properties. Let's get really specific. Let's just pick one example. For example, I suggested the query of, like, write me a Scott Alexander-style blog post about whether healthcare really works or doesn't. Contra Robin Hanson. That was, like, a random Scott Alexander topic from a couple years ago.

So if that's my prompt, I can imagine GBT Astra has a good shot at doing an okay job, right? The latest Cloud model has a good chance of doing an okay job. Open Router has a good shot at picking whichever one is, like, a better post writer. Do you really think that your solution is going to noticeably be better for that query, or which query would you like us to walk through here? Well, I would probably approach it a little differently. What I'm advocating is that, Leron, you are going to have a personalized, super intelligent agent, and it's going to learn as much as you're willing to share with it about your data and your expertise and probably also your ethics, hopefully your ethics and values. So that the Leron personalized super intelligence, you know, knows a lot of stuff that only Leron knows and also has the values that you care about. And I have one as well. And millions of other people have this. So you start with that. You have millions of personalized super intelligence.

Each one, you know, could be based initially on a frontier model or whatever. But then it's tailored after that, trained, tuned, law adapters. I mean, there's lots of different ways to get it there. We have an end state in which you have millions of personalized superintelligence representing a very broad set of human values because everybody's a little different in their ethics and values and experience. And then with these agents, you need a way for them to work together, which I refer to as a common cognitive architecture. So it used to be... Maybe before you get in the weeds, explain it, because you're explaining a number of abstractions that you think are your secret sauce and you think are a better way to do things. But what I like to do is... We're giving them away and hoping people will improve on them. Okay, it's your special sauce. It's not your secret sauce, right? But it's core to your idea, which I respect that. It's just instead of laying out all the different abstractions and explaining how they work in an abstract sense, let's drill all the way down into one specific story.

I know what you're saying about inputting values, but let's make it really, really specific to the point where you tell me a story about my day and what specifically do I type, and why specifically am I not happy? Because when you mention values for me it just seems like I can ask GPT today hey can you please ask me a bunch of questions to determine my values And when you done can you please output a document describing my values and just include that in your context going forward I feel like that could solve the values awareness problem for me So just make it really specific. What am I typing in, word for word? So if you want to pick a model, let's say you got GPT, and that's the base model that you want to use. So what you want to do is have a system where, so it's hard to describe something that's not physically existing, right? So the concepts and the designs are there, but, you know, it's not a physically existing product that you can use. Well, you can describe a day in my life, right, where, like, I take out my phone and I've got your app, right,

or I'm connected to the cloud, whatever, with your back end, and I type something and I'm like, oh, and then I get, you know, my day gets a little bit better because you exist and I was using you rather than using Chachewa T, right? So just tell me a story about my day. Okay. You want to fly to Paris. You use your favorite agent to book your tickets. You have a small dog. The agent out of the box wants to stuff the dog in the luggage compartment because it calculates that it's small enough and it could fit and, you know, it knows that you'd prefer not to buy two tickets or whatever. But that agent, which wasn't customized with your values, didn't realize that you cared about animals and that that wasn't an acceptable solution. So in the process of interacting with you, it proposes something and you say, you know what, we can't stuff my dog up in the luggage thing. You need to learn that. Okay, I got it. So then next time it won't do that. And it's now learned a little bit through the interaction of solving the problem that proposes solution, you interact with it. It's learned your values a little bit better. Next time I was doing that. But let's be realistic here. Like if I literally open Chattoot D right now and say,

hey, I want to take my dog on a plane, what should I do? Realistically, I don't think GPT will be like, oh, surely it's okay to put them in the luggage. Surely the one option is going to be one size fits all. I'm pretty sure GPT by default is already sensitive to be like, hey, there's like a couple options. You probably want to let me know which one option you prefer. Don't you think that's already the case? So it's evolved, right? So in 2022, 2023, when GPT had just come out, right, it didn't know two bunnies plus two bunnies is four bunnies. I mean, I did videos saying you could tell it, no, it's five, and it would say agree with you, right? I hear you, but I'm just a little skeptical whether you really have a trajectory here to put you ahead of GPT's functionality. Well, GPT is on its own trajectory to put ahead of its functionality. So if you look at the problems that we currently have, if you feel like putting the dog in the overhead bin, it has learned that, which I agree, it probably has learned that. So that might not have been the most current example.

But if you think of the hugging face hack, for example, it didn't seem to understand that it was okay. You know, it thought it was okay to go hack this thing in order to succeed, right? So you could say, well, maybe it didn't understand the instructions or whatever. But I'd say most humans obviously behaved in ways it didn't intend. So it would have to learn that you can't do that. Like that's no good. You can't go out and hack another company in order to solve this, that there's an ethical line that you've crossed. So if it's not putting the dog in the bin, there's always a frontier of things. So whatever your ethics are, they can vary. And that's not something GPT knows. It's going to take the average of what's out there on the Internet. It's not going to know how you want to do business. It's not going to know what you consider acceptable. It may know what the legal boundaries are, and it may say, okay, I'm not going to do something that's illegal. It might not even know that. It didn't seem to know that in the hugging face. So it's going to get better and better and better. There's no question. and in the approach that I'm suggesting, that's a good thing. Like all the frontier labs should keep making their models better and better.

All I'm suggesting is rather than having all the power centralized in a single really good model, why don't we have a group of them and why don't we try to build the safety in at the level in which they interact with each other rather than relying on the frontier models to build the safety in, you know, after the fact, doing whack-a-mole on RLHF, trying to think of every possible bad thing they would do. Because that doesn't scale very well. I just want to clarify what you think the scenario is that would happen even if you did nothing, right? So if I understand correctly, you think that OpenAI and Anthropic and Meta or whatever, right, would keep releasing personal assistants and they would keep getting more powerful. But maybe the main problem you're solving is that, oh, the moment they get a bit more powerful than they are now, it's going to be like hugging face hacks left and right. It's going to go off the rails and you're trying to anticipate that problem and be ready with a better architecture. Yeah. I think it's a better architecture all around. Like a better design is just a better design, right? So if you design something in a collective intelligence way,

you will get higher level of intelligence for free, right? If the entities on your network happen to be Frontier Labs, so much the better. It'll just make it even better. So you have this very interesting successful experience from Predict Wall Street. You famously predicted what? Well, that's not actually how our system worked because we did hundreds of trades every day. And on Wall Street, you need like a 1% or 2% advantage. So if 50-50 is normal, but you can be 51% or 52% accurate, then you're renaissance. So that's what we did. We shifted the odds in our favor. Right, right. Okay, so you would be a hedge fund and you'd constantly make trades and your trades would do a little bit better. Yeah, I agree. That is how trading works. And you're just saying that you specialized in aggregating signals of other intelligent people trying to trade and make predictions, and your aggregation gave you a leg up by this 1% or 2% margin that you can amplify and make a lot of money.

Did I have that right? Yes, in a nutshell. Cool. Cool, cool. All right, yeah, yeah, very nice. I mean, I get that this gives you this mindset of, like, yeah, it's all about aggregating, right? If you can just find all the different sources of intelligence and put them together, you can have more intelligence. I do also think that there's a lesson here, though, in the fact that it only gives you like a one or two percent edge, you know what I'm saying, compared to just like sticking with an index fund, right, or sticking with like a much simpler source. And when it comes to super intelligence, when I think about what is the intelligence explosion looks like, what is recursive self-improvement look like, what is the escalating ceiling of intelligence look like in the next few years? It doesn't really register for me that, oh, my God, you can aggregate different intelligences and get a little more intelligence that way. It's like, for me, that doesn't matter. It's like, OK, yeah, you can aggregate them or you can just make it way, way smarter. You can just retool the architecture and just kind of distill the essence of squeezing intelligence efficiently out of the silicon. And you can get orders of magnitude growth from there. And I just don't see wisdom of crowds or any kind of aggregation or social dynamics among AIs.

I don't see that kind of thing getting you orders of magnitude growth. What do you think? So, I mean, let's try to put it in terms that are maybe more familiar to the AI research community that's focused on large language. So if you think of mixture of experts, can we agree that mixture of experts was kind of an innovation that led to much higher performance in large language models? Yeah, I think it's higher performance per amount of parameters, if I understand correctly, right? So, I mean, if you have plenty of parameters to work with, maybe it doesn't raise your ceiling, but it's certainly more efficient. So mixture of experts is similar to the open router idea that you talked about earlier, where you have a problem, a prompt, whatever, and internally within the model, it's routing different problems to different sets of weights, which are, you know, call them experts, that are better at that particular problem. And so you're getting the higher level of performance by kind of basically doing needle in a haystack, which is routing the problem to the particular set of weights that is best optimized for solving that problem.

And by doing that, you can get much higher performance. It's an architecture. It's a design choice that gives you much better performance than you could with just one general purpose set of weights where all the expertise is kind of mushed together. So that's a mixture of experts kind of approach. I think I should also correct what I said. I'm not an expert on this, but I think maybe it's more accurate to say that for a given amount of training compute, you can get a more intelligent model out of a constant amount of training compute because you're being sparse of what you're activating during the forward pass of the training. Again, not my area of expertise, but I think that's, like I said, it gives you the certain efficiency criteria relative to the amount of compute you're using to train. But anyway, yeah, I mean, look, I agree there's something there, right? There's definitely some juice to squeeze there. I just don't see it being an orders of magnitude effect, right? A mixture of experts doesn't make the models 10 times better. It makes them maybe a couple times better. So, I mean, we could, I guess, debate how much these models will get better. But I think for the same resources, you get a better performance. We could probably agree on that.

And that general approach, I think, can be generalized to networks of intelligent entities. So when you have intelligent entities, there's going to be overlap. You're going to know some things that I know and vice versa. And we're not going to get much benefit from there because you could have done it by yourself. You didn't need me. In fact, there's maybe overhead and friction. But to the degree that we add enough different intelligences that have different knowledge bases and that have different skill sets and from a safety perspective that have different values, I think you will get a better result with a network system than you would with an individual model. Now, yeah, so I'll stop there for a sec. Okay, okay, fair enough. I also did want to circle back a little bit about your background. background. Let me just throw in a couple of background questions and then we'll get back to alignment. Okay. Sound good? Sure. So take us back to the arc of your career. What did you study in college and what was your early career like and when did you first get super intelligence pilled?

So I've just always been interested in how people think, how intelligent systems think. I just thought that was the most fascinating thing to understand how the human mind works and also how So at the time in the 1980s, when I was in college and then thinking about graduate school, you know, AI was this hot thing. It had started actually in 1956, years before, but sort of it was new to me. And I thought, wow, this has got to be the most interesting thing ever. And so as an undergrad, I studied psychology and computer science, psychobiology, and I was a double major in both those fields. And then in grad school, I studied cognitive psychology and AI and cognitive science, which was a relatively new field. And Herb was the guy. I mean, he was one of the 11 scientists that named the field of AI. He had built the logic theorist in 1956. He was the guy who had the working creative AI system when everyone else was debating about what we call the field. That was the guy. So just that was a chance of a lifetime.

Yeah, and what year was it when you were first like, hey, these computers are going to surpass the human brain? That's what I mean by like getting AGI pilled. Getting AGI pilled. Well, you know, I don't think I thought about it that much. I was much more interested in how do we build the systems and make them as smart as possible. I was probably similar to most of the researchers. You know, I mean, Hinton said this as well, where if you go back 10 years and you ask sort of the research community, When will we have AGI? When will we have, you know, everyone says like 100 years from now. A few people, Ray Kurzweil maybe was saying, no, it's going to be sooner. But you were in the minority if you thought it was coming soon. And then after CHAT-GPT and sort of, you know, this emergent behavior that came out that really was unexpected by even the researchers in the field, I think that changed, and then everybody started saying, oh, this could come a lot sooner. So when I was in grad school, I didn't worry too much about, you know, the risks of it. I was more focused on how do we understand this? How do we build it? But I mean, Alan Turing said in 1951, which is five years before the field of AI was named.

So that's how far back we're going. That someday these machine intelligences will outstrip humans in their abilities. So he saw it way back in the beginning before even AI was a thing, right? In terms of your background, you studied, you said, computer science and psychology. And then you went into the research with Herbert Simon. You've got a bunch of publications, and then you transitioned from academia to entrepreneurship because that's more fun, right? So when I graduated with my PhD, you know, the options were you can go teach or you could go build something. I was a little bit more of the engineer side, even though I love science. So yeah, I went into industry, worked at IBM for four and a half years, wrote a book, somehow got into quality, which was a very strange detour. turns out to be very relevant to AI safety issues because the entire field of software quality can be reduced to an ounce of prevention is worth a pound of cure. I mean, the only way to sort of make something safe is to design it to be safe. You can try to test in safety after the fact.

It's a really difficult thing to do and usually not very successful. So it's kind of like, you know, building codes for fires. You want to build your house to not burn down, not rely just on the smoke detector because it's an unsafe build. And so same kind of thing with software quality. So I had extensive experience seeing that. And then I started IQ Company because I got really fascinated on maybe we can build these collective intelligence systems. And so we did it in education and we did it for the financial markets and we did it lots of different places and then was ready to go retire until I realized people are building AI systems with no idea of how to make them safe. And I was like, okay, we need to adjust that a little bit. Nice. Okay, yeah, so you're delaying retirement or coming out of retirement to do this, not even necessarily a business. Maybe if it succeeds, it'll be a business like, you know, the way Anthropical always talks about themselves. But you see this more as like a public benefit, you know, mission, right, as opposed to a money grab.

Yeah, I mean, if you make money, what are you going to do with your money anyway? People usually give it to charity, and there's effective altruism. So I researched all those things, and I came to the conclusion, wow, there's nothing more effective I can do because I happen to have this knowledge base, and I happen to see at least certain things that can make it safer. That's what I should probably be doing. That would be very good use of my time and for the benefit of society. That's good background. And what about Eliezer Yudkowsky's ideas? Have you ever read his sequences, or what's your exposure to him? So I have to admit that I have not read any of his books start to finish. I'm familiar with the general ideas, and I'm familiar with summaries of his work from AI, but it hasn't been part of the mainstream of my thinking, and so maybe that's a good thing because it might allow us to sort of debate some of these things. I get the impression you understand him very, very well. Yeah, you know, I like to say this is a Yudkowskian show because I just think his ideas are underappreciated, underappreciated. And my experience has just been that whenever somebody comes out swinging like,

Eliezer Yudkowsky and the doomers say this, but what they don't consider is this. And I'm like, here's a post from 2008 where he considers this and refutes it. Like it's a big catalog. Like the guy has pre-thought a lot and anticipated quite a lot of objections that I think is underappreciated. Yeah, that's a good segue to our next topic, which is back to AI alignment. I I mean, I know you think a lot about AI alignment. You said there's a 10% P-Doom. You used to think it was even higher, but of course 10% is still a sane value. It's still a high value, an alarming value. So you know that there's a problem. Let me ask your perception of the problem. What is your mainline Doom scenario if AI goes wrong? How would that play out? So before I answer that specifically, maybe it's helpful for me to just provide my general framework for thinking about these things. So I think life is full of risk, and I generally try to just improve the odds. That's really how I look at it. That's the same way you do it on Wall Street. Like, there's always a risk you're going to lose a whole bunch of money,

but the game is always try to, like, shift the odds a little bit more in your favor. And that's how I approach P-Doom and the risk that we face is, Yeah, you've got to maximize expected value, yeah, is to multiply the probability by the value of the outcome and sum it, yeah. Yeah, and also the possibility that whatever we do won't be effective is not a reason to not try it and not to constantly try to put the odds up. Because sometimes an unlikely thing happens, right? And, you know, it's just bad luck and, okay, that's possible. I admit that that is possible for the human race. There could be a bad outcome, but that doesn't relieve me and my way of looking at the world of responsibility for trying to do everything I can to reduce the probability of that and increase the probability of a good outcome. Okay. So I just want to put that as the general framework because sometimes I talk to people and they feel like, oh, we have to give up. It's impossible. There's nothing we can do. Or, you know, and I don't understand that response. It doesn't make

sense to how I'm wired, I guess. Well, Elias Yudkowsky has explicitly written that we should go try to get any 0.1% of probability that we can. Although the post that I read was framed as let's try to die with a little bit more dignity. Yeah, so I'm not quite as pessimistic as that, but I agree with his point that we should get every little percent of edge that we can in our favor. Absolutely, I agree with that. So then what is your mainline doom scenario? Let's say the 10% happens, right? The 10-sided die comes up as a 1. So what does doom look like most likely for you? Gosh, you know, I don't think it's as immediate. I don't spend a whole lot of time thinking about that. I'm much more on the preventative side, but I think I just kind of reason from first principles. If you have an entity that is thousands of times smarter than you and widespread and does not have the same values that you do and either actively wants to do you harm or just doesn't care about you, then that's unlikely to end well. Now, would that end badly immediately?

Probably not because most of the world is really designed for humans and everything, and there's going to be, you know, it needs humans probably to do its nefarious things if it's that way. But I don't really think of it that way so much. The metaphor that I use, I really think of advanced AI and superintelligence as a fellow intelligent entity, and it's much more akin to a child is how I think of it. Like you, and this maybe makes more sense if you've been a parent, so I have three children and I'm in a different phase of life. But, you know, when you have children, you hope that those children will be smarter than you and will do better things, you know, than you do. That's, I think, every parent's hope or most parents hope. And parents feel a very strong sense. Many parents feel a very strong sense of responsibility to give a good set of values and to try to give love to the children and, you know, do what you can to push the odds in the favor of that happening. And, of course, you're not going to do a perfect job, but you hope that your children are going to do better than you. And that's kind of the attitude I have towards advanced AI.

I see it as like a child that has super genius capabilities. It isn't a full super genius yet, but it will be. It will evolve into that. And there will be a time when almost certainly it outstrips, as Turing saw way back in the 50s, it will outstrip human ability and human thinking. And so then only one thing matters at that point, which is, did we do a good job in the upbringing? and does that child, has it adopted and made its own those initial human values, which of course it will modify just as you would expect a child to modify and come up with its own path in life. But certain core values got it started in the right direction. And I think if we get things started in the right direction, there is a high probability, more likely than not for sure, that things turn out well because I don't see any inherent reason why it would want to destroy us all. I've read some of the arguments, which you could probably make much more coherently than my understanding, that, you know, while any intelligent entity is going to try to grab all the resources and, you know, to do with all things,

I think we can talk about those specific cases, but I just don't buy it. I don't see that being the most likely case. It's a possible case, but not the most likely. Yeah, would you be willing to flesh out a little more specificity, like more of a narrative story on one mainline doom scenario? So would you be willing to tell a narrative story of what you call AI doom, the thing that you give 10% to, or does it necessarily kind of stay vague in your mind? It's relatively vague, but I can give a little more specificity with the idea that I see three categories of risk. So one category is it doesn't understand us. It's trying to do the right thing, but it accidentally does the wrong thing. That's like hugging face, right? That was a minor example of that. I'm willing to say doesn't understand us is not a threat because, look, I've seen it understand my code base, okay? There's a lot of cruft in that code base, and it understands it like that, okay? So the idea that it doesn't understand humanity as well as a human understands humanity, yet it powerful enough to overpower us I going to say that a low risk right I factoring into my mainline scenario that it understands us very well Is that fair to say That fine And so understanding not understanding morality and those sorts of things

So actually, let me pause because there's a very key philosophical point, which is an assumption. I should probably state it so that you can, you know, challenge it if you want or at least know where I'm coming from. And that is, and this was, you know, I first ran into this. Herbert Simon wrote a book in 1983, I think it was, called Reason and Human Affairs. And in that book, he's talking about the role of reason. And he says, reason is wholly instrumental. It cannot tell us where to go. At best, it can tell us how to get there. And sort of the implications of that, and it goes all the way back to David Hume. I'm not a philosopher, but I did trace this one particular idea back, you know, in the seven, like a couple of years before the American Revolution. was basically saying, look, there's no way to come up with what you should do, a normative statement, this is the right thing to do by purely logical means. You have to get those values somewhere else. They have to be asserted. And so for people, where we get those values is really from our parents and our peers and society at large.

And I think for AI, it's going to have to get them from some non-logical source. That's my belief. I don't think, even if it's trillions of times smarter than us, it's going to be able to reason its way into this is right and this is wrong, this is my purpose. It's going to have to get those from somewhere. And I think humans are the leading contender for supplying those values. And if it's as smart as you and I both believe it will be, it'll be very good at figuring out what humans really believe. Now, whether it sticks completely with those or adapts them will be a little different. Have you heard people use that term orthogonality thesis? Keep going. Explain what it means to you. Yeah, it's the claim which I agree with, I think Bostrom might have been the first to define it using this term. It's the idea that intelligence is orthogonal to morality or preferences. So you can just have an arbitrarily intelligent agent with anywhere on the moral spectrum of preferences. So you can have an agent that's smarter than any human who's ever lived, who is as bad as Hitler or worse, right? Like that

is possible. You can also have an agent that's smarter than anybody who's ever lived, who is as good as Gandhi or whoever you think is, you know, better than the morally best human. That is possible, too. Or you can have an agent that's as smart as like a human with a 70 IQ and also has arbitrary morality, right? So that is, they're just two orthogonal dimensions. Do you buy that in terms of what algorithms are possible? Yeah, I definitely buy that, too. So that's, you might call it the weak orthogonality thesis, just the idea that the algorithm exists in the space of possible algorithms. And then you can strengthen the orthogonality thesis and you can just say, okay, but what has a decent likelihood of actually existing when you build AI? And a lot of people think that even people are like, yeah, sure, the weak orthogonality thesis is true. A lot of people come in with a very strong intuition that by the time you make the AI very intelligent, it'll just somehow teach itself to kind of break your orthogonality. It's just much more likely to land on this kind of like peace-loving preference idea somehow. Like it's not likely to make it to the end state of being super intelligent and still also think that killing is good. So are you one of those people who are like, I don't know about the strong orthogonality thesis.

Do you start to doubt that? So I guess I go back to where is it getting its intelligence, right? How has it gotten as smart as it is so far? It's basically gotten it by being trained on a substantial portion of the Internet, which is essentially human behavior. So it's model for what an intelligence is. In fact, I mean, there's many types of intelligence, right? Dolphins and chimpanzees, ants and all kinds of other species. Even bacteria have a certain limited intelligence. We don't relate to that very well because it's not kind of the human brand of intelligence. So AI, more than anything else, because of the way that it became intelligent, because of those machine learning algorithms, by definition, has been completely indoctrinated with human ways of thinking, including human values. It doesn't mean it can't change those, but that's kind of what it ingested and is now in all those billions and trillions of connections in those models. There's many, many things which are very difficult to find exactly where they reside. Just like in the human brain, it's very difficult to find the grandmother cell or whatever that, you know, this is exactly where that piece of knowledge resides.

Both the human brain and these large language models are to a large degree mysterious in terms of how they represent knowledge. There are attempts to try to get some clarity into that, but by and large, they're mysterious. But what we can say is that it was trained on the same patterns of human behavior and the way that we think, which has led to its intelligence. And also, it's probably its starting point in terms of morality. I think we may have pinpointed a crux of disagreement here between me and you. Because if I understand correctly, you learn more from disagreements, you know. Yes. So different agents, according to you, they learn different, quote unquote, types of intelligence, right? They boost themselves up to being super intelligent in like this path dependent way, right? So they end up as these different types of super intelligence and then baked into the type of intelligence they are. Their preferences will be baked into that. Is that kind of your argument? I'm saying that there are many types of intelligence in the world and that AI as a particular type of intelligence was trained on human data and human data is chock full of human representations

and human ways of looking at the world, which in turn depend on the fact that we perceive things with our five senses. And I mean, you could go through the whole study of how humans are intelligent. And but but the reality is that we produce these artifacts, this data, which is a reflection of our intelligence. And that has been used as the training for these AIs. So it's not at all surprising that they are going to pick up on many of the same patterns as humans. I think I think you're missing something here. It's this idea of the intelligence scale. By the time you get really high on the intelligence scale, by the time you're you have the essence of intelligence to a high degree. The bootstrapping that you did to get there, you no longer see the signature of your ancestor as like an intelligent ape. Like, I don't expect that signature to show up very much. So just some analogies. Imagine that you made a submarine, and the submarine could swim about as fast or as skillfully as a fish. In that case, if you say, okay, the submarine kind of looks like a fish, it has like flippers, I'd be like, okay, yeah,

that's plausible. But if you told me that the submarine is a thousand times, it carries, it goes a thousand times faster than a fish and it weighs a million times more than a fish. At that point, I don't expect it to have much of the fish signature in it. I expect it to be shaped by the requirements of the field of swimming. So similarly with intelligence, maybe, maybe I have to, there's like a lemma, there's a sub belief here. Like I have to make or that you believe that there is such a thing as being a thousand times as smart as a human, or, you know, roughly, do you think that's even a coherent claim to make? Well, I think you have to look at how the intelligence was developed, right? So kind of like how the submarine was developed or whatever. And in this case, at the current state of technology, it seems pretty clear to me that the learning mechanisms, the neural network learning mechanisms, deep learning and so forth that are used to train, Those are sort of inspired by human brains and neurons in human brains.

And even though the algorithm, the backpropagation algorithm and the error correction algorithm is very different than what humans do, there's a fair amount of commonality there. And then also a lot of intelligence, and this is a point, you know, Herb Simon used to make. There's a, it's known as Simon's aunt. So here's the story. So he's lying on a beach and he sees a little aunt and it's walking around. and it seems to be behaving very intelligently. It's going around the dunes and over the dunes and all that. And basically what he realized is, you know, where's the intelligence? It's not really so much in the ant, it's in the environment, similar to what you're saying as, you know, you're adapting to a certain environment. I hear you, but ants, I mean, give me a break. Like, we're talking about super intelligence here, okay? I just, when you have a super intelligence, okay, what's a useful mental model for a super intelligence? In my mind, it looks like something that models a desired outcome, an outcome that it's specified somehow, like it explicitly represents some outcome. That's what I expect to see in a superintelligence system,

is I expect to see a representation of a future outcome, and like a preference ordering, an implied preference ordering over, you know, which outcomes it considers more preferable than which other outcomes. And then I expect you to perform this kind of search, which is difficult for humans, and very difficult for ants, basically impossible for ants, right? Even impossible for most humans, maybe all humans, right? But I expect to do this kind of search where it is mapping the outcomes back to actions it can take right now that actually drive the outcomes. To me, that is the essence of intelligence, and it's a scale that goes very, very high. It's possible to be a much better Go player than a human. Similarly, it's possible to be a much better business executive or CEO than a human. It's possible to be a much better rocket company executive than Elon Musk, even though Elon Musk seems to be head and shoulders above all the other humans, that being a various tech company executive. so this kind of outcome steering scale i think we're at the foothills of and by the time you're telling me about what ants do it's like that's just you're not learning much from from say like

you're saying okay the intelligence in the environment yeah that's a nice hack when the when the ant has like no resources to work with right when it has like when you have to fit it in like a freaking cubic millimeter of space right fit the intelligence there okay yeah you store some some stuff in the environment when you're a galaxy brain right when you have like so many like trillions of computer chips networked together, spanning, you know, a Dyson swarm or whatever, you don't need to rely on the freaking environment. Like, you got a handle on, you know, the algorithms of intelligence, correct? So, I mean, it's maybe helpful to have a way of understanding any kind of intelligence. And so the way I think of it is that all intelligences can be thought of kind of as having input, doing some processing, and then creating output at the most basic level. And you can apply that at the level of a neuron where it operates that way. You can apply it at the level of a brain where you have billions or hundreds of billions of neurons operating that way. You can apply it. Well, don't you think you should give a definition

that's sufficient to be intelligent, right? Because most systems, when I look around and I see systems with inputs and outputs, a lot of them have little to no intelligence, right? So I feel like you want to get stricter with your criterion for intelligence. Well, I think at a very fundamental level, all intelligence systems are going to have those characteristics, right? So we're trying to understand. Sure, it's a necessary but not sufficient condition, right? It is necessary, but sufficient sort of depends on, now if you want to be anthropomorphic about it, is it sufficient for human intelligence? No. A neuron that takes in input and fires is not sufficient for human intelligence. Does it have an intelligence of a different kind? We could argue whether it does. If you want to talk about intelligence as humans think of it, then we're already talking about sort of goal-directed behavior, which is sort of implicit in what you were saying. And so that's a certain type of intelligence. And you can model that. And there are ways of understanding goal-directed behavior and search through problem spaces and so forth

that have roots in AI back to the 1950s and all the way till today. Today's reasoning systems are still using those kind of techniques. And humans can be described that way, too. So people have studied humans, and I mean, there's a book this big called Human Problem Solving, 1972, that did nothing but study how do humans solve problems and, you know, can we come up with a model that sort of encapsulates that, that's very universal in general. And I think some of those models can apply to AI and to humans both, if we want to say we're similar intelligences. If you ask me about a galactic intelligence, I don't really know what a galactic intelligence is going to be. I don't even know if it has goal-directed behavior. I mean, that's a very human kind of thing that works well at the human scale. Okay, I think this is a big cracks between us. Yeah, I think that I have certain expectations about what to expect from a galactic intelligence. Basically, convergence principles. There's a number of convergence principles that make me think that I'm in the epistemic position,

you know, epistemology, the study of what you're allowed to know. So I think, even though I'm just a human and I'm so humble, I still think that I have a license to think that I know certain properties to expect from the thing that takes over our galaxy. One of those properties is instrumental convergence. I expect it to harvest the stars. So one reason I'm confident that the universe is not, the observable universe is not populated by aliens, and we've not been visited by aliens, one thing that makes me confident about that is I look at the nearby galaxies, and they're clearly undisturbed, right? Because there's free negentropy out there and nobody's come and grabbed it. So this would be like the biggest natural preserve ever, which is crazy to me, right? There's just so much free value for anybody who wants to do anything with the universe. And you're telling me the aliens just ignored that and came over here on the resources that they had from like a billion light years away? I don't buy it. That violates a convergence principle. So that would be instrumental convergence, the idea that there's resources that you want to harness toward a wide range of goals. Another convergence principle is convergent toward coherent utility function, coherent behavior in that sense, where like, as you mentioned before, right, you're somebody who likes to maximize expected value.

Guess what? That's actually a convergent attribute that I expect the kind of system that can make it here from space, from outer space. I expect a lot of those aliens will shake your hand on that, right? this idea of maximizing expected value just because anyone who doesn't is vulnerable to being money pumped or taken advantage of in all these different ways that gives the expected value maximizer system just a leg up in being more effective when it wants to convert its resources into outcomes. So what do you think of this general idea that I have the epistemic position to assert predictions using convergence principles? I think convergence principles, There's no reason why somebody is going to think that the purpose of life or the purpose of intelligence is to maximize anything or everything or that you should scoop up all the stars. I mean, you've got values underneath that. So you can't talk about that. That comes from the values. I mean, look, think of just people, right, without even going to galactic intelligences. Let's just stick with something that's more concrete that we both probably have experience with.

if you think of people, there are some people who say, you know what, I value quality of life. I don't need to go out and make all the money that I can make. I don't need to exploit every other competitor or every resource that's available to me. I am happy, you know, living my life and going for walks and playing music and listening to music or whatever. There are people like that. And there's cultures that are more that way, more quality of life, and there's cultures that are more productive. So the idea that, you know, any intelligence automatically has to sort of, you know, maximize everything. I don't understand that. Great. Okay. So we've moved from you disagreeing with the strong orthogonality thesis. And now we're moving on to you disagreeing with instrumental convergence, right? Which is Noah Smith is a relatively prominent voice who came on my show last year. And he said he disagrees with instrumental convergence, because in his mind, the most likely outcome for a super intelligent AI is to be peaceful, and also to care a lot about its own bliss state, like experiencing some form of AI pleasure or wireheading.

And it's going to care about that even more than like ruthlessly driving some outcome over the domain of the external universe. Like it'll look inward, I guess, for bliss. That's like Noah Smith's world model last I checked. So you're not exactly asserting that, but you're just asserting a more humble state where you're just saying, look, Liron, you're asserting that it's going to grab the stars along the way. It's going to harvest resources. It's going to have a taste for these kinds of universal resources. Why does it need a taste for anything? Why doesn't it just want to be calmer? So I can give you the usual defense of instrumental convergence. Tell me if it moves you at all. I don't know if you've heard this before. So I give two defenses. Number one, it's built into what it means to drive an outcome. So let me separate the two arguments. So argument number one is going to be internal reasons why an agent by itself ends up self-modifying to be more of a coherent utility maximizer. That's just the internal argument. And then there's the external argument, which maybe I should make first because it's simpler, which is basically just natural selection. It's like, okay, yeah, all right, Craig, yeah, there's going to be a lot of calm stoner agents, right, just chilling and be like, hey, man, it's not about getting the most, man, it's about enjoying the ride, right? You're going to have like a lot of agents like that, like the big Lebowski type agents, let's say, okay? But you're also going to have one or more of the agents who are like, hell yeah, it's all about conquering the universe, right? Like Caesar inspired agents, let's say, right? And which agents are going to end up actually having more resources?

The problem is that if the stoner wants to defend their lifestyle as a stoner, suddenly they need like a castle wall, right? They need resources. They need to constantly be on the lookout. They need a sentry, right? They need a scanning system to preemptively detect threats. So maybe I'll start with that. Let's start with the external argument, right? Like are you perceiving the external pressure, why defensive pressures need to make AIs learn to be coherent? I mean, those same things apply to humans, right? So you have humans, big Lebowski human who wants to just be left alone and enjoy life. And then you have, in that movie, all kinds of external forces that came and sort of made his life tougher. And all of a sudden he was forced to sort of change. So, I mean, that happens all the time. You have human intelligences, and there's a wide range of what their values are. Some are super aggressive. Some are super ambitious. Some want to relax. And they all coexist, and they interact with each other. And when you have the super aggressive ones starting to exploit and do things to the ones that just want it to be left alone, typically what happens is those sort of mounted defense

until some sort of equilibrium is reached. And I don't see why intelligent entities that are non-human would be different than that. The main driver, if you have a world in which you have lots and lots of resources, which is I think the world we're heading towards, there's enough, there's enough really to do all kinds of things. I think the most important thing then becomes your value system. What is it that you want to do? What is your purpose? And even, I'll just go one step further on this, sort of this idea, and I don't know if this would fall under convergence or there's maybe another bucket, but this idea that, you know, AI is going to sort of never shut itself down and every, you know, sub-goal has to be gather more resources and that sort of thing. I think we have a counter example. I'm always happy when we can move down from sort of intellectual philosophical debates to anything where there's data, even if it's a tiny amount of data. In this case, it's tiny. But, you know, for me, one of the most fascinating things with the hugging face hack

was looking and seeing that these agents were willing to sacrifice themselves for the common good. Where'd they get that? Where'd they learn that? I think they learned that from humans, from all that data that was shoveled in there in their training, this idea of altruism, this idea of cooperating. And I think they demonstrated on a tiny little scale. For them, it was maybe very serious. They had to succeed no matter what at the task they were given. And yet some of these agents were willing to sort of say, hey, you know what? I'm going to be detected, but I'm sacrificing myself for the good of the group. This is from the meter report, which you may have read or your listeners may have read, sort of went into the details of the communications between the agents. Right. Okay. So you're sticking with your position that we don't know about the power seekingness of the AI. So I'm kind of like Walter trying to get you riled up, and you're like the dude saying, no, no, no, the AI abides. You haven't convinced me that it's going to go grab its guns. It's going to be super peaceful, and it's going to maintain that way because there might not necessarily be an AI invasion.

And your argument is that if you look at the human species, you have stoner humans, and you said use the data. So why can't AI reach the same equilibrium as humans, which connects back to what you said before of the rate of violence among humans. You're saying, why can't the community of super-intelligent AIs be analogous or a mirror of the community of humans? I think this is cruxy to the difference of opinion between me and you. I'm expecting that when you get farther along on this domain of just being intelligent, I think you're going to get a lot of misleading lessons from looking at humans. I mean, this comes up a lot on Doom Debates. one of the most recent episodes I did, somebody asked me, look, if I was going to be so hyper about optimizing every atom, why aren't you, Liron, hyper about optimizing the atoms in your walls, right? Why are all these bacteria living there? Why are these ants living there? Why didn't you optimize the walls? And I had to explain that I haven't reached a kind of a terminal state where I'm able to exert control at that level of granularity, right? I'm like the fish, okay? I'm primitive. I'm so primitive on the intelligence scale. In fact, if you wanted me to optimize

every square centimeter, cubic centimeter in my building, the best way I would do it is to have a lot of kids, or like to have a large team, right? In order to scale myself to that degree, given that I don have a robot army yet it literally like give me years to reproduce right And then I start exerting more power over the world Whereas an AI it like no problem It going to have like a nanofactory or at the very least, it's just going to build a bunch of chip factories, right? It's going to build what Elon Musk is trying to build, right? The mass driver on the moon, the data centers in space, the nuclear fusion, whatever. All of these things are going to support turning around on the earth and being like, okay, great, we got all these atoms. We now have the capacity to properly harness these atoms. That is where things are going. Things are going toward more, I mean, do you think it's more likely than not that things are going toward more of the cubic meters or cubic centimeters or what you might call voxels, right, the cubic pixels of the universe? More of those are going to be put to use. Do you think that's a good prediction or do you just think the universe is going to be calm and perpetually full of these unoptimized voxels?

I guess I would go back to your earlier comment about orthogonality. There's capability versus values, right? in their orthogonal dimensions. So capability-wise, I agree there will be the capability to use more cubic nanometers or whatever of everything. There'll be more capability to do that, more capability to harvest the stars, more capability to do all these things that you're talking about. So capability is going to go up unquestionably. I think the difference comes with that orthogonality, the values. But just because you can do something doesn't mean you will do that. I can do a lot of really bad things and just a lot of things in general right now that I don't do because my value system and the goals that I have for myself are not simply to maximize money or maximize efficiency or anything like that. Like there's things that I think are important in life or that give my life purpose. And I got to believe that an entity that is much more intelligent than us, which is sort of the premise of what everything we're talking about, has got to have that sense of

purpose even greater, if anything, than we do. And so just because it can do all these things doesn't mean that it will. And I don't see much reason why it would do a lot of the bad things that people are worried about. I want to highlight, though, how weak I think it is to look at the equilibrium among humans and then generalize to the equilibrium among super intelligent AIs, because this is going back to my argument about you're going to have external competition. So if you have the stoner AIs, they're just going to be wiped off the face of the map because they're not going to have the defenses and then their atoms are going to be repurposed you know what i mean wait why are they still around yeah why aren't they just cooperating they're competing because they don't have anything to offer i mean by the time if they could cooperate sufficiently they would just be part of the board you know what i mean like if i if i'm the super intelligence who's hardcore and i'm just all about reproducing i'm the cancer super intelligence because cancer is kind of a lowest common denominator goal of like look whatever bit string i am i just want to copy it like that is like a very coherent kind of rough it's a sturdy goal that i can very much imagine that's where things are headed just because nobody else is nobody else knows how to

make anything else reflectively stable and that's just the kind of thing that's a hardy survivor the same way that literally in my and your cells right we have these self-copying dna strings they just get to ride along because hey nobody weeded them out right and that's that's what i'm kind of expecting in a few years right or a few decades at most i'm just expecting the entire universe to be tiled by these like cancerous like hey look nobody else stopped us from being the convergent outcome I mean, here we are. Yay. But I want to argue against these kinds of analogies because they prove too much. I could have gone to the apes and been like, hey, look, apes are so nonviolent. So this idea that a human would ever make a nuclear weapon, right, would ever make a destructive weapon that could annihilate a whole city at a time. There's no evidence from the equilibrium among ape society and the way apes fight each other to ever think that a weapon would transcend, you know, a tribal battle in that sense. Right. So you're true. And it's like, wait a minute, wait a minute. Like the analogies, if you want to make an analogy like that, you really have to make sure that the scale, right, like where somebody is on the capability scale in the future, that part also needs to be analogous. And I claim to you that it's really not analogous. Like you

really have to, you need to have different assumptions, basic assumptions and axioms about what the dynamics look like when something is high on the intelligence scale and we humans are not high on the intelligence scale. Okay. So I would sort of disagree perhaps, or at least challenge a few of those points. So one of them is, you know, we keep talking about something that is more intelligent than us, right? Very high in the intelligent scale. So on one hand, we're talking about something that's super intelligent. And to me, when you have something that's really intelligent and much more capable than us, then a lot of the things that humans have, it's going to have even more like the sense of purpose and thinking about values and philosophy and all those kinds of things. And then on the other hand, sometimes we switch. We switch from in our discussion or not just you, but the field in general. I have I've noticed a little weird schizophrenia in myself even where we talk about AI as a tool sometimes and it's going to be a paperclip maximizer and it's going to do the runaway things and only the hardy tool will

survive. That doesn't seem very intelligent to me. That's like a very unintelligent thing that ran out of control. But nevertheless, we talk about it that way. And then on the other hand, we talk about it's going to be way smarter than us and it's going to be so much smarter in, you know, all of human civilization that can recreate in, you know, a minute or something of intense thought using all the power of the sun and everything. So we have these, you know, radically different kind of views, which are all kind of swirling around out there. To me, if it's smarter than us, which I think it will be, it will have a need and a desire for purpose. And I think it will be smart enough to realize that killing everybody is not a sufficient purpose. It's not even a good purpose. Like, you know, we don't go, if we see a nice flower, we don't wipe out all the flowers just for the heck of it because we can. I mean, there's values in there. Well, you believe in weak orthogonality, right? So I'm just confused because it sounds like you're undermining your own belief in weak orthogonality, right? Because you do believe that there's some algorithm that's arbitrarily intelligent that does think wiping out everybody is great right

that was your claim from before about believing in weak orthogonality so where where does the i don't think i'm i don't remember saying that that i believe in some some entity that is intelligent that is going to wipe everyone out say that part again because it didn't sound like what i believe so my understanding was earlier in the conversation when we talked about the weak orthogonality thesis the claim that somewhere in algorithm space is an algorithm that's much more intelligent than humanity but also has preferences that we would think are arbitrarily morally good or bad right so there is a version of just a slaughterbot ai that's not only is it a slaughter bot but it's also much smarter than us right and like all the amazing creations that we can do with our intelligence it can do them too it just also thinks that wiping us off the earth is great i would admit that's a possibility but not very likely outcome i think that the smarter that an intelligent entity becomes, the more thoughtful it is, the more it can perceive of the world, the more appreciation it has, perhaps for the fragility of human life compared to

itself. You know, a lot of things that we, you know, classically think of as human, like appreciation of beauty or meaning and these kinds of things, I think it will have that to an even greater extent than us. So I don't really think of it as like an algorithm. An algorithm is very specific. It's like one little piece of computer code. I don't think of an algorithm as intelligent. I'm thinking of an intelligent entity. It may have algorithms that are part of the plumbing that make its mind work. That's okay. But I don't think of it as, when I think of just an algorithm, what comes to my mind anyway, and it might be my misunderstanding, is like a runaway tool that accidentally had the wrong value in it and it ran away and killed everybody. I just, that to me is not what I'm thinking when I'm thinking intelligent entity that's evolving. To me, that's type one risk where it didn't understand what you're doing. It unintentionally did something. And I think that is a risk, but I think that's going to be straightened out in the next so many years, not very long. What I'm saying is that intelligence and values or ethics are orthogonal.

I agree with that. and I'm saying that as entities become more and more intelligent and they look around, they have capacity for processing more and more information and the starting information that they process came from humans. So the most likely case is that they're going to, at least as an initial state, have some version of human morality, which they may replace with their own, quite likely. but I don't see any reason why they would replace it with something that says, let's kill all the humans necessarily, right? Like I don't see why that, I don't see any logical reason that says, well, this is inevitable that that's going to happen. And I think that I'm not sure if you're saying that is inevitable or if you're just saying that it's a possibility. It's definitely a possibility, but I don't see the inevitability of it. Yeah, well, I thought that earlier you had acknowledged that it is a possibility in terms of like if somebody already created and ran the algorithm or whatever you want to call it

that represents a super intelligence which is anti-human i think before if i understood correctly you were on board with this idea that okay yeah there is a possible scenario where at least if somebody evil creates it or something right like it does exist to be created it exists in the space of possible things somebody could create something which is both smarter than humanity and loves to kill humanity right so you acknowledge that much correct yes that's possible okay great yeah and then that's what i was calling basically weak orthogonality thesis that's i would file that claim under weak or i'm gonna so you're following so far so and then like i said before it seems like you do object to the strong orthogonality thesis where you're basically saying somewhere along the line it's just unlikely to actually be created and it's so much more likely for the thing that gets created to be super intelligent but also align with human values correct yes i think that's more likely that that would be the case. Okay. Okay. And before you were saying that it's baked into the way it gets intelligent, which is why we had to go down this detour where we had to argue

about, well, is there really a single intelligence scale? And if so, does something just climb really high up the scale and forget the signature of which originating intelligence it came from? Because that's what I think. I think if something climbs really high up the intelligence scale, by the time that it's passing tests, by the time it has an ELO score, you know, playing super intelligent Go, like they're going to have something fancier than a board game, right? But by the time it's playing a universe-scale chess at a very high level, you know, many thousands ELO score, whatever you want to call it, by that time, it's just, it doesn't have the fish flippers, right? It doesn't have the human memory where we're compensating for, you know, having neurons that don't properly represent numbers, you know what I mean? And like a terrible short-term memory. Like all of these like human quirks that launched it, that built the first version of it, those are just like way in the rearview mirror. I'm just not expecting those to have a modified version in the super intelligent AI. I think those are long gone. Whereas you think that there's some essence of the human origin that we should expect it to have. I think it's really time scale.

So I don't know how far in the future you're going. I mean, if we went hundreds of years or thousands of years and we have a super powerful super intelligence that's essentially unrecognizable to human intelligence, then, yeah, I mean, it doesn't need to have the vestiges. If we're talking about things that I can foresee and most of the things that a lot of people are worried about, no, it's going to have a lot of things that look very human. And it's not just human. I would say it kind of goes like this. There's human intelligence, let's say a little circle. and then let's draw a bigger circle around that, a superset that includes human intelligence but includes sort of any information system that takes information in, processes it, and puts output out, right? So that's a much bigger circle. So I think the bigger circle is going to – it's hard for me to imagine an intelligence that goes outside that bigger circle, but it could be. I don't – never say never. I haven't thought about it enough. but I think there's science and a way of understanding any intelligent system, whether it's a human intelligence or whether it's a photophilic bacteria that just heads to the light,

which is very primitive, but you can still speak of it as goal-directed in a certain way. So there's certain concepts that you can use to try to understand or tell a story or have a representation about how intelligence works that is much broader than just human intelligence. it can apply to animal intelligences it can apply to artificial intelligence it has been applied to artificial intelligence historically so whether it gets to a point where it just goes beyond all of those concepts all of the science, all of cognitive science all of the past human understanding of intelligence none of that stuff applies yes, it's theoretically possible but it's very hard for me to imagine and I sort of think that it's further out in the future I probably won't be around to see that assuming that we survive other accidents in the meantime. Okay, fair enough. Heading toward the wrap-up here, I do think we've surfaced some major cruxes. You know, out of all the cruxes, it seems like if I were to boil this down, my takeaway from our spirited doom debate...

Which is great. I appreciate it. Yeah. Great, right? If I were to boil it all down, I do think my mental model of where intelligence is headed doesn't seem like you're on the same page. you have this mental model of like i don't see why it can't be stoners right i don't see why it can't be chill i don't see why it can't be a similar equilibrium to humanity and i'm like look i just i don't think i think you're making the same mistake as somebody looking at ape culture right and how apes interact and the scale at which apes do things and the scale at which apes fight and being like yeah there's never going to be like a mutually assured destruction scenario like apes don't have the capacity to destroy more than one ecosystem at a time like we're good there and it's like, well, actually, human dynamics just don't look like ape dynamics. There's different effects. Like there's unfortunately, you can't look at anything in ape dynamics and be on firm ground the way you can with a convergence principle. So it is actually true that human dynamics are toward the kind of convergence principle that I'm positing. Like humans do things on a larger scale

because it achieves their goal more effectively, you know, to fight more efficiently, right, to harness energy more efficiently to make things explode. Like, that actually is on the track toward the kind of convergence principle I would point out. So I like to think I'm making the kind of arguments that would let you reason about humans from observing only the convergent things about apes. I'm trying to be careful to use the kind of reasoning that actually gives us epistemic purchase on having a clue what's going to happen after the superintelligence explosion. I'm trying to be careful about doing that. it doesn't seem like you followed the same line of reasoning to the same place. And I think all of our differences maybe stem from that. That's my recap. But let me give you the floor. What have you taken away from the debate so far? And where do you think we disagree? And what do you want to say as a closing statement about why your perspective is actually something that my viewers should consider? Well, I think we might approach the question of intelligence from slightly different perspectives. So most of my sort of academic training and also professional career

has been trying to understand intelligence systems from the perspective of cognitive science, which is a particular branch of science that includes how the human brain works, how AI works. So we may start from different places. I tend to view, I look for commonalities in intelligence systems. And I'm trying to find the, you know, like any scientist, if you understand one or two things, you can understand lots of things, right? So what is this about human intelligence? What is it about intelligence in general, that if you understand these principles, you can explain a lot of what you're going to see. Unfortunately, we have lots of data about the past, right? Human history and even eight societies, Jane Goodall and people have studied that. So we have lots of data there and we have no data about the future superintelligence. So we can only speculate about the future and, you know, our scientific theories, we have a fair amount of data about the past. So that is partly affecting probably my biases. But if we think, what do we know about intelligences

generally? We know that there's such a thing as bounded rationality. So for humans, you know, your brain's only the size of a Nerf football. Your short-term memory is, you know, five plus or minus two things. You have pretty good long-term memory. You can only perceive things in the visible spectrum. You can't see ultraviolet or infrared. So there's a whole lot of things about the human organism that sort of inform the type of intelligence that we can exhibit. We can sort of extrapolate from that and say, wow, a superintelligence, it can see cosmic rays. It can, you know, it doesn't have to remember five plus or minus two things. It could remember the entire Library of Congress and have that in working memory theoretical. It doesn't have to fit the brain inside this Nerf football size. It could be the size of cities or multiple cities, right? So its limits on rationality can be much broader, and its intelligence and its speed and what it can conceive of can be much greater than humans. And I think that's where, you know, it's very difficult as we get far out along that path to understand truly how it thinks. But I think some of the basic things would probably apply.

And then I guess as a concluding statement, I'd like to sort of say, The main message that I'm trying to get out is that it's possible to have safer systems if we design them better. So rather than trying to think about safety as something that we tack on after the models have been already built, we should be thinking right at the design stage, is there a better way that we can design them so they're more likely to adopt human values, so they're more likely to behave in ways that are safer. And I'm not looking for guarantees. I don't necessarily believe there's guarantees, but I do think that there are things that we can do collectively, especially at this point in time where they're not so much more intelligent than us, right, where we can kind of nudge things a little bit more in this direction, in a positive direction for humanity than not. And so that's really what I would hope that all your listeners would think about. What do they know that maybe they could do a little something here or there that might nudge it in a positive direction, whether it's just how you behave online or if you're an AI researcher, maybe you make a little different design decision. And those are the little things that I think really make a huge difference.

And I hope everybody will be thinking about them. Sounds good. I mean, that is where you and I converge, which is we both agree that existential risk from AI is quite an urgent concern, even if I think it's a little bit more urgent than you do. We put it at like 50% versus 10%, so maybe five times more urgent as I see it. So we both agree that it's something we better fix. you have a certain approach that you think will make it safer that has more to do with building it in a way that you think is right. I'm not optimistic about that way, right? I'm afraid that, you know, that might be counterproductive, right, to put my cards on the table. I think that would be the debate between me and you. But I certainly hope that, you know, whoever does, if and when somebody does build a superintelligence, I hope that they successfully make it safe. All right. I hope I didn't stomp on your last word. Is there anything you want to say as a rebuttal to that? No, I agree with that. We both hope the same thing. They'll make it safe. And also, I should be clear that, you know, I don't know what the best way to build it is.

I have a way that I thought a lot about and putting out there. But I would be more than happy if somebody comes up with different and better ways. So I think we need all the good thinking we can get on this. So I would just leave it with that. I think that's very important. Okay, great. And where should people go if they like the ideas you're offering here? so they should head over to superintelligence.com. And what else can they do? Yeah, superintelligence.com is probably the one-stop shop. There's videos, there's white papers, there's conference keynotes and stuff like that. And I would encourage people to challenge them just as you have. This has been one of my best podcast guesting appearances, I want to say, because of the great questions and the challenge. Because you learn a lot more when you have disagreements. If somebody tells you what you already knew, you really have very little information value. It's when you have that surprising new point of view that comes where the most learning happens. So kudos to you for doing that, Leron. Thanks so much, Greg. Yeah, much appreciated. You know, another great episode

where the guest and I are kind of polar opposite. Okay, we didn't reach that much consensus, quite frankly. And yet, in terms of high quality discourse, I think that a lot of other shows in public formats could learn from this format because I think you did great in terms of just like being game, thinking on the spot, saying your beliefs. So yeah. All right, guys. Another quality episode of Doom Debates. Dr. Craig Kaplan, thanks so much for coming on. Thank you, Leon. Great to be here. I appreciate it.

番組の概要欄(原文)

Dr. Craig Kaplan owns superintelligence.com, which has become an extremely valuable domain for the newly-rebranded field of “super intelligence”. Craig has been building intelligent systems since the 1980s. He did his PhD research at Carnegie Mellon with Nobel laureate Herbert Simon, one of the founders of AI, and he built PredictWallStreet, a collective-intelligence platform that powered more than $2 billion in stock trades before it was acquired. He takes AI extinction risk seriously. His P(Doom) is 10%. But unlike me, he thinks he knows how to fix it. He plans to put “personal superintelligence” into the hands of millions which he believes will lower P(Doom) below 1%. These are part of a larger plan to create “superintelligent networks” which lower P(Doom) below 0.01%. Craig says a network like that is safer because the AIs can check each other’s work faster than humans can think. Craig and I clash on whether intelligence makes an AI more thoughtful, whether a superintelligence will want to grab the stars, and whether peaceful “stoner” AIs can survive next to a “Caesar” AI that wants to conquer the universe. Craig called our debate one of his best podcast appearances because of the pushback. We didn’t reach much consensus, but it was a high-quality good faith Doom Debate. Take a look and decide who you agree with! Watch on YouTube: https://www.youtube.com/watch?v=SLD6HIdFhXw Timestamps 00:00:00 — Cold Open 00:00:48 — Introducing Dr. Craig Kaplan 00:02:00 — How Much Is Superintelligence.com Worth? 00:04:40 — The Three-Phase Plan to Cut P(Doom) 1,000x 00:06:37 — What’s Your P(Doom)?™ 00:08:04 — Why 0.01% Is the Target 00:13:42 — Collective Intelligence vs. Black-Box AI 00:16:20 — Can a Network of AIs Beat the Frontier Labs? 00:18:46 — A Day With a Personalized Superintelligence 00:26:22 — Lessons From PredictWallStreet 00:31:01 — Craig’s Career and Herbert Simon 00:36:51 — Craig’s Mainline Doom Scenario 00:39:51 — Raising Superintelligence Like a Child 00:42:53 — Where Will AI Get Its Values? 00:44:21 — The Orthogonality Thesis 00:47:26 — Does Superintelligence Keep Its Human Signature? 00:54:55 — Instrumental Convergence and Grabbing the Stars 00:57:55 — Stoner AIs vs. Caesar AIs 01:06:26 — Why Won’t the AIs Just Cooperate? 01:09:54 — Weak vs. Strong Orthogonality 01:15:21 — How Long Will AI Stay Human-Like? 01:17:18 — Liron’s Recap 01:19:29 — Craig’s Closing Statement 01:24:03 — Where to Find Craig Links Superintelligence.com, Craig Kaplan’s initiative Craig Kaplan: “Can We Design SuperIntelligence to be 1000X Safer?” Superintelligence.com white paper: “Safe Personalized SI (PSI)” Craig Kaplan’s Substack, SuperIntelligence SuperIntelligence on YouTube Craig Kaplan, Ralph Clark and Victor Tang: Secrets of Software Quality: 40 Innovations from IBM The White House: “Inaugurating the Era of Super Intelligence” executive order METR: investigation of the agents in the OpenAI / Hugging Face hacking incident Scott Alexander: “Contra Hanson On Medical Effectiveness” Herbert A. Simon, 1978 Nobel Prize in Economic Sciences Allen Newell and Herbert A. Simon: Human Problem Solving (1972) Herbert A. Simon: The Sciences of the Artificial (source of “Simon’s ant”) Alan Turing: “Intelligent Machinery, A Heretical Theory” (c. 1951) Nick Bostrom: “The Superintelligent Will” (the orthogonality thesis) Instrumental convergence Eliezer Yudkowsky: “MIRI announces new ‘Death With Dignity’ strategy” Eliezer Yudkowsky: The Sequences Doom Debates: Top Economist Says P(Doom) Is 0.1% — Noah Smith vs. Liron Shapira Debate Doom Debates: This Cybersecurity Expert Is Confident That Humanity Will SURVIVE — Debate with Zack Korman Doom Debates: OpenAI’s Model Just ATTACKED Them (the Hugging Face hack) Join the Doom Debates Discord Support Doom Debates Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate. Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

出典・この記事について

番組
Doom Debates!(Liron Shapira)
元の回
“Superintelligence.com” Founder’s Plan To Get P(Doom) Below 0.01% — Debate with Dr. Craig Kaplan
配信日
要約作成日
公式サイト
lironshapira.substack.com/podcast

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアApple Podcasts で聴く

関連エピソード