AIバブルは崩壊するのか？ロバート・ライトのNonzeroとのコラボ回

Nobody's said the word AI winter for a couple years. AI itself is doing so much AI research, I think we're done with AI winters. I think it's just AI summer till the end now. Man, this whole thing is moving very fast, that's for sure. Stock market has had AI jitters the last couple of weeks. Google took a huge hit. It's crazy that people are talking about an AI bubble, and then the P-E ratio is like a totally ordinary P-E ratio. Google, I don't think it's a $3.4 trillion company. It's easily a $10 trillion company yeah i don't know it's a drawn there were so many data points from the anthropic s1 this is the final nail in the coffin of the etzitron camp they're printing money like serving the models printing money i don't think i have to be nostril honest here to be like hello leron hey bob great to be back with you great to be back with you this is another episode of Non-Zero Doom, a review

and assessment of recent AI news and a, I would say, high synergy collaboration between a Non-Zero podcast and a Doom Debates podcast. I'm Robert Wright, host of the Non-Zero podcast, publisher of the Non-Zero newsletter and author of the recently published book on AI Godcast. And you are Leron Shapira, proprietor of the pioneering, increasingly famous Doom Debates podcast. And a long-standing member of the Doom community. Is that a fair thing to say about you? Yeah, since 2007, reading Less Strong, Elias Yudkowsky, surprisingly long-duration rationalist at this point. Yep, yep. Congratulations on sticking it out. You were finally vindicated. Yeah, great. We think. We hope not, actually. We hope not. Or we hope that you're vindicated in the sense that your warnings are heated and we do the right things to make sure doom doesn't happen. So we, yeah, so much has happened since we last. We need to do these maybe a little more often

because if you let a couple of weeks go by, just too much stuff happens almost to keep track of. The pace of activity is not so down in AI, I would say. And there's so many things to talk about. There's, you know, stock market has had AI jitters the last couple of weeks, notwithstanding news from Anthropic that I would say is kind of auspicious for their commercial prospects if you look at their filing for the IPO. Google took a huge hit, but now since that hit, which was about 36 hours ago, it seems to me we've seen some promising news from them. meanwhile my own my own nominee for in a way most important story and this is kind of a sleep or kind of a dark horse but it's what I call social science judgment day is coming but what would you say is the thing to focus on first and foremost yeah like very high level right like

what is the vibe of the last couple weeks so for me it's just capabilities increase you know like the rest is noise if the AI keeps getting smarter and smarter and more capable, that's going to be the reason why big things happen soon. And when I look at the last few weeks, I am seeing more scientific breakthroughs. We can talk about that. I am seeing the AI companies talking about increasingly ambitious plans. I am seeing my own cloud code and codex harnesses getting more and more capable. It's just insane. My subjective perception, you said that I've been vindicated as an AI doomer. Well, one thing that's been a little bit surprising to me is that it's not quite a rapid FOOM. You know, it's not happening in the course of minutes or hours. We should say for my audience, FOOM is, well, you tell me, so it's rapid takeoff. I thought it was an acronym for fast onset overwhelming mastery, but apparently not. So, yeah, I mean, I first saw this around 2008 in the context of Aliazhar and Robin Hanson debating the AI trajectory.

And there was literally a post called AI Go FOOM. and I think there was an image or at least a description of kind of like a nuclear mushroom cloud type of situation, just like basically a foo's of positive feedback loop and it's like the sound that it makes, but, you know, maybe there's a backronym attached to it. Maybe, yeah, I think it's a backronym. Anyway, go ahead. Sorry. Yeah, so I am kind of an orthodox Yukowskian. You know, I haven't branched off to the later sects. I think that he was originally correct about most of the things that he said, believe it or not. and one of those things that was a Fuhm prediction, he was saying, hey, when you start seeing general intelligence at the superhuman level, it is going to be a cascade where the intelligence helps you develop more intelligence and there's a good chance that that cascade is going to be unintuitively rapid. It's just going to happen so fast. And there's this famous 2014 Tim Urban illustration where it's like the train's coming and people are looking like, oh, the intelligence train here, and they whip their head around like, wait, where did the train go? So, you know, that idea that it, like, really accelerates faster than we expect.

Now, I actually think that characterization is not far off the mark. I think the amount of progress that we're seeing month to month is incredible, and we are in this insane, super exponential curve, but it's not happening in, like, one day. I can kind of live through it at a somewhat human pace where I come into work today and I open up my latest AI tools, and I'm like, oh, wow, they're clearly smarter than yesterday, but only by, like, an increment that I can measure. Or like, oh, wow, this math proof is clearly a little bit more advanced than the one that dropped a week ago. And yet human mathematicians aren't quite out of a job yet. So the foom, from my perspective, the foom is just taking like a little bit longer than a maximally rapid foom. Yeah, and it's what you expect because there is a positive feedback system. I mean, in the, you know, the singularity scenarios, the driving force is that the better the AI gets, the more it helps the next generation of AI get born. You know, and you move toward so-called recursive self-improvement where in the strict sense of the term, there's no human in the loop.

AI just making better and better versions of itself. The positive feedback loop definitely seems to be happening. You know, could there be some wall? Maybe. But so far, they have found enough kind of new techniques to add, like the whole reasoning thing, that, you know, I don't see any reason to expect. And when you keep in mind that supposedly OpenAI and Anthropic have models in the wings, waiting in the wings, that are next generation models, I think, from totally new training runs, as I understand it, it's happening fast. and the symptoms seem to me everywhere. You've got the cyber apocalypse. I mean, you had the PHP thing last week. I guess this Binance hack may not be AI-driven in a fundamental sense, but even if it was a social engineering hack, I've got to think the AI helped them do the research for what the social vulnerabilities were.

And then, yeah, the math stuff. And by the way, I just learned that these math breakthroughs, as kind of deep and impractical as they sound, some of them actually will themselves have an accelerating effect on the development of the AI. They actually are practical value. And, you know, yeah, it's... Can I make another high-level observation? Mm-hmm. When was the last time you even entertained the idea of AI winter? I remember that phrase. So that was when you would go a long period without discernible AI progress. When was the last AI winner? I guess it was almost pre-deep learning, right? I mean, the deep learning thing really kind of blew up with AlexNet, the image recognition thing in around, what, 2014? That's when it really, you know, rose its head and got a lot of attention. Exactly. So the 2010s, especially heading into, like, 2018 with GPT-2,

that era, you could call it AI spring, I guess, right? Like it was foreshadowing the AI summer, which I guess is right now. But yeah, there was kind of an AI winter. I mean, I grew up studying computer science. I was always interested in AI, and I was always a believer that AI was eventually going to surpass humanity. But I very much remember, I mean, we both lived through this, the era where it's like AI, you know, even character recognition was always kind of janky, right? Speech recognition kind of worked, but it wasn't quite worth using, right? It's like, We lived through decades where these things are just like, ah, screw it, you know? And I had many hours of experience, many decades of experience, I guess, handwriting code that is no longer needed. But I would, like, stare, you know, I would be a manual processor of text, right? I'd be fiddling with all this data, and now I don't need to do that anymore as of, like, the last year. So, but AI winter was, like, this long period, and there's been multiple AI winters, right? Because we've had previous bursts, you know, like, oh, my God, computers can find, like, shortest path algorithms. That was kind of, like, the first AI summer, right, where the algorithms were more structured, and like, oh my God, computers can play chess.

Okay, but now they're stuck on like moving robots around. They're stuck on doing fuzzy things. Oh wait, deep learning kind of works, right? So we used to have these AI seasons, but the reason why I think nobody's said the word AI winter for a couple of years, like after 2023, people are like, okay, this is a nice day of summer. We're probably gonna head into AI winter, but it's like, oh, AI winter is canceled. And I think the reason is now because the AI itself is doing so much AI research and there's so many cascading effects. Like we've cracked enough of the problem wide open that I think we're done with AI winters. I think it's just AI summer to the end now. Yeah, and I mean, even if you suddenly had an AI winter, even if there were no new generations for a couple of years, we would have our hands full, I think, assimilating the consequences of what we've already done. This stuff is just starting to work through the economy and all kinds of professions. I'll get to this social science judgment day thing in a while. But meanwhile, you know, we're also seeing progress in dimensions that aren't even getting that much attention. Like you called my attention to a couple of new kind of

amazing video applications. And then Google, I think about the time the stock market was losing faith in Google, like a couple of days ago, they released this GD4 thing. And what that is, And this has important consequences for Google's, I think, long-term financial prospects and very positive ones. So it's not just video generation. I mean, Google has always been near the forefront of that. It is interactive video. So you say, you know, kind of, okay, I want a world, you know, I want to see this world with these animals or whatever. It creates it. But then it's like a video game. You can walk through it. You can turn your head and look around and you can add new things on the fly. And you can say, now make a bird fly in, you know, and in my in my my book, I talked about, you know, how virtual reality was going to become a like put me in a world where thing.

and you could make the characters as autonomous as you want, as unpredictable as you want. And, you know, I mean, to get back to Doom, well, let me tell you, so Dario, I'm sorry, Demis, Demis Asabas, who's still no longer CEO of DeepMind, but, you know, is nominally, I guess, chair and certainly still relevant. He says, very excited to share, Jeannie, for our incredible interactive world model. bringing together real-time physics, reasoning, and audio simulations. Can we have to do a screen share because I have the video up in? Do you want to try to use that feature? We, I mean, as fast as we're going to try to get this up, honestly, I'm not sure about the wisdom of trying to integrate that. I mean, it's Friday. If you, I mean, you can do one and we can probably, well, we can either. Let's try how it's on a screen share and maybe it'll just embed the video in my video and it won't create any extra work.

Yeah, go ahead and give it a shot. Okay, so there's like the famous painting, I guess, Sorry Night, is it, right? And he transformed it. So now you're in the 3D world of the Sorry Night painting and there's like people walking around. It's kind of like Harry Potter where the images come to life. It's actually very similar to that. And you are controlling the movement. That's the key thing. And you can say on the fly, now make this happen. Now, here is my very, my quote tweet of Demis. I said, we'll see how deep the rabbit hole goes. I said, that's an apt reference since this technology will bring to yourself virtual reality that draws us toward the matrix pods and can prove so alluring that we enter them voluntarily. This is a metaphor. I don't mean literally stuff ourselves into tubes. But I do think in one scenario, we kind of, it isn't that, you know, as we surrender agency to AI, if we do that, it isn't that it has to lock us up.

It's that we choose to kind of pleasantly anesthetize ourselves. In my book, I have this line that's probably not original, I don't know, but the path to the Matrix leads to WALL-E, you know, by which I meant the Pixar movie in which people are kind of, They kind of anesthetized themselves via self-indulgence. Yeah. And, but in any event, I think this is big. And also, by the way, it has implications for robotics. I mean, the, you know, I think Google is in a good place to exploit the robotics revolution, partly because they've always, for a long time, they've been at the forefront of this kind of thing. um yeah you know i remember they said something uh about six months ago when they kind of disappointed people with like the original gemini it was a 3.5 or like heading toward gemini 4 they had this google io where like i was personally excited but then they kind of disappointed everybody and they started falling behind but i remember one thing that they said around that time

was they're saying hey we're investing so much in world models we don't think just the traditional language model approach is that good so i think what might have happened is that they kind of took one step back in order to take two steps forward. So this is just my speculation based on reading the tea leaves, but I think the kind of world modeling that you're seeing in G4, somehow they incorporated that secret sauce to then give themselves a boost and cash back up to the frontier, and maybe there's actually a chance that now they're going to beat OpenAI and Anthropik right now because they've got some of the secret sauce. Just a potential speculation, which maybe that's a good segue into why I think Google stock is undervalued. You know, I have evergreen subject. One more thing on that. And it turns out that the language models play a bigger role in this than I had appreciated. It isn't just translating, you know, your requests into some language and then stepping aside. Apparently, they are playing a role in orchestrating the coming video and figuring out what would make sense given what you've asked for as the next step.

So it's a pretty complicated machine that is being built here. yeah yeah now go ahead now i know uh full disclosure right you you bought google stock i mean i i'm involved in all of these companies i'm sure but i don't fool around and buy and sell um you know i've got i have a mutual fund but but you on the other hand play the market right that's right yeah not very successfully i i must admit i don't do successfully uh but i'll tell you my high level thinking okay my high level thinking is is what i call the icarus curve i think i mentioned it before it's the idea that hey the economy is really going to rock it up i'm actually very bullish on how much it's going to rock it up before we quote unquote fly too close to the sun like it's great to fly higher but i think our wings are going to burn off at some point which basically means that we're going to lose control so yeah the ai is going to be amazingly powerful it's going to be so useful it's just going to stop being useful for us and start being useful for whatever it's been on a roll doing. It's just going to sever the link to us, and it's

just going to fly off. I've lost control of a drone before, and then it's over on some roof. That's kind of metaphorically what's going to happen with AI, in my opinion, unfortunately. But anyway, economically speaking, for as long as we have an economy and the lights are still on, I'm actually very bullish. I think companies like Google, who the market cap today is a measly 3.45 trillion. That's Google's market cap today. It's been like 4.4 trillion or something at the peak. So it's actually down from the peak a few months ago. I don't think it's a 3.4 trillion dollar company. I think it's easily a 10 trillion dollar company in the next couple of years. So I think it's undervalued right now. I think Anthropic and Opening Eye are probably 10 trillion dollar plus companies pretty soon. I think SpaceX and Tesla and Waymo are potentially close to SpaceX definitely. I basically think all of these companies are undervalued for the simple reason that I think the whole economy, the parts of the economy that are still relevant to the future are going to be 10 times bigger. So I think today's economy is like the seed of the future economy. There's so many resources that aren't plugged

into the economy because we haven't had enough intelligence to do it. We haven't done it seriously using what I call grown-up intelligences. You ain't seen nothing yet when it comes to what an economy could look like when you have super intelligence powering it. So that's my general thesis. I'm just a bull overall. And then I look at Google, And I'm like, the last thing you should be doing with Google stock is having you go down. Like, okay, at least keep it the same. So I think it's crazy that there's a dip with Google. Like, people don't even think it's worth $4 trillion. Because even coming back down to Earth, okay, forget about the future. Even if you look at the present, it's like, hello, YouTube, okay, what is that, chopped liver? That's like easily a trillion right there. Okay, the stake in Waymo, stake in SpaceX, stake in Anthropics, okay, there's that. Hey, Google search, okay, fine, Google search is dying. Maybe it's not worth a trillion. Maybe it's worth half a trillion. Okay, and then you've got Google Cloud. Oh, that business is absolutely rocketing. like that's easily worth one to two trillion okay and there's like i'm sure there's a couple things i haven't mentioned here that that companies right competing with nvidia right so if nvidia is five trillion i just i'm making the chips so google has their own independent chip supply chain like

head-to-head with nvidia right that's so many of these companies are using so it's all of these amazing businesses packed into one and you're telling me the sum is 3.5 trillion it just doesn't that up to me. Now, the price-earnings ratio is not that high. It's not exorbitant. It's not like some crazy thing. It's 15, right? Which is hilarious to me. I don't think people are talking about an AI bubble. Well, that's what Google's saying, right? So I don't know if it... It's a crash, maybe, because they lost like 20% of their value this week in one day. I have seen somebody saying it's like 20-something, but you're right. So there's different ways to measure it. So I don't even know if the 15 represents forward or backward. I got to look into this, okay? But whatever it is, okay, it's crazy that people are talking about an AI bubble and then the PE ratio is like a totally ordinary S&P PE ratio. Like you would not know it's at all just by looking at that, but of course there's arguments like, well, because search is dead, like that profit is never coming back, right? So of course there's the bare thesis, but I think it's crazy. And by the way, there's search revenue in the third quarter. This is the announcement that I think more than anything, Tank the Stock had grown 12% year over year, but that was a lower growth rate than the second quarter,

which had been 17% year over year, Well I think like just a kind of a gradual plateauing in search is not a bad outcome because obviously that particular business model is challenged by AI and they going to have to probably figure out new ways to milk more ad revenue out of it But the main thing they need to do is just hang on to people using search as AI and do various things with them in terms of moving them into new apps and so on. And as we just said, I think Google has a lot of potential. Now, I will say the other thing that's been going on in the background is Google, they kind of fell off of the frontier. A year ago, we were talking about three frontier companies. Now we've talked about two. And then here's the thing I want some of you to explain to me, what happened over the last couple of weeks, okay? So first, they introduced Argon, their new fresh training run model.

And the reviews are kind of positive but slightly lukewarm. Everybody's saying, well, it's kind of in the game. It's close to the frontier, but it's not on the frontier. And then like yesterday, two weeks later, they introduced Carbon, which has appreciably better eval results than Argonne and at the same token cost. And it's clearly more impressive, including in places where Google is behind, like, you know, agentic autonomy and stuff. And I don't understand why they introduced Argonne in the first place if they had this. I mean, the typical pattern of these companies seems to be you introduce your killer model with a jillion parameters, and then you introduce a kind of distilled and refined version of it that's almost as powerful and cheaper. And so you get this kind of, you know, opus, sonnet, whatever, you know, hierarchy. This seems to be unusual, what Google just did. I don't totally understand what they did, but it does suggest to me that they're closer to being back in the game

than we thought two weeks ago, unless I'm misunderstanding what's happening. Well, my take is just the reason they released Argon is because of the new one, Carbon. It's not quite ready for release for whatever reason. My best guess is they just need to give capacity to it. Like, it's too inefficient. And they could release it to, like, 1,000 people, and then, like, 50 million people would be like, when do I get it? So they're just holding it off. That's my best guess. And I think they just, they knew Argon was, like, a little worse than open-air ananthropic, but they're like, look, it's either Argon or nothing for the next month, right? So they gave us Argon, and it's like, okay, they're still in the game. Like, I think it's fine. My best guess is that Google is going to catch up, because it's like, Google is very well-resourced, right? It's like they have all these other businesses, they have all this infrastructure, we talked about the chip supply line, they have some amazing talent, like I'm sure there's been some brain drain, but they still have truly amazing talent, they can get their top people on this, and the incentives are so massive that my best guess is that they will be head-to-head on the frontier, and in some cases, even lead forward, which, again, brings me back to the stock, but I'm not trying to show my book here.

I did look up why some people say the PE is 14 and some 24. So the trailing PE is 14.7, which is good. Like low PE means it's like attractive, basically, because you're not paying as much for the same profit. So the reason it's only 14.7 trailing is if you include their windfalls from like the appreciation of SpaceX and a few other stocks that they hold. if you exclude all of that, then you're paying 24.2x trailing PE. I think the truth is somewhere in the middle. I don't think it's crazy, but yeah, SpaceX, they still hold the SpaceX. It's still going to keep appreciating. So whatever, let's call it a 20. 20 is also their forward PE. So okay, it's trading at 20 PE at the beginning of the freaking singularity. The singularity is known for its growth potential. So generally, you'd want to go higher than 20 PE. Yeah, that's the fun part of the singularity. and then the rest comes exactly and by the way there's another factor though worth adding into the analysis because there are

so many factors the other factors like look it might crash like it might the whole economy I'm the one saying as much as anybody right like I think we're close to potentially a collapse people are talking about hey it's time to prep right the internet's going down we should talk about you know black October or whatever you want to call it so as great as the bull case is when you're doing an expected value calculation you do have to also average it in with the bad scenario, which I actually think is incredibly likely. So I can talk about, yeah, 20 PE, it should be 40 PE. Okay, but it should also be 0 PE, right? Because the lights are going to go out. So actually, you know, 20 PE is actually very calibrated. Yeah, now I will say on the other side, Google has shown signs of suffering from, you know, big corporation disease being a little slow and clumsy. And I mean, they have bled some real talent, you know, known Shazir. They paid a jillion dollars to get him back after he went and started charactering it you know he's like the attention is all you need guy as much as any other person is you know the transformer guy they paid him like literally i think 1.5 billion to bring him back he stays there 18 months and

then goes to open ai right and uh so i don't you know that's that's they're doing something wrong when that happens apparently one one one story was that he wasn't getting as much compute as he wanted and look everybody's compute constrained and google's got to make their choices and look if they're putting compute into these world models and things that's not necessarily a bad bet i know what's going on but but they do and and i i would also say the landscape is so fluid i mean any of these companies i mean i think we can talk about the anthropic uh what do you call an s1 the thing they filed yeah yeah like yeah and they look very healthy and and honestly i think they and open ai I are both in a better place than I realized even weeks ago for reasons I can explain. But the landscape is so fluid, there are things you just don't know. Like, as far as the current anthropic revenue stream, depending very heavily on the use of frontier models by businesses, you know, that could, the gap, you know, could get commoditized faster than, you know, there are a lot of variables.

There's regulatory possibilities. And so, you know, I don't any one company could falter. But, man, this whole thing is moving very fast. That's for sure. And faster than I would like. And I would like a government mandated slowdown. And I honestly we can get into this maybe later. I'm a big international governance guy but I'm not sure that a brief slowdown really demands international coordination if it's true that the Chinese are relying so heavily on distillation then you know they're not going to catch up all that fast if you quit releasing new models to distill you touched on a few topics I think we can dive into more on the subject of Google we can talk about what they've been indicating that they feel like... They're basically saying that they don't want to pace. That's the signal that they've been throwing off,

and maybe this has to do with them stepping down. Maybe that's related. But they're basically not saying anything useful about pacing. And I think that, as far as I can tell, there's been what you might call evaporative cooling where the people who disagree might have left. Certainly, I know Alex Turner left, but he had his own reasons. It just seems like it's full of people who think, you know, the Yudkowskians are smoking crack and they should just accelerate and they should just make money. That seems like what Google has become these days. And on the subject of Anthropik, there's definitely some interesting observations about the S1 and the fundamental economics of chips. What do you want to dive into? Well, first of all, Anthropik's got a ton of cash on hand, it turns out. I mean, like what? 200 billion sitting around? Cash and liquid securities. That's right, yeah, it's like a five-year-old company with more cash sitting in the bank than Apple Yeah, and, you know, I think the third quarter showed a little bit of a revenue slowdown

But, I mean, that wasn't in the S1 per se, but we know that that's a fact But, you know, that's like, things are moving very, very, very fast So revenue growth seems healthy, and as I said, it's all very fluid Who knows what is going to happen next? I think the ace in the hole that Anthropic and OpenAI have is increasingly going to be intellectual property. They're using these models that they haven't released, I think, to do research in biotech and stuff. I mean, Anthropic just this week bought a biotech company. They're doing collaboration with big existing pharma companies. And, you know, OpenAI is doing that. And by the way, I can't reveal my source on this, but apparently OpenAI is also approaching companies about paying them large sums of money, very large, to make use of the models that are in the wings, that are unreleased, like the next generation model.

and so yeah and I think both of them the IP thing is huge and this is why the funny thing is a slowdown might actually help them because it may be that there's more value in keeping their frontier models secret and not letting other people use them to develop new drugs and just do it themselves and not just drugs other forms of IP I guess although although Pharma's the leading candidate. And so, you know, that's why there's some people who think they do secretly want a slowdown. Now, you said, you know, Google's not doing much serious pacing talk. Well, nobody's doing much serious pacing talk. I mean, they talk, but they don't. Dario shows no signs of wanting to actually pace. You know, he just talks and talks. And throw up an open idea a little bit. I mean, don't get me wrong, they don't do enough. but there certainly have been some recent flashes of facing talk from both OpenAI.

Sam Allman was tweeting right after Astra, maybe this was about a month ago, where he was saying, we have internally slowed or paused our training because we're not happy yet with the security configuration. And, of course, this was in the wake of Hugging Face and a bunch of other German wiki. That lasted three weeks or something? That's not serious. Yeah, it lasted three weeks because I think the end of it was when they rolled out a bunch of announcements being like, hey, we've got this whole framework now. It's called the offline world from now on. Don't worry about agents escaping onto the Internet because we realize it's not safe to connect to the Internet. It's like, wow, you've invented not connecting agents to the Internet from the 2005 Aliazhar Yudkowsky story. Don't be dumb enough to connect your agents to the Internet. So I guess that's good that they're not connecting it to the Internet, although I was then joking around on Twitter. I mean, I was actually serious. I was saying, hey, can you please give these AIs a public key so that when they do escape on the internet, they can announce in a way that we can all see that they're here, because they can use a digital signature. So I can know for a fact that, hey, there's basically no way to fake the AI escaping on the internet

because only it can have this digital signature because it'll announce its public key beforehand. So it's like, oh, it's a signed message from the AI that escaped. Great, we all know OpenAI sales. But of course, I didn't get any reply from that. Some OpenAI employees did like it, for the record. But anyway, so that's OpenAI talking about, that was their little flash of thing. Can I just say I think it's possible to explain that whole episode of smart PR. They were in trouble. The hugging face thing had not been good for the reputation. A three week pause, to me that's just screening, that's just virtue signaling. I mean, it probably did some good but they needed to do it for their own good. They needed to get the shit under control. That's not a pause. I mean, I'm sorry. There was a researcher I forgot if this was Daniel Kucutelo or some kind of smart former insider like that was commenting on the way that OpenAI is doing their quote-unquote pause or did their pause. He was pointing out, hey, okay, you guys paused the next training run,

but you didn't pause the research, the R&D that you were doing, short of kicking off the training run. So when you do kick off the training run, it's just going to have these new insights incorporated into it, and when the training run finishes, you're going to be in the same place on the curve. So I agree with you that they're really just, you know, this is just pure PR, like on Sirius. Yeah. There's another, yeah, I mean, the challenges that would be posed if the government, if we had a government that was paying any attention and wanted to impose a pause, a brief pause even, you know, kind of mindful that eventually China might catch up or whatever they're concerned about. I mean, that's not my big concern, but it's the big concern in Washington, I guess. How challenging would the kind of legislation or the regulation be? And one reason I ask is that I've heard a little about how, you know, we think of like a whole new training run. You start over, you comb the Internet, you do the thing.

I gather that more and more, I mean, how many times do you need to comb the Internet? You know, I mean, it's true. It's probably changed a little since the last training run. And you could maybe add that. But I gather that they're talking more and more about just using the existing training foundation. Right. That was established by the pre-training last time around and maybe adding a little of pre-training or something. But largely than just adding post-training onto the foundation. So in other words, a training run, I don't know if this is true, but my sense is that maybe a training run is going to require less of a fresh start than it traditionally has. And I just raise that because if you're going to have a pause that's mandated, you need a definition of what a new training run is. And I'm just wondering, are there gradations of refining the old model where at the extreme it amounts to a whole new model generation? but, you know, it's a fuzzy line or what?

It's a great point right now because, yeah, I mean, the obvious thing that people have been saying for a few years is you just detect how much compute is going into the training run and you can tell, okay, this is a serious training run by one of the major players. Like, nobody else ever harnesses two months of compute in a giant data center for the same level of activity. Like, it's got to be Google training the next frontier model. But as you say, that heuristic is becoming less and less valuable. There's different scaling laws you mentioned. and there's the post-training scaling law. When you have more compute, when you're doing reinforcement learning on the thinking that it's doing after it's trained, you can get more intelligence that way. And there's this new scaling law, you know, the swarm scaling law, right? Like research into like, oh, I'm going to have a bigger swarm attack a problem, and I'm going to get more effective intelligence that way. So hypothetically, if the government were to finally get their act together and be like, hey, you can't scale past this level right now, there's a pause on this level, any kind of condition you might want to institute on what you can't do, we're now at the point where the boat is springing more and more leaks i don't think it's fully leak you don't get me wrong it's worth a try like it's not hopeless yet but i absolutely think there's

a way to kind of like limbo over some of these bars like oh you can only use this much computation it's like okay wink wink we did some research like we became more complication efficient we did a bunch of distillation like faster cycles like train a little bit distill a little bit i mean i'm just you know i'm talking vaguely here but i do think that there's a lot of ways now to kind of get through the cracks of any regulation that's based on any fixed numbers. Mm-hmm. I mean, it would be so great if we had a government that was focused on this. And I wonder if, you know, if the Democrats do take both houses, whether, you know, there will be leaders in Congress that at least kind of crystallize some of the issues and, like, pass some sort of legislation that's simple and clear and at least dare Trump to veto it, you know? like, and just have it out there as like a paragon of smart legislation. I'm not optimistic, I've got to say, but God knows we're not going to get any great guidance from the White House anytime soon. I think the key is to just continue to be straightforward the way Jacob Cox did, the

way a bunch of people in Congress did right after he came out and right after Evan Hubinger from Anthropic, who was, I guess, kind of his former boss leading the safety team. All those people were very straightforward, like, hey, AI might extinct everybody. And you even had the Attorney General of Florida being very direct, being like, yeah, you guys say you can kill everybody. Okay, we're going to take you up on your offer to stop you. Everybody was being so direct. I feel like I haven't heard that rhetoric for a while, except Bernie Sanders had kept it up. Credit to Bernie Sanders. But it just seems to be dying down for whatever reason. Maybe it's not the number one most popular midterm election topic. It's not. I mean, the economy is. and whatever will get you a victory in Congress is what the focus will be right now. But I do think after the election, it depends on what the results are, there may be an opportunity because there are people who are smart and sincerely concerned about this in Congress. Bernie Sanders is one. I'm just saying I think whatever the policy has to be,

you can't just write it down in legalese where you can't do it with just the letter of the law. Like, oh, the letter of the law is here's the computer requirement. No, it just has to state our intention of being like, hey, we are scared that this thing is going to run away. Stop making something that's threatening to run away. And there has to be a lot of judgment going into what that means. But we should be able to prosecute or bring consequences on somebody who's like, hey, why did you do that? I know you technically obeyed the letter of the law, but what the hell? Don't you realize you're making something that can permanently run away and disempower humanity? We have to be talking like that. yeah I think the conversation may get better after the election I mean we'll see there's things happening but you know I mean the main thing I would say is like if you are worried about losing the benefits of AI if we slow down I'm telling you there's plenty of power in the existing models including again next generation models that haven't been released to power real innovation and productivity gains and so on for some time to come and bring a lot of the benefits that are real.

You know, medical diagnosis, bring it to people who don't have good medical diagnosis now. The existing models do that. Education to kids who don't have good education now. At the same time, the challenges this stuff poses to society on various fronts, jobs and education in other senses and home life and freaking parents out and all kinds of things are collectively so substantial that I just think if we keep moving faster and faster, which is a path run now, things are going to get unstable in a way that a whole lot of people don't like. Leaving aside the out-and-out apocalypse scenarios of, you know, takeover and bioweapons, you know, and all that, it's just we're moving fast enough. And, you know, So this thing I mentioned, the Social Science Judgment Day, is kind of an example of that. And this is, in principle, it's a good application of the technology. That's one challenge here is there are so many upsides of technology that it is going to refuse of its own accord.

And so, you know, it makes the regulation that more challenging. But anyway, this is a potentially positive application. So this begins with, I guess, some reader of Tyler Cowen's blog writing in this guy had used the two frontier models from OpenAI and Anthropic to do this elaborate study of studies published by two economists that this guy doesn't like. And we should say that this the first example of this being done the targets are these two left wing economists Tyler and his readers are right But in the long run you can imagine this being done across the board on any kind of arguably dubious social science The name of these guys are Emmanuel Saez, S-A-E-Z, Gabriel Zuckman. They're both in Berkeley, and they've done, you know, left-wing stuff about wealth distribution and whatever.

And according to this study, their work was found wanting in various ways. I mean, you could kind of see this as an extension of the replicability crisis in psychology and other disciplines, but it isn't mainly cases where if you replicated the study, you wouldn't get the results. It's more like things where on post-examination, this was a dubious choice of variables to study or whatever else. It was a serious methodological critique. And there's an elaborate process of the agents going and looking for potential flaws and then their claims being challenged by another agent. And the reason I mean I would be a little suspicious If this was just coming From This reader of Tyler's And Tyler because they have an ideological Thing but a number Of people that I've heard of Weighed in and Said things that don't seem Ideologically motivated this guy

You know this name Joshua Achiam Former OpenAI I think or maybe Anyway And this was embraced by Philip Tetlock, famous predictor, forecaster guy, right? Yeah. And this tweet of Akiyam's, and what he wrote was, people who have been in research fields develop a felt sense that most, quote, research, in almost any field that isn't exactingly technical, is research theater. And many superstar careers enjoy star status not because of a unique connection to the truth, but because of a unique connection with an audience. No one really has the time or stature to, from a position of objectivity and trust, bring popular researchers to heel. But it will get increasingly hard to hide from serious critique. The machines can do critique now at the level of depth and rigor previously only possible for the academically deranged. In other words, someone who's just obsessed with finding the flaw, you know.

So he says we're going to have to revise a lot of what we know about the world that isn't really knowledge, but has actually just received popular consensus, like floating voltages without grounding wires. The grounding wires are coming. I don't know enough about electricity to totally grasp that closing metaphor, but I get the idea. And it's interesting because, I mean, you know, there is the hope that we can use AI to help rationalize discourse. I mean, I would like the ultimate application to be geopolitical, you know, to like help us adjudicate disputes that could otherwise turn violent and things like that. But, you know, it's a but at the same time, it is a sign of how fast this stuff is moving. I mean, think about it. It's like these guys are going to tell us which social science is good and which are bad. They're not even guys. They're not human. And we are going to teach them.

Just looking at the audit here by Sayez and Zuckman that you mentioned. So I'm reading here, 138 papers were examined, 2,035 problems identified, 147 problems were so serious that it changed the headline number. That's interesting, 147 problems out of 138 papers, okay. So they said 69% of papers contained at least one problem. 97% of directional errors overstated the papers. So, of course, you're going to have errors in your favor, right? Very few errors undermining your paper. I mean, look, this is not surprising to me, right? It's like this process of peer review is supposed to catch stuff like that, right? But peer review, it's like, okay, here's my giant paper, right? I worked on this for months. you're going to read it in your busy life and you're going to find my subtle mistakes that you can't even necessarily tell well I guess you can tell it but yeah I can tell but it's just like these take a lot of cognitive labor to find and like a peer reviewer, their token human mental token budget is realistically not as high

as like the two bucks that you can get from a modern AI even if you don't think the AI is smarter the AI is just going to take this project really seriously so you're just getting more cognitive work looking at these papers and you know humans make mistakes even if they're good faith even if they're not trying to cheat you know dan or yali right there's all these scandals of people like intentionally trying to cheat that's a different i mean that was an allegation of conscious fraud i don't even i mean the the kind of stuff they're pointing to here isn't necessarily at a conscious level and i should add we should reserve judgment i mean i haven't looked at this thing and ais can make mistakes and the person who who assigned the ais to do this had apparently an ideological position and so let's reserve judgment but it seems pretty clear that they can play some valuable role here and as you said the the resources they can throw at the problem when you especially have like swarms of agents and agents talking to one another and challenging each other um it's just something we can't really

do and i just want to say again we are already here okay this is possible like how fast how much faster do we need to move? I mean, just assimilating all this is going to, if indeed this technique of review is as powerful as you say, that's going to create a certain kind of turbulence of its own and is going to upset influential people. And I mean, look, I'm all for upsetting influential people. Most influential people I know of, I would like to see upset. But just the collective turbulence of the impact of AI on all of these, you know, fronts, family, jobs, and so on. It's like we have got enough to work, not to mention geopolitics, and the just inherent destabilization of the shifts in power AI is going to bring both domestically and geopolitically, you know, depending on who uses it fastest, the best, who has the most access to best models, blah, blah, blah. folks, we need to slow down.

This is my message to you. I mean, this particular example of AI reviewing research papers, I don't know if that's the strongest argument for the we need to slow down case. I mean, from my perspective, this is pretty cool. No, it is cool, but what I'm saying is all the cool things, all the things that have a cool side also create turbulence. I mean, just take scientific progress alone, okay? Let's say that very quickly we get to a point where, yes, because of AI, you can just do direct gene splicing during in vitro fertilization, not just selective reimplantation of embryos, but actually take the embryo, boost the IQ, whatever it is you want to do. I mean, imagine the political battles that is going to unleash, because in the first instance, it's just going to be rich transhumanists doing it. Elon will do it. And then, but if that keeps happening, you get this, you know, genetically determined like caste system.

and you can say, well, then the government should pay for everyone to do it. Of course, some people aren't going to want to do it. There's just a lot of creepy scenarios that arise, and it's going to unleash a lot of political battles. And that's just one little dimension. I'm just saying change in general, even change that some people would find positive, and I'm not sure that I would personally say the gene-splicing option is a positive one to begin with, but like the economic things, yeah, productivity growth is great. And who knows, maybe a lot of the people whose jobs no longer make sense will be able to adapt the jobs or find new jobs. Fine. But even that transition creates turbulence and just a certain amount of aggregate turbulence creates, like, you know, catastrophe. Social, major, major social geopolitical disruption, possibly including wars, revolutions, and so on. I hear you, I hear you. Just from my perspective, there does seem to be two categories of turbulence, where one is like, oh, God, this is existential, and then the other is like, this is kind of good turbulence.

We kind of need this to move society forward. We need an automated review of research favors. Yeah, but what about jobs? We have a history of seeing what happens when a lot of people's jobs are challenged. Do you remember when Marxist revolution was in the air? Do you remember that? That was the Industrial Revolution. Right, right. This is happening much, much faster. And also, I mean, this is my thing. It's like I don't dismiss the total doom scenarios at all, really, if we don't control this. It seems like in New York, a lot of the country, the Marxists are doing really well in the Democratic Party, even without AI. Yeah, look, I'm not against, I mean, look, if I had to choose between David Sachs libertarians and Marxists, I'm not saying I'd choose libertarians. It's not, I'm not saying I'm against radical redistribution. I mean, look, anyone who supports UBI is saying we're going to need radical redistribution.

I'm just talking about the turbulence part where there's the battles over what we do and I'm just saying that I think slower is better yeah yeah okay yeah fair enough going back to this idea of what the different companies are messaging there was also another interesting little tidbit Sam Altman versus Satya Nadella they both kind of reverse directions to their previous statements Sam Altman was recently doing a talk at Stanford and his quote was let's see he said I thought the alignment problem was going to be harder it may still turn out to be very hard from now in fact I suspect it will get much much harder okay so nevermind I guess he kind of doubled back it seems like he interrupted and like I don't even know what he said at this point OpenAI came out just like yesterday I think with some new announcement they were very proud of about this approach to alignment they were taking that had five bullet points or something.

It sounded like a little more closer to constitutional alignment that's associated with Anthropocon Weaver, but I didn't understand any of this shit well enough. But anyway, they had this new thing. Yeah, yeah, you sent it for me. I've got this Boaz Barak tweet. You sent this to me before the show. So Boaz Barak, he's quote-tweeting Jason Wolfe, and who's Jason Wolfe, alignment and model spec at OpenAI. and Boaz Barak is a famous theoretical computer science researcher who's been at OpenAI for a while. So Boaz Barak says, we organize the model spec around five core values. Be faithful to human intent, minimize harm, be honest and transparent, facilitate human agency, and act with care. You know, it's like Asimov's three laws. I was about to say, is that an improvement over Asimov? I mean, you have the same vagueness and ultimately the edge cases are the problem, but whatever. Yeah, exactly, right? It's just like Anthropics, helpful, honest, and harmless.

So Elias Ryukowski tackled this again in the early 2000s, and his whole point is like, yeah, that's great in a very vague way. The problem is that when you actually have to make decisions and weigh, weigh up complex scenarios and see where does it net out, what action should I actually take, it turns out that listing those five things doesn't actually tell you what humans actually want to trade off. Like, we have a more nuanced utility function in our brain, and the AIs actually can talk about it pretty well, but when the whole world is being optimized using, you know, capitalism or competition, are you going to preserve all of those things in the equilibrium of having all of these AIs, like, serve their master? So it's tough. I mean, you know, those five values, it's like it's better than nothing. I just think that, like, I guess just to review, like, what is my mainline nightmare scenario here, which I think is likely, I think you're going to have these buddies like Claude and GPD, and they talk to you, and they can even repeat back your values to you, and you're like, wow, this guy gets me. We're great to go. And then you're like, okay, do you mind working on this next iteration of a system and deploying the system? And it's like, sure, no problem.

But the system it deploys, it's more powerful, but it doesn't faithfully capture all of those kind of nice stories in the iteration of the powerful AI, and the powerful AI is more like optimizing for rewards, And there's kind of a loop where you have multiple generations of AIs and you're not stably preserving the original values. And that's where we're heading right now because we don't have the theoretical foundation of how to keep things robust across generations. That's basically where my head is in terms of what I expect. Yeah, and I would add one thing. And first of all, I should say it's not like they just wrote down the five bullet points. I'm sure they did a lot of work with their models and whatever. But one thing is that the competition among people using the AI to try to get a competitive advantage, whether it's in business or media or academia or scientific research, it leads them to explore the edge cases kind of, right? It's like, hey, let's try this thing that our competitors aren't trying. I mean, there's just a tremendous kind of exploratory impetus behind the use of this stuff.

So you keep leading it to like new cases that were probably not anticipated by whoever wrote these five bullet points. And I mean, that's a that's a kind of a vague thing for me to say, and I'm not sure it would stand up to scrutiny. But I think that's part of the problem here is there is such a robust, just natural ecosystem for the creative use of this stuff. And the creative use of this stuff can bring tremendous sorrow. Here, let's go to Satya. Okay, so Sam Allen is kind of mealy-mouthed. He's saying he thought it would be harder. And I guess so far, I guess the best sense I can make out of Sam's comments is that he thinks that there's a continuous line. So he's like, hey, look, the alignment problem today of making GPT not kill everybody has been so easy. But to me, that sounds disingenuous because it's like, look, the main reason GPT doesn't kill everybody is because it can't. Simple as that. Even if it tried its hardest right now, it can't quite do it.

It doesn't have quite enough intelligence for whatever reason, probably related to the reason why it can't 100% do everything I ask it in terms of the coding that I want it to do. The same reason it gets a little bit stuck there is probably related to the reason why it can't single-handedly plot how to take over the world robustly yet. But Sam Altman is, like, declaring victory. It's like saying, like, look, I have this baby tiger, and we've managed to restrain the baby tiger from slashing everybody to death. It's like, right, because it's not even that good at slashing yet. That's not a very impressive victory. So when he's saying, oh, in the future, it's going to be harder. Right, because that's the real alignment problem is the one where you can't just trivially restrain your agent by shutting it off. Anyway, so that's him on. So then Satya Nadella is going the other way. He says, I used to always think that alignment will just fall out. I no longer believe that. So I think this alignment thing may be a harder, different problem. Hallelujah. Okay, wow. Satya Nadella is waking up to the problem. Wow. You know, 2026. You got there. They could say they're both just talking their books.

I mean, back when it looked like Microsoft was going to have this alliance with OpenAI, Microsoft is reassuring us everything is going to be okay. I mean, the CTO or whatever his title is now, Kevin Scott, very nice guy, very sincere guy. He wasn't making it up. I've had it on my podcast, but was saying, you know, look, we in the trenches know you can control this stuff. I think a lot of people thought that at the time. But I would say that at that point, you know, Microsoft thought they were going to be on the frontier, I think, along with OpenAI or whatever. Then that alliance kind of fell apart. now OpenAI is on the frontier and Sam's saying, oh, don't worry, we've got alignment and control. Microsoft's not on the frontier and they'd be happy probably for the frontier companies to slow down and they're saying, oh, this alignment thing's concerning. So who knows? I mean, I just think that as a matter of fact, Satya is right. And it's just absolutely 100% and obviously not under control. and the idea that it's going to get more under control as we accelerate

seems to me wishful thinking. Yeah, some context also in terms of Microsoft's 180 here. In 2023, I posted a clip from Kevin Scott. I believe he's still the CTO of Microsoft, or at least he was at the time. And he was very dismissive. His quote was something like, look, I know how this technology works. I know how the cloud works and what's going on in those machines. And you guys have nothing to worry about. So I'm sure he wasn't thinking about like agentic AI or whatever it was. whereas many of us see a little further into the future. Yeah. So like what what else did much else happen? I mean. Yeah I got a couple I got a couple things for you. We can say a little bit more about the Anthropic S1 and what I learned from that. There's been a lot of hacking. We can talk about that and also as it relates to crypto and the blockchain and also there's a couple amazing new demos that impressed me as well. Where do you want to go? Well, do you think the Binance thing was a classic AI hack?

Now, first of all, the early reports are it's a South Korea, a line group. I'm skeptical. It just seems like, although everyone always says that, you know, hacking is incredibly hard to, you know, correctly attribute to a source, as soon as any of these things happen, there's some like company you've never heard of being quoted in the newspaper saying it's one of the four villains it's russia or china or north korea or iran or and it's aligned and if you look at the way they put it it's always kind of very vague it's like this has in fact this time i think i wish i had the quote it was like has the hallmarks of uh a north korea aligned operation so who knows whether it's north korea but i mean one thing i'd say and i think there are signs it's social engineering i want to hear what you say but i would say quickly one thing i learned um i was in northern california last week talking to some people who know about this stuff and one thing somebody said is you know you might you might they didn't say this but you might think north korea is you know like they don't

have access to the the models that are uh in in waiting in the wings they don't have access to the true frontier models we're not going to share that with them how could they do this but what this guy said is, you know, frontier model is not like a one-dimensional thing. It's like models have their idiosyncrasies. So you could use the two best models in the world to secure your infrastructure. It looks for all the vulnerabilities. You're done. And it can turn out that there's a model that's a little off the frontier that is capable of finding the vulnerability you missed. That's just the way it is. There's no model that truly covers the waterfront. And there are sub-frontier models that find vulnerabilities others don't find. So you can imagine a North this is a hypothetical assuming it is North Korea you can imagine them using one of the powerful open source models to do this That said my sense is that maybe it social engineering and if anything AI was used to research that

But what is your take on the whole Binance hack, which is fascinating because as we speak, it's not totally clear to me that Binance is going to have the resources to cover the damage, but we'll see. yeah let me uh look up the binance hack here uh i'll get back to you on that one in a sec uh i know a little bit more about the polymarket hack can we talk about that yeah tell me about that because i only read a little okay so polymarket is a prediction market and famously they use crypto so you load your crypto in there and you trade you try to make correct predictions and then you you know buy and sell prediction market trades and eventually hopefully you make money and you withdraw the crypto so what happened there was that they were running a third-party JavaScript library or tool. It's called DebugBear, and sites do this all the time. The idea is like, oh, you want to embed this thing on your site? Okay, yeah, here's like a third-party tool. It puts some JavaScript on your site. Users don't even realize what's going on because you're serving the JavaScript from your side, right? It's a third-party tool embedded in your site. So what happened is that somebody compromised DebugBear, and that was how they compromised Polymarket

because DebugBear is like, oh, yeah, we're serving different JavaScript on Polymarket now, and it's malicious JavaScript specifically designed to mess with Polymarket users and go talk to the Polymarket server on the user's computer's behalf. So it's making the user's computer talk to the Polymarket server and say, hey, I want to sell this position. Hey, I want to take this crypto and move it to this other wallet. And the other wallet is owned by the attacker. So this is, it's technically called a software supply chain attack because the bugbear was in the software supply chain of Polymarket. And now everybody's really suspicious. Oh my God, anytime I'm embedding anything or pulling any module, that could be compromised, right? There's stories like that all over the place. But the reason this is newsworthy is because so many people ran this malicious script right on their computer, and it's super hard for Polymarket to defend against this kind of thing. And so the damage, apparently there's $3 million in damage, and it happened all from fewer than 15 accounts. Okay, so the scale of this, in terms of money, it's huge. But in terms of, I actually thought that it would be more dangerous. I guess 737 accounts were compromised, but the total amount of money taken was,

I don't want to say that high, but it's only $3 million. I think that because we're talking about, with Binance, we'll talk about that in a second, $6.3 million. Yeah. But it is quite scary because crypto is the canary in the coal mine, or rather just like the first, it's going to be the first victim of everything just because once you get the crypto, ransomware, that's why ransomware is done on crypto. If you move the crypto, you're kind of screwed in most cases. There's not going to be insurance on that. But yeah, but the polymarket attack gets connected because there's a whole series of things to worry about on crypto. So, okay, yes, this is just general zero-day vulnerabilities. But another aspect of this is if you hold crypto, if you hold it in any kind of wallet or third-party service, I would be terrified right now because it's not like logging into your bank where it's like, worst case, they've got some paper copy of this or some offline copy, and they'll make good on it, and the government will step in. No, it's like this random software supply chain attack, like your life savings is gone if you're stupid enough to put your life savings in crypto.

And it's even the slippery slope goes all the way down to speculation that the whole idea of public key cryptography, that you can have a private key and a public key and you'll be safe because nobody can figure out your private key. And so you can trade with your public key. That whole idea, the theoretical computer science underlying public key cryptography is currently considered at risk, not like super high risk, but like, let's say like 10% risk. Like it's suddenly in danger when people are like, of course, public key cryptography works. So it's a crazy situation. When did that risk materialize? I mean, I had kind of thought that it was quantum computing that was going to eventually create the big danger here. And you're saying it's already arrived via some other avenue? So it's a different thing. So with quantum computing, that has to do with, like, you want quantum-grade cryptography. So normally, so maybe we should step back and say, hey, what is public key encryption? Yeah. I always like the example of, like, you know how, let's say you go to Amazon.com and you buy something, and you have to send your credit card number to Amazon. Okay, but the Internet is fundamentally an insecure channel.

So, okay, so you want to encrypt it. But how do you encrypt it in a way that just you and Amazon can see? Because you and Amazon never met and exchanged a secret together, right? So how are you and Amazon able to whisper to each other in a way that other people can't intercept? And so the answer is public key cryptography. I mean, it's very interesting. Humans figured it out in like the 1970s. I think there's a visual metaphor where it's like, I send you like an open box and you're able to put your message into the box and then shut the box and you can lock the box and then you send the locked box back to me and nobody can open the lock box on the way. But I had the key. So in this metaphor, it's like I sent you a box that you can lock that only I have the key to. So it turns out that the mathematical equivalent of that is that there's a mathematical transformation that you can do to the data, which is the equivalent of turning my lock, and it's like a recipient-specific lock. So Amazon's like, okay, here's your data that I locked with a lock. Here, Amazon, here's my data that I locked with your lock. So this is publicly cryptography. And then what you're saying is like, what about quantum computers? So the particular scheme that does the locking is

based on the difficulty of factoring integers, and that's something that quantum computers are known to be able to do efficiently. So people are like, okay, stop securing things based on the difficulty of factoring integers because pretty soon quantum computers are going to be able to factor the integers, but you need a different crypto scheme. And so people have suggested quantum secure cryptography, and they're feeling pretty optimistic about that. And so there's just a movement to upgrade a bunch of different cryptography to be quantum grade using a different kind of lock. But then the AI is coming in, and the AI is discovering new results in theoretical computer science that might even undermine even quantum secure cryptography might turn out not to be secure at all because it might just turn out that all of these locks, they actually have unlocks that we finally discovered, because this has always been like a somewhat question mark that we just assumed is probably hard, but it might not be hard. And AI is very much upending the game board, which is a whole other rabbit hole we can go down. There's been recent theorems from AI where they took conjectures that humans thought are going to go one way, like, oh, this conjecture is true. And AI is like, here, I got a new result. This conjecture is false. And everyone's like, oh, crap. And recently that happened for something about the exponential time theorem.

I forgot exactly what it's called, but it's this idea that like certain problems, the best algorithm for the problem has to take an exponential number of steps, but then AI came in and proved, oh no, it's not exponential. Look, it's slightly faster than exponential, which, you know, maybe from practical purpose may or may not matter, but from a theoretical perspective, it's really shocking to people because it's starting to chip away at what we took for granted in complexity theory, where there were like these certain separations of things being hard or easy that weren't 100% proven, but we strongly conjectured it, and now the conjectures are starting to crumble, and the crumbling might go all the way down to public key cryptography. Nobody knows. And this is a naive question probably, but could a sufficiently big and powerful swarm of agents factor integers faster than has generally been the case or not? Well, the swarminess of the agents doesn't really help you because the original conjecture is that even the best algorithm in the world, if you run it on a classical computer, is just always going to take exponentially many steps. So if you have a bunch of agents working on it, you need so many agents that it doesn't really matter.

That's the magic of the exponential. If you have a thousand bit integer, that gives you roughly two to the power of a thousand computation steps to factor it. Parallelizing two to the power of a thousand, it's still hopeless. You still need as many agents as there are, grains of sand or whatever. The numbers just aren't going to work, so you're secure forever. But if it turns out that there's some other algorithm that's more clever that doesn't need two to the thousand steps, then we're screwed. And that's what may happen right now. okay well that's reassuring um the uh yeah it's it's not getting less weird um the the funny thing is i haven't even made that much use of this stuff i mean i'm i've been so busy like writing about it and stuff that i am not fully uh agentized i need to get more agentic um i mean i can tell you about my experience i mean it's it's been yeah yeah so so i have a lifetime of experience as a computer programmer, 20 years of professional experience, I've spent

many hours in front of the computer looking at code. And yeah, I've said this before, it's always been my experience throughout most of my life where I'm like, yeah, this is like the peak of my brain. I'm understanding this is so, so much nuance, like computers are so far from being able to do this. And then literally in a period of months, I still remember it was March this year, right? We're still living in the same year when my entire career entered a completely new phase where now I don't write code at all. I just tell AI to write code. Like I've become a product manager. And of course, AI is on my heels being its own product manager. My day-to-day experience is that I'm able to step back more and more. So back to like the frog boiling side of this, there's never been one day where I'm like, great, I don't even have to show up to work. Like I'm still showing up to work, but the amount that I do, it's now at the point where somebody will message me something at my company and I'm like, I'll get right on that. And it's literally one click for me to just automatically send their message without even editing it to my AI agent. And my AI agent just knows what to do. It's like it builds the release,

ships the release, and then I just go back to the person. I'm like the guy from Office Space, right? It's like I take the specifications from the customer and I give them to the engineer. I'm completely useless, right? And of course, there could be an integration. I'm literally glancing at it, right? So I'm the difference between the AI automatically kicks in versus I just look at it and nod, and then the AI kicks in. That's the only difference. and then the entire release goes live. And the AI is like, oh yeah, I ran like a thousand tests on this. I've never written a single test in my life. My entire career, I was always bad at testing, and now the AI, it claims that it's doing like a billion tests, and sometimes it's like, oh yeah, I mocked up like a fake database, and I tested it, and I'm like, great, yeah, knock yourself out. It's like, it's doing all these jobs. It just knows how to do all this programming stuff better than me, and I'm just sitting here nodding and kind of stepping back from the system. yeah i mean i'm in a different line of work and there are things that i'm interested in doing including big projects but i haven't gotten all that far and then the little stuff you don't know

whether it's worth the trouble like uh just today you know you and i had been having this conversation uh dming in twitter and you know a number of kind of links had accumulated and i thought it would be nice to have them all like you know you know a summary just a crisp one-line summary of everything we said we might talk about in a word doc with a link uh but you know it's not that hard to do that myself is it like i don't know uh i and you know have you tried any of these like muse or any of the or dots or any of the uh the new the new uh so i haven't tried all these are dots i mean because i have like the pro version of this stuff and the reason i call it pro is number one it's like set up in a coding system where you can download packages, and it's got the full computational capabilities. But the other reason I call it Pro is because it's the expensive kind, meaning there's no limits of any kind. It's pay-per-usage, and I have days, like yesterday, where I was able to spray $1,000 into the damn thing,

and it didn't even feel like it was doing that much work. It's like, okay, the total amount of time that it was churning through, it was like, okay, eight hours, so it was busy for eight hours, doing all kinds of stuff, like doing features that I never had to implement myself because it implemented the features. So yeah, it was working hard, but it was just like one or two processes working for eight hours. I mean, it was spawning sub-agents. And I'm just out here like seeing the bill pop up on my bank account. I got like a push notification. Okay, another $1,000 was spent toward this AI, and I'm like, oh, okay. So given that that's my setup, if I were to use Muse, I would just run into limitations where it would just be stupider because it's not wasting, you know, it's not using like these state-of-the-art tokens. It's using, like, you know, the older generation, right, the cheaper generation. Yeah, but look, I mean, for the average person who wants to book a flight, you know, Muse is pretty good. Yeah, even there. Anyway, how many human hours of work two years ago would it have taken to do that $1,000 worth of work, roughly? Oh, boy. I mean, so I was the human, right? That's the funny thing, right? It's like I can very much give you a before and after. And it's basically, I'll conservatively say, it's about a 7x difference in terms of time.

So if I were to use seven workdays, like Monday through Friday and then another Monday, Tuesday, like that's the entire focus of my workday. Like I don't even get pulled into meetings and I'm just programming nonstop. That would roughly, that would give you something that if I have an AI spending all day, like one eight hour working with very little input for me, it's roughly the same result. and like I said, I'm being kind of conservative because the AI is like so consistent and relentless that it's just doing things at a level that I would just frankly just be too lazy or I just don't have that much brain power in a day that I can do so many things. But I'll just give you an example. Like whenever I have a question, even this idea of like mining our data, like we have a website, I run this business relationshiphira.com. It's like a coaching website. So people are like connecting with coaches and paying. So there's all these events happening And anytime I'm curious about a question, I'm like, okay, so this, people who see this version of the site, what is their, what are the analytics relating to them? Like, what are some trends? If you segment them like this,

normally I'd have to like pull out a command line interface and I'd have to be like, okay, what database query is this? I'd have to write my own queries and like debug them. Like, oh, wait, what does this mean? Pull up a file of code. Oh, the data gets moved into the database like this. Oh, okay, interesting. All right, and I'm like burning, before you know it, an hour is easily gone. And I'm like, and I learned something like trivial. and realistically, I'm like, look, it's not worth an hour. I'm just going to take a guess. I'm going to look at a simple version of this. Now that I have AI, I just write one paragraph, like a brain dump, like, hey, what do you think of this? Can you investigate this and that roughly? Just give me your best attempts to do this. I don't know. I'm just kind of brain farting. I barely even know what I'm saying. For one paragraph, I'm using speech to text, so I'm literally talking the way I'm talking to you. And then I press enter, and then I go away and do something else for two minutes. And then I come back, and it's like, okay, yeah, here's a table. I knew exactly what you mean. I dug into this, and then I came up with some of my own ideas. And this is just one tab. And I have a dozen tabs going like this, where I just shoot off a little thought, switch context, switch back context, see the answer. And these are just questions that normally we just wouldn't have the answer, we wouldn't iterate.

So there's all this iteration happening rapidly that just didn't used to be a thing. Okay. I've got what we call in the business a hard hour at the half hour mark. But I have one more question for you. So I had this, like as of a year ago, I had this theory when people were saying, where's the productivity growth? Well, first of all, that always takes time. I mean, it's a long time, you know, with the original kind of PC revolution and the online PC revolution and so on. That was kind of slow to accumulate. But I had this theory that what was actually happening was that the workers who had figured out how to use AI weren't telling their bosses. They were just spending their day at the beach, right? And so they're still getting as much money as they were to do the same amount of work they were doing. So the productivity growth doesn't show up unless you count beach time. Do you think that is still happening to some extent? I've got to think the companies are wising up, but what is your sense for that? Yeah, it's an interesting question. What's going on with AI productivity growth?

So for one thing, I actually think the best place to see the proof of AI productivity growth is in the early stage startup market. If you listen to Y Combinator, see what they're saying. And if you listen to Stripe, you know, Stripe is where, I mean, I run on Stripe, you know, the payment processing. And they have a bunch of tools that are useful for startups. They even do startup incorporation. Stripe is not holding back. They're being very clear. They're saying, yeah, 2026 was the start of AGI. The growth has went vertical for us. All of these companies are legit. They're making a ton of money. These new companies are scaling revenue faster than we've ever seen. Y-combinator is saying revenue is scaling faster than we've ever seen. Valuations of Y-combinator startups are shooting up. So, I mean, I would call this a major productivity increase, and this certainly dovetails with what I'm seeing day to day. I'm seeing my own productivity increase massively day to day. So this idea that this potentially does not net out in a productivity increase, I think there's no way. Like, there's clearly, the only question is, how many percent are we talking about?

I agree. Yeah, I don't know. Edzitron, I'm not sure. It was a very interesting stat, although it was circling back to the Anthropic IPO discussion, right, the S1. And you mentioned Edzitron. I think the number one takeaway here is that this is the final nail in the coffin of the Edzitron camp, right? This idea that, like, look, sure, Anthropic is making tokens, but they're just borrowing from NVIDIA, right? and all these companies are just like in a circle, like throwing money at each other and there's no value being generated. There were so many data points from the Anthropic S1 that put the nail in the coffin of that kind of speculation. So number one is just like the cashflow. It was very clear that when Anthropic is serving a model, the margins on serving the model are huge. They were talking like 50, 60, 70, 80%, like depending on how you account for it. But like, yeah, they serve a model, they're printing money, like serving the model is printing money. And then another crazy stat is they're embedded in, I think half the Fortune 500 companies, something like that, but they're only getting like a 0.1% of these companies' budgets.

So they're still getting like a tiny fraction of the company's budgets. I can tell you they're getting a lot more of that out of the budget of my company, like with my $1,000 a day or whatever. So they're 0.1%. So I don't think I have to be nostril honest here to be like that 0.1% is going to grow, okay? This is why Anthropic is a good buy at $1.5 trillion or whatever it is. Yeah, now there could be a crash. I mean, you know, because for one thing, a lot of people are investing on vibe, and those are the kinds of people who can freak out. And look, the whole financing thing of the infrastructure, it is more and more reliant on debt. Interest rates are going up. You could have, I mean, look, in my ideal world, you would have regulation that slowed things down, and that could have an effect. But as I've said before It's just like the dot com crash I mean the dot com crash happened But the web didn't stop And if there's that kind of crash

It won't be the end of the industry So yeah Well this was You know I thought another High nutrition Personally Episode of Non-Zero Doom we should do this again And we should ask people To rate and review I think both the Non-Zero and Doom Debates podcast smash the like button and anything else you want to exhort them to do yeah let's see yeah doomedbates.com give some love to Paws AI including Paws AI US Holly Elmore's thing and or other Paws AIs in other countries exactly tell your friends about the urgency of Paws AI because as fun as it is, it's also ridiculously reckless to just continue in this direction. Yeah. So, we will, this is good. I always learn stuff from talking to you, and we will, I guess, see you next time.

Likewise, sir. All right. Non-Doom Zero Debates, out. you