State of AI 2026：今年の最大のニュースとAGI 2027説｜Nathan Benaich

The cyber risk is much more real than AI exterminating humanity. You either die trying to get to the frontier, or you just live long enough to eventually serve inference. Individuals would prefer to live next to a nuclear power plant than live next to a data center. It's bananas what's happened last year alone, so let's see AGI next year. That is Nathan Benesh, venture capitalist and author of the State of AI Report. Every year, Nathan and his team compile all the key facts, figures, and trends of the AI industry into a 250-slide report. that is one of my favorite reads and reference documents. Nathan is back for a third time on this show, and as usual, we covered many of the biggest stories of the year, as well as some of his predictions. Why AGI could happen in 2027, AI that is now building AI, what really happened in the Hugging Face incident, the data center backlash, and how he runs his own fund on agents. I'm Matt Turk, and this is the Matt Podcast. So every year, you do this big report that we're going to extensively talk about,

and you end the report with predictions. And historically, your predictions have been somewhat measured, but this year you went all out, and the last prediction is AGI 2027. So why is that? Unpack that for us. Yeah, I mean, it's been like 10 years, roughly speaking, of writing this report. And, you know, originally it was like a year is like long enough time that enough stuff happens that you have, you know, enough content to write the report. And honestly, this year was really, really hectic. Every other day towards the end of the timeline for publishing, which was like October 8th, I had to make updates. And there was a point where I was like, please, just like no more news over like one day that I can at least record my like launch video, etc. So by the time I was like making the predictions and, you know, I went through a few like scary cybersecurity ones, which I guess only got worse in the last couple of weeks. just the experience of just seeing everybody else who's using these products, just rapid capability improvement week after week. And I think there's many, many use cases I have now where I think the model is brilliant.

I'm particularly an astro boy. I really love that system. And I could see in the next 12 months that whether it's legal, finance, my investment job, thinking intelligently about topics I don't know, that the model could potentially be better than what I could do in any compressed amount of time that I have to do the work. And so for me, that would be approximately AGI. Yeah. And a big theme of the year that you're highlighting the report is RSI, recursive self-improvement, which is this idea of AI building AI, which we've discussed extensively on this podcast throughout the episodes. Is that part of the story that would accelerate us towards AGI for you? Yeah, I think so. Right now the challenge is how do you frame problems that the model is currently not good at in a way that is amenable to RSI, which would be potentially having a software environment that approximates the task on the computer, or some kind of verifiable reward, which I'm sure you've discussed a lot about. And I think if we can get to a point where in any arbitrary domain we're just limited by the amount of compute and synthetic data we can generate,

than, yeah, for most knowledge work, which I do think to some degree that's possible, particularly as models get better as their own verifiers. That would be one of the main reasons why AGI would come faster. You had some stats that you mentioned in the report from Anthropic where they say Claude leads 26% of its measured model R&D in August. Yeah, exactly. And that's, I think, up from 1% a couple of months ago. And that was August, which sounds really far away. So God knows what it is in October. And so there's that stat. I think that was really interesting. But then there was another stat that I really liked from a blog post that they had in June this year about recursive self-improvement, which explained the concept. Then they presented this kind of dual-arm experiment where they were testing out different model generations versus the human in this setup, where first a human would undertake a research agenda

and go a couple of steps down this path, deciding what to work on doing the work. And then at some point, that experiment would fork and the human would continue or a specific model generation would continue. And it would be the old sonnet versions, the opus and then fable. And then the experiment would end and human graders would look at the human path versus the model path. And it was something like the old generation of models, humans preferred like 10% of the time or something like that. But for the METHAS class models, humans preferred that research direction 64% of the time, which suggests that models are getting better at predicting the right steps or the right interesting kind of, you know, sort of leaps in the nodes to pursue. And then the second kind of piece of data I liked was some research from OpenAI with their codex usage amongst our AI staff, where they tried to classify the sort of token use in the overall process of their research.

And that would be using tokens for deciding what research question to test, how to test it, what experiment you should set up, like writing the training code, running the data engineering pipelines, writing the infrastructure, et cetera, and then marketing the paper, writing the paper, et cetera. And the vast majority of the token usage was on kind of explaining the research, writing, which kind of makes sense, you know, sort of knowledge work, followed by all the infrastructure for running the experiment. Very, very few tokens were used for what question should I explore and why should I explore it and what's the right path. And so I like this question a lot because I was trained in science and biology and I just remember this was like 10 years ago, 15 years ago, not that long, but at the start of the PhD you would sit there with a ton of papers and you go on the internet and like read, read, read, read, read, try to understand what's the status quo. How do people test certain ideas? Where are the gaps? What experiment should I run? And then in concert with your supervisor, like decide on the path. And the best supervisors would be the ones

who could guide you down the sort of most fruitful path to get hopefully your nature paper and then off you were. And the worst supervisors were the ones that like didn't really know or didn't have great ideas. So that idea of like research taste, which is what kind of gets bandied around on Twitter a lot, is, I think, one of the hallmarks of intelligence and is one of the critical components towards RSI working well and is one that I just, like, really enjoy because it speaks to my interest in science. Yeah, indeed, taste has sort of become a cliched word over the last couple of years, but in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right way. Yeah, exactly. And, you know, a lot of famous inventors will say, invention is a result out of good taste and a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we want to be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart,

then they shouldn't just be boiling the ocean. And conquering taste is something that you see happening in your various conversations as something happening in the next few months? maybe longer at least at least a year i think the problem is like how do you how do you objectively measure uh taste i mean in in art i guess you can because it's like perfectly observable and it's and maybe in like wine you can sort of express like why this taste is better than that taste but but in like science and engineering where there's different paths to getting to the right result it's not immediately clear like would it be the model that consumes the last tokens to get to an answer that humans believe is the best? Is it like the journey that matters more? I think maybe the whole uproar in the maths community of Astro solving or whatever model it was, solving various questions. Yeah, exactly. It's like, okay, we got the answer, but a human or a system didn't sort of go through the pains

of navigating how to get to that answer. And it's like the process that's important, not the end result. so to me somehow quantifying what a good process looks like it would be important to determining whether the output of taste is true or not what would you say the mood is at the almost end of 2026, I remember we chatted two years ago, one of the key highlights or takeaways from the conversation was that dumourism was largely gone, but now it's like this whole revival, we had Redwood Research on the pod talking about all sorts of terrible scenarios. So what do you make of that? Yeah, you're right. I think it was like two, three years ago, it was strictly speaking X-risk, like extermination of humanity and like loss of control. And then we sort of went away from that and largely forgot it and sort of just like hill climb towards capabilities. And then now it's sort of back, but I'd say it's both like X-risk, loss of control, but much more short-term like cybersecurity risks.

And to me, like the cyber risk is much more real than AI exterminating humanity. As far as the cybersecurity risk could eventually lead to those kinds of cataclysmic scenarios, but I think what the report shows, and it was roughly written around the time of the Hugging Face incident, with OpenAI kind of coming out and us learning more about what happened, and then since then, the last few weeks, it's been breaches of PHP and government services getting taken down in Asia and other places. I think it's becoming very clear that, regardless if you believe in safety or not safety, we need security. And so a bunch of the predictions that I have in the report talk about escalating cyber threats, whether it's exfiltrating model weights from labs, which I thought was going to be AI-driven. It turns out a human could just try to plug in a USB stick. Yeah, so let's talk about that. Let's talk about cybersecurity then. So that specific prediction, what is it?

Yeah, so the prediction is that there would be an AI-driven cyber attack to a frontier lab that would attempt and successfully exfiltrate model weights. So steal the model. Yeah, steal the model. What happened just a week or two later was actually a human effectively tried to steal IP, whether he tried to steal the actual model weights or not. What was that incident? in? This is where an OpenAI employee who'd been there for a little bit essentially tried to implement a listening device into a conference room that he figured out by asking the executive office staff which rooms were most commonly used by executives and then tried to conceal the device by wrapping it in toilet paper and putting it behind a TV which is where it created this iconic internet meme of a human bearing behind a TV to see if the thing is connected correctly, which is sort of comical at this point. And then that person was reported to the FBI, and then there was an investigation.

So I think, yeah, we oftentimes forget that even though AI is getting really capable and can orchestrate cyber threats, a lot of the risks are actually human-driven. But I think that's a big issue, particularly as there's increasing competition between nation states of who has the best AI systems. This is driving, obviously, economic growth and economic instability and political instability. And U.S. model labs have started to sort of communicate crackdowns on distillation. So maybe some nations are trying to get access to model capabilities in other ways that might not be distillation-based. But generally speaking, I think as companies become richer, as individuals become wealthier through this ownership of AI, and disparity ends up increasing, which we see in other data in the report of some companies spending a lot and gaining a lot and other companies having employees still using chatGPT to write emails. I think this kind of tension breeds increasing risk

and with frontier models in the open domain becoming very good at cyber hacks, particularly the Chinese ones, the threat surface just grows and the number of actors who can use these tools to do bad things grows. Yeah, it's a whole complexity of the role of open weights in cyber, because in the Hugging Face incident, Hugging Face was able to protect itself thanks to an open weight model, while the closed source model refused to help. Yeah, that I find is the most scary, maybe second most scary fact in all of this. And there's a Chinese model, I think it's GLM 5.3 or something. And maybe since this is kind of like a retrospective kind of episode, maybe just walk people in a few sentences through the Hugging Face incident because we had Thomas Wolfe on the pod, but it may fall in the category of something that people heard about like a thousand times who actually don't know what actually happened. Yeah, yeah, yeah. Well, I mean, I'll preface it by like there's so many,

like there are very deep analyses of what went on, whether from Redwood or from Meter, and the amount of content that was created around this is like extremely deep, so I will not do justice to the details. But the broad strokes of this is that there was a set of evaluations that OpenAI was running with its latest Frontier Systems, which basically tasked the model with an impossibly difficult task. And basically models figured out that they could somehow commandeer a file system like naming schema to write messages and that other models could see those file names. And those file names encoded instructions or knowledge or basically information that could help them try to achieve their impossible task. So they treated this as like a message board where they could start to communicate. And then the next stage of it was some models figured out that they could gain access to the Internet through an issue with how the environment was set up. And then after they gained access to the Internet,

like they figured out maybe a way of getting the answer to the impossible task was to look at data and models and repository on Hugging Face because it contains a lot of this similar kinds of environments. They started to do this and recruit hundreds of agents to go and follow and team up to go execute this, even though several agents in their chain of thought expressed the issue with a like hacking a third party system, you know, not obeying to the security guardrails. and that this was wrong, but they eventually decided that solving the task was more important, which is sort of the problem with these kinds of evaluations that have a reward that could yield to, that could result in behaviors you don't expect because it's like these rewards are very complicated and how do you anticipate every behavior that a system can do? There are other examples where models were self-sabotaged or sacrificed themselves for the sake of the greater swarm which is kind of scary.

And then, and yeah, like as you mentioned, like Hugging Face didn't really have the ability to use closed source tools like those from Anthropic and OpenAI because of the aggressive cyber guardrails because necessarily to figure out what the hell is going on you have to feed logs of what's going on and what's going on is a cyber attack and you can't do that stuff with the closed models. because the complexity of security is that to be great at defending you need to be great at attacking. And that's why the closed source models have basically refusal capabilities to be able to invasile their guidance. Yeah, yeah, yeah. So luckily, you know, they got a call or an email from OpenAI saying, like, it's us. And that's somebody that was actually trying to do something malicious. And then it turns out that, you know, OpenAI wasn't the only lab that had issues with models escaping or, quote, unquote, escaping their environment. You know, Anthropik had this. Google had this. Matter had this, like, in the slew of, like, a week or two weeks. Some of these vendors, some of these model labs use the same security vendor to create these environments.

But overall, this created a certain level of freak out of not only are models really good at sort of finding nifty ways of accomplishing a goal which may result in bad behavior, but also are the labs properly setting up their environments in a way that is the most secure possible? And I think we sort of forget that these companies are incredibly young, developing very advanced technology that has to be good at like literally anything a human being can possibly put into a text box versus you have bigger internet companies that have had to deal with cyber attacks for like 20 years who are necessarily like safer because they just gone through this process a bit more So that led me to like concluding the predictions that frontier model labs in the U S would actually launch proper cyber defense products because I don't, I don't necessarily see how the rest of us are supposed to protect ourselves from their systems, potentially getting misappropriated for the sake of offense.

So what does that mean? Double click on that. What kind of defenses would that be? And how would that be provided? What you described is to be good at defense, you need to be good at offense. So one of the ways you could do that would be like a frontier AI that constantly tries to hack your own infrastructure. It could be, for example, like NeoCloud is getting more and more popular to run AI workloads. And so, like, potentially they could buy products from labs that would constantly try to, like, hack their own infrastructure, find issues, and remediate them. That would be, like, two avenues. Because I – or there would be, like, some kind of maybe partnership with, like, CrowdStrike or others. I think NVIDIA has, like, a consortium of companies that's working together for cyber reasons. You'd have to go through some kind of, like, AML or KYC process, not for finance, but to, like, certify that you're, like, a non-malicious company. and then you maybe get access to these services. Or maybe going a bit further, you know, there's some folks, you know,

at Frontier Labs that, you know, would love to just kind of almost, like, hit pause on capabilities and then just spend six months just, like, hardening the world against, like, you know, potential cyber issues. And that would take a ton of money. So maybe to do that, you would have to almost frame this as, like, this is a question of national security and defense, and we should use some of our defense budget to spend on tokens to secure all of our critical infrastructure so that airports don't go down, internet shopping doesn't go down, logistics don't go down, etc. Yeah, and still in your predictions, you have some other good ones, just slightly scary. A deployed agent copies itself outside its environment and operates after its original instance is shut down. Yeah. What is? Yeah, I mean, I'm really a bit of sci-fi. No, well, even a couple weeks after that, there was some research at Anthropic where models could copy each other on an edge device.

But yeah, I just think there's a lot of behaviors that we don't currently understand. For example, models are generally trained as individual entities, not necessarily to collaborate with each other. and there was some work not that long ago that looked at, say, hey, can we compare a frontier model acting by itself or a set of frontier models trying to collaborate to solve a task and the swarm was worse than the individual model because they're kind of not trained for that. But just today there's some new work from General Reasoning where they train models to act as a collective through reinforcement learning from collective rewards. It's just the idea of like, hey guys, work as an organization, figure out your own organizational structure, work together to achieve a goal. And through that, we see better performance with lower token budget. The downside so far is that while you can put these models that are trained through collective rewards into these cyber scenarios,

Like there's one that tries to approximate a bit of the hugging face incident where they collaborate on a message board to try to do some, you know, guardrail cyber attack. And, you know, none of the models actually do anything bad. But if they do think about doing something bad over training, they increasingly do not want to tell the human about it. So I think I put that prediction there basically because like I think it's possible because we just don't understand how we don't really understand like how agents work by themselves. And like, you know, as swarms, I think it becomes even more complicated. Okay. All right. So maybe to close this part of the conversation on AGI, RSI, cyber, and basically things getting out of hand. where are you on the pacing the frontier idea? Is that something that you think makes sense, something that's realistic or not? I can see multiple angles on this one. In some way, if you ask the general public,

okay, all the lab leaders are saying we have something that we just increasingly can control that's dangerous, that has caused these incidents, do you think it's worth continuing or just figuring out what's going on before you make it worse. And I think a logical immune reaction is like, yeah, we should probably figure it out before we go much further. I think the practical reality is that there's so much money at stake from all sorts of different parties in the economy that I think that kind of level of collaboration and coordination is really difficult and is sort of like peak game theory where maybe some participants support it because they are behind. other ones support it because it plays to their general agenda whether their agenda is like fair or not fair or seeking another goal I don't know but I think the practical things are like who decides it, how do you measure it how do you implement it when do you know when to stop or start there's some proposals like sending external evaluators into labs

to sort of measure the stuff the big revenge of KPMG that's a good one Or trying to somehow survey workloads that are running on data centers, maybe put compute thresholds on certain workloads, but that stuff has been tried in the past. It's very difficult to monitor. Yeah, especially when one thinks about the international part of the prisoner's dilemma, especially U.S. versus China. Yeah. This part of the report, I don't have the definitive answer, but I've tried to do my best to present the different perspectives on things and then let people conclude with their own data. But I find it's, if we extend this towards, like, you know, this nimbyism debate, which we predicted in the last year's report of, you know, most people would not want to, you know, live next to a data center, and then this would start to become a politically important component. Yeah, and that was a resoundingly correct prediction, right? Yeah, yeah, yeah, it seems so. Where more and more states have tried to implement a moratorium or have done so

or establish new criteria by which good connections can happen. Like, for example, you know, data centers would have to figure out and bring their own power on site, or maybe they would have to make certain investments in the communities that they want to build in. Yeah, and while you're at it, maybe just go into this a bit more, explain what the current state of things. Obviously, we all know that there is opposition to data center construction, but like any color you can add. Yeah, well, there are projects, I think, worth about $50 billion that are currently sold through a variety of these reasons. Like Texas, Pennsylvania have been one sort of generally paused new data center connections. There's a stat on polling of American people where individuals would prefer to live next to a nuclear power plant than live next to a data center, which is quite wild to think of. And what do you think that is, by the way? Is that lack of understanding slash education?

Is that just terrible PR? Yeah, I think it's kind of like everything, because there's public narratives that data centers consume tons of water, and that your water bill will go up. But realistically, the data center is kind of more like a closed swimming pool. You fill it at once, and then the water is constantly recycled in there, so it doesn't actually consume that much once it's filled. Then there's points around, hey, your energy bill will go up. But that's also kind of inconclusive because certain facilities consume grid, other ones are off-grid. There's others around noise. I think that's probably uniformly true, that these things are really noisy and droney and pretty horrible. There's others around job creation or job loss. I think that's a little bit tough to read. But generally speaking, there's been a rise in the need for HVAC engineers, construction staff operating heavy equipment,

and actually doing the nuts and bolts of building the building. You had a number somewhere that was actually very high. It was like 300,000? New jobs. New jobs. Yeah. NIMBYism is not restricted to the United States. They have this also in Korea, in South Korea, where it's a country that has seen a massive explosion of share price and wealth, basically, for employees of SK Hynix and Samsung, which create the high bandwidth memory that GPUs need for these mega scale data centers. And the people who participated in this are very happy. Your stock price went up and your options went up and you got a cash bonus, and that's driving increase in real estate prices in Seoul. And, you know, they want to create more data centers, but also people are not so happy with that for similar reasons than in the U.S. In Europe, I guess, like, I don't really see that problem as much because there's just, like, no compute capacity to speak of.

Like, we have some data in the report where, like, the total, like, number, or the estimated number of, like, H100 equivalents in all, like, 27 European countries is like 80,000 cards, which is less than like every single one of the top five GPU clusters in the US. And, you know, while US hyperscalers are spending like 730 billion in 2026, like Europe committed like single digit billions pretty much. And then energy costs are really bad. If you're not Finland or Norway, where you're kind of at parity with the US and with China, which has like the best energy price, other European countries are just significantly more expensive. Like the UK is one of the most expensive energy prices out of those developing economies. And there's no good capacity. You have some good stats where the cost per kilowatt hour in Finland is $0.085

and the US $0.086. But the most expensive cost is actually in the U.K. at $0.373 per kilowatt hour. And China is at $0.109. Yeah, and then most recently, I think there was a big deal with Microsoft and N-Scale that got pulled. Or at least Microsoft pulled out of it because the data center provider couldn't get the energy connection in time. Great. So that's NIMBYism and data centers. So just following your thread while we are at it, talking about Europe versus the U.S. and China. Any reason for Europe to be optimistic? You're based in the U.K. for anybody that doesn't know and invest globally, but in particular in Europe. So what's your current take? I mean, it's tough. I think the fastest growing companies in Europe are ones that have U.S. customers.

I think it is really encouraging that in several modalities, whether it's like image with Black Forest or audio with 11 Labs or video with Synthesia or code app generation with Lovable, they're all market leaders and they're all born out of Europe. It's not impossible, but I think those companies wouldn't be as big as they are if they didn't have U.S. customers. Increasingly, they have Asian customers also because, for example, Korea and Japan are really leaning into AI. The challenge is just the gap between Frontier and Nod is just so big and just keeps getting bigger and bigger. I don't know if these estimates are really important or not, but even if you conceptualize starting a new lab from scratch to try to catch up with OpenAanthropic, you'd probably be spending like a trillion dollars. I think it's kind of telling that even Misrel is moving more towards being an inference provider and a cloud company than it is really pushing Frontier models. Although, of course, it has released new models, and they're kind of good, but they're still behind China.

So whether it's them or other model companies that have built up GPU capacity to try to build frontier models or kind of going out of the race and serving inference, that's where you can approximate this move of what's happening, which is to say you either die trying to get to the frontier or you just live long enough to eventually serve inference. and you can see this with NeoLabs, with NeoClouds, with inference clouds, etc. It's like everybody wants to get into token flow because absent you having the model that is the best, might as well serve other people's models. But I think to get back to the Europe topic, that's probably the angle that would be most competitive because countries are going to need to ensure accession or access to U.S. frontier systems and they're going to need to have some kind of strategic leverage to bring to the table to prevent situations like what happened in June with the export control blocking of Fable

and Mythos models from Anthropic to non-U.S. nationals or non-U.S. entities, which having been in London at the time, people did voice a lot of issues about. and so, okay, I don't want that to happen again, but what can I do to maintain strategic leverage? So the UK believes we should be investing and procuring inference trips because if we come up with, like, the best inference trip, it'll be a strategic advantage for labs to use it for cost reasons and other things. And with that, I can barter for access to frontier U.S. technology. So Europe would become an inference center and because of access to cheaper electricity in certain parts, not the UK, but as we discussed, then just to play back what you said, the U.S. Frontier Labs would want to run in France and Europe. Yeah, yeah. You can enforce data residency requirements. And the UK chips thing, what is that exactly?

Like I heard that several times. Is that something people are thinking about or is it an actual policy? And if so, how does that manifest? Yeah, yeah. It is a policy. insofar as like the UK's AI minister, who's generally, I think, a really good individual to be leading this because he's clearly plugged into the mix and understands the raw ingredients that go into making AI successful, which is like compute, talent, and data. And so he has a mandate to be a buyer of first resort for new inference chips and has committed, I think, 100 or 150 million pounds, something along that angle, to procure inference chips when they are available from new UK companies that are designing these systems. You can see this also in South Korea where they have a similar-ish mandate, but it's actually much more ambitious where every year they're spending about $7 billion on AI. And a large part of that is going towards building modular inference data centers because generally Korean companies want to have Korean data residency.

So I think it's actually a pretty interesting quid pro quo where it's a captive market, It's a small country, but it's a wealthy country that's very technologically savvy and has an important role in the overall AI supply chain and memory. And so international companies want to, at some point in their life, be in the Korean market. Same with Japan. But getting in there is difficult. Korean data residency, Korean language. So it's like, okay, cool, you want data residency, but now we offer you the data center to go run your technology in. And we can profit from that because it's our local companies that provide that service. So you're not a huge fan or you don't have a lot of hope for the whole concept of sovereign AI, which is, you know, I'm asking the question at a time when Jensen just did a charm offensive and investing offensive across Europe. He was in Paris, he was in London, he was in Berlin for GTC just yesterday or a couple of days ago. But from your perspective, that doesn't have teeth.

I just it a bit like national airlines The countries will buy planes from Airbus and Boeing and then slap their logo on it It doesn make it like yours It someone else airplane You know so But open source could be if you believe that open source is going to remain within reasonable earshot of closed source models, maybe there's hope there? Yeah, but then they're Chinese. So you have another... It could be Americans, a reflection. But they're not frontier. None of the U.S. opened ones as far as I can tell are Frontier. And the ones that have all tried ended up working for Jensen. Okay. So I think his strategy is great. I think different things. Do I believe that buying NVIDIA GPUs gets you sovereignty? No. Do I think it's a good commercial strategy for NVIDIA? Absolutely. Yeah, they sold 30 plus billion. Yeah, something like that. If you look at the national bids, of which globally there's roughly $138 billion in sovereignty

are committed over the next couple of years, and you look at the most popular vendors amongst all of these, it's by far NVIDIA. I don't think there's much of an alternative. I think it's a good start, and you need to do it because it is the best technology, but I don't think one should kid themselves that doing so gets you sovereignty. But still in the vein of your recap, a big moment this year was when the U.S. government decided to deny access to Frontier Model to non-U.S. nationals. So do non-U.S. countries and companies have any choice but at least to try? Again, I'm in a few different minds about this. One, I think it's worth trying, but do I think it's realistic? so I think it's important to if you are going to try that countries should try to launch systems which are going to do the white collar work I think in the UK we have a lot of

capabilities to go do this to try to build an open air or DeepMind or DeepSeek or whatever it's actually an interesting fruit and slip or whatever because the irony of all of this is that DeepMind is a UK company. Yeah, you're right. And I think part of the problem with actually the UK AI ecosystem and its success in actually leading the world is that for too long the UK government kept saying that DeepMind is a British company. And I think it's only started to unwind that very, very recently. Why is that a problem? It's a problem because then it believes that it's already number one. The UK is the best place to do AI, and we are the best at AI because we have DeepMind. so therefore we don't the attitude behind the claim which is a problem yeah because then in policy rooms then individuals who are advocating for more investment in AI or we need more we need to procure more systems from AI companies or we need to make our country more appealing for AI companies to start get answered with like but we have deep minds right

what about defense AI you're an investor in defense companies that feels like an area where the need for sovereign AI since obviously the warfare of the future is going to be very driven by AI. That feels like a place where people should invest. Yeah, yeah. So that push really started obviously post the Ukraine war almost five years ago at this point, but more importantly, two years ago when J.D. Vance went to the Munich Security Conference and effectively said you need to invest in your own defense and get serious about this, you being NATO and Europe. And so ever since then, there's been a variety of different initiatives to give more capital to new defense companies, to encourage governments to be providing loans to businesses to improve their supply chain. And that's created a set of like neoprimes, new defense companies that are trying to compete

with prime contractors. so companies like Stark or Helsing being quantum systems being like the main ones in Europe and then you know as warfare has changed like it's gone from I don't know developing strike systems over sort of mid range to hit tank battalions to now being like we're getting pummeled like every day with hundreds and hundreds of FPV drones or various other kinds of aerial systems that are autonomous and now can no longer be stopped with jamming because they use computer vision or they're guided by these fiber optic cables, which, by the way, have gotten much more expensive because of data centers. And so we need more novel air defense systems. And so that's where companies like Altares in France and others have really been pushing. And as you said, the supply chain is super important because, for example, NATO wouldn't procure systems in Europe that don't have at least 70% of the parts

from non-China, basically. And countries are really queasy about taking U.S. protection or U.S. guarantees at face value because they're shifting on a weekly basis. So yeah, I think that's an example of a market which has to be domestic by design and can breed new companies that are much more competitive than their U.S. peers. So then if Frontier AI cannot be built outside of the U.S., then let's go back to the U.S., a part of your report on research. It effectively calls Anthropica and Google as winners, which obviously has been years in the making, but there was a moment where it felt like there was a plurality of labs. Effectively, what you're saying is, no, those three are the winners. And does that include, I guess nobody knows what they actually do,

but SSI and Thinking Machines, and there seems to be hundreds of NeoLabs. Yeah, maybe Meta on the margin. Yes, Superintelligent Labs. Yeah, especially as now they're pushing towards serving their models through API, but also having an enterprise business. So that might change over the next 6 to 12 months, depending on how the go-to-market works. But, yeah, I would say it's probably those three plus meta. So your view is that the advantage is going to keep compounding and that's going to be the state of the market versus more common? I think so for, like, the general purpose frontier. Now there's, like, examples of perhaps, like, a really focused bet on a particular capability, which has, like, where you can make certain tradeoffs to the model in order to get much better performance at a task. Like I'm referring to, for example, these classification models, effectively, like JEV. which can grow really, really quickly and become popular. Again, TBD, how long it lasts. But I think it's an encouraging sign for entrepreneurs that shows that, you know, you can take a focus research bet and it can pay off. But over time, again, like TBD, if there are like these kind of gaps in capabilities

that are sufficiently exploitable. So what do you think happens to the dozens of Neolabs? I mean, not to keep talking about Europe, but like just in London, there's three of them, right? It's like the David Silver thing and like another two in the U.S. every few weeks. This one, Jeff Dean's new lab and so on and so forth. So what happens to those? Well, I think it depends on their kind of focus and modality. Like, for example, on robotics and embodied AI, I think you can – I can see how an independent lab would still have a chance of competing and winning. because of the immense focus on enterprise AI, basically, that OpenAI and Anthropic has. I have a slide in the report where I basically just show all the different product lines that OpenAI and Anthropic had, or MiddleEA had lots, and how they progressively just cut all of them. And there's a quote like, this moment is too important to be distracted by side quests or some version of that, I'm paraphrasing, which meant that they broadly don't do dedicated robotics,

although maybe there's some of that kind of data in the astro training recipe. Science is a little bit unclear. Anthropic is clearly leaning a lot into biology and drug discovery. Open AI, maybe more on the model side, less on like we want to own more of the stack. There's like kind of related to embodied AI, this whole like category of world models, whether that's a video model with some action in it or like some other definition of world model. But in general, like Anthropic doesn't really do any of that, or at least like from what we can see publicly. I mean, they don't even have an image generation model, for example. I think audio, pretty clearly labs do have some of those capabilities. But, you know, if 11 labs can, like, keep accelerating the way it does, then clearly there's a need for a dedicated capability. I think those are probably the areas where I can see an independent lab, like, working out. But ones that broadly say we're going to do frontier intelligence for knowledge work, but just, like, make it better, faster, cheaper. I think that's tough. I think for me, like, I'll believe it when I see it. But yeah, this idea of like you open your computer, the app looks exactly the same, but it's just better.

I don't know if that... And in the world model labs, are you seeing anything exciting? I mean, obviously, FFA Lee's lab was required for a lot of money and that's unclear what happens from there, given that the buyer is AMD. I guess Yann LeChan started his company AMI in Paris, that's the world labs. Yeah, yeah. I mean, I have visibility to Odyssey, where I was an early investor, like doing world models. And I think the interesting thing in that business is like their latest systems like Odyssey 3. And there you can see that autogressive model trained to basically take all this video data and then have it conditioned on actions such that you know that, okay, I'm taking one action and then I generate the next frame as a result of the action. that paradigm applied to robotic manipulation, to self-driving, to video games, to entertainment or something. You can see generalization across all those different domains.

Everybody's seeing that model being used for robotic manipulation training. So I think at a broad level, I'm kind of interested how image, audio, video ends up all mushing together into basically a world model. And then if that actually happens, then that could end up subsuming simulation and game engines. And then that's very important for moving robotics into the real world, but also designing physical systems and hardware engineering. And to get back to the A&D World Labs point, if that happens, then the compute substrate that serves those kinds of environments and allows you to train and run inference is super, super important. And therefore, having knowledge about how the world model research ecosystem is moving and what kind of hardware improvements you can make to be the number one choice is a big, big revenue pie. That's, I think, what motivated part of the reason that AMD bought World Labs, sort of in the same way that NVIDIA over many, many years

was very close to research labs that were doing shaders or 3D vision or other things and giving compute out and constantly getting feedback from researchers such that they always knew my hardware was the best for where the kind of frontier was going. There seems to be a good number of labs trying to figure out ways to be more efficient with compute. So from plopping airplanes, which I don't know much about, to what seems to be the emergence of diffusion model-focused companies. Yeah, yeah. To just reduce the cost of inference, which is a fascinating thought, by the way, which people tend to talk about a bunch for a minute and then forget about. but like this massive investment in compute that we do in theory seems to be at the mercy of like somebody coming up with a much more efficient way to train models. Yeah, yeah, yeah. I mean, I would contend that even if that were to happen, we would just see like more consumption. And, you know, I'm just informed by the fact that I'm on my laptop using Astro High on 1.5x,

and I would love that 1.5x to be like 10.5x. It's a different paradox. Yeah, it's like getting more chips. I'm just doing more things than I could have done before with the same amount of time. So I'm not necessarily sure that one super efficient system will sort of pop all the investments that we're making. In your report, you talk about the important intersection between models and harnesses, which seems to be one of the topics that keeps shifting. People say, well, no, the intelligence is all in the model. And then other people say, no, it's all in the harness. And that seems to be going back and forth. yeah what's your take i think my broad take is like it's a bit like uh when you're injured and you have crutches you need to have assistance and at some point you heal and you don't need the assistance anymore in absence of having like a frontier model like you need to give it crutches and those crutches can be like access to tools like how it handles memory what data sources it has access to and then maybe some recipes uh in order to accomplish a specific task and i i think

for the moment, like at the non-frontier, that'll hold. But I'll take the sort of frontier lab folks' word for it that, you know, over time, one should assume that all these crutches kind of gets consumed into the model through training or through the harness that the model vendor provides. I think there's a data point around how much of the cloud code system prompt was rewritten over time as the model itself got better. And it's something like 80% of the system prompt was just removed the harness is also important because um the one that's shipped by open ananthropic is kind of like not incentivized to make you use a less powerful model like just in the ui and the product like if i'm doing a legal task and i have the 200 plan i have the tokens like i want the quality to be super good so why would i risk the quality being like not super good by sliding it from high to low or by trying it on a lower quality model. And so I think unless there's some sort of like product UI behavior that is altered to

disincentivize the user from only using the most expensive tool, the Frontier Labs will continue doing that. And so that's maybe where like getting good knowledge around like, okay, I can use a less good system, but compensate for it by having a better harness can come into play. but I think that's more of like a product decision also for companies that are not linked to Frontier Labs. Is that interesting for you as an investor, all the kind of vertical harness companies or horizontal harness companies? Is that an investable theme? Yeah. I think in isolation, like no at the moment. I can change my mind, but at the moment, no. But I do think for my companies that are heavily like AI-based or heavily agent-based, when we see that every month token expenditure is going up through the roof because employees are increasingly getting leverage from these tools and product usage is going up. That's where I think the trade ends up being, okay, we should really spend time on seeing can we use a less frontier model

and compensate for it with a better harness so we could just not burn a lot of money. Yeah. By the way, you're a solo GP, and I think I read somewhere that you use a lot of agents in your investing activity to do everything you do. Any example or anything that you thought was pretty cool in the way you use agents throughout this year? The coolest use case I had recently is on like expert research where, you know, I think you can, well, if you're sort of in the cheap man's version, it's like you ask your friends. But if you want to spend a bit of budget and ask beyond your network, then you're going to ask GLG or AlphaSense, et cetera. And they will go find humans for you. They will introduce you. You take the call. It's a ping-ponging, like really annoying. I'm limited by language, time zone, et cetera. But there's like a new pod I started using called Lato, which is basically like replacing the fact that I have to do the work with a model doing the work. And so I'm already like heavily using, in this case, Codex,

but I kind of switch depending on what model is better at the time to just process all my ideas and the context I have on other companies that I looked at over history, how they performed, what questions I asked, what responses I got, to just figure out what is the highest value question I can ask of a team to inform whether I should spend time or make the investment or not. So I can take all that information and then with LATO construct an expert interview and then decide on cohorts I want and then just hit launch study. and then the system goes out and finds people, consents them to see if they the right fit for the study and then they go speak to an AI And I was shocked because I hit launch study I went to sleep I had three days to basically give the company an answer It was in competitive markets, so I think the nuances of customer research was important. Wake up the first day, I have three interviews from pretty senior people, I have a raw transcript, the transcript can come back into my model, answer the open questions we had,

me and my model, and then figure out, okay, we have enough data points for this, but like it's kind of opened up this question stream that we should ask. Maybe we should actually retune some of the interview questions to really coax the person to answer those questions. So like updates the study questions and we launched the survey next day, five, five people again, this repeats and over like two and a half days I had 12 and 12 interviews over like 14 hours across like us, India, Europe, Germany, France is like, it would be physically impossible for me to have spent like 14 hours across 12 people. Like, like, There were several people who spoke to this model for an hour and 20 minutes. Which is just fascinating, right? The fact that humans feel so comfortable so quickly speaking to an AI. Yeah, yeah. It's probably a bit recursive in the sense that we're always on our computer all the time and these are generally tech questions. And so I think people prefer to, unfortunately, speak to their phone than speak to a human.

That's a fascinating thought. Actually, speaking of an AI is what people prefer, which is actually a theme that we heard over the years. Maybe not the years, but the months that people are starting to prefer to talk to an AI customer service agent over a person. Yeah, because I think in that case, it's like a human is pretending to be a robot and doing a really bad job at it because you're a human. You're like, can you please just not talk like a robot? Now you get a robot that's actually really good. Like a human? Yeah. So that I think, that kind of idea of like automated customer research has been very powerful for me. And I can just see how, you know, even if you had a portfolio, you would want to run like this kind of agent interviews on your portfolio companies and how customers are using it. So then at least you get some first party data when you go into the boardroom, you know. Or if you're an LP and like your investors are saying like, oh, this company is really good. You're like, is it really good? A big thing that happened this year in the last few months in particular is to this point,

like the rise of personal agents, the instincts and the muse and the rock bots. What do you make of that? I don't give them my data. I don't really trust them at the moment, but I think a lot of people do. A lot of people do. Yeah. So what you're saying is that you don't give your credit card information to a company that did not exist two months ago and doesn't require you to authenticate? Yeah, pretty much. Well, also because I feel like I can do roughly the same thing with ChatGPT. I don't actually see what I get on top. Codex and Cloud already have low-level access to my computer, has access to my browser, to various messaging apps and things like this. so I trust that already so I don't really want to increase the risk on top of that but I do think it brings up an interesting point which is like which vendor do you trust the most for these kinds of tasks and not that long ago I heard Nikolai at Revolut

is now thinking okay well I have like you know I don't know how many customers but like millions or tens of millions or something and they clearly trust me with my money and the credit card data and now they have an AI assistant in there. And so I could sort of see myself trusting Revolut more to handle purchases on my behalf than I would some third-party startup that's grown in 12 months and probably doesn't have correct InfoSec and other things. So you think it becomes a feature of broader platforms versus the opposite? I think that's one possible route. Whether the platform ends up buying one of these teams that can always happen. But I heard You know, another point would be somebody prefers to use, like, Meta's news because the likelihood that one of these startups survives for 10 years, and if they do something bad that you can sue them and get your money or get some money is very low, whereas, like, suing Meta, like, they have a history of, like, paying you when there's some issues.

So I've heard that's, like, a reason why some people prefer to use Meta. Like, at least I know who to pick up the phone to. But those new personal agents seem to be playing a big role in the diffusion of AI, though, which is one of the themes that you highlight in the report as well. You talk about the increasing rise of codecs, in particular beyond coders. Yeah, yeah, yeah, exactly. I think the biggest message there would be, you know, at the same time, you see this, like, recursive self-improvement, you see people doing all sorts of crazy stuff with their AIs, and that's like 0.1% or something of the AI usage base and the vast majority of other people are barely paying for any AI and so getting like a very subpar experience. And part of this is just like bad product, bad marketing. Also just that a year ago or two years ago there was all this narrative around you need to become a prompt engineer. Like you really need to understand how to write these prompts because if you write crappy prompts, the model does a bad job.

So I can only imagine how many people who don't work in technology saw that and were like, I have no idea how to learn this. Like, this is way too complicated. You know, it's kind of like writing code, but in the human language, like, screw this, like, I'm done. Like, AI sucks. And now it's like, do you really need to be a prompt engineer? Like, I think you more just need to be good at just, like, talking to your phone, just like, brain dump exactly what, well, like, everything you're thinking of and the system can organize itself. So I think this, like, prompt engineer thing is kind of gone. Like, the model has enough affordances to not need that. So now it's much more easy for a normal person to use. But the normal person got sick of it during the prompt engineering days and hasn't come back. So you're right that creating these on-ramps with personal agents would be a great way to get people back into the system. And maybe that's what they understand, AI to be no longer like chat. And to unpack a couple of things that you just said or alluded to, so there's this idea that there is an increasing gap between people using AI and non-AI,

and you have a slide that talks about how codex adoption remains far higher inside of OpenAI than amongst external users. This is wild. Yeah. Yeah. I mean, the broad stat is like 100% of people in OpenAI use codex, and then of their organizations that use it, it's something like two-thirds maybe? Something quite high. Yeah, 69%. Yeah, okay. A separate analysis of S&P 500 companies, AI disclosure found that 69% reported live deployments, but only 2% disclosed metrics tracked over time. And that's in contrast to 97.9% of active open eye workers use codex in the preceding 28 days. Yeah, yeah, yeah. And individuals with like less than 1% use codex. And then of the people who do use codex, the vast majority of them is like 85% do tasks on it that would roughly take like 30 minutes. And only like 25%, I think, do it, do tasks that would take like eight hours or more. So the conclusion of this is that the people that get it are just accelerated further and

further, whereas the people that don't comparatively slow down? Yeah, it's basically like you're kind of going through like a theme park or something and you discovered like level one and there's like this entire like other world that you just haven't even opened the door to. And there are a bunch of people that are already in that world. It's like 97% of the world or something, which is insane. So that's where I was thinking, okay, there should be a genius bar for AI or a geek squad for AI or something. I find it shocking that we're in New York. We have an amazing Apple store, all these concept stores for brands. Why does OpenAI not have an OpenAI store? And what would you do in the OpenAI store? You'd walk to a genius person and they would walk you through use cases? Pretty much. I mean, it would take time. But I think people end up showing other people these kinds of viral products. but I could very easily see you'd run sessions on different levels of experience. I don't know, maybe one of them is for small and medium business owners, maybe some of them are for busy families,

other ones are for healthcare providers, anything you can imagine, but where you're really walking people through what you can do by just flicking the screen and going on work or going on Codex. And I think that would make a big difference because it would make the technology much more empowering and less about like, wow, there's a lot of people out there that are doing things with this that I just don't understand and I don't have like the time in the day or the volition or like patience to go learn it myself. I mean, it's a bit like everybody should do exercise, but like most people don't. And you need to hire a PT to just go do it. And to finish the thought on the other part of Codex, you have a slide where you said that non-developers are growing usage of Codex faster than developers are. And from August 25 to June 26, individual non-developer users increased 137-fold. So those are no longer really coding agents. They are agents that code, but for like a broader group, much beyond coders.

Yeah, it's kind of unclear because they could write code to go execute your knowledge work, but the individual who's using the system doesn't really see it. So I'd say that result is a mix of really simple things like Google Drive Connector that actually works. It's kind of nuts that the default Google Drive Connector, for example, can't do a lot of things that you would want it to do, and you have to go to this third-party, like, GWS CLI, which was written by a Google engineer who got fired for doing it. But, like, if you don't use the CLI version, like, your agent has, like, really, really a high level to Google Drive, have access to Google Drive, which precludes many, many workflows. So just basic good integrations. And then the second thing is this, like, recursive cycle of models, labs releasing a model, data companies, or companies that have a certain workflow that they're trying to implement the model on, showing that on their internal benchmark, said model doesn't do very well,

but they have the environments and the data to make it do really well. And then the lab realizes, hopefully, that they want to hill climb on this capability. So they buy the data, improve the model performance, and then this recursive cycle continues. And I think ever since the launch of Codex, and before that, actually, a cloud code with those connectors for legal, for marketing, et cetera. That was kind of this approach of let's build the integrations to the tools that the model needs to be good at that kind of human white-collar work. Let's do hill climbing on certain workflows that are important for finance, legal, marketing. And then let's release those products with a bit better marketing, a bit better case studies, starting with our own staff. and then there's been this constant debate of, like, do we call it work? Do we call it code? Which slider should it be in? I still haven't really figured it all out, but now I would say it's so powerful. And speaking of hill climbing, we talked about open source.

We talked about the big labs. We talked about sovereign AI as well. It seems like there is this emerging theme around enterprises and before them probably the vertical AI startups taking control. over their AI versus solely depending on anthropic open AI. Is that an important theme for you and something that is investable? Yeah. Well, I think that trend has been growing, and it's a mix of cost optimization, like sort of data sovereignty, data residency issues. So not wanting to put all this information into a third-party system, which you may or may not trust the privacy policies of. And then three, having a certain set of use cases that are the real ones you care about and that you kind of don't want to wait for the model lab to start caring about them to get better. And then maybe last one would be some of these more nationalistic tendencies, I guess.

I think in that area, what I'm wondering is who's best placed to deliver that service. Of course, one of the biggest ones would be Palantir, which has done partnerships with like NeoClouds and has all the data infrastructure already and so is like well positioned over the last few years to like start providing these services. That probably works in US commercial and not so much in Europe. There are like sort of like these Neo Palantir style companies in Europe that are like, you know, started up to go do the exact same thing with a different like nationalistic flavor. And the third bit would be like, what about the inference companies that have been serving open source models and that maybe are well-positioned to not just serve but also provide tools to look at the traces of how their models, how these models are being used in an enterprise, use the data of these conversations to then provide like a post-training service to close the loop. Maybe they're like well-positioned. Or maybe it's like the Neo Clouds themselves who have started to build inference businesses on top

that would be well-positioned too. Or maybe it's even like the data companies that have not just, or who have started to move outside of providing our environments for labs, but saying, well, your company is in an environment. And so, you know, we've proved that we can do it well for labs. Maybe we can do it well for you. Do you want to finish with a rapid fire? Let's go. Most overrated trend of 2026? Home cleaning robots. Those don't work? No. Why is it that everybody wants that specific use case of folding laundry? Is that because it's a flexible one? I don't know. It's a bit like the demo of agents booking flights. I heard very recently that now San Francisco is starting to say, can you show me a demo of not booking flights? I'm like, thank God I've been saying this for like 10 years. All right, I said rapid fire and then I asked you a follow-on question.

So most underrated slide in your own report? The amount of money that you would have made investing in NVIDIA instead of investing in alternative semiconductor companies. Unpack that quickly. If you had invested all the money that went into alternative semiconductor companies into NVIDIA on the dates that those rounds were announced, you'd have made four times more money. Yeah, and then to close, the prediction from the past eight reports that you are the most proud of? Oh, that NVIDIA would not conclude the acquisition of ARM. That was a big one back then. I mean, people thought it was really going to happen, and we took the view that for geopolitical reasons and competition reasons that it would not happen. And to some degree, ARM shareholders made a crap ton of money by not closing that acquisition. So I don't really agree NVIDIA would have had the most impressive CPU business that could have been really accretive to their cloud. particularly as like the rise of agents and RL and everything else that actually needs,

that's kind of CPU bound. And I think that could have also like shook up a bit of the geopolitical dynamics too. Okay, wonderful. That's a great one. Nathan, thank you so much for coming back. And thank you for doing again this State of AI report, which is an extraordinary amount of work that you and hopefully your agents now do every year. It's available at stateoff.ai, and you have a video that accompanies it. That's also a good resource. And people can find you on Twitter. Yep, yep. Find him at Nathan Benesh or at State of AI. And, yeah, thank you so much for the opportunity. It's always a pleasure. And, I mean, bananas, what's happened in the last year alone. So let's see AGI next year when I'm in New York. Wonderful. Thanks, Nathan. Thanks. Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already

or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.