数学を襲った雪崩がAI開発にも来る――AI研究の自動化と再帰的自己改善の見通し

If you had asked me earlier this year how far away we are from superintelligence, I'd probably have said at least a better part of a decade. But over the last three months, I've updated my views a ton. I've been debating people both on and off the podcast about this question. And of course, I've been watching the progress of AI on verifiable domains and thinking about what that implies for AI research itself. And I've come to the view that we're only a couple of years away and possibly only a couple of months away from AIs that are as good as the best humans at AI research. And I've also come to believe that automating AI research will compress so much progress into a single year that soon thereafter, so by 2030, and plausibly as early as next year, we'll have AIs that are at least as good as the best human experts in every field. I know this is a wild claim, and it's one that I myself didn't believe even a couple months ago. Let me walk you through how my thinking has changed. Before we get into the more speculative arguments about what will happen in the future,

I want to point out that AI is already dramatically speeding up further AI progress at the frontier. For example, Meter, which is an independent evaluation organization, spent three months embedded inside of Anthropic, and afterwards concluded that Anthropic's rate of AI progress on well-scoped tasks is 2x what it was a year ago, and that a, quote, majority of this acceleration today is likely attributable to Anthropic's extensive use of AI to do R&D, end quote. OpenAI published a blog post about this in September, where they showed that the heaviest 10% of AI users amongst OpenAI's researchers went from using about $1,000 a day of internal coding agents at the beginning of the summer to over $7,000 a day by the end of August. And also in September, Anthropic put out a similar post on its own AI uplift. They reported that Claude now leads 26% of Anthropic's AI R&B work, up from under 1% as recently as February.

In thinking about how big a deal it would be for AIs to automate AI research, and how much that would speed up AI progress, A really important update for me has just been seeing how fast the capabilities of AI models in mathematics have exploded. Current AIs are slicing through well-scoped problems in maths like a hot knife through butter. It was only two years ago that I asked Dario why LLMs aren't able to use their vast knowledge across so many different fields in order to connect together different ideas and make new discoveries. And lo and behold, here we are. In a wholly verifiable domain like mathematics, AIs are doing the search and match discovery to such a great extent that hundreds of important results, which the top mathematicians and computer scientists in the world had spent decades trying to resolve, are being churned out as side quests to hype up future model releases. It's crazy how fast this progress has happened. As recently as 2023, these models couldn't reliably add four-digit numbers. In 2024, they could barely solve any problems in a high school math competition.

In 2025, they could win a gold medal at the International Math Olympiad. To earlier this year, where they finally started to solve some niche open problems. To a month ago, where OpenAI solved a millennium price problem in 88 hours. I'm reminded of a line from the novel East of Eden. Quote, and there is one short thing about the fall of gods. They do not fall a little. They crash and shatter. Now, it's true that progress has screwed disproportionately towards the verification-laden parts of mathematics, like finding counterexamples, rather than, say, coming up with new theory. Toby Orte recently wrote a great essay about this. He writes, quote, But another way to say that same exact sentence would be this. Current AI systems have solved a millennium-priced problem, all without the ability to ask powerful questions or find profound new fields of mathematical inquiry.

And this ability to solve well-scoped problems is going to be an even bigger deal for machine learning research than it has been for mathematics broadly construed. That's because the goal of mathematics research is not just to prove or disprove long lists of conjectures, but rather to extend the human understanding of mathematical concepts. Coming up with the idea of topology or groove theory, or going back even further, the Cartesian grid, matters far more than knowing exactly how many one-unit-apart pairs you can make from n dots. We mainly care about such abstruse questions insofar as they might elicit new mathematical theory. But in AI research, the priorities are exactly reversed. AI labs mainly care about how any new algorithmic recipe empirically makes training more efficient or elicits smarter models. They don't care about theory for its own sake, and all the progress that we've made in deep learning over the last decade and a half has not required much new theory anyways.

The equivalent of just grinding through random conjectures would be enormously valuable in machine learning in a way that it just isn't for mathematics. AI research is actually surprisingly similar to the kind of mathematical progress which current AIs are disproportionately good at, which is unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domains. For example, let's talk about the transformer architecture. This is the big breakthrough that enabled modern LLMs. My friend Sholto Douglas, who is an AI researcher himself, has this perennially half-finished blog post where he explains how you could have invented the transformer two years earlier than people actually did. And maybe this video will finally get into quick stalling and complete his essay. But in the meantime, I'll summarize a bit of the discussion I had with him about the contents of his essay. Before the Transformer paper was published in 2017, many researchers had already identified attention as a way to better parallelize training

and communicate information across longer sequences. Chris Ola, who was an active researcher at the time, has said, quote, at that time, my sense is that the transformer architecture was a natural combination of many ideas like attention and residual networks, which had independently been effective, taken to quite aggressive lengths, end quote. In other words, none of the core pieces were a surprise to researchers active in the field. So then why didn't people invent the transformer sooner? One reason was that this new architecture was only obviously better than the state of the art at the time once you connected together half a dozen additional tweaks and innovations. And though almost all of them had been known in the literature for years, no one knew how to combine these puzzle pieces together in just the right way. Doing so required great research taste, of course, but it also required tons and tons of small experiments. But what I've just described above, which is persistently and intelligently matching dozens of different combinations of ideas, is exactly the kind of capability that AIs have shown themselves disproportionately skilled at

in their contributions in mathematics. This is all to say that even the Transformer, which is arguably the biggest breakthrough in ML since AlexNet kicked off the deep learning revolution in 2012, is the kind of thing that Quad7 could have invented if Chris Ola asked to try really hard to get this attention thing to work better. Now, I want to clarify that some parts of AI research are in fact less verifiable than math. That's why OpenAI hasn't fired all of its researchers, despite having a model that can apparently autonomously solve Millennium Prize problems. But I think that AI research is verifiable enough that AIs will soon be as good at it as the best humans. Now, explaining why requires getting into the nitty-gritty of how AI research actually works. And because those details are interesting, but technical, I've saved them for after the conclusion. So if you're curious and you want to hear them, you should just keep watching until the end of the video. But in the meantime, I'm going to give you my estimate of when we're going to have such AIs. If I had to pick my median year for when automated researchers will be as good as the very best human AI researchers at doing AI research, I'd guess 2028.

And that's assuming that the AI labs go full steam ahead and don't slow themselves down intentionally or aren't slowed down by a government. Why do I think that AIs could get this good within two years? Well, in every domain where we can check answers and give AIs thousands of rounds of practice, For example, competitive programming contests or well-scoped open math problems. They've gone from worse than most humans to better than the best humans within just a few years. And as I'll argue after the conclusion, AI research has exactly the same verifiable properties. Of course, a big part of what makes the top human AI researchers really good at their job can't be directly verified. For example, their ability to bet years in advance on the right research direction. But as we've already seen in mathematics, models generalize far beyond what they were directly trained on. OpenAI's model was obviously not trained by solving other Millennium Prize problems. It was trained on lots of less ambitious, more tractable math problems.

And from those, it generalized all the way to solving a problem which no human had ever cracked before. And I start the same in AI research, where the combination of thousands of rounds of practice on verifiable research tasks, plus the fact that each generation of models is broadly smarter than the last will make models excellent even at the parts of AI research that we can't directly check. Now, I want to be clear that there is a pretty decent chance that hitting this milestone takes longer because some important parts of AI research remain hard to automate. This is just a very hard thing to try to predict in advance, and I'd guess that there's at least a 25% chance that it doesn't happen until after 2032. But conversely, I also think that things could happen much faster, and we hit this milestone in 2027. Some of the most well-informed researchers I know, people who have a very good track record of making predictions over the last few years, expect AIs to beat the best humans at AI research as early as next year. And 2027 is also the straightforward implication of putting the new report from Meter that says that Anthropics' rate of AI progress has doubled from a year ago into a take-offs model.

As former guest Ajayi Kautra put it in a blog post about this report, quote, So the reason my median year is not 2027 is just that I uncertain about how accurate this kind of straightforward modeling is But we should not be surprised and we should be ready for a world in which we get to this milestone next year Once AI is as good as the best humans at AI research, algorithmic progress will get a lot faster for three important reasons. One, the AI will be much faster. Two, each of them will be very smart. And three, there'll be a lot more of them. Let's start with faster first. Today's AIs already do work at about 10 times human speed, and automated researchers would also be working around the clock, nights and weekends included.

As a result, each individual automated researcher will be cramming roughly 50 times more work every calendar week than a human researcher can do. Of course, that doesn't mean that AI progress will immediately accelerate 50x. Those automated researchers would still be bottlenecked by compute for experiments, which aren't running any faster. So there would be diminishing returns. But even with extremely steep diminishing returns, this alone would make algorithmic progress proceed many times faster than it currently does. And this is before we counter the fact that in the scenario I'm talking about, every automated researcher will be at least as good at AI research and engineering as the very best humans today. My sense is that around 80% of the algorithmic progress at a given frontier lab is driven by its 50 top researchers. It's not for nothing that Zuckerberg was willing to pay hundreds of millions of dollars to acquire the best talent in the industry. And so you get another big speedup in algorithmic progress from the fact that every single engineer and researcher in this automated company will be at the level of the very best

humans in today's frontier AI labs. And finally, there will be a lot more automated researchers. Now, throwing more workers at a problem isn't always helpful. For example, a swarm of agents probably isn't going to be able to write a great novel much faster than a single agent could. But other tasks lend themselves much more to paralyzation. For example, look at this animation that I'm overlaying, which shows a bunch of different claws working together to formalize Fermat's last theorem. This feat had thousands of modular subtasks, each of which could be completed independently and where each agent could locally check whether their work had succeeded. AI research is closer to formalizing a theorem than it is to writing a novel because there's a lot of independent modular work to do and there are verifiable proxies for progress. That means a swarm of agents can try out a thousand different approaches and see which one performs best, and they can do this independently for each different part of the pipeline. AI companies are already made up of dozens of different teams working independently on different parts of model development.

One team, for example, improves how the pre-training data is filtered. Another designs a particular kind of synthetic RL environment. Another writes a better attention kernel, and so on. In domains like this, where different streams of work can be done and checked independently, dumping in lots of persistent hill-climbing agents tends to speed up progress a lot. Again, there will be steeply diminishing returns just from adding more parallel researchers. Anybody who's ever worked in the team before knows that having 100,000 collaborators doesn't mean that you'll go 100,000 times faster, probably not even 100 times faster. But given the nature of AI research, it's implausible that you wouldn't be going at least a couple times faster from having orders of magnitude more automated labor. So the combination of faster, smarter, and more automated researchers will speed up AI progress a lot. But will this pace continue? Will it get even faster? Or will it slow down? That question is even more important than what the initial pace actually is. Two competing trends are relevant

here, and the subsequent rate of progress will come down to which one ends up being more powerful. On the one hand, because of RSI, automated researchers will continue getting faster, smarter, and more plentiful. On the other hand, AI research will be getting harder and harder, as all the easy ideas get eaten away by the swarm. By the way, ideas getting harder to find is a thing that already happens. AI companies have to quadruple their R&D compute and double their workforce every single year just to maintain the same rate of progress. So if the rate at which AI gets smarter is faster than the rate at which ideas get harder to find, then we get something called the software-only singularity. In this scenario, even without increasing compute, the AI swarm rapidly makes smarter and smarter versions of itself until, within a matter of months, we end up with a superintelligence that scrapes against the physical limits of our current ships. But if ideas get harder to find faster than RSI makes AIs better at research, we'd end up with something which I'd call a spark. we get something like three to six years of AI progress at the current rate in a single calendar year.

And then things would simmer down until we build more compute, as even these super genius automated researchers need to run bigger experiments before they can make further breakthroughs. As to which one will actually happen, I think it's basically a coin toss. If pressed, I'd give a 50% chance to the Spark scenario and a 50% chance to the software-only singularity. But to be clear, even in this Spark scenario, where we make something like four years of progress in the first calendar year after we automate AI research, things would feel insane. That's a jump as big as GPT-3.5 to GPT-6 Astra in just 12 months. To remind you of the scale of progress that this leap would compress, GPT-3.5 was able to write a passable limerick and autocomplete a trivial Python function, but otherwise it was mostly a useless toy. Whereas Astar can build whole video games from scratch and churns out so many field-metal-worthy mathematical contributions that we become totally desensitized. In the last four years, we've gone from a preschooler to a savant intern.

If you made another jump that big, but starting in 2027 or 2028, you'd probably get a mine which would be vastly superhuman in domains that are even slightly verifiable. For example, it'd be better than NVIDIA at coming up with the next generation of AI accelerators. It would make more money than the current human versions of Citadel and Jane Street if it was given equal access to the market. If you gave it remote access to a material science experiment lab, it'd be better than the rest of humanity at finding a room temperature superconductor. now at this point you might say how will even super intelligence which is trained in a data center beat human experts in domains where the basis of that human expertise is years and years of real world experience the way we're currently making ai smarter is by rling them on tasks that are easy to automatically grade from within a data center for example do the million new lines of code that the AI wrote to pass all the unit tests? Did the AI solve more open math problems? But of course, most of the world is not like this. I made this point in a recent essay.

How do we train an AI to get really good at building a business from scratch? How about winning court cases, or having a profitable day of trading in the markets, or helping a candidate win an election? The rollout here requires interacting with the real world, and you can't recreate it from just within a data center. So how will superintelligence get good at this kind of stuff? Well, let me be clear. I don't think we can just bake the answer to every real world problem into the weights. Rather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp up quickly in new contexts, even in hard to verify domains like the ones mentioned above. Getting really good at any complex task requires two things. One is domain specific knowledge, and the other is a more general set of skills, the kind of capabilities that capable humans have, like being able to seek out relevant information, make a plan, push forward on that plan through mistakes and obstacles, keep track of what you've learned, and collaborate with other people. During RSI, automated engineers will build

tens of thousands of times more RL environments than we have currently, and these will be far larger and longer running and higher quality and more open-ended and covering a much wider range than today's environments. The AIs will learn to make their way through unfathomable amounts of virtual worlds that are each to today's video games what a Bugatti is to a horse and buggy. These games will be far more difficult and messy and sprawling and realistic than anything that has ever been made for human entertainment. They will require AIs to cooperate or compete with teams of other AIs over long periods of time. For example, an AI might be dropped into a simulated economy of millions of other AIs and told to co-found a successful company. And it's not like the AI will be stuck just thinking about the ideal corporate structure. It'll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques. The AIs will also train themselves on endless backtests of everything in the history of science and tech and politics and culture.

every discovery, invention, war, election, and quarterly earning that has ever happened. For example, here's one prediction training task. Can the AI figure out general relativity given a training corpus that includes no information beyond 1700, which is far earlier than any human was on track to discover GR? Another task might be, given every single newspaper article until 1962, can AI predict how the Cuban Missile Crisis is going to unfold? And it's not just the training data that these automated researchers will be improving. They will also radically improve the basic learning algorithms themselves through breakthroughs in online learning and sample efficiency, both of which have verifiable proxies, as I'll discuss later. In the same way that humans are much more efficient and effective at learning from data than current LLMs are, future AIs could be far beyond humans at their learning efficiency. Now, why do I think we'll see lots of transfer from verifiable tasks to these more general skills? Well, one reason is that even at one ten-thousandth or a millionth of the scale of RL training that will be happening during the intelligence explosion, we're already seeing lots of generalization.

If you point clot code or codex at a project that it's never seen before, it knows how to orient itself and figure out how everything fits together and then tackle an ambitious assignment. Of course, it's hard to measure how much models have improved at tasks that are not easily verifiable, because if we could, those tasks would be verifiable. But you can just go talk to a new model and compare it to an old one. In my experience, AIs have gotten much better at things that have very little to do with code or math. For example, helping me research the abstract questions I needed to answer while writing this essay. The other big piece of evidence for transfer is human evolution. Our general purpose mental architecture was developed by evolution. and evolution is an optimization process with a really crude verifier. Did your genes pass on or not? In the ancestral environment, the tasks which influenced that outcome were things like hunting and fending up predators and keeping your standing in your tribe. These are a tiny subset of the kinds of tasks that humans are able to tackle today. Evolution did not specifically optimize us to be able to write a Pulitzer Prize winning biography or make advanced semiconductors or lead nations of millions of people or come up with a new branch of mathematics But in optimizing us for survival on the Savannah

evolution built a general purpose learner. And in training these AIs, we're running a similar process, except we're doing so deliberately with tasks that are much more closely related to the ones that are needed for success in today's world. And unlike evolution, AI companies are trying to improve transfer during training. For example, by making sure that performance improves on difficult tasks that were held out of training. If even dumb, blind evolution can get this much transfer out of building a mind that is effective at propagating itself in the savannah, then the deliberate effort that AI companies are putting into building a flexible and general intelligence that can rapidly ramp up in new contexts should advance much further. Now, even a super intelligent learner can't figure out everything from inside a data center. So even these AIs would need some real world experience before they can beat the best humans. But I think they'll be able to get that experience much faster than people expect. And that they'll be able to beat top human experts within months of being deployed. And honestly, one of the things that's guiding my intuition here is that I have a lot of 20 something year old friends who I've watched accomplish incredible things and build world leading companies and ventures in domains that they entered into with basically

zero years of experience. History is full of super competent young people who ran circles around the world that they found themselves in. By the age of 27, Napoleon had stacked victory after victory against far larger armies from the other great powers of Europe. By 25, just two years after he got into the oil business, Rockefeller was running one of the world's largest refineries. Rockefeller and Napoleon needed experience, yes, but only a couple years worth, not decades. And Whatever was going on inside these guys' brains, an AI should be able to replicate or even exceed. There's no reason to think that humans are at the ceiling of learning quickly from experience. Furthermore, unlike Napoleon or Rockefeller, AI is going to accumulate years of deployed context in just a few days of calendar time. For example, imagine that you wanted these future AIs to run a Fortune 500 company better than its current CEO. How quickly could the AI ramp up at the shop? Well, if you let it shadow that CEO, on its first morning, it could read every email and Slack message and document that the company has ever produced.

By the end of its first day, copies of it have interviewed every employee and major client in parallel. Obviously, they wouldn't have access to feedback that arrives after many months of serial time. For example, how well they did on the next quarterly earnings. But even given these kinds of bottlenecks, we should expect these future post-Spark AIs to be able to get to top human performance in all domains within months of being developed. Distill 2028 or 2029 in this median scenario. And remember, even in this Spark scenario, all of this happens in just the first year or two after AI research is fully automated. Things don't just stop there, because even if the software progress peters out, AIs will continue stoking the fire by adding more and more compute. Even without help from AIs, global AI compute has been growing roughly 3x a year for the last half decade. And the AIs that come out of the Spark will be jaggedly superhuman and helping us continue scaling up this compute. They'll be extremely good at designing new AI chips and laying out the designs of new FABs,

even teleoperating robots to help build parts of FABs and data centers. So in the Spark scenario, AI capabilities continue to improve, but because their progress is driven by building more chips and data centers, and robots, it takes years to get to the same endpoint that the software-only singularity would have achieved in months, which is a superintelligence that is to us what we are to chimps. Appendix. So earlier, I promised to explain why I think AI research is largely verifiable. So that's where I'll start. After that, I'll go through what I think are the strongest objections to my arguments about RSI. Now, one thing I want to call out at the very top is that because both pre-training and post-training are verifiable, and I'll make my argument for why later, so is progress on what is the most galling and important efficiency that AIs have against humans, namely their lack of sample efficiency. If you want a neural network to gain some capability, you have to feed it far more data than it would take a human to learn the exact same thing.

But every kind of sample efficiency is measurable. For example, you can measure how much data it takes to reach a given pre-training loss, or how many tasks it takes to reach a given score in an RL environment, or even how much feedback and time within a single session it takes to reach a desired level of performance just from in-context learning. Sample efficiency and online learning are really two sides of the same problem, which is how to learn from real-world data quickly and efficiently. For decades, people have recognized this as the deepest open problem in machine learning. AI is equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problems, even when they are very difficult, as long as they are verifiable. Okay, now let's get to why I think that AI progress is largely verifiable. Why progress on sample efficiency or online learning, or the other problems in ML, is captured by verifiable proxies. There are roughly three broad buckets of AI R&D. Pre-training, post-training, and inference efficiency.

And as I'll explain, each has strong verifiable proxies that are easy to train against. And since together, these comprise all of algorithmic progress, that means that AI progress in general is easy to train against, which in some sense shouldn't be that surprising, because we can make models better at things that can be tested inside a data center, and AI training is literally happening inside a data center. As we talk about these proxies, you should keep in mind that verifiability is valuable for two reasons. So first, it allows AI companies to directly RL a model to get better at the specific parts of AI research through thousands of rounds of practice. But the second reason verifiability is important is that these proxies allow each agent to empirically check whether they've beaten some score on a leaderboard before they merged their contributions. And so they can work much more effectively in parallel. Okay, so with that in mind, let's start by talking about pre-training. So as you're building GPT-7, you can RL that model to get better at finding pre-training compute multipliers.

For example, you can make an RL objective that GPT-7 has to minimize how much compute it uses to train a GPT-2 size model to a fixed pre-training class. And once GPT-7 is built, we can now deploy this model, or a swarm of these models, to figure out how to train a GPT-3 size model to the lowest possible perplexity. And this is because loss on a held-out test set, which is the basis of free training research, is extremely easy to verify. Of course, what works at small scale doesn't necessarily work at large scale. And it requires taste to anticipate exactly which small-scale improvements will actually bear fruit when they're composed together in a bigger model. But this kind of intuition is also trainable. For example, you can make an RL environment where GPT-7 has to predict whether an algorithmic improvement that it made at the 100 million parameter scale will also hold at the 10 billion parameter scale. And while this taste bench environment is more expensive to RL on than the nano GPT speedrun,

it's still not that difficult to train on if you have frontier lab scale compute. So even important parts of research tastes are verifiable. And if you just automated pre-training, you'd already radically speed up AI progress. And there's a couple of reasons why. First is that pre-training leads to very general improvements in the model's intelligence and capabilities. And there's a deep explanation for why this is true, which has to do with the link between compression and intelligence. And if you want to learn more about this, by the way, I highly recommend this excellent video by 3Blue1Brown on the topic that I'll link to below. And the second reason is that as you make pre-training improvements, you can now train a smaller and smaller model that is just as good as a previously larger model, but which trains faster and more cheaply. And this not only speeds up pre-training, but it also means that your experiments and RL rollouts atop the space model are now more efficient. And since your automated researchers are just instantiations of this now smaller model, you can now support larger researcher populations too.

And the third reason is that because smaller models train faster, this actually tightens the RSI loop itself. During RSI, labs probably won't be doing three months long pre-training runs that are spread out across multiple data center campuses. Instead, the army of automated researchers will be finding algorithmic innovations so quickly that by the time that a giant run finished, it would already have been made obsolete by a much smaller model that first began training only a week ago. It's similar to how if you launch a spaceship to Alpha Centauri with today's technology, it would probably be overtaken in transit by one that was launched a century from now. Furthermore, since you're training smaller models faster and on a scale where you can run thousands of experiments daily, you won't have to bet that small-scale results will hold up in bigger models. You've already tested at the scale which you'll be training at. Now, how much better could base models get even at today's parameter sizes? I think there's probably a ton more potential. We are far from the limits of pre-training. For example, the AI Lab Magic recently announced that they pre-trained a model

that matches the leading open-source-based models today, like DeepSeq v4 Pro, for just half of the compute that was used to train the original GPT-3, which would roughly cost $500,000 today. Orders of magnitude further improvements seem totally plausible. Now let's talk about post-training. There are three different kinds of verifiable proxies for post-training. First, your automated researchers can try to improve your post-training algorithms such that if you want to RL on a new set of environments, it takes less compute or data or time, All three of which are easily measurable. Second, your automated researchers can work on making models better at learning on the fly. And you can easily measure progress on this. For example, you can measure if you drop your model into an environment that was never trained on. How quickly does it learn what's going on and figure out how to get a good score? And how fast does it get better at a video game that it's never played before? And you can have more real-world examples of this too. If you want to see what these could look like, check out this great benchmark that ByteDance released called EdgeBench for these kinds of tasks. And third, you can just measure AI R&D capability directly.

Anthropic, for example, has this internal benchmark called CoBench, which contains around 500 real problems that the researchers and engineers have solved this year. And every benchmark is also an RL environment. And an RL environment like this would allow Anthropic to have their models practice redoing the last three to six months of human progress at the frontier. And the third big area of AI research is inference efficiency. What this involves is designing your model and the surrounding software to get more juice out of your compute And this is especially verifiable For example you can easily measure GPU utilization or KB cache size per token or total flops in a forward pass or latency and so on. Now I want to deal with the strongest objections to my arguments about RSI. I want to be clear that these bottlenecks are serious and they do tamp down the probability of the most aggressive possible takeoff. This is why I think we'll probably make something like a GPT 3.5 to GPT 6 size jump in 2028, rather than some crazy overnight singularity

that starts tomorrow. But I think these bottlenecks won't be especially crippling, that the neck of the bottle is actually pretty wide. Let me explain why. Okay, so the first objection, and this is an objection that I had until recently, is that we'll be bottlenecked by continual learning, or the lack thereof. I spent the past year and a half arguing that the main thing missing from today's AIs is this ability to learn from on-the-job experience. Every new session is their first day on the job, and in each session, they can only retain a couple hours worth of information so they don't accumulate the kind of feedback and know-how that arrives over the course of weeks or months. Last December, this was my main reason for doubting that we'd automate AI research anytime soon. As I wrote then, somehow this automated researcher is going to figure out the algorithm for HEI, which is a problem that humans have been banging their head against for the better half of a century, while not having the basic learning capabilities that children have, I find it super implausible. But I've somewhat changed my mind on this. Here's why I don't think that the current lack of continual learning in these models will stop

full automation of the best AI researchers by 2028. First, AI companies might just solve continual learning by then, with architectures that are much like the ones they already have. Previously, I thought that we were bottlenecked by some crazy breakthroughs that would allow models to update their own weights at the end of every session. But now I think that scaling up in-context learning is enough. And so you'll have sessions that are effectively many months long, and this will allow a single instance to learn effectively on the job. And people have published many, many promising ideas that would let the labs achieve dramatically longer effective context windows. And the second reason is that you might not need a continual learner to solve continual learning. As I explained earlier, online learning is an ML research problem with clean, verifiable proxies. If you'd asked me two years ago, which would be harder to solve, Navier Stokes or Continual Learning, it wouldn't have been obvious to me that the answer was Navier Stokes. Now, of course, it turns out that Continual Learning did turn out to be harder, given the fact that the labs haven't cracked it yet. But are we so confident that it's that much harder that the non-online learning models a year or two smarter

than the ones that are currently solving Millennium Prize problems won't be able to crack it? even if they're amnesiacs like the current models are? These models are out here solving millennium prize problems while still being these derpy savant interns that have no long-term memory. It's like watching somebody do cartwheels down the track and still keep pace with Usain Bolt. If they were actually allowed to run, I'm sure they could go even faster. But who's to say that they can't win this race while still cartwheeling? Now, all this being said, I think there is a serious chance that we don't solve continual learning within the next few years and that models are not able to compensate for this missing capability through their other advantages. That's why I put a 25% probability that we won't automate ILIA-level top human researchers until after 2032. One important objection people have is that we'll be bottlenecked by finding the right verifiable proxies. All the verifiable proxies that I've listed above assume the current paradigm, they would argue, but figuring out what objective to verify in the first place, choosing which hill to have GPT-7 climb

is itself an important part of AI research. This is what one might call research taste. And how are you going to train for that? Before 2018, almost nobody was saying that minimizing loss on next token prediction is going to be the most tractable path to AI progress. So nobody would have bothered to build nano-GPT-style speedrun environments where the agent swarm has to minimize how much time it takes to get to a certain pre-training loss. Wouldn't there be multiple new paradigms that we have to cross to get from here to superintelligence? And won't each new paradigm require some associated proxy that we don't know ex-ante? How will the automated researchers find these during RSI? I have two responses to this objection. First, the transformer was visible on, in fact invented to beat, the existing translation benchmarks. A similar story is true for reasoning models and the pre-existing coding and math benchmarks. Proxies made before a paradigm leap seem perfectly capable of capturing large step changes in performance. This is because AI learning algorithms are quite general. And second, it's not that difficult to come up with paradigm-agnostic outer-loop proxies.

For example, how many Millennium Prize problems does your new model solve? And if you train your model on data before 1905, how long does it take before it discovers general relativity? How fast does it master totally unseen video games? And so on. All in all, I think the other objections to the impact of RSI are much stronger, and I'd be very surprised if this ended up being the reason that RSI was not a big deal. The other big bottleneck that people point out is data. Isn't data one of the main inputs into AI progress? How will RSI happen without that scaling up at the same rate? My answer is that data will continue growing during RSI, but the AIs will be the ones compiling it. Producing better training data for both pre-training and post-training is itself an ML research problem, and AIs are getting increasingly better at doing it themselves. Now, it is true that data is an extremely important input into AI progress. Together with Jerry Hahn, who's a student at Princeton, I actually did some research into this. We found that for pre-training, over the last six years, more than three times the algorithmic progress has come from data improvements than model improvements. However, I don't think that people realize that these data improvements in pre-training are very much in line with normal AI research work.

It's basically figuring out which techniques for filtering existing data or generating new synthetic data empirically lead to lower loss on a held-out corpus. A similar thing seems to be increasingly true for RL. DeepSeek just published a paper where they talk about how they post-train their new model. They say that changes to the basic RL algorithm have not mattered nearly as much, and that, quote, systematic improvements in the scale, diversity, and verifiability of synthesized data and environments account for essentially all of the observed gains, end quote. And the way in which these environments are made is itself getting more and more automated. They write, quote, we invest in large scale automated pipelines for data synthesis and environment construction. Using difficulty and correctness as reward signals, we iteratively train the model to construct better tasks, end quote. Now I want to talk about the idea that AI progress will be bottlenecked by experiments. So earlier I made this extended analogy between the current acceleration in mathematics and what we should expect to happen in AI research. But there is one big difference between AI research and mathematics.

In math, you can simply reason your way with pen and paper to new discoveries. Whereas in AI research, you need to run actual experiments to validate your ideas. And these take compute and time. And as the number of automated researchers grows, each gets access to a smaller and smaller slice of the total experiment compute pie. My answer to this is that each agent will still get access to a ton of compute, basically. Frontier Labs have been more than quadrupling their compute every single year. And if this trend continues, then by the end of 2027, OpenAI could almost give 10,000 agents, so the amount that were involved in solving the Millennium Prize problem, each enough compute to run a GPT-3 size training run every single day. Keep in mind that the final model, trained in the original Transformer paper in 2017, took three days on 8P100s, which would cost about 50 bucks if you went to rent it today, or about one ten-thousandth of the training compute of GPT-3. If the top human AI researchers, like Norm Shazir and Elias Satskova,

had enough experiment compute to train a GPT-3 size model every single day, they might make slower progress than they currently do, but it would be extremely surprising to me if they failed to make meaningful progress altogether and just sat down and started complaining about how small their cluster was. Now, how productive do you think an AI lab would be if it had 10,000 great researchers, each with this much compute and each thinking 10 times as fast? You wouldn't want to write a check to this Neolab? The nature of research and experimentation at the labs today reflects that researcher time is extremely scarce relative to compute. Top AI researchers individually get paid hundreds of millions of dollars a year. So the way labs use compute today tells us very little about how efficiently they could use it if compute rather than researcher time was the key bottleneck. There are two main ways in which the RSI agents forum will use experiment compute much more efficiently than researchers today use it. First, immediately they can trade labor for compute. To save valuable researcher time, experiments today are, and should be, very wasteful with compute.

So instead of painstakingly setting up and carefully monitoring tons of small-scale pre-training experiments, A human researcher today might instead kick off a couple of huge multi-agent RL runs in parallel. But if you're bottlenecked by experiment compute, you can just do more of the former. And when those automated researchers are doing an RL run, they could read every single rollout transcript and immediately pause and rebuild the whole environment if some task is broken. They could be doing crazy amounts of small-scale ablations of their models to make them oh-so-slightly better. And when a team of agents decides to combine their compute together in order to do a much bigger run, they could spend many subjective decades designing the most informative pooled experiment. And once it's done running, they could spend further subjective decades comprehensively analyzing it before designing a new experiment. The other big way that this huge agent swarm could remain productive, despite the fact that each individual copy will have access to less compute, is that these automated researchers could come up with a better science

for learning from small-scale experiments. Scaling laws themselves are a good example of this. The GPT-4 paper says that they were able to predict the model's final loss using models trained with 10,000x less compute. One of the things that the smarter and faster automated company could be doing is trying to discover more scaling phenomenon like mu-p, which lets researchers anticipate the optimal high parameters for a larger run from a 100x smaller model. How many more similar discoveries like this are there? I remember chatting to an AI researcher who thought that learning from extremely small neural networks could be incredibly informative. He put it this way, The gods above must work from the same principles as the ants below. I want to thank my friends who paired with me through significant parts of writing this essay. Max Behrens, Gabe Freed, and Thomas Larson. I'll see you on the next essay about superintelligence, but in the meantime, if you want to get notified when that video essay drops, please subscribe on YouTube. All right, I'll see you next time.