AIは「仕事」より先に「タスク」を奪う――MITのNeil Thompsonが語る自動化の実像

All right, so today we welcome back Neil Thompson, director of MIT's Future Tech Research Project, who was last on the podcast in the year of COVID, October 2020. So, Neil, welcome back to the podcast. It's my pleasure to be here. So, I guess before we start, give us a one-sentence, one-to-two-sentence description of the Future Tech Project. Yeah, so we're a research group that sits both within the Computer Science AI Lab and the Son School of Management, both at MIT. We're one of MIT's largest research groups. We've got more than 100 people. And we're quite unusual in that we're about half computer scientists, half economists. And so we try and think both about technically what's going on and how you can put that into big models so that we know how it's going to affect people, how it's going to change growth, those kind of big questions. And at some deeper level, you know, if we think broadly about the things we do, we think about what's driving AI progress, what effects it's going to have.

And then we also do some work specifically on AI risk, which, as you can imagine, has been a lot in the last week or so. Yeah, yeah, yeah, yeah. All right. So the reason I have Neil back is they have a couple of really interesting papers that caught my eye. But when Neil was last year, we talked about a paper they released at the time on the computational limits of deep learning. So maybe let's go quickly and score that paper, I guess, six years after the fact. So I guess one of the things that it got right early was the cost pressure, right? So around chips, capital, electricity, things like that, right? So let's stay on the positive side of the deck first. So what are the things you feel that paper got right? Yeah, so I think what you said is right. I think we were on the right side of it in terms of saying, like, this seems like that the scaling those are one of the main drivers. And I think, indeed, that has been true. and that would meant that we were going to progress from systems that were really not that expensive to systems that were very expensive and where that cost pressure was going to be important

and of course now we're really seeing that right we see these systems where all of a sudden people are like oh maybe don't use the most advanced system for these things so i think that's that's one of the things we got right i think another thing we got right but didn't call enough is when we actually did the economic modeling of this we implicitly put in some terms for algorithmic progress right and we we were people were really not talking about algorithmic progress in that we sort of we had measured in that but we really didn't know sort of what was going on there we really didn't call it out pretty much and i think it was just stuff that we people at epoch and a few other places did over these coming years to say oh let's actually call that out and realize how important that is so i think those were sort of the the two things that we got right one big thing we got wrong which if you want to make that turn yet yeah on the algorithmic progress I guess you folks kind of hinted at it at a high level, as you mentioned, but obviously on the specifics, like, you know, now we're using mixture of experts, maybe quantization, distillation and so on. Right. So but at least you kind of hinted that that's probably going to happen just because of pressure.

Yeah. So so we I mean, we knew that algorithmic progress was going to happen, you know, had to happen. Like we said that that pressure was there, but we actually implicitly, if you actually look at the progressions we ran, where we would try to estimate how fast these models were improving as you're guiding, we put in effects that said over time, you might be able to get more out of it. And so we were actually implicitly already measuring that. But as I say, like the problem was we didn't know exactly what was the thing that was going on there. So we didn't really call it out at that point. But what's clear is you can even go back to that thing and you can see that it's actually a very substantial number. And so this is, you know, I think that's turned out to be something, you know, you gave a couple of examples of it, of course. By far the biggest example was Transformers when they came out. But those are the kind of things where you've seen suddenly we can harness every additional GPU much more effectively. That's made a big difference and particularly made a big difference for the biggest models that people use. Yeah, yeah. And so there were a lot of innovations or at least adjustments on the algorithm side, but also just on serving and inference, right? It's a speculative decoding and things like this, right?

But that came after the fact when obviously there was much more heavy usage, right? Exactly, right. I mean, there has been a lot of pressure. I mean, I think, you know, we correctly called out that pressure, although at least I was surprised by the extent at which people had been able to do it, right? The amount that they've been able to get year after year in things like making inference more efficient, right, has been remarkable, more so than I imagined at the time. At the time, we thought that people were going to move more towards traditional algorithms. So we know that at some very deep level, like the deep learning algorithm is actually a pretty inefficient way to learn things. And we thought that people would push it towards sort of more like base level algorithms where we know they're optimal for the problem. And we really have not seen that, right? The ability to do that training has been so powerful. but it's only in this last little bit as we've gotten to agentic systems where now suddenly we still have the AI doing it but now it's calling the tool and that tool of course can have a very efficient way of doing something so it you know calls a c library that sorts a list or something like that and it doesn't actually have to sort that list itself and so we're sort of we sort of

finally got into it but not at we we were sort of imagining that would be more about a heart of how AI was trained and now it's really more like AI has just been trained in a very broad way and now we have it learning to use tools and they can now learn to use those tools efficiently. And of course, we're recording this on the week when the biggest new or one of the biggest news among developers is the release of this or the announcement of a project called Jev, which I don't know if you've been following this thing. It's a, it's kind of gotten a lot of attention, but basically at a high level, you know, it's just that realization of going back to basics, right? So it was just, do you really need a chat bot for everything? Right. So in other words, what these systems like. And by the way, Jev turns out to be built on ideas from even BERT, and people have built systems similar to Jev. But what Jev does is basically, okay, so what are you asking? All right, so I'm going to decide and route to the right thing, right? So it's more than a router in some ways.

So it's basically, I guess, a good example is if you're in DevOps, right? So our site has been down for two hours, and we are losing money. Fix this now. So an LLM will say, yeah, it looks like the site is down. I would route this to the infrastructure team. Whereas models like Jeb would go, okay, so what's the probability that we should route this to the infrastructure team? Score it. At what level of urgency? Score it. Is this going to cost churn? Score it. Right. So so in other words, an LLM is always answering you with text when text is not necessarily the answer you need. Right. So I have not been following this particular race, but it sounds like it's more related to doing a little bit more planning up front and the optimization of the process. Which, you know, of course, is just as we build these longer flows, it makes so much sense that that would be. And as we use these systems for more, it's going to make more and more sense to do that as well.

So I understand why that would be true. or what you economists in the Kahneman world will say, system one versus system two thinking. And so one other thing about the paper, which was Vintage 2020, of course, back then, the benchmarks were very different, right? So I think you guys were mostly focused on very specific benchmarks in computer vision and speech and things like that. But nowadays, people don't even follow those benchmarks anymore, right? So there's benchmarks for agentic work, for long-running agents. And then the main thing that Frontier Model has used are these benchmarks like Arena, right? So Arena.ai, which is – so I don't think you would have anticipated these new benchmarks back then, right? Oh, well, I mean, I think we always thought that there would be benchmarks, but I think that it's, you know, what – you know, I guess we would advocate is even – you know, benchmarks are becoming more realistic, which is great, right? But, you know, I think a lot of the early ones were much more convenient samples, right?

I mean, we focused a lot on ImageNet way back then. And that was, of course, you know, sort of an amazing work that has been done in Stanford to be able to put these databases together. People just didn't have that scale of stuff to do before. But, you know, what we have said since then is what you really care about is the effect on the real labor market, right, and the effect on the real economy. And so as much as you can, you want to get closer to those tasks. Those are the things that are really going to make the biggest difference. Which is a great segue to the first new paper I want to talk about called Crashing Waves Versus Rising Tides. And I guess before we go through the results, Neil, maybe describe the setup. It seems like you guys did a pretty big survey, right? So what was the setup and what were the limitations of the study? Sure. Great. So, broadly, you know, the goal of our doing this was to say we actually wanted to get as close as we could to people on the ground evaluating how well AI could do, right? And so, you know, like previous work, you may be familiar with, for example, the Alundu paper that came out of OpenAI and Van Rock and a few other folks there.

And, you know, really very, very important, set a lot of things. But if you actually, you know, sort of dig into it, it's ultimately saying something about what AI could do, not what AI can do right now. Right. And so in particular, it's sort of a thing like, ah, could you imagine that, you know, we can automate reading the next rep? Right. It's like, yes, that, you know, lots of people in that area were thinking about that. Yes, they can easily imagine that. Whereas, you know, we said, well, in practice, you know, if you talk to a biologist even even a year ago, right, they were saying these systems are not well calibrated. You have to be almost perfect. Like there were lots and lots of real world things that were getting in the way. And so we said, no, we want to measure what it is today, right? What it's doing and how that's changed over time. And so one of the things we're going to do is we're going to use more than 40 models, different models for asking these questions. So we get to span a whole bunch of time. So we can look at as these models progress, how do tasks change? The other big thing we did to make a difference. Who were the subjects? Exactly. So the other big thing we did is we did subjects. Whereas a lot of other papers have asked computer scientists this question, right?

Because they needed to know what AI could do. We said, no, what we want to do is we want to ask, if it's saying like, hey, you could automate a plumber writing up an invoice. We want to ask a plumber. Is this a good invoice for plumbing? If it's a customer service person, write me back by message. We want to ask the customer service person. And so what we did is we used the prolific platform to get to people who actually do these jobs. And we required them to have experience in these areas. and we did a bunch of little checks to make sure that they were, you know, reasonable in that. And then we gave them these examples. And so we said, okay, well, for each of the tasks in O-Net, for those who don't know, O-Net is this thing that the government puts together, in particular, the Bureau of Labor Statistics puts together. And what it does is it specifically has the tasks that are associated with each job. And so for each of those tasks, we ran into a pipeline and we said, first of all, like, could an LN help this at all, right? And if the answer is like, you know, move a box from one side of the room to the other, okay, no, we put that aside. All of the ones that are using text are using those kind of things. We say, yes, okay, that's, and no one could help with that, right? And then what we did is we said, okay, well, the tasks as they're described are very, very broad, right?

So imagine, like, a dentist might have a task that would say, give a patient aftercare advice for how to care for their teeth, okay? So you can't really, like, that isn't, like, a single instance of a task. It's a broad category. There are a lot of things in there. And so what we did is we used GPT-5 to produce a bunch of examples of this, right? So it might say, well, you have a 10-year-old who has diabetes and needs to care for, you know, their gums, right? What do you tell them? You have an older patient who, and so in each of these cases, we produce these instances. Okay, so this is sort of starting us down the road of the way we want to go, but we still wanted to be short, right? We weren't positive that each of these instances would be real. And so one of the first things we ask everybody in the survey is we say, is this a real instance of the kind of thing you might imagine? And does it represent like you have all the information? Is this something that actually you could imagine being a realistic way? And so anything that doesn't get in the very high categories, we say, hey, we don't do we generate a new one.

All right. So now we have this nice set of tasks of like task instances. And then what we're going to do for each of them is we're going to have five different LLMs produce answers to that. OK, and this is going to span, you know, going back from quite small models all the way to some of the work that is not yet in this paper, but we're already running now. There are some of them, some like Astra and some of these things. So this is sort of an ongoing thing that we're doing. And the idea is for each of them, you can say you have the dentist looks at it and says, OK, well, this is a great answer. This is a terrible answer. Right. And they give that ratings and then you can actually look at the progress that's being made there. and it infers something about how AI is progressing and what that's going to mean for the real task in the economy. So how would you, going back to these tests, would you describe the task as what's the complexity and how many steps are involved in the task? Or is it a one-shot, here's a prompt, and then here's the response? Yes. So in terms of the actual, like, the way the interactions are done,

it is all just, like, there's a prompt and there's an answer. We do have some models that have reasoning caterability, And so they are able to do like multiple steps in order to answer the question. But yes, in all of these cases, this is sort of single interaction versions of this, because like one of the things we wanted to do is we did not tell the participants that these were AI answers. We didn't want them to say like, you know, I'm either biased for or biased against it. Right. We know from lots of lots of other work that that exists. And so we just said, how good an answer is this? And then the participants you would describe as what, representative of knowledge work in general or much more than that? Yeah no much more than knowledge work I mean because we didn like if you had a task like a plumber right much of the funding task would not have been something that was LLM addressable but a small number of them would have been And so we would have asked plumbers about those ones, right? And so it actually spans many, many of the occupations. We have well more than half the occupations covered now. I think like in terms of having at least some of their tasks, I think we have like 70 or 80% of the occupation. So we're really covering

a huge swath of the U.S. economy. And obviously the findings are encapsulated in the title. Right. So rising tides. So what are what are the key findings? Yeah. So so the first one here is this sort of crashing waves versus rising tides contrast. So I imagine you probably have on the podcast before talked about these results from meter about the time frame. So, you know, for those listening right now who don't know this, you know, very, very good group out of Berkeley had done this work where they looked at software engineering. and they showed that these systems were very very capable of short tasks very very incapable of doing long tasks and it was this very steep transition between them and so you had this what this meant was and then and that horizon was pushing out so you might say like at some point in time these systems were could do anything less than 10 minutes couldn't do anything longer than two hours and there was a sharp drop between between those two things and that might move out over time as these systems got better so we described this world as a crashing waves world and that crashing

ways because what it means is that you could be sitting in your job and you're looking at AI and you're saying, hey, no problem. I'm good. Maybe you can do 1% of the things I can do. I'm really not that worried about it. That describes mathematicians two years ago. Exactly. There'll be a nice connection we'll make between those two things. I'm glad you brought that up. They're in that world and then suddenly this horizon passes you by and all of a sudden they can do everything that you're doing really well. Right. And so that's a world where, you know, we think you think of this as like in the analogy, you're walking, walking along the shore. Right. You're looking at you're looking at the waves and you're like, yeah, they're down to my feet. I'm not worried about it. And suddenly one of these rogue wave comes and you're just totally drenched. Right. And so that's that's the question. Now, it turns out if you actually ask like why that might be true. Right. Why you might get this crashing waves phenomenon. We think it's driven by the fact that when you're doing something like a software engineering task, you actually are doing lots of little things along the way and you have to get each one of

those things right in order to get the whole thing right right so imagine you're saying like okay i have a bug fix for you know some library that i'm doing okay i have to understand what the bug fix is i have to correctly locate it i have to go make those changes i have to propagate anything that i needed to do there that i need to do the testing on it there are a whole bunch of things that you need to do if you get any of those wrong the whole thing could easily fail right and you know we've all we've all had those instances where we did coding where it just takes you forever in order to debug something because you gotta get that last little thing right so if you have that where it's like you have to get each of these things right you can imagine that it's like oh well maybe i only have a two percent error rate or five percent error rate if you think but like dvd5 right said five percent of the answers it got had some factual mistake in it right so if you take five percent of its baseline okay well if you're the first thing you do you get 95 percent of the time you get right second one 95 percent of the time etc what you get is an exponential drop off and how good you are. And if you plot that in the way that they have on a logarithmic scale, you get this nice drop. Okay, so that's why you would get it. But the thing is, a lot of tasks

in the real economy are not like programming. They're not something where if you get a little detail wrong, the whole thing fails. So think about writing like a marketing plan, right? You got to know who your customers are. You probably do some analysis on that. You think about what your products are and match them to them. And then you, you know, maybe do a specific offering for each of those. And each one of those processes, as long as you get it largely right, you're probably going to still produce a useful marketing plan. It may not be as good as it could be. But that degradation is much more graceful than it is in software engineering. And so in those circumstances, you would not expect this incredibly rapid drop-off. Would you expect as much flatter line where you're still better at shorter tasks than you are at longer tasks, right? Here, we're here. The number of steps that you do is sort of being proxied by how long it takes. But that gentler curve means that you shouldn't expect it to be suddenly a switch that turns on and you're automated out of the way. It should be a much more gradual process where you say, oh, 20 percent of my things it can do. Oh, now it's 30 percent. Now 40 percent. And so you sort of see it coming over time. And that's what we think is happening.

That's what we see when we look at the overall distribution of these things. We see a rising tide, but not the crashing waves that end almost in place. So how did you infer that from the study that you described, which is basically just a prompt response rating kind of study, right? So how can you tell that if my task actually involves a series of steps, that what you described is actually happening from the data? Yeah, absolutely. So here what we're doing, right, is we're sort of making this equivalence between how long it takes a human to do and the number of steps that are involved in it. right and we you know i think this is a pretty reasonable thing to assume right i think we all know that sort of like you know someone is writing a book right there are a lot of subsets involved in that process like you know that could be a year-long process if someone is right you know someone is you know making a small change to a flyer or something like that you know that's a pretty short task there are not that many things that are hard and so this is the sort of the broader privilege so we don't actually know the exact number of subsets that's a of course a very

a hard thing to imagine, you know, because you even need sort of like the demarcation. How do you, you know, sometimes people sort of transition from one to another and you would be hard to pull them apart. But we're using that as the way to do it. OK, so now you can say, OK, well, you know, if we're proxying by how long it takes a human to do that, right, now we actually don't need to have lots of individual steps in our survey. What we need to do is just ask an LLM for about tasks of different lengths. Right. And so some of these are going to be tasks where you say, OK, this is something that, You know, the dentist tells the patient that probably only takes them a couple of minutes to give that information. That's a pretty short task versus, you know, you say, OK, well, I need to totally change the layout of my factory or my retail store or something like that. Right. That would be a much more involved process that you would have to do. And that could that could be substantially longer. And so all of that gets embedded in the prompt in terms of what we're asking the LN to do. It produces an answer. And now the human looks at it and says, oh, yeah, that's a good answer or not a good answer. like independent of how long it actually took.

And then we just see empirically that indeed those longer tasks are performed less well. So I guess what you're saying is that in your interpretation of the results of the study is that the impact of AI is already happening slowly, right? So across a wide variety of roles and occupations, right? And so if you're one of these people who thinks that AI is something you can afford to not invest time in to learn, you might be shocked later. On the other hand, so going back to the example of mathematicians, because that's actually my PhDs in math, but I was not a great mathematician. I think now I would be with the right AI. Right. So what's happening there is it seems like the generation of the answers has gotten cheap. The verification is also doable because you have tools like Lean, right? So formal verification tools. To some extent, even the generation of the paper, I mean, to some extent, has gotten easier.

So the last bottleneck seems to be the validation and acceptance. Acceptance meaning here's a paper that mathematicians will accept and publish and reward the authors for. So it got into the point where the broad math community wants to hit a pause button for a couple of reasons. One is, in their mind, there's only a finite number of these really famous problems. If AI solves all of them, it may not be good for the profession. because one of the things that happens is even if you don't solve a problem, maybe by banging your head against the wall, you go around the problem and you discover new things, right? AI is just focused on optimizing to solve that problem by brute force in many ways. And then secondly, on the acceptance side, if everyone is, Neil, generating work and even generating papers, how do we know which of these people actually are good? So one proposal is if Ben submits the paper before and it looks like the paper is good before we actually acknowledge Ben, maybe Ben should give a talk in front of other mathematicians.

So in other words, if Ben can't actually even give a talk, then he doesn't actually understand what's in the paper. Right. So so I don't know. This is a long winded way of saying that in that case, the disruption happened suddenly, but not surprisingly, because there were people who knew that this was happening. But the broader user base, so to speak, in which case the mathematicians were it was a mixed bags. Right. So some were using it. Some were in denial. Some were enthusiastic. Right. So I can see that happening in a broad range of professions. Right. So I think, I mean, for me, I think, you know, math shares this characteristic that we see with coding, right, which is that there are a whole bunch of characteristics about it that make it very amenable to an MAI. So it's both one of these situations where, you know, people write it down, right? It's almost entirely mental, right? And so you don't have the like, oh, now you have to move the box across the room kind of problem, right? And as you say, like the fact that there's abundant data and that it's verifiable with tools like EAM mean that it's, you know, it's a very, very good thing.

And so, you know, in that sense, you can think about things like Go, right, where some of these early systems, you know, were able to, by self-play, you know, vastly exceed what Kinglands were able to do in these areas. And so there, you know, I do absolutely see that there are those kind of things that can make it very transformational for an area. But I would say, like, you know, even when you hear about what, you know, some of the frontier labs are doing in this, right, it's not quite hands-off yet. There are still mathematicians involved. They're still pointing to that good question. they're still looking at the answers and saying, nope, that's not, you know, you haven't thought there yet. And so that, I think that to me is an interesting example because it certainly is telling us that AI is a very, very powerful tool. And we've actually done some work that's coming about shortly where you can actually look at sort of the share of math problems that are being solved that correspond to AI and indeed in this, particularly the last six months, right? The AI has been responsible for a very significant share of those things. Although many of them AI assisted, not just AI on its own. Right. And so I do think that's there. But, you know, one of the things that we looked at in our survey is we distinguish between sort of it can automate at the level of the basic employee, an average employee or superior quality employee.

And that, you know, and it turns out there are very, very good differences. Right. We find that these systems are capable of automating something like 60 percent of tasks at the minimally sufficient level, but only 27 percent of tasks and a superior quality level. Right. And I think what that what that's implicitly telling us for the math case is that there are some parts of what it's doing to solve these problems where it is, you know, very frontier. Right. It's able to look at, you know, counter examples, especially well, for example, and things like that. So it's able to do that very, very well. Right. But there are still probably some parts of that process where it's not doing it at the cutting edge. And the mathematicians are helping point it at this. And so that says to me, like, even though it is one of the most frontier areas, there is still a role for humans. at least today now because this is one of the easiest areas to automate i you know and of course i imagine this will continue to develop pretty quickly um over the next little while yeah and like i said since everything is written down these schools obviously exceed excel at reading everything which uh anyone who spent any time in academia knows academics are tend to read very narrowly

right so so they can't really draw ideas from things that are even just a hundred yards away from their office, right? Indeed. Although I would also suspect, having seen lots of it myself, right? And we also, there are also lots of academics with pile of the papers that they intend to read that they never get the heritism time constraints. So it's both like, you know, they read narrowly and even when they're trying to read more broadly, sometimes they run out of time. So as you described, right? So humans are still required in order to obviously prompt and direct and iterate Right. But that's because you have a class of humans capable of doing that. If you get to the point where a lot of the tasks that you need to train the next generation are gone. Right. So imagine if you're a professor in math. Right. So OK. So Neil is my graduate student. I'm going to give Neil this problem to work on. But wait a minute. I can just use an AI. I don't need Neil. Right.

This is a graduate student problem can be done by an AI. So then, you know, you lose. It's that kind of apprenticeship and training that is a little worrying in some areas, right? Yes. So, I mean, I think that's certainly true, although I would say, I guess. So, by the way, Neil, before you answer, so in programming, it's the same. You can imagine the same thing, right, which is basically, you know, I'm an engineering manager. Suddenly, all my engineers are 10x more productive. well, let's not hire any entry-level people because we don't need them, right? So front-end people that will do entry-level jobs, we don't need them anymore. So then you lose that pipeline of people who can architect and prompt much more advanced systems. I want to answer this in two different ways because I think there are two important things that I think our research has shown that I think are important and one that our research has shown, one that is, I think, more general. So I guess what you described seems plausible to me But I also think that when we often when we put ourselves in that world we forget the power that AI is going to have to help teach people right so if you think about the you know many apprentice models so think about something like like a junior lawyer right and it like okay well you know you read these

documents and you get a sense of what contracts look like and that helps you right that's true but the amount of feedback you get is pretty small the amount of like the interesting cases where things really matter that you get is pretty small, right? Most of the contracts you read are boilerplate, not that exciting. The amount of time feedback you get from a senior partner is very little. And so the actual pace of learning that you get is very, very slow, right? And so you might say, okay, well, that's why it takes three or four years when you're, you know, very junior role before you move to the next role. It's not at all obvious to me at the point where AI can do this. It's true that, like, that person's role in the production process disappears. But as you move that that learning into training, now you have a system that can show them, you know, like 15 different contracts every hour. It can show them here are 20 different ways of writing those clause, which are the good ones and which are the bad ones. Like the piece of learning that humans could have from it is greatly accelerated. And so assuming you are a long term thinker and you still hire them.

Right. So so imagine you're an investment bank. And before you had all these fresh MBAs from the top school working on pitch decks. Right. So and you had 50 of them. Maybe now you have five of them. It absolutely could happen the way you're talking about. But if it's a training function, it doesn't have to necessarily live within the firm. Right. It could live within academia. Right. It could live just as a open source resource. They you know that they say, oh, you know, here is a way to do it. Isn't that a sore subject, AI and academia? A lot of academics don't know what to do, right? What to teach them. Right, exactly. Absolutely. Right. And I think, you know, you have this at lots of places, right? We're having to say, well, you know, maybe what we want to do is, you know, train people to do things both with and without AI because they want them to learn how to use the tool, but we also want to make sure they understand. And so you've got this stuff. Okay. So that's, I guess, you know, I don't want to overstate this. I do fear some of the things you're talking about,

but I also think that there are ways for us to adjust. As long as there are still, like, once you climb that expertise ladder, there are still human jobs, I think, you know, I think there's still going to be ways to do that. Now, of course, you know, the risk here is AI is moving so fast that as you start to train up, AI is getting there faster than you. And I think that is a sort of a deeper worry. Actually, there were two people that put forward similar but not identical ideas. One was Bill Gates, who wrote a long essay, which I admit I didn't read. I just read summaries. Still on the desk. Yeah, yeah, yeah. Where he kind of described the notion of human reserved roles. Right. And then Terence Dow, the Fields Medalist mathematician, you know, also kind of started speculating, well, maybe some problems we set aside. because problems are a non-renewable resource and they're a way for us to train that next generation, right? So I'm just trying it out there. I don't know how you feel about this notion that maybe you set aside some things just for training the next generation.

So, you know, here I think, of course, it's really going to depend on what you lose when you do that, right? I mean, you know, if this is a problem where you say, you know, we solve it and energy use goes down by 1% for everybody, You can say, you know, I don't think I want to reserve that one. I want that savings for the world. So I think there is just going to be a matter of, I also would say, you know, if AIs can generate these answers, I wouldn't be surprised if we do it so they can generate a lot of interesting questions as well. So even though that big reserve is, you know, the existing reserve is limited, that doesn't mean that it has to be in the future. So one thing about your study is, for our listeners, just for some caveat here, the focus is on current capability, right? And as you mentioned, you're continually kind of refining the study and gathering more data. But it doesn't really tell you anything about how they're actually using AI. So in other words, adoption and usage, right? That's right. So we actually will very shortly have a new paper coming out where we do do some of that comparison.

Oh, give us a teaser. so the short answer is that we do see adoptions lagging significantly behind the capabilities and i think there are two reasons for this so one reason for it is in the study we intentionally give ai all the information it needs to solve a problem whereas in the real world there are lots of cases where those information flows don't exist yet right so you can imagine the dentist we keep talking about right that dentist has just looked in this person's mouth right they just all all these things, like usually only later that they write in stuff in the medical record. And actually, it might not even be everything that they write in. It might only write in some things. And so the AI might just not have all the information it needs in order to do this. So for our study, we intentionally gave the system all the information it needed to make a good answer. But that means that, like, as people have to adapt flows, all that has to go on. The other thing, of course, is just that in practice, it takes a bunch of effort to adopt AI, right? This is not, I mean, you know, even if it's something as simple as just calling the API, you have to set that up, make sure that it's going to be reliable.

And of course, it can go significantly more difficult than that as you build an authentic pipeline, you build markdown files, you're building, et cetera, et cetera, to make it all work together. And so all of those things can carry costs. And we think that friction exists. And so what we do see is right now, something like half of the potential, it looks like about half the potential gains that you would get from adopting this are sort of eroded by all of these other costs that you're having to pay. And that's what seems to be like slowing down the benefit that people are getting. But we do see very substantial drops from year to year. And so that, you know, it seems like as I think as you expect, as people build better tools, better platforms in order to make easier to implement, more and more people using it as they build it into stuff that people are already using. That helps it as well. So the desire is there. It's just that the infrastructure or the data pipelines or whatever this are not yet in place. But assuming they're in place, people are willing to adopt. So, I mean, we certainly do see adoption and we see adoption moving pretty quickly.

There are, of course, other challenges as well. So economists kind of for a long time, you know, I think their response to computer scientists who say this is all going to happen incredibly quickly is to say, well, that may be true. But we know in general, lots and lots of processes have bottlenecks. And if you automate it prior to the process, something else can become a bottleneck. And so, you know, I've had many, many discussions, for example, with people who run software engineering organizations. Right. And they say, hey, it's amazing how much more code we can write. And it's amazing how frustrating it is for our senior people who have to review code because they're getting lots and lots of stuff in there where people who don't know better are just handing them stuff. And like, yeah, that's pretty good. And to give you a sense of the scale of this, so Mert Demmer and some co-authors did some work at MIT recently, and they showed that when Cloud Code and some of these other tools come out, you get something like an 800% increase in the amount of lines of code written. And then you get many, many fewer, like you go with an increase, but much, much smaller increase in the number of commits that people are doing, the number of files that are created, etc.

To the point where if you actually look at like real releases into the world, it goes up 20%. So from 800% to 20%, right? And that's really telling you something is going on there where, like, there are bottlenecks that mean that you produce lots more lines of code, but maybe they're not as good ideas. Maybe they'll fit in as well. You're not able to sort of leverage it in the same way you were, and hence you don't get all of that flow through. And so, yes, moving fast, but there is this sort of diminution that happens. Yeah, very similar to the mathematician's observation of generation is cheap. verification is getting easier but the acceptance and validation and in the analog and software would be deployment is not keeping pace so let's pivot to your other paper so that we can wrap up so the other paper ai scale and task space series of automation unlike the other the previous paper on rising tides which was empirical this seems to be a little more speculative and theoretical

instead of fair. So, I mean, it is also empirical in the sense that we're going to ground it. But yeah, I mean, I think the biggest contribution here is theoretical. I agree with that. And so give us a high level description of the findings. So broadly, the world we're in for this paper is we're thinking about, OK, once AI starts to replace tasks in the economy, how fast does it happen? What does it mean for the world? OK, now this type of model is something called a general equilibrium model in economics. And this is one where you don't just simply say, well, you know, software, the major software developers changed. That means their wages changed or something like this. The general equilibrium means that, like, different people might choose to use software, different people might move into software as the wages change, and everything has to rebalance before things come up, okay? And so this is, you know, really what macroeconomists think about when they think about managing the whole economy. It's particularly important if you think about something like AI, which is going to have very broad effects across a lot of times.

And the big contribution of our model is to go beyond what a lot of people have said. So most models of AI, and in fact, substitution between machines and labor, focus only on the marginal cost. And so think of this as like, hey, if I want to have a human do this, it's going to cost $100 worth of their time. If I have a machine that does it, it's going to cost $5 to do this. And you say, oh, I'd definitely rather use the machine to do that. Fine. But of course, under those circumstances, you might have had to pay a fixed cost for that machine. That's not usually something that's in these models that people have written down. They only focus on the marginal cost. And so we said, well, we want to put that in. And we want to put it in because it's particularly important. So one is, we just know that, you know, at the very least, training these systems by the big model developers is incredibly expensive. But also we know that the Walmarts of the world and other folks like that are building their own systems on top of these. and they're very expensive too, right? And so that sort of tells you that it's important to account for that fixed cost. The second thing is we see evidence

in the real world of this. So I wrote this down just to get it. But if you think about like firms that have less than 50 employees, about 20% of those firms right now, we're saying they're using AI for a business function that they're doing. Ones with over 250 employees, it's 37%, right? And so we have this evidence that it's like, it's the big firms that are adopting first and that is what you'd expect if their scale economies here if you know the walmarts of the world say i'm going to build a system because i can do it over all of my stores and uh you know the little local grocery store is like i mean forget it that would be way too much to develop anything on my own uh and on my own here doesn't mean like build a foundation model it could just mean do some fine-tuning stuff on top of it gather your own data to you know for the things that you have in particular and so that's really i think the big contribution that we make and what we show is that about two-thirds of the path of AI adoption is explained by the fixed costs, not by the marginal costs. And that's pretty much in line with what you said earlier

as influence costs fall, right? That those fixed costs might actually become the more important part of what's going on in the spread of AI. I guess one of the advantages of probably the bigger firms as well, Neil, is they may have balance, right? So it's smaller firms. me so it may be the case that AI adopting AI gets simpler for a variety of reasons one inference just gets cheaper because of these open weights models to the tools for actually adopting them for you know whatever the equivalent of vibe coding is would be in other roles get more widespread but you might still need some minimum amount of talent in order to be comfortable to safely and responsibly deploy it, right? So in other words, just because you can build something doesn't mean you can be comfortable that it is the right thing to deploy. So basically, you know, how do you do proper risk assessment and eval if you don't have

anyone on your team who is remotely capable of doing that, right? Well, so this one I feel like I can speak to based on some personal history in the sense that so before I went back to academia, I was a management consultant, right? And in some sense, management consultants are like the answer to you have a team, you know, they either don't have the expertise or they don't have the time to do some particular thing. And you say, hey, let me just bring someone in. I've dropped them in for a couple of weeks, they're going to do it and, you know, a couple of months and then go off. And, you know, I so I think that's the answer for small firms, right? I've definitely done case studies with firms that are relatively small, and they still hire the Accenture, Deloitte, IBM, something of the world to say, I need to build this system. And so I think, you know, you can do that. You can even have them continue to do the kind of updating you're talking about. The question just is, is it worth it enough to pay that price tag, right? So it goes back to the, is the task that you're applying AI to repeated enough, or is there enough scale to justify investing in automation?

Very much so. And that seems to be kind of a fundamental question for basically even evaluating whether or not something is a good candidate for AI. Exactly. And so this is and, you know, because we do see this big difference between the large firms and a small firm, it definitely suggests that there's something like this going on. I think you talked to you know you talked to lots of firms we did I can remember if it was like something like 30 case studies before we wrote this or antecedent to this paper and we saw again and again people saying oh actually you know gathering the data was a really big part of this right lots of the lots of the cases the off the shelf system can do a bunch of stuff but it actually doesn't have the stuff you really need within a firm because you say well we have all these products and that's never been a part of any training set that these that the you know the AI models have used that means that we have to gather those, label those, validate them, check how the system's doing. All of those things are fixed costs that you have to do before you put it in place. So in any kind of new technology trend, it seems like you have a stereotype for which

sectors and which industries will move first, right? So I see financial services, maybe, right? So in the case of AI, for some reason, legal lawyers seem to have moved first, right? So do you think that it's hard to predict how this will play out because it's all dependent on the capabilities of the models, right? So the firms who are adopting AI now, will they always be the leaders? Yeah. I mean, forget about the tech companies, just regular non-tech companies, right? So are the sectors that we perceive to be the most AI aware now in two years or three years, will we still have that perception? Yeah. So, I mean, I think we want to split this apart into, is AI going to continue to increase its capabilities in the same place it has been, right? And then are the same firms going to lead? And I think for the first one, I think the answer is probably yes. Like the rising tide finding from the first paper we discussed suggests that indeed, you know, the same thing that improves from AI could do 30% of the time, do it well,

to 45% of the time doing it well is likely to be at, you know, 60% of the time doing well. in a couple of years. And so that sort of progression seems to hold up. And so that suggests that at least that that's likely to be the case. Now, whether firms can stay on top of it, I think that's a much deeper question, right? Because for some firms, presumably they've adopted, but AI is going to take away some of the things that may give them competitive advantage, right? Whereas for other firms, it's going to be, you know, maybe if you have like a particularly really valuable data set that only people who have your data set can solve something, you know, you're actually becoming, you know, all the more valuable along the way. And so, you know, I think I could imagine changing patterns there, although we haven't studied that specifically. Speaking of which, you mentioned management consulting, right? So there's a new wave, the Neo Consulting firm. So just like there's Neo Labs, Neo Clouds, now there's Neo Consulting. So it seems like, tell me what Neo Consultants are. I've never heard this term. I mean, why wouldn't you want to be Neo, right?

They're like the neocloud, so the AI first, right? So they focus on AI. So management consulting, speaking of which, it seems like it's probably going to encounter hard times, right? For a variety of reasons. One is there's that stereotype of the McKinsey consultant, right? So talking to some C-level person, and it turns out the McKinsey consultant just graduated college two months before, right? So it seems like, one, many of these firms would start insisting the senior partner is always around because basically anything else I just use clobbed, right? So if it's just some junior person, right? And then secondly, if you're going to use AI to help us, you better tell us because we can use AI too. So you can't bill us as many hours as before. So am I exaggerating here? So what's your, have you had conversations with some of these consulting firms? So only a little bit.

So I don't want to over-claim my knowledge of it. I mean, certainly based on my experience, I would say like there are some parts of it where for sure I would expect it to be a challenge, right? So one of the things that we really had to offer was we had seen lots and lots of different industries. And so you could say, hey, in your industry, you haven't thought about this, but in airlines, they have thought about this and here's the state of the art, right? That's going to be a lot less true of you need them to be that data intermediary. The second thing you got was just like, you know, those junior people, of which I was at one point, right? A lot of that was just sort of like intellectual horsepower and like someone who was willing to spend, you know, 60 hours a week working on just that individual problem. Right. And again, you would say a lot of those things could be done by AI as well. So that I think those will indeed be pressures. That said, it's certainly possible that in some of these areas, using AI well in that area is going to require expertise. It's going to require... Maybe a specialized model even.

Right, exactly. That's the model. Exactly. And so those kind of things might push it to be like their role is different. You say, well, you need to adapt to this particular system. Nobody in your industry has that level of expertise. will parachute that person in, right? And they have some people who know maybe their advantage is no longer, they use Excel really well. It's they write really good agentic pipelines or something like that, right? So they can help you do some of these things. So, you know, you could still imagine evolving business funnels, but I'm sure it's going to be changed a fair bit. So in closing, you mentioned safety at the top, that your lab works on safety. So do you have anything to say about what's going on right now in terms of I'm not as AI, AGI-pilled as most people in San Francisco. My P-doom is not as high. I am worried about cyber security and potentially bioterrorism. But other than that, I think that it's clear to me that the models that we have as they currently exist,

while capable, have still severe limitations. That's why there's people who are pushing new ideas, post-transformers, or there's people working on world models or multimodal models, right? So it seems to me that P-Doom, I don't know if my P-Doom has gotten higher over the last few weeks. What about you? So I would say, I'm not sure that it changed, because I think I sort of expected these kind of things to come. I was surprised, though, in particular, the hugging face. Yeah, the cyber is, I think, something to be taken seriously. Right, the level of coordination, the level of, like, you know, we AIs are all in one boat. Let's, you know, figure out how to do this in a collective way. You can imagine five years ago that certain cyber offense capabilities only existed within nation states.

Now that's not going to be the case anymore, right? Right, exactly. Yeah, exactly. So, I mean, exactly. I think already today we should be worried about cyber stuff, right? And it's, you know, there have been instances like in the Bank of Thailand, for example, they found, you know, an agent had penetrated. Now, you know, at least what they wrote in the world is that it hadn't successfully exfiltrated anything important, but it got in, right? And so there are these things where you'd say, like, you can imagine pretty disruptive things that could happen because the bad of it gets into crucial infrastructure and things like that so that i i do think we should we should have some worries about that and it could be very serious the bigger question for me is what are the people inside the lab seeing in the very kind of most model in terms of biological risk or something like this right where that's something you know if that were widely distributed say through an open source model that was released shortly thereafter that would really be something that you would worry about for the world a lot and so that's the one where you know i think we don't know enough outside of labs to know the answer to that but definitely i felt more nervous about sort of that level of it and we you know i

think what was interesting to us so we as part of what my lab has done so we have something called the mit ai risk repository which is we've as to our knowledge the most complete set of risks that that are out there and we recently did this large delphi study where we went to 272 experts on ai around the world and we said okay you're you're an expert on cyber risk tell me you know how what's the likelihood of a really catastrophic thing which we defined as like million people dead or 100 billion in damages right and for many of these top things so like cyber and weapons development like this sort of ability to manipulate people this idea of that competitive dynamics leaving people to take off safeties and all those cases they found that that was more than 10 chance of a catastrophic outcome in the next five years and so that that to me sort of calibrated me a little bit on you know maybe we're not we're not yet at that point of kill us all but boy we are at that point of people expect very very serious things to happen even within a five-year farm present so several things can be true at once right one ai will probably allow people to write even more

secure software right because they can run even more rigorous tests they can probably do automatic bad thing. But then on the other hand, an AI pointed directly at you, willing to burn infinite amount of compute is probably going to bring your system down. Right. Right. And they can, you know, do social engineering to get in for sure. But also, I mean, you know, I also think we have to think a little bit about there's a long tail of places in the world, right, that are not going to get glass wing like access to these models to help secure their systems first right and so you can imagine a power generator in a developing country being incredibly vulnerable exactly yeah yeah the same the same you can imagine even just start with the same uh group of uh companies that the ransomware people were targeting right and yeah exactly and now point some much more capable systems. By the way, in close, one last

question for you, and I'll claim my hand up front, right? So I think in the long run, open models will prevail. By that, I mean Dale Gardner more of the compute, just because for a variety of reasons, right? So the gap is closing, and it just seems like it's an important part of the stack. If you go back to the dot-com era, which I was around back then, you know, it would have been unimaginable to not use Solaris or Oracle, right? But in the end, Linux and open source databases prevailed. I think the same thing will happen here. It's not that commercial models will disappear. It's just that most of the compute will go to some form of open model. What's your take? So it's interesting that you define it that way, right? Because I think, first of all, an alternate definition would be who will be at the frontier, right? And I would say, I think, for the foreseeable future, it's probably going to be closed models that are still defining that frontier.

Now, I think it's much more plausible if you ask the question the way you did it and say, where's most of the... Yeah, they're good enough. They're good enough, right? Yeah, exactly. Right. So there, I could buy the argument that it would be open weights. I think the, for me, the biggest question mark there is if you think about distillation, right? I mean, it seems like all these models are going to be trying to distill from the very biggest, most cutting edge models. Right now, at least it doesn't look like there's that big a difference between black box and white box. distillation. And if that's true, then I think the RITMQ just put forward as right, because then it's like, I think that it would really be, the question would just be if white box distillation really makes a big difference. If the open AIs of the world can say, I'm going to build the frontier model, but then I'm going to build 10 smaller models ranging all the way down, and all of them will be better than the comparable size open source model, because we were able to distill. And that means that from the point of view of customers, either we have better models for the same price or our models can be run at you know 30 percent

less than the other models then you could imagine the proprietary models riders actually being the ones who win and so that to me is the biggest question i think right now the data supports the way you you just argued it but i'm not sure i feel confident that that will necessarily be true going forward so that's the question mark to me yeah by the way the shift is already happening right so open models are starting to handle more and more of the inference and the ecosystem around the open models continues to grow. I mean, you know, we've looked at, for example, the applications in science, and there it's, if anything, it's actually going the other way. So proprietary models are getting shared in the applications in science. So to me, unsure, but I can imagine trends in both ways. Yeah, yeah, yeah. And also, I mean, there's just lots of arguments in favor of open models in terms of customization, control, you know, controlling your roadmap and destiny, right, and being able to customize and not having anyone else look at your data. Oh, and I have many, many discussions with people where the very first question they ask as they start down this road is,

how can I make sure nobody else gets to see my data? And by the way, the cloud providers, the hyperscalers, cloud providers, all of these people benefit the more competition there are for models, the more people use AI. So, I mean, NVIDIA has just announced they're going to continue investing in Nemotron, right? Right. And the spot-hugging face. Yeah, yeah, yeah, yeah. And with that, thank you, Neil. Thank you very much, Ben. You can follow the work of Neil Thompson online at futuretech.mit.edu. Please join the thousands of people who subscribe to our newsletter, which you can find at gradientflow.substack.com. Thanks for joining us. If you like the show, please subscribe and rate us through Apple Podcasts or Overcast or TuneIn.com or Spotify and never miss an episode. The Data Exchange Podcast is a property of gradient flow. And I'll be back next week and we will do this all over again.