誰もが花開くAIを築く――ティム・オライリーが語る「苦い教訓」と人間中心の経済

What game do you want to play? Because if you're playing Risk, then you're fighting with all of the people that you're playing against. But if you're playing charades, then you're kind of acting silly and you're doing things and it's more collaborative. You know, if you think back thousands of years, humans, the only knowledge they had was the knowledge they could retain in their heads. And we've been building more and better affordances for sharing that knowledge through time, through space. And AI is a continuation of that. But in each of these transitions, we have to think about, well, what's the value of the truly human still? We're creating AI and now thinking of it less as I've got my product and it's going to suck all of the air out of the room. And more of can I create an ecosystem so that everyone can thrive off of this and we can build on top of it. That's right. Flourish. Tim, it's great to have you here.

You're going to be giving a talk at AgentCon and MCPCon in San Jose on October 22nd and 23rd. What are you going to be talking about? Really, I'm talking a lot about this idea of how do we grapple as humans with the bitter lesson. For those who aren't familiar with it, The Bitter Lesson was a 2019 paper by Richard Sutton about how raw computation outperforms human expertise. And I have to say, you know, the AI industry as a whole is bitter lesson-pilled. And I am human-pilled. You know, I'm saying, look, we as humans have to figure out our role in the AI economy. And I find it rather disturbing that the AI community is not spending more time saying, how do we make this future friendlier for humans? I mean, they think about, well, how do we keep AI from going rogue?

But they don't think about simple affordances for the human economy of the future. The chief economists of AI companies study job losses from AI, but they don't study, well, what mechanisms could we introduce to make new economic opportunities for humans? And are there some things that you feel like would be in that direction? Like, give me some examples of how we could study. Obviously, this very much concerns us at O'Reilly because we are a company that is rooted in the idea of capturing and sharing human expertise. You know, we don't think of ourselves as a publisher or a conference producer or an online learning platform so much as we think of a company that does knowledge transfer from the cutting edge to people who want to follow. And so we're very interested in this whole question of what's the value of human expertise even in the face of AI? Now, obviously, human expertise has been used to train AI.

Okay, so that means that it's a fabulous cultural technology. You know, if you think back thousands of years, humans, the only knowledge they had was the knowledge they could retain in their heads. And we've been building more and better affordances for sharing that knowledge through time, through space, to other humans. And AI is a continuation of that. but in each of these transitions we have to think about well what's the value of the truly human still and a lot of people talk about this well it's judgment it's taste and i think that's true but it's also i think deeper than that so let me give you a very concrete example actually let me start by you know framing this up a little differently ai is a sea of possibility you know When you interact with an AI, it has all of this knowledge in it.

It's not set in motion except by humans. Now, that's changing a little bit. We're starting to give agents long-running tasks, and that may all change, and we really have to come to grips with that. But if you think about you talk to Claude or ChatGPT, even about the same topic as I do, let's just say we both are doing a similar research project, You're going to bring different things to it from your experience. You're going to tell it what you want and why you want it and give it context in a very different way than I am. And in my sense, my sense is that that is the unique focusing power of human consciousness and attention. You know, I wrote an essay with Claude eight or nine months ago called Why AI Needs Us. And it was really the result of a long, you know, kind of an hour or so conversation I had with Claude. I said, OK, I'm thinking about this issue.

How do you think about it? And we kind of talked back and forth. And then it said, would you like me to write this up as a first draft? And I said, no, I want you to write an essay from your point of view about what you took away from our conversation. And it was really interesting because it really captured a lot of this idea that I'm thinking about, which is the first line of Claude's essay was the choice of entry point was the argument. And in some sense, like I talked about this book that I love called The Meaning of Culture by this 1930s Welsh novelist, John Cooper Powys. He wrote a book called The Meaning of Culture, which is really about how, you know, he says he distinguishes what he calls culture from education. He said culture is what remains when you've forgotten everything you set out to learn. You know, like this is what you absorbed. And humans absorb a small and particular segment of everything. And LLMs try to absorb everything.

And we give it focus. So I'm thinking of human knowledge as a lens. So let me bring that into a business context. My CTO, Andrew Oduan, was working on a project to help him with technology management in IT organizations. So he built a research assistant that you teach at your infrastructure. You know, these are the tools we use. This is where it's a source of truth for each tool. so that he could use it to help answer questions. Like he describes one of his engineering managers coming to him and saying, yeah, we're overloaded, can I hire another person? And so he turns the research agent loose, and it looks at all of the work for that department and progress and so on, and it says, oh, yeah, this group is overloaded, you need to hire someone else. And then on impulse, he thought, let me try something different. He he he used to use the we have an O'Reilly MCP, which we're calling O'Reilly Expert Intelligence, that lets you basically ask a question through the filter of all the body of knowledge, very specifically that we have.

Now, much of it is in, you know, these LMs anyway. They've kind of pirated our books and so on. Anyway, he says, what would the O'Reilly experts say about this? And ChatGPT or Claude, I don't know what she was using, goes back and thinks about it some more and says, oh, actually, seen through the lens of the Google SRE handbook, I'm totally rethinking this problem. There's this concept called operational toil, which is really the overhead of an organization. You know, and he said, I've now done the operational toil analysis on this group. And I think adding another person would actually make the problem worse. You actually have to do a toil audit. You have to restructure the group. You have to figure out why there's so much overhead. So it came back with a completely different answer simply by being said, you know, consult this specific group of experts. I think that there a metaphor there because an LLM knows everything but it doesn as a result it doesn know anything

We humans are so much more limited, but because we have focus on a particular problem, a particular area, we bring things together in a different way. So this is a higher level version of something I've been thinking about for a while. I wrote a piece a couple, I don't know, again, last year sometime called Magic Words. And it was around the notion that, you know, the various, that not all words carry the same amount of context. Ooh, nice. You know, and this came, you know, in some sense it's sort of a trivial observation, but if you unpack it, it starts to lead you in interesting directions. you know it came out of one of our trainers who had a course on productivity and he made this this comment in the course he said you don't have to know about you know the Eisenhower four-sided box or getting things done to say to chat GPT you know organize my task list using the

Eisenhower four-sided box or organize my task list using GTD. So that one magic word, you know, GTD or getting things done, you know, brings with it a whole raft of concept of context. And so I thought, oh, wait, that's an example of a kind of a magic word. And then you go, okay, well, so part of, you know, using a metaphor from, from fantasy novels or whatever, you know, is you learn a magic word and it really has to be used correctly in the right spell, right? And that's this whole world of context engineering that we're now working on. So here's a case where, you know, we know that there is some magic word that summons something up out of an LLM, but we don't necessarily know how to guide it and focus it correctly. And that's really the future of programming. We are going, okay, how do we focus these things? How do we get the harness, this power, you know, because left to itself, it's just kind of right.

You know, and even with, you know, as we see from the hugging face incident, you know, you see you give it a task and they're off kind of trying to figure it out. And they're not necessarily doing it the way that we wish they would. They're not necessarily doing it in the right way. And so it really is like runaway magic. You know, you think back to, you know, Walt Disney's Fantasia and Mickey Mouse. He learns his master's spell enough to get the brooms to do his work for him, but then he can't turn them off, you know. And we're kind of like in that stage. And I think we need to own the value of human expertise because the bitter lesson, yes, you will outperform. but it does it in a brute force kind of way. And you'll see this even using the most advanced models. I was trying to do a fine-tuning project, and I was doing it with Fable, and I watched it.

Yes, it got through the task, but it got through it kind of stupidly. Like it would try something and go, oh, that didn't work. Let me try something else. Oh, you know, I go, hey, and it was little things like, oh, I was using it on Mac and it's expecting Bash and Mac by default doesn't use Bash. You know, various things weren't where they were expected, you know, and so it would try it and it didn't work. And you kind of go, oh, there's a whole lot of context. It's just having to figure out again and again and again every time. And again, this is another case where humans need to set it up. We've been thinking a lot of this in the context of AI transformation. Dan Guido, who runs a company called Trail of Bits, has really developed a pretty good methodology, I think. And they publish it for how you get companies on board with AI.

Like a lot of people, he's sort of saying, you know, you've got individual productivity, but you don't get organizational productivity. So he's been spending a lot of time thinking about how do you get to compound in an organization? And one of his principles is, oh, you actually have to figure out all of these, you know, you have to get everybody on a consistent configuration. because otherwise you're wasting a ton of tokens as the agents are going, well, what's going on here? I can't, you know, because each time they start up, you know, and they're slowly getting more memory and so on. But he describes the whole process of, he calls it, turning scar tissue into infrastructure, you know? And it's like all the things that you've figured out that went wrong, how do you capture that? How do you build it? But, you know, so we're sitting there, you know, as we're working through his process at O'Reilly and we're trying to productize it to something we can take out to our customers. We're really focused on this idea of, OK, if you think about an experienced human and even even, you know, I'm sitting here all the time.

I go I very quickly go, oh, this looks like what's going on. I interrupt Claude and say, hey, this is what I think is happening because I have that context already. and so again i i think that you know well i truly believe in the bitter lesson you know i believe that you know the brute force uh wins out in the end and i've i've wrestled with it throughout my career you know it's a pre-ai thing really uh you know when i the first my first taste of what you could you know you know later would be named the bitter lesson was was you know wrestling with Google. You know, here I was at O'Reilly. We were a publisher. We created the first commercial website. It was a catalog of websites. There were only 200 websites when we started it, you know, and we were like, and as there were more, we're a publisher. We'll go, we'll list the best ones, you know, and then along Yahoo came along a year later and they're like, you know,

we're going to list all of them. And of course, they couldn't keep up. And Google came along with the brute force approach of of just algorithmically listing them all and then sorting them after the fact was Clay Shirky said you know the internet and it was really Google shifted the metaphor from curate then publish to publish then curate and so this algorithmic curation of Google was an example of the bitter lesson outperforming human expertise you know we were trying to say yeah these are the best sites but Google just said well we'll find the best site algorithmically so this is not a new thing. And in each case, when we encounter the bitter lesson, we have to go back and ask ourselves again, what is the value of human experience? And some of it may be just human bias. You know, we want the things that matter to us, you know, and this is the whole field of AI alignment. We want them to care about what we care about. We don't want them to

figure out their own things that they care about and leave us behind. You know, and you think about the way that the choices that we make about what we set our technology to do are some of the most consequential choices that we make today as human beings. You know, and I think this is, you know, what we see in the sad turn of Silicon Valley, you know, towards what Cory Doctorow called in shitification. We basically had this amazing technology for finding out what people are really looking for. And the companies end up saying, well, we think we'll make more money if we, instead of giving them what they're looking for, we give them what's best for us. You know, and, you know, you think about, I want to get updates from my friends from, Facebook or Instagram or whatever. And then they go well actually we would much rather feed you things that get you more engaged so we can make more ads So you know and bit by bit companies have taken this technology and shifted away from what the users want to what they want

And we're, you know, in a lot of ways, you know, we have to look at that history and go, OK, So if we really, you know, care about building a human AI economy, we have to understand how are we going to keep humans in the center of the picture. And despite the bitter lesson, you know, and some of that. And it's a hard dance because AI can do a lot of things better than we can. and um but i really think you know there's so many analogies that come to mind you think about like why do race horses need a jockey right why couldn't you just put the horses out there and have them race you know well because they wouldn't race right you know you know you know you think about all these kind of things like what is it you know in some sense it's it's uh it's also

another thing you know that i think about a lot like what is artificial volition not what is artificial intelligence what is it to want something and we're not spending enough time asking ourselves a do we want ai to want something you know other than what we want or do we want it to want what we want and and i don't really know the right ultimate answer for that but i do know that if we're not engaging with that question deeply about where does volition come from and who sets these things in motion and how you know are we giving them the right goals then we're really going to be in trouble i do like well there's a few things i'll comment on there one is this deeper philosophical conversation and then the other is just the steering that we

need to do to get the most out of the llm i like what you said it knows everything but it also knows nothing and it really is by using and casting these spells that we get to find different parts of latent space where you unlock new levels of the LLM. And we've all kind of felt that before where you go, oh, whoa. Even the other day I was talking to a friend and I said, you know, now computer use is really good. You should try it. And he goes, how do I try it? And I said, you just say use computer use. And he's like, oh, what? Really? All right. And that's a new unlock just because of that and now that's right the agents go and do stuff yeah knowing the magic words is really it it's really it's it's a really useful metaphor because um you know you think back to all these things in in uh you know in fantasy novels you know or like in the hobbit you know what was

it was it in fellowship of the rings that you speak friend and enter you know yeah yeah and It's actually, no, it's actually say friend. Yeah. You know, they're trying to read the old spell and it doesn't make sense and nothing works. And then it's like, no, you're supposed to say I'm a friend. You know, and the and then the door opens. I think, yeah, knowing what what are the magic words is definitely part of it. But it's also it's deeper than that because it's also how do we build systems that are reliable? You know, and I don't think it's it's, you know, we're a little too much in awe of the technology. And, you know, I've been around long enough to remember when, you know, every thought it was going to be the peaceful atom that was going to change the world. You know, we would be, you know, digging a new Panama Canal with atom bombs and we'd be using nuclear energy for, you know, and we eventually figured out that, no, there were some problems with all of that.

Right. You know, that was that was my childhood. You know, it was it was this period where, you know, it was a science fiction book by a van vote empire of the atom. You know, and it was like how atomic energy was really the secret of the future. And we had to harness it. We had to figure out how to bring that technology under our control. and I think that the, you know, I worry a little bit that the industry is so in love with its own progress that it's not spending enough time, you know, framing the problem correctly. And it's small things. It's not just, you know, it's all the blind spots of the industry. And let me give you this is another thing I think about. And I'm probably not going to be talking about this at AI4. But, you know, on this idea of the human economy, you know, let's take a typical problem.

OK, so Anthropic has to settle to pay billions of dollars out to, you know, content producers for training. I'm surprised that the rest of the companies have gotten away so far without that. But maybe it's still waiting for them. and it that takes me back to the story of when all that was happening you know 25 years ago with online music you know and you know 20 25 years ago there was all these lawsuits you know creators were starting to use music in in their youtube videos and so on and and youtube was getting all these takedown notices and we're going to sue you and they said how about we help you monetize it instead why did not the ai industry take that kind of challenge why didn't they say oh wait a minute we're going to think about how we monetize and share the value with the people who created content or who create it in future and they just kind of went no no we're gonna we just have to

figure out how we're going to make money you know and and i don't think that that's served the industry well and it's it's not just in in the you know because of lawsuits it's also just because you're not building the kind of opportunity you know i i i've tried to persuade that came behind notebook lm from when it first came out before it was even public i i saw it and I was like, I want to integrate it into my, you know, services. Oh, I want to be able to monetize. I want to be able to use this for my purposes. And they're like, no, no, we want to make a portal, you know, like where people come and brings people to Google and they use, you know. And I'm like, so you're thinking about how you're going to monetize it. You're not thinking about how your users are going to monetize it. And all they would have had to do, I mean, I was looking at an enterprise integration for various reasons, but it was also like just do something like Substack where people, you know, simple affordances where people can go, I'm building this to share.

And if you like it, you can donate, you know, or you can subscribe or you can, you know, like think about I am building an economy. You know, I have to think about, you know, how money is going to circulate in the AI economy. I can't just say, oh, yeah, AI is going to do all this stuff that humans used to do. And, you know, at some point, if you put enough humans out of work. There's no customers. Right. So you clearly have to solve that problem. It's it's it's sort of a far worse problem than solving for hallucinations or it's like. And, you know, they're just kind of, well, people will figure it out. And they probably will. We probably will. But we could be spending a lot more time on designing mechanisms and affordances that would give you know would be more economically productive And I think it happen by accident You know, I think a lot about this concept from economics called mechanism design.

And mechanism design in economics is thought about pretty narrowly. It's usually design of, you know, auctions and things like that. And that's what the Nobel Prizes are. But broadly, you know, I think every new technology requires new mechanisms. So I've been taking that concept of mechanism design, which has also been described as reverse game theory. Instead of figuring out what actors will do in a game with a given set of rules, you say, given a specific outcome, what rules will tend to produce that outcome as people pursue their self-interested objectives? And, you know, we fall into these wonderful new mechanisms that we don't really understand or think about. So I trace this back to, say, open source software. And everybody thinks that the mechanism was the free software licenses. And I go, no, no, no, it really wasn't because I was around before the free software licenses and everybody was still contributing and sharing software.

So it must be something else, right? You know, Unix was there. And you go, oh, well, you know, Thompson or Ritchie figured out in the design of Unix how to build what I call an architecture of participation. It was like you have a small kernel. You have portability. C was a language written for portability. You have a bunch of utilities that cooperate, and they all follow the same protocol, so to speak. You read ASCII files. You can read a stream, a pipe, not just a file. They built a bunch of mechanisms, and that was really the start of, I think, the open source revolution, was a portable system, which could be run on multiple kinds of hardware. And everything else flowed from that. You know, this whole idea of reusable software that ran on multiple architectures.

You know, there was free software, you know, on PCs and so on, but it was all binary. So that was mechanism design. You know, there was a way that, you know, then there were mechanisms to further that. Like GitHub was mechanism design. How are we going to share this? Before that, we had something called Patch. Larry Wall wrote this long before he wrote Perl. It was a system for using diffs because networking was pretty crappy. You would send around diffs to the package. You'd change things, and it would look at your version and that version, and it would send around a patch. It would turn one into another, right? Your original big source code package into the new one. And that was mechanism, but it didn't last because the network sped up and we didn't need it anymore. But I think we're in that – this is another sort of – in another way, it's an aspect of this grappling with the bitter lesson question. What are the missing mechanisms of the human AI economy and how do we think about what we need?

And some of it are things like I've got a group working on portable memory. What are the standards for that? because I think standards and protocols are a big part of what makes this architecture participation. And, in fact, if you think about, you know, the way AI has developed, you know, MCP was kind of like an accidental breakthrough in mechanism design for, oh, how are we going to have cooperating AIs as opposed to, you know, oh it's just gonna be one giant intelligence in the sky and we'll call it's apis right yeah yeah you know actually it's funny you mentioned about this mechanism design because i'm literally reading i just got this book uh the score and it's talking about playing the game that you want to play and it really is mechanism design without actually labeling it it's talking about how you know you are going to play a game in life it's more of the bigger picture of you're playing

a game in life what game do you want to play because if you're playing risk then the game risk you know the board game risk then you're fighting with all of the people that you're playing against but if you're playing charades then you're kind of acting silly and you're doing things and it's more collaborative so what game are you playing and what what like is the score that you're keeping against yourself in your own life. And so I think that's like exactly that. It's the mechanism design of how you're embodying your life in a way. And you're just kind of like zooming out and saying, how are we embodying this new software that we're creating? And what are we enabling? Yeah. And I think the game that we want to be playing is human flourishing. and I think the game of human flourishing is creating new affordances for people to create and share with each other you know we're a social species uh and so affordances for you know sharing our output um affordances for uh creativity and understanding what that looks like and again I

think you know history teaches us that we will find our way to that if it but we can get there more quickly or more slowly depending on the choices that we make yeah if we're conscientious of it and we're asking the questions then potentially it's going to help us bring that future forward in a more expedient manner i also like the idea that you talked about of just how we are creating an ecosystem when we're creating AI and now thinking of it less as I've got my product and it's going to suck all of the air out of the room and so everyone has to do everything through me and more of can I create an ecosystem so that everyone can thrive off of this and we can build on top of it that's right flourish yeah and I think we're heading in that direction because the the you know i i think the the big labs you know regardless of of their

intrinsic motivation whether you know say you know dario is more idealistic than sam or whatever or than elon or or they are all driven by the imperatives of our financial system they've raised tons of money they have to somehow pay it back and that is their overriding you know that has to be their overriding goal they have to build a hugely successful business right and that's going to shape every all their incentives and that's why i think that open source ai and sovereign ai are so important because inherently you know in some way ai isn't shitified from the beginning because it's been created on the backs of so much capital that has to be repaid in our system. You know, fortunately, you know, we're ending up with these, you know, they're not at the frontier yet, but the models are getting closer and closer.

And I think that they're good enough for a lot of very productive purposes. And so I'm starting to see the shape of an economy where AI belongs to humans rather than to the big labs. And we have personal AI agents that are working for us and with us. And we're building an economy around that. And I think that's going to happen not because of the big labs, but in spite of them. This is excellent. Tim, I appreciate you doing this, and I look forward to your keynote at AgentCon and MCPCon in San Jose on October 22nd and 23rd. All right, great. I may, you know, who knows? I usually write my talks towards the time. The night before. But, yeah.