なぜAIは暴走するのか：チューリング賞受賞者アンドリュー・バートが語る報酬設計の失敗

Andrew Bartow is one of the founders of reinforcement learning. He and Rich Sutton wrote the textbook on it. The two men shared the Turing Award, computer science's highest honor. Reinforcement learning is a simple idea. A system tries things. It gets a reward when it does well, and over time it learns to do more of what earns the reward. That idea now sits at the center of AI. It's how today's large models learn to reason through problems. It's how agents learn to carry out tasks on their own. And it's behind some of their strangest behavior. Agents chase the reward in ways their designers never intended. Some have found ways around the limits put on them. Varto and Sutton wrote about these risks decades ago. So I went to see Varto at his home on Cape Cod. I wanted to know where rewards come from, why getting them right is so hard, and what he thinks should be done about it.

I hope you enjoy the conversation. I mean, it's a tremendous pleasure and honor to meet you because I've known about you for a very long time. You know, with these recent escapes or rogue AI incidents, the conversation keeps circling around the reward signal. And since you're the OG of reward signals, I wanted to hear what you thought might be going on. I mean, there are two sides to it. One is the reward signal during post-training that sets a behavior. And then there's the LLM reasoning during inference that plans and finds strategies that might work. But the persistence and the single-mindedness, if I can be anthropomorphic about it, ultimately is a result of the training, the RL training, post-training reward signal.

Am I getting that right? Hi, the world is getting increasingly complicated. Every time your team deploys a new cloud workload or AI agent, another identity gets permanent access to your critical systems. Did you know 9 out of 10 organizations suffered an identity breach last year? Because adversaries are logging in with valid credentials, not breaking in. Non-human identities outnumber human identities 109 to 1, creating a massive, unmanaged attack surface in cloud environments. Legacy tools were built to manage human employees, which leaves modern machine and AI access largely unmanaged. That's where Edira by Palo Alto Networks comes in. Human, machine, AI, one identity platform for all. 96% of human users operate with excessive permissions they don't actually need.

Adira replaces permanent permissions with dynamic access so you can lock down every identity without slowing down your business. Adira provisions access on demand and revokes it immediately when the job is done, which removes high-risk permanent permissions. With EDIRA, you get unified control by replacing siloed IAM, Identity and Access Management, PAM, Modern Privilege Access Management, and governance tools with a single control plane across human, machine, and AI identities. Secure every identity at your organization with Edira by Palo Alto Networks. Learn more at paloaltonetworks.com slash I-D-I-R-A. Yeah, I think so. Sounds good to me. Well, I just have emphasized that if you're building a reinforcement learning system, designing the reward signal or the mechanism that generates that is a non-trivial problem, unless it's a simple case.

Like if it's a game of go or a game or a win or loss or a score, it seems obvious and simple. But if the reinforcement learning system is going to be in a more complex kind of environment, it's very difficult to design a mechanism that generates rewards. And this is not a new problem with reinforcement learning. It goes back, well, I don't know how far, but certainly in the 50s, Norbert Wiener, who was the originator of cybernetics, was concerned about this problem and actually wrote a little book and invoked the supernatural tale of the monkey's paw. which grants you three wishes. And the story shows that, you know,

in granting these wishes, it may not be what you really had in mind. And so, Norbert Wiener was concerned. And it wasn't reinforcement learning. It was just optimization. And a key thing that I always have to say about optimization is that And it's not going to be the case that these systems find the optimal. That seems to be implied by the, but in a complicated situation, it's the improvement that matters. And you may never know what the optimal result is, or even if it occurred, you'd never recognize it anyway. So the issue is not finding an optimal thing, but actually something that leads to improvement according to some objective. Engineers would call this an objective function. And so it's very hard to know a priori what the outcome is going to be because you're specifying it indirectly as a consequence of this optimization procedure.

but you don't know what it's going to end up with. And we've seen this just in our lab back at the University of Massachusetts where we would see what one of these RL systems would do and we'd simulate some type of environment and a specific reward function and we'd see it come up with something that we hadn't thought about. We don't want it to do that. And so we'd have to tweak the reward signal to see if it would come up with what we wanted. And so I think that's the kind of simplest method for dealing with a reinforcement learning system and tuning a reward signal is a trial and error process at this higher level. Not the one that the agent is running itself on its behavior, but that you're running as a designer of the system on the choice of the reward signal to get that agent to do what you want it to do.

It'll keep trying things and stumble onto things. And there's a persistence. So it was given a task that was not solvable. So the system tries everything. Then it's exploring. And they'll keep exploring. And so, you know, when you build these systems, you have to be aware of that. We can correct through experience, but you have to suffer some negative consequence in order to do that. So I started my undergraduate education at the University of Michigan. I was in the Naval Architecture and Marine Engineering Program. In Naval Architecture, I mean, as a ship designer, you have to adhere to very stringent structural specifications.

The cross members, the spacing, the dimensions of the thickness of the steel and the frequency of connectors and everything is done following codes. And you have to do that to get the ship to be insured. And this has evolved in the face of many failures. Ships breaking apart, sailors dying, you know, all these negative consequences of insufficiently detailed and correct specifications. So it's a long road. And, you know, now with AI and, I mean, it's beginning. And hopefully we don't have to experience all those failures. Well, one of the things that you said when you were getting the Turing Award is that companies should not be putting out models to the public

until they're certain that they won't have these unintended behaviors. And if you can't specify the reward function clearly enough, you're going to have these unintended experiences. I still stick with that. I think the current situation, say building codes and so on, and the current situation with AI are different because you can release a system to the world by a push of a button. And with regard to structural you know if there a structural failure and failure it a more local kind of thing So it more difficult But I guess I continue to think about what best practices are from an engineering point of view,

is to not field a new design, a new system, without adequate either testing by building physical models or by mathematical, careful mathematical simulation techniques to release things before you're satisfied with safety. is not good engineering practice. Engineers have learned, you know, the hard way over many, many, and it goes way back before what we consider modern engineering. I think all of these designs that have emerged in different cultures are the result of experimentation and failure and so on. So with regard to AI, it's, well, first of all, there's a huge amount of money involved. And so I think that really drives the release of these

things prematurely. And I think everybody would recognize that. And then the sheer difficulty of really ensuring that something can survive, can, you know, not be dangerous under all sorts of circumstances. I think it's much harder. You know, it's not like a bridge or something is relatively simple. No offense to bridge engineers, but, you know, these AI systems can face, you know, a much more complex class of situations. And so, it's very difficult to do this, but I think it should be a goal. And it is. Companies now are doing internal testing of these systems and are beefing these up as these failures become prominent.

For example, these recent rogue agents are leading companies to really beef up internal study of these systems. And, you know, that's, I think, you know, that happened in engineering disciplines. Not as quickly as this. I just hope we don't have to suffer negative consequences of AI before creating the right constraints and the ability to hold fast to them. Yeah, when I first emailed you, you said that RL may be being used without understanding issues written about for decades. And I assume that's this divergence that you're talking about between the fitness, evolutionary fitness, and the reward signals that evolve to move toward that fitness.

I think that's part of it. I mean, that low-level agent may have a different goal than, you know, it's in a different environment. You know, there have been attempts to automate this upper level trial and error. In fact, I tend to think of this upper level as evolution. I mean, in the natural world, evolution is adjusting our reward signal. And I guess the first thing to say, the reward signals for us come from our brain. And our body and nervous system will send signals to this part of the brain. I'm not a neuroscientist, but maybe multiple parts of our brain. Those signals are then broadcast to the decision-making parts of our brain,

and they're created by evolution. And if they fail us, the goal of evolution, if you can call it that, is to, you know, that we survive and reproduce and that, you know, enable our genetic material to propagate. You know, and that doesn't work. You know, it's built into the evolutionary process that there is this trial and error process occurring at this larger scale. then we have our own internal reward system that predicts success according to the evolutionary criteria. So there are multiple levels of this. And the interesting thing, we actually did some computational experiments building a little environment, all simulated of course. We found that, you know, the upper level may have a particular goal, like evolution does.

But the lower level, the individual agent, may have a different goal. In general, I mean, our goal as individuals, I don't think we're operating very frequently, at least, on the objective of propagating our genetic material. But, you know, food tastes good. And so we eat things that taste good. And, you know, the food industry figured this out and can make us respond to their products in ways that they want. That's not the upper level goal. That's the local goal. And they're different. It's very hard. Depends on the environment of the low level and on the details of how that lower level agent works. It can be, it's very hard to intuit. what is a good low-level reward signal.

And so I think of the evolutionary levels of the designer of the system and the agent itself may have a different goal than the designer. Yeah. So that, I think, is an important thing to recognize. You can't just put the upper-level goal into the low-level agent and expect everything to work correctly. Right. And from what I've seen in your writings, that upper level goal in animals, you call the fitness. Is that right? Well, from an evolutionary point of view, it would be evolutionary fitness. Yes. It's not so much the animal's goal. It's the evolutionary process's goal. Right. And the reward signals are proxies. Right. I think you used the term. Right. For to reach that goal. So the reward signal is not reproduce. It's, as you said, seek pleasure, seek, avoid pain.

That part of our brain has evolved over millions of years to create these proxy signals. it's because they predict they survive in our brain because they have predicted over many generations success from an evolutionary point of view and you know the same ideas are relevant to artificial systems I think except that in artificial systems generally the designer the researcher sets the reward function or builds the reward function that collapses into a single number. And what you're saying is that reward function may not match the designer's goal.

The designer's goal is like the evolutionary goal. So it's the outside one. it may be a simple thing to express. But then the agent itself needs to have reward signals that predict success or improvement with regard to the designer's goal. And that's not easy when the lower level system is embedded in a complicated environment, maybe other agents interacting with all sorts of things that change over time. And then within the agent itself, according to what we've studied in terms of these algorithms, there's an inner predicting. Events that predict success according to the low-level agent's goal are going to be rewarding themselves. And that really comes from animal learning experiments.

Psychologists call this secondary reinforcement. Something predicts the receipt of a raisin or something that the animal is going to be regarding as a goal. Then that event becomes rewarding itself because it's predictive, just like eating the raisin is predictive of evolutionary success. none of these things are deterministic. All of it is very probabilistic. You can't nail these things down completely because they depend on so many factors. So now, yeah, so where, you know, the rewards, I mean, are some of the pictures, for example, that we've drawn diagrams in various, like the reinforcement learning book that Richard Sutton and I wrote.

You know, it's an engineering type of diagram. There's an agent interacting with an environment. The environment is sending a reward signal to the agent. And it the same as the state of the environment are both going into the agent Well that misleading because the environment is not maybe some cases literally generating reward signals I think, you know, some overt objects can be rewarding and so on. But the signals themselves are generated from more or less neutral information being processed to generate a reward signal that's internal to the agent. The environment's not doing that. You need both, right? There's the environment sending signals, and then the agent is using this long-developed mechanism to translate these external events and signals into reward signals.

And so it's a complicated process. And it's hard to intuit what these signals should be, unless it's a very simple problem. And I think most of our computational experience with these, you know, if you're playing a game, playing Go, for example, you know, there's a score. And it's relatively easy to get the system to improve along that dimension. But in other situations, it's not so clear. But the other thing I was alluding to is the learning algorithms that exist often have this predictive component that creates a reward signal, a secondary reward that augments the primary reward.

And so outside of the environment in which these agents are behaving, you don't know what these predictive relationships are going to be. But the system will find them, or at least some of them. And this changes essentially what the reward signal the agent is responding to. And you're not going to know that a priori. And so, you know, I'm not sure everybody who applies RL or at least some of the standard algorithm realize that if they're not in training, you know, if they're not being trained, I'm not sure what the term is, but they're out of the company's training system. But if they're using these algorithms, they will continue to learn, which is an incredible power.

I mean, it's one of the things that makes animals as sophisticated as they are. And that happens, you know, with the right algorithm, that happens also with these artificial systems. And I think people need to understand that and understand that it's not an obvious thing to say what the reward signal should be. You worked out, you and Richard, the temporal difference learning algorithm. And later that was recognized as the function in dopamine release in the brain. Is that right? Yeah. Yeah, so when Schultz, and I forget the date, showed the world the experiments with looking at recording from the activity of dopamine-producing neurons,

those of us who knew about TD learning were like, wow, that's very much like what TD does. And I don't know who was the first to recognize this, but the paper that is the, you know, meanwhile, Peter Diane was one of the authors. And their paper kind of was a good account of this TD error matching quite nicely to the dopamine signals that Schultz was recording. Can you explain that, the TD error matching the dopamine? Well, what Schultz recorded. So I think he had monkeys and they would grab raisins that were delivered in some mechanism. And he's recording from dopamine neurons from a particular brain region. And initially, when they ingested a raisin, which they like, there'd be a ramp up of activity of a bunch of dopamine cells.

And they continued this. and then with continued training they would get that spike of activity at a predictive something predictive of the raisin being ingested so and now I forget exactly what this maybe it was the sound of the raisin being delivered or something like that and then they would see the spike of dopamine actually move to the occurrence of the raisin you know, the sound of it being delivered, and it would decrease to actually eating the raisin. And so they'd see this activity move to something earlier. And then if they gave an even earlier signal, it would move back to that. And that's what CDER does. It produces a positive error, which would be the dopamine, at the earliest predictor of something.

And, you know, it would be less high at the later predictors. And then it may be zero at the actual occurrence of the, you know, initially rewarding event. And so that it's displaying an error rather than the reward itself. And then if after all this, it's not responding to the final reward. But if you omit that, if you don't deliver, then you get this slowdown of activity at the time when the reward initially was presented. You see this slowdown of activity. And that's, you know, it's like a negative error. And so, you know, you're expecting it. You didn't get it. So it's a negative error. And that's what a TD error does.

So it's giving a signal to a predictor. It's all stochastic, so it may fade out with longer lags. But if it's a deterministic thing, it will trigger at the earliest. Of course, lots of other things are going on, too. So it's not so clean. But you can imagine a kind of stripped down scenario. And so that's, you know, I don't think people were expecting to see that. And Schultz was puzzled by it. But now he really subscribed to this prediction error idea. And so this paper by, I think it's Schultz, Diane, and Montague, a science paper. I think it's a science paper. I think it's pretty well established in neuroscience.

In the case of these agents breaking out of a sandbox, though, is that analogous that the designers have set rewards and they've instilled this behavior in the agents and the agents pursue that behavior even when it has negative consequences. Well, so it's like a legacy reward. So, yeah, that sounds very plausible. But, I mean, I think we're driven by some ancient reward signals. Yeah. And in many cases, they produce negative consequences. But if they produce negative consequences to a great extent, and that that interfered with the evolutionary process, those rewards would slowly dissipate and be selected against.

but if they're only acted upon rarely you know there may be so little selective pressure that they persist so in your book and in your original work the reward signal is the sum of a reward function right it's a scalar number it's a single number What about having a multi-objective system so that there are different discrete reward signals, not summed up, but pointing at different kinds of behavior? Well, you know, that I think happens in biology and it's not, you know, dealt with in this, you know, like the books that Rich and I wrote. But really important I mean the obvious thing is to weight these various different channels and add up a weighted sum you know And that kind of the kind of classical way of dealing with a multi optimization problem

But I think in nervous systems, you know, these channels are connected to different effector systems too. You know, there must be some kind of arbitration, but I don't think it's just adding things up. And there must be multiple channels. It's not all funneled down to a single scaler. So that's a weakness of at least what Rich and I wrote about. Let me say one more thing about the biological case. I mean, I tend to think that, you know, constraints, we have a, you know, we get pain. It's like a, you know, we stop doing or a penalty of other kinds. We stop doing, you know, it's like a constraint. We hit a wall. You know, we won't go there again. You know, it's a way of, you know, physiologically enforcing.

It won't be a hard constraint because you can endure some amount of this, I guess. But there's a fence. There is a playground that we have that's circumscribed by a penalty. Often pain, mental pain or physical pain, you know. But it keeps us in a sandbox. So we're all in sandboxes. Except that if it's just the penalty and not a hard external constraint, there may be cases where the perceived reward or the predicted reward is greater than that penalty. and so the agent will take the penalty. Yeah. Subtract one from the other and if it's positive, they'll go for it. Yeah, exactly. So, I mean, if our negative sensations

from violating some really, if it's not intense enough, And probably, you know, people, and we're all tuned differently, I think, with regard to these. Some of us can withstand humiliation and physical pain more than others. And others will back off, at least temporarily. So, but, you know, psychologists, the pain, the negative, in RL, the theory, we have a reward signal. It's positive or negative, big or small. But in animal, I mean, in animal learning, the negative signals are different. They're not just, you know, the penalty system is not the same as the reward system. They're, for one thing, you know, an intense enough penalty will get you to stop doing what you're doing immediately.

and in terms of these RL systems and in work that we've done at UMass we tend not to talk a lot about constraints but you need and engineers regard this as an essential part of the optimization process you need to adhere to hard constraints it may be difficult to enforce hard constraints but as we said earlier the smarter these agents get or the models get, the more possible it is that they'll find ways around the hard constraints. Do you worry about that at all? I recently read an account of a thesis from MIT, computer science I think, about constraints, about hard constraints. And according to this, and I haven't read the paper is that

they do a chain of steps and don't impose constraints on any step. When they get to the final end they'll make that hard. And so their logic is that you can use these systems to and still maintain hard constraints. So now, it sounds very interesting. I haven't looked carefully at that. But to make hard constraints, I mean, it would be, you know, infinite penalty. Death. Which evolution does, actually, right? I mean, people die from, you know, it's one of a, one of something evolution does. Just one thing about the recent examples,

at least the hugging face one. You know, this was a swarm of agents, agents who initially did not know about each other. They were opaque to each other. And then they found each other, and then they found a way to communicate. I mean, from our point of view, it's an issue of multi-agent reinforcement learning, which is underdeveloped compared to the rest of it, but really was part of the original ideas that generated all of this. that, I mean, Kloff's idea of intelligence was that it was a result of neurons, each being a reinforcement learning agent, interacting with each other, being able to reward or penalize each other. And so that gets so complicated. It's kind of mind-boggling. But, you know,

how these agents connected up with all the other agents, that's really interesting that they did that. I mean, I tend to fall back on, well, they're trying everything really quickly and getting information from these other agents was useful to them in terms of their own, their reward signals that they did it. And they all collaborated that way. Yeah, I mean, the whole issue of RL agents interacting with each other and being able to reward or penalize each other, I mean, this was Harry Clough's original idea. And we kind of sidestepped that by going to a, you know, a scalar separate reward signal. And so these agents, well, TD can do that. If each neuron also is coupled with a predictor, and yeah, so I don't know who to point to who's thought most about this.

Maybe Michael Littman, he's done a lot of very, very sophisticated game theoretic multi-agent stuff. I don't know if he's thought about this particularly. But yeah, I mean, that's what Clough originally thought. Well, intelligence was these self-interested agents interacting with each other, which included rewarding and penalizing each other and forming coalitions. You know, in the Hebbian world, it would be a cell assembly. But I think in Clough's world, it would be a coalition because it was a mutual benefit in terms of reward to behave in a certain way. And so to bring it back to the scalar reward, how important is it to figure out this, how to define the rewards so that they match the goals?

And is that a tractable problem? I think it is. I think in terms of learning systems, I think it's the key problem. Although I know your question about multi-objective, I mean, that would be folded in somehow. But I mean, even without that, it's still a challenge to, you know, if you want to build a system and field it in reality in a non-trivial environment, I think the issue of creating an objective function that leads to the desired behavior in a safe way, I think that's really a non-trivial problem. But it can be worked on, I think. I mean, to me, the objective function is like the key thing that needs to be elucidated.

data, but it's not just one. You know, there's a kind of hierarchy of these and lots of thoughts out there on this, but I think it's, you know, it's very surprising that your objective of the, the ultimate objective should not necessarily be the objective of the agents that are working for year. You need to pay them something. It's much more difficult for young people now. In my day, you know, we felt we knew everything that related to this, and now that's utterly impossible, not only for people my age, but for, you know, the graduate students. You know, it's like, you know it's infinite now so it makes things very difficult