GeminiとAGIへの道筋:Koray Kavukcuoglu が語る製品・アーキテクチャ・再帰的自己改善

Gemini and the path to AGI with Koray Kavukcuoglu

Google DeepMind: The Podcast46分
GeminiとAGIへの道筋:Koray Kavukcuoglu が語る製品・アーキテクチャ・再帰的自己改善

GeminiとAGIへの道筋:Koray Kavukcuoglu が語る製品・アーキテクチャ・再帰的自己改善

Google DeepMind: The Podcast

0:0046:57

要約

Google DeepMind の Koray Kavukcuoglu が、Hannah Fry との7年ぶりの対話で、単一モデルによる汎化、製品を通じたAGIへの道、Geminiのアーキテクチャの進化を語った。再帰的自己改善(RSI)については、研究パイプラインの一部をエージェントに任せる段階で、完全なループにはまだ遠いと述べている。安全性やベンチマークの飽和、競争についての見解も示した。

この回のポイント

  • ●Koray はAGIにテストは存在せず、人々が信頼して意思決定を委ねるほど「AGIらしく」感じられると述べ、製品をAGIへの踏み石と位置づけた。
  • ●かつての個別プロジェクト中心から、科学分野の課題を解けるよう単一モデルを訓練する方向へ重心が移ったという。
  • ●Geminiは静的なものではなく、各世代が手作業で組み上げられるカスタム品であり、将来はアーキテクチャも変わると見ている。
  • ●RSIは研究パイプラインの一部をエージェントが担う段階で、Gemini自身が次期Gemini開発に使われ、計算資源の一部をエージェントが実験に使っていると説明した。
  • ●安全性では、報酬を与えず手を加えない思考の連鎖(chain of thought)を監視可能に保つことを重視している。
  • ●ベンチマークの飽和は進歩の表れであり、AGIの測定ではなく能力の穴を見つける手段だと位置づけた。

章立て

  1. 7年ぶりの再会と単一モデルへの集約

    かつて分野ごとに分かれていたチームが、汎化する単一モデルの可能性のもとに統合されてきた経緯を語る。

  2. 製品こそがAGIへの道

    AGIに明確なテストはなく、製品を通じた有用性と信頼が進歩の梯子になるという主張を説明する。

  3. 汎化と進化し続けるGemini

    狭いタスクの最適化より汎化を重視する姿勢と、世代ごとに手作りされるGeminiのアーキテクチャを語る。

  4. 世界理解・効率・LeCun批判への見解

    モデルの物理的理解の限界、人間の脳との効率差、Yann LeCunの立場、人間データの重要性を議論する。

  5. 再帰的自己改善(RSI)

    RSIの定義と現状、Gemini自身が開発に使われている点、RSIが実現するかへの見方を述べる。

  6. 安全性と競争

    思考の監視、マルチエージェント、数十億ユーザーでのアライメント、減速論や政府との協力に触れる。

  7. ベンチマークと7年後の展望

    ベンチマーク飽和の意味、ユーザーフィードバックの役割、7年後のAIの姿を語る。

解説記事

Google DeepMind のポッドキャストで、Hannah Fry 教授が SVP 兼 Chief AI Architect の Koray Kavukcuoglu と対談した。2019年にロンドンのオフィスを案内された時から7年、AlphaGo や AlphaFold といった個別プロジェクトの時代から、Gemini による汎用知能の追求へと変わった道のりが語られた。

個別プロジェクトから単一モデルへ

Koray は、かつては複数のことを探索していたが、現在は汎化を生む単一のモデル・技術の可能性が見えていると述べる。核融合の制御のような科学プロジェクトも続けるが、狙いは特定問題を解くことから、その科学領域で実行できるモデルを訓練することに変わったという。方向性を決める要素として、AI研究者としての見立て、製品、科学の3つを挙げた。

製品こそがAGIへの踏み石

最も印象的なのは「製品がAGIへの道」という主張だ。Koray によれば、AGIにはテストもベンチマークもなく、人が信頼し、意思決定を任せるほど AGI らしく感じられるようになる。AGIは北極星であり、次の一歩は製品が教えてくれるという。レストランの予約のような作業を除外せず、全部解かねばならない点が製品の意義だと語る。また、狭いタスクは機械学習がすでに解き方を知っており、知能の本質は未見のものへの転移だとして、スケーリングによる汎化を主ルートとした。

Geminiは静的な単一アーキテクチャではなく、世代ごとに手で組み上げられるカスタム品で、5年後には別のアーキテクチャを使っているだろうと述べる。新しいアイデアは「ファネル」から取り込まれ、複数チームが現行版と次世代版を並行して開発している。

世界理解、効率、人間データ

モデルの理解について、Koray は現状では視覚的に正しくても、流体、気流、熱伝達などで数値的に正しいとは言えず、工学的な正確さが課題だという。エネルギー効率では人間の脳との差を認めつつ、目標は人間の知能の複製ではなく、人の役に立つ知能だとした。Yann LeCun がLLMを袋小路とみる点には、世界理解という目標は共通で、別アーキテクチャの探求もファネルの一部だと応じた。Sutton の「苦い教訓」を踏まえた人間データの問いには、モデル自身の探索は重要だが、人と協働できる道具であるために人間データを軸に据えるべきだと答えた。

再帰的自己改善(RSI)の現在地

RSIは、モデルが新しいデータやアーキテクチャを探索し、実験し、検証して次世代を訓練するループ全体を指す。Koray は完全なループにはまだ遠いが、パイプラインの一部にはモデルに任せられる領域があると説明する。Gemini自身が次期Gemini開発に不可欠であり、計算資源の一部をエージェントが実験に使っているとも述べた。RNNからTransformerのような飛躍をAIが起こせるかについては、いつかは可能と信じるが、まだ見ていないという。RSIが起きる確率を問われると、自分は偏った標本だと断りつつ、研究とは証明されるまで未知のことを信じて進める主観的な営みだと答えた。

安全性、競争、ベンチマーク

安全面では、報酬ハックを許さない環境設計、訓練中の監視、実運用でのリアルタイム監視を挙げた。特に思考の連鎖には報酬をかけず手を加えないことで、行動に先立つ思考を監視できるようにするという。マルチエージェントでも通信を観測・理解可能に保てば同様の監視が有効とした。研究から実際の製品開発に移り、安全は開発と一体であるべきだと語る。減速論には、安全に進めない場合は進むべきでないが、基準が不明なまま止める理由はないとの立場だ。競争が安全判断を妨げることはないとし、外部専門家や政府との試験を挙げ、説明が十分でない点は認めた。

ベンチマークの飽和は、モデルが多様な領域で向上している証拠であり進歩の姿だとする。ベンチマークは能力の穴を示す手段にすぎず、ユーザーのフィードバックも重要だと述べた。

まとめ

この対話の核心は、AGIを到達点ではなく、信頼と有用性が積み上がる過程として捉える視点にある。製品開発は寄り道ではなく研究そのものだという立場は、AI を事業に組み込む日本の企業や開発者にも示唆的だ。RSIについても誇張せず、現実には部分的な自動化が進んでいると語った点は、過度な期待や不安を冷静に見直す材料になる。

文字起こし(英語・自動生成)

Welcome to Google DeepMind, the podcast. I'm Professor Hannah Fry. Now, the first time I met our guest on the podcast today, he was dashing around Google DeepMind's offices in London, scribbling equations on whiteboards, directing blue sky projects that would fundamentally alter the trajectory of modern technology, all with the rather audacious goal of creating intelligence. If you're interested in AGI, and at the end, you need to work on agents, and that is something that we have been doing for a long while. Quite a lot has changed since then. These days, Coray Cabacholo has swapped London for California and those whiteboards for the even more monumental task of driving one of the world's most advanced generative AI ecosystems. He is steering the development of Gemini, now in its fourth iteration, and overseeing the rollout of AI to billions of people at once. But perhaps things still haven't changed all that much. He is still pursuing that same goal of creating intelligence. Only now, the technology is finally catching up with his ambition.

And is he still dashing between projects? Well, let's find out. Corai, a real pleasure to welcome you to the podcast. Thank you very much. I don't know if you remember in 2019 when you gave me a whistle-stop tour of the Google DeepMind offices in London. Do you remember this? I do, I do. And it was like, over there is the AlphaGo team, this is the AlphaStar team, there's the AlphaFold team. I mean, everyone was kind of in their distinct groups. I mean, things have sort of changed a bit since then, right? A lot of things have changed since then. Like we used to fit on a floor and like we could walk around the floor. I mean, it was a big floor, but it's still a floor. Now things are a lot bigger. And this is what happens when something starts working, right? Like we were exploring many different things at the time. whereas now I think the biggest change is we know that there is the potential of building a single model, a single technology that is going to deliver a lot of generalization,

a lot of results, and that sort of brings everything together and makes it bigger and bigger and bigger. Because I remember I was talking to you back then about the work that you were doing on Fusion, for example, which was like a very sort of constrained, specific, scientific application. Do you still see those kind of things as important or do you think that you can still get there from this one model approach? At least for me, it's mostly the better. The reason for that is, yes, we did all those different things and they all taught us something. And when you look at, even on the fusion side, we were doing RL, we were doing control. But the potential, the opportunity that we have right now is this single algorithm that can actually do all those things. And then more importantly, actually, like being able to train something that can do all those things. So like we would still do these kinds of science projects, but a big part of the aim now is a bit different. A big part of the aim is to be able to train the model that can actually execute in that scientific domain for that scientific problem.

So like the idea that this comes back to the model and you train the model to be successful in that, rather than solving a single problem there, is the biggest difference. Because once you actually train the model, then you can solve many problems there. That's the big difference. But then if you put those two things together, the team is much bigger than it was before. Yes. But also you're all pursuing this kind of common goal. Presumably there's all kinds of different avenues, all kinds of different projects that could combine to that end. How do you decide what to pursue? There are several things that combine there. One is, obviously, as AI researchers, we have an idea of where AI should be good. The second one is products. Products are critical. Why? Because through those, we know what people need AI for, right? Or you can explore where AI can be useful. You can't do this in a black box.

You need to understand either on the consumer side or on the developer and enterprise side. You need to understand how you can actually embed this technology. And then third one is science, as you said. Obviously, we know that, I mean, we all believe that science is the biggest driver of progress and making the models better and better at different scientific domains and then being able to take on and help with research in those different scientific domains. These three all combine together. You mentioned products there. I really want to ask you about this because I know that you've argued before that products are the route to AGI. Give me the strongest version of that argument for me. The strongest version of the argument is we don't have a test for AGI. We'll never have a test for AGI. It's not like we are going to have a benchmark and then you're going to solve that benchmark and then you say, okay, this is AGI or not. AGI is something that if people find it useful, if you trust it, if you can work with it, the more you trust it to make decisions for you

or to make decisions with you, then it will start feeling like AGI. Because with that trust, you will afford a lot of, just like you are collaborating with a person, you will afford a lot of decision-making there. And that implies generality in order to be able to solve problems in real world. And that's where you get with the products. AGI to me, therefore, is like this sort of, I explain it as like the North Star, the goal, yes. But it doesn't give you, as you ask, like, what is the next step you should do? Products give you that. Like, they are sort of like the stones that you go through. And each one of them, like, helps you solve another thing. Right? And through those, we really figure out, like, I mean, like, the idea that we can create chatbots, but people just don't chat with it. People actually start asking questions. and then we start getting into, oh, okay, people actually, like we can build things that are useful in people's daily life. We can build things that are useful in people's work. Then we can get into the economical usefulness. Then we can look at, okay, what are the areas of economy

that actually an intelligence technology should be impactful on? Those are the ones that define, of course, software engineering and coding, but there are different areas of knowledge work. So trying to build AGI without products in the loop, I don't think it's going to happen. Almost like the usefulness of it is part of the definition of what makes it AGI. Exactly. Like, otherwise, why would you call it intelligence if you know that it's not useful to you at all? Like, how is it intelligent then? Right, yeah. Start with the application, essentially. Exactly. Do you think when, slash, if you get there, will AGI just turn up as like a Gemini app on your phone? I mean, what's it going to, you know what I mean? How will people use it? This is another thing that I have a little bit of a different perspective on. Again, I don't know what it means for it to turn up or we get there. I think it is more about the journey because go back decades, go back 15, 20 years ago,

there was always a definition of AGI. We always tried to quantify it in terms of these benchmarks or in terms of these tests. and we always thought that if we do this, then it will mean that the models achieved some kind of intelligence and at least we have solved some area of intelligence, right? We have always achieved those benchmarks. We always thought we didn't actually capture the gist of the intelligence and you can see that this is the regular cycle that happens with research. What is important is, do we actually have a technology that we trust more? Every day, every month, new models coming up, and we see this. We see models becoming more useful in our lives. We see us trusting these models more in our work, in our daily life. I think that's the progress. That's the ladder that we should be climbing. I'm not sure about the threshold on when we say it's AGI or not, because also certain things we do, the models do really well. Certain things they are really bad at. And many of those things that the models are really bad at,

slowly they are getting better. Like if they're a bit spiky at the moment in terms of what they can do. Yeah, I mean, they're much less spiky than what they were, but still they're spiky. I guess what you're saying is that you're sort of incrementally making it less spiky as time goes on. And that is by making the models part of our life. In terms of what people want, though, I mean, they want an agent that will book a flight for you, that will sort out a restaurant for you, that will interface with your calendar, that will talk to travel agents. But I know that when we spoke to Nenad Tomashev, about this, it's really hard to do. You have issues with cybersecurity, with agent protocols. I mean, there's a lot of faff, basically, in getting this stuff to actually work. So I guess my question is, when you're building these commercial products, does that friction end up pulling you away from advancing intelligence itself? Not at all. Because if we don't, then people get to ask the question, you can do all these things, but you can't book a restaurant for me. and that is the definition of spikiness and if you're talking about the general intelligence

like if i can solve all these like hard problems i gotta be able to book a restaurant surely right surely so that's that's part of it i wouldn't make too much of a differentiation between all these kinds of tasks are important that is why let's define intelligence as that and these kinds of tasks are not important and that's the importance about products when you do those you have to solve everything you don't get to choose or i'm going to solve this or i'm not going to solve this we are not coming up with benchmarks that define intelligence we are trying to build it i really like that idea but i also it does make me wonder about whether you're giving yourselves a sort of bigger challenge than anyone else because you're trying to integrate gemini into maps and documents and email and calendar and so on which have like billions of users each um that's much harder surely than just saying here's this narrow constrained version that we're building conceptually harder but also easier once you unlock that point about generality

i think that that that sort of frees you up i always worry about over optimizing for a single task because with machine learning we already solved that. We know how to do it. The parameters of that is very, very simple, right? Like we know that we have a class of models that if you have a narrow task and if you have data, that model will learn how to solve that task. I mean, this is what you were doing with Atari all those years ago. I mean, like even before that with deep learning, we basically verify that you can train a neural network on data and if you have enough data and if you have a model, you'll get the result right that is that i don't think anyone questions that anymore there were times we were questioning that like we are not questioning that anymore so then like when we think about intelligence it's not about a collection of these narrow things intelligence is the generalization over these intelligence is transfer from these into something that you haven't seen before

that's the capability that's what amazes people with lms because we are just inputting the knowledge there it actually generalizes into all sorts of things that you didn think before that you didn think that you build a model for that kind of task but it actually does that That the kind of generality that you want That is why we think about scaling these with more data and more capabilities is the main route to intelligence. One of the nice things that happens is you train a model, and then, of course, a lot of people take it, and then they try different kinds of tasks and benchmarks that they have been working on. And there are some domains and tasks that no one thought about, and people try and say that, okay, this model is doing this well on this kind of thing. And to me, those are the ones that I really like because all of a sudden, okay, this is exciting. We definitely did not optimize for it. Because it's the generality. Because it's the generality. I'm not claiming that these models generalize and transfer to very unknown things. I think on that part, we are still very, very early. But the goal of building intelligence, the goal of building AGI has always been the only single goal.

and we are trying to find ways on how to do that in the best way that we can. There's nothing else that is a goal for Gemini other than that. There is this perception that Google DeepMind now has a lot of eggs in the Gemini basket, as it were. How confident are you that you would notice the next big thing, the next big advance, and be able to implement it in your models? A hundred percent. I think that's a good question. But then that comes down to Gemini is not a single architecture that stays static. It's not like that, right? Like it's the pursuit of intelligence. And right now we have architectures that give us the most efficiency in terms of getting gradients from the data that we have and update that model. and we keep exploring and researching for different architectures that will enable us to do that. Pretty sure, five years from now, we will be looking at different architectures.

My goal is to make sure that it's like a funnel. Right now, we have a particular architecture that we are using. It's different from what we used six months ago, which is different from what we used six months ago, and it has been evolving. and it has been evolving by taking ideas and new developments from this funnel. So when you say this funnel, is the idea that sort of at the wider edge of the funnel, you're experimenting, you're trying out different things, but actually putting it in a position where it can reach a bit of users is a different game altogether. Exactly. Because once something ends up in products and such, maybe a lot of people have the idea that it's pretty much fixed and it doesn't change much. whereas the way I look at every generation of Gemini is a custom job. It is put together by hand and there is no other version of it because the next one is going to be different. We go and put together the next one by hand as well, looking at everything that is there. Obviously, it's what you learned from the previous one. That's why we need the large teams

because while you are working on the current version, there's a whole team working on the next version and there's a whole team working on the next, next version. so that people build ideas slowly and they go into what is right now. The current Gemini that you see, it has a lineage of at least a year. Right. During the year, maybe we release like six different models, so they all overlap. So how different then is the Gemini that I could have access to and the ones that you're playing with? the ones that I'm playing with or we are playing with we have a lot more options and depending on the time it might be very different or it might be close it really depends on what is on the pipeline if you ask me this question before we released argon I would say very different now that we have argonaut and we have other models in the pipeline it's closer but like if you ask me what are the models you're going to play with in the next like

two three months obviously it's going to drastically change so then with the things that you've been playing with do you think that you're starting to see sparks of AGI I mean maybe more than sparks actually do you think that we're there again like I love this question is AGI is it AGI or not or when do we get to AGI I think I asked you this in 2019 right i don't know i don't know what i said like at that time you said seven years and one month seven years um here's what i would say when i look at the progression of things that are happening this i believe is definitely the path to agi but like to me it really does not matter if we call something agi or not you go back 20 20 years ago like you would definitely call this an idea right but then like you live in this world and you see those spikes and the lows and the highs and then you can say okay like there are still these things that it doesn't do once it can book restaurants car that's the last thing um just talking about 20 years ago

i mean when you did your your phd i mean you've been researching in this area for for decades right i know that some of your thank you sorry you don't look it I know some of your early research was on learned representations, right? So this idea of how a system kind of builds its own picture of the data that it sees. In terms of the models that you get to see, sort of how much conceptual understanding do you think there is? Okay, let me ask you a question, Mike. Like, how would you know the difference? I don't know. You're the researcher. So that's the whole purpose of doing the generative models, right? Because you don't know exactly, by building a generative model, what you try to hope is that as you can query the model and as you can ask more questions about hypothetical situations, and if the model is giving you realistic answers, then you believe that it has produced accurate representation of the world. Whatever the data that you put in is the definition of the world, by the way.

And we can always argue the gap between that world and the real world. But at the end of the day, that's what we believe the understanding and the intelligence comes from. And that's the predictive power, right? Because if you have a model, if you have an understanding that you can predict things that can happen, hypothetical situations that can happen, even the ones that you haven't seen before or the model hasn't seen before. Like the canonical example would be in the early days, pink elephant. There's no pink elephant in my data. Can I make my model imagine a pink elephant? Yes, the models can imagine pink elephants now. Does it mean that they can actually, they've learned the whole world and they understand? They understood a lot of things, but I don't think that we are there, that they understand the physics and the dynamics and the collisions well enough. I don't think that they understand the fluids well enough. I don't think that they understand the airflow or the heat transfer and all those kinds of things well enough. Right now, the models are visually correct.

I'd like them to be numerically correct in the engineering sense of it. Right, and correct in the sort of physics sense. Exactly. Okay. Do you think that's still to come? Do you think we'll get there? A hundred percent. Right. Yeah, we'll get there. Because these are computational machines that can do any kind of computation. If we actually train them in the right way with the right data in the right environment to predict the right things, they should be able to do accurate numerical computations that can be part of that engineering flow. And to me, that is important because, again, engineering is the thing that makes the world work. Right. Because otherwise you don't end up with this full representation of the world that we're living in. I know one of the other things that you're working on in the early days was about the efficiency of your systems. You know more than anybody how much it takes to train a frontier model now, right? And I sort of wonder about, do you think that that is just what intelligence costs? When you compare it to sort of a biological system, the famous line is that human brain requires less energy than the light bulb.

Do you think that we're maybe missing something fundamental? That it takes so much more energy to use this version of intelligence than the one that you see in biological creatures? I mean, clearly there's a difference. Clearly there is a system that requires much less power. So hence, there is a lot more that we need to achieve to get to that human level efficiency from that point of view. At the same time, we are not building biological systems. We are building digital systems. And yes, they consume more power. And there's a gap there. The way I would think about it is there are definitely systems, both software and hardware. that can be more efficient and more powerful than what we have today? Will it actually lead us in the exact path as the human brain? I don't think so. And I don't think that's our goal. We are not here to necessarily replicate the human intelligence

either in the wetware or the software form. As I said, our goal is to build intelligence that is helpful to us. And with the current technology that we have, we are building algorithms that can be efficient. There are some things that we can do a lot more efficiently than human brain with our current systems. And I think that complementarity is critical. Because there are some people who say that this isn't the right track. I mean, I'm thinking about Jan Nakund here, for example, who I know that you worked with closely when you were at NYU. He sort of says that LLMs are a bit of a dead end and that actually we should be thinking about something different altogether. You see it differently. What do you think that he's missing? I think, like, the general idea that we need to understand our world and we need to build a genetic model for our world, I think that's a shared goal. But obviously, like, in Jan's case, like, he has a specific idea, and he's someone who has proven that he has good ideas, right? Like, I mean, I work with him. He chose you as a PhD student. Right, yeah. It was a private idea.

So, like, what he says is he doesn't think LLM, says the neural network architecture is the successful one. That's great. Like, I mean, right now I feel like they are a successful architecture. But as I said, five years from now, it might be a different architecture that we are using. And that's perfectly fine. That's the research he's doing. That's the research we are doing as well. That's the funnel that we need to continue doing. We can't get stuck on what works right now. And imagine that five, ten years from now, where we predict that the whole world is going to change, that our model is not going to change. I don't think so. There's also this idea about human data. I'm thinking here about the bitter lesson which Richard Sutton wrote actually wrote it in the same in 2019 the last time we properly spoke on camera It was the most pivotal year in the history For those two events alone but the argument that he makes in that is that you can end up constraining a system by putting too much human data in it the difference between AlphaGo and AlphaZero once you take away that human data you can end up with a system that can accelerate beyond

what it was previously capable of. I want to get your take on this. Do you think there will be a point when we step away from human data and really see things take off? Yes and no. I wouldn't categorize it as step away, but there a lot of data that the model needs to generate itself and explore by itself And through that it can figure out strategies in the human data that you don have So that's real, right? And you can exactly see this in an environment which is fully observable and constrained like Go. The model can explore things that the humans did not because still that search space is a big search space. it's small compared to the general language of exploration space but by enabling the model to explore by itself you are giving it more opportunity but I don't think it has anything to do with moving away from human

data especially again if our goal is to build intelligence I think it is critical that we anchor ourselves on human data because we need models that are useful for us in our world that we can use as tools, that we can actually rely on, that we can work together with. And that relies on understanding how humans work and humans like to work. I think it's a really persuasive argument, the way that you put it, that why would you possibly not want to put the human at the center of it? This other thing that everyone's getting excited about at the moment is recursive self-improvement. How do you define that? When we think about recursive self-improvement, we think about a system that can train itself, which means that if you think about the whole loop of researching a model, that whole loop that a model can do by itself, that it can explore new data, that it can explore new architectures, that it can run experiments to verify

if it is going to do the research in this domain or in that domain, that it will verify the success of those, and then train the model, and then ideally train multiples of these models so that then it reaches, okay, this can be the next generation of it so that it's that self-improvement loop. We are a bit far away from that. Where we are is more closer to, there are pockets of areas. If you think about, we talked about the whole pipeline of research that needs to happen to build a model. and there are pockets of that pipeline that we can rely on models to do the work by themselves and go over different hypotheses and build models and train models and look at the results. And as this gets more and more and more, we keep getting closer to RSI. I think you're right that sometimes the way that it's spoken about is almost as though you go in and you hit a button and it says, give me the new version of Gemini,

like Gemini 13 and then 14 and 15. Like that is sort of how it's... I think that's how a lot of people imagine. But to get there, first you need models which will understand, first of all, what is the goal? Second, what is the data? Then what it means to do research? What it means to actually come up with a better algorithm? What it means to do experiments, at least in our own space, right? So there's a lot of exciting research that happens there. and there are improvements that are happening. Of course, when you talk to anyone in Frontier Labs, this is one of the areas that all Frontier Labs, obviously us included, we are very focused on, we are investing in. But also I don't want to give the impression that at least for us it's not like we have a full RSI loop that, yes, we are pressing a button and it does everything by itself. But it is true that Gemini is one of the most critical things that we rely on while building the next version of Gemini. There is a percentage of compute that runs in Gemini

that is under the control of the agents, that they run experiments. And I think that is just going to happen more and more. Do you think there will be a point then where you press a button and it gives you Gemini 13, 14? As long as we are clear about what we expect from the 13, 14, yes, there might be, But, I mean, the trick is about, like, anyone who has built something, it is always about, like, what do you actually want? Like, what do you want that goal to be? Again, the goal is not RSI for the sake of RSI. The goal is to be able to build something that is useful for us, that we can use. And the definition of that is very hard. There is no single benchmark that tells you what that definition is. Of what you're actually aiming for. There's one other thing about RSI that I wonder about. there's almost this sort of implicit assumption within it that there'll be a flywheel, that the better it gets, then it can start to accelerate and really take off. How can you be sure of that and that you won't end up in a situation

where actually there's sort of a cap on its capability and then you'll just get to a position where there's marginal gains? Either it's possible. I don't think anyone is sure, but I don't think we only do things that we are sure. Like, that's the whole idea of research, that we explore things that we think are possible. And if they are possible, then they work. And that's where we are. I mean, agents are very good at climbing hills that already exist. But then thinking about the, I mean, we've spoken a little bit about the change in different architectures, different approaches. The jump from RNNs to Transformers, for example, sort of like a completely different hill altogether. Do you think an AI system can make that sort of a jump? At some point it will, yes. that's what I believe you can ask me, are you sure? no, I'm not sure, but I believe it is possible so that's what we should explore and that would be a real test that would be a good test that would be a real creativity and yes, it could be and I think it should be possible I don't think we are there, at least I haven't seen

but I think it is possible that we will get there do you think then the RSI could end up being this genuine scientific phenomenon as opposed to just I mean, I think at the moment it's sort of a story that we're telling ourselves. 20 years ago, the whole AGI was a story. Now we all take it for granted. It's part of our lives. That's the journey that we should be in. If you're a betting man? No, if you're a researching, exploratory, frontier-seeking person. What would you put the odds on that RSI will happen, proper RSI? I mean, look, I'm biased. my biased sample is like obvious like a sample from yes it will happen all the time until i'm convinced that it won't then but that's the thing about research right research to me is the most subjective thing that anyone can do in the world the measurement of research is very objective it has to be but why someone goes and researchers a topic that until you actually prove that it is

true it is not known is a very subjective phenomenon and that is you your creativity and your motivation as a researcher. So I want to also talk to you about safety with Gemini. Because one of the things that you were researching was reinforcement learning for years. And presumably, during that time, you spotted algorithms coming up with all sorts of ingenious ways to work their way around the constraints that you try and place on them. How do you mitigate against those now that we're in a world of agents? The old and the new doesn't matter. I think as systems get more intelligent, of course, they will find ways to accomplish the task. Now, the question is, and a big part of research, is building the right environment. Because in the right environment, where cheating or shortcuts are not rewarded, then the model won't learn about those. It is important that we create the right environment.

That's number one. It is important that we actually very carefully monitor the training if the agent for some reason builds up those cheating behaviors. And then obviously in the real world, that is why you need to continue monitoring the agents and make sure that that monitoring is always on, on real time. such that if the agent deviates from what you intended to do, you can actually interject. And one of the most important things for this is designing architectures that are monitorable. Like right now, our systems have a chain of thoughts, right? They are not visible to the end user. These are literally the model's thoughts. And those thoughts being pristine, they are being untouched, is very critical for safety. We pay a lot of attention not putting rewards on those thoughts so that the agent's thoughts are in a way unfiltered.

And at execution time, we can monitor those because every action that the agent will take will actually first manifest in its thoughts. Together with the actions, we monitor the actions of the agent, the environment, what it does, the side effects on the environment, and the agent's thoughts to build a detector of, is the agent still executing on what the user intended. But it's not just agent. I mean, agents as well. I know a lot of your earlier work was about sort of creating teams of agents to achieve a particular objective. And now we're really in an era where we're seeing teams of agents working together to sort of do slightly more underhand things. Yes. How do you monitor those? The job gets more complex, especially with multi-agents. There's an explosion of intelligence that you can expect to happen. but as long as you keep the thoughts you keep the communication channels observable, detectable and understandable then I think you can still the same idea of monitoring

of course like at scale still works hence that is why always anchoring on what I find really useful is sometimes people question do the thoughts need to be in natural language it has two effects one regularization two we can actually introspect we can actually study what is going on both agent to agent but a human can study i guess a human can so and and that is important but i i do sort of wonder how um now that you've got these systems with multi-agents that that you know are sometimes working together but also that you're creating products that are going out there to billions of people quite literally has how you think about safety changed through that time inevitably in what ways in every possible way like i mean like from the days where first like building intelligence was more of a like an academic research exercise where we were like we were training these things

on games and like the capabilities that the that the models and the agents had did not expand beyond that game, beyond that environment. But especially with coding and tools, like the capability that the agents have in the digital world is like being able to get out of that environment. So then the safety and security aspects of both training the model and deploying the model becomes extremely important. Safety and development should always go hand in hand. Like this is where I think we moved from science of researching AGI into building AGI which is a lot more of an engineering activity You don imagine building anything that is not safe And that the mentality that we should all have that we only build things that are safe. But I still get comfort from knowing that there are many different ways we have found to develop technologies that can be sensitive. You can create like

air gaps or like sandboxes around them and you can develop that technology and you can build something that is safe to use in real world do you think that building things that are used by so many people simultaneously makes it easier or harder to end up with something that aligns with human values because they're going to stress test it right oh yes i mean i mean like both ways right like it's easier because then you get feedback from so many different type dimensions so many different directions, but also at the same time, it is obviously harder to get to that level. This is not any different from the conversation we had about building something narrow versus building something general. If I were concerned with building something that is only useful for a particular task, I wouldn't have too much of an alignment problem. I know what task I'm solving. But because I'm building something that is useful in general,

that is why we are talking about the alignment problem because the task comes in real time from the user. And you have to make sure that the model follows what the user wants. So, yes, it is harder, but the fact that I'm doing this across a large population means that I can actually understand the true way that I can get the model to tune for whatever user wants and everyone wants something else. There's been a lot of talk recently about this idea that the lab should slow down their research. What's your take on all of it? I think my take on it is if on this path we are having trouble finding the next step, that is the step that we're comfortable, that we are safe with, then obviously we shouldn't take. If it leads to a slowdown compared to whatever hypothetical criteria that people have in mind, that should happen. But I don't think that there is necessarily a reason to slow down or stop

based on a criteria that we don't know either. Every lab, I can speak for our lab, that we should be building technologies that we know that we can release to the world and that it's a safe technology to use. We have to be clear about the side effects of those. We have to be clear about the risks of using these. That is why we are very careful about researching the mitigations for the model that making sure that the alignment is there, making sure that the model doesn't do things that we think are clearly unsafe. We work with experts outside of the company to give them the models so that they understand the capabilities and they can help research both the safety aspect, but also use it in their domains to understand the impacts of the models. So all this machinery is there in a way, like if you go back two years ago, releasing the models, did not involve any one of these steps

because we knew that the impacts of the downstream impacts of the models were much more limited. Because the thing is, I mean, I think that you've proven over and over again that safety is really genuinely something that you care about very deeply. Yes. I think that there's this fear, I suppose, in the general public that more generally, across the board, it might get to a point where it's difficult to make a sort of independent safety decision when ultimately the labs are in a race with one another. Are you getting that sense from people? I am definitely getting that sense. And I think we are not doing a great enough job to explain all the mitigations we are putting in the model and all the risks that we foresee, but also how we deal with those risks. Again, I can talk for myself. The competition aspect is not something that gets in the way of doing the right thing.

And it shouldn't. Right? I can understand the worry there, but this is why we also work with governments. Before releasing a frontier model, we work with the government to do the safety testing. We are working in ourselves to make sure that we build mitigations that we think are going to produce a model that is robust and safe enough for people to use without unintended side effects. All these things are there because we care about the technology and the technology being safely in people's hands. I mean, I think that there is this big conversation about competition. Can you even tell who's winning? I guess this isn't really a question about benchmarks, right? But lots of these benchmarks are becoming saturated. They're becoming contaminated by training data. I mean, everyone's talking about it being a race. But can you even tell who's in the league? I think we all have different intuitions and notions about what quality means

and where we latch on to in terms of quality. But your question about benchmarks is right. And I think in the grand scheme of developing intelligence, benchmarks are serving a different purpose. As I said, benchmarks are not there to measure, oh, did I reach AGI or not? Like, nowadays, benchmarks are there to either measure the capabilities of models in a particular domain, or if you are doing really bad on a benchmark, like, there are, like, you should think about, okay, maybe there's an interesting question there that people actually find useful and your model is not doing well there. So the example that I give is if I don't show any math data to this model, I don't expect to know about math. You're not going to teach the model to ride a bicycle and then expect that it's going to ride a motorbike. That's not going to happen. That's not the kind of generalization we are looking at. So if a benchmark tells you that you're not doing well,

Like these 1-2% differences is a different story. But to me, the most important thing about benchmarks, if a model is doing really well, there's something interesting there because either there's a bug or a mistake somewhere or there's some interest. And if people find that benchmark interesting, there's some interesting task that we didn't think about. And if we want models to be useful in our world with people, then I need to train my model to be good at that task. Of course, what I'm going to do is I'm going to figure out how can I formulate that. I'm going to create environments. I'm going to collect data. And then I'm going to train my model with the expectation that it's going to do better on the test. That's why gradually we see all these benchmarks getting saturated because the models are getting better in many different areas. That is what it should be. That's what progress looks like. And then, of course, then we explore new benchmarks. What has happened in the history of machine learning over years is playing out over weeks. Mm-hmm. But then do you think this will keep going, that it's just you saturate the benchmarks,

you have the benchmarks, you saturate those benchmarks, and so on, and so on, and so on? That's how we train the models, don't we? Yeah, that's fine. Right? Like, I mean, I'm not very, like, concerned about the benchmarks themselves. Like, I mean, at the end of the day, we optimize the models to accomplish some tasks. It is more important to pick those tasks right. That is where products are useful. As I said, I know that as AI researchers, there are some tasks that we want the model to be more useful at. And as scientists in different areas of science, we know that we need the models to be better and useful in those areas. But it's like the benchmark's a way to highlight maybe the gaps that exist in the model. And I guess the some billion users that you have are also pretty useful for that too. Of course it is, because as I said, there are many things that is very hard to codify as a benchmark. So the feedback that you get from the users and that experimentation is a big guiding point. And this is even true for things like coding.

Coding is not just about writing correct code. Like two years ago, all the models were reaching almost top human level on competitive coding competitions. but like it's not like the code that we write day to day of course you need the models to be able to work with a software engineer and the software engineer should find the methodology of the way the model works useful for them so that they use it so even things like coding and scientific research you are going to work with a mathematician like you have to work with a mathematician you're going to work with a chemist you have to work with a chemist you need to understand what they expect from a model which is different to just solving an olympiad exactly yeah okay so let me ask you this then if i come back here in seven more years seven more years seven more years okay who's going to give the tour then i don't know maybe not probably not me gemini that's possible

maybe why not you where will you be i don't know like seven years is an eternity but like what i hope is that we'll have robots here that will hopefully like help with a lot of things i mean i hope that seven years time when you visit yes i will be here and like we'll have the conversation but also at the same time like genuinely the reason i i said i have not me is like i hope that i enjoy like growing a team i hope that there will be many people here who you would be excited to have a chat with what do you think aj i will look like by then will you look like well be like act like will it be able to book me a restaurant finally finally yes i think that's the that's the real that's the real benchmark that's the real benchmark my hope is and i think maybe naively i've been saying this for a long while i expect that like we will think that it is normal that we have this really intelligent thing that we can access through our phones

or our tablets. But also, in our professional lives, I think the opportunities will be much, much, much bigger. Because any question you have, the amount of progress you can make in the minimal amount of time will be very different. Something to potentially look forward to. Indeed. Kaur, thank you so much for joining me. Thank you very much for the conversation. Thanks a lot. There are plenty of people who are out there making these convincing arguments that a focus on improving the current generation of models might be a distraction from the next big breakthrough. But I have to say I was very persuaded by Corey's idea there about using people as a compass. Because intelligence doesn't exist in a vacuum. And if AGI is ultimately about matching or surpassing human capabilities, then understanding what people actually need from these systems doesn't just make commercial sense. it might be one of our best routes to making sure that they are aligned with us you have been watching google deep mind the podcast with me your host professor hannah fry as ever we have

plenty more episodes coming up later in this series make sure to subscribe so you don't miss them

番組の概要欄(原文)

Seven years after their last conversation at Google DeepMind’s London office, Professor Hannah Fry reunites with Koray Kavukcuoglu, SVP of Google DeepMind and Chief AI Architect at Google. Back then, DeepMind was focused on specialized breakthroughs like AlphaGo and AlphaFold; today, Koray is leading the charge toward general-purpose intelligence with Gemini. In this episode, they discuss why real-world products are essential to building AGI, how Gemini’s architecture continuously evolves, and the topic on everyone's mind: recursive self-improvement.

出典・この記事について

番組
Google DeepMind: The Podcast(Hannah Fry)
元の回
Gemini and the path to AGI with Koray Kavukcuoglu
配信日
要約作成日
公式サイト
deepmind.com/podcast

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアSpotify で聴くApple Podcasts で聴く

関連エピソード