トム・ミルトンが語る、米中AI合意をどう検証するか

Your most likely route to the Nobel Prize for making AI go well is that you are the one who develops the verification tools that enable a deal. You would need to retrofit like every single data center in China and every data center in the US. Like it just seems to me like it would require a level of cooperation that's completely impossible. When I speak to my non-AI friends about this, they kind of say, why can't you just turn it off? And you do always have this fallback that you could turn data centers off. We could turn all data centers off in the world, but That's extremely costly. Lots of what verification is doing is reducing the cost to complying with some kind of slowdown. Just by looking at the power, you can be like, okay, great, there's no backwards path is happening here. As the power is being used, it fluctuates. So you can see all of these little spikes, the really small ones. That happens at the start of each layer. A marginal AI safety, like really talented AI safety researcher, could have a huge impact on verification. I think if you started on this today, by the end of the year, you'd probably be one of the best five people in the world working on verification or one of the most useful five people in the world.

Today, I'm speaking with Tom Milton, who is the CEO of Emoto Design, which is a company that builds hardware for scientific research, biosecurity and AI verification. And it is above all the last of these that I'm interested in talking about with you today, because I feel like the problem you're trying to solve is both very hard and very, very important. Pretty much any time I look at AI news, it looks to me like there is another lab CEO or another letter signed by thousands of employees insisting that they want to slow down the progress of AI development, but they are personally unable to do so for fear of losing the frontier or ceding the frontier to either another American company or really in the long term, China. And it feels like so much of what we would want to do on AI safety in basically any capacity is bottlenecked on this ability to trust that if everyone wants to slow down, the other actors are in fact also slowing down.

At Emoto, you are building tech which would allow us to do this. Could you tell me exactly how this tech might work? Absolutely. We'll start by setting aside the fact that I think a unilateral slowdown could also be good. We'll go on from here, assuming that people are shooting for multilateral actions. When we think about a slowdown, the problem is trust. And that's why the problem is knowing what is happening at another AI developer, whether it's a country or a company. and we can then break that down into some of the inputs of development. You could think about the employees as one input, think about the software algorithms and data, you could then also think about the compute. And we have focused on the compute. This is the physical chips that AI development and deployment runs on and it's a really attractive place to do our verification, to do our checking whether someone is slowing down because they are very specialised to the AI setting. you need many thousands of them and at least for now they all need to be in the same location so it's a great place to start for monitoring what different parties are doing on AI.

And so you're building ways that you could look at compute and make claims that would be verifiable by someone else about what that compute is being used for. What kinds of claims would you be aiming to verify? Exactly so you've said the word claims there to explain a bit of language at the top we say there is a prover who makes a claim and a verifier who then checks that claim. So a prover Or it could be an AI company. Exactly, it could be an AI company, whether that's controlled by a country or not, essentially the developer. And then the verifier, this could be another country, it could be a third party, whether it's an embedded auditor or an international organization. And the claims that people want to verify depend kind of on what your goals are. So recent examples might be OpenAI claiming that they have stopped training after their latest RL accidents. So you might want to verify the claim, we are not doing any reinforcement learning inside our company. And as AI safety science advances,

I think we'll have more specific claims that we want to verify. So right now we can say, this data center is on or off, there is no AI activity happening, or there is some. We might want to go all the way through to details like, this safety training technique has to happen, or this type of training technique is too dangerous and we promise we're not doing it or these data were held out. We are or aren't using neural ease. But most of that requires science and AI development to... To understand which claims you would want to be able to make to begin with. Exactly. And so the verification R&D process is allowing us to make more and more fine-grained claims. So we're not stuck at this very costly level of, oh, the data center is literally off and therefore I know for sure there's no training happening there. Yeah, when I speak to my non-AI friends about this, they kind of say, why can't you just turn it off? And you do always have this fallback that you could turn data centers off. We could turn all data centers off in the world, but that's extremely costly. Inference is valuable.

Providing access to the models is valuable. And lots of what verification is doing is reducing the cost to complying with some kind of slowdown. So at first it might be turning the data center on or off. Then it might be serving only inference. And being able to prove your only serving inference allows you to still serve inference whilst being worried that training would be dangerous and should be stopped. Then we might want to do something like resume small scale AI safety experiments, alignment experiments, for example. And this is where beginning to make more and more precise claims about how compute is being used is more and more useful. It both reduces the cost to coming to a deal, but also opens up a bunch of value for humanity in still being able to use the compute for something and still being able to use the IIS for something. Great. Let's zoom in on this example of trying to verify. Someone claims that they're using a given data to do inference and not to do training. And, you know, you can imagine this being useful if OpenLI wanted to verify to someone that they had in fact pause the training run.

Or you could imagine it to be useful in some grander scenario where the US and China have agreed to pause training for... come out of time. AI2040 plan A imagines this kind of thing. From what I understand you've built a prototype that is working towards this that works on a small that could verify that a small cluster of GPUs is in fact being used for inference and not training. Could you at a high level walk me through how that would work? So we've got a pilot of a system that uses what we call inference recomputation. So if you were thinking about monitoring the compute, you could go all the way down to the GPU level and you could try and look at the instructions that are running on the GPU. We operate at a slightly higher level so that it's less invasive and we look at the network traffic. So there's this front-end network connecting the GPUs to each other and also to the outside world and we look at the information on that network. We install a network certifier. What this basically does is forward on the information to our verifier node. The information

still goes on to its original destination. It's just duplicated. And then on the verifier, we run a recomputation. And you can think about this as simply rerunning the inference that happened. So normally your input tokens go into your model. You do your inference and you get your output tokens. What we are doing is then feeding those through the model again and checking whether they came out the same. And this allows us to look at the front-end network, look at the information on that network and confirm whether or not it came from the model that we all agreed should be running. And of course, there's some non-determinism in these models. You need to account for this. We use an algorithm called DIFFER that looks at some of the probability of choosing different tokens and on a long section of text can have quite high confidence as to whether these tokens did indeed come from the right model. And if no training is happening, then that model is staying constant and therefore you know that it's only doing inference because of the way the model would be changing over time.

Yeah, so the model is staying constant is one thing. The other thing is there aren't packets, there aren't bits of information on that network that can't be explained by the RIMC computation. So another way of describing this is bounding the unexplained communication. So if you can explain all of the communication that's happening, then you can know that all of that comes from the agreed inference. if there is this unexplained information on the network, then there is a workload that you don't know about happening. Interesting. Okay, so just to lay this out again, you're claiming to me, here's a set of GPUs that I have, here's the model that I'm serving on these GPUs to these customers, and you're going to be able to trust me because you are going to have a part within my data center where you can rerun the model that I'm claiming I'm using, and it will produce the same tokens as the ones that are actually being sent to external users. And in addition to that, I can also put network taps into your DIT server

and they will also be able to certify that there's no other things going on other than the inference that I'm also able to recompute. Is that correct in the summary? I have the verifier. You're coming in to check me. Let's say I'm a verifier, yeah. You're going to put in your server that does the rerunning. The network tap, tap stands for test access point, is a piece of fairly standard data center networking equipment. And all that's going to do is forward the packets on to the verifier. So this is the piece of equipment that basically bridges your verifier hardware to my Prover hardware. And the main thing that you're going to do when you come in is install that tap to forward the packets on to you. So you get a copy of them as well as me having my original copy of them. And you've built a small scale version of this and it works. So I would be able to run this, and whatever you're running on your GPU, I don't need to look into your GPU in any kind of invasive level of detail. I can just look at what my own repoputation is providing, and I can be certain that you're running the model that you claim. What scale does this work at?

So we've got this running on about 30, 32 H100s, H200s, at our demo research cluster in Sheffield. Great, okay, that's cool. And are you hoping that this basic prototype would be the kind of thing that could be used in a deal between the US and China. I mean, this strikes me as requiring quite a lot of mutual trust. I need to trust that when I'm putting, I guess, my server within your data center, you're not then tampering it. You need to trust that the network taps that I'm putting in your data center, they're not somehow picking up on IP that you don't want to give to me. Am I right in thinking this requires a lot of mutual trust? So I think the things that get implemented in future deals are likely to, rather than relying on recomputation, end up relying on zero knowledge proofs. Well, recomputation is the part where I rerun the model weights. Exactly. And the important part of this is that rerunning meant that I had to have access to the model weights as the verifier. You're the verifier. You had to have access to my model.

And it was within the security boundary. It was within my data center. So I had some confidence over the security of the model. I was fairly sure that by putting it within my walls, you weren't stealing the model. But we would much prefer if I didn't have to share the model with you at all. And the way we can do this is with zero-knowledge proofs. And there are a few teams already working on this. We're working to implement this. This is ongoing work. There's not an established way to do this. Well, there are some established zero-knowledge machine learning approaches. the main problem with what's kind of referred to as KML or zero knowledge proofs for inference is that they're really slow they're very computationally demanding many orders of magnitude more computational overhead to run using the zero knowledge approach although this is like coming down and seems like something that actually could hill climb with the AIs quite effectively the reason I'm hopeful on implementing in the type of system we're talking about

is that we don't need to check every single packet on the network. So although it may be thousands of times slower to generate this proof, actually only checking one in a thousand gives us some reasonable confidence. This is the way you can sample some of the packets that are being sent around without having to do with every single one. Exactly. And you know that they're not scooping you in some way. Yeah, some sampling and the fraction you need to sample is roughly on the game theory you expect of their cheating and how worried they'd be about being caught and the probability of getting caught. So right now, a few of our team are implementing some of these zero-knowledge proofs on top of the prototype I just described, so that rather than having a recomputation server, that verification server is running some of this zero-knowledge system, and therefore the prover doesn't need to share the model weights with the verifier, and that's going to be much more palatable on an international scheme. Great, great. And as I understand it, this is just one of the ways where you could verify you're looking at a specific type of information,

you're looking at the network traffic or whatever, to understand what a data center is being used for. But I believe we have here with us a prop in the studio which tries to establish the same thing, aka that inference and not training is happening, by looking at the power that is consumed by a data center. Could you walk me through exactly how that works? And for our audio listeners, we have a prop here. You can follow along on video if you want to see exactly what it looks like. But we'll do our best to describe it in the requisite level of detail. Yeah. So in addition to monitoring the network, there are a bunch of other like physical things we can try and monitor about a data center and about compute. One would be how the memory is being used, doing random challenges to see what's in memory and then correlate that to a workload. we can generate these zero knowledge proofs we could do something like proof of work which gives you like cryptographic proofs that the gpu all of it the gpu's resources are being consumed and then another thing we can do is monitor something like power so the gpu is drawing a

lot of power we all know that data centers use lots of power but as the power is being used it fluctuates. So the power consumption of the compute changes with the operations that it's doing. So in this box here, we have a kind of mock setup of a simple server tray. We have an NVIDIA GPU, a Blackwell series GPU. And the hardware that's part of the verification here is this small inductive loop. So this is wrapped around the power supply to the GPU. and then we can see on our oscilloscope here how the power is fluctuating over time. So we've got time on X and time on Y. And you can see, firstly, a forward pass and then a backwards pass and then a break. So what are we running on this view? What did we run on this view? I think we ran a nano GPT or something GPT-2 or something like this. And you can see actually a bunch of properties just from the power consumption.

you can see all of these little spikes, very small ones. That happens at the start of each layer. And you can actually tell this is a 12-layer model from those spikes. Wow. And you see there's a lot of information that leaks out of this. In the field of security, these are referred to as side channels or covert communication channels. There are all sorts of crazy ways that nation states and other hackers use these for steal information. But we can also use these as a way to monitor the workloads that are happening. And the idea here is that the amount of power that a GP given GPU or even a whole data center is drawing when it's doing a certain action, like a backwards path, is distinguishable enough from a forwards path that just by looking at the power, you can be like, okay, great, there's no backwards path happening here. Is that correct? It's kind of fingerprinting. Workflows have fingerprints. They have communication patterns you see on the networks. They have power patterns as well. And we might not be able to, as a whole data center level, tell this is definitely a backwards pass but there are fluctuations that are like canonical they're kind of

representative or are often a fingerprint of doing different workloads and another kind of easy example here is that if you go and read news reports about frontier data centers you'll read about all of these grid problems that happen where there are these really large swings as you do your synchronous training so they all do similar actions at the same time they all do a bunch of compute and then a bunch of stuff moves across the network while the compute becomes more idle and you see these big power swings um so much so that meta have published loads of papers on how to mitigate these power swings um so we might not be able to tell to the fine detail we have here because we're on a single gpu level but there are still fingerprints that we can use to understand what's going on and add this into a scheme like networked apps or use it as a preliminary tool in the early stages of some kind of coordination. Interesting, okay. So my takeaway from this is that there do in fact exist ways

if you know that there's a data center that's being used for some kind of, some AI activity, there are in fact ways to know from the outside what is going on in that data center without requiring super invasive access. Ideally, you're going to have to have some level of cooperation, but you probably don't need to let someone in and see absolutely everything. The question with access tends to be how easy is it to spoof it and how specific can your knowledge about what's happening be? From this, I think we're unlikely to know about the exact details of how fine-tuning is done or what data were held out during training we did or didn't hold out our dangerous biological data. But we can, for a start, say the data centre is on or off. we might be able to then say something more fine-grained, like this is consistent with an inference workload, this is consistent with synchronous workloads like pre-training. The precision is therefore one of the things we trade off

by not being inside the data center. We also trade off the adversarial robustness, where by being further away, we have now more opportunities. It's easier for them to achieve, yeah. Yeah, if one of the pre-training teams from one of the Frontier model companies spent a bunch of time trying to make training look like inference. As measured by something like power. Exactly. Would they be able to do it? I suspect so. They'll take some efficiency hit. Whether that efficiency hit is very small or very large is still up in the air. Interesting. And presumably there are some kinds of training that look a lot like inference anyway. Like if you're doing long horizon RL, doesn't that end up looking fairly similar to inference? Again, depending on how much level of access you have. Yeah, so as soon as you roll out and you're generating your inference tokens, obviously this is inference. We mitigate this in our network tap-free computation system because the model weights aren't updating, which is an important part of inference. But in this case, because we're further away, we have less resolution.

We can make a much less fine-grained claim. Cool, that makes sense. Okay, well, yeah, my overall kind of take on this is like, it does seem like there is a way given a data center that is doing activity there is a way to know things about what's going on in that data center this is a good thing this this would enable some level of coordination i feel like the other half of the puzzle that we would need to know is there aren't data centers somewhere that are doing things that you know that we don't even know about that are doing things like training um as far as i understand it's not something emoto is working on but should we be optimistic that these are solvable challenges too yeah this is something we're working on. I have a lot of uncertainty here and get persuaded either way. I mean, you talk to national security people about these things and one day they'll persuade you this is totally trivial because they're so good at spying. And the other day they'll say this is just like totally impossible. There is no way we can find a hidden data centre. I, like, a few things that help me think about this. Firstly, we will have all of the normal spying resources formerly called National Technical Means

we will have countries using satellites and human intelligence and signals intelligence spying on them It's truly easy to hack up an AI anyway apparently The easiest way for China to know whether US front of labs are racing is for them to have terrible security and for them to just spy on what's happening anyway Short of that, there will be some sets of signs of cheating on deals I expect the larger the project I imagine the easier it is to catch a project, could China hide a thousand dollar project that sounds easy could they hide a million dollar probably a billion, a trillion ten trillion, increasingly crazy numbers, seem increasingly difficult to catch I do think the intelligence community will be able to do a reasonable art of this and under scrambles under like quick races to put together a deal, a very large amount of our verification

and trust in other countries will come from the intelligence community. And just unilateral surveillance, basically. Yeah. Yeah. Exactly. And, you know, I'm sure there will still be networks. I expect the U.S. will call on the Five Eyes to contribute to their spying. But there are some mechanisms there. Interestingly, when you talk to national security people, even the ones who think it was so good at finding stuff on satellites, you guys just don't know how amazing we are at spying. Even they think it is valuable for people to be working on stuff in the open. OSINT, open source intelligence, there are projects, new projects using satellite imagery to detect covert data centers. And one of the reasons that they think this is valuable is because even though the intelligence community means something, knows something, sorry, that doesn't mean they can tell the world about it. And you see this in the nuclear setting in which intelligence communities

sometimes commission open source projects so that there can be some public information. So they commission them to do things that they actually already know. They already know it. They don't want to show that they know it. They can't show that they know it because they don't want to reveal their sources or methods or like show their hand. So it might be important that some of this happens separate to the IC so that there can be, like, international, multilateral, public, like, discourse about this information. The takeaway being that, sorry, even if our espionage is good enough, it would still be very valuable to have other means of verifying this stuff. Yeah, and I think this should be, this should encourage people who are interested in working on that to work on it, even though the intelligence community might do a reasonable job on it. Okay, interesting. So some of these technical problems may well be solvable, and we will have ways of verifying these kind of claims through espionage or maybe if we cooperate enough to install some of this hardware on each other's data centers or things like this. But it still feels like the field is incredibly small.

You know, Emoda, I think you've got 50, 60 employees. Only some of them are working on this anyway. Like what needs, and you know, the scales you're working at, they're very small. You know, this here is just one GPU, right? I'm pointing at the prop. And your prototype, like you said, was also a pretty small scale. like what needs to happen between now and uh a moment where we could actually use some of these prototypes using these technologies to verify a literal deal between the us and china so like a lot that needs to occur right yeah so i'll talk a bit about some of the work that needs to happen and then afterwards i'll talk about who could do it and who should do it um the technologies that are being developed at the moment i think are a kind of prototype pilot stage so that a few organizations working on this. I'll start a list and then inevitably forget someone. But Lucid Computing are working on some of these technologies including some similar things to what I've shown here. There is the Singapore AI Safety Hub who also have a network certifier system as do

the Future of Life Institute are working on some of these technologies. And finally a company called Attestable who are working on zero knowledge proofs for inference and and proof of work. These are all at the prototype pilot stage. I think within a couple of months, most of them certainly will be at the point where they can be red teamed, they can be tested. So before you get to a deal, you want all of the parties to trust it, understand it, and also for it to actually work. We don't do red teaming just to build trust. We also do it because we do want to work. And by red teaming, you mean figure out, you set it up and you figure out whether the prover could actually cheat can find a way to deceive whatever, you know, whatever technique you do. So in red teaming what we would do is we would set up a pilot installation and then we would get an adversarial team to come and try and break it. And this can be from either side. Either a prover who tries to cheat the system and do the illegal workload whilst making us think it's the legal one

or you might also want to red team from the other direction as the verifier and try and steal information you shouldn't be able to get. Steal the model weights or something like this. And both sides of those might be important for actual international installations. And we see historical precedent for this. In the nuclear setting, the scientists around the world, not just in the US and the Soviet Union, but in a bunch of middle powers, were developing tools to detect where the nuclear tests had happened. And this was important to underpin deals about test bans, so partial test ban treaty, for example. And these tools are things like seismology equipment for detecting tremors in the Earth. Before deals happened, before the US and Soviet Union agreed they trusted that this would work, they ran a joint verification experiment in which Soviet scientists went to the US, US scientists went to Soviet Union and ran tests on one another's infrastructure.

And I think we're going to need this game. They're running a test on, like, a literal atomic bomb being detonated and seeing whether their tools work in measuring the detonation. Yeah. So there are these great photos of them doing these exchanges and having dinners together on all sorts And I think we will need the same between countries ultimately ideally between the US and China I'd be excited to see the same between AI companies. The near-term version of this is that a bunch of the organizations working on this verification tech will be running exercises in which they install verification technology on pilot example data centers and then get red teams from the AI companies and from governments to come and try and break it. And support from governments and the AI companies is going to be needed to make that happen. Pretty indiable here. Yeah, exactly. And each of them have different knowledge bases that outside of their setting, we don't necessarily have. So governments have access to the national security ecosystem.

This is interesting for two reasons. One, they have cyber capabilities that are unmatched. And having their input on this is going to be really valuable. And secondly, they are already well versed in verification. So if we take the UK as an example, there are departments that deal with both chemical verification and nuclear verification for those respective weapons conventions. DSTL doing the chemical part, AW Blackness doing the nuclear part. So we have their national security capabilities that we want to learn from and also their verification experience, which is going to be really invaluable in verification and red teaming. And then in terms of the labs, they have the infrastructure knowledge. They have the knowledge on where the frontier is, what the algorithms look like, how you might cheat it, possibly. And what sorts of claims you'd want to make about what it is that you're doing. And that is invaluable in red teaming, testing these systems, and making sure they actually work. We want this to work.

We're not here to sell a product. This is all open source IP. We only develop it because we want it to work. And this means that collaborations with governments, labs, other groups who can tell us it doesn't work is completely necessary. Do you have much insight into how much the labs are doing on verification? Are they doing verification R&D right now? And if so, why are they not doing it? Especially if you're on Topic or OpenAI, I met many members of your race. Most senior staff have said that they want verification technology to exist. Yeah. Like, what's the state of how that's going right now? Probably if you're an outsider looking at the fraction that the senior staff and the junior staff describe verification as part of their theory of victory, you would expect 10 multi-hundred-person teams working on this very important thing. I suspect in reality we see much less than that, maybe single digits at best. The outside information is scant, but on the anthropic side, we see in their frontier safety roadmap, they talk about verifiable inference, developing a prototype by the 30th of September, 2026, that can prove which model was running.

And on the open air side, we see occasional job openings that include verification in the list of tasks in the RSI security engineer area. So I'm hopeful there are a few people coming on to this yet, but I'm pretty sure there is not a significant effort that is proportional to the amount that people talk about this being necessary and to the amount that this is part of people's theory of victory. Yeah, I mean, it feels like it's almost everyone's theory of victory, right? Like, the chance that AI goes blistering, like, a lot of people believe that there is a chance that AI progresses blisteringly fast. Whether or not this happens is unclear. And if that is possible, then it seems very difficult to deal with the situation safely at all, unless you can coordinate. So it feels like it's like a crucial element of most people's theory of victory. Unless maybe Dario, in his most, like, hawkish of moods, thinks maybe just winning is all that you need to do. but I don't know if this is even what he believes. I think the ball case, the strongman case against verification from the labs would be

we do need to slow down but it's just going to be in the US. It's going to be domestic. And in a domestic case you can rely on US law you can rely on people with guns coming in interviewing and that you don't need sophisticated tools on top of embedded evaluators, for example, and that perhaps you can slow China. Either they're not actually taking part in the race, which would be one argument, or... They beat China. They beat China. Or the Dario case in which he describes the need for export controls to make sure that China are a long way behind and to extend the lead, such that there can be a domestic slowdown without being worried about China. And that will be a case where the importance of verification tools is lower. I do still think the middle powers will want this. If you are the Netherlands or France or Germany or Singapore, et cetera, et cetera,

then you're looking around saying some people are developing this very powerful system that affects my citizens that I have almost no control over. In that case, you still might want verification. whether you actually have the levers to persuade the US to implement it is a different question. Yeah, yeah. And I know when we spoke about this like a couple of days ago, you also made the point to me that the lab itself may want verification if it is operating huge AI swarms doing AI R&D. If you've got, you know, all this compute that has basically been handed over to, you know, Claude 10 or something, and you want to be sure that Claude is only doing inference, which is what you've asked it to do, and not doing all these secret training runs, then also verification hardware would be very useful for this kind of purpose. Verification hardware ultimately lets you check that untrusted hardware is doing what you think it should be doing. And in the international verification setting, the untrusted hardware belongs to an adversary and belongs to the other side.

Our adversaries aren't necessarily AI Live employees, but actually AIs themselves, in which whilst at first Anthropic might not have nation-state level cyber capabilities, as the models improve, you might be able to model them as a nation-state level actor. And in this case, you might need a very similar stack of verification to verifying another country's data center, because it's not just the US government verifying Anthropic, they're also verifying Claude, and Claude is becoming increasingly powerful. and it's much more China-like in its capability. Because in this scenario, Claude is doing, I don't know, it's optimizing the way it's serving inference, perhaps, and he wants to actually be sure, oh, Claude is not spoofing us and secretly doing a training run. And in this case, he would also want something like this. Yeah, and even the AI companies trusting their own infrastructure. Yeah. So in the hugging face attack, Claude kind of escapes in some ways.

We don't think it managed to get access and modify the model weights. Or not Claude, but the persistent internal model, whatever it is. Yeah. In recent attacks, we've seen models escape their sandboxes. There aren't reported cases of them managing to get into the influence infrastructure itself, the GPUs, and change the weights. But you could envisage during a training scenario. Actually, one of the things that a model tries to do is change the weights themselves. And then this begins to look a bit like the influence recomputation demo that we were talking about. earlier in which you want a similar stack you want to be able to reliably tell that the weights on those gpus hasn't changed during an evaluation um uh claude or chat gpt hasn't taken it upon itself to modify its own weights so it's the capacity evaluation now it wants to okay exactly but i think we could like we really could see this as being an emergent property happening soon and whilst we were happy just trusting these software-only logs,

we might really want to work towards these much higher-trust hardware-backed logs and verification primitives. So the idea being this would be another reason why a lab would possibly be intensified to develop a hardware-backed link. Yeah, exactly, where basically there was kind of a security overlap with a verification overlap in which you start not seeing your hardware as something you perfectly control, but seeing your hardware as something that's a bit untrusted, your data center is untrusted, and therefore you yourself want to verify it rather than just an outside verifier wanting to verify it. Interesting. And especially if the AI is capable of tampering with logs or whatever, then if you put hardware, we're a really good fallback option here. Yeah, and this is where there's a bit of an overlap between both international deals and technologies we might want right now in trustworthy tamper-proof logging, recording of what's happening on a cluster for rerun even by embedded evaluators, for example, to understand what happened in an AI incident after the fact. Close to a black box in an airplane.

No. Okay, so just to recap again, we would want labs to get involved because they have the expertise in exactly how their infrastructure works, exactly how their training works, and what kind of claims they want to be making about their training. We would want government to get involved because of their expertise in cyber and in verification tools in general. And we would also want a certain amount of R&D funding to flooding into this space. Have you seen any movement from governments that are interested in promoting verification, whether that's middle powers or anything else? What's happening here? How quickly can things move? Yeah. So this year has been a huge shift in verification. We started the year by having to claim that we were interested in verification for normal market reasons. Maybe using a model in a medical setting means that you have to prove the model is correct for certification reasons. But as soon as Mythos came out, this changed. Governments suddenly said, pattern matched it to weapons of mass destruction and said verification is needed. So the most recent development on the government side is the Dutch Prime Minister at the UN putting down a research agenda on compute verification.

So this means that in the inboxes of all 193 member states will be this research agenda. and more and more countries are now becoming interested. I don't think we've seen any government funding on this yet, but I'm hopeful that this year we will begin to see R&D efforts funded by governments, but also involving government employees as well. Interesting. When I think about this, like, we're using hardware-based verification and ideally, I mean, you are shooting for this to be able to underpin a potential US-China deal on slowing down. down. And again, I think it's always worth reminding ourselves, maybe the progress of AI doesn't end up seeming so fast that we need this. That would be great. That would be great. But in the world where it's so astonishingly fast, maybe we have some incidents where the US and China actually want to come together and make a deal. It sounds to me, if we use the kind of tech that Emoto is developing, we would need to retrofit every single data center in China and every data

center in the US. It just seems to me like it would require a level of cooperation that's completely impossible. Are you actually imagining American data center operators allowing Chinese officials to come in and install network taps in their data center? I'm genuinely asking, is that what you're imagining you might have to do? This is one example. I think there are a range of invasivenesses, a range of robustness of verification tools. There are groups working on software-only solutions. It would be less invasive. Less invasive, no on-site, needed perhaps. Then there's a question for those on how adversarially robust to a cheater they are, all the way through to things like FlexHEGS, Flexible Hardware Enabled Guarantees, a compute verification scheme described a few years ago in which you have on-chip mechanisms changing your actual GPU design to include some piece of verification hardware. Well, if you can enforce what kinds of things are done on the GPU, for instance. Yeah, exactly.

And we're talking about something that's kind of in between. it's retrofitable, but it would have to be installed in a data centre. This has precedence. So in theory, you'd like to talk to experts in data centres around the world, but like, we have this for chemical weapons, right? There are actually people going to on-site industrial chemical weapons industrial chemical plants. Yeah, yeah. We would actually like to install a bunch of these technologies in government and privately held data centers around the world in this scenario. And as a reminder, I'm not talking about current frontier models. It's easy to reference to what's politically feasible now. We're talking about systems that are approaching and exceeding human intelligence in which we are worried about them taking over the world and ending humanity. This is the only scenario in which we're really thinking about literally doing this. It's a weird world. but yet still

we have some examples where there is international coordination without the risk that another species takes over and this is nuclear weapons and chemical weapons these continue to provide serious catastrophic risks to humanity so in the chemical weapons convention part of that is an inspection regime so this is run by the OPCW who are in The Hague in the Netherlands and they have a a group of scientists, of inspectors who come from all of the member states and they are able to do challenge inspections in facilities around the world. And in the standard inspection you give a country and a company 48 hours notice. You say I'm going to come to your chemical plant in China in 48 hours, expect to see me at Beijing airport and I expect to be supported to the facility. They turn up with all of their equipment and are taken to the facility where they do an inspection. So they inspect declared sites that are declared by the country to the OPCW.

And when they get there, they do a few things. One is kind of paper-based audit type verification in which they look at all of the receipts, understood what's purchased, understood the processes that are being undertaken there. And then they also do inspection using a bunch of chemistry equipment, something like a mass spec in which they are checking for detecting illegal chemicals that are prohibited under the chemical weapons convention and they can use this mass spec to test out a bunch of chemicals that are on site in these facilities and be like okay these are the these are the legit ones and none of this is prohibited yeah and we want something like this for AI yeah we want something for AI and the useful analogy here is the chemical industry does have a bunch of IP the exact makeup of your chemicals is really important IP to a bunch of corporations in the same way or a similar way that the model weights are IP to the AI companies. And therefore, to protect this, the OPCW actually have these confidential closed modes on their mass spec. So when you go and use it in a lab, obviously it reports you all of the information in your own chemistry lab.

But they have software on top of it that, rather than returning the full spectrum, just returns a pass-fail. And that's what we need in AI. We need systems that maintain confidentiality. They say, yes, they're following the agreed international deal, or no, they're not. But they don't necessarily give away the input and output tokens. They don't give away the confidential model weights. So I think we should take a bit of hope from this in terms of international trust. But there are still disanalogies. These are not the same technologies. This might be much faster. Previous deals have taken a long time to negotiate. It might be easier to cheat in this way, maybe because the surface area is much larger and more complex. Yeah, it could be. It's unclear. I think it's unclear. I think people dismiss too easily, too readily, without realising the actual great lengths that we've gone to before and still go to in nuclear, in chemical. And the same is true for the amount of effort that was put into developing these schemes.

So maybe only 50 engineers in the world working on AI verification right now on compute verification. But we look at nuclear and the US government ran Project Vela, this big ARPA program, in which they spend in today's money billions of dollars on developing seismology tools and satellites to dissect atmospheric nuclear tests and things like this. So it is true that this might be harder. There are disanalogies. but we should be putting in at least the same if not more effort. We wouldn't even know if it was harder right now because we're not putting the same amount of effort towards it. Exactly. And if there's only 50 people it would be quite easy to 10x. Yeah. I find this like both worrying and exciting. Yeah. And the worrying side is this is an extremely important technology in which many people's theory of victory root through verification. And right now whilst I think the teams working on this are great I think our team is great and many of the companies and non-profits working on this are talented. We do not necessarily have it handled. We don't have it handled to the level we should

for this important technology. So this should be an alarm bell. But at the same time, it is an exciting, ambitious research project. I look at something like Project Vela, these projects to make seismology devices, and think, wow, that must have been an amazing, intense R&D experience with all of this time and money to do something really ambitious. and the same exists in AI verification where I think this year and next year we will begin to see these really ambitious projects start and we need the founders and the technical talent, all the support we can get to rapidly scale the field and start doing ambitious work to solve this really hard problem. And presumably people within the labs could also pivot to working on this. They're just the best place people to do some of this stuff. Yeah, I think it seems likely to me that a marginal AI safety, like really talented AI safety researcher, could have a huge impact on verification. If it's true that the labs only have one or two people working on this, you could very quickly run a verification team

that had a really strong inside view on what was actually needed, both because you are quite good at assessing what claims should be verified, but also because you have knowledge of the infrastructure. And I think if you started on this today, by the end of the year, you'd probably be one of the best five people in the world working on verification or one of the most useful five people in the world. So people working at labs should really consider whether they should jump onto this inside or outside of labs. And this may be quite an open-ended question, but I want to get the bird's eye view of what are the things that most need to happen between now and a deal getting made? So we currently have prototypes, you know, and let's say that the deal focuses on things like being able to verify that only inference is happening. Ideally, we would want to start making more fine-grained claims, but I sort of agree with AI2040's vision that a pause on further training is a good place to start while you figure out other stuff. Let's say that's the thing we wanted to verify. How many problems still need to be solved? Ideally, you want a way to do this that doesn't involve handing over model weights

to the verifier as it currently works in the Emoto prototype. What else are things in big buckets that really need to happen? Yeah, so the model weight problem is solved by using some zero-knowledge proofs. Or will be solved. Yeah. We have early versions of this. Others also have versions of this implemented. A testable, for example, have fairly advanced versions of systems, version-knowledge proofs on inference. So I'm fairly sure that gets solved. I think there are a few forking directions here. So... What's most important is in the, you know, two to 20 days in which the panic deal is put together, that there are tools that are ready. A bunch of these will have to just be existing tools. So making sure that between now and this deal, the intelligence community have been collecting on... Where tools are going. Yeah, exactly.

Right now, it would be really great if the US and China started spying on where the computers. it would be really good if they started collecting information right now so that if they had to say where are all of the GPUs in China is there an illegal, like a hidden project happening How many are unaccounted for? So there are a bunch of things like this that would be really useful to happen now both in a closed setting and an open setting so projects to quantify compute basically things that feed into information that would be really useful in a scramble then there are things that are long lead time. These need to happen sooner rather than later. So if we do think a version of a deal needs network certifiers, we're working on some of these. Network certifiers are helping you understand what is being passed between networks and GPU. The version of a network certifier we're working on is an FPGA, a field programmable gate array, a kind of reconfigurable chip that you can plug into your network

and it will store a cryptographic log of the network traffic and then this is part of being able to implement these zero-knowledge proof demonstrations that a cluster is only serving inference and not training. Now, the problem is we might need a lot of these. We need to develop it. Developing FPGAs is very slow. You need to check they work. You need them to have been security tested, red-teamed, and then you need to manufacture prototypes and then scaled-up production. And then you also need to get everyone who would use them to trust them and not think of their sauce. So a bunch of this needs to happen. All of this should happen as soon as possible. I've also seen arguments that some of this should happen later. And what matters more is preparing for those 20 days. They're probably network certifiers. Maybe they could be implemented very quickly, but only if you've done loads of prerequisites work. So if it was next month that this happened, we would simply not be able to do it. We would not use any of these. We would use any of these hardware mechanisms. Zero, yeah.

I think we would probably just shut down the data centers. In this crazy world where something really insane happens. And then the other is probably sending in some kind of auditors or something like this. Like in person, yeah. Yeah. We might also do things like natural technical means, using infrared satellites to spy and see whether data centers turned on or not. if you have a bit more time it's possible we can have this whole sequence of prototype development, security testing, red teaming, scaled up production mutual trust, etc. in time if we do have time for that then doing that as soon as possible is good because the lead times on these are really long even if Claude is very good at FPGA design and the Chinese model is very good at verifying that the Claude one is is not um uh has not got any like back doors in it um you still have to like fabricate

them and build up all of the pc and that's a big supply chain which if you're using on every data center in the world eventually that's going to have to be absolutely massive presumably yeah exactly yeah um so doing this as soon as possible is really important because of those wrong lead times um but then at the same time um it is important for some groups to be focused on this scramble on the memo that lands on the president's desk it's like within three days these are the actions you can take that will give you some trust okay you should get x y and z turned down switched off this is how much compute China have this is where their frontier model currently is um these are the humans who are working on it this is where they're all located um etc yeah actually I mean this is not really what you work on but do you have much of an inside view on what you would want an early stage deal to look like? What are we agreeing to, do you think? Let me think about this. So, it depends how scary things are. Yeah. Shutting down seems really extreme.

But maybe it doesn't seem extreme. I say it seems extreme. But, like, OpenAI have now twice stopped training. So, globally agreeing that there shouldn't be training, doesn't seem inconceivable. Since we've seen it on a single domestic case, maybe we'll see it on a complete domestic case at some point, then maybe we'll see it internationally. So I think just like stopping training is the main thing that we might see. I think the other interesting thing we've seen is I've forgotten which AI company it was, which Chinese AI company, but they provided a dashboard of their latest URL. Oh, the URL, I saw this. I can't remember which company it was. That is total research transparency extremely early. And that's kind of... It would be great to see more of this. Again, this is just a level of trusting the documentation they give rather than having anything kind of like... It is, but... More of that would be great. More of this. Are we going to see Anthropic do this? Are we going to see MIO do this?

I hope we'll see something. What are the first unilateral verifiable claims you expect the last to be making? So I can imagine a world where it's pretty soon that Anthropic builds into their API a way of verifying to the customer that they are in fact serving Opus 5.5 and not the secretly shittier model to people who are accusing them of serving on Twitter or something. Are there other prosaic verification claims you'd expect in the market to cater to automatically? So, that one is definitely the first one. Although, we have seen unilateral but untrustworthy or not verified claim from Open Air already on Pussy Training. Which, I think if we were having this conversation at the start of the year would have seemed completely insane. So I use that to frame like whether I should think the future things that currently seem insane are actually insane. So we have already seen them talk about poison training. We now see Anthropic talking about being able to prove a specific model. I think we might want groups to be able to make claims about their security capabilities.

so if you were to ask the question what claims we want to make, a simple way of doing this would be to go through a responsible scaling policy and look at each of the claims they make, like we will reach this kind of security level this is what access lab employees will have and not only release will not have passed this king million threshold some of those are example prosaic but still relevant to catastrophic risk claims that we might want to be verified But actually most of those are kind of hard to operationalise in a way that can be easily verified They're often not that precise in a way that I can check with a fairly simply written programme. I can maybe check it as a human. I mean, with a lot of those, I'd imagine that implicitly we're relying on whistleblowers, the potential that a whistleblower would report them as the implied verification mechanism. Yeah, and whistleblowers add to each of these verification systems.

This is always on top of what we're doing in addition to the technical tools. Do you think much at Emoto about who your Chinese counterparts might be or how you would reach out to just engineers working in China? You're open-sourcing yourself already. I don't know if you know of anyone in China who's reading this stuff or is interesting? Yeah, so... Do you think about this? There are already track twos happening. Yeah. And those kinds of efforts are what's needed. And I think this is to mostly academics is the first landing point for these kinds of things. But I think it would be valuable to think about how to make it most legible to Chinese counterparts. But I really want to understand if I'm at the right level of optimism or pessimism here. like when you describe the prototype and you describe how this works, should I be thinking oh yeah, this is something which we are technically on track to achieve or is this still miles away? I don't have the kind of expertise to know this. What should I what should my position be? I think the current position is something like

it's tractable but not solved. There are small but competent teams working on this but we still need a moonshot R&D project to happen. This is in the critical path to many people's theory of victory and right now the people who work on this could completely change the field. A marginal person here is going to really accelerate progress. The thing that I'd be most excited about are significant lab efforts happening. The thing that would most help me sleep at night is if the major AI developers had dozen-person teams who are focused on making sure that verifiable deals can happen. I think if you are a safety researcher or security researcher in a lab, your most likely route to the Nobel Prize for making AI go well is that you are the one who develops the verification tools that enable a deal. There was someone who pioneered seismology equipment for nuclear deals.

There were people who made these satellites. The team who are going to actually make international deals on AI verifiable are not yet defined. and I think sometimes we should find this worrying. We should also find it exciting that you can have a real counterfactual impact on the outcome of AI by putting your effort into this. And what might have been so much that it's people at the labs that are doing it just because they understand everything about how the infrastructure works. It is important that stuff happens outside the labs as well and I'm excited about the work we're doing, excited about the work that other teams are doing. But inside the lab, you have a few advantages. You have access to the frontier. You know how fast the frontier of air is going. You can get insights on the algorithms and infrastructure that is used to serve the models and also information on the compute itself. The other thing is that the labs are a key stakeholder in international deals. I imagine a day in which the president phones up Sam Altman and says, Sam, I want a deal with China.

I don't trust them. Can you verify it? And Sam probably then phones up Chief Scientist Jakob. and Jakob phones up his reportees and says, did we ever do that work on verification? And we need to make sure they say yes and they understand it because AI lab employees and government employees are going to be the ones who are asked for their opinion when it comes to a deal. So they have the skill set, they have the knowledge, the labs are well capitalized, but also they are the very people who will get asked whether this works. So if they can be the most up-to-speed people on it, the ones developing it, then that's going to really help when it comes to coming to the table on international deals. Yeah, okay, that's very convincing. And the final thing I really want to wrap my head around here also is, like, how true is it that so many of these theories of the victory do route through verification mechanisms? You mentioned, okay, if export controls are comprehensive enough and expansive enough, maybe we don't even need to worry about trying to catch up. I mean, there's so many other things that might come into play, like, why is it that you're so convinced that without formal verification we're in a really tricky position?

Yeah, I don't think I am convinced. The world looks worse. But I do think that less trustworthy mechanisms will be useful. Some people kind of frame verification as needing these formally verified, zero-knowledge-proof, perfect systems that are perfectly China-proof. I don't think that's the reality. I think when it comes to international coordination, stuff does go on a bit of trust. It goes on some spying and some national technical means. You see this in previous deals in nuclear. Like what, for example? Yeah, so in early nuclear agreements, you have, I think in START 1, a clause that on request, you'll put your heavy bombers outside so they can be counted by satellite. You can cheat by doing things, for example, don't put your bombers outside. But this builds up trust. We have many layers to this system. It's not the verification tool on its own. So I don't think it's actually fair to say we are completely scuppered without this. I still think unilateral slowdowns might work.

I think person to person looking people in the eyes and saying, like, are you actually developing this? Plus buying, plus auditors. Plus just domestic unhappiness with AI might be enough to be like, you know, it might not even be that this fear of China is the real bottleneck. And if, you know, if the voters are mad enough about AI, I feel like that could also be a forcing function, whether or not we have the verifiable technology in place. Yeah, I think it depends where in the government holds power and how much the, like, ASI and the China Hawk and National Security Center of Power pull the lever versus the, like, political job loss, caring about what... Yeah, like, constituency arguments hold power. And this is just unclear. Yeah, I think this is unclear. I think in this setting, it's likely the national security thing kind of wins out. Especially if AI is as impressive and as fundamental. Yeah, and, like, at the same time,

presumably this camp is, like, spending a bunch of effort trying to persuade the public about this as well. And there's a bunch of fear-mongering from all sides. I think it looks very weird and very uncertain. I am so not sure that verification is actually 100% needed. I do think people should think about it as optionality and optionality for a deal. It just increasingly looks like we're in the world where you need it. Whereas a year ago, no one was calling for this. We have incrementally seen junior lab employees, lab leaders, governments call for this. And we might be in the world where we win because there are warning shots and we want to slow down. And in that world, we might need verification technology. Perhaps we get away with it without it, but I would be much more happy if it did exist. And we should see it as optionality, but we should also be very worried if very few people are working on it, which I really want it to exist and be as trustworthy as possible. I also think that we should care about a bunch of the similar tech stat

for post-ASI or power of OGI reasons and diffusion of power. The verification tech stack. Yeah. Especially the cryptographic primitives and zero-natured proofs around inference, or inference that involves flea homomorphic encryption, or even things like confidential computing, where you can, like, serve the inference without necessarily giving the person with the model full access to the input-output tokens. Those seem really useful for making sure having this, like, new powerful intelligence in the world benefits all of humanity rather than causing... Oh, yeah, why is that? So we have intelligence equal to ourselves. This is doing lots of economic... making lots of economic progress. If the only option is to route all of the input output tokens through Anthropic or OpenAI, that does seem bad, right? I don't think... I don't want them to have all of the world's IP going through their data centers.

I would really like it in that scenario if we had cryptographic primitives in which we could have stronger safeguards over IP, whether it's my personal healthcare data or whether it's these chemicals being developed by a company. How is that related to the verification tech? Well, I think this is quite similar to some of the technology around knowledge proofs and also discussion around trust execution environment and confidential computing. Because what you're building is a way for me to give my data because I'm querying Claude and I can be very confident that there's no way Anthropics sees that data and my confidence we rely on some of the same technological primitives and another thing we've not really talked about is that in addition to training, I think we go data centers are turned off then we say we're serving inference and not training I think the next thing or somewhere in this, maybe even before is I promise I'm running the safety classifier.

So in addition to Anthropic providing a fingerprint of the virus by their endpoint, I imagine domestic and possibly international regulators saying, and please prove that you are running this classifier. And that looks similar to the zero-knowledge techniques that might be used to fingerprint models. Can you give me a vague impression of how you would verify that they're running a particular classifier? What is it? I think it's a model. Yeah. It shows similar zero-knowledge proof and that when the US government export-controlled Fable and said your classifier isn't good enough, you can imagine them saying, please use this new classifier and just prove to us you haven't turned it off because we're worried that you're going to slip back to the old one because it keeps a run or like few of the things get false positive flags.

That's really a classifier. And presumably for that zero noise proof you would need... What actual level of invasiveness? You don't know what level of access the US government in this case would require. Nothing. Nothing. Yeah, because it's a cryptographic proof. It is what it is. You don't need access. It doesn't mean that all of the it gives you this correctness claim not this completeness claim. So they could also be running like illegal fables somewhere if you don't know about it without the classifier. But if you were to say randomly query the API and be given like the US government you were trying to audit Anthropic and what you want to do is make sure the classifier is still running and they have this zero knowledge proof on the classifier then you could send off a thousand requests a day and keep checking and keep checking does it still have the classifier proof on it. Okay, yeah, that makes sense. So, point being here, you want verification for concentration of power, for normal, like, you know,

more mundane safeguard reasons, and you want it for a deal. And in all these worlds, it's like a super useful technology to have ended up building. The question then does come up of whether these are similar tech trees for international deals or for prosaic verification. I think in some cases, something like the zero knowledge proofs do overlap the software solutions. But you do have to be careful pretending that your international deal verification is the same as domestic verification. So, for example, in the pilot I've described, we talk about ripping out a bunch of the back-end network. It would be crazy for one of the labs right now to rip out a bunch of their back-end network to make a prosaic domestic argument about what they're doing with their review. Because that would be such a costly way to do something. Extremely costly. Doesn't make a lot of sense. Ripping out the back-end network is not going to be... Yeah, make it much less efficient. It's inference, perhaps, without reconfiguration of the cluster make it, like, perhaps totally useless for doing any pre-training. Now they've got a cluster that can only do

one thing. That's very annoying. They don't want that. And you would only do that in a more adversarial setting. Yeah. Let's talk a bit about hardware R&D in general. So at Emodo, you don't just work on verification. You've also built all sorts of, like, other cool things. And I'm quite interested because this gives you like a privileged insight to how good the AI models are at working with hardware. There's like a good blog post that I remember you wrote back in July where you talk about designing a wake-up light basically with a one-shot Fable prompt where I think previous models had failed and Fable could just immediately design the circuit board and basically everything you would need for the light and then you could send it off and actually have that light delivered to you. I think you called this that vibe cadding. which for our audience CAD is the computer-aided designs of software, the healthy design of physical hardware. I'm curious, are there any engineers at Omodo right now who you would describe as vibe-cadding in their day-to-day job?

We actually vibe-cad very little of our work right now. So we've got a team of 60 mechanical electronic firmware software engineers. And as you said, it's not just verification hardware we're working on. We work with ARIA, the UK's Advanced Research and Invention Agency, to make brain implants and brain imaging systems, laser systems for imaging crystals and clouds, all sorts of fairly advanced hardware. The Frontier models can contribute in little chunks right now. And Fable was really the first model that could contribute at all. I think it's kind of like GPT 3.5, GPT 4 in terms of its software capabilities. It can write little scripts with some help, but an engineer will look at it and be like that is a very ugly design um but i i do expect this to you know even between fable 5 and 5.1 and opus 5.5 and 6 astra um we've already seen improvements in

both the like electronics taste that they have but also then the mechanical design capabilities and I expect this to really change. I think it will change our work, but I think it will also change the way that people interact with physical products. So it's currently not changing the way your engineers are doing that job, your hardware engineers are doing their job all that much. And this is because we've run experiments in which we rerun old projects. I've actually got a couple of boards here. Oh yeah, let's see them. We're looking at some boards that Tom has brought in. So if you have these other two over here, these are two small circuit boards, one of which was designed by one of our engineers a few years ago to monitor the power being consumed by a GPU similar to the demo we've got behind us but rather than using an inductive monitor we are actually measuring the current across a resistor and this is something that takes an engineer 10 to 30 hours to design and this is then the Fable one-shot equivalent and right now this is extremely ugly

the double version is quite ugly and the only thing this has to do is what measure the voltage it's got an ADC and analog to digital converter and a microcontroller essentially a small computer and it is monitoring the voltage current across this resistor and it's using that to calculate power and then it splits it out of USB or USB-C cable here so there are what four or five integrated circuits on this board. It has to do a few things in doing this design. It has to choose the parts to come up with an overall architecture. It then has to create the footprint for those parts. So each of these components, you can see they have lots of these legs. We're looking at small components on the circuit board right now. Yeah, exactly. So you have an integrated circuit. It has lots of pins on it, we need to first, in our CAD software, draw the shape of this circuit board. Sorry, of this component. We call this a footprint.

So what the AI does is it looks at the data sheet of that component and it extracts the information. There'll be a drawing in there. About how large each component is. Yeah. So we have to come up with a schematic first. So this is how all the components are going to connect together. And then we draw what the components look like. we put them on a PCB and we call this layout and then we route between them as we connect basically a dot to dot of the pins that need to be connected based on the schematic. And there's like okay at some of these processes and quite bad at others. So right now they can do fairly good high level schematic design. The schematics are okay. When they're doing the layout, they seem to make quite poor choices. They put things in weird places that then make future steps really hard and they are tragically bad at rooting. They're really bad at doing this dot-to-dot between parts. Between different subcomponents. Yeah, exactly. And actually, if you look at this one, it makes all sorts of weird choices that a human would never make. Like, none of these are...

They're all really ugly, curved lines. You would just never do this. I don't have the taste of this. But to be clear, this circuit board, which is supposed to measure... What is it, power or current? Or am I... It matters things that then contribute to power. It can measure things that contribute to power. But it does work. It does work. The thing that Fable designed, it actually does work at the thing that it's supposed to do. So I think it's just like with early software, where you have software engineers saying, oh, yes, it works, but if I had to debug that, that would be impossible. Interesting. And something like this, it's going to be really hard to rework. And if an engineer had to go into the final design and make modifications, it's a bit of a rat's nest of things. Whereas this is not true of the version of this circuit that your engineer had designed. Reviewing that can be quite simple. If you wanted to cut traces and make modifications, it would be much more simple. and to debug. This has made some weird choices and done some weird choices. And the one that your engineer designed a few years ago, that took 10 to 30 hours. And now a crappy version of that is replicable in Fable with one friend. Yeah, so I think the hardware time horizon of Fable 5 was

something between 10 and 100 hours for... Interesting. At some quality trade-off. At some quality trade-off, but consistently getting things that worked in a way that I think will really change how we interact with the physical world. So this is my wake light example. We're looking at another circuit board that was used in the wake light. Yeah. You might know Lumi wake up lamps, like sunrise lamps. They are not bright enough to wake me from my deep sleep. So what I wanted was a really powerful video light, like a 60 watt LED panel, like the one you might see in this podcast studio. But I wanted to slowly ramp this up. So I said to Fable, where I've bought this LED light from Neewer, and it has a DMX connector on this. It's an industry lighting standard that allows you to control the light. What I need you to do is make me a PCB that controls that. Where a PCB is? It's printed circuit board, a circuit board that can control that DMX connector,

and it talks a communication protocol called RS485. And it worded away over the course of something like 10 hours. it designed this PCB that I Which software is it using? So this What's it outputting? Yes, this is running in code code and it is then using a piece of software called KiCAD, which is an electronic CAD software and it outputs a file called a Gerber so what it outputs is essentially the copper of every layer of this board and if you look at it, you'll basically see a trace of how you want the copper to be This is a two-layer PCB. So there's two copper layers and some other layers in between. And then it provides a file that then gets sent to a PCB fabricator, in this case, a PCB fabricator in China, who makes the board and then also puts all the parts on. So there's also a file that's got a file. So you can ask Table to create you this Wakelight or this board

that you can put into the Wakelight that will meet these specifications. It'll do it. You can send the file directly to this manufacturer in China, and it'll come back and it'll work. I think the really interesting thing about this is we're now at the point with software where I can have pieces of software that work exactly how I like. If I want a markdown editor that works exactly how I want it to, then I can have that. If I want a website blocker on my computer that doesn't work the way that any other website blocker on the internet I can find works, then I can have that. I think we're moving to a point where we can also have hardware that works just the way we like it. So this Wake Up Lite, it has a customer base of one. it works exactly how I want it I have this really bright 10,000 lumen light in the morning and I think we'll move towards having hardware that is kind of customised in this way and this is going to end up with a really different hardware kind of physical experience for all of us in personal lives and also in science and in business and all sorts. That's interesting going back to how Emoto uses the models today like at what point in time do you expect that

per engineer you'll be spending the same amount on tokens as like a SaaS company spends per software engineer on tokens today. Is it like a rough guess? Yeah, I would guess this happens next year. Wow. Do you know what the current amount is? Is the current amount very low? Do you know what the current amount is for your hardware engineers? It's like quite low. Few of our engineers saturate the most premium subscriptions from Anthropic or OpenAI. How much this changes depends on A, how we adapt our workflows given AI systems. and secondly on how much the labs choose to hill climb on some of these problems when I look at some of these they really seem RL friendly something like routing a PCB really seems like it probably has a verifiable reward you could come up with an RL environment to hill climb against and if they do this then I could imagine the way we work would change quite significantly it's funny you mention this because one thing I thought is

it does strike me that hardware design companies like Emodo, you guys could just start designing RL environments to sell directly to the labs presumably, like you're working with these models every day and you're, you know, it would probably be not that much effort, or I mean it would be some amount of effort, but would that be something you would ever do? Do you expect other manufacturing companies to start doing this? I think it depends why you exist, we do not exist to optimise profit if we were doing this, I think maybe many of us would just go work at the labs or a quant or something. But rather, we exist because we think there's a bunch of tech that needs to be developed to make humanity go well. Therefore, our engineering team is focused on making that technology. If for some reason working on our own environments is in the critical path to that, then maybe. Otherwise, no need to. We're not a weisty company looking for some beachhead market. Rather, we are an engineering team making technologies and biosecurity and AI verification and accelerating science where a bunch of this differential

technology development we think has to happen very quickly if there's a distraction from that then we would have to do it yeah that makes sense what about giving models actuators so presumably at some point you can automate more and more of the things that you create like what actuators do you give the models currently and like which ones might you expect to give them like physical actuators in the real world in the near term so we have Claw currently has access to 11 3D printers that it controls and it can put prints on exactly it can these are FDM OSX 3D printers it has control of 11 of them it can print and it can view the camera but apart from that it doesn't have much actuation the direction that hardware R&D and prototyping goes in is slightly unclear to me if you think about software Claw is really good at software engineering at least in part because it can check whether it worked. It can compile the software and see whether it can run. It can come up with the software tests that it can run against.

Probably about a year and a half ago is when we suddenly saw this proliferation of the AIs writing loads and loads and loads of tests and that improving how well they could write software. The question then becomes, in hardware, what is the equivalent of compilation and what are the equivalent of tests? And there's two directions here. One is that you necessarily need to make something. You need to prototype it on a printer or on a CNC, computer, numerically controlled lathe or milling machine to actually check whether the part works. Because you need to produce it in the physical world and then experiment. Yeah, and people doing it, robotic actuators experimenting with it, whether it's humanoids or specialist robots. An alternative direction is that actually simulation is good enough that you need to make things much more infrequently. So if you had really good finite element analysis and you could simulate the mechanical properties of a part, or you have good enough simulators of circuit boards, then perhaps that is the verifiable checkpoint, the compilation equivalent.

and then rather than having to have this really fast prototyping lab, you can actually do this in software. Wow. And that would be really relevant because as I understand it, the time horizon of models on software and tasks where there is a unit test that you can run to see if you're making progress. I mean, time horizon on tasks like that is like way longer. So you would expect the models to rapidly get better if they have access to simulations that they can run. I think ultimately you do need to make sense of things. So we're betting on the need for rapid turnaround prototyping. the nice example of where rapid turnaround prototyping accelerates fields is Formula 1 Formula 1 is one example of engineering where the UK is actually still at the forefront and if you speak to an F1 engineer they talk about a wing breaking in Monaco in practice on a Friday and overnight on Friday they've made a new wing and on Saturday morning someone is flying out to Monaco with it in the seat next to them to be installed and they able to do this because they have this really concentrated and specialized supply chain around F1 both in manufacturing and a bunch of small machine shops in the Motorsports Valley

that can make these parts. And our ambition in rapid prototyping Emodo is that we can have the same response type on making hardware for science to excel in its science breakthroughs, but also making hardware for AI verification, for biosecurity, roughly for differential technology development, for developing DEFAC, defensive technologies, that we need in this rapid AI transition. So you're trying to prepare your company to be able to rapidly iterate on prototypes that the AI is suggesting to you as quickly as possible. Yeah, I mean, the AI, I think there'll still be... There'll be a lot of human design work still. I think it's a very good bet, even without the AI's being good at hardware, because we are often bottlenecked by prototyping. If we look at a serial-time, like, 10-week engineering project, we'll often be waiting five weeks for components to arrive, whether this is machined mechanical components or printed circuit boards. And having that really fast feedback loop is what's needed to run really quickly at these hard engineering projects that we're working on.

And what about, as you climb further and further up, you're automating more and more of the work. How do you actually think about the risks there? So, you know, I'm sure you've seen Anthropic have a new wet lab where Claude can, as far as I understand, basically run experiments. And currently, humans do all the physical lab work there. But how do you think about managing the risks of what is happening? Okay, you're not running a wet lab. Perhaps the risk factors are different. But how do you think about how you manage the risk if you automate more and more manufacturing? How do you know the models aren't doing naughty things there? Maybe you could say it's like in automated wet lab. I think I've roughly not thought about this. Okay, no, no. I think it looks at first like similar safeguards to cybersecurity and like classifiers running on top and these normal things. There may be specific choke points that you care about. So in biology, it's often talked about DNA synthesis screening because there is a nice bottleneck, like a choke point that you can use to monitor what you can access.

And what use activity involves that? And kind of concentrated, there's an intervention point you can get to. I'm not quite sure what the intervention point is in hardware design, at least in part because we have very broad diffusion of the tools you need to do harmless things. so this is weapons development or something else. We have 3D printers come everywhere. Are we, in the US, there have already been, perhaps even enacted laws to do with screening 3D prints that they're not 3D printing components of guns. And this is really hard to do because 3D printers are very abundant. You can very easily hack them and control them yourself. yeah um so how would that be installed at the firmware level of the 3d printer or you're not sure yeah so you i i don't think i endorse this yeah uh i think this is too surveillance state

and isn't the right trade-off i think the right trade-off might exist in biology because of the ability for a single actor to do something so catastrophic um but i think this would is not necessarily proportional to the risk in a bunch of hardware design. The easiest place to do this is at the slicer. So when you go to 3D print something, you have a 3D model, a 3D file that you give to a piece of software that basically plans out how the printer is going to make it. At that stage, you still have a fairly nice high level of abstraction of what the part is. but like with synthesis screening you can just you can chunk bits you can make it unrecognizable um it doesn't like almost impossible to enforce uh something like this no no another thing i want to get your take on is you want to you want an r&d firm uh and it seems clear to me that uh there's big big waves coming for the world of r&d uh maybe there's two bit two big things to single out one

as we've just been talking about, capabilities in AI will make what's possible, will vastly improve the space of what's possible and how fast it's possible. And then two, there's billions of dollars of philanthropic funding flooding into selective R&D spaces. And I feel like this could change the world of research and development pretty dramatically. What do you think actually needs to happen to the funding landscape, the institutional landscape, for us to make the most of this? Yeah. So one way I like to describe Emodo is we have the fit-for-purpose engineering R&D lab for this AI era. And the reference class I use, so in order to turn capital into scientific or engineering progress, you need someone or something to do the work. So DARPA didn't have any lab benches. They have their performers, their primes, these engineering companies and academic groups that do work for them.

So when we look at this new wave of philanthropy, there needs to be groups who can do this research. And one way to describe the motto is the fit for purpose prime for these catastrophic risks. What are some of the historic primes that you think were a good example? Yeah, historic primes would be Skunk Works. They would be the canonical prime for turning US government capital into new types of jet plane. And there's a company called BBN, Bolt, Baranek and Newman. They were the prime, the contractor who did the ARPANET contract, the thing that kind of led to the Internet. And they did this for ARPA at the time. and then we are the fit for purpose prime for biosecurity, for AI verification for emerging risks. And what makes you fit for purpose rather than just existing legacy companies? Yeah, so one is the fact we take the risks seriously and are embedded and knowledgeable about them and have team members who understand the landscape

of AI, understand the landscape of catastrophic risks and if you were to go and interact with one of the normal engineering consultancies in the UK is like much beyond their remit and knowledge. The other is that we have set up a hardware team that has a track record of delivering on like speculative hard R&D projects. We've made brain implants. We've made brain imaging systems. We've made high-speed communication systems for quantum networking devices, all sorts of things like this, and have focused not on producing like PowerPoints for our clients, but on producing products. So in about four years, we've delivered 100 hardware projects and have this well-oiled machine that can turn capital into physical sciences, R&D progress. Amazing engineers, a well-run workshop. And we think that is going to be one of the crucial elements to turn this new philanthropic capital into impact in the real world. Well, what about government funding as well?

So it's like, you know, government is looking to fund and steer scientific progress in one way or another. and the speed of scientific progress is potentially accelerating massively, like how should governments be thinking about how they're allocating, you know, capital, like the MHRA or, you know, like all these different institutions? Yeah, yeah. Does it be... UKRI in the UK gives out on the order of £10 billion a year. I think it's going to be extremely hard for them to adapt because I think the pace is going to be so high. And people are used to proposing one, two, five-year research programs in which they talk about deliverables that I think will go from taking years to taking months or weeks. And this means you're going to need to change either the way that you give out your funding. So rather than being goals that take five years, goals that are much shorter, and then iterating on decision making. So every two weeks they come back to you and you re-evaluate how they should spend their money.

or giving large block grants in which they have a lot more freedom over those time periods. But the main struggle, I think, is just going to be the pace of change and whether they can really adapt to it. You can't have UKRI setting out a 20-year research time horizon and these elaborate, long research plans if these are going to be torn up in a few months. The other interesting aspect here is government moving from the largest kind of public good funder to philanthropy competing on the same scale. Now, they won't compete on the same scale as the US government. They give out enormous amounts. But if you were to look at the UK, UKRI giving out £10 billion a year, it seems feasible that philanthropy, these AI lab philanthropists, the Anthropoc Institute, Anthropoc co-founders. The Urban AI Foundation, Coefficient Giving, all of these groups could be giving out tens of millions of dollars a year.

Sorry, tens of billions of dollars a year, where maybe single digit billions could be spent on R&D in the UK. And suddenly, they're like actually a considerable ballpark of our whole research ecosystem. and it's a big question for how research groups here in the UK can adapt to absorb this capital, to adapt to the types of work that these philanthropists want them to work on, but can also come up with working patterns that are much faster and conducive with powerful AI systems. And that basically involves figuring out ways to prototype things quickly figuring out, I mean having people who know how to use the models Yeah, know how to use the models, change their scientific workflow So it might be that the way you start adopting AI in science and in physical sciences, R&D and hardware R&D, is you just give them your normal human tasks. But it might be with this kind of different brain type, you should be following a different process. Maybe you run much more stuff in parallel because the cost of parallelism has suddenly gone down.

So rather than doing lots of serial time experiments, you should do all of your experiments in parallel. and you have lots of ways like this. So I think you'll also, a long way of just being good at using the models is can you change your R&D workflows to match the differential advantages that the AIs have? Perhaps a slightly ill-formed question, but I feel like if you're someone who's running a company, you might have a take on this. Have you seen increasing levels of automation change the culture of your company at all? What individual engineers are able to do is more depending on each other less because they're collaborating with the models more? Do you feel like you have a sense for how the culture of Emoto's been changed in the past even just year? Yeah. I think the culture hasn't shifted very much, but there are examples of the way that working has kind of changed. The majority of our projects are basically the same, but we're now at a point where small electronics projects, they're not asking for a firmware engineer anymore.

Like, oh, I can probably take that task on myself. Or a mechanical engineer. From where as you're designing software for a specific hardware object. Another example would be recently we had a mechanical engineer who's working on a project to do with sizing particles in the atmosphere, making a particle sizer that can go on a drone to measure particles as part of one of RA's programs. Particles in the air. Yeah, and he needed a differential pressure sensor. He needed to take this off-the-shelf component, put it on a PCB, put it on a circuit board, connect it to a computer. Normally, a mechanical engineer would go to one of our electronic engineers for this. It was a very simple design. It would only have taken a couple of hours for an electronic engineer to do. But instead of doing that, they worked with Fable to design the PCB and were suddenly able to do it themselves. so I think the main change is going to be that engineers are able to take on

many more disciplines that aren't their own like for a long time or some reasonable time they'll still lead in their own discipline but increasingly say oh actually I don't need to go to the resourcing meeting, find an electronic engineer who can be resourced onto this to give me advice on how to set up these power supplies rather they are empowered by the models to do it themselves I think this means many more teams of engineers who are able to deliver much more ambitious projects much more quickly because they suddenly have extra skill sets that they didn't have before. And what about making sure Omodo remains at the frontier of using the models as much as you're able to? Is the main thing that you have basically just, I want to hire people who will use the models as much as they can and that's the only way I can? Or do you do top-down stuff actively after having hired people too? So we have a... This is really hard to think about because I find the capabilities of the models like viscerally scary, not just in terms of how good they are.

Not just like the traditional risks talked about on this podcast and the bio and AI cyber and things like this. But you interact with them and you can't help but think, wow, my job is going to change so much. This is like very destabilizing and strange and scary. um the main thing that we've done to get people to understand where capabilities are is by running trackathons so we when fable 5 came out we prompted it on a few harder design tasks and were in awe or in like complete shock uh i had pure clawed psychosis for like three days which all i would do was prompt them on the hardware what kind of stuff are you making I was making like this PCB. I was making like fidget toys, all sorts. And unfortunately, the US government export controlled it. So my psychosis ended and I was able to go back to doing normal work. But you don't really grok how good the models are until they suddenly are really good at something that you're good at

or you're adjacent to being good at. So what we've done on this is held like 10 hackathons with six of the team at a time for a whole day in which we just like rerun old projects. who get an intuition for where they're good and where they're bad. And it does lead to a weird atmosphere. The models are really good in some places and really bad in others, but it is a strange experience seeing something that you love doing and a really skilled at suddenly be done by a computer. So I'm both excited about the capabilities that we'll have, but also worried and scared about what the future looks like for a company and for work. And this is underrated by many people. Is your mainline expectation that there'll be companies that are good at adopting this technology into their workflows and they will just dramatically out-compete the companies that won't? Is this kind of how you see things going? Is that the wrong way to think about it? I think that is one direction. I think there's already stuck in the system. So there's already a lot of waste in the system.

We already out-competed many of our competitors and exist because, in many cases, we're able to accelerate scientists by tenfold. So what's another fivefold that is caused by the market? How do you measure a tenfold acceleration of scientists? This is something we do. So with ARIA, an ARIA creator, what they call their grantees, has to submit milestones that they're going to hit. so what this is really great for us because we um then go and help them on one of their projects and we can see whether their milestones were delivered early in a bunch of cases we've delivered milestones like 12 months early on a two three year r&d project um and in some cases delivered things that they think would have taken like 18 months in the course of like a few weeks so we're already like delivering very quickly but haven't come to like dominate all of science and this is because of like markets not being like perfectly efficient all of these other dynamics

so I do think that companies might be able to move much much more quickly what this means for how your business models evolve is then an interesting question in hardware so we are originally set up as a consultancy, we are paid by other people to do hardware r&d work for them increasingly we're describing ourselves as a hardware r&d lab or a physical sciences r&d lab now not least because you know consultancy is not not cool yeah um uh but because as we speed up at doing the engineering work in the middle we want to take on more of the upstream and downstream parts so when we think about our like high impact work around AI risk. This means taking more responsibility for identifying the projects we should work on, more time doing a kind of course prioritization and thinking about the inputs, and then also more time thinking about the spin-outs and how things get deployed. So we don't do engineering work for

the sake of doing engineering work. We do engineering work because we want to see good in the world, and we can't necessarily rely on doing some engineering work and then putting the product on a shelf publishing a nice GitHub repo with some open source designs. You want it to go where it needs to be. Exactly. So increasingly, we will be owning much more of the output. So this doesn't necessarily mean for-profit spin-outs. It could be for-profit spin-outs. It could be non-profits. It could be open sourcing designs. We've partnered with others for them to deploy it. And the move to being much faster at the middle step will mean we can do more of that and also take on more of the pie, more of this. like impact pipeline taking back to personalised hardware what piece of personalised hardware are you most excited to build? Do you have anything that you daydream about? This was one that I had for a long time the wake up light I had thought about making this for years and I had just not had the skill set I really couldn't have done that and now I have

a physical product that's my own I I recently made a distance tracker for cycling. So when I go cycling with my partner, one of us will inevitably drop the other. And rather than looking behind and always seeing whether they're still on your wheel or you've dropped them, I made a system that uses an ultra-wideband radio. It sits on your handlebars and it shows how far behind they are. What is it looking at, sorry? So it has a radio on each handlebar. Okay, yeah. and ultra wideband radios for some physics and electronics that I don't understand can tell distance. So one sits on my handlebars, one sits on her handlebars, and on a little screen it shows how far behind she is or how far ahead she is. And then I can tell, oh, no, I'm cycling too fast. I've dropped her. And that was designed with Fable? And it was designed with Fable. And there are all sorts of weird products like this

that I'm excited about. And the existence of these tools, I think, causes a really significant return to people who have many ideas who didn't necessarily have the execution skills. It's the golden age of the ideas guy, right? Yeah, exactly. I feel, like, massively uplifted. But I mostly feel massively uplifted because I was an incompetent engineer. I mean, most of our team don't feel very uplifted, but that's because they were already very good. So it's not that helpful. Yeah. I'm interested in how you feel running an engineering company in the UK right now. How do you feel about the UK's prospects going into this world where AI can do more and more of all the productive work that the humans are doing? I think at least for the last five years, being in the UK has been a significant advantage to us rather than a disadvantage. And being in Sheffield in the north of England has also been a weird advantage. and the main weird advantage is it's not been noisy so if we were in London if we were in San

Francisco we would have been dragged into a local minima we mean like dragged into the current venture capitalist hype cycle whereas there's a bootstrapped company in Sheffield we've been able to like build an engine that does amazing hardware work and is not distracted by the newest Chachi BT wrapper and this means we've got an amazing team of 60 who deliver hardware work I think as well as any team in the Baywood. And we've been able to work on weird things. If we were a venture-backed San Francisco startup, it's unlikely that we would have, at the start of the year, been able to start working on AI verification under the premise that maybe you need an international deal. Like the VCs, they were not into that. So I think being here has given us access to underpriced talent, really good engineers who the rest of the world has not noticed. I also think that they that our team have honed lots of skills that don't get honed

if you live in the bay it's like honest good delivery that always works the I think the place where this changes is if Frontier AI becomes really important to our delivery pipeline and the UK doesn't get access anymore that's the existential essentially yeah So we already saw this. I had my three days of Claude Psychosis where I was using the AI systems to do all of this electronics design and mechanical design. And then I it was a Saturday morning. I woke up. I said to Fable 5, keep going. It returned an error. I said, keep going again. It returned an error. I went on Twitter. Fable access had been cut off. and suddenly I had lost the skill set. I had the previous day been able to do some electronic design and then I had reverted and I no longer had this skill. And imagine building a company on that. You're building a company on having access to this intelligence and it gets taken away just like that.

That is catastrophic. And my worry would be that as Frontier AI becomes more and more capable, the first thing will be we will have a compute crunch. So the US will have as many data centers as it can. It will be serving as many tokens as it can, but there will be way more demand for tokens than there are tokens that can be served. And at this point, what you start doing is you start prioritizing domestic supply. So instead of serving a motor with tokens, you serve your Boston, San Francisco equivalent with tokens because you want to develop your own economy with the amazing hardware progress that can be enabled by the AIs. And what this means is that we begin to lose access to AI systems. Now, what can the UK do about this? Well, of course, there's already a special relationship. There are already reasonable ties. This is hopeful. But I think the risk is that the US are unwilling to export model weights outside of the US unless you create very strong security claims.

So the export of the weights is a marginal cost unless they get stolen, in which case that's catastrophic. So unless the UK has sovereign compute that is at high enough security level, I'm increasingly worried that we will no longer have access to these frontier models and that this could hold back UK science, UK R&D, UK companies, the quality of living, ultimately as it gets implemented as part of the healthcare system, etc. So building ultra secure compute in the UK would be like an absolute priority? Yeah, I think the case against this is whether or not you actually need the frontier of capabilities. Like, are our engineers fantastic? Yes. Are they literally the best people in the world at electronic design? Like, I don't know, maybe, but possibly not. Was it okay that we had, you know, person number 10 or person number 100 or person number 1,000? They were still, like, amazing at doing it.

And likewise, I think we probably see the same with AIs. where they saturate at a certain point and some tasks don't benefit from increased intelligence. There's not just an infinite dose response curve. Exactly. So this is where I can envisage there being much less of a problem for companies no longer having access to the frontier. Like some problems, they're just sonic problems. Some problems are fable problems. But not all work is a fable problem. you can still I think it would still be really bad for your country not to have access to the smartest intelligence because there are many problems that may well yield to these very significant intelligences and you still probably need some domestic commute for optionality even if there is still a free international market for compute you still have a lot of benefit to having it on so on and so forth So it's still worth doing even under these conditions. Yeah. And I think you also likely need a bunch of this high security compute

for your national security as well, where it seems likely that AI systems will massively transform the intelligence community. If you think about their ability to process fast quantities of data in the same way that analysts do in GCSQ, you could see them really being bottlenecked by GPU access. and I don't think GCHQ are going to be passing lots of their classified information through like random Swedish data centre that's serving Astra this week. Well we did have a PLA official who was querying Moonshot's Kimmy and it ended up going to Claw. I can't remember. Amazing. Yeah I can't remember because there was some Chinese government official too. So some people were doing this but they shouldn't be because you would be. So I think on the secure compute side there is this like make sure we still have access to front-end models. there is also this like clear necessity for some parts of government to have access to this for like national interest. Yeah, that sounds compelling to me. Cool. Well, Tom, it's been a real pleasure.

Thank you for coming on. It's been a pleasure. Until next time.