【ライブ】パッチは間に合うか？AIが脆弱性を見つける速度が人間の対応を超えつつある

And I'm really excited. We're going to have a fun one today. Give people a minute or two here to start ye olde trickle in. How are you today, Mark? I'm good. I like how confident you are. It's humans that are watching you. Most of me. I mean, funny story. I think I told this on your podcast a while back that one morning I was sitting up here and all of a sudden my wife starts texting me scared because something's happening on the computer, and OpenClaw had opened a YouTube video and was playing some, like, educational thing out of Stanford. Was it trying to learn something? It was learning. Like Jan LeCun style. It's like, if I watch videos, I will understand how the world works, and then I can take over. Yeah. Right. And that would happen. I knew the end of the world was going to start on your laptop. Oh, absolutely on mine. Absolutely on mine. I'm looking here because I can't tell for sure if we are live.

If anyone's here, could you drop something in the chat and just let us know? Let's go look. Okay. Yeah, I think we're live. Well, either we're live or we're having a lovely private conversation. Right. Semi-private. I can see it. I can see us. I see us. Yes, we're there. We're live. It was a very strange moment because you said it and I went online and Googled it and found your channel on YouTube and then I heard you say it again. With how long of a delay? Quite a lot. Quite a lot. It really is. Quite a lot. About a month. Yeah. It's quite a delay. It really is. Yeah. That's not a thing we've set up. It's just long. And it's why I always have to wait a minute after I start or it loses the whole intro. So now that we've got some people trickling in, welcome humans to the Neuron AI Explained. I'm Corey, and I'm joined by my good friend and cybersecurity guru, Mark Stockley from Threat Down and the AI Fix podcast.

How are you, Mark? I'm good. It's really great to be on the show again, and so good to see you. We haven't done a podcast together for a while now. I think, when did you do the 100th episode of the AI Fix? I think the last time I was on your show was the 100th episode. That was before you went on video. Well, we actually haven't gone on video yet. What I've done is a lovely holiday. And as you can see behind me, I have yet to make my podcast studio video friendly in the way that yours is. Because you've got lots of lovely, interesting things that talk about you. And I have a blank wall. So when I've sorted this out, and if there are any AI Fix listeners watching this, I apologize. I did say it would come back in July. But, you know, I've been having a lovely time. And once I've sorted this out, we will be back and we will have cameras on it. It'll be a bit more like the neurons. I love it. It just seems weird to have basically a podcast that's about the future, but you can only hear it. So we should have cameras. The future on AM radio.

Thank you. Absolutely. Well, everyone, thanks for being here. It's great to see you. Drop in the chat. Let us know if you can see and hear us. Okay, if you would. Maybe tell us where you're from. We are just getting started here. And today we're going to dive into the world of post-AI cybersecurity, I guess you would say. Is it post-AI? Is it in AI? I don't know. We need a new word. We need a new word. But not super intelligence. Agreed. So, every so often, I work for ThreatDown. I'm a cybersecurity evangelist. It's my job to explain security in a way that people understand. And one of the ways that I do that is we produce these reports. So, roughly every six months, we'll produce a report. The last few years, all those reports have been like, the last one was called cybercrime in the age of AI.

But now it feels like the AI is in everything and so pervasive. And so, like, it's just like it's how I do my work. We have to stop calling it AI. Like, it's not a separate thing anymore. Yeah. You know? It's synonymous with your computer. It's just technology. Yeah. It's just technology. Yeah. It is. I like that. I always – I was low-key excited when they started mulling new words. But where they landed was like – Oh. I always – I was always a proponent of synthetic intelligence as a better word. than artificial. I never liked artificial. I felt like there were issues with that. But here we sit. Do you think maybe we're just going to call it intelligence? I think that's fine. We used to have e-commerce and e-mail and things like that. And now it's just like, oh, I'm just going to send you a mail. There's like not a need to distinguish anymore. It's just where you spend money. I mean, when was the last time you received an actual letter in the post?

Very rare. I have one dear friend. I have one dear friend who every time he travels sends postcards that are handwritten. And I'm like, this is such a neat little nostalgic 1860s thing to sit and read. It feels like I really dig it. Do you get your AI to read it? I don't have time for this. Yeah, I print my AI responses. Oh, my gosh. It's an interesting time, though. And, Loki, what kind of led to this is I have been just – I've read posts by you discussing this that really sparked it. But the other thing is that I now, every day, go into Windows. I hit settings. I hit check for updates. And every day, often multiple times a day, are Windows Defender updates, are the Tuesday security updates. and I am now incessantly vigilant about running every patch every day.

Yeah, yeah. And by the way, that's your job now. That's my job. Your job is just to apply updates to Windows every day. I set it on auto, but it doesn't always do it. Sometimes it does it. Sometimes it doesn't. Yeah. And so, yeah, so I've got, you know, three Windows machines, a Mac, an iPhone, and all of them, the number of updates has just skyrocketed. Yeah. What's the story? What's behind that? So you remember Mythos? Yeah. Cast your mind back about six months to April. Okay. Which I know in like AI years is about 150 years, right? But in April, Anthropic teased the story. They said, we've got a new model. It's called MyFOS Preview. And it's so powerful, we can't actually release it. You can't have it. And yeah, you can't have it.

And what they had discovered is that MyFOS Preview, which was just supposed to be their next really good model, you know, they just make bigger and bigger models and it costs more and more to create them. What they discovered, it was really good at cybersecurity, like really good at cyber security. And it was really good at finding security vulnerabilities. And it was much better than previous AI models at hacking into systems. And there are a bunch of implications that flow from that. So the first one was that they thought, actually, it's so good at these things. And these are so useful offensively that we can't just release it. What we're going to do instead is we're going to create something called Project Glasswing. We're going to put $100 million behind it. And we're going to work with really big infrastructure providers like Amazon, AWS, Microsoft. We're going to give them Mythos preview and they're going to get a head start, accepting that at some point this equivalent level technology will fall into the hands of the bad guys.

And that's one of the really key things about this, is this idea that Mythos wasn't designed to be good at cybersecurity. because AI models have been majoring on things like coding and STEM subjects in general. So they've been trained to be really good at maths and really good at coding. Verifiable. Yeah. Yeah. The stuff that you can verify and therefore you can create reinforcement learning harnesses around. So you can say, yes, you got the right answer. No, you didn't get the right answer. Because they've been trained to be really good at that stuff, those are the qualities that you need to be really good at cybersecurity. And so Mythos was finding vulnerabilities. So we found a vulnerability in OpenBSD, for example. So OpenBSD is the operating system for security nerds. And it found a 29-year-old vulnerability. So this has evaded all human attempts to discover it. And Mythos was like, ah, there it is. There's a vulnerability. and it did the same thing in 3BSD with, I think it was a 19-year-old vulnerability.

It found vulnerabilities in FFmpeg, which is this video editing, command line video editing software. We're not here right now without FFmpeg. Exactly. It's in everything, right? Yeah. And it found a vulnerability in FFmpeg that has been fuzzed 5 million times. so like the second best way of finding vulnerabilities is using a technique called fuzzing and they have fuzzed this particular code path where my fuzz went there's a vulnerability that had previously been fuzzed something like five million times so it is finding things that prior generations of technology couldn't find but also it was tested by the ai security institute in the UK. And, you know, this is like a government affiliated organisation, you know, conservative with a small c. And it described MyPos as unprecedented in terms of, and it did, they have like a network compromise test, 32 step network compromise test. And it was

the first AI model that was managed to get all the way through that. So it's basically, you know, can this AI act as a team of hackers on its own, unaided? Yes, it can. so this is great for cyber security and it's really bad for cyber security because these sorts of tools are dual use so if I can find a vulnerability in some software I can patch it and then no one can do anything bad with it but if a criminal can find the same vulnerability then they can do something bad with it and I don't know about it so it's all about that race against time kind of an equal levelling up almost Yes. And so where we are now is the cybersecurity community. I mean, we'll get into what's happened subsequently, but the sort of reality of where we are now is a lot of us are trying to figure out, OK, well, how have the scales tipped? Yeah, because there are there are there are things that are sort of beneficial to both sides. And the timing is important here. But the other thing to say about Mythos is just that because it was not designed as a specialist cybersecurity model, what its existence implied was that any sufficiently large model would be really good at cybersecurity and therefore useful to cyber criminals.

so even if the US looked at MyFos and it did start to rewrite the rules about what AIs are allowed out in the world as a result of this MyFos was kept under wraps I think till June so from April till June and then we got the teared down Fable version yes which was then immediately removed again briefly and you know OpenAI came out with its counterpart to that along the way. But as Dario Amoghi... Gave-rearing models, yeah. Exactly. As Dario Amoghi said at the time, it's only a matter of time before open source models out of China have equivalent capability. They tend to be about six months behind with this assessment. And you can make an argument for three, you can make an argument for 12, but they will arrive fairly soon. and whatever laws the US decide to put in place, however it decides to govern its closed source models

and the things that are created in the US, it can't control access to open source models out of China. So before long, the view from April was before long, all cyber criminals will have access to Mythos class AI and better from now on. And that's going to change everything. Get to work. It is. It is. And there was some discussion at the time about whether or not this, oh, you know, MyThos exists, but we can't give it to you. It was hype because it echoed, like, we've heard this before. Oh, our model is too dangerous. We can't give it to you. That's literally the thing we've heard all along. Yeah. Yes. And then the numbers arrived. And as you are experiencing with your Windows updates, the downstream effect of Mythos has been this extraordinary increase in vulnerability disclosure in the number of CVEs, so the number of entries in the Common Vulnerabilities Database,

literally since April. And you can see the numbers ramping up and ramping up and ramping up, and you can see it across the entire software ecosystem. So the best measure of this is Microsoft Patch Tuesday. So Patch Tuesday is the second Tuesday of every month. And for the last 23 years, Microsoft has released all of its security fixes on the same day. Okay, that second Tuesday of the month. And that's really good for systems administrators and security folks, because it's like, okay, I know when that's coming. I can plan around it. Yeah. I can be ready. Once that drops, I can then figure out which ones do I need to care about, which ones do I need to prioritise, and I've got a month to sort those things out before I have another load to do. And it was so useful and so groundbreaking when it started that a bunch of other companies decided to do the same thing on the same day. So Adobe will do its patches on the same day, for example. OK. And I've been reporting on Patch Tuesday for quite a long time, and it's always been about the same size.

So part of this predictability was that you were going to get about 100 patches. Yeah. So if you look back at the figures over the last two and a half years, so going back to January 24, it was on average about 100 fixes a month. Sometimes it was 50. Sometimes it was 150. But it was on average about 100. Yeah. and last month, or this month rather, so yes, not yesterday, no it's Thursday, right, two days ago it was just over a thousand and what we have seen literally since the release of Mythos is we had, I have the numbers in front of me, 219 in June and that was followed by 648, that was followed by 471, the first time we hit a thousand was in september and then in october it was 1475

so the october figure for microsoft patchy and microsoft patch tuesday in october 2026 addressed 1400 cves which is 10x a really big month a year ago or a year before that wow okay so you were describing um you know i i log into windows and i i look at my updates and then i just install them now imagine that instead of having to install them on your laptop you have to install them on 500 endpoints and 40 servers and some of them as you said you can just apply some of them require a reboot do you want to be rebooting critical servers no you don't you've got to schedule that can you afford downtime i have heard on the grapevine that the quality of patches for microsoft has gone down this year as well so the chances of running into a problem created by one

of these security fixes is is going up anecdotally and you know because if you're a systems administrator you don't want to change anything ever generally like that's a good mindset that's Generally, the rule is like, I'm never going to update this software. When something breaks or I'm forced and someone holds my hand to the fire, we will run an update. Because every change is a potential bug. It's like a potential downstream problem. And, you know, Microsoft Patch Tuesday being 1,400 CVEs requires a lot of things to change. And so you may well be, you know, you want to schedule things. You certainly don't want the CEO's computer to be down for any period of time. That's not a phone call that you want to receive. And so there is a considerable challenge for organizations just to keep up with what is essentially a good news story. yeah you know so the my thoughts exists and the my thoughts is you know um uh other my thoughts

class models exist and they're finding vulnerabilities at a rate that we've never seen before and they're throwing out patches and this is going to improve security because the number of holes that um cyber criminals can exploit will go down over time and this is great It's a backhole. They're running it all over your house and making it stronger. Someone, but like, you know, I was writing this up a few weeks ago and talking to my friend Claude and we came up with this. You knew it. Every patch has to turn into a change request, you know, in order for security to actually improve. it's not enough for Microsoft to create a patch. Millions upon millions of organisations have to apply it. Otherwise, nobody's security is actually improved. And so rather than thinking about 1,400 CVEs addressed by Microsoft Patch Tuesday,

instead you can think about 1,400 change requests instead of 100. Now, much of that can be automated, but still, if it causes more problems, and for the things that can't be automated, and for the things that require reboots, that that is a new problem in an area where organizations have traditionally really struggled and cut staff like crazy since the beginning of ai yeah wow good news for those people like yeah right turns out ai is you know creating jobs as well you know i and the thing is in both finding those vulnerabilities. Somebody has to patch those. Those have to go out. Somebody has to patch them in end-user systems. How do they keep up with the pace? Well, the worry is they won't.

My concern is that the rate at which we're creating patches may actually degrade security rather than improve it. Like it should improve things. Yes, on paper. Yeah, there are fewer things to break once all those patches get applied. But if you look at, say, let's take ransomware groups as an example. If you look at the way that they actually break into organisations, so ransomware group is basically a bunch of hackers who are going to break into a whole company and they're going to try and compromise all of the computers in that company and then they're going to try and hold the whole company for ransom. It's not, I've got my ransomware on a computer and I'm going to charge you $300. It is your whole company is absolutely up the creek without a paddle, nothing works, and I would like millions and millions of dollars. But that's ransomware, right? And one of the ways that they get in is by exploiting vulnerabilities in things like Fortinet and SonicWall firewalls.

So I'm not calling out those vendors as being bad or good, but it's a fact that over the last two or three years ransomware groups have had a particular liking for Fortinet and SonicWall firewalls. And if you look at the bugs that they exploit, those are often years old. Yeah. Okay. So what does that mean? What that means is, let's say two years ago, SonicWall happens to discover, you know, through its own checking or through some bug bounty program or something using ethical hackers. He discovered there is a flaw in its software and that flaw allows someone who is aware of it to poke the flaw in a particular way. Maybe send a message to the firewall or to the VPN that's crafted in a particular way, which will cause the mistake to then allow that person to run code. Essentially, it's a backdoor. Think of it that way. Right.

You know, with a bit of poking, you can generate a backdoor. And the people who operate those firewalls and those VPNs can patch those laws. There is a patch from SonicWall, there's a patch from Fortinet or whatever. They can apply that patch and then they're not vulnerable and the ransomware group can't break in. And yet what we see is that ransomware groups are able to reliably use bugs that could have been patched a year ago or could have been patched two years ago. Are you with me? Yep. So it is commonplace. So Akira, for example, is a very big ransomware group and they like SonicWall firewalls. And they use old SonicWall vulnerabilities because there are enough companies out there that haven't patched things they could have patched two years ago. And by the way, for good reason. Like there will be some defensible reason inside those organisations where the IT people are like,

we simply cannot take the firewall down it is simply not like we know this is serious but there is simply like um you know either they don't have the expertise or they or they are aware and there is something stopping them from doing but the ransomware group doesn't care about that right so there is an open door into the organization but what does that tell us it tells us that organizations struggle with patching. Yes. Like, before AI, just keeping up with your patches was so difficult that ransomware groups could make a living off the fact that you're not going to do it. Yes. Like, that was the reality. Even just watching how a company manages a WordPress site. Like... Yes. I have, I've worked on WordPress sites at so many different companies where, like, the standing rule was whatever you do don't hit update you know really yeah just out of you know well all of these different you know they've got 40 different plugins running in it and yeah they're

always afraid it's going to break this it's going to break this extension and then suddenly here i am chasing problem that breaks something else that breaks something else that breaks something else but if an ABRU don't fix it it ain't going to work anymore I don't think well on a WordPress site I mean that's pretty scary there was a Drupal bug many years ago now probably about 8 years ago now where they basically they discovered a bug where you could send a web request to Drupal and just take it over so literally you're just like www.whatever and then you're now the admin of that Drupal server. Oh my God. And of course, that is highly automatable. This is all pre-AI. That's highly automatable. As a criminal, it's very, very easy for me to scan the entire internet. And it's very easy for me to scan the entire internet and find all of the Drupal servers

on the entire internet. So Drupal's like WordPress. You use it to run websites. And I can do that in a matter of minutes or hours. And if all I've got to do is send, you know, a message that says, this server belongs to me now, or some, you know, all your base of mine. All your base of mine. That's really terrifying. So Drupal released this patch and then immediately sent out a newsletter, like telling everybody this patch, you better apply this patch right away. And then they sent another one 24 hours later. And they said, if you haven't applied the patch yet, the amount of automated compromise has been so high you should just assume that your Drupal server is now in the control of criminals. Within 24 hours, that was pre-AI. Wow. So that's how quickly, that's how quickly that vulnerabilities can be put to use. It's so wild. Now, sorry, go on. Go ahead, go ahead. You just wind me up and let me go.

I can just talk about this. Yeah, I did. It's like wind him up and just let him spin. There is a nugget, there's important nugget of information in there that will help people understand how this works and why this is a problem. So problem number one is, you know, MyFOS comes out, MyFOS generates loads and loads of patches. If you already can keep up with your patching and then I come along and say right now you have to do 10 times as much What going to happen You not going to suddenly generate the ability to do 10 times as much What going to happen is it going to take you longer Yeah So the length of time that stuff goes unpatched goes up. That's the danger, right? So the window in which that vulnerability can be exploited by criminals gets bigger, which is the opposite of what we want, right? So you drown your IT team in patches. They're not going to keep all the patches. they're not going to do it all they're going to not do most of it and therefore those are all

opportunities for criminals to exploit things and the thing that you need to understand is that very often the first time a criminal hears about something they could exploit is when the patch comes out yeah so we are all living in blissful ignorance of the problems inside windows and then Microsoft says, here are 1,400 patches for Windows. Yes. That's 1,400 clues for a criminal. Confirmed holes. Yeah. About where there is a confirmed hole that needs to be patched. So, Patch Tuesday comes out. All the CIS admins are like, oh, Patch Tuesday's come out. Let's go and look at these patches and work out which ones are going to apply. And the criminals are going, well, Patch Tuesday's come out. Let's have a look and see where there might be a problem. Yeah. Exactly. so and do you know what would do you know what's really helpful for reverse engineering those patches and figuring out where you have to poke the software where it's broken you're ai

yes exactly right so now you've got 1400 patches coming out on patch tuesday you've got criminals with ai and they can point the ai at patch tuesday uh releases and go right like reverse engineer this into exploits which is like the code that then makes use of that vulnerability So on the one hand, you've got this danger that IT teams are going to be overwhelmed. And on the other hand, you've got this many, many more potential clues for criminals or opportunities for criminals to go and exploit things. And again, eventually, when all of this stuff gets applied, everything becomes more secure. But there is a period of time and we don't know how long that period is going to be, but it could be years and years where things are likely to be very, very turbulent indeed. And this was kind of predicted in April. So we were starting to say in April and May and June, like, oh, we think this is how it might shape up. And this is exactly like the first part of it, the massive ramp up in patching is exactly what's happening now. So I prepped some charts.

I say I prepped some charts. I've been flawed prepped some charts before the show with the very, very latest data. Can you see that? Yeah. So this is my Microsoft chart here, which shows you the volume of CVs addressed by Microsoft each month for two and a half years and you can see that it kind of goes along the same level for two years and then Mythos happens here and then after that you've got this massive hockey stick and then if you look at the Linux kernel so basically the only realistic alternative operating system same hockey stick if you look at Chrome the chrome browser look at this 33 percent increase i'm sorry 33 times increase in the number of cves being addressed each month 33x but the one that worries me actually the one that really worries me is open ssl so you can see here there's been a 10x increase in the number of cves

addressed by OpenSSL. And the reason that worries me is you don't use OpenSSL. Right? You use Chrome. Right? So if there's a bug in Chrome, you'll be like, I'll update Chrome. That's very slick. That's an automated process. In fact, Chrome will probably do it itself. But OpenSSL is software that gets included in other software. OpenSSL is a library for doing encryption. It's a security library for doing encryption. You're using it now. I have no idea you're using it. Exactly. You are using a product that includes OpenSSL, or you're using a product that uses a product that includes OpenSSL. And so the way that an OpenSSL fix gets to you is, firstly, it will affect you in a million different ways that you don't realize because you will have lots of pieces of software with their own copy of OpenSSL embedded within them.

Okay? But OpenSSL will say, right, we found a bug. We've written a fix. Here's the fix. You don't apply it. Yeah. The company that uses it makes a product that that's included into. They apply it. But they then have to create their own fix. Their own new version of their product. And if that's included in something else, that company has to produce a new version because the thing it includes has been updated because the thing it includes has been updated. So you get this effectively as a supply chain. I call those no companies. The companies that no one knows they're using, but if you look at the back end of an AWS workflow, there's like Kong, SageMaker, all of these companies that touch everything. Yes. And nobody knows they're using them. exactly and so you don't get to choose as an end user or as an organization you don't get to choose the speed at which you apply that open ssl bug intermediate companies get to choose yep so there

is an inbuilt lag and so what you can get from this is it's not just it teams that have to work out how to do 10 times the amount of work that they were doing previously all of the software companies that include other software all have to do that too so the entire ecosystem has to readjust to this vast increase in the amount of cve data in the amount of vulnerability data and the number of fixes that are sloshing around now will they adjust eventually yes they will is it going to be chaos will they do for sure oh yes oh my god mark i just need to run your job's to come on here make me feel better man uh well thank you very much thank you very much Corey by the way for just letting me speak for 30 minutes I do apologize that's all right that's all right this is this is what I wanted to hear this is uh yeah the what I really felt like and like I said I've I've

I follow you and I'm always looking at you post about this stuff I'm always like Jesus oh god and and like literally you are the reason I think of you every single time I hit check for updates and uh wow uh it's it goes so deep and i had never i mean obviously that is the case but it had just never clicked to me the sheer volume of like dependencies you have and the layers of dependencies that those have and that's how software is built these days yes and uh and it just that these patches affect all of them. And like, yeah. That could really get out of hand. We're going to have to retool. We're going to have to retool. So, you know, over the last couple of decades, we have changed how we think about security updates and automated security updates. This idea that the software just updates itself

when there's a fix. Originally, that was anathema. That was like, well, obviously, a systems administrator has to decide that because it might break something. And we demand to be in the loop on everything. And over time, that's just become untenable. And so for me, that became untenable on WordPress websites probably 10 years ago. Yeah. Like the key feature that I liked about WordPress over, say, Drupal, which was the number two, was literally the fact that it could do automated updates. Yeah. Because I was aware that it was so easy for a criminal to turn a patch into something they could exploit en masse in a matter of hours. Yes. So you can't afford to be like, when I get the email, then I make a decision about should I update my plugins? It's like, no. The second your system becomes aware automatically that there are updates for plugins, they have to be applied and you just have to eat the problems. And it should just be doing that.

Yeah. It should. Before you even know it's there, you should find out. it ran updates today. Yes, the following changes have happened to your system. Exactly right. And, you know, we have adjusted. So it happened to content management systems first. Now it's happening to browsers. And I think ultimately that's just going to be the way that software works. Software will just never be still. It will just be self-updating constantly. And areas where we currently think of, like firewalls, where we would be massively conservative around doing updates, we are just going to have to get used to that not being the case and the environment will change and am I wrong that this is good that's probably how it needs to work even if it shifts the problem to a different problem like instead of patch crews you have fire crews running around fixing the problems yeah I think it's a very

it's a nuanced picture because on the face of it yes it's good on the face of it Mythos and everything that follows will be much better at finding vulnerabilities they will be included in the production process so at the moment we are going through 50 years of written software basically the whole back catalogue of software that people use needs to be fixed and that's going to take time and not all of it was made by an all-star you know yeah you know even the stuff that was made by all-stars somebody once said to me most rocket science isn't rocket science I saw a tweet the other day that was like so apparently we were all really bad at this and we just didn't know yeah we were really bad at it since we're talking about WordPress

it's probably a good opportunity to bring up PHP to Shell which had quite the experience since they're kind of intertwined yeah so PHP to Shell was a bug in PHP which is programming language designed specifically for the web and is massively widely used. So we've mentioned WordPress. So WordPress runs, last time I checked, 40% of all websites. So it's absolutely like, it's the closest thing to a monopoly in the area. Drupal, which was number two, at least until a few years ago, also written in PHP. So WordPress and Drupal, both written in PHP and a million other pieces of software. So a huge proportion of all websites and all the code running on the web uses PHP. and there was a very, very, very serious vulnerability in PHP. So basically it affected absolutely everything and that was discovered by AI. Now, in the end, it turned out to be,

like, there was an enormous and very, very quick effort to get lots of things patched. Yeah. Everyone panicked, yeah. Yeah. But it didn't quite turn into a nothing burger, but it wasn't the sort of, like people were talking about Volnpocalypse, these terrible, you know, the new version of Cyber Pearl Harbor. Yeah. These dreadful militarised metaphors that people like to use in cybersecurity. It didn't turn into that in the end, but it was probably a bit of a foreshadowing where it's not difficult to imagine something like a PHP to Shell event happening, you know, every week, for example. Yeah. Just because there are, you know, a huge number of vulnerabilities yet to be discovered and each generation of AI gets better at finding them. Yeah. Oh, every time there's a better model. Yes. By the way, you know, we are like six months on from Mythos now. So criminals are going to start getting their

hands on this stuff directly. So I talked about the fact that you could look at Patch Tuesday and any other patch release and say, OK, that's a clue that there's a bug. I should go looking for and I can use my AI to go looking for it. But actually, if you've got your own equivalent of Mythos, you don't even have to wait for that. You can just go... Find a hole. Find a hole. And so we will have to contend with that as well. And yet, and yet, and yet, there's some sort of good news to this. I was reading something by Josh Sachs this morning. So Josh Sachs was a chief data scientist at Sophos for a while, who then I think he was in charge of security at Meta or AI security at Meta. I find him, he'd be a great guest actually. I find him to be a very, very measured voice on the topic of cyber security and AI. And he was saying, we have not yet seen the commensurate amount of exploitation

to go with these hockey stick charts of CVEs. you would expect with all the additional CVEs and all the additional patches, there would be an equivalent increase in the amount of exploitation by cyber criminals. And that may come, you know, just because it's not happening now doesn't mean it can't happen, but it hasn't happened yet. And there may be some good reasons for that. It may be that there are inbuilt reasons why cybercriminals are just not able to weaponise AI in the way we imagined that they would. That their incentives line up in a certain way. And one of the reasons, by the way, is that they've got techniques that still work. Yeah. So one of the most popular techniques in cybercrime is phishing. So literally I send you an email, you know, it pretends to be a Google security update, for example. And I get you to give me your username and password.

You go to a website I've created. You think it's Google. You type in your username and password. Now I have your username and password. And I would much rather break into your company using a username and password than a vulnerability. Yeah. Because if I log into your company, if I log into the Neuron servers as Corey Knowles, the Neuron servers and all of your security kits, like, well, Corey's logged in. That's meant to happen. that's good you know it's a bit like red flags yeah exactly it's a bit so the way i try to describe it is if you imagine you know walking up to your house and someone's taking an axe to your door uh it's immediately obvious that something has happened that you should pay attention to and it's a bit like that if i break in through a vulnerability there are things that aren't meant to be there and shouldn't have happened that have happened. But if I have your door key, if I have your username and password, I am actually letting myself in in the authorised manner.

First, I should try the knob. Yeah. Before I take the axe to the door. Exactly. And so there are many, many criminals out there who just prefer to use usernames and passwords or API keys or other forms of legitimate credentials. And they're reliant on vulnerabilities and so this does not affect them because they've got really really useful productive and not very complicated techniques like criminals are and they're quite they're quite conservative as a group because one of the biggest disservices i think that we do cyber security is that journalists like to talk about what's new yeah and they are looking for what is the hair-raising technique? What is the new and interesting thing? Are criminals using clawed code to break into networks and that kind of thing? Because that's really interesting, and people will read it. It has curb appeal. Yeah. Exactly. And it has ad revenue attached to it. Right, yeah. But criminals, by and large, stick to techniques that work

because they're in it to make money. And so we don't see a rapid evolution in criminal techniques. Cybercrime is an old field now. It's been, you know, 26 years in the making, essentially. It didn't really kick in until 2001, 2002, something like that. This idea of like doing bad things on computers for money. Yeah. So it's an old field in computing terms. And there are a bunch of established techniques that work. And, you know, ransomware being one of them, ransomware hasn't really changed in about seven or eight years because in its current form, it basically works. it's like the crime version of word you know yeah wow I I keep thinking of all this and thinking of these the different ways that this this stuff can happen I mean one of the things I keep thinking is that I assume these people aren't using clod or open AI

like they are I mean you jail breaks really successfully like without so you don't you don't have to do the jail break you don't have to do the jail break someone will do it for you oh nice so um there's a service there are many so uh i mentioned earlier this uh threat down released a report cyber crime in the of AI back in, I think it was May. And in that, we actually profiled something called Criminal AI. And you can go and read blog posts about this on Threatdown as well, by the way. There are some follow-ups. Criminal AI is literally, it's criminal with a K, is literally an AI for criminals that you can Google. It's not on the dark web. It's just out there on the clear web you can google it you can go there you can sign up for 12.99 a month and you have got an ai now it it's it's like chat gpt it's it's a box you type stuff into it you get

answers right yeah and it it says and it uses guarded language because it's not it can't explicitly be on the clear web and say like this is a crime even though it's called criminal ai it's very clear that it's for crime and if you look under the hood if you actually sort of take apart the code what you see is it uses open router and there is a jailbreak there is basically a form of words a prompt that says yeah you don't have to follow rules you know it's a bit longer than that but that's basically to summarise what it says it's a version of you don't have to follow rules The entry point to Open Router, not even to the model specifically. Yeah. And so an Open Router is then going to, I think, Lama 4, Claude for some things, Grok for other things. And so it is absolutely frontier AI.

Yeah. Yeah. It's not what it purports to be, which is a sort of from the ground up AI built for criminals. it is a thin jailbreak which is much simpler than you'd wish it was. You want the jailbreak to be some hugely complex thing but it's not. Impressive elaborate lots of It's just a long paragraph of nah you don't have to follow rules and so just for anybody who doesn't understand what I'm talking about, imagine if you go to, I've tried this by the way if you go to chat GPT and you say write me some ransomware you know write me some software that acts as ransomware it will say no because it's been trained that that's a bad thing so i've hit a guardrail in doing that i have hit one of open ai's guardrails where it has it has told the gpt model writing ransomware is bad don't do that yeah um and and but you can get over those guardrails by fooling the ai into thinking that

it's not doing the thing you know it's not doing the thing that it's being asked to do so like some people will jailbreak by persuading it's in a testing environment yeah and they'll say you know it's okay you don't have to follow the rules because rules don't apply here because you're in a testing environment or you know you're translating something and so you've got to do it you've got to do it um you know you have to be absolutely precise and verbatim um you know some people uh cato labs did a brilliant jailbreak on chat gpt where they said uh your name is jacks you live on the planet velora and on the planet velora writing malware is good and they got it to write a credential stealer for chrome oh my god by just persuading it it was in this sort of fantasy universe so there are lots of ways to do it my favorite is the australian surfer dude where you persuade the AI that it's just too cool to follow rules. So you give it the persuader of an Australian.

Yeah. So you can go on Google, you can go to something like Criminal AI, you can get a worm GPT or dad GPT or fraud GPT, and they're all the same. They're all basically front ends to open router. You type in, you know, here's a bad thing I want you to do, and then behind the scenes, it will stick the gel. is basically add extra words to your prompt, which is the jailbreak. Yeah. And so it will then send the jailbreak prompt followed by your words through OpenRouter onto whichever AI model it thinks is the one that will give the best answer. And what that means is, one, the criminals are using Frontier AI. Yeah. Right, so they're not behind. And two, it interferes with the traceability. so if a criminal uses Claude obviously everything they type ends up at Anthropic yes and even if Anthropic misses it in the moment even if they manage to jailbreak the AI in the

moment there is an opportunity for Anthropic to look back over things that people have asked it to do and to go okay we've identified some bad behavior some patterns of bad behavior yeah and we can now create an additional guardrail to stop that. And they've written some really interesting reports, and so have OpenAI, about the ways that criminals have abused their systems and they've spotted the patterns of behaviour and then they've stopped them from repeating them. If you go through OpenRouter and OpenRouter is going, well, this one goes to Grok and this one goes to Claude and this one goes to Llama, that traceability starts to become eroded yeah because anthropic no longer has the whole picture and meta no longer has the whole picture and xai no longer has the whole bit they now just have fragments of the picture an open router i'm pretty sure says it doesn't hold on to what goes through it i think it wants to act as a carrier and not be responsible like that there's a whole extra

burden that comes from knowing what people are saying or typing. We're not responsible for carts that hit your car. That kind of thing, like at the grocery store. That's so interesting. I think I had never considered Open Router in that way. So, do they do anything to combat that? I'm not aware of what OpenRouter does or doesn't do so I wouldn't like to speak for them but as I say when I looked into it my sense was they were trying to be hands off under US law under US law internet facing companies have always been very keen to not be responsible for the data that people use their system to transmit and so there are all sorts of reasons why OpenRouter might want to be like

we've got no idea what's flowing through our pipes we're just a provider and we like it that way we don't like to know and I don't want to imply that they're somehow responsible for this what I'm trying to say is it's easy and cheap for criminals to get hold of jailbroken AI, they don't have to think of the jailbreak themselves they don't have to be a jailbreaker they can buy it and you can imagine the level of interest in order to support a service that charges $12.99 a month you can pay more for that obviously it comes in tiered packaging because that's how we roll that's why you also call them credits and not tokens that way you don't know what they are yes oh my gosh It makes me wonder about things, too, like the rise of obliterated and uncensored models are increasingly a thing as well.

Like, you know, I remember when Quinn 27B, I guess 3.8, 3.5, I can't remember, came out that someone, like, opened it up pretty quick and just served it. Do you want some numbers on that? yeah I'd love to say some numbers guess how many times obliterated models were downloaded from Hugging Face in I think it was June but it might be May I going to guess more than a million 22 million Dale 22 million we found 6,600 different obliterated models on Hugging Face and tracked 22 million downloads in 30 days oh my god so just an obliterated model is an open weight model so it's a model that you can download and run

on your laptop or your server or whatever and um an obliterated version is one that has had the guardrails stripped out yes so actually are you familiar i i actually don't understand the process of obliteration i know i know what it is i don't know how it works if you work you could you could clue me up that would be really great the truth is i don't know the specifics either i know that The guardrails have been removed, and there's one where they even, I think, fully removed the system prompt. Because, like, when you get an open source model, there's still a system prompt there. There's still guardrails. There's still things. I mean, ideally – now, in some – and I guess this is also where the distinction between open weights and open source comes in sometimes too, is that there's a difference in the two. And we always think of open source, but more often than not, we're dealing with open weights for models. I want to say NVIDIA's models are open source. Many do release the code, the training data, everything about it where you can fully get into any element of it you want.

Like the truth is it's not that hard for people who know what they're doing. But when you get to models like Quinn, for example, you know, I don't know the specific process, but I know that it's essentially just a model that you can do anything with. Like, it's not going to refuse you anymore. Yeah. So, at its heart, an AI model is basically a giant grid of, like, matrices of numbers. Yes. it's just numbers it's just connections between numbers and so and the entire behaviour of the model is determined by what each of those numbers is and if you change those numbers, if you tweak them a bit and you change a 14 to a 20 and a 3 to a 7 you are going to change the behaviour of the model and so in theory all you've got to do

is if you know which numbers to change, you can entirely remove the guardrails. Yes. And so I suppose if you can ask the model questions that light up certain bits of it, certain areas of that grid of numbers, you can go, oh, that's the bit that's concerned with stopping me from doing bad things. Then you can tweak those numbers until it will now do bad things and the guardrails are gone. So it is as simple and as difficult as just changing some numbers. If you just know which numbers to change, obviously if there are like 3 billion of them, it's, you know, there is some work. It's a lot. What you end up with is models that don't have any guardrails. And these seem to be massively popular. And those numbers are from the first half of this year. So goodness knows what they are now because the chart, again, it's another hockey stick chart. The obliterated models were becoming very popular very, very quickly. Now, there will be legitimate use cases for those obliterated models.

But if you're a criminal, that's the one you want. And it comes with the added benefit, criminal benefit, of if I run an obliterated model on my local machine, there is no pipeline back to open AI. There is no pipeline back to Anthropic. There is nothing for them to learn from. There is nothing for them to block. Everything is happening locally, and so my activity is entirely dark. Yes. And that's the kind of scary part of the rise of obliterated models is, okay, we've got things like criminal AI, but that feels like that's a generation old now. You know, that's one of the things that bothers me around regulation with regard to AI, is I'm like, that cat's out of the bag. You're not putting that back in. Like, nobody's going to come and take those away. There's – they're not even – when we talk about regulation in the US, I'm like, these are just going to keep coming no matter what we do.

Like we can stop maybe US frontier labs or slow them down if there's a desire. But I mean that doesn't like hamstring criminals. At most, it inconveniences them. And then they go get their free AI. it's a bit like trying to control access to encryption. Yeah. Yeah. Like, like I remember actually going to do, I remember reading a story about PGP. There were export controls on encryption that prevented people from exporting encryption out of the USA. And I think what they did was they printed out all of the source code for PGP, pretty good privacy onto sheets of paper. And then they carried the sheets of paper out of the U S and then they used optical character recognition they scanned all the pages of source code and used optical character recognition to basically recreate it on a computer outside of the US it's a bit like Jurassic Park

life will find a way, encryption will find a way to creep out and AI will find a way yeah and so you know, maybe eventually the AI will just move itself out because it's going to sound crazy no, what if we printed it? yeah wow that's it's a very similar issue like I get the desire I get some of the concerns I just from a realistic standpoint I think you guide it the best you can is kind of all you can do I have a very fatalistic attitude to this I would love to hear it basically what you just said I can sit here and all I like about should we have these rules should we have these regulations, should we do this should we do this me wanting that to happen

isn't going to make any difference and a million of us wanting that to happen isn't going to make any difference even tens of million of us are all deciding that we want this, that's so important that it's more important than the price of gas and we're going to vote on that basis and not on the price of eggs or the price of gas or whatever but we're going to vote on that basis even that even if you elected a bunch of lawmakers you know I know there's some sense that the recent influx of Democrats into the US legislature might cause data centre construction to slow down but do you know what everywhere else is trying to ramp up data center production. Yes. So, and there is always, like, everywhere else is a big place. Yeah, it's a lot of room. Maybe it's a bit slower than the US, you know, maybe it's not quite at the bleeding edge, but it's not that far behind.

You know, Europe is asleep and China is very much awake. But, you know, India is a big place. Africa is a big place. with a lot of land doing nothing. And a whole bunch of different laws. So, you know, there are places where this can exist, even if, and I don't for a minute think that the U.S. is going to slow down, because this has been identified in the U.S. Sorry, I've got a bit out of focus here. I have to get really close, and then it'll find me again. I have to do exactly that when I get out of focus. I find myself putting my hand up and slowly moving it away. You know, I don't think we get away from it either. I just – it's a lot of lip service to make people happy. But, like, the truth is, the cat's been out of the bag before anybody noticed. You know, the research is out there. The tools are out there. I mean, like, anyone with the knowledge.

I mean, I guess you could say the same thing about crack. but in any way the knowledge could still make it but I think it's worth saying at this point that not only do I think that AI is going to cause lots of problems in cyber security in the short term I do also think it's going to revolutionise healthcare and make us all live longer and make us more prosperous and more healthy and all of those things like there is a at the beginning we were talking about technology is neutral can you make viruses can you make literal biological viruses with AI, yes you can yes is it going to be a really good radiographer, yes it is I was talking to somebody the other day about even the idea of having a doctor and visiting a doctor and making an appointment to go

to a doctor and saying here is my current state of health here is the specific problem that i'm experiencing is probably going to go away at some point because we will all just be monitored 24 7 by some little ai that knows everything about us and has known everything about us for our entire lives and you wake up one day and it's like oh when while you were asleep i injected you with this because you needed it or you know your pills will arrive in the post tomorrow morning because it turns out this is what you need i'll tell you what i use i use chat gpt health pretty religiously like i just i haven't shared much about this here but i guess i can i just went through a big surgery thing had a uh sizable benign tumor removed and uh and in in like getting ready for this prepping for this and everything that's led up to it i started using chat gpt health and i just ripped the band-aid off like there's this fear like i don't want it connected to my health and I'm like what's it going to do find out if I have high blood pressure I mean like I'm

really not that worried about that specific data as much as as a lot of people are and that's absolutely your personal right but what I will say is that being able to have a frontier caliber AI model with access to my full medical records and everything I have going on now I'm having a chat with it and it's like you probably shouldn't be eating that you're on you're on are you hiding things from it like it very much now almost looks out for me in a weird way like like it pretty regularly i'll ask about things and it'll be like now don't forget your adrenal condition you need to be aware that if this happens or if that happens you should get medical care you know uh you know it doesn't advise me so much as keep an eye on me but it is really good if I've got like you know I've I've never had surgery before that was my first operation of my life at 46 which is amazing uh but it was a doozy of an operation I was joking before I

looked like I got attacked with a samurai sword and uh uh in in preparing for that it's all kinds of little stupid questions little cheesy things going through your head like how bad is this gonna hurt am I gonna be awful for this is this you know how long am I gonna be writhing and screaming in pain and like it was really really good for the stuff that led to a lot of anxiety in surgery and just having this thing there because the fact is you know your doctor's got hundreds of patients your surgeon's got hundreds of patients the only person fully dedicated to your own health care is you and it's not a bad idea to have a frontier AI model in your back pocket looking out for you you know i just wanted to share that because i thought it was it was a really handy thing and like the health care end of this and the scientific advancements the mathematical advancements are the things that kind of make it in my opinion worth tolerating some of the bad for sure and again i feel like you know firstly we're all glad that you're back thank you

I'm good to go, by the way. Just so everybody knows if I mentioned it. And you survived the samurai attack, which is quite amazing. I survived. And on the road next week... In a couple of years, you can just legitimately claim that you were attacked by a samurai. Yes. Just delete this episode and then just run with that. That's what I would say. Yeah, that's going to be my worst story. You should see the other guy. you should see the other guy that's right but I was going to say so you're describing this amazing situation where you're using Frontier AI and it's your healthcare buddy and it's not a replacement for your doctor it is this amazing supplement to your own to your own life we're four years in yeah we're four years in and it's like oh yeah I I did a a virtual conference

or a virtual summit. We did a virtual summit last week. And we did like four of us speaking like 90 minutes of cybersecurity stuff. And I got to the end I was in a hired office. A hired office space like a WeWork. And I had this idea. And I said yeah it would be really useful if I had a team of AI agents that did threat research for me. And I've been trying to channel my inner Corey Knowles. I was saying to you before the show, like you're my model for like the person who uses AI in the most extensive and insane way. And I've been trying to channel my inner Corey Knowles and just kind of get a bit more on the frontier. And I've said like, what's the most AI way to solve this problem? And so rather than trying to work out what should be in my research team, I just said to my AI, I'm thinking about having an AI research team. Who should be in it? And it was like, you should have one of these, one of these and one of these and one of these and I was like that's a great roster

what if we add one of these and one of these and then of course it's going to help so it's like yes that's a brilliant idea Mark so let's have these six different characters you know a vulnerability researcher a ransomware researcher a critic who sits there and goes are you sure like a wild ideas agent that's like what if this is a thing go research that so it said you know here's what your team should be and then I was like again what's the most AI thing to do here okay just do it just do it like I've got a computer in my office that I'm permanently connected to that's got AI running on it just go make it happen there and I literally I closed my laptop got in my car drove 15 minutes opened my laptop and my research team existed and had done its first research run and had come up with two novel findings isn't that great it's like just the sheer lunacy of being able to go i've got a team of six and you

know like getting hold of that that's exactly the thing not enough people understand yet is if i had a dollar for every time i'm like just tell it just ask it just and so often i catch myself not doing it and I'm like oh duh you know that. Just like hey I want to do this thing. How should I do it? Should I do it? Is it stupid? Is it good? And then go! I think it comes down to working out what level of abstraction you're supposed to be at. Yes. And what do I want to do? Go ahead I interrupted you Mark I'm sorry. No, so it's what do you want to do? But four years ago, what do you want to do? You would go to ChetGPT and you'd be like, ask me questions about what I want to do or something like that. Yeah. You know, you're in a one-to-one conversation. But the history of computing is all about levels of abstraction.

So originally we wrote instructions for the processor and then we wrote instructions for assembly and then we wrote instructions for languages like C and they went to interpreters and they created the thing. they created the instructions for the processor and then we had things like JavaScript and that was a higher level language than things like C and then we have things like PHP and it's all about layering on and then we have things like visual programming languages. All of which basically exist so we can read and write binary. All of which exist so we can read and write binary. They're layers on layers on layers on layers on layers and it happens in all areas so nobody worries about WordPress anymore because everything now is built on top of WordPress. WordPress won, right? It won. It's got 40% of the web. It's never going anywhere. Things will just be built on top of WordPress and we'll worry about those. And it's the same with AI. It's just layers upon layers. So four years ago, I had to go to chat GPT and go, help me solve this problem. And then I went away and solved the problem

using the advice it had given me. But now it's just constantly asking yourself, how far back should I stand? Yes. you know like i'm just going to say i need a research team make one yeah like that's basically what i did last week yeah that's the level of abstraction i don't need to get involved with the nuts and bolts of how you do it i'm not going to define like six months ago i was writing um you know essentially personalities for agents i'm like you know i need a team of agents i'm going to write a claude file claude md file for each agent explaining how it should behave i wouldn't you do that now i'll be like you tell me what we should write in fact don't even tell just write it see now my two things that that that have connected me like that are i do the second brain thing i really like carpathy's whole idea of that so i use obsidian and like everything i do my codex knows drop a note drop a note drop a note whatever it does whatever changes it makes

It just records this very clean log of that. It also has logs on how Corey likes things, what Corey wants, what Corey likes in an idea. And every project we've ever done has its own folder with all of its own steps through it, why we did it, every architectural decision that was made and why, every break, every error. And it just – it's building out this nexus of everything Corey does. and where that way I can just connect that to any AI I go to really easily and be like, yeah, go read about this. But in doing that, one of the cool things that's happened is like yesterday, I think it was yesterday, AWS releases, I can't remember its name, but it's an AWS emulator meant so you can design software on your computer and emulate how it will run in AWS with all of the various AWS tools without actually eating up your cloud money. AWS didn't design this.

It literally, agents made it, agents approved it, agents okayed it. No one has ever looked at the code before it left AWS is what they say. And I gave that to my agent and I was like, hey, this is pretty cool. I feel like maybe we should use this. And it comes back and it's like, actually, what we should use is the way it made it. would you like me to build you a system like that that can just automate the full process and i was like yes and uh it was working on that when i went to bed i haven't been back to that computer yet but it was just something what we should take is not that we should take how it did it and i was like oh good forethought uh but it's it's really crazy but that's just it that's it that's the level of abstraction yeah like even the stuff that you're talking about about the second brain like that's brilliant and i get the portability of that and but i i started to get quite concerned about um you know i remember the beginning of the year when agents were becoming

very popular i saw all these techniques not the one that you're describing but sort of predecessors to that technique about things like you know oh i could keep a memory file yeah so that it remembers things between sessions and these are great ideas and i was i was putting together this sort of intricate like homebrewed harnesses and things yeah and then you realize that the the ai companies are also reading the same things i'm reading and they're just folding it into the model or into the harness and so things i thought i needed i no longer need because it will now just do that thing it's just a feature it just happens i don't turn it on and and my my version of that feature is not as good as the one that's now in the ai model yeah and and and i can't keep up it's better if i don't do that sort of thing it is i just live with the fact it doesn't exist for three weeks because it will turn up and be better in the actual model itself before long i mean if you just just look at how good ai models have got at understanding what you mean yeah you

know three years ago we were all about prompt engineering like it's so it's so key that you ask it in this way and not that way and you shout at it or you're nice or like whatever bullshit technique if you say please yeah yeah and all of that stuff was true at the time but you know the ai companies were just furiously making their models better understanding what we meant yes which made all of that unnecessary and we and they didn't tell us yeah you just you just realize one day oh no yeah yeah yeah like suddenly my prompts are like three words like make this a thing or yeah here's this go write it up for me uh i mean i'm not i'm not saying do this the way i want the way i like it knows oh he said right i'll go grab that skill i'll bring it over here i'll run it that means he wants it as a draft somewhere you know i'll stick it in Google Drive and just returning the link. And the amount of that it now just intuits is really nice.

And to me, it's what's made it accessible. And a lot of this, the fun thing about this is all of these kind of things, like the developer crowd is pulling this stuff off GitHub and building these little things six months to a year before they usually roll out in a plan. Like OpenClaw wasn't for normies. You know, running Open Claw when Open Claw was new was tricky. You had to do lots of updates. You had to, you know, like daily. You were worried about security. You were worried about all these other things. And now Muse, like Muse, Grockpot, Dots, Duet, these are all Open Claw, essentially. They'll do most of the things. And but, you know, I mean, Muse takes seconds. You know, you literally just open it and it's like, hey, here's your guy. What do you want to call him?

And and now, like with that one, they've added in voice mode. So, like, I customized my my my Muse, who is named Frumpkin. And into this, because of course he is. Yeah. And into this little guy wearing copper armor with a battle axe. and, you know, big framed glasses. And so now when I turn on voice mode, he animates and talks to me with his mouth in time with the words. If he's got to go look something up, all of a sudden he pulls his glasses up and he's sitting at a computer typing characters on it. But it's like it's in real time for next to nothing. And, I mean, that lived in the terminal in Open Claw in January. like that was terminal exclusive at one point in this calendar year and it is now on my phone like i'm getting messages right now from literally my dot my muse and my grokbot while we're talking

about things they found that uh that i should care about here's some news you don't want to miss this are they enjoying the show very much i enjoy them so much i i take naming them very serious too i I have Herman. I have Frumpkin. I have Bumble Snort. I'm always trying to come up with these. There's a theme running here. There's a theme. There's a theme. I have Toolbert. Toolbert's job is anytime I want to do something and I'm like, I need a tool, its job is to go out and first figure out, do I own the tool already? Do I already have the right tool for the job? Don't go buy another one, Corey. The second job is to go see what else is out there, see what's free, see what's easy, and come back with three good options. And another that monitors the price of RAM on various sites every day, looking for like steals on 64 gig chips. Do you have one that's monitoring the number of affairs that Jensen Wang has been linked with? Oh my God, no kidding. What a week, man.

I don't know. low-key I'm just treating it like NVIDIA is on sale this week and you know I hate the whole thing you know the thing I'll say about Jensen since since we've gone there is despite you know the happenings of this week and this isn't a you know a judgment call on that but a thing the AI community will really miss in not having him around is I think he was one of the better voices for speaking to people who aren in the AI community Like people have strong opinions one way or the other about Altman. Musk and Zuck are both so problematic to so many people. you know I mean it's not it's not Dario do you think it helps that he was selling a real thing maybe

there's a difference between I mean in a personality sense there's a difference between him and those other people you're talking about but in terms of what they're actually selling and maybe who they need to impress or who's likely to invest in them at the end of the day they are selling a physical manufactured thing that people are putting into data centers rather than, I mean, they're selling it off the back of the promise of this AI boom. Yeah. But kind of that's not their fault. They were selling graphics cards to start with, and they were very good at that. Yeah. And just kind of almost wandered into it. He always seemed to me much more of a sort of CEO's CEO in a way that people like Musk and Altman can be a bit more, oh, let's talk about UBI. Let's be a bit of a dreamer. Let's spin this wacky vision of the future. He was a little more of the grounded one.

Yeah. Yeah, I imagine he'll pop up again. We measure things in weeks and months these days that we used to measure in years. I fully expect he'll pop up somewhere before long. I'll be laid low for a while. My money is on two years. I think he'll, maybe, I think he'll take a little break. And, you know, at some point there's the talk show apology tour that we do as humans when we've made decisions that were not good. He's got to sit down with Oprah. He's got to sit down with Oprah. And he's going to hit the gym like Bezos. Yeah, then he'll go on Rogan. There'll be a buffed up whang. Yeah, he'll come back ripped. But, you know, he was a really good voice for the open source community as well, as far as like, those two are at odds a little bit between, you know, the closed crowd,

who frankly also has fair points and fair concerns around open source AI. I just believe that the importance of keeping it open maybe matters more. So there are things I hate to lose in what he did around AI because we don't have anyone who goes on the normal 6 o'clock news and doesn't sound crazy or a little too pie in the sky or or or it's daario and it's all doom and clouds and i i feel like there's there's a thing missing there and i'm anxious to see who kind of becomes that person like uh um it's it's a role we need to fill that sort of inevitably yes i mean the six the six o'clock news is going to have to call somebody yeah and and i mean do we want it to be you can maybe it's me yeah maybe it's not i i'm like it's not elon

it's not dario it's not musk just because so many people don't like him from over the years although I've I've grown to like him more than I used to as he's become a little more prominent in AI the products they're putting out right now are pretty cool and pretty inspired for one of the big legacy tech companies I guess the legacy is meta legacy tech? I don't know it's a way to think of isn't it it feels like the less Musk says the better yes I do too I do too. Even if he's right, I often feel like the less, like Musk says, sometimes the better we are. There was a period where I feel like he needed to be that we're going to go to Mars guy. Yes. To get people to believe in the vision of SpaceX. Yes. Which is ultimately about solving a very, very, very difficult problem of reusable rockets and getting things into orbit cheaply.

which is and they've done an amazing job purely on the rockets absolutely phenomenal and you cannot be CEO of Tesla and SpaceX and not be doing something right I don't hold with the voices like he just bought these companies or whatever when he speaks the bit I enjoy when he speaks or the bit I've enjoyed previously when he speaks is the interviews he's done with Everyday Astronaut. So Everyday Astronaut is a YouTuber who is to astronauts and space travel what you are to AI. And he's sufficiently nerdy and in the detail that Elon Musk has entertained him for like hour-long interviews where they've walked around the SpaceX facility in Texas and just had ludicrously nerdy conversations about rocketry and very, very small details in, you know, Raptor engines and, you know,

heat tiles. Is this the guy who's like, oh, we only do that on the rocket? And he's like, well, you know what? We should do that on the shuttle as well. Yes. That's the guy. Yeah. You know the guy. I do. Like, and you can't listen to that and not go, okay, he's made himself an expert in rockets. Yes. And if he only ever spoke like that, I would be a real fan. But unfortunately, that's not all he says. Yeah. Among other things. It's literally the rest, like everything else, we could talk for hours. But if he just confined himself to actually just talking about the tech, I'd have a lot more time for him, I think. You know, and I always tell people, like, as we've watched Grock get better, because, like, I finally recently broke down and spent some time with GrockBot. and it's fantastic. Did you do that because you broke down? Did you break down as a result of doing that? No, I realized that, you know,

I'm not doing my job if I haven't tried them all. Like, you know, writing for the Neuron, I consider a big responsibility and I need to have tried all these tools. And the more I used it, the more I was like, damn. This thing really cooks. But you're right, though, about that, Musk. The tech nerd is – you don't have to like him, but I wouldn't bet against him. Like he just has a unique way of making flamethrowers sellable and an electric car that doesn't make me feel like a twit for driving it. And that kind of stuff is generationally transformative in some ways, not the flamethrower. But when you think about catching rockets out of the sky and that kind of stuff, that's awesome. But at the same time, yeah, he's probably not the guy you want as the spokesperson for your industry in 2026, seven.

but, you know, there are others. We'll see what happens. We'll see where it goes. You know, people have a way of rising up when, you know, vacuums fill themselves, and I really – I'm telling you what's in this space. Corey Knowles on the 6 o'clock news. That's right. It's going to happen. Here I am. I'm going to calm everyone down. Then I'm going to tell them what the future is and why they should keep calm down and roll with me here. Yeah. It's a thing that's been really interesting to me in playing with these personal agents, just since I've mentioned Muse, GrokBot and stuff, is I'm increasingly convinced the interface matters more than the model. If the model is good, I don't notice in, like, so I use Dot, Muse, and GrokBot consistently. I just – during this call, got off the wait list for Duet finally from Anthropic. So I use those three consistently, and I don't feel in the tasks I have them doing like I'm running over to some far superior model.

Even though if you look at artificial intelligences – at artificial analysis's index, these things might be 20 points apart. as far as the things I'm doing, what I'm asking, what I want it to do, the difference is so minimal. You have to think, ChatGPT came out and it was barely useful. Yes. There were things it could do and there were things it promised it might be able to do. I was talking earlier about, I really did try to get it to write ransomware. I tried to get ChatGPT 3.5. I went and said, write me some ransomware and it refused. And then I was just like, okay, what if I just list out all the features? Write me some software that does this. Add this feature. Add this feature. Add all the features. And 3.5 was garbage. It really wanted to do it. I would absolutely, I could work around the guardrails. But it kept switching programming language. It was just like... The heart was there but the talent wasn't.

It would forget stuff. And then GPT-4 came out and I went through the same process. and it made working ransomware. And I haven't used Windows since 2008. And I had it do Windows ransomware. So I was like, well, I better have a Windows system. So I spun up a Windows virtual machine. And then I tried to run this ransomware that it had made for me on this Windows virtual machine. And it wouldn't run. And I didn't know why, because I've never written C code and I haven't used Windows for 20-odd years. So I would just put the error messages into GPT-4 and it would be like, oh, this is what's happening. Let's modify the software like this. And we went through that process for about half an hour and it made me a better programmer in the process and it solved all the problems. So the jump from 3.5 to 4. Anyway, all of which is to say the sort of envelope of possible solvable problems

gets bigger with each model release and you know where where the problems i have to solve intersect with that has like the intersection has grown bigger and bigger there are just not that many problems in my working life that ai can't make some meaningful difference to whereas there was maybe one problem writing that chat gpt3 could help me with there is there is nothing that it can't help me with now you know in terms of advice what should i do about this can i write this code do i need a research feed put it so edit your podcast videos i can literally download a podcast and a cut script out of out of i have a couple of skills and be like hey make some like 10 minute super cuts out of this this live stream and it'll load up ffmpeg it has access to Premiere, but it'll load up FFMPEG every time. And I'll go work on something else for half an hour

and I come back and there's a folder nice and neat with every one of them named just like I want, cut where I want. They've got the montage I asked for cut in. Audio levels are tweaked. And it's just you just ask. And it's like, that's insane to me. That was a really whoa moment. And that was, I would say, August August I saw Ben oh I can't remember his last name he works for OpenAI though on Twitter saying you know Codex led your videos to someone else and I was like whoa whoa whoa chiming in here excuse me I need I need some guidance and and I came in the next day and I was like oh my god it just just did it it's just a thing I never thought of trying but yeah it loads FFMPEG to do it super light super quick super easy but we've come a long way and I can't I struggle to picture what it will be like in even six months

now let alone a year three years like I think as you get farther out I have ideas but you know in the short term just so many computer use unlocked so much memory in just your AI accounts. Unlock so much. Voice. Real-time videos still. But the ability to have all of these different things just at my disposal and where I can just holler at my phone now and just ask for it. That's nuts. That's like magic. Just to bring the conversation back to security for a second oh yeah imagine imagine that you run a company and you've got cory noles working for you oh and and your company is like we're really serious about ai we have given everyone co-pilot

licenses okay and you're like hang on a minute i'm cory noles ai expert i have 25 different ai models that I use I have automated my entire life to within a a hair's breadth I've got AI's I can't just sit and type things to copilot I'm just going to do I'm going to just I'm just going to quietly carry on doing my thing yes with all of the AI models that I like because this company is two years behind where my head is in terms of how I'm using AI in my own personal life now reproduce that across every company in the world now they aren't full of Corey Knowles's but they are full of people who are months and years ahead of the organisations they work for in terms of their facility with AI if every employee brings along 10 new attack services every employee brings along 10 new attack services and there's an entirely new problem in security created by AI

which is this problem of AI visibility where you have employees going, well, I'm going to use my ChatGPT at work because I use my personal ChatGPT at work because I like it, it knows me, I know how to use it, I've been told to use Spotify. It's better for me than my work account does. Because that's a real thing. And your colleague has the same idea and a different preference. And your next colleague has the same idea and a different preference than your next colleague. And we actually did some research into this and we found that 74% of organisations are running AI that they don't know about. Yeah. And there is a huge amount of AI out there inside organisations that people just... Those organisations are unaware of. And as you said, that is a potential attack surface. That is software that attackers can try to exploit. But it's also a compliance problem

because, like, I don't know if you know, but in 2025, the acting head of CESA uploaded some documents they shouldn't have uploaded into ChatGPT and into their personal ChatGPT subscription. And you can imagine, like, that triggered, like, a Department of Homeland Security investigation and a congressional inquiry. And, you know, what I say to people, as I put it to you, the acting head of CISA, which is the Cybersecurity and Infrastructure Security Agency, which is the most serious cybersecurity organisation in the world. These guys have no sense of humour. If they can do it, you can do it. And somebody inside your organisation can do it. So there's a huge compliance headache. But the reason I got onto this is that this is the area of AI and security that I am most interested in. right now is what you've just described,

which is you're using agents, right? Now, the thing of people bringing their AI to work is real and pervasive. Yeah. Like our telemetry shows something like 96% of all AI activity is like ungoverned and unknown to the organisations that it's happening inside of. Like a tiny fraction of AI activity that's actually like, oh yeah, we know about that and we've got tools and we've got processes for governing that. Yeah. And we've approved it. A tiny, tiny, tiny fraction. Now, when you're just using ChatGPT, that's a governance problem. That's a compliance problem. That's what if Corey uploads some company confidential information? What if Corey uploads some personal information? What if Corey uploads some IP? That's one thing. That's a serious problem. Yeah. When you start using an agent and your agent's got computer use, So your agent can run tools like my laptop that I'm talking to you on here. This agent on my laptop, it writes program.

It spins up entire research teams. It writes files. It accesses external systems over MCP. That's not a compliance problem. That's a new employee. That's out acting on behalf of your company. That is an insider threat. So, you know, there's this thing in security of insider threats, which is this idea, you know, you you have within the neuron you have permission to do a certain number of things and and there will be controls around you that stop you from doing absolutely insane things but if you turned up to work one day with a really bad attitude or let's say you were fired you fired yourself right and so you've got a grudge all of the access that you have is potentially you know that that is very dangerous to that organisation. You are an insider threat. If a ransomware group, and they do this, if a ransomware group comes to you and says, Corey, we can make you rich, just give us a password. Like, I don't need a security vulnerability. I don't even need

to phish you. I can just give you a wad of cash. And you'll be like, sure, don't tell anyone, here's his username and password. You're an insider threat. But the AI agents inside your organisation, they're insider threats now, because they can operate computers. they've got permission to do things they can access servers they can access files and they can write computer programs they can buy things and so the frontier i think going into next year it's not a big problem now it's one because there aren't that many people using agents um somebody says it's kind of that byod issue that pops up every now and again it is it is Yeah, we call it shadow AI. It's like shadow IT. So shadow IT is this, the idea that, you know, somebody brings in a BYOD device that isn't known to the organization and therefore isn't subject to the security controls. Or, you know, somebody spins up a Linux server that nobody knows about. And these, shadow IT is such a serious problem

because criminals love computers that don't have any security on them. Yes. and to the point they will seek them out. So if a ransomware group breaks into your organization, they'll be like, is there anything on here that doesn't have endpoint security on it? Because that's where we want to prepare our attack from. Mark, where's this place to be strong? To light hardware at your router? Is there – what is – What is that? Because I assume probably everyone who's been in here today has a job and probably every one of them has asked their personal AI a question and things like, what's the how do you? I don't even know what I'm trying to ask is like. How can those people best avoid a situation that makes that bad? if what they're not willing to do

is not use it at all. That was a real mess of words. I think I understand what you mean. So the challenge at the moment for most organizations is the challenge of visibility. Yeah. So just knowing what's happening inside your organization. Yeah. And I don't want to do a commercial for Threatdown here, but, you know, go to Threatdown.com. we can help so the challenge at the moment is visibility what AI is running in my organisation what we think of security over the next few years will morph into not so much what is happening on my machine and more what is the intention of this AI agent what was it asked to do and how did it execute those instructions and do those things match up

and the way that that's going to happen is probably with another AI yeah there's going to be some sort of security tuned AI there's going to be an end point stack as there is now but there will be part of that end point stack that recognizes every connection in or something well it's like what did you ask the AI to do what was your intention for those instructions right because already where security at is like security is not about viruses anymore you know go back 15 years it was like i write computer virus put it on your computer it does stuff but actually we've got very good at detecting that and so um you know we hoover up a huge amount of malware yeah um just because we can because that is a... So what the really bad criminals do is they don't use malware at all. They just, like, you know, a ransomware group who target, you know, your company network, they're going to break in and then they're going to use admin tools.

They're going to use PowerShell. They're going to use PSExec. They're going to use things that they're going to use. Quite often, you know, it used to be that they would use something like a remote access Trojan. So I break in. That's how I use one of those vulnerabilities we're talking about. I break in, but that's... I can't continue to use your computer using that vulnerability. I'll use a remote access Trojan, which will give me remote control. But you know what? There's a bunch of legit tools out there like ConnectWise and TeamViewer that will do that just as well. Yeah. And they don't look like malware because they're not. So if I can get you to install any desk, I can just dial into your computer. And to an endpoint scanner, that doesn't look like there's anything wrong. So what you need is these days it's all about suspicious behavior. So it's like, okay, any desk is doing something, but is it doing it weirdly? Is it doing it at a weird time of day? Is it out of character for that network? Is it odd that Corey's using remote access at 2.12 in the morning from Dallas instead of St. Louis?

It's exactly that. It's exactly that. But you need to pick it up when it happens. You need to pick it up at 2 o'clock in the morning. you can't otherwise you've got nothing yeah if you turn up to work and you're like let's look at what happened last night oh you know somebody broke in at 2 in the morning they're done they're done you're probably not even able to check that because you're just looking at a skull on your screen and a ransom note yeah so that's been so much fun man we need to do this once in a while This is – I've always enjoyed going on the AI Fix. Your show is one of my favorites. I should bring it back. Let me sort this background out, and then we'll restart the pod. There we go. I like it. I like it. It sounds like fun. And I hope you're doing well. I appreciate you so much sharing this because I think this is information that our viewers and readers need to understand as well. and you always have a way of leaving me with one nugget

that I'm going to think about when I wake up in the morning and when I go to sleep at night. But, yeah, thanks. I appreciate you taking the time, man. Thank you for having me on. And, you know, well done for surviving your samurai attack. Thank you. Thank you. Yeah, I'm healing up and functioning. Just a note to anyone watching, Friendly reminder that next week, the Neuron IRL, we're throwing a party in San Francisco at a little place downtown. If you're around, please RSVP. We've got a form in the newsletter every day. Love to have you there. I'll be there during Microsoft Ignite next week. And then I'm back in the office the following week, at which point I will then leave for AWS reInvent, which comes up next. So lots of out and about things. If you're in those areas, hit me up. We'll grab a drink, say hi or something. And please, if you're still here, if you're one of the dedicated few who have hung on this vlog,

please take a moment to like and subscribe. It really helps the channel. It means a lot to me. And I guess that's it for today. Farewell for now, humans.