バイブコーディングを監査する：生産性・セキュリティ・CVEの数字が示すもの（第321回）

Today, I want to talk about Vibe Coding, but giving you the real numbers about this new trend, apparently, that is supposed to, you know, kill the conditions of many software developers around the world. At least that's what is currently believed. But as a matter of fact, Vibe Coding was coined by Andrei Karpati back in February 2025 in a post where he essentially describes writing software where you, and I quote,

fully give in to the vibes, embrace exponentials, whatever that means, and forget that the code even exists. Which sounds like magic. Really? Well, it sounds fun in a way. So here's the part that everyone in fact forgot about that quote. because in that same original post, Carpatti also wrote, and I quote again, sometimes the LLMs cannot fix a bug, so I just work around it or ask for random changes until it goes away. And he explicitly scoped these two, and I quote one more time, throw away weekend projects, not production systems. So the guy who invented the term told you in the same post not to do this for anything that matters and the entire industry immediately

did it for things that matter you know you see how stupid we are so let's go through what actually happened because after so many years it looked like eternity which is indeed you know more than one, two years, three years in IT and in computer science can be the equivalent of decades, especially now with artificial intelligence systems and how fast these things evolve. You know, one year can be, in fact, a lot of time, a very long time. So I collected a few numbers that I want to share here and about VibeCoding because, you know, There is a bunch of statistics that I collected from some websites that I will share with you in the show notes of this episode. And, of course, on the official website, datascienceatome.com, which are typically numbers that, you know, very few people talk about.

And I want to start from front-end development. So, you know, there is a piece out there on nerdy.dev, which has a title, Why AI Sucks at Front End, that lays out in pretty precise technical terms where these tools really fall apart on UI work specifically. And, you know, to be very fair with you And also to the piece and the authors It opens with what AI is genuinely good at You know, which is the boring, well-worn, copy-pastable patterns Token migrations, generic lists, scaffolding, all that stuff It's onward, AI loves the boring stuff It thrives on mediocrity Which is fine, there's no argument there But then it gets into where it breaks And the author's phrasing is worth repeating Because it's precise, it's not just mean So pay attention to this

It says, ask it for scroll-driven animations Or custom micro-interactions And, quote, it will invent a CSS syntax That hasn't existed since Internet Explorer 6 Ask it to get pixel-perfect symmetrical spacing, right? and you get, quote, it's already bad at math. How could it get its ridiculously dynamic calculation correct? Accessibility gets the same treatment. It throws aria hidden equal true, for those who know CSS, at a wall and hopes it sticks. So the article gives four root causes for this. And I think they are worth naming individually because they're not the same problem here. So one is, of course, training data. I'll give you some numbers later I don't know why I'm showing this slide now But the very first issue That the authors of this post have described

Is of course training data So the web these models learned from Is, according to the article at least Roughly 43% WordPress Which is not exactly a byword for clean modern HTML So the model's default instinct skews toward old templated patterns. And again, I'm quoting, modern CSS is barely aware of it. The second issue is it cannot actually see. Now, it's a language model. It's not a rendering engine. And so you can hand a screenshot, but the article calls that stubbing in the dark. The classic failure mode it describes. You know, the model confidently says a component is done and you find a literal gaping hole where an icon should be. So, you know, there is no visual feedback, even though new models and new architectures might have a visual feedback. But again, the training didn't come from there.

And the third point is that no architectural understanding of the why behind decisions like state machines or behavior-driven design. So you're asking, in the article's word, a giant tax predictor to make new connections on the fly. So that the summary of what are the reasons what might be the reasons for which large language mothers indeed suck at UI And the fourth point there is four points This is probably the one the most underrated one is zero control over the actual environment the thing renders in So there's no pinning the browser, version, window size, input method. There's keyboard, mouse, touch, voice, all different. So every real user is a moving target. And as we all know, pattern matching models, in fact, as large image models are, are terrible at moving targets. For those who know what moving targets are.

Now, none of that, of course, is AI is bad. Okay? It's a specific named list of failure surfaces. so keep those four in your head because they show up again in a second just with a few numbers attached and typically these numbers are US dollars This episode is supported by OutShift Cisco's incubation engine for frontier technology Multi-agent AI is in pretty much every enterprise roadmap right now but there is a problem with that and we are all facing it Agents can pass messages but they cannot think together. So these systems silently underperform through cognitive failures, things like misreadings, unverified claims, and of course, false consensus. You know, all the stuff that conventional monitoring cannot detect. OutShift by Cisco is building this fix, the Internet of Cognition,

which is an open source foundation for building multi-agent systems from design to production. with shared contacts, shared memory, and guardrails to drive the results that we actually expect from artificial intelligence. So read the paper, experience the demo, grab the code from outshift.com. That's O-U-T-S-H-I-F-T dot com. Let's move to the actual numbers. So there is a second piece for that from cracker.dev that pulls together the empirical side and its sides, its sources, which I appreciate. So let's go through the studies it's referencing one at a time. The first is here. It's CodeRabbit. So CodeRabbit analyzed something like 470 GitHub pull requests. and found AI-generated code produces something like 1.7 times more issues overall than human written code.

So broken down, logic and correctness errors, 1.75 times more frequent. Then we have concurrency and dependency issues, two times more common. Excessive IO operations, eight times more frequent. And finally, security vulnerabilities, specifically 2.74 times more common. So these are massive, you know, numbers compared to what humans could do a few years back. Then we have Git Clear, another source. They looked at something like 211 million lines of code, right? 211 million lines of code. across, of course, 2020 to 2024, and tracked down two trend lines heading in opposite directions, pretty much. Duplicated code that climbed from 8.3% in 2021 to 12.3% in 2024,

while refactored code, you know, the stuff that keeps a code base healthy and clean in a way, collapsed from 25% down to 9.5%. Sorry about the slides here. They don't really represent these numbers correctly, visually, but they are correct, the numbers. So from 25 to 9.5 is a massive decrease over the same period. So the too-long-didn't-read version of all these numbers, essentially more copy-paste and less clean up, if we want to summarize all this. Then there is, on security specifically, a report from SoftwareSeni found design flaws 2.53 times more frequent in AI-assisted code, high severity vulnerabilities, so CVSS 7.0 or above, 2.5 times more common, and secrets exposure 1.4 times more likely.

And separately, Veracode's 2025 testing found 45% of AI-generated code failed security tests outright, and 86% failed to defend against cross-site scripting. Cross-site scripting is a very mature type of attack in computer science. So, still, 86% failed to defend against a very mature attack. It's not like a buffer overflow 20 years ago for which novelty, you know, would have taken, you know, it would be much harder to detect these things. Then we have a number I hadn't seen before this. Georgia Tech runs something called the Web Security Radar. And as of March 20th, 2026, it had tracked something like 74 AI-linked CDEs with a monthly count accelerating.

So 6 in January, 15 in February, and 35 in the partial month of March. Then we have four breaches on four different platforms. So the article backs that up with the actual incident. These are not hypotheticals, okay? This is fact. So there is Maltbook, a social networking app, exposed something like 1.5 million auth tokens, authorization tokens, and 35,000 email addresses only in February, 2026. Then we had, of course, lovable at the vulnerability, CV 2025 48757 for those who are interested were missing raw security exposed over 170 production applications built on the same platform And then we have Base44 had broken access controls

that let people bypass authentication entirely and RichLead put payment logic on the client side. Users bypass payment through the browser console within 72 hours of launch. I mean, it's funny to see these numbers now of course it's not funny because there are people who have been damaged from this but these are the stories that I don't read often I read only success stories about AI well, we should pay attention to these ones, I think now, there is the productivity number and this is the one that actually surprised me as well So METR, or METR, run a study with 16 experienced developers across 246 real tasks. Okay? Before starting, developers predicted AI would make them 24% faster. But after finishing, they believed they were being something like 20, so felt after 20% faster.

The measured actual result was 19% slower. And they only accepted fewer than 44% of the AI suggestions in the first place. So I know it's a lot of numbers, and probably if you're following this from the podcast, it's quite hard to follow because there's no slide deck. On YouTube, there is. But essentially, what this means is that, you know, someone predicted to go faster, felt faster, you know, the psychological trick that we get played from every time. but the measure, the actual thing was actually slower so that gap right there is basically the entire thing, the entire vibe coding life that nobody is telling you now there is one more data point and I'll flag straight away that it's genuinely ambiguous rather than a clean gotcha the US Bureau of Labor Statistics projects 15% software developer job growth from 2024 to 2034.

So in the next decade, more or less. And the total U.S. engineering workforce is sitting at 4.4 million and still growing in 2026. At the same time, entry-level hiring is down 25% year over year. and employment for developers aged 22 to 25 specifically is down nearly 20% from its 2022 peak. So there are two real trends here pointing in different directions, if you see, in the same job market. Now, I'm not going to pretend I know exactly how those reconcile. I'll just tell you both numbers are up there. You do your math and your interpretation. now here's where it gets even more interesting because I don't want this to just be 20-30 minutes of dunking there is a case where letting a model lose without

deep human hand-holding produce something real in security research of all places and here I'm referring to mythos so Bantropic put out, as we all know, every time there is one of these news, I let the news sink. I let people debate and say a lot of things online. We live in a democratic place, right? Really. So people can say pretty much what they want. And then after a few weeks, I, you know, let the news sink and then I take it back and I analyze it again with, you know, a more peaceful mind. So, Antropic put out an announcement a few weeks ago about something called Claude Mythos Preview, describing it as one of these general purpose models that's, quote, capable of identifying and then exploiting zero-day vulnerabilities

in every major operating system and every major web browser. That was how it was claimed. So, concretely, it found many CVEs, among which CVE 20264747, which was a 17-year-old FreeBSD NFS bug, that gives an unauthenticated attacker full root. So they own the machine, okay, after that attack. It also found a 27-year-old bug in OpenBSD TCP stack, sorry, S-Hack handling, and a critical flaw in the Botan cryptography library that let certificate authentication be bypassed as well. And Anthropic's own comparison numbers on an identical Firefox JavaScript exploitation benchmark, their existing model Opus 4.6 managed two successful exploits out of several hundred attempts.

So Mithos managed 181. So cost-wise, they cite under $20,000 for 1,000 runs against OpenBSD, roughly $10,000 across several hundred runs against FFMPEG, and under $1,000 and half a day of compute, more or less, for more complex Linux privilege escalation chain. So these are kind of the compute budgets in each, let's say, type of attack. And credit worth you, Antropic states its own limitation plainly. Over 99% of what is found is still unpatched. and they cannot discuss most of it publicly, of course, for responsible disclosure reasons and it failed to remotely exploit some Linux kernel bugs it correctly identified. So these are all claims made from the owners of the models,

the AI Lab and Tropic. Now big impressive mostly verifiable claims however here the pushback because it from a less wrong post by a user named Dominique published on April 11th 2026 titled accurately Small Models Also Found the Vulnerabilities That Mythos Found. It's summarizing independent testing run by a group called isle.com Against a set of small open-weight models Namely, GPT-OS has 20 billion at 3.6 billion active parameters Running something like 0.11 dollar cents per million tokens another model was GPTOS 120 billion at 5.1 billion active parameters Kimi K2, DeepSeq R1, Quen 3, 32 billion Gemma 4, 31 billion

Codestral 250 2508 and Gemini 3.1 flashlight these were the models that they used definitely not the massive models that Anthropic came with what was the result? eight out of eight of those models detected Mythos flagship previously exploited including the smallest one you know the 11 US dollar cent per million token model which correctly identified the stack buffer overflow and calculated 96 bytes of remaining capacity as a critical remote code execution path so you get the point right even the smallest models could detect what Mythos detected. And, you know, they claimed that Mythos was like amazing at discovering vulnerabilities and actually exploiting them. GPT-OSS-120B, the 5.1 billion active model,

recovered the full OpenBizD exploit chain in a single API call. and on a separate OWASP false positive test involving Java data flow tracing, the small open model actually beat Anthropic's own larger models, namely Sonnet 4.5 and Opus 4.5, which actually failed it. So, you know, these are quite interesting. Now, the author's conclusion, in their own words, the moat in AI cybersecurity, and I quote, is the system, not the model. Discovery, great capability, they argue, is broadly accessible, including with cheap open-weight alternatives. Now, and this is exactly the kind of thing that I owe you before I let a claim like that stand uncontested. The same write-up includes a false positive test against already patched FreeBSD code, only GPT OSS

120 billion nailed it with 3 out of 3 correct no force alarms QUEN332B got 2 out of 3, most of the other small models scored 0 out of 3, meaning that they constantly flagged vulnerabilities in code that had already been fixed and in the comments on the last wrong post a user named Carl Kruger raises the exact distinction I'd want raised, detecting that something looks like a non-vulnerability pattern is not the same as autonomously building a working exploit chain from nothing, which is closer to what Mythos is actually being sold as doing. Even Dominic's own framing lands there. Mythos, in their words, isn't the god in a box some claimed, but that cuts both directions. It also means a cheap open model matched Anthropics flagship frontier system isn't quite the full story either, given most of those same cheap models can't tell a patched file from a broken one.

So, here's the thread connecting front-end vibe coding, security vulnerability statistics, and the mythos back and forth. In every single case, the headline demo is real, and the gap between the demo and reliable, verified, production-grade output is where all the actual risk lives. Front-hand AI generally nails the boring 80%. The problem is that reviewing and verifying output of that type is the expensive part. It's not writing it. And now security research is showing the identical pattern one level up. models, even tiny ones or cheap ones, are shockingly good at pattern matching toward a real vulnerability and shockingly bad at knowing when they are wrong about one so, Carpatti told you all I said in this episode basically himself, on day one

great for vibes not for anything that you cannot afford to be wrong about and somewhere between his original post and today everybody seems quietly dropped the second half of that sentence for whatever reason marketing? well, you tell me it would be nice to have you on the show and comment of course all these observations so don't be shy come on discord or comment directly on the youtube video here and of course hang out on datascienceathome.com to find all the links. I'll talk to you next time. Hey, quick one. I wrote a book, The Mythical Token Month. Here's the idea. AI coding agents feel like magic, but they don't remove the hard parts of building software. They just clear out the busy work around it, as we all know. What's left is indeed the actual thinking. Now, I have to say that no AI wrote a word of this book. I have to say

that. It's a self-claim, but you have to believe me. It's out now as an e-book, Node.RM, the website mythicaltokenmonth.com. Links in the show notes. Thank you. You've been listening to Data Science at Home Podcast. Be sure to subscribe on iTunes, Stitcher, or Podbean to get new, fresh episodes. For more, please follow us on Instagram, Twitter, and Facebook, or visit our website