新議会はAI災害から私たちを救えるか｜Rob Wiblin

We now know the makeup of Congress during what may turn out to be the two most important years for AI that ever occur. Democrats won big time, far more than anyone predicted. They'll have at least 250 seats in the House of Reps, and they'll have 56 out of 100 seats in the Senate, including seats in Mississippi, South Carolina, and Kansas, a place they hadn't won in 100 years. And they won Texas of all places by 11 points, swooping every office in the state. AI wasn't a top three issue in this election, but it was maybe the fourth or fifth most significant driver of the voter backlash. It's a place where government policy and what voters say they want are just clearly massively out of sync, all of which makes it more likely that Congress takes major action on AI during this crucial window in its development. So what's likely to happen? And what are the most likely ways for things to go really well in the next few years? Let's start with the bad news, at least from my perspective. Despite the massive shift in Congress and the role that AI played in it, passing a strong standalone AI bill still won't be at all simple. The first threshold one has to cross and have a shot at passing legislation

is to get 60 votes in the Senate to overcome the filibuster. That is actually probably possible for an AI bill. The issue is a lot less partisan in Congress than you might expect, given Trump's strong stance against pacing the frontier or any binding AI regulation. Among the general public, Democrats and Republicans feel roughly the same about AI, which is to say remarkably negative. 80% of Americans want the government to keep AI safety rules, even if it slows down development, including almost 80% of Republicans. And while 67% of Americans say they think AI is advancing too quickly, just 2% think that it's advancing too slowly. Plus, a number of Republican senators have been leaders in the push to act on AI, including Josh Hawley, John Kennedy, and Marsha Blackburn. And we can guess that many are more quietly sympathetic. But the president has a veto over any legislation passed by Congress. And overcoming that would require 67 votes in the Senate and 290 in the House of Reps. With overall concern at today's levels, that is going to be pretty hard to achieve, or it would require watering down the rules to the point that they're not especially useful. But that is far from the end of the road. There are at least four ways Congress can make progress right away, even with a veto hanging over it. The first is to pass the

common sense measures that are already popular in both parties, some of which might get 70, 80, or even 90 votes in the Senate. For example, today it seems entirely possible that the Tony's government can and is stealing model weights from US AI companies. In a sense, that actually gives them more extensive access to frontier models than the US government itself has. Nobody in Congress likes this situation, and legislation could demand that companies fix it before they keep going training more powerful models. There are probably also bipartisan majorities in Congress for restoring and strengthening export controls to China and law enforcement against chip smuggling, which I think would strengthen the US's negotiating position if it ever does want to push for an AI treaty with China. It's less clear, but there are also indications that there might be congressional supermajorities in favour of things like whistleblower protections for AI staff or mandatory incident reporting to Congress, and also so-called kill switches. As the water level rises, the range of provisions that can attract majority support in both parties is only going to keep growing. Second, there's must-pass legislation, the bills that keep the government funded, like the annual defence bill. These are hard for a president to veto because it means a government

shutdown or no defense budget. So this is where negotiated deals between parties get done. The most obvious thing to try to include in such a deal is proper funding to build out Casey, the likely home of USAI regulation, something that I'll come back to later. The third pathway is the least significant, but I'll mention it anyway. Cabinet secretaries and many other senior officials need to be confirmed by the Senate. So senators can grill nominees for AI-relevant roles and vote down or hold up anyone whose views on AI are badly out of step with those of the Senate itself. The fourth pathway is quite big. It's to hold hearings and collect information. There's so many things that the Senate and the House should demand that company staff answer under oath. For example, what is the full story of the hack where an AI swarm took over an OpenAI research cluster? We're still not being told. OpenAI also says it has evidence that using looped transformers didn't reduce monitorability much. What are all of the experiments they conducted to assess that? Over at Anthropic, what evidence do they have that pushing ahead with more and more recursive self-improvement is safe or not? What models are OpenAI, Anthropic, and Google training right now?

And how much compute are they using? And have these companies conducted any safety tests with results so embarrassing that they didn't publish them? These are all great questions to ask and probably quite fun for members of Congress as well. Between them, these four pathways could get a lot done over the next two years, but they're all about working around the veto, not overcoming it. Unfortunately, I don't think just working around it will be enough. To make it through this, we need a much bigger shift in policy than anything I've been describing. At a minimum, we need to slow down the AI race while models are around the human level to give AI companies breathing room to test the nature and personality of their models check over the work that they doing and hone their alignment techniques Without that extra breathing room it seems more likely than not that we produce super intelligent machines that are misaligned but completely capable of hiding that fact, and that this or some other screw-up leads to total disaster. The companies themselves are begging to be given this time, but as far as I can tell, we don't yet have a supermajority in Congress in favor of rules sufficient to accomplish that. That is a very serious problem because more and more evidence is coming out suggesting that it will be possible to fully automate AI R&D as soon as 2027

and that this will lead to a big speed up in research progress, including a new report from MEDA that landed last week. For that reason, we desperately need California to take big steps immediately as a stopgap measure until Congress can step in, and I hope to cover that in a separate video. But the events that did more than anything else to build support and Congress to act were the Hugging Face hacks and Jacob Coxon's resignation. To change Trump's perspective or get a supermajority in Congress to overcome his veto, my guess is that it will take another demonstration of what AI can do or is willing to do that's at least as shocking to people as that was. That next one shot could take many different forms. At the low end, we could have another hugging face incident, but one that demonstrates long-term power seeking by the AI or that interrupts services ordinary people use. At the high end, we could see AI cyber capabilities disabling critical infrastructure, or a rogue AI that can't fully be shut down, or even an AI-caused pandemic. The days and weeks after such an event could well be the most pivotal ones in AI policy, the critical moment that sets us on a better trajectory or saves us from a much worse disaster later.

Some people should be trying to improve policy today, both for the direct benefit and to build experience and relationships. But if willingness to act can predictably soon be 10 or 100 times greater than it is today, Clearly, many people should also be preparing for that moment. That means being ready to work 100-hour weeks and a bubble thinking through and drafting excellent legislation now while there's still time to think carefully and get it right. Anyone preparing for that moment should expect a very different political environment from the one we're in today. I see three main ways that things can predictably change. Up until now, AI regulation has done everything it can to improve safety while doing as little as possible to interfere with company activities. If a warning shot is severe enough, the public mood may be so furious that that constraint could disappear or even flip on its head. Interfering with model training or deployment or just generally lashing out and interfering with AI companies may come to be viewed by the public or politicians as actively desirable rather than something that needs to be minimized. The second big shift is that the typical legislator in AI regulation discussions isn't then going to know much less about the topic.

So things are going to have much less of a technocratic vibe. And thirdly, the public is going to be a lot more involved and politicians will have to sell the steps they're taking to the general public, which is going to mean boiling it down to a core simple message and communicating why what they're doing is actually going to deal with the problem. My guess is that the simple message more informed groups are going to rally around will be ban superintelligence or ban superhuman AI. It's a simple idea that's easy to explain. It makes intuitive sense that it would help. Unlike banning AI entirely, it's going to seem feasible. And polling shows the US public is already overwhelmingly against the training of superintelligent machines, ones that totally dominate human experts in all domains. It's a sufficiently natural idea that there's already legislation introduced in the House and Senate to accomplish that goal, ban superintelligence. Needless to say, there are massive implementation challenges that people preparing for this moment need to figure out ahead of time, such as what's the best treaty to immediately propose to China to enable both countries to pace the frontier? What specifically are we going to be limiting or banning? And how does the policy shift over time? One promising answer is that rather than limiting training ones, say,

we require AI companies to spend a minimum share of their compute serving inference to outside customers. If you set that minimum to 100%, that's basically a research pause that keeps every AI product running. And it's actually immediately good for AI users who will be able to buy inference at much lower prices. The exit plan would be to gradually turn this constraint down to, say, 70%. But if we get new warning signs, that's a dial that we can turn back up again to pace the frontier more strictly. Effectively, a regulator could set this number in order to achieve whatever speed of AI progress it thinks is desirable, the same way the Federal Reserve sets interest rates in order to keep the economy not too hot and not too cold. This stuff needs careful forethought and legislative drafting involving technical experts familiar with how the frontier companies work. Beyond the big public slogans, there are a few ideas that are too niche to ever be rallying cries, but that I think are worth developing out now and probably worth including in any emergency response. The thing I'm most excited by is the idea to throttle recursive self-improvement using the specific mechanism of prohibiting AI companies from using recent models for their research

or any other work. The rule would be something like, for internal purposes AI companies can only use models that finished training at least say six months ago and which have already been released publicly The companies couldn complain that it ridiculous and makes their work impossible Presumably, it wasn't impossible to do their jobs six months ago. And for that reason, it wouldn't by itself stop AI research progress entirely. But it would definitely slow it down, and it would make it much harder for it to ever become truly explosive, or for alignment to go off the rails very abruptly. That's because any capabilities gains couldn't be fed back into the recursive self-improvement cycle for at least six months, and would also have at least six months for extensive alignment tests of a model before it gets its own hands on the control wheels and is able to design its successor. The policy also has a more significant slowing effect. The stronger and faster recursive self-improvement would otherwise be going, which is exactly the effect we want. If six months turned out to be too little, we could just increase the delay. And if the figure is extended out to one year or 18 months or two years, you're incrementally coming closer and closer to an outright ban on recursive self-improvement.

We ideally don't want to interfere with research aimed at improving safety and alignment in particular. And to that end, companies could potentially apply to Casey for exemptions for particular projects. To give companies less reason to pass off capabilities work as alignment work, the rules could require that any work conducted with the help of more recent models has to be published in full as soon as it's completed, which would mean that any capabilities research would effectively immediately be shared with competitors. This approach does have some problems, some weaknesses. If a company found an older model was misaligned in some way that they hadn't appreciated at the time, they wouldn't just be able to replace it with a new one that they think is more aligned. We'd need to have some workaround to address that. But overall, this really seems like an idea with legs that's worth developing in detail. If Congress is willing to take any significant steps to limit AI company research practices in the near future, this is the most valuable rule that I'm aware of to attach to a must-pass bill. The next most promising idea, at least for me, is banning scarily high, no-chain-of-thought capabilities. In short, you don't ban models that are too capable full stop.

Rather, you ban models that are able to accomplish too much without writing anything down in their chain of thought along the way. If models can't do very much without a chain of thought, it'll be a lot easier for us to monitor them, and they'll have a much harder time scheming against users. A rule like that would force companies to go back and figure out how to design AI with a fundamentally safer architecture that has much less serial depth. Another approach people have suggested is banning excessive use of reinforcement learning to generate super persistent agency, because those RL environments seem to play a very outsized role in causing AI misbehavior, at least currently. Perhaps you could set a maximum amount of RL VR generated data that can be used while in training models. Another possibility is that frontier AI training and research should have to be conducted in fully sandboxed, air-gapped facilities. I don't know if those ideas can be made to work well in practice. We're going to need clever technical people to think hard and figure that out. Those more technical ideas are where I would put my own energy. But alongside ban superintelligence, I expect the slogan with the broadest support

to be something much more concrete. Stop building data centres. Data centres work directly on the ballot in 40-odd places in the midterms. And the anti-side won all but one of those elections, often by a ratio of 3 to 1. And assuming a serious warning shot, preventing more compute coming in line, is a step in the right direction, for sure. It's a great place to start building a broader coalition as well. But if, like me, you're most worried about loss of control to AI, or bio or cyber misuse, or extreme concentration of power, it's probably not the highest leverage thing to push for individually. The thing is, if we're already at the point where loss of control is possible, simply not adding more chips doesn't make the problem go away. At best, it holds it steady. To lower the risk, you need to restrict company research and development practices, like I've been describing. Or you need to start actively disconnecting chips from existing data centers. And you also can't allow chips to be relocated to data centers outside the US. That's not much of an improvement. So to be properly effective, a ban on data center construction would have to be paired with stronger export controls as well.

So data center bans by themselves are top of my list, but they could be one useful part of a broader response package that avoids effectively pouring more fuel on fire while we use other policies to help put the fire out. whatever mix of bans and limits we end up with there's one other piece of the puzzle that would be really easy to underrate which is making sure the government can actually accurately tell how much danger we're still in if you want to pace the frontier of ai development for years rather than just months it becomes necessary to progressively add global coordination and enforcement and compute controls all of which is possible in principle but raises the difficulty quite a bit to keep escalating like that it will be necessary for political willingness to act to keep rising in line with the seriousness of the threat An experience so far shows that simple facts about what AI models can do and the misaligned things they are doing and the sloppy company practices that makes it all possible that has done far more to change minds than anything else. That means it's incredibly valuable for Congress and the executive to get more transparency into AI companies. For instance, companies should be required to immediately share any significant accidents

or warning signs, and Casey should automatically get access to any tests or research results conducted by the companies. And really, they should have their own auditors placed inside the companies, keeping tabs on what they're doing following the embedded bank supervisor model. On top of this, the U.S. government needs its own expert AI body that can make sense of this enormous amount of incoming information and implement regulations that are passed and adapt them to circumstances over time. Unfortunately, an organized government agency with top AI experts can't be thrown together in a few weeks in a crisis, which is why it's urgent that Congress properly fund the body best place to play this role, which is the Centre for Advancing Innovation and Standards for Superintelligence, or KC. There are great people there, but it's just way too small and underfunded relative to the scale of the role that we need it to play. It's getting outclassed, mugged really, by the equivalent agencies in the UK. KC runs on about $15 million a year with just a few dozen staff, while the UK's AI Security Institute has over 100 technical staff alone and about eight times the budget.

Britain just committed another 300 million pounds to it over the next four years, plus hundreds of millions more for new agencies on AI biosecurity, cyber and rogue AI incidents. Adjusting for its economy, the UK is now spending something like 50 times as much building state capacity in AI, which is a massive credit to the UK, but it's also completely nuts. The effect is that in a serious emergency, the US might be forced to delegate a lot of AI risk assessment to the UK government because it hasn't built the capacity to perform core functions for itself. Both parties in Congress and the White House should want to urgently rectify this. Unfortunately, this is exactly the kind of thing that can go into the next government funding bill. Even the White House has asked to nearly double Casey's budget next year, and I think Congress should go much further again. There's another high-level challenge to deal with here. Almost every step that I've described, including a bigger Casey, all of it, hands more power to the executive branch. And that power could be used to strong-arm companies into doing things that have nothing to do with safety, if that's what the executive wants to do. Ideally, we would have built in safeguards against this,

such as well thought through options for companies to challenge decisions in the federal courts if they're capricious or violate the First Amendment or just aren't credibly justifiable on safety grounds. And the details are really going to matter here. It's probably too dangerous to allow companies to get a preliminary injunction to preemptively brock rulings from, say, Casey. But we do want them to be able to object and overturn clearly political decisions promptly within two weeks, if at all possible. In the worst case, if Congress concludes that the executive simply isn't going to enforce rules that it passes, or it's only going to use them to pursue partisan political objectives, Congress may be forced to just play an assisting role, helping states like California and Texas do the heavy lifting, at least for some period. The traitors here are pretty difficult to deal with, and they're another thing that has to be thought through ahead of time, not in the middle of an emergency. Here's a final trap to watch out for. Until now, the expectation has been that when Congress passes a big set of AI rules, it's going to preempt the states and severely limit their ability to regulate AI. By default, though, I think anything like that should be kept out of an emergency bill

after a warning shot of some type. By that point, states may well have strong rules on the books and regulatory systems and agencies that have actually been set up and are functioning. And given the goal is to immediately strengthen safeguards, it would be totally perverse to delete those rules, which could have the net effect of, at least temporarily, loosening controls. The midterm results substantially increase the chance that Congress will act to reduce AI risk, which is great. But it's nowhere near guaranteed that this will happen in time or happen in a really smart way. If you understand how frontier models actually get trained, this work needs you really badly. There are people working on these topics at the AI Futures Project and Secure AI Project, among many others. And some politicians who have deep familiarity with these issues, most notably Scott Weiner, are going to be joining Congress for the first time soon, and they'll no doubt be looking to develop proposals. It's a cliche that when a crisis finally comes, what gets done depends on the ideas that are lying around. It's a cliche because it's mostly true. Political will on this topic is only going to go in one direction. Our job is to make sure that when the moment comes, the ideas lying around are good ones,

ideas that will actually fix the problem. That's something people listening to this show are unusually well-placed to help with. If we pull it off, these might just be the two years during which an AR catastrophe is averted. and on that note I'll speak with you again soon