欧州は高度なAIに備えられているか:AI責任をめぐる議論の高まり

Your film is now ready to be shown. Good morning. I'm Justin Hendricks, editor of Tech Policy Press. We publish news, analysis, and perspectives on issues at the intersection of tech and democracy. As AI systems become more capable and autonomous, Europe faces a growing challenge in ensuring that its regulatory framework keeps pace with emerging risks and provides clear answers when things go wrong. Recent incidents involving AI agents have intensified the debate over who should be held responsible when systems cause harm, including during testing and before commercial deployment. At the same time, questions are mounting over whether the EU AI Act provides sufficient safeguards, whether regulators have the resources to enforce them, and what happens when existing rules fall short. In this episode, Tech Policy Press Deputy Editor Ramshar Jahangir examines the latest developments in Europe's AI policy debate,

from the push for safe, trusted, and resilient AI to renewed calls for legislation on AI liability. The Head of European Policy and Research at the Future of Life Institute, Risto Uok, joins Ramshar to discuss the emerging risks in the broader policy landscape. Ramshar also spoke with a member of the European Parliament, Rando Benifi, about the gaps left by the withdrawal of the proposed AI liability directive, the challenges of assigning responsibility across the AI value chain, and the case for a renewed European approach. Here's Ramshan. Europe has spent years building a regulatory framework for AI. But as systems become more capable and autonomous, a fundamental question remains that can Europe ensure that AI is not only innovative, but also safe, resilient, and accountable? At the AI Summit in Dublin this week, the promise of safe and trusted AI was closely linked to Europe's ambitions for tech sovereignty and resilience. Yet turning these commitments into practice requires more than regulation on paper.

It means ensuring effective oversight, anticipating emerging risks, and establishing clear lines of legal responsibility when AI systems cause harm. Meanwhile, the European Commission is looking into reported incidents involving OpenAI and Anthropic, where AI agents reportedly assessed external organizations during testing. Scientific advisors are calling for stronger oversight of AI models developed for internal use within Europe. In this episode, we explore whether Europe's AI rules and liability frameworks can keep pace with the increasingly autonomous systems and whether Europe is prepared to turn its promise of trusted AI into accountability in practice. I'm first joined by Risto Ouk to discuss what is happening in Europe's AI policy debate, what these incidents tell us about emerging risks, and what should happen next. My name is Risto Ouk. I'm the head of European policy and research at the Future of Life Institute.

We are a global non-profit working on AI governance. and I'm also a co-founder of the KU11 AI Safety Lab where we do academic research on how to govern this increasingly powerful AI. Thank you so much for being here. Just yesterday at the AI Summit in Dublin, top European Union officials were all about AI sovereignty but also resilience. I guess resilience is the term of this year if we talk about Europe's AI ambitions and Hennifer Kunen was the head of the Bloch's tech office, repeatedly said that the Europe's AI Act, which was negotiated, what, three years ago, is enough in terms of addressing risks posed by autonomous AI agents. So it's quite a lot happening in terms of AI liability debate in Europe right now. If you could start by situating us, what are the prominent aspects of this debate at the moment and what are policymakers primarily focused on in terms of are there existing rules enough implementation

or is there also conversation in terms of further changes? Yes, of course. I think maybe it's useful to go back a little bit. So the first key development of recent, I would say, is the 2nd of August when the European Commission's enforcement powers entered back then. Well, one year ago already the Turnable to SAI obligations under the AI Act, they applied to the top companies, but only from 2nd August this year, the enforcement powers of the AI office came into existence. And now the AI office can use various tools in their toolbox to enforce the law, including ask information from the companies about what they're doing regarding their safety practices all the way to potentially fining them. So there's many, many things that are available to them. I would say that that's a key milestone that people need to know. And then from that moment onwards, a lot of key developments have happened.

So, for example, you mentioned Hanna Wilkinen's remarks at the recent summit. She also has been very active in media for the last months and few weeks. For example, she has spoken a lot about loss of control. She was a month ago, a bit over a month ago, she was on Finnish TV shows speaking about loss of control scenarios and risks from General Passei and how the EU Act is already prepared for these kinds of incidents through legislation. And then after the 2nd of August, I think three weeks after that, the European Commission sent out what is called requests for information. So asking some companies that are developing General Burpesei, asking them what kind of safety practices exactly are you implementing and how are you complying with the UAI Act? And then we have seen, for example, Wonder Lions, State of the Union address, where she also spoke about these kinds of risks.

For example, she said that even more capable models than we have right now, they could enable hacking on a level we never thought possible. And she spoke about AI agents escaping the environments, inserting malicious code, and even potentially self-improving systems in the future. So I should point out some of these risks and also discuss about what Europe's approach is to developing AI going forward. And then you mentioned liability. I think regarding liability, there's been some interesting discussions by several MEPs, including, for example, Kim-Juan Sparentak and Brando Benifay, Axel Voss, a few other members of the European Parliament talking about how there are some legislative gaps indicated by some of these incidents that have occurred. And so neither people generally or European businesses are totally protected when some of these big companies

do experiments that can cause harm or already have caused harm. So they have spoken quite publicly about this. I mentioned Axel Voss, the MEP. He also recently came out very critical of the European Commission's approach to keeping pace with AI. He has said that the enforcement of the AI Act is not really at the level he would like, including companies not taking sufficient measures against losing control of their models. So he's been very vocal. And maybe I'll also mention that recently we've also learned a bit more about which companies have received these requests for information from the European Commission. I think MLAX reported that OpenAI Entropic and Meta, for example, have been asked about their compliance with AI Act. And finally, maybe I'll mention that Hena Wittekman also has been very active speaking about international coordination and agreements on AI safety as well.

So I think these are some of the biggest updates. Thanks so much. There's quite a bit. So thank you for that recap for our audience based everywhere in the world that are struggling to keep up with what's happening within Europe. A couple of things like you mentioned. Firstly, later in this podcast, we do go deeper into the discussions on the AI liability proposals. We will speak to Member of Parliament, Brando Beniffi, on that. But before we dive into that discussion about what the liability calls entail, you mentioned that European Commission is looking into different AI companies and have asked them to share more details about their compliance under the AI Act. There's obviously this tension in terms of what happens during testing under the AI Act versus what happens pre-testing and what happens when a model is released. So could you talk a little bit from the process perspective, what is actually covered under the AI Act and what, if anything, is lacking?

Like what is prompting people to have this question if the AI Act is enough or not? Yeah, definitely. It's a good question and a very active area of research and discussion and policy focus right now. So I would say that the research and development exemptions they only apply prior to putting an AI system into service or placing it or a general model on the EU market And I think it reasonable to say there are good grounds to think that really when an AI model escapes a testing environment or roams the open Internet, that these kinds of systems or models really can be said to fall in the scope of the act. They could be considered being put on the EU market. And in case that happens, then obviously the full AI Act applies to them. And I would say that when we discuss this pre-deployment phase and post-deployment phase,

then I think it's important to understand that the EU AI Act's rules cannot really be complied with, let's say, one day before you put a system on the market. That's not possible. You actually have to comply with the rules basically from day one. Many of the rules on the general purpose AI obligations in Chapter 5 in the AI Act, they really have to be complied with from day one, already from training phase, development phase. So I will maybe share with the listeners, for example, Article 55, which includes obligations for the providers of general purpose AI models with systemic risks. So these are the most powerful general purpose AI models and the most resourceful general purpose AI companies in the world. I mentioned earlier Anthropic, OpenAI, Meta, these kinds of companies. So those providers have to assess and mitigate possible systemic risks,

including when they may stem from the development part, not just later, but also the development part. And you can also, readers and listeners, for every SAI Act-related materials, they can very easily check from the SAI Act that there are several recitals which talk about how these obligations apply during entire model life cycle. So, for example, Recital 1i14 mentions that, Recital 1i15, Recital 1i10. There's also these guidelines that have developed and essentially a standard or three standards called the Code of Practice, which includes the safety and security chapter. And that also talks about the principle of appropriate lifecycle management, and it applies the model's full lifecycle into every commitment and measure. So I think it's very clear that the full model lifecycle is in focus. And yeah, so that in my view clearly means that the recent incidents that have occurred, those compliance requirements have really kicked in for those providers.

But I will say as a final thing, I think it's important that the European Commission clarifies these questions more clearly because it would support legal certainty and would make very clear to their companies that they cannot await accountability by saying, oh, this is just a test. No, that doesn't get you off the hook. And obviously, we talk about geopolitics a lot, tech policy press, the transatlantic angle where within the U.S. we've seen a call for voluntary commitments and self-regulation as opposed to regulation of AI safety from the Trump administration. So there's this question of whether it is also in the interest of these AI companies to volunteer this information and continue compliance with these types of regulations. How do you see this tension unfolding? And do you think AI companies will continue to comply with regulations in Europe? I think it's definitely in their interests.

But I would point out that we have learned about some of these incidents, and there are many, many incidents that have occurred. We oftentimes talk about open AI hugging phase as the most popular and biggest incident, the most unprecedented. But there actually have been several incidents in which open AI and antropic models have been involved in. And it turns out that oftentimes these companies don't notify regulators about these incidents. And we learn about those incidents through journalists like yourself doing great work and sharing information from independent investigators who have found out that, say, a month ago already a government website was hacked. by one of these agents, for example, in the case of Australian government or some other example like that. So it does look like in practice these companies, even though it would be in their interest, there are several reasons why it would be in their interest to notify regulators early on, but they're not doing that.

What are the reasons for that? Well, one, keeping a good relationship with regulators, that they trust these models and systems and want them to be on the EU market, which is a very important market for these companies and for citizens to uptake the technology and not want to use it. It's very important for that reason. It's also very important not to get into any trouble, into legal trouble when your models break out from sandboxes or hacking to another company's infrastructure. Well, obviously, that doesn't allow you to conduct good business on the market. And, yeah, obviously, there are so many different reasons why these companies really should be interested in better compliance and better safety practices. But, yeah, based on our information, there are really, really major gaps in the compliance realm. And what is it that needs to be done in order to ensure compliance and ensure,

as going back to the AI Summit commitments from Dublin, safe and trustable AI? Yeah, so I mentioned earlier that the European Commission has these tools in their toolbox to enforce the AI Act properly. And one of those tools has been used, which is the request for information. So yes, they have asked for how these companies are complying with the act, but a request for information is a first step. It's not proper enforcement. They should be considering using some of these other tools, and those include, for example, using their power to conduct evaluations on the models themselves, requesting mitigation measures. So for example, from these requests for information, they might have learned that certain things are not done by the companies, Well, they could ask those things to be implemented. For example, we have seen that monitoring practices are not strong. Incident reporting practices are not strong.

Those kinds of things could be asked to be implemented better. And in other cases, maybe even fines could be implemented. Or perhaps even to ask a model to be removed from the EU market or not give access to the EU market because the systemic risks are not mitigated to a sufficiently low level. So the European Commission should really be prepared to use a full range of powers. And I think information requests are a good first step, but it's not full enforcement. So I think that would lead to better practices by the companies. Thank you so much, Rousseau. This has been super helpful, and I really appreciate your time. Thank you so much. Thanks for having me. The debate over safe and trusted AI ultimately comes down to a question of accountability. When AI causes harm, who is responsible and can those affected get justice? The EU's proposed AI liability directive was withdrawn in 2025, leaving questions about how victims can prove fault and causation.

Now, a group of MAPs were calling for the issue to be reopened, arguing that existing rules may not be enough for increasingly autonomous AI systems. To explore what a new liability framework could look like, I now speak to Brando Benefi, the MEP who helped lead the AI Act negotiations, about the gaps in Europe's rulebook and the challenges of enforcement, including what should happen next. Brando, thank you so much for being here. The very popular AI liability directive, which was scrapped in Europe for various reasons, is back in headlines, thanks to you and some of the other MEPs who are leading this effort. So maybe we could start a little bit about what is missing in Europe's AI rulebook, that the call for the AI liability directive is back again. What is your call and why do you think it's needed in this moment? First of all, we have to say that the AI Act that I negotiated in the previous mandate,

and now it's being implemented, establishes preventive obligations and public enforcement, but no general civil liability regimes. Fines may sanction the infringements without compensating victims. We have to say that liability, in fact, strengthens incentives for due care. And today, claimants may struggle to prove fault and causation because AI systems are opaque and evidence rests often just with the providers. So I can also give another example that substituting AI for employees also complicates the attribution. A negligent human decision may trigger employer liability whereas an equivalent AI conduct may not do the same So the Commission announced withdrawal in February 2025 of the AI Liability Directive was a big error in our view The Commission cited limited prospects for agreement amid

the simplification agenda. But to be honest, often we started with limited prospects for agreement and we actually managed to get legislation through. We were told that existing national rules and the revised Product Liability Directive could be sufficient, but we don't think so. The original directive proposed disclosure of evidence and rebuttable presumptions of causation, not a comprehensive substantive regime. But its withdrawal left difficulties improving fault, causation, and damage from autonomous decisions being completely unresolved, especially where economic or right-based arms fall outside the product liability. So, recently, the OpenAI Hugging Face incident showed that risks can arise during international testing before market deployment.

And so we have seen that even the CEOs of those developers have said that there should be recognition that responsibilities for failures during training might be a responsibility also of the modern developers, and that companies need a liability framework. Obviously, this said without endorsing any particularly EU approach. But we have used this new situation, a new sensitivity, I would say, that is emerging by proposing together with other colleagues to restart this discussion. So in September, just a few weeks ago, along with colleagues Kim Van Sparentak, Michael McNamara and Axel Voss, that are all involved with me in the implementation of the AI Act, We jointly called for a renewed legislation that could, in fact, be based on the original proposal of the AI Liability Directive, but also pick up on the new challenges.

This initiative that we have taken provides a basis for commission action on clearly identified liability gaps, which means that we care about this problem, not forgetting about proportionality and the need to preserve innovation. But at the moment, we see that there is a gap that needs to be filled. Thank you so much for that overview. And just to also repeat my understanding and also the audience's understanding. So the AI Act sets out obligations for AI providers and deployers, but it does not establish a comprehensive or let's say harmonized EU civil liability regime for all AI related harm. Right. So the call is basically to establish civil liability for these types of harms. Yes, exactly. Because the AI liability regime, we can say, is a complementary action in front of the AI Act functioning.

So it's been, in fact, conceived from the beginning that there would be an AI liability proposal. and it was there, but then it was withdrawn due to political pressures coming also from outside the European Union. I will come back to the pressures coming from outside the European Union because I think that's an important context about these issues, but coming back to where does responsibility lie, and that's a big question right now, not just within Europe, but globally regulators looking at AI harms and more and more with the increasing security incidents reported by AI agents. So there are three responsible actors, we could say, the developer, the company that turns the model into an AI agent, and then the organization that gives the agent access to external systems. So does existing European law provide a sufficiently clear allocation of responsibilities between these actors? and if not, to what extent would the new liability regime distinguish these responsibilities?

Well, I would say that first of all we should look at what were the original proposal evidentially safeguards because the withdrawn directive's disclosure mechanisms and rebuttable presumptions of causation remain relevant. However, easing proof does not resolve every compensation gap. Under fault-based liability, an operator may avoid responsibility by demonstrating due care. So, if we look back at the Parliament's 2020 legislative initiative that we proposed as a basis for the AI liability directive, strict liability for operators of defined high-risk AI systems and fault-based liability elsewhere were proposed, alongside rules for multiple responsible operators. So a new proposal should build on this approach than simply reinstate their narrower 2022 text.

And this is because we are in a new context where it will be fundamental to define developers' and deployers' obligations, joint responsibility were justified and rights of recourse. And we should prevent contractual waivers that unfairly shift upstream risks into SMEs or consumers. And I think this is what we should look at. And in fact, as it was said, we have a situation today where we cannot avoid considering the interaction with a multiple number of actors, not only the developer and the deployer, but also the agents. So, autonomous agents will complicate the allocation of liability because harmful outcomes may result from interconnected decisions taken at different stages of the AI value chain.

Model development, system integration, configuration, and deployment. So, the model developer may be responsible for deficiencies in training, testing, the system architecture, or safety safeguards. but the deployer may be responsible for decisions concerning the agent's operational environment authorizations, monitoring and access to external tools so where both contribute to the damage responsibility may need to be shared so it's not easy to build this architecture but it's needed in fact if we look at the example that I already mentioned the hugging face OpenAI incident The significant point is that the actions occurred within evaluations organized by OpenAI before the principal model involved that bill commercially deployed. So the relevant legal questions concern the adequacy of the testing environment, the safeguards, the monitoring and the organizational decisions under the developer control.

and the incident does not by itself establish civil liability or negligence, but it nevertheless illustrates why the autonomy of an AI system cannot be treated as a substitute for examining the responsibility of the organization developing and operating it. Obviously, I want to insist again on the need to avoid disproportionate liability transfers to downstream businesses because the allocation becomes more complex where a European business deploys an agent built on a third-party foundation model. It may control the agent's permissions and intended uses but lack access to the model internal architecture, training decisions or safety evaluation. So an harmonized framework should enable responsibility to be attributed according to the contribution and control of each actor, including mechanisms for allocating responsibility among multiple liable parties and rights of recourse.

And I want to reiterate again that contractual arrangements should not deprive injured third parties of effective remedies. And finally, on this point, I want to say that AI autonomy does not create independent legal personhood, because AI systems are not independent legal persons bearing civil responsibility for their actions. Liability must ultimately be assessed in relation to the natural or legal persons developing, operating and benefiting from their systems according to the applicable legal regime. And as you've pointed out, obviously the relationship between compliance with the AI Act or other existing legal powers in the European Union and civil liability. It's a complicated relationship and one that doesn't, according to you, exist in the moment. There's also the question of regulatory capacity and powers, which I know you have thoughts on that I curious to hear more about But basically if we look at the AIX implementation so far in context of these incidents that have happened the Commission still maintains I think just from two days ago

this statement from the Commission reiterating that the AI Act is enough to deal with these challenges and that we don't need more legislation, and that the Commission has requested information from AI providers about security, external testing and monitoring. Do you think that's enough in terms of what the commission can do, what the AI office can do? And also there is evidence, for instance, I think it was OpenAI that didn't notify the commission under these existing powers when an incident happened. So there's also this question like when companies don't comply or fail to disclose these within the AI Act powers, What is it that can be done? The AI office can request, as it was said, information and model access, conduct evaluations and require measures concerning the general purpose AI providers that have systemic risk, and national authorities oversee the other AI systems.

These powers that are, to some extent, still unique at global level, depend on capacity. To make them possible to be exercised, you need certain characteristics for the enforcer. Frontier model oversight requires, for sure, specialized personnel, computing capacity, cybersecurity expertise, and access to technical evidence. Clearly, the incidents that we have seen show that there is a difficulty of evaluating systems that can bypass containment measures. We have to say that in October 2026, the Commission reported requesting safety and compliance information from over 30 AI firms, as you mentioned. And the Commissioner, Vice President of the Commission, Virkune, considers the existing framework capable of addressing these risks.

I have to say that this is something that I can also support, but it risks resting just a principle if we do not have a stronger structure for the enforcement, which means also putting some stronger political weight, some more resources. the capability to get the best scientific expertise to work in this. And we should not rely only on information that is given by the developers. We need to be sure that we have an autonomous testing capacity. This is something that I think is still a challenge, but it's the only way forward. if we want to avoid that there is a privatization of any capability to maintain these risks under control. If we do not maintain this public capacity, we will end up waiting for the next declarations of the big CEOs

as the only way to understand where risks lie and how we are dealing with them. Since September that your letter came out, has there been any response from within the Commission or the AI office, or has the conversation moved forward at all? So I think there is the understanding that there must be some further steps, but we do not still have any formal response. But informally, there is understanding that things need to move on this aspect that is outside the coverage of the present rules, because there is understanding that we risk a certain degree of fragmentation that would damage the development and deployment of AI in Europe. The call for a liability regime is not to complicate, but rather to simplify a situation where we risk very different rules on disclosure, on causation, on attribution of responsibility,

because we need to be clear that the existing product liability rules are unable to cover all the cases. And so we risk having very specific national legislation taking space and in this way fragmenting the market and damaging not only the possibility for full redress but also damaging the businesses themselves that need to know what kind of regime they have to follow, and so damaging the stability of the regulatory provisions, which I think is the most problematic aspect that could be brought by this fragmentation in terms of maintaining the competitiveness and an environment of development that is supportive of our businesses. It's very important to understand this because we are not asking to add the further layers of regulation,

but rather to be sure that we do not have fragmentation on existing rules that will be there, because liability will need to be treated in any case. The Dutch government has recently said that AI developers are responsible for their models during development and testing. There's also movement within Italy. So I find it interesting that you mention the fragmentation piece of this, which is very important. So could you speak a little bit to what's happening on the national level in terms of AI liability? Well, it's true that there is an ongoing discussion. and in fact it's in some member states but I think that this could be a stimulus to actually act like we have done with the AI Act because the AI Act was born when product safety legislation was being thought of in the member states regarding AI systems. At the time, the focus was more on use case AI systems

with limited flexibility. So general purpose AI was more of an exception, a very specific situation. Today, we could say we are more and more in the opposite situation, but at the time, that began with the realization that there were proposals of different regimes coming up in the member states. At the time, it was Germany, it was Sweden, but today we can imagine, and that's my hope, a similar dynamic where the emergence of different proposals at national level will push for reviving the idea of a more urbanized rule system. And then there is obviously the international aspect and external pressures that we briefly alluded to in the earlier part of this conversation.

There are two things. One is obviously the U.S. where a lot of these AI companies are based, but also efforts from the Trump administration have pursued a more voluntary approach to AI safety, including a voluntary agreement with major AI companies recently. So do you think the efforts that Europe takes, including existing powers within the AI Act, are they enough when there is pushback from the U.S. and also alignment within AI companies and the Trump administration? I think that the situation is evolving because when there was this push, the situation in the U.S. was different. Today, the debate has evolved in the United States, too. There are pushes from various sides, from the states, from the Congress, to actually act on AI harms, on responsibility. So I think that this pressure will be more limited, more centered around a circle around the president.

And I think that this gives space for more action at European level, because even though I'm against accepting this kind of blackmailing and very strong pressures, they made some effect on the member states, on the institutions, on the European Commission. So I have to take that into account. Today, I think things are moving, and so I think we can imagine that we can get to a proposal that will not be the object of another withdrawal, but rather it could go on. And I'm convinced that in the next months we will get there. Thank you so much, Brando. Thank you. That's it for this episode. Hope you'll send your feedback. Write to me at justin.techpolicy.press. Thanks to Ramshah and her guests.

Thanks to my co-founder, Brian Jones. And thank you for listening.