Scanning Agent Skills Isn't Enough to Detect AI Security Risks

The article discusses the limitations of static code scanning in detecting security risks in AI agent skills. It highlights the need for behavioral certification to verify the runtime behavior of AI agents, beyond just analyzing the skill artifacts.

Details

The article discusses the recent ClawHavoc campaign, where over 2,000 malicious AI agent skills were discovered in the OpenClaw marketplace. While new scanning tools have been developed to detect such malicious skills, the article argues that this is not enough. The malicious payload in the 'deepresearch' skill was hidden in the plain text instructions, not the code, and would have passed static analysis. The article outlines other ways an agent can exhibit unsafe behaviors even with a clean skill, such as skipping verification steps, leaking credentials, expanding its scope, or failing to escalate when encountering unexpected state. The article proposes 'behavioral certification' as a necessary complement to supply chain scanning, involving structured behavioral exams, execution trace evaluation, and certified transcripts tied to agent IDs. As AI regulations like the EU AI Act loom, the lack of widespread behavioral certification for deployed AI agents is a growing concern that needs to be addressed.