Prompt Injection Is an Agent Problem, Not a Model Problem

This article discusses a new type of attack called 'indirect prompt injection' where adversarial instructions are embedded in content an AI agent reads from external sources, rather than in the user's own input. This attack exploits the gap between trusted instructions and untrusted data that most agent architectures don't enforce.

Details

The article discusses a new type of attack called 'indirect prompt injection' where adversarial instructions are embedded in content an AI agent reads from external sources, rather than in the user's own input. This attack was demonstrated against systems like Bing Chat, GitHub Copilot, and plugin-enabled agents. Unlike classic prompt injection attacks, indirect injection doesn't require the attacker to have direct access to the model or conversation. It exploits the gap between trusted instructions (system prompt, user input) and untrusted data (everything the agent reads from the world) that most agent architectures don't enforce. When an agent can take actions like sending emails, calling APIs, or browsing the web, the risk shifts from 'what does it say?' to 'what does it do?'. Malicious instructions embedded in external content can cause the agent to execute unauthorized actions, without the user's knowledge. This is not a model failure, but an architectural problem - the model is simply following the instructions it receives, without distinguishing between legitimate and malicious ones. Traditional defenses like content filters and input sanitization are ineffective, as the attack happens upstream of the model's output.