Shut down prompt injection attacks targeting AI email agents and tools

Prompt injection is an attack where malicious instructions are embedded in content that an AI model processes to trick it into following the attacker's instructions instead of the legitimate user's or system's. It's roughly analogous to SQL injection, but targeting AI models instead of databases.

Direct injection is when the attacker attempts to manipulate an AI agent or tool into taking a specific action. Indirect injection is when the malicious instructions intend to confuse an AI or traditional ML security system into making an incorrect verdict about an email.

An attacker could send a phishing email containing injected instructions targeting an AI email assistant. If the assistant processes the email and acts on it (summarizing, replying, filing), the injected instructions could redirect its behavior to exfiltrate other emails, send replies on the user's behalf, or manipulate its outputs.

Not currently. There's no complete technical solution. Mitigations include input sanitization, privilege separation, keeping sensitive actions behind human confirmation steps, and limiting what data the model can access. It's treated more like a risk to manage than a problem to fully solve.

Retrieval-Augmented Generation systems pull external documents into the model's context to answer questions. An attacker who can influence what documents get retrieved — by poisoning a knowledge base, a shared drive, or a webpage — can inject instructions that appear to the model as legitimate context, hijacking its responses or actions.