Sander Schulhoff: Why AI guardrails fail every red team test

Guardrails (LLM classifiers around your model) can only be tested on a vanishingly small fraction of the effectively infinite prompt space, and human red-teamers consistently bypass them in 10–30 attempts. They tend to create dangerous overconfidence without meaningfully reducing risk for determined attackers.

If your AI is a read-only FAQ bot with no actions and no privileged data, your primary risk is reputational, not systemic. Once an AI can send emails, modify databases, browse the web, or control robots, any action it is permitted to take can be coerced by a prompt injection or jailbreak.

Classical bugs can be patched with high confidence; LLM behavior cannot. Even after fine-tuning or patching specific failure modes, the underlying model remains vulnerable to countless unseen variations of the same attack pattern.

Combine classical cybersecurity with AI awareness: sandbox code execution, strictly scope what each agent can read or write, and assume any accessible data can be exfiltrated. Proper data and action permissioning can neutralize entire classes of prompt injection, as in containerizing model-written code instead of running it on the main app server.

Frameworks like CAMEL infer the minimal permissions needed for a given user request (e.g., only ‘send email’ but not ‘read inbox’ for a simple outbound message) and grant only those. This can block many prompt injection attacks that appear later in untrusted content, though it cannot solve cases where dangerous capabilities (read + write) are legitimately required together.