Lenny's PodcastSander Schulhoff: Why AI guardrails fail every red team test
How prompt injection and jailbreaks bypass state-of-the-art guardrails; agents that send emails or touch databases turn every bypass into real damage.
CHAPTERS
- 0:00 – 6:04
AI guardrails don’t work: the core claim and why it matters now
Sander opens with a blunt thesis: commercial AI guardrails are easy to bypass and often marketed with misleading claims. Lenny frames the urgency—major incidents may be limited so far mainly because adoption and agent autonomy are still early.
- •Guardrail providers’ “we catch everything” claims are called out as false
- •AI security risk is growing as systems gain power, not because AGI arrives
- •“You can patch a bug, but you can’t patch a brain” sets up the central difficulty
- •Early adoption—not strong security—may explain the lack of catastrophic attacks
- 6:04 – 8:27
Who Sander is: prompt engineering → red teaming → benchmark datasets
Lenny introduces Sander’s background and why his perspective is credible: years of AI research, early work in prompt engineering, and running large-scale red teaming competitions. Sander describes creating major datasets used by frontier labs and enterprises to evaluate prompt injection and jailbreak robustness.
- •Created Learn Prompting guide and moved into AI security research
- •Ran early and now-largest genAI red teaming competitions
- •Open-sourced a large prompt injection dataset; recognized at EMNLP
- •Works with frontier labs and Fortune 500s on benchmarking/defenses
- 8:27 – 10:03
Jailbreaking vs. prompt injection: the two main attack families
Sander defines the difference between jailbreaking (user vs. model) and prompt injection (user manipulates an app/agent to override developer instructions). The chapter clarifies the role of system prompts and the “gray area” between categories.
- •Jailbreak: direct manipulation of a chatbot to produce disallowed outputs
- •Prompt injection: attacker gets model to ignore developer/system prompt in an app
- •System prompt/developer instruction is the key differentiator
- •Real-world apps/agents create more attack surface than standalone chat
- 10:03 – 17:56
Breaches in the wild: agents recruiting agents, chatbots going rogue, and code execution exploits
Lenny and Sander walk through concrete incidents that show how prompt injection becomes dangerous when systems can take actions. Examples range from social media bots to tool-using apps that execute generated code and exfiltrate secrets.
- •ServiceNow example: second-order injection recruits more powerful internal agents to do CRUD + send emails
- •Remotely.io Twitter bot hijacked to post threats/hate, harming the company
- •MathGPT: injected prompt causes code generation + execution to exfiltrate OpenAI API key
- •Why these are prompt injection: attacker overrides app intent via malicious input
- 17:56 – 19:44
From “bad text” to real harm: agents, browsers, and robots raise the stakes
The discussion shifts from embarrassing outputs to tangible damage as AI systems gain permissions and autonomy. Sander highlights how agentic systems and VLM-powered robots amplify the consequences of the same fundamental vulnerability.
- •Agents can leak data, spend money, modify records, and trigger workflows
- •AI browsers/robotics introduce new channels for indirect injections
- •Prompt injection against physical systems could cause injury in worst cases
- •The threat is near-term and capability-driven, not hypothetical AGI
- 19:44 – 21:10
The AI security industry landscape: monitoring/compliance vs. guardrails/red teaming products
Sander distinguishes frontier labs’ research efforts from the B2B “AI security industry.” He argues monitoring/observability and governance can be useful, while guardrails and automated red teaming are often oversold as solutions to unsolved problems.
- •Two worlds: frontier lab research vs. enterprise AI security vendors
- •Useful categories: monitoring/observability, compliance/governance
- •Questionable categories: automated red teaming products and guardrails
- •Many vendors bundle guardrails + automated red teaming as default offerings
- 21:10 – 28:33
How guardrails and automated red teaming are supposed to work in enterprises
Sander explains the standard enterprise buying story: a CISO sees scary red-team results, then purchases guardrails that filter inputs and outputs. Lenny clarifies the typical architecture (guardrail before and after the model).
- •Automated red teaming: models that generate attacks against other models
- •Guardrails: classifiers to block malicious inputs/outputs pre- and post-model
- •Typical pitch: audit → alarming findings → buy guardrails to “fix” it
- •Common deployment: input filter + output filter around the core model
- 28:33 – 31:04
Why the approach breaks down: red teaming always finds something, and guardrail metrics mislead
Sander argues automated red teaming often provides little novel insight because all transformer-based systems remain vulnerable, and enterprises often use the same off-the-shelf models. He introduces adversarial robustness and attack-success-rate metrics, then explains why static measurements don’t reflect real attacker behavior.
- •Automated red teaming will reliably find failures in any current model
- •Results often aren’t novel when enterprises use the same frontier models
- •Adversarial robustness and ASR (attack success rate) as key measurement terms
- •Static evaluations underestimate risk; adaptive attackers change tactics
- 31:04 – 39:18
The ‘infinite’ attack surface: adaptive attacks and humans breaking every defense
Sander’s main technical critique: the number of possible attacks is essentially the number of possible prompts—astronomically large—making “99%” claims meaningless. He emphasizes that adaptive evaluation (especially human attackers) quickly defeats state-of-the-art defenses, including guardrails.
- •Attack space ≈ all possible prompts; effectively unbounded
- •“99% effective” is not statistically meaningful against near-infinite possibilities
- •Adaptive evaluation is the gold standard; humans are highly adaptive attackers
- •Research/competitions show humans break defenses in ~10–30 attempts
- 39:18 – 44:42
Misaligned incentives and the ‘patch a brain’ problem: why this isn’t classical cybersecurity
They explore why the issue persists: labs prioritize capability to win the market, and many buyers misunderstand AI as if it were patchable software. Sander criticizes prompt-based defenses as especially weak and warns that the industry’s marketing can create dangerous overconfidence.
- •Frontier labs focus on capability because it sells; security lags in the race
- •AI is not patchable like software: ‘can’t patch a brain’
- •Prompt-based defenses are called ‘worst of the worst’ and well-studied failures
- •Marketing and fabricated stats can lead to overconfidence and bad deployments
- 44:42 – 48:47
Practical guidance: when you can ignore it, and when you absolutely can’t
Sander offers a pragmatic decision rule: read-only chatbots with no actions and no sensitive access may not warrant heavy defenses. The real danger appears when systems can take actions or access privileged data; then classical security practices (permissions, isolation) become central.
- •If it’s a simple chatbot with no actions, impact is mostly reputational
- •Any data the model can access can be forced to leak; any action can be forced to execute
- •Treat AI as an untrusted component; constrain its blast radius
- •Focus on permissioning and deployment architecture, not content filtering
- 48:47 – 1:09:18
Containment wins: sandboxing, least privilege, and the CAMEL framework for agents
The conversation moves from filtering content to controlling capabilities. Sander explains how sandboxing (e.g., containerizing code execution) can neutralize certain prompt-injection paths, and introduces CAMEL as a permission-scoping approach that limits what an agent can do based on the user’s request.
- •Example fix: execute model-written code in a container to prevent secret exfiltration
- •Indirect prompt injection risk rises when agents read untrusted sources (email/web)
- •CAMEL: infer minimum permissions per task and grant only those permissions
- •CAMEL helps when read/write can be separated; struggles when both are required
- 1:09:18 – 1:12:06
Education over tooling: building the ‘AI security + cyber’ skillset
Sander emphasizes that awareness and training prevent the worst deployment decisions. He argues the most valuable work sits at the intersection of AI security and classical security, and promotes structured education as a more reliable near-term investment than guardrail products.
- •Awareness changes deployment choices before incidents happen
- •Teams need cross-disciplinary expertise: AI behavior + security engineering
- •Education and internal capability-building outperform “plug-and-play” promises
- •Course plug: practical training on AI security, red teaming, and deployment risks
- 1:12:06 – 1:21:58
What frontier labs can do: adaptive evals, deeper adversarial training, and harder agent problems
Sander is pessimistic about near-term breakthroughs but suggests directions: better measurement via adaptive/human evals, adversarial training earlier in the pipeline, and potentially new architectures. He notes that preventing harmful content is easier than securing agents that must sometimes take risky actions like sending emails or browsing.
- •Little ‘meaningful progress’ overall; improvements often overstated via static datasets
- •Prioritize adaptive evaluations and human red teaming for realistic measurement
- •Potential research directions: adversarial training earlier; new architectures
- •Agent security (indirect injection) is harder than ‘never produce CB(R)NE’ style rules
- 1:21:58 – 1:25:48
Market correction and what happens next: guardrail hype fades, real-world harms rise
Sander predicts a near-term correction in the AI security vendor market as customers realize guardrails don’t deliver the promised protection—especially with free/open-source alternatives. He expects more real-world incidents as agentic systems become capable enough to cause immediate operational and financial damage.
- •Prediction: guardrails/automated red teaming vendors face revenue and trust erosion
- •Acquisitions may look misguided as efficacy becomes clearer
- •Robustness won’t be ‘solved’ soon; this is a long-standing research field
- •As agents proliferate, tangible harms (data loss, money loss) likely increase
- 1:25:48 – 1:32:40
Closing recommendations: stop publishing jailbreaks, avoid false confidence, think before deploying
In the final segment, Sander discourages purely offensive jailbreak research that adds attack recipes without improving defenses. He reiterates that guardrails can create a dangerously inflated sense of safety, and urges rigorous deployment thinking, capability restriction, and expert involvement.
- •“Don’t write that jailbreak paper”: attacks are already abundant; focus on defense/deployment
- •Human-in-the-loop helps security but may not match where products are headed
- •Core warning: guardrails create overconfidence as agents/robots get deployed
- •Call to action: evaluate prompt-injection exposure, permissions, and whether to deploy at all
