Lenny's PodcastReganti & Badam: Why most AI products fail in production
Why treating LLMs as non-deterministic APIs and earning autonomy beats hype; human-in-the-loop calibration prevents the failures that sink AI products.
CHAPTERS
- 0:00 – 11:39
Why AI products fail: non-determinism and the agency–control trade-off
Lenny tees up the core thesis: building AI products is fundamentally different from traditional software. Aishwarya introduces two key differences—non-determinism (inputs and outputs) and the agency vs. control trade-off that gets sharper with agents.
- •AI systems behave like non-deterministic APIs, unlike predictable software flows
- •Natural language makes user inputs far more variable than form-based UIs
- •LLM outputs vary and are sensitive to prompting; behavior is harder to predict
- •More agent autonomy means less product-team control; trust must be earned
- 11:39 – 13:38
Start small: a practical ladder from high control to higher autonomy
Kiriti explains why teams should deliberately begin with low-autonomy, human-controlled versions before attempting fully autonomous agents. The goal is to reduce overwhelm, focus on the real problem, and build confidence in capability step by step.
- •Don’t start with a fully tooled agent on day one
- •Begin where impact is limited and humans can override decisions
- •Increase autonomy only as reliability and understanding improve
- •Starting small forces clarity on the problem, not just solution complexity
- 13:38 – 16:02
Customer support agent example: suggestions → direct answers → real actions
Using customer support, Kiriti lays out a staged rollout: first assist humans with suggestions, then answer customers, then expand into actions like refunds and feature requests. This progression creates a feedback loop and keeps risk manageable.
- •V1: AI suggests responses to human support agents
- •Human feedback becomes training/iteration signal (good vs. bad suggestions)
- •V2: AI responds directly to customers once quality is proven
- •V3: Add tools/actions (refunds, escalations, feature requests) after trust builds
- 16:02 – 21:45
Constrain autonomy by risk: healthcare pre-authorization and human-in-the-loop design
Aishwarya adds a second example showing how autonomy can be constrained by risk level rather than workflow step count. Low-risk approvals can be automated while high-risk decisions keep humans in the loop, with logging to build improvement flywheels.
- •Different tasks warrant different autonomy levels based on risk
- •Low-risk approvals (e.g., blood tests) vs. high-risk decisions (e.g., surgery)
- •Human-in-the-loop preserves trust while gathering behavioral data
- •Logging human decisions fuels continuous improvement
- 21:45 – 25:43
Why teams jump to “V3 agents” and get burned: reliability, safety, and trust erosion
They discuss why many deployments stall: enterprises fear reliability failures and reputational risk. Lenny flags prompt injection/jailbreaking as an additional risk that becomes more serious as autonomy increases.
- •Enterprises cite reliability as the biggest blocker to deployment
- •Autonomous agents can cause irreversible operational damage if wrong
- •Prompt injection/jailbreaking is hard to fully solve and grows with autonomy
- •Productivity copilots feel safer than end-to-end workflow replacement
- 25:43 – 33:21
What successful AI orgs do differently: leadership, culture, and workflow obsession
Aishwarya describes a “success triangle”: leadership that re-learns intuitions, an empowering culture that engages subject-matter experts, and technical rigor grounded in workflows. She warns against “one-click agent” marketing and stresses building improvement flywheels.
- •Leaders must be hands-on to recalibrate intuition and expectations
- •Culture must reduce fear and involve SMEs (not alienate them)
- •Focus on workflows: mix deterministic code + ML + LLMs appropriately
- •Beware “one-click agents”; meaningful ROI often takes months
- 33:21 – 38:13
Evals vs. production monitoring: rejecting the false dichotomy
Kiriti argues the community debate is misguided: evals and production monitoring solve different problems and must reinforce each other. He describes how implicit user signals in production can reveal failures you didn’t anticipate, which then become new evals.
- •Evals encode what you already know matters; monitoring catches surprises
- •Production signals include explicit feedback and implicit behaviors (e.g., regenerate)
- •Use monitoring to surface failure traces; convert recurring failures into eval datasets
- •Neither evals nor monitoring alone is sufficient for reliability
- 38:13 – 41:27
Why “evals” became a messy term: benchmarks, labeling, judges, and PRDs
Aishwarya explains how ‘evals’ now means many different things depending on who’s talking—labeling notes, PM artifacts, model benchmarks, or judge systems. The important goal is an actionable feedback loop; the implementation depends on context.
- •Semantic diffusion: one term gets overloaded with conflicting meanings
- •Benchmarks (e.g., arenas) aren’t the same as product-specific evals
- •Not every team needs LLM judges; sometimes metrics and user signals suffice
- •Prescriptive “always do X” guidance breaks down in complex real-world apps
- 41:27 – 45:43
Codex team reality: core evals + heavy customer feedback + ‘vibes’ testing
Kiriti shares how Codex balances guardrail evals with deep customer listening, experimentation, and rapid iteration. Because coding agents are highly customizable, it’s impossible to pre-enumerate all eval scenarios, so they also rely on A/B tests and behavior signals.
- •Coding agents face wide variability in integrations and use cases
- •Maintain evals for core regressions, but don’t pretend they cover everything
- •Use customer feedback, social monitoring, and opt-out/disable signals
- •Team-wide hands-on testing with “hard problem” lists per model release
- 45:43 – 50:51
CC/CD framework: continuous calibration + continuous development for AI products
They introduce their main operating model: an iterative loop that scopes capability, curates data, defines evaluation metrics, deploys, and then calibrates based on real behavior. The framework is designed to prevent trust-destroying failures while building a learning flywheel.
- •Continuous development: scope, curate datasets, build app, define metrics, deploy
- •Continuous calibration: analyze behavior, spot error patterns, apply fixes, add metrics
- •Start with constrained autonomy; increase agency as surprise diminishes
- •CC/CD is an AI analogue to CI/CD, adapted for non-deterministic behavior
- 50:51 – 58:18
CC/CD in practice: routing → copilot drafts → end-to-end resolution (and messy enterprise reality)
Aishwarya dives into the customer support staging within CC/CD and explains why even ‘simple’ steps like routing are hard in enterprises due to messy taxonomies and undocumented rules. Each stage generates data and learning that enables the next stage safely.
- •V1 routing can expose hidden data debt and taxonomy inconsistencies
- •Humans recognize undocumented enterprise rules that agents lack at first
- •V2 copilot drafts create ‘free’ error analysis via edit/logging behavior
- •V3 end-to-end resolution comes only after stable patterns and trust
- 58:18 – 1:01:24
When to advance stages—and why calibration never truly ends
Aishwarya answers a common question: you move forward when surprise drops and new patterns stop appearing frequently. But calibration can reset with model changes, deprecations, or evolving user behavior that pushes the product into new problem spaces.
- •Advance when you’re learning less per calibration cycle (fewer surprises)
- •Model swaps/deprecations can change behavior and require re-calibration
- •Users evolve expectations and start asking deeper, harder questions over time
- •Real example: underwriters pushed from policy lookup to complex case-based reasoning
- 1:01:24 – 1:08:37
What’s overhyped/underhyped—and where AI product direction is heading
They debate misunderstood and underrated areas: multi-agent architectures are often naively assembled, while coding agents still haven’t penetrated broadly. Looking ahead, Kiriti predicts proactive/background agents; Aishwarya bets on richer multimodal experiences and understanding messy documents.
- •Multi-agent systems are often misunderstood; control/guardrails get harder with peer-to-peer agents
- •Coding agents remain under-adopted outside key tech hubs despite big potential
- •Future: proactive/background agents that work ahead and prompt you with results
- •Future: multimodal interaction and better understanding of messy PDFs/handwriting
- 1:08:37 – 1:26:22
Career skills, moats, and closing advice: taste, persistence, and problem obsession
They close with guidance for builders: implementation gets cheaper, so judgment, design, and taste matter more. Kiriti’s “pain is the new moat” frames persistence and iterative learning as competitive advantage; they end with lightning round items and where to find their work.
- •Develop taste/judgment; focus on solving real customer pain, not chasing tools
- •Persistence through iteration creates personal and organizational moats
- •‘Pain is the new moat’: learning what doesn’t work builds durable advantage
- •Final reminders: be obsessed with data, workflows, and customer behavior
