CHAPTERS
- 0:00 – 2:33
The hidden PR review backlog created by AI coding
Claire frames the core problem: AI makes it easy to generate lots of pull requests, but human review capacity doesn’t scale. This creates a growing queue that slows shipping and wastes attention on low-value reviews.
- •AI-generated PR volume is overwhelming many teams
- •Engineering leaders feel the pain most acutely
- •Review queues become the new bottleneck after adopting AI coding
- •Goal: keep shipping velocity without sacrificing safety
- 2:33 – 3:04
Why you don’t need humans to review every AI PR
She challenges the default assumption that every PR must be reviewed line-by-line by a human. The proposed solution is to put AI in the loop to triage and only require humans where risk is meaningful.
- •The common objection: “we review all human PRs, so we must review all AI PRs”
- •In 2026 workflows, AI can safely reduce mandatory human review
- •Focus human attention on PRs that truly require judgment
- •Use automation to clear low-risk changes faster
- 3:04 – 5:05
Intercom’s approach: AI-approved PRs can be safer (with metrics + compliance)
Claire cites Intercom’s system for scoring and auto-approving PRs, emphasizing they optimized for quality and safety—not just speed. She also explains how auditability and policy alignment can satisfy SOC 2/HIPAA-style constraints.
- •Intercom increased PR throughput significantly by auto-scoring/approving
- •AI-approved PRs were ~5x faster to approve
- •Lower revert rates for AI-written/AI-approved code
- •Compliance is possible with auditable, defensible policies and logs
- 5:05 – 6:05
Technical inspiration: Diff Vader/Rewind-style risk scoring signals
She references another implementation that breaks risk into concrete signals like blast radius and CI status. These references help shape the components of her own agent design.
- •Risk signals can include blast radius, correctness, and workflow checks
- •CI/actions status is part of determining readiness and safety
- •Real-world writeups provide a blueprint for building your own
- •Motivation: remove “boring” low-risk PRs from the queue
- 6:05 – 9:08
Why Vercel Eve: a practical framework for enterprise Slack + GitHub agents
Claire introduces Vercel Eve and explains why it’s her default choice for deploying internal agents. Eve’s directory-based structure, multi-channel support, sandboxing, and managed connectors reduce setup friction.
- •Eve agents are organized as a directory of instructions, skills, and code
- •Built-in channel support makes Slack/GitHub interaction easier
- •Sandbox execution helps for tasks like checking out repos and inspecting diffs
- •Vercel Connectors simplify OAuth/refresh tokens and enterprise integrations
- 9:08 – 10:08
The Codex prompt that kicked off the entire bot build
She shows how minimal the initial Codex prompt was—just a clear objective to review PRs after checks pass, classify risk, and auto-approve low-risk changes. From there, she steered Codex to implement it as an Eve agent.
- •Simple starting prompt: review after checks are green, score risk, auto-approve low-risk
- •Light steering to choose Eve as the implementation framework
- •Few iterations were needed to get to a working repo
- •Emphasis: you don’t need a huge spec to begin
- 10:08 – 11:39
Setup shortcut: using Codex browser automation for GitHub/Slack configuration
Claire highlights that the hardest part is often SaaS setup—permissions, manifests, and app configuration—not code. Her “meta tip” is to use browser automation to click through setup flows while she verifies and handles 2FA.
- •GitHub app + Slack bot setup is permission-heavy and tedious
- •Codex “Chrome browser use” can navigate configuration screens
- •Human stays in the loop for verification and 2FA
- •Great for any complex third-party tool configuration
- 11:39 – 12:10
What the agent does end-to-end: read diff, score risk, approve or escalate
She outlines the core workflow: the agent reads the PR and diff, produces a risk score with evidence, approves low-risk PRs, escalates to humans when needed, and requests changes for blockers. This mirrors a human review process while scaling throughput.
- •Reads PR content + exact diff
- •Scores risk and publishes rationale/evidence
- •Low risk: approve; medium/high: escalate to human
- •Blockers trigger a “request changes” style outcome
- 12:10 – 13:11
How it works technically: GitHub events → Eve channel → sandbox checkout → output
Claire explains the integration flow: a GitHub App triggers on PR readiness, Vercel routes the event to the GitHub channel, then Eve spins up a sandbox to inspect code and run skills/tools. Results include review output plus Slack notifications for the team.
- •GitHub App triggers after PR is ready (post-checks)
- •Vercel GitHub integration forwards events into Eve’s GitHub channel
- •Sandbox checks out repo, inspects diff, runs evaluation skills/tools
- •Outputs approval/status + comment; then notifies Slack
- 13:11 – 14:11
Inside an Eve agent: channels, instructions, skills, and minimal files
She breaks down what’s in the agent repo: a GitHub channel, concise instruction text, a PR review skill, tools for reading PR context and making decisions, plus a Slack notifier. The key message is that the implementation can be surprisingly small and maintainable.
- •Core pieces: GitHub channel + instructions + review skill + tools + Slack notifier
- •Instructions are short (a few paragraphs/bullets) but effective
- •Most logic can live in markdown/text plus TypeScript where needed
- •Eve’s structure encourages simplicity over over-engineering
- 14:11 – 15:42
Trigger rules and permissions: reduce noise and only run when it matters
Claire notes the importance of configuring when the bot runs so it doesn’t spam early or on incomplete PRs. Permissions are scoped to PR contents, checks, metadata, and the ability to comment/approve as appropriate.
- •Avoid triggering on every PR update before checks finish
- •Scope GitHub permissions to PRs, file contents, CI/actions, metadata
- •Use rules to ensure the bot runs at the right lifecycle stage
- •Automation handles setup, but teams should verify access boundaries
- 15:42 – 17:13
Six-component risk scoring model and thresholds (low/medium/high)
She explains the risk rubric: blast radius/change surface, reversibility, security/data impact, operational impact, and verification gaps (tests/CI completeness), among others. A numeric score maps to low/medium/high bands that determine whether humans must approve.
- •Risk factors include blast radius, reversibility, security/data concerns, ops impact, and test/CI coverage gaps
- •Repo-specific categories: docs low risk; feature logic medium; auth/billing high
- •Diff size alone does not define risk
- •Thresholds: <24 low; 25–64 medium; 65+ high; medium/high require human approval
- 17:13 – 21:46
Merge Mommy in action: three PR examples (auto-approve, blocked, escalate)
Claire demos the bot on real PRs: a docs PR scored low but blocked due to merge conflicts, a docs PR that receives an approval signal, and a larger deprecation/refactor PR scored medium and escalated. She also explains why a human still clicks approve due to repo rules.
- •Docs-only PR: low risk but blocked when merge conflicts exist
- •Docs PR: low risk and marked approved by the bot
- •Deprecation/refactor PR: medium risk due to change surface/API behavior; not auto-approved
- •Repo rules may require a human approval even if the bot “approves” (gray check as signal)
- 21:46 – 24:13
Recap: build your own + add evals to keep the agent honest
She summarizes the build recipe—Eve agent + PR-reading skill + risk score + GitHub/Slack connectors—and reiterates the benefit: faster cycle time with attention focused on risky changes. Finally, she recommends running evals and logging each review so engineers can audit and improve the agent over time.
- •Use Eve, write concise instructions, implement PR read + risk scoring
- •Connect to GitHub for PR context and Slack for escalation/ops flow
- •Result: faster PR cycle time and higher throughput
- •Add evals/logging for continuous improvement and safety validation
