CHAPTERS
- 0:00 – 3:47
AI skepticism vs. “no excuse for rigor” on hard engineering problems
Claire and Ankur open by challenging the common staff-engineer belief that AI can’t help on the most complex technical work. Ankur argues that agents enable more rigor—more experiments, more benchmarks, and more iteration—than humans typically sustain manually.
- •Pushback on the idea that AI fails on “complicated things”
- •Agents can run more rigorous experiments than typical human workflows
- •Rigor becomes cheaper: fewer excuses for untested assumptions
- •Framing: many decisions and tasks now belong “below the agent line”
- 3:47 – 5:18
Using coding agents to diagnose and speed up real production queries
Ankur describes how Braintrust uses agents to identify slow query patterns and reproduce them for systematic optimization. The emphasis is on defining success criteria (latency, throughput, indexing cost) and letting agents explore solutions from database literature.
- •Reproducing slow queries based on real user patterns
- •Defining tests/success metrics so agents can iterate independently
- •Applying database indexing and execution ideas at scale
- •Using production or production-like data for more realistic results
- 5:18 – 9:32
Exhaustive benchmarking: column stores, execution engines, and a solution matrix
They dive into Ankur’s current work: testing many open-source column store formats and execution engines to find the best combination. The key advantage is the ability to run exhaustive permutations that would be too time-consuming for a human team.
- •Trying “every” open-source column store format and engine
- •Computing the performance matrix across combinations
- •Why this breadth of benchmarking is rarely feasible manually
- •Agents as force multipliers for infrastructure/platform decisions
- 9:32 – 11:33
Why risk-averse teams are wrong: benchmarks, peer review, and hidden regressions
Ankur explains how even strong engineers often skip or hand-wave benchmarks under time pressure, especially for database changes. With agents, teams can systematically validate both wins (query speed) and tradeoffs (indexing slowdown), surfacing issues earlier.
- •Typical benchmark gaps: focusing on a few tests and “bullshitting” the rest
- •Example: bloom filters discovered via week-long continuous experiments
- •Benchmarking both query performance and indexing cost
- •Agents reduce the cost of thoroughness and expose regressions sooner
- 11:33 – 14:16
Production safety, practical quality, and continuous iteration beats theoretical perfection
Claire and Ankur distinguish theoretical human-perfect code from practical reality: attention decay, context loss, and limited cycles. They argue that agent-driven persistence and repeatability often improves real-world quality and enables bigger technical bets.
- •Practical quality improves with sustained, consistent iteration
- •Humans lose context and stamina on tedious technical work
- •Agents enable “run at the problem” persistence
- •Business case: background progress without tying up staff for months
- 14:16 – 15:16
The “agent line” and reclaiming maker time (no meetings after noon)
Ankur introduces the “agent line” framework: if an agent can solve it given the meeting’s information, it likely belongs below the line. He pairs that with schedule design—protecting maker time—to write code deeply and effectively alongside agents.
- •Definition: can an agent solve it with the info being discussed?
- •The agent line keeps rising as tooling/integrations improve
- •Pushing the agent line via skills and integrations inside the company
- •Maker schedule tactics: blocking afternoons for focused coding
- 15:16 – 20:32
Running 4–6 concurrent agents: tmux sessions, local constraints, and remote experiments
Ankur describes a pragmatic setup: multiple foreground agents in tmux, each tied to a workstream, plus a remote environment for compute-heavy benchmarks. They also cover the current limits of off-the-shelf background agents for complex systems (ports, isolation, collisions).
- •Foreground agent concurrency: ~4–6 sessions in tmux
- •Naming and isolating parallel workstreams per agent
- •Port collisions and why complex software is harder to containerize neatly
- •Remote/cloud environments for high-scale, multi-day experiments
- 20:32 – 23:09
Sustainable AI usage: flow state vs. productivity anxiety (and closing the laptop)
They discuss the emotional side of agentic coding: some people rediscover flow, while others feel constant pressure to be “kicking off agents.” Claire advocates chunking time with AI, and both emphasize boundaries to avoid always-on work habits.
- •Two camps: renewed joy/flow vs. anxiety/burnout
- •Avoiding the feeling that you must run agents constantly
- •Chunking work time with AI for better focus and enjoyment
- •Personal boundary example: closing the laptop at dinner
- 23:09 – 26:16
Evals, demystified: machine learning shifts “how” to “what”
Ankur explains evals as the core mechanism for specifying outcomes in AI systems—defining success rather than implementation. He frames evals as the modern PRD: prose plus examples, encoded in measurable ways so models can explore the solution space.
- •ML changes programming from “how” to “what”
- •Transformers as an inspiration: define the objective and measure it
- •Evals as modern PRDs: examples + quantifiable criteria
- •Focus on success definitions enables creative solution search
- 26:16 – 30:21
Live demo: building a documentation-answer eval with a generated scoring function
Ankur walks through creating a dataset of real documentation questions, running a prompt to answer them, and asking a model to generate a scoring function. They highlight why this is safer in a sandbox/playground than letting agents loose on a local machine.
- •Creating a question dataset from real doc queries
- •Using MCP/context tools to ground doc answers
- •Letting a model generate evaluation criteria and scoring code
- •Sandboxed agent environments reduce risk vs. local bash autonomy
- 30:21 – 30:57
Why “vibe checks” aren’t enough: avoiding whack-a-mole with systematic evals
They contrast rigorous eval pipelines with the common alternative: testing one or two examples and generalizing. Ankur supports vibe checks, but argues that without evals you end up fixing one issue at a time after shipping, never knowing what regressed.
- •Common anti-pattern: a couple examples and a guess
- •Vibe checks are useful but insufficient alone
- •Whack-a-mole failure mode after shipping
- •Evals provide aggregate tracking and regression detection
- 30:57 – 33:53
Encoding taste: using a designer’s judgment to raise quality without replacing them
Ankur shares how a designer (“David”) provides periodic taste-based reviews that are then converted into eval criteria. Claire addresses fears about “building your replacement,” and they argue that codifying taste lets experts scale their impact and raise the quality bar.
- •Designer-in-the-loop: periodic vibe checks on top of quantitative evals
- •Turning qualitative feedback into scoring functions for future runs
- •Capturing taste increases reach and consistency across outputs
- •Codifying expertise elevates the expert rather than replacing them
- 33:53 – 37:31
Lightning round: carving vs. constructing, and why CI/CD is the bottleneck
Ankur explains that AI makes it easy to overbuild, so product work becomes “carving”—removing complexity to reduce confusion. On throughput, he emphasizes investing in CI so teams can safely move faster, and reframes engineering as building platforms for agents.
- •AI accelerates feature creation, increasing the need to remove/trim
- •Default response to confusion: simplify rather than add complexity
- •Throughput strategy: improve CI to earn speed safely
- •Core team job: build feedback loops/pipelines (for evals and for code)
- 37:31 – 39:08
When agents fail: reset, improve the eval, and re-run (plus hand-writing the eval)
Asked about a go-to prompting strategy, Ankur prefers restarting with better evaluation scaffolding rather than wrestling with a stuck session. He shares an example where a vibe-coded eval became unusable, so he hand-wrote the eval to understand the problem and quickly finish the migration decision.
- •Preferred fix: close the session, strengthen evals, restart clean
- •Avoids accumulating brittle, confusing “agent-made” evaluation code
- •Example: 3,000-line messy eval script replaced by a hand-written eval
- •Principle: humans focus on the eval/feedback loop; agents do the rest
- 39:08 – 40:11
Closing: where to find Braintrust and Ankur (and hiring)
They wrap up with links to Braintrust and Ankur’s socials, plus an invitation to talk about evals and observability. Claire closes with standard show outro and subscription prompts.
- •Braintrust: braintrust.dev and @braintrust
- •Ankur: @ankrgyl; open to chatting
- •Hiring call for people excited about rigor and evals
- •Podcast outro: like/subscribe, ratings, and where to listen
