CHAPTERS
- 0:00 – 0:32
Using Fable to eliminate import cycles and speed Python app boot via laziness
Rahul describes putting Fable to work on deeply entrenched monolith problems: breaking Python import cycles and making the app load modules lazily. He emphasizes learning where models fail so the team knows what to re-attempt with each new model release.
- •Asked Fable to “fix all our import cycles” in a large Python monolith
- •Also tasked Fable with making the app boot lazily despite many modules
- •Significant progress and some changes merged into the codebase
- •Focus on identifying model breakpoints to guide future trials with newer versions
- 0:32 – 1:33
Shadow-running Fable in CI: choosing verifiable, non-product tasks
Rahul explains why CI/testing infrastructure is an ideal early proving ground: it’s measurable, low-risk, and easy to validate empirically. The team ran Fable-written CI logic in shadow mode and found it consistently faster than the existing implementation.
- •Selected a use case with empirical verification and rich data output
- •Targeted non-product surface area (CI/testing suite) for lower risk
- •Ran in shadow mode for weeks to compare performance safely
- •Observed consistent speedups versus the prior approach
- 1:33 – 3:36
Dynamic workflows as test-time compute: orchestrating many sub-agents to cut CI 66%
Boris explains his model-evaluation “tests” and how dynamic workflows helped when Fable struggled. With orchestrated sub-agents continuously profiling and iterating, they drove CI p50 from ~18 minutes to ~6 minutes and even scheduled follow-up runs to confirm gains in production-like conditions.
- •Dynamic workflows orchestrate sub-agents in parallel/serial as needed
- •Framed as a new kind of test-time compute you can “dial up”
- •CI p50 reduced from ~18 minutes to ~6 minutes through repeated profiling/optimizations
- •Workflow can schedule itself later to collect real production data and keep iterating
- •Ends by reporting results via artifacts like charts
- 3:36 – 4:16
Loops vs dynamic workflows: repetitive automation vs open-ended optimization
The conversation contrasts two paradigms for long-horizon work. Loops suit repeatable tasks with known steps, while dynamic workflows are better for exploratory or emergent multi-step efforts like performance tuning where the next step isn’t known in advance.
- •Loops = repetitive work with predictable steps (e.g., babysit PRs, fix CI, rebase)
- •Dynamic workflows = adaptive work for unknown sequences (e.g., system optimization)
- •Mental model shift: relinquishing more of the workflow to the agent
- •Horizontal slice: automate the task everyone does daily (review, PR feedback, dead code)
- •Vertical slice: end-to-end ownership for a project/experiment lifecycle (e.g., Tag shipping experiments)
- 4:16 – 5:18
End-to-end experiment shipping with agents: minimal human involvement after kickoff
Boris describes an agent-driven experiment pipeline: create the experiment, land the PR, monitor metrics, adjust exposure, and later ship the winning variant. Humans primarily initiate and approve, while the agent handles the multi-week operational follow-through.
- •Agent creates and lands experiment PRs, then schedules self-checks
- •Monitors exposure balance and ramps exposure over time
- •Returns weeks later to propose shipping a variant with a new PR
- •Human stays mostly out of the loop beyond initial ask and approval
- •Illustrates long-horizon autonomy plus routine-based reminders
- 5:18 – 6:49
Claude Code setups: vanilla terminal workflow vs background-heavy context gathering
Austin shares a simple, terminal-based setup designed to learn model behavior, while Rahul describes a shift toward background-heavy sessions for information gathering across many services. Constraints of local dev in a complex microservice environment push more work into remote/background agent workflows.
- •Austin: iTerm2, no IDE, minimal plugins/MCPs, claude.md, adversarial review
- •Rahul: increasingly background-heavy; most sessions are context gathering
- •Uses local Claude Code when hands-on debugging or local context is required
- •Local dev becomes a constraint with many services (DBs, queues, Redis, etc.)
- •Newer models require less hand-holding—“let them cook” leads to moving workloads off laptop
- 6:49 – 7:38
Agents across the engineering lifecycle: from ideation to production monitoring and security
Rahul outlines how Ramp embeds agents throughout software delivery—from identifying bugs and log issues to writing and reviewing code and checking production behavior post-deploy. They also apply agents to security-oriented workflows like finding bugs and issues.
- •Coverage across idea generation, debugging, notifications, coding, review, and monitoring
- •Agents help interpret logs and system signals to surface problems faster
- •Post-deploy validation: verify production behavior matches intent
- •Security lens: using agents to find bugs and issues earlier
- •Goal is end-to-end velocity and reduced manual toil
- 7:38 – 9:29
Building for future models: reduce scaffolding, increase tools/context/agency over time
Instead of over-optimizing for today’s model limitations, Ramp aims for what models will be capable of in the near future. As models improve, they remove previously necessary harnesses and invest in giving agents the tools and access needed to act like coworkers.
- •“Build for what’s coming next,” not just the current model snapshot
- •Repeatedly remove scaffolding as models outgrow harness constraints
- •Provide more tools, more context, and more agency to hit coworker-like UX
- •Aim for a declarative, task-based interface (what to do, not how)
- •Treat failures as signals about missing tools/context rather than permanent blockers
- 9:29 – 12:08
Guardrails and trace-driven debugging: least privilege, correct traces, and layered defenses
Rahul explains how they keep agents safe and effective by studying individual traces rather than only aggregate benchmarks. They define the “correct trace” for a workflow, then shape agent behavior via prompts/tools while enforcing least privilege with read-only keys and layered defenses supported by security.
- •Focus on individual traces: identify the correct command/tool sequence
- •Diagnose failures as context/tool-access gaps and refine accordingly
- •Layered defenses enable more agency while constraining risk
- •Least-privilege access patterns (e.g., read-only service keys)
- •Security team builds network/key infrastructure and also uses the agents themselves
- 12:08 – 13:25
Cost controls and code quality: constrained-risk deployments and evolving review focus
Ramp scales agent usage by picking tasks with bounded downside and investing in verification loops in CI/CD. They also evolve code review—using their own review bot and codified team “skills”—as models stop making certain classes of mistakes, shifting what humans should spend attention on.
- •Prefer tasks where worst-case impact is highly constrained
- •Prepare stronger verification loops, especially in CI/CD
- •Human review focus shifts as models improve; stop checking obsolete failure modes
- •Built an internal code review bot using Inspect/background agents
- •Teams codify institutional knowledge via skill files and memories to standardize checks
- 13:25 – 13:53
Culture of experimentation: broad access, no token budgets, and empowered bottleneck-breaking
They attribute adoption to Ramp’s experimentation culture and minimizing friction for engineers. By avoiding per-engineer token/tool budgets, everyone can develop fluency and share learnings, accelerating decentralized innovation.
- •Company-wide norm: experiment, move fast, accept that some attempts won’t pan out
- •Engineers get broad access to tools—no strict token budgets imposed
- •Shared language and common tooling improves collaboration and diffusion of practices
- •Bottom-up identification of bottlenecks; teams feel empowered to automate
- •Leadership supports adoption by enabling rather than tightly prescribing usage
- 13:53 – 16:07
Glass and Inspect: AI coworkers via web/Slack with multiplayer PR sessions
Austin and Rahul explain Ramp’s internal products: Glass for non-technical users and Inspect as a tool-rich “digital coworker” for builders. Inspect integrates with core systems (GitHub, Linear, Slack, Datadog, Sentry, Zendesk), can be invoked from Slack, and provides link-based, collaborative sessions with a VM per PR.
- •Glass: home base for non-technical users to interact with coding agents
- •Inspect: agent with broad tool access for tickets, issues, errors, and PRs
- •Web access plus strong Slack-based invocation drives viral adoption
- •Each PR can have its own VM; sessions are link-based and transferable
- •“Multiplayer” collaboration: others can take over or join ongoing agent work
- 16:07 – 17:32
On-call Assistant: Claude Code-powered AI SRE for incidents and root-cause PRs
Austin describes an on-call agent that runs on every incident—support escalations and system issues alike. Built from proven local Claude Code workflows, it’s packaged into a guarded container that produces root cause analyses in incident Slack channels and can propose fixes via PRs.
- •Runs on every incident assigned to engineers (support + system incidents)
- •Delivers root cause analysis directly into the incident Slack channel
- •Incident responders interact with the agent during live response
- •Built locally first (skills/MCPs/prompts), then packaged to run continuously
- •Emphasizes Unix-like composability of Claude Code as an execution primitive
- 17:32 – 20:49
Automation at scale: more agent sessions than humans, decentralized builders, and ROI-first spend
They discuss how automations increasingly trigger Inspect sessions more than humans do, and how the org structure remains decentralized with a platform abstraction as bedrock. Cost strategy prioritizes ROI and capability—use latest frontier for humans, cheaper models for automations, and follow up with top spenders to platformize successful patterns.
- •Inspect sessions increasingly initiated by scheduled/external triggers vs humans
- •Decentralized creation of automations; platform team maintains core abstractions
- •Platform consolidates “desire paths” when many teams want similar capabilities
- •No per-user token limits; instead rely on defaults, batch/flex APIs, cheaper models for automations
- •Cost management via outreach to unusually high spenders to learn, correct, or platformize
- 20:49 – 21:58
Advice for CTOs: focus on the rate of change and build 3–6 months ahead
Rahul’s closing guidance is to look beyond today’s model snapshot and instead track the trendline of rapid improvement. Building for where models will be soon helps avoid shipping systems that are obsolete by launch and positions teams to capitalize on rising intelligence and agency.
- •Pay attention to the rate of change, not just current capabilities
- •Assume rising intelligence/agency will expand what agents can do
- •Build for what’s likely 3–6 months out to avoid instant technical debt
- •Avoid playing catch-up; by the time you ship, the target may have moved
- •Use scaling trends to guide architecture and investment decisions
