How I AIHow Intercom 2X'd engineering velocity with Claude Code | Brian Scanlan
CHAPTERS
- 0:00 – 1:06
Imagination becomes the bottleneck: why AI feels different now
Brian and Claire open with the core shift AI brings to engineering: output is no longer constrained by typing speed or tooling friction. They frame the episode around a new ceiling—imagination and decision-making—rather than execution capacity. The tone is set: Intercom is seeing major throughput gains and wants far more than 2X.
- •AI removes the “physical limits” of producing code
- •Engineering throughput is up ~2X in months, with ambition for 10X
- •The new constraint becomes idea quality and imagination
- •AI makes long-standing backlogs feel tractable (even “backlog zero”)
- 1:06 – 3:45
What Intercom set out to prove: 2X PR throughput across all of R&D
Claire introduces Brian and the promise of the episode: Intercom materially increased shipping velocity using Claude Code. She positions Intercom as having met the AI moment both in customer product and internal engineering workflows. The focus shifts from theory to measurable operational change.
- •Intercom approached internal AI adoption like a product rollout
- •Goal framed as doubling R&D throughput using PRs as a proxy
- •AI transformation spans product, engineering, and design workflows
- •The episode will reveal tooling, workflow, and measurement details
- 3:45 – 5:02
Why Intercom went all-in on AI: product strategy made internal adoption inevitable
Brian explains why engineering adoption was easier at Intercom: the company already committed to AI-first product thinking. They believed engineering would be transformed too, but waited for tooling/model maturity to make it truly disruptive. The org’s stance is proactive impatience, not cautious experimentation.
- •AI-first product strategy created organizational pull for engineering adoption
- •Earlier tools showed promise but weren’t fully transformative
- •Clear internal expectations: engineering work will shift dramatically
- •Adoption accelerated once a real “breakthrough moment” arrived
- 5:02 – 7:02
The breakthrough: Opus-era inflection and the post-holiday ‘everything changed’ moment
Claire and Brian recount the model inflection point where AI shifted from autocomplete help to idea-to-execution amplification. Brian ties the acceleration to late-year model improvements and the Christmas break when usage exploded socially and internally. Returning to work felt like the entire landscape had changed.
- •Shift from “massaging prompts” to simply expressing intent
- •Opus-era capabilities drove step-function change in usefulness
- •Holiday hacking time accelerated skills and confidence
- •Intercom committed to going “all-in on Claude Code” after the inflection
- 7:02 – 12:51
Demo: the PR throughput dashboard and why PRs are a practical leading indicator
Brian shows Intercom’s dashboard tracking merged PRs per R&D head and the share generated by AI tools. They treat the engineering org as a product: instrumented, measured, iterated. PR count is acknowledged as imperfect, but useful to validate adoption and momentum without sacrificing quality.
- •Tracking merged PRs per R&D head across all of R&D roles
- •AI attribution: % of PRs generated via Claude/Cursor/etc.
- •PRs as a “crude but useful” leading indicator for adoption
- •Quality expectations and high-trust culture reduce metric gaming concerns
- 12:51 – 14:29
Agent-first work: reimagining workflows from first principles (and finding new bottlenecks)
Brian argues that technical work will become agent-first and should be treated as a fundamental workflow redesign. Intercom focuses on where time is spent and moves tasks to agents so humans operate at higher levels. As throughput rises, constraints shift—CI once melted, now code review is the bottleneck.
- •Vision: every technical workflow becomes agent-first
- •Redesign work rather than merely “pushing people to go faster”
- •Bottlenecks move: CI failures → optimized → code review becomes limiting
- •Current building blocks are enough to keep reworking workflows even before future model gains
- 14:29 – 16:43
Cost vs speed: treating AI token spend like a strategic investment
They discuss the rapidly rising AI bill and how it resembles hiring entire new offices. Intercom’s current strategy is speed-first: default to powerful models and optimize cost later once benefits compound. Brian notes this isn’t feasible for every company, but the payoff can justify an “investment phase.”
- •AI spend is increasing in lockstep with usage and throughput
- •Intercom defaults to high-capability settings (e.g., large context)
- •Cost optimization deferred until after capturing the velocity upside
- •Acknowledges tradeoffs: cheaper models can work and sometimes improve discipline
- 16:43 – 21:43
Measuring quality beyond PR counts: time-to-ship, incident signals, and external research
Brian shares how Intercom checks that higher velocity translates into real customer value. They track time from first code to public release updates and monitor incident patterns, while also collaborating with a Stanford research group on code quality signals. Their current findings suggest quality is improving, not degrading.
- •Trailing indicators: shipping volume and time-to-news-channel updates
- •Operational signal: no clear increase in incidents/outages observed
- •External perspective: Stanford research analysis of code quality trends
- •AI magnifies strengths/weaknesses—mature engineering foundations matter
- 21:43 – 23:06
Demo: shipping a Rails redirect in the monolith—fast changes with guardrails
Brian demonstrates a small change in Intercom’s large Rails monolith: add a redirect (with a lobster emoji). The demo highlights how Claude Code can locate the right place to edit and propose opening a PR quickly. The underlying theme is making routine work trivial so humans focus on higher-leverage problems.
- •Working inside a mature, multi-million-line Rails monolith
- •Claude Code identifies redirect location and proposes a PR
- •Prompting “hints” to the agent to guide urgency/intent
- •Reframing: “coding redirects” is not where human creativity should go
- 23:06 – 30:16
Raising standards with a custom PR workflow: the ‘Create PR’ skill and enforcement hooks
Intercom found AI-generated PR descriptions were drifting toward low-value code summaries rather than intent. They built a ‘Create PR’ skill that extracts session context to produce better descriptions, then enforced it via hooks that block default PR creation. Quality is treated as a system property: measured, judged, and fixed.
- •Problem: PR descriptions became worse with AI—too code-focused, low intent
- •Solution: a ‘Create PR’ skill using session context for meaningful descriptions
- •Governance: hooks block PR creation unless the skill is used
- •LLM judge used to measure PR description quality over time and confirm improvement
- 30:16 – 39:20
Telemetry as a first-class product: Honeycomb dashboards + session capture to S3
Brian outlines Intercom’s telemetry stack for Claude Code usage: events into Honeycomb and full session data into S3, plus warehousing in Snowflake. The goal is to avoid flying blind—understand which skills are used, where users struggle, and how to improve the platform. They also describe anonymization and privacy considerations.
- •Skill-level events piped to Honeycomb for org-wide visibility
- •Session data collected (with anonymization) and stored in S3
- •Additional analysis flows into Snowflake for warehousing and reporting
- •Telemetry enables discovery: adoption, success rates, friction points, and skill ROI
- 39:20 – 42:06
Inside the skills platform: distribution, IT-managed installs, and quality tiers
They tour Intercom’s internal skills repository and how it’s deployed to hundreds of laptops. Instead of relying fully on the plugin mechanism, Intercom uses internal IT systems to sync plugins reliably. Skills are organized with different quality bars: minimal core plugins for everyone, and rigorously evaluated developer tools for broad use.
- •Skills live in a centralized GitHub repository with heavy activity
- •Deployment choice: IT-managed sync to disk for reliability and easier debugging
- •Core plugins are minimal and safe for all roles
- •Higher-tier developer tools require evals/tests and stricter maintenance
- 42:06 – 53:44
Deep dive: the ‘flaky specs’ skill—how it evolved from 1X to ‘100X’ capability
Brian explains how a focused, testable goal (fix flaky tests) and a large backlog of historical data enabled rapid skill improvement. The skill is built around checklists, feedback loops, and continuous self-updating: when it encounters a novel fix, it updates its own guidance. Over time it becomes a dependable, low-maintenance building block with evals guarding against regression.
- •Start with a bounded goal: reliably fix flaky tests in a large test suite
- •Use historical flaky-spec data as training/reference material for patterns
- •Feedback loop: skill updates itself when novel cases are solved
- •Fan-out approach: fix one case, then locate and repair similar failures
- •Evals ensure model/tooling changes don’t degrade performance
- 53:44 – 1:06:10
How internal agent use reshapes customer product: agents need CLIs, APIs, and ‘helpful hints’
Claire asks how Intercom’s internal experience changes how they build for customers. Brian argues many agent workflows default to “build it yourself” unless SaaS products become agent-friendly—through better APIs, CLIs, MCP-style interfaces, and embedded guidance. They explore the idea of agent-oriented UX: discoverability, automation paths, and reducing agent frustration during setup.
- •Agents often choose to implement features (e.g., feature flags) instead of buying SaaS
- •Products must become agent-friendly: CLIs/MCP/APIs and stepwise automation
- •“Helpful hints” in command outputs guide agents toward success paths
- •Agent-friendly interfaces often become more human-friendly (discoverable verbs, clearer help)
- 1:06:10 – 1:18:45
Lightning round: culture shift, enabling skeptics, and leadership ‘permission’ as the unlock
They close with reflections on how AI changes engineering culture—faster feedback loops, more variety in work, and more fun. Brian emphasizes that adoption requires leaders to push past risk-aversion thoughtfully, grant permission, and take accountability so teams can explore. The episode ends with practical advice on meeting skepticism and continuing to iterate toward agent-first operations.
- •AI improves morale through faster loops and eliminating low-leverage work
- •Leadership role: push through barriers carefully (security, production access, data tools)
- •Permission + accountability unlocks experimentation across roles
- •Clear stance matters: “work will be agent-first” shapes culture and urgency
- •Skeptic management: enable, guide, and iterate rather than debate indefinitely
