Lenny's PodcastArchie Abrams: Why Shopify bans KPIs and optimizes for churn
Through Shopify's year-plus holdout experiments and 100-year vision; intuition and lower funnel friction beat conversion-rate KPIs that quietly gate growth.
CHAPTERS
- 0:00 – 0:46
Why optimizing funnel conversion rates can backfire (and what to measure instead)
Archie opens with a warning about how teams get seduced by local funnel conversion rates and end up gaming the system. He explains why Shopify pushes teams to focus on absolute outcomes (more people succeeding) rather than percentage rates that can be improved by simply adding friction upstream.
- •Local conversion-rate goals often create perverse incentives between funnel-stage teams
- •It’s usually easier to raise a conversion rate by making the previous step harder
- •Absolute counts (e.g., # activated) can be healthier than ratios
- •Getting more people “in the door” may lower conversion rate but increase total success
- •Metrics should align incentives across the full journey
- 0:46 – 3:56
Archie’s role, Shopify’s scale, and what makes their growth philosophy unusual
Lenny introduces Archie Abrams and previews Shopify’s unconventional operating model: long-term vision, intuition-driven roadmap, and growth practices that look nothing like typical SaaS. Archie then shares eye-opening scale stats that frame why Shopify’s growth challenges are unique.
- •Archie leads a 600+ person org across product, design, engineering, data, ops, and growth marketing
- •Shopify is ~10% of US e-commerce; many consumers don’t realize they’re using it
- •2023 GMV of ~$235B (compared to the size of a national economy)
- •Preview of Shopify’s distinctive choices: KPI bans in core, churn focus in growth, long-term experiment holdouts
- •The conversation will unpack systems, structure, and concrete growth wins
- 3:56 – 6:26
Sponsor break: embedded analytics and customer insights tooling
A brief sponsor interlude highlights tools for embedded analytics (Explo) and for consolidating customer research and feedback (Dovetail). The ad sets up a theme of instrumentation and insight that contrasts with Shopify’s selective use of metrics.
- •Explo: low-code embedded dashboards and AI-assisted reporting for end users
- •Dovetail: AI-first hub for qualitative customer insights, summaries, and feedback analysis
- •Emphasis on speeding up insight generation for product teams
- •Transition back into Shopify’s growth and measurement philosophy
- 6:26 – 10:00
Why Shopify “optimizes for churn”: lowering barriers so more entrepreneurs can try
Lenny challenges Shopify’s counterintuitive stance on churn and retention. Archie explains that Shopify’s mission is to increase entrepreneurship, so the business is designed to reduce barriers to starting—even if many businesses fail—because a few winners drive most of the value.
- •Shopify lowers friction to starting a business, accepting that many attempts will fail
- •Unlike typical SaaS, Shopify isn’t maximizing retention of every individual merchant
- •A power-law outcome: a few massive successes make the cohort economically valuable
- •Most revenue scales with merchant success (payments/services), not just subscription fees
- •Analogy to angel investing: portfolios are driven by outliers
- 10:00 – 14:59
Measuring success with cohort GMV (power laws) and accepting long feedback loops
Instead of obsessing over churn curves, Shopify evaluates cohorts by the total GMV they generate over years. Archie describes why this long time horizon changes what “good” looks like and how it forces better thinking about what truly matters to the business model.
- •Primary lens: total cohort GMV over 1–5 years, not per-merchant retention
- •GMV is power-law distributed; outliers dominate outcomes
- •Cohort GMV translates to revenue, gross profit, and reinvestment capacity
- •This creates a long feedback loop that doesn’t fit quarterly metric thinking
- •Growth uses leading indicators, but is ultimately accountable to long-term value
- 14:59 – 20:40
Long-term holdouts: discovering pull-forward effects and hidden high-value segments
Archie explains what Shopify learns by revisiting experiments months and years later. A significant portion of “wins” fade over time (often pull-forward), while some neutral-looking changes uncover unexpectedly valuable merchant pockets—especially when reducing early monetary friction.
- •Common pattern: short-term lifts often don’t produce long-term GMV lift
- •30–40% of apparent short-term winners show no durable long-term effect
- •Pull-forward is a frequent culprit (accelerating outcomes that would happen anyway)
- •More rare but valuable: neutral early results that create long-term GMV gains
- •Reducing early monetary friction can unlock merchants who otherwise would have quit
- 20:40 – 22:56
How Shopify’s experimentation system enforces long-term accountability
They “call” experiments quickly but preserve cohorts for long-term evaluation, enabling fast shipping without losing the truth. Archie details Shopify’s two holdout layers and the automation that forces teams to revisit outcomes at 3, 6, 9, and 12 months.
- •Two holdout strategies: ongoing global holdouts (e.g., 5% across changes) and cohort-based splits for new merchants
- •For new-user experiments: run 50/50 briefly, ship winner to 100%, but keep cohort assignment for later analysis
- •Experimenters are automatically pinged with updated results over time
- •Core long-term outcomes include GMV and gross profit, plus merchant milestones
- •System design prevents teams from “declaring victory” and moving on forever
- 22:56 – 26:35
Big growth wins: onboarding personalization, dunning lessons, and “ship neutral” intuition
Archie shares practical categories of wins and two concrete examples. One “win” (payment failure notifications) looked great short-term but didn’t matter long-term; another (preconfigured store sections) looked neutral short-term but drove major downstream GMV. Shopify will often ship neutral results if intuition says it’s better for merchants.
- •Consistent opportunity areas: monetary friction levers and onboarding/onramp quality
- •Collecting more signup info can enable better personalization and guidance
- •Dunning/payment failure alerts can produce short-term lift without long-term GMV impact
- •Preconfigured store-building blocks: neutral early conversion, strong long-term GMV via better stores
- •Cultural stance: if A/B is neutral and intuition favors a version, Shopify may ship it
- 26:35 – 29:46
Monetary friction explained—and why Shopify prefers absolutes over conversion rates
Archie defines “monetary friction” and then expands into Shopify’s metric philosophy: rates can be gamed, absolutes align incentives. He argues teams should optimize for the total number of merchants progressing, not local conversion percentages that encourage upstream restriction.
- •Monetary friction includes trial dynamics, incentives/credits, and price points
- •Discounts aren’t always “lower quality”; they can causally extend runway for entrepreneurs
- •Conversion-rate optimization often leads teams to add friction to earlier steps
- •Shopify emphasizes absolute counts (more merchants succeed) rather than ratios
- •Lowering friction may reduce LTV per user but can lower CAC and increase total value
- 29:46 – 33:03
Inside Shopify Growth: R&D vs growth marketing, enablement tooling, and support experience
Archie breaks down how the growth org is structured and what each group owns. Growth is split between product/engineering/data (“growth R&D”) and channel execution (“growth marketing”), with a notable emphasis on internal tools and customer support experiences as part of the end-to-end journey.
- •Two big groups: Growth R&D and Growth Marketing
- •Growth product scope: landing pages → signup → onboarding → monetization → engagement
- •Enablement pillar builds experimentation, comms, BI, and MarTech tooling for growth and beyond
- •Customer support experience (tooling + help center + AI) sits within growth
- •Growth marketing covers paid acquisition, affiliates, email/content, and SEO
- 33:03 – 42:02
Goals, forecasting, and measuring impact without getting trapped by funnel stage KPIs
Lenny presses on how such a large org sets goals. Archie explains the macro model (cohort value and payback guardrails) and how channels and product teams use guardrails while still being judged by long-term incremental cohort value—rather than whether a local conversion rate moved.
- •Top-level goal: incremental cohort value (GMV/GP) over multi-year horizons
- •Efficiency matters via payback guardrails and channel LTV:CAC boundaries
- •Growth marketing uses channel guardrails; content/SEO is evaluated similarly over time
- •Experiments are called quickly but monitored long-term for true incremental impact
- •Conversion-rate movement alone is treated as largely irrelevant if total outcomes improve
- 42:02 – 48:04
Building a 100-year company: the founder-driven “how” and technical architecture as strategy
Archie describes Shopify’s deep commitment to a 100-year vision and how that shapes product investment decisions. A key founder-led principle is that technical architecture (“the how”) determines long-term strategic optionality, so Shopify spends executive attention on engineering foundations many companies would dismiss as implementation detail.
- •Shopify prioritizes building the best product for merchants over short-term revenue pulls
- •Foundational belief: the biggest brands today won’t be the winners 100 years from now—startups will be
- •Every ~6 weeks, leaders review every R&D project with Tobi in significant detail
- •Tobi’s view: technical architecture drives strategy in a technology platform company
- •Example: extensive debate on CSV importer architecture as long-term infrastructure
- 48:04 – 54:34
Why Shopify bans KPIs/OKRs in core—and how GSD enforces quality through approvals
Archie explains that KPIs and OKRs are effectively banned in core product work, where decisions rely on conviction, data as input (not override), and a consistent “taste bar.” He then shares how Shopify’s internal project system (GSD) and “ok-to-ship” approvals create accountability without conventional KPI scorecards.
- •Core teams use data, but aren’t governed by target KPIs or OKRs
- •Tradeoff: less metric gaming and more bold product bets, but more subjectivity
- •A small set of leaders (e.g., Glen, Tobi) maintains a consistent quality/taste bar
- •GSD (“Get Shit Done”) is Shopify’s internal PM system with standardized project artifacts
- •Nothing ships without an “ok-to” approval from the relevant group lead
- 54:34 – 1:01:08
Core + growth collaboration: productive tension, trust, and the “no wizards” principle
Growth is allowed to touch any surface area, which creates intentional tension with core’s quality and principles. Archie explains that the solution isn’t rigid boundaries—it’s trust, shared commitment to quality, and principled design constraints like avoiding onboarding wizards while still achieving wizard-like simplification.
- •Growth and core are designed to operate on different time horizons and incentives
- •Collaboration relies on human trust and reliable follow-through on quality commitments
- •Disagreements often center on UX quality vs short-term growth lift (e.g., moving a button)
- •Shopify’s “no wizard” principle avoids setup flows that bypass the real product
- •Example: prefilled sections achieve guided setup without a wizard flow
- 1:01:08 – 1:06:35
Growth meets sales: building hybrid journeys and shifting from attribution to incrementality
As Shopify added a sales motion, the company had to reconcile two previously separate funnels: self-serve and sales-led. Archie shares the operational and measurement challenges of hybrid journeys—especially how legacy attribution models break—and why Shopify prefers incrementality testing for budgeting decisions.
- •Sales introduces different scale and requires more qualitative insight and empathy
- •Shopify is moving from separate funnels to hybrid journeys (self-serve ↔ sales)
- •Hybrid paths break traditional measurement (LTV models, attribution, and channel credit)
- •Example: an ad drives a self-serve signup that converts via sales—value gets lost in old systems
- •Preferred measurement north star: incrementality testing over multi-touch attribution
- 1:06:35 – 1:17:45
Marketing without a CMO + Udemy discounting lessons + lightning round and personal leadership
Archie describes Shopify’s decentralized marketing model—growth marketing, revenue marketing, brand, PMM, and consumer marketing embedded throughout the org—enabled by strong founder/president intuition but sometimes messy in practice. He then shares what he learned at Udemy about discounting as a lever (value signaling, urgency, and the emotional job-to-be-done), and closes with lightning-round recommendations and a reflection on his father’s influence.
- •Shopify has no single CMO; marketing leadership is embedded near the work (growth, sales, brand, core PMM, consumer)
- •This speeds execution but can create coordination mess; cohesion relies on strong top-level intuition
- •Udemy discounting: high list price signals value, deep discounts match willingness-to-pay and drive action
- •Education purchases often satisfy an emotional job (feeling progress), which discounting + urgency amplifies
- •Lightning round highlights books, media, an AI music tool, a planning motto, and empathy-driven leadership lessons
