Lenny's PodcastThe UX Research reckoning is here | Judd Antin (Airbnb, Meta)
CHAPTERS
- 0:00 – 0:52
“User-centered performance”: when research is a checkbox, not learning
Judd opens with a critique of teams using user research symbolically—late-stage “validation” studies that won’t change decisions. He frames a healthier mindset: research should try to falsify assumptions, not confirm them.
- •“User-centered performance” as signaling customer obsession vs. seeking truth
- •Late “quick studies to validate” are often too late to affect outcomes
- •Mantra: don’t validate—falsify; actively look for how you might be wrong
- •Ego and confirmation bias push teams toward research-as-justification
- 0:52 – 5:23
Who Judd Antin is and why this episode is “spicy”
Lenny introduces Judd’s background across Meta/Facebook and Airbnb, and his current consulting work. The episode’s goal is set: unpack what’s broken in how research is practiced and integrated, and what must evolve.
- •Judd’s leadership history building research practices at major companies
- •The episode will challenge common assumptions about UXR’s role
- •Themes: impact, integration, business alignment, and misuse of research
- 5:23 – 7:31
Why “The UX Research Reckoning Is Here” struck a nerve
Judd shares how his post unexpectedly spread widely and sparked intense debate. He clarifies misconceptions—he’s not blaming researchers—and highlights pushback, especially around his call for profit focus.
- •Surprise at the reach and intensity of the response
- •“Reckoning” as taking stock, not doomposting
- •Critiques: ‘throwing researchers under the bus’ and leadership accountability
- •Most surprising backlash: anti-capitalist resistance to profit orientation
- 7:31 – 8:54
Layoffs as a signal: the system isn’t delivering research impact
The conversation turns to why UXR was hit hard in layoffs and what that reveals about organizational failures. Judd argues research remains crucial, but companies often set it up as a reactive service function that can’t demonstrate value.
- •Layoffs prompt a “moment to take stock” about research value
- •Impact problems stem from both researcher practices and org integration
- •ZIRP-era overhiring exposed weak operating models for research
- •Service-function positioning leads to low influence and easier cuts
- 8:54 – 14:14
Macro vs. middle-range vs. micro research—and why “middle” disappoints
Judd introduces his framework: macro (strategic/market), micro (usability/implementation), and middle-range (globular user understanding). He argues the industry over-invested in middle-range work that’s interesting but often hard to operationalize and easy to dismiss.
- •Macro: strategy, market, TAM, competitors, forward-looking innovation
- •Micro: usability, pixel-perfect execution, rapid product improvement
- •Middle-range: broad “how do users feel/behave” questions lacking business targeting
- •Middle-range triggers: “obvious” critiques, post-hoc bias, and low actionability
- 14:14 – 17:31
The structural fix: integrate research from the start (not at the end)
Judd explains the vicious cycle: research is brought in too late, so it can’t shape questions, builds weak relationships, and produces lower-impact outputs—leading to more sidelining. The remedy is continuous engagement where researchers help frame the right questions early and throughout.
- •Reactive research can’t shape the question, only react to it
- •Being “in the room” changes framing, speed, and decision quality
- •Constant partnership turns researchers into a living repository of insights
- •Breaking the cycle requires both process change and relationship depth
- 17:31 – 19:54
Traits of great researchers: the five-tool (plus) Swiss Army knife
Lenny asks how to tell if a researcher is great. Judd argues the best researchers are multi-method—strong in both qualitative and quantitative approaches, and increasingly technical.
- •Formative/generative research for innovation and context
- •Evaluative/usability research for product quality and micro improvements
- •Rigorous survey design as scalable signal
- •Applied statistics to interpret data and experiments
- •Technical skill: SQL historically; increasingly dashboards and AI/prompting
- 19:54 – 21:10
How to interview and evaluate researchers for multi-method thinking
Judd shares his approach for hiring: give a “juicy” open-ended problem and look for a layered plan across time horizons and methods. He also emphasizes team composition—individual T-shapes plus collective coverage across the full toolset.
- •Use scenario questions to reveal method flexibility
- •Look for plans that scale: “day vs. week vs. month” approaches
- •Avoid “hammer/nail” specialization that narrows the answer space
- •Build team-level coverage where specialists complement each other
- 21:10 – 29:13
Business focus without losing empathy: finding the overlap of user and profit
Judd argues researchers must speak the language of business outcomes while preserving user-centered practice. He gives concrete steps: learn the funnel, OKRs, strategy docs, and earnings narratives so insights can be used like a scalpel against real business problems.
- •Goal is the overlap (“Venn”) between user value and business success
- •Researchers should learn metrics, funnel stages, and strategy/OKRs
- •Read shareholder letters/calls (or internal equivalents) to gain fluency
- •Impact increases when insights are tightly aligned to a business lever
- 29:13 – 32:54
Intuition vs. evidence: how to check your gut (and use researchers to do it)
They explore the right balance between intuition-led product judgment and evidence-led learning. Judd frames gut as pattern-matching plus bias, advocating System 2 thinking and “wisdom of the crowd,” with researchers helping teams surface blind spots and improve shared mental models.
- •Intuition is valuable but also where cognitive biases live
- •Use System 2 (slow, analytic thinking) as a practical gut-check
- •Wisdom of the crowd works when inputs are diverse and independent
- •A key research success metric: teams won’t hold key meetings without them
- 32:54 – 44:55
PM–research friction: common tropes, A/B testing limits, and hindsight bias
Judd lists recurring PM misconceptions: research is slow, PMs can do it themselves, and A/B testing should replace research. He also addresses the “we knew this already” reaction, explaining hindsight bias and narrative fallacy as reasons teams dismiss insights.
- •“Research is too slow” vs. reality: can be day/week/month based on need
- •“I can do my own research” and why method/rigor matters (garbage in/out)
- •A/B tests tell you what happened, rarely why—research fills the ‘why’ gap
- •Hindsight bias: “Everything is obvious if you already know the answer”
- •Narrative fallacy: we retrofit neat stories that distort how learning happened
- 44:55 – 46:53
Actionable research outputs: frameworks, not ‘research says build X’
Reacting to Patrick Collison’s framing, Judd argues research should improve the team’s mental model and decision-making framework. Recommendations can be valuable, especially for micro issues, but researchers overstep when they present a single “therefore build this” directive for complex problems.
- •Best research delivers: what → so what → then what (without overreach)
- •Micro findings can directly suggest fixes (e.g., UX clarity issues)
- •Macro and complex middle-range work should produce shared frameworks
- •Decisions remain team-based, blending constraints, strategy, and evidence
- 46:53 – 56:49
How teams should leverage researchers: unified insights, ruthless prioritization, and field closeness
Judd’s advice spans both sides: researchers should broaden skills, deepen business literacy, and communicate crisply; companies should stop siloing insights and integrate research into product workflows. PMs should help prioritize and actively participate—sometimes literally in the field.
- •Researchers: diverse skills, business fluency, strong communication to different audiences
- •Fewer projects done better; avoid spreading one researcher across too many teams
- •Companies: integrate UX research, market/consumer insights, DS, and VOC into one system
- •PM partnership: ruthless prioritization and shared ownership of outcomes
- •Field involvement builds shared understanding and faster alignment
- 56:49 – 1:03:40
Staffing and leverage: the ‘right’ researcher ratio and why startups still need insights
Judd rejects a universal researcher-to-team ratio, arguing the real unit is the relationship: you have enough when key teams have consistent research partners. He also challenges startup hiring patterns that exclude research, arguing a strong insights generalist can accelerate pivots and reduce guessing.
- •Organizing principle: protected, consistent partnerships—not broad coverage maps
- •Create “pain” by prioritizing depth over thin coverage to justify headcount growth
- •Early vs. late stage needs differ, but research can add value at any stage
- •Startups: insights reduce “moral luck” in pivots by adding evidence to judgment
- 1:03:40 – 1:14:34
Hot takes to close: NPS is broken, dogfooding has blind spots, then lightning round
Judd argues NPS is a poorly constructed metric with weak comparability and recommends CSAT instead. He also cautions that dogfooding is useful for spotting issues but risky for prioritization because internal users differ sharply from real customers; the episode ends with quick personal recommendations and wrap-up.
- •NPS critique: flawed scale design, labeling, mobile UX, and benchmark comparability
- •CSAT as a better-behaved and more meaningful satisfaction metric
- •Dogfooding: good for issue discovery; dangerous for deciding severity/priority
- •Lightning round: books, shows, interview question, products, and stoic motto
- •Where to find Judd and his call to ‘get next to your researcher’
