Aakash GuptaThis AI Expert's Method Will Change How You Do Customer Research
CHAPTERS
- 0:00 – 2:31
Why AI user research feels unreliable—and the rigorous alternative
Aakash and Caitlin frame the core problem: most AI-driven research is messy because people ask for synthesis too early. Caitlin’s thesis is that good AI research simply replicates the disciplined, human analysis workflow—analysis first, then verification, then synthesis.
- •AI research often fails due to skipping rigorous steps
- •Gold standard: replicate how trained researchers analyze data
- •Three phases: analysis → verification/stress test → synthesis
- •Goal is reliability (less hallucination) and faster turnaround
- 2:31 – 5:57
What the full workflow covers: surveys, interviews, and automation vs agentic tools
They outline what will be demonstrated: applying the same disciplined process to interviews and survey data. Caitlin distinguishes “agentic” setups (Claude Code) from more common automation (Zapier/n8n/make), emphasizing that the process matters more than the tool.
- •Same phases apply across interviews, surveys, and short feedback
- •Agentic tools are less common in most teams; automation is more typical
- •Interview transcripts are harder because they’re long and messy
- •Systematic prompt sequencing improves reliability for tough datasets
- 5:57 – 8:22
AI-moderated interviews: market reality and why craft still matters
Caitlin discusses AI moderation tools and the wide variance in question quality and bias handling. Aakash underscores that user research is a craft—even leading AI labs can produce mediocre moderation—so the episode focuses on analysis workflows rather than AI interviewers.
- •Many AI moderation tools exist; quality ranges widely
- •Bias, leading questions, and shallow probing remain common issues
- •Anthropic’s showcased AI moderation didn’t impress Caitlin
- •Takeaway: prioritize learning rigorous research practices over hype
- 8:22 – 11:30
Step 0: Load context separately (two-step prompting) to prevent instruction loss
Caitlin introduces “context loading” as a dedicated prompt to onboard the model like a new teammate. Separating context from tasks reduces overload, increases compliance, and avoids premature synthesis.
- •Treat the model like a new team member who needs onboarding
- •Separate prompt #1 (context) from prompt #2 (analysis)
- •Long prompts can cause models to drop instructions
- •Explicitly tell the model: internalize only; do not analyze yet
- 11:30 – 17:04
Why Claude for analysis—and how markdown transcripts improve accuracy and scale
Caitlin explains why she prefers Claude’s default nuance while noting Gemini’s strength in frequency-focused summaries. She also introduces converting transcripts into structured markdown to reduce bloat, avoid token limits, and improve model comprehension.
- •Claude tends to produce more nuanced analysis by default
- •Gemini can be stronger for quick, evidenced top themes/frequency
- •Markdown conversion structures transcripts for better retrieval/accuracy
- •Reduce transcript “bloat” by extracting relevant info + metadata
- 17:04 – 24:12
Step 1 (Interviews): Per-participant analysis—extract value anchors, fragile points, and ratings
Instead of asking for themes, Caitlin forces the model to comb through each transcript and extract foundational elements per participant. She defines key constructs (value anchors, fragile points, stability rating) and specifies quote rules so outputs stay faithful to the source.
- •Force file-by-file review before any theme synthesis
- •Extract per participant: value anchors (retention drivers) and fragile points (churn risks)
- •Add a stability/churn-risk rating with explicit scoring criteria
- •Define quote length and fidelity rules (avoid “cleaning up” customer voice)
- 24:12 – 27:01
Tooling tradeoffs: Dovetail and specialized platforms vs LLM control
They compare dedicated research tools with direct LLM workflows. Caitlin highlights Breda.ai and Reveal as promising, but argues that platforms often reduce analyst control over granularity, framing, and methodology compared to carefully designed prompts.
- •Platforms can accelerate time-to-insight but reduce control
- •Breda.ai: strong prompting leads to nuanced analysis quickly
- •Reveal: grid/hypothesis-style mapping supports analyst judgment
- •LLMs + prompts provide customizable rigor for specific goals (e.g., retention)
- 27:01 – 35:05
Step 2 (Interviews): Verification via contradiction checking to avoid cherry-picking
Caitlin adds a stress-test step where the model searches for contradictions within each participant’s statements and revises risk ratings accordingly. This reduces the chance of AI “smoothing” inconsistencies and strengthens traceability for stakeholder scrutiny.
- •Verification catches issues that ‘look fine’ at first glance
- •Contradiction pass: detect inconsistencies across the same interview
- •Define what is/isn’t a contradiction with examples
- •Model can revise earlier ratings (e.g., moderate → high churn risk)
- 35:05 – 36:16
Survey analysis setup: why you must code first (not jump to patterns)
Switching to churn survey data, Caitlin explains that pattern-finding on raw open text leads to missed or miscategorized results. She introduces qualitative coding (labels/buckets) and contrasts inductive vs deductive approaches—starting inductive to avoid force-fitting.
- •Most people misuse LLMs by asking for themes immediately
- •Coding = labeling responses into meaningful, analyzable buckets
- •Inductive coding lets categories emerge; deductive uses a preset taxonomy
- •Starting inductive helps prevent AI from force-fitting responses into your tags
- 36:16 – 46:15
Survey Step 1: Create a codebook, enforce coding rules, and compute frequencies safely
Caitlin demonstrates a structured coding prompt with clear constraints (one primary code, mutually exclusive categories). She adds a safeguard for math consistency by instructing the model to use code (e.g., Python) for counting rather than unreliable text-based arithmetic.
- •Enforce mutually exclusive primary codes for clean quantification
- •Codify methodology with explicit rules and criteria
- •Have the model generate a recoded/exportable CSV for auditing
- •Instruct frequency counts to be computed via code to avoid math errors
- 46:15 – 51:23
Adding intensity ratings: a better alternative to generic sentiment for churn feedback
Rather than labeling everything ‘negative,’ Caitlin layers an emotional intensity scale (soft exit → angry exit) calibrated with definitions and examples. This helps prioritize which issues matter most, especially when combining intensity with coded reasons.
- •Sentiment analysis is often unhelpful in churn contexts (everything skews negative)
- •Use calibrated intensity levels to separate soft vs frustrated vs angry exits
- •Provide few-shot examples plus reasoning to align ratings with your interpretation
- •Rank codes by intensity to guide prioritization (not just frequency)
- 51:23 – 57:41
Step 3 (Surveys): Audit the AI’s own coding and ratings (CYA reliability layer)
Caitlin runs a verification/audit pass to recheck consistency—especially high-intensity (4–5) ratings and potential miscoding. The model surfaces corrections (renamed/refined codes, downgraded exaggerated intensity), producing a more defensible dataset before synthesis.
- •Run an explicit audit pass; models often catch their own mistakes
- •Recheck high-intensity items to prevent over-escalation in reporting
- •Audit can refine taxonomy (e.g., ‘missing favorites’ → more precise ‘lack of control’)
- •Outputs become more presentation-proof and traceable under questioning
- 57:41 – 1:08:40
Claude Code agentic ‘mega version’: parallelize interviews + surveys with reusable context files
Caitlin demonstrates an agentic setup in Claude Code where prebuilt agents pull from stored context/methodology markdown files and run both analyses simultaneously. This reduces manual prompting, supports reuse across projects, and produces polished markdown outputs for sharing.
- •Agents encapsulate multi-step workflows and retrieve from context documents
- •Parallel execution: interviews and surveys run at the same time (time savings)
- •Claude Code can locate files, troubleshoot paths, and output markdown deliverables
- •YOLO/bypass permissions exists but carries risk; use cautiously
- 1:08:40 – 1:12:29
Final outputs: executive summaries, traceable quotes, and how to use results responsibly
They review the generated markdown reports: executive summaries, per-participant tables, and traceable quotes/timestamps—plus a corresponding survey report. Aakash closes by reinforcing the core lesson: rigorous step-by-step prompting and verification enable trustworthy AI-assisted discovery.
- •Deliverables include executive summary + detailed per-participant evidence
- •Traceability enables spot-checking quotes and validating conclusions
- •Survey and interview outputs can be triangulated after parallel runs
- •Closing reminder: don’t skip analysis/verification; tailor synthesis to your role
