Aakash GuptaAI for Product Managers: 10X Growth with Smart Experimentation
CHAPTERS
- 0:00 – 1:46
How AI is reshaping experimentation (and why most teams still don’t test enough)
Aakash and Frederic open by framing AI as the biggest shift in experimentation in years, then quickly land on the core problem: most teams don’t A/B test most releases. The bottleneck is not ideas—it’s execution, especially the build phase, which keeps experimentation slow and developer-dependent.
- •AI is the largest driver of change Frederic has seen in experimentation
- •Most orgs run far fewer experiments than they could due to build constraints
- •Developers are the limiting resource; experiments wait 1–2 sprints
- •Teams over-index on prioritization meetings because shipping tests is hard
- •Sets up why AI’s main leverage is speeding the experimentation loop
- 1:46 – 4:18
The 4-step experimentation loop and where AI fits in each step
Frederic breaks experimentation into four steps—idea/assumption, build variations, configure targeting & KPIs, and analyze results. They map AI’s potential onto each step, arguing AI can heavily automate build/configure/analyze, while humans remain most critical in ideation and business framing.
- •Four steps: ideate/assume → build → configure (targeting+KPIs) → analyze
- •Build has historically been the hardest and most tool-promised, least-solved step
- •AI makes experimentation faster and more accessible across the org
- •AI can detect patterns and suggest next steps when results are inconclusive
- •Humans still needed to define hypotheses, constraints, and success metrics
- 4:18 – 6:55
AI-powered ideation: better prompts, better context, and “UX memory”
They discuss how generative AI (especially newer models) improves idea generation when fed rich product and business context. Frederic introduces the concept of an experimentation knowledge base that gives AI “UX memory,” helping teams avoid re-testing old ideas blindly and learn from historical experiments.
- •GenAI can propose test ideas when given page + business/user context
- •Best results come from strong prompting and structured context (projects, docs, screenshots)
- •PMs usually don’t lack ideas; they lack speed to test them
- •AI can retrieve prior experiments and outcomes across teams (avoid duplicated tests)
- •“UX memory” can flag prior results and suggest whether re-testing makes sense
- 6:55 – 14:12
Human-in-the-loop roles: PM, data scientist, and AI analyst across the workflow
Frederic clarifies responsibilities: PMs contribute business context and constraints; data scientists validate assumptions, modeling choices, and bias; AI accelerates configuration and analysis at scale. The takeaway is a two-layer review process—AI drafts, humans harden—especially for high-confidence decisions.
- •PM supplies business context, internal constraints, and what success means
- •Data scientist challenges AI outputs, checks plausibility and bias, selects modeling approach
- •AI can automate breakdowns/pattern detection beyond human manual analysis
- •If metrics aren’t defined/measurable, AI can’t rescue the experiment
- •Framework: AI drafts → PM sanity checks → DS validates and strengthens
- 14:12 – 16:05
Sponsor break (Jira Product Discovery + AI PM Certification)
Aakash pauses the conversation for sponsorships, highlighting tools/courses aimed at improving discovery and AI product management. This break separates the initial framework discussion from the deeper historical and technical exploration that follows.
- •Jira Product Discovery for discovery/prioritization/roadmapping connected to Jira delivery
- •Maven AI PM Certification (cohort-based learning) promoted with a discount code
- •Sets up transition into a historical view of AI’s impact on experimentation
- 16:05 – 19:43
Two waves of AI in experimentation: ML (2016) → GenAI (2022+)
Frederic provides a historical model: the ML wave improved targeting, traffic allocation, and analysis; the GenAI wave unlocked content generation and assistants; newer multimodal models enabled “vibe coding,” making it possible to turn prompts into working experiments rapidly. This evolution culminates in prompt-based experimentation inside experimentation platforms.
- •ML wave: AI targeting, multi-armed/contextual bandits, automated analysis
- •GenAI wave: content generation, RAG-based assistants, faster interpretation/support
- •Late 2023+: new models make prompt-to-production workflows feasible
- •“Vibe coding” enables rapid prototyping; experimentation needs rapid learning, not just building
- •Prompt-based experimentation aims for idea→running experiment in minutes
- 19:43 – 22:50
AI targeting: predicting intent to personalize who sees what
They dive into AI targeting as a practical, mature ML application: scoring users by intent (conversion likelihood, churn risk) based on observed behavior. Instead of manually defining segments, teams use AI-generated scores to decide eligibility and tailor experiences like discounts more efficiently.
- •AI assigns an intent score based on browsing behavior and signals
- •Replaces manual segmentation at the configuration step
- •Common use case: show coupons only to users who need the incentive
- •Analogy: a salesperson reading customer buying intent in a store
- •Positioned as adjacent to bandits, but focused on targeting/selection
- 22:50 – 28:16
Multi-armed bandit vs contextual bandit: speed vs accuracy vs personalization
Frederic explains how bandits shift from classic A/B testing (balanced splits and statistical certainty) toward performance-optimized allocation. Multi-armed bandits route more traffic to the best variant overall, while contextual bandits personalize variant selection per user segment or individual context, typically requiring high traffic to learn quickly.
- •A/B tests optimize for confidence; bandits optimize for performance during the test
- •Multi-armed bandit: allocate more traffic to the best overall variant (faster wins, less accuracy)
- •Contextual bandit: learn which variant works best for each user/context (hyper-personalization)
- •Use cases: media headlines (MAB) vs coupons/homepages personalization (contextual)
- •Both approaches typically require substantial traffic for fast learning
- 28:16 – 30:01
Opportunity detection: learning from the 80% of experiments that ‘fail’
Because most experiments don’t lift primary metrics, teams often need to dig into breakdowns to find insights. AI opportunity detection automates this drill-down, surfacing segments (e.g., mobile users) where a variant performs better and suggesting targeted personalization or follow-up tests.
- •Only ~20% of experiments are ‘successful’; most are inconclusive or negative
- •Analysts traditionally drill into segments to find why results differ
- •AI can automatically crunch data and highlight promising segments
- •Example: variant works on mobile but not desktop → personalize or retarget
- •Helps convert non-wins into learning and next actions
- 30:01 – 32:03
Sponsor break (Mobbin + Maven courses)
Aakash inserts another sponsorship segment focused on design benchmarking and cohort-based courses. This marks the transition into GenAI features like content generation and RAG assistants within experimentation tooling.
- •Mobbin for real-world UI pattern benchmarking and stakeholder alignment
- •Maven for expert-led courses; curated recommendations link provided
- •Leads into content generation and AI assistant discussion
- 32:03 – 37:45
GenAI in experimentation tools: content generation + RAG assistant for stats and code
They cover two GenAI-driven capabilities: content generation for rapid copy/messaging variants and RAG-based assistants for in-tool guidance. The assistant can recommend statistical methods (frequentist vs Bayesian) and advanced techniques like CUPED, plus generate implementation snippets for feature flags via SDKs.
- •Content generation: generate alternative copy for popups, banners, messaging tests
- •RAG assist/chatbot: answer platform questions and interpret results inside the tool
- •Assistant can recommend statistical engines and explain tradeoffs (frequentist vs Bayesian)
- •CUPED can reduce experiment duration using pre-experiment covariates/history
- •AI can generate code snippets to implement feature flags and variations faster
- 37:45 – 41:31
Vibe coding meets experimentation: prompt-based experimentation and why it matters
Frederic argues that building faster isn’t the end goal—learning faster is. Vibe coding accelerates prototypes, but only live experiments reveal true user behavior at scale; prompt-based experimentation bridges this by creating production-ready test variations directly on the website from prompts, enabling “vibe experimenting.”
- •PM goal: learn faster and kill bad features sooner—not just ship faster
- •Prototype tests can mislead; real behavior appears only at scale in production
- •Prompt-based experimentation generates variations directly on the real website/app
- •Combines creativity/speed of vibe coding with rigor of experimentation discipline
- •Enables minutes-not-weeks iteration on live experiments
- 41:31 – 49:15
Live demo: from prompt to running experiment (sorting change) + governance workflow
Frederic demonstrates creating an experiment by prompting a live site to change default sorting (price low→high). They discuss how the system finds elements, generates JS/CSS, checks accessibility and mobile, and produces a shareable preview; then they outline a stage-gated workflow where design, engineering, and data can quickly review before launch.
- •Demo: prompt changes default sorting without selecting elements manually
- •AI extracts page context, identifies targets, and generates JS/CSS as needed
- •Built-in checks: accessibility, mobile coverage, and impact preview
- •Governance: keep checkpoints, but lighter—share simulation link for review
- •Engineers can inspect generated code; data team validates goals/metrics
- 49:15 – 54:21
Beyond text prompts: mockups, sketches, and generating new UI elements
They show how the system can start from a Figma-like mockup upload or a rough sketch to generate front-end changes. Examples include building a newsletter popup with AI-generated imagery and creating new UI behaviors like quick view and add-to-cart actions, iterating through multiple prompts.
- •Input modes: text prompt, mockup import, sketch-based generation
- •AI asks clarifying questions when info is missing (e.g., what image to generate)
- •Outputs include structured reasoning plus generated JS/CSS
- •Can create net-new UI elements (popups, banners) and layout changes
- •Iterative prompting enables refinement (notifications, quick view, add-to-cart)
- 54:21 – 1:01:45
Measuring AI features: adoption, outcomes, experience—and PBX-specific speed metrics
Frederic lays out a measurement framework for AI features: usage/adoption, user outcomes, and user experience feedback (e.g., thumbs up/down). For prompt-based experimentation, they add operational metrics like time from first prompt to live, number of prompts required, and how often developers still need to intervene.
- •Three core AI feature measures: adoption, outcome, and experience feedback
- •Feedback loops matter because unhappy users silently stop using AI features
- •PBX metrics: prompts-to-experiment, time-to-live, developer involvement rate
- •Goal: reduce experiments requiring code from ~80% to ~20%
- •Measure value delivered (speed to learning), not just raw usage
- 1:01:45 – 1:07:07
Measuring RAG systems: accuracy, relevance, context quality + LLM-as-judge evals
They define three technical RAG quality metrics—accuracy (faithfulness), relevance (answers the question), and context quality (right/updated sources). Frederic describes operationalizing this with evaluation suites and an LLM-as-judge approach to score relevance and generate related questions as a proxy check.
- •Accuracy/faithfulness: answers must match documentation and be correct
- •Relevance: response must address the user’s actual question
- •Context quality: retrieved docs must be current and appropriate
- •LLM-as-judge can score outputs and generate similar questions to test relevance
- •Avoid stopping at usage; validate trustworthiness and helpfulness
- 1:07:07 – 1:15:11
Experimentation excellence and common PM misconceptions (Booking.com, traffic, speed)
Frederic names Booking.com as a standout due to its deep experimentation culture where nearly nothing ships without testing. He closes by addressing common PM objections—experimentation slows delivery, there isn’t enough traffic, or discovery is sufficient—arguing instead that experimentation accelerates learning, can start small (flags → rollouts → A/B), and complements discovery by measuring what users do versus what they say.
- •Booking.com exemplifies rigorous experimentation culture
- •Start simple: feature flags → progressive rollouts → A/B testing at scale
- •Misconception 1: experiments slow delivery; reality: they speed learning and iteration
- •Misconception 2: not enough traffic; consider conversions and avoid over-trusting tiny qual samples
- •Misconception 3: discovery is enough; discovery + experimentation provide full picture
