By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howCHAPTERS
Jump to publisher timestampsChapter summaries
Design Arena as a product-led benchmark for creative model performance
Grace Li explains that Intelligence benchmarks frontier models in hard-to-verify, subjective domains by embedding evaluation inside a creation product called Design Arena. Users come to build websites, images, or videos, and their interaction signals become real-world evaluation data that Intelligence uses to help model teams improve.
- •Intelligence focuses on benchmarking models in creative, non-verifiable domains
- •Design Arena is positioned as a default place to build creative outputs with models
- •User interaction signals (requests, edits, preferences) are transformed into evaluation signals
- •The company feeds these insights back to frontier labs to improve model behavior in subjective tasks
Turning user behavior into model diagnostics (without hurting UX)
Grace details how the product captures human-computer interaction and converts it into measures of user enjoyment and preference. She emphasizes that the labeling/evaluation must be embedded into a good product experience rather than asking users to explicitly rate everything.
- •Users arrive with a creation goal (e.g., build a website) and generate iterative feedback through edits
- •Intelligence acts like a product manager converting raw signals into actionable model feedback
- •The goal is diagnosing where models fail so teams can improve them
- •Evaluation data collection must be integrated into UX rather than relying on explicit user labeling
Why Grace left big-company leverage for startups
Grace describes her path through big tech and quant, and why she wanted a setting where effort could translate into outsized impact. She frames early-stage startups as uniquely leveraged when the product aligns with strong market demand.
- •Big-company environments can feel low-leverage even with high individual effort
- •Early-stage startups can compound effort into disproportionate outcomes if the market wants the product
- •She describes the work as hard but rewarded by faster feedback and returns
- •This leverage mindset set up the motivation to start the company
From “AI can code games” to “AI can’t make them fun”
Grace explains the company’s earliest idea: building a gaming company after noticing models could output functional games. The team found the deeper problem was not correctness but taste—games worked but weren’t enjoyable—leading them toward creative evaluation as the core challenge.
- •Initial concept was building games generated by models
- •At the time, models could only output limited code (e.g., one long file) and workflows were clunky
- •Games were functional but lacked good design, story, and addictiveness
- •This highlighted the gap between verifiable correctness and subjective quality
Non-verifiable domains need a “North Star” signal
Grace challenges the idea that models can only reliably improve on domains with clear right answers. She argues humans improve in subjective domains because there are learnable signals, and the missing piece is a scalable way to define what “better” means.
- •Prevailing view: models improve fastest on verifiable domains and struggle on subjective ones
- •Grace argues subjective improvement is possible because humans demonstrably improve in those skills
- •The bottleneck is the lack of a clear objective function for subjective quality
- •The team searched for scalable ground-truth signals of preference
The kernel of Design Arena: a double-sided preference system at scale
Grace describes realizing games (and creative outputs) can be evaluated via population behavior signals like engagement and retention. This led to the idea of a double-sided product: humans use models to create things they love, while models learn what people love at scale.
- •Engagement/retention can function as a ground-truth signal for whether something is “good”
- •The product aims to serve humans while simultaneously collecting preference data for models
- •Grace frames it as superintelligence accessing humans to learn preferences at scale
- •She says Design Arena scaled to millions of users across many countries
Comparison-based UX: improving creation while collecting preference labels
Grace explains why showing multiple variants is both a better user experience and a better evaluation mechanism. Rather than asking users to label outputs explicitly, the product structure naturally elicits preferences through choices and iterative refinement.
- •Comparison between multiple variants produces higher-quality preference signals
- •The multi-option format mirrors how designers present mockups to clients
- •Explicit labeling would harm UX; signals must be embedded in the workflow
- •The product design aligns user success with benchmark data collection
Why evals matter: diagnosing subjective failures requires population-scale data
Grace explains that model improvement depends on knowing where models break, and subjective failures can’t be reliably diagnosed from a single person’s opinion. She argues that in non-verifiable domains, evaluation must come from large-scale preference data rather than individual expert judgment.
- •Evals guide improvement by revealing where models fail
- •Individual preference doesn’t indicate population-level preference or cohort-specific needs
- •Subjective verification is better done with population-level systems than single experts
- •Scale is required to distinguish real model issues from idiosyncratic taste
TikTok as an analogy for building a subjective verifier
Grace uses TikTok to illustrate how platforms can infer subjective quality through behavior rather than relying on “high taste” individuals. She emphasizes that different cohorts value different outputs, so the system must learn from aggregate behavior signals.
- •Behavioral signals (like time spent) can serve as verification in subjective domains
- •TikTok’s strength is algorithm + population data, not a single arbiter of taste
- •What counts as “high taste” varies across cohorts
- •The analogy supports Intelligence’s approach of using large-scale real usage data
Preference drift and forecasting what users will want next
Grace describes seeing how user stated preferences change over time and how this drift itself becomes valuable. She notes that some frontier labs are thinking about using these trends to predict what people will want in future quarters.
- •Intelligence’s vantage point comes from large-scale data across users
- •Stated preferences shift over time, and Q1 wants can differ from later in the year
- •The company observes both stated vs revealed preference framing
- •Some labs consider using preference drift trends to shape future model outputs
Many intelligences: models optimize for different demographics and cohorts
Grace argues that “better” is not universal in subjective domains because different labs target different user cohorts. She describes this as resembling an ads marketplace, where the objective function varies by user and task.
- •Different frontier labs may target different demographics and preferences
- •Subjective quality depends on which cohort you are optimizing for
- •This implies different post-training data is needed for different targets
- •Grace likens the landscape to an ads-style marketplace with varied objective functions
A marketplace for specific intelligence (not one general model)
Grace lays out her view that the future may involve chaining specialized components rather than relying on one expensive general model for every task. She positions Intelligence as a marketplace that selects the best model for a user’s price, time, and preference constraints, without being tied to a single provider.
- •Specialized models can be preferable when tasks don’t require maximum general intelligence
- •The future system may chain multiple specifically intelligent components
- •Diversity of providers (open source, application layer, labs) supports a marketplace role
- •Intelligence aims to route users to the best model given constraints, rather than only offering its own
Scaling fast with a small team: alignment, trust, and “never giving up on a person”
Grace explains how small teams can move quickly by maintaining shared problem-solving frameworks and high trust. She argues that speed collapses when teammates stop engaging with disagreements and instead default to compliance or internal narratives about others being irrational.
- •Misalignment in decision-making frameworks is a major source of slowdown at scale
- •Small teams can afford deep resolution of disagreements instead of giving up
- •A culture of trust reduces the need to second-guess motives or reasoning
- •Grace describes this as building a shared framework so people stay aligned while moving fast
Hiring for learning velocity, action bias, and handling rejection while building the future
Grace describes the top trait she seeks in early hires: the willingness and ability to learn anything quickly as the environment changes. She shares the “idiot in motion” principle—acting beats overthinking—and offers a reframing of rejection as often less absolute than it feels, before closing on what it means to “live in the future” and what she wants to scale next.
- •The most important hire trait is high learning ability amid rapid change
- •An under-leveling teammate becomes a compounding liability in a fast-moving environment
- •“Idiot in motion” beats “genius at rest”; doing creates learning and luck surface area
- •Rejection feels amplified to the recipient; she shares a recruiting anecdote to illustrate
- •Working on the frontier means running experiments that make the team early to new capabilities; next focus is scaling what works with growth, hiring, and internal systems
