CHAPTERS
- 0:00 – 0:42
Roadmap to 10–100x better AI: six levers beyond just scaling
Sarah and Elad set up the core question: what will make AI systems 10x–100x better from here, even before waiting for next-generation frontier models. Elad lays out a practical framework of six capability areas that can dramatically expand usefulness on today’s models.
- •10x–100x improvements can come from product/system layers, not only bigger base models
- •Six improvement buckets: multimodality, long context, customization, memory, recursion, and model orchestration
- •Many of these advances are ‘when’ not ‘if’ and are arriving quickly
- 0:42 – 1:42
Multimodality and long-context: making models handle real-world inputs and entire codebases
Elad explains why multimodal inputs/outputs and longer context windows are foundational upgrades. These changes enable more natural interaction (voice/images/video) and allow models to reason over large bodies of information like full repositories.
- •Multimodality: text/voice/images/video as both inputs and outputs
- •Longer context windows enable feeding whole documents or entire code repos
- •Example: coding use cases benefit from repo-level context rather than piecemeal prompts
- 1:42 – 3:16
Customization as the next frontier: fine-tuning, data quality, and task-specific performance
The discussion turns to model customization—how to adapt general models to specific tasks and domains. Elad frames customization as a major source of near-term gains and tees up recent news: OpenAI expanding fine-tuning and research on AI-generated feedback.
- •Customization includes fine-tuning, RAG, data cleaning, labeling, and other adaptations
- •You can get big gains without waiting for ‘GPT-7’—start with existing models
- •Two timely developments: OpenAI fine-tuning access and Google work on AI vs human feedback
- 3:16 – 5:18
Why OpenAI fine-tuning matters: enterprise use cases and shifting narratives
Sarah explains why OpenAI enabling fine-tuning on more capable models is a meaningful step, especially for enterprise. She contrasts general scaling-law progress with the practical reality that most applications are task-specific and benefit from adaptation.
- •Fine-tuning has existed, but expanding it to stronger models and enterprise workflows changes adoption
- •Real-world usage is task-driven; customization can outperform generic behavior in-context
- •Open question: is this a research mindset shift or primarily a commercialization move?
- 5:18 – 7:07
ChatGPT as proof: RLHF as fine-tuning that created a step-function in usability
Elad argues the market learned fine-tuning’s power when ChatGPT launched. RLHF turned a capable but less accessible base model into a widely useful product by aligning outputs with what users actually wanted.
- •ChatGPT’s utility jump came largely from RLHF, not a fundamentally new base model
- •Human preference ranking created a strong alignment layer for end users
- •Enterprise analogs: fine-tune for proprietary medical, HR, or company-specific knowledge
- 7:07 – 8:56
RAG explained: trust, citations, cost, and freshness through retrieval
Sarah breaks down Retrieval-Augmented Generation as a way to constrain answers to a chosen corpus while preserving reasoning. She highlights why teams adopt RAG: improved trustworthiness, lower retraining costs, and fast incorporation of new information.
- •RAG retrieves from a specified document set (legal, internal docs, medical research)
- •Boosts trust via controlled sources and potential citation/grounding
- •Avoids frequent retraining; improves freshness and reduces compute cost
- 8:56 – 10:57
Hallucinations and RLAIF: grounding outputs and replacing human raters with AI feedback
Elad connects RAG to reducing hallucinations and addresses the broader concern that models fabricate information. He then summarizes Google’s work suggesting AI-generated feedback can substitute for human feedback in some settings, lowering the cost of alignment.
- •Hallucinations are a key critique; RAG can restrict answers to known-valid sources
- •Regulatory concerns often stem from hallucination risk in high-stakes domains
- •RLAIF: using AI to grade AI outputs can match humans for some tasks and scale training
- •Example reference: MedPaLM 2 results implying AI can exceed expert accuracy in narrow domains
- 10:57 – 11:41
Social data and open source momentum: X training on tweets and Meta’s LLaMA bet
The conversation shifts to social networks and the model ecosystem. Elad cites X’s intent to train on platform data and notes Meta’s emergence as a major sponsor of open-source LLMs, with LLaMA/LLaMA 2 gaining developer traction.
- •Platform data (e.g., X/Twitter) may power distinctive consumer-oriented models
- •Meta is positioning itself as a key backer of open-source LLMs
- •LLaMA 2 is becoming a widely used open model in developer/enterprise ecosystems
- 11:41 – 13:19
Why Meta open-sources models: the MySQL analogy and building a durable ecosystem
Sarah compares Meta’s strategy to Facebook’s past role in hardening MySQL for internet scale—investing in an open alternative to avoid vendor lock-in. She argues that a sufficiently capable, well-supported open model can attract developers and become self-reinforcing.
- •Analogy: Facebook improved open tech (MySQL) to avoid dependence on commercial vendors
- •Meta benefits from an ecosystem alternative to expensive labs that may compete with it
- •If baseline models are high quality, developer adoption can sustain the ecosystem
- 13:19 – 15:08
Open source isn’t free: Linux/IBM lessons and Meta’s strategic motivations
Elad challenges the assumption that open ecosystems sustain cheaply, citing IBM’s massive Linux investment. Sarah clarifies Meta’s incentives: use in core consumer business, cost-sharing via community contributions, and constraints posed by centralized training requirements.
- •Historical parallel: IBM funded Linux heavily to counter Microsoft’s dominance
- •Meta could keep models internal—so open-sourcing implies strategic ecosystem goals
- •Community coordination is harder with centralized training and high compute costs
- •Meta likely monetizes indirectly through stronger consumer products, not B2B licensing
- 15:08 – 16:00
AI-native consumer engagement: chatbots, ads, and new app behaviors
Sarah outlines how generative AI is already capturing consumer attention through tools and experiences like chatbots and creative apps. She suggests Meta wants direct access to these capabilities to drive engagement and ad business without relying on external labs.
- •Consumer pull: Character-style experiences and generative creation tools draw attention
- •Meta testing/interest in chatbots points to engagement as the monetization path
- •Owning model access reduces dependency risk and protects core ad-driven economics
- 16:00 – 18:30
A new window for social startups: gen-AI as a modality shift beyond ‘Twitter again’
Elad argues social innovation has been stagnant since TikTok’s rise, partly because new networks must displace entrenched incumbents. Generative AI may reopen the design space by changing content creation and interaction modalities, creating room for both incumbents and startups.
- •Displacement is hard in social; time/attention shifts drive winners and losers
- •Startup social often regressed to clones (political variants or ‘early Facebook’ remakes)
- •Gen AI introduces new modalities that could redefine social product quadrants
- •Open question: biggest winners could still be incumbents, but startups have a new opening
- 18:30 – 20:05
AI-native feeds and cold-start: lessons from Toutiao/TikTok and ‘cold start on content’
Sarah draws from Toutiao and TikTok as early AI-native aggregators that solved relevance cold-start with rich behavioral modeling. She proposes that the next opportunity may be cold-starting the content itself—using generation to create engaging feeds and give creators superpowers.
- •Toutiao bootstrapped personalization without explicit preference selection
- •Rich per-user engagement models improved relevance and labeling loops
- •Next frontier: generative AI can help create content, not just rank it
- •Creation tools can seed network effects by making compelling assets easy to produce
- 20:05 – 23:53
From creation tool to network—and founder advice: chase low-hanging fruit, avoid ‘no GPU before PMF’
Elad and Sarah debate whether new generative tools become standalone social networks or utilities that feed existing platforms. They close with entrepreneurship guidance: in a new tech wave, prioritize easier markets and faster customer value over multi-year moonshots—culminating in Elad’s mantra.
- •Tools like Midjourney/Pika enable creation; question is ‘GIPHY vs Facebook’ outcome
- •Many founders over-focus on enterprise/infrastructure while consumer social may be underexplored
- •Early wave strategy: do the easy market opportunities first; save hardest markets for later
- •Slogan: ‘No GPU before product/market fit’—avoid heavy spend before clear demand
