By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

Thank you for 1m downloads of the podcast and 2m readers of the Substack! 🎉 This is the audio discussion following The Winds of AI Winter essay that also serves as a recap of Q2 2024 in AI viewed through the lens of our Four Wars…

Alessio Fanellihostswyxhost

Aug 2, 20241h 23mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Singapore studio catch-up and why they’re recording together
Alessio and swyx open with a long-awaited one-on-one recorded in Singapore, explaining why it took time to schedule and how the podcast has grown. They discuss travel, a depleted guest backlog, and the plan to cover a lot of news in a rapid-fire format.
- •They frame this as a rare one-on-one due to heavy travel and prior guest backlog
- •They note the show’s growth and the need to rebuild the guest pipeline
- •They set expectations for a faster, news-heavy recap format
Sovereign AI Summit takeaways and what countries should invest in
They recap hosting the Sovereign AI Summit in Singapore and what surprised them about the sophistication of questions. The discussion frames sovereign AI as a productivity strategy for shrinking workforces, and raises the practical question of where countries should place bets across models, infrastructure, GPUs, and agents.
- •Alessio describes Singapore impressions and the event they hosted
- •They list attendees like Nvidia, Goldman, Temasek, GIC, and Singtel
- •They outline sovereign AI decisions: foundation models vs data centers vs GPUs vs agents
- •They note senior finance leaders asking about alternatives like state space models
- •swyx argues AI engineering is a path for countries without massive GPU clusters
Re-centering on the “Four Wars of the AI Stack” framework
They reintroduce their recurring framework—GPU Rich vs GPU Poor, Data Quality Wars, Multimodality Wars, and RAG/Ops Wars—and explain why it still holds up months later. This sets the structure for bucketing the last few months of developments.
- •They name the four wars and use them as an organizing lens
- •They argue most AI news still fits these buckets even after months of change
- •They note they’re recapping multiple months of updates at once
Frontier model momentum: Claude 3.5 Sonnet, interpretability, and benchmark limits
swyx highlights Claude 3.5 Sonnet’s sustained top position on certain public leaderboards and calls it a meaningful shift away from an OpenAI-centric world. They discuss the interpretability angle—sparse autoencoders and feature “control vectors”—and why classic benchmarks like MMLU are increasingly insufficient.
- •swyx says Claude 3.5 Sonnet has held a top spot on LMSYS-style benchmarks for over a month
- •He describes a shift where tooling is being built primarily for Claude instead of OpenAI
- •He summarizes interpretability claims: training sparse autoencoders to identify features and steer behavior without retraining
- •Alessio compares Claude vs OpenAI on their internal summarization workflows and describes it as a “vibes” advantage
- •swyx argues MMLU is saturating and no longer reflects real capability differences
Llama 3.1 as “the synthetic data model” and the open recipe for post-training
They turn to Meta’s Llama 3.1 release, emphasizing not just the model, but the explicit paper detailing how synthetic data is used across multiple domains. Both stress that the key shift is having an open path to improve smaller models without relying on GPT-4 or Claude outputs.
- •Alessio focuses on training/fine-tuning on model outputs and the opportunity to do this without using GPT-4 outputs
- •swyx says synthetic data is now established; the next question is how to use it well
- •swyx lists domains where Llama uses synthetic data: code, math, multilingual, long context, tool use, ASR, and voice generation
- •They describe Llama’s paper as providing a usable roadmap/recipe for others
- •They discuss a broader “vibe shift” where Anthropic and Meta releases are treated as serious peers
The mid-tier frontier and “vibe shift” across labs: Phi, Gemma 2, and Mistral Large 2
They discuss how successive releases affect perceptions of labs’ trajectories, citing Phi and Gemma improving over versions. Then they assess Mistral Large 2 as less exciting than Llama 3.1, raising questions about Mistral’s positioning between open weights, licensing, and differentiation.
- •swyx argues release-to-release momentum is a key signal of a lab’s health
- •He cites Phi improving from earlier skepticism and Gemma 2 becoming widely used
- •They describe Gemma 2’s improved reception compared to Gemma 1
- •They discuss Mistral Large 2’s muted excitement relative to Llama 3 and note its research/non-commercial licensing
- •swyx argues Mistral needs to be “superlatively good” in at least one dimension and suggests they’ve ceded the open-source “crown” to Meta
Nvidia ecosystem lock-in and the efficiency frontier: FlashAttention 3 and custom ASIC economics
They shout out FlashAttention 3 and use it to illustrate how optimization work reinforces Nvidia’s advantage. swyx explains why Nvidia’s ecosystem is hard to replicate, but also relays an argument that very large training runs could justify custom ASICs economically.
- •Alessio highlights FlashAttention 3 as an example of small teams impacting the whole industry
- •swyx says FlashAttention 3 works for Nvidia H100s, reinforcing Nvidia’s ecosystem advantage
- •He argues validating and scaling these optimizations is expensive for competing hardware ecosystems
- •He relays an argument that at sufficiently high training costs, custom ASICs become economically rational if they cut costs significantly
- •They briefly connect this to ongoing interest in custom silicon
Noam Shazeer/Character.AI serving-scale tricks and hybrid attention architectures
They discuss a Noam Shazeer post describing practical inference-cost reductions at Character.AI, including int8-related ideas and hybrid global/local attention. The conversation broadens into a view that future models will likely mix architectural building blocks rather than rely on pure transformer stacks.
- •swyx recounts Noam Shazeer sharing “five tricks” for large-scale LLM serving at Character.AI
- •They react to a claim comparing Character.AI inference volume to a fraction of search traffic and discuss multi-turn dynamics
- •swyx highlights native int8-related training as sensible given quantized inference targets
- •He explains Character.AI’s global-local hybrid attention ratio and connects it to other hybrid research trends
- •They suggest there is still significant room to improve inference efficiency
GPU poors and on-device LLMs: Mozilla Llamafile, Chrome’s Gemini Nano, and Apple Intelligence routing
They shift from data center scale to on-device execution, surveying approaches from universal binaries to OS/browser-integrated models. Alessio questions differentiation at small sizes, while swyx emphasizes routing/adapters in Apple’s approach and argues the open web may depend more on browser-native models like Gemini Nano.
- •swyx frames on-device models as the “GPU poors” part of the stack war
- •He highlights Mozilla’s Llamafile vision: fast local inference even on CPU and a cross-architecture executable approach
- •They survey layers: Llama.cpp/MLX-style runtimes vs vendor-integrated models in Chrome and Apple’s OS
- •Alessio questions whether users will even notice model differences at smaller sizes
- •swyx argues Apple Intelligence is effectively a routing layer (including adapters and external model routing) and discusses implications for privacy and cost
Quality Data Wars: NYT lawsuit dynamics, licensing deals, Reddit, and creator-rights disputes
They summarize the ongoing legal and deal-making backdrop: news orgs choosing to fight or partner, platforms monetizing user-generated content, and high-profile creator cases. The segment covers Reddit’s licensing revenue, crawling restrictions, and entertainment industry lawsuits against music generators.
- •They note the NYT lawsuit continues with limited public incremental updates
- •Alessio says OpenAI’s strategy is pushing NYT to prove originality/value of its content
- •swyx contrasts “fight vs partner” approaches and notes other outlets may do deals instead
- •They discuss Reddit’s significant licensing revenue and scrutiny around exclusivity and crawling rules
- •They mention Scarlett Johansson’s public complaint about voice similarity and the RIAA actions against Udio and Suno
Synthetic data and math breakthroughs: AlphaProof, IMO scoring, and “jagged intelligence”
They cover Google’s AlphaProof performance near an Olympiad gold threshold and situate it within broader work on verifier/search-driven math solving. They contrast spectacular narrow-domain performance with basic failures in other areas, labeling the gap as jagged intelligence and questioning how general intelligence emerges.
- •They describe AlphaProof as missing an IMO gold threshold by a narrow margin and note perfect scores on attempted questions
- •swyx mentions related work and competitions focused on mathematical reasoning/search/verifiers
- •They introduce “jagged intelligence” to describe uneven capability across tasks
- •They debate whether progress leads to one general model or a mix of specialized models
- •swyx proposes a sequence: capability in a domain, then generalization, then efficiency
Multimodality War: GPT-4o voice rollout reality, live demo constraints, and Meta’s multimodal roadmap
They discuss the gap between impressive voice demos and delayed rollout, including the challenges of staging a live real-time voice system in a noisy venue. Then they shift to Meta’s multimodal plans, noting Llama’s eventual multimodality via adapters and highlighting Chameleon as an early-fusion approach.
- •They note GPT-4o voice still wasn’t broadly available at the time and discuss beta rollout expectations
- •swyx describes the AI Engineer World’s Fair live demo and the technical challenge of audio feedback/self-voice suppression
- •They say Meta’s Llama is expected to become multimodal and draw connections to prior Meta voice projects
- •swyx highlights Chameleon as natively early-fusion for vision-language-text interleaving vs late-fusion adapters
- •They mention controversy around releasing open weights without certain capabilities and community attempts to add them back
PaliGemma and business OCR-style multimodal workhorses; Google’s developer experience push
They give a focused update on small multimodal models, especially CoPaliGemma for extracting structured information from PDFs. They also discuss Google’s internal product tension (AI Studio vs Vertex) and the opportunity to rebuild a developer-friendly platform experience.
- •They describe PaliGemma as a small release and CoPaliGemma as increasingly important for PDFs and structured extraction
- •swyx says CoPaliGemma combines a retrieval approach with a vision model to improve document extraction workflows
- •Alessio says Google has been executing well across several releases and platform features
- •swyx describes Vertex’s reputation for complexity and AI Studio as a more developer-friendly alternative
- •They frame this as Google’s chance to better serve AI engineers who dislike heavy cloud bureaucracy
Renaming RAG/Ops to “LLM OS”: what LLMs need to plug into (tools, ops, and product primitives)
swyx renames the RAG/Ops war to “LLM OS,” arguing the ecosystem must support more than chat: coding, tool use, and agents. They debate the value of ops vs capabilities, discuss why current tooling is fragmented, and outline key ops categories (frameworks, gateways, monitoring/tracing).
- •swyx proposes “LLM OS” to capture the broader integration layer beyond chatbots
- •He suggests the GPT builder interface reveals “every button as a startup” opportunity
- •Alessio describes investing in E2B and frames code execution as an OS primitive for LLM apps
- •swyx outlines three ops buckets: frameworks, gateways, and monitoring/tracing
- •They criticize fragmented tooling integrations and question what truly needs to be AI-native vs reused from existing ops stacks
Vector DB wars to memory layers, GraphRAG hype, and the agent tooling/protocol reality check
They argue vector databases are too low-level and that higher-level memory layers are emerging, especially for conversational memory. They touch on GraphRAG enthusiasm with skepticism, then move into agent tooling and protocol standardization challenges, noting incentives push teams toward proprietary approaches.
- •swyx says vector DBs (cosine similarity + retrieval) aren’t sufficient out of the box and must move up the stack
- •They discuss “memory databases” for long-term conversational memory vs document retrieval
- •They mention GraphRAG interest and warn that “everything is a graph” waves often fade
- •They discuss agent tooling momentum (including a rebranding/pivot around OpenDevin/All Hands AI)
- •They describe why agent protocol standardization is hard early on because companies want to own proprietary standards and monetize first
Macro trends above the four wars: model cost depreciation, vertical “AI employees,” and post-MMLU benchmarks
They close with higher-level trends: rapid shifts in price-efficiency frontiers, the implications for startups and product planning, and why verticalized “service as software” businesses may capture value sooner. They also list alternative evaluation directions beyond MMLU, and end on a tongue-in-cheek safety note referencing CrowdStrike.
- •swyx describes a tightly moving price-efficiency frontier and says cost per intelligence is dropping very fast
- •They discuss whether future frontier releases change the efficiency trend and emphasize distillation from larger models
- •Alessio argues many companies want AI to do labor rather than buy tooling, citing examples like Brightwave and Dropzone’s alert analysis
- •swyx enumerates multiple benchmark directions beyond MMLU (reasoning, instruction following, code, context use, function calling, vision, multilinguality) and distinguishes general benchmarks from product-specific evals
- •They end with a joke that CrowdStrike could serve as a “kill switch” for agents, then wrap up and mention AI News’ growth

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.