The Twenty Minute VCZico Kolter: OpenAI's Newest Board Member on The Biggest Questions and Concerns in AI Safety | E1197
CHAPTERS
- 0:22 – 1:29
Zico’s background: CMU machine learning leadership and joining OpenAI’s board
Harry opens by asking Zico to quickly establish his credentials and current roles. Zico explains his academic position at Carnegie Mellon and his recent appointment to OpenAI’s board, setting up why he’s focused on both capability and safety questions.
- •Head of CMU’s Machine Learning Department and long-time ML professor
- •Recently joined OpenAI’s board and what that implies about his perspective
- •Framing the conversation as both technical foundations and safety/governance
- 1:29 – 4:18
How modern LLMs work: next-word prediction—and why that still yields intelligence
Zico explains the core training objective of LLMs: predicting the next token from large-scale internet text. He argues that dismissing LLMs as “just word prediction” misses the scientific surprise that this simple objective can produce coherent, seemingly intelligent behavior.
- •LLMs trained on internet-scale text to predict next words/tokens
- •Generation is repeated next-token prediction chained into long responses
- •Why “just word prediction” doesn’t imply lack of intelligence
- •LLMs as a major scientific discovery of recent decades
- 4:18 – 7:11
Are we running out of data? High-quality text scarcity vs untapped multimodal and private corpora
The conversation turns to whether model progress will stall due to data limits and whether synthetic data is required. Zico argues that while high-quality public text is finite, the world contains far more usable data (including multimodal and enterprise/internal data) than what models currently consume.
- •‘Another Wikipedia’ doesn’t exist—premium public text is limited
- •Training corpora sizes (tens of TB) are small relative to global data
- •Large reserves in multimodal data: video, audio, time-series signals
- •Compute and usability, not raw availability, are the real constraints
- 7:11 – 9:09
Using multimodal data well: compute explosion and cross-modal generalization challenges
Harry presses on what makes video/audio data hard to leverage. Zico highlights the huge size gap between raw video and text, and the resulting compute burden, while arguing multimodal signals are central to human intelligence and likely essential for future model capability.
- •Video/audio are orders of magnitude larger than text representations
- •Compute and training efficiency are the biggest barriers
- •Need better ways to learn and transfer across modalities
- •Rationale: humans rely heavily on spatiotemporal/visual experience
- 9:09 – 12:13
Why performance may keep improving even with fixed data: scaling, repeated passes, and better extraction
Zico explains why model performance doesn’t necessarily plateau even if new data were limited. He points to continued gains from larger models on fixed datasets and the idea that current methods don’t extract maximal information—synthetic or self-generated data may help unlock more value.
- •Bigger models can improve even on the same dataset
- •Re-training/iterating over data can still yield gains
- •Current algorithms don’t fully extract information from available data
- •Synthetic-data loops can sometimes improve downstream performance
- •Open research question: how to maximize information extraction
- 12:13 – 14:09
Many small models vs a few big general models: no equilibrium yet
The discussion moves to whether the future is dominated by small efficient models or large general-purpose systems. Zico says the field hasn’t found a stable ‘steady state’ and notes he personally defaults to the largest available models for generality, with small models becoming useful once tasks become repetitive and well-defined.
- •No clear equilibrium for model size vs application yet
- •Large models win for general-purpose, one-off, evolving workflows
- •Small/specialized models make sense for repeated, narrow tasks
- •Expect ongoing experimentation before the landscape stabilizes
- 14:09 – 16:14
Perceived diminishing gains are often benchmark and usage artifacts—real gains show up in harder tasks
Responding to claims that improvements are now marginal, Zico argues this is partly due to people asking models the same ‘easy’ questions. In demanding workflows—especially coding and transcription—he sees major gains that let users operate at higher levels of abstraction.
- •Benchmarks and user habits can hide meaningful improvements
- •Many common prompts are already ‘good enough’ on smaller models
- •Harder tasks (coding, lecture transcription) still show strong gains
- •Users’ imagination and workflows lag behind model capabilities
- 16:14 – 17:49
Commoditization of models and industry consolidation: why training from scratch won’t stay viable for most
Harry asks how commoditization changes the competitive landscape. Zico predicts consolidation and argues many companies training models ‘because they can’ will find it economically unjustifiable as strong open-weight and leading closed models become widely available.
- •Rapid proliferation of both open-weight and closed models
- •Early “me-too” training runs may not be economically rational long-term
- •Likely consolidation among model providers
- •Most firms won’t sustainably train frontier models from scratch
- 17:49 – 19:09
Compute as the third pillar: scaling still works, but economics and efficiency matter more than hard limits
The conversation turns to claims of compute-driven diminishing returns. Zico says scaling laws still suggest room to run, while emphasizing that the real constraints are cost trade-offs (training and inference) and that algorithmic efficiency may outperform brute-force scaling.
- •Skepticism about “compute plateau” claims; scaling laws still trend upward
- •Compute remains a major driver of capability improvements
- •Brute scaling may not be the most efficient route to progress
- •Economics (training and inference costs) increasingly shape decisions
- 19:09 – 22:41
AGI vs profitable products: defining AGI as a year-long collaborator and embracing uncertainty
Harry asks whether companies are chasing AGI or just products. Zico defines AGI as a system comparable to a close collaborator over a year-long project, says he now believes it could arrive in his lifetime, and stresses huge uncertainty in timelines.
- •AGI definition: functionally equivalent to a close collaborator over a year
- •Not mutually exclusive: product progress can also move toward AGI
- •Updated belief: AGI possibly within lifetime, but timing uncertain (wide range)
- •Embodiment likely slower; virtual collaboration is the nearer target
- 22:41 – 27:55
Enterprise adoption realities: data fears, on-prem demands, and why RAG fits access-control constraints
Harry asks about enterprise reluctance to share data and potential movement back on-prem. Zico argues much fear stems from misconceptions about training vs inference, explains why ‘dump all data into a model’ fails due to access rights, and describes RAG as a pragmatic pattern that respects permissions.
- •Enterprises often misunderstand what gets trained on vs what’s queried
- •Central obstacle: access control/permissions within corporate data
- •RAG (Retrieval Augmented Generation) as a practical, permission-respecting approach
- •On-prem vs cloud tension is partly perception and communication/marketing
- 27:55 – 32:40
Misinformation and the collapse of trust: AI accelerates a return to ‘trust your tribe’ epistemology
Zico addresses deepfakes and misinformation, arguing the biggest harm is not believing falsehoods but believing nothing—eroding shared objective reality. He frames this as a reversion to how humans historically formed beliefs (trusting close associates), with institutions and brands becoming focal points for trust—though contested.
- •Primary risk: societal disbelief and erosion of objective shared facts
- •AI didn’t invent misinformation but dramatically increases scale and speed
- •Historical lens: humans evolved relying on trusted in-groups, not recordings
- •Brands/institutions may become trust anchors, but trust is fragmented already
- 32:40 – 37:14
Regulation and governance: easier to regulate downstream harms than fast-changing technical internals
Harry challenges whether governments can regulate highly technical AI systems. Zico supports regulation in principle but warns that technical prescriptions quickly become outdated; he suggests focusing on regulating uses and harms (e.g., libel, fraud) while acknowledging AI’s volume/velocity may require updates.
- •Governments should play a role, but technical rules can become obsolete fast
- •Better leverage: regulate downstream applications and harms
- •Existing laws may cover many cases but need updates for scale/velocity
- •The technology’s pace makes durable policy design difficult
- 37:14 – 45:05
Zico’s top AI safety hierarchy: specification failures, jailbreaks, and why this multiplies every other risk
Zico names his highest-priority concern: models can’t reliably follow specifications, enabling prompt injection and jailbreaks. He argues this is a foundational ‘cybersecurity-like’ vulnerability that becomes far more dangerous as LLMs are embedded into agents and broader systems, amplifying risks like cyber offense and other misuse.
- •Core problem: unreliable specification-following and instruction hierarchy
- •Prompt injection/jailbreaking as a pervasive exploit class
- •Risk grows sharply when models become agents acting on untrusted inputs
- •Analogy: a widespread ‘buffer overflow’ we don’t yet know how to patch
- •This weakness magnifies bio/cyber/terror/fraud risks downstream
- 45:05 – 51:13
Open-weight release trade-offs: why today’s openness feels manageable, but future capabilities may cross a line
Harry asks whether highly dangerous AI should ever be ‘open.’ Zico rejects simplistic nuclear-weapon analogies, supports open-weight models for research and ecosystem health, but argues there will be a capability threshold—especially in autonomous exploit discovery—where broad release should pause; he prefers closed-first as a buffer to understand risks.
- •Nuclear weapons analogy is imperfect; AI is dual-use with many benefits
- •Open-weight models are crucial for academic research and ecosystem progress
- •Current frontier models may be safe enough for openness (today), but not forever
- •Future red line: models that can rapidly find/exploit vulnerabilities at scale
- •Closed-first releases can provide time to evaluate and mitigate before openness
- 51:13 – 57:37
Beyond sci-fi: correlated failures in critical infrastructure and the case for practical safety now
Zico contrasts ‘rogue AGI’ narratives with nearer-term systemic risks. He presents a scenario where AI-embedded infrastructure (e.g., power grid software) creates catastrophic correlated failures—whether malicious or accidental—and argues the outcome matters more than intent, reinforcing the need for concrete safety measures today.
- •Most safety work should focus on practical, present risks—not only sci-fi
- •Critical infrastructure + agentic systems increases correlated-failure risk
- •Catastrophe can occur via bugs/misconfigurations as much as malicious intent
- •Intent debates can be irrelevant when real-world outcomes are identical
- •Need deployment discipline, safety structures, and risk planning now
- 57:37 – 1:03:45
Optimism with conditions, then quick-fire: ‘post-architecture’ era, data surprises, board responsibilities, and global safety
Harry closes by asking whether Zico is optimistic and then moves into quick-fire questions. Zico expresses optimism contingent on safety, shares that he’s shifted away from architecture-centric thinking, emphasizes the power of large-scale uncurated data, describes his OpenAI board role as AI/safety expertise, and argues safety must be globally coordinated.
- •Optimistic stance: wants to use these tools, but safety is necessary
- •Belief update: architectures matter less than once thought (‘post-architecture’)
- •Belief update: large-scale ‘internet data’ learning can outperform heavy curation
- •OpenAI board: quarterly meetings; role is to advise on AI and AI safety
- •Geopolitics: competition exists, but AI safety requires global cooperation
