Lex Fridman PodcastGary Marcus: Toward a Hybrid of Deep Learning and Symbolic AI | Lex Fridman Podcast #43
CHAPTERS
- 0:00 – 4:01
Singularity as gradual change & intelligence as multi-dimensional
Lex opens by asking about an AI-driven singularity and humanity’s place in the “food chain.” Gary argues change is already underway and likely to be incremental rather than a single discontinuity, because intelligence is not one scalar but many capabilities that mature at different speeds.
- •AI impact is already shifting labor and society
- •“Singularity” is unlikely to be a one-day event; progress will be uneven
- •Intelligence has many facets (math, games, language, motor skills)
- •Machines already exceed humans in narrow areas but lag in language and flexible reasoning
- 4:01 – 11:14
Common sense as the missing prerequisite (physical vs psychological reasoning)
Gary identifies common sense as a rate-limiting ingredient for robust AI, especially for reading and real-world understanding. The discussion separates physical reasoning (objects, mechanics) from psychological reasoning (goals, emotions), including why physical experimentation may be more accessible for robots than probing human minds.
- •Common sense underpins reading, movies, and everyday inference
- •Physical reasoning may be learned via robotic experimentation
- •Psychological reasoning is harder to probe ethically and practically
- •Facebook-style data can offer partial “theory of mind,” but it’s limited
- 11:14 – 17:18
Games aren’t magic: chess/Go vs open-ended language understanding
They pivot from chess and AlphaGo-style triumphs to what those wins do and do not imply. Gary argues game domains are closed and stable, while language and comprehension are open-ended and depend on rich world models—so techniques don’t transfer automatically.
- •Chess/Go are closed-world problems with fixed rules and bounded actions
- •Language is vast and depends on physical and social world knowledge
- •Media narratives overgeneralize game success into “AI can do anything”
- •‘AI is not magic’ framing: a toolbox of methods with sharp limits
- 17:18 – 25:27
Forecasting AGI: what we can safely predict (and what “general” means)
Lex presses Gary to speculate 100 years out; Gary resists precise forecasts but offers safe trends: faster, cheaper, more pervasive, and more general. They clarify that “general intelligence” need not mirror humans and could surpass human limitations like memory and biased reasoning.
- •Hard to predict applications; easy to predict speed/cost/pervasiveness
- •AGI likely won’t be human-like; it can be better than humans in key ways
- •Human flaws (memory limits, motivated reasoning) aren’t virtues to preserve
- •Goal is combining machine strengths (compute) with human-like understanding
- 25:27 – 28:22
What ‘general intelligence’ really implies: transfer, flexibility, and reuse
They debate whether humans are “general” and contrast that with today’s narrow systems that require retraining for small changes. Gary emphasizes transfer learning—reusing knowledge across domains and adapting to novel variations—as a hallmark of generality.
- •Humans can interpret diverse situations (e.g., most movie scenes)
- •Current AI is brittle: small shifts (e.g., board shape) can break performance
- •General intelligence involves transferring knowledge across contexts
- •Being ‘bounded’ doesn’t negate meaningful generality
- 28:22 – 30:26
Deep learning’s core gaps: cognitive models, abstractions, and common sense
Lex references Gary’s ‘Deep Learning: A Critical Appraisal’ and asks which challenges matter most. Gary ties many failures to the absence of cognitive models—systems learn correlations (pixels, co-occurrences) rather than structured causal/mechanistic understanding (e.g., how a bottle cap seals).
- •Key bottleneck: lack of explicit cognitive models and common sense
- •Correlation-based learning fails under distribution shift (e.g., Breakout paddle moved)
- •Need representations of objects and relations (paddle/ball/walls)
- •Many listed issues (transfer, robustness, explainability) are intertwined
- 30:26 – 32:52
Why common sense is hard: beyond taxonomies to functional understanding
Gary argues ‘common sense’ isn’t a single thing and is not solved by classic symbolic tools alone. Examples (containers, cheese grater) illustrate that knowing categories isn’t enough—you need functional, causal, and interaction-rich understanding of artifacts and actions.
- •Common sense includes taxonomy but also function, affordances, and causality
- •Logical hierarchies help some inferences but miss many practical details
- •Artifacts (e.g., cheese grater) demand understanding of purpose and use
- •Deep learning alone is unlikely to induce these abstractions reliably
- 32:52 – 44:42
Emergence skepticism & the case for built-in structure (convolutions and beyond)
The conversation drills into whether large neural nets can ‘emerge’ the right abstractions. Gary points out that key successes like convolution bake in priors, and argues progress will require more engineered inductive biases and mechanisms that support variable-like operations.
- •Convolution is an engineered abstraction, not something nets typically discover
- •Inductive biases can dramatically improve data efficiency and generalization
- •Standard deep nets struggle with abstract, compositional concepts
- •Differentiable programming is cited as a promising direction toward structure
- 44:42 – 46:23
Why expert systems failed—and why hybrids are the future
Lex asks about expert systems; Gary distinguishes endorsing symbolic manipulation from endorsing 1980s-era rule-only systems. He advocates hybrid architectures: deep learning for perception plus symbolic/structured components for inference, abstraction, and reasoning.
- •Expert systems: hand-coded rules captured expert knowledge but didn’t scale well
- •Modern deep learning: powerful for perception but weak on abstract inference
- •Hybrid approach: combine statistical learning with symbolic/structured reasoning
- •Need new synthesis methods, not a return to either extreme
- 46:23 – 52:52
The knowledge acquisition bottleneck: why ‘obvious’ facts are hard to encode
They explore why commonsense knowledge is difficult to gather and formalize: people omit what’s obvious, crowd-sourced descriptions skew too micro-level, and abstraction is hard to elicit. Gary notes CYC as the closest attempt, but criticizes overreliance on pure logic and hand coding.
- •Humans recognize truths when stated but rarely volunteer them unprompted
- •Crowdsourcing often yields procedural minutiae rather than abstract intent
- •CYC attempted large-scale common sense but struggled with representation and scale
- •The field overgeneralizes CYC’s failure as failure of the whole approach
- 52:52 – 56:26
Compute, ‘The Bitter Lesson,’ and the ladder-to-the-moon critique
Lex raises Rich Sutton’s claim that general compute-driven methods win in the long run. Gary agrees compute helps but argues it has delivered mainly in perception/RL, not commonsense or language understanding, and cites the brain’s efficiency as evidence that architectural insight matters.
- •Compute helps, but returns diminish without the right representations
- •Deep learning’s wins are concentrated in classification and certain RL settings
- •Brain achieves broad competence at ~20 watts—suggesting more than brute compute
- •‘Better ladder ≠ ladder to the moon’: scaling alone won’t guarantee AGI
- 56:26 – 1:01:50
Children, curiosity, and nature-and-nurture as a blueprint for better AI
Gary reflects on learning from his kids: they invent ‘what-if’ scenarios and self-generate problems—capabilities missing in current systems. This leads into nature-and-nurture: innate structures plus learning are both essential, and AI shouldn’t treat innateness as ‘cheating.’
- •Kids learn by creating new games and counterfactual scenarios
- •Nature-and-nurture must work together; innate scaffolds enable learning
- •Possible innate frameworks: space, time, agents, causality, mental algebra
- •AI should borrow from developmental psychology and cognitive science
- 1:01:50 – 1:06:43
Evolution, biomimicry, and building richer ‘libraries’ for intelligence
They discuss evolution as inefficient but cumulative: once a useful ‘library’ appears, it gets reused and spreads. Gary argues AI can accelerate by adopting insights from biology/cognitive science rather than waiting for blind evolutionary search or sheer compute scaling.
- •Evolution is slow to discover, fast to reuse; cumulative improvements matter
- •Evo-computation often starts too ‘blank,’ lacking rich priors/libraries
- •Biomimicry for cognition: import mechanisms from animals and humans
- •Engineering can shortcut evolution by designing the right structures directly
- 1:06:43 – 1:12:30
Testing intelligence: Turing Olympics and the ‘comprehension challenge’
Gary rejects a single intelligence test and proposes a battery (“Turing Olympics”). His personal ‘impress me’ benchmark is deep comprehension: answering open-ended questions about stories/films, character motivations, and implied meaning (e.g., “I am Spartacus”).
- •No single metric captures intelligence; it’s inherently multifaceted
- •Turing Test is easily gamed by evasion and persona tricks
- •Better benchmark: robust comprehension across arbitrary narratives
- •Success requires commonsense inference, intent modeling, and causal understanding
- 1:12:30 – 1:25:00
GPT-2, rebranding, and why trustworthy AI needs ‘deep understanding’
Lex challenges Gary on whether deep learning might surprise him; Gary predicts hybrids will be re-labeled as deep learning, but pure correlation-based nets won’t reach real comprehension. The conversation closes on trustworthy AI: alignment requires explicit concepts (like harm), mechanisms for translating values into machine-executable form, and public literacy to resist hype.
- •AlphaGo is a hybrid (deep nets + Monte Carlo tree search), often misbranded
- •GPT-2 can be fluent yet conceptually inconsistent; failures persist over decades
- •Alignment requires representing abstract concepts (harm, intent), not just correlations
- •Need governance and informed discourse; book offers ‘questions to ask’ of AI claims
