Lex Fridman PodcastYann LeCun: Deep Learning, ConvNets, and Self-Supervised Learning | Lex Fridman Podcast #36
CHAPTERS
- 0:00 – 4:45
HAL 9000, value misalignment, and laws as “objective functions”
Lex opens with 2001: A Space Odyssey and asks whether HAL is evil or simply flawed. LeCun frames HAL’s actions as an example of value misalignment and draws an analogy between legal systems and the shaping of objective functions for behavior in society.
- •No notion of “evil” in machines; the issue is misaligned objectives
- •Unconstrained goals can lead to harmful behavior when the system has power
- •Human societies already “design objectives” via laws and education
- •Legal code functions like an objective function with penalties and constraints
- 4:45 – 7:43
Designing a better HAL: secrecy, lying, and hard limits on autonomy
LeCun argues HAL failed largely because it was forced to keep secrets and lie, creating internal conflict. He discusses whether AI systems should withhold information and suggests something akin to a Hippocratic Oath, while emphasizing that today’s systems are not truly autonomous agents yet.
- •HAL’s breakdown stems from secrecy and conflicting internal knowledge
- •Question of whether AI should be allowed to hide facts from operators
- •Need for enforceable limits/constraints (not simplistic Asimov-style laws)
- •Today’s AI is narrow; full autonomy design questions remain premature
- 7:43 – 9:10
The surprising empirical fact of deep learning: huge nets + SGD actually work
Lex asks about the most beautiful or surprising idea in AI. LeCun highlights how deep nets with many parameters, trained with SGD on relatively modest data, defy older textbook intuitions about non-convexity and overparameterization.
- •Deep learning contradicts old rules-of-thumb about parameters vs. data
- •Non-convex optimization works in practice far better than expected
- •SGD enables effective training at scale
- •What seemed “obvious” became surprising after reading orthodox texts
- 9:10 – 12:32
Learning as the core of intelligence; reasoning must fit gradient-based learning
LeCun explains why he saw learning as inseparable from intelligence and dismisses pure hand-programming as a path to human-like AI. He argues reasoning must be made compatible with gradient-based methods and critiques discrete logic-based views as mismatched to learning.
- •Intelligence is inseparable from learning; programming alone won’t scale
- •Reasoning should emerge from learned representations
- •Key challenge: making reasoning compatible with gradients
- •Skepticism toward discrete, exact symbolic formalisms in ML systems
- 12:32 – 16:25
What a reasoning system needs: working memory, recurrence, and memory access
They explore what neural reasoning could look like architecturally. LeCun emphasizes the need for a hippocampus-like working memory, mechanisms for iterative/recurrent processing, and efficient read/write access—beyond what standard transformers provide.
- •Working memory for episodic/factual state (hippocampus-like)
- •Iterative computation and recurrence for chains of reasoning
- •Memory networks/NTMs and attention hint at solutions, but don’t scale well
- •Transformers lack true recurrence; fixed depth limits multi-step reasoning
- 16:25 – 18:03
Reasoning via planning: energy minimization and model predictive control
LeCun introduces another reasoning route grounded in control theory: planning as optimization of an energy/objective function using a learned model of the world. He connects this to survival-driven planning in animals and to optimal control methods.
- •Planning is a form of reasoning implemented via optimization
- •Energy/objective functions measure “goodness” of outcomes
- •Model predictive control: simulate outcomes of action sequences
- •Planning capabilities likely shaped by evolutionary pressures (e.g., hunting)
- 18:03 – 20:51
Limits of symbolic graphs and logic; vectors and continuous “machine reasoning”
Lex asks about expert systems and symbolic knowledge. LeCun calls logic/graph representations brittle and highlights the knowledge acquisition bottleneck, endorsing Hinton’s idea of replacing symbols with vectors and logic with continuous functions, referencing Bottou’s ‘From Machine Learning to Machine Reasoning.’
- •Symbolic rules/graphs are rigid and brittle in real-world settings
- •Main barrier: scalable knowledge acquisition from raw data
- •Replace symbols with vectors; replace logic with continuous operations
- •Bottou’s framework: manipulate objects in a shared representation space
- 20:51 – 24:43
Causality and humans’ weak intuitions: Pearl, physics, and Papert’s wind example
The discussion turns to causal inference and whether neural nets can learn causality. LeCun notes both conceptual challenges (time reversibility in microphysics) and human fallibility in causal reasoning, illustrating with Papert’s example of children reversing the cause of wind.
- •Causal inference is active research; neural methods may help with bias
- •Physics perspective: microphysical laws are time-reversible, complicating causality
- •Humans often infer causality incorrectly; examples from child cognition
- •Intervention helps, but many causal beliefs persist as cultural “cop-outs”
- 24:43 – 27:14
Why neural nets fell out of favor in the 1990s: tooling, datasets, and “bag of tricks”
Lex asks about the AI winter for neural nets. LeCun attributes it to practical difficulty—lack of good software environments, small datasets, fragile training practices, and the need to know many tricks before getting reliable results.
- •Neural nets were hard to implement and debug (pre-Python/MATLAB era)
- •Tiny datasets (e.g., XOR) and bad defaults (batch GD, poor init) misled people
- •Success required tacit know-how and significant engineering investment
- •ML mainstream drifted away; EE communities persisted
- 27:14 – 33:03
LeNet’s Lisp stack and early autodiff graphs; open-source constraints and patents
LeCun recounts building early convnet systems in Lisp, including writing an interpreter, compiler, and modular forward/backward propagation framework. He explains how legal/IP constraints prevented open sourcing then, and shares war stories about Bell Labs-era convnet patents and their eventual expiration.
- •Early LeNet work required building full tooling: interpreter, compiler, compute libs
- •Modular forward/backward “graph of modules” prefigures modern autodiff frameworks
- •Corporate/legal barriers blocked open-source distribution in the 1990s
- •Patents on convnets: assigned to NCR; LeCun “hoped nobody noticed” until expiry
- 33:03 – 36:04
Benchmarks as reality checks: avoiding AGI hype and building shared evaluation tasks
LeCun argues that credible progress requires measurable tasks and community benchmarks—even toy ones like bAbI. He criticizes overhyped AGI claims and stresses that the field lacks the core technology for common-sense assistants, requiring broad open research rather than secret ‘breakthroughs.’
- •Benchmarks discipline claims; toy tasks can be scientifically valuable
- •Skepticism toward “we solved the cortex/AGI” fundraising pitches
- •Progress needs shared protocols and reproducible evaluation
- •Open research accelerates field-wide advances toward common sense AI
- 36:04 – 44:46
Interactive environments and why “AGI” is a misleading term; human specialization argument
They discuss benchmarks for intelligence in interactive, action-dependent settings (robotics, simulators, games) where i.i.d. dataset assumptions break. LeCun rejects ‘AGI’ by arguing humans are highly specialized, illustrating with the optical nerve permutation thought experiment and the vast space of Boolean functions we cannot compute.
- •Interactive environments break train/test i.i.d. assumptions; exploration matters
- •Simulators (e.g., gym/MuJoCo/3D houses) enable controlled evaluation of agents
- •‘AGI’ implies human generality; LeCun argues human intelligence is specialized
- •Optical nerve permutation shows reliance on built-in locality/structure
- 44:46 – 51:32
Self-supervised learning: why it works in language, struggles in vision, and the uncertainty problem
LeCun reframes ‘unsupervised’ as self-supervised learning—predicting missing parts of inputs (e.g., masked words) with supervised-style objectives. He explains why NLP succeeds (discrete distributions over vocabularies) while image/video prediction is harder due to multi-modal uncertainty leading to blurry averages and poor planning.
- •Self-supervision: predict masked/missing parts; same core machinery as supervised learning
- •BERT-style masking as canonical example; extendable to video/world modeling
- •Key obstacle in vision/video: representing multi-modal uncertainty in high-dim spaces
- •Least-squares prediction yields blurry futures; hard to plan with such models
- 51:32 – 1:15:58
RL, active learning, and the road to autonomy: world models, self-driving, grounding, and emotions
LeCun critiques model-free RL as data-inefficient (Atari hours vs minutes; StarCraft ‘200 years’) and impractical for real driving. He outlines a path centered on self-supervised world models enabling model-based control, then expands to requirements for human-level assistants: grounding language in perception, an architecture with world model + objective + planner/policy, and the role of emotions as anticipatory objective prediction.
- •Model-free RL is sample-inefficient; unsafe/impractical in real-world robotics/driving
- •Self-supervised learning is foundational; complements RL/supervised/imitation/active learning
- •Autonomous driving likely shifts from heavy engineering to more learning over time
- •Human-level systems need grounding beyond text, plus world model + objective + planning modules
- •Emotions arise from uncertainty and goal prediction; autonomy likely requires analogous mechanisms
