Lex Fridman PodcastVladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
CHAPTERS
Engineering vs science of intelligence: imitation, understanding, and Turing’s legacy
Lex and Vapnik contrast building systems that imitate humans with the deeper goal of understanding what intelligence is. Vapnik frames engineering as functional imitation, while science seeks underlying principles—setting up his central thesis about predicates and invariants.
- •Engineering intelligence focuses on usefulness and behavior imitation, regardless of internal mechanism
- •Scientific understanding of intelligence is harder and borders on philosophy
- •Turing’s influence steered the field toward engineering goals
- •Vapnik introduces the idea that understanding may hinge on predicates and invariants
Propp’s 31 narrative units as a model for “few powerful predicates”
Vapnik uses Vladimir Propp’s analysis of folktales—31 recurring structural units—as an example of compact, universal predicates that explain many phenomena. The discussion positions Propp as evidence that a small set of abstractions can capture complex human behavior.
- •Propp’s ‘Morphology of the Folktale’ proposes 31 reusable narrative units
- •These units generalize beyond Russian tales to movies and serials
- •Predicates can be few, while behaviors/invariants can be many
- •The analogy motivates searching for similarly compact predicates in vision
What Vapnik means by ‘predicate’ and how invariants arise from data
Lex pushes for a definition of predicate; Vapnik reframes it as a function used to measure properties, often via inner products, which yields invariants on specific data. Symmetry and ‘structuredness’ are floated as candidate predicates for images, akin to vocabulary used by music critics.
- •Predicate as a function (not just a true/false logical statement)
- •Invariants are measurable properties of a specific object/image derived from predicates
- •Examples: degree and types of symmetry; informal notion of image ‘structure’
- •Analogy: critics use a limited vocabulary (predicates) to describe music
Plato’s world of ideas vs world of things: projection via invariants
The conversation ties predicates to Plato’s Theory of Forms: abstract ideas (predicates) project onto reality (data) through invariants. Vapnik suggests intelligence is the ability to connect these worlds by applying abstract predicates to observed reality.
- •Two worlds: abstract ideas (predicates) and concrete reality (things/data)
- •Invariants act as projections of ideas onto specific instances
- •Intelligence as combining ideas with reality to produce useful invariants
- •Lineage hinted: Plato → Hegel → Wigner → modern learning theory intuition
Strong vs weak convergence: why predicates help learning
Vapnik introduces strong convergence (function-level closeness) and weak convergence (agreement on integral/functional properties via inner products). He argues predicates correspond to selecting which weak properties must match, shrinking the admissible set of functions and improving generalization with limited data.
- •Strong convergence: functions become close in norm (e.g., squared integral difference)
- •Weak convergence: convergence of inner products/functional properties for all test functions
- •Predicates define which integral properties must be preserved
- •Matching predicate-derived averages on training data restricts admissible functions
Admissible function sets, VC dimension, and why ‘good predicates’ are rare
Vapnik formalizes ‘good predicates’ as those that dramatically reduce the admissible hypothesis space, lowering capacity (VC dimension) and data needs. The ‘duck test’ illustrates that some predicates constrain effectively while others don’t.
- •Goal: reduce hypothesis space while retaining the true target function
- •Good predicates shrink admissible sets substantially; bad ones barely constrain
- •Reducing VC dimension reduces needed training data
- •‘Looks/swims/quacks like a duck’ as useful predicates vs irrelevant ones (e.g., ‘plays chess’)
Deep learning critique: convolution as a single predicate and missing invariants
Vapnik critiques deep learning as relying on a limited, hand-designed predicate (e.g., translation invariance via convolution) and then optimizing within a narrow function class (piecewise linear networks). He argues the field should explicitly build admissible sets by stating invariants rather than relying on architectural heuristics.
- •Neural nets framed as large families of piecewise linear functions
- •Convolution interpreted as enforcing a particular invariant (translation)
- •Architectures implicitly define admissible subsets without explicit predicate reasoning
- •Vapnik’s preference: specify invariants/predicates directly to shrink function sets
Discovering new predicates: contradiction hunting and ‘privileged’ clues
Pressed on how to find predicates, Vapnik suggests a physics-like method: locate contradictions where current invariants fail, then add a new predicate to eliminate the failure. He calls this a brute-force but workable approach and remains skeptical that machines will discover the “smartest” predicates unaided.
- •Method: find cases where the current theory/predicate fails (contradictions)
- •Add a predicate that enforces the missing invariant and retrain/resolve
- •Example alluded to from Vapnik’s diabetes discussion (predicate-driven improvement)
- •Open question: whether machines can systematically find best predicates
Symbolic AI and logic systems: ‘logic is not enough’ without life/experience
Lex asks whether logic-based symbolic AI could discover good predicates; Vapnik argues it can’t, because predicates must be grounded in reality and lived understanding. Propp is used again as an example of someone extracting invariants from deep familiarity with narratives, not pure deduction.
- •Vapnik’s skepticism of purely logic-based predicate discovery
- •Need for grounding: ‘you should know reality / know life’
- •Propp’s success framed as knowledge-driven abstraction, not formal logic alone
- •Common sense discussed but left unresolved and intentionally narrowed back to digits
How hard is 2D image understanding? MNIST as a ‘not too simple’ intelligence test
Vapnik argues that achieving SOTA MNIST performance with 100× fewer examples would require genuine intelligence—new invariants and weak-convergence thinking. Lex presses on the jump to natural images and 3D projection complexity; Vapnik insists progress must start with the simplest nontrivial challenge.
- •MNIST-with-few-examples as an ‘Einstein simple but not simpler’ benchmark
- •Success would imply discovery of meaningful invariants/predicates
- •Debate: does MNIST transfer to general vision/common sense?
- •Vapnik predicts human-interpretable principles (e.g., degree of symmetry) will emerge
Data vs functions: why Vapnik focuses on shrinking hypothesis space, not dataset curation
Lex explores whether selecting ‘right data’ (teacher examples) is itself a path to intelligence. Vapnik keeps the emphasis on controlling overfitting by restricting the function class via admissible sets and predicates; he notes 60 examples per class may be enough to rely on the (non-uniform) law of large numbers.
- •Overfitting framed as choosing from too large a function set with limited data
- •Core lever: reduce admissible function set using predicates
- •Distinction between law of large numbers vs uniform law (learning requires uniformity)
- •Vapnik aims for regimes where fewer examples suffice due to strong constraints
Language as an out-of-scope frontier: ‘not for the 21st century’
Lex brings in Chomsky and asks whether language is central to idea formation; Vapnik declines, calling language vastly more complicated than vision benchmarks. He reiterates the strategy: master simple tasks first to extract principles before tackling language-level complexity.
- •Vapnik views language understanding as too large/complex for current era
- •Skepticism that language can be reduced to simple symbol manipulation
- •Preference for extracting predicates from constrained vision tasks first
- •Belief that even current digit-recognition practice is misguided if it needs 60,000 samples
Most beautiful learning-theory ideas: uniform convergence and the power of weak convergence
Asked for the most powerful idea in learning theory, Vapnik highlights uniform convergence (VC theory) as essential for learning guarantees. He then elevates weak convergence as the underused tool that naturally supports admissible-set construction and yields computationally clean (closed-form) solutions in rich Hilbert spaces.
- •Uniform convergence: simultaneous convergence over function classes enables learning
- •Why pointwise/‘for any fixed function’ convergence isn’t enough for ERM selection
- •Weak vs strong convergence as the two fundamental modes in Hilbert spaces
- •Closed-form solutions seen as a sign of using the right mathematical instrument
Reasoning, heuristics, and philosophy: asking good questions and building ideas before implementation
Lex probes reasoning, recurrence, and heuristics; Vapnik stresses formalizing what you want first and is wary of adding mechanisms without necessity. He describes philosophy as ‘understanding life’ that precedes implementation, and argues progress came from decades of revisiting the same problems until weak convergence clicked.
- •Vapnik downplays recurrence, focusing on specifying the problem precisely
- •Hardest part of reasoning: asking good questions
- •Heuristics may help engineering, but science seeks principled formulations
- •Philosophy as the upstream source of ideas; math as implementation/verification
Music, poetry, and privileged information: using artistic descriptions to uncover predicates
A long Russian-language interlude about music leads into a striking proposal: analyze critics’ vocabularies as candidate predicate sets. Vapnik recounts an experiment where a poetry scholar described MNIST digits using poetic imagery; those descriptions served as ‘privileged information’ that improved learning—hinting at deep, transferable abstractions.
- •Critics’ descriptive vocabularies as windows into high-level predicates
- •Different composers evoke different predicate ‘languages’ (Bach vs Chopin)
- •Poetic digit descriptions used as a second modality (‘privileged information’)
- •Suggestion: cross-domain abstraction mining (art → vision) might reveal universal predicates
Mortality and unfinished work: urgency to find universal predicates across domains
Lex asks about death; Vapnik expresses limited fear but regret about unfinished intellectual projects—especially connecting predicates across music, art, and vision. The conversation closes with reflections on literature thinkers’ insight into life and a return to the MNIST challenge as a concrete path forward.
- •Mortality framed as losing time to complete long-term research ideas
- •Desire to collaborate with music theorists/critics to systematize predicates
- •Admiration for literature scholars and managers as people who ‘understand life’
- •Closing encouragement: revisit the challenge problem and reconvene after progress
