Stanford OnlineStanford CS230 | Autumn 2025 | Lecture 2: Supervised, Self-Supervised, & Weakly Supervised Learning
CHAPTERS
- 0:05 – 3:07
Course goals and why this lecture is industry-driven
Kian introduces himself, explains CS230’s in-person lecture focus, and previews how the class emphasizes practical decision-making in AI projects. He sets expectations for an interactive, conversational format with real-world examples.
- •Instructor background and industry perspective (Workera)
- •What students should expect beyond the online videos
- •Themes for the quarter (project decisions, adversarial, RL, RAG/agents)
- •Lecture roadmap and emphasis on practical judgment
- 3:07 – 8:40
Deep learning recap: supervised learning loop, model components, and what can vary
A quick recap of the canonical supervised learning pipeline: inputs, outputs, architectures, parameters, loss functions, and gradient descent. The section frames “what changes” across tasks (data modality, outputs, architectures, losses, and training choices).
- •Model = architecture + parameters; how deployed systems conceptually work
- •Gradient descent and loss as the feedback mechanism
- •Inputs can be image/text/audio/video/structured data
- •Outputs can be classification, regression, or generative
- •Loss design as a creative/critical skill (e.g., YOLO)
- 8:40 – 14:32
Neurons to networks: multi-class labeling, one-hot vs multi-hot, and capacity/overfitting intuition
Kian reviews the neuron (linear + activation) and extends binary classification to multi-class/multi-label setups. He highlights common project mistakes around labels and builds intuition for model capacity vs dataset size and overfitting risk.
- •Logistic regression view of a neuron; sigmoid for probabilities
- •How to modify output layer for multiple classes/labels
- •One-hot vs multi-hot vectors; label-schema pitfalls
- •Layer/neurons notation used in the course
- •Capacity vs data diversity: deeper models can overfit small datasets
- 14:32 – 18:33
What networks learn inside: feature learning, encodings vs embeddings, and why distance matters
Using a face-trained network as an example, Kian explains how early layers capture edges while deeper layers capture higher-level semantics. He contrasts feature engineering with end-to-end feature learning and introduces embeddings as encodings with meaningful geometry.
- •Early layers: low-level features; deeper layers: semantic parts
- •Feature engineering vs feature learning (end-to-end)
- •Encoding vs embedding (distance has meaning)
- •Why vector-space distances enable search and retrieval
- •Preview of interpretability/visualization later in the course
- 18:33 – 34:18
Case study 1 — Day vs night classification: defining scope, collecting data, and choosing resolution
Students co-design a simple classifier and quickly discover that task definition and edge cases dominate difficulty. The discussion focuses on dataset diversity, resolution trade-offs, and using humans as a fast proxy to validate assumptions.
- •Task definition changes difficulty (location-specific vs global)
- •Hard cases: indoors, dawn/dusk, weather, extreme latitudes
- •Dataset composition and distribution matter
- •Resolution trade-off: information loss vs compute/iteration speed
- •Human experiments as rapid proxies; example target ~64×64×3
- 34:18 – 42:02
Case study 2 — Trigger word detection: cascaded assistants and why labeling strategy matters
Kian explains why voice assistants use a cascade of models and zooms into the trigger-word detector. Students explore how to collect audio data and why accents, demographics, speech rate, and background noise drive real-world performance.
- •Cascade: activity detection → keyword spotting → heavier intent model
- •Collecting positives/negatives; choosing confusing negatives (e.g., “deactivate”)
- •Distribution gaps: accents, age, gender, cadence, noise environments
- •Audio as sequential data (time steps; sample rates)
- •Practical tip: borrow known audio hyperparameters from existing projects
- 42:02 – 48:54
Human experiment: weak labels vs time-localized labels (and the cold-start problem)
A classroom listening exercise demonstrates that simply knowing a clip contains the keyword is much harder than knowing where it occurs. Kian motivates time-localized labels as a dramatic speedup in learning and discusses trade-offs between labeling effort and model data needs.
- •Binary clip-level labeling vs time-localized labeling
- •Why localized labels reduce required data by orders of magnitude
- •Cold-start mitigation: richer supervision early, simpler labels later
- •Sequential sigmoid outputs; binary cross-entropy applied per time step
- •Avoiding label imbalance where “all zeros” looks highly accurate
- 48:54 – 55:21
Synthetic data pipeline for keyword spotting: auto-labeling at scale + expert architecture guidance
Kian describes a practical recipe: record short positive/negative word clips, gather abundant background noise, and synthetically mix them with a script that auto-generates labels. He emphasizes that architecture choice often comes from expert heuristics and experience.
- •Three-source dataset: positive words, negative words, background noise
- •Scripted non-overlapping insertion enables automatic label generation
- •Rapid scaling: hours of recording → millions of training examples
- •Augmentation ideas (speed, pitch/frequency shifts)
- •Why consulting experienced practitioners saves weeks of trial-and-error
- 55:21 – 1:04:55
Case study 3 — Face verification: why pixel comparison fails and how embeddings solve it
The lecture moves to face verification for campus facilities and explores the pitfalls of naive pixel-wise matching. Kian introduces the encoder approach: run images through a network to obtain a vector embedding, then compare embeddings with a threshold.
- •Problem setup: ID photo vs camera photo match
- •Why pixel distance breaks (lighting, background, translation/scale/rotation)
- •Need higher resolution for fine identity cues (e.g., ~412×412×3)
- •Encoder network produces 128-D vectors for semantic comparison
- •Threshold selection trades off false positives vs false negatives
- 1:04:55 – 1:12:46
Training the encoder with triplet loss: anchor/positive/negative and decision-driven loss design
Students derive a loss that pulls same-identity embeddings together while pushing different identities apart. Kian formalizes triplet construction and explains how this training trick enables robust verification despite appearance changes.
- •Triplets: anchor, positive (same person), negative (different)
- •Objective: minimize d(anchor,positive) and maximize separation from negative
- •Triplet loss structure and why a margin (alpha) stabilizes training
- •Data augmentation helps when few images per identity exist
- •Core lesson: loss function design creates the learning environment
- 1:12:46 – 1:20:22
From verification to identification and clustering: k-NN search and k-means on embeddings
With a trained embedding space, Kian shows how to reuse it for new tasks without retraining the network. Identification becomes nearest-neighbor retrieval over stored vectors, and clustering becomes a standard unsupervised algorithm applied in embedding space.
- •Verification (pair match) vs identification (who is it?) distinction
- •Store embeddings for the database; query with camera embedding
- •k-Nearest Neighbors for identification; threshold for “unknown person”
- •k-means for face clustering (phone photo albums)
- •Centroids as practical summaries for efficient comparison
- 1:20:22 – 1:26:33
Self-supervised learning: contrastive pairs via augmentation (SimCLR) to avoid manual labels
Kian motivates self-supervision by the cost of labeling and shows how augmentations generate positive pairs automatically. Contrastive learning trains models on massive unlabeled datasets, with compute—rather than labeling—becoming the dominant constraint.
- •Key idea: two augmented views of the same sample should embed similarly
- •Examples: rotations, crops, occlusions/patching as supervision signals
- •Contrastive learning (SimCLR) as a modern pretraining workhorse
- •No explicit identity labels needed; supervision comes from transformations
- •Shift in bottleneck from labeling to compute and scale
- 1:26:33 – 1:34:07
Self-supervision in language: next-word prediction and emergent behaviors
Using simple fill-in-the-blank prompts, Kian illustrates how next-token prediction induces surprisingly broad capabilities. He frames emergent behavior as capabilities that arise from scale and objective design, not explicit labeling.
- •Next-token prediction as self-supervised learning for text
- •Examples show learning of semantics, facts, probabilistic reasoning, and inference
- •Context co-occurrence drives representations (tea vs coffee cultural priors)
- •Parallel to vision: general representations emerge without task-specific labels
- •Preview of emergent behaviors in RL (e.g., AlphaGo strategies)
- 1:34:07 – 1:39:47
Weak supervision and multimodal embeddings: naturally paired data and shared representation spaces
The lecture closes by explaining weak supervision as learning from naturally occurring pairings (captions, subtitles, audio-video). Kian highlights how shared embedding spaces connect modalities and points to ImageBind as an example of aligning many modalities through central pivots like text and images.
- •Weakly supervised learning: leveraging naturally paired modalities
- •Examples: image captions, subtitles/transcripts, audio-video, music-title, medical pairings
- •Why text often becomes the shared ‘pivot’ modality
- •Shared embedding spaces enable cross-modal retrieval and reasoning
- •ImageBind demo: aligning text, image, and audio in one vector space
