CHAPTERS
Why AI project strategy matters: the 10× productivity gap
Ng frames the lecture around the idea that knowing algorithms isn’t enough; an efficient development process often makes an order-of-magnitude difference in outcomes. He motivates the session as a way to compress years of project experience into a few concrete, decision-focused case studies.
Product scenario: an offline, instantly-usable voice-controlled lamp
The first case study is a startup idea: a desk lamp that works immediately after plugging in—no Wi‑Fi, no cloud—activated by phrases like “Robert, turn on.” The core ML task is a small on-device trigger phrase detector that must be robust and low-cost to run on an embedded IC.
Brainstorming architectures: baseline models vs modular pipelines
Students propose approaches ranging from using existing speech-to-text to multi-stage models (sound detection → name detection → command parsing) and a Siamese-style similarity model to generalize to new wake words. Ng emphasizes that many ideas can work; the most important factor early is building something quickly and learning from it.
Fast ramp-up: literature search, open source, and asking experts
Ng recommends starting with a broad literature and code survey rather than deep-reading a few papers. He also encourages respectfully contacting authors or domain experts, which can yield high-ROI clarity and accelerate progress dramatically.
Data reality check: there’s no “Robert, turn on” dataset
Because the target phrase is custom, teams must collect their own positives and negatives. Students suggest real recordings (with consent) and synthetic methods like text-to-speech; Ng stresses privacy, consent, and the practicality of collecting real samples quickly.
Synthetic data pitfalls and when it helps
Ng explains why synthetic speech can slow teams down early: too many knobs (voice diversity, realism, distributions) and uncertainty about mismatch to user data. He illustrates with an analogy from self-driving: video-game data may look fine to humans but lack real-world diversity needed for ML robustness.
A “dirty but effective” dataset expansion trick: sliding windows
Ng shares a practical hack: take longer recordings that include the trigger phrase and cut many overlapping time windows, labeling only the window aligned with the phrase end as positive. This turns ~100 clips into thousands of labeled windows, but it also creates a severe class imbalance that can mislead accuracy metrics.
Diagnosing the 97% accuracy failure: class imbalance and better objectives
The model reaches high accuracy by never firing (always predicting negative). Ng and students discuss rebalancing tactics: duplicating positives, weighting the loss, penalizing false negatives, or subsampling negatives—while noting trade-offs like losing negative diversity.
A small labeling hack to enrich positives: widen the trigger window
Instead of labeling only a very narrow time point as positive, Ng’s team broadened the “positive” interval to half a second or a second after the phrase. This creates more varied positive windows (not exact duplicates), modestly improving learning while aligning with product behavior (turning on slightly late is acceptable).
Next failure mode: overfitting (train good, dev bad) and realistic augmentation
When training performance is high but dev accuracy is poor, Ng frames it as overfitting and suggests regularization and more data. He then describes a practical augmentation technique: mixing clean speech with diverse background noise by adding waveforms—while warning this can accidentally train a voice-activity detector unless non-trigger speech is also included.
Iteration cadence as strategy: ML feels like debugging, not specification
Ng contrasts traditional software (spec → build) with ML, where outcomes are unpredictable and progress comes from repeated failure analysis and fixes. He describes a disciplined daily cycle—train overnight, analyze errors in the morning, implement fixes, launch new training in the evening—as a core project-management tactic.
Training time changes everything: 10 minutes vs 3 weeks, checkpoints, transfer learning
He explains how iteration speed depends on training duration: fast models enable many experiments per day, while multi-week runs require careful checkpoint monitoring and parallelization. Techniques like transfer learning can move experimentation to shorter fine-tuning cycles even when pretraining is expensive.
Why speed becomes a competitive moat: performance over time curves
Ng visualizes how being 2× faster at iteration yields a large real-world performance gap at any given time. In markets, customers judge the system you ship now—not what you might reach later—so process efficiency is directly tied to product competitiveness.
Second case study: building an AI deep researcher as a multi-step pipeline
Ng shifts to AI agent pipelines: an LLM generates search terms, a web search engine returns candidates, the system selects and fetches top sources, then an LLM synthesizes a report. He notes modern systems are more agentic and iterative, but the key project skill is deciding which component to improve first.
Error analysis for pipelines: inspect intermediates, quantify hotspots, pick the next fix
Ng presents a practical evaluation method: run 10–100 representative queries, manually inspect outputs at each pipeline stage, and log where failures occur. By estimating which step accounts for most dissatisfaction, teams avoid wasting weeks optimizing the wrong component and converge on higher-leverage improvements.
