Stanford OnlineStanford CS229: Machine Learning - Linear Regression and Gradient Descent | Lecture 2 (Autumn 2018)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Linear Regression, Gradient Descent Variants, and the Normal Equation Explained by Ng
- Ng motivates linear regression with a Portland housing price dataset, defining the hypothesis, parameters theta, training examples, and the notation used throughout the quarter.
- The learning goal is to minimize a least-squares cost function J of theta, which Ng optimizes with gradient descent by repeatedly stepping opposite the partial derivatives scaled by a learning rate.
- Ng contrasts batch gradient descent, which scans the whole dataset per step, with stochastic gradient descent, which updates on one example at a time, oscillates without converging, but is much faster on large datasets.
- For linear regression specifically, Ng derives the normal equation using matrix derivatives and the trace operator, giving the optimal theta in one step as (XᵀX)⁻¹Xᵀy.
- Throughout, Ng answers student questions on choosing the learning rate, why the gradient is subtracted, when to stop SGD, and what to do when XᵀX is non-invertible.
IDEAS WORTH REMEMBERING
5 ideasLearning algorithm design rests on a few key choices.
Ng says the workflow, the dataset, and how to represent the hypothesis are decisions made in nearly every supervised learning algorithm, which is why he spends time on the concepts and notation.
Linear regression minimizes half the sum of squared errors.
The cost J of theta sums squared differences between predictions and true prices over all training examples, with a one-half added to simplify derivatives; Ng defers the justification of squared error to next week's generalized linear models.
Linear regression's cost surface has no bad local optima.
Because J of theta is a sum of squares, it is a quadratic bowl whose only local optimum is the global one, so gradient descent will eventually converge to the global minimum.
The learning rate must be tuned empirically.
Too large a rate overshoots and can make the cost increase, while too small makes training slow; Ng tries values on a roughly doubling scale, often starting near 0.01 with scaled features.
Stochastic gradient descent trades exactness for speed on big data.
It takes noisy steps from single examples and never fully converges, but it makes much faster progress and gives good parameters; Ng notes decreasing the learning rate over time shrinks the oscillations.
WORDS WORTH SAVING
4 quotesIf you have the right notation, you can solve some problems much more easily.
— Andrew Ng
gradient descent and linear regression is definitely still one of the most widely used learning algorithms in the world today.
— Andrew Ng
we just make up names of stuff, and sometimes the names aren't great.
— Andrew Ng
I would almost always just use Batch Gradient Descent because it's one less thing to fiddle with
— Andrew Ng
High quality AI-generated summary created from speaker-labeled transcript.
