Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks

Afshine Amidi 0:05 Cool. Hello, everyone. Welcome to Lecture 2 of CME295. So before we start, I wanted to give you a heads-up about two, uh, logistical things. So the first one is, uh, with Shervin, we reviewed the recording of Lecture 1, and we couldn't help but notice that the audio was suboptimal. So what we're doing for this lecture is to have another setup. But then one issue is my voice will not be, you know, propagated in the room. So I guess I have one question. Can any-- everyone hear me very well, even from the back? Okay, cool. Great. So that's point number one. Point number two is about the final exam. So the final exam right now has a placeholder date for, uh, Wednesday, I think December the 10th. But, uh, just a heads-up that we're trying to see if there's a way to move that to earlier in the week. So we'll let you know when that's finalized, but for now, uh, this is still, uh, TBD, but we'll make sure to let you know. Cool. So with that aside, let's go into today's topic. But before we do that, as always, what we will do is we'll just quickly recap what we saw in the previous episode. So if you remember,

Afshine Amidi 1:30 Lecture 1 was all about introducing the concept of self-attention. And if you remember, what self-attention is, is that each token is attending to all other token in the sequence through this, uh, mechanism of attention. So you have these notations of queries, keys, and values. So here, the idea is that the query is going to ask which other tokens are most similar to itself by comparing query and key. And then once that's done, uh, basically, we will be taking the associated value. So we saw that, uh, self-attention mechanism can be expressed in-- with this formula, so softmax of query times key transpose over square root of dk times v. So I hope this formula is familiar for you. Um, so just know that this formula is highly optimized. You know, these are like big matrix multiplications that, uh, our hardware is, uh, very, um, you know, capable of doing, is very optimized in doing that. And all of that to say that we also introduced the architecture of the transformer, which you can see on the right. So, uh, here, if you remember, the transformer is composed of two main components, so the encoder on the left side and the decoder on the right side. And, uh, the transformer was initially introduced in the context of machine translation. So you can think of the left side as processing the input text in the source language, say English, and the right side is responsible for decoding the translation in a target language, let's say in French. And this multi-head attention layer is where the self-attention mechanism happens. And, um, I remember there was a lot of questions regarding, you know, um, it's called multi-head attention layer, so there are several heads. What does that correspond to? So in the transformer paper, which is the attention is all you need paper, you have this figure which actually represents each of these heads. And you can think of each head as an opportunity for the model to learn one way of projecting the input into being a query, a key, or a value. So just, uh, being a bit more clear. So for instance, for the query and the key, so this is like each heads. So the number of, like, these little boxes is basically the number of heads. And to better visualize and understand what this means, um, I also wanted to show, I guess, what is being shown in the paper, which is the way to interpret what each of these heads do. So we have this concept called attention map, which basically, um, tries to represent the value of each of these query.product query. So in this example, what we're interested in is to see which other token is being most similar to token it's. And so what we do is we take a look at the quantities, the query that is representing it's.product all other keys, and we look at what other keys are leading to a high value of the .product query times key. And when you do that, basically what you see or what the paper sees is you have these two words, so application and law, which are highlighted as being, you know, with a high attention weight. So attention weight here being, uh, the dot product of the query it's and the key, uh, you know, for each of these tokens. And I guess there is also a way to interpret those. And here you can see that the tokens that are being kind of highlighted are law and application, which basically makes sense because the token eats is referring to law. So basically, the model needs to kind of learn how to associate these words with what happened before, and eats is also referring to application, which is also another way of kind of explaining why that is the case. And so here, what the authors chose to do was to show these values as a function of these different heads. So for instance, on the left side, these are the kind of intensity for heads, for let's say the first heads, and then the second heads shows that, you know, the intensity is very high for law. So basically, long story short, these heads may learn, I guess, different ways of, uh, kind of figuring out what, what words matters. Yeah. Great question. So the question is, when we're doing all these computations, are they going through different, uh, different MLPs? The answer is that we're going to have different projection matrices for each of them. And so, um, you can think of, um-- So we had this like detailed example that actually kind of, uh, went through that, where each head is going to have its own projection, and in parallel, you're going to have that computation that's going to happen. So each head is going to have one result here that is then going to be concatenated and then projected once ag-once again with the output matrix. So yeah, long story short, it's highly parallelized, and, uh, it's basically just like projections and like here you have like some matrix multiplication and softmax. Does that make sense?

Speaker 8:30 Yeah.

Afshine Amidi 8:30 Cool. A-any other questions on this? Cool. So I guess this is just a way to illustrate the conversation we had for, you know, lecture one. I, I know there were some questions about, about, um, these, uh, attention heads, what they do. And I guess looking at the attention maps is one way of making sense of what they mean. Cool. With that, I highly recommend that you read the Transformers paper, "So Attention Is All You Need." So it's a very dense paper. It's just a few pages long. Uh, but I hope that what we-- you've seen in lecture one, you'll be able to digest the content in a way that will kind of make sense to you. Cool. So with that, we're going to start the actual meat of what we're going to discuss today. So surprisingly, this transformer architecture, which was introduced in twenty seventeen, is actually an architecture that has kind of, um, you know, still stayed relevant along the years. And there are a few components that have slightly changed, and we're going to see which ones they are. So there are like some slight variations. But overall, today's models are, we're going to see, all more or less based on the initial transformer architecture. So we're going to try to divide the class in, in, in two parts. So the first one is what I'm going to cover, which is what are the parts of the transformer that are important and that had some variation. And in the second part, Shervine is going to talk about, um, I guess the nomenclature of today's models and how they relate to the original transformer. Cool? Okay. So let's start with the first

Afshine Amidi 10:37 important concept that's in this architecture, and this is the position embedding. So if you remember, here we're letting tokens interact with all other tokens in a direct fashion, so they have direct links. But contrary to things like RNNs, where you have a sequential dependency, where you process each token one at a time, here you're basically losing this idea of a token being processed before another one, so you kind of lose this position information. So as a result of that, we need to somehow quantify positions at-- uh, sorry, tokens at each position and try to inject that information when the transformer is processing the, the inputs. So how are we going to do that? So the original transformer paper authors, they chose to have a dedicated embedding. And when I say dedicated, what that means is each position has one embedding. So position one has one embedding, position two has one embedding, et cetera, et cetera. And what they chose to do is to add that embedding to the input token embedding So for instance, if I say a cute teddy bear is reading, A, which is position number one, will be represented by the token representing the token A, plus the embedding representing the first position. Yeah. It's a great question. So the question is, are the position embeddings learned or static? Both. Both as in the authors have tried both, and we're going to see what the second one is. But I guess, I guess here, let's suppose that they are learned. So what does that mean? So that means that basically you need to learn embeddings for each position, and, um, the problem with this approach is that you're very much dependent on what is in your training set. So for instance, um, like here, if you have somehow a text that always has something that is happening at position number two, your learned embeddings will kind of have that bias kind of learned. So that's like one limitation of that. Second limitation is you can only learn positions up to the max number of position that is in your training set. So let's suppose you train your transformer on sequences that are up to, let's say, I don't know, five hundred and twelve, let's say. You can only learn position embeddings up to that position, right? Yeah. Uh, so the question is, uh, I guess how do you parameterize that? So I guess what you do is you have a kind of a placeholder of a position-- learnable position embedding, let's say, between like position one and five hundred and twelve. And basically, when you do your training, you're just letting these, these weights be learned through the regular, you know, gradient descent, all these things. So yeah. So this is like the first method. Uh, but as I was mentioning, the-- it has its limitations, um, because you can only learn embeddings of positions up to the max position that is present in the training set. So for instance, if you have at inference time a position that is beyond the position that was, uh, in the training set, well, you have not learned that. So you need to find a way to kind of infer the value. So that's the second limitation. And, uh... But yeah, but on the pro side, I guess you're just letting your model learn and, uh, we've seen that, uh, gradient descent does wonders when it comes to just, you know, learning from the data. Um, so yeah. And for these reasons, these methods was something that the authors said that was performing well, along with a second method, which is different, which is around

Afshine Amidi 15:37 having an arbitrary formula for each dimension corresponding to a position embedding, and we're going to see that now. So first method was, you know, you have one embedding per position and you just learn that. Second method is you have one embedding per position, but you're not going to learn that. You're going to have something that is predetermined that you're going to use. And we're going to see that what the author chose was a formulation using sine and cosine. So it can feel kind of weird, you know, why did they choose this? But we're going to see why that makes sense. So the idea here is for a given position, let's say M, have a vector of size D model. So D needs to match the dimension of your token embeddings because of course you're adding them. And what you're going to do is for every index, you're going to compute the corresponding value with respect to these formulas. So what are these formulas? So it's basically sine of something times M, and we're going to see what that something, something means. And then the second one is cosine of something times M. So who remembers trigonometry formulas? Cool. Everyone. [laughs] So before we go into that, let's just simplify the notations. Let's just assume that this big quantity that you saw is actually something like omega. So let's suppose it's omega as a function of i times M. And you note omega i as being this, you know, quantity, so ten thousand to the power of minus, uh, two i over d model. Let's suppose you, you construct your embeddings to have this way. Then I guess I want us to think about why that would make sense. 'Cause if you think about it, what you want is to represent positions in a way that reflects the following facts. Words that are close together are likely to be more relevant as opposed to words that are further together, right? So if you have, uh, two words that are, like, just one position apart versus ten, ten thousand position apart, what you want is that the one that is one position apart is more similar than the other one. So let's see if the formula makes sense. So let's suppose you have two position embeddings, so one at position M and the other one at position N. And let's suppose you compute, you know, all the values from this predetermined, um, formula. Well, it turns out, if you remember your trigonometry formulas, so cosine of A minus B is equal to cosine of A, cosine of B plus sin-sine A, sine B, right? Well, turns out that if you express cosine of omega I factor of M minus N, this is something that you obtain. It's just the I-- like, the identity that I mentioned. Well, it just turns out that this quantity is just one component that appears when you do the dot product of these two position embeddings, right? 'Cause basically here, when you do the dot product of position M and position N, what you do is you take the first position, you multiply them, then you plus the second position, you multiply them, et cetera, et cetera. And then you come here. It's sine of this times sine of this plus cosine of this times cosine of this, right? Which is just this quantity. So at the end of the day, what you realize is that when you perform a dot product of these embeddings, you end up with a sum of cosine that are a function of the relative distance between M and N.

Speaker 20:29 [inaudible]

Afshine Amidi 20:32 Yeah. What do you mean by pair, by the way? Exactly, yeah. So basically-- So the question is, um, yeah, so the closer they are, the more similar they are. So that's the intuition, that basically this way of formulating, uh, the embeddings is trying to approximate or it's trying to mimic. So with that, you basically obtain a dot product that is just a function of M and N, the relative distance between them, actually. Um, and just as a reminder, so why do I care about the dot product? It's because if you remember, in the embedding world, when we try to quantify the similarity between two embeddings, what we do is typically something that is involving the dot product of these two. So you typically have the cosine similarity, but cosine similarity is just dot product over the norm of each embedding, so, which is basically a dot product, right? So that's why we care about, uh, the dot product, and here we see, great, it's a function of the relative distance between the two. So in particular, again, if you remember your trigonometry class, cosine of zero is one. And the higher this number, the lower the value of cosine of this number, right? Of course, it's, uh, periodic, so what I'm saying is not necessarily true past two pi or, sorry, uh, pi. Just goes the other way. But I guess what I'm trying to say is for M equal to N, you have basically a sum of cosine of zero, and it is the value at which this quantity is maximum. So when M is equal to N, this quantity is maximum, which means basically if you're looking at the position itself, it's the most similar, which basically matches our intuition. And so now when you plot the values of the embeddings, this is what you obtain. So here in this graph, on the Y-axis, I'm basically representing all the embeddings for each of the, let's say, fifty positions. And on the X-axis, it's basically values along a given vector across several dimensions of a vector. So if you take, let's say, the first row, you're looking at the first or number zero position, depending on how you index your vector, and you're looking at all the values of the position zero embedding. And so you see that, uh, for low dimensions, this value kind of goes up and down very frequently, so it's more high frequency. And when the dimension is high, it basically takes a lot of time for the value to go up and down, so it's basically more low fr-frequency. So this relates to this omega i that I mentioned, this omega i. So omega i is very high for low values of i, which is the dimension, and is very low for high values of i. So this basically just determines how quickly your cosine and sine basically vary. Cool. So this is what the original authors have tried, and basically what they said, what they noted was that using this method leads to comparable results compared to the learned one. But here we have a big advantage, 'cause it can extend to any sequence length, not just the sequence lengths that you saw at training time, and this is one of the reasons why, you know, this may be something that is preferable. So yeah, this is the intuition, and this is what the authors chose. Now fast-forward to twenty twenty-five. I guess you may ask me, are we still using that? And the answer is kind of. So we're still using this idea of we want far, I guess, tokens to be less similar than closer tokens. But we're not injecting the embedding like they did, and we're going to see why. 'Cause

Afshine Amidi 25:57 if you remember, what you care about is determining how similar tokens are in the self-attention computation. And where does the self-attention computation happen? In the attention layer. But here, what, what did I say? I said, "Let's compute these embeddings, and let's add them here." But actually, what we want is to reflect this similarity in the attention layer. Do you have a question? Yeah. So in this first method, yes. So the question is: Is it added to the input feature? Yes. Yes, just add. Yeah. Yeah. But the problem is, we mostly want these, I guess, intuition to hold true in the multi-head attention layer. So this is one of the reasons why people have tried different variations and, in particular, have these position embeddings intervene directly in the attention layer as opposed to the input. 'Cause basically, when you do it at the input, just here, fair, okay. It's going to be roughly something that is going to go into this attention layer, but it's kind of indirect. So what we want is to directly kind of do something about the attention formula that would, I guess, reflect the fact that we want close tokens to be more similar compared to further tokens. And the way we do that is, if you remember, the self-attention layer is basically the softmax of QK transpose over square root of d times V. So what we want is to add a little something inside that softmax, which is basically where you quantify how similar a token is to another token. You want to add a little something to reflect the fact that some tokens, they're supposed to be more similar compared to that token compared to oth-others. So there's a, a, a few methods that have tried to kind of have some variation of that. So for those of us who know, like, paper T-T5, the T5 paper, which we're going to see a little bit, uh, later, um, they have tried these kind of relative position bias by learning the bias term, which is in the formula above. So what they did was, let's suppose you have a given distance between the positions M and N. So their idea was, let's learn that. Let's basically bucketize all, you know, M minus N into some buckets, and let's just have the model learn these quantities that are then going to be injected in the softmax. Yeah. Mm. So question is: Does that pose a problem that the, the bias is here with respect to the probability at the end 'cause it has to sum to one? Well, you can do whatever you want inside the softmax, 'cause the softmax is going to normalize it anyways. So you can think of the bias as being something that is maybe more negative for things that are far apart compared to things that are closer together. So T5 says, "Let's learn them." We have another method from, um, I guess this Train Short, Test Long paper, which introduced this method called ALiBi. ALiBi, uh, stands for Attention with Linear Bias, I believe And what they did was say, "Instead of learning those biases, let's actually have a deterministic formula, which is at a function of the difference-- the relative d-difference between those two positions." And they said that, so they kind of had some results. So you know, all these papers, they always kind of compare based on one another to kind of see which one is, um, is, I guess, more performant. But the reality

Afshine Amidi 31:02 is, is that in today's models, most models actually use another kind of position embedding method, and we're going to see it now. So this method relies on rotating the query and the key vector by some angle. And I guess you can think of it as, you know, you have your query, you have your key, let's suppose, in a 2D space. So what you're going to do is rotate your query by some angle that is a function of its position, and you're going to rotate the key vector by some angle that is a function of its position N. And I guess, how do you do that, by the way? I wasn't supposed to show this thing. So let's suppose you have a vector, by the way, and you want to rotate that vector, 'cause I had the answer on the slide. But I guess how would you go about this? I guess for people who just want to kind of talk about this with intuition. So who here has done, uh, I don't know, rotations in, uh, in space? Yeah. Mm-hmm. Mm-hmm. Yeah. That's correct, yeah. Mat-mat-matrix co-multiplication, and you're going to use a quantity that's called rotation matrix. And, uh, the ro-rotation matrix is, uh, expressed as follows. So it's basically a two-by-two matrix in the 2D plane that has cos-- uh, cosine of, uh, this angle, minus sine of this angle, sine of this angle, cosine of this angle. I'm looking at the time. There's actually-- It's, it's quite simple to just show that it works, uh, but we may run out of time. Do you wanna-- You want me to quickly show you that it's indeed a way to, uh, rotate a vector? Okay. So here, as a reminder, what we're trying to show is that we can use a matrix multiplication to rotate a vector in 2D space. So let's suppose we have the following vector. That is something that you can, um, you can quantify with, uh, two dimensions. Let's say x and y. You can express your vector in 2D space with this, right? But I guess this, if you note R the norm of the vector and Phi the angle with respect to the x-axis. You can also write V as R with vector cos of-- cosine of Phi and sine of Phi. Right? So if you multiply the rotation matrix with this V, what you're going to obtain is some multiplication of cosine, minus sine, sine and co-cosine, uh, of this and that. And I will leave this exercise for you, but you can show that, uh, rotation times this V can be expressed as R of cosine of theta plus V and sine of theta plus V. So this is a quick proof. So I'll just kind of leave you the multiplication of the rotation matrix and V. But you will obtain these, like, trigonometric identities that will have, I guess, lead you to this formula. This basically shows that if you multiply this matrix and this vector, you're basically rotating the vector by this angle. Yeah. So, uh, question is: Why do you want to do this? It's a great question. It's my next, uh, slide. So this is just like a little intro. Um, so I guess here, just going back to these methods, what we want to do is to quantify the similarity between tokens and have close tokens be more similar compared to tokens that are more afar. So the problem that we had with the previous methods was-- So in the first method, this learned embedding, you always had this, uh, overfitting issue, because basically, when you learn these biases, it always depends on which training set you have. So maybe, uh, your data set is in a way that, let's say, tokens that are close are kind of similar, but in a different way compared to what you see at inference time. And this ALiBi method It didn't have that learnable component, but this was quite restrictive 'cause it's a very simple formula after, after all, right? Just the relative difference between N and M. So I guess people have tried different ways of coming up with something that kind of tells you that, I guess, an embedding that reflects the fact that you want further positions to be less similar than closer ones. So in this method, we're going back to the sine and cosine world from what the author had proposed, and think about similarity from the lens of, I guess, cosine and sine functions. So this is a little intro, and their method is called RoPE. I'm not sure if you've heard of it. So it stands for Rotary Position Embeddings, and we're going to see that this method-- So why, why do we care about this method? So this method has two great things. So the first one is that if you rotate the query and the key, you will end up with a quantity that will be a function of the relative distance between the two. That's going to be very nice, and that's why I, I wrote this thing on the blackboard. Not sure if you can see, by the way. But, um, we won't have time to go into the mathematical detail, but if you want to just, you know, express these things at home, uh, this is just a foundation. And so in particular, if you remember your attention formula, so you have, you know, query times key transpose. So if you rotate the query by an angle M and the key by an angle N, what you're going to end up is a formula that has the rotation matrix of angle theta and, I guess, N minus M. And this is great because this is a function of the relative distance between these two positions. Okay, so why do I talk about this in detail? Well, it turns out that most models these days, they use RoPE, which is why it's important. And I would say another thing, it's, it's maybe a little bit hard to get the intuition as, as to why that works. But hopefully, the explanation that I gave regarding the, you know, sine and cosine at the very beginning can help you just build that intuition. And speaking of that, it turns out that the upper bound of the attention weight given by the query and the key is such that we observe a long-term decay, meaning that as M minus se-- M is large, we do see the upper bound that gets kind of smaller and smaller. While you see these, uh, little oscillations, it's not, like, perfect either, but we do have some mathematical, uh, I guess, results as to just the upper bounds kind of decaying over the long term. Cool. Any questions on this? Yeah. Yeah. Exactly, yeah. So the question is the relative distance is captured in the rotation matrix, and yes. Yes. Oh, yeah. Ah, it's a great question. So question is, uh, what is, uh, theta? So theta is actually fixed. So do you remember this omega that I talked about here? Here? So it's basically some function of I and D. I actually kind of, uh, passed it quite quickly. But what I showed you is in the 2D space. But here, we're in, uh, this D dimensional space, which is greater than two. So I kind of glossed it very quickly, but the way you extend this method is by having these 2D, 2D space kind of by block. But then the theta is a function of, uh, typically something that you fix, but a function of I, which is the dimension, you know, um, it's basically between one and D, D over two. It's a function of that, and it's a function of, uh, D as well. So it's typically-- You, you, you will see this theta as being roughly equal to omega I, which is this, this one, more or less. More or less. Sorry? Oh, as it-- So the question is, uh, so that it has the same dimension as the latent dimension. So well, here you have a product of matrices, so you need to have, like, the dimensions match. So I guess your answer, yeah. Does that make sense? I'll take that as a yes. Okay. So I, I spent a bunch of time on this because, uh, I think this is actually quite important. Uh, a lot of models use this and, uh, yeah, I guess the intuition is not super obvious. So I hope this was helpful. Okay, so this was, uh, position embeddings. Yeah. Oh, yeah. Oh, yeah. So the question is about, uh, how do you obtain this curve? So this is actually a curve that I believe is shown mathematically 'cause basically-- So it's kind of, uh, complicated. We're not going to write down the formula, but if you're interested, so in this paper, in the Roformer paper, there's an appendix where they show mathematically that it is upper bounded by some quantity, and this is what, this is what this is showing. Yep. Great question. Cool. So position embeddings is one part of the transformer that has changed a little bit, and we've seen how that changed and why. So now we're going to see another

Afshine Amidi 43:42 component of the transformer that has also a lit- a little bit changed, and that component is the layer normalization. So if you remember, the transformer architecture is, uh, composed again of encoder, decoder, and then you have the components inside. So you have these boxes that say add and norm. So what do they mean? So basically, here, what we do is we take the input as well as the output of this sublayer, we add them together, and then we normalize. So this is a little trick that the authors do, and it is shown in practice to improve convergence and just make the convergence be quicker. So the idea is as follows: If you have a vector, sometimes the components of the vector can be super large, sometimes they can be super small. The idea here is to kind of normalize the components of your vector within some range, some normalized range. So the way you're going to do that is you're going to take your vector and then subst- subtract it by the mean, computed mean, which is basically the sum of its components, and normalize it by basically its, uh, standard deviation. And what you're going to do is you're going to learn two quantities. One gamma, which is going to be the rescaling factor, and then beta, which is, um, another factor, another term as well that you learn. And you're going to let these two quantities be learned by your model. And so in practice, as I mentioned, so what this does is it helps with training stability and with convergence time. So this was a technique that was used in the original transformer paper. Uh, I just want to call out that there has been some changes since then. Um, so we went from normalizing the input plus the output of the sublayer to having a sum of the input and the sublayer of the normalized input. So in other words, what we've done is to change where the normalization is located. So here in the, the transformer paper, it was what we now call a post-norm version. And nowadays, we use a pre-norm version, which basically consists of having the layer norm right before the vector goes into the sublayer. And sublayer here can be either, uh, the attention layer or the FFN. But not only that, there is also another change. So nowadays, people, they do not use layer norm. They use something else called RMSNorm, RMS root mean square normalization, which is basically a variation of what you've seen before. So instead of computing this, basically what people do is they just normalize X by the root mean square of the, of the components of X, and they learn gamma. Only gamma. So why do they do that? Basically, they show that the convergence, uh, properties, they're basically comparable, but here you have fewer parameters to learn, so it's basically quicker. Yeah. Mm. Mm-hmm. Good question. So I guess the question is, what is the intuition behind normalizing? So the intuition is that if you look at your model, you have several layers. In some layers, your mo-- your vector-- your activation, to be more precise. So do you know the, the vector that you see that goes from here to here is basically called the activation? Sometimes the activation has extreme values in o-one part of its component, sometimes in another part. And the model is typically having trouble in learning the weights in each of these layers if these activations, they vary too much So the idea is to bring the values of the components of the activation to some range that is not too, you know, far off in some direction. So in case you're interested, there is this keyword, internal covariate shift, which is basically the term that is given to the phenomenon that I'm describing here. So yeah, that's the intuition. Yeah. Oh, great. Great question. So question is: What is the difference between this and batch normalization? So batch normalization is normalization across the other dimension, which is the nor-- the dimension of the batch. So let's suppose you have a bunch of vectors. What you do is you normalize each component with respect to all the other components for the same dimension, but of, of the other vec-vectors. So you can think of it as just another way of normalizing. Uh, but that being said, uh, when it comes to these, like, transformer-based models, typically people use layer norm, probably because empirically it works better, but also because, uh, batch norm is you-- you're basically also dependent on the batch, and it can be-- it can introduce some, I guess, differences between training and then inference. So that's basically the reason why. Cool. Great. So we've seen position embeddings, we've seen layer normalization. So now we're going to see a third

Afshine Amidi 50:39 important component of the transformer, which is the attention. And in particular, I guess there's something I've not really emphasized, but when you do self-attention, you're basically letting every token interact with all other tokens. So when you look at, let's say, a matrix that shows all the interactions, so you have, like, N, the sequence length, and, and the sequence length, you basically have O of N squared complexity, which is a lot, especially as N goes longer. So people have tried to kind of approximate this, you know, O of N squared into something that is a little bit more tractable but does not lose the performance. So there was this paper in twenty twenty that came out, LongFormer. So what it did was just try different versions of the attention by restricting the window at which it operates. So here, instead of letting each token interact with everyone, each token only interacts with its neighborhoods. Yep. Oh, again, a great question. So the question is: Do you do that after the, uh, attention matrix, uh, computation? You, you mean the softmax, right? Softmax. So you raise a great point, which is, if you do the softmax of everything, why, why would you do this? [laughs] So in practice, there is a bunch of implementations that does a clever, uh, so it's called, like, tiling, bunch of clever, uh, operations that do not involve this huge matrix operation that you see in the softmax. Uh, I guess it's, uh, like, I guess softmax of QK transpose over-- You're not going to compute the whole thing. You're going to have a little bit, like, s-some cleverness in how you compute that, basically. Yeah, so the question is: Can it be, um, uh, something that's compared, c-c-comparable to convolutions? And we're going to see that in a second. But yeah, you have some, uh, I guess, similarities with the vision world, and we're going to see that in a second. Cool. So nowadays, when you have local attention like this, like this, people use the term sliding window attention. So when they use that term, they, they mean this, which is basically just restricting the attention to neighboring, uh, tokens. And nowadays, what people do is, in some layers, they will have local attention. In some others, they will have global attention, and they interleave these layers. So depending on the model, they, you know, typically try different combinations. So there's not, like, a set, uh, recipe, but it's typically something that is u-used nowadays. So the window here, I mean, for illustrative purposes, uh, the window is super small in my illustration, but, um, I guess nowadays with the sequence lengths that can be very big, you can think of this window as being O of, uh, several thousands. So it, so it seems pretty sizable. And, uh, just to give you another example, so back to the convolution, um, um, a-- like, comparison that you mentioned. So, um, you have some architectures, and here I'm going to take the example of, uh, Mistral, that has this sliding window attention at every layer. But then, uh, when you think about it, the token here can attend up to the token here, but then the token here can attend up to the token here, et cetera, et cetera. So I guess if you think about it- It's kind of similar to the idea of the receptive field in computer vision. So I'm not sure if you're familiar or kind of from the computation-- computer vision world, but if you are, um, what this means is just taking one token and trying to think what other tokens has this token effectively interacted with. Which is basically the question that people also sometimes ask when they do convolution. They're like, "Okay, so this value, what are the values did it actually see?" So yeah, you can also think it, uh, this way.

Afshine Amidi 55:39 Okay. So first variation is instead of doing the full, you know, N by N attention, people sometimes do local attention. The second variation that is kind of orthogonal to all this is to not have one projection matrix per head, but to share projection matrices across heads. So here the idea is you have, let's say, H heads. The idea is you're going to have some number of projection matrices for the query, but then what you're going to do is to group the projection matrices for the key and the value that you're going to share across several heads. Now you may ask, why do you share projection matrices for the keys and the values, but not for the queries? Is this a question, yeah, that you, you're wondering? Well, here I guess, you know, people do things to just try out if something works or not. I guess here you can intuitively think that, uh, you know, the query is basically wondering if something is similar to another thing. So I guess you can ask yourself this question in different ways. So it may make sense to kind of keep that diversity. But one of the core reasons why we choose to group projection matrices for the keys and the values and not the query is because when you decode, you perform attention between the current word and all the words before. So in other words, every time you generate a new word, you're going to attend that word to all the words before. So the keys and the values, they're going to come up a lot, right? So every time you want to decode something, you need to kind of attend to all other things again and again. So we're going to see in, I think, next lecture, there, there's something called the KV cache, which basically saves the values of the keys and values. And one thing that we want is for that cache to not become too big. So if you share projection matrices across heads, just allows you to save a little bit of space, a little bit of memory. That's the, that's the why. Does that, does that roughly make sense? Okay, cool. So speaking of that, there are some variations as to, I guess, uh, just how many projection matrices to, uh, share. So you have the extreme example where all H heads, they all share the same projection matrices for V and Q-- and K. And in that case, this method is called MQA, Multi-Query Attention. You have the, you know, uh, in between case where you share G, where G is the number of groups, G projection matrices for V and K, and then you have, uh, groups of size, uh, let's say H, uh, over G. So this one is called Group Query Attention, GQA. And then you have the case that you are very familiar with from the transformer, which is every head has its own, um, query projection, key projection, and value projection matrices, and this is the standard multi-head attention. Cool. So I'm going to just take, uh, like a pause to see if, uh, what I mentioned makes sense. Wanted to soon, uh, transition into Shervine, but before I do, I just want to make sure that what I discussed is, uh, kind of making sense roughly. Yep. Mm-hmm. So the question is, do we apply this to self-attention and cross-attention? I don't want to spoil, uh, the show, so I'm going to talk about something later on. But, um, where is my trans-- So if you take a look at the transformer architecture, we're going to see soon that what we care about most is what is in the decoder because we actually forego of the encoder in nowadays models. We, we have not seen this yet, but I'm just telling you. And so this typically comes into play for the masked self-attention in the decoder But this technique can be applied in all attention layers. But I guess what I'm telling you is modern LLMs, they are decoder-only models, which basically only have the decoder part of the transformer. We're going to see this in a second. Which is the masked self-attention. So I'm not going to say too much because Shervine is going to cover that, but yeah, just a quick peek of, of what we're going to see. Cool. Yeah. Yeah. Great question. So, uh, question is, when do you know which one you should use? I would say the choice is always driven by a few factors. One is how well it performs. Second is how, how much do you care about things like latency costs? Um, and so it really depends how big is your model, how much you want to save on, let's say, compute. Uh, like, you know, how is your, uh, input length? Like for instance, if you have like a shorter input length. So here, what we wanna do is to avoid having to do all these things for the whole, you know, O of N squares, uh, thing. So I guess all of these come into play. So I would say it's not straightforward as to what the answer should be, but I would say, uh, a lot of, uh, recent models, they tend to share, to share projection matrices. So typically, I would say GQA is what you would see, but, um, it's not necessarily the case for all models. Cool. We're running out of time, so I'm going to have Shervine here for the second part.

Afshine Amidi 1:02:42 Thank you. Okay, great. Thank you, Afshin. So we're going to continue, uh, this lecture with, uh, some deeper dive into the kinds of models that we have in the transformer landscape. Um, and then we're going to do a deep dive into one specific architecture that is very useful for classification settings. So, so first, we're going to come back to the architecture that we saw together, uh, last time. So this traditional encoder-decoder architecture where you have both components. And so you have the original transformer paper from twenty seventeen that had this architecture, but also later on, you see more architectures that built on top of it. So here we talk about the T5, uh, family of models. So T5 is a paper that's like-- So, so it's an abbreviation of like multiple Ts. So the first one is transfer, and then it's text-to-text-transformers. So this is where the T5 naming comes from. And then it's derived into multiple versions. So the T5 is like the vanilla paper, and then it had MT5. M stands for multilingual, uh, where, um, like there was some more work on the data it was trained on, as well as, uh, like the vocabulary that it was computed, uh, over. And then you have a Byte-T5, which is, uh, some sort of tokenizer-free method of all of this, where you forego of the fact of tokenizing, and instead of that, you, uh, basically operate at the byte level. So like byte is like bytes. And basically, you have a vocabulary size that is much smaller. So instead of having like a O of thirty K, uh, you have, um, two to the power of eight, so like a byte is eight bits. And then, uh, you can represent every character in two bytes. So that's what they do. Um, okay, great. And then one thing I want to mention regarding the T5 family is that the objective function changes a bit with what the original transformer, uh, did. So the original transformer did next token prediction for the training task. But the T5 family, what they did is that, um, they operated on the so-called span corruption task. So basically, you would have your sentence as an input to the encoder, and instead of putting everything to the encoder, you would leave blanks. And this is what we call, you know, span corruption. And then the span corruption could be one or multiple tokens missing. So if I want to give an example, so for example, my teddy bear is cute and reading. So you could have my teddy bear span corrupted and an is reading. So that could be one potential, um, encoder input. And then you could have up to N. Uh, I mean, N is the parameterization of the number of spans that are corrupted. So you would have like N such tokens here. And then the T5 families, uh, calls them sentinel tokens. So if you see sentinel tokens, they represent a span of corrupted tokens. So you have them here in the encoder, and then the decoder's work is to find each of these spans, um, in series. So you start with a token that denotes the first corrupted span, and then you start the decoding process until it hits a prediction that predicts the next, uh, sentinel token up to the N plus one one, where, um, the kind of tokens that are decoded between two consecutive sentinel tokens corresponds to the s-- corrupted spans that are recovered. Yep. So like this parentheses is basically a shift from the next token prediction, uh, objective function. Yep Yeah. So the question is, can you elaborate with the decoding process? So what you said regarding reconstruction is exactly right. So you have sentinel tokens that denote some missing text. So what you want to decode is that missing text. So the decoder output will be exactly like each span reconstructed. Yeah. And then if you want to know how training works, you do a teacher forcing, um, mechanism where you, like, input everything in the decoder and then try to reconstruct e-everything at once. So, uh. Any other questions? Great. Um, and then now we're going to talk about another class of transformers where, uh, basically you have this encoder-decoder str- structure where you just forget about the decoder and then just deal with the encoder. So you might tell me, "Okay, hey, you cannot do, um, generation-- the generation with this?" And then I would, I would respond to you, you know, "Yes, that's exactly the point." Uh, so this encoder and has-- So it has encoder representations that can be used for, um, like tasks that might be more geared towards classification. So, um, like sentiment extraction, uh, token classification, like all of these things that used to be done with, uh, specific language models, uh, they can be done with the encoder parts of the transformer. And we're going to, uh, dive deeper, uh, l-- a bit later into, uh, three, um, like key encoder-only models. So BERT, which is, uh, like the central one, uh, I would say in this landscape, and then two other architectures, uh, DistilBERT and RoBERTa, that are, uh, investigating on axes of, uh, improvement. So it's going to be good to see. So, okay, great. And then, uh, there is one last class. So as Afshine just mentioned, uh, like today's LLMs, they remove the encoder part altogether. And then when you have no encoder, you don't have your encoded embeddings at the end of your stacked encoders that could be, uh, be fed to the cross attention, so this module disappears, disappears altogether. So each decoder that is stacked just has masked self-attention and an FFN, uh, as part of it. Sorry. And, uh, yeah, that's basically, um, you know, something that has, uh, caught up, uh, since then because when you look at the popularity of each of these models... Uh, so you used to have, uh, this, like, transformer-like architecture that was popular towards the beginning, where the main hypothesis was that, like, the encoder part is very useful for, like, getting to the, uh, like decoded, uh, representation. But as time went, people realized that your compute budget could be best invested in the decoder only. Uh, and then, um, you know, you had-- there, there has been more investment into the kind of task that is, I would say, easiest to scale up and generalize, next word prediction, as opposed to the task I just mentioned for T5, which might be, uh, kind of more bespoke. So you need to, like, corrupt things and then... You know. So it's more complex, whereas, like, next word prediction is, I would say, like the simplest thing you can do. And it showed, uh, like it proved to be wonders and well aligns with the task of being a, a helpful kind of chatbot, which is like today's applications, I mean. And then the decoder-only architectures. So I'm not going to talk about them today, uh, but it's going to be the central part of the next lectures, uh, when we're going to talk about, uh, LLMs more and more. Okay, awesome. So now

Afshine Amidi 1:11:38 we can dive deep into, you know, as promised, into encoder-only architectures and with BERT. So first, we're going to start by, you know, seeing what does BERT mean. So BERT, it's an acronym that denotes bidirectional, uh, encoders representations from transformers. And we're going to see together how does each section of this acronym cor-- what does it correspond to? So let's just start with, uh, the encoder part, which is like the easiest to, to grasp. So as we said, we just drop the decoder. So like this encoder from transformer is basically, you know, exactly what it means. Now on the other part. So, uh, so why do we talk about bidirectionality? So it's a way, um... So, so the paper's result is kind of remar-remarkable because we are able, from a given input, to get, um, like output representations that have attended to everything, you know, for each token. And, um, this is the case because since we only have the encoder, we have this self-attention layer that truly attends to every other token. And this is in contrast with the masked self-attention that you have where we said that the mask is making the attention mechanism causal. So every token can attend to itself- And to the tokens before it. And this is, by the way, something that the authors, uh, like discuss a lot in the paper saying that, you know, GPT came out. GPT, they're not truly bidirectional. And then these encodings that can be used for classification task, they truly are. Any questions? Yep. Yep. That's right. So, uh, the question is, when you don't have the mask, each token can attend to each other. You know, that's exactly right. And then the mask is exactly there to prevent links from tokens to like those that, that kind of come after them. Yep. Okay, great. So I just want to put this paper in context and, you know, the field of NLP was booming back then. So you had another landmark paper that, that same year, which was called ELMo, so Embeddings from Language Models. And I would say that the timing of that, uh, paper was a bit unfortunate because it truly had new insights of, uh, also building bidirectional representations, but it just turns out that it came the same year as a transformer. Like, as, as a kind of transformer-based, like, same line of work. So it got a bit masked by it. So ELMo, just to give the main lines, it was based on a bidirectional LSTM, where you had multiple layers, um, stacked on top of each other, and basically, you were able to build a bidirectional representation for each word, uh, thanks to this, um, architecture. And so why didn't it, uh, get, uh, you know... Why didn't it become as popular as BERT? It's because you had the same, uh, downsides as the previous models, where basically it's hard to scale, uh, because of this recurrence. Um, and, um, you know, I think a lot of you, you know, when you think about ELMo and BERT, you don't think about these papers, you know, at first because they are part of, like... they are characters in Sesame Street. And, you know, I grew up in a place where I, I did not know about Sesame Street. So like, for me, ELMo and BERT is like paper names, but, you know, it might be, you know, what represents to you, um, you know, at first. So I thought that was quite funny. And then, you know, researchers, they are generally quite, um, you know, playful. They have, uh... Like, they try to fit their acronyms into, like, themes. So, you know, if you look at paper names, you, you'll get, um, entertained. Okay, great. So to dive deeper into what I mentioned to be the goal of encoder-only models. So you have your set of tokens as input, and the goal here is to perform tasks that might be, uh, focused on projecting some representation somewhere, so typically classification tasks. And here the, uh, like the way BERT works is very specific. So you have two kinds of tokens that you will see, uh, are kind of structural here. So you have first the CLS token, which is-- which stands for classification, and it's basically a placeholder token that is put at the beginning of the sequence that will then, uh, at the end of the whole attention and then projection and, you know, the whole encoder mechanism, be projected into an embedding that we can then, uh, use for classification. So yeah, CLS is just some kind of placeholder that will, uh, carry the bidirectional information of your whole input. And another, uh, token that might be, you know, useful to see is the SEP token, so it's like separator. Um, and then you will see, uh, very soon we're going, we're going to see the objective functions that it operates on. It's, uh, it aims at separating, uh, two sentences. Uh, yep. So this is the very high level. And then, um, so one thing that is very interesting to see with this model is that you, you have this concept of multi-stage training. So you don't train the model, you know, in one shot. You do it in multiple stages. So the first stage is aimed at being aligned with the task of interest. And, uh, so this is what we call pre-training. And this pre-training, we will see in detail, is done with two objective functions that are respectively, uh, MLM and NSP. So MLM stands for masked lan-language model, and NSP is the next sentence prediction. So masked language model, it's a way for a model to learn the internal structure of the, uh, of the input. And then NSP, it might be, um, seen as a way to see, like, if the ordering of sentences makes sense. So we're going to, like, to go, like, a bit deeper into each. But it's a combination of objective function that the authors, um, you know, assume to be helpful for learning, like, general, uh, general embeddings of high quality. Okay. So the second thing that I want to mention is that once you have all of this, you have a further stage where you keep the embeddings that you have learned, and then you attach to it some other network, like typically a linear projection, to then fine-tune, uh, what's-- like the embeddings that you have learned into some target task. So this is what we call fine-tuning. Yep. So yeah, so, uh, so the question is, um, do we still have just an encoder here? Because a next sentence prediction is, you know, could be seen as a, a decoder task. So it's actually... I'm going to go in detail. The next sentence prediction task is actually us putting two sentences, one after the other, and predicting whether they are truly consecutive. Um, yeah, so it's a classification task. Yeah, great point. And, uh, yeah, we're going to see that in detail very soon. So before we see that in detail, um, I'm going to discuss the pros and cons, uh, that this method usually has. So, um, so regarding, uh, like, on the pros side, like, this pre-training, um, mechanism can be done on, like, fairly unlabeled data. So you still have this, like, next sentence prediction task, but this is something that you control because you know which sentences follow each other, so it's something that you kind of, you know, you know, and it's kind of self-supervised. And the masked language model task, we're going to see how it, um, you know, how it consists, but, um, but it's also something that is unsupervised. So this is a very interesting way to learn interesting embeddings out of, uh, unlabeled data. And then, uh, in practice is that we see that this unlabeled data leads to helpful representations being learned, and then you need very little data to build on top of it. So in the fine-tuning stage, you just start from these very nice embeddings that you have learned, and you just have a few weights to tune. And then this usually leads to, um, performance that exceeds state-of-the-art back then. And, um, and then regarding, uh, you know, downsides, we have, uh, you know, of course all the text generation, uh, kind of tasks that are out of reach because we don't have a decoder. And, uh, also we can see this two-stage process and the fact that we need to further tune embeddings as something that's, that can be, you know, other hurdles. So it might be, uh, you know, when you compare this kind of methodology, uh, with respect to more traditional methods that can be, you know, one shot, this could be seen as a downside. Okay, great. And, uh, you know, I have some, uh, you know, names, uh, regarding, uh, variants here. We're going to dive deep into two of them a bit later on. Okay, so I want us to focus on the original transformer, and we're going to go step by step into what happened to it and then how we get the BERT architecture. So this is what we had in the original transformer paper that we saw last week. And, uh, what BERT did is extract the encoder part of it and attach basically these new, um, objective functions that I just mentioned alongside some new set of tricks, uh, when it comes to representing, uh, tokens. So I think we're going to go, uh, you know, we're going to go through them one by one. So first of all, one, uh, interesting note is that it uses a specific tokenizer called WordPiece. So you can see it as a tokenizer that learns on your training set based on, uh, you know, merge rules that maximize the likelihood. So basically, you have some huge training set and you train a tokenizer that merges atomic tokens together to build, uh, your target vocabulary. And there's some order of magnitudes. I mentioned O of 30K, so it's typically, you know, uh, I think this is the size that they chose in this paper. And in general, um, like, vocabulary sizes in these sorts of paper are in the order of magnitude of ten to the power of four, ten to the power of five, except for our token-free method that we saw, uh, like, Byte T5, which is, like, this very restricted set of tokens, like, two to the power of eight, which is like two fifty-six. You know, apart from this specific case, you, you always have, um, kind of this order of magnitude. And, uh, okay. And basically, we are going to make use of the tokens that I mentioned, um, like, at the beginning. So you have the TLS token that is going to carry the bidirectional representation of all your, uh, sequence. And then you have SEP tokens to separate the two sentences, uh, towards the next sentence prediction task. And there is something that I haven't talked about just yet. Uh, so I think maybe we can talk about it at the time of the deep dive. So basically, in order-- So in order to have this, like, masked language model task, so of course you need to mask some tokens. So we're going to see the technique that is, uh, basically applying these sorts of mask and where, and at each fr- at, at which frequency. And we're going to see, with respect to the output, what is going to be our task based on the resulting representation. So I'm just going to, like, pass quickly on it here, but we're going to come back, uh, very soon. Okay, great. Um, so one big piece of news regarding your input embeddings. So, like, j- what stays the same? So you still have these huge- Dictionary lookup of embeddings for each token that you're going to learn, and you're going to additively add the positional encoding that we saw, uh, at last lecture, um, and then Afshin talked about earlier in this lecture, which can either be hard-coded or learned. Uh, I think the au-authors here just, uh, use an, a hard-coded version here, uh, but I'm not too sure. Like, but it's roughly the same performance usually. But there is something new. There is the introduction of a new kind of encoding called segment encoding that is going to still be additively added to tokens, uh, with the exception that we just have two, uh, two possible encodings there. So you have segment A that represents the first sentence, and then segment B that represents the second sentence, and it's supposed to help with the NSP task to represent, you know, what could be features that can helpfully represent a sentence that precedes another one. Uh, so at least that was the hypothesis emitted by the authors. We're going to see it has been challenged later on, but, uh, you know, this is one of the key concepts that was introduced here. Any questions so far? Yep. Yep. So y-great point. So the question is, what does segment encoding like e-- do at all? So it's basically something that is learned. So you have two indices, like two basically embeddings that you can learn. You just additively add them, uh, you know, s-segment A for like tokens in, in the first part of the sentence and then segment B in the second part, and you just learn, uh, you know, you just learn it with gradient descent. So you do nothing with it. I think there was another question here. Yep. That's right. So the question is, uh, is every token in the same sentence going to have the same segment encoding? Yes. Yep. Okay, awesome. So I'm seeing I might need to speed up a tiny bit. Um, so yeah, the second thing that I want to highlight here is that we take the encoder parts of the transformer and, uh, you know, nothing new here, it's just like the same. We have self-attention followed by, uh, these like FFN. Uh, so this is where you're going to get your bidirectional, uh, nature of your encodings. Um, and then basically, we're assuming that training both MLM and NSP on it is going to help us, uh, like, learn helpful projection matrices in this encoder that can be suited for any classification task. Okay, great. And then I, you know, as promised, I talked about, you know, detailing more about like what this MLM task was going to do. So when you, uh, look at the input, you're going to, uh, basically have your input sentence and replace at random some tokens, uh, so it will be replaced either with the token mask, so it's in eighty percent of the time. In ten percent of the time, the token selected for this like MLM, uh, objective function are not going to be replaced at all. So we just say, you know, it's the same token, just predict the same token. And then ten percent of the time it's going to be changed to some random other world-- uh, word. So at the end of it, you have this subset of tokens you're going to perform your MLM task on, and then this is, you know, how it's composed. Uh, okay, great. And so basically the intuition here is that when you want to predict what a token is, you need to know about its context. So we're going to force the model to learn about what surrounds it left and right. So kind of this is the bidirectional, um, property of this architecture put into action. Okay, great. And now we're going to talk about like the next sentence prediction task, where, um, basically the process here is to put-- is to select two sentences from a given corpus and then present them, um, you know, side by side in order fifty percent of the time and in some random order the other, um, kind of part of the time. And the goal is for some classification head on top of the CLS token to determine whether A and B are consecutive. So that's all it does. And again, so the assumption is that it also helps, uh, learn some useful embeddings. Okay, great. And, uh, here I'm going to present the notation that is used in the paper. So the paper is something I recommend reading. It's a very nice read alongside Attention Is All You Need. I would say it's, uh, like a landmark paper. I think it has like one seventy K citations, something, uh, really impressive. Um, and, uh, basically what we called N in the original transformer paper is now called L. H, that was called D model, is the dimension of our embeddings and A is, uh, the number of attention heads with what's called a little H. So here I'm just showing these new notations for information, just to, like, give you the mapping. But, uh, of course, together with Afshin, we're going to stay consistent with the original notations we had, so just, like, for information. And, uh, one interesting note is that you will see that the BERT model, uh, when you look at some, um, repository of models such as Hugging Face, so usually it's, it's given in several versions. Uh, so you will see sometimes like cased, uncased. Uh, so this denotes what kind of pre-processing was done to the data, whether you only have lowercase, um, words in there or whether casing matters. So, uh, based on your, uh, task of interest, it might be, you know, one thing you want to choose from. Okay, great. And, uh, you know, give some orders of magnitude from the paper on, um, you know, what values were chosen for, uh, for, for each of these. So if I remember correctly, I think the original transformer had like twelve stacked encoders and decoders. Um, yeah, so I think it's-- like some, some numbers here were kind of taken from there. And, um, yeah. So yeah, or-order of magnitudes is, is a hundred million parameters. Any questions so far? Okay, great. Um,

Afshine Amidi 1:33:24 and then now we're going to talk about the fine-tuning stage, where the goal is to basically take whatever we have learned at pre-training stage and then freeze those weights. And instead of training again on the same weights, you have some classification, a linear layer that is put on top of either the CLS token or on top of tokens of interest, and you're going to learn about the linear embeddings there. So you're going to have some, uh, classification task. So you can either, you know, freeze all these like pre-trained weights and just, you know, train on these small weights, or you can just retrain the whole thing. I think there is like a multiple classification schemes, and some of it will be influenced by your willingness to retrain a lot of the network and how different the classification task is with respect to the original pre-training task. Um, okay, great. And then just to give some examples of what could be fine-tuning tasks for, you know, tasks of interest. You can have a sentiment extraction where you build a classification layer on top of the CLS token, and we give some other, um, you know, example. A question answering, where basically you are given some input and the goal of the model is to detect beginning and end spans of the response. So it's basically some objective function that is at the token level. Okay, great. So now I propose that we do some deep dive into one specific, uh, example, uh, you know, our favorite example. Uh, "This teddy bear is so cute." Let's just see how, um, BERT works in practice. So basically what you would do in an uncased setting is take the sentence, like pre-process it, uh, like j-just put everything lowercase. Then you apply the WordPiece algorithm on whatever tokenization mechanism that your, uh, like tokenizer has learnt. So for example here, you know, apparently it had all these merge rules. So this is kind of the tokens that appear in the vocabulary. And then as I promised, we add the CLS token at the beginning, and then we add the SEP token. And you also have these like, um, you know, pads tokens that, uh, are basically, um, used to fill out the se- the sequence until the end, 'cause when you train, you train by batch, and then batches are composed of matrices that have a, a fixed length. Uh, yeah. So let's just have like this deep dive a bit the same as, uh, like what we had in the transformer, uh, deep dive last time. So you have the embeddings that are learned. So it's like some huge, uh, lookup table, uh, between the index of each token and a learned representation. So you add to it the position embedding of the token and something that's new here, the segment embedding. So exactly as we said, it's an embedding that is added to each token. And the same embedding is added to the same tokens. So to all the tokens of the same, uh, segment A and then some other embedding B is added to all the tokens of the segment B. Okay, great. So now you have, similarly as a transformer, something that is position aware and context aware. Um, so not context aware just yet, but, uh, segment aware here. Uh, okay, great. And, uh, basically, um, it's-- it goes through this encoder architecture, and then at the end of it, for the case of sentiment extraction, we do not care about the embeddings that are, uh, corresponding to each token that is not the CLS token. Because what we care about is the output embedding of the CLS token where we plug A linear layer, uh, that will learn some, uh, cal- classification task. So any question here? So does the fact that we drop all the output embeddings apart from the CLS token make sense? Yep. Yep. Yeah. So the question is, uh, what does this, uh, you know, FFN... Is that right? What does this FFN correspond to? So basically you have a map between the dimension of the output embedding and your task of interest, which might be classification of, um, you know, you know, either-- is either positive or negative. So you have some hidden layer with some, with some length. So you have, like, typically two matrices to learn the projection from one to that hidden layer and then the hidden layer to the output, and then you learn these weights in order to do the classification task based on the embeddings that it has learned here. Yeah, great question. So that was the question I was waiting for. So the question is, why do we throw away all these other, uh, you know, output embeddings so we don't need them, need them here? So we're in a classification task. We have put as a convention to operate on top of some token, like the CLS token. And then the magic of this token is that all these self-attention mechanisms that are in the encoder has mixed the representation of each of the other tokens in that representation such that the output embeddings, they are context-aware. So basically it's a, it's an embedding that is geared towards classification, and this is what we say, you know, will be what we plug in to that, uh, you know, linear layer. And, um, so when I say, you know, we throw out all the other embeddings, it's actually something that we do for the case of classification. But if we do classification at the token level, then each of them may be used. So for example, when I said question answering, if you want to detect whether, um, you know, uh, like a given token is the beginning of a, of an answer, uh, beginning of end of an answer, you would typically have two FFNs that respectively predicts the start and end of the answer, and you would apply it on each of these embeddings. Any other questions? Yep. So the question is, "What is query key and value for CLS?" So it's going to be the same process as, as all the other tokens. So you learn here a representation for the embedding, for the token CLS. It gets projected to, uh, you know, query, it gets projected to key, it gets projected to value, it does all its attention computation. And at the end, you get this embedding that is basically, um, you know, that has intended to all the other ones. So it's like the answer is, it's the same as for the other tokens. So yeah. So just treat it as one token, you know. Could be any token and, and it's the same. Okay, awesome. So I see we have five more minutes and I'm going to, uh, quickly go through the end of it. And, um, you know, one thing that was great with BERT, and it was also, you know, great with ELMo, is that you have embeddings that are contextual. Um, and here, uh, you can see that it's very easy to learn any classification task that you want based on these learned embeddings. So this is a flexibility that was greatly appreciated here, and it's widely used in the industry. So anything that comes with sentiment detection or, like, other classification-related tasks, it's very common to use a BERT-like model for it, uh, nowadays. And, um, I'm going to talk a bit about its limitations now. So, um, as you see in the original paper, I think the context length was of size, uh, five hundred and twelve. So it was typically limited in this, um, kind of early paper. Afshin has mentioned of techniques as to how we can further grow this context size without making the complexity go completely, you know, basically, like, high. Uh, and then you have some, uh, approximation methods where you compute local attention and so on, and then the use of these tricks helps you grow the size of the context while staying within reasonable, uh, computation requirement bounds. And I'm going to talk about two other limitations that we're going to see what other models try to remedy. So there is one where, um, you know, the latency could be seen as high. You have, uh, still one ten million parameters for a BERT-base. It's quite a lot. You know, is there a way to make this, you know, smaller and faster? And then the second one is, uh, basically you have these two objective functions of MLM and NSP. Are those two truly helpful? Is there any way we can simplify the pre-training process? So we're going to see, uh, in a second how. Um,

Afshine Amidi 1:43:30 so first, regarding this second limitation that we had, this, uh, basically, um, this basically, uh, you know, sensitivity to cost. So, so who here has heard of distillation? Okay, great. Few people. So I just want to, like, flash this quote from, uh, Hinton, uh, Vinyals, and Jeff Dean, uh, which I think is very, uh, informative and, uh, kind of-- It's a good mindset to have to see that the distribution that is output by a given model is actually super helpful to know what it has learned. So it's-- the soft targets contain almost all the knowledge, so it's like some lecture from, from them that contain this, this quote, and I feel-- So, so basically they were at the origin of the concept of distillation later on, and you quickly see in practice that basically learning the distribution of a model is more helpful than directly learning the hard labels. So you have this concept of teacher and student models, where basically, um, you know, the goal in distillation is going to be to map this output distribution of a smaller model directly to your more complex model rather than the hard labels. And the, um, you know, the objective function that you use for this to minimize this is the KL divergence, which basically says, in a world that is described by the teacher, the teacher T, how bad is it to model with the student S? So it basically kind of tries to assess how close the student distribution is going to be to the world T. And, um, one interesting note that you will see here is that you find the cross-entropy loss if your YT distribution is just a hard label. So if you have just a one at one position and then zero everywhere else, you have minus log of, uh, you know, YS, which is very interesting. And then what the DistilBERT folks have done-- So by the way, it's a very clean paper. It's like four pages, but super impactful. So it says that, you know, if you reduce the number of layers by two, you have a lot of gains and almost the same performance, uh, which is kind of very remarkable. And they basically used distillation as a way to retain that performance. So this has been, like, the key alongside the, the, like, diminishing the number of layers. And last, I'm going to talk about RoBERTa, that, uh, studied the fact that removing the NSP, um, objective function led to no decrease in performance almost, so they just dropped it. And they added some tricks such as, um, you know, dynamically doing the masking. So for example, for a given piece of text, at each epoch, so every time you see the same kind of text, you would change the masking. And they also had some data strategy where they saw that the model was vastly undertrained, so they increased the diversity and the size of the data a lot, and they saw that, uh, kind of benchmark performance increased, uh, quite a bit on the same benchmarks. And that's it for today. Thank you, everyone. Have a great weekend.