Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training

1. 0:00 – 7:19

   Introduction

   1. SPSpeaker0:05

      Cool. Hello, everyone, and, uh, welcome to Lecture 4 of CME 295. So today is Friday, October the 17th, which means that the midterm is one week away. So before we start, I'm just going to go over some logistics to make sure, you know, we're all aligned on what to expect. So the midterm will take place next week, same time. Instead of, uh, an hour and fifty minutes, it will be an hour and thirty minutes. So it's 3:30 to 5:00 in this classroom. So it's like, uh, business as usual. Um, in terms of topics, the midterm will be about lectures one, two, and three, which we had, and this one, which is lecture four. So just to give you, uh, like, an overview of what you can expect in the midterm, there is going to be, uh, some multi-choice questions along with some free-form questions, but they're mainly going to be about the things that we've seen in class. So if you watch the recordings or attend the lectures and, you know, just go through the slides and know the important formulas, I think, uh, yeah, you'll be-- you'll be fine. So I know that you may have questions until, uh, next week, so that's why after this lecture with Shervin, we will be holding, uh, office hours. So feel free to, you know, come to us and ask us any questions. And, uh, of course, we'll be fully available between now and next week. So in case you have any questions, feel free to, uh, ping us on Ed. Um, and yeah, we'll make sure to respond. Um, cool. Uh, I also know that a number of you are auditing this class, so in case you're still interested to take the midterm for some reason, uh, maybe, uh, because you have an upcoming interview, uh, just, uh, tell us so that we can just expect the number of, uh, like copies to print. So we'll be printing this on Monday, so just let us know over the weekend in case you're interested. Cool. So that's for the midterm, and then the second piece of news is the final. So we said that we were working on the dates, so we finally finalized the dates, which did not change. So it's Wednesday, December the tenth. Okay, so a little bit late, 7:00 PM to 8:30 PM. Uh, so it's a slot that we have. Uh, the location is different from this one, so it's in this room. And the final will only cover the second part of the class, which is basically lectures five to nine. Any questions on this? Yeah. Oh, yeah, good question. So is it closed notes? Yes. Yeah. Yes. So question is, what is the format of the multiple choice? So you'll have, uh-- So we did not finish writing the exam, but it's going to be something like you have a question and you-- let's say you have, like, three, four possible answers, and then you just choose the, the, the one that's, that's correct. Something like this. And you'll also have some free form, uh, like you, you'll have to just, like, answer in your own words. Yeah. Yeah, uh, thanks. So question is, are we allowed to take anything? So it's closed book. So, like, n-n-not like-- Yeah, nothing. [chuckles] Just a pen. Yep. Uh, question is no calculator? You will not need calculator. [chuckles] But speaking of the cheat sheet, so I'm not sure if we mentioned. I think we did. So there is a cheat sheet for this one which we cannot bring to the exam, but you can use for your, uh, just for, uh, you know, just your studying. Uh, that's on the website, class website. So yes, recommend looking at it. Cool. So super clear for everyone? Very cool. Well, okay. As always, we'll be starting the class just recapping what we saw in the previous lecture. Um, so if you remember, we basically studied a new kind of architecture, which was called the Mixture of Experts, uh, which is such that if you have an input, what you want is to not necessarily activate all the parameters. And so you are in a setting where you have multiple experts, uh, and in the forward pass, you only activate some of them, so that's the sparse MoE. You also have the dense MoE, which basically weights the outputs as a function of, uh, the output of the gate. So we saw that this architecture was used in LLMs, and it was mainly used to be able to scale these LLMs without incurring, uh, an expensive cost at inference time because you don't want to activate all the parameters. The second thing that we saw was, uh, just defining what an LLM was, and in particular, how you could decide on what the next token prediction is. So we saw three methods. First one was we called, uh, greedy decoding, which was always taking the highest probable token. The second method we saw was beam search, where we kept track of the k most probable sequences. And then the third one was sampling. So we're not doing, um, most probable, we're not, uh, keeping track of the highest probable sequences. What we do is we sample the next token with respect to the distribution that we get as outputs, and then we saw there's this, uh, hyperparameter that's called temperature that allows you to tweak how, uh, spiky you want your distribution to like versus, uh, not. And we also saw some inference optimization techniques which are used in practice to avoid having, uh, like, a big cost at decoding time. So I'm not going to just mention everything, but I would say KV cache, for instance, is, uh, is an important method. So yeah, just, uh, recommend just knowing what it is, along with the other ones. And with that, we're going to start Lecture 4. And actually, I was really looking forward to today because Lecture 1, we saw what self-attention was, what a transformer was. Second lecture, we saw some of the tricks that, uh, people use today and some of the variations from the transformer. We introduced what an LLM was last lecture,

2. 7:19 – 13:26

   Pretraining

   1. SPSpeaker7:19

      and this lecture, we're finally going to see how these LLMs are trained. So today, we're going to focus on LLM training. And the first thing that I'm going to say is if you've been in the ML field for, let's say, more than a few years now, uh, you may have noticed that traditionally, if you had a task, what you would do is train a model specifically for that task. So let's suppose, like, ten years ago, let's suppose we had a task which was around detecting spam. You would train a model specifically to detect spam. So you would train on the training set, eval on the validation set, and then test on the test set. If you had another use case, let's suppose sentiment extraction, you would train a model specifically for that, and so on and so forth. But one could argue that these tasks, they're not completely disjoint. They're all involving just understanding the text. So one could argue we could find a way to somehow leverage the knowledge that we acquired during training for, let's say, one task and reuse that for another task. So this method has a name. It's been around for some time, and it's called transfer learning. So the goal of transfer learning is to not always start from scratch if you have a new task. It's to start with some pre-trained model, and we're going to see what pre-trained is, and then tune it for your task instead of starting from scratch. Well, it's basically the paradigm on which LLMs are trained. So the idea here is that all these tasks, they involve understanding language. So what we're going to do is have what we call a pre-training stage, which involves training your LLM on vast amounts of data to just understand what language, what code is, and then have a second stage of, quote-unquote, "tuning." And we're going to see a little bit what that tuning is. But in that second stage, we're going to take our pre-trained model and somehow find a way to tune the weights to adapt to a specific task. So as an example, uh, here, uh, we would pre-train a huge model, and then suppose for spam detection, we would somehow tune it for that. Uh, sentiment instruction, same. We would tune it for that, and so on. So this is just to take the example from before. And the idea here is in order to obtain these models, we're not going to start from scratch. Cool. So okay, so now we're going to see what pre-training is. So pre-training is by far the most expensive, both in terms of compute cost, you know, everything, part of the training. So what it does is taking a huge amount of data and training your LLM to just predict the next token. And here by data, what I mean is basically everything you can find. So it can be, uh, you know, text in English, can be text in other languages. It can be even code. Can be, uh, code in different languages. Can be basically the whole internet. We're going to see some of the datasets that people use for that, but you can think of this as just training your model to try to predict anything that's written. And as I mentioned, the objective here is to predict the next token. So if you remember, our LLM is a text-to-text model and most likely a decoder-only model in more than ninety percent of the cases. So what it does is it takes some input text, and it tries to always predict the next token in an iterative basis. So in terms of the datasets that are used, you will see the term Common Crawl a lot on papers. It's basically a dataset composed of anything you can find on the internet. So I think they have something like three billion pages per month. So if you go on their website, they have a hu-huge archive. So there's a bunch of other websites as well that you can find in there. So for instance, Wikipedia articles, any, like, social media as well, like Reddits. I know there are a lot of Reddit conversations in those, uh, in those datasets. You have a lot of codes, and of course, you have a bunch of places for that. You have GitHub. You have Stack Overflow, all these, like, forums that talk about codes. So all of this is meant for your model to just understand the structure of the language and codes. And in terms of size, so it's measured in terms of tokens, number of tokens, and one order of magnitude that I want you to remember is, is on the order of hundreds of billions or even trillions or even tens of trillions of tokens. So I'll give you an example. So GPT-3 was trained on three hundred billion tokens and, for instance, Llama 3, which was, I believe, published, uh, last year, was trained on fifteen trillion tokens. So these are huge datasets.

3. 13:26 – 16:34

   FLOPs, FLOPS

   1. SPSpeaker13:27

      So before we go further, I want to introduce two notations, and I think one of them I introduced, I introduced it last lecture. The reason why I want to talk to you about these notations is they're used everywhere to talk about how much compute s- uh, some model needs. So the first notation is FLOPS, which stands for floating operations, and what it is is it's a unit of compute. So the higher the FLOPS, the more operations are involved. 'Cause by definition, FLOPS is the number of operations that involve floating point numbers. So floating point numbers, you can think of them as just like numbers with decimal points. So in terms of order of magnitude, training an LLM is on the order of ten to the power of twenty-five FLOPS. And the way you obtain FLOPS, so usually it's like a complicated formula, but in your mind, you can think of it as something that is a function of the size of your data, so the number of tokens that you train it on, and the number of parameters of your model. So there's not, like, a universal formula because it also is a function of the architecture. So you can think of, for instance, MoE-based LLMs as requiring, let's say, less compute because only some parts are activated compared to, let's say, dense LLMs. But you can just think of it as it's a function of the number of tokens and parameters. It's like all of the product between the two, more or less. And then there's a second notation that I want to introduce, which is also FLOPS, but it's different. So here, FLOPS stands for floating point operations per second. So it's a measure of compute speeds. So it's basically how fast can your hardware execute these operations. And so you also have, like, some order of magnitudes here. Uh, but if you're into, uh, let's say, GPUs, you will see that in the description of GPUs, they always indicate FLOPS, and we will see that in a second. But I just want to call out that FLOPS here are usually all caps, although you may see some papers that use one for the other, which is confusing. So I just recommend, uh, just contextu- conceptualizing this notation with respect to the sentence that it is in, because sometimes people actually switch the two. But this is the common notation. So far so good?

4. 16:34 – 24:49

   Scaling laws, Chinchilla law

   1. SPSpeaker16:34

      Cool. Okay. So now we know that we have a pre-training step. We know it involves a lot of compute. We know it, it involves a lot of data. We know that our model is large. So what people did was trying to see how the performance evolves as a function of model size and training size. And there's this one paper called Scaling laws for neural language models that was published in 2020 that, uh, performed a bunch of experiments by varying these parameters. And what they found was the more compute you have, the better your model learns about predicting the next token. Same for dataset, uh, size. So the more-- the bigger your training set, the better it is, and the bigger your model, the better it is. So for some time, I think between 2019 and 2024, you were seeing models that were larger and larger, just people just building things that were bigger and bigger. B-because according to these experiments, um, the performance was just getting better. So something else that they noticed was bigger models tend to be more what they call sample efficient. So what that means is for an equal amount of tokens that is processed, you will have a better performance with a bigger model compared to a smaller one. But then you can wonder, you know, um, we don't have unlimited compute. You know, compute is expensive. Uh, it, uh, you know, it has a lot of drawbacks. So, uh, you have a fixed compute, and people also try to answer the question: given a certain amount of compute, how can you fix your training set size and your model size in a way that's more optimal? 'Cause, um, here, uh, you need to decide how big is your model. So what they did is they fixed a unit of compute, which is the color of these curves, and they tried training models of different sizes with different training set size. And what they saw was that there was always a sweet spot here which followed some kind of relationship. And in particular, this is a table that summarizes quote-unquote, "the optimal set of number of parameters and training set size," which is sometimes called the Chinchilla law. And what they realized was if you have an amount of training set size that's about twenty times the model size, then you're spending your compute in a quote-unquote, like, an optimal way. And in particular, GPT-3, for instance, I think it was like a hundred and seventy-five billion parameters, if I remember correctly, but it's only trained on three hundred billion tokens. So this one, for instance, is, according to this, really under-trained, quote-unquote. I think there was a question. Yeah. Mm. So yeah, the question is: Do they fit the neural architecture? So I think, um, by now everyone agrees that LLMs are transformer-based decoder-only models. So everyone uses the same model. Yeah. So you can assume that when I say LLM here, it basically means, uh, decoder-only transformer-based models. Yeah. Question is architecture change does not play a big role. So that's what they say, actually, in their paper. They say the thing that changes the most is the amount of tokens on which you train and the size of your model. Cool. Any other questions? Yeah. Oh, yeah. Good question. So the question is, uh, is there some kind of transfer learning between different versions of models? Um, so for a lot of these models, they're actually closed source, so they don't exactly reveal these things. But, uh, I guess it's an interesting question. Um, one that I cannot answer in a general way, so maybe, I think, is the best answer I can give you. Um, but in any case, uh, when you look at, um, some of these papers, they always state how much it costs to train this, and it's always in the order of, uh, you know, millions. It's, it's always an expensive step regardless. Cool. Um, just, uh, speaking of that, um, so pre-training has a lot of challenges. One of them is cost. So, uh, when I say millions of dollars, it's a minimum. I think it can even cost tens of millions of dollars or sometimes hundreds of millions of dollars. It takes a lot of time. And, um, people have been mindful of, uh, the impact on in the environment, so they've also been including the ecological costs. So the other, uh, challenge is that the pre-training step is on data that is up to the time at which you pre-train your model on. So what that means is that the knowledge that you acquire from training on this data set can only go up until the date at which you cut your data set. So this date is called the knowledge cutoff date. And so what that means is your base model, your pre-trained base model, does not know-- has no way to know by itself knowledge that occurred after this date. And speaking of that, a lot of papers, they've tried to edit knowledge, inject knowledge. It's always tricky because, uh, there's not a clean way to, um, you know, change the weights in a way that does not penalize some parts. So I guess what people want to do is inject knowledge but not regress in some other domains and this, this is a very hard problem. And of course, you know, these models, they try to predict the next token and, uh, there's this question of, uh, what if, uh, it just generates something that it has seen at training time? So what we call plagiarism. Uh, so there's always a risk. So these are all the challenges. I just want to illustrate when I said the knowledge cutoff date. So if you go on, let's say, the OpenAI website or Google websites to look at the model cards, you will always see... So I'm not sure if you can see from here, but, um, there is always a line on knowledge cutoff date which tells you on when the pre-training of this model was done. So here, for instance, GPT 5, uh, was released a few weeks ago, and here it says the knowledge cut-off date is, uh, September 30th. So you can guess that they've done their pre-training around that stage. Cool. Any questions on the first part? Everyone good? Perfect. So

5. 24:49 – 31:09

   Training optimizations overview

   1. SPSpeaker24:49

      in this first part, we've seen that pre-training was a crucial step of the LLM training process, and we've seen all these big numbers, and one could wonder, h- well, how can you train such a big model on such a big amount of data? Like, how do people do that? So this is what we're going to see here. So just, uh, what, uh, what I mentioned, so LLMs, you can think of them as decoder-only transformer-based models. So in order to train your model, you need that. You need a lot of data. But then if you look at your architecture, you see that a lot of the operations involve matrix multiplications. And I guess I have a question for you. What is the kind of hardware that loves matrix multiplications? GPUs? Yes. So you also need GPUs, actually, more than one. Yeah. You had a question? Oh, uh, question is GPUs for inference. So this one we're going to focus on training. But, um, requirements for GPUs, they differ a little bit between training and inference. But in this part, we're solely focused on training. And speaking of GPUs, uh, I guess, uh, it's not GPUs everywhere because, for instance, Google, they've developed their own hardware that's called TPUs. Uh, but any non-Google, um, uh, Google-based models, they've, you know, most likely been trained on GPUs. Cool. So in order to train your model, what do you do? So first of all, you have your LLM, which is now-- So this is this model, but, uh, now we're representing with a box just for simplicity. You initialize it. Uh, it's like, um, you know, a lot of parameters, so you can think of, uh, the scale as being somewhere around like billions to hundreds of billions of parameters. So huge model. And what are the steps involved to train a model? Well, what you're trying to do is to tune the weights so that the model can learn how to generate the next token. So you have one step called the forward pass, where you have a bunch of data that you're trying to pass through the network. And, um, while we do that, I just want to call out things that are important to note that we need to somehow save in memory. So when you do this forward pass, you have something that's called activations, which are basically the values at each layer that are needed in order to compute the loss. So the loss tells you how off you are compared to, uh, the label that you want to train this on. And so the amount of memory that you will use here is dependent on a lot of things. It's dependent on the model size, which impacts the number of activations. It's dependent on how big your batch of data is for training, and it's dependent on, uh, how, uh, large your context length is, because if you remember, uh, here we have O of N squared complexity because of this self-attention operation where N is the sequence length. So you have all these parameters that come into play. So once you do the forward pass, let's suppose you compute the loss, you know how off you are compared to your label. Now, the next step is to somehow tweak the weights in a way that minimizes the loss. So how do you do that? There's this-- another pass called backwards pass. So what this pass does is quantify the direction where the loss is going to be minimized. It's called a gradient. You take the gradient of the loss with respect to each parameter. Well, gradients, they also need to be saved somewhere in memory. And then you have finally the weight update, which is where you know where the direction at which your loss is going to be minimized, so you apply that update to your weights, and you typically use optimizers out there. Like, have you heard of Adam optimizer? Yeah. So Adam optimizer, just a fancy version that has some additional quantities, uh, which keep track of, uh-- which are basically a function of the gradient. So you have the first moment and the second moment, which is basically an average of the-- moving average of the gradient and the squared gradients and all these quantities. So the first moment, the second moment, you also need to somehow save them somewhere in memory. So it's a lot of things to save. Well, okay, breaking news, memory is not unlimited. Memory is limited. And so here what we have in front of us is the description of a GPU. Uh, I think so, yeah, H100, which is a very good GPU. And you will see that in that description, there is a line on GPU memory. So GPU memory is your amount of memory per GPU. It's, uh, eighty gig for this one. It's quite large. So it's in-- on the order of tens of gigabytes. So you need to store all these things in 80 gigabytes, which is, uh, not a lot. So w-what are we doing? Wh-what will you be doing? So I guess the idea is to leverage not one, but several GPUs in order to somehow distribute the load across CPUs. And in order to do that, you have several methods, which we will see in a second. So the

6. 31:09 – 35:51

   Data parallelism with ZeRO

   1. SPSpeaker31:09

      first set of methods is called data parallelism, also known as DP. So what this set of methods does is it distributes data across GPUs so that this forward pass and backward pass, they can all be done kind of independently. And so the idea here is to divide the batch of data across devices. And then, um, in order to do that, of course, you need to have a copy of the model per device, 'cause of course you need to compute the activations. You know, you need to compute all these things. Um, but when you do that, you're able to reduce the memory that is linked to the batch size. So that's called data parallelism. Yes. Uh, question is, how about, uh, gradient updates? Well, uh, it's a great question. So how-- what do you do when you have independent computations here and there? Well, the gradient is just the average of the gradients, uh, for this, for this thing. So you have some communication in between the GPUs that basically aggregates the gradient for, for the updates. So I have a question for you. Is this the answer to everything? Like if we just scale up like this for, I don't know, a lot of GPUs, is it, is it, is it great? Always great, or do we have, like, cons? Oh, yeah. Uh, great point. So yeah, you have to fit one model. So yeah, that's, that's a great point. So the second point that I will add is you have an additional cost, which is called communication cost, because you need to somehow communicate between your GPUs in order to aggregate some quantities. So your training is going to be slower. It's good. You can scale up the memory. Of course, you need to, uh, fit a model on a, on a device, and we will see what-- how we can do to do that. But you will be incurring those communi-communication costs, so it's not all, you know, great. So speaking of the memory and the fact that we want to, I guess, be able to at least store a model per, um, per, uh, device. So people have realized that there is actually a lot of duplication. And there has been a paper on wanting to deduplicate this duplicate information, and this method is called ZeRO, ZeRO Redundancy Optimization. And the idea is that in each-- on each GPU, you know, you store the same parameters, you store the same gradients, you store, store the same optimi-optimizer step-- states. So the idea here is how about we shard-- we partition those quantities across GPUs. So the first variation is around sharing the optimizatio-optim-optimizer states, so meaning we partition those states across the GPUs. So this reduces the memory by a lot. We can also partition the gradients, and we can also partition the parameters. So here we have no redundant information. Things are just partitioned. Well, the problem is you're going to have even more communication costs, but at least it allows, uh, for us to decrease the memory load on each GPU. So this is ZeRO. So there's ZeRO one, ZeRO two, ZeRO three, and I guess the variation that you will choose will be a function of how sensitive you are to, I guess, training time and how big is your model and whether this will be an actual problem, or are you just fine with just storing everything? So that's one set of methods. So this set of methods is, again, data parallelism. So it's basically you having independent sets of data that are handled by different GPUs. Well, you have another set of methods

7. 35:51 – 38:26

   Model parallelism

   1. SPSpeaker35:51

      that's called model parallelism. So model parallelism tries to parallelize the operation even within one batch. So there's a bunch of methods. I don't want to sound too like a catalog, so we'll not go through them all by one by one. But I will just call out a few that are worth noting. So if you remember last lecture, we talked about MoE-based LLMs and how sequences were being sent to different experts Well, there is a way to distribute that across GPUs via this expert parallelism techniques, which is, uh, having, let's say, one expert on a device, another one on another device. So that's one thing, uh, worth noting. Uh, another one I will say, so tensor parallelism is, uh, when you have big matrix multiplications to somehow cut that in a way that decreases the, uh, memory required for that. Okay, and maybe the last one I will say is pipeline parallelism. It's when you consider a forward pass as involving several layers. So you're going to say that one GPU is going to only be responsible for, let's say, layers one to three, and then another one, layer three-- uh, sorry, four to five. Four-- sorry, four, five, six, and so on and so forth. Um, so you also have that kind of parallelism. But anyways, there's a bunch of techniques, and the ones that I mentioned, they fall in the bucket of model parallelism. Make sense? No need to know the details on there, but I think just, like, knowing that there are several methods and just a rough idea, I think is a, is a good, good thing to have in mind. Cool. So what did we do? So we realized that during the training process, we had to save a lot of things in memory. So what we saw was techniques that reduce the burden of having memory per GPU, so we are trying to distribute that across GPUs. So we saw data parallelism and then the ZeRO method that has some extra optimizations, and we saw model parallelism as well.

8. 38:26 – 52:37

   Flash Attention

   1. SPSpeaker38:26

      So now we're going to see another technique that leverages the structure of the GPU. And you may have heard of this technique. It's called Flash Attention. It was actually, uh, developed, uh, here at Stanford, uh, in 2022. And in order for me to talk to you about this technique, I want to tell you more about what a GPU is com-composed of. So if you look under the hood, well, a GPU is very complicated, and I'm for sure not, uh-- I don't know everything either, but what I know is that we have two kinds of memories in a GPU. So you have one kind of memory that's big, but relatively slow, that's in the HBM. And then another kind of memory that is fast, but much, much smaller, which is on-chip, next to the-- where the compute happens. That's called the SRAM. So you have HBM and SRAM. HBM has something around, uh, you know, tens of gigabytes, so it's like the GPU memory that you saw in the description. SRAM is much smaller. It's like something around, like, several, like, you know, tens of megabytes, let's say. So it's much smaller, but then it is like ten times faster. So this one is, uh, a few terabytes per second, let's say, and the SRAM is, uh, tens of terabytes per second. So it's like a noticeable difference in speed. So what we want is to somehow leverage the strength of these kinds of memories in order to speed up the attention computation in a, in an exact way. So what do I mean by exact way? So what I mean is we're not making any appro-approximations to the computation. What we're doing is we're just leveraging the strength of these components and sending the computation in a, in a clever way. So if you remember, the self-attention computation is done with this very important, uh, formula. So it's softmax of queries and keys over some scaling factor times V. So this allows queries to interact with everyone else. Um, so in matrix form, you can think of queries as being, uh, uh, as having the number of rows equal to the sequence length, uh, and then, uh, columns to being the, the dimension of the query, and then, you know, same for key and value. So you have these big matrix multiplications. So if you do it-- if you do this computation the standards, the vanilla way, what you would do is store them in the big but slow memory component of the GPU. So you, you would store it in the HBM. So here is what you would do if you were to not do any optimization. So you would take those matrices from the big but slow HBM, perform the computation, and then write it back to the HBM. And then you would read that result again from the HBM, compute the softmax, and then write it back to the HBM. And then you would again load this plus the value matrix, multiply them, and then write them to the HBM. See, there's like a lot of, uh, read and write to the HBM, so it's a lot of, uh, data transfer. Which actually becomes the bottleneck. So a GPU is very, very fast, but then you spend a lot of time just loading your matrices from the memory. The reason why you do that is because of the softmax op- softmax operation. So do you remember what a softmax does? So it normalizes the quantities so that they sum to one, but it's row dependent, meaning that each row needs to sum up to one. So in a sense, you need that computation to happen first before you do your softmax. Like, uh, if you just, like, look at it like that, you're-- you would think, yeah, you, you need to do the whole thing first. Well, turns out that you don't need to do everything at once, and this is the core idea behind Flash Attention. So what Flash Attention does is it tries to minimize the amount of read and write from and to the HPM, and instead takes small blocks, and it's called tiling. The method is called tiling. It takes small blocks that it sends to the SRAM so that it gets computed from end to end before being sent back to the HPM. Does that make sense? So the idea is, let's send small matrices into the SRAM so that it does the whole, you know, full end-to-end computation and just send it back to the HPM. 'Cause we want to minimize the amount of read and write from the HPM. So here's how, how you do it. So you remember the softmax, uh, computation with the query and the key and then the value. Well, what you would do is to cut your matrices and then proceed step by step. But then there's a cool trick that I want to talk to you about, which is that you don't need to compute the whole matrix inside a softmax in order to achieve the whole softmax computation. 'Cause if you think about it, let's suppose you have a whole matrix and then you have like different, let's say, columns or like submatrices, S1 to Sn. Well, softmax of this huge matrix is equal to this matrix where the softmax is taken with respect to each of these submatrices up to some scaling factor. So this is the core trick, and, uh, if you want to be convinced of it, just look at the softmax formula. It's like exponential of something over some quantity, which is shared across the row. So this scaling factor will just, uh, fix this with respect to that. So with this in mind, what we will do is take each respective slices of these matrices, do the whole computation, and then populate the corresponding, uh, entry in the output matrix. So we will do that between, let's say, the first slice of the query and the first slice of the keys and the values, and then we will repeat for the other slice until the end. And then we will repeat for the other queries as well until the end. So what the paper explains is how this scaling factor is being computed. So this one is some formula that I did not put on the slide. So it's not necessary for you to memorize the formula, it's just the idea, and the idea is exactly this trick. So once you do that, you basically end up with only one read from the HPM, and these like tiled quantities are stored in the SRAM, and then they're read from the SRAM, which is very fast, and then computed, and then back to the SRAM. And then at the end, in order to accumulate the results, they're being sent back to the HPM. So just to make sure we're clear, so in green, it's basically when it's read from the SRAM, and then in blue is from the HPM. You have a question? Yeah. Yeah. The question is, do you take the whole row or a portion of it? So you can take a portion of it, but just for illustrative purposes, here we take the-- This is just for illustrative purposes. You can think of your mo-- your matrix as being completely, uh, you know, like a grid and then, uh, you just like multiply accordingly. But yeah, this is just for illustrative purposes. Yeah. Yeah. Yeah. So the question is, uh, are you computing alpha and all these quantities on the fly or do you have some estimation? So all this is exact, and they're computed in an iterative basis. So the way it works is when you populate the output, you will keep on having some extra quantity that will adjust for that Think of it as just some formula that works. Uh, so yeah, I highly recommend looking at the paper. They actually explicit that quite, quite a lot. So yeah, the paper is FlashAttention, Fast and Memory-Efficient Exact Attention with IO Awareness. All of the links are in the, the slides, but yeah, I highly recommend just looking at the exact formula in case you want to be convinced. Cool. But the idea makes sense overall? Any questions on this? Cool. Well, this was Flash Attention, but there's actually another idea from the paper. So this was only the first idea, which was around making the attention computation faster. The second idea is, given that the attention computation is faster, now let's, let, let's try to be smarter about the backwards pass. Because when you compute in the backwards pass, when you compute the gradient of the loss with respect to a parameter, the chain, chain rule will surface some activations that you need to have in memory. Well, here, given that computing these activations is very fast, one idea is to just not store activations from the forward pass, or at least not store everything, but instead, in the backwards pass, compute these activations again. So it's called recomputation. So when you do the forward pass, you compute an activation to be able to compute the loss. But then instead of saving it to reuse it during the gradient update, you will just discard it and then recompute the activation during the backwards pass with this very, uh, fast technique. And when you do that, it's actually quite remarkable. So you do more operations. Uh, so GigaFLOPS is, uh, you know, so just a derivative of FLOPs. So it's the number of, uh, operations you, you're doing. So with FlashAttention, you're doing more operations because you're actually recomputing things. But then, uh, so you also see fewer read and writes from the HBM. So here, for instance, it was forty point three in the standard way, and then four point-- So it's like almost a 10X reduction. But then you see that the runtime is also smaller. So this is very remarkable because usually when you recompute things, you are saving memory, but at the expense of, uh, runtime. You're just taking longer. But here you're not only taking less amount of time, you're also saving memory. So you're basically having everything as the best of all worlds. So this is what FlashAttention, uh, mentioned. So you will see that there are some, uh, I guess variants of FlashAttention. So FlashAttention-2, FlashAttention-3. And I would say that these are more adaptation of these methods to the current infra because each new GPU comes with new, um, pros and cons or strength and weaknesses. And, uh, I guess there's always a way to make these optimizations better. Uh, but yeah, I would say FlashAttention is quite a common trick, and I think it's a very good thing to know. So does that make sense? Cool. I'll take that as a yes. Okay, cool. Okay, so last thing. No, we have a few things, um, uh, that I want to talk to you about. So, you know, you have

9. 52:37 – 56:00

   Quantization

   1. SPSpeaker52:38

      your LLM with a bunch of weights. These weights are all floating points. So you can think of them as, you know, just being numbers with a bunch of, uh, numbers after the decimal. One natural question you can ask yourself is, do you really need to know that much like precision after the decimal point to be able to do a good job? I guess, put another way, can you just like cut your precision in some way to save on memory, but then keep the same performance? So this is the idea behind quantization. So quantization is the process of converting the precision of a number from, let's say, one, uh, setting to another. To-- In order to better understand that, I think it's important to know how floating points-- Sorry, floating point, uh, numbers, how they are encoded. So in practice, they're just a bunch of bits. And here you have some bits that are responsible for, let's say, the exponent, some that are responsible for the mantissa, which is basically how granular, granular your number is, and then one that is about the sign. So you have a bunch of representations of floats. So the most common ones are in this table. So you have, uh, so single point precision, half precision, um, floating point sixty-four, floating point, uh, brain float sixteen, that are each having different granularities for the three dimensions. So if we take the first two rows, which are the two most common, uh, representations that you will see out there, floating point sixteen is only represented on sixteen bits compared to thirty-two. So it takes basically half the memory if you, if you want to think about it this way. Uh, but then it's less precise, meaning it's less granular. So I guess one idea that we could have is to somehow decrease the granularity of these weights and these numbers, hoping that it will not impact performance too much. So-- And a thing that I want to, uh, mention is, you know, back to that, uh, GPU description. You see that there is also a bunch of information around the compute speed. So I'm not sure if you can see very well on the-- Can you see very well on the slide? So I, I'll read it. Uh, the compute speed as a function of which kind of representation you're using. So here, if you're using FP64, which is this super granular way of representing numbers, you only have thirty-four teraFLOPS of compute speed. But then if you're using, uh, let's say FP32, which is, uh, half of it, you can, uh, kind of double your compute speed, and so on. You have all these, like, other, um, numbers as well. So the idea is you can save on memory, you can also go faster. So with this

10. 56:00 – 1:02:31

    Mixed precision training

    1. SPSpeaker56:00

       in mind, I just want to touch on one last technique before giving it to Shervine, which is mixed precision training. So the idea behind mixed precision training is to leverage different granularities of float representations such that it will not hurt the performance too much, but allow you to save on memory and do things faster. So the idea behind mixed precision training is you have your models or you have your model, you keep your weights in high precision, so FP32, and all the operations that you do in the forward and backwards pass, you're going to do it in a lower precision. So in this case, FP16. But then the weight update will be done still in FP32. So the authors of the paper, when they, when they did that, basically what they realized was the performance was not degraded too much. But then you had a lot of savings on memory, and then it was running faster. So now you may wonder, well, why are you keeping the, the weights at a high precision, but not the activations and so on? Um, so I can offer you just a bit of intuition. So whenever you perform a forward pass, you're performing that on a set of data. But the set of data can be noisy in itself. So maybe all the decimal points after-- So all the numbers after the decimal point may not be as useful. So we can think of this update as being more like, you know, in which direction should the weights go to go to a more optimal state. So it doesn't require them to be super precise. But then it's much more important to keep your weights, so the weights of the model in high precision, in order to not accumulate errors due to quantization. That's one way of just thinking about it. But the long story short is if you reduce the granularity of some numbers, then you will have a lot, lot of benefits and not a lot of disadvantages in terms of the performance. Cool. Yeah. That's a great question. So the question is: Do you apply this, uh, technique to all weights of all layers, uh, or just to some of them? So there's been a lot of papers on that. Uh, and the answer is there are some variations to it. Um, and so the answer is not necessarily. So I do have some pointers, happy to share, share them with you, but, um, some parts may be more important than others. So this strategy is l- a little bit, uh, like a high-level idea, but there are always some variations, uh, from setup to setup. Yeah. So is your question that people rely on the, like, uh, relationship between compute and-- sorry, not compute, parameters and token from the Chinchilla paper, uh, but it may be different from their setup, which may introduce something? Is, is it basically your question? Hmm. Mm-hmm. So the question is, uh, is there also an optimal, uh, precision to use? So I would say people use different things. There's not, like, a set way where everyone does, uh, something. But, um, I would say that these, uh, scaling-- like this Chinchilla law is actually something that some authors, they try to reproduce for their own model. So, uh, actually, uh, for instance, the Llama 3 paper, there's a whole part around trying to, uh, have some relationship between, uh, you know, if given a fixed amount of compute, what is the optimal, uh, you know, number of tokens and then, uh, number of parameters. That is, uh, unique to their setup. So what people do is they do this thing themselves on a small amount of training, uh, training set and on smaller models so that it doesn't cost them too much. They try to see what the relationship is for their setup, and then they extrapolate. And I believe that's how Llama 3-- So I think there's a model that-- So Llama 3, four hundred and five billion parameters. I think they had the whole section just justifying that they came up to this number with some experiments that they ran. So to your question, um, I think it highly depends on the kind of model that someone is training, and they'll probably have this, uh, analysis, I guess, with respect to their own setup. Does that answer your question? Cool, cool. Okay, good. Yeah. Yeah. Yeah, yeah. The question is, uh, is there something you can do about the range? So there are several types of quantization. We'll not go into details here, but, uh, I can give you like, uh, zero-point quantization and ABT-SMAC quan-quantization, which are different techniques that play with these ranges. Uh, there's also like a quantization technique that Shervine will cover that also talks about that. Um, so yeah, stay tuned for this. But, um, I think zero-point quantization and ABT-SMACs are, are ones that maybe you can look into for this. Cool. So with that, I'm going to give it to Shervine. Thank you, Afshin. So now we have seen, uh,

11. 1:02:31 – 1:09:21

    Supervised finetuning

    1. SPSpeaker1:02:31

       how, uh, pre-training worked. We're going to see together, uh, the next stages of training. Um, so as we saw with Afshin, pre-training was a way to, um, build the intuition to the model about how language was constructed and about, uh, the main characteristics of what is contained in the training data that is general enough. So this is why we had corpuses that spans huge corpora of texts that are typically on-- of the size of what you find on the internet, so typically very large and raw and conveying, um, aspects of language. So now you might want to ask yourself, how can we make such knowledge actually helpful? So this is something that we're going to discuss just in a second. But before we discuss about what could motivate further training, I want to come back to some very simple example that will, um, surface what could be such a need. So let's take our fav-favorite example about our teddy bear, and let's say we have a very, um, practical use case. So we have thoroughly loved our teddy bear, uh, but now it has become a bit dirty, so we need to wash it. And you might want to ask your favorite LLM, um, if you can put it in the washer. So with the description that Afshin mentioned of LLM pre-training, I wanted to ask you all what was your opinion about what could the output to this be? So any guesses? Yep. Yep. Yeah. Excellent. Yeah. So the, the answer, the guess was maybe another question. So I think that is a great guess that could definitely be it. Uh, so here in the example, we put some other sentence that could be likely, but the, the gist is the same. It has been trained on next token prediction, not being an assistant or someone that is helpful to you. So this is why, uh, it will try to mimic, uh, what could be the pattern here and what could be potential likely next words. So for example, the example we put here is something that relates to, uh, like how the teddy bear is composed of, um, and, uh, because it's in the same domain as, uh, kind of teddy bear. And then when you talk about washer, then maybe, uh, it has been trained on data that includes, uh, materials of teddy bears. So that could be it. Um, and I have a question with you all. Um, are you happy with it? Nope. Yeah, exactly. So this is why we're going to see what we can do with the model to make it helpful to you. And this stage is called fine-tuning. So we're going to see, um, that it enables us to, uh, like in the case of general-- like the general case of LLMs to make the LLM, um, helpful assistance. But also, if you have a specific use case, you can also use this technique to tune the general representation of the language towards your task of interest. Uh, but we're going to focus first on, uh, what is done, uh, for LLMs in general. Uh, so first I want to define some terms. So model fine-tuning is commonly known as SFT. SFT is, uh, short for supervised fine-tuning. So the supervised part suggests that we need some labels, and it's actually the case. So it's pairs of inputs and outputs that we provide the model to be trained on, and this is why, uh, it's called supervised. And then fine-tuning is, uh, the term that refers to, uh, refining the weights that have been already trained. So you, you start from the pre-trained weights and you further train the model on additional data to get a fine-tuned model. Um, and, um, and then one interesting note that I will tell you all is that the objective function, even though it's the same, will differ a bit from the pre-training task. So in the pre-training task, you start with the BOS token, beginning of sentence, and you fit, uh, all your corpora of data trying to predict the next token. But here, since you have, uh, this supervised fine-tuning setup where you want the model to do something useful in cases of interest. So let's say when I ask the model about washing my teddy bear. So washing my teddy bear is actually an input that I give to the model. It's, it's fixed. I don't want the model to parrot what I say. I want the model to be a helpful assistant to what I, uh, condition it on. So in this setup, the input is not a location where you would predict the next token. You would not do teacher forcing on it, but rather start from this input and then predict the next token onwards. And then you're going to tune what the optimal, um, like, distribution of next token can be based on this input. So any questions on the overall idea of SFT before we dive in, uh, a bit deeper? Yep. So the question is we don't put a l- a loss over the input. That's right. So you start from the input, and you start predicting the next token, and then the last calculation starts from there. Yep. Everything good? Okay. So, uh, what I wa- uh, what I was just telling you, um, is that SFT can be done to tune your own, uh, task of interest. But also it's something that is being done at, uh, the-- one of the main stages of training LLMs as you use it every day. And then this transition from, uh, the model being a good representation of language to a useful assistance is a subcategory of SFT called instruction tuning. So instruction tuning comes from the fact that we want the model to answer instructions. So

12. 1:09:21 – 1:37:53

    Instruction tuning

    1. SPSpeaker1:09:21

       now I'm going to, like, show with the graph what I was just mentioning regarding predicting the next word. Uh, okay, so actually just in a bit first, we're going to look at the data. So, um, ash-- with Afshine, we saw that the composition of the data for pre-training was basically the whole internet, where you had, uh, sources that were, uh, carrying knowledge such as Wikipedia, um, and in general, like, masses of text of English that, um, that tells, that tells the model about how English and other languages work, as well as coding. And here it's slightly different. For instruction tuning, we're going to look at data that presents to the model how it can most helpfully respond to instructions. So you can divide this in several categories. So here we present some categories that could be helpful to users of LLMs. For example, uh, story writing. You know, you're interested in writing a story. You give it a set of instructions, and then, uh, you have a ground truth that is attached to it. Uh, other, um, examples include, uh, poem creation, list generation, explanation and, and many more that are part of the data mixture of SFT. So basically, you, uh, run all these supervised fine-tuning, um, training with the input that is given, and then you train, uh, the model on predicting the output. And I'm going now to show, uh, what it looks like. So, uh, let's say your instruction is here. So you ask your model to do something. Um, so it could be formulated in a different way, but just to keep it short, I just said, "Do X." And then the yellow part is where you start fitting your objective function on. Um, does this make sense? Great. So as I was, uh, mentioning, the data mixture in-- includes, um, instruction, um, geared data. So for example, you had all of these, uh, kinds of tasks that, uh, users might ask for at inference time. So it's what I put under the category Assistant Dialogues. Um, and these days, uh, since you have all these, uh, large models that already exist out there, and let's say you want to generate a new one, you don't necessarily have to start from scratch. So all of these that I mentioned were originally human-written. So you had all these instructions that people gathered, and you had, uh, expert linguists that had a set of instructions on how to write the best answer that was fluent, helpful, and, um, everything that was geared towards maximizing your happiness as a user. But these days, you can use these already trained LLMs to generate such data. So typically, uh, when you have a large model that, that is very performant, you can, uh, feed it such instructions, sample some generated outputs, and then have a human or some other LLM review the quality So it's something that speeds up a bit, uh, the curation of such datasets. So I just wanted to emphasize on the fact that it's not only human-generated these days, but you can have some assistance. And besides all of the categories that you care about, so typically, uh, math or, um, you know, how you write a proof, how things follow each other or codes, uh, like high-quality code bases. You also have other aspects that, um, that are very important which when you release a product like this to the wider population that come under the umbrella of safety. So, uh, you want your model to be helpful but also harmless. So you also have, uh, in the data mixture oftentimes a subset that enforces the model's behavior not to, uh, repeat some of the harmful content it might have seen on its pre-training corpus. So it will include techniques that might, um, reject some, um, some, some user prompts. So you might have tested, uh, you know, on your own the limits of an LLM today if you submit queries that might lead to something that, that is considered bad. It would just say, "Okay, sorry, I cannot answer this." And this "Sorry, I cannot answer this" is, you know, could be done via regex, but very rarely because it's not scalable. And then people actually embed this property as part of the model. So, um, you have techniques that, uh, like reject user queries that are seen as harmful or, uh, there is some other, um, phenomenon such as hedging that nuance the, um, the output of the model in order to not make blanket statement as well. So you might have datasets that includes such, um, flavors. And you can have many more, um, depending on the task of interest that you're dealing with. But what I'm listing here is what usually the LLMs out there that aim at being general assistants gather. They tend to be like general tasks, um, that they are being trained on. Any questions on the data? Yep. Yep. So the question is, the story writing example doesn't precise what kind of poetry, uh, the story should be about, so it might be ambiguous. How can it generalize? So that, that is a great point. So we rely on the model's, uh, knowledge that it has accumulated at pre-training time to be able to generalize beyond the examples that it has been fed. So let's say now you have, um, a story about, uh, you know, poetry, and from pre-training it knows about all kinds of poetry that exist out there. So you could imagine after feeding such an example at supervised fine-tuning time that you could give more details to this prompt and then the model has the ability to generalize the generation of the story with respect to those other attributes. So you can have-- I think, like, the example that you mentioned is a great, um, example of, uh, the, the, the magical ability of LLMs to adapt to natural language. Uh, and all of that has to do with the kind of distribution it has seen in the past and what we teach it. So, uh, I think, like, the concept itself of writing stories is the key learning to the model that it can gen, uh, kind of generalize. Does, does that make sense? Yeah. Okay. Great. I think there was another question here. Yep. So the question is about the term of alignments. Um, is this post-training, uh, what is called as being aligned? So you're ahead of me. Uh, in a few slides we're going to talk about it. Yeah. Okay. Great. Great questions. Uh, any other question? Awesome. So, uh, and then just like Afshin did it for the pre-training part, I'm going to give some orders of magnitude. So the same GPT-3 and Llama-3 papers that Afshin referred some numbers from don't actually give their, uh, statistics in terms of token numbers for the, for the instruction tuning side, but actually in number of examples. So you see you, you have about thirteen K used by GPT-3 and, uh, ten million for Llama 3. And, uh... Okay, so let's do some quick estimation. Let's say each example is about a thousand token. When you multiply that with the number of, of examples, you see that the size of the datasets used for SFT is several orders of magnitude, uh, lower than the one used for pre-training. So the mental model that you need to have here is that pre-training is a lot of data. You want to learn about general, um, characteristics about language. And an SFT is more exactly what's, uh-- So someone was mentioning just now regarding alignments, was aligning the goal of the model to being, uh, suited for your tasks. So typically very high quality datasets and much more, um, like concise and precise in terms of number. So like, uh, lower-- like less order of magnitude. Okay, great. So now let's try this exercise again. What do you think could be the answer to our instruction tunes now model to the same question? Anyone wants to take a guess? [laughs] You want to try again? Yeah. So the, the, the answer was yes, you can put in the teddy bear in the washer, except that you shouldn't put your teddy bear in the washer. You should hand wash it. Um, so it will give you the answer that is helpful to the user and indeed, uh, this time respond to the query. So which is, which is pretty nice. So now we talked about all the things that are great about supervised fine-tuning. Now that I'm going to detail a bit more about what could make it hard or challenging and then motivate the optimization part that we're going to see afterwards. So first we talked about it a, a bit. Uh, when we have these datasets, da-- these data mixtures for SFT training, we need high quality data and when you say high quality data means oftentimes human involved in the loop. You need to make sure it abides with all the rules and all the characteristics that the user will care about. So typically it's highly, um, involved. So originally these first models, they were almost all human, but now you might have some, um, some mixture of human and, uh, generation. Um, and but one good thing about these things is that the datasets it's-- they are trained on are reused. So, so it's a work that you do once and you can complement with respect to time, uh, but it's still very expensive in time and resources. So the second point I want to mention comes back to a point that someone just made regarding, uh, the distribution of your SFT dataset and how it would align with actual inference distribution. So in the case of the story regarding a specific kind of poetry, the distribution of prompts that precise the kind of poetry seems close enough for generation-- generalization to be made. But you could think about examples that widely differ. So let's say you ask for a story that is in some movie plots and it differs from the stories that you might, uh, like, see in, in textbooks. So this could be slightly out of distribution and it could have trouble generalizing to that. So, um, prompt distribution is very important and aligning that prompt distribution with respect to the target task is of interest. Yep. Yeah, great point. So the question is about, let's say you have SFT tuned your model and now you put a training input back to the model. Will you get the same story? So it has to do with the phenomenon that Afshin was, uh, mentioning regarding, um, gen-- uh, memorization. So in practice you will not see the same story most likely because the sampling that you're doing is, uh, with a non-zero temperature. It will, uh, maybe have the same flavor but not word, word for word. Um, then the flavor of that story, if you have the exact same prompts, might be the same. You know, it depends on what it has seen as at pre-training time. Um, but yeah. So the answer is that if I had to give a guess on that precise example, it might be the same flavor but definitely not the same wording. Um, yeah. So and it might be a different story, um, if the sampling goes through some other region of the space. You know, stories are highly creative and the pre-training, uh, corpora of, of like data mixture has, uh, like all kinds of stories in there. So like the specific, uh, example of story is likely to generate different, um, different like outcomes. Yep. So the question is, uh, will how much the model wanders around depend on the temperature parameter? And yes. So typically when you have a higher, higher temperature you also-- So it's also called more creative and it's for that reason. It might also, uh, generate things that are less likely, uh, in, on its output distribution with respect to what it has learned and this is exactly, you know, a case of something that we learn. So yeah, the answer is yes. So but I-- So the question is, is there any way to tune that? So when you tune the temperature you, you exactly do that. Yeah. Or were you thinking about something else? Yep. So the question is, you know, is there something else you could do to have like, um, variations in the response? So, uh, for that I have a simple but hard solution. You would need more data in that category that, uh, will ground the model into seeing what kind of distribution we target with such queries and I think that will be one way for the model to generalize more. Yeah. So I think like it will be working on the data part Was there another question? All good. So yeah. So this is a very relevant question because it touches on the topic of generalization. So yeah. So data mixture matters a lot and having points that are, um, sparse enough in the distribution space to give the gist to the model of what it needs to learn rather than repeating the same story again and again will have a lot to do with generalization powers. Um, and then this is something we are going to see in a second. How do you evaluate such models? Um, the feeling that you get as a user on how helpful a model is tends to be subjective. So how can you put some number on it is going to be a key topic of interest. And then also, um, very soon we're going to see how to make the computations not that expensive. So Afshin talked about training optimization techniques, but now, uh, when we're at the fine-tuning stage, maybe we can go one step further and, and, and see what simplification assumption we could make. Okay, awesome. Now let's dive into one of these challenges, which was the evaluation part. So people have decomposed what they care about in categories. So I'm listing here some dimensions that are evaluated against and that give some quantitative number. So you have a general language where this benchmark that is popular, um, is generally one score that users-- that people report. It's MMLU, um, which is Massive Multitask Language Understanding. So it, it has, I think, uh, fifty or so tasks in there where the model is being evaluated against and you have some score that you can compare with. You have, uh, benchmarks on reasoning, uh, on, uh, math reasoning as well as code generation. So here you have all sorts of acronyms and there is even way more. So, uh, benchmark, um, basically the, the setting up benchmark has been, uh, one area of research. So you always see more and more benchmark coming in because people tend to optimize for those that exist and then, uh, there are some gaps that might be filled by further ones. And the GSM8K stands for, um, grad school-- So yeah, grad school math. 8K stands for the number of examples. Uh, yeah, I think so. Or may-maybe G stands for something else, but it's, uh, like high school topics basically. And one very interesting, uh, pattern that people have seen as these models came out and as they were evaluated against these benchmarks is that sometimes for a same kind of model, you see all of a sudden a spike in, uh, numbers, uh, with respect to some of these benchmarks without a clear explanation behind it. And there is this paper that I highly recommend taking a look which, um, explores the phenomenon of training on the test task. Not the test set, the test task. So when you have benchmarks regarding, uh, math reasoning, let's say, it matters a lot if the model has, has been trained on data that relates to that kind of reasoning versus not. And this is why when you type in those kinds of benchmarks, you will see that there are sets of so-called auxiliary training that can be used for a model to be trained on the same domain. And, um, and in that way it enables, um, like the fact of comparing models with respect to that specific, uh, capability. So the-- what the paper, uh, tries to convey here is that if you want to compare models between each other, you need to compare, um, the training mixture it has been trained on and then ensure parity in terms of training on the test task. So you need to make sure that, for example, the two have been trained on the test task or not. And if it's one and the other not, then it might not be a good thing to compare, uh, them. It doesn't give you like the intrinsic value of the model. So yeah, that's an interesting, um, phenomenon here. Uh, any questions? Okay, great. So we can move on. Uh, so one thing that I was mentioning here is that it's very hard to get a sense of how good a model is even with respect to benchmark scores, because oftentimes what happens is that as they come out, people design, um, data on the training side that resembles more what we try to solve on the benchmark side. So sometimes you end up with models that score great everywhere, but you as a user don't necessarily see, uh, added value to it. Uh, and, uh, you know, it's not the fault of models, it's not the fault of the benchmarks. It's just that it's very hard to give a number that conveys the value to you of the model. So this is why people have come up with other, uh, techniques to put a number on, uh, model evaluation. And you might have heard of Chatbot Arena. Have any folks heard of it here? Yeah. So it's a website where models can be submitted and, um, users come in and they ask their questions and they're presented responses from two models, and they're being asked to judge which one is better. And then with some pairwise computations done on the website side, they come up with some ranking in the end that ranks model with respect to, quote-unquote, "user preference". So it's a sort of a, a number that's puts on a-- like, that is being put on the vibe seen by the user and is it, um, all perfect? Actually, uh, no. So it suffers from several, uh, issues that are, uh, like another set of issues that are hard to deal with. So, uh, like among them, when you have a new model that come in, you have some noise at the beginning with respect to which, which other model it's being compared against. And these first few steps actually, um, influence quite a bit the actual ranking, which makes it, um, you know, kind of a brittle, uh, property. And, uh, there are-- there is a paper that actually shows that it's easily possible to rig such a leaderboard. So I'm not sure if there is any evidence that it has been done in the past, but, uh, you know, y-you can use any model. If you ask the question, "Who are you?" It's just going to say who, who it is. If you ask ChatGPT, "Who are you?" It's going to say, "Hey, I'm ChatGPT. I'm a helpful assistant." And this paper, uh, you know, observed on this very simple property that it was able to detect what model was being evaluated against. And let's say you have an adversarial player, it could rig the ranking just by selecting the right model, so it's not, um, foolproof. And then on other aspects, so, um, some of the benchmarks that evaluate these models, they're actually curated by experts that know very well about the target distribution of what a given prompt should give. So it can-- uh, so it has a set of guidelines that says clearly, "Okay, this is factual. This is non-factual." So they're able to determine what is good versus bad. And you as a user, you might not know about it 'cause let's just take the example of the teddy bear that you wanted to put in the washer. If I don't know that I need to hand wash my teddy bear, and if I have a detailed response regarding, uh, you need to wash it with the machine wash cold, and I could, as a user, find these pieces of advice helpful because they are actionable to me. But actually, are they factual? That is a whole other set of issues that you as a user wouldn't be able to tell on a lot of queries. And, uh, another challenge that we have here is a user preference. So who here likes it when there are emojis in LLM responses? Yes? No? Yeah. Yeah, personally, I do. And I think there are, like, uh, strong opinions, and some people don't like it at all, so they will down rank such, such responses. But actually, it's something that the user should be able to, to tell and choose. And then the distribution of people who choose the best model with respect to the distribution of the wider population that is going to use this model-- these models is going to be different. Um, you know, and oftentimes, and I think the emoji case is a good example, uh, 'cause I think generally emojis are popular in the wider population, but I think domain experts might not like it as much. So I think that the-- that mismatch is one that you would see here. And then the last point I will mention here is the safety side. So you as a user, you don't really like it when a model rejects your prompt. When you ask about something and it says, "Okay. Hey, sorry, I cannot answer it." So there will be a bias towards responses that actually responds to your query rather than, uh, respect some safety principle that might be eventually the intended product decision. So there is also this kind of bias that pops up here. And as I was mentioning, evaluation is a hard problem. So you have all these angles, uh-- all these angles that you can ex-explore, and it's not one number that's going to tell, tell you what you care about. It's the combination of all of these. And in the end, it's, um, it's tailored to what you actually need. So you would need to see the strength and weaknesses of a given model and to determine, like, which one corresponds to your use case. Okay. So I'm going to come back to one question about alignment here. So we're going to see at the next lecture a further step into aligning the model to do what you want to do in a step called preference tuning. And then the combination of fine-tuning and preference tuning, um, which is-- which comes after pre-training, is what we call alignment of the model. So, like, these two steps are called alignment. And I want to call out one other thing. Uh, there is one step we didn't mention here that is called mid-training that has been emerging very recently, which consists in a step just after pre-training in aligning the kind of data that the model is being trained on to the tasks that you really care about. So it's the same pre-training objective, but aligning the kind of task and the kind of datasets, uh, to something that you care about. And, uh, yeah, so it's an emerging trends. I have not mentioned it, but just so that you know, mid-training is something that you would have between pre-training and fine-tuning. All good? Okay, great. So now we're going to tackle one aspect of the challenges that we had mentioned regarding fine-tuning, which is computational expense, like, uh, the fact of being computationally expensive. And

13. 1:37:53 – 1:45:16

    Parameter-efficient finetuning with LoRA

    1. SPSpeaker1:37:53

       we're going to look at it with one, uh, well-known technique called LoRA, which is a technique to fine-tune your, uh, weights at the fine-tuning stage in an efficient manner. So it's widely used, it's, um, yeah, and, and it saves a lot of compute. So, uh, when you look at your weight matrices, instead of directly fine-tuning the whole weight matrix, the LoRA technique decomposes the fine-tuning between the weights of the pre-trained model and additional weights that it decomposes into a low-rank multiplication. So, uh, in this formulation, the pre-trained weights that you have is frozen, so this W zero is frozen. And then B and A are the matrices that you need to tune. So B and A are typically, um-- So they have dimensions that will match the, like, number of rows and number of columns, respectively, for, for the number of rows of B and number of columns of A. But then the dimension of the columns of B and then rows of A is R, which is the rank of these matrices, which is typically taken very small. So the dimensions of W is typically hundreds or thousands, and the dimension of R is typically, um, you know, up to ten or O of ten. So as you can imagine, it results in much less weights to train. Um, and then I will mention just other techniques to improve efficiency, uh, and then ease the fine-tuning. Uh, some, uh, methods called prefix tuning and adapters. So, um, they're explained in the, in the class textbook, but we're not going to dive in, uh, deep into them, um, because they're, like, uh, less commonly used, but just so that you know. Okay, so let's walk through in detail how, uh, does LoRA work. So you-- when you want to fine-tune your model, so you have all these ways for which you have already learned a distribution at pre-training time. So one, um, naive way of operating fine-tuning would be to directly iterate on these weights. But what we're doing here, as we said, is to decompose this into the weights that we have already pre-trained on and this product of matrices. And what LoRA says is that you can do a forward pass on both these terms and then adds these, uh, quantities in the end. And, uh, yeah, A and B is, um, like, is going to be a characterization of your task of interest. So let's say you have a model that, uh-- So Afshin was mentioning the kinds of tasks that you can maybe specialize your model on, for example, spam detection. So you would take your pre-trained model, uh, instantiate this B and A in weights, like in the weights of your model, fine-tune on them, and then what you, you will have is that B and A are going to be specific to this task of spam detection. Similarly for sentiment extraction and so on. So it's a very nice property that you can start from a base model, further tune your weights, and then have your A and Bs that are task-specific. Um, so I want to comment now on where were these, um, you know, matrices being learned. So in the original paper, uh, of LoRA, it's mentioned training only on the, uh, attention matrices. But later on, uh, people realized that it might not be the place where, um, it has the most positive impacts on, uh, on performance. And then there is a blog that, uh, came out a few weeks ago that, uh, studies this into detail. Uh, and then, uh, they realized that the feed-forward blocks are actually those where putting LoRA is most beneficial. So today, typically, both these components, uh, carry LoRA matrices, but the bulk of the, of the performance improvements is actually contained in the feed-forward, uh, block. And then I want to mention two interesting properties about, uh, LoRA. Uh, so when you fine-tune with LoRA weights, uh, you want to use a higher learning rate, so typically, uh, ten times more is the, is the guidance. And then, uh, one interesting fact is that it doesn't perform as well when you train it with larger batch size. So I don't have a good theoretical explanation to give you for each of them. It's more empirical observations that have been emitted. But to give the main lines behind the mindset shared from people who study this phenomenon, um, the first one might be guided by the rank of the LoRA matrices that might be small and As a result, like, given the regions of space that it needs to explore, you need higher learning rates. And then the second one, uh, the hypothesis here is that the training deni-- uh, the training, um, like, dynamic of products of matrices is different than a full matrix, and this is where the, you know, increasing batch size, uh, phenomenon occurs. So it's basically the, um, explanations-- tentative explanations given there. So now we're going to explore, um, an optimization of this. Uh, but before I dive deep into, uh, that part, uh, does anyone have any questions on the LoRA part? Yep. Yep, great point. So the question is, uh, do you do grid search, uh, on the rank? So you could. So it's a design choice, the rank R. Uh, typically, people would have done it before you, so you have an idea of what rank could be, uh, well suited for your, uh, for your use case. Uh, so you could definitely do grid search, or you could just pick a popular value. I think a four is, is used commonly. Uh, you can just go with, with one of them. Um, yeah. And then the re-- we are going to see that the reduction in number of parameters is so huge already that reducing, uh, you know, even more maybe doesn't matter that much. Like, that initial reduction, reduction by orders of magnitude goes a long way, uh, and then that is just a hyperparameter tuning. And you can see a given rank in a given setup as being a design choice. Uh, okay,

14. 1:45:16 – 1:47:21

    QLoRA

    1. SPSpeaker1:45:16

       so we have two minutes, and we're going to quickly cover quantized, uh, LoRA. So the techniques that Afshin mentioned just now, uh, regarding quantizing weights, uh, and, um, and so reducing the memory footprint is something that we're going to see just here. Uh, so when you look at these matrices, W zero and A and B, what people have done in this paper is to quantize the, uh, weights of W zero into a format that is very smart and then, uh, compute and iterate on these matrices that are being learned, A and B, in full precision, which is in that case BF16. Um, and then, uh, the quantization of these, uh, frozen weights is super smart. It's in a format called NF4, which assumes that the weights are distributed normally, and it splits the space into quantiles rather than buckets of fixed size, which puts about the same num-amount of number of, uh, values into each. So it kind of optimizes the bits that you use for encoding. And what it does is that it does, like, a double quan-quantization process. So it quantizes the weights and then in a second step, it quantizes the quantization constants that we didn't see at length here. But basically, when you want to convert your, um, your full weights in and out of quantized states, you have constants that are generated, and they propose to, to quantize these constants as well. So, uh, this method generated sixteen X, um, VRAM, uh, savings, and then the double quantization methods, uh, gave some extra savings, but not that much. But it's interesting to know. And then we're exactly on time. Uh, thank you for your time.

Episode duration: 1:47:26