Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)

1. 0:00 – 0:10

   Introduction

   1. YDYann Dubois0:05

      So let's get started. Uh, so I'll be talking about building LLMs today. Um,

2. 0:10 – 0:16

   Recap on LLMs

   1. YDYann Dubois0:10

      so I think a lot of you have heard of LLMs before. Uh, but just as a quick recap,

3. 0:16 – 0:19

   Definition of LLMs

   1. YDYann Dubois0:16

      uh, LLMs, standing for Large Language Models, are basically

4. 0:19 – 1:16

   Examples of LLMs

   1. YDYann Dubois0:19

      all the chatbots, uh, that you've been hearing about recently. So, uh, ChatGPT from OpenAI, uh, Claude, uh, from Anthropic, Gemini and, and Llama and other type of models like this. And today, we'll be talking about how do they actually work. So it's gonna be an overview 'cause it's only one lecture, and it's hard to compress everything. But hopefully, I'll touch a little bit about all the components that are needed to train, uh, some of these LLMs. Uh, also, if you have questions, please interrupt me and ask. Uh, if you have a question, most likely, other people in the room or on Zoom have other-- have the same question, so please ask. Um, great. So what matters when training LLMs? Um, so there are a few key components that matter. Uh, one is the architecture. So as you probably all know, LLMs are neural networks, and when you think about neural networks, you have to think about what architecture you're using. Another component which is really important, uh, is the training loss and the training algorithm,

5. 1:16 – 1:20

   Importance of Data

   1. YDYann Dubois1:16

      um, so how you actually train these models. Then it's

6. 1:20 – 1:33

   Evaluation Metrics

   1. YDYann Dubois1:20

      data, so, uh, what do you train these models on? Um, the evaluation, which is how do you know whether you're actually making progress towards the goal of, uh, LLMs? And then the system component,

7. 1:33 – 1:41

   Systems Component

   1. YDYann Dubois1:33

      so that is, like, how do you actually make these models run on, uh, modern hardware, which is really important 'cause these models are really large. Um,

8. 1:41 – 1:47

   Importance of Systems

   1. YDYann Dubois1:41

      so now more than ever, system's actually really an important topic, um, for LLMs.

9. 1:47 – 1:57

   LLMs Based on Transformers

   1. YDYann Dubois1:47

      So those five components, um, you probably all know that LLMs, and if you don't know, LLMs are all based on transformers or at least some version of transformers.

10. 1:57 – 2:00

    Focus on Key Topics

    1. YDYann Dubois1:57

       Uh, I'm actually not gonna talk about the architecture

11. 2:00 – 3:02

    Transition to Pretraining

    1. YDYann Dubois2:00

       today, uh, one, because I gave us here a lecture on, um, transformers a few weeks ago, and two, because you can find so much information online on, uh, transformers. But I think you can-- it's-- uh, there's much less information about the other four topics, so I really wanna talk about those. Um, another thing to say is that most of academia actually focuses on architecture and training algorithm and losses. Um, as academics, and I've done that for a lo-- big part of my career [chuckles] , is simply, uh, we like thinking that this is, uh, like we make new architectures, new models, and it, it seems like it's very important. But in reality, honestly, what matters in practice is mostly the three other topics, so data, eval-evaluation, and systems, uh, which is what of-- most of industry actually focuses on. Um, so that's also one of the reason why I don't wanna talk too much about the architecture, uh, 'cause really the rest is super important. Um, great. So overview of the lecture, I'll be talking about pretraining. So pretraining, uh, you probably heard that word. This is the general word.

12. 3:02 – 4:17

    Overview of Language Modeling

    1. YDYann Dubois3:02

       This is kind of the classical language modeling, uh, paradigm, uh, where you basically train a language model to essentially model all of internet. And then there's a post-training, which is a more recent paradigm, which is taking these large language models and making them essentially AI assistants. Um, so this is more of a recent trend since ChatGPT. Uh, so if you ever heard of GPT-3 or GPT-2, that's really pretraining land. Uh, if you heard of ChatGPT, which you probably have, this is really post-training land. Uh, so I'll be talking about both, but I'll start with pretraining. And, uh, specifically, I'll ta-talk about what is the task of pretraining LLMs and what is the loss that people actually use. So language modeling, this is a quick recap. Uh, language models at a high level are simply models of probability distribution over sequences of tokens or of words. So it's basically some, uh, model of P of x1 to xL, where x1 is basically word one, and xL is the last word in the sequence or in the sentence. Um, so very concretely, if you have a sentence like, "The mouse ate the cheese," what the language model gives you is simply a, a probability of this sentence being uttered by a human or being found on, on, online.

13. 4:17 – 5:15

    Generative Models Explained

    1. YDYann Dubois4:17

       Uh, so if you have another sentence like, "The the mouse ate cheese," uh, here there's grammatical mistakes, so the model should know that this, uh, should have some syntactic knowledge. So it should know that this has less likelihood of appearing online. Uh, if you have another sentence like, "The cheese ate the mouse," uh, then the model should hopefully know about the fact that usually cheese don't eat mouse. Um, so there's some semantic knowledge, and this is less likely than the first sentence. So this is basically at a high level what language models are. Um, one word that you probably have been hearing a lot in the news are generative models. Uh, so this is just something that can generate, models that can generate sentences or can generate some data. Uh, the reason why we say language models are generative models is that once you have a model of a distribution, you can simply sample from this model, and now we can generate data. Uh, so you can generate sentences, uh, using a language model. So the type of models that, uh, people are all currently

14. 5:15 – 6:36

    Autoregressive Models Definition

    1. YDYann Dubois5:15

       using are what we call autoregressive language models. And the key idea of autoregressive language models is that you take this distribution over words, and you basically decompose it into the dis-- into the distribution of the first word, multiply the-- by the distribution of or the likelihood of the distribution of the second word given the first word, uh, multiply it by P of the third word given the first two words. Um, so there's no approximation here. This is just the chain rule of probability, which you hopefully you all know about. Uh, really no approximation. This is just one way of modeling a distribution. Uh, so slightly more concisely, you can write it as a product of, uh, of Ps of the next word given everything which happened in the past, so of the context. Uh, so this, this is what we call autoregressive language models. Again, this is really not the only way of modeling distribution. This is just one way. Uh, it has some benefits and down- some downsides. One downside of autoregressive language models is that when you actually sample from this autoregressive language model, you basically have a for loop, which generates the next word, then conditions on that next word, and then regenerate another word. So basically, if you have a longer sentence that you wanna generate, you... it takes more time to generate it. Uh, so there are some downsides of this current paradigm, but that's what we currently have, so I'm gonna talk about this one. Uh,

15. 6:36 – 7:49

    Autoregressive Task Explanation

    1. YDYann Dubois6:36

       great. So autoregressive language models. At a high level, um, what the task of autoregressive language model is, is simply predicting the next word, as I just said. So if you have a sentence like, "She likely prefers," uh, one potential next word might be dogs. And the wa- the way we do it is that we first tokenize. So you take these words or sub-words, you tokenize them, um, and then you give an ID for each token. So here you have one, two, three. Uh, then you pass it through this black box. As I already said, we're not gonna talk about the architecture. You just path it through... pass it through a model, and you then get a distribution, a probability distribution over the next word or over the next token. And then you sample, uh, from this distribution, you get a new token, and then you detokenize. So you get a new ID, you the- detokenize, and that's how you basically sample from a language model. Uh, one thing which is important to note is that the last two taps, uh, two steps are actually only need, needed during inference. Uh, when you do training, you just need to predict, uh, the most likely token, and you can just compare to the real token which happened in practice, and then you basically change the weights of your model to increase the probability of generating that token. Um,

16. 7:49 – 8:48

    Training Overview

    1. YDYann Dubois7:49

       great. So autoregressive neural language models. So to be slightly more specific, still without talking about the architecture, uh, the first thing we do is that we have all of these... Oh, sorry. Yes.

    2. SPSpeaker7:59

       On the previous slide, when you're predicting the probability of the next token, does this mean that your final, like, output vector has to be the same dimensionality as the number of tokens that you have?

    3. YDYann Dubois8:09

       Yes.

    4. SPSpeaker8:10

       How do you deal with, like, if you have more tokens, if you're adding more tokens to your corpus over time or something like-

    5. YDYann Dubois8:16

       Yeah.

    6. SPSpeaker8:16

       How does that work?

    7. YDYann Dubois8:16

       So we're gonna talk about tokenization actually later, uh, so you will get some sense of this. You basically can't deal with adding new tokens. I am... I'm kind of exaggerating. There are methods for doing it, but essentially people don't do it. Um, so it's really important to think about how you tokenize your text, and that's why we'll talk about that later. Uh, but it's a very good point to note is that you basically... the vocabulary size, so the number of tokens that you have, is essentially the output of your, uh, language model. So it's actually pretty, pretty large. Okay, so autoregressive neural language models.

17. 8:48 – 10:50

    Tokenization Importance

    1. YDYann Dubois8:48

       First thing you do is that you take every word or every token, you embed them, so you get a, um, some vector representation for each of these tokens. Um, you pass them through some neural network. As we said, it's a transformer. Then you get a representation for all the word in... all the words in the context. So it's basically representation of the entire sentence. Uh, you pass it through a linear layer, as you just said, to basically map it to the number, so that the output, the number of outputs is the number of tokens. Uh, you then pass it through some softmax, and you basically get a probability distribution over the next words given every word in the context. And the loss that you use is basically, it's essentially a task of classifying the next token, so it's a very simple kind of machine learning task. So you use a cross entropy loss, where you basically u- look at the actual target that happened, which is a target distribution, which is a one-hot encoding, which here in this, in this case says, "I saw a..." The real word that happened is cat, so that's a one-hot, um, distribution over cat. And here this is the actual, uh... Do you see my mouse? Oh, yeah. This is the distribution that you generated, and basically you do cross entropy, which really just increases the probability of generating cat and decreases all the o- the probability of generating all the other tokens. One thing to notice is that, as you all know, again, uh, this is just equivalent to maximizing the text log likeli... uh, the text log likelihood, 'cause you can just wri- rewrite the, the max over the probability of, um, this autoregressive language modeling task as just being this minimum over... I just added the log here and minus, which is just the minimum of the loss, which is the cross entropy loss. So basically minimizing the loss is the same thing as maximizing the likelihood of your text. Any question? Questions? Okay. Tokenizer. Um, so this is one thing that people usually don't talk

18. 10:50 – 13:30

    Tokenization Process

    1. YDYann Dubois10:50

       that much about. Tokenizers are extremely important. Uh, so it's really important that you kind of understand at least, uh, what they do at a high level. So why do we need tokenizers in the first place? Uh, first, it's more general than words. So one simple thing that you might think is, "Oh, we're just gonna take every word that we will have." You just say every word is a new, is a token in its own. Um, but then what happens is if there's a typo in your word, then you might not have any token associated with this, this word with a typo, and then you don't know how to actually pass this word with a typo into a large language model. So what do you do next? And also, even if you think about words, words is a very... Like, words are fine with, like, Latin-based languages, uh, but if you think about a language like Thai, you won't have a simple way of tokenizing by spaces 'cause there are no spaces between words. Um, so really, uh, tokens are much more general than words. First thing. Second thing that you might think is that you might tokenize every sentence character by character. You might say A is one token, B is another token. Uh, that would actually work and probably very well. The issue is that then your sequence becomes super long, and as you probably remember from the lecture on, on transformers, uh, the complexity, uh, grows quadratically with the length of sequences. So you really don't wanna have a super long sequence. Um, so tokenizers basically try to deal with those two problems and give common subsequences a certain token, and usually, how you should be think about is ar-around, uh, an average arou-- every token is around three, four letters. Um, and there are many algorithms for tokenization. I'll just talk about one of them to give you a high level, uh, which is what we call byte pair encoding, which is actually pretty common, one of the two most common, uh, tokenizers. And the way that you train a tokenizer is that first you start with a very large corpus of text, and here I'm really not talking about training a large language model yet. This is purely for the tokenization step. Uh, so this is my large corpus of text with these five words. Um, then you associate every character in this corpus of text a different token. Uh, so here I just split up every character with a s- different token, uh, and I just color-coded all of those to-tokens. And then what you do is that you go through your text, and every time you see pairs of tokens that are very common, the most common pair of token, you just merge them. So here you see three times the, the, the tokens T and O next to each other, so you're just gonna say this is a new token. And then you continue. You repeat that. So now you have T-O-K, tok, which happens three times, toke with an E,

19. 13:30 – 16:00

    Example of Tokenization

    1. YDYann Dubois13:30

       that happens, uh, sorry, two times. Uh, and token, which happens twice, and then E-X, which also happened twice. So this is the... If you were to train a tokenizer on this corpus of text, which is very small, that's how you would, uh, finish with a tokeni-- with a pre... like a trained tokenizer. Uh, in reality, you do it on, on much larger corpuses of text, um, and this is the real tokenizer of, uh, actually, I think this is GPT-3 or ChatGPT. Uh, and here you see how it would actually separate these words. So basically, you see the same thing as what we gave in the previous example. Token becomes its own token. So tokenizer is actually split up into two tokens: token and izer. Um, so yeah. That's all about tokenizers. Any question on that? Yeah.

    2. SPSpeaker14:16

       How do you deal with spaces and how do you deal with punctuation tokens?

    3. YDYann Dubois14:20

       Yeah. So actually, there's a, a step before tokenizers, which is what we call pre-tokenizers, which [chuckles] is exactly what you just said. Uh, so this is mostly... In theory, there's no reason to deal with spaces and punctuation separately. You could just say every space gets its own token, every, um, uh, punctuation get its own token, and you could just do all the merging. The problem is that... So there's an efficiency question. Actually training these tokenizers takes a long time, uh, so you're better off-- because you have to consider every pair of token. So what you end up doing is saying, "If there's a space, this, this is very..." Like pre-tokenizers are very English specific. You say, "If there's a space, we're not gonna start looking at the, the token that came before and the token that came afterwards." So you're not merging in between spaces. But this is just like a optimiza-- uh, like a computational optimization. You could theoretically just deal with it, um, the same way as you deal with any other character.

    4. SPSpeaker15:15

       And-

    5. YDYann Dubois15:15

       Yeah.

    6. SPSpeaker15:16

       When you merge tokens, do you delete the tokens that you merged away or do you keep the, the smaller tokens that you merged away?

    7. YDYann Dubois15:22

       Um, you actually keep the smaller tokens. I mean, in reality, it doesn't matter much because, um, usually on a large corpus of text you will have actually everything. Uh, but you usually keep the small ones, and the reason why you wanna do that is because if in case there's a, as we said before, you have some, um, s-some grammatical mistakes or some typos, you still wanna be able to represent these words by character. Um, so yeah. Yes.

    8. SPSpeaker15:48

       Are, are the tokens unique? So I mean, say in this case, T-O-K-E-N, is there only one occurrence, or could... Do you need to leave multiple occurrence so they could have...

20. 16:00 – 20:50

    Evaluation with Perplexity

    1. SPSpeaker16:00

       take on different meanings or something?

    2. YDYann Dubois16:01

       Oh. Oh, I see what you say. No, no, no. It's every token has its own, uh, unique ID. Um, so a usual... Th-this is a great question. For example, if you think about a bank, which could be bank for, like, money or bank like water, um, it will have the same token, but the model will learn, the transformer will learn that based on the words that are around it, it will... should associate that. I-I'm saying... I'm being very hand-wavy here, but associate that with a, with a, with a representation that is either more like the, the bank f- money side or the bank, uh, water side. Um, but that's the transformer that does that. It's not a tokenizer. Yes.

    3. SPSpeaker16:39

       Yeah. So you mentioned during tokenization, keep the smaller tokens you started with, right? Like if you start with a T, you keep the T, and then you tell your tokenizer to the standard, "Allow encode token." So let's say, um, maybe you didn't train on token, but, like, in your data you are trying to encode token. So how does the tokenizer know to encode it with token or to read a T?

    4. YDYann Dubois16:59

       Yeah. It's a great question. You basically... When you... So when you tokenize, so that's after training of the tokenizer, when you actually apply the tokenizer, you basically always choose the largest, uh, token that you can apply. Uh, so if you can do token, you will never do T. You will always do token. Um, but there's actually... So people don't usually talk that much about tokenizers, but, uh, there's a lot of, of computational benefits, uh, or computational tricks that you can do for making these things faster. Uh, so I really don't think we... And honestly, I think a lot of people think that we should just get away from tokenizers, um, and just kind of tokenize character by character or bytes by bytes. Uh, but as I said, right now there's this issue of, like, length. Uh, but maybe one day, like in five or ten years, uh, we'll have different architectures that don't scale quadratically with the length of the sequence and, uh, maybe we'll- Um, yeah, move away from tokenizers.

    5. SPSpeaker17:51

       So can you share with us the drawback? Why, why do people wanna move away from the tokenizer?

    6. YDYann Dubois17:55

       Oh, um, yeah. So think one good example is, uh, math. If you think about math, actually, numbers right now are not tokenized. So for example, three hundred and twenty-seven might have its own token, which means that models, when they see numbers, they don't see them the same way as we do. And this is very annoying because what-- I mean, the reason why we can kind of generalize with math is because we can deal with every ca- every letter separately, and we can then do composition, where you know that basically if you add stuff, which is the same thing as adding every one separately, plus, like, whatever the unit that you add. Uh, so they can't do that. Um, so then you have to do, like, special tokenization. And, like, o- one of the big changes that GPT-4 did, uh, is changing the way that they tokenize, uh, code. So for example, uh, if you have code, you know, you have, like, often in Python, these four spaces at the beginning. Those were dealt with, uh, kind of strangely before. Um, and as a result, like, the model couldn't really understand, uh, how to deal with code. Uh, so, so tokenizers actually matter a lot. Um, okay. So I'll move on, uh, right now, but we can come back later on tokenizers. Great. So we talked about the task, the loss, the tokenizer. Let's talk a little bit about evaluation. Uh, so the way that LLMs are usually evaluated is what we call-- is using what we call perplexity. Um, at a high level, it's basically just your validation loss. Uh, the slight difference with perplexity is that we use something that is slightly more interpretable, which is that we use the average per-token loss, and then you exponentiate it. And the reason why you exponentiate it is because you want-- I mean, the loss, uh, has a log inside and you-- like, one, humans are actually pretty bad at thinking in log space, but two, logs depend on the base of the log. Uh, while wh- when you exponentiate, you basically have everything in the, uh, kind of the vocabulary size, uh, unit. Um, and the average per token is just so that your, your perplexity is independent of the length of your sequence. Um, so perplexity is just two to the power, uh, average of the loss of the sequence. Um, so perplexity is between one and the length of the vocabulary of your tokenizer. Uh, one, it's simply-- well, if you predict perfectly the thing which, uh, every word, then every word will have basically product of ones. Uh, so the best perplexity you can have is one. If you really have no idea, you basically predict with one divided by, uh, size of vocabulary, um, and then you do simple math, and you basically get perplexity of size of vocabulary. Uh, so the intuition of perplexity is that it's basically the number of tokens that your model is kind of hesitating between. Uh, so if your, if your model is perfect, it doesn't hesitate, it know exactly the word. If it really has no idea, then it hesitates between, uh, all of the vocabulary. Uh, so perplexity really improved. That's perplexity on a standard dataset between twenty seventeen and twenty twenty-three.

21. 20:50 – 24:30

    Current Evaluation Methods

    1. YDYann Dubois20:50

       It, it went from a kind of seventy tokens to less than ten tokens over these five, six years. So that means that the models were previously hesitating between seventy words every time it was generating a word, and now it's hesitating between, like, less than ten words. Uh, so that's much better. Uh, perplexity is actually not used anymore in academic benchmarking, mostly 'cause it depends on the tokenizers that you use, uh, it depends on the actual data that people are evaluating on, but it's still very important for development of LLMs. So when you, when you actually train your own LLM, people will still really look at the perplexity. Uh, one common other way, and now more common in academia of evaluating these LLMs, is just by taking all the classical NLP benchmarks, and I'll give you a few examples later, and just kind of aggregating everything. Um, so collect as many automatically evaluatable benchmarks and just evaluate across all of them. Um, so one such ev-- uh, or actually two such, uh, benchmarks are what we call, uh, HELM, which is from Stanford. Another one is the Hugging Face Open LLM Leaderboard, which are the-- probably the two, two most common ones right now. Um, so just to give you an idea, in HELM, there are all of these type of tasks, which are mostly things that can be easily evaluated, uh, like question answering. So think about many different question answering, uh, tasks. Um, and the benefit with question answering is that you usually know what is the real answer. Um, so you can-- The way that you evaluate these models, and I'll give you a concrete example in one second, um, is that you can just look at how likely the language model is to generate the real answer compared to some other answers. And that's essentially, at a high level, how you evaluate these models. Um, so to give you a specific example, MMLU is probably the most common, um, academic benchmark for LLMs. Uh, and this is just a collection of many question and answers in all of those domains. For example, college medicine, college physics, astronomy, and these type of topics. And the questions are things like, so this is in astronomy, "What is true for type One A supernova?" Then you give, uh, four different potential answers, and you just ask the model which one is more likely. So eith- either many different ways of doing it. Either you can look at the likelihood of generating all these answers, uh, or you can ask the model which one is the most likely. Uh, so there are different ways that you can prompt the model, but at a high level, you know which one is correct, and there are three other mistakes. Um, yes.

    2. SPSpeaker23:20

       Given that we're kind of creating these, like, unconstrained text as an output-

    3. YDYann Dubois23:25

       Yeah.

    4. SPSpeaker23:25

       How do you evaluate a model if it gives something that's, you know, semantically completely identical but is not the exact token list that you expect?

    5. YDYann Dubois23:35

       Yeah. So that's a great question. I'll talk more about that later. Here in this case, we don't do unconstrained. So the way you would evaluate MMLU is basically either you l- l-- you ask the first question and then you look at the likelihood- Of the model generating A, the likelihood of the model generating B, C, and D, and you look at which one is the most likely. Or you can ask the model out of A, B, C, D, which one is the most likely, and you look at whether the to- the most likely next token is A, B, C, or D. So, uh, you constrain the model to say it can only answer these four things.

    6. SPSpeaker24:08

       When you say you constraint the model-

    7. YDYann Dubois24:10

       Yeah.

    8. SPSpeaker24:10

       Do you mean you constraint it with a prompt, or do you mean that of its whole probability distribution layer outputs, you only are comparing the outputs of, like, you're only comparing the A token to-

    9. YDYann Dubois24:20

       Yeah. So, uh, in the second case I gave you, you would do exactly the-- uh, actually, you would do both. You would prompt the model saying A, B, C, or D, plus you would constrain to only, uh, look

22. 24:30 – 1:44:24

    Academic Benchmark: MMLU

    1. YDYann Dubois24:30

       at these two-- these four tokens. In the first case, you don't even need to generate anything. So in the first case, you literally just look, given that it's a language model, it can give a d-distribution over sentences. You just look at what is the d- the likelihood of generating all of these words. What is the likelihood of generating the second choice? And you just look at whether the most likely sentence is actually the real answer. So you don't actually sample from it. You really just use P of X one to XL. Does that make sense?

    2. SPSpeaker25:01

       Yeah.

    3. YDYann Dubois25:01

       Uh, that being said, evaluation of open-ended questions is something we're gonna talk about later and is actually really important and really challenging. Yes.

    4. SPSpeaker25:10

       Earlier, you mentioned that, um, like, um, metrics like perplexity are not ca-- are not, like, usually used because it depends on, like, how you do your tokenization, some design choices. I was wondering if you could speak more to that.

    5. YDYann Dubois25:23

       Oh, um, yeah. So think about perplexity. I told you perplexity is between one and vocabulary size. So now imagine that ChatGPT uses a tokenizer that has, like, ten thousand tokens. But, uh, Gemini from Google uses a tokenizer that had a hundred thousand, uh, po-potential tokens. Then actually, the Gemini one will, will have, like, the upper bound of the, the perplexity that you can get is actually worse for Gemini than for ChatGPT. Does that make sense?

    6. SPSpeaker25:52

       Yeah.

    7. YDYann Dubois25:52

       So that's just an idea. It, it's actually a little bit more complicated than that, but that's just, like, one, uh, first order bit of where you can see that the to-- the tokenizer actually matters. Um, great. Okay. So evaluation challenges, there are many. I'll just talk about two really briefly. Uh, one, as I told you, there are two ways of doing evaluation for these MMLUs. Actually, there are many more than two, but I gave you two examples. Um, and it happens that for a long time, even though that was a very classical benchmark that everyone used, uh, actually different, uh, different companies and different, um, differ, uh, uh, different organization were actually using different ways of evaluating MMLU. And as a result, you could-- you get completely different results. For example, Llama 65B, uh, which was the first model of, of Meta in the Llama series, uh, had on Helm sixty-three point seven accuracy, but on this other, um, benchmark, had, like, forty-eight point eight. Um, so really the way that, that you evaluate, and this is not even talking about prompting, this is really just kind of the, the way that you evaluate the, um, uh, the models. Prompting is another issue. So really, there are a lot of inconsistencies. It's not as easy as it, as it looks. Uh, first thing. Yeah, sorry.

    8. SPSpeaker27:08

       How can we make sure that all these models aren't trained on the benchmark?

    9. YDYann Dubois27:12

       Okay. [chuckles] Second thing. This is a great question. Uh, train test contamination. Uh, this is something which I would say is really important in academia in, uh, given that the talk is mostly about training large language models. Uh, for companies, it's maybe not that important because they know what they trained on. Uh, for us, we have no idea. So for us, it's a real problem. Uh, so there are many different ways of trying to test whether, uh, the test set, uh, sorry, whether the test set was actually in the training set. Uh, one kind of cute trick, um, that people, uh, in, in the lab-- on Tatsu's lab have found is that what you can do is that given that most of the dataset online are not randomized, you can just look at-- and, and that language models, what they do is just predict the next word. Um, you can just look at the entire test set. Uh, what if you generate all the examples in order versus all the examples in a different order? And if it's more likely to generate a thing in order, given that there's no real order there, then it means that probably was in the training set. Does that make sense? Um, so there are many-- that's, like, one of them. There are many other ways of doing it. Train test con-contamination, again, not that important for development, really important for academic benchmarking. Great. So there are many other challenges, but, uh, I'll move on for now. Great. Data. Um, so data is a-another really big topic. Um, at a high level, people just say, "Oh, you basically train large language models on all of internet." What does that even mean? Um, so or people sometimes say all of clean internet, which is even less defined. Um, so internet is very dirty and really not representative of what we want in practice. If I download a random website right now, uh, you would be shocked at what is in there. It's definitely not your Wikipedia. Um, so I'll go really briefly on, like, what people do. Um, I can answer some questions, but I mean, data is on its own is a huge topic. Uh, basically, first, what you do is download all of internet. What that means is that you use, uh, web crawlers that will go on every webpage on internet or every webpage that is, um, on Google, uh, and that is around two hundred fifty billion pages right now, um, and that's around one petabyte of, of data. So this is actually a Common, Common Crawl is one web crawler. So people don't usually write their own web crawlers. What they do is that they use standard web crawlers, and web Common Crawl is one of them, uh, that basically every month adds all the new we-websites that were added on, uh, internet that are found by, by Google, and they put it in a big, uh, basically a big dataset. Um, so that's on Common Crawl, you have around two hundred fifty billion pages right now, so one E six gigabytes of data. Once you have this, uh, so this is a random webpage, like literally random, uh, from this Common Crawl, and what you see is that, one, it really doesn't look at type of things that you would usually see. But actually-- So this is an HTML page. Uh, it's hard to see, but if you look through, you will see some content. For example, here, uh, "Test King World is your ultimate source for the System X high performance server," and then you have three dots, so you don't even-- the sentence is not even finished. That's how a random internet looks like. Uh, so of course, it's not that useful if you just train a la-large language model to generate things like this. So what are some of the steps that are needed? First one, you extract the text from the HTML, so that's what I just tried to do by looking at, uh, basically the correct tags. Uh, there are a lot of challenges by, uh, through this. For example, extracting math is actually very complicated, but pretty important for training large language models. Um, or for example, boilerplates. A lot of your forms will have the same type of headers, the same type of, uh, footers. Uh, you don't wanna repeat all of this in your data. Um, then you will filter undesirable content, uh, so not safe for work, harmful content, PII. Uh, so usually every company has basically a s- a blacklist of websites that they don't wanna train their models on. That blacklist is very long, [chuckles] and you basically say, "If it comes from there, we don't train on this." There are other ways of doing these things, is that you can train a small mo-model for classifying what is PII, removing these things. Um, it's hard. Every point here that I'm gonna show you is, like, a hard amount of work, uh, but we're gonna go, go quickly through it. So filter undesirable content. Second or fourth is the dupli-- de-duplication. As I said, um, you might have things like headers and footers in forms that are always the same. You want to remove that. Another thing that you might have is a lot of URLs that are different, but actually show the same website. Um, and you might also have a lot of like, uh, um, paragraphs that come from, like, common books that are basically deduplicated a hun- a thousand times or ten thousand times on internet. So you have to deduplicate. Also very challenging, uh, because you have to do that at scale. Once you do deduplication, you will do some heuristic filtering. You will try to remove low-quality documents. Uh, the way you do that are things like rules-based, um, filtering. For example, if you see that there are some outlier tokens. If the distribution of tokens in the website is very different than the usual distribution of tokens, there's probably some outlier. If you see that the length of the words in this website is super long, there's something strange going on on that website. If you see that the, the website has only three words, eh, maybe is it worth training on it? Maybe not. If it has, like, ten million words, maybe there's something also, uh, wrong going on that page. Um, so a lot of rules like this. Yes.

    10. SPSpeaker33:02

        Why don't we filter out undesirable content from our dataset instead of kind of putting it in as, like, a supervised loss, right? Like, can we not just say, like, you know, "Here's this, like, hate speech website, let's actively try to-- Let's actively penalize them for generating-"

    11. YDYann Dubois33:20

        We'll do exactly that, but not at this step. That's where the post-training will come from. Uh, pre-training, um, the idea is just to say, "I wanna model kind of how humans speak, essentially, um, and I wanna remove all these, like, headers, footers and, and menus and things like this." Uh, but it's a very good imp-- uh, like, idea that you just had, and that's exactly what we'll do later. Next step, model-based filtering. So once you've filtered a lot of data, what you will do... Uh, so that's actually a very cute trick. Uh, you will take all of Wikipedia, and you will look at all the links that are linked through Wikipedia pages, 'cause probably if something is referenced by Wikipedia, it's probably some high quality website. And you will train a classifier to predict whether something comes from-- whether a document comes from one of these references, uh, from Wikipedia, or whether it's from the random web. And you will try to basically say, "I want more of the things that come from Wikipedia references." Does that make sense? So yeah. So you will train a m- a machine learning, uh, model. Usually also very simple models, 'cause you need to do that really at scale. I mean, just think about the two hundred and fifty billion pages. Uh, next one, you will try to classify your data into different, different, um, domains. You will say, "Okay, this is entertainment, this is books, this is code, this is, like, these type of domains." And then you will try to either, um, up or down-weight some of the domains. Uh, for example, you might say, uh, you might see that actually if you train more on code, then actually your model becomes better on reasoning. So that's something that people, uh, usually say in a very hand-wavy way. If you train your model more on code, actually it helps reasoning, so you wanna up-weight the coding, uh, distribution because that helps for general language modeling skills. Uh, books is usually also another one that people usually, um, up-weight. Entertainment, they usually down-weight. Uh, so things like this. Of course, you wanna do it-- So people used to do it maybe, uh, kind of heuristically. Now there's entire pipelines that we'll talk about of how to do these things, uh, slightly more, um, automatically. And then at the end of training, uh, usually train, um, after training on all of this data that we saw, you usually train on very high quality data at the end of y- of training a la-large language model, where you decrease your learning rate. Uh, and that basically means that you're kind of over-fitting your model on a very high quality data. So usually what you do there is, like Wikipedia, you basically over-fit on Wikipedia and you over-fit on, like, hu-human, uh, data that was collected. Um, the other things like continual pre-training for getting longer context, I'm, I'm gonna skip over all of these things. Uh, but just to give you a sense of how hard it is when people just say, "Oh, I'm gonna train on internet," that's a lot of work. Um, and really we haven't figured it out yet. So collecting world data is a huge part of practical large language model. Uh, some might say that it's actually the key. Yes.

    12. SPSpeaker36:27

        Yeah. I have a question about the data. So here's the question. So usually when you start with, like, the tera- terabytes of data, after I go through all that steps, what is the typical amount of data I have remaining? And then, like, how, how large a team does it typically take to go through all the different steps you talked about?

    13. YDYann Dubois36:43

        Sorry, how lo-- Is your question, how large is the data after you filter?

    14. SPSpeaker36:47

        Yeah, after you filter, and then to go through all the step, how large a team do you need to go through, like, the fil- the all the filtration steps you mentioned?

    15. YDYann Dubois36:54

        Uh, how slow is it, or-

    16. SPSpeaker36:56

        How, how, like, how lo- how many people would you need-

    17. YDYann Dubois36:59

        Oh

    18. SPSpeaker36:59

        ... to be able to do this work?

    19. YDYann Dubois37:02

        Uh, okay, that's a great question. I'm gonna somewhat answer about the data, uh, how large is the dataset, uh, at the end of this slide. Uh, for number of people that work on it, um, that's a good question. I'm actually not quite sure, but I would say... Yeah, I actually don't quite know, but I would say it's probably even bigger than the number of people that work on kind of the tuning of the pretraining of the model. Uh, so the data is bigger than kind of the modeling aspect. Um, yeah, I, I don't think I have a good sense. I would say probably in Llama's team, which have, like, seventy-ish people, I would say maybe fifteen work on data. Uh, I th- yeah. All these things, you don't need that many people. You need a lot of compute also, 'cause, uh, for data, you need a lot of CPUs. Um, so yeah, and I-I'll answer the second question, uh, at the end of this slide. So as I just kind of alluded to, really, we haven't solved data at all for pretraining, so there's a lot of research that, that has to be done. First, how do you process these things super efficiently? Uh, second, how do you balance the kind of like di-- all of these different domains? Uh, can you do synthetic data generation? That's actually a big one right now. Uh, and w- because we don't have, uh, we'll talk about that later, but we don't have enough data on the internet, um, can you use multimodal data instead of just text data? And how does that improve even your text performance? Um, there's a lot of secrecy 'cause really this is the key of most of the pretrained, pretrained large language models. Uh, so for competitive dynamics, uh, usually these, these, um, these companies don't talk about how they do the data collection. And also, there's a copyright liability issue. They definitely don't wanna tell you that they've trained on books even though they did, um, because if not, you can, uh, sue them. Uh, common academic benchmarks, uh, so that will kind of answer what you asked. Um, it started-- So those are the smaller ones. It's-- The names are not that important, but it started from ar-around one hundred and fifty billion tokens, which are around, uh, eight hundred gigabytes of data. Now it's around fifteen trillion of to-- fifteen trillion tokens, which is also, uh, the size of the models that are-- Right now, the best models are probably trained on that amount of data. So fifteen trillion tokens, uh, which is probably, I guess, two order of magnitude bigger than that, so eighty, uh, E3 gigabyte. So that would be around one hundred to one thousand times, uh, filtering of the Common Crawl, if I'm not mistaken. Um, so yeah. One very co-- one very, uh, famous one is the PAL. So this is a academic benchmark, the PAL, and we can just look at what distribution of data they have. It's things like, um, arXiv, PubMed Central, uh, which is all the, the biology stuff. Uh, here it's Wikipedia, you see Stack Exchange, um, some GitHub, and some books, and things like this. Um, again, this is on the smaller side. So this is, if we look at here, this is on two hundred eighty GB, so in reality, it's like a hundred times bigger, so you cannot have that much of GitHub and on, on Wikipedia. Um, in terms of closed source models, uh, just to give you an idea, uh, Llama Two, um, it was trained on two hun- two trillion tokens. Llama Three, fifteen trillion tokens, which is currently the best model that we know on how much it was trained on, which is the same thing as this, the, the, the best academic or the biggest academic benchmark, which is fifteen trillion tokens. GPT-4, we don't really know, but it's probably in the same order of magnitude, or it's probably around that actually. It's probably around thirteen, um, from leaks, if the leaks are true. Um, great. So scaling laws. Um, any other questions on data before we go to scaling laws? Sorry, I know I'm giving you a lot of information, but, uh, there's a lot into training in large language models. Great. Scaling laws. So the idea is that what people saw, um, around twenty-twenty, or at least from a long time, but they've been able to kind of, uh, theoretically show it or empirically show it since twenty-twenty, is that the more data you train your models on and the larger the models, the better the performance. This is actually pretty different than what you see in this class. In this class, we teach you about overfitting. Overfitting doesn't happen with large language models. Uh, larger models, better performance. Um, it's something that really took a long time for the community who took this type of class to realize. Um, but for the exam, overfitting exists. [laughs] So okay. The idea of scaling laws is that if, given that you know that more data and larger mo- models will always give you better performance, can we predict how much better your performance will be if you increase the amount of data and the size of your model? And surprisingly, it works. Uh, so here you see three plots from a very famous paper called "Scaling Laws" from OpenAI. Um, here you see on the x-axis compute, so how much did you train, like how much compute did you, did you spend for training? And here you see test loss. So this is essentially, I mean, it's not perplexity, but it's your validation loss. Um, so it's a log of the perplexity. And if you put these two on, uh, log scale, uh, then you see that, uh, the, um, the performance or, like, the, this, the, sorry, the, the scaling law is linear. Uh, that means that if you increase your compute by a certain amount, you can, you can say by how much your test loss will actually decrease. Same thing with data and same thing for parameters. If you increase the data set size, your loss will in- will decrease by an amount that is somewhat predictable. If you increase the number of parameters, it will decr- the loss will decrease by am-amount which is somewhat predictable. This is really amazing. Um, very surprising. I mean, it looks innocuous when you look at these type of plots, but that's crazy 'cause it means that you can predict, uh, how well we're gonna perform in two, three years, depending on how much compute we will add, assuming that these things will hold. There's nothing theoretical about it. Um, yes.

    20. SPSpeaker43:05

        Uh, two things. One, what is the loss that they're using here if there's perplexity?

    21. YDYann Dubois43:09

        Uh, so it's, it's, you know, I said perplexity was like two to the power of the loss, so this is the-

    22. SPSpeaker43:14

        Oh

    23. YDYann Dubois43:14

        ... the p- the power-

    24. SPSpeaker43:15

        Oh

    25. YDYann Dubois43:15

        ... of the perplexity.

    26. SPSpeaker43:17

        And then the second thing is when you, like, increase the number of parameters or you increase the total data set size and you're going over that data set more and more times, doesn't that just inherently increase your compute? Like, is all of this work obviously just how you're doing that-

    27. YDYann Dubois43:31

        Oh, yes

    28. SPSpeaker43:31

        ... or is there something specific to it?

    29. YDYann Dubois43:32

        No. This is a great question. So the compute here is actually a factor of two things: the data and the parameter. What I'm showing here is that you can, um... Well, actually, we're gonna talk about that in details, but basically, if you increase the number of parameters, you should increase the number of data that you have. Um, so you actually don't go multiple times through the same data set. No one does epochs in, uh, a large-- uh, at least not yet.

    30. SPSpeaker43:56

        Okay.

Episode duration: 1:44:30