Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 1 - Transformer

1. 0:00 – 3:54

   Introduction

   1. AAAfshine Amidi0:05

      Cool. Hello, everyone, and, uh, welcome to CME 295, Transformers and Large Language Models. So, my name is Afshin, and I will be teaching this class with Shervin, who's in the back. And, uh, before I start, I'm just going to introduce ourselves. Um, so we're twin brothers and, uh, we actually had kind of a similar background. So we both went to a school in France called Centrale Paris, and then we each went our way. So on my end, I went to MIT, and then Shervin went to Stanford to do the ICME master's program. And after that, I guess, uh, our, um, industry background is very similar as well. So I first went to Uber, and then Shervin came to Uber as well, and then Shervin left to Google, and I went to Google. And then very recently, I joined Netflix, and Shervin joined Netflix as well, and we've been working on large language models. Um, so yeah, I guess we have, like, uh, technical backgrounds and mostly oriented towards LLMs. Okay. So why are we doing this class? Um, so since 2020, uh, Shervin and I have been specializing in NLP, and we've been giving this class, uh, in a format of a workshop that was, uh, done in a yearly basis, so in 2021, 2022, 2023, 2024. You know, ChatGPT came in 2022, and suddenly there was a lot of interest for LLMs. And so, uh, it's actually last spring that we started to offer this class as a Stanford course that is now called CME295, and this is the second instance. Cool. Um, so what can you expect from this class? So first of all, LLMs are basically, uh, everywhere now, and I guess our goal here is twofold. So the first one is to learn about the underlying mechanism that makes all this work, and we're going to see the transformer, which is the foundational architecture, uh, that makes all this work. And then the second thing is to know how these LLMs are trained and where they are applied. So in case you're still wondering if this class is good for you, um, I would say that this class is great for people who just in general have an interest in this field, either because you wanted to make it your career goal, uh, if you want to be a, I don't know, research scientist or an ML scientist, or if you want to, uh, develop, like, a personal project that relies on, uh, LLMs to some extent to just, like, knowing the caveats, I guess what works, what doesn't. Or just say if you're in a separate field and you just want to know how this whole AI, gen AI, LLMs thing works and how you can apply it to your domain. Okay. So now in terms of prerequisites, I would say that at, at a very minimum, um, you should have some foundations in ML, like basically know what a-- how a model is trained, what a neural network is, uh, and also some, uh, basics in linear algebra, so basically how matrices are multiplied, for instance. Uh, but even, even if you have kind of a developing, um, uh, I guess competency in these fields, I guess it's fine. We'll still be here to help you out, but I guess, uh, this is, like, the ideal set of prerequisites.

2. 3:54 – 9:40

   Class logistics

   1. AAAfshine Amidi3:54

      Cool. So still on the logistics. Um, so this class will be held every Friday from 3:30 to 5:20, and it will be held here. So this class is two units, and you have the choice to either take it as a letter or a credit/non-credit. So as you could tell from the, from the setup, uh, we're basically recording this class, and if you cannot, for some reason, attend this, uh, you know, this time, this slot, uh, we'll make sure with Shervin to, um, make the recordings available either tonight, like every Friday night, or on Saturday. So in terms of the grades, um, so what we're doing for this quarter is to have two exams. So one is the midterm, which will be happening, uh, during our fifth instance, which is October 24th. And then the second exam will be the final exam, which will be held, uh, like, in, uh, on the, you know, in the week of the December 8th. So date is still TBD, so we'll let you know. Cool. Um, so every time we have a lecture, we'll be posting the slides and the recordings on the website. And in case you're interested, we also have the syllabus in there, so you can know a little bit what are the topics that we'll be talking about. Uh, and, uh, the class textbook is, uh, this Super Study Guide Transformer & LLMs. Uh, so we have a copy here in case you want to take a look. Um, so yeah, I guess a lot of the concepts that we have in this class will actually be in the book, so I guess it's a helpful way to follow this as well. And also we did, uh, some kind of very short, condensed version of this whole class that we called The VIP Cheat Sheet. So this one is available on GitHub in case you're, um, you're interested. Uh, and yeah, we've also translated it into a number of languages now. Um, by the way, if, uh, your language is not there, uh, let us know and, uh, yeah, happy to work on that as well together. Okay, cool. Uh, I think it's the last things on the logistics part. So in terms of announcements, we'll be posting things on Canvas. In case you have any questions, you can, of course, reach out to us. Uh, but there is also a tab on Canvas that's called Ed. Uh, I'm sure you're familiar. Um, so yeah, just click on that, just post your question, and then Shervin and I will be responding. And, uh, yeah, I guess, uh, to reach out to us, you have this mailing list or just, like, uh, you know, we're just two, so just, uh, ping us. Cool. So on the logistics, do we have any questions so far? And one thing I forgot to mention is that given that we're recording this class, um, I guess if you're asking a question, it may not be super clear for the viewer what your question was, so I'm going to make an effort to just repeat your question. It will sound weird, but yeah, I'll try to not forget. But yeah. So yeah, any questions so far on the logistics? Yeah. So the question is whether there are, like, coding parts in the exams. So the answer is no. So the exams will purely focus on concepts that we see in class, and actually, it's not meant to, you know, trap you. So I guess if you follow the class, if, uh, you know, you see the slides and, like, the concepts that we see, should be fine. Yeah. Oh, yeah. Uh, question is, uh, if you're wait-listed, what do you do? Um, I think... So by experience, you know, a lot of people will, uh, kind of finalize their schedule. Some people will drop, some won't. In case you're still wait-listed, uh, you know, come talk to us, but I'm pretty confident, uh, you know, it's going to be okay because I think the wait list right now is, like, six. So yeah, things should be fine. Cool. Yeah. Uh, they will be on the website, and we'll make sure to, um, also post the link on Canvas. Yeah. Uh, so the question was where are the slides, and they're on the website. Cool. Yeah. So question is on the weighting of the exams. So yeah, there is no homework. So fifty percent is midterm, fifty percent is final, and, uh, no, no grades, I mean, no weights apart from that. And in particular, I mean, if this slot is conflicting with something, just, uh, keep in mind that we are recording this, so I mean, it's fine if you, if you cannot attend the same session. Yeah. Sorry? Oh, um, is the question that the final is about just the second half of the class? Um, we've not, uh, we have not, uh, written the exam yet, but I think this is something we're thinking of. So yeah, the final is probably going to be the second half, about the second half of the topics. Cool. Okay. Long story short, fifty percent midterm, fifty percent exam, uh, final exam, and, uh, yeah, it's a fun class. Cool.

3. 9:40 – 22:57

   NLP overview

   1. AAAfshine Amidi9:40

      So with that, I'm going to just slowly start the, the class. Um, so another thing that I wanted to mention was every time we're talking about something, you will see that at the bottom of the slide, there will be a source. It's mostly for... So first to credit whatever we're quoting, but also, uh, for you to kind of dig into those material a little bit more in case you're interested. Uh, because of course, we have only, like, two hours per week, and we only have, uh, nine or ten weeks, so there's nowhere n-near the, you know, uh, enough time for us to cover everything. And the second disclaimer is you will see that the field is full of abbreviations. So I myself was completely scared of them when I started. Uh, but hopefully by the end of the class, you will have a mental mapping of what these abbreviations mean with respect to what they correspond to. Um, so yeah. So if you have that mental mapping towards the end of the class, then we'll know we did a good job. So with that, um, let's start, and I guess we will start at the very high level because I will just, uh, assume that, um, I guess we're starting from scratch. Um, and we're going to talk about NLP in general. So NLP is going to be our first abbreviation. So NLP stands for natural language processing, and it is a field that is around, like, manipulating text, just computing things with text. And at a very high level, you can basically classify NLP tasks into three buckets. So the first bucket is what we call classification. So we have an input text as an input, and then what we want is to predict something. So one example is you have a movie review, and you want to predict whether, uh, the sentiment is positive, negative, or neutral. Uh, so that's one example. We can also have intent detection, uh, just, you know, knowing what, for instance, the person want to do. So let's suppose you say, "I want to create an alarm for tomorrow." So the intent here is create an alarm. So also to detect the language. So for instance, if you write, I don't know, in French, you want to detect that that text is in French. Topic modeling. The second category is what we call multi-classification. So we still have a text as input, but this time we predict more than one thing. So you have a number of tasks in that bucket as well. So one that is very popular is called Named Entity Recognition, AKA NER. So what that task does is, given an input text, we want to basically label some specific words, like for instance, identifying whether something is a location or a time, and so on. And then you have some other tasks as well that are a little bit more on the linguistic side. Uh, I think they're less trendy now, but I guess ten years ago it was something that people would study a lot. Uh, so part of speech tagging, which is about just figuring out which word is, uh, you know, a noun, a verb, et cetera, or some, uh, parsing-related tasks, so dependency or constituency parsing. And then the last bucket, which is very popular these days, is the generation bucket. So you have an-- a text as input, and you also have text as output. And here the length can be variable, meaning you don't know what the length of your output text will be beforehand. So here you have, uh, several tasks. So for instance, you have machine translation. So for instance, something in English, and I want it to, let's say, uh, German. Question answering, so typically, you know, the ChatGPT, Gemini that you're using, you know, the assistant. So you ask a question and you have a response. And then you have like other, um, tasks as well, like summarization. You want to summarize an article, let's say, or just generate something. So something can be generate codes, generate a poem. It can also be a lot of things. Cool. So now what we will do is go through these tasks one by one to just illustrate what people typically handle with. So we're going to start with the first bucket, which is the classification bucket. And here we're going to illustrate this with the sentiment extraction task. So let's suppose we have a sentence, "This teddy bear is so cute." We want our model to predict, you know, this to be a positive sentiment. So typically what you would use is, you know, data sets that are around, you know, sentiment extraction data sets. So I mentioned movie reviews, so this is IMDb critics, but you also have reviews about products, so Amazon reviews or, uh, you know, tweets. Now, I guess it's called X, so X posts. And the way you would evaluate such outputs would be by typically using traditional classification metrics. Um, so you have accuracy, which is, you know, how many-- what, what is the percentage of the observations that you correctly predicted? But you also have two key metrics, which I'm just going to, uh, remind. Uh, I'm not sure if everyone knows about them. So one is precision, which is out of all the positive predictions that you made, which ones were correct? And then the second one is recall. Out of all the true labels, how many of them did you correctly predict as being positive? And, uh, you have, uh, this metric called the F1 score, which basically takes the harmonic mean of precision and recall to just give you one number. So now you may wonder, you know, why do you need all these metrics? So the short answer is that sometimes you have tasks and data sets where your classes are very imbalanced. So for instance, you can have, I don't know, ninety-nine percent of your data set that is a positive label and then only one percent of the data set which is negative. And so here, if you take like a metric like, uh, accuracy, it would be very misleading because if you have a model that would predict everything as the majority class, then you would have a, you know, great classifier, but that's not the case. So that's why precision and recall really play a role. So that's, that's for the first one. Okay, so now let's move to the second category of NLP tasks. So this, this one is the multi-classification category. So you have an input text, and you predict multiple things. And, uh, we're s- illustrating this with the NER task, which as I mentioned, is about identifying the category of given words. And so here, for instance, we want to identify, uh, teddy bear as being an entity. Um, I guess for that you would use classification metrics, but not at the sentence level, but more either at the token level or at the entity type level. And by, by that I mean, let's suppose you have a category, um, let's say location, and you want to know how well you're predicting words in that category. So you would typically aggregate these metrics, um, as a function of that. Cool. Okay, let's go to the last category, which is, uh, as I mentioned, the most popular one. So this one is text in, text out. So I'm illustrating this with the machine translation task, which is around translating a text from a source language to a target language. So here you have the example with English to French. So, "Cute teddy bear is reading," uh, "L'ours en peluche mignon lit." Um, so for that, I guess it's harder to get datasets because here you, you need to have pairs of text. So you have a very popular dataset that's called WMT, which stands for Workshop On Machine Translation, and that one contains a bunch of paired sequences in different languages. Uh, so for instance, you have the English French, English German coming from the European Parliament's dataset, for instance. Um, okay, so to evaluate those, the-- to evaluate the performance of your model, it's actually a lot more tricky because as you can imagine, you can have many different ways to translate something. I'm sure many of us in the room, uh, are, you know, bilingual, triling-trilingual. Um, so yeah, that's, that's what is making it this hard. So in the past, people have used several rule-based metrics to do that. So one that you may have heard is BLEU. BLEU stands for Bilingual Evaluation Understudy, and it is a measure of how well your translation stands with respect to a reference text. Same story for ROUGE, which, uh, is actually a suite of metrics, um, but kind of captures that in a different way. And, uh, you will see that the machine learning community is, uh, funny because, uh, bleu, I'm not sure if you know, in French means blue, but rouge means red. So I guess they tried to kind of, uh, uh, add some, some fun in this. Um, but the problem with these metrics is that you always need a reference text. So you basically need labels. And in practice, having labels is very cost expensive. It takes a lot of time, a lot of money to, uh, get, uh, labels. And we will see later in the class that with, uh, the progress that we have made in the LLM space, or that the community has made in the LLM space, we can actually forego of these reference-based metrics and go towards a more reference-free, uh, kind of metrics, and we will see that later on. And then the last metric that I will say that people, uh, sometimes use is called perplexity, and perplexity only looks at the probabilities that are output by the model, and it basically quantifies how surprised the model is by its outputs. So BLEU and ROUGE, the higher the better. Uh, perplexity, the lower the better. And I guess, um, LLMs have been a kind of a hot topic since twenty twenty-two, but actually the field goes way back, way before that, uh, that year. So in the eighties, w-we'll see it in a second, but, uh, there's a class of models that were actually kind of thought of even in the eighties, and the nineties we had, uh, LSTMs that we'll see also in a second. Um, but the problem was, during that time, we didn't have the internet, we didn't have a lot of compute, and I guess this was one of the limiting factors which prevented these models from-- like the n-- uh, the models from today from being trained. And then more recently, uh, we've had several advances. So Word2vec was really one of the, uh, kind of pioneering, um, work in just, um, computing meaningful embeddings, and we'll see it in a second. And then, of course, we had the Transformers, which were, uh, part of a paper that was published in twenty seventeen, which is basically at the foundation of, of the models that you see today. And then, uh, you know, these models, they just, uh, were scaled up both by compute, uh, but also in, in terms of like the data that was used to train them. And, you know, that's how LLMs were dubbed and I guess these are more like the twenty twenties. But yeah, I guess we'll see those. Cool. Any questions on, I guess, the high level? Everyone good? Cool. So

4. 22:57 – 30:28

   Tokenization

   1. AAAfshine Amidi22:58

      I guess the first question that I want to ask ourselves is, what we want to do is to have a model that handles text. But models, they understand numbers, they don't really understand text. So we need to somehow do something with that text to make it more quantifiable, something that a model can understand. So if you look at a sentence, for instance, a cute teddy bear is reading, you first need to ask yourself, how can you cut this sentence to pass it to a model? So this part is called tokenization, and what it entails is basically cutting the text with respect to some arbitrary unit of text. So there are several ways of doing this. I guess the first way is doing it completely arbitrarily. So here, for instance, you would have A, that would be one unit of text. Cute would be another unit of text. Teddy bear would be another one, and so on. And by the way, the unit of text is called a token, which is why, uh, the method, method is called tokenization. Another way would be to just separate by words. But I guess, uh, we would have, you know, uh, always pros and cons. Uh, I guess, um, one of the goals that we want to achieve is for us to then be able to represent these tokens in a meaningful way. So one con with doing this at the word level is you will end up with words that look similar, but they are actually considered as different tokens. And I guess the limitation here is you will need to compute embeddings for these similar yet different tokens and somehow make their embedding similar. So I'll give you an example. So let's suppose I have the word bear, and then you have another word, plural, plural form, bears. So these two words, they are very similar, just one is singular, the other one is plural. If we go ahead with the word level tokenization, then we will end up with just two different entities which are basically, yeah, just considered as different. Same with run and then runs, you know, variations of verbs. So for that reason, people have, uh, kind of, um, dug into a category of token- tokenizers that are called subword tokenizers, which is around leveraging roots of words in order to find what are the common roots that we can find is in these words. So for instance, for bear and bears, you would have the bear particle that would be, uh, kind of shared. And so I guess the pro is that you get to leverage the root of the words, but then the con here is that your sequence would be longer, and we will see why this is a con, um, I guess later on. I guess I can give you, um, a preview. Um, so the complexity of these models is also a function of the sequence length. So the more you ha-- the more tokens you have to process, the more time it will take for your models to-- model to run because it needs to basically process all these tokens. So that's one con. So pro is it leverages the root of words. Con is it just makes your sequences longer. Okay, you have a last category of, uh, kind of ways of tokenizing things, which is just going at the character level, just like taking all characters. So here, I guess, uh, you and I, when we write, I don't know, a message, we typically have, uh, sometimes misspellings, and with the subword, uh, way of tokenizing things, you may not, uh, be able to recognize a word that has been misspelled. And this is something that the character level tokenizer can, uh, I guess, take into consideration. But here the problem is you have a sequence length that's much, much longer, which will make your model, I guess, take much more time to process, uh, the sequence. So that's one con. And then, uh, the other con is, I guess when you want to represent each of these tokens, I guess it's very hard to know what a representation of a letter really means. Like, what does the representation of the letter U mean? Very hard. Cool. So I have, uh, just a, a quick recap. So word level is a super naive way, super simple way of, um, I guess, dividing your text into arbitrary units. Um, but then the problem is, as we mentioned, we do not leverage the root of words. And, uh, I did not mention this, but, uh, there is a term whenever you cut something and then, um, at inference time when you want to make a prediction, I guess one, uh, prerequisite that you have is that you need to have the token that you saw at training time. You need to have it in your training set. And the problem is, let's suppose at inference time you cut your text into words, and let's suppose you have not seen a word at training time. You will need to mark it as unknown. And so this thing is called OOV, out of vocabulary. So luckily, the subword level tokenizer mitigates that problem. Uh, so you have, uh, like, um, a lower risk of, uh, OOV, but still you, you can have. And as we mentioned in terms of the pro, uh, you leverage the roots of the words. Um, and then, uh, character level, uh, you know, it's robust to our misspellings and our casing errors. Uh, but the problem is it makes computations just, like, much slower and, uh, your sequences would be, like, very, very long, uh, which will also make your, I guess, inference time much higher. That sound good? I guess this is really the foundation of, uh, I guess how to handle things with text, but yeah. Does that make sense overall? Cool. Okay, so now-- Okay, what we did is we took an input text. What we did is we cut it into parts that are basically tokens. So in order for our model

5. 30:28 – 53:23

   Word representation

   1. AAAfshine Amidi30:28

      to understand it's-- these tokens, we need to find a representation for each of them. So here, uh, we're going to take a look at this. So that's called word representation or more, um, I guess, in a more correct way it should be token representation. So we want to find a way to represent each of these tokens. So the simple and naive way to do this would be to just assign a one-hot vector for each word or for each token. So for instance, let's suppose if-- let's suppose we have a vocabulary of three tokens: book, soft, and teddy bears. We would have, let's say, soft that is a one, zero, zero vector, teddy bear that is, let's say, a zero, one, zero vector, and book that is, let's say, a zero, zero, one vector. So this is called a one hot encoding, OHE, we typically see. So cool. Yeah, this is a, a way to represent of our tokens but, um, basically what people want to do is compare these tokens to basically see which ones are more similar to what other ones. So common similarity measure that people use is something called cosine similarity. Not sure if you have heard of it. Um, so we can think of it as just seeing what angle these vectors make in the n-dimensional space. And if, I guess, they are pointing in the right-- in the same direction, then maybe they're similar. Maybe if they're orthogonal, maybe they're kind of independent. And if they're n-- completely opposite, then maybe they're opposite. That's basically the mental model we want to, uh, go into. So the problem is, if you represent your tokens in a one-hot fashion, you will end up with all your vectors being orthogonal to one another. So that's a problem. So ideally, what we want is for tokens that mean the same or similar to basically have a high similarity. And for tokens that are not similar, like, on, like, about different things, to be more like orthogonal. So here I-- just, uh, for illustrative purposes, uh, teddy bears are, are soft, so you want teddy bear and soft to be, I guess, with a high similarity. And let's say teddy bear and book, which is kind of independent, you want them to be closer to zero. So that's what you want. That's what you have with one-hot encoding, and that's what you want. Yeah. Uh, sorry? Oh, I see. Uh, the question is, why do you care about the norm? Um, so I guess cosine similarity is actually normalized by the norms. So it's, um, dot product. Oh, you mean why did I just put dot product here instead of do... Oh, I see. And your question is, why do we not care about the norm? Um, uh, cool. I guess, uh, the viewers know the question. Um, I guess these measures, they're all, you know, measures. They're all ways to try to capture these, uh, kind of similarity, uh, things. Um, so I guess why do you not care about the norm? Hm. I guess it's how people have tried to kind of quantify that. Um, I, I guess you will need to see how your vectors are trained and whether the norm would be indicative of something. Um, I guess the best answer I can give you is, uh, I guess this is a measure. This is not the perfect measure. Um, yeah, people may use also that product as a, as a measure. But yeah, I don't have, like, a great answer for you. Cool. But as long as you capture, I guess, how these vectors, they're, um, they're pointing, I guess typically what you care about is the angle between them. Um, but yeah. Typically, you don't really take into consideration the, the norm. Cool. Any questions? Any other questions? Yeah. Yeah. Yeah. Yeah. It's a great question. So question is around size of vocabulary and how that would inform the choice with respect to word, subword, and how that changes across languages. It's a great question. Um, so I would say it really depends, first of all, on the task that you're trying to achieve. If your task is just about one language, you will just, uh, take that same language. You would typically go with a subwords tokenizer just because of the reasons that we mentioned here. Um, so I guess subwords is a nice trade-off between being able to identify, uh, words by their roots, like leveraging that, but also running less into the OOV, uh, risk. Um, so, uh, in terms of the size, I know that people they've, uh, you know- Like, try different things. I think, uh, typically for English, you would target something on the order of tens of thousands of, uh, vocabulary size. Uh, but, you know, like, nowadays, the models, uh, they are multilingual, they're also about codes. So you will see that the vocabulary size now is sometimes on the order of, of hundreds of thousands. Uh, okay, so with respect to Chinese, so I guess you have this, uh, you know, difference in characters that you're using. So for Latin, I guess, so it's the alphabet we're all accustomed to, but of course, for the other ones, uh, you'd have something similar, but in, uh, I guess the, the target language character. Um, so yeah, I would say order of magnitude, tens of thousands for one language, hundreds of thousands, if it's like multilingual. Um, yeah, these are the order of magnitude that you want to target for. Cool? Yeah. Great question. So the question is, how do you get those embeddings? So it's actually the next slide. So we're going to talk about this. Cool. Great. So okay. So now that we know that the one-hot encoding is not a, a good way to represent tokens, what we want to do is to learn those embeddings from the data. So I mentioned that there was this, um, you know, uh, paper that came out in the twenty tens, so I think it was twenty thirteen, that was called Word2vec. And the reason why it, it was so popular is because they showed a very intuitive and interpretable way of seeing these embeddings, uh, 'cause they were saying, saying something like, okay, king is to queen what this is to that, like Paris is to France, what Berlin is to Germany. So there was basically a way to make sense of the embeddings. So now the question is: how did they do that? So they had two ways of computing these, uh, embeddings. So one way was called Continuous Bag of Words, the other one was called Skip-Gram. But they all rely on the same idea, which is, let's just leverage text that we have and then try to predict something that is part of the text based on, let's say, the context. So for instance, Continuous Bag of Words, the goal is you take into consideration the words that are around a given target words, and your goal is to predict that target words. And Skip-Gram is kind of the opposite. You go from, uh, a target word, and you want to predict the words that are around it. So I guess this task is commonly called a proxy task, 'cause at the end of the day, in this exercise, what we care about is not necessary to predict the next word, or at least not yet. Our goal is to learn a representation of these words that are meaningful. And so here, the idea is, if you have a model that somehow knows how to predict, let's say, the next word, then it means that your model has some understanding of how language works, which is basically what you want. You basically want an embedding that is reflective of, uh, I guess what language is, which is, uh, you know, king and queen are, you know, are similar. You know, Paris and France, like this is a capital. You, you want to have these associations embedded in the representation. And let's go through a very simple example of what that looks like. So here in our example, let's suppose that our proxy task is about predicting the next word. So here, what we take is a very vanilla neural network model, which basically receives a vector of size V, has some multiplication and a bias term to get, uh, like a hidden state, and then, uh, another set of, uh, multiplications to get our, uh, final vector. So here it's basically a, a very simple neural network. Uh, so the input is of size V. The hidden layer is of size D, which is typically much smaller than the vocabulary. So vocabulary is typically like tens of thousands or hundreds of thousands. So D is typically hundreds. Like seven sixty-eight, for instance, is, is one example, uh, of dimension. So it's much, much smaller. So what we're trying to do is to learn the word representation through this proxy task. And what we're going to do is try to consider the words as inputs and predict the next word. So let's go with the first word of the sequence. So by the way, I use token and words interchangeably. Um, so let's suppose we have the word A, and we want to predict the next word, which is the word cute. So what we do is we take the word A, we take the one-hot encoding representation And we pass it through the network. So here, uh, if you're familiar with neural networks, so here you have, uh, I guess, a multiplication between, I guess, a matrix and, and this vector. So you have a hidden state representation, which is a vector of size D. So here, let's suppose it's point two and point nine, so D equal two. And then you have, I guess, another pass here. Uh, and then you get after softmax a set of probabilities which are around seeing what is the next word. So in this example, we have, uh, a vocabulary of size six. So the first word is predicted with probability point two, second word point four, and then the other words are all point one in this example. So let's suppose that we want to somehow be able to maximize our prediction to be the second word of the vocabulary, which is the point four. So we basically compare the prediction with, uh, I guess, zero one zero zero zero, which is a representation of the second word of the vocabulary. And then we, you know, do the backprop. We, you know, update the weights. Not sure if everyone is familiar with that part. Uh, but the idea here is once you obtain a prediction, you compute the loss, so typically cross entropy, which will determine how far off you are from the true answer. And based on that difference, you're going to update the weights in order to make your prediction closer to the truth. So that's what you do. And then you repeat that process. Let's suppose you take the word cute, which, as we said, is the second word in the vocabulary. So the one hot encoding representation is zero one zero zero zero. So you go through that, uh, you know, network. Uh, you have a hidden state, like the vector is point eight and point four. You do that again. And what you wanna do is to predict the next token. And here is teddy bear. And so you see now your model in this example is predicting the next word to be kind of like uniform, but you want to somehow maximize the probability for teddy bear. So you go about doing this again and again for all the words. And at the end of the day, you obtain a model that learns how to predict the next words, which is basically the proxy task. And what you're going to do is to take the representation that the model learns, which is the, the green units. So what happens now is every time you have a word, you just represent that and as a one hot encoding representation, and you just like multiply this with these weights, and then you obtain the green representation, and that's is your word representation. Does that make sense? Yeah. Yep. Yeah, great question. Yes, great question. So the question is about what does V correspond to and why there's only six. So yes, in this example, we only have six possible words, which is basically the vocabulary size. Just like very, um, kind of a toy example because in practice there's, uh, many more. Um, so I guess, uh, that's one of the challenges with language. So you can technically have many variation of words, which is why if you take a word level way to divide your text into tokens, you can end up with a vocabulary that's like very big because you need to account for all the variations of given words. Uh, and the other thing that I want to point out is, um, let's suppose you have a vocabulary size of six, and it's the six words that you saw at training time, but what, what happens is if at inference time you have a word that you have not seen at training time? And so the answer for that is typically what people do is they reserve a spot for what they call, uh, an unknown token or out of vocabulary token. Which is basically, um, you can think of it as, you know, a bucket for everything that we could not ha-- we, we were not able to identify. So if let's suppose at inference time you have a token that you were not able to identify, they will all take that representation, which is the unknown token representation. Um, and it's, by the way, something that, I guess, the word level tokenizer has kind of trouble to do because you will have a much bigger chance of having out of vocabulary tokens. Word lev-- uh, subword level will have a lower chance. And then character level, uh, I guess you, yeah, you don't have that problem. Does that answer your, your question?

   2. SPSpeaker48:30

      Yeah.

   3. AAAfshine Amidi48:31

      Okay. Any-- yeah. Great, great question. So first question is, when do you know when you're done? So the thing with the proxy task is when you train your model, I guess your true objective is to not really learn, I mean, in this case, to learn how to predict the next word. Your objective is to have meaningful representations. But what you can do is to somehow track the loss function for the proxy task that you're pursuing, but then also taking into consideration that this is not necessarily your end goal. So I guess, uh, one, uh, you know, very reasonable way of going about doing this is just to wait until your model converges. So here what you do is you track the loss as a function of... So there's this term epoch, just how many times your model sees the training set, and, uh, so you compare these, these different curves, and when this converges, this is typically like a good time to, you know, stop, stop the training process and, like, just see if that makes sense. Depends on your downstream task, of course. But that's one. Um, okay. So your second question-- Sorry, can you repeat the second question? Yeah. Yeah. Oh, great question. So, so the question is, how do you know when the generation stops? I guess, like, otherwise it, it will stop, uh, it will never stop. So yeah, exactly. So you have some special tokens. Typically, you have end of sequence, uh, end of sequence. So typically, when you have the end of sequence token generated, then it's when, when it stops. All right. So second question was, um, what informs the size of the hidden layer? Um, I would say it's a trade-off, uh, because you want the embedding to be rich enough that it can be informative for your downstream task. So for instance, if you want to somehow get an embedding of, let's say, your sentence, um, and if you want to, let's say, do a very, very specialized task, super, uh, you know, um, like with a lot of different outcomes, maybe you want a vector that recaptures that. So maybe you want a bigger vector. But if you had like a very simple task, maybe a smaller vector make-- makes sense. Um, so I guess the size of your hidden dimension also impacts the complexity of whatever you're running after. 'Cause, of course, if you have longer vectors, you'll have more computation, so your inference would be probably more, uh, expensive, et cetera. So I guess there's a lot of factors. So I guess just to recap, one is how complicated your downstream task, task is. Second one is, uh, how sensitive are you with like latency, cost, all these things. Uh, so it's really a trade-off. But out there, you would typically see embeddings of, you know, uh, oof hundreds or thousands. Um, of course, you know, these models, uh, they've been growing, so number may change, but that's the, the order of magnitude that you're looking at. Right. This is indeed empirical. Yeah. Yeah. I guess you can also rely on, like, what others found, uh, and just go from that. Yeah. But seven sixty-eight and... Yeah, like, these numbers are, are things that people typically take. Yeah. Cool. Yep. Good question. So, um, the question is, how can you distinguish words that are spelled the same but in different contexts? So you're way ahead of me. Uh, so this is basically the basics, and we're going to tackle, uh, methods that can tackle these problems of just contextualizing the word in the sentence. Uh, so yeah. So w- we'll see that in a, in a bit. Cool. I'm not, uh, uh, I'm not on time, so I'll, I'll try to get moving. Um, so okay. So

6. 53:23 – 1:06:47

   Recurrent neural networks

   1. AAAfshine Amidi53:23

      now what we did was see how we could learn representations of tokens. But I guess you may also want to get representations of, you know, sentences or, you know, pieces of text. So one very naive way to do that with what we saw before is to take something like the average of words, let's say, the word representations. But the problem is you, you know, lose a lot of, uh, meaning, you lose, uh, the order, you lose, um... And I guess here, I think you pointed out very well, uh, the representations that you learn are token specific regardless of where they're at. So that's why we have a class of models that aim at capturing the sequential nature of, you know, how text appears. So we're going to talk about RNNs, which stands for recurrent neural network. So what RNNs do is instead of processing words one at a time, what they do is they keep a hidden representation of the sentence so far, and they consider tokens one at a time. Uh, so as I mentioned before, this technique was actually u-- uh, introduced, uh, like a fair amount of time ago, so in the '80s Um, and what this model does is it takes into consideration the order at which words appeared or tokens appeared. And so in this example, um, you start the, I guess, processing at the very beginning of the sentence. You have some, uh, dummy, um, hidden states that is called A, typically noted A or H. It's called a hidden state activation or even sometimes a context vector. And you have some kind of a mo-module that takes into account the hidden state so far and the word at time step T. So here, time step one. So here, what it does is it takes in the meaning of the sentence so far and takes into consideration the words that is happening now, and it produces an output vector that here can be used to, you know, try to predict the next word. So for instance, here we have this hidden state and this, um, uh, the representation of the words that then, you know, you have some matrix multiplications in this, in this blue box, and you have an output vector that you try to, you know, train on predicting the next words. And then you keep on doing that by, by keeping this hidden state, keeping track of this hidden state. And so you repeat the process. And I guess the way you would interpret this hidden state is it's a representation of the seq-sequence process so far. So the good thing with RNNs is now the word order matters, and you're also able to encode the sentence in a more natural way. So let's see roughly how it works. So we have the same, uh, favorite example. So a cute teddy bear is reading. So you would have the token A, you find the one hot encoding vector, you pass it through your network, you compute the hidden state, you try to predict "cute," but then you keep track of the hidden state. And then you input that into another, uh, module, and then, uh, you also consider the next words. So you consider not only the word itself, but also the hidden state of, uh, I guess the sentence so far, and you try to predict the next word again and again and again. So this is RNN. So RNNs were used for a bunch of tasks and just like mapping that back to the categories that we saw before. For classification purposes, you can basically use the hidden state of the last word in your sentence. For instance, if you want to predict, uh, you know, the review, like the sentiment of a review, you would take basically the, the last, last vector here and try to project it into the space of, uh, the predictions or the labels that you want to predict on. So for instance, if you want positive or negative, we basically project that vector into that space. You can do that here. For multi-classification, so you would basically have the representation of the token of interest, and you would project that. Or for generation, you would basically process the whole source text and then have, uh, kind of a, the context vector, aka activation vector, aka hidden state at the end of your processing, which will then be used to decode the output prediction. So this is how you would use an RNN, uh, for like each of these tasks. So the reason why you have not really heard of RNNs like this, like these days is because they had, uh, you know, some pros, but a lot of cons. So one of the cons is that the meaning of the sentence is basically solely encapsulated into this hidden state. So you have this problem of long ra-- long range dependencies, which basically, um, like impacts your ability to, quote-unquote, "remember" what the model saw in the past, which is why you have another class of models that try to build on RNNs. So this one is called LSTMs, Long Short-Term Memory. And the goal of that extension is to have a way to somehow keep track of the things that are, quote-unquote, "important" to remember on top of the hidden state that we talked about. So here you have, uh, A of T, which is, uh, you know, your activation. Um, like basically the, the, the sequence so far encoded in there. And then you have another quantity that you track that is called the cell state. It's, uh, noted C here. So this architecture aims at, I guess, improving that piece, but I guess it's-- it was not perfect either. But yeah, so that was, uh, like the main, um, issue of RNN-based methods, which is That they have this issue of kind of forgetting what was in the past. So you will see in the literature that this phenomenon is called vanishing gradient. And the reason why it's called that way-- So I know we're kind of running out of time, but I'm going to just explain that part. So in order for you to predict, let's say, the last words, you're basically dependent on every hidden state that came before that. So far so good? And so whenever you want to update the weights of your model to match the prediction here with the actual prediction, when you do the backpropagation, you somehow need to take into account that the last-- the value here is basically not only a matter of this computation, but also this computation or this computation that basically happen in a sequential manner. So you have this kind of, uh, um, this phenomenon of trying to, I guess, backpropagate through time. But the problem is, in practice, when you kind of write that down, so it's a very ugly formula. But when you write that down, it ends up being a product of, of a bunch of quantities that can-- So if it's greater than one, then it's excluding. If it's less than one, it's, uh, vanishing, 'cause if you multiply a lot of things that are less than one, it just goes to zero. So I guess if you have something that you're trying to update that goes to zero, you basically have trouble just like, you know, doing your updates. So that's a high-level intuition. This is not the focus of this class, which is why I'm not going into the detail of these ugly formulas, but I hope you get the idea that, uh, for kind of remembering things from the past, it, it's not doing a great job because of this, uh, sequential, um, I guess, uh, characteristic. Does that make sense? Okay, I hope the next thing will make a bit more sense, but before that I'll just recap what we saw. So our goal is to represent text. So we first started with represented-- representing words or tokens, which was what we tried to do with Word2vec. And we saw that, you know, it was a good way to leverage, uh, you know, proxy tasks to learn this representation. But we had a, a bunch of, uh, limitations, and one of them that you mentioned was that this was not aware of the context, uh, and also the word order didn't count. Um, and so you have this, uh, other class of, uh, methods that is able to take into consideration, uh, the words, but then, uh, they have some trouble keeping track of things when the sequence gets very long, and you have this problem of vanishing gradients or long range dependencies. So whenever you see this term, it's basically referring to that. And also another thing that I ha- I have not mentioned, but the computations are very slow. So when you want to train these models at training time, in order to predict this word, you basically need to compute all the hidden states before. So when your s- when your sequence gets very long, it just gets, uh, like it just takes a l- a very long time. So for all of these reasons, for, for, I guess all these reason-- W-what reason? So for the fact that, um, the model has trouble remembering things from the past, people have tried having more direct connections between something and the thing from the past, and this is the idea behind attention. So what attention does is it tries to have a direct link between what we're trying to predict and something from the past. So in this example, let's suppose I'm trying to translate an English sentence into a French one. So here, I guess the input sentence is given. I'm, you know, computing the hidden states. You know, I'm processing words one at a time. This is my traditional RNN. So, "A cute teddy bear is reading." So here, I have a hidden state that I'm then decoding. And you can imagine that when wanting to generate the next word of my translation, it would be great if I knew what word I'm trying to predict. Or in other words, it would be great if I could take a peek at a certain area of the input text. So the idea behind attention is to have a direct link between what you're trying to predict and things before. This is the idea behind attention. Um, and so it was introduced in twenty fourteen and, uh, yeah. Again, this is trying to kind of solve for these long range, uh, dependencies issues. And so yeah, this example want to do that. And this

7. 1:06:47 – 1:13:53

   Self-attention mechanism

   1. AAAfshine Amidi1:06:47

      concept is going to actually be key for this class 'cause we're going to see that the attention mechanism is the thing that is going to make everything, I mean, most of the things work. And this is actually the main principle that the transformer paper relies on. So the transformer, which is the core architecture that we will see in this class, has been introduced, like, or was introduced in twenty seventeen in this paper named Attention Is All You Need. So even from the title, you can see that, you know, the authors wanted to just rely on that, on that part. So what the authors tried to do was to move away from this sequential way of processing the text, and instead let the model just have direct connections with all parts of the text at once. So that is called self-attention. Uh, so they tried that on translation tasks, and they just realized that, you know, it was, uh, giving great results. So back to the example that we're still using. A cute teddy bear is reading. Here what we would say is that in order to compute the representation of the token teddy bear, we're going to look at all the other tokens in the sequence at once and directly, with direct links. So I guess back to your question. Here, we would have a representation of teddy bear that would be unique to the context that it is part of. So back to your question about river bank and, uh, uh, robbing a bank. Like, here, the bank would have, um, different representations. So this is the idea. I guess, does the idea roughly make sense? And again, this is called the self-attention mechanism. Shervine, how am I doing on time?

   2. SPSpeaker1:09:02

      Late.

   3. AAAfshine Amidi1:09:03

      Late? Okay. [laughs] Cool. Um, so... Okay. So this is the idea. Uh, so now I'm going to just introduce another set of ideas, which is more terminology, but it's going to be very important. So when you want to express something in terms of something else, we use the words key... Sorry, query, key, and value. Q, K, and V. So in this example, our goal is to figure out what other tokens is the query teddy bear more similar to. So here the question is, okay, you have a query, and you want to see what other tokens are most similar. And so what you're going to do is to look at all the other tokens, which are basically composed of keys and values. So you're going to compare the query to the key to quantify how similar your query is to a given key and take the corresponding value. So we'll see that in this example. So let's suppose you want to express teddy bear as-- in terms of everything else. What you're going to do is you're going to take the query teddy bear, and you're going to compare that query with all the other keys to see which element is most similar and then weigh the more similar ones and take their associated value. So that's a very high level, uh, idea of, I guess, how these things are. Of course, we're going to see exactly how they work, but that's the general idea. Okay, cool. And speaking of query, key and value, we will also see that one benefit of expressing things this way is that we can express doing the self-attention computation across the whole sequence in a matrix format. And GPUs love matrices, so it's really, like, made for the hardware that we have. Um, and I guess what I mentioned here can be expressed in a form of softmax of the query and the key, which is basically a way to get, uh, some kinds of weights of which values will be more important. So for instance, if a value is more important, you'll have m- bigger weights, and another one will be less important, you'll have a smaller weight, and you basically, uh, multiply that by the values. So don't worry, we'll have a, a detailed example after. So if it still feels very, you know, high level fuzzy, don't worry. We'll have a, a detailed, uh, walk-through. And yeah, so this is, uh, how it works. Okay, cool. Any questions on how self... Or what self-attention is? Yeah. Great. Great question. So the question is, uh, what is value? What is key? I guess, how do you get those? What do they mean? Um, so first of all, I just want to say that these quantities, they're learned, so you are not fixing them. But from an interpretation standpoint, you can interpret that the key is there for you to figure out which one is most similar to the query, and the value is the actual value That is associated with that, with that element. So here you will have something like you want to express this in terms of all the values, so the weights in your weighted average will be basically the dot product between-- basically between the query and the key, and the value will be the actual vector that you will use. But again, these, these things are learned and there's something I've not mentioned, but, uh, you mentioned it correctly. So we're going to actually do projections to obtain these quantities, and these projections are actually learned by the model. Sounds good? Okay, cool. Uh, so with that,

8. 1:13:53 – 1:29:53

   Transformer architecture

   1. AAAfshine Amidi1:13:53

      uh, we have, uh, fifteen minutes, right? To, uh, talk about the architecture. Um, okay. So at a very high level, in order to make the self-attention mechanism happen, the authors propose an architecture that is composed of two parts: an encoder, which is on the left side, and a decoder, which is on the right side. So the application that they have is translation. So what will go through the encoder is the input text in your source language, and what is going to go through the decoder is the target language that you're predicting. So the high level idea is you're going to compute meaningful embeddings from your input text by passing them through the encoder, and you want that self-attention mechanism to apply, meaning you want to compute representations of each token as a function of others, and you do that by using a layer called the attention layer. So multi-head attention layer, but multi-head is just doing this computation, uh, you know, uh, in different ways to just allow the model to learn different representations or different projections. But the idea here is you're going to input your input text, and all the tokens in your input text are going to attend to one another. So for instance, a cute teddy bear is reading. You are going to compute the representation of all the tokens in this text basically as a function of others, and you're going to do that with the encoder, so here with the multi-head attention. Uh, and then, uh, you have a feed forward, uh, layer, which is just to kind of let the model learn, uh, another kind of prediction. And what, what you're going to obtain at the end of your encoding process is rich representations of the tokens from the input sentence. So far so good? But now your goal is to actually translate the input sentence. So what you're going to do is to start your translation with, let's suppose, the beginning of sentence token. It's your first token. And what you're going to do is use all the representations from your input sentence in order to figure out what to predict next. So this, what I just said, is the cross, cross-attention layer, which is the, the one that is the second sort of... This one, which basically, you know, you see-- I'm not sure if you see the arrows, but there are two arrows coming from the encoder, one arrow coming from the decoder. Can anyone tell me what the arrow from the decoder stand-- like, represents? Is it query, key, or value? I guess there is one over three. Thirty-three percent chance. Who wants to try? [laughs] Is it key? Is it key? Okay. Hmm? Query. Query? Okay. So the way to think about it is you're trying to ask yourself what are the words from the input that matter, right? So basically you want to know, given your query, what are the elements from the input that matter? So here, this arrow is indeed the query 'cause this is the thing that you want to figure out, and the keys and values are actually coming from the encoder, which are basically coming from the input sequence. Um, and then you have another attention layer, which is this one, and that one is trying to figure out what other tokens of the output sentence that you're decoding is going to be useful to predict the next token. So let's suppose you, you know, start, uh, decoding and you say, uh, un ours en peluche, which is in French. Uh, to, to predict the next word, you want to basically figure out where the tokens translated so far that are going to be useful to predict the next word. So this is what this attention layer is about, and it's called masked because it only looks at the tokens translated so far. It does not look at tokens that were not translated because, of course, they were not translated, so there's no, no way. Like, on the right side of the, of the token that you're trying to predict. Cool. Um, so at a very high level, you have this attention layer, which is present in the encoder, which is present in the decoder, but it has several, I guess, use cases. So the attention layer here aims at computing embeddings from the input sentence as a function of themselves, and then the ones from the decoder. So the first one, the masked self-attention layer, aims at expressing something as a function of everything that has been decoded so far. And the second one, the cross attention layer, tries to express things as a function of what is-- what has been seen in the input. So here, given that you are having direct links to different tokens, you don't have this sense of order, right? Because in the-- on an RNN, you were basically expressing things, you know, one at a time, so you had some sense of the word order. But here you don't have it 'cause it's like a direct link, which is why you have position encodings, which are there to inform on the position of the word in the sequence. So we're not going to dig into that today, but I just want to call that out. So at a very high level, and we're going to see this in a detailed example, uh, what we do is in order to translate a sentence from source language to target language, we're first going to tokenize the text, so, you know, divide it into arbitrary units. We're going to learn an embedding for these tokens. So this is what the input embedding is about. Then we're going to ha-- to add some encoding with respect to the position. We're not going to talk about it to-today, but just good to note. And then we go through the encoder. So the encoder tries to figure out how to express things as a function of other things from the input. So it does that in the multi-head attention layer, and then it goes through a feed forward neural network, which is just a way to just project, uh, the, uh, the vectors to just, like, uh, have some more degrees of freedom, uh, to learn things. And then once you have these representations from the input, you're then going to start your translation. So you start with the BOS token, and what you're trying to do is figure out what the next word is. So you're going to see, okay, what are the words that were translated so far that are useful for a translation? So this is what the masked multi-head attention layer does. And then you have another attention layer, which is about expressing things as, um, as a function of what was in the input, which is the cross attention layer over there. And then you have a feed forward neural network, uh, to again, give some more degrees of freedom. And at the end of the day, you have a vector that you then go through softmax, and it just is a way for you to guess what is the next word. So you have a vector of size, vocabulary size, and you're going to use these values to determine what is your next word. Easy, right? Any questions on this? Yeah. Oh, right. That's a great question. So the question is what does head mean? So I guess I went too fast. I kind of ignored that part. But when you do the attention, self-attention computation, you basically make queries interact with keys and then take the corresponding value, but nothing prevents you from doing that several times. So the term heads is given to the projection matrices that you use to obtain the query key and value. And when you have several heads, what you're doing is you're allowing your model to learn different projections. So it's basically an additional degree of freedom for your model to learn different associations between your vectors. So it's a great question. Uh, so typically it will be noted lowercase h, number of heads, uh, and this is what this corresponds to. Does that answer your question? Cool. Very cool. Um, okay, we have a lot to discuss, but... Here, I have a slide actually for this. [laughs] So this is the multi-heads that you were mentioning. So we're basically running the self-attention computation several times in parallel, again, with different projection matrices that, uh, the model learns. So in case you have a computer vision background, it is similar to having multiple filters in your convolution. Uh, so it's very-- it's the same idea, but, um, I guess it's different here. Yeah. Great question. So the question is, uh, are the projections different? So we're not, typically not constraining things. We're just letting the model learn. But in practice, it just tends to learn different ways of saying the same thing. Um, so yeah, typically there is no constraint. Of course, you have, you know, papers that dig into, you know, how, how about if you change this? But typically, you don't have any constraints. Cool. Uh, great. Okay. I will just mention one trick, another trick that, uh, transformer, uh, authors use. So it's called label smoothing. Who has heard of label smoothing here? Okay. So, uh, one new thing here is in NLP, when you want to predict what comes next, there's typically more than one way. Like, when you say, "What a great day," "What a great lecture," "What a great book," "What a great..." There's always multiple, uh, choices. Like, there's more than one way of filling that gap. So label smoothing is a technique that tries to intuitively address that. And what it does is, instead of saying, "Predict this word a hundred percent, it's this one, there is no other words," what it does is it says, "Okay, predict this word, but there's a chance it's not this word." And in practice, what it does is it takes the one-hot encoding, and instead of saying it's a one, zero, zero, zero that you need to predict, it says it's, uh, actually one minus epsilon and then epsilon over V, uh, uh, minus one, I guess, um, is what you're trying to predict. So in practice, it's a, a method that tends to make your model be more unsure, like it would be less sure about this prediction, because you always tell it, "Okay, uh, try to predict this," but actually it's possible it's not, it's not the correct value. But in practice, the authors see that it tends to improve metrics like BLEU, which is a kind of proxy metric for translation tasks. So yeah, I think this method is, like, pretty general for NLP. So yeah, it's a good, good one to know. And with that, I think there's about twenty-ish minutes left. So yeah, Shervine is going to, uh, walk you through an end-to-end example. And with that, you... Yeah. The lu... Oh, you? So I guess here we can think of this as, um, something, I guess some quantity. It's not defined. So some quantity. Uh, and, uh, yeah, the delta is like the, like one hot if you want. Yeah. Something like this, yeah. It can also be a constant. Yeah. Yeah. Uh, so question is, uh, is there a, a relation with explore and exploit? It's an interesting one. Um, mm, so why would softmax give it for free, by the way? Right. But I guess it was still... So I guess at the end of the day, what you're trying to do is to compare your prediction with respect to the label. So I guess the question here is, do you want to compare with one, zero, zero, zero, or do you want to compare with something that is o- not one, zero, zero, zero? So I guess your softmax does not allow you to do that. Exactly, yes. So I'm not sure if this was super clear, but this is actually the label. So what we're trying to predict is not one, zero, zero, but we change the label in a way that makes the model, uh, I guess, predict something as less sure, I guess. Cool. Thanks. And with that, yeah, Shervine. Okay, great. Thank you, Afshin. And, uh, yes, so we saw basically how the transformer worked, and now we're going to piece it all together with the one specific example. Um, okay, great. So let's take our favorite example again. So a, a cute teddy bear is reading, and then we'll go,

9. 1:29:53 – 1:41:58

   Detailed example

   1. AAAfshine Amidi1:29:53

      um, like all together through each step. Um, so first we ta- we start with tokenization. Um, so as we said, we, we can use any, uh, arbitrary decomposition to decompose this into tokens. Um, and then as someone mentioned, you know, you need to have some way to indicate, uh, like the start and the end of a sequence. So typically, this is done with the BOS and EOS tokens, so you add them. Okay. So now let's focus on the composition of each token representation. So you have its embedding that is learned, and then as Afshin mentioned, in order to have an idea of what is the position of the words or the token, I should say, as part of the sequence, you have some added information that is in the form of a position embedding. And here, the original paper, uh, uses the convention of some sines and cosines that it adds additively to the representation. So it's like a element-wise addition. Okay, great. So now you have a position-aware embedding for your token, and you repeat that for each of your tokens. Okay. So now you can see all of these embeddings in the format of a matrix, which is of size D model, which is the size of your embeddings. And then the other dimension is the length of the sequence. Um, so typically N. So all makes sense so far? Any questions on the inputs? Okay, great. So now we will send this representation through the encoder. Okay. So as Afshin said, you have this, uh, concept of self-attention, and the way you perform self-attention is that you take this input and project it on three spaces. So you project it to the space WQ, you get queries. You project the same embeddings in the space WK, you get keys. And you do the same for values, you get your values. And then WQ, WK, and WV are learned by the model. They're like, basically projection matrices. So far so good? Okay. So now with all of that in mind, you can apply the formula that Afshin mentioned, that is the self-attention formula, which is softmax of QK transpose over square root of DK times V, uh, which gives you another matrix, uh, out of all of this. Now, let's pause for a second and look at how this computation is done in practice and, you know, what every step means. So let's look at Q. You know, when you compute Q, yeah, basically you project your embeddings into that space. What do you obtain? You obtain a matrix where each row represents a given query. When you say K transpose, it's basically the same kind of matrix, but transposed, where each column represents the key representation of each token. Now, let's mix them together with the matrix, uh, multiplication. So when you multiply each of them, you see that each row represents the projection of the query over each key, such that when you take the, like, matrix multiplication and get the softmax of all of this, you get a probability distribution of the projection of the query over keys for each query. Like, each line will have this. And yeah, I don't know if anyone asked the question regarding, you know, why do we scale by square root of DQ, uh, D-DK? So it could be DQ as well, because matrix multiplication, like the dot product here enforces the fact that DQ equals DK. And basically, what you see is that these dot products, as the dimension of key and queries grows, it will tend to grow as well. So you want to normalize, uh, these dot products, and this is why you divide by square root of, of the dimension of, of keys. Okay, great. And then now you have your softmax of all of this, and then you multiply it with the matrix V. And, um, this is what Afshin, uh, explained as having, uh, the query, uh, projected on the space of keys and then multiplied by the corresponding value. So the value is the representation of the corresponding, uh, key that we project on. So you end up with a weighed sum of values for each query. Okay, great. Does that make sense so far? Okay, awesome. And someone asked, you know, what is the multi-head stuff? So you are right. It's not just single one, one time that it's done. It's actually done H times. And, uh, what you obtain is, like, all of that is done in parallel, and at the end, you obtain H such matrices, and you concatenate them with respect to the columns. And at the end of this, you have another projection matrix that you call WO that will project all of this back to the original dimension of embeddings. So it's a way for a network to basically have a dimension invariant way to, bless you, to go from, uh, like the, the original dimension back to the original one. Any questions? Yep. So the question is regarding, uh, H. Is it possible to get the same result each time? And then if you concatenate the same, same thing, you know, will it be helpful? Uh, did I get the question right? So what makes it different? So it's the magic of gradient descents. So the network has an objective function at the end. It has degrees of freedom. Its incentive is to build a representation that will be helpful to learn the next word. So it doesn't have an incentive to copy the same thing or do the same mechanism, and this is why in practice you see the model converge towards building different representations that it can then concatenate and then project into something, uh, useful. So what, what makes it such that you don't have the same thing? Nothing. Like, you don't have any constraints, but the nature of the learning that you let the model have, uh, makes it do so in practice. Any other questions? Yeah, and that is a great question. I mean, it's, um, like, typically, yeah, gradient descent does wonders. Okay, great. So now that we have gone through the self-attention layer, um, you have another component that is the FFM, and I think there was a question just here regarding How to choose the dimension of the hidden layer with respect to the input and output. So when Afshin mentioned Word2vec, typically you have a smaller dimension than the input o-ou- outputs. But here, actually, the hidden layer is of a bigger dimension than input o- and outputs, and the rationale for that is that you want to have enough degrees of freedom for the model to learn useful representations. So it's a way to complexify the features that you learn, um, and yeah, interesting to know. Okay, great. And you don't have just one encoder module, you have actually N of them, big N, uh, in the original paper. And at the end of all of this, you have an encoded, uh, context-aware set of embeddings, and then each of these set of encoded embeddings will be those that will be fed to the N decoders. So you have, like, a stacked, um, like, a stacked, uh, like, succession of N encoders. You have a stacked succession of N decoders, and the last representation of the encoder is what you will feed to the cross-attention of each decoder. Um, so yeah, we're going to see that more in detail. So how do we even start the decoding process? So you start with the BOS token, basically saying to the model, you know, "Hey, we need to predict the next word. Uh, you know, let's start." So what happens to the BOS token at the very beginning? So you feed it to the decoder, and then similarly as before, uh, for the encoder, you have a self-attention layer. And as Afshin mentioned, the self-attention layer, um, is causal. So the attention will be done on the same token and the tokens that precede it. So on, on this first BOS token, you don't see a difference because it will just attend to itself. But when you have, uh, like, other tokens that you want to decode, you will have this difference in where you attend with respect to the encoder. Okay, great. Goes through the self-attention layer, and then you have what I mentioned to be the, um, cross-attention that takes as keys and values these encoded embeddings as inputs, and then the queries are those of the-- that come out of the self-attention layer. Okay, great. And then once you do this cross-attention, you have, just like in the encoder, an FFN component that, uh, makes the representation richer. Was there a question there? No? Um, and then at the very end... So you do that, all of that N times, and at the very end of the decoding process, you have a linear projection and a softmax layer to turn, um, the prediction of the next word into a probability distribution over the vocabulary. Okay, great. So we saw how to do that for the next word here, and, uh, basically, you do that again and again. So you have found your next token, and then, uh, which is like a one hot, one hot basically, um, encoding of what you want, and then you take that embedding and then put it back in the decoder and continue this process. And when do you stop? It's a question for you all. When you hit the EOS token, yeah. Yeah, exactly. Okay, great. And with this process, it's basically how the authors of this original, uh, like, landmark paper did machine translation, so this is typically the use case that was presented. Um, yeah. Any questions? Okay, awesome. And with that, thank you for your attention. [audience applauding]

Episode duration: 1:41:58