Stanford CS25: V2 I Introduction to Transformers w/ Andrej Karpathy

1. 0:00 – 0:47

   Introduction

   1. SPSpeaker0:06

      Hi everyone, welcome to CS25 Transformers United V2. This was a course that was held at Stanford in the winter of 2023. This course is not about robots that can transform into cars, as this picture might suggest. Rather, it's about deep learning models that have taken the world by the storm and have revolutionized the field of AI and others. Starting from natural language processing, transformers have been applied all over, from computer vision, reinforcement learning, biology, robotics, etc. We have an exciting set of videos lined up for you with some truly fascinating speakers, skip talks, presenting how they're applying transformers to the research in different fields and areas. We hope you'll enjoy and learn from these videos.

2. 0:47 – 3:19

   Introducing the Course

   1. SPSpeaker0:47

      So without any further ado, let's get started. This is a purely introductory lecture, and we'll go into the building blocks of transformers. So first, let's start with introducing the instructors. Uh, so for me, I'm currently on a temporary depo from the PhD program, and I'm leading AI at a robotics startup, Collaborative Robotics. We're working on some general purpose robots, uh, somewhat like a robot. And, um, yeah, I'm very passionate about robotics and building existing learning algorithms. My research interests are in reinforcement learning, computer vision, generative modeling, and I have a bunch of publications in, uh, robot autonomy, driving other areas. Uh, my undergrad was at Cornell. It's somewhere in the book Cornell. It's a nice to meet call.

   2. SPSpeaker1:33

      So I'm Steven. I'm currently a first year CSP speaker. Uh, previously did my master's at CMU and undergrad at Waterloo. Uh, mainly into NLP research, anything involving language and text. Um, but more recently, I've been getting more into computer vision as well as multi-modal. And just some stuff I do for fun. A lot of music stuff, mainly piano. Some self-promo, but I post a lot on my Insta, YouTube, and TikTok. So if you guys want to check it out. Um, my friends and I als- are also starting a Stanford Piano Club. So if anybody's interested, uh, feel free to email me or DM me for details. Um, other than that, you know, uh, martial arts, bodybuilding, and huge fan of K-dramas, anime, and occasional gamer.

   3. SPSpeaker2:14

      [laughs] Okay, cool. Yeah. So my name's Rylan. Uh, instead of talking about myself, I just want to very briefly say that I'm super excited to take this class. Uh, I took it the last time it was off... Sorry, to teach this class. Excuse me. I took it the last time it was offered. I had a bunch of fun. Um, I thought we brought in a really great, uh, group of speakers last time. I'm super excited for this offering. Um, and yeah, I'm, I'm thankful you're all here, and I'm looking forward to a really fun quarter together. Thank you.

   4. SPSpeaker2:39

      Yeah. So fun fact, Rylan was the most outspeak- outspoken student last year. And so if someone wants to become instructor next year, you know what to do. [laughs] Okay, cool. Um, let's see. Okay. So what we hope you will learn in this class is, first of all, how do transformers work? Um, how they're being applied, just definitely. And nowadays, like, we are pretty much using them everywhere in, uh, AI, machine learning, um, and what are some new interesting directions of research, uh, in these topics. Uh, cool. So this class

3. 3:19 – 3:35

   Basics of Transformers

   1. SPSpeaker3:19

      is just an introductory. So we'll be just talking about the basics of, uh, transformers, introducing them, talking about the self-attention mechanism on which they're founded. And, uh, we'll do a deep dive more on like, uh, models like BERT, GPT, stuff like that. So great. Happy to get started.

4. 3:35 – 5:01

   The Attention Timeline

   1. SPSpeaker3:35

      Um, okay. So let me start with presenting the attention timeline. Uh, attention all started with this one paper, Transform... uh, Attention is All You Need by Vaswani et al. in 2017. That was the beginning of transformers. Um, before that, we had the Prehistoric Era, where we had models like, uh, RNNs, LSTMs, um, and simple attention mechanisms that didn't work for scale at all. Uh, starting 2017, we saw this, uh, explosion of transformers into NLP, where people started using it for everything. I even heard this quote from Google. It's like, "Our performance increased every time we fired our linguists." [laughs] Uh, so that was 2018. After 2018 to 2020, we saw this explosion of transformers into other fields, like vision, um, bunch of other stuff, and, uh, like biology, AlphaFold. And it... and last year, 2021, was the start of the generative era, where we got, like, a lot of generative modeling. Uh, started, like, models like Codex, uh, GPT, DALL-E, stable diffusion. So a lot of, like, things happening in, uh, generative modeling. And, uh, and we are... we started scaling up in AI. Um, and now the present. So this is, uh, 2022, and, like, the start of 2023. And, uh, now we have models like, uh, ChatGPT, Whisper, um, a bunch of others. And we are, like, scaling onwards, uh, without stopping now. So that's great. Uh, so that's the future.

5. 5:01 – 6:10

   Prehistoric Era

   1. SPSpeaker5:01

      So going more into this, um, so once there were RNNs, uh, so we had sequence-to-sequence models, LSTMs, GRU. Uh, what worked here was that they were good at encoding history, but what did not work was they didn't encode long sequences, and they were very bad at encoding context. Uh, so consider this example. Consider trying to predict the last word, uh, in the text, "I grew up in France, dot, dot, dot. I speak fluent, dash." Here, you need to understand the context for it to predict French. And, uh, like, attention mechanism is very good at that. Whereas if you're just using LSTMs, uh, it doesn't, uh, work that well. Uh, another thing transformers are good at is con- uh, more based on content is... Sorry. Also context prediction is, like, uh, finding, uh, attention maps. If I have, uh, something like a... word like it, what noun does it connect to? And, uh, we can give a, like, a probability attention on what are the possible activations. Um, and this works better than, uh, existing mechanisms.

6. 6:10 – 7:30

   Where we were in 2021

   1. SPSpeaker6:10

      Okay. Uh, so where we were in 2021, we were on the verge of takeoff. We were starting to realize the potential of transformers in different fields. Uh, we solved a lot of, uh, long sequence problems like, uh, protein folding, AlphaFold, offline RL. Uh, we started to see few-shot, zero-shot generalization. We saw multimodal tasks and applications like generating images from language. So that was DALL-E. Um, uh, yeah, and it feels like ancient, but that was only like two years ago. And this is also a talk on transformer that you can watch on GitHub. Uh, which... Yeah. Cool. Um, and this is where we was going from 2021 to 2022, which is we have gone from the verge of taking off to actually taking off, and now we are seeing unique applications in audio generation, art, music, storytelling. We are starting to see reasoning capabilities like common sense, uh, logical reasoning, mathematical reasoning. We are also able to now get human enlightenment and interaction. We are able to use, uh, reinforcement learning and human feedback. That's how ChatGPT is trained to perform really good. Uh, we have a lot of mechanisms for controlling toxicity, bias and, uh, ethics now, and a lot of also, a lot of developments in other areas like vision models.

7. 7:30 – 10:15

   The Future

   1. SPSpeaker7:30

      Cool. Uh, so the future is a spaceship, and we are all excited about it. [laughs] Um, and there's a lot more applications that we can enable. Um, and, uh, it'll be great, uh, if you can see transformers also work there. Uh, one big example is video understanding and generation. That is something that everyone is interested in, and, uh, I'm hoping we'll see a lot of models in this area this year. Uh, also finance, uh, business. Um, I'll be very excited to see GPT author a novel, uh, but we need to solve very long sequence modeling, and most transformer models are still limited to like four thousand tokens or something like that. So we need to do, uh, make them generalize much more better on long sequences. Uh, we are als- we also want to have generalized agents that can do a lot of multitask, uh, multi-input, uh, uh, predictions like Gato, and, uh, so I think we will see more of that too. And finally, um, we also want domain-specific models. So you might want like a GPT model that's good at like maybe like, yeah, health. So that could be like a doctor GPT model. You might have like a lawyer GPT model that's like trained on only on law data. So currently we have like GPT models that are trained on everything, but we might start to see more niche models that are like good at one task, and we could have like a mis-mixture of experts. So it's like you can think like this is, uh, like how you'd normally consult an expert. You'll have like expert AI models, and you can go to a different AI model for your different needs. Um, there are still a lot of missing ingredients, uh, to make this all successful. Uh, the, the, the first of all is external memory. Uh, we are already starting to see this with, uh, models like ChatGPT, where, uh, the interactions are short-lived. There's no long-term memory, and, uh, they don't have ability to remember or store conversations for long term. Um, and, uh, this is something we want to fix. Uh, second, uh, we're-- second is reducing the computation complexity. So attention, uh, mechanism is quadratic over the sequence length, which is slow, and, uh, we want to reduce it or make it faster. Uh, another thing we want to do is we want to enhance the controllability of these models. Like, uh, a lot of these models can be stochastic, and we want to be able to control what sort of outputs we get from them. Um, and you might have experienced with ChatGPT, you just refresh, you get like different output each time, but you might want to have a mechanism that controls what sort of things you get. Uh, and finally, we want to align our state-of-art language models with how the human brain works. And, uh, uh, we are seeing this search, but we still need more research on seeing how they can be more important. Okay. Thank you.

   2. AKAndrej Karpathy10:14

      Great.

8. 10:15 – 10:39

   Transformers - Andrej Karpathy

   1. AKAndrej Karpathy10:15

      Hi. [laughs] Uh, yes, I'm excited to be here. Uh, I live very nearby, so I got the invite to come to class, and I was like, "Okay, I'll just walk over." Um, but then I spent like ten hours on those slides, so it wasn't as, as simple. Uh, so yeah, I'm gonna talk about, uh, transformers. Uh, I'm going to skip the first two over there. I'm not going to talk about those. We're gonna talk about that one. Just to simplify the lecture since we don't have time. Um,

9. 10:39 – 1:00:30

   Historical context

   1. AKAndrej Karpathy10:39

      okay. So I wanted to provide, uh, a little bit of context of why does this transformers class even exist? So a little bit of historical context. I feel like Bilbo over there. I joined, uh, like telling you guys about this. Um, I don't know if you guys saw Lord of the Rings. Uh, and basically, I joined AI in roughly 2012 in full course, so maybe a decade ago. And back then, you wouldn't even say that you joined AI, by the way. That was like a dirty word. Uh, now it's okay to talk about, but back then, it was not even deep learning, it was machine learning. That was the term you would use, uh, if you were serious. But now, um, now AI is okay to use, I think. Uh, so basically, do you even realize how lucky you are potentially entering this area in roughly 2023? Uh, so back then in 2011 or so, when I was working specifically on computer vision, um, your, your pipelines looked like this. Um, so, uh, you wanted to classify some images, uh, you would go to a paper, and I think this is representative. Uh, you would have three pages in the paper describing all kinds of a zoo of kitchen sink of different kinds of features, descriptors. And you would go to a poster session and in computer vision conference, and everyone would have their favorite feature descriptor that they're proposing, and it was totally ridiculous. And you would take notes on like which one you should incorporate into your pipeline because you would extract all of them, and then you would put an SVM on top. So that's what you would do. So there's two pages. Make sure you get your Sparse SIFT Histograms, your SSIMs, your Color Histograms, Textons, Tiny Images, and don't forget the Geometry Specific Histograms. All of them had basically complicated code by themselves. So you're collecting code from everywhere and running it, and it was a total nightmare. Uh, so on top of that, it also, uh, didn't work.

   2. SPSpeaker12:10

      [laughs]

   3. AKAndrej Karpathy12:12

      This would be, I think represents the prediction from that time. Uh, you would just get predictions like this once in a while, and you'd be like, you just shrug your shoulders like that just happens once in a while. Today, you would be looking for a bug. Um, and worse than that, every single, um, every single sort of field, uh, every single chunk of AI had their own completely separate vocabulary that they work with. So if you go to an- if you go to NLP papers, those papers would be completely different. So you're reading the NLP paper, and you're like, "What is, uh, this part of speech tagging, morphological analysis, syntactic parsing, coreference resolution? What is NP, VT, KJ?" And you're confused. So the vocabulary and everything was completely different, and you couldn't read papers, I would say, across different areas. So now that changed a little bit starting two thousand and twelve when, uh, you know, uh, Alex Krizhevsky and the colleagues, uh, basically demonstrated that if you scale a large neural network on large dataset, you can get very strong performance. And so up till then, there was a lot of focus on algorithms, but this showed that actually neural nets scale very well. So you need to now worry about compute and data, and if you scale it up, it works pretty well. And then that recipe actually did copy-paste across many areas of AI. So we started to see neural networks pop up everywhere since two thousand and twelve. Um, so we saw them in computer vision and NLP and speech and translation in RL and so on. So everyone started to use the same kind of modeling toolkit, modeling framework. And now when you go to NLP and you start read-reading papers there, in machine translation, for example, uh, this is a sequence-to-sequence paper which we'll come back to in a bit. You start to read those papers, and you're like, "Okay, I can recognize these words. Like, there's a neural network, there's a parameter, there's an optimizer." And it starts to read like things that you know of. So that, uh, decreased tremendously the barrier to entry across the different areas. And then I think the big deal is that when the transformer came out in two thousand and seventeen, it's not even that just the toolkits and the neural networks were similar. It's that literally the architectures converged to, like, one architecture that you copy-paste across everything seemingly. So this was kind of an unassuming, uh, machine translation paper at the time proposing the transformer architecture. But what we found since then is that you can just, uh, basically copy-paste this architecture and use it everywhere, and what's changing is the details of the data and the chunking of the data and how you feed in. And, you know, that's a caricature, but it's kind of like a correct first-order statement. And, uh, so now papers are even more similar-looking because everyone's just using transformer. And, uh, so this convergence is-- was remarkable to watch and unfolded over the last decade, and it's pretty crazy to me. What I find kind of interesting is I think this is some kind of a hint that we're maybe converging to something that maybe the brain is doing because, uh, the brain is very homogeneous and uniform across the entire sheet of your cortex. And okay, maybe some of the details are changing, but those feel like hyperparameters of like a transformer. But your auditory cortex and your visual cortex and everything else looks very similar. And so maybe we're converging to some kind of a uniform powerful learning algorithm here. Uh, something like that I think is kind of interesting and exciting. Okay, so I want to talk about where the transformer came from briefly historically. Uh, so I want to start in two thousand and three. I like this paper, uh, quite a bit. It was the first sort of, um, popular application of neural networks to the problem of language modeling, so predicting, in this case, the next word in a sequence, which allows you to build generative models over text. And in this case, they were using multi-layer perceptron, so a very simple neural net. The neural nets took three words and predicted the probability distribution for the fourth word in a sequence. Uh, so this was, uh, well and good at this point. Now, over time, people started to apply this to a machine translation. So that brings us to sequence-to-sequence paper from two thousand and fourteen that was pretty influential. And the big problem here was, okay, we don't just want to take three words and predict the fourth. We want to predict how to go from an English sentence to a French sentence. And the key problem was, okay, you can have arbitrary number of words in English and arbitrary number of words in French, so how do you get an architecture that can process this variably sized input? And so here they used a LSTM, and there, there's basically two chunks of this which are covered by the slack, by the, um, by this. But basically have an encoder LSTM on the left, and it, uh, just consumes, uh, the wor- one word at a time and builds up a context of what it has read, and then that is-- acts as a conditioning vector to the decoder RNN or LSTM that basically goes chunk, chunk, chunk for the next word in a sequence, uh, translating the English to French or something like that. Now, the big problem with this that people identified I think very quickly and tried to resolve is that there's what's called this, um, encoder bottleneck. So this entire English sentence that we are trying to condition on is packed into a single vector that goes from the encoder to the decoder. And, uh, so this is just too much information to potentially maintain in a single vector, and that didn't seem correct. And so people were looking around for ways to alleviate the attention, uh, sorry, the, um, encoder bottleneck, as it was called at the time. And so that brings us to this paper, uh, Neural Machine Translation by Jointly Learning to Align and Translate. And, uh, here, just quoting from the abstract, "In this paper we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of the basic encoder-decoder architecture and propose to extend this by allowing, uh, the model to automatically soft search for parts of the source sentence that are relevant to predicting a target word," um, yeah, "without having to form these parts or hard segments explicitly." So this was a way to look back to the words that are coming from the encoder, and it was achieved using this soft search. So as you are decoding in the, um, uh, the words here, while you are decoding them, you are allowed to look back at the words at the encoder via this soft, uh, attention mechanism proposed in this paper. And so, uh, this paper I think is the first time that I saw basically, uh, attention. Um, so your context vector that comes from the encoder is a weighted sum of the hidden states of the, uh, words in the, um, in the encoding. Uh, and then the weights of this sum come from a softmax that is, uh, based on these compatibilities between the current state as you're decoding and the hidden states, uh, generated by the encoder. And so this is the first time that really you start to like, uh, look at it and, and, uh, this is the, uh, current modern equations of the, uh, attention. And I think this was the first paper that I saw it in. It's the first time that, uh, there's a word, attention, used as far as I know, uh, to, uh, call this mechanism. So I actually tried to dig into the details of the history of the attention. Uh, so the first author here, Dmitry, I, um, I had an email correspondence with him, and I basically sent him an email. I'm like, "Dmitry, this is really interesting. Transformers have taken over. Where did you come up with the soft attention mechanism that ends up being the heart of the transformer?" And, uh, to my surprise, he wrote me back this, like massive email, which was really fa-fascinating. So this is an excerpt from that email. Um, so basically he talks about how he was looking for a way to avoid this bottleneck between the encoder and decoder. He had some ideas about cursors that traversed the sequences that didn't quite work out. And then here, "So one day I had this thought that it would be nice to enable the decoder RNN to learn to search where to put the cursor in the sourced sequence. This was sort of inspired by translation exercises that, um, learning English in my middle school involved. Your gaze shifts back and forth between source and target sequence as you translate." So literally, I thought this was kind of interesting that he's not a native English speaker, and here that gave him an edge in this machine translation that read to-- led to attention and then led to transformer. So that was-- that's really fascinating. Um, I expressed a soft search as softmax and then weighted averaging of the bi-RNN states. And basically, um, to my great excitement, this worked, this worked from the very first try. So really, I think interesting piece of history. And as it later turned out that the name of RNN Search was, uh, kind of lame, so the better name Attention, uh, came from Yoshua on, uh, one of the final passes as they went over the paper. So maybe Attention Is All You Need would have been called like RNNS or just [audio glitching] But we have Yoshua Bengio to thank for, uh, a little bit of better name, I would say. So apparently that's the, the history of this, which I thought was interesting. Okay. So that brings us to twenty seventeen, which is Attention Is All You Need. So this attention component, which in, uh, Dmitry's paper was just like one small segment, and there's all this bidirectional RNN, RNN and decoder, and this Attention Only paper is saying, "Okay, you can de-actually delete everything." Like what's making this work well, very well is just the attention by itself. And so delete everything, keep attention. And then what's remarkable about this paper actually is usually you see papers that are very incremental. They add like one thing, uh, and they show that it's better. But I feel like Attention Is All You Need was like a mix of multiple things at the same time. They were combined in a very unique way and then also achieved a very good local minimum in the architecture space. And so to me, this is, uh, really a landmark paper that, um, uh, is quite, uh, quite remarkable and I think had quite a lot of work behind the scenes. Um, so delete all the RNN, just keep attention. Uh, because attention is operates over sets, and I'm going to go into this in a second, you now need to positionally encode your inputs because attention doesn't have the notion of space by itself. Um, they... Oops, did I-- [laughing] I have to be very careful. Um, they adopted this, uh, residual network structure from ResNets. Uh, they interspersed attention with multi-layer perceptrons. They in- they, uh, used LayerNorms, which came from a different paper. They introduced the concept of multiple heads of attention that were applied in parallel. And they gave us, I think, like a fairly good, uh, set of hyperparameters that to this day are used. Uh, so the, uh, expansion factor in the multi-layer perceptron, uh, goes up by four X, and I'll go into like a bit more detail, and this four X has stuck around. And I believe, like there's a number of papers that try to play with all kinds of little details of the transformer and nothing like sticks, uh, because this is actually quite good. Uh, the only thing to my knowledge that stuck, uh, that, that didn't stick was this, uh, reshuffling of the LayerNorms to go into the pre-norm, uh, version, where here you see the LayerNorms are after the multi-headed attention or feed forward, uh, they just put them before instead. So just reshuffling of LayerNorms, but otherwise the GPTs and everything else that you're seeing today is, uh, basically the two thousand and seventeen architecture from five years ago. And even though everyone is working on it, uh, it's re-proven remarkably resilient, which I think is really interesting. Uh, there are innovations that I think have been adopted also in, um, positional encodings. It's more common to use different rotary and relative positional encodings and so on. Uh, so I, I think there have been changes, but for the most part, it's proven very, uh, resilient. So really quite an interesting paper. Now, I wanted to go into the attention mechanism. Um, and I think I sort of like the way I interpret it is not, is not similar to the ways that I've seen it presented before. So let me try a different way, um, of like how I see it. Basically, to me, attention is kind of like the communication phase of the transformer. And the transformer interleaves two phases. Uh, the communication phase, which is the multi-headed attention, and the computation stage, which is, uh, this multi-layer perceptron or P tool. So in the communication phase, uh, it's really just a data-dependent message passing on directed graphs. Uh, and you can think of it as, uh, okay, forget everything with, uh, machine translation and everything. Let's just-- we have directed graphs. At each node, you are storing a vector. And then, um, let me talk now about the communication phase of how these vectors talk to each other in this directed graph. And then the compute phase later is just a multi-layer perceptron, which now, which then, um, basically acts on every node individually. But how do these nodes talk to each other in this, uh, directed graph? So I wrote like some simple Python, uh, uh, like I wrote this in Python basically to, to create one round of communication of, uh, using attention as the, uh, direct, as the, um, message passing scheme. So here a node has this private data vector, as you can think of it as private information to this node. And then it can also emit a key, a query, and a value. And simply that's done by linear transformation, uh, from this node. So the key is, um, what are the things that I am, um, uh... Sorry. The, the query is what are the things that I'm looking for, the key is what are the things that I have, and the value is what are the things that I will communicate. And so then when you have your graph that's made up of nodes and some random edges, when you actually have these nodes communicating, what's happening is you loop over all the nodes individually in some random order. And you, uh, you're at some node, and you get the query vector Q, which is I'm a node in some graph, and, uh, this is what I'm looking for. And so that's just achieved via this linear transformation here. And then we look at all the inputs that point to this node, and then they broadcast what are the things that I have, which is their keys. So they broadcast the keys. I have the query. Then those interact by dot product to get scores. So basically, uh, simply by doing dot product, you get some kind of a, um, unnormalized weighting of the interestingness of all of the information in my, uh, on, in the nodes that point to me and to the things I'm looking for. And then when you normalize that with softmax, so it just sums to one, uh, you basically just end up using those scores, which now sum to one in our probability distribution, and you do a weighted sum of the values, uh, to get your update. So I have a query. They have keys. Uh, dot product to get interestingness or, like, affinity. Softmax to normalize it, and then weighted sum of those values flow to me and update me. And this is happening for each node individually, and then we update at the end. And so this kind of a message-passing scheme is kind of like at the heart of, uh, the transformer, uh, and, uh, happens in a more vectorized, uh, batched way that is more confusing and is also interspersed with, interspersed with, uh, layer norms and things like that to make the training, uh, behave better. Uh, but that's roughly what's happening in the, uh, attention mechanism, I think, on a high level. Um, so yeah. So in the communication phase of the transformer, um, then this message-passing scheme happens in every head in parallel, and then in every layer in series, um, and with different weights each time. And that's the... that's, that's it as far as the multi-headed attention goes. And so if you look at th- these encoder/decoder models, you can sort of think of it then in terms of the connectivity of these nodes in the graph. You can kind of think of it as like, okay, all these tokens that are in the encoder that we want to condition on, they are fully connected to each other. So in... when they communicate, they communicate fully when you calculate their features. But in the decoder, because we are trying to have a language model, we don't want to have communication from future tokens because they give away the answer at this step. So the tokens in the decoder are fully connected from all the encoder states, and then they are also fully connected from everything that is before them. And so you end up with this like triangular structure of... in the directed graph. But that's the message-passing, uh, scheme that this basically implements. Um, and then you have to be also a little bit careful because in the cross attention here with the decoder, you consume the features from the top of the encoder. So, um, think of it as in the encoder, all the nodes are looking at each other, all the tokens are looking at each other many, many times, and they really figure out what's in there, and then the decoder when it's, it's looking only at the top nodes. So that's roughly the message-passing scheme. I was going to go into more of an implementation of a transformer. I don't know if there's any questions about this bit.

   4. SPSpeaker27:23

      Uh, you explained a little bit about self-attention and multi-headed attention, but what is the advantage of this?

   5. AKAndrej Karpathy27:30

      Yeah. So, um, self-attention and multi-headed attention. So the multi-headed attention is just this attention scheme, but it's just applied, uh, multiple times in parallel. Multiple heads just means independent applications of the same attention. Um, so, uh, this message-passing scheme basically just happens, uh, in parallel multiple times with different weights for the query key and value. So you can almost look at it like in parallel, I'm looking for... I'm seeking different kinds of information from different nodes, and I'm collecting it all in the same node. It's all done in parallel. So heads is really just like, uh, copy/paste in parallel. Um, and, uh, layers are copy/paste but in series. Maybe that makes sense. And, uh, self-attention, when it's self-attention, what it's referring to is that the node here, um, produces each node here. So as I described it here, this is really self-attention because every one of these nodes produces a key query and a value from this individual node. When you have cross-attention, you have one cross attention here, um, coming from the encoder. That just means that the queries are still produced from this node, but the keys and the values are produced as a function of nodes that are coming from, uh, the encoder. So, um, I have my queries because I'm trying to decode some, the fifth word in the sequence, and I- I'm looking for certain things to become the fifth word. And then the keys and the values in terms of the source of information that could answer my queries can come from the previous nodes in the current decoding sequence or from the top of the encoder. So all the nodes that have already seen all of the en- all of the encoding tokens many, many times can now broadcast what they contain in terms of information. So I guess to summarize, the self-attention is kind of like... Sorry, cross-attention and self-attention only differ in where the keys and the values come from. Either the keys and values are produced from this node, uh, or they are produced from some external source like, like an encoder and the nodes over there. But, uh, algorithmically it's the, it's the same, um, mathematical operations.

   6. SPSpeaker29:39

      Question.

   7. AKAndrej Karpathy29:40

      Okay.

   8. SPSpeaker29:40

      There's two questions, but the first question is, in the message-passing graph paradigm, what would layers

   9. AKAndrej Karpathy29:48

      So, yeah. So, um-

   10. SPSpeaker29:49

       [inaudible]

   11. AKAndrej Karpathy29:56

       So think of, um... So each one of these nodes is a token. Um, I guess, like, they don't have a very good picture of it in the, uh, transformer, but, like, um, like this node here could represent the, uh, third word in the output in the decoder. And, um, in the beginning, it is just the embedding of the word. Um- And then, um... Okay, I have to think through this analogy a little bit more. I came up with it this morning. [laughs]

   12. SPSpeaker30:34

       [laughs]

   13. AKAndrej Karpathy30:35

       Actually, I came up with it yesterday. [laughs]

   14. SPSpeaker30:38

       Uh, we have another question. One example of instantiation would be a good instantiation of nodes as in blocks word embedding?

   15. AKAndrej Karpathy30:50

       Uh, these nodes are basically the vectors. Um, I'll go to an implementation... I'll go to the implementation, and then maybe I'll make the connections, uh, to the graph. So let me try to first go to... Let me now go to, with this intuition in mind at least, uh, to NanoGPT, which is a concrete implementation of a transformer that is very minimal. That's why I worked on this over the last few days, and here it is, uh, reproducing GPT-2 on open web text. Uh, so it's a pretty serious implementation that reproduces GPT-2, I would say, and, uh, provided enough compute. Uh, this was one node of eight GPUs for thirty-eight hours or something like that, if I remember correctly. And it's very readable. It's three hundred lines, so everyone can take a look at it. Um, and, uh, yeah, let me basically briefly step through it. So let's try to have a, a decoder-only transformer. So what that means is that it's a language model. It tries to model the, um, the, the next word in the sequence or the next character in the sequence. So the data that we train on is always some kind of text. So here's some, uh, fake Shakespeare. Sorry, this is real Shakespeare. We're gonna produce fake Shakespeare. So this is called the Tiny Shakespeare Dataset, which is one of my favorite toy datasets. You take all of Shakespeare, concatenate it, and it's one megabyte file, and then you can train language models on it and get infinite Shakespeare if you like, which I think is kind of cool. So we have a text. The first thing we need to do is we need to convert it to a sequence of integers, uh, because transformers natively process, um, you know, um... You can't plug text into transformer. You need to somehow encode it. So the way that encoding is done is we convert, for example, in the simplest case, every character gets an integer, and then instead of "hi there," we would have this sequence of integers. So then you can encode every single, um, character as an integer and get, like, a massive sequence of integers. You just concatenate it all into one large, long, one-dimensional sequence, and then you can train on it. Now, here we only have a single document. In some cases, if you have multiple independent documents, what people like to do is create special tokens, and they intersperse those documents with those special end-of-text tokens, uh, that they splice in between to create boundaries. Uh, but those boundaries actually don't have any, um, uh, any modeling impact. It's just that the transformer is supposed to learn via back propagation that the end of document sequence means that you should wipe the memory. Okay, so then we produce batches. Uh, so these batches of data just mean that we go back to the one-dimensional sequence, and we take out chunks of this sequence. So, say, if the block size is eight, uh, then b- the block size indicates the, um, maximum length of context that your transformer will process. So if our block size is eight, that means that we are going to have up to eight characters of context to predict the ninth character in the sequence. And the batch size indicates how many sequences in parallel we're going to process, and we want this to be as large as possible, so we're fully taking advantage of the GPU and the parallels on the boards. So in this example, we're doing, uh, four by eight batches. So every row here is independent example, sort of. And then every, um, every, uh, every row here is a, is a small chunk of the sequence that we're going to train on. And then we have both the inputs and the targets at every single point here. So to fully spell out what's contained in a single four by eight batch to the transformer, uh, I sort of, like, unpack it here. So when the input is forty-seven by itself, the target is fifty-eight, and when the input is the sequence forty-seven, fifty-eight, the target is one, and when it's forty-seven, fifty-eight, one, the target is fifty-one, and so on. So actually, the single batch of examples that's four by eight actually has a ton of individual examples that we are expecting the transformer to learn on in, um, in parallel. And so you'll see that the batches are learned on completely independently, but the, uh, the, uh, time dimension sort of here along horizontally is also trained on in parallel. So sort of your, your real batch size is more like B times T. It's just that the context grows linearly for the predictions that you make along the T direction, um, in the, in the model. So this is how the mo- this is all the examples that the model will learn from this single batch. So now this is the, uh, GPT class, and, uh, because this is a decoder-only model, um, so we're not going to have an encoder because there's no, like, English we're translating from. We're not trying to condition on some other external information. We're just trying to produce a, uh, sequence of words that follow each other, or are likely to. So this is all PyTorch, uh, and I'm going slightly faster 'cause I'm assuming people have taken 231N or something along those lines. Um, but here in the forward pass, we take this, uh, these indices, and then we both encode the identity of the indices just via, uh, an embedding lookup table. So every single integer has a, uh, we index into a lookup table of vectors in this nn.embedding and pull out the, the, um, word vector for that token. And then, um, because the message... Because transformer by, by itself doesn't actually... It processes sets natively. So we need to also positionally encode these vectors so that we basically have both the information about the token identity and its place in the sequence from one to block size. Now, those, uh, the information about what and where is combined additively. So the token embeddings and the positional embeddings are just added exactly as here. So this X here, uh, then there's optional dropout. This X here basically just contains the set of, uh, words, um, and their positions, and that feeds into the blocks of transformer, and we're gonna look into what's block here. But for here, for now, this is just a series of blocks in the transformer. And then in the end, there's a LayerNorm, and then you're decoding the logits, uh, for the next, um, word or next integer in a sequence using a linear projection of the, of the output of this transformer. So LM head here, uh, short for, uh, language model head, is just a linear function. Uh, so basically positionally encode all the words, feed them into a sequence of blocks, and then apply a linear layer to get the probability distribution for the next, uh, character. And then if we have the targets which we produced in, uh, the data loader, and you'll notice that the targets are just the inputs offset by one in time, then those targets feed into a cross entropy loss. So this is just a negative log likelihood, typical classification loss. So now let's drill into what's here in the blocks. Uh, so these blocks that are applied sequentially, um, there's again, as I mentioned, this communicate phase and the compute phase. So in the communicate phase, all the nodes get to talk to each other. And so these nodes are basically... If our block size is eight, then we are going to have eight nodes in this graph. There's eight nodes in this graph. The first node is pointed to only by itself. The second node is pointed to by the first node and itself. The third node is pointed to by the first two nodes and itself, et cetera. So there's eight nodes here. So you apply... There's a residual pathway in X, you take it out, you apply a LayerNorm, and then the self-attention, so that these communicate, these eight nodes communicate. But you have to keep in mind that the batch is four. So because batch is four, this is also applied. Uh, so we have eight nodes communicating, but there's a batch of four of them all individually communicating among those eight nodes. There's no crisscross across the batch dimension, of course. There's no batch dimension anywhere, luckily. Um, and then once they've changed information, they are processed using the multi-layer perceptron, and that's the compute phase. So, um, and then also here we are missing, um, we are missing the cross attention, um, and, uh, because this is a decoder-only model. So all we have is this step here, the multi-headed attention, and that's this line, the communicate phase. And then we have the feed forward, which is the MLP, and that's the compute phase. Uh, I'll take, I'll take questions a bit later. Then the MLP here is fairly straightforward. Uh, the MLP is just individual processing on each node, um, just transforming the feature representation sort of at that node. So, um, applying a two-layer neural net with a GELU non-linearity, which is, just think of it as a ReLU or something like that. It's just a non-linearity. And then MLP is straightforward. I don't think there's anything too crazy there. And then this is the, uh, causal self-attention part, the communication phase. So this is kind of like the meat of things and the more... most complicated part. It's only complicated because of the batching and the implementation detail of how you mask the connectivity in the graph so that you don't re-- you can't obtain any information from the future when you're predicting your token. Otherwise, it gives away the information. So if I'm the fifth token, and, um, if I'm the fifth position, then I'm getting the fourth token coming into the input, and I'm attending to the third, second, and first, and I'm trying to figure out what is the, what is the next token. Well, then in this batch, in the next element over in the time dimension, the answer is at the input. So I can't c- get any information from there. So that's why this is all tricky. But basically, in the forward pass, um, we are calculating the, the queries, keys, and values based on X. So these are the keys, queries, and values. Here, when I'm computing the attention, I have the queries, uh, matrix multiplying the keys. So this is the dot product in parallel for all the queries and all the keys in all the heads. So that, uh, I th-I mentioned-- I failed to mention that there's also the aspect of the heads, which is also done all in parallel here. So we have the batch dimension, the time dimension, and the head dimension, and you end up with five-dimensional tensors, and it's all really confusing. So I invite you to step through it later and convince yourself that this is actually doing the right thing. But bas-basically, you have the batch dimension, the head dimension, and the time dimension, and then you have features at them. And so, uh, this is evaluating for all the batch elements, for all the head elements, and all the time elements, the simple Python that I gave you earlier, which is query dot product key. Then here we do a masked fill, and what this is doing is it's basically clamping, uh, the, um, the attention between the nodes that are not supposed to communicate to be negative infinity. And we're doing negative infinity because we're about to softmax, and so negative infinity will make basically the attention of those elements be zero. And so here we are going to basically end up with, uh, the weights, um, the, the sort of affinities between these nodes. Optional dropout, and then here attention matrix multiply V is basically the, um, uh, the gathering of the information according to the affinities we've calculated, and this is just a weighted sum of the values at all those nodes. So this matrix multiplier is doing that weighted sum. And then transpose contiguous view because it's all complicated and batched in five-dimensional tensors, but it's really not doing anything. Optional dropout and then, uh, a linear projection back to the residual pathway. So, uh, this is implementing the communication phase here. Then you can train this transformer, um, and then you can generate, uh, infinite Shakespeare, and you will simply do this by... Because our block size is eight, we start with, uh, some token, um, say like, uh, I use... In this case, um, you can use something like a new line as the start token, and then, um, you communicate only to yourself because there's a single node, and you get the probability distribution for the first word in the sequence. And then, um, you decode it for the first character in the sequence. You decode the character, and then you bring back the character, and you re-encode it as an integer, and now you have the, the second thing. And so you, you get, okay, we're at the first position, and this is whatever integer it is. Add the positional encodings, goes into the sequence, uh, goes into the transformer. And again, this token now communicates with the first token, um, and then its identity. And so you just keep like plugging it back. And once you run out of the block size, which is eight, you start to crop because you can never have block size more than eight in the way you've trained this transformer. So we have more and more context until eight, and then if you wanna generate beyond eight, you have to start cropping because the transformer only works for- Eight elements in time dimension. And so all of these, uh, transformers in the naive setting have a finite block size or context length. And, uh, in typical models, this will be one thousand twenty-four tokens or two thousand forty-eight tokens, something like that. But these tokens are usually like BPE tokens or sentence piece tokens or work piece tokens. There's many different encodings. Uh, so it's not like that long, and so that's why I think they mentioned we really want to expand the context size when it gets gnarly because the attention is quadratic in the mini case. Now, if you want to, um, implement an encoder instead of a decoder attention, then all you have to do is this mask load, and you just delete that line. So if you don't mask the attention, then all the nodes communicate to each other, and everything is allowed, and information flows between all the nodes. So if you want, uh, to have the encoder here, uh, just, uh, delete, um... all the encoder blocks will use attention where this line is deleted. That's it. So you're allowing, um, whatever this encoder might store, say, ten tokens, like ten nodes, and they are all allowed to communicate to each other going up the transformer. And then if you want to implement cross-attention, so you have a full encoder-decoder transformer, not just a decoder-only transformer or GPT, then we need to also add, uh, cross-attention in the middle. So here there's a self-attention piece where all the... there's a self-attention piece, a cross-attention piece, and this MLP. And in the cross-attention, uh, we need to take the features from the top of the encoder. We need to add one more line here, and, uh, this would be the cross-attention, uh, instead of... I should have implemented it instead of just pointing, I think. Uh, but, um, there will be a cross-attention line here, so we'll have three lines because we need to add another block. And the queries will come from X, but the keys and the values will come from the top of the encoder. And there will be basically information flowing from the encoder strictly to, uh, all the nodes inside X. And then that's it. So it's very simple sort of modifications on the decoder attention. So you'll, you'll hear people talk that you kind of have a decoder-only, uh, model like GPT. You can have an encoder-only model like BERT, or you can have an encoder-decoder model like, say, T5 doing things like machine translation. So, um, and in BERT, uh, you can't train it using sort of this, um, language modeling setup that's autoregressive, and you're just trying to predict the next element in the sequence. You're training it to a slightly different objectives. You're putting in like the full sentence, and the full sentence is allowed to communicate fully, and then you're trying to classify sentiment or something like that. Uh, so you're not trying to model like the next token in the sequence. Uh, so these are trained slightly different with masked, uh, uh, with, uh, using masking and, uh, other denoising techniques. Okay, so that's kinda like the transformer. I'm going to continue. So yeah, maybe more questions.

   16. SPSpeaker45:37

       [background noise] Uh, so what we are importing information is essentially fully connected graph to the multi and do it, and when we're like building this graph with the tag, you know, would the transformer still perform, um, if like it's a dynamic graph that, that the connections change in every time step, and we also have some feature information for different things. This is like we are enforcing these, um, constraints on it by just masking, but is it aware of the enforcement and things like...

   17. AKAndrej Karpathy46:11

       Um, so I'm not sure if I fully follow. So there's different ways to look at this analogy, but one analogy is you can interpret this graph a- as really fixed. It's just that every time we do the communicate, uh, we are using different weights. You can look at it that way. So if we have block size of eight in my example, we would have eight nodes. Here we have two, four, six. Okay, so we'd have eight nodes. They would be connected in, in, um... you lay them out, and you only connect from left to right.

   18. SPSpeaker46:35

       Yeah. I mean, but for different problem that might not be the case. Like, yeah, it will have a graph where the connections might change as a problem.

   19. AKAndrej Karpathy46:42

       Uh, why would the connect... Usually, the connections don't change as a function of the data or something like that.

   20. SPSpeaker46:47

       I mean, like in biology with like an actual graph, like with like dynamic models where like [background noise] we may not know. I'm wondering if like there's any-

   21. AKAndrej Karpathy47:00

       I don't think I've seen a single example where the connectivity changes dynamically in the function of data. Usually, the connectivity is fixed. If you have an encoder and you're training a BERT, you have how many tokens you want, and they are fully connected. And if you have a decoder-only model, you have this triangular thing. And if you have encoder-decoder, then you have awkwardly sort of like two pools of nodes. [chuckles] Uh, yeah. Uh, go ahead.

   22. SPSpeaker47:25

       I guess there's... I think if there's like a different type of problem that maybe not be the, the computation. But like, it's like... My question is, um, I wonder if you know much more about this than I know actually. But like, um, do you have a sense of like, you know, if you ran like a correlation study, which parts of the result, like in my head, I'm thinking there's original connections and, um, you also have

   23. AKAndrej Karpathy48:13

       Yeah. It's really hard to say. So that's why I think this paper is so interesting is like, yeah, usually you'd see like a path, and maybe they had path internally, they just didn't publish it. Uh, all you can see is sort of things that didn't look like a transformer. I mean, you have ResNets, which have lots of this. Uh, but a ResNet would be kind of like this, but there's no, um, there's no self-attention component, but the MLP is there kind of in a ResNet. Um, so a ResNet looks very much like this except there's no... Uh, you can use LayerNorms in ResNets, I believe, as well. Typically, sometimes they can be BatchNorms. So it is kind of like a ResNet... It is kind of like they took a ResNet and they put in a trans- self-attention block in addition to the preexisting MLP block, which is kind of like convolutions. And MLP would strictly speaking be convolution, one-by-one convolution. Um, but I, I think the idea is similar in that MLP is just kinda like a, you know, typical weights, uh, nonlinearity weights or operation. Um, and but I will say like, yeah, it is kind of interesting because a lot of work is, is not, uh, is not there, and then they give you this transformer, and then it turns out five years later it's not changed even though everyone's trying to change it. So it's kind of interesting to me that it's kinda like a package, came like a package, uh, which I think is really interesting historically. And, um, I also talked to a, uh, paper authors and, uh, they were unaware of the impact that transformer would have at the time. So when you read this, uh, paper actually, uh, it's, it's kind of unfortunate because this is like the paper that changed everything, but when people read it, it's like question marks because it reads like a pretty random machine translation paper. Like, "Oh, we're doing machine translation. Oh, here's a cool architecture. Okay, great. Good results." Like, it's, it doesn't sort of know what's gonna happen. [laughs] And so when people read it today, I think they're kind of confused, uh, potentially. Like, having, like having... I will have some tweets at the end, but I think I would've renamed it with the benefit of hindsight of like... Well, I'll get to it. [laughs] Yeah.

   24. SPSpeaker50:08

       [background noise]

   25. AKAndrej Karpathy50:21

       Yeah, I think that's a good question as well. Uh, currently, I mean, I certainly don't love the autoregressive modeling approach. Uh, I think it's kind of weird to like sample a token and then commit to it. Uh, so, you know, maybe there's some ways to, um, uh... Maybe there's some ways, some hybrids with diffusion as an example, which I think would be really cool. Um, or we'll find some other ways to like edit the sequences later, but still in the au-autoregressive framework. Um, but I think diffusion is kind of like an up-and-coming modeling approach that I personally find much more appealing. When I sample text, I don't go chunk, chunk, chunk and commit. I, uh, I do a draft one, and then I do a better draft two, and that feels like a diffusion process. So that would be my hope.

   26. SPSpeaker51:05

       Okay, I also have a question.

   27. AKAndrej Karpathy51:06

       Yeah.

   28. SPSpeaker51:06

       Uh, so yeah, I really like the graph analogy for the pixel-based vision like what's happening. Would you say like the, like this like self-attention is sort of like, uh, computing like a edge weight using like the dot product on the notion that... And then once we have the edge weight, we just like multiply it by the values and then just propagate it?

   29. AKAndrej Karpathy51:25

       Yes. That's right. Yeah.

   30. SPSpeaker51:27

       And do you think there's like a, like analogy with like graph neural networks and, uh, self-attention could be-
10. 1:00:30 – 1:11:35

    Thank you - Go forth and transform

    1. AKAndrej Karpathy1:00:30

       I don't know. Okay, yeah, that's it. [laughs] You can read the tweets later, but, uh, that's where I'll just end here. I'll just leave this up. And maybe... [laughs] So, sorry, I just found this tweet. So turns out that if you scale up the training set and use a powerful enough neural net like a transformer, the network becomes a kind of general purpose computer over text. So I think that's a kind of like nice way to look at it. And instead of performing a single fixed sequence, you can design the sequence in the prompt. And because the transformer is both powerful, but also is trained on large enough, very hard dataset, it kind of becomes this general purpose text computer. And so I think that's kind of interesting when you look at it. Yeah.

    2. SPSpeaker1:01:12

       Um, actually, we have three points we want to talk.

    3. AKAndrej Karpathy1:01:16

       Yeah.

    4. SPSpeaker1:01:18

       Um, so like, uh, I guess, like, like for me, I learned about phone number first.

    5. AKAndrej Karpathy1:01:31

       Mm-hmm.

    6. SPSpeaker1:01:31

       And then you get called, you know, like that would be a function. Uh, if you have an RN.

    7. AKAndrej Karpathy1:01:41

       Yes.

    8. SPSpeaker1:01:41

       So, um, and I first learned about, like, transformer, um, I mostly like, all right, like this happens, uh, durable-

    9. AKAndrej Karpathy1:01:53

       Yes

    10. SPSpeaker1:01:54

        ... sort of thing. But now we see kind of ideas that like maybe there's a meadow or a gradient. And I guess my question specifically is how much do you think it's like, is pre really like most of it? You know, like is it really because it's mostly more efficient or do you think, is there any sort of like opinion you have that like maybe transformers have like a specific inductive bias for compute or ingredient-

    11. AKAndrej Karpathy1:02:24

        Yeah

    12. SPSpeaker1:02:24

        ... in a way that they find it?

    13. AKAndrej Karpathy1:02:26

        So I think there's a bit of that. Yeah. So I would say RNNs, like in principle, yes, they can implement arbitrary, uh, programs. I think it's kind of like a useless statement to some extent because they are not... They're probably... I'm not sure that they're, they're probably expressive because in a sense of like power in that they can implement these arbi- arbitrary, uh, functions, but they're not optimizable and they're certainly not efficient because they are serial computing devices. Um, so I think, uh, so if you look at it as a compute graph, RNNs are very long, thin compute graph. Um, like if you stretched out the neurons and you look-- like take all the individual neurons, interconnectivity and stretch them out and try to visualize them, RNNs would be like a very long graph and like bad. And it's bad also for optimizability because I don't exactly know why, but just the rough intuition is when you're back propagating, you don't wanna make too many steps. And so transformers are a shallow wide graph. And so from, from supervision to inputs is a very small number of hops, and it's along residual pathways which make gradients flow very easily. And there's all these layer norms to control the, the gradient, the, the, the scales of all, all of those activations. And so, uh, there's not too many hops, and you're going from supervision to input very quickly, and it just like flows through the graph. So, um, and it's, it can all be done in parallel, so you don't need to do this, uh, encoder-decoder RNNs. You have to go from first word, then second word, then third word. But here in, uh, in transformer, every single word was processed completely as sort of in parallel, which is kind of a... So I think all these are really important because, you know, all these are really important, and I think number three is less talked about but extremely important because in deep learning scale matters. And so the size of the network that you can train, uh, uh, gives you, uh, is extremely important. And so if it's efficient on the current hardware, then we can make it bigger.

    14. SPSpeaker1:04:15

        When you mentioned that you're dealing with like multiple modalities of data, you sort of feed it all together. How does that process work?

    15. AKAndrej Karpathy1:04:22

        Yeah.

    16. SPSpeaker1:04:22

        How does that actually work? Do you like kind of leave the different data at different token or is it like-

    17. AKAndrej Karpathy1:04:27

        Yeah

    18. SPSpeaker1:04:27

        ... okay, that would be the

    19. AKAndrej Karpathy1:04:29

        No. Um, so yeah, so you take your image and you apparently chop them up into patches. So there's the first, uh, thousand tokens or whatever, and now I have a special... So radar could be, um, also, but I don't actually know the native representation of radar. So, uh, but you could, uh, you just need to chop it up and enter it, and then you have to encode it somehow. Like the transformer needs to know that they're coming from radar. So you create a special, uh, you have, you have some kind of a special token that you, um, like these radar tokens are slightly different in the representation, and it's learnable by gradient descent. And, um, like vehicle information would also come in with a special embedding token, uh, that can be learned. Um, so, um-

    20. SPSpeaker1:05:09

        So have you like those like poorly

    21. AKAndrej Karpathy1:05:11

        Actually, you don't. It's all just a set.

    22. SPSpeaker1:05:14

        And even the placement coordinates, you know, which can be the same.

    23. AKAndrej Karpathy1:05:19

        Yeah, it's all just a set, but you can positionally encode these sets, uh, if you want. So, um, but positional encoding means you can hardwire, for example, the, the coordinates like using sinusoids and cosines. You can, you can hardwire that, but it's better if you don't hardwire the position. You just-- there's just a vector that is always hanging out at this location. Whatever content is there just adds on it, and this vector is trainable by background. That's how you do it. Yeah.

    24. SPSpeaker1:05:47

        I don't really like this approach or... I think they're kind of elegant. Like, they seem to work, but it seems like there's sometimes hard situations where you would actually want them to put some structure of the location or presentation or whatever else. Do you have any ideas like how you might do something better?

    25. AKAndrej Karpathy1:06:03

        Um, I'm not sure if I understand the question. So I mean, the positional encoders, like they're, they're actually like not, uh, they have... Okay, so they have very little inductive bias or something like that. They're just vectors hanging out in location always, and you're trying to, you're trying to help the network in some way. Um, and I think the intuition is, is good, but, um, like if you have enough data, usually trying to mess with it is like a bad thing. Um, like trying to, like trying to enter knowledge when you have enough knowledge in the data set itself is not usually productive. Uh, so it like really depends on what scale you want. If you have infinity data, then you actually want to encode less and less. That turns out to work better. And if you have very little data, then actually you do wanna encode some biases. And maybe if you have a much smaller data set, then maybe convolutions are a good idea because you actually have this, uh, bias coming from all filters. And so, uh, but I think, um- So the transformer is extremely general, but there are ways to mess with the encodings to put in more structure. Like you could, for example, encode sinuses and cosines and fix it. Or you could actually go to the attention mechanism and say, "Okay, my-- if my, uh, image is chopped up into patches, this patch can only communicate to this neighborhood." And you can-- you just do that in the attention, uh, matrix. You just, uh, mask out whatever you don't want to communicate. And so people really play with this because the, uh, the full attention is, uh, inefficient. So they will intersperse, uh, for example, layers that only communicate in little patches and then layers that communicate globally, and they will sort of do all kinds of tricks like that. So you can slowly bring in more inductive bias. You would do it, but the inductive biases are sort of like they're factored out from the core transformer, and they are factored out in the, uh, in the connectivity of the nodes, and they are factored out in the positional encoders, and you can mess with this per computation.

    26. SPSpeaker1:07:50

        And how you could try to, uh, augment things with the base of the transformer. So then I think there's probably use with this type of thing. Usually I'd like to organize it. How do you imagine this being solved completely?

    27. AKAndrej Karpathy1:08:02

        So there's probably about two hundred papers on this now, uh, if not more. Uh, they're kind of hard to keep track of. Honestly, like my Safari browser, which is... Oh, it's on my computer, like two hundred open tabs. Um, but, um... Yes, I'm not even sure if I, if I wanna, like, pick my favorite, honestly. [laughs]

    28. SPSpeaker1:08:23

        Yeah. I was thinking about the, this very interesting talk from NeXT this year. And you can think of a transformer as like a CPU. I think this process is fixed by construction, something like four thousand different times. And then like now, now any given the CPU, what you have is like you store variables, you have memory. So it's like if I want to interpret a program with CPU, I just like do it multiple times. So maybe like you can use a transformer like that to get there.

    29. AKAndrej Karpathy1:08:45

        The other one that I actually like even more is potentially keep the context length fixed, but allow the network to somehow use, uh, a scratchpad. Okay? And so the way this works is you will teach the transformer somehow via examples in the prompt that, "Hey, you actually have a scratchpad. Hey, hey, trans-- You basically, you can't remember too much. Your context line is finite. But you can use a scratchpad, and you do that by emitting a start scratchpad, and then writing whatever you want to remember, and then end scratchpad. And then, uh, you continue with whatever you want." And then later when it's decoding, you actually, like, have special logic that when you detect start scratchpad, you will sort of like save whatever it puts in there in like a external thing and allow it to attend over it. So basically, you can teach the transformer just dynamically because it's so, um, meta-learned. You can teach it dynamically to use other gizmos and gadgets and allow it to expand its memory that way, if that makes sense. It's just like human learning to use a, a notepad, right? You don't have to keep it in your brain. So keeping things in your brain is kind of like the context line for the transformer. But maybe we can just give it a notebook, and then it can query the notebook, uh, and read from it and write to it.

    30. SPSpeaker1:09:46

        Maybe you can use the transformer to train another transformer. [laughs]

Episode duration: 1:11:40