Stanford CS329A Self-Improving AI Agents | Part 1 | Course Overview

Dr. Aakanksha Chowdhery 0:05 Welcome everyone to fall quarter, and welcome to CS329A. Um, I hope you made it to the right class. This class is on self-improving AI agents. Anyone made it to the right class here?

Speaker 0:18 Yes.

Dr. Aakanksha Chowdhery 0:19 Yes? Okay. Uh, how many of you are here, uh, and doing master's at Stanford? Okay. PhD? Okay. A few of you. And then undergrad? Okay. So master's is the big crowd here. Okay, cool.

Azalia Mirhoseini 0:38 There's a good balance. Yeah.

Dr. Aakanksha Chowdhery 0:39 It's a really good balance. Um, so we'll introduce ourselves. So I'm Aakanksha. I have been working in large language models for a while, and I'm Adjunct Professor at Stanford, and I'm also in research, uh, at a startup called Reflection AI.

Azalia Mirhoseini 0:52 Hi, I'm Azalia Mirhoseini. I'm a, I'm an Assistant Pro-Professor in the CS department. And, um, uh, Aak-Aakanksha and I met each other when we were at-

Dr. Aakanksha Chowdhery 1:05 Google Deep-

Azalia Mirhoseini 1:06 ... Google, Google Brain. Google Brain.

Dr. Aakanksha Chowdhery 1:07 Google Brain. [laughs]

Azalia Mirhoseini 1:08 Yeah, back in the day, and then Google DeepMind. And I also worked on Claude at Anthropic and Gemini at Google, uh, DeepMind. And we are so excited to, uh, teach this course again. This is the second time we're teaching. Of course, we have made some updates both to the lectures and also the way we are, uh, kind of conducting everything during the, the, uh, for the course. Uh, and we are-- we were just so amazed by the amount of interest, uh, for, for people to join this course. So, uh, we are excited to have you here.

Dr. Aakanksha Chowdhery 1:41 Uh, the course website is at cs329a.stanford.edu. We have updated the website in terms of all the lecture materials and, and the papers that you'll be reading, as well as, uh, what's the homework schedule to expect, um, the project proposal and project structures as well. So in today's lecture, we'll start with an overview of large language models, uh, just, uh, how the scaling trends have evolved over the last, uh, five years. And then we'll focus more on what we will cover in this class. And then from there, uh, in the second half of the, uh, this lecture, we'll cover the course logistics. And that part you should pay a lot of attention to because that influences your grades. So let's take a look at an overview of the trends for large language models. So one of the aspects that has been mind-blowing since, uh, GPT-3 came out is that, uh, there is a, uh... that as you scale up the number of parameters in a model, uh, they get better. And what this means is, uh, that language

Dr. Aakanksha Chowdhery 2:42 models like BERT and T5 were good, but as we increased the number of parameters, they got really better. They, they performed much better. And what this led to is something called a scaling loss for large language models, uh, which are used in building the base model for the pre-trained base model. And, uh, as you increase the amount of compute, so there are three graphs here. As you increase the amount of compute on the X-axis, the loss function on the Y-axis goes down. And, and the more compute you have put in, the more, uh, the tes-test loss goes down, which leads to a better model. So that was one big aspect that led to GPT-3 and then all the subsequent ChatGPT, PaLM, um, Gemini, and so on. And then similarly, as you increase the dataset size on X-axis, uh, the test loss goes down further on, uh, Y-axis. So that's another aspect of, like, another axis of scaling that leads to better language models. And, uh, the fourth one is of course-- the third one is, of course, parameter count. So if you increase the number of layers in a transformer or the amount of parameters that are there in the language model, that improves the loss, uh, value and that, that leads to a better model. And, and this has been the foundation for a lot of the... Uh, this has been the foundation for a long time, uh, until last year where this was starting to hit some kind of a saturation point. So from 2018 to almost 2024, um, this graph is a little bit outdated, but, uh, from 2018 to 2024, the, as the model size has gone up, uh, almost consistently, we have been increasing the, the size of the model. So you started with something like BERT was three forty million parameters, GPT-2 was one point five billion parameters, and then GPT-3 was one seventy-five billion parameters, and then, uh, say, PaLM was five forty billion, and then, uh, there was a bunch of subsequent models, and GPT-4 is estimated to be, uh, there, which is like trillions of parameters. So you can imagine that there has been an exponential growth in just the model size, uh, of, for large

Dr. Aakanksha Chowdhery 4:43 language models. The large stands for that they're growing in size, and that has been one of the key areas in which, um, the models have been getting better. So why do we care about the models, uh, getting bigger? What does this lead to? So as I was explaining that as the models get bigger, uh, they continuously improve in performance on, uh, natural language benchmarks, on various other benchmarks, reasoning benchmarks, and so on. And then other interesting aspect that I'll, uh, will show you i-in a moment is that they learn how to do few-shot learning. So earlier you have to fine-tune the model for a specific domain, but just by giving a few examples, the model can follow that template and, and reason in that same vein, which makes it extremely easy to prototype, uh, things. And then the third aspect is that as the models become bigger, they have emerging behavior where the capabilities like reasoning only emerged in larger models. So just to understand few-shot learning or zero-shot learning here. So if, if the model is given a question like, or, or a task like translate English to French, uh, it's able to give, uh... So this is called a prompt. Like it's given a task, translate English to French, and then it's asked to translate cheese to whatever is the French word. If the model can predict the answer without, uh, actually being trained on the specific task, then that would be called zero-shot learning. And then few-shot learning- In addition to giving the inst-- uh, description of the task, we're also giving a few examples. In this particular case, it's given translate English to French, and it's given a few examples of the translation from English to French, and then it's asked to translate cheese, and it's able to translate that. And you can see that this capability shows up in the large language models across a large number of tasks, and that has been one of the key areas that GPT-3 and PaLM and, um, other models enabled, and then that, uh, subsequent innovations, uh, came from reasoning aspects of this. So on the reasoning side, um, as the models have become bigger, they have this emergent behavior of reasoning which Azalia will tell us a little bit more about.

Azalia Mirhoseini 6:48 So as the models become bigger, not only we have this kind of a predictive scaling laws k-property that we kind of know how the loss function's gonna go down as we throw in more compute data and parameters in the model, there are new kind of behaviors that appear in the model that they didn't have before, and we never could predict that until we had these bigger models and saw these kind of behavior in them. And one of the most important, uh, one of those behaviors is the emergent, uh, is the chain-of-thought behavior. And for those of you who are not familiar with this, here is how it works. So in a usual prompt, prompting of a model, we can ask a model, in this case a, a math problem, um, and then we can give it a one-shot example. Here is a math problem, here is the answer. Now I ask you a similar but not exact, not exactly the same math problem, and the model is expected to answer that. Um, and the model might be able to use the, the one-shot example, or it might not be. But we can do better here by not only give an example in... but also provide the reasoning or the chain of thought on how we got to that answer. So in this case, the question is a very simple question. Roger has five tennis balls. He buys two more cans of tennis balls. Each can, each can has three tennis balls. How many, how many total tennis balls does he have now? And instead of just saying eleven, like we are walking the model through how to get to this. Um, Roger started with five balls, two cans of three tennis balls is six, and, uh, five plus six is eleven. Uh, and with this example, then the model ha- uh, has become familiar with the process to get to the answer, and it can leverage that to answer new problems. Well, obviously this is a very simple problem. Any, like, one B parameter model nowadays can solve this. Doesn't need a, uh, an example or a few shot or a chain of thought. But the chain-of-thought property itself is a property that is holding to this day, and it's a very, very important property for the reasoning models and all the thinking models and all the prog-- and many, much of the progress that we have seen, um, in the past year or so. Now, um, this property again appears as we have larger and larger models. Here are three models, Lambda, GPT, and PaLM. And as you can see, the chain of thought, uh, and this is some results on the math, some math data set. And these models, when they're smaller, like eight billion parameter for Lambda or, uh, G- for GPT around seven billion, they can't really benefit from chain of thought. Like, it doesn't do anything for them. But as they become larger, uh, or for a PaLM model, again at eight billion, it can leverage this chain-of-thought property and learn from the reasoning that is, and the, the, the process that is provided in the context in order to solve problems better. Um, and not only that, there are other kind of abilities to solve new task as we increase the size of the models. Like for example, models can all of a sudden solve mod arithmetics better or solve, um, a word unscramble problem just by having this larger size and this property. You see that all of a sudden it appears at a certain size. Um, and because of all of this, we still, as in frontier [laughs] companies and labs, are still interested in pushing the, the, the scaling laws and not only get the natural kind of progression, but also we are, we will see more and more of these kind of emergent behavior that we are, uh, that is going to be very important. Now, um, a little bit

Azalia Mirhoseini 10:51 of history here. So ChatGPT was launched in, uh, November 2022. Obviously it was one of the most successful, uh, kind of apps or software ever created. Uh, the time to reach one million users for it was, um, five days [laughs] , which is significantly, um, faster than any, than many of the other famous, um, software that you can see there or services that you can see there. And there were pieces of, of kind of innovation on top of just scaling up the parameter size of the model that were very important to make ChatGPT what it was and to make that kind of leapfrog and over the previous models like GPT-3 and so on. And two p- two important piece here were, w- were, um, the instruction tuning and, um, the reinforcement learning from human feedback, which, which we are gonna briefly mention in the next slides, but throughout this course we are gonna learn more and more about them. Um, so let's first... And then, um, we're gonna learn about all of these in more details, but let's just take a step back and see how the process works. So the pre-training step is the step that- The, the beginning of any language model training, uh, that's the easiest step, and we just train the model to predict the next token out of all sort of text and data and so on. And then this is a fine-tuning step that was a key differentiating factor for a model like ChatGPT. Um, and let's start with, um, uh, the first aspect of fine tu- fine-tuning, which is kind of alignment with human preferences. So when a model is pre-trained, it has no sense. It has seen all data on the internet, on the bo- in the books, everywhere, but it has no sense of like what is right and wrong and what is... It just have-- it just statistically knows about the state of the world, but does it-- it doesn't exactly know things or know how to follow instructions. So the next step in training these models, uh, is the, the, the, the, the step where we try to steer the AI models to follow the goals and preferences, uh, and values of humans. Um, and this is still a big problem. It's a big issue. We, we haven't like mastered it yet, but as the models become-- as we progress, models are becoming better and much more, uh, kind of like, uh, powerful in the things that they can do. Um, so it turns out that we can fine-tune the models to be aligned. For example, uh, here are some graphs that show what if we take our base, uh, base model that's just pre-trained and fine-tune it for things like sensibleness or safety. And these are like, uh, these are data that we have collected. These are high-quality data for kind of showing the model what it-- what we mean by, by what's safe and what's unsafe or what's sensible and what doesn't make sense. And these are like more cura- like more highly curated data that we pass the model for fine-tuning and to make the model more aligned with the values or the, or the preferences of humans. Um, so going back, let's just walk through this process again. So the first stop when we take a pre-trained language model is, uh, fine-tuning with next token prediction, which follows the same process as pre-training, but this time on data that is much more higher quality. Uh, for example, this could be data from books, creative essays, uh, data that companies pay a whole lot, millions of dollars or hundreds of millions of dollars potentially to buy this data that's really high quality, and then we train the-- we fine-tune the model on those, and the model becomes much better as a result of that. Um, the, the next step is instruction tuning, and by instruction tuning, what we mean is that we show the model again, uh, trad-traditionally in the past two years, this instru-in-instruction tuning data is like a combination of human-generated data or templates and then, and then, and the synthetic data. But the way this instruction tu- fine-tuning works is that we do have, uh, instruction and then question-answer pairs, uh, where the model learns how to follow the questions and answers them. For example, here, an instruction for, uh, tuning dataset could be about the weather, like, "Please answer the following question. What is the boiling po-point of nitrogen?" And then this is the answer. Um, or we can have chain of thought, uh, fine-tuning, where we, um, show the model how it can walk through a process to get to the answer. And then, uh, we have the label, we have the results over here. And this is like these pairs of like questions and instructions and answers. And again, this, this dataset is very, um, is, is-- there's a lot of effort that goes into this dataset and the quality of dataset and the, uh, generality of it, uh, has a lot of impact on the quality of the model that we get at this stage. Um, and after this point, then the model becomes more and more like, um, like the models we see today. Like you can ask them questions, you can go back and forth with them. They, they have this sense of like how to, uh, perceive a question and how to walk through kind of, uh, some sort of like process to get to, to an answer. And the next step after instruction fine-tuning is this process of, um, using human preferences and fine-tuning the model based on that. And this process is called RLHF or reinforce- reinforcement learning from human feedback. Um, the way this is different from the previous step is the way we create the objective function and, uh, create the data. So in this case, instead of like just creating some supervised prompt and labels, we are basically creating a reward model out of human preferences. So companies spend a lot of money, uh, asking humans, sometimes experts, sometimes just humans, normal humans, uh, about questions and the answers generated by the model. And we ask the, the humans to rate them to see which one is correct and which one is not correct, and that's how we can create some sort of a reward model here. Um, and this reward model is instead, instead of human, then we can use the reward model in order to kind of like guide the parameters of our LLM towards, uh, generating answers that are in line with this reward model. So basically, we want the, the generations by the model, uh, to be such that this reward model says, "Yes, these are good generations." Um, we're gonna learn more about this here, but I wanted-- we wanted you to have, um, the high level idea of what RLHF means. Uh, and we can collect-- we can have different reward type, different types of reward. For example, we could ask the model to reward a pair of answers based on their correctness or helpfulness or specificity or harmlessness, all sort of reward functions that we can define. And depending on how we want the models to be and what we care most about, we can weight these different reward models and kind of use that in the RLHF process. So these step at a, at a high level, of course, there's a lot of effort goes into that. But like pre-training and then, um, fine-tuning on higher quality data, instruction tuning and RLHF are were the core kind of components that made, um, a model like ChatGPT over, over something that existed over all the large models, um, like GPT-3 and prior models. So that made a huge difference in how good and capable the models were. Um, uh, so, so that was very

Azalia Mirhoseini 19:27 cool [chuckles] up until last year or a year, year, year and a half ago. Pre-training and then fine-tuning were the big pieces. Uh, but since a year and a half ago, it turns out inference is also a frontier for making the models much more capable. Um, so... And that o-open like a whole set of-- whole lot of new research and directions and ways that we can make the model better. Um, so let's think ab-- let's talk about it a little bit. Like this is a work we did in my lab last year. Uh, last summer. It, it was called Large Language Monkeys. Um, and it's kind of inspired by the infinite monkey theorem, and it's related to impr-inference scaling. Um, how many of you have heard of the infinite monkey theorem? Okay, that's a good amount. So the idea of in-- the infinite monkey theorem, uh, what it says is that it's not proven, but [chuckles] what it says is that if you have a monkey and a kind of a typing machine, and then we keep letting the monkey type all day forever, um, after some time, you will have the wor-works of William Shakespeare in, in the things that the monkey types. Um, so we were inspired by that to name our project. Uh, but the idea here was that let's, let's have the LLM be the monkey, and, uh, let's, uh, ask the model over and over again to solve a given input problem. Uh, you can think of it as parallel sampling or parallel generation. So instead of asking the model once to solve the input problem, you can ask it many times. And also, let's assume you have some sort of a verifier or selection mechanism, so you can pick which one of these generated responses were correct. Um, and then you output that in your system. That becomes the out-- the final output. So the model, instead of generating one answer, generates many, many answers, and then your verifier selects which one of them is correct and then outputs one of the correct ones. Um, you can think of a verifier as, say, some unit test. If the problem is, uh, generating code, the verifier can-- could be the unit test that we run against the code, and we see which, which one of the code generations passes all the unit tests and so on. Um, the reason this is very-- this is a, a-- this can be a possible possibility here is that models are, um, do not, are not, uh, there is some va-variance in the way the models generate responses. It's not deterministic. And you can even control the variation in the model responses with this thing called temperature. So you can force the model or encourage the model to generate more different responses as you ask a question. Um, and here, now let's see some of the results of this repeated sampling. For a bunch of tasks like m-math and coding benchmarks, we increased the number of samples per problem from one, which is the normal one, uh, to ten thousand. And here we are showing the coverage or the fraction of problems that are solved by at least one of these samples. And what we are seeing here is that, um, so the red line, the red dashed one, was the GPT-4.0 model. And while these blue and green lines, which were the Llama eight B, Llama three eight B, and seventy B, were worse than GPT-4.0 model with one sample. But if we increase the number of samples from these models, in all of these cases, they do better than the GPT-4.0 model. And what it says is that it kind of seems like the model or models already know a whole lot more than what you get out of them when you just ask them once, right? So this is the kind of the underlying property of inference scaling, because it seems like we can get a whole lot of-- a whole lot more capability and problem-solving kind of ability from these models if we just bring in this inference scaling. Here we are showing repeated sampling, but there are many other ways to do that. And we are-- So the reason it's called inference scaling is that we are not touching any of the parameters of the model. The model is fixed, and we are just like at inference time, we are creating different ways to generate, to kind of produce generations from the model and use them to generate higher quality answers or answers to harder problems for the model. Um, the interesting part here is that for some of these problems, out of these ten thousand solutions, maybe like three or four of them were correct for a single problem. So it shows that how, how kind of, um- Uh, how important it is to, to scale the inference to really get to the core capability of the models.

Speaker 24:38 Um, but what is the difference, like, um, with, like, traversing all the possible answers and just, just pick out, like, the correct one? Like, if the number of samples that you have here, like, I saw, like, you, you need, like, ten thousand times of the sample.

Azalia Mirhoseini 24:54 Mm-hmm. Sorry, what's the question?

Speaker 24:57 What is the difference of this with, um, with, like, traversing all the possible answers and, uh, and s- and just select one?

Azalia Mirhoseini 25:04 So the possible answers need to be generated somehow by the model, right? So, so this is like the space of all possible answers to the model is way bigger, like, like, right? We can, like, if you do, like, a tree search or a random search, it's, like, way bigger, right? There's no way to, like, hit... Even, like, imagine, like, we are asking a human, like, "Just sit down and solve this problem," right? There is a limit to, um... Of course, there's no limit to human creativity, human's creativity, but there is a limit to how many generations we can have, right? Or if we randomize things, there's a whole lot set of answers that can be generated and it's -- and this is, like, highly, still highly sample efficient, right? Ten thousand is not a whole lot, especially, like, for example, for the math, um, F2F, some of these problem are really hard, like IMO level problems. Like, these are really hard problems that a tiny model like a seven and a Llama-eight-B can solve.

Speaker 26:08 Um, I was wondering how, how does the latency compare to, like, when you do actually generate all the samples, like, compared to that you get for all, right? Like, how does the latency-

Azalia Mirhoseini 26:19 Yeah, that's a great question

Speaker 26:20 ... is there a trade-off there or no?

Azalia Mirhoseini 26:21 Yeah, of course, there is a trade-off. Um, and we, we have -- we touch on that in the, in the lecture that I go more details towards this. The parallel samples, the good thing about them that they're -- they can be run in parallel. Um, so from a latency perspective, that's less of an issue. But of course, there is the frontier of cost trade-offs between how much compute you're spending and, and we will look into that. And it varies. It varies depending on the type of and complexity of the problems. Yes.

Speaker 26:51 So far we've been into when you have a verifiable domain, like, mapping to go in. How do you handle it when you don't have-

Azalia Mirhoseini 26:58 Yeah

Speaker 26:58 ... a verifiable domain?

Azalia Mirhoseini 26:59 Yes. Great question again. It's a very hard t- Uh, when you have verifiers, it's much easier. When you don't have verifiers, there's this whole set of research on how to train LLMs judge or LLMs reward functions or LLMs with tools becoming, and we are gonna learn about all that. We have a lecture just dedicated to verifiers, so.

Speaker 27:21 But-

Azalia Mirhoseini 27:22 Yeah.

Speaker 27:22 Just had a quick question, sorry. So, um, does this generalize to any positive temperature? Um, so, like, as the, like, when as the temperature goes to infinity, the kind of loss-

Speaker 27:35 No. I, I think-

Speaker 27:36 ... saturation more slow.

Azalia Mirhoseini 27:37 Yeah, we can, like, have the temperature being too high because it's just, it's gonna be gibberish, like the model doesn't generate. So we-- There's actually, like, a research project last year was, like, on tuning that or, like-

Speaker 27:49 I see

Azalia Mirhoseini 27:49 ... figuring out what temperature. Usually, if you go beyond, like, one point two or so it's, it's not great. Yeah. [chuckles] Yeah. But there, there are other tricks that you can do to kind of, kind of, uh, make the model be more, um, di-- like, have more diversity in the type of answers, uh, and that can be useful. Uh, all right. Um, so let's look at this.

Azalia Mirhoseini 28:20 So now, again, going back to the events of last year, year and a half, um, we had the DeepSeek model. I think it came out in-

Speaker 28:30 December

Azalia Mirhoseini 28:30 ... Decem-

Speaker 28:31 Like Christmas.

Azalia Mirhoseini 28:32 DeepSeek came out December twenty twenty-four. Uh, oh, wow.

Speaker 28:36 It's, it's been a-

Azalia Mirhoseini 28:37 [laughs]

Speaker 28:38 ... short time. Short time or long time, I don't know. Both.

Azalia Mirhoseini 28:41 Uh, so, so when models like DeepSeek came last year. So basically, a core kind of innovation in DeepSeek and then the o1 series Gemini Thinking and such were that now we are kind of bringing this fine-tuning and test-time scaling together because with this test-time scaling and the models themselves, we have this new, like, kind of engine that we can generate a whole lot of synthetic data. Um, for, for example, for problems like for math problems where we know the answer, we can have the model generate different, like, kind of answers that leads to that final, uh, final kind of, uh, golden answer. Or for the coding problems, we can use the model to generate tons of data during test-time scaling, uh, tons of, like, quality data to solve coding problems, and that becomes part of our training set now. We can use that to fine-tune the model, um, to, to become better. So that kind of bringing these two together became a big piece in thinking models, uh, and reasoning models. Um, and this, this is like a-- this is like a, an area that is like really, really important. There is just so much to be explored in this because again, the- there is no-- it's kind of open-ended, right? There's no boundary in how good the models can become with test-time scaling and then bringing that back to the process of training the model or fine-tuning the model to become better, uh, is very exciting, and that's kind of the self-improving piece that we are very excited about. Um, so a little bit about reasoning models. Um, in the Language Monkeys, what we were showing was the coverage. Like, if we had access to a verifiers, how, uh, we, we can see the kind of a log linear kind of scaling laws with the number of samples. Um, and o- OpenAI, when they released o1, uh, last September, um, they showed a, the, a log linear relationship this time with pass@one for these like har- hard, um, AIME, uh, SAT Benchmark, which is like a set of really difficult, uh, math questions. So as they're sho- what they're showing here is that as they're increasing test-time compute, and this is in log scale, um, uh, the, the accuracy, the pass@1 accuracy of the model goes up, and this is for test time alone. Uh, previously, uh, it... this has been shown for training, but it's very interesting that this kind of scaling applies to test time as well, um, without changing the parameter count of the model. Um, um, I'm gonna walk you through, like, one ex- a few examples of how this kind of reasoning or test-time, uh, scaling, uh, kind of is, uh, works with models like o3 and, or Gemini. So one motivation here is that for difficult problems, just like, uh, humans think a lot more, spend a lot more time or consider many different strategies, uh, thinking models do the same. And they may use a chain of thought or other techniques to just go, uh, go about, like, solving a problem. Um, and there are different steps to that. For example, um, problem analysis is one step. Like, the model first sees a problem and tries to analyze it. Um, then it can do the task decomposition, breaking a task into kind of simpler tasks that are more, uh, addressable. Um, then there is the self, uh, evolution strategies where the model kind of tries something, sees the feedback, for example, runs some tests on the code or use a, uses a calculator or just judges the answer, and it can use that feedback to optimize itself. Um, there is self-correction, and there are alternative proposals. If something doesn't work, the model can backtrack and kind of, um, try a different approach. These are some key princip- principles that the model, uh, probably at some point, uh, pieces of it were trained as part of the training kind of dataset, were curated by humans. But a big part of it is also the model kind of acquiring these skills during this fine-tuning process and RL, like RL from synthetic data process that the model does. Um, here is an example of, like, how, uh, o1 does analysis. Um, uh, the model's asked to write a bash script, um, uh, that takes a matrix and then outputs, uh, the transpose of the, that matrix. The model starts to ju- just like humans, like, starts, like, thinking what are the pieces, important pieces here? So the user is requesting a bash script. Um, let's understand the input and o- output, uh, formats of this matrix, um, and, and so on. So the model just thinks itself. Um, and amazingly this kind of like... it's like a progression of chain of thought. This is like helping the model finding the answer, but the difference between chain of thought and this is that the model itself is producing this chain of thought. Um, there's of course task decomposition, like in this case for the approach of br- the, the model thinks about parsing the input, building the matrix as an array of, uh, as, as an array of arrays, and so on. Um, there's also self-correction, which is a big important piece of, um... this is, this is an important piece here, and I don't know how many of you have seen that when trying with the models that you can see the thinking traces. Uh, and this is actually pretty common. Like, the model starts something and then in the middle says, "Wait, the correct for- Like, there's something wrong maybe, um, or... and maybe I need to fix that." And it's very, very cool that the model can do that itself. Um, so reasoning models when they came out, like models like o1, compared to a model like GPT-4o, which wasn't a reasoning model, um, they tend to be better in obviously the reasoning task. For example, in math calculation, data analysis, programming and such, they all kind of outperform GPT-4o, but not necessarily in personal writing or editing text and so on. Yeah. Question?

Speaker 35:15 Do you think the gains due to this asking for a reasoning trace is due to actually the act of generating this reasoning trace or just the act of asking the model to say or to re- decomposition at all? Like for example, if you said like, "Think step by step, but don't generate out loud. Like, don't actually generate the thought process," which was think about it without saying it.

Azalia Mirhoseini 35:39 Yeah. Um, I repeat the question. Like, do you, you're saying that is it the... Do you think the reasoning models are becoming what they are is based on the... I- is it based on the loud way of thinking?

Dr. Aakanksha Chowdhery 35:53 I, I think the question is that, uh, is it because of the, um, reasoning traces or is it because of think step by step, the chain of thought? Is it, like, is it the data that's making a difference or is it the, the instruction? Um-

Azalia Mirhoseini 36:05 It's... I mean, so the data basically has made the model kind of become a generalized thinker, right? Like, the model can... And the, all of it, all of these steps, like breaking down a problem, being able to backtrack, being, being able to do analysis, all of these are these skills that the model has learned during this thinking optimization process that now generalize to other things, and they're helping the model right now.

Dr. Aakanksha Chowdhery 36:34 So I think the other way to look at it is, as Azalia presented, repeated sampling. So if you had just a base model right after pre-training, it would be able to do some reasonings. But at the same time, it will generate different kinds of reasoning chains, and it doesn't quite know which one is correct. So a lot of the- What we will cover in train time or test time scaling really comes down to it learns which is correct. So pass@1 accuracy goes up, which is what you were seeing in the reasoning models, as opposed to, like, pass@K or coverage, uh, which is what you were seeing in the repeated sampling-

Azalia Mirhoseini 37:06 Mm-hmm

Dr. Aakanksha Chowdhery 37:06 ... results.

Azalia Mirhoseini 37:07 Yeah. And, and one other way to think around, like, yes, these kind of, like, kind of deep thinking and things that the model generates is actually helping the model every time to solve a new problem. Um, and although it's, like, very expensive for the model to generate those things, still, like, we want them to generate those because it le- le- leads to better answers. Um, any other questions? Yes.

Speaker 37:37 Can you use a different model for the reasoning step versus generating the final answer? Like, maybe a smaller or like a

Dr. Aakanksha Chowdhery 37:48 Um, so typically the reasoning capabilities have gone up with the model size so if, if anything, uh, you would use the larger model-

Azalia Mirhoseini 37:57 The larger

Dr. Aakanksha Chowdhery 37:57 ... for reasoning traces and then maybe you collect a bunch of reasoning traces and then have a smaller model summarize the answer.

Azalia Mirhoseini 38:03 Yeah. So the models like this is, like, at least currently, things may change. They like their own traces more. Even if their, the traces are coming from a better model, they tend to like their traces, their own generated traces more, and we're gonna have sh- talk about SWERL. This is one of the papers that, uh-

Dr. Aakanksha Chowdhery 38:24 The multi-step reasoning

Azalia Mirhoseini 38:24 ... the multi-step reasoning that we talk about. What if we bring a different model, not in the context of, uh, generating reason traces, but in, in the context of, like, evaluating a- and giving feedback, and we, we're gonna see that as well. But models surprisingly or not surprisingly, they like their own traces a lot more.

Speaker 38:44 Uh, for reasoning models, how are they taught to reason? Is it a hard-coded set of sequences they're taught to do traces or are they somehow fine-tuned to say, "We need to do this much or this kind of thing which will cause the traces to generate?"

Dr. Aakanksha Chowdhery 38:59 So I don't think there's a published piece of work that really covers this but, um-

Azalia Mirhoseini 39:03 It's both, yeah

Dr. Aakanksha Chowdhery 39:04 ... but it's a bit of both. At the same time, the base model did have the thinking capability to begin with, right? It would go do in some-- go do some amount of thinking. One thing that we will cover in the class is, uh, this notion of outcome rewards model and process reward model in how you can use that feedback to get the model back.

Azalia Mirhoseini 39:25 Yeah. There's always some bootstrapping like, okay, here are different ways. Like chain of thought itself, which is heavily used in the instruction tuning of dataset tell-- shows the model some ways of, like, how to think. Um, and there could be, like, some templates showing, uh, for the model fine-tuning, but the things that a model if you just ask them any-- ask it any question and it comes up with that, there is a whole lot of generalization in that. So models have gone way above, like, what the data or instructions that are used for training them. Yeah. Yes.

Speaker 40:02 Uh, in repeated sampling, is there a way to make the number of samples dependent on problem difficulty?

Azalia Mirhoseini 40:07 Please, uh, speak up.

Dr. Aakanksha Chowdhery 40:08 The number of samples, can they be dependent on problem difficulty?

Speaker 40:12 No, just in general.

Dr. Aakanksha Chowdhery 40:13 I don't think there's a published piece of work that does that. Uh, but, uh-

Azalia Mirhoseini 40:18 Yeah, we just... So, so when, when-- So there is follow-up work to that where we use a reward model and then you can use that. So assuming the reward model has some notion of complexity, if you haven't solved it, it can guide, like, more sampling or whether it's repeated or, or parallel. Uh, but that's definitely an interesting, uh, direction to explore. Yeah. Okay, we need to get to the... Oh, we haven't... Right.

Dr. Aakanksha Chowdhery 40:48 Yeah. Let me cover these. Okay, so we-- you now know about large language models, and then you heard about how they learned how to think, and then they learned how to reason. So what's next, and why does this-- why is this course relevant? So a lot of what this course is about is that large language models as chatbots or as, uh, reasoning models are basically still single turn, like, or, or just, like, in the chat format. So they're not accomplishing a task for you. They're fun to interact with, but they're not necessarily accomplishing a task for you. What has happened this year, uh, almost in the last few months, and it's been surprising, is that agents like Claude Code or Deep Research have really enabled people to do real-world workflows. Um, so they are agentic workflows which can achieve tasks that you ask them to do end-to-end. So for example, if you want to go research and figure out where should I, uh, rent a home, uh, perhaps for, uh, the entire, uh, year for, uh, if I want to take a class at Stanford, the model would actually be able to do a whole bunch of analysis and, like, go look at a lot of different websites and actually summarize the results and give you pros and cons of different places. This used to not be possible before. So this, uh, we'll cover a little bit of how that, that becomes possible and how this course allows you to learn about that. And then Claude Code, for example, if any of you are Codex, uh, which is the coding agent from OpenAI, if you're using that, what you'll see is that just by giving instructions in English, you can modify files or you can, like, figure out test cases and, and whatnot. So it, it really, uh, is become a coding productivity tool in, in the day-to-day workflows for software engineers at this point in time. So what, what is the transition from LLM to agents? So as I was giving you the example of Deep Research and of, uh, coding agents, basically the goal-- Basically now the, uh, model can be given a goal, and it will plan out the steps, then it will go interact with, uh, the environment, and based on the feedback, it will correct its steps until it achieves the goals or it will come back and say, "I cannot achieve the goal." So this notion of, like, having some notion of a goal, uh, taking actions towards that goal, getting the feedback- And then deciding when to stop, that's what makes agents, uh, different from the chatbot times of what we were doing before. And, um, this also might require interacting with tools, uh, which are external to what the model itself is doing, getting some inputs from there. But it still stays on track of like whatever task it's, uh, choosing to accomplish, which might mean that it has to have some form of memory to keep track of the task that it's trying to accomplish. So what has been, uh, accomplished? So like in, in, in several cases, in simpler cases, you can achieve these, uh, um, these end-to-end goals, like deep research can like be accomplished end-to-end. But in most scenarios, you are still having very static workflows. So today, what exists is closer to agentic workflows where you have, um,

Dr. Aakanksha Chowdhery 43:49 some sort of an input where you're giving a model the goal and then, um, what this slide is showing you is that one model is perhaps, um, giving the output like a solution, and another model is judging it and then deciding based on that whether the solution should be accepted. So that's like one orchestration framework that's possible. Uh, deep research will be more of the second one where you're basically generating mult-- uh, you're calling the LLM on multiple possible inputs and then aggregating the, um, things to get the output which will be a summary of the deep research results. So this is a very cartoonish abstraction of what agentic workflows might look like, and you can generalize from there. But instead of having like very open-ended loop that I was showing you here where you're supposed to go take an action from the environment and come back and do some feedback, um, it's easier for open-ended problems to like construct this graph by hand of like how a human would do it and then like get this feedback perhaps from an LLM which is the LLM evaluator here. So a lot of the real-world workflows still have this paradigm, but in, in certain cases we are starting to see signs of life for what's possible in this slide, and coding being one and research being another one. So typically, the workflows will have things like LLM calls. So you would basically call an LLM with an instruction or some form of like input, and then you're asking the LLM to give you an output. It will have some form of verifiers. We'll cover verifiers in a whole lecture by itself. It will have some form of critics or judges, which is effectively LLM as a judge, uh, paradigm. Uh, there might be tool calls. For example, for deep research you have to actually go search the web to figure out what exactly is the content that you should be looking at. So there might be tool calls of that form or, uh, getting, uh, some value around what exactly is the weather. Um, and similarly, search would be another tool call. And then you would orchestrate these in, in some form of a workflow. So a simplest workflow here would be prompt chaining. So similar to reasoning models where the task gets decomposed into subtasks, prompt chaining will simply be like you have chained a bunch of different subtasks that you're supposed to go accomplish to achieve the end-to-end goal. There might be routing. Routing works for complex tasks where you say, "Okay, if this is very complex, then go do this more complicated set of LLM calls, and if it's less complicated than do this simpler workflow." Um, there might be parallelization where you... Like deep research is one example where you, you can have multiple LLM calls work simultaneously on researching different keywords that you give as input to the LLM and then, uh, finally you aggregate the output. Or you might break the task into independent subtasks and then, uh, combine, uh, the solution. Or you might have this notion of orchestrator where you basically are actually using some sort of a, in, in simple words, LLM manager. So a central LLM is actually doing the planning. So in Claude Code you actually start to see that, that there's some notion of like a plan that it comes up with and then based on that plan it will like make subsequent LLM calls. You might have an evaluator or a judge. So instead of like getting feedback from real world like user or from some sort of a actual running a unit test, it ac-actually might just use LLM as a judge and we will have, uh, some homeworks that will cover this aspect. And then you might have verifiers. Verifiers are things where you can actually verify the output. So in code, for example, if you run the code, how do you know that this code is correct? So typically as software developers we write unit tests. Uh, so similarly verifiers might be running some kind of unit test to check whether what LLM generated is correct. And in domains which are verifiable, uh, math, code, um, and other domains that are more rule-based, uh, this verification is a good way to give feedback back to the model so that, um, it can correct its steps. So most of these, um, workflows mean that the, uh, the LLM needs to be better at planning, it needs to be better at multi-step reasoning, and

Dr. Aakanksha Chowdhery 47:51 it needs to be better at self-improvement. Like when it makes mistakes, it needs to be able to correct itself. So effectively these are sort of new paradigms that the current set of LLMs just with reasoning were not quite co-accomplishing, and these are some of the topics we do plan to cover in subsequent lectures. And just to, uh, drive the point home around, uh, coding agents, so this is a very simple example. This is actually, um... I should have updated this slide to for Claude Code. But basically what this is showing is that you have an LLM agent and, uh, it's interacting with the computer these days just with a terminal. You give it an instruction where you might tell it that you want to implement a test, um, and then it has some navigation of repository, searching of files, all of these tool calls around like viewing files, editing lines, and then it will go execute a bunch of commands in the terminal, and based on the output it might be like, "Okay, I need to go edit this other file," or, "I need to go look at this other file." So if you look at this loop, this very much mirrors, um, what was... So this was not quite reliable last year, and it's just starting to get reliable in what we are seeing in the, uh, realm of coding agents now. Yes.

Speaker 49:01 Can I ask why you think it's more reliable than, I guess, the general architecture is? I mean, I would say it's not changed that much. Like, the reasoning or-

Dr. Aakanksha Chowdhery 49:11 I mean, I think the paradigm is very much the same. It's, it's mostly a matter of po- more powerful models and then better RL. The RL, uh, the verifiable rewards is working. The training time scaling is working well.

Speaker 49:24 That's... How, how could you, I guess, enhance really, you could get, I guess, better performance without necessarily just throwing more compute power at everything?

Dr. Aakanksha Chowdhery 49:37 I mean, compute or data, right? One of those two. But, like, just also once the models start to get better, there's the self-improvement loop that kicks in. Uh, because you can now generate tasks and the tasks become more reliable. So in the published pieces of work, if you look at, say, Code Monkeys, which we covered last year, um, if you can generate unit tests, um, for whatever code the model generated and then see if the generated unit tests are making things better, then that, that, that's a very good way to verify things, right?

Azalia Mirhoseini 50:09 Yeah, like model-based. It seems like to some extent if we have good verifiers, um, we can make the model generate the, the right code.

Dr. Aakanksha Chowdhery 50:21 Yeah.

Azalia Mirhoseini 50:21 Um, but the question is how do we do that? And there are ways to do it, and we, we, we would be happy to... for you guys to take research projects in that domain too. But obviously it's an active, um, research area, but there are ways to make, um, improvements that we can discuss.

Dr. Aakanksha Chowdhery 50:38 I, I think what you're alluding to is this notion of generator verifier gap.

Azalia Mirhoseini 50:41 Yeah.

Dr. Aakanksha Chowdhery 50:41 So it's easy for models to generate a whole bunch of nonsense or sensible set of like reasoning traces or like useful set of content. But at the end of the day, whether that's useful or not, we need feedback loop for that. And if you're creative writing, how much feedback can you get? So human feedback ends up becoming a bottleneck. In domains where you can have good feedback, that's where it's possible to continue to improve the model. And that, like, but veri- getting robust verification, a lot of robust verification is hard. So, uh, Azalia will cover a paper that she did in her lab about like how to combine verifiers and... But veri-verification continues to be one of the bottlenecks in this space to make them better.

Speaker 51:20 Got it.

Dr. Aakanksha Chowdhery 51:21 Yes.

Speaker 51:21 Um, just like intuitively, I think pre-training, you get abilities and files within that such a large model.

Dr. Aakanksha Chowdhery 51:28 Mm-hmm.

Speaker 51:29 So just like to me intuitively, I'm trying to figure out why there's such a big jump.

Dr. Aakanksha Chowdhery 51:33 Mm-hmm.

Speaker 51:33 Because like we're sort of providing more signal-

Dr. Aakanksha Chowdhery 51:36 Mm-hmm

Speaker 51:36 ... and we're trying to elicit abilities that I guess shouldn't exist during pre-training as well. So yeah, I, I guess like I'm just not-- it doesn't fully resonate with me. Like-

Dr. Aakanksha Chowdhery 51:48 I, I think this is still an active area of research, so there is different set of opinions around what really improves the model. Is it RL or is it like the, the pre-training the diverse data by itself? And I, I don't think there's a single point of consensus at this point in time. Both processes help. Um, what you just said, uh, which I will repeat for the class, is that if pre-training is the place where, uh, the model after repeated sampling should be able to at least have one solution correct out of if it was generating a large number of samples, then, uh, this notion of giving feedback should improve the passive one accuracy but should not improve the model. Why is it like such a big jump? But I think, um, that whole loop is not completely well understood. It's like the first signs of life as it starts to get commercialized, but I think there's a lot more research still open in this area.

Speaker 52:37 Yeah.

Dr. Aakanksha Chowdhery 52:37 So that's only one, um, way of thinking about it. There are other, um, at least some signs of life which say that you can continue to do RL and then that will continue to improve the model. Yeah. Okay.

Azalia Mirhoseini 52:51 Yeah.

Dr. Aakanksha Chowdhery 52:52 So, uh, so this is, uh, basically covering a lot of, uh, the same set of points, but, uh, one abstraction that is worth taking away from this particular chart is that even when you give the model a goal, it has to clarify the user intent. So it's like not always obvious to it what the mo- user wants. And then it might go search for relevant files and then whatever, um, set of, uh, actions it chooses to take, oftentimes it needs some form of verification. In this particular case, it's based on, um, passing the test and it might actually generate the test that it needs to pass, which, um, which, uh, is also, uh, mentioned here. But the, the key idea is that if you give it a task, how does it come back and how do you know that it's gonna complete what you asked it for? Oftentimes, the users will not specify the problem well enough. So asking-- clarifying the user intent so that it knows what to go look for and how to verify it starts to become, uh, important. And in models like o3, I think they have seen a lot of these traces end-to-end. So they are able to like do the planning, the reasoning, the multi-step reasoning, and like come back with the conclusion. But, uh, a lot of this, um, is needed for, um, completing tasks end-to-end or having an end-to-end goal. And where this has been super useful is, uh, if you have repetiti- uh, repetitive tasks like code migrations or version upgrades, or if you need to restructure the code base. Or if you have tasks that involve data engineering, where you have to basically do some sort of extract, uh, the data and then do some sort of clean up, clean up on the data, or if you have to do some sort of data warehouse migration. So a lot of this work tends to be extremely, um, repetitive, and it's much easier to just delegate it to coding agents. Uh, same for unit tasks. Oftentimes, like much easier to like start to give it to, uh, coding agents. Another area where agents have become, uh, extremely prevalent is in customer support. Uh, it's one of the most thankless jobs that, uh, when, when you are on the customer support side of things you have to do, and using LLMs to do that definitely makes, um, streamlines the experience in interesting ways. So one simple set of examples ends up being like you can use them to do live transcriptions, so that gives you a very nice record and, and you're seeing that even for meetings these days. Another simple is-- example is that you can have knowledge assist. So if you have a database of information, the, the customer support agent does not need to know everything. They can consult with the LLM and, and get an answer and surface the relevant article, uh, which is better than having, um, just search, um, like an index and search by itself. Uh, smart reply. So in, in chat, oftentimes this has existed for a while, but you can use an agent to, like, give, uh, chat responses. And then having a call summary can help you, um... You can use, uh, the call summary to really, uh, improve the customer experience by itself as well. So overall, I think in the customer support area, there's multiple, uh, companies that are going after this area, but, uh, there are different segments of the problem that can be addressed using LLMs, and they have been super useful. And then there's also the end-to-end stuff that's starting to happen here. And then the third example that we will use, and actually some of the homeworks, is that if you have very complex topics and, uh, you want to provide a comprehensive report, um, earlier it used to be that you had to do a literature review, and then you had to summarize each paper, and then you had to synthesize these things. And these days, you can just give it to an LLM and actually do it for you and, um, study maybe more articles than you would. So typically, if you give it an example like, um, say, twenty twenty-two Winter Olympics opening ceremony, it will identify what references to go look at. Then it will construct an outline of like, okay, these re-references are relevant or not, and then summarize each of the references, uh, as to what is the relevant content, and then combine them to create a full-length article, um, which is, um, very impressive in certain cases, and, and you'll get to try it in one of the homeworks. And then even, uh, more, uh, forward-looking, um, these research agents are starting to be used for, uh, as AI scientists. So basically, they're starting to assist scientists in, say, uh, solving math problems or solving science problems. So here the LLM is used as a brainstorming thing to come up with ideas. So the idea generation phase. Then the experiment iteration phase, it might actually help you iterate on the experiments that you are hoping to work on. This is from the AI scientist paper. And then in the paper write-up phase, it will help you improve, uh, the, the paper write-up by itself. And what's interesting, uh, is that even, even though these models hallucinate just the i-- uh, just the notion that they can come up with so many different set of ideas, as an AI scientist, it might actually brainstorm or, like, give you ideas outside what you would have thought if you had just, like, taken a bunch of courses or, or if you're a researcher and you've been in the field for a long time. Sometimes the reading the web, uh, allows these LLMs to come up with ideas that are way outside the box and can be a very good brainstorming, um, way. Uh, so these, these have been super useful for, in, in the AI scientist style of work. Okay, so let's move to course logistics. But before that, do we have any questions? Yes.

Speaker 58:10 Yeah. So I just want to clarify. So, um, is reasoning and chain-of-thought baked in training, or is it just, uh, prompt engineering that was accidentally discovering these kind of capabilities in larger models? And if so, uh, are there other emergent behaviors besides, uh, reasoning and chain-of-thought that you think might be discovered?

Dr. Aakanksha Chowdhery 58:30 So, um, I mean, it was not baked in by, um, by design. Uh, it was, uh, it was basically discovered as in, like, it was... I mean, we gave it hard problems, and then we saw that it was basically, uh, by having reasoning chains, it was doing better. So the GSM eight K was the first paper that kind of showed signs of life of this. And then, uh, with larger model, uh, like PaLM, we actually saw that this was a very big deal. And one of the examples there was that it could explain jokes, which was very impressive. And then from there, the reasoning models have emerged. So it was not-- But it has read all of the web, so it's n-- it has definitely seen data which is, like, more, um, methodical and systematic.

Azalia Mirhoseini 59:11 But the reasoning models are trained to be reasoning more and more, right? So it's-- they didn't-- The entire reasoning is not an emergent behavior. The reasoning models are trained to be, to be thinking. But, like, the models are convergent, so the models are going to be trained such that they know when to-- they need a lot of reasoning and when they don't and, uh, and to generate the answers. But chain-of-thought originally was an emergent behavior. They noticed that, oh, if we explain things, the model gets better.

Speaker 59:45 Are there any kinds of other, um, emergent behaviors in larger models?

Dr. Aakanksha Chowdhery 59:49 I don't think it's-- I would see it as emergent behaviors per se, but I think as I was mentioning, we usually go looking for certain things, right? So in the agentic workflows, what we're looking for is planning, which is a form of reasoning. We're looking for multi-step reasoning-

Azalia Mirhoseini 1:00:02 Mm-hmm

Dr. Aakanksha Chowdhery 1:00:02 ... which we will cover in class as well. And then we're looking for self-improvement or self-correction. So all of these capabilities would be nice to have and, like, what gets the models there is a set of questions that are worth-- like, there are papers on this kind of thing.

Azalia Mirhoseini 1:00:15 Yeah. Like self-correction, backtracking, things like that is like, yes, you could call it emergent, but you could also say-- think about it that they have seen this kind of behavior, and it's kind of like it's reinforced in the way that they're fine-tuned. So it's... Yeah, it's hard to say. Okay, um, so let's go through the, the, the logistics for the class. Um, here is

Azalia Mirhoseini 1:00:45 a list of all the amazing topics that you're going to learn about. Um, they are, um... Yeah. This is very cool.

Dr. Aakanksha Chowdhery 1:00:55 I, I think the main thing to remember here is that the overall theme stays the same. Uh, and then we'll also have guest lectures around from, um, folks in frontier AI labs and covering things all the way from, say, how have, uh, has-- how has post-training evolved, or say even multimodal agents and, uh, in robotics. So it's, it's a, it's gonna be a mix of lectures and, uh, guest lectures and, um, and of course your project presentations.

Azalia Mirhoseini 1:01:23 So here is some logistics that you all know. The, the prereq are there. Just make sure that you, you are comfortable with these prereqs as you, uh, register then in the class. So we have the external website, but, uh, make sure that you're, uh, you check Canvas. That's where we send out the latest updates at all times. Um, and, um, we will also upload the-- we try to upload the lectures before we start every, every, in every class so you have access to the lectures. Uh, and we will also have-- we already have the due dates for all the assignments and all the projects, so you can check it out. Um, so this quarter we are going to have three homeworks. That's the difference from between this class and, uh... That's one of the differences. So we have one more amazing homework for you all that we have, uh, designed, and the TAs have done a great job be-- uh, and that's helps you, um, hopefully learn, uh, learn these topics better and more in depth. Um, and then we also have a course project. So the course project is where you can kind of unleash your creativity and, like, your, your way of, like, building agentic systems or going deeper into a question, uh, and, and designing experiments around it and see what works and what doesn't. Um, we will provide some, um, kind of examples or suggestions, but it could be completely on you how to design this project. We'll have some examples from last year, like the successful projects from last year as well. Uh, it's, it's gonna be uploaded to, um, Canvas or, uh-

Dr. Aakanksha Chowdhery 1:03:06 Actually, we'll display them on the website.

Azalia Mirhoseini 1:03:08 Oh, on the website. Oh, cool.

Dr. Aakanksha Chowdhery 1:03:09 Yeah.

Azalia Mirhoseini 1:03:09 The public website, so you can all see that. Um, so yeah.

Dr. Aakanksha Chowdhery 1:03:14 So for the course project, I think the main thing to remember is that, uh, we will have API credits, and you can work in teams of two to four people, right?

Azalia Mirhoseini 1:03:24 Yeah. Or up to four, I guess. [laughs]

Dr. Aakanksha Chowdhery 1:03:27 Up to four? Okay. [laughs]

Azalia Mirhoseini 1:03:28 It can be one if you really want that, but we, we suggest you team up with others. Um, at the very least, you have more credit collectively, so you can run more experiments. But then you might also, uh, find a friend, collaborator along the way, and you can, um, um, do something bigger. Um, and we are going to have this, um... What is this? The course project. Yeah. We-we're gonna, uh, we're gonna let you know about some, uh, some of the previous years and some ideas that you can take. Um, here are some course project examples that are acceptable. For example, a new evaluation data set or a new benchmark. Um, you can design a project around the reliability of a, of an agent-agentic system that already exists. You can take a benchmark that exists and try to hill climb on it with whatever idea, great idea that you have. Um, and then... Or you can just be-- we are going to have a whole lot of papers covered in this course, and we have-- we are showing all of that to you. They're all on the website. So as you, as we give these lectures, we kind of expect you to also have read those papers or read the papers along the way and develop better and deeper intuitions on how these methods work. And y-your project could always be improving those or, like, questioning one of the decisions that they have made and trying to, uh, change that or, uh, or evaluate that. Um, negative examples are, like, a survey paper or, like... We just don't want-- We want something researchy here. So we just don't want, like, just an app that you put together, and you just show us something. It has to be like, "Here is, like, here is the hypothesis. Here is the question that we wanted to answer. Here is the type of, like, improvement we wanted to see or the type of, like, uh, properties we wanted to analyze, and then we have built that," right? We want something, something that, um, uh, something more than vibe coding basically in this, um-

Dr. Aakanksha Chowdhery 1:05:32 Uh, in, in terms of milestones, I think the-- it's worthwhile to remember that you wanna start early. So it's-- you need to have a project proposal, um, somewhere around, I think we, we put it early October.

Azalia Mirhoseini 1:05:44 Yeah.

Dr. Aakanksha Chowdhery 1:05:44 And then, uh-

Azalia Mirhoseini 1:05:45 It's on the website, yeah.

Dr. Aakanksha Chowdhery 1:05:47 It's on the website. And then for the midterm project presentation, we do want you to have some progress. So two weeks after the project proposal, we do expect you to have, like... So start thinking about, like, what experiments you want to run by the time you get to the project proposal so that you actually go run them for the midterm projects. And so we do expect you to have made some progress in the midterm project presentation. It should not just be, "Here is a proposal." And then the final report has a lot of weight, and then the final poster presentation, uh, will be the end of the quarter.

Azalia Mirhoseini 1:06:20 Um, yes.

Speaker 1:06:22 Let's see. If, uh, we do a project that is similar to other ones that we have seen in previous years, like CS190. So say, like, uh, all our own things technically across the classes.

Azalia Mirhoseini 1:06:39 Hmm. I, I think so if you add to it, right?

Speaker 1:06:46 Okay. Okay, okay.

Azalia Mirhoseini 1:06:46 Yeah.

Dr. Aakanksha Chowdhery 1:06:46 CS190, uh, 190 is independent study, right?

Speaker 1:06:49 Yeah, independent study.

Dr. Aakanksha Chowdhery 1:06:51 Yeah, yeah. That's fine.

Azalia Mirhoseini 1:06:51 I, I think-

Dr. Aakanksha Chowdhery 1:06:52 I mean, as long as you're doing actual work, and it's not just like reusing the work from-

Azalia Mirhoseini 1:06:56 [laughs]

Speaker 1:06:57 Yeah, yeah.

Dr. Aakanksha Chowdhery 1:06:57 Could be an exact copy of course. Yeah. Yeah, yeah.

Azalia Mirhoseini 1:07:01 Um, we had, uh, we had students publishing papers out of their projects, uh, in the last quarter. So, uh, in the last, last time we taught the class. So that's something you could-- it could be one of you or many of you this time, again, uh, turning your research project, working more on it, and then, uh, turning it into a publication at, um, uh, conferences. Um, and so here is a save the date [laughs] for, for our poster presentation. It's gonna be December 12, 4:00 to 6:00 PM. So, um, we, we would want you to be there and present the posters. We'll have people from industry joining, and you can... That's where you can brag about what you've done and just meet new people. Um, here is the course, uh, grading rubric. So, uh, we have three homeworks. That's 50% of your, uh, of, of your entire grade, and then another 50% is on the project. And the duration of these are kind of like aligned with how much, uh, how much the grading, um, grading is allocated to each of these. Um, and we, uh, of course, expect you to, uh, honor the honor code. Um, and let's see. Office hou- hours will be posted on Canvas. Um, you can ask your questions. We really, really encourage you to ask questions on EdStem, uh, and make it public so others... Probably if you have a question, um, it's very likely others have the same question. So please help us, like, to broadcast this to everybody. Um, um, EdStem is, um, again, you can ask questions, and then Gradescope is where you submit your project milestones and everything else. Yeah, the homeworks and everything. Um, we have the following late policy that we think is kind of like between the generous kind of like, um, side, uh, uh, on the gener- generous side of things. And because the class is big this quarter, we really can't make any exceptions, so try to use these, um, kind of late days, uh, wisely. Uh, audits are not allowed, but we will have the, the course, the-- we will have the videos on-

Dr. Aakanksha Chowdhery 1:09:22 Course, uh, videos on YouTube eventually. Yeah.

Azalia Mirhoseini 1:09:24 Eventually. And that's it. Any questions? No? All right.

Dr. Aakanksha Chowdhery 1:09:35 Okay. Thanks everyone.