How We Scaled Kimi K2.5 | Zhilin Yang's full GTC 2026 Keynote

Speaker 0:06 [upbeat music]

Speaker 0:12 Hi, everyone. Thank you so much for the introduction. It's great to be here. Uh, it's great to have this opportunity to share with you guys, uh, some of our latest progress and efforts. So, uh, one of our major pursuits, uh, is to build better open models, and we believe in democratizing intelligence. With open models, you can deploy anywhere. It can be on your local servers, it can be on the cloud, and you can access every single bit of the weights in the modeling that instead of just, uh, using a black box. And this is one of the slides that I took from Jensen's, uh, talk earlier this year at CES. So as you can see, open models are quickly, uh, closing the gap, uh, with, uh, proprietary models and it's reaching the frontier. And we believe that with better and better open models, we're going to, uh, make intelligence more accessible to anybody in the world, in

Speaker 1:12 every corner o- of the world. But open models cannot be just open, they c- have, have also to be, uh, great. So, uh, in this talk, we're going to discuss how we make open models great. Uh, so a- as we know, uh, scaling is a primary driver, uh, for a lot of progress, maybe all of the, you know, major AI developments that we have witnessed in, in the last few years. And here we're going to discuss how we scale our model in different dimensions. So on the left-hand side, the first figure you see here is kind of the, the standard scaling law. So you- on the x- x-axis, you have the, uh, log of the number of training tokens, and on the y-axis, you have the log loss. And as you scale the number of training tokens, you get a lower loss. But here the point is, we're, we're not going to just scale the number of training tokens, but we also want to, want to improve, uh, the token efficiency. Uh, meaning that we want to move this curve to the left-hand side so that we can achieve a lower loss, a much lower loss using the same number of training tokens. And this can be achieved by having better architectures and optimizers, as w- as we'll discuss in, in our later slides. And the second scaling dimensions that we're very interested in is to scale the context length. So as you can see in the second, second figure, uh, if we increase the context length, then we can, uh, have a much higher accuracy in terms of predicting, uh, the token loss at a given position. Uh, and this means that we can increase the capability of the model to achieve more complex tasks by increasing, uh, the context length. So this is the second scaling dimensions that we're going, going to talk about. And the third scaling dimension is the number of agents. So we introduce this new learning paradigm of agent swarms, where we can-- we don't just rely on a single agent, but we also, uh, orchestrate a swarm of agents that can accomplish the sub-task in parallel so that we can increase the task capacity. And we can translate all of this in- into the language of agents. So if you look at token efficiency, it's mostly about having a stronger prior so that you can, uh, have more efficiency when you do agent IO to search for a, uh, better solution. And we think about long context, it's, it's mostly about increasing the context then, so that you can have a longer-running agent. It can probably run for, uh, days or even weeks or month to accomplish more, uh, uh, more, more tasks, more, uh, complex tasks. And about-- and for agent swarms, it's, it's, it's another dimensions that, that add to it, and at the end of the day, we're going to have a swarm of agents that each of them have a super long context, and each of them have a very strong prior for us to, uh, search, uh, in this

Speaker 4:16 entire agent IO system. All right, so we're gonna start, uh, from token efficiency. So this is one of the most, you know, classical figures in the history of machine learning, right? So it's taken from Ka- Kaplan et al., and it basically says that if we scale proportionally the number of training tokens, the model parameters, and also the amount of compute, we can get lower and lower loss. And this is, you know, one of the major breakthroughs that, that the entire community, uh, has achieved in the last few years to, to get, uh, better intelligence. But here, what, what we're interested in is, is to have better and better, uh, token efficiency. And, uh, here's the thing. So one thing that I would like to emphasize is that token e- eff- efficiency is not just about efficiency. It's actually also about improving the upper bound of intelligence. So here's, here's, here's why. So, uh, suppose you have, uh, say, fifty trillion tokens, fifty trillion high-quality tokens, and then you apply this new optimizer, maybe the Muon optimizer, and then all of a sudden you have a two times token efficiency. So it means that it's almost like magic that you get, uh, equivalently one hundred trillion tokens. Uh, and nowadays we are scaling towards the data war, and we're hitting, you know, the data war, and the amount of high-quality data is quite limited. And if we suppose that it's a constant amount, then we increase the token efficiency, it means that, uh, we're going to get better intelli- intelligence out of it. It's not just about infrastructure efficiency, it's about, you know, better, uh, intelligence. So, so this is why we spend, you know, a lot of efforts in this aspect, because it's going to push, uh, the frontier of, of intelligence. And Muon optimizer is one of the things that we have heavily invested in, uh, since

Speaker 6:17 last year. So it's a second-order optimizer, and basically every single gradient update is transformed in a way that each entry is orthogonal to each other. And this is very different from the traditional Adam optimizer. And if, if we implement this optimizer properly, you can get a two times token efficient- efficiency improvement. Uh, so we, we are one, uh, we are the first, uh, work, we published the first work to demonstrate that Muon optimizer is actually scalable for LLM training. And these are two key techniques that e- we employ to make it effective for large scale training. So one of them is weight decay. It, it is critical for scaling to larger models, and the second is, uh, we wanna ensure a consistent RMS updates compared to Adam. So we have this, uh, uh, adjustable coefficients that is applied, uh, to each update so that, uh, the, uh, resulting RMS is going to be comparable to Adam. And, uh, to make Muon, uh, memory efficient across all this, uh, NVIDIA GPU clusters, we also develop a distributed Muon optimizer implementation that partitions the states across, uh, the data parallel group so that we can have a very efficient imple- implementation for the Muon optimizer. And these are some of the results, uh, that were presented in the paper. So as you

Speaker 7:47 can see, with, with the same number of parameters and the same number of training tokens, we just replace the original AdamW optimizer with the new Muon optimizer. It's going to, to improve the performance across the board, uh, signi- uh, significantly. But there was, uh, this new challenge that we encounter when we try to scale it up further. When we try to scale Muon for a 1 trillion parameter model, we encounter a new issue, uh, about training instability. So as you can see on the left figure, uh, we, uh, we observed that the max logits quickly explodes and quickly exceeds 1,000, and the typical values for, uh, for, uh, training, uh, for this max logits i- is about, say, 50 or maybe less than, uh, 100. But for, uh, for Muon, it quickly exceeds 1,000. And at the same time, we observe training, uh, divergence on the left-hand side. If you look at the training loss, uh, it goes down a bit, but then at the end of the day, it explodes and it cannot converges, uh, as expected. So this is one of the technical challenges that we have to, to address. Uh, and the solution to this is to introduce this new technique called, uh, QK clip. So basically what it says is that for each attention head in this entire neural network, we're going to... in the forward pass, we're going to compute the max logits, and then we're going to calculate a dividing factor that can be applied to each, uh, key projection as well as the query projection so that we can, uh, sort of clip the maximum, uh, value of the query and the key to, to sort of constrain it into a, a given range so that, uh, we're not going to have explosion anymore. Uh, so these are some of the empirical results. On the left-hand side, there are two curves, but they are strictly overlapped with each other. So these are the training curves before and after applying the clipping technique. So you can see, uh, the clipping technique does not affect, uh, the training loss, uh, uh, uh, decrease at all. But on the right-hand side, if we inspect, uh, the intermediate metric, if we inspect the max logits, it's going to be effectively, uh, constrained. So it first explodes as before, but at the value of 100, it's going to be clip at a constant value for a long time, and then after a certain number of steps, it will just naturally go down. Uh, so the neural network sort of, uh, find a way to, uh, constrain the maximum value of the, uh, max logits to ensure a stable, uh, training process. And at the same time, it doesn't affect, you know, the training convergence as shown in, uh, in the, in the left figure. So we employed this technique in our, uh, K2 model training and successfully scaled it to 1 trillion parameters.

Speaker 10:49 And this is, uh, the first example of a large-scale Muon training in the history of machine learning. And the second dimension that we're very interested in is, is long context. So this is another figure. It's probably less known. Uh, it's, it's, uh, one of the hidden gems in these papers. Uh, so instead of just, you know, pushing down the training loss by training on more tokens, it has some, uh, in- uh, insights from another perspective. So as we can see, this is a comparison between transformers and LSTMs. So on the left-hand side, we can see the transformers achieve a lower training loss given the same number of parameters and the same number of training tokens as expected, and this is why transformers become, you know, the, you know, the sort of the de facto architecture that people are using right now. But on the right-hand side, it's very interesting to see that transformers are actually better because it can improve through the whole context. So the x-axis is the token index in context, and if you increase the token index, you can see that the, the training loss of transformers actually drop by a lot. If you just continu- continually increase the context length, the loss just continuously drops down. But if you look at, you know, the curve of LSTM, it just is saturated after a certain, uh, number of tokens. It means that transformers have this bet- better capability of capturing longer context, and this is, this is what makes it, uh, better, uh, be- because if it, if it go back to like 10 years ago, uh, people use LSTM for tasks like machine translation, but it is not good for, for example, understanding entire code base or running a super long agent trajectories to solve, uh, a, a, uh, for example, writing Linux kernels, uh, s- from scratch. It's not going to be accomplishable by LSTMs. So this is a very s- uh, much needed capability in the era of agents because tasks are becoming harder and harder, uh, and we need longer and longer context. Uh, so the research idea here is to develop a better architecture so that we can, uh, efficiently scale to a longer context length and at the same time, achieve a lower per token loss at, uh, larger token indices. And this is the motivation, uh, f- for which we introduce, uh, this new architecture

Speaker 13:21 called, uh, Kimi Linear. And it's, uh, contains this new, uh, linear attention variant called, uh, Kimi Delta Attention, which improves the original gated, uh, delta rule, GDR, uh, by improved recurrent memory. I will show the details later. And at the same time, we're going to mix linear attention layers with full attention layers using a one to three ratio so that it can balance between, uh, these long

Speaker 13:51 context capabilities and at the same time, having a more efficient, uh, implementation. So this is, uh, some of the formulation. Uh, the basic idea is simple. If you look at linear attention, uh, in the original formulation, uh, the memory is going to be global. So there is a global single decay factor that is applied along the way. So it means that, uh, basically if, i- uh, there are only two cases. In one cases- in one case, you're going to forget basically everything, and you are not going to retain any information. And in the second case, you can choose to retain, you know, almost everything, but at the same time, you, you don't have the capability to leave out some of the unnecessary information in this long context. So we introduce this key idea of having a fine grain, uh, decay factor as shown in this highlighted alpha term. So it's going to, uh, instead of s- being a scalar, it's going to be a, a diagonal matrix, which controls, uh, the decay rate for each channel so that we can have two possibilities. For some of the channels, we can, uh, decay really, really slow, meaning that we can retain this long context information across a very long, uh, range. And at the same time, for the other channels, we can sort of quickly forget, uh, the information from the past indices to refresh it and observe new information. And this is, uh, to increase the e- expressivity, uh, of this model. And of course, to leverage modern GPUs, we have to use this chunkwise, uh, formulation so that we can, uh, parallelize the computation on modern GPUs. So the first equation here is the chunk, uh, chunkwise, uh, formulation of, uh, Kimi Linear. But as you can see, this is going to bring massive infrastructure, uh, challenges because of this newly introduced alpha term. Uh, because now it is a matrix in- instead of a scalar, it cannot easily be factored out. So to achieve a efficient implementation, uh, we rewrite the entire equation into the, the bottom three equations. Uh, so i- we introduce this matrix inversion operation, as well as introducing, uh, the, uh, cumulative decay factor so that we can implement this entire thing in parallel without sacri- sacrificing, uh, any efficiency. And more importantly, this is not an approximation. It's an exact, uh, mathematically equivalent formulation so that we can achieve much efficient implementation without sacrificing, uh, any loss in terms of, uh, performance. So it's going to be as efficient as, uh, you know, previous, uh, linear attention variants, but at the same time, much more expressive. So these are some of the results that we obtained, uh,

Speaker 16:53 using a fair comparison. So on the left-hand side, we see the performance on two different types of tasks. So MMANU is a short context task. So for short context tasks, uh, Kimi Linear achieved a better performance compared to MLA and GDN. And at the same time, for longer context tasks, such as Ruler, uh, Kimi Linear is, uh, also better than the variants, uh, the other variants, while being much more efficient compared to MLA. And when we scale the context length further to, for example, one million tokens or even longer, uh, it's going to be much more efficient, uh, compared to, uh, the baselines. And this is also, uh, the first architecture that can outperforms full attention across, across the board, uh, including short context tasks, uh, long input tasks, and long output tasks. So these are two, uh, key dimensions that, uh, we are

Speaker 17:54 interested in. And the third dimension, uh, is the agent swarms. So here is a diagram to showcase how we designed this agent swarm, uh, paradigm to solve some of the more complex tasks compared to single agent paradigms. So here we have an orchestrator- Or you can call it a main agent. It's responsible for orchestrating tasks. It has different options. For example, we can spawn, uh, a group of sub-agents and assign new tasks to these sub-agents. Or you can collect the results, uh, from the return of these sub-agents. And you can sort of, uh, performing s- uh, this process in a iterative way, and at the end of the day, you can acco- accomplish a more, uh, complex task compared to using one single agent. And it's analogous to, to human society. For example, if we build a, a company, uh, we need different roles, and we need, for example, uh, orchestrator or maybe we need a CEO to, to decompose and assign the tasks to different roles, and then at the end of the day, uh, the entire organization is going to have, uh, to move towards the same goal. And here, for example, in this case, we have... maybe you have the AI researchers, uh, you have the web developers, you have physics researchers, and they, they can study different topics. And at the end of the day, you just collect the results and spawn, uh, a group of fact-checkers and web developers and file downloaders to, um, to assemble the results to- into a, a, a single report. And this is another perse- uh, perspective to, to look at, uh, this new paradigm. So the x-axis is the complexity of the task, uh, and the y-axis is, is the execution time. And the complexity of the task is measured by the accuracy of, uh, a group of models, uh, on, on such task. So we can see with agent swarms, it's going to, uh, substantially, uh, inc- uh, reduce, uh, the execution time, uh, compared to, uh, uh, compared to single agents. Uh, it's going to be more effective. Uh, and this means that we can scale, uh, this agent swarm paradigm to, uh, for example, if you run this, uh, agent swarms with 100 or maybe even 1,000 sub-agents, you can ac- accomplish a complex task within a certain period of, of time that is tolerable for, um, for, to pro- producing real economical value. And we can s- we can certainly scale it in different dimensions. We can scale the inputs. For example, we can download and read hundreds of sources or even maybe thousands of sources in parallel, or we can output, uh, write a 100-page literature review, uh, in, uh, in parallel. Or we can take actions at scale, we can perform data analysis, uh,

Speaker 20:56 for 10 different tasks, and al- also it is orchestration at scale. You have to learn to design sub-tasks and aggregate the, the results. And technically, uh, we define some new objective functions to guide the learning process of our agent swarm system. So there are three reward functions, uh, uh, reward objectives that are, uh, considered here compared to, uh, the conventional single agent IO learning. So the first term is what we call the instantiation reward. It incentivizes sub-agent instantiation to prevent, uh, uh, this, uh, serial, uh, collapse, uh, phenomenon from happening. So basically, we don't want it to default to single agent execution. We want to encourage, uh, the, uh, parallel executions, especially, uh, when we s- uh, when, when it's early stage in training. Uh, and of course, we can decay the weight for, uh, this instantiation reward term over training course because, uh, when it learns to, uh, learns, uh, parallel execution, we can reduce, uh, the weight. And the second term here is, is finish reward. Uh, and it is used because we observe one of the things in training, uh, that, uh, some of these, uh, sub-tasks are just created but never finished. So it's almost like it's going to hack the first term by just spawning a bunch of sub-agents and the task might be too complex or maybe the task just doesn't make sense. And here we use this finish reward to basically encourage that each of the sub-tasks should have a relatively high ratio of, uh, completion instead of just spawning a bunch of, uh, pseudo tasks which needed to be, uh, meaningful. Uh, so this is the second term that we use. And of course, we use the same, you know, decay strategy. We use a relative high weight at the beginning of training, and we decay it to a relatively low weight at the end of training. And of course, the third term is the standard term, uh, is, is the outcome reward, uh, is going to measure whether the entire task, uh, is completed. Uh, and then we're going to add the three terms in our, our reinforcement learning, uh, system. And of course, we have to build, you know, the entire infrastructure because, uh, right now you need to support the parallel execution, and then you need to support different reward functions, uh, and, and to, you know, maximize the efficiency of the entire agent swarm IO system. So here is three, uh, different things that, that we have, uh, tried scaling. Uh, the MELLP optimizer, uh, improves token efficiency. And Kimi Delta attention, uh, in the Kimi linear architecture improves long context. And we also have the agent swarms paradigm to

Speaker 23:58 further, uh, create a new dimension of scaling. And all of this put together, uh, we created, uh, Kimi K2.5, a new model that we just released, uh, over one month ago. Here's a short video to demonstrate some of its capabilities

Speaker 24:24 [upbeat music] [clapping]

Speaker 25:19 So, yeah, there are a lot of interesting things, capabilities that we discover from the model. For example, uh, it merges the visual capabilities with coding capabilities. So a lot of new things just emerge out of it. It can read a video and then produce a website that, uh, sort of replicates or style transfer the original video. And all of these are due to successful, uh, and stable training, uh, at the pre-training stage. So this is also one of the, uh, most beautiful curves that, uh, I observe in my life. Uh, so this is the training curve of the K 2.5 base model. Uh, so as you can see, it went through, uh, over fifteen trillion tokens. And of course, in K 2.5, we additionally trained another fifteen trillion tokens. And the entire, uh, train, uh, the training process is just so stable. There's no loss spike, especially when we introduced this new Muon optimizer. We didn't observe any spike. And this smooth, stable training process produces a very stable outcome that, uh, a very strong base model that we can fine-tune on top of it, uh, to achieve, you know, new capabilities as, as we introduced and saw in the video, uh, the video. And this is also, of course, uh, trained on NVIDIA H100 GPUs, and you should know in this H100 cluster contains two, uh, TB RAM and eight GPUs. Uh,

Speaker 26:49 they are connected by NVLink. And, uh, one of the... another, you know, key innovation of Kimi K2.5 is that it is the first open model with native joint vision, uh, text capabilities. So if you look at previous open models, usually their visual capabilities are added on top of a text base. Meaning that, for example, if you train the text models for twenty trillion tokens, and then on top of it, you do another two trillion, sort of a post-training process to add additional visual capabilities on top of it. Uh, but for K 2.5, it's different in the sense that we fuse the training process of vision and text from day one, so it's called early fusion here. We start from, you know, 0% of the progress. So from day one, we're going to merge, uh, the vision and text tokens. And as shown in our pr-preliminary experiments, it outperforms, uh, late fusion. And some of the new capabilities that we observe also come from this training recipe. Uh, for example, if you wanna do vision to count, you really have to merge vision and text into a single brand to achieve that. If you separate these two brands, it's not going to happen. You have to align these two modalities into a shared embedding space, uh, entire, uh, a shared representation space, s-so as to, uh, achieve this. And another interesting thing that we observe is that these two modalities can actually enhance each other. So that's, uh, that's been, uh, long been a challenge, that if you add vision capabilities into a text model, it's going to somewhat, uh, uh, hurt the text performance. But here, we found that if you train it properly, these two modalities can actually enhance each other. So this is one of the key findings that, uh, we observe in, in, in our training. Uh, so first vision improves text. So this is so interesting. So before vision RL, the performance, uh, in the first column, and then we have the performance after vision RL. So here, vision RL refers to a process that we only use vision tasks. So there is no text tasks involved here. We only have vision tasks. For example, we teach the model how to, how to count, uh, how to answer some of these visual QA, uh, problems, uh, without any, for example, math, any coding problems in, in, in this space. But we observe that it's going to improve the performance for even, you know, reasonably heavy text task. And on the other hand, text also improves vision. If you have a very strong text base, you, you actually don't need any vision SFT data in the training process. And this is the approach that we adopt. So it's called zero-vision SFT. Uh, basically, we don't have... We, we have basically zero vision SFT data, and the only SFT data that we have is the text SFT data, and then we do a joint RL over text and vision. And you can see that we can achieve, uh, almost state-of-the-art performance across the board on, on vision tasks without any vision data. So it, it's clear that, uh, if you have a strong text base, it's also going to improve, uh, the vision if, if you, uh, align these two modalities into a shared space in your, in your pre-training. And also, uh, these are some of the examples of, uh... Yeah.

Speaker 30:26 As, as I show in the video, so it's, it demonstrates strong capabilities of, uh, uh, visual design and front-end coding, and this also emerges from our, uh, vision text through training. So, uh, after all this, so this, uh, all about Kimi K2.5, and as probably, uh, you probably know, we released our new architecture, uh, yesterday, uh, in our tech report. It's called Attention Residue. So here I'm also gonna briefly talk about our new work, uh, which serves as a, a sneak peek into our next generation architecture that we're probably going to adopt in, in our later models. Uh, so here the motivation is, is quite simple. Can we apply some of our techniques that we use in, in the, in the temporal dimension and we, uh, we just take some of the inspirations and then apply it to the depth, uh, dimension. So, uh, and it starts from this residual connection. So I still remember, uh, listening to, to, uh, Cami's talk, uh, at a tutorial in, in ICML 2016, 10 years ago. Uh, so it was a brilliant idea. So basically before ResNet, uh, nobody was able to train deep networks. If you increase the depth, if you increase the number of layers for neural networks, nobody was able to train it because you observe this gradient exposure and gradient vanishing, all this, you know, stability issues. But then after the introduction of, of ResNet, we can train, you know, an arbitrarily large number of layers. You can stack as many layers as, as you want, and you, you, you don't have to worry about the training stability issue, uh, and stuff. And as discussed in Ilya's talk, uh, two years ago, it basically says that residual connection is a variance of LSTM, but just rotated 90 degrees. So how do you understand this? If you look at the LSTM, it's, it's a variance of recurrent net, right? And it's a recurrent, uh, model process. So we're going to take the hidden states from the last step, and then we're going to have some gating mechanism, some function to produce the current state, right? And if you look at the, uh, the depth dimension, residual connection is basically the same. We're going to take the outputs from the last layer, and then we're going to apply some sort of function on top of it to produce, uh, the current, uh, uh, the current outputs of, of the current layer. It's just the formulation is different. For example, for residual connection, we're going to use a fixed addition. We're going to, uh, have this, uh, we're going to add, uh, the previous hidden states, uh, with the current outputs. It's just the formulation that's different, but the basic idea is the same. It's a recurrent net applied, uh, in the dimension of depth. And... But on the other hand, we can think about, uh, reformulating, uh, this, uh, this function. Instead of having an LSTM, can we have an attention in the dimension of depth? And it's going to create new possibilities because attention have, has been demonstrated to be so successful, uh, in the transformer era. So what we're gonna do is not just to take the last hidden state, but we're going to consider all the previous hidden states and use the attention operation, the attention mechanism, to assemble and aggregate all of these previous hidden states to compute the current state. So this is ex- exactly attention rotated by 90 degrees. Uh, it's sort of we view it as a natural generalization of residual connections, uh, in the LSTM an- a- analogy. Okay, and here is the detailed formulation. So on, on the left-hand side is a standard residual. As I said, it's basically LSTM rotated by 90 degrees. And the second figure is attention rotated by 90 degrees. So what we do is to collect all the previous hidden states and have a simple attention operation on top of it to produce the current layer's outcome. Uh, and of course, to increase the efficiency, to reduce the infrastructure, for example, communication and memory overhead, we also design a new variant called, uh, block attention residue on the right-hand side. So basically the idea is, is also simple. We're going to divide, uh, all the layers in the neural networks into multiple blocks. For example, each block can contain, say, 16 layers, or it can contain maybe four layers. And then for each block, we're going to, uh, apply, uh, this attention residue only on the outputs of each block. But within each block, we also, we still adopt, uh, the standard residue. So this is going to reduce a lot of overhead while having minimal, uh,

Speaker 35:29 loss in terms of training accuracy. And these are some of the, uh, impressive results that we achieved, uh, uh, on this new architecture. Uh, so on the scaling law, we can, uh, improve the token efficiency by, uh, 24%, uh, meaning that if you have 50 trillion high-quality tokens, now you just magically have, uh, uh, over 60 trillion, uh, tokens. And then for the validation loss, uh, you, you can also observe that, uh, it's, uh, consistently lower than, uh, the original curve, uh, demonstrating, uh, the stability across optimization, and also achieves the best improvement, uh, on some of this coding, math, and reasoning-heavy tasks, as shown in the benchmark results of GPQA, uh, math, and HumanEval. So the entire community, uh, keeps moving forward, uh, and we're happy that we can... we're able to contribute to, to the community with, uh, you know, new technologies. And some of these are have, you know, some of these technologies have been sort of standard and de facto for a long time. But as you can see, we still see a lot of opportunities to improve it, to, uh, to have re- revolutionary new design to achieve better performance. If we multiply all these gains together, you can actually have a much better model. So Adam was invented in 2014, uh, and now we scale an open source Meowclip, a, a drop-in replacement for, uh, for Adam. And I'm sure that, uh, if you're training a, uh, transformer LLM, uh, it's going to be much better if you use Meowclip instead of Adam. And attention was invented, uh, over eight years ago, and then now we have Kimi Linear, which is a linear version. We don't have to use full attention across all layers. Uh, we can have linear attention that performs better on short, long context at the same time. And also, residual connections are now, uh, also challenged. Uh, we scaled an open source attention residue. So I think one of the, uh, interesting things about our era is that we sort of adopt a different mindset for, for doing research. So if we go back to ten years ago, it's mostly about publishing a new idea. Uh, but then I think the lack of the rigor of the experiments, it's very hard to produce, uh, solid experimental results. But now we have this scaling ladder. We have, uh, enough resources to, you know, train the model and run, you know, different-- at different scales. We can have, you know, a whole set of benchmarks to measure the progress. So it is-- it becomes easier to make confident and solid conclusion out of it. And this is one of the reason why we are observing, you know, ne- new progress, uh, on these ancient techniques. And I'm sure that we'll see more and more, especially in the open source community. I think we're going to have more and more even better architectural and, you know, optimization, uh, improvement in, in the next few years. All right, so to summarize, we're going to, uh, keep scaling our models. And so these are three dimensions. Uh, for example, we, we see, uh, we, we see, uh, different architectures and optimizers that optimize all three, you know, dimensions. And we'll keep, you know, seeing new dimensions for scaling. Uh, agent's ROMs is, is not the end. And we are glad that we can, uh, move forward with the entire open source community to achieve better and better, uh, you know, intelligence. Thank you so much. [audience applauding]