By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Scaling open models via token efficiency, long context, and agent swarms
- The keynote frames progress in open models as scaling along three axes: token efficiency, context length, and multi-agent (swarm) execution.
- For token efficiency, the speaker describes scaling the Muon optimizer for large LLM training and solving 1T-parameter instability via a per-head query/key clipping method called QK clip.
- For long context, the speaker introduces Kimi Linear with Kimi Delta Attention and a mixed linear/full attention stack to improve efficiency and performance across short- and long-context tasks.
- For complex task capacity, the speaker proposes an orchestrated agent-swarm training paradigm with new reward terms to encourage parallelization and completion of subtasks.
- The talk culminates in Kimi K2.5, described as a stably trained open model with early-fused joint vision-text training, followed by a preview of a next-generation “Attention Residue” architecture that applies attention across depth (layers).
IDEAS WORTH REMEMBERING
5 ideasOpen models matter most when they’re both accessible and frontier-grade.
The speaker argues that open models let people deploy anywhere (local servers or cloud) and inspect full model weights rather than relying on a black box, but emphasizes they must also be highly competitive to matter.
Token efficiency is treated as a path to higher capability, not just lower cost.
They frame progress as scaling not only the number of training tokens, but also shifting the scaling curve via better architectures/optimizers so the same amount of data yields lower loss.
A second-order optimizer (Muon) is presented as a major lever for scaling open LLMs.
They describe investing in the Muon optimizer, claiming about a 2× token-efficiency improvement over AdamW when implemented with key techniques like weight decay, RMS-consistent updates, and a distributed state-partitioned implementation.
Training stability at trillion-parameter scale required a targeted attention-logit control mechanism.
When scaling Muon to a 1T-parameter model, they report logit explosion and divergence, and introduce “QK clip,” which clips query/key projections per attention head to constrain max logits while keeping the loss curve essentially unchanged.
Long-context capability is positioned as essential for long-running agents and complex tasks.
They motivate long-context scaling by contrasting transformers vs LSTMs, highlighting that transformer loss keeps improving across the context while LSTMs saturate, and propose Kimi Linear with Kimi Delta Attention plus a 1:3 mix of linear and full attention layers.
WORDS WORTH SAVING
5 quotesSo one thing that I would like to emphasize is that token e- eff- efficiency is not just about efficiency. It's actually also about improving the upper bound of intelligence.
— Unattributed
It's not just about infrastructure efficiency, it's about, you know, better, uh, intelligence.
— Unattributed
And the solution to this is to introduce this new technique called, uh, QK clip.
— Unattributed
And this is, uh, the first example of a large-scale Muon training in the history of machine learning.
— Unattributed
So this is also one of the, uh, most beautiful curves that, uh, I observe in my life.
— Unattributed
High quality AI-generated summary created from speaker-labeled transcript.
