By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 2:15
Scaling Laws and the Episode Setup
Anjney Midha introduces scaling laws, the 2022 Chinchilla paper, and Arthur Mensch's role in it. He explains how Mensch co-founded Mistral with two former Meta researchers and previews the topics of the conversation.
- •The 2022 paper showed data sets mattered more than sheer model size
- •Mensch co-authored it while at DeepMind
- •Mistral was founded with Guillaume Lample and Timothée Lacroix, who worked on Llama
- •Mistral 7B and the new Mixtral mixture of experts model are introduced
- •Preview of open versus closed models and compute, data and algorithmic innovation
- 2:15 – 4:40
The Chinchilla Paper and Compute-Optimal Scaling
Mensch explains how earlier scaling-law work led to a misconception that model size should grow much faster than data. His team found that compute, model size and data should scale together, which led to Chinchilla.
- •Earlier work advised scaling model size much faster than data
- •Papers in 2021, including work at DeepMind on Gopher, made this mistake
- •The corrected rule is that 4x compute means roughly 2x model size and 2x data
- •This scaling stays consistent as everything grows, and multiple runs confirmed it
- 4:40 – 8:26
Founding Mistral and Llama's Over-Training Insight
Mensch describes how the three founders knew each other from school and parallel careers at DeepMind and Meta. He explains that ChatGPT signaled an opportunity for a more open approach, and how Llama showed that over-training smaller models cuts inference cost.
- •The founders met through school and earlier collaborations in Paris
- •Mensch's DeepMind work included RETRO, Chinchilla and Flamingo
- •ChatGPT signaled an opportunity for a small team distributing the technology openly
- •Llama went beyond Chinchilla by training on more tokens for cheaper inference
- 8:26 – 13:19
Mixtral and Sparse Mixture of Experts
Mensch explains how Mixtral's sparse mixture of experts works and how it differs from dense models. He covers the cost and latency benefits and the training and deployment challenges.
- •Dense layers are duplicated into experts with a router sending each token to two of eight
- •46 billion total parameters but only about 12 billion executed per token
- •Mixtral is on par with Llama 2 70B while about six times cheaper or faster
- •Challenges include mathematically correct training, hardware efficiency and communication between experts
- •An open source package based on vLLM was released for inference
- 13:19 – 17:56
Why Mistral Chose Open Source
Mensch explains that Mistral's open approach is both ideological and pragmatic. He recounts how information sharing drove AI progress until companies became opaque after GPT-3, and describes the open-weights, modular philosophy and the open core business model.
- •Progress from 2012 to 2022 was driven by the free flow of information
- •Communication largely stopped around 2020 to 2022, which the founders regretted
- •Developers want deep access to modify models
- •Open weights allow interpretability and adaptation to proprietary data
- •Pre-trained models should be neutral, with customers setting their own editorial approach
- 17:56 – 21:26
Open vs. Closed Models and Community Innovation
Mensch gives his view of the performance gap between open and closed models, saying it is around six months and closing. He describes how the community has built on Mistral 7B and why he expects open models to converge with proprietary ones.
- •Mixtral performs similarly to GPT-3.5, and internal models sit between 3.5 and 4
- •The gap is about six months, and open source moves faster
- •Mensch compares open source in AI to Linux on the internet
- •Community work includes 128K context extension, image encoders and DPO from Hugging Face
- •Mistral 7B has run on laptops, an iPhone and a stuffed parrot
- 21:26 – 23:49
Is Open Source Less Safe?
Asked about the common view that open models are less safe, Mensch argues that current models are a compression of the internet. He says scrutiny of open models helps find misuse and build countermeasures, as with cybersecurity.
- •Current models are compared to the printing press as a way of making knowledge flow
- •Misuse risks exist for both open and closed models
- •Closed APIs can be exploited, including through fine-tuning access
- •Open scrutiny is likened to how breaches are found in operating systems and networks
- •Midha adds that many red teamers raise the odds of finding biases and risks
- 23:49 – 31:47
Advice for Policymakers: Regulate Applications, Not Math
Mensch describes what Mistral tells regulators, including that independent safety checkers need open models. He argues that models are like programming languages and that regulation should target applications, not the technology or training FLOPS.
- •Empirical monitoring is best done through open source
- •Independent product-safety evaluation needs strong open models they can own
- •Regulation burden and the need to stay at the frontier are risks to open models
- •A model is like a programming language, while the application is the product to regulate
- •Pre-market FLOPS thresholds are not a good measure, and strong evaluations are favored
- 31:47 – 35:08
Next Frontiers: Data Efficiency, Reasoning and Adaptive Compute
Mensch names reasoning and training inefficiency as the weak points of current models. He discusses data quality, adaptive compute, and how to judge whether models truly reason.
- •Training is inefficient compared to the brain, by a factor he puts around 100,000
- •The frontier includes data efficiency, reasoning and adaptive compute
- •Better data filtering and new techniques are still to be invented
- •Whether models reason or retrieve is hard to tell, but what matters is whether it works
- •Evaluating out of distribution, for example in mathematics, helps, and mathematicians like Terence Tao are using LLMs
- 35:08 – 37:25
The Future of Applications: Specialized Models, Games and Enterprise
Mensch predicts that in five years developers will use specialized small models inside complex systems, tuned for low cost and latency. He expects more interactive interfaces, with examples in games and enterprise personas.
- •Developers will want the smallest, lowest-latency model suited to each task
- •Different LLMs will serve different parts of complex applications
- •The internet in five years will be much more interactive
- •Games could use swarms of small interacting models
- •Enterprise applications could use different personas on the same model with different prompts and fine-tuning
- 37:25 – 39:02
Call to Action for Builders
Mensch urges application makers to build with Mistral models. He says the stack is clear and efficient and can run on a few GPUs or even a MacBook Pro, then Midha closes the episode.
- •Builders should take Mistral models and build applications
- •The stack is becoming clear and efficient
- •A couple of GPUs, or even a MacBook Pro, are enough to start
- •Software development today differs greatly from a year ago
- •Mistral aims to enable application makers to build as fast as possible
