No Priors Ep. 40 | With Arthur Mensch, CEO Mistral AI

(electronic music) Open source AI models have completely changed the landscape of technology over the past year. One tiny team of ex-DeepMind and Meta researchers in France has made a huge splash recently, Mistral. This week, Elad and I are joined by Arthur Mensch, the CEO and co-founder of Mistral, who recently released Mistral 7B, an Apache 2.0 licensed open source model that has changed people's mental models about what can be done with small models. Arthur, welcome to No Priors.

Thank you for inviting me. I'm very glad to be here.

Okay. So, just six months ago, when we met, you were leaving DeepMind to start Mistral. It takes real guts to look at the scale of dollars and compute that OpenAI, and Google, and others have amassed and say like, "We want to play in this game too, and it's important we do." Uh, tell us about the inspiration to start.

Me, Guillaume, and Timothée were, uh, I guess, pretty early in the field and, uh, it's- it had been 10 years that we had been doing machine learning. And we do- we did know where to start from and how to make a good model with a limited amount of compute and money. Well, not so limited, but at least more limited than, uh, where we were coming from. Uh, and so I- I think that's why it get us started. The various companies we were in move into directions that we hadn't anticipated when we joined the company. And, uh, we decided that there would be a very good opportunity for creating something that would be a standalone company in Europe, uh, focusing on- on making AI, uh, better, focusing on making frontier AI, and focusing on making, uh, open source AI as a core value.

Maybe we can talk about each of those pieces. So, 10 years in, uh, machine learning before, you were a co-author on the Chinchilla's scaling laws paper, you worked on, um, uh, the sort of mixture of experts ideas early. Can you talk a little bit about what your research directions were at DeepMind?

Yeah. So, I come from an optimization background, so my focus has always been, uh, for the last 10 years to make algorithms more efficient and to, uh, use better the data that we have to make, uh, models with good prediction performances. Uh, and so when I arrived at DeepMind, uh, I joined the LLM team that was 10 people at the time. And very quickly, I started to work on retrieval augmented models. Uh, so with a paper called RETRO, uh, that I co-led with my friend, Seb, uh, Bordeau, who is still at DeepMind. The point there was to use like very large databases during pre-training, uh, so that we didn't force knowledge into the model itself and we would tell the model that it would have access to an external memory anyway. And so it was working quite well. We could actually lower the perplexity, let's say. That's what you work on when you make LLMs. Uh, there were some limitations that we- uh, that- that I think the community has started to address quite, quite well. And that was at the time when retrieval methods weren't really, uh, mainstream. Uh, now they've become completely mainstream. So that's the first project I did. I worked on, um, on sparse mixture of experts, uh, also quite quickly, uh, because that was related to my topic of post-doc, which was optimal transport. So optimal transport is a setting where you have, uh, I guess, tokens, you need to assess them- assign them to devices and you need to make it sure- to make sure that there's some good, uh, assignment in between the two of them so that the devices don't see too many tokens. And, um, as it turns out, uh, the way you do it is with optimal trans- optimal transport is a mathematical framework to do it correctly. And so we- I started to work on introducing this to sparse mixture of experts. And very quickly, we started to move onto, uh, scaling laws. So, how do you actually take the method that is working at a certain scale and try to predict how that will evolve with the scale, the number of experts, the- the amount of data you see? Uh, and so that's, uh, a j- a work I- I've- I've done with many colleagues as well, uh, on how do you adapt the scaling laws for dense parameters, uh, for dense models, uh, to a setting where you wanna predict the performance not only with relation to the size of the model but also the number of experts, 'cause that was the s- similar thing I worked on. And then connectly worked on Chinchilla, which is, uh, I think a major paper in the history of LLM, uh, also with Seb, uh, Jardin, uh, Laurent, many other people. Basically, the story was that everybody was training models on too few tokens, uh, because of the paper from 2020 that happened to be not very well, uh, executed. And so what, uh, we observed is that, uh, you could actually correct that. And so instead of training very large models on very few tokens, you should actually grow the number of tokens as you grow the- the size of the model, which if you think about it, makes a lot of sense because you don't want to have infinite size model looking at, uh, a finite size- a- a finite number of tokens. And similarly, you don't want to have a finite size model looking at an infinite number of tokens.