a16zHow Open Source Became AI's Backbone | Inferact with a16z
EVERY SPOKEN WORD
45 min read · 8,864 words- 0:00 – 1:00
Intro
- MBMatt Bornstein
The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source world?
- SMSimon Mo
If moderation is never solved, in the future people will go to open way by default because that is where you know for sure you can control your guardrail for trusted use cases.
- EBElena Burger
Can you talk about where vLLM sits in that stack?
- SMSimon Mo
vLLM is a inference engine. It is kind of like databases and operating system and other critical software to power AGI, NVIDIA, AMD, Google. Their newest chip will make sure vLLM can run on them, and in a lot of cases they use vLLM as a benchmark. We're bridging almost a 10 X gap. For proprietary models, there is a regular mode and fast mode, but for open weight, every provider can offer potentially even 10 different levels of speed.
- EBElena Burger
Five years from now, open source AI models, have they closed the gap with frontier models?
- SMSimon Mo
Capability-wise, I don't really see a big gap, not even today, because-
- 1:00 – 5:10
What Is vLLM & Why Serving LLMs Is a Fundamentally Different Problem
- EBElena Burger
Today we're here with Simon Mo, co-founder of Inferact and a lead maintainer of vLLM, the open-source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Bornstein, an a16z general partner. Simon, Matt, thank you so much for joining us. I think first we should start, uh, with open-source AI and kind of the more recent history of open-source AI. So vLLM actually has its origins kind of back in 2022, pre-ChatGPT, and your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems. So can you talk about what made serving an LLM so fundamentally different from the ML workloads everyone already knew how to run?
- SMSimon Mo
Yep. Good to be here. So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs, and it, it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LLM's response quickly and efficiently. So this typically means we need to handle, uh, differences in input distribution, how long each request is, output distribution, which is non-deterministic, and batching and scheduling a lot more in the, at the core of the inference engine.
- EBElena Burger
Yeah. Um, and I, the, so, so the team, the project has been around for about four years now, um, but the company is a little bit more recent, so I want to throw this out to either of you. Matt, I know you've known the team and obs- uh, observed the team for a very long time, so at what point did you see this sort of transitioning from being, you know, a, a much beloved open source project to critical infrastructure and then a company?
- MBMatt Bornstein
Yeah. I mean, I, I think you have to go back a, a little bit, right? O- open source, um, wa- was the norm for AI models early on, right? I mean, we literally had this company called OpenAI, which-
- EBElena Burger
Mm-hmm
- MBMatt Bornstein
... you know, it's become a little bit of a joke. It's not as open as it once was, or not nearly as open as it once was, but, uh, but, but early on, all, all the frontier AI work was being open source or, or at least released into open weights, which is a little bit different than true open source. Um, and, and people could mostly run these models sort of on hardware or computers that they already had.
- EBElena Burger
Mm-hmm.
- MBMatt Bornstein
Um, I'm, I'm curious, Simon, like maybe you remember, like what was like the first model that like you actually needed to go out and get special software and like a, you know, a special set of computers to be able to run?
- SMSimon Mo
Probably BERT.
- MBMatt Bornstein
Yeah.
- SMSimon Mo
And before that it was like ResNet for computation, like, uh, like images, com- computer vision classification. So ResNet already need to run on, uh, NVIDIA K80, which is kind of one of the first GPU-
- MBMatt Bornstein
Oh man, that's a blast from the past [chuckles]
- SMSimon Mo
... on, uh, AWS and other places and, and but way over... But even at this point, ResNet you can still kind of run on a, uh, commodity, even CPU devices, it's just very slow. But for BERT, where running at it is like, wow, you have to run it on GPU to make it anything faster and efficient for anything translation or any task. So that was like before 2020 even. Yeah.
- MBMatt Bornstein
It's so funny thinking about this. I, you're, I'm like all the memories are flooding back.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Like Hugging Face had like 1,000 BERT variants on it.
- SMSimon Mo
Yeah, exactly, yeah.
- MBMatt Bornstein
And you'd have to go find like the right BERT variant for your particular task.
- SMSimon Mo
Yeah. Yeah, yeah.
- MBMatt Bornstein
And yeah, and you're right, you had to... Some people, I guess, had their own GPUs and could run it, but, but yeah, a lot of people had to go, you know, provision cloud-
- SMSimon Mo
Exactly
- MBMatt Bornstein
... GPUs and stuff like that. That's really funny. And so yeah, so, so look, I mean, um, uh, you know, BERT was an early-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... language model, um, that, uh, you know, newer models are much bigger, much more sophisticated, take up a lot more memory, a lot more compute and, and, and, um, you know, so vLLM really from the early days, right, was about running these, you know, more powerful models-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... that, um, you know, that, that, that you couldn't just sort of do it, you know, figure it out on
- 5:10 – 8:26
When Open Source Became Critical Infrastructure
- MBMatt Bornstein
your own.
- EBElena Burger
Yeah. Yeah. Um, and uh, I, I think like to get us to, to this present day, you know, um, I, I, I think it would be good to, to talk about kind of at what point it really became critical infrastructure for these, you know, even larger open source models, and when did we even start to see these larger open source models kind of come into the field?
- SMSimon Mo
Well, we really see the criticality of, um, the software stack is, um, it's always about bringing open frontier intelligence. This is, open frontier intelligence is the catch, catchphrase for the K3 model released this time, and it's also about vLLM's mission about inferencing infrastructure. So if we look at it, when do they start being critical is when people are relying on it for their application, for their day-to-day productivity. So- Maybe 2023 when GitHub Copilot and ChatGPT are where, uh, is something that people just cannot live out, uh, with anymore. Uh, so at that point, open weight model is already backing and become a cornerstone of the how people are living every day.
- MBMatt Bornstein
Yeah.
- SMSimon Mo
And at that point, we will need the accelerator running open weight model and open weight software to, uh, open source software to make sure it works well. Yeah, and have all flexibility and control over it.
- MBMatt Bornstein
Yeah, it's sort of, it's sort of interesting from a startup standpoint. Um, uh, you know, like, like you mentioned sort of GPT-3 or like early ChatGPT, um, those closed source solutions were starting to become critical to like a small group of people-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... around that time.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
And open source existed, but it was a little bit of a curiosity or sort of an enthusiast thing. Um, a- as the frontier has expanded, as, you know, particularly with closed source models, more and more open source has been dragged in as kind of like critical behind it, like if that makes sense. Like at, like at any given point in time, including now, I think models from OpenAI and Anthropic are, are kind of more widely used and more critical kind of in general than, than open source models. But I do think we passed a threshold in like, I wanna say about a year ago-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... where a bunch of smaller companies or like new application companies, as they were trying to figure out, "How do I really build an AI without just being a wrapper on top of OpenAI?" The answer to that question turned out to be open source. I mean, this is what Cursor did, this is what sort of Decagon and Harvey are, are in the process of doing now. And a, and a bunch of other like really, really strong application level startups sort of made the determination, "We can't build just on closed source. We need to do our own mid-training, our own post-training, our own sort of inference and deployment tricks."
- SMSimon Mo
Yeah.
- MBMatt Bornstein
And all of that means it must be built on top of open source. Like, you know, the, the closed source vendors won't, won't give you the access to do this. So my read is like kind of a year-ish ago-
- SMSimon Mo
Mm-hmm
- MBMatt Bornstein
... open source became really central in a way that's not always visible because it's, it's deeply embedded in some of these products, but, you know, some of the most innovative products and applications now, um, you know, re- really d-depend on this very deeply.
- EBElena Burger
Yeah. Yeah, and can you talk about where vLLM sits in that stack, where, where we do have these larger enterprise companies that are choosing to use open source models? Like where, where does vLLM sit
- 8:26 – 13:35
Where vLLM Sits in the Stack
- EBElena Burger
in the stack for them?
- MBMatt Bornstein
Yeah, I mean, you should like-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... just about everybody uses vLLM.
- SMSimon Mo
Yeah, just-
- MBMatt Bornstein
You should describe it, yeah
- SMSimon Mo
... just about everybody uses vLLM. vLLM is a inference engine. That means its job is to turn available GPUs into a running endpoint for intelligence. So that means it is kind of like databases and operating system and other critical software to power, uh, this, uh, economy or power the AGI that everybody really uses today to ensure they can have, uh, cost effectiveness, efficiency, reliability, and also always staying on the frontier. Because for vLLM, we support more than 1,000 model architecture up to today, and a lot of those are proprietary, but also a lot of those are open weight, right? And a lot of those model architecture, when they're becoming, transitioning from a research prototype to world accessible open weight model architecture, they are live on vLLM on immediately. So that's what a process we call day zero model release. And additionally, vLLM also work closely with all the hardware vendors, so that means across like Nvidia, AMD, Google, and Amazon, Intel, and a lot more, their newest chip will make sure vLLM can run on them, and in a lot of cases, they use vLLM as a benchmark to make sure it runs well on them. So this kind of fusion of where models run and where hardware, uh, where it get meets-- gets to meet the hardware is where the magic happen, and this is where vLLM is.
- MBMatt Bornstein
And, and you've told me some of the behind the scenes stories, like it's actually not easy these days [chuckles] around model releases.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
It's like a lot of human drama-
- SMSimon Mo
Oh, yeah, yeah, yeah
- MBMatt Bornstein
... in addition to like technical work.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
I, I guess, are there any stories there that you think are okay to share?
- SMSimon Mo
Oh, it's actually a very fun co-design process because from model labs point of view, right, these are brilliant researchers who have built this model. Now their biggest question becomes, "How do we get this out of the world and make sure everybody's able to use it and run it well?" And we have worked with model labs that are, um, very... n- just because they just use vLLM already in production or in re-their research process, they will just done everything for you. 'Cause this is a moment when we go to them, it's like, "Hi, we're the vLLM team. We would like to support your open source model. We would like to offer, in a way, this kind of open source, but white glove service to get your model running well on our architecture." And then you return, we get the model labs like, "Oh, we'll get it working already because we're running it for the RO process. Here you go. Just review our code and merge our pull request." And then on the other end, we really have model labs that just don't know how this can work, so these-
- MBMatt Bornstein
Because systems is not like their core.
- SMSimon Mo
Yeah, 'cause systems is not their core, and they have been training, or maybe they have their internal inference engines that just don't know how it will adapt to the open weight ecosystem. And by the way, this is also a very much a multi-party kind of involvement process. Every model release typically involves, um, the, the model lab, involves, uh, primary or secondary hardware vendors, involves us, involves Hugging Face, who are the model format and like model, uh, hub vendors. And then depending on the appetite of the model lab involves 10 or 20 different kinds of release partners. These could be inference clouds, these could be public hyperscalers, whoever is gonna run this model and you want to them to ensure success-- the model are running successfully, right? So, uh, even up to today, if you look at the K3 model release, it's a whole partnership and a drive to make sure that the model is... Once the model is released, because it's just- A few terabytes of files sitting on the internet that people are actually going to be use it really, really well. E- even from the beginning of 2023, 2024, if you remember when Mistral dropped their first model, they just drop a torrent link for PTP, and then everybody's like struggling and trying to get it up and running. And then we're working behind the scene with the Mistral team trying to get the inference engine support working, uh, in vLLM. This is one of the most ex- uh, probably early on exciting weekend that we ge- are able to spend on this. And then, and, uh, the, after the weekend when everybody's trying to really analyze what's going on, and, uh, Monday, Tuesday, well, Mistral and us just announced, "Here, you can run it on vLLM successfully here," and everybody will able to easily reuse a lot of the work and bu- start building on top of it.
- MBMatt Bornstein
That was sort of a fun time where like enthusiasts like me could just-
- SMSimon Mo
[chuckles]
- MBMatt Bornstein
... like scramble to like download the model and like-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... get it running somewhere.
- SMSimon Mo
Yeah, yeah.
- MBMatt Bornstein
I'm glad the professionals-
- SMSimon Mo
[laughs]
- MBMatt Bornstein
... have taken over because it never worked like very well.
- SMSimon Mo
Yeah, it's very fun.
- MBMatt Bornstein
But it was kind, it was like a fun moment in time.
- SMSimon Mo
Yeah.
- EBElena Burger
So, so to bring things, you know, forward to the present, um, I think open source models and also, you know, distillation have, have been in the news recently. Um, Inferact signed the NVIDIA Open Weights and American AI leadership letter, uh, that, that was signed by also a16z, Meta, Amazon, dozens of other companies. Can you just talk about, um, you know, your decision to sign that and, and sort of what, what you were really kind of responding to, uh, in the market and, and kind of in
- 13:35 – 16:42
The Open Weights Letter & Why Open AI Development Must Be Protected
- EBElena Burger
the news?
- SMSimon Mo
Yeah. So for us, what really want to stand behind is open weight absolutely matters in the ecosystem. The world cannot just be controlled by proprietary APIs and where open weight, open development and research of these models are blocked or banned, right? Uh, the pledge that Inferact really signed up for is we want to help and foster this ecosystem, where we are typically in a little bit downstream of this ecosystem, right? Inference engine are not part of the pre-training process nor the RL process, but we are where the model actually meets the world. And from what we're seeing, people are just really using their imagination and ability to materialize this imagination of open-weight model. They're able to leverage these open-weight models so much effectively.
- MBMatt Bornstein
There's almost two pieces to this, right? There's like the, the cost thing-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... where it's like the closed models are too expensive, and then there's sort of the control thing where I, I, I want to sort of be in control of my infrastructure and, and, and control the model, right? If I need to extend it or, or put on my own guardrails or anything. I, I'm just curious, have you heard from, from customers, like are, are both those things important to them? Or, or like are they kind of willing to pay as long as they have the control? Or, or maybe there are different use cases.
- SMSimon Mo
I think like it fluctuate over time, so control matters a lot over the last few years, and then cost just started to matter over the last few months.
- MBMatt Bornstein
Mm.
- SMSimon Mo
So cost really matters starting from people trying to migrate off, uh, right, uh, their expensive coding plan and like, uh, every skyrocketing token maxing spend. But control has always been on, in the backbone of this, is they want to even, in a way, to control the cost, right? But also it's about controlling the system performance against what they're paying for. So, uh, there are, for example, for a voice agent company, they want to control their own model so that it can make sure the model actually respond by the required time. So the customer-
- MBMatt Bornstein
Mm
- SMSimon Mo
... when they're on the phone, they can ensure the agent is responding, uh, according to a SLA. And this sometimes is only you can do with your controlled intelligence, um, because you know the whole hardware you're running and the whole system you're monitoring, versus signing up for relying on your critical infrastructure with a proprietary API where they might go down any time or have violation of the contract any time.
- EBElena Burger
Yeah. Simon, you also, um, to, to go back to the cost point, you actually make the point, um, in an essay you recently wrote about the release of Kimi K3, that actually the economics is besides the point, and it's actually, you know, in, in the case of these, you know, just really, really great open-weight models that are on the frontier that were designed by really brilliant researchers, like, like these models are in some cases just as expensive as, you know, the closed source models. So in those cases, kind of what is the point of running them and kind of what do we learn architecturally in the, in the course of running them?
- 16:42 – 19:57
K3 Economics: Bridging the Gap Between Open & Proprietary
- SMSimon Mo
Yeah. So first on cost, it's not necessarily they are as expensive as a proprietary model, but rather first the cost, cost discourse has been discussed over and over again with even Geo 5.2 a few months back. So open-weight model are sometimes definitely a lot cheaper and, but for this model, there's a big sort of s- step change where we're bridging almost a 10X gap, but strike somewhere in the middle, where Kimi K3 is not as expensive as Claude or GPT Sol, but it is a lot more expensive than Geo 5.2. Why is that? And I do believe this is the point of we're pricing intelligence with the market correctly and understanding where it is and... But then the majority part of the discourse should be focusing on, wow, this model is bringing a Opus 4.8 level model to our own infrastructure that I can use, I can run, I can fine-tune, I can be able to- Understand exactly how many tokens do I need, understand the exact performance profile. The reason here, for example, is for proprietary model, there is a regular mode and fast mode, and that's only the two switch here. But for open weight, when you're running it, every provider can offer potentially even 10 different levels of speed, going from like the slowest mode, which can be a lot cheaper, to, um, 400 tokens per second, uh, almost, uh, up to 500 in many cases that for some workloads. And this is, uh, typically two X or three X faster than the fast mode out there today. So this kind of level control even in terms of performance, and then let alone control over how customer interacting with the model, control over data retention. Keep in mind, Fable doesn't have da- zero da- data retention policy, and at least the, a lot of the data need to be staying there, and let it control security and compliance a lot more. Um, yeah, this is why I'm particularly excited about K3, not just from the cost perspective, but a lot more on bringing this level intelligence to something people can own.
- EBElena Burger
In terms of, you know, calibrating things like speed, calibrating other things, just sort of on the back end, what needs to happen and, and kind of what are you seeing your users do, and like, like who, who is being really clever about this?
- SMSimon Mo
So we do see users are able to get the maximum benefit out of this model when they en-enable fast mode.
- EBElena Burger
Mm-hmm.
- SMSimon Mo
Like what I'm talking about here, of course, is vLLM's own fast mode, getting up to 400, 500 tokens per second, because it is really a big step change from like the-- especially when developer interacting with, uh, the model. They can see, "Oh, I can really just get my task done faster here." And the model are now no longer stuck in syncing, rather it is just executing, executing and interacting with the environment. So for premium developer blocking focus task, we're seeing it's very benefiting. But also K3 are just be able to, uh, have the ability for people to modify it and f-fine-tune on top of it, allow them to make it better for their own workload, and this is definitely happening today as well.
- MBMatt Bornstein
Can,
- 19:57 – 28:51
Licensing Evolution: From Apache 2 to Commercial Terms
- MBMatt Bornstein
can you just explain what the, um, licensing term is for the most recent open source models-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... compared to the past-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... and, and, like why you think they're doing that?
- SMSimon Mo
Oh, yeah. So historically the, um, the open weight model are just like Apache 2, like our software, which is like take it, modify it, do it whatever you want with it. Here is a gift to the world. And then recently the model lab are trying to understand a way to economically fund their own model development. After all, model training and research are, and the data are very, very expensive. So we have been starting to see, uh, terms even to the Llama days for, for when Meta was releasing Llama, they do have a term of if your daily active user or like annual recurring revenue exceeds some threshold, please enter into a commercial agreement with Meta specifically, right?
- MBMatt Bornstein
I, I do remember the-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... the numbers were like specifically chosen at that time, that you could go find it was like two companies in the world-
- SMSimon Mo
Yeah. [chuckles]
- MBMatt Bornstein
... that like fit the definition that they had excluded from their license.
- SMSimon Mo
Yeah, exactly. But like people have taken a hint from that, especially now th- the labs are trying to figure out a way to economically fund it, especially when they re- open source a model, everybody can just take it and run it themselves, whereas nobody will use their API anymore in many cases, while their API currently still taking up shape, right? And now we're seeing a very healthy ecosystem development starting from even Minimax recently ha-- when they're releasing the M2.7 model, they have a term specifically focusing on, uh, usage. And Kimi initially also has like if you have derivative, derivative works, like this is kind of big news back then was, um, Fireworks and Cursor about how they build on top of Kimi model. Yeah.
- MBMatt Bornstein
And, and it's, uh, if I c- I, if I could just expand on that a little, like-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... I, I, don't think it's greed, at least what I've seen from open source model labs, right?
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Open source models, really what we're talking about are open weights.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Right? And, and it's just not software, right? Like a, an AI model is not software-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... at the end of the day. Um, and so open source software used to be supported by people donating their time or big companies kind of authorizing their employees to donate their time.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
So it was sort of like a bulk, in-kind, you know, donation of people's time. That really doesn't work in AI.
- SMSimon Mo
Yep.
- EBElena Burger
Mm-hmm.
- MBMatt Bornstein
Right? Like, like I can't just like go home at night and like train a frontier open-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... source model with friends-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... for fun. Like we need millions or billions of dollars of computing resources-
- 28:51 – 32:21
Why Open Source Inference Is the Only Way to Scale Agents
- SMSimon Mo
Yeah. So scale comes from a few points. It comes from whether or not you can run this gigantic model on a data center rack and, but also it comes from can you optimize it to the extreme point, to the speed of light so that you are getting the most value out of it? But also you need a whole community and collaboration and partners, uh, of effort to validate it and making sure that there's no that last remaining bugs that appear like 0.0001% of the time. So this is where scale, scaling up the deployment and then making sure more people are running it as sc- uh, uh, like largest footprint possible will ensure everybody's experience of running this model is better. And this is where kind of... This is why open source inference is the current leading way right now instead of closed source inference engine. And, uh, frankly, right, all the, a lot of the open, uh, a lot of the open... Sorry. A lot of the inference cloud and API as a service today do use and leverage open source inference engine under the hood. And the reason to do that is just there's just so much battle tested and learned recipes and, uh, things that they can b-build on top of. And this is where open source is the current leading way to run models.
- EBElena Burger
Something that I find kind of funny in like looking back at not just the history of vLLM and Inferact, but also a company like OpenRouter or even Ollama, all of these different teams kind of got started around 2022 and 2023, some of them even before ChatGPT, in your case, uh, built around open models. And what kind of special thing do you think was happening at that time, just like in the world of research and in the world of AI, where we see these companies now and we look at them and you think of vLLM as mission-critical or OpenRouter as mission-critical, um, and what do, what do you think was going on at that time where even before we had, you know, a big consumer use case, we had, you know, teams like yours that were, were building out these things and, and kind of how did you guys know, I guess is, is the question?
- SMSimon Mo
I think, uh, there's two parts to this. Our team always have a open source kind of angle. We're from UC Berkeley, a long tradition of open source software and our system research principles.
- EBElena Burger
Mm-hmm.
- SMSimon Mo
Um, but also just curiosity overall. Everybody was so curious about how are these AI models gonna benefit the humanity and how can we use it better? And this is where open source is where you meet the most mission-aligned people together, right? OpenRouter, Ollama, as you mentioned, they're all great partners and friends in the ecosystem because we're there to understand how everybody else can leverage, uh, AI model better and making sure they have a good use of it.
- EBElena Burger
Speaking of, you know, leveraging open-weight open source models really well, so I think another thing that was in the news really recently, um, was Hugging Face using a Chinese open source model to help contain a cyberattack-
- SMSimon Mo
Yeah
- EBElena Burger
... uh, carried out by a rogue unsandboxed OpenAI model that was being tested. Um, so like what can we take away from something like that? And can you just first, I mean, walk us through your impressions and understanding of what happened and then kind of what, what you think the, the takeaways there are?
- 32:21 – 36:57
The Hugging Face Incident & Why Guardrails Break Down
- SMSimon Mo
Yeah, like this kind of goes back to our previous point about control. So for the Hugging Face incident, they break it down pretty care- uh, pretty, pretty, pretty well on their website and blog, so really thank for their transparency there. In the end, it's about all the closed proprietary model API, their guardrails are a little bit arbitrary, but also very difficult to enforce.
- EBElena Burger
Mm-hmm.
- SMSimon Mo
That means they have so much false positive in the guardrails that will have legitimate use cases just being blocked. And this is like a evergreen problem, even in the social media days. Like how do you, uh, design content filters, uh, correctly and like be able to do the moderation correctly? So if moderation is never solved, which is gonna be very, very hard, then there's always a place where you have a model where you know and trust that you are publishing to and to be able to use from. So Hugging Face, they have to use open-weight model. But in the future, we'll also see for the trusted use case, people will go to open-weight by default, because that is where you know for sure that, uh, the guardrail is, uh, lessened or you can control your guardrail for trusted use cases. Uh, even for us, like this is actually applied to us today, uh, where, uh, a lot of the Anthropic model do... are banning frontier AI use research, and then when we're studying GPU kernels, even as a invalid memory access error, we are triggering the, the red line. And so a lot of our developer within Inferact and for vLLM, um, are like retreating from using Fable-05 because you, you have a two-hour job and you trigger the, the, the red line, which is false positive, and then you have to lose all of your work. And so a lot of our developer are using like Kimi K3 today even, just, uh, making sure-- Because it's similar quality and it has a, uh, guardrail that makes sense to us, there goes we're using it.
- MBMatt Bornstein
I think your social media analogy is a really apt one.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
'Cause like in both, in both cases, what's kind of happened is you've taken like distributed human activity and kind of centralized it in one place. Like, like before social media, people talked to each other-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... right? And published articles-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... and like messaged each other-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... about on message boards. But then you centralize it all under one kind of like profit-seeking enterprise, and all of a sudden the incentives change like a lot, right? And we, we have a specific carve-out for social media, which is you're not responsible for what people say on your platform.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
And that allowed the moderation problem to be tractable, I think.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Right? It's like, okay, we're gonna do our best to eliminate obviously illegal things, um, you know, like things that are just gonna degrade experience on the site, you know, you know, in, in, in really bad ways. But like, you know, if it's on the edge, you know, it's something that's maybe out of fashion to say, but not illegal-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... like we're just, we, we, we just can't police everything. We can't be the world-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... police of like all human communications. I think something similar is sort of happening in AI, right? Where a lot of work, not just talking, but work, is kind of being consolidated in one place. If I need to write code or create a spreadsheet-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... or get advi- you know, health advice or like, like anything-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... it's all happening in like one or two sites.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
And they don't have that carve-out, right?
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Like they don't have that exemption of, of like, "Hey, we're not responsible for what actually happens on the site." And like some of them, especially, you know, Anthropic, like is, is kind of going further than even-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... what would be sort of like legally required, and they're-
- 36:57 – 43:32
Building a Company from an Open Source Project
- EBElena Burger
a company.
- SMSimon Mo
Yeah, Ian, as a co-founder, he has always been thinking about open source and where-- how do you support open source better? And then now with experience from Databricks and Anyscale at, and even Arena, which is a, a public platform. And now at Inferact, his focus really stem from how do we build such a... In a world where this software is being used by so, so, so many cr- uh, people on such a critical project, how do you make sure they're getting the best quality and where the value come from, right? If the open source movement and open source inference engine is a starting point, then what are the gaps that we can fill as Inferact and then what are the gaps that we should absolutely put in? Ian has always been open source first. So for whatever we need to build, we would like to build them in open source. And then for whatever value we can deliver, we want to really close the last mile and making sure that, uh, customer and partners are getting the best out of it.
- EBElena Burger
Just, I guess, to close this out, five years from now, uh, do you think open-weight, open source AI models, have they closed the gap with frontier models completely?
- SMSimon Mo
Mm-hmm.
- EBElena Burger
Are frontier models always one step ahead?
- SMSimon Mo
Yeah.
- EBElena Burger
Kind of how, how do you see that shaking out?
- MBMatt Bornstein
Five years.
- SMSimon Mo
Five years is a lot. [laughs]
- EBElena Burger
Five years. [laughs] All right. All right. One year.
- SMSimon Mo
One year.
- EBElena Burger
One year.
- SMSimon Mo
Um-
- EBElena Burger
Yeah, five years we're all, [laughs] you know, who knows?
- MBMatt Bornstein
We're either, you know, we're all gonna be like just floating around in our Wall-E pods-
- EBElena Burger
Yeah. [laughs]
- SMSimon Mo
[laughs]
- MBMatt Bornstein
... on spaceships and, you know.
- EBElena Burger
Exactly.
- SMSimon Mo
Progress, right?
- MBMatt Bornstein
Yeah. [laughs]
- SMSimon Mo
Yeah. Um, for, for me really, at, at this point, there's kind of a point we haven't talked too much about is what really differentiate open s- open-weight model from closed-weight model, right? In the end, there's not much differentiation. It's, uh, more about their distribution strategy and go-to-market strategy. And the capability wise, I don't really see a big gap, not even today, because for how these model are coming to being, they're really starting from the first principle, right? You have a computer cluster, you have training data, and you have brilliant researchers, uh, that group together and really to build this amazing artifact that is this mo- pre-trained model and then later our own, uh, post-trained model, and that the world can use. But if you look at the ingredients, right, the, one of the most important part just the data. It's about who gets what data and then what are the environment you are building to let the model improve on itself and make better, right? One of the very useful, uh, benchmark that we have on Arena for, uh, for, for K3 has been front-end coding, right? That means for Moonshot, they have built some of the best environment for front-end coding, right? They have published amazing demo on the ability for this model to code and then see what the rendered is, and then kind of continue looping and this iterative process. Now, this is about their environment to improve the model. It's not about just source data. It's not about where to get the data from. Rather, it's who can build the best environment and who can make the most sort of, uh, optimization and algorithmic choices to leverage all those learning from this environment. So the next year is all gonna be about that. It's about how open way model labs are differentiating and really getting the model to meet the real world and have this kind of what people are popular today, like recursive self-improvement almost, to really improve the model overall. And so really project out in a year, there's not gonna be any difference.
- EBElena Burger
Yeah.
- MBMatt Bornstein
Yeah. And, and you've used this term brilliant researchers a few times.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Um, there are brilliant researchers everywhere in the world, clearly. Um, why, why do you think, you know, in the US all the smart researchers are working on closed models and in, and in China all the smart researchers are working on open models?
- SMSimon Mo
I mean, from my point of view, they are attracted to interesting problems, not necessarily on the open or closed stance, but rather-- but however, open-weight model does give people a really, really good boost on the impact of such models. So that is, uh, like a plus.
- MBMatt Bornstein
Mm.
- SMSimon Mo
And I, I think all the brilliant researchers are attracted to how to improve the model overall, right? Like, actually one interesting point about this, um, uh, m-maybe fairly technical, uh, for, for this Kimi K3 model is they removed a, uh, rotary positional embedding. So RoPE has always been there for a lot of the transformers model. And guess who removed it? Is the inventor of RoPE, who are-
- MBMatt Bornstein
Oh, that's so funny
- 43:32 – 46:01
The Distillation Debate: Is It Critical or Incidental?
- MBMatt Bornstein
part of it?
- SMSimon Mo
I, I, I would lean to the latter part, specifically, as I mentioned previously, environment matters so much today, so these are our own environments, right? These cannot be distilled. Like, you don't have other people's environment to really distill a copy from. It's about constructing it, understanding... also understanding the learning process. We cannot distill how the model learns with the environment. A lot of these are just not doable today. Um, there are things potentially you can do with rewriting the data sets, right? Making better pre-training data. But again, you can do it with any models.
- MBMatt Bornstein
Mm.
- SMSimon Mo
Any models are, are going to follow instruction, are going to be useful in terms of utility there. So I really don't think from currently what we're seeing, uh, this is a big cornerstone of what's powering the progress today. In the end, what's powering the progress is still just, um, really smart people with very interesting algorithms, data environment, and they will produce, of course, compute, they will produce the models.
- MBMatt Bornstein
I think, yeah, it has really interesting policy implications. I, I, I tend to agree with you, by the way-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... that, that, you know, we have smart people everywhere working on a bunch of smart things, and it's-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... not about, you know, distilling data from any one place. Um, it has really interesting policy implications, right?
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Because it doesn't... You know, it's almost tempting if you're, if you're sort of, you know, in the White House to say, "Oh, sure. We'll just, we'll just turn off distillation. All our problems will be solved." But, like, you know, I think it's more the case that they're just, you know, smart people doing interesting things, and so it's, so it's like how do we, how do we kind of like adapt, adapt to that, I think is a good question.
- SMSimon Mo
Yeah, and creative innovations, right? Like, one part in my essay we kind of mentioned that open source and open way really helps innovation because it set out this racetrack where everybody can learn from each other and see where each person, like, every, uh, each player is in this racetrack, and then you are able to improve and stand on shoulder of each other kind of to improve yourself. So, uh, it's, that is where everybody can move forward faster.
- MBMatt Bornstein
Yeah. And w- one thing we're looking for a lot from an investment standpoint is, um, is people doing more open source model training all, all over the world, not just in China-
- SMSimon Mo
Yeah
- MBMatt Bornstein
... you know, not just in the United States, but all-
- SMSimon Mo
Yeah.
- MBMatt Bornstein
Right. Because, you know, you get that sort of magic of collaboration when everybody's doing it.
- SMSimon Mo
Yeah.
- MBMatt Bornstein
And achieve global harmony in all, all senses.
- SMSimon Mo
[laughs]
- EBElena Burger
Well, I think that's a good note to end on. Um, Matt, Simon, thank you so much for joining us. Thanks for tuning in.
- MBMatt Bornstein
Thanks so much.
- SMSimon Mo
Cool. Thanks, Shannon. [upbeat music]
Episode duration: 46:17
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode 78-6dUROziQ
