The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

1. 0:00 – 2:01

   Intro Song by Suno.ai

   1. AFAlessio Fanelli0:00

      [ on-hold music] Hey everyone, welcome to the Late in Space podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and today we're in the Singapore studio with Swyx.

   2. SWswyx0:12

      [chuckles] Hey, uh, this is our long-awaited one-on-one episode. Uh, I don't know how long ago the previous one was. Do you remember? Three, four months?

   3. AFAlessio Fanelli0:23

      No. Yeah, it's been a, it's been a while.

   4. SWswyx0:25

      It's been a minute. And people really enjoyed it. It's, it's just that really I think our travel schedules have been really difficult to get this stuff together. And then we also had like a decent backlog of guests for a while. I think we've kind of depleted that backlog now, and we need to build it up again. [chuckles]

   5. AFAlessio Fanelli0:38

      [chuckles]

   6. SWswyx0:38

      But it's, it's been busy and there's been a lot of news, so we actually get to do this like sort of rapid fire thing. I think some people-- You know, the podcast has grown a lot in the last, uh, six months. Maybe just in reintroducing like what you're up to, what I'm up to, and, um, why we're here in Singapore and stuff like that.

   7. AFAlessio Fanelli0:55

      Yeah. My first time here in Singapore-

   8. SWswyx0:56

      [chuckles]

   9. AFAlessio Fanelli0:56

      ... which has been really nice. This country is really amazing, I would say. First of all, everything feels like the busiest part of the city. Everything is skyscrapers. There's like plants on all the buildings, or at least in the areas that I've been in, which has been awesome. And I was at, uh, one of the offices kind of on the south side, and from the 30th floor you can see Indonesia on one side and you can see Malaysia on the other side.

   10. SWswyx1:19

       Mm-hmm. Yeah.

   11. AFAlessio Fanelli1:19

       Um, so it's, uh, quite, quite small. One of the people there said their kid goes to school at the border with Malaysia, basically, so they could drive to Malaysia every day. [chuckles]

   12. SWswyx1:27

       [chuckles]

   13. AFAlessio Fanelli1:28

       Just to go pick her up from school. Yeah, and we came here, we hosted, um, with you the Sovereign AI Summit Wednesday night. We had, uh, a lot of, a lot of folks-

   14. SWswyx1:36

       Nvidia, Goldman, Temasek, uh-

   15. AFAlessio Fanelli1:38

       GIC-

   16. SWswyx1:38

       Singtel

   17. AFAlessio Fanelli1:39

       ... Singtel. And we got to talk about this trend of sovereign AI which maybe we might cover on another episode. But basically, how do you drive-- If you're a country, how do you drive productivity growth in a time where populations are shrinking, the workforce is shrinking, and AI can kind of supplement a lot of this? And then the question is, okay, should I put all this money in foundation models? Should I put it in data centers and infrastructure?

2. 2:01 – 5:49

   Swyx and Alessio in Singapore

   1. AFAlessio Fanelli2:01

      Should I put it in GPUs? Should I put it in agents and whatnot? So we'll touch on some of these trends in, in the episode, but it was a fun event, and I did not expect some of the most senior people at the largest financial institution in Singapore ask about state space models and some of-

   2. SWswyx2:16

      Yeah

   3. AFAlessio Fanelli2:16

      ... the alternatives. So it's great to see, um, how advanced the conversation is sometimes.

   4. SWswyx2:21

      Yeah. I think that that is mostly people trying to listen to jargon that is being floated around as like, oh, what could kill transformers? And then they jump straight there without actually exploring the fundamentals, the basics of what they'll actually put to work. That's fine. It's a, it's a forum to ask questions, so it's-- you want to f- ask about the future. But I feel like it's not very practical to spend so much time on, on those things. You know, part of the things that I do at Late in Space, especially, um, when I travel, is to try to ask questions about what countries that are not the US and not San Francisco can do because everyone feels a bit left out. Y- y- you feel it here as well, and, uh, I'm trying to promote alternatives. I think AI engineering is one way that countries can, can capitalize on the industry without building a hundred billion dollar cluster, which is one-fifth the GDP of Singapore. Um, [chuckles] and, and, uh, a-and, and so, you know, my, my pitch at the, uh, summit was that, uh, we would-- Singapore would be the AI general nation. We're also working on bringing, um, the AI General Conference to Singapore next year together with ICLAIRE. So yeah, we're, we're, we're just trying my best and, you know, I'm, I'm being looped into various government meetings to, to try to make that happen.

   5. AFAlessio Fanelli3:32

      We'll, we'll definitely be here next year. We'll be-- I, I'll be back here very often.

   6. SWswyx3:36

      [chuckles]

   7. AFAlessio Fanelli3:37

      It's, uh... It's really nice. So.

   8. SWswyx3:38

      Yeah. Awesome. Okay. Well, well, we have, uh, you know, a lot of news. Uh, how do you think we should cover?

   9. AFAlessio Fanelli3:44

      Maybe just recap since the framework of the Four Wars of AI-

   10. SWswyx3:48

       Mm

   11. AFAlessio Fanelli3:48

       ... is something that came up-

   12. SWswyx3:49

       People keep coming back to it

   13. AFAlessio Fanelli3:50

       ... end of last year.

   14. SWswyx3:50

       Yeah.

   15. AFAlessio Fanelli3:51

       Um, so basically we'll link in the show notes, but the end-of-year recap for twenty twenty-three was basically the Four Wars of AI, uh, which we picked GPU Rich versus GPU Poor, the Data Quality Wars, the Multimodality Wars, and the RAG/Ops Wars. So usually everything falls back under those four categories. So I, I'm pretty happy that seven months later, uh, it's something that still matters given that-

   16. SWswyx4:16

       It still kind of holds up.

   17. AFAlessio Fanelli4:17

       Yeah. Most, most AI stuff from eight months ago, it's really not that relevant anymore. Um, and today [chuckles] we'll, we'll try and bucket some of the, the recent news on it. We haven't done a monthly thing in like three months, so three months-

   18. SWswyx4:31

       Yeah

   19. AFAlessio Fanelli4:31

       ... is a lot of stuff.

   20. SWswyx4:32

       That's mostly because I got busy with the conference. [chuckles]

   21. AFAlessio Fanelli4:35

       Yeah.

   22. SWswyx4:36

       Um, but I, I do want to-- I actually, I, I do want to get, get, get back on that horse or, or maybe just do it weekly so that I don't have such a big lift that I don't do it. I think the activation energy is, is the problem really. Um, so yeah, uh, I think frontier model-wise, it seems like Claude, it has really carved out a posis- persistence space for itself. You know, for a long time, Anthropic was kind of like a clear number two to OpenAI, and with 3.5 Sonnet, at least in like the some of the hard benchmarks on LMSYS or coding benchmarks on LMSYS, it is the undisputed number one model in the world, even with 4o Mini. Uh, and we can talk about 4o Mini and, and benchmarking later on. But for Claude to be there and hold that position for, uh, what is almost-- what is more than a month now in, in AI time is, is a big deal. There's not that much that people know publicly about what they-- what Anthropic did for Claude Sonnet, but I think it's, it's still a, a huge achievement. It, it marks the beginning of a non-OpenAI-centric world to the point where people on Twitter have canceled ChatGPT. That's been a trend that's been going on for a while. We talked about the unbundling of ChatGPT, but now, like new open source projects and tooling, they're just built for Claude. Like, they don't even use OpenAI. That's a s- a strategic threat to OpenAI, I think, a little bit.

3. 5:49 – 6:35

   GPU Rich vs Poors: Frontier Labs

   1. SWswyx5:49

      Uh, obviously, OpenAI is so big that it d- doesn't really care about that, but, uh, for Anthropic is a big win. I, I think like to, to see that going and to see Anthropic differentiating itself and actually implementing research. Uh, so the rumor is that the scaling monosemanticity paper that they put out, um, two months ago- Was a, a big part of Claude 3.5 Sonnet. I've had off-the-record chats with people about that idea, and they, they, they don't agree that it is the only cause. Um, so I, I was, I was thinking like this is the only thing that they did. But, um, you know, people, people say that, uh, there's about four or five other tricks that they haven't disclosed yet that went into 3.5 Sonnet. But the scaling on the semanticity paper is a very, very good read. It's a very long read, but it basically says that you can find control vectors of control features now that you can turn on to make it better at code

4. 6:35 – 10:37

   GPU Rich Frontier Models: Claude 3.5

   1. SWswyx6:35

      without really retraining it. You just train a whole bunch of sparse autoencoders, find a bunch of features and just say like, "Let's up those features," and suddenly you're better at code or suddenly you care a lot about the Golden Gate Bridge. These are the same things to the model. That is a huge, huge win for interpretability because up till now we were only doing interpretability on toy models, like a few million parameters, a model of Go or chess or whatever. Claude 3 Sonnet was interpreted and usefully improved using this technique. Wow.

   2. AFAlessio Fanelli7:05

      Yeah, I, I think it would be amazing if we could replicate the same on the open models to then-- because now we can use Llama 3.1 to generate synthetic data for training and fine-tuning. I think obviously Anthropic has a lot of compute and a lot of money, so once they figure out, okay, this is what we should make the model better at, they can kind of like put a lot of resources. I think in open source it's probably gonna be a more distributed effort, you know? Like I feel like News has held the crown of like the best fine-tuning data set owners for a while, but at some point that should change, hopefully, you know. Like other, other groups should, should step up and I think if we can apply the same principles to like a model as big as 4 or 5B and bring them into like maybe the 7B form factor, that would be great. But yeah, Claude is great. I told ChatGPT a while ago, every small podcaster run for latent space, it runs both on Claude and on OpenAI and Claude is definitely better most of the time. It's not a benchmark, it's just vibes. But when the vibes are good, uh, the vibes are good, you know?

   3. SWswyx8:02

      We run, uh, most of AI News summaries on Claude as well. Um, and but and I always run it against OpenAI. Sometimes OpenAI wins. Um, I, I do a daily comparison, but yeah, Claude, Claude's very strong at summarization and instruction following, which is something I care a lot about. So when you talk about frontier models, MMLU no longer cut it, right? Like, uh, we have reached, reached like 92 on MMLU. It's going to like 95, 97. It just means you're memorizing MMLU. Like the it there, there's some fundamental re- irreducible level of mistakes because of M- uh, MMLU's quality. Uh, we talked about this with Clementine on the Hugging Face episode and so we, we need to, to see what, uh, what else. What is the next frontier? I think there are 10 directions that I outlined, uh, below, but we'll talk about that later. Yeah, should we move on to Llama 3?

   4. AFAlessio Fanelli8:43

      Yeah, 3.1. I guess that to make sure to good differentiate between the, the models.

   5. SWswyx8:49

      Yeah.

   6. AFAlessio Fanelli8:49

      But yeah, we have a whole episode with, uh, Tomas Shalom from, um, the, the Meta team which was really, really good and I'm glad we got the podcast to come out at the same time as the model. Um-

   7. SWswyx8:59

      Yeah, I think we're the only ones to coordinate for the paper release for the, for the big launch, the 4.5 launch. Zuck did a few interviews, but we're the only ones that did the technical team interview. [chuckles]

   8. AFAlessio Fanelli9:07

      Yeah. Yeah, yeah. Zu- I mean, they were like surfing or something with the Bloomberg person. We should get invited to surf with Zuck but I think the-

   9. SWswyx9:16

      I would, I, yeah, I would be down to-

   10. AFAlessio Fanelli9:16

       To the-

   11. SWswyx9:17

       [laughs]

   12. AFAlessio Fanelli9:17

       To the audience the, the technical breakdown was-

   13. SWswyx9:19

       So, so behind the scenes, you know, uh, one for listeners, one thing that we have, we have a tension about is who do we invite? Because obviously if we get Mark Zuckerberg it'll be a big name, then it will cause people to download us more, but it will be a less technical interview 'cause he's not on the research team. He's, he's CEO of Meta. And so like, yeah, I think it, it's this constant back and forth. Like we want to grow as a podcast, but we want to serve a technical audience and we're trying to do that and tr- thread that line because our currency as podcasters is the people that listen to it and we need big names but we also need to serve our audience well and I think, um, if we, if we don't do it well... This actually goes all the way back to George Hotz when, when after he re- finished recording with us he, he said, "You have two paths in the podcast world. Either you go be Lex Fridman or you stay, you stay small and niche." Uh, and we, we definitely like, we like our niche. We, we think it's a, it's a good niche. [chuckles] It's gonna grow but at the same time we s- I, I still want us to grow. I, I want us to grow on YouTube, right? And, and so, uh, that's, that's always like a, a Meta thing that not to get too Meta.

   14. AFAlessio Fanelli10:15

       No, not that Meta.

   15. SWswyx10:17

       [laughs]

   16. AFAlessio Fanelli10:17

       The, the other Meta. Um-

   17. SWswyx10:18

       Uh, yeah, so Llama 3, yeah.

   18. AFAlessio Fanelli10:19

       I think to me the biggest thing is the training on outputs. Like every company is just hiding the fact that they've been fine-tuning and training on GPT-4 outputs and you cannot technically do it but obviously OpenAI is not enforcing it. I think now for the first time there's like a clear path to how do we make a

5. 10:37 – 15:41

   GPU Rich helping Poors: Llama 3.1: The Synthetic Data Model

   1. AFAlessio Fanelli10:37

      7B model good without having to go through GPT-4 or going to Claude 3 and we'll kind of talk about this later but I think we're seeing maybe the, you know, not the death but like selling the picks and shovels is kind of going away and like building the vertical things is like where most of the value is actually getting captured at least at the early stages. So being able to make small models better at specific things through a large model is more important than yet another 7B model that I can try and use but at the end of the day I still need to go through the large labs to fine-tune. So that to me is the most interesting thing. You know, it's such a large model that like it's obviously amazing but I don't know if a lot of people are switching from GPT-4 or Claude 3.5 to run 4.5B. I also don't know what the hosting op- hosting options are as far as like scaling, you know. I don't know if, if the Fireworks and Togethers of the world how much capacity they actually have to serve this model because at the end of the day it's, uh, it's a lot of compute if some of the big products would switch to it and you cannot easily run it yourself. So, um, I don't know but to me the synthetic data piece is definitely the most, the most interesting.

   2. SWswyx11:46

      Yeah. Uh, I would say that it is not enough now to say that synthetic data is real. Uh, I sh- I actually shipped that in the original email and then I changed that, uh, in the, in the so that what you see now in the, in the podcast description. But because there it is so established now that synthetic data is real therefore you need to go to the next level which is, okay, what do you use it for and how do you use it? And I think that is what It was interesting for Llama 3 for me. If you, if you read the paper, 90 pages of, uh, all filler, no killer is, is, or something like that is, is what the people were saying. Very, very like for once a frontier model with proper paper instead of a marketing blog post. And, uh, you know, they, they actually spelled out how they'd use syn-synthetic data for a few different domains. So they have, uh, synthetic data for code, for math, for multilinguality, for m-long context, for tool use, and then also for ASR and voice generation. And I think that, yeah, y- okay, now you have the license to go distill Llama 3, 4, or 5B, but w- how do you do that? That is the sort of the next frontier. Now you have the permission to do it. How do you do it? And, uh, I think people are, you know, gonna reference Llama 3 a lot, but then they can use those techniques, uh, for, for everything else. W- you know, in our episode with Thomas, he, he talked about syn- uh, like I was very focused on synthetic data for pre-training 'cause that's, that's my context, that's my conversations with Technium from News and all, and, um, all the other people doing synthetic data for pre-training and fine-tuning. But he was talking about post-training as well, and for, uh, everything here was post-training. Uh, in, in fact, I wish we had spent more time with Thomas on, on this stuff. Uh, we just didn't have the paper beforehand. [laughs]

   3. AFAlessio Fanelli13:20

      [laughs]

   4. SWswyx13:21

      Uh, but I think like when I call Llama 3 the synthetic Synthetic Data Model is you have the license for it, but then you also have the roadmap, the recipe, because it's in the paper. And now, like now everybody knows how to do this. Uh, and, and probably, you know, o- obviously, like OpenAI is probably laughing at us 'cause they w- they did this like a year ago, but now it's in the open.

   5. AFAlessio Fanelli13:40

      I mean, they can laugh all they want, but they're coming for them.

   6. SWswyx13:42

      [laughs]

   7. AFAlessio Fanelli13:43

      Um, I, I think, I mean, that's definitely the biggest vibe shift, right? It's like obviously Llama 3.1 is good. Obviously Claude is good. Maybe a year and a half ago you didn't get the benefit of the doubt as like an OpenAI competitor to be state-of-the-art. You know, it was kinda like, oh, Anthropic, yeah, those guys are cute over there. They're trying to do their thing, but it's not OpenAI. And like Llama 2 is great, but like it's really not a serious model, you know? It's like just good enough. I think now it's like every time Anthropic releases something, people are like, "Okay, this is like a serious thing." Whenever like Meta releases something, it's like, okay, they're at the same level. And-

   8. SWswyx14:17

      Mm-hmm

   9. AFAlessio Fanelli14:17

      ... I don't know if OpenAI is kinda like sandbagging the GPT next, you know?

   10. SWswyx14:22

       They're releasing wait, waitlists. [laughs]

   11. AFAlessio Fanelli14:23

       Yeah. I don't-- [laughs] And then they kinda, you know, yesterday or today they announced the search GPT thing-

   12. SWswyx14:29

       Yeah

   13. AFAlessio Fanelli14:29

       ... behind the waitlist, you know.

   14. SWswyx14:30

       Th- this is the Singapore confusion. Wh- when was it that something happened?

   15. AFAlessio Fanelli14:32

       Yeah. When was it? Yeah.

   16. SWswyx14:33

       Because it happened yesterday US time, but today Singapore time. [laughs]

   17. AFAlessio Fanelli14:36

       Yeah. So Thursday. [laughs]

   18. SWswyx14:38

       Um-

   19. AFAlessio Fanelli14:38

       It's been really confusing. But yeah, and people are kinda like, "Uh-oh. Okay, OpenAI, I don't know if we can take you seriously now." So-

   20. SWswyx14:46

       Well, no, uh, the, one of the, um, uh, AI Grants, um, employees, I think Hersh tweeted that, you know, you can skip the waitlist, just go to perplexity.com. [laughs] And it was, it was a really, really sick burn for the, uh, OpenAI search GPT waitlist. But y- their implementation will have something different. They probably like train a sp- dedicated model for that, you know? Like it-- they'll have some innovation that we haven't seen.

   21. AFAlessio Fanelli15:08

       Yeah, data licensing obviously. Um-

   22. SWswyx15:10

       Uh, data licensing. [laughs] Yes. We're optimistic, uh, you know, uh, but the, the vibe shift is real, and I think that's something that is just worth commenting on and watching and, um, yeah, how the other labs catch up. I think what you said there is actually very interesting. The, the trend of successive releases is very important to watch. If things get less and less exciting, then it's a red flag for that company. And if it's-- things get more and more exciting, it means that they-- these guys have, have a good team, they have a good plan, good ideas. Um, so yeah, like, uh, I will call out, you know,

6. 15:41 – 18:26

   GPU Rich helping Poors: Frontier Labs Vibe Shift - Phi 3, Gemma 2

   1. SWswyx15:41

      uh, the Microsoft Phi team as well. Um, Phi-1 was kind of widely regarded to be over-trained on benchmarks, and Phi-2 and Phi-3 subsequently improved a lot as well. I would say also similar for Gemma, Gemma 1 and 2. Uh, Gemma 2 is currently leading in terms of the, uh, local Llama sort of vibe check eval [laughs] informal straw poll. And, and th- and that's only like a month after release. They released at the, um, AI Engineer World's Fair. And, um, you know, like I didn't know what to think about it 'cause Gemma 1 wasn't like super well-received. It was just kind of like, here's, here's like free tier Gemini, you know? But, but now Gem- Gemma 2 is actually like a, a very legitimately widely used, uh, model by, by the open source and, uh, local Llama community. So that's great until Llama 3 7B came along. [laughs]

   2. AFAlessio Fanelli16:28

      Yeah.

   3. SWswyx16:28

      And so like the, uh... And we'll talk about this also, like just the, The Winds of AI Winter is als- also like what is the depreciation schedule on this, on this model inference and training class? Like it's, it's very high. [laughs]

   4. AFAlessio Fanelli16:39

      Yeah. I'm curious to get your thought on Mistral, uh, everybody's favorite sparkling weights-

   5. SWswyx16:44

      [laughs] Yeah

   6. AFAlessio Fanelli16:45

      ... um, company.

   7. SWswyx16:46

      Yeah.

   8. AFAlessio Fanelli16:46

      Um, they, they just released the, you know, Mistral Large Enough.

   9. SWswyx16:49

      Large- Mistral Large 2.

   10. AFAlessio Fanelli16:51

       Yeah. Large 2.

   11. SWswyx16:52

       Uh, so this was one day after Llama 3, uh, presumably because they were speaking at ICML, which is going on right now. Uh, by the way, uh, Brittany is doing a guest host, uh, thing for us. She's, she's running around the poster sessions doing what I do, which is very great 'cause I couldn't go 'cause of my visa issue. I have to be careful what I say here, but I think because we still want to respect their work, uh, but it-- Mistral Large I would say is like not as exciting as Llama 3. That-- I think that is very, very fair to say. It is, yes, another GPT-4 class model released as open weights with a research license, not a commercial license, but still open weights, and that's good for the community. But it is a step down in terms of, uh, the general excitement around Mistral compared to Llama. I think that would be fair to say, and I would say that to Mistral themselves. So the, the general hope is, and I f- [laughs] I cannot say too much just 'cause I have, I've had offline conversations with the people close to this. The general hope is that th- they need something more. You know, of the 10 elements of like what is next in terms of their frontier model boundaries, Mistral needs to make progress there. They made progress here with like, um, instruction following and structured output and multilinguality and all those things. But I think to stand out, you need to basically pull a stunt. You need to be a superlatively good company in one dimension. And now unfortunately, Mistral does not have that crown as open source kings. Uh, you know, like, like a year ago I was saying, Mistral are the kings of open source AI. Now Meta is They've lost their ground. Uh, by the way, they've also deprecated, uh, Mistral 7B, eight by seven B, and eight by 22 B, right? So n-now there's only, like, the closed source models that are at API platform. So

7. 18:26 – 21:56

   GPU Rich: Mistral Large

   1. SWswyx18:26

      has Mistral basically started becoming more of a closed model op-- uh, proprietary platform. I don't believe that's, that's true. I believe that they, they, they're still very committed to open source, uh, but they need to come up with something more that people can use. A-and that's a, that's a grind. I mean, they have, what? Six hundred million dollars to do it.

   2. AFAlessio Fanelli18:43

      Mm-hmm. Right.

   3. SWswyx18:44

      So that's, that's still good. Uh, but, you know, people are waiting for, like, what's next from, from them.

   4. AFAlessio Fanelli18:49

      Yeah. To me, the perception was interesting. In the comments of the release, everybody was like: Why do you have a non-commercial license? You're not making any money anyway from the inference. So I feel like the AI engineering tier list, you know, is kinda shifting in real time, and maybe Mistral, like you said before, was like, "Hey, thank God for these guys. They're saving us in open source. They're kinda like speed running GPT-1, GPT-2, GPT-3 in open source." But now it's like they're kind of moving away from that. I haven't really heard of that many people using them at scale commercially just from, you know, discussions. Um, so I'm curious to see what the next step is.

   5. SWswyx19:25

      Yeah, but also you're sort of US-based, and maybe they're not focused there, right? So-

   6. AFAlessio Fanelli19:29

      Yeah. No, exactly. That's the-

   7. SWswyx19:31

      The, the... It's a very big elephant, and we're only touching pieces of it. It's, it's blind, you know, blind leading the blind. [chuckles] Uh, I, I will call out, um, you know... They, they have some interesting experimentations with Mamba, and Mistral Nemo is actually on the efficiency frontier chart that I drew that is still relevant, so don't discount Mistral Nemo. But Mistral Large, uh, otherwise, like it's an, it's an update, it's a necessary update for Mistral Large V1. But other than that, they're just kind of holding the, the line-

   8. AFAlessio Fanelli19:58

      Mm-hmm

   9. SWswyx19:58

      ... not really advancing the field yet. That'll, that'll be my statement there. So those are the frontier big labs.

   10. AFAlessio Fanelli20:05

       Yes.

   11. SWswyx20:05

       And then now we're t- we're gonna shift a little bit towards the, the smaller deployable on-device solutions.

   12. AFAlessio Fanelli20:11

       Yeah. First of all, shout out to our friend Tri Dao, uh, who released FlashAttention 3. FlashAttention 2, we kinda did a deep dive on the podcast. He came on, uh, in the studio back then. It's just great to see how small groups can make a big impact on a whole industry, just like by making math better. So it's just great to see. Uh, just wanted to give Tri a shout-out.

   13. SWswyx20:33

       Something I mentioned there and something with-- that always comes up, even in the Sovereignty AI Summit that we did, was what-- does Nvidia's competitors have a, have any threat to Nvidia? You know, AMD, like Madex, like, um, Etched, you know, which, which caused a lot of noise with their Soho chip as well. And just the simple fact is that Nvidia has won the hardware lottery and people are customizing for Nvidia. Like FlashAttention 3 only works for Nvidia, only works for H100s. And, like, this much work, this much scaling, this much validation going into this stuff is very difficult to replicate or very expensive to replicate for the other hardware ecosystems. So not impossible. I actually heard a really good argument from one of, um... I think it is Martín Casado from a16z who was saying basically like, yeah, like absolutely Nvidia's hardware and ecosystem makes sense and obviously that's gon- that's contributed to its like, I don't know, like, it's like the most valuable company in the world right now. But current training runs are like one hundred million to two hundred million in cost. But when they go to five hundred million, when they go to a billion, when they go to one trillion, then you can ac-actually start justifying making custom ASICs for your run.

   14. AFAlessio Fanelli21:43

       Mm-hmm.

   15. SWswyx21:44

       And if they, if they cut your costs by like half, then you make your money back in one run.

   16. AFAlessio Fanelli21:49

       Yeah, yeah, yeah. Martín has always been a, a fan of, uh, custom ASIC. I think he... They wrote a really good post maybe a couple of

8. 21:56 – 23:45

   GPU Rich: Nvidia + FlashAttention 3

   1. AFAlessio Fanelli21:56

      years ago about, um, cloud repatriation-

   2. SWswyx21:58

      Oh, yeah [chuckles]

   3. AFAlessio Fanelli21:58

      ... and, uh, custom ASIC, so-

   4. SWswyx21:59

      I think he got a lot of shit for that, but, uh-

   5. AFAlessio Fanelli22:00

      Uh-

   6. SWswyx22:01

      ... it's becoming more consensus now, I think. So Noam Shazeer blogging again, fantastic gift to the world. This guy, nonstop bangers. Uh [laughing]

   7. AFAlessio Fanelli22:10

      [laughing]

   8. SWswyx22:11

      And so he's at Character.AI and, uh, they, they-- he put out a post talking about five tricks that they use to serve twenty percent of Google search traffic as LLM inference. A lot of people were very shocked by that number, but I think you just have to remember that most conversations are multi-turn, right? Like in the span of one Google search, I will send like ten text messages, right? So obviously there's like a ratio here that, uh, that matters. It's obviously a flex of Character.AI's traction among the kids because I have tried to use Character.AI since then and I still cannot for the life of me get it. I don't-- Have you tried, uh-

   9. AFAlessio Fanelli22:46

      I tried it but yeah, it's definitely not, not for me.

   10. SWswyx22:47

       Yeah, they launched like voice. I tried to talk to it. It was just so stupid. I just [laughing] I didn't like it myself but, uh, this is what it means-

   11. AFAlessio Fanelli22:55

       But please still come on the podcast, Noam Shazeer.

   12. SWswyx22:57

       Yeah, yeah, yeah.

   13. AFAlessio Fanelli22:57

       Sorry, we didn't mean [chuckles]

   14. SWswyx22:58

       No, no, no. It's, it... Because like, uh, like I, I don't really understand like what the use case is for apart from like the, the therapy role play homework assistant-

   15. AFAlessio Fanelli23:06

       Mm-hmm

   16. SWswyx23:06

       ... uh, type of, type of stuff that is the norm. But anyway, one of the thing-- most interesting things, the detailed five tricks, um, one thing that people talk a lot about is native int8 training. I, I got it wrong in our Thomas podcast. I said FP8 is in-int8. And I think like that is something that is a easy win. Like I-- we should, we should basically when we're getting to the point where we're over-training models 100 times past Chinchilla ratio to optimize for inference, the next thing is actually like, hey, let's stop using so much memory when, when training because we're not-- we're gonna quantize it anyway for inference. So like just let's pre- let's pre-quantize it in, [chuckles] in training. So that makes a lot of sense. The other thing as well is this concept of

9. 23:45 – 28:14

   GPU Rich helping Poors: Noam Shazeer & Character.AI

   1. SWswyx23:45

      global-local hybrid architecture, which I think is basically going to be the norm, right? Uh, so he has this formula of one to five ratio of global attention to local attention, and he says that that is, that is, uh, that works for the long-form conversations that Character has. Okay, that's great. And like simultaneously we have independence research from other companies about similar hybrid ratios being the best for their research. So Nvidia came out with a Mamba Transformer hybrid research thing, and their, in their estimation, you only need seven percent transformers. Everything else can be state space models. Jamba also had something like between like six to like 30 to one and, uh, it basically every form of hybrid architecture seems to be working at the research stage. So I think like if we scale this, uh, it makes complete sense that you, you just need a, a, a mix of architectures and, um, it could well be that the transformer block instead of transformers being all you need, transformers are the global attention thing and then the, the local attention thing can be the state space models, can be the RWKVs, can be another transformer but just, uh, limited by a sliding window. And I, I think like we're slowly discovering like the, the fundamental building blocks of AI. One is transformers, one is something that's local, whatever that is, and then, you know, who knows what else is next. Uh, may- the, the other, the other stuff is adapters, but we can talk about that. But yeah, headline is- The Gnome, maybe he's too confident, but I, I mean, I, I believe him [chuckles] . Gnome thinks that he can do inference at 13X cheaper than the Fireworks and the Together, right? So, like, there is a lot of room left-

   2. AFAlessio Fanelli25:18

      Mm-hmm

   3. SWswyx25:18

      ... uh, to, to improve inference.

   4. AFAlessio Fanelli25:20

      I mean, it does make sense, right? Because, like, otherwise, I don't know-

   5. SWswyx25:23

      Otherwise Character would be bankrupt by now [laughs] .

   6. AFAlessio Fanelli25:23

      Yeah, exactly. I was like, they would be-

   7. SWswyx25:25

      [laughs] .

   8. AFAlessio Fanelli25:25

      They would be losing a ton of money, so, um-

   9. SWswyx25:28

      They are rumored to be exploring a sale. Um-

   10. AFAlessio Fanelli25:30

       Yeah

   11. SWswyx25:31

       ... so I'm sure money is still an issue for them, but I'm s- also sure they're making a lot of money, so I... it's very hard to tell because it's not a very public company.

   12. AFAlessio Fanelli25:39

       Yeah. Well, I, I think that's one of the, mm, things in the market right now too, is like, hey, do you just wanna keep building? Do you wanna, like, just not worry about the money and go build somewhere else? Kinda like a maybe inflection and adapt in some of m- these other non-equi hires, licensing deals and whatnot. So I'm curious to see what companies decide to stick with it.

   13. SWswyx26:00

       It's strong... Like y- uh, I think Google or Meta should pay $1 billion for Gnome alone.

   14. AFAlessio Fanelli26:05

       Mm-hmm. Right.

   15. SWswyx26:06

       The, the, the purchase price for Character is a o- $1 billion.

   16. AFAlessio Fanelli26:09

       Mm-hmm.

   17. SWswyx26:09

       Which is super underpriced.

   18. AFAlessio Fanelli26:10

       Which is nothing at their market caps, right?

   19. SWswyx26:12

       It's nothing. It's nothing.

   20. AFAlessio Fanelli26:12

       Like Meta's market cap right now is-

   21. SWswyx26:15

       Is like 2 trillion

   22. AFAlessio Fanelli26:15

       ... 1.15 trillion-

   23. SWswyx26:17

       Yeah

   24. AFAlessio Fanelli26:17

       ... because they're down 5%, 11% in the past month.

   25. SWswyx26:21

       What?

   26. AFAlessio Fanelli26:21

       Um, yeah. So if you pay $1 billion, you know, that's like 0.01%-

   27. SWswyx26:27

       Yeah, yeah, yeah. That's nothing

   28. AFAlessio Fanelli26:27

       ... of your market cap. And they paid, they paid $1 billion for WhatsApp, and they buy 1% of their market cap-

   29. SWswyx26:33

       Yeah

   30. AFAlessio Fanelli26:33

       ... on that at the time, so.
10. 28:14 – 35:33

    GPU Poors: On Device LLMs: Mozilla Llamafile, Chrome (Gemini Nano), Apple Intelligence

    1. SWswyx28:14

       into, uh, their products by the, by the vendors. So Google Chrome is building Gemini Nano into the browser. The next version of Google Chrome will have Nano inside that you can use like window.ai.something, and it would just call Nano. There, there'll be no download, no latency what-whatsoever 'cause it runs on your device. And there's Apple Intelligence as well, which is Apple's version, which is f- in, in the OS accessible by apps. And then there's, there's a bun- long tail of others. But like, yeah, your, your comments on those things.

    2. AFAlessio Fanelli28:42

       My, my biggest question is how much can you differentiate at that model size? You know, like how big it's gonna be, the performance gap between all these models and like, are people gonna be aware of what model is running? You know, right now for the large models, we're still pretty aware of like, oh, is this Sonnet 3.5? Is this GPT-4? Is this, you know, 3.1, 4 or 5B? I think the smaller you get, the more it's just gonna become like a neutrality, you know? So like you're not gonna need a model router for like small models. You're not gonna need any of that. Like they're all gonna converge to like-

    3. SWswyx29:17

       Well-

    4. AFAlessio Fanelli29:17

       ... the best possible performance at this point

    5. SWswyx29:18

       ... actually Apple Intelligence is the model router, I think. Uh, they have something like 14... I, I did a count in, in my newsletter, like 14 to 20 adapters.

    6. AFAlessio Fanelli29:26

       Mm-hmm.

    7. SWswyx29:26

       And so based on your use case, they, uh, they'll, they'll route and, and load the adapter or they'll route to OpenAI. So there is some routing layer. Um-

    8. AFAlessio Fanelli29:34

       Yeah, yeah.

    9. SWswyx29:34

       To me, I think a lot of people were trying to puzzle out the strategic moves between OpenAI and, and Apple here because Apple is in a very good position to commoditize OpenAI. There were some rumors that Google was working with Apple to launch it. They did not make it for the launch. But presumably Apple wants to commoditize OpenAI, right? So, uh, you know, when you, when you launch, you can choose your preferred-

    10. AFAlessio Fanelli29:55

        Mm-hmm

    11. SWswyx29:55

        ... external AI provider, and it's either OpenAI or Google or someone else. And that puts Apple, Apple at the center of the world, uh, as, as with the ability to make routing decisions and, um, I think that's probably good for privacy, probably good for the planet 'cause you're [laughs] -

    12. AFAlessio Fanelli30:11

        Yeah

    13. SWswyx30:12

        ... you're not running like oversized models on like your, your, you know, your, your spellcheck, uh, pass. And, uh, I, I'm generally pretty positive on it. Like, yeah, I, I'm not concerned about the capabilities issue. It meets their benchmarks. Apple put out a whole bunch of proprietary benchmarks 'cause they don't like to do anything in the way that everyone else does it [laughs] . So like, you know, in the Apple Intelligence blog post, they like... I think like all of them were just like their internal human evaluations, and only one of them was an industry standard benchmark, which is, which is IF eval. Which is good, but like, you know-

    14. AFAlessio Fanelli30:41

        Yeah, yeah

    15. SWswyx30:41

        ... why didn't, why didn't you also release your MMLU? Oh, 'cause you'd suck on it. All right [laughs] .

    16. AFAlessio Fanelli30:45

        Well, I, I actually think all these models will be good. And on the Apple side, I'm curious to see what the price tag will be to be the default. Right now, Google pays them $20 billion to be the default search.

    17. SWswyx30:57

        I see.

    18. AFAlessio Fanelli30:58

        Uh, I, I wonder-

    19. SWswyx30:58

        The rumor is it's zero.

    20. AFAlessio Fanelli31:00

        Yeah, yeah. I mean, today, even if it was $20 billion-

    21. SWswyx31:02

        I see

    22. AFAlessio Fanelli31:03

        ... that's like nothing compared to like, you know, NVIDIA's worth $3 trillion. So like paying 20- even paying $20 billion to be the default AI provider like would be- Cheap compared to search, given that AI is actually being such a core part of the experience. Like Google being the default-

    23. SWswyx31:18

        Um

    24. AFAlessio Fanelli31:18

        ... for like Apple's phone experience-

    25. SWswyx31:20

        Yeah

    26. AFAlessio Fanelli31:20

        ... really doesn't change anything.

    27. SWswyx31:21

        Yeah, yeah.

    28. AFAlessio Fanelli31:22

        Becoming the default AI provider for like the Apple experience-

    29. SWswyx31:25

        Yeah

    30. AFAlessio Fanelli31:25

        ... would be worth a lot more than this.
11. 35:33 – 37:41

    Quality Data Wars: NYT vs The Atlantic lawyer up vs partner up

    1. SWswyx35:33

       million, $200 million a year of, of, uh, licensing deals just because you wanted to pick that war, which ideologically, I think they're absolutely right to do that. But, um, you know, the other people-- Uh, The Verge did a very good interview with, I think The Washington Post. I'm gonna get the, the, the, the outlet wrong. The, The Verge did a very good interview with a newspaper owner-editor on why they did the deal with OpenAI. And I think that listening to them on like their thinking through like the, the reasoning of like the pros and cons of picking a fight versus partnering, I think is very interesting.

    2. AFAlessio Fanelli36:08

       Yeah. The, uh, I guess the winner in all of this is Reddit, which is making over 200 million just in data licensing to OpenAI and some of the other AI providers. Um, I mean, [laughs] 200 million is like more than most AI startups are making. [laughs] Than they just, um-

    3. SWswyx36:22

       Yeah. So I think that was an IPO play because Reddit-

    4. AFAlessio Fanelli36:25

       Yeah, yeah, yeah

    5. SWswyx36:25

       ... conveniently did this deal before IPO, right?

    6. AFAlessio Fanelli36:27

       Totally.

    7. SWswyx36:27

       Is it like a one-time deal and then, you know, the, the stock languishes from there? I don't, I don't know.

    8. AFAlessio Fanelli36:31

       Yeah. No, well, their IPO has done... Well, I guess it's not gone down, so in this market they're up 25%, I think, since IPO. But I saw the FTC had opened a inquiry into it just to like investigate. So I'm curious what the antitrust regulations are gonna be like when it comes to data. Obviously, acquisitions are blocked to prevent kinda like stifling competition. I wonder if for data it will be similar, where, hey, you cannot actually gate all of your data only behind $100 million plus contracts because otherwise you're Stopping any new company, uh, from building a competing product, so, um-

    9. SWswyx37:08

       Yeah, I-- that's a serious overreach of the state there-

    10. AFAlessio Fanelli37:11

        Yeah, yeah

    11. SWswyx37:12

        ... I would say. As a, as a, as a free market person, I want to defend... This is weird. Like I'm a free market person, and I'm a content creator, right? So I want to be paid for my content.

    12. AFAlessio Fanelli37:19

        Right.

    13. SWswyx37:19

        At the same time, I, I believe that, you know, people should be able to make their own decisions about, uh, all these deals. But UGC is a weird thing-

    14. AFAlessio Fanelli37:27

        Mm-hmm

    15. SWswyx37:27

        ... 'cause UGC, UGC is contributed by volunteers.

    16. AFAlessio Fanelli37:30

        Yeah.

    17. SWswyx37:31

        And the, the other big news of Reddit is that apparently they have added to their robots.txt, like only Google should index us, right? 'Cause we did the deal with Google, and that's obviously blocking OpenAI from crawling

12. 37:41 – 41:03

    Quality Data Wars: Reddit, ScarJo, RIAA vs Udio & Suno

    1. SWswyx37:41

       them, Anthropic from crawling them, you know, Perplexity from crawling them. Perplexity maybe ignores all robot.txt-

    2. AFAlessio Fanelli37:46

       [laughs]

    3. SWswyx37:47

       ... but that's a whole different, uh, other issue. And then the other thing is I, I think this is big in the, the sort of normie worlds, um, the actors, you know, Scarlett Johansson had a very, very public Apple Notes takedown of OpenAI. Only Scarlett Johansson can do that to, to Sam Altman and then, you know... I was very proud of my newsletter for, for that day. I called it Sky Fall because the voice of the, that voice was Sky, so I called it Sky Fall. And but it's true, like, uh, you, uh, there's-- that, that one she can win. Um, and there's a very well-established case law there. And the YouTubers are in the music industry, the RIAA, like the most litigious, uh, section of the creator economy, um, has gone after Udio and Suno, you know, Mikey from our, uh, our podcast with him. And, uh, it's un-unclear what will happen there, but, um, it's, it's gonna be a very costly legal battle for sure.

    4. AFAlessio Fanelli38:34

       Yeah. Um, yeah, I mean, music industry and lawsuits name a more iconic duo, you know? So-

    5. SWswyx38:40

       Yeah

    6. AFAlessio Fanelli38:40

       ... I think this, uh, to be expected.

    7. SWswyx38:41

       So I was-- Yeah. I think last time we talked about this I, I was, uh, pretty optimistic that this, something like this would reach the Supreme Court and with the way that the Su- the Supreme Court is, is making rulings, like we just need a judgment on whether or not training on, on data is transformative use. 'Cause I think it is. Literally, we're using transformers to, to do transformative use. So then it's open season for, for AI to do it, and comparatively, the content creators and owners will lose out. They just will.

    8. AFAlessio Fanelli39:08

       Yeah.

    9. SWswyx39:09

       'Cause right now we're paying them money out of fear of lawsuits. If the Supreme Court rules that there are no lawsuits to be had, then all the money disappears.

    10. AFAlessio Fanelli39:16

        I think people are probably scraping latent space, and we're not getting a dime, so-

    11. SWswyx39:20

        [laughs]

    12. AFAlessio Fanelli39:21

        ... it is what it is.

    13. SWswyx39:22

        Yeah. Uh, no, you can, you can support with like an $8 a month subscription-

    14. AFAlessio Fanelli39:25

        Yeah. [laughs]

    15. SWswyx39:26

        ... and, and that pays for our microphones and travel-

    16. AFAlessio Fanelli39:27

        Yeah

    17. SWswyx39:27

        ... and stuff like that. Yeah, it's, you know, it's not, it's definitely not worth the amount of time we're putting into it, but it-

    18. AFAlessio Fanelli39:33

        Well

    19. SWswyx39:33

        ... it's a labor of love.

    20. AFAlessio Fanelli39:34

        Yeah, exactly. I-

    21. SWswyx39:36

        Synthetic data.

    22. AFAlessio Fanelli39:36

        Yeah. Yeah, I mean, I guess we talked about it a little bit before with Llama, um, but there was also the AlphaProof thing.

    23. SWswyx39:43

        Yes.

    24. AFAlessio Fanelli39:43

        Uh-

    25. SWswyx39:43

        Just before I came here, I was working on that newsletter.

    26. AFAlessio Fanelli39:46

        Yeah. Yeah, Google trained. They almost got a gold medal. I forget what the-

    27. SWswyx39:50

        Yes, they're one point short of the gold medal.

    28. AFAlessio Fanelli39:51

        Yeah, one point short of the gold medal.

    29. SWswyx39:52

        It's a, it's a remarkably-- I wish they had more questions. So the, the, the, so the International Math Olympiad has six questions, and each, each question is seven points. Every single question that the AlphaProof model tried, it got full marks on. It just failed on two, and then the cutoff was, was like sadly one, one point, uh, higher than that. But still, like it was a, it was a very big-- like a lot of people have been looking at IMO as like the next gold prize, grand prize in terms of what AI can achieve and betting markets and Evie atop Kowski has, has, has updated in saying like, "Yep, like we're, we're pretty close." Like we, we basically have reached it near gold medal status.

    30. AFAlessio Fanelli40:31

        Mm-hmm.
13. 41:03 – 45:33

    Quality Data Wars: Synthetic Data, Jagged Intelligence, AlphaProof

    1. AFAlessio Fanelli41:03

       Yeah.

    2. SWswyx41:03

       And it also doesn't generalize. So one thing that people are talking about is this concept of jagged intelligence. 'Cause at the same time we're having this as a discussion about being superhuman. You know, one of the, um, IMO questions was solved in 19 seconds after we gave the question to, uh, AlphaProof. At the same time, language models cannot determine if 9.9 is smaller than or bigger than 9.11. And part of that is 9.11 is an inside job. But [laughs] it's, it's a funny... And that is someone else's joke. I don't know. I really like that joke. Uh, but it's, it's jagged intelligence. It's, it's a failure-

    3. AFAlessio Fanelli41:37

       Yeah

    4. SWswyx41:37

       ... to generalize because of tokenization and because of whatever, and what we need is general intelligence. We've always been able to train dedicated special models to, to win prizes and do stunts, but the grand prize is general intelligence. That same model does everything.

    5. AFAlessio Fanelli41:51

       Is it gonna work that way? I don't know. I think like if you look back a year and a half ago, and you would say, "Can one model get to general intelligence?" Most people would be like, "Yeah, we can keep scaling." I think now it's like, is it gonna be more of a mix of models? You know, like can you actually do one model that does it all?

    6. SWswyx42:10

       Yeah, absolutely. I think GPT-5 or Gemini 3 or whatever, uh, would be much more capable at this kind of stuff while it also serves our needs, uh, with like, with everyday things. It might be a... It might be completely uneconomical.

    7. AFAlessio Fanelli42:26

       Mm-hmm.

    8. SWswyx42:26

       Like, why would you use-

    9. AFAlessio Fanelli42:27

       Right, yeah

    10. SWswyx42:27

        ... a giant ass model to, to do normal stuff? But it is just a demonstration of proof that we can build super intelligence for sure. [laughs] And, and then, you know, everything else follows from there. But, uh, right now we're just pursuing super intelligence. I always think about this, uh, you know, just reflecting on the GPU Rich Poor stuff and then now this Alpha Geometry stuff. I used to say you pursue capability first, then you efficient- make, make it more efficient. You make frontier model, then you distill it down to the AP-7B, 70B, which is what Llama 3 did. And by the way, also, OpenAI did it with GPT-4o and then distilled it down to 4o Mini, and then Claude also did it with Opus and then with 3.5 Sonnet, right? That suitable recipe, I in, in fact, I call it part of the deployment strategy of models. You train a base layer, you train a large one, and then you distill it down. You add structured output generation, tool calling and all that. You add the long context, you add like this, this standard stack of stuff in post-training that is growing and growing to the point where now OpenAI has opened a team for mid-training-

    11. AFAlessio Fanelli43:26

        Mm-hmm

    12. SWswyx43:26

        ... that happens before post-training. I think like- One thing that I've realized from this AlphaGeometry thing is before you have capability and you have efficiency, there's an in-between layer of generalization that you need to accomplish. You need to do capability in one domain. You need to generalize it, then you need to efficient-size it.

    13. AFAlessio Fanelli43:45

        Mm-hmm.

    14. SWswyx43:46

        Then you have good models. [laughs]

    15. AFAlessio Fanelli43:49

        That makes sense. I, I think like maybe the question is how many things can you make it better for before generalizing it, you know? Yeah, I don't have a good intuition for that but-

    16. SWswyx44:00

        Uh, we'll talk about that in the next, uh-

    17. AFAlessio Fanelli44:01

        Yeah

    18. SWswyx44:01

        ... next thing. Yeah, so we can skip Nemo- Nemotron's worth looking at if you're interested in synthetic data. Uh, multimodal labeling I, I think has happened a lot. We'll, we'll maybe we'll just jump to multimodal model now.

    19. AFAlessio Fanelli44:10

        Mm-hmm. Yeah, we got a bunch of news. Well, the first news is that, uh, 4o voice is still not out even though the, the demo was great. I think it-- they're starting to roll out the beta-

    20. SWswyx44:21

        Yeah

    21. AFAlessio Fanelli44:21

        ... the next week.

    22. SWswyx44:22

        So, so, uh, I am subscribing-- I subscribed back to ChatGPT Plus.

    23. AFAlessio Fanelli44:25

        You gave in?

    24. SWswyx44:26

        I gave in because [laughs] they're rolling it out next week. So you better be on the, the cutoff or-

    25. AFAlessio Fanelli44:30

        Nice bait

    26. SWswyx44:30

        ... you're not gonna get it. [laughs]

    27. AFAlessio Fanelli44:31

        Nice bait somehow, man.

    28. SWswyx44:32

        I, no, I said this. I said when I talk about unbundling on ChatGPT, it's, it's basically 'cause they had nothing to offer people. That's why-

    29. AFAlessio Fanelli44:38

        Mm-hmm

    30. SWswyx44:38

        ... people are unsubscribing 'cause like why keep paying $20 a month for this, right? But now they have proprietary models. Oh yeah, I'm back in, right? Like [laughs] I want-
14. 45:33 – 47:34

    Multimodality War: ChatGPT Voice Mode, OpenAI demo at AIEWF

    1. SWswyx45:33

       like the GPT-4o talk at the AI Engineer World's Fair. But second of all, because it was presented live at a conference with large speakers blaring next to you and it is a real-time voice thing, so it's, it's listening to its own voice and it needs to distinguish between its own voice-

    2. AFAlessio Fanelli45:46

       Right

    3. SWswyx45:46

       ... and between the, the human voice.

    4. AFAlessio Fanelli45:47

       Yep.

    5. SWswyx45:48

       And it needs to ignore its own voice. So we had opening engi- engineers tune that for our stage [laughs] to make this thing happen, which is absurd.

    6. AFAlessio Fanelli45:56

       Yeah.

    7. SWswyx45:56

       It was so funny, but also like, uh, you know, shout out to them for, for doing that for us and for the community, right? Uh, 'cause I, I think people wanted an update on voice.

    8. AFAlessio Fanelli46:05

       Mm-hmm. Yeah, they definitely do care about demos. Uh, not much to add there.

    9. SWswyx46:09

       Yeah.

    10. AFAlessio Fanelli46:09

        Llama 3 voice.

    11. SWswyx46:10

        Something that maybe is buried among all the Llama 3 news is that Llama 3 is supposed to be a multimodal model. It was delayed thanks to the European Union, apparently. I'm not sure what the, the whole story there is. I didn't really read that much about it. It is coming. You know, Llama 3 will be multimodal. It uses adapters rather than being natively multimodal. But I think that it's interesting to see the state of La-- of Meta AI research come together because there was this independent threads of Voice Box and Seamless Communication. These are all projects that Meta AI has launched that basically didn't really go anywhere because they were all one-offs. But now all that research is being pulled in into Llama. Like Llama is just subsuming all of FAIR, all of Meta AI into this thing. And, and yeah, you can see, uh, a Voice Box mentioned in Llama 3 voice adapter. I was kind of bearish on conformers because I looked at the state of existing conformer research in ICM, Clear and, and NeurIPS, and they were far, far, far behind Whisper, uh, mostly because of scale, like the, the sheer amount of resources that they're dedicated to. But Meta is, is approaching there. It's-- I think it's, um, they had 230 hours, 230,000 hours of speech recordings. I think Whisper is something like 600,000. So Meta just needs to 3X the budget on this thing and they'll do it, and, uh, we'll have open source voice. [laughs]

    12. AFAlessio Fanelli47:31

        Yeah. And then we can hopefully fine-tune on

15. 47:34 – 50:54

    Multimodality War: Meta Llama 3 multimodality + Chameleon

    1. AFAlessio Fanelli47:34

       our voice, and then we just need to write this episode instead of actually recording it.

    2. SWswyx47:38

       I should also shout out the other thing from Meta, which is a very, very big deal, which is Chameleon, which is a natively early fusion vision and language model. So most things are late fusion, basically. Like you freeze an existing language model, you freeze an existing vision transformer, and then you kind of fuse them with this, that thin adapter layer. That, that is what, uh, Llama 3 is also doing. But Chameleon is slightly different. Chameleon is interleaving in the same way that Idafix, uh, the, the sort of, um, dataset was, was doing. Interleaving natively for image generation and, uh, vision and, and, and text understanding. And I think like once that is better understood, that is going to be better. That, that is the more deep learning pilled version of this, uh, the more GPU Rich version of, of doing all this. I asked Yita this question about Chameleon in his, in his episode. He did not confirm or deny, but I think he, uh, he would agree that that is the right way to do multimodality. And now that we have-- we're proving out that multimodality is valuable to people, basically all these half-ass measures around adapters-

    3. AFAlessio Fanelli48:41

       Mm-hmm

    4. SWswyx48:41

       ... is going to flip to natively multimodal. To me, that's what GPT-4o represents. It is the train from scratch-

    5. AFAlessio Fanelli48:47

       Yep

    6. SWswyx48:47

       ... fully omni modal model whi- which is early fusion. So if you want to read that, you should read the Chameleon paper, basically. That's what is my, is my whole point.

    7. AFAlessio Fanelli48:56

       And there was some of the Chameleon drama because the open model doesn't have image generation.

    8. SWswyx49:02

       Yeah.

    9. AFAlessio Fanelli49:02

       And then there were, uh, fine-tuning recipes.

    10. SWswyx49:04

        It's so funny.

    11. AFAlessio Fanelli49:05

        And then the leads were like, "No, do not follow these instructions to fine-tune image generation."

    12. SWswyx49:10

        Yeah, yeah. Don't do that at all. Yeah, yeah. That's just really funny. I don't know what the, the... I, I-- okay, so yeah, whenever image generation is concerned, obviously because of the Gemini issue, you know, it's, it's very tricky for, for large companies to, to release that, but they can remove it, say that they re- point out exactly where they remove it and let the open source community put it back in. [laughs] The last piece I had, which I kind of deleted, was, uh, ju-just a special mention, honorable mention of Gemma again with PaliGemma, which is one of the smaller releases from Google I/O. I think you went, right? So PaliGemma was, was mentioned in there?

    13. AFAlessio Fanelli49:43

        Uh, I-

    14. SWswyx49:44

        I don't know. It was, it was one of the-

    15. AFAlessio Fanelli49:44

        Not, not on the... Yeah, yeah, one of the workshops, I think.

    16. SWswyx49:46

        Very, very small release.

    17. AFAlessio Fanelli49:47

        Yeah.

    18. SWswyx49:48

        Um, but, uh, CoPaliGemma now is being talked a lot about as a, as a late fusion model for extracting structured text out of PDFs. Very, very important for business work-

    19. AFAlessio Fanelli49:57

        Yeah, I know

    20. SWswyx49:58

        ... workhorses.

    21. AFAlessio Fanelli49:58

        Yes.

    22. SWswyx49:59

        Uh, so apparently it is doing better than Amazon Textract and all the other state-of-the-art, and it's just, it's a tiny, tiny model that does this, and it's really interesting. It's a combination of the Omar Khatab's, like, ColBERT sort of retrieval approach on top of a vision model, which I was severely underestimating PaliGemma when it came out, but, like, it's continuous to come up. Like, there's a lot of trends, and it-- again, this is making a lot of progress here, just in terms of their, their applications in, in real-world use cases. Like, these are small models, but they're very, very capable, and they're very good basis to build things like CoPaliGemma.

    23. AFAlessio Fanelli50:31

        Yeah, no, Google has been, it's been doing great. I think maybe a lot of people initially wrote them off, but between, um, you know, some of the Gemini Nano stuff, like, uh, Gemma 2, PaliGemma, we'll talk about some of the KV cache and context caching, um-

    24. SWswyx50:45

        Uh, yeah. Yeah, that's, that's a RAG wars, yeah

    25. AFAlessio Fanelli50:47

        ... for it to be building. So there's a, a lot to like, and our friend Logan is over there now, so he's excited about everything they got going

16. 50:54 – 52:55

    Multimodality War: PaliGemma + CoPaliGemma

    1. AFAlessio Fanelli50:54

       on, so yeah.

    2. SWswyx50:55

       I think there's a little bit of a fight between AI Studio and Vertex. And what Logan represents is, is-- So he's moved from DevRel to PM, and he was PM for the Gemma 2 launch. Vertex has this reputation of being extremely hard to use. It's one, one reason why GCP has-

    3. AFAlessio Fanelli51:10

       Yeah

    4. SWswyx51:10

       ... kind of fallen behind a little bit. And, and so AI Studio represents, like, the developer-friendly version of this, like the, the, the Netlify or Vercel to, to the AWS, right? And I think it's Google's chance to reinvent itself for this audience, for the AI engineer audience that doesn't want, like, five levels of auth IDs and org IDs and-

    5. AFAlessio Fanelli51:29

       Right. Yeah

    6. SWswyx51:29

       ... policy permissions just to get something going.

    7. AFAlessio Fanelli51:32

       True. True. Um, yeah, we wanna jump into RAG Ops Wars.

    8. SWswyx51:37

       What to say here? I, I think that what RAG Ops Wars are to me, like the, the tooling arounds the ecosystem, and I might need to actually rename this war, I think.

    9. AFAlessio Fanelli51:46

       War renaming alert.

    10. SWswyx51:48

        [chuckles]

    11. AFAlessio Fanelli51:48

        What are we calling it?

    12. SWswyx51:49

        Uh, LLMOs.

    13. AFAlessio Fanelli51:51

        LLMOs.

    14. SWswyx51:51

        Because it, it used to be when you're-- when, when the only job for AIs to do was chatbots.

    15. AFAlessio Fanelli51:58

        Mm-hmm.

    16. SWswyx51:58

        Then RAG matters-

    17. AFAlessio Fanelli52:00

        Yeah

    18. SWswyx52:00

        ... then Ops matters. But now we need AIs to also write code. We also need AIs to work with other agents, right? That's not reflected in any, any of the other wars. So I think that just the, the whole point is what does an LLM plug into with the, with the broader ecosystem to be more capable than an LLM can be on its own?

    19. AFAlessio Fanelli52:19

        Yep.

    20. SWswyx52:20

        I just announced it, but [laughs] yeah.

    21. AFAlessio Fanelli52:22

        Yeah, yeah.

    22. SWswyx52:22

        This is something I've been thinking about a lot. I-I... There's a blog post I've been working on. Uh, basically my tip to other people is if you wanna see where things are going, you go open up the ChatGPT, GPT Creator. Every single button on the GPT Creator is a potential startup.

    23. AFAlessio Fanelli52:34

        Mm-hmm.

    24. SWswyx52:35

        Um, EXA, uh, is, is for search, the knowledge RAG thing is for RAG.

    25. AFAlessio Fanelli52:40

        Yeah. We invested in E2B.

    26. SWswyx52:42

        Yeah, congrats. Is that announced? I don't know if [laughs]

    27. AFAlessio Fanelli52:44

        Oh, it's announced now.

    28. SWswyx52:44

        It's announced now.

    29. AFAlessio Fanelli52:45

        By the time this goes out, it'll be.

    30. SWswyx52:46

        What, uh, briefly, what is E2B?
17. 52:55 – 55:31

    Renaming Rag/Ops War to LLM OS War

    1. AFAlessio Fanelli52:55

       Mistral to add that in. They have this open source, uh, Claude Artifacts clone using E2B. It's a-- I mean, the amount of, like, traction that they've been getting in open source has been amazing. I think they went in, like, four months from, like, 10K to a million containers spun up, um, on the cloud. So I mean, you told me this maybe, like, nine months ago, 12 months ago-

    2. SWswyx53:15

       Yeah

    3. AFAlessio Fanelli53:15

       ... something like that. Uh, you were like, uh, what you literally just said, every ChatGPT plugin can be-

    4. SWswyx53:21

       A business, a startup

    5. AFAlessio Fanelli53:21

       ... or, like, uh, can be a business startup.

    6. SWswyx53:22

       Yeah.

    7. AFAlessio Fanelli53:23

       Um, and I think now it's more clear than ever than the chatbots are just kinda like the, uh, band-aid solution-

    8. SWswyx53:31

       Yeah

    9. AFAlessio Fanelli53:31

       ... you know, before we build more, more comprehensive systems and, um, yeah, EXA just raised a Series A, uh, from Lightspeed, so, um-

    10. SWswyx53:38

        I tried to get you in on that one as well.

    11. AFAlessio Fanelli53:40

        Yeah, yeah, no, I read that.

    12. SWswyx53:41

        [laughs] I'm trying to be a scout, man. I don't know. [laughs]

    13. AFAlessio Fanelli53:45

        Um, so yeah, this is a giving as a VC, early-stage VC, like, giving capabilities to the models is, like, way more important than the actual LLM ops, you know, the observability and, like, all these things. Like, those are nice, but, like, the way you build real value for a lot of the customers is, like, how can this model do more than just chat with me? So running code, doing analysis, doing web search.

    14. SWswyx54:10

        Hmm. I might disagree with you. I, I, I think it's com-- They're all valuable. They're all valuable.

    15. AFAlessio Fanelli54:15

        Yeah, well-

    16. SWswyx54:16

        They're all valuable. So I will disagree with you just on, like, it, it, uh, I find ops my number one problem right now building Smalltalk. [laughs] Building AI news, building anything I do, and I, I don't think I'm happy with all the ops solutions I've explored.

    17. AFAlessio Fanelli54:28

        Mm-hmm.

    18. SWswyx54:29

        There are some 80-something ops startups.

    19. AFAlessio Fanelli54:31

        Right.

    20. SWswyx54:31

        [laughs] I nearly, you know, started one of them, but we'll briefly talk about this ops thing-

    21. AFAlessio Fanelli54:36

        Yeah, yeah

    22. SWswyx54:36

        ... then we'll go back to, go back to RAG. The central way I, I explain this thing to people is that all the model labs view their job as stopping by serving you their model over an API, right? That is unfortunately not everything that you need-

    23. AFAlessio Fanelli54:48

        Mm-hmm

    24. SWswyx54:48

        ... in order to productionize this API. So obviously, there's all these startups. They're like, "Yeah, we are ops guys. We've, we've done this for 30 years. Uh, we will now do this for AI." And the, the 80 of them show up, and they all raise money. And the, the question is, like, what do you actually need as, as, like, sort of an AI-native ops layer versus what did you just plug into Datadog?

    25. AFAlessio Fanelli55:08

        Mm-hmm.

    26. SWswyx55:09

        Right? Uh, I don't know if you ever, you have dealt with that because I'm not, like, a super ops person, but I appreciate the importance of this thing. I think there's three broad categories, which is frameworks, gateways, and monitoring or tracing. We've talked to, like, uh, I interviewed Human- Humanloop in London, and after-- You've talked to a fair share of them. I've talked to a fair share of them.

    27. AFAlessio Fanelli55:29

        Yep.

    28. SWswyx55:29

        So the frameworks would be, uh, honestly,

18. 55:31 – 1:02:57

    LLM OS War: Ops War: Prompt Management vs Gateway vs Observability

    1. SWswyx55:31

       I won't name the startup But basically what this frame company was doing was charging me forty-nine dollars a month to store my prompt template, and every time I make an inference, it would F string-

    2. AFAlessio Fanelli55:42

       Mm-hmm

    3. SWswyx55:42

       ... call the prompt template on some variables that I supply, and it's charging forty-nine dollars a month for unlimited storage of that.

    4. AFAlessio Fanelli55:49

       Yeah.

    5. SWswyx55:49

       It's absurd, but, like, people want prompt management tools. They want to interoperate between PM and developer. There's some value there. I don't know what the right price is.

    6. AFAlessio Fanelli55:59

       Yeah, yeah.

    7. SWswyx55:59

       There's some price.

    8. AFAlessio Fanelli56:00

       Well, I was at... I'm sure I can share this. I was at the Grab office-

    9. SWswyx56:04

       Mm-hmm

    10. AFAlessio Fanelli56:04

        ... um, and they also treat prompts as code, but they build their own thing to then import prompts and-

    11. SWswyx56:08

        Yeah, I wanna check prompts into my code base as a developer, right? But maybe do, do you want it outside of the code base?

    12. AFAlessio Fanelli56:13

        Well, I think it's like how do you... Well, you can have it in the code base, but, like, what's like the prompt file? What's like, you know, it's not just a string. How do you better-

    13. SWswyx56:21

        There's string and model-

    14. AFAlessio Fanelli56:22

        How do... Yeah

    15. SWswyx56:22

        ... and config.

    16. AFAlessio Fanelli56:24

        Exactly. How do you pass these things? Uh, but I think, like, the problem with building frameworks is, like, frameworks generalize things that we know work, and like right now we don't really know what works. So-

    17. SWswyx56:35

        Yeah, but some people have to try. You know, and the whole point of early stage is you try it before you know it works. [chuckles]

    18. AFAlessio Fanelli56:39

        Yeah, but I think like the, the past, if you see the most successful open source frameworks that became successful businesses are frameworks that were built inside companies and then were kinda spun out-

    19. SWswyx56:50

        Yeah, yeah. Okay

    20. AFAlessio Fanelli56:51

        ... uh, as projects. So, uh, I think it's more of ordering.

    21. SWswyx56:53

        So we're more of a vertical, vertical build instead of horizontal build. [chuckles]

    22. AFAlessio Fanelli56:57

        I mean, we try to be horizontal build, right? And it's like, where are all the horizontal startups? So [chuckles] that's it.

    23. SWswyx57:01

        There are a lot of them, they're just not that... They're not gonna win-

    24. AFAlessio Fanelli57:05

        [laughs] Well, my-

    25. SWswyx57:06

        ... uh, by, by, by themselves. By themselves. They-

    26. AFAlessio Fanelli57:08

        Yes.

    27. SWswyx57:08

        I think some of them will win by sheer excellent execution and, and then, but, like, the market won't pull them. They will have to pull the market.

    28. AFAlessio Fanelli57:16

        Yeah. Well, but that's the thing, it's like, uh, you know, take like Julius, right?

    29. SWswyx57:20

        Yeah.

    30. AFAlessio Fanelli57:20

        It's like, hey, why are you guys doing Julius like the same as Code Interpreter? And yet they're pretty successful. A lot of people use it because they're like solving a problem, and then, uh-
19. 1:02:57 – 1:06:15

    LLM OS War: BM42 Vector DB Wars, Memory Databases, GraphRAG

    1. SWswyx1:02:57

       is memory. Long conversation memory. I have on me this, um, uh, Bee, which is one of the personal AI wearables. I'm also getting the Limitless, uh, personal AI wearable, which is like I j- I just want it to record my whole conversation and just repeat back to me or let me, let me find, m- augment my memory. As a, as a-- I'm sure Character AI has some version of this.

    2. AFAlessio Fanelli1:03:17

       Yeah.

    3. SWswyx1:03:17

       Like, everyone has conversation memory that is different from factual memory. And right now, vector database is very oriented towards factual memory, document retrieval, knowledge-based retrieval. But it's not the same thing as conversation retrieval, where I need to know what I've said to you, what I said to you yesterday, what I said to you a year ago or three years ago. And that's, that's a different nature of retrieval, right? Um, so there's a, a... At, at the conference that we ran, GraphRAG was a, was a lot of, uh, focus for people-

    4. AFAlessio Fanelli1:03:42

       Mm-hmm

    5. SWswyx1:03:42

       ... the marriage of knowledge graphs and RAG. I think that this is commonly a trap in ML, that people are like, they discover that graphs are a thing for the first time. They're like, "Oh yeah, everything's a graph. Like, the future is graphs," and then nothing happens. Very, very common. This happened like three, four times in, in the industry's past as well. But maybe this time is different. [laughs]

    6. AFAlessio Fanelli1:04:00

       Maybe. [laughs] Unless.

    7. SWswyx1:04:02

       Unless. Unless. [laughs] So this is a fun-- This is why I'm not an investor. Like, you have to get the time-

    8. AFAlessio Fanelli1:04:10

       Yeah, yeah

    9. SWswyx1:04:10

       ... the, the this time is different. Because no ideas are really truly new.

    10. AFAlessio Fanelli1:04:13

        Mm-hmm.

    11. SWswyx1:04:14

        But some-sometimes this time is different. [laughs]

    12. AFAlessio Fanelli1:04:18

        Maybe. Maybe.

    13. SWswyx1:04:18

        And so memory databases are one form of that, where, like, they're focused on the problem of long-form memory for, uh, for agents, for assistants, for chatbots and all that.

    14. AFAlessio Fanelli1:04:28

        Yeah.

    15. SWswyx1:04:28

        I, I, I definitely see that coming. There were some funding rounds that I can't really talk about in this sector, and I've seen that happen a lot. Um, yeah, I have one more category in LLMs, but, uh, any comments on memory-

    16. AFAlessio Fanelli1:04:37

        Yeah, no, I, I think that makes sense to me that moving away from just semantic similarity I think is the most important because people use the same word with very different meanings, especially when talking, you know. When writing it's different, but yeah.

    17. SWswyx1:04:50

        Yeah. The other direction that vector databases have gone into, which, um, LanceDB presented at my conference, was multimodality. So, uh, Lance, uh, Character AI uses LanceDB for multimodal embeddings. That's just a minor difference. I don't think that's like a quantum leap in terms of what a vector database does for you. The other thing that I see in LLMs world is, is mostly, um, the evolution of like, uh, just the, the ecosystem of agents, right? The, the, the agents talking to other agents and coordinating with other agents. So I interviewed Graham Newbig at, uh, I-Iclea, and he since announced that they are pivoting open-OpenDevin or broadening OpenDevin into All Hands AI. I'm not sure about that name, but, uh, it, it is the... It is one of the three LLMOS startups that got funded in the past two months, uh, that I know about, and maybe you know more. They're, they're all building, like, this ecosystem of agents talk- working with other agents and all this, all this tooling, uh, for, for agents. To me, makes more sense. It, it is probably the biggest thing I missed in, in doing the Four Wars, the need for startups to build this ecosystem thing up, right? So the big categories have been taken. Search, done. Code interpreter, done. There's a long tail of others, right? So memory is emerging, then there's like other stuff.

    18. AFAlessio Fanelli1:06:00

        Mm-hmm.

    19. SWswyx1:06:00

        Uh, and so they're, they're focusing on that. Like, so to me, uh, browser is slightly different from search, and BrowserBase is, is another company I invested in that is focused on that, but they're not the only one in that category by, by, by any means. I used to tell people, "Go to the Devin demo and

20. 1:06:15 – 1:08:26

    LLM OS War: Agent Tooling

    1. SWswyx1:06:15

       look at the four things that they offer," and just say, "Each of the, each of those things is a startup."

    2. AFAlessio Fanelli1:06:18

       Mm-hmm.

    3. SWswyx1:06:18

       Devin, since then, they spoke at the conference as well. Scott, uh, was super nice to me and then, uh, actually gave me some personal time as well. They have an updated chart of their plans. You look at their plans, they have like sixteen things. Each of those things is a potential startup now.

    4. AFAlessio Fanelli1:06:30

       Mm-hmm.

    5. SWswyx1:06:31

       And that is the LLMOS. Everyone's building towards that direction because they need it to do what they need to do as a, as an agent. If you believe in the agent's future, you need all these things.

    6. AFAlessio Fanelli1:06:40

       Yeah. You think the agent OS is its own company? Do you think it's an open standard? Do you think-

    7. SWswyx1:06:48

       I would love it to be open standard. The reality is that people want to own that standard. So we have-- We actually wound down the AI Engineer Foundation w-where the, where the first project was the Agent Protocol, which E2B actually donated to the foundation, because no one's interested. Everyone wants to be VC-backed-

    8. AFAlessio Fanelli1:07:04

       Right

    9. SWswyx1:07:04

       ... so then they wanna own it, right? So there's just-- It's too early to be open source. This-- The-- People will keep this proprietary, and more power to them. They need to make it work. They need to make revenue before all the other stuff can happen.

    10. AFAlessio Fanelli1:07:14

        Yeah. I'm really curious. You know, we're investors in a bunch of agent companies. None of them really care about how to communicate with other agents. They're so focused internally, you know. But I think in the future, you know, I talked about this on-

    11. SWswyx1:07:27

        I see. You're, you're talking about agent to a other external agents.

    12. AFAlessio Fanelli1:07:29

        Yeah. So I think-

    13. SWswyx1:07:30

        I'm not talking about that yet

    14. AFAlessio Fanelli1:07:31

        ... yeah, I wonder when, like... Because that's where the future is going, right? So today it's like intra-agent-

    15. SWswyx1:07:37

        I see, I see

    16. AFAlessio Fanelli1:07:37

        ... connectivity. You know? At some point it's like, well, it's not like somebody-- I'm selling into a company and the company already uses Agent X For that job, I need to talk to that agent, you know? But I think nobody really cares about that today, so I think that's usually-

    17. SWswyx1:07:51

        Yeah. So, uh, I think that that layer right now is OpenAPI. Just-

    18. AFAlessio Fanelli1:07:56

        Yeah, yeah, yeah. Exactly

    19. SWswyx1:07:56

        ... give me a RESTful protocol-

    20. AFAlessio Fanelli1:07:57

        Yep

    21. SWswyx1:07:57

        ... I can, I can interoperate with that. RESTful protocol only does request response, so then the next layer is something I have worked on, which is long-running request response, which is workflows, which is what Temporal was supposed to do before, let's just say, management issues. Yeah, but like, you know, RPC or some kind of... You know, I, I think the, the dream is, uh, and this is one of the, the, my problems with the LLM OS concept is that do we really need to rewrite every single thing for AI native use cases? Shouldn't the AI

21. 1:08:26 – 1:10:43

    LLM OS War: Agent Protocols

    1. SWswyx1:08:26

       just use the, these things, these tools the same way as humans use them? The reality is for now, yes, they, they, they need specialized APIs. In fu- in the distant future when these things cost nothing, then they-

    2. AFAlessio Fanelli1:08:37

       Right. Yeah, yeah, yeah

    3. SWswyx1:08:37

       ... can use it the same way as humans does, but right now they need specialized interfaces. The layer between agents ideally should just be English, you know, like the same way that we, we talk. But like it- English is too underspecified-

    4. AFAlessio Fanelli1:08:50

       Yeah, yeah, yeah

    5. SWswyx1:08:50

       ... unstructured to, to make that happen, so.

    6. AFAlessio Fanelli1:08:52

       Yeah.

    7. SWswyx1:08:52

       So, yeah.

    8. AFAlessio Fanelli1:08:53

       It's interesting because we talk to each other in English, but then we both use tools to do things-

    9. SWswyx1:08:58

       Yes

    10. AFAlessio Fanelli1:08:58

        ... to then get the response back.

    11. SWswyx1:09:00

        For those people who want to dive in a little bit more, I think AutoGen, I would, I would definitely recommend, uh, looking at that. CrewAI. There are established frameworks now that are working on inter agent communication layers to coordinate them, and not necessarily externally from company to company-

    12. AFAlessio Fanelli1:09:15

        Yeah, yeah

    13. SWswyx1:09:15

        ... just internally as well.

    14. AFAlessio Fanelli1:09:16

        Mm-hmm.

    15. SWswyx1:09:16

        Between if you have multiple agents farming out work to do, do different things, you're gonna need this anyway. And, uh, I don't think it's that hard. It's, it's they are using English, they're using s- some, some mix of English and structured output. And yeah, if you have a better idea than that, let us know.

    16. AFAlessio Fanelli1:09:30

        [laughs] Yeah. We're listening.

    17. SWswyx1:09:32

        [laughs] So that's the four wars discussion. I think I want to leave some discussion time open for miscellaneous trends that are happening in the industry that don't exactly fit in the four wars or, or are a layer above the four wars. So the first one to me is just this trend of open source. Obviously this overlaps a lot with the GPU poor thing, but I want to really call out this depreciation thing that I've been working on. Like, I, I, I do think it's probably one of the, the bigger thesis that I've come, that I've had in the past month, which is that we now have a rough idea of the dep- depreciation schedule of, uh, this, this sort of model spend. And, yeah, I, I basically drew a chart. I'll link it in the show notes, but I drew, drew a chart of the price efficiency frontier of, as of March, April 2024, and then I had listed all the models that list that sit within that frontier. Haiku was, was the best cost per intelligence at that point in time. And then I did the same chart in July, two days ago, and the whole thing has moved, and Mistral is like deprecating their old models that, that used to be in the old frontier. It is so shocking how predictive and tight this band is. Very, very tight band,

22. 1:10:43 – 1:16:45

    Trend: Commoditization of Intelligence

    1. SWswyx1:10:43

       and the whole industry is moving the same way. And it's roughly one order of magnitude drop in cost for the same level of intelligence every four months. My previous number for this was one order of magnitude drop in cost every 12 months.

    2. AFAlessio Fanelli1:10:55

       Mm-hmm.

    3. SWswyx1:10:56

       But the timeline has accelerated because, uh, GPT-3 took about a year-

    4. AFAlessio Fanelli1:11:00

       Yeah

    5. SWswyx1:11:00

       ... to, to drop, uh, order of magnitude. But now GPT-4, it's really crazy. [laughs] I don't know what to say about that.

    6. AFAlessio Fanelli1:11:06

       Yeah.

    7. SWswyx1:11:06

       But I just wanna-

    8. AFAlessio Fanelli1:11:06

       Do, do you think GPT Next and Claude 4 push it back down because they're coming out with higher intelligence, higher cost? Or is it maybe like the-

    9. SWswyx1:11:17

       It, it should-

    10. AFAlessio Fanelli1:11:17

        ... the timeline's going down because new frontier models are not really coming out at the same rate?

    11. SWswyx1:11:22

        Interesting. I don't know. That's a, that's a really good question. Wow. I'm stumped.

    12. AFAlessio Fanelli1:11:26

        [laughs]

    13. SWswyx1:11:26

        I, I don't have-

    14. AFAlessio Fanelli1:11:27

        You're like, "Wow, you got a good question on this."

    15. SWswyx1:11:28

        Yeah, I don't have a, I don't, I don't, I don't have an answer. No, and I mean-

    16. AFAlessio Fanelli1:11:30

        Yeah

    17. SWswyx1:11:30

        ... you have a lot of good questions, but like, uh, I thought I had solved this, and then now you came along with it. The first response is something I haven't thought about. Uh, yeah. Yeah, so there's, there's two directions here, right? When, when the cost of frontier of models are going up, potentially, like SB 1047 is gonna-

    18. AFAlessio Fanelli1:11:44

        Right. Yeah, yeah, yeah

    19. SWswyx1:11:44

        ... make it illegal to train even larger models for as well. I, I, I think the opposition has increased enough that it's not gonna be a real-

    20. AFAlessio Fanelli1:11:51

        Mm-hmm

    21. SWswyx1:11:51

        ... real concern for people. But I think every lab basically needs a small, medium, large play, and like we said in the, the sort of model deployment framework, first you choose, you pursue capability, then you pursue generalization, then you pursue efficiency. And, and what we're talking about here is, is efficiency.

    22. AFAlessio Fanelli1:12:07

        Yeah, yeah.

    23. SWswyx1:12:07

        Right? Like that now we care about efficiency. The, this-- definitely one of the emergent stories of the year that has happened is efficiency matters for 4o, 4o Mini and 3.5 Sonnet in a way that in January we would... nobody was talking about.

    24. AFAlessio Fanelli1:12:20

        Mm-hmm.

    25. SWswyx1:12:21

        And that's great.

    26. AFAlessio Fanelli1:12:22

        Yeah.

    27. SWswyx1:12:23

        Regardless of GPT Next and Claude 4 or whatever, or Gemini 2, we will still have efficiency frontiers to, to pursue. And it seems like doing the higher capable thing creates the synthetic data for us to do the efficiency, efficient thing, and that means lifting up the... Like, I, I had this difference chart between Llama 3.0 8B, Llama 3.0 70B, versus their 3.1 differences-

    28. AFAlessio Fanelli1:12:48

        Mm-hmm

    29. SWswyx1:12:48

        ... and the 8B had the most, uh, uplift across all the benchmarks. Right? It makes sense. You're, you're training from the 4 or 5B. You're distilling from there, and, and it's gonna have the, the biggest lift up. So the best way to train more efficient models is to train the large model. [laughs]

    30. AFAlessio Fanelli1:13:00

        Right. Yeah, yeah.
23. 1:16:45 – 1:20:44

    Trend: Vertical Service as Software, AI Employees, Brightwave, Dropzone

    1. AFAlessio Fanelli1:16:45

       they charge a lot of money, more than you would expect because hedge funds and, like, investment banking, investment advisors, they're used to paying a lot of money for research. It's like the labor. They don't even care that you use AI. They just want labor to be done.

    2. SWswyx1:16:57

       I'll mention one, one pushback. But, uh-

    3. AFAlessio Fanelli1:16:58

       Yeah, yeah

    4. SWswyx1:16:59

       ... as, as a hedge fund, we used to pay for analyst research out of our brokerage cost-

    5. AFAlessio Fanelli1:17:03

       Mm-hmm

    6. SWswyx1:17:03

       ... and not read them. And that, that-- to me, that's, that's my-

    7. AFAlessio Fanelli1:17:06

       Yeah

    8. SWswyx1:17:06

       ... risk of, of Brightwave. But, but, you know, I-

    9. AFAlessio Fanelli1:17:08

       No, but I, I think the-

    10. SWswyx1:17:09

        As a consumer of research, I'm like-

    11. AFAlessio Fanelli1:17:11

        Well, if, if we wanna go down the-

    12. SWswyx1:17:12

        [laughs]

    13. AFAlessio Fanelli1:17:12

        ... the rabbit hole, there's a lot of pressure on funds for, like, a OPEX efficiency, so th-there's not really capture, uh, researchers anymore in most funds, and, like, even the sellside research is like-

    14. SWswyx1:17:22

        I see. So, so-

    15. AFAlessio Fanelli1:17:23

        ... not that good

    16. SWswyx1:17:23

        ... taking them, uh, from in-house to external thing.

    17. AFAlessio Fanelli1:17:25

        Yeah.

    18. SWswyx1:17:26

        Yeah, yeah.

    19. AFAlessio Fanelli1:17:26

        Exactly.

    20. SWswyx1:17:26

        That makes sense. Yeah.

    21. AFAlessio Fanelli1:17:27

        Um, so yeah, we s- you know, we had Dropzone that does security a-analysis. Same, people are used to paying for managed security or, like, outsource SOC analysts. They don't wanna buy a AI tool to make the security team more productive. So, um-

    22. SWswyx1:17:39

        Okay. And what specifically does Dropzone do?

    23. AFAlessio Fanelli1:17:41

        They do, uh, SOC analysis. So not SOC like the compliance, but it's like when you have security alerts, how do you investigate them? So large enterprises, they get like thousands of phishing email, and then they forward them to IT, and it's IT or security person, the tier zero, has to go in and say, "That's a phishing email. That isn't. That isn't." So they have an agent that does that. So the cost to do-- like, for a human to do the analysis at the rate that they get paid, that's like $35 per alert. Dropzone is like $6 per alert.

    24. SWswyx1:18:10

        Yeah.

    25. AFAlessio Fanelli1:18:10

        So it's, it's a very basic economic analysis for the company whether or not they wanna buy it. It's not about, "Is my an-analyst gonna have more free time?" Like, "Is it more productive?" So selling the labor is, like, the story of the market right now. Um-

    26. SWswyx1:18:24

        My version of this is I should start a consulting services today-

    27. AFAlessio Fanelli1:18:27

        Mm-hmm

    28. SWswyx1:18:27

        ... and then slowly automate myself, my, my employees out of a job, right?

    29. AFAlessio Fanelli1:18:32

        Mm-hmm.

    30. SWswyx1:18:33

        Is that fundable?
24. 1:20:44 – 1:23:31

    Trend: Benchmark Frontiers after MMLU

    1. SWswyx1:20:44

       you have function calling. Here, I think Gorilla, API Bank next is, uh, pretty, pretty consensus. I, I've got nothing there apart from, yeah, like all models need, need something like this. Vision now is like multi- like multimodality the, the vision is the most important. Um, I think like ViBEV is actually the, the state-of-the-art here. I, you know, open to, open to being corrected. And then multilinguality. So basically, like these are the 10 directions, right? Uh, post-MMLU, here are the frontier capabilities. If you're developing models or if you're, if you're encountering a new model, evaluate them on all these elements, and then you have a good sense of how state-of-the-art they are and what you need them for in terms of applying them to your use case. So I just want to get that out there.

    2. AFAlessio Fanelli1:21:19

       Yeah. And we had the ARC-AGI thing. How do you think about benchmarking for, you know, everyday thing or like benchmarking for something that is maybe like a hard to reach goal?

    3. SWswyx1:21:30

       Yeah. This has been a debate, uh, for, uh... That's obviously very important and probably more important for product usage, right? Here, I'm talking about benchmarking for general model evals and then there's a, there's a schism in the AI engineering community or criticism in the AI engineering community that it did not care about, enough about product evals. So Hamul Hussein, uh, led that and I, I had a bit of disagreement with him, but I, I acknowledge that. I, I think that is important. There was an oversight in my original AI engineer post. Uh, so the job of the AI engineer is to produce product specific evals for your use case and there's no way that these general academic benchmarks are going to do that because they don't know your use case.

    4. AFAlessio Fanelli1:22:06

       Yeah.

    5. SWswyx1:22:06

       It's not important. They will correlate with your use case and that is a good sign, right? These are very, very rigorous and thought through. So you want to look for correlates, then you want to look for specifics and that's something that only you can do. So yeah, ARC-AGI will correlate with IQ. It's, it's an IQ test.

    6. AFAlessio Fanelli1:22:21

       Yeah. Yeah. Yeah.

    7. SWswyx1:22:22

       Right? How well does IQ tests correlate to job performance? 5%, 10%.

    8. AFAlessio Fanelli1:22:26

       Mm-hmm.

    9. SWswyx1:22:27

       Not nothing, but not everything.

    10. AFAlessio Fanelli1:22:29

        Yeah, yeah.

    11. SWswyx1:22:29

        And so it's important.

    12. AFAlessio Fanelli1:22:30

        Anything else?

    13. SWswyx1:22:31

        Super intelligence. [laughs] We, we can... You know, we, we try not to talk about safety. Uh, my, my favorite safety joke from our dinner is that, you know, if you're worried about agents taking over the world and you need a button to take them down, just install CrowdStrike-

    14. AFAlessio Fanelli1:22:42

        Yeah [laughs]

    15. SWswyx1:22:43

        ... on all, every agent and you have a button that has just been proved at the largest scale in the world to disable all agents, right? So that safe super intelligence, you should just install CrowdStrike. [laughs] That's what AI subscribers should do. [laughs]

    16. AFAlessio Fanelli1:22:56

        That's funny, except for the CrowdStrike people. Awesome, man. This was great. I'm glad we did it. I'm sure we'll do it more regularly-

    17. SWswyx1:23:03

        We should do more

    18. AFAlessio Fanelli1:23:03

        ... now that you're out of visa jail.

    19. SWswyx1:23:05

        Yeah. Yeah. I, I think, uh, you know, AI News is surprisingly helpful for doing this. Like, uh-

    20. AFAlessio Fanelli1:23:09

        Yeah

    21. SWswyx1:23:10

        ... yeah. I, I, I had no idea when I started. I just, I just thought I, I needed a thing to summarize Discords, but now it's becoming a proper media company. Like, yeah, 1,000 people sign up every month. Uh, it's growing.

    22. AFAlessio Fanelli1:23:21

        Cool. Thank you all for listening.

    23. SWswyx1:23:23

        Yeah.

    24. AFAlessio Fanelli1:23:24

        See you next time.

    25. SWswyx1:23:24

        Bye. [outro music]

Episode duration: 1:23:35