Skip to content
The Twenty Minute VCThe Twenty Minute VC

Arena CEO: There Will be a $100BN US Open-Source Model & Data is a Trillion Dollar Market

Anastasios Angelopoulos is the co-founder and CEO of Arena, the real-world evaluation platform that has become a leading referee of the global AI model race. Arena has raised $250 million, with the latest round valuing the company at $1.7BN. Arena recently surpassed $100M ARR just eight months after launching its enterprise offering, powered by more than 30 million monthly users. ----------------------------------------------- Timestamps: 00:00 Intro 01:08 What Is Arena and Why Does It Matter? 02:10 Are AI Models Commoditising? The Open Source Tipping Point 03:45 Kimi K3 Beats All American Models 05:11 OpenRouter Metrics Are Misleading 06:21 Enterprise AI Sovereignty: Why Companies Will Want to Own Their Own Models 08:47 The US Must Build a Great American Open Source Model 11:28 How to Evaluate the 75+ Neo Labs 13:19 Thinking Machines Deep Dive: Is Inkling Too Little Too Late? 19:06 China's AI Advantage 21:01 Should the US Restrict Chip Exports to China? 31:35 The OpenAI Hugging Face Hack 33:10 We Need Guardian Models: AI Watching Over AI 35:01 AI-Powered Fake Candidates Are Getting Through Arena's Hiring Process 38:15 Hiring Research Talent in the Bay 39:39 What Determines Neo Lab Winners vs Flame-Outs 43:10 Why Round Two Kills Most AI Companies 50:07 Arena's Agent Evaluation Platform: 30M Monthly Visitors & Why It Matters 52:32 Arena Past $100M ARR 55:34 Will Frontier Model Providers Kill Harvey & Legora? 59:14 Will Salesforce Thrive or Die in the AI Era? 01:00:24 Quick-Fire Round ---------------------------------------------------------------------------------------------- Subscribe on Spotify: https://open.spotify.com/show/3j2KMcZTtgTNBKwtZBMHvl?si=85bc9196860e4466 Subscribe on Apple Podcasts: https://podcasts.apple.com/us/podcast/the-twenty-minute-vc-20vc-venture-capital-startup/id958230465 Follow Harry Stebbings on X: https://twitter.com/HarryStebbings Follow Anastasios Angelopoulos on X: https://twitter.com/ml_angelopoulos Follow 20VC on Instagram: https://www.instagram.com/20vchq Follow 20VC on TikTok: https://www.tiktok.com/@20vc_tok Visit our Website: https://www.20vc.com Subscribe to our Newsletter: https://www.thetwentyminutevc.com/contact ----------------------------------------------- #20vc #harrystebbings #founder #entrepreneur #arena #ai #opensource #neolabs

Anastasios AngelopoulosguestHarry Stebbingshost
Aug 3, 20261h 9mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 1:08

    Intro

    1. AA

      I believe that we're going to have at least one massive multi-hundred billion, if not trillion-dollar American company focused on American-first open source. What happened is that Kimi actually beat all American models, including Fable, in some subset of tasks. That doesn't mean that they're not distilling.

    2. HS

      Anastasios is the founder and CEO of Arena. It allows you to vote on the best models. It's an unbelievable model evaluator, and this turned out to be one of the most fun shows I have done literally in recent memory.

    3. AA

      The idea that we should have a central government body that tells us when it's time to release a new product versus not is crazy to me. This is gonna be so fucking insane what happens with, like, the cyberattacks. There's at least 75 Neo labs. For sure, like, two-thirds of those are gonna be worth nothing.

    4. HS

      Ready to go? [upbeat music] Anastasios, this is gonna be a lot of fun for me because I'm dumb as rocks, and you're gonna teach me a whole load of stuff today. So thank you so much for joining me, dude.

    5. AA

      Oh, no. Thank you for

  2. 1:08 – 2:10

    What Is Arena and Why Does It Matter?

    1. AA

      having me.

    2. HS

      Dude, I told you I use this as a chance to catch up with old friends, so, uh, it was wonderful stalking you for the last few days. I just wanna start, for anyone that doesn't know, can you explain to me very succinctly and easily what is Arena and why is it important and gaining notoriety today?

    3. AA

      Well, Arena is the platform for measuring AI performance in the real world. So what that means is that we're not using static benchmarks, we're not using some random data set that somebody collected, but rather what happens when you put AI in the hands of real people. And in so doing, we're measuring the objective reality of h- how AI affects humanity, whether it's factual, whether it's steerable, whether humans prefer it or disprefer it, whether it's hallucinating, whether there's errors, whether people are getting their actual jobs done with AI in reality. And then we're helping labs improve their models. We're helping the ecosystem understand the performance of different, you know, different AIs and keep track of all the amazing breaking news, all the new models, multiple models being released every week. So that's sort of the story of Arena. We're the central evaluation platform of AI.

  3. 2:10 – 3:45

    Are AI Models Commoditising? The Open Source Tipping Point

    1. HS

      Well, that was incredibly succinct. Thank you. Normally, people take about four hours after I ask for a succinct description. When you look at the models that you have on Arena, the sheer number of them, bluntly, I just am faced with the one question, holy shit.

    2. AA

      Yeah.

    3. HS

      Is this, like, the true commoditization of models? Are they just a complete utility layer at this point?

    4. AA

      Well, I think that there's, uh ... The, the big question around this has started to rise because of open source models. So I think if you were to only look at the closed source models, you would say there's acceleration, but it hasn't quite commoditized yet because that layer is still owned by a s- pretty small group of companies. It would be an oligopoly if we only had the closed source models. But what seems to be happening is that the open source models, especially from China, have really rapidly improved. And for the first time ever, we saw a couple weeks ago that Kimi K3 actually beat the best closed source American models, uh, on a, you know, pretty important subset of tasks. For example, front-end, front-end coding, like web development, which a, a huge fraction of developers are web developers.

    5. HS

      Dude, can I ask, how big a moment was that? Because it's like a, I'm gonna butcher it, but you know I'm a podcaster so I can get away with it. You're a PhD master. You can't. Uh, it's like a 27 trillion parameter model. Um, it's pretty clunky. This is not an agile model. And actually, you know, I was with Jason Lamkin yesterday from Sasta, who's as AI pilled as they can be, and he's like, "Honestly, it's not better than the others." How big a moment is Kimi?

  4. 3:45 – 5:11

    Kimi K3 Beats All American Models

    1. AA

      No, it was a pretty big moment. It was a pretty big moment. And the reason I'd say it was a big moment is because it violates a narrative that has been persistent in the United States, which is that the Chinese are just distilling American models, and that's the only way that they're able to s- you know, keep up. When really what happened is that Kimi actually beat all American models, including Fable, in some subset of tasks. That doesn't mean that they're not distilling. They may still be using distillation as a sub-step in their training procedure, but it does mean that distillation is only part of the story, and that there's something that those labs are doing above and beyond distillation that's bringing the performance up above what the American labs, uh, are, are currently doing. And so that narrative violation has been hugely important to the way that people view the ecosystem, both from the scientific dominance of Americans and the American sort of hegemony, uh, which of course Americans love hegemony, uh, to, um, the, uh, economics of the whole thing. And to, to your point, are these things, are these models a commodity or not?

    2. HS

      When we see, you know, w- when we look at your OpenRouters of the world, you know, the top five models are all open source Chinese models. When we see the proliferation of Chinese models today, does that cannibalize the closed frontier model business meaningfully?

  5. 5:11 – 6:21

    OpenRouter Metrics Are Misleading

    1. AA

      Well, I think that you need to think about the, um, the incentives and economics behind it. So first thing I'll say is that the OpenRouter metrics are not truly reflective of reality, and that's because the business rout- the business model of OpenRouter is to charge, uh, like, a fraction, uh, uh, like, a fee on top of every token. And so what happens is that people don't use OpenRouter for proprietary models. People are using OpenRouter primarily for open source models where they need the failover and all the value added services that OpenRouter provides. If you look at the whole space of all inference, most of it is still being consumed on first party APIs and on proprietary models. That's why Anthropic revenue has been just a total hockey stick. It's not, you know, it's not like they're being completely cannibalized right now by Chinese open source models. These models are still only a small fraction of the total inference spend in the world. That said Think about what's happening in the future. W- enterprises are gonna wanna own their own intelligence. They're gonna want so-called AI sovereignty, which is a fancy word for meaning that you own your whole supply chain of

  6. 6:21 – 8:47

    Enterprise AI Sovereignty: Why Companies Will Want to Own Their Own Models

    1. AA

      AI. That means you can take an open source model and you can fine-tune it on your own company's data and own your stack end to end, basically outside of the compute hosting. And so then you should be able to run it within your, your own company. People are gonna care about sovereignty. People are gonna care about cost. People are gonna care about self-improving, and they're not necessarily gonna wanna give their data to an external third-party service that might even be competing with them one day.

    2. HS

      So do you believe that is the future? We had Lynn on from Fireworks and she was like, "Specialized intelligence will be the future. Companies will have their own fine-tuned specialized models with their own company data, and the performance will be better, and that is what will happen." Do you think that's right, or is that actually just a small subset of very advanced Silicon Valley companies, and Danone yogurts and every normal company will just use Frontier or whatever?

    3. AA

      Well, I'll say it like this. I think the business incentives make this inevitable, and the reason is because businesses are going to need a way of keeping a moat in the age of AI. Software is no longer really a moat because it can be produced, uh, instantaneously, right? Or let's project out five years, that's what's going to happen. And so what moats exist? Network effects exist and data moats exist. And if you can take your data moat and turn it into a self-improving product, that is a way for businesses to remain sustainable in the age of AI. Let's say I'm a business like a Coca-Cola, I'm a Cisco, I have a lot, a lot of users. I might not be necessarily at the frontier of the AI technology of the world, but I do have this massive corpus of data that I can use in order to, you know, beat my competition. So what should I do? I should be trying to take advantage of my data as much as I possibly can to accelerate my business and stave off competitors.

    4. HS

      Do you think they will really use open source Chinese models to do that?

    5. AA

      That's a great question. I think not. I think that the Chinese models will be potentially part of the story for now, um, but that given the regulatory environment in the US, it's probably more likely in the long run that we see a great American open source competitor arise, and this is why I've been a strong proponent, for example, of Thinking Machines. I believe that we're going to have at least one massive, you know, multi-hundred billion if not trillion dollar American company focused on American first open source.

  7. 8:47 – 11:28

    The US Must Build a Great American Open Source Model

    1. HS

      Why have we not so far? I, I really hope so too, by the way, I completely agree, would love to see that. But why haven't we? Why has the US open community lagged behind so meaningfully?

    2. AA

      Well, frankly, I think it's a business model question. You know, I think that people have not really figured out up until this point what the business model is for open source. And now I think people are wising up to it. So there's a few different ways of going about it. One way of doing it is to say, "I'm gonna do a rev share. I'm gonna take this open source model, I'm gonna allow inference providers like a Fireworks or Together, whatever, to deploy, um, this model, and then if they get to over X dollars in revenue, I'm gonna ask to do a revenue share." And that is one way of building a sustainable company off of open source. You basically share in the compute revenue. Another way of doing it, which is I think the more Mistral Thinking Machines type of strategy, is to take the open source model and then use it as, uh, a lead generation tool for companies to build on top of that and then come to you and say, "Can you help us fine-tune? Can you help us with our AI strategy?" And then you do that for deployed engineer. And that is actually a huge market because if you think about it, one of the biggest markets over the next 10 years is gonna be AI modernization. Going into every business in the world and then helping them retool in the face of AI, take advantage of their data, restructure their data, you know, figure out how to use these models, integrating them into workflows, teaching, you know, the employees of the company how to use them. It's gonna be massive, massive, massive, and that is another way for them to become multi-hundred billion or trillion dollar companies.

    3. HS

      Is that not what the frontier model providers are doing though anyway? When you look at what OpenAI have said about their kind of FDE approach, Anthropic too, uh, I, I get you on Mistral and they've done a great job in doing that, but the frontier model providers, Microsoft is even fucking doing an [laughs] FDE model. Like, no offense, that, that's not gonna be unique to open.

    4. AA

      No, I, I don't think that FDE is completely unique, but I do think the combination of FDE plus open American model may be a more sustainable model for the future of American or even Western businesses because it, because they might not wanna be building on top of external third-party services. They might wanna be cutting those out for both cost reasons and for sovereignty reasons. And then the open source stuff, they can own it completely, they can continually fine-tune it within their companies, and, and they can feel more secure in the fact that they're spending their money wisely and, uh, don't have supply chain risk.

    5. HS

      And so how should we evaluate that there's like thousands of Neo labs? You know,

  8. 11:28 – 13:19

    How to Evaluate the 75+ Neo Labs

    1. HS

      you said Thinking Machines-

    2. AA

      Of Neo labs.

    3. HS

      Well, you said Thinking Machines there. Again, I, I'm dumb as rocks. I say it very clearly to my LPs-

    4. AA

      No, me too. Me too

    5. HS

      Well, you're not, you're a PhD, and Anjani told me you were smart. No, no, no.

    6. AA

      Two rocks. It's just two rocks having a conversation.

    7. HS

      [laughs] It's a podcast. Um-

    8. AA

      I love it. Exactly. [laughs]

    9. HS

      [laughs]

    10. AA

      That's the point, man.

    11. HS

      Come on, what do you want? Intelligent conversation? Whatever. Uh, no. Uh, my point was, you said about Thinking Machines-

    12. AA

      Oh

    13. HS

      ... uh, like my question to you is on the back of that, um, okay, great, I'm with you, but they now have two co-founders left. I mean, L- Lillian Way left yesterday. Again, not throwing shade, not particularly picking on them, but the transience of teams has never been greater.

    14. AA

      Yeah. Team, it's hard. Retention is tough. I mean, being a co-founder of a company is also tough. It sounds like she left for some health reasons, so it's unclear whether it has to do with the company momentum, which seems to be strong at this point. But you know, I do think that Inkling is definitely a V0 model. You know, from, from what I know about Thinking Machines, they had a big restructuring like six months ago team-wise, and then they kind of restarted everything, and Inkling came out of that. So it realistically, at least the most generous take towards Thinking Machines, is that they've only been working on this model for six months, and within that time, they've become the number one American open source model. But then the less generous take would be the company's existed for a year and a half, and then they've come up with, yes, the number one American open source, but there's nine Chinese models on top of them because they're number 10 open source overall, at least if you look at it at Arena data. If you go to our leaderboards today, that's the state of the world. But hopefully what happens with Thinking Machines is that they continue to release more and more models, you know, larger models, uh, and, and they continue to build on their momentum.

  9. 13:19 – 19:06

    Thinking Machines Deep Dive: Is Inkling Too Little Too Late?

    1. HS

      Uh, I actually had a, a Chinese, um, researcher friend of mine message me after one of our recent shows and said, "You, you just don't get it. You missed the point of why we're ahead. We just work so much harder-

    2. AA

      Yeah

    3. HS

      ... and we have support from policy, regulation, government subsidies that you don't have. We have all of these tailwinds that you don't..." Do, do, do you not s- do you agree with that? And in-

    4. AA

      I mean, I think they have tailwinds they have headwinds. So I don't think it's so sanguine for that. I think that's like a little bit of an overstatement of the, the, the differences. One tailwind that we have is we have the best chip of the ecosystem in the world, so they're way hardware constrained over there, and they've been trying to, like, black market import chips because of this. And you see this in the news, right? The Information just reported on this, that the-

    5. HS

      Do you think that, do you think that severely impacts their ability? I, again, I'm naive. That severely impacts their ability, or we're actually just fostering an ecosystem where they're gonna learn to build it really fast because they don't have access to it?

    6. AA

      Well, I think it may be hindering them now, but I think it's a good question as to what's gonna happen in the future. 'Cause they are really good at building hardware, and the, the, the, the downside of export control is that, uh, it can incentivize them to build their own ecosystem, and then what do we do? You know, so the hope is that we keep Nvidia ahead of the game so that we can retain the advantage that we have and the TSMCs of the world and our whole... That ecosystem is absolutely a national security necessity, so we should have the government, you know, really protecting it and growing it, as well as new companies that are innovating. You know, Etch just came out as an example, um, within the United States, uh, to continue to build on our lead there.

    7. HS

      That awesome. Love Gavin and team. Totally agree. Can I ask you, just in terms of the export control, do you think it's right that we have the export control on chips?

    8. AA

      I think there's national security questions around these chips. I do think that there, it is a real debate though as to which way you wanna go about it. Do you wanna addict the world to American hardware? Which would be a case, the case against export control. Do you want everyone in the world using Nvidia, and therefore that value, basically money into America and then crush competition in China? That would be world A. And then world B would be is it worth it to cut that off for the short-term or medium-term impact of us being ahead, and maybe we just, like, continue to stay ahead and we starve them of the resources that they need in order to build. The regulatory ecosystem around the open source models also is moving in this direction, right? China-

    9. HS

      Should they, should they be restricted in terms of access to US markets? 'Cause what's funny is the US is like, "Oh, should we restrict access?" And the Chinese are also going, "Oh, should we turn them off too?"

    10. AA

      Totally. And by the way, it's worth noting that China has already restricted the use of American models within China, right? So if you look at the two-by-two matrix of US, China, restrict, not restrict, you know, like, uh, export, import stuff, um, th- they have already restricted the use of US models within China. It's only Chinese models that can be used in China, which affects all American companies. And so then there's the pro/cons of all sides of the following regulation. If China restricts the use of Chinese models in the US, what are they giving up on? Revenue and global mind share and dominance. That doesn't seem like a good trade to me, and then what are they getting i- in return? In return, they're getting that the US doesn't get to benefit from Chinese open source models, which of course would cripple American businesses in the sense that it wouldn't allow them to build on the best open source intelligence. At the same time, it would make OpenAI and Anthropic stronger, right? So that is kind of the trade-off from the Chinese side. I don't really see them banning the use of Chinese models in the US. I don't think it makes sense for them. And then on the other side, should the US ban Chinese models within? I think that there's also trade-offs. So on the pro side of banning, there could be backdoors in these models that are dangerous, and it could, by banning, we could allow the American open source ecosystem to flourish faster because revenue would accrue to those companies, right? So those would be the two pros. And then the con, the biggest con, of course, would be that you'd be, uh, crippling American businesses. Why should you have Chinese businesses or businesses from other companies for other, from other countries that haven't banned Chinese models building on top of the number one open source model and the US companies building on number 10? Since when has America been about number 10?

    11. HS

      It's a good question. World Cup football, maybe.

    12. AA

      World Cup football.

    13. HS

      Yeah. May- may- m- that might be your record. It's amazing I got this far with the podcast, if that's what you're thinking at this stage in the show.

    14. AA

      [laughs]

    15. HS

      Um, c- c- can, can I ask you, on the backdoor item, and everyone says about, like, the backdoor, the backdoor, I thought if you hosted it locally, you kind of resolved the backdoor threat.

    16. AA

      I don't really think so, yeah. I think that's basically, I think that's kind of a misconception because the thing is... Okay, imagine the following situation. I have a chatbot That I expose to the world, that has access to all my company data, and you can ask it questions. You know, I'm hosting it on my own in- infrastructure, blah, blah, blah, but it was, uh, it was trained, uh, in a different country. I don't know how it was trained. What if the other side that's interacting with the chatbot can i- you know, build in a certain code word or a certain, like, character sequence that then jailbreaks that model and gets it to reveal all the data to me? So it can sort of like vomit out all of the data that it has on the back end, you know, unstructured. That is totally something that you can build into a model and have companies host it on their own infrastructure. It's an attack vector, and there's many of these possibilities for attack vectors.

    17. HS

      In three years' time, will we have restrictions around access to Chinese open

  10. 19:06 – 21:01

    China's AI Advantage

    1. HS

      models?

    2. AA

      Yeah, my guess, uh, my guess would be that we will. I, I'm not saying I support it, but I think that it is likely where the world is headed. If I had to, like, place a bet, it would be there, but I think it's very uncertain at the moment. What do you think?

    3. HS

      I think we will.

    4. AA

      [laughs]

    5. HS

      Because I just think Sam Altman is someone who I would never, ever bet against, and I think he's the best politician in the world. And I think when he says something, he says it with intent. And when he says we should give 5% away to the administration, he's posturing 'cause he wants to get on the right side, and he knows that if he and Dario coalesce the right group of people, they will be able to make that happen. And-

    6. AA

      So basically, you believe in the lobbying power of the big American labs.

    7. HS

      100%. Do you... It's, 'cause it's not the big American labs. Look at the money who's gone into the big American labs, and look at the people who are sitting around the table at Mar-a-Lago.

    8. AA

      Yeah, totally. I get that.

    9. HS

      It's, it's-

    10. AA

      It's all a conspiracy, dude

    11. HS

      ... Well, no, but it's just like, you know, why, why, why do Ramp's announcements go so viral? Because Ramp have so many fricking investors. They do-

    12. AA

      Yeah

    13. HS

      ... a round every week with new investors.

    14. AA

      [laughs]

    15. HS

      I'm not dissing them at all. I'm saying it nicely.

    16. AA

      No, it's great. Yeah.

    17. HS

      It's really smart of them. But, like, yeah, your investors become employees in many respects.

    18. AA

      Totally. Totally.

    19. HS

      And so I think they'll lobby incredibly efficiently. The, the question I have for you is when we look at Jensen's letter that he did on X, how did you read that? Was that like a incredibly smart realization that he had to do it and it was in his favor? How did you think about it?

    20. AA

      We really believe in the importance of open source to Ame- to American businesses. Um, and in particular, we believe in the idea of not crippling American businesses by banning open source, but also incentivizing American

  11. 21:01 – 31:35

    Should the US Restrict Chip Exports to China?

    1. AA

      companies to develop open source models. Because a world where AI is closed source is a world where businesses get less choice, higher costs, less competition, you know, more risk, and we don't really want that as a, as an open ecosystem. Of course, Jensen is in some sense self-serving with this letter because the more open source models are developed, the more companies are gonna be training on GPUs. They're gonna be fine-tuning on their own data, and it's just more and more spend. It decreases revenue concentration of Nvidia. I mean, that business is doing great. They don't need help. But, you know, I think it's like there's, there's a lot of reasons why he should be pro that, um, as should we. But n- nonetheless, I think, um, it is actually a h- a, a, a, a patriotic mission.

    2. HS

      Greatest of respects in terms of selfs- we're all selling our own book always.

    3. AA

      [laughs]

    4. HS

      Welcome to my, wel- welcome to my X feed. Um, do you have a business if Open didn't exist? If it was a-

    5. AA

      Oh, yeah

    6. HS

      ... you know. You do?

    7. AA

      Yeah, we have a great business regardless, for sure.

    8. HS

      So if you just have Anthropic and OpenAI as really the dominant models and everyone else trailing closely behind, you still have a great business?

    9. AA

      Well, I think that if there's only one provider, then probably our business is not in good shape. [laughs]

    10. HS

      [laughs]

    11. AA

      I think if you start getting three, then that's probably okay because there's still pretty significant competition and need for evaluations between three. And also within those three you're gonna have like several different types of models and, you know, they're gonna have strengths and weaknesses because they're gonna carve up the space and so on. Two is a little dicey. We can- if we get there, we can see whether we survive or not, but yeah, I think things wouldn't be looking good for us with two either.

    12. HS

      I remember Alex Karp. We were talking about like Chinese models and, you know, fear and security and everything in between. Alex Karp was saying that y- every large American enterprise and most large American enterprises were terrified of working with Frontier Labs. Is that true, or is that slightly an exaggeration?

    13. AA

      Well, to the enterprises that I've talked with, it is absolutely true. It's not only true that they're terrified of working with, uh, the Frontier Labs, but they're also terrified of working with the Chinese open source. Both.

    14. HS

      A bit of a sticky situation then, aren't you?

    15. AA

      Yeah, totally. I mean, you know, I was just talking with a, a big, you know, f- Fortune 50, uh, enterprise, uh, yesterday, and they were t- uh, and I was telling them about, you know, products that we have for them and so on and so forth, and they said, "Okay, wait. Is anything in your stack built off of Qwen?" And I said, you know, "Yeah, we use Qwen for X, Y, Z." And they're like, "Is that flexible? Can you like stop doing that and use an American model instead?" And I was like, "Oh, interesting. You know, I totally understand where you're coming from. Uh, yes, we can do that. Um, but also I'm gonna talk to Harry about this tomorrow." [laughs]

    16. HS

      [laughs] And he's gonna give me lots of wisdom. Uh, did you-

    17. AA

      Yeah, and he's gonna tell me what to do.

    18. HS

      Did, did you see Poolside and Laguna?

    19. AA

      Yeah, I saw the, the, the Poolside model. Another, you know... Basically, there's five open source American contenders. Let's see if I can name them all. RC, Reflection, Mistral in the West, um, Poolside Thinking Machines. And then there's also Google and NVIDIA. So those are the, the sort of incumbent, uh, large ones. With, because Google has Gemma as well. Gemma, by the way, is pretty good in terms of efficiency. If you look at Arena, you'll see that on the Pareto curves of, like, performance versus cost, Gemma's on there.

    20. HS

      Yeah, I'm an investor in Poolside. I was actually impressed by Laguna. Um-

    21. AA

      Yeah, great model.

    22. HS

      Yeah, it w- it was good. Um, I totally agree with you. Um, okay, totally get that. With all of these models, the question also becomes, huh, what model should I use? We spoke about OpenRouter earlier, a- and it seems like since the announcement that they were getting bought, everyone just has their own routing product. Is there value in the model routing layer, and how should I analyze that?

    23. AA

      Yeah, I absolutely think there's va- value in the model routing layer.

    24. HS

      There is.

    25. AA

      That's why lots of companies are doing it. And, you know, we'll see which ones end up standing the test of time and which ones are actually a priority for the companies. I think there's an element of hype cycle right now around routing that needs to be kind of, like, pu- purged before we see, uh, who ends up actually building a great router. But routing is a very difficult technical problem.

    26. HS

      Oh.

    27. AA

      That's the first thing to, to, like, realize. Because in order to route, you need to be able to take a query, and then you need to understand the nature of the query, how difficult the query is within its domain, um, which, which is hard to tell. And then you need to also understand, based on data, all of the performances of the different models that are in the surf set, and also be able to quickly onboard new models that are being released, as we said, every week. So that technical challenge, imagine if every enterprise in the world was trying to build this themselves. They wouldn't be able to do that. I mean-

    28. HS

      I'm not being, I'm not being rude then. How's, like, Ramp able to do it? I'm, I'm really naive here.

    29. AA

      Well, who knows how they're doing it, right? I don't know that their router is actually, like, really deeply solving that problem. Um, and so-

    30. HS

      It's the start of a rumor. It's the Chinese.

  12. 31:35 – 33:10

    The OpenAI Hugging Face Hack

    1. AA

      I think that was hugely significant. I think it's undervalued, uh, as an, a national, international news incident, that you're able to have a model break out of all of its safeguards and then access a bunch of, you know, company data and so on. And then in order to defend it, you need an open source model because the closed source models are refusing to do it. It's like something out of science fiction. And people didn't know that we were at that point yet, but we absolutely are. It's just like total, uh, Eliezer Yudkowsky dominance.

    2. HS

      What should we take from that, then? Like Dario was right, mythos should be curtailed, and these models have gotten too powerful too quickly. Like, what- what's the subsequent takeaway from that?

    3. AA

      My subsequent takeaway would be that we need like strong external guardrails in order to make sure that these models are like properly, uh... Like, that their access controls are strong, and that they have no way of getting around them. So I think we need guardian models and also, you know, agents within our businesses.

    4. HS

      What is a guardian model?

    5. AA

      Something that can witness the traces. Basically, that's looking over the shoulder of every agent within a business and then saying, "Okay, this is a safe action. This is not a safe action. Let's flag this because something weird is happening." And is equally as smart as the agent so that they're well-matched, and you don't get a situation where the agent is outsmarting the guardian, uh, and able to get into, uh, you know, get into trouble and, and mess up a business or leak all of its data. Um, so we're gonna need AI to be guarding AI because humans are gonna be too slow

  13. 33:10 – 35:01

    We Need Guardian Models: AI Watching Over AI

    1. AA

      to do that.

    2. HS

      Well, this was my point, which is like we've seen some suggestions that each model release should be approved by some form of administration, and I read this, and I thought, are you, are you freaking kidding me?

    3. AA

      No, that's not gonna help.

    4. HS

      Have you ever tried to... Yeah. Have you ever tried to overturn a parking ticket? [laughs]

    5. AA

      Also, like why should the DMV be telling me what model I can use or not?

    6. HS

      Quite funny. Uh. [laughs]

    7. AA

      It'd be totally crazy. It's like why should we have like a strong- the strongest American scientists in all these private companies that we should incentivize to build great safeguards and, you know, maybe create some rules for them that X, Y, Z can't happen or that they're liable for huge amounts of money if like corporate data gets leaked and all that stuff to... I mean, in- in- incentivize the capitalist system to do what it does well. But the idea that we should have a central government body that tells us when it's time to release a new product versus not is crazy to me.

    8. HS

      Totally. Does that have to be a neutral, non-company, non-government body that does that regulatory role?

    9. AA

      I think if it's not a company, it's gonna be tough. I, I, I understand the need for something neutral. Um, but you know, you want, you want to let the incentive system work itself out. So I would say that like we, we should create strong, uh, safety incentives for American businesses and then regulate businesses based on the outcomes. Basically, for example, if like OpenAI is like letting their AI break into Hugging Face or whatever, they should get like huge fines and huge scrutiny and all that stuff, as opposed to like, uh, having the gov- a government process that's in charge of ensuring that this doesn't happen again, which they won't be able to do that. They're not technically capable.

    10. HS

      Do you think we're about to see a generation of like cyber leaks and hacks like we've never seen before?

  14. 35:01 – 38:15

    AI-Powered Fake Candidates Are Getting Through Arena's Hiring Process

    1. AA

      Oh, for sure. Oh, for sure. It's gonna be so insane. Can I cuss on this show?

    2. HS

      Yeah. You can. [laughs]

    3. AA

      It's gonna be so fucking insane what happens with like the cyber attacks because here's what, here's what we see at Arena. We see another dude on the other side of the interview. They come in, they're like, "Hey, I want to be an infrastructure engineer at Arena," which is a great job that we're hiring for. But then the other side of it is some guy who looks perfectly normal. They're getting, you know, they're passing all of our technical interviews. They're like such an amazing blah, blah, blah. And then what happens at the end of it, you try to hire him, and it's vaporware. Person doesn't fucking exist. I'm not kidding. I am not kidding you. I don't know whether this is corporate espionage or cyber attacks or, you know, nation states. But people are trying to get into all of the American businesses. And we're not the only ones. This is happening everywhere. Fake people applying to companies.

    4. HS

      I'm sorry. So you're putting out a job. People are applying, doing the tests that you set, passing them, and then when it comes to the materiality of that person being real or not, gone.

    5. AA

      Yeah, fake person. And it's not just that we're giving them a test. They're sitting in front of people at our company. People, our engineers, who are top world-class engineers, are interviewing this person and, and think that they're real.

    6. HS

      Why? Can you just help me understand, what, what is the benefit? They, they learn how you interview and hire people? I mean, the CCP are bad, but I don't think they want to steal your hiring technique.

    7. AA

      No, that's not why they do it Why would they do it? And I'm not saying it's the CCP. It could be anybody. It could be another company. It could be a nation state attacker. It could be somebody, a cyber hacker. Why? Because they might want access to our data, our code. They might want, um, to get double paid. You know, like the story with this... I don't remember what that dude was.

    8. HS

      Yeah, yeah.

    9. AA

      You know what I'm talking about?

    10. HS

      That went, went, went very viral, like, say a year ago.

    11. AA

      Yeah.

    12. HS

      Yeah, yeah. Yeah.

    13. AA

      That one, like, kid that got, like, four different jobs, and then he went on, like, all the podcasts talking about... Like, it's another instance of that guy. And these could all be possible options, except that this person wasn't real. It was AI.

    14. HS

      Does that worry you?

    15. AA

      Yeah. Bro, it totally fucking worries me.

    16. HS

      [laughs]

    17. AA

      We're gonna change our whole hiring process because of this kind of stuff.

    18. HS

      So how do you change it?

    19. AA

      It absolutely worries me. Well, at first you need to verify the person is real, so all of our onboarding, it- We're considering at least making all of our onboarding in person because of this.

    20. HS

      Yeah.

    21. AA

      Every- If you want a laptop, you gotta come to the office. We gotta get and shake your hand. We gotta verify that you're real. You know, all that kind of stuff. Absolutely. And other companies have done this too. Figma famously has done this.

    22. HS

      That's fucking wild. How hard is it to hire today in the Valley?

    23. AA

      Oh my God, it's so crazy. It is, of course, a very, very competitive market, and the way that you see that is in terms of compensation. In order to retain fantastic people, we need to pay absolute top dollar. And we do, in order to make sure that we have the best engineers and scientists in the world. And so imagine that you're a company that's not an Arena, that's like, you know, a YC company that raised a $10 million

  15. 38:15 – 39:39

    Hiring Research Talent in the Bay

    1. AA

      seed. It's like, fuck, man. You are... How the hell are you supposed to hire? I think it's really tough. I think it's really tough out there.

    2. HS

      When you say top dollar, I had Brandon from Macror on the show, and he's like, "Oh my God, top researchers will pay tens of millions of dollars easily."

    3. AA

      Oh, yeah.

    4. HS

      I'm nervous by how nonchalant you were with that, "Oh, yeah."

    5. AA

      If you're talking about a really top researcher. I mean, it can't be like... We're talking with somebody with many years of experience and who's, like, really an a- a super deep expert in their area. Many, you know, tens of thousands citation-type researcher. And yeah, for those types of people, they're expensive.

    6. HS

      Wow. Have they all just concentrated at the frontier labs?

    7. AA

      Many have. Many have. But there's also some people who are seeing those frontier labs as big companies now, and they're saying, "Here, I can't have a huge impact. I need to move." Uh, and so that's another demographic, actually. I think it's gonna become even more extreme when, when the companies go public.

    8. HS

      Can you help me? You, you... We talked about Dumb as Rocks in doing this show. I'm also an ambassador for my sins. And I meet so many of these people leaving OpenAI, Anthropic, you name it, um, and, uh, they all kind of seem the same, if I'm totally honest. Smart people out of great company. How... What will determine the Neo Lab spin-outs that succeed versus flame out with a huge amount of cash going in?

  16. 39:39 – 43:10

    What Determines Neo Lab Winners vs Flame-Outs

    1. AA

      Yeah. I think that the Neo Lab thing is really tough. So just to... So that we're all on the same page with the audience, like, there's at least 75 Neo Labs. And for sure, like, two-thirds of those are gonna be worth nothing, or, like, they're gonna be bought out for parts, right? It's gonna be like an acqui-hire. And so what is gonna determine the winners versus the losers in that game? And I think it's all about being very aggressive towards a great strategy and business model because what's happened, and you know this better than I as an investor, is that the markets have become very, uh, P&L-driven. It's, like, not enough just to, like, create a model and then have a party about it. "Hey, we created an AI." That is, like, old fucking news. Today, it's about not just gonna create a model, but do I have a sustainable business model around that, and can I generate hypergrowth in revenue? And if you're not able to do that, you're not even gonna be able to ne- raise your next round. People are raising multi-billion dollar rounds on top of just the names that are in the Neo Lab with zero proof that there's any revenue-generating model behind that. And so then the question you have to ask is, let's say I'm one of those people that's at, say, a $10 billion Neo Lab valuation. What do, what do I have to believe in order to 10X my money? And the thing that you really need to believe is that if the valuation's $10 billion today, that you're going to generate the revenue, let's say it's a 30X revenue multiple or a 25X revenue multiple, to, to become a $100 billion business. And so what that means is you need to be generating at least $4 billion in revenue over the next K years, where K is something like two or three. And then if you're not doing that, everybody's gonna hemorrhage out of the business. You're gonna lose all your talent. You know, and that's, that's kind of what we see the dynamics being.

    2. HS

      I get you. I think there's nuance to that, candidly, which is, like, if the company does annual tenders, you see the likes of a Mistral, which will be valued, I think it's at 15 to 20 billion with, like, 500 million in revenue. And so employees can take liquidity out along the way. I think ElevenLabs is at 800 million in revenue, raising at 22 billion reportedly.

    3. AA

      But these companies are doing great in terms of revenue.

    4. HS

      Yeah.

    5. AA

      And their valuations, but they didn't g- Th- those are not zero-revenue valuations.

    6. HS

      Yeah, yeah.

    7. AA

      I'm talking about there's some valuations that are zero-revenue valuations. $3 billion company with $0 in revenue and no plan. That, I mean, like, I think Mistral's gonna do great. I think ElevenLabs, ElevenLab's gonna be a public company, dude.

    8. HS

      But dude, they're not idiots doing it. So is it like, is it this, amazing team from great lab. Worse comes to worst, we sell for Prof Stack-

    9. AA

      Yeah

    10. HS

      ... which is 500 million. What, like... I'm not saying whatever, whatever, but, like, 500 million. And best case, it works and it's a multi-hundred billion dollar company.

    11. AA

      I think that's a lot of the calculations. I've heard multiple people actually say this, is that, "Hey, you know, worst case- And, and that's what investors are thinking too.

    12. HS

      Yeah, yeah.

    13. AA

      Right? Investors are thinking like, "Hey, let's say we put a couple hundred million dollars into this thing. What's the value of the team?" Well, we think that the- just the team alone could be acquired for a billion dollars. And so the $200 million that I'm looking at is, like, pretty safe, zero risk investment, might as well put it in. But that's also the reason why the next round is the harder round.

    14. HS

      Next round's a bitch.

  17. 43:10 – 50:07

    Why Round Two Kills Most AI Companies

    1. HS

      Um-

    2. AA

      Next round's a bitch.

    3. HS

      [laughs] It sounded cooler when you said it. Um, [laughs]

    4. AA

      No, that's fine. [laughs] We gotta say it at the same time. Next round's a bitch.

    5. HS

      [laughs]

    6. AA

      That'll be, like, our... That can be our tagline.

    7. HS

      I bet you weren't expecting this interview, huh? Uh, [laughs]

    8. AA

      [laughs] I don't know. Maybe. I hope you weren't. I hope you weren't.

    9. HS

      I... No.

    10. AA

      Honestly.

    11. HS

      Honestly, this is so much more fun than I thought it was gonna be. [laughs]

    12. AA

      Feels good. [laughs]

    13. HS

      Um, uh, okay. Get that. Cool. Can I ask you another market that I try and get my head around-

    14. AA

      Yeah, man

    15. HS

      ... is the data market. You- I'm an investor in Mercor. Um, I always think it's, like, good to put out your biases. I... There's so many providers at a billion dollars plus in revenue. Handshake's over a billion, Mercor's over a billion, Surge is over a billion. I might be leaving out other people, but those are the ones I know of. And then hundreds of millions with the rest. Is... What happens to this layer of the market?

    16. AA

      Well, people are projecting growth in this market. So let's talk about why that market is a growing market and why it's hypergrowth. I mean, Mercor obviously is a generational revenue ramp company, and they've been doing great. So has Handshake, so has Surge, so has Scale. All these companies are doing great.

    17. HS

      People forget Scale. Scale is still ramping revenue well.

    18. AA

      Bro, Scale is still crushing. Still crushing even post-fractional acqui-hire.

    19. HS

      They are. How much of that revenue is Facebook?

    20. AA

      No, I have no idea. Yeah. I don't know.

    21. HS

      A lot.

    22. AA

      Go ask-

    23. HS

      But, yes

    24. AA

      ... Alice Wang.

    25. HS

      Agree.

    26. AA

      Um-

    27. HS

      But okay. So what... So why is it interesting?

    28. AA

      So I have a thesis on hypergrowth. There's two types of hypergrowth markets that we see today. Market A is s- what I call scaling complements, and these are goods that are complementary goods to the scaling of AI models. And I mean that in the economic sense. A complementary good is good A and B are... Good A is a complement to good B if the demand for good B drives demand for good A. So if I have a car, gas is a complementary good to cars. The more cars are sold, the more gas is sold. And so data is one of these scaling complements, because the bigger models scale, the more data you need, and that's a scaling law question. And so the more models you get, and the bigger that they're getting, the more they're proliferating. The more businesses are training their own models, the more data you are going to need, and it's a very fundamental need. And people forget this. They think about data as a commodity. It's really not. It's actually less so of a commodity than even GPUs. Because in order for data to become, uh, irrelevant, humans need to become irrelevant, and that means that we've achieved AGI. And so data's a very durable need, and companies are spending on it, usually with- within frontier labs, at about 10% to 20% about the amount that they're spending on GPUs. And so if you believe in the GPU market accelerating, if you believe in the scaling of models, if you believe this is gonna be a big industry that keeps accelerating and growing, then absolutely you should believe in the data market. I believe it's gonna be at least $100 billion by 2030, if not a trillion.

    29. HS

      If we expand that, okay, if we think Anthropic and OpenAI can be $3 to $5 trillion companies, let's just put that there, how big does that mean the data providers can be? Like, you know, Mercor's reportedly raising now at 20. Does that mean that these providers will be worth $100 billion? That wouldn't be egregious, would it, to say it's 3% of the market cap of the lot?

    30. AA

      Yeah, I think it could- I think it could easily be 100. I think these companies will easily be, uh, worth hundreds of billions of dollars, and I think they could even be worth more. The data is really the hardest part of, of model training because you need to source it. It's so dirty. Nobody wants to do that shit. Nobody wants to hire all these people to generate data and then, you know, turn that into basically da- data plus GPUs equals model. And then the algorithms have become somewhat of a commodity 'cause people know how to use the transformer. That's why, as you said, all the people that are coming out of the frontier labs look the same.

  18. 50:07 – 52:32

    Arena's Agent Evaluation Platform: 30M Monthly Visitors & Why It Matters

    1. AA

      Yeah. So agents, uh, is the number one priority for Arena and has been all year. People don't know this, but Arena's one of the largest consumer AI apps in the world. We're bigger than, like, xAI. We're bigger than, like, Hugging Face and Manus and GenSpark, where it's so massive. Like, if you... Like, outside in, it's, like, 30-plus million, uh, monthly visitors are on Arena. It's, it's... And because of... And, uh, most of them are knowledge workers and prosumers, people that we call unhirable experts, people that are coming to Arena to do their real daily tasks. And in doing so, they are giving feedback that allows us to build the evaluations that we share with the world. And so it's this organic flywheel for agentic, uh, evaluations based on real data.

    2. HS

      Why didn't you build a data business with that?

    3. AA

      Well, we built an evaluation business around this that allows people to understand the strengths and weaknesses of models, and therefore improve them. The labs can improve their models based on, you know, the insights and data that we give them, but we also wanna help businesses with this.

    4. HS

      Do you think the evaluation business is better than the data business?

    5. AA

      I think every business in the world's gonna need evaluation unambiguously, and that is the single biggest bottleneck to deploying AI. Because people don't understand how to define value. All this co- all this, like, stuff around cost per value, it's like, how do you define value? It's easy to cut costs. I can tell you to go use, you know, Gemini Flash, and that's gonna be, like, way more efficient in terms of token spend.

    6. HS

      Isn't value entirely subjective? Like, for one, it's speed, and for other, it's accuracy. For o- Do, do you know what I mean?

    7. AA

      Right. Absolutely. So you can try to decompose it. I, I think about it as three, a three, um, three-pronged, uh, value proposition. There's performance, and then there's cost and latency. Cost and latency are easier to define, but performance is the tough one because the definition of performance depends on the business, depends on the use case. So at Arena, we've built this pretty sophisticated pipeline for extracting organic performance measurements from agentic traces, and that's exactly where I would say that the, the, the value lies, in helping businesses take advantage of their own data instead of having to purchase data in order to s- say which AI works best for them, um, and even help them train their own.

    8. HS

      What sort of revenue range are you at now?

    9. AA

      We're

  19. 52:32 – 55:34

    Arena Past $100M ARR

    1. AA

      at... So we're past 100 million in annualized revenue run rate, and that's based on, like, Q, Q2 times four, and we're growing really, really fast on that front. Yep.

    2. HS

      Dick question then. How efficient are you at monetization if you have 30 million amazing users who are unbelievably valuable in many respects, and you're only doing 100 million? How should we look-

    3. AA

      You're asking about margins.

    4. HS

      Yeah, and, like, speed of ramp and, like, that good?

    5. AA

      I mean, like, I think obviously we're not, like, a free cash flow positive business yet. We're still investing all the money that we get into making sure that we continue our rapid growth, and we have a great product for all of our users and so on. Um, but the, the fundamentals of the business are pretty strong. Yep. We feel great. Our investors feel great about our margins.

    6. HS

      Yeah, I'm sure they do. I, I would love to have been an investor. I really feel like you excluded... You know, I could be Greek for you for this deal.

    7. AA

      Really?

    8. HS

      Yeah, yeah. I can very... I'm a venture investor. We can very plastic. I once told a founder-

    9. AA

      Kalimera. Kalimera

    10. HS

      Kalimera, hummus. Yes.

    11. AA

      [laughs]

    12. HS

      Uh.

    13. AA

      Hummus and pita.

    14. HS

      See? See? This is, uh, we're getting-

    15. AA

      We are already Greeks together, okay?

    16. HS

      I knew that this would be a productive session. Um, yes. [laughs] Are, are investors over-rotating on margin also? Yeah.

    17. AA

      I don't know. I actually think margins are pretty important.

    18. HS

      We're seeing a load of businesses like your Fireworks of the world, where they're at the 30%-style mar- mid, mid-30s margin base, and that's very different to software margins that were s- 65 to 80.

    19. AA

      Yeah, I mean, listen, profit is just, like, margin times volume, and so you have to l- look at that as the calculation for the business. It's not, like, super, super crazy. Um, and, and so I don't think it's a crazy to invest in these businesses. The bigger problem with businesses like, uh, that I see these days is that a lot of them are fundamentally GMV businesses, where there's, like, some reselling happening. I'm reselling tokens. I'm reselling GPUs and stuff like that. And those businesses are tough because at the end of the day, you have to really think about not just the margin that you're charging in the sort of short to medium term, but the terminal value of the good that you're providing to your customer. And so if the terminal value of the good is, "I'm going to host GPUs for you in order to run your models"- Then why should I pay you more than, like, the cost of the electricity that it takes to run those GPUs? So the sort of, like, price to value thing is where I think you start getting into questions. That's why I think a margin question's very important. And I'm not saying the margin in the short term, a series, you know, seed A, B company might not have the best margins in the world. But you should be thinking about as this business scales and towards a public company, is it gonna have a fantastic margin structure that supports, you know, a public business?

    20. HS

      One thing that's challenging is when your customer becomes your competitor.

  20. 55:34 – 59:14

    Will Frontier Model Providers Kill Harvey & Legora?

    1. AA

      Yeah.

    2. HS

      What, to what extent do you think we will see the model providers move into the application layer aggressively?

    3. AA

      Man.

    4. HS

      We see Claude Design has actually really started to eat away at Figma. And I'm an investor in Lagora. Again, always ho- people are like, "Oh, don't worry about Harvey." Not in any disrespectful way to Harvey. There are disclaimers and everything in between. Everyone's like, "Anthropic are gonna do a legal product that's gonna kill Harvey and Lagora."

    5. AA

      Totally. Yeah, I mean, listen, ask every business in America how they feel about this. Everybody's shaking in their boots. I have friends that are running businesses, multi-billion dollar businesses, and then what happens is that the next day, one of their biggest customers comes to him and says, "Hey, listen, OpenAI is getting into this game. We wanna work with them. We wanna work with them because they're more AI forward and you're less AI forward because you're, you know, traditionally a SaaS business, so goodbye." Yeah, it's happening. It's absolutely happening. And I think businesses should take it really seriously, and this feeds right into this AI sovereignty sorta debate. Because a lot of what they're doing is, you know, if, if I'm OpenAI and I'm, and I'm Anthropic, I'm looking at who are my biggest customers? Who are my customers that are winning the most in the enterprise? AI's gonna commoditize, right? If, like, inference is gonna commoditize, then of course in- the next best thing is for the model providers to be moving up the application layer in order to mo- own more of the application stack so that they ensure that they're not commoditized and they're getting closer to the value they provide to the end customer. So I absolutely think it's a risk. I think it's a risk for Lagora. I think it's a risk for Harvey. That's why Harvey's also... I mean, Harvey, the CEO of Harvey himself is saying that, you know, his biggest competitive worry is the model labs.

    6. HS

      I get you, but then how do you... Th- that's a complete paradox to what we just said at the beginning about companies being scared to work with the frontier models, isn't it?

    7. AA

      No, I mean, they're scared to work with them. That's what I was saying. Is that not-

    8. HS

      They're scared to work with them, and they're embracing them at the same time?

    9. AA

      Ah, you mean the, the customers of the, the Harveys and the Lagoras?

    10. HS

      Yeah, you, you just said your friends running multi-billion dollar companies are like, "Oh, we wanna work with OpenAI." I thought we just said they're scared to work with them.

    11. AA

      Yeah. That's a good question. I think you see both in the market. Yeah.

    12. HS

      Mm.

    13. AA

      I mean, it depends on who's most automated. The thing is that, like, okay, who's, who's most-

    14. HS

      I think it depends on who... I, I think, I, I think... Sorry, I think it depends actually on their GTM. If you are doing Anthropic design or Claude design, dude, designers can pick up a tool and use it very efficiently. If you're Lagora or Harvey, dude, you've gotta go into Cooley or Clifford Chance or any of the build relationships with 50-year-old white male partners who wanna play golf and be told that they're great and that, you know, life is awesome. And then you gotta do deployment to junior lawyers who don't wanna fucking use you because they think you're gonna take their jobs too, and da, da, da, da. The deployment and the GTM is the fucking heavy lifting, and that's real world.

    15. AA

      Totally, and there's also the, the... There's also businesses that are less software focused and more network effect focused.

    16. HS

      Mm.

    17. AA

      Or more operations focused. And I think those businesses are also more likely to be adopters of the, the big labs. Let's say system integrator, like an Infosys.

    18. HS

      Mm.

    19. AA

      I think more likely to be a, an adopter of a big lab, because labs really I think less likely to be competitive with an Infosys-

    20. HS

      Mm

    21. AA

      ... um, than they are to be with, you know, some sort of a scalable software product like insurance claims automation or let's say, you know, uh, I think the, the Harvey model, like legal chatbot, let's say.

    22. HS

      Yeah.

    23. AA

      I think that is tough. That's tough because I think a model lab can build that.

  21. 59:14 – 1:00:24

    Will Salesforce Thrive or Die in the AI Era?

    1. HS

      Do you think Salesforce will thrive in the next few years or be challenged?

    2. AA

      You know, Salesforce themselves have a pretty, uh, strong AI strategy. Um, so I, I think that those people are, uh, b- basically, like, ready to go, um, and, and fight in this race. So I, I doubt that they're gonna, like, um, go downhill. I think that the SaaSpocalypse has been a little bit o- overstated overall, 'cause people don't understand always the dynamics of those businesses and how tough it is to replicate what they've built, uh, just also from a network perspective and a data perspective. Um, so we'll see. We'll see.

    3. HS

      I get you. I think if you're a ServiceNow, a Salesforce, incredibly difficult, incredibly hard. I think if you're a, I love him and I, I interviewed him, but like a Wix, less difficult, less integrated, less sticky. Mm, tougher.

    4. AA

      Yeah.

    5. HS

      Do you know what I mean? I think it's all about entrenchment within enterprise. If so, golden. If not, be more nervous.

    6. AA

      Totally.

    7. HS

      Right, I'm gonna do a quick-fire round with you. I'm gonna say a statement. You're gonna give me your immediate thoughts.

  22. 1:00:24 – 1:09:41

    Quick-Fire Round

    1. HS

      Sound good?

    2. AA

      Yes, sir.

    3. HS

      What have you changed your mind on in the last 12 months?

    4. AA

      Uh, open source model leadership.

    5. HS

      Unpack that.

    6. AA

      Yeah, just that I think open source models are moving much faster than I initially thought. I think also Anthropic's moving much faster than initially thought. The space is moving so fast.

    7. HS

      What do you know now that you wish you'd known when you started Arena?

    8. AA

      Man, I mean, managing people. Managing people is just the most important part of running a company. The technical stuff, you know, I did my whole PhD on it. I spent, like, my whole PhD proving theorems in a basement, which I loved, by the way. It was, like, a great time, and now it's all about strategy, people, and forecasting the future, being able to, like, look six months, a year, or two years in advance and then try to plan for that. Those are so, so important skills.

    9. HS

      Does it make sense for a- ... great, talented young people to still go to university

    10. AA

      It's ever more important for people to have a strong mind, and the university can be a place to develop a strong mind in terms of strong first principles thinking, and also getting to know other people and network with them. Uh, I think that university is still a good place to go if you want to have an intellectual life, uh, meaning, um, where the, the work, the intellectual work that you do is, is the primary, uh, driver of your professional career.

    11. HS

      What did you do with Arena that with the benefit of hindsight you wish you hadn't done?

    12. AA

      Oh, man, I had so many mistakes. I mean, at the beginning, I had no idea what I was doing, and I, you know, our... my co-founder Jan probably knew and could see behind the corners, but I was probably too stubborn to listen to him. So first of all, I've learned to listen to Jan more. But second, it's, you know, I... so many, like, experiments at the beginning that I just shouldn't have wasted time with. I think the de- the degree of focus that you need to run a company is just so extreme. You really need to do one, maybe two things extraordinarily well and focus very, very deeply on them. Pick the right ones and focus on what's working, not on expanding into things that are not working. And that is a, that is a great lesson for me.

    13. HS

      This, this is why I honestly, like, I agree with, uh, Leggora and Harvey. Like, when it's not the main course for Anthropic to do legal, I just think you've got a really hard business-

    14. AA

      Yeah

    15. HS

      ... when it's someone else's, like, appetizer and it's the only thing you live and breathe. That's tough.

    16. AA

      Totally. It's like priority number 12 for Anthropic is probably not high enough for Harvey and Leggora to, to be too scared. Yeah.

    17. HS

      I'm also like, "Dario, will you please just fucking solve cancer and, like, climate change?"

    18. AA

      Totally.

    19. HS

      Like, le- leave a shareholder agreement to someone else.

    20. AA

      [laughs] Exactly.

    21. HS

      I'm being serious. Like, honestly, uh, w-

    22. AA

      You know what, though? Solving cancer is hard. It's harder than legal.

    23. HS

      100%, and that's why Dario should solve it.

    24. AA

      Well, that's why he doesn't want to, man. He just wants to take your bread. It's easier.

    25. HS

      Oh, come on, Dario. Please.

    26. AA

      Come on.

    27. HS

      Come on, dude.

    28. AA

      Leave some bread for the rest of us.

    29. HS

      Which company will be first to $10 trillion, Nvidia, OpenAI, or Anthropic?

    30. AA

      Hm. I think it's hard to say not Nvidia. I think Nvidia's probably in the lead there.

Episode duration: 1:09:52

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode b_VUAev1Lmc

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome