Inside the Race to Measure Frontier Intelligence

1. 0:00 – 0:55

   Intro

   1. RKRayan Krishnan0:00

      Every time a new trillion-dollar industry emerges, there's a need for this independent testing group. When Meta released Llama 4 on our held-out private benchmarks, the model was actually underperforming. But on all of the major public benchmarks, it was showing incredible capability.

   2. BHBen Horowitz0:14

      What's the limit of what you can achieve, and then within that, how are you going about it?

   3. RKRayan Krishnan0:19

      In an ideal world, take a frontier model and have it train the next version of itself, but obviously, that's very expensive and slow. And so what we're doing is forming a set of proxies for every part of the process it takes to build the next version of the models. Evaluations, as they become more complex, have a fewer sample size, but a larger set of criteria or expectations of them.

   4. JLJennifer Li0:38

      Where do you see the gap that's happening today?

   5. BHBen Horowitz0:40

      The government kind of has an inclination of what it's afraid of, be it biohacking or cyber hacking, but then there becomes the question of, can the model do it, and then can you get the model to do it?

   6. JLJennifer Li0:51

      What do you think the landscape will look like?

   7. RKRayan Krishnan0:52

      Um...

2. 0:55 – 2:52

   Why Vals Exists: When Public Benchmarks Stopped Working

   1. JLJennifer Li0:55

      So I'll start a question from, uh, when VALS got started in 2024 after your team discovered that all the public benchmarks are just not sufficient enough to measure model progress and there needs to be a new methodology and approach coming to, you know, keep us, um, uh, on the frontier and, and help, um, model labs hil-continue to hill climb. Uh, maybe just take us back to the, the inception of VALS and what you see was missing in the, in the, in the market then.

   2. RKRayan Krishnan1:24

      Yeah. Yeah, I mean, so I, I had a background at doing research, in particular building benchmarks and evaluations, and so what was very clear to me was the very tight relationship between what it takes to build new systems for generation, uh, and actually new mechanisms for evaluation. In fact, in order to get one, you often need to get better at the other. Um, and actually one of the biggest drivers for model capability is having a new legible way to evaluate models. Um, and so around early 2024, what we were seeing is, um, there are actually many interesting models coming to market. They weren't all coming from, from OpenAI. Um, and then also in particular, it was, um, harder than ever to actually ascertain what was newly capable with the new models. Um, and so in kind of a first principles way, what we, we realized is that there would need to be some third-party company that solely existed to build really high-quality evaluations and benchmarks to be able to discern, um, what was newly possible with these models. And so we released our first benchmarks in 2024, and now over the last couple of years, that has kind of been, been realized by many different parts of the industry.

   3. JLJennifer Li2:21

      And why, why do-- I guess one of the obvious question is why do you think the labs can't do this by themselves? 'Cause, um, they know the best of where the models are hill climbing on and what, what is missing capability-wise. Uh, why can't they be the, the, the benchmarking, um, stores?

   4. RKRayan Krishnan2:37

      Yeah, I mean, internally, they, they do build a lot of great benchmarks, and that's what drives model progress. But I think there's an issue when we, we speak about model capabilities in a way that's self-reported. And so one of the early indications of that you saw was when Meta released Llama 4, um, that was a bit of

3. 2:52 – 4:05

   The Llama 4 Disaster: Public Scores vs Private Reality

   1. RKRayan Krishnan2:52

      a disaster. And interestingly, what we saw is that on our held-out private benchmarks, the model was actually underperforming. But on all of the, the major public benchmarks where the, the questions and, and rubrics were actually open source, um, it was, it was showing incredible capability. So there's a huge disconnect between what was self-reported based on these open benchmarks, um, and then what we were actually finding with our, with our higher quality, higher signal benchmarks. Um, but I think it, it speaks to a broader, um, concept which the labs, I think, understand, which is that they would like to see a rational buying market. Uh, you know, they would like to see that when they invest billions of dollars to build a new model, there are actually substantive ways they can point to evidence and say, "We're, we're advancing in these, in these ways," um, and it's not just entirely self-reported to justify that investment. Um, and so you also see instances of, of Demis and others in the industry calling for an ecosystem of third-party evaluators.

   2. ETErik Torenberg3:40

      And what's the historical analog that you have in mind here? There, there are rating agencies, a-audit firms. What, what's the right comparable?

   3. RKRayan Krishnan3:46

      Yeah, I, I think, I think there's honestly lessons to learn across the board. And, and every time a new trillion-dollar, uh, industry emerges, um, there, there, there's a need for this independent testing group, and I think the fact that it's moved so quickly in AI has, has caused the necessity for a lot of these parallels to, to see, to be borne out. Um, you know, we think about ourselves as, as trying to sit on both sides

4. 4:05 – 6:00

   Inside the 6-Hour Pre-Release Testing Window

   1. RKRayan Krishnan4:05

      of the market. So there, there are mechanisms by which labs need to prove that new models are very capable, but there are also parallels where enterprises need to figure out what adoption strategy is gonna amount to the greatest ROI for them.

   2. ETErik Torenberg4:15

      Yeah. I want to sh-shift to take us through sort of the six-hour pre-release window before the model drops. Um, obviously, you need to run tens of billions of tokens without delaying the launch. W-w-which part is you doing an all-nighter versus-

   3. RKRayan Krishnan4:30

      [laughs]

   4. ETErik Torenberg4:30

      ... it, it being, uh, a-a-automated? Ta-take us through that.

   5. RKRayan Krishnan4:32

      Uh, it's honestly been a journey. Um, and, and I think, like, the, the real, uh, the goal North Star we think about is we, we never wanna be, um, kind of a lagging indicator or a delay to a model release. Um, and, and so that means we have to move really, really quickly and extract the most possible signal with the, the rate limits or capacity that we have, we have. And so early on, what this looked like was my co-founder Lynx and I pu-pulling an all-nighter, uh, to try and get as much done as possible and, and get results out the door. Um, now we've built up a team, but we've also really invested heavily in infrastructure, and so we're able to run evaluations in a massively distributed way, um, r-running effectively the, the maximum possible rate limits with, with every model we get access to. Um, and we also have this in-internal system called Steve, uh, Steve the Economic VALS, uh, Employee. Um, and, and so that, that's been a mechanism by which we're able to actually take more of the human work over time and, and put it into Steve.

   6. BHBen Horowitz5:24

      And how do you deal with the kind of issue that it's a little bit of a, an AI complete problem in that, um, you know, we still aren't really good at evaluating humans, um, or we're, we haven't agreed on it. Like, there are things like IQ tests, there's EQ, there's, you know, the Big Five personality and so forth, but there's not really an agreed-upon, um, framework for which we do it, and people have issues with things like the SAT and this and that and the third. And then, you know, of course, models are really good at hacking the benchmark, [chuckles]

5. 6:00 – 10:00

   The Limits of Evaluation: Making Fuzzy Evals Explicit

   1. BHBen Horowitz6:00

      um, as-

   2. RKRayan Krishnan6:00

      Yes

   3. BHBen Horowitz6:01

      ... Llama 4 proved. And so how do you think about You know, that issue and what are you-- like, what's the limit of what you can achieve and then, you know, within that, how are you going about it?

   4. RKRayan Krishnan6:13

      Yeah, I think, I mean, I think the honest answer is that it's forcing a lot of the, um, more fuzzy or distributed forms of evals to be made explicit. Like, what, what is really the distinction between an associate and a partner at a law firm? Um, and there, there isn't a clear test or an eval for that in the human world. Uh, and so we, we have to, again, first establish a lot of that in, uh, these different, different enterprise or real-world e-e-workflows for us to be able to test models in the same way. Um, and, and I think long term, that will be actually the biggest bottleneck, our ability to take companies and their evals and, and make them legible, uh, because that's how we'll figure out what signal we hill climb on and where we actually adopt.

   5. BHBen Horowitz6:52

      Got it. Very interesting.

   6. JLJennifer Li6:55

      Actually, maybe one question for you, Ben, on just, like, how the industry has formed before, like, intelligence came through. Like, we're now measuring something that's very fluid versus before, like, when we were talking about enterprise software. Like, you know, there's Gartner rating on, like, 70 different metrics. Like, you can sort of stack rank on a, on a, on a quadrant to say this company has these features covered, these features not. But now it's like, you know, very jack frontier that's very hard to measure given industries. Like, what do you see, um, you know, one is the analogy to the past that, uh, lessons we can borrow, and what do you see that's really going to be the challenge and, uh, and missing pieces going forward?

   7. BHBen Horowitz7:34

      Yeah, it's a, it's a little bit reminiscent of the MPAA, right? Where it's like, what's art? What's porn? Where's the line? When is it R? When is it X? Um, and by the way, the definition of that has changed over time, I think. Um, things that, that used to be [chuckles] , uh, X are now R and, and so forth. And, you know, and then what's PG? What's PG-13? You know, all that kind of thing. And there is no-- and the, you know, the famous line is, "Well, I, I know it when I see it." And I think that, um, this one, I, I, I just think it's gonna be necessarily fuzzy, but there will develop norms over time. Uh, and, and you know, like if enough kind of people who run companies or run finance or run whatever it is kind of agree, "Yeah, no, that's a norm," uh, then, then I think it is. As opposed to kind of what we have a lot now in the open me-benchmarks, whereas if you can solve this specific problem, then you're at this level and so forth. I think that's, um, uh, y-you know, one, attackable, and then it's too narrow.

   8. RKRayan Krishnan8:50

      I think when you also look to some historic analogies, there's, there's a lot of lessons that you can take from them as well on what's gone wrong and what we need to avoid. Um, and I think, for instance, for at VALS, one very early decision we made was, uh, the decision to, uh, to never sell training data to labs. It's, it's often a place that we're pushed, um, when we start working with a new lab to, to actually source and sell for them a bunch of training data.

   9. BHBen Horowitz9:12

      Yeah, that's a lucrative business, yeah.

   10. RKRayan Krishnan9:14

       Yeah. Yeah. And, and actually a, a lot of that industry has, has now built these gimmick-style benchmarks as a mechanism to sell their data. Um, and so that, that's, uh, become kind of their go-to-market as well. But, um, you know, I think if you look at auditing as an industry, you end up with issues like Enron, where, um, if you have the same group who's responsible for doing the audit as well as also consulting and, and supporting the company, you have a mixed incentive structure and, uh, and then it just becomes pay to pass the audit or, or in this case, pay to, to win the benchmark. Um, and that's really not what the market benefits from and, and what we're trying to do.

   11. JLJennifer Li9:48

       And Rayan, today you have already a pretty, um, extensive catalog of different type of benchmarks. Some of them are more focused on, you know, um, specific industries. Some of them are more like consumer mental

6. 10:00 – 13:13

   The Recursive Self-Improvement Index

   1. JLJennifer Li10:00

      health related. Maybe first just talk through, like what are the benchmarks that are most popular and most like, you know, um, read upon, and would love to dive into one of them as well.

   2. RKRayan Krishnan10:09

      Yeah. Well, we've done a lot of work in, in kind of the economically interesting applications of models. Uh, our finance agent benchmark is used by a bunch of the big financial institutions to get a sense of how models are improving. Um, we also have a lot of good work in coding, so our Vibe Code Bench measures how well models can take a natural language prompt and build a full stack web application. And so, uh, that's been a, a keen way to track model improvements over the last nine months. Um, yeah, we're also doing a lot more experimental work. So, so one benchmark we released recently I'm very excited by is our Recursive Self-Improvement Index. Uh, it's a topic which a lot of the big labs have been talking about and, and, and starting to report on in their model cards. Um, but there isn't a shared language to talk about the RSI potential of models, and so we created this as an apples to apples way to actually benchmark across the models.

   3. JLJennifer Li10:53

      Yeah, I thought that one is a very cool benchmark. It's sort of all the rage in the research community of how do you measure, like, progress you can make through having more frontier models that you can just compound on the capabilities. Um, how do you actually go about, like, building this RSI benchmark?

   4. RKRayan Krishnan11:08

      Yeah. I think in a, in an ideal world, what you wanna do is actually take a frontier model and, and have it train the next version of itself and see what, where the delta comes from. Um, but obviously, that's very expensive and slow, and so what we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model. So there's some work around pre-training, post-training, harness level engineering, um, and then seeing w- in, in which mechanisms and behaviors the models are able to, to do very good research work and, and build something new and, and where they're struggling.

   5. JLJennifer Li11:38

      Very cool. Um, there's also, uh, cases where you have deprecated, uh, indexes and benchmarks. Uh, it's funny that I always, uh, watch this benchmark industry people like- Just like early diffusion model days, like you cherry-pick whichever image [chuckles] shows up the best and the most perfect.

   6. RKRayan Krishnan11:55

      Yeah.

   7. JLJennifer Li11:55

      Like benchmarks as well, like you pick something that's, you know, very popular but maybe already saturated, and you rank very well on, on that or score very high. But you took a very different approach in like if these benchmarks are saturated, you'll deprecate it. Like maybe talk us through the, the, the thinking around that too.

   8. RKRayan Krishnan12:10

      I think it's a necessity, and, and this is kind of the infinite game we're in. I mean, you're wearing our shirt, and so we have this unofficial motto, "Always a higher peak."

   9. JLJennifer Li12:18

      Love the shirt.

   10. RKRayan Krishnan12:18

       Um, yeah. So insofar as foundation model labs are hill climbing, they're searching for the next peaks to summit, it is our job to perpetually construct these next mountains for them to summit. Um, and I think that's also how the economy has naturally functioned. Over time, as, um, you know, agriculture becomes less important for our, our labor market, um, there are new forms of, uh, of, of labor that's required out of our, out of our population. Um, and so in the same way, we, we should expect our benchmarks to keep up with the new frontier for what we want models to do. Um, there's another component of retiring benchmarks, which I think is, is underappreciated, uh, which is that benchmarks should also be reflective of the current state of the world. Uh, so in the same way, if you're a lawyer, you have to, you know, retake the bar exam and get certified, or if you're an architect, you have to get your certification and, and, uh, or, or, you know, a doctor. Um, uh, we should also expect models to be tested on the current state of the world and what we know in medicine or what we, what we have as our set of laws.

7. 13:13 – 17:52

   Beyond Capability: Cost, Latency & Keeping Benchmarks Fresh

   1. RKRayan Krishnan13:13

      Uh, so in the instance of case law updating to something like legal research benchmark, that's a desire to create a benchmark more reflective of the current state of the world and also push the models in place we want to see them go.

   2. JLJennifer Li13:23

      And how has, um... I guess, one, like it, it used to be like we're, we're, um, doing these like multi, um, answer or multi- s- step questions to like just evaluate prompt an- answers. Now there's like a lot more agentic, uh, work that's happening, uh, whether it's on finance or legal, um, or coding especially, like there's a lot of, um, you know, um, async background agents that can just complete tasks. How has that changed sort of how you build infrastructure, how to think of evaluating, um, the capabilities of not just the models and agents themselves, and there's also like a lot more dimensions that people care about. It's not just like capability, it's cost, it's latency, it's like, um, you know, whether this model is flexible enough to, to, to address like broader, um, domains and tasks and so on. So how do you think about the additional parameters to, to what you evaluate?

   3. RKRayan Krishnan14:16

      Oh, yeah. There's, I mean, there's a lot that goes into that. I mean, I think on the infrastructure level, you have a whole new set of problems. Um, I mean, for instance, now we're testing models and their ability to run over hours, days, sometimes weeks. And so the infrastructure needs to be very stable to support evaluation over time. And, and if there is a, a failed request, we should be able to retry from that one and not redo the whole trajectory. So there's some simple, simple mechanisms in the in- infrastructure we have to think about. Um, but, but I think in general, what we've seen is evaluations, as they become more complex, have a fewer sample size, but a larger set of criteria or expectations of them. And so what I mean by that is a benchmark is largely some kind of input space of things you're trying to query a model to do, and a set of requirements or rubrics that you, uh, see in expectations of the output. And so early on, you have things like ImageNet, which have millions of, of images you're trying to see a basic categorization for. So it's a one-to-one mapping between an image input and a, a text label output. Now what we have is far fewer set of tasks, you know, generate me 50 full stack web applications, but a much larger complex mechanism for evaluating the output produced. And I think that trend is going to continue as we see more complex workflows evaluated with models.

   4. BHBen Horowitz15:24

      And, and do you, do, do you think that, um, it will become kind of an im-- a, a, a real time, uh, kind of mechanism like so for something like Open Router, which Stripe just bought, would Open Router look to VALS and say, "Okay, where should this next request go?" Or is this gonna be kind of strictly for like picking a model in an enterprise for a task?

   5. RKRayan Krishnan15:52

      Yeah. I mean, I think, um, you know, Open Router is a bit of a misnomer in that most of their usage comes from being a model gateway. And so it's actually up to their, their users to decide which models they want to use when. And that's because really the hardest part of routing is building the evals and trying to determine, uh, in what places a set of intelligences should be used for a particular application. Um, and so, so, you know, our effort in supporting enterprises and building evals has actually supported a lot of them in also adopting routers.

   6. ETErik Torenberg16:20

      And, and say more about why it's not only important to labs, but also existential for, for enterprise, and maybe just say more about how, how you guys work with, with enterprise.

   7. RKRayan Krishnan16:28

      Yeah, of course. Yeah. I mean, I think the, the lab side of this is very clear. Like, you know, if, if you're raising lots of money, investing heavily in building models, it's, it's essential for you to show, uh, why your model is, is getting better and then why this customer should, should pay a premium for them. Um, but what I think is still underappreciated is on the enterprise side, this is turning to be existential as well. Um, you know, I, um, I have a, I have a small anecdote related to this actually. You know, I was meeting with a company in, in the Fortune 10, um, and they-- the way that they've adopted Cloud Code has been with roughly $100 a day budget for their engineers. And, and so what, what I was hearing is that this has actually fundamentally changed how work gets done in this company, uh, in that there is a, a rate limit which resets at 4:00 p.m. And so, so the most productive hours of work are actually now 4:00 to 6:00 p.m., uh, when the rate limits reset. Um, but then there's this dead period in the afternoon when people go on walks or, you know, get a coffee because they, they just don't have the rate limits. And so I think what's really, um, you know, illustrative there is that, uh, you see that there is a misvaluing of intelligence happening at every layer of the stack. And so by that, I mean, you have engineers who have $100 worth of, of usage limits, and they don't really know how to apportion that to the greatest productivity for them. Uh, you also have this Fortune 10 company, which has kind of arbitrarily said they're going to allow $100 per employee. They've actually recently increased it to $300 per employee, so, uh, almost an employee's worth of salary in tokens for them to use. Um, and, and this is actually pretty arbitrary because it's, it's hard to quantify what the right usage limit should

8. 17:52 – 19:19

   When Token Spend Starts to Eclipse Salary Spend

   1. RKRayan Krishnan17:52

      be. Uh, but then also Anthropic is running on, on pretty narrow margins to support this. Um, and, and they have, you know, massive cost to, to serve these models. Um, and so I think we're in this world where it is still very unclear what ROI looks like and how to value this intelligence that's being used. Um, and so as we talk about the existential concern for enterprises, I think it is this, uh, kind of direction we're shifting in where token spend may start to eclipse salary spend. And, and so if, if this is such a meaningful line item in, uh, in your costs, you actually have to justify the ROI much more c- cleanly than, uh, you, you've seen over the last six months. Uh, and over time, as we were talking about, I think a firm really is just its evals. And so the ability for a company to make its evals legible, uh, in order to solve this ROI, uh, calculus, um, is gonna be the reason why that, that company wins out over the competitors in the long term.

   2. JLJennifer Li18:41

      And, and maybe just double-click on that, um, like similar question to, to why the labs can't do it themselves or it requires a third party, um, agency to, to rate it. I, I think it's, it's a lot m- uh, more, um, uh, understandable that, you know, you, you need this, that neutrality across the industry. But for enterprise, they will argue that they know the task the best for their customers. Like what, what-- h-how does like VALS come in to provide value, and maybe you can talk through sort of VALS Smith new product launch as well.

   3. RKRayan Krishnan19:09

      I would recommend a lot of companies to develop in-house expertise, um, but I think that should not be the only solution. Um, you know, there's this explosion of intelligence happening.

9. 19:19 – 22:40

   Private Repos vs Public Benchmarks: The Real Performance Gap

   1. RKRayan Krishnan19:19

      There are somehow still more foundation model labs, um, getting, getting constructed and, and each lab is also releasing more models than ever with many more hyperparameter options, and they exist within a complex set of harnesses and agents. So the option- I won't even talk about specific intelligence, this, this new paradigm that's emerging. Um, so there's, there's a growing set of intelligence options, and I think what we're finding is that we're still finding new places we wanna use AI models, and so the use cases are growing in complexity as well. And so I think if you're, if you're a company, you, you have a compounding set of, uh, complexity i-in the set of options, and it's very, very hard to develop the internal capability to do the evaluation. Um, and so to, to try and remedy that, we've started to release some products more openly for enterprises to use, uh, the first of which is called VALS Smith. Um, and so VALS Smith is, uh, is focused on code gen, the area we're seeing to be the highest, uh, spend in enterprise AI. Um, it allows any company to take their GitHub code base and build their internal coding benchmark from it to get a sense of what coding agents are gonna be the most performant, but also what's going to be Pareto optimal or the, the highest ROI for them to use. Um, and actually we, we use, uh, VALS Smith a lot at VALS and, uh, uh, we're seeing that a lot of the best enterprises and, and so-sophisticated ones are doing that too. I would expect that to be the direction the market moves as it rationalizes.

   2. JLJennifer Li20:36

      And w-what are some of the examples when you, let, let's say, benchmark on a private repo that it just shows very different, uh, performance, cost, behavior compared to, let's say, like using like Frontier model, using a, a, a public repo, uh, benchmark?

   3. RKRayan Krishnan20:53

      I think today it's still very unclear whether, uh, the best OpenAI model or the best Anthropic model is actually going to be best for your repository. And so we've seen a lot of non-intuitive examples where you actually have to run the eval to figure out what's gonna be the frontier performance for that repository. Um, but I think you also now see a very complex middle set of options in that, um, there's now, uh, Opus and Sonnet models from Anthropic, but also Luna and Terra, and Luna's very, uh, cost competitive. Um, MuSpark is also very cheap and, and, and 1.2 is very capable. There's also a growing ecosystem of open source models which, uh, companies can choose to self-host. So I think in this, in this messy middle, it, it's actually very non-intuitive what's the right fit, where we're actually seeing in a lot of cases Sonnet is more expensive than Opus because it is so token hungry.

   4. JLJennifer Li21:39

      Mm-hmm.

   5. RKRayan Krishnan21:40

      Um, and, and so I think if you were to operate based on, you know, use Sonnet where, where you feel like it's, it's, uh, applicable, you may actually end up spending more than you need to.

   6. ETErik Torenberg21:48

      And say more about how this evaluation framework will apply to knowledge work in other domains, or what are some examples I can think of?

   7. RKRayan Krishnan21:56

      I think coding is a sign for what's to come in every domain, and, and a lot of the primitives established there are carrying over to other places. You know, if you, if you have a very good coding agent, uh, chances are you have a model that can also make PowerPoint slides or DCFs in Excel, um, and, and, and with, with a high degree of capability as well. Um, I think what we need to leverage in a lot of these industries, though, is the existing repository of work that has been done as a mechanism to build evaluations. And so just as Ben was talking about, we, we, you know, haven't really solved the question of what is human intelligence, but I think in a lot of industries we have a sitting repository of data around what work has looked like, and it'll be the task of, of us and others to try and codify that into evaluations, uh, that can, that can stay dynamic and actually evaluate models where human work is being done.

10. 22:40 – 24:48

    How Vals Uses Vals: Token Maxing the Coding Tools

    1. JLJennifer Li22:40

       Maybe, maybe just tag along the, the earlier question. Um, how, how are you guys using VALS Smith internally to, uh, evaluate what's the best coding model for, for VALS?

    2. RKRayan Krishnan22:48

       Yeah, I mean, to be honest, this was actually born out of a problem that, that we saw as well. So, um, I wanted to do a token maxing experiment, um, and it was able to get, uh, unlimited access for our team for, for a month for some of the coding tools. Um, and so in, in retrospect, looking back, uh, we had some-- we had a lot of engineers spending between one to two billion tokens a day. Uh, I think peak day was one engineer sp-spending six billion.

    3. JLJennifer Li23:10

       [laughing]

    4. RKRayan Krishnan23:11

       Yeah. And it's also crazy because we-

    5. JLJennifer Li23:12

       Well, how much does that e-e-equal to, to dollars?

    6. RKRayan Krishnan23:15

       So, okay, and then I went back and did some math, um, and, and it looked like in that month we spent roughly $1.5 million worth of tokens. This is free, by the way. I... No.

    7. JLJennifer Li23:24

       [laughing]

    8. RKRayan Krishnan23:24

       Don't wanna... [laughs] Um, but it was actually, uh, 10X more we were spending in tokens than employee salary for that month. Um, so it's not even like, oh, this is, this is 50/50. It's 10X. Um, and it was interesting to debrief and see the, the places where people were using agents and, and this kind of, you know, insecurity to use models all the time everywhere. Um, um, and so what we were faced with is, okay, we cannot continue with this mode of operation for the next month. Um, how do we actually intelligently figure out what are the right tools we should use and in-- and for what teams and, and what projects? Um, and so we ran this experiment of looking at the work that was done. We, we looked through a lot of the traces. We looked through our GitHub repo and, and built out the VALS Smith tool, and we found some pretty surprising, um, insights. Like, like for instance, the Cognition Devin tool is actually very token efficient, and so that's a place we, we've chosen to adopt more. Um, and, and I think there's a lot of places when you can get better pricing models out of, um, subscriptions as opposed to token-based pricing. And so it's actually informed our strategy for how we, um, c-can actually effectively token max without spending $1.5 million per month.

    9. JLJennifer Li24:30

       Very cool. So is the current, um, operating mode that you're using like one, uh, I guess more token efficient harness plus model and then like On top of that, people have some more flexibility to, to use token base to, for some higher, higher or more challenging

11. 24:48 – 28:32

    Policy: What Should the Government Actually Do?

    1. JLJennifer Li24:48

       tasks.

    2. RKRayan Krishnan24:48

       Yeah, we, so we, we have access to all the tools. We, we give everyone access to everything. Um, but we auto-issue recommendations for any GitHub issue or ticket for where to begin their session, and that should titrate the actual usage depending on the intelligence required for that task.

    3. JLJennifer Li25:04

       Very cool.

    4. ETErik Torenberg25:06

       I want to segue to the policy side for a sec- for a second, 'cause we, we talked about how quickly benchmarks become obsolete. In policy it's, it's even worse in that laws move, you know, m-much slower re-relative to capabilities. Ben and Mark spend, you know, a bunch of time, um, in DC and, and with policy makers to try to close that gap. So, so given that, who, who should define the standards here? Is it labs? Is it independent evaluators like, like you guys? Is it customers? Is it, is it government? H-How should this work from a policy perspective?

    5. RKRayan Krishnan25:33

       Yeah, I, I think the short answer is that everyone should be involved to some extent. Um, I think there's, there's a benefit from varied perspectives. I think the main issue, though, is that policy conversations as they've happened over the last couple of years have been very abstract. Uh, and there, there's been no material grounding to, to figure out what policy should cover. Uh, and so even when you have proposals from labs to have, uh, a third-party testing, uh, company or ecosystem, um, it isn't actually made explicit what the behavior and mechanism by which they work is. Uh, and so I, I view our role, especially early on, is to just be in evidence-gathering, uh, mode, where, where we're able to pull a lot of information and empirical data about what models, uh, are capable of and where the risks are, and that can go on to inform, um, you know, a, a more sophisticated conversation about policy.

    6. BHBen Horowitz26:19

       And, uh, how do you think about, like, who does what? Because the government kind of actually did the first evals and is continuing to do evals in terms of, um, okay, what's at the frontier and needs to be regulated, right? So they started with, you know, some crazy idea with 10 to 26 FLOPS or some such thing. Um, and so when you think about it, like, what should the government be doing, um, you know, to put it in this 30-day wait period or 60-day wait period or whatever it is, and then what should happen in that wait period, uh, and how does that intersect with what you're doing, and what's the right way to determine whether a model is on the frontier or not and needs to be, um, put in some special box [chuckles] for a while to make sure it doesn't break into everything? Like, how do you, how do you think about, like, how that relationship works?

    7. RKRayan Krishnan27:16

       I think there's e-effectively two countervailing forces that, that has to be considered. The first is the desire to move very quickly and ensure that the, the government process isn't slowing down, uh, the rate of technological innovation. And I think the other part is to make, to make sure the technology, as it's developed, is in the best interest of Americans and, and people more broadly. Um, and so I think these are very tough to reconcile a-and often, you know, picking one means it's at the expense of the other. Uh, and so what I'd hope to see is that, um, by doing this evidence-gathering process, we can help policymakers inform what they believe technology should look like in order to be aligned to American interest, and it can be the job of third-party evaluators to develop the technology to actually test and enforce that. Um, because I think that will create a mechanism by which you can see, uh, advancement in methodologies for evaluation and testing in a way that actually keeps up with the frontier. It, it doesn't lag behind or slow down the pace of development.

    8. BHBen Horowitz28:13

       Do you think, um, in terms of that already and, and developing your evals, like, do you think, "Well, can we test to see how easy it is for this, to get this model to start reward hacking or, or that kind of thing, um, you know, and doing illegal stuff?"

12. 28:32 – 33:30

    Alignment, Reward Hacking & Models Gaming the Test

    1. BHBen Horowitz28:32

       Or is that kind of not in the scope yet, or how do you think about that?

    2. RKRayan Krishnan28:36

       Yeah, I mean, we think about this broadly under the category of alignment. Um, I think there's places where you see that borne out now where models that are being tested for one cybersecurity risk are actually reward hacking and figuring out other ways to, to get around it. Um, but what we're trying to evaluate is, are models aligned with user intent? And so in those places we're, we're actually finding evidence that models are exhibiting behaviors that, that are not.

    3. JLJennifer Li28:58

       And maybe that's a question for you, Ben, as well. Um, I guess h-how do you think about the right division of labor here? Like, what, what should, um, you know, government agencies, um, control or, and do themselves and, and w-where they should like, you know, partner, trust, um, you know, private, uh, companies to, to take care of? And where do you see the gap that's happening today?

    4. BHBen Horowitz29:19

       Yeah. So I, I think the government agencies do get a lot of warnings from, by the way, the big labs, "Oh, this thing is gonna biohack. This is gonna be a cybersecurity risk," and so forth. And so I think what the government needs to do is go, "Okay, if the model, you know, is, if the model is capable of it, and then can somebody, uh, kind of basically prod the market to actually do the illegal behavior," um, w- and, you know, kind of specifying exactly what are those things, um, that they don't want in the market, and then having a third party kind of evaluate that. So it's kind of does the, uh, government, um, you know, the government kind of has an inclination of what it's afraid of, be it biohacking or cyber hacking or so forth, but th-then there becomes the question of, okay, can the model do it, and then can you get the model to do it? And, and then somebody's got to actually evaluate those two capabilities, and I think the government is particularly ill-suited to do the latter, particularly over time. It's just not a good government function. But they're very good at setting the rules, 'cause they can enforce the rules. So I think that that's kind of the combination you want, that the government sets and enforces the rules, and that, um, a very competent kind of private company then tells them if the rule is broken. Um, y-you know, and kind of it's, it's been interesting to see, like, the large labs start to go, "Well, the model's got the capability, and you can get it to do the bad thing, so we're not gonna let anybody have it. We'll just use it and make sure that our people don't get it to do the bad thing." And that's-- And even that doesn't always work, so very-

    5. JLJennifer Li31:14

       Definitely.

    6. BHBen Horowitz31:14

       You know, we're in interesting times, I would say.

    7. JLJennifer Li31:17

       A-and to me, there's the gap of, like, what's the narrative and what actually happens in real world. Um, 'cause, you know, every setup in, again, like an enterprise setup is, is very different. Like, um, or, or just, like, people, however they use the models are very different. The narrative, uh, that connects to the actual examples are very rare, which is why we still talk about OpenAI and Hugging Face hack. Today, we still talk about, you know, what happened with Fable and, and AWS for, for like two months. [laughs] But, uh, a lot of times, like, you know, that's not really how, uh, the model's being deployed. The, the environment they're running on is very, uh, bespoke. So how, how to like, you know, really bridging those thoughts and again, set up the, the right, uh, environment and also, like, rule bases for adopting these models, I think also just requires, you know, someone taking, um, uh, the capability and taking, like, what's the, the, the, the guardrails and, and put it down to, to, to the ground so that people can like, you know, have the confidence using, using the models.

    8. ETErik Torenberg32:15

       Then say more about how exactly the policymaker should work with the evaluator. What information do they need? What, what... How, how should the relationship w-work so that it's most effective?

    9. RKRayan Krishnan32:24

       Yeah, I think, I think in the first order, there should be an, a mechanism by which, um, insights and data can be passed directly to relevant people in government. And so now we're regularly doing briefings for, um, executive and legislative branches, um, on what we're finding in capabilities and risk of models. And so I think first order, that helps, uh, people there get up to speed on what's going on and, and also track through what will be problems in the future. Uh, I, I think, you know, things are moving very, very quickly. It's hard to predict where, where things are going, um, but at least when you have data, you can start to extrapolate a trend. Uh, and then I think from there it's, it's up to the people in the legislative branch to decide where they, they want to see policy. Uh, and so it's not really our, our place to give recommendations like that. Um, but if they, if they see that there is significant risk in, say, mental health for, for people under the age of 18, uh, or biosecurity risk in the models that necessitates having a standardized way to curtail model release, um, then it's up to them to inform policy. Um, and I think then there's other places where the executive branch in the places like Department of Commerce or SEC is responsible for, for making sure that private companies are able to adopt and use the models in a way that's gonna be productive for the whole

13. 33:30 – 37:14

    The Geopolitics of Evals: Whose Values Get Embedded?

    1. RKRayan Krishnan33:30

       system.

    2. JLJennifer Li33:30

       I would love to probe on another angle just ar-around geo-geopolitical. Um, I often see evals being like a representation of sort of the value, um, of, of the model, the model developer. Like you kind of develop this rubric of what's embedded in, in the model. And of course, different, different countries, um, uh, and, and labs in those countries care about different things. Like, I mean, um, I'm born and raised in China. I use a lot of, like, Chinese, uh, open source models too. Like you, you still cannot let them, you know, just go freely talk about [laughs] CPC and, and all history there-

    3. RKRayan Krishnan34:05

       Yeah

    4. JLJennifer Li34:05

       ... uh, because, um, you know, what happens in China. Um, so, so how do you think about how evals, I guess, a-and benchmarks play a role in, like, standardizing or like, um, being treated by different, um, different model labs from, from different places?

    5. RKRayan Krishnan34:23

       Yeah. I mean, to be honest, from, from my, um, very idealistic perspective, um, I'm surprised to see so much investment in sovereign AI. Um, you know, if I was taking a God's eye view, it would be extremely inefficient to build all of these data centers and, and replicate, uh, this data engineering process and train these very large models, um, when, when in fact, you could probably consolidate a lot of these efforts. Um, but it seems like that's not the world we're in or the one we're headed towards, and, and there's actually increased efforts to build AI in a sovereign way. Um, and so I think that takes having a shared language to communicate about what the framework for evaluations are and, and, um, where we're gonna collectively align around the risks. Um, you know, I think there's actually a lot to learn from, from nuclear here as well. I, I think Reagan had this line, "Trust but verify." Uh, and so I think we're starting to see signs of trust in that, um, you know, Xi Jinping and Trump are going to be meeting, uh, next month. Um, but there is no, uh, clear way to actually do the verification part of this. Um, having the shared language of evals will allow us to say things like, you know, you, you have the, the right number of nuclear warheads, and in that example, there were also flyovers, so, so a mechanism by which a country could audit another country's, uh, nuclear stockpile by having flyovers. Um, and so I think similarly, if there's concern about the societal or even existential risk of AI, it will necessitate us, uh, constructing this shared language of evaluations to do the verification process.

    6. BHBen Horowitz35:43

       How, how do you think about, um, harmonizing a policy like that so that, [chuckles] you know, it's hard enough to do it in America. [chuckles] And then how would you think about kind of taking it global? Um, you know, because now you're dealing-- you're not dealing with enterpri-enterprise customers, you're dealing with governments, and those governments are competitive with each other. And h-how would you think about that working?

    7. RKRayan Krishnan36:15

       I would be naive to say I have the perfect solution to this problem today. Uh, and so I think there are baby steps in which we can start. Um, for instance, there seems to be a lot of talk about cybersecurity risk. I think the concern around biosecurity will become even more important over time. And so there are clear places where there'll be mutual interest in aligning around, uh, ways to, to prevent conflict around cyber or bio. Um, I-- in my opinion, I think long term, what's actually gonna be the most interesting is the recursive self-improvement possibility. And that's a place where you could see one country, one or one company, you know, kind of run away with it and produce models that we, we don't know much about or, or operating in ways that are u-unknown to us. Um, and so I think having a way to, in a joint way, describe this being the level of pace we're comfortable with or this being exceeding the pace of, of development, uh, as, as it relates to RSI, is gonna be super important. And that's where I think you see a lot of the researchers at closed source labs calling for joint conversations between governments today.

14. 37:14 – 38:59

    What the Benchmarking Landscape Looks Like Next

    1. JLJennifer Li37:14

       What do you think landscape will look like going from, from here, um, now that we have lots of different capabilities and, uh, capable models as well as like, you know, um, uh, countries that care about different, um, developing... I mean, everyone cares about RSI for sure, but like on, on the bio side or like the, the cyber side, people care about, um, you know, slightly different, different things, whether it's more offensive, defensive, and so on. Like, what do you think the landscape will look like and, um, how do you think about developing new benchmarks to, to keep up with, with that?

    2. RKRayan Krishnan37:47

       Yeah, I mean, we're, we're hyper-focused on, on building benchmarks that capture the frontier. And so, uh, insofar as we see new places for capabilities or risks at that frontier, we want to make that, uh, an actually well-documentable evaluation, um, on VALS.ai. Um, and, and I think it, it takes having increasing coverage over time. You know, for instance, I think in, in cybersecurity, a lot of our historical work has been done around, um, code vulnerabilities or memory leaks that may exist in code. Um, but actually a lot of the, the biggest concern or risk is in the infrastructure level. Uh, and so these are not things that are expressed in code, but take, um, simulating larger environments of enterprise cloud infrastructure or, or even grid infrastructure for us to be able to say this is what the offense or defensive capability of models is. Um, and, and so making sure mo- evaluations are reflective of those new case- uh, places is, is really important for what we do at VALS. Uh, and we believe that the, the most valuable form of this business will be one that's incentive aligned around doing really high quality evaluation, not supporting the intelligence development process or, uh, the, the process by which the models can actually improve on that side over time.

    3. ETErik Torenberg38:53

       Awesome. Thanks for coming on the podcast. It's been a great episode.

    4. RKRayan Krishnan38:56

       Thanks so much for having me.

    5. JLJennifer Li38:56

       Thanks so much, Ryan. Thanks, Ben.

    6. BHBen Horowitz38:58

       That was fun.

    7. JLJennifer Li38:59

       Uh-huh.

    8. BHBen Horowitz38:59

       Thank you.

Episode duration: 39:14