The Twenty Minute VCThe Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO
EVERY SPOKEN WORD
70 min read · 13,585 words- 0:00 – 1:05
Intro
- HCHarrison Chase
I think we spend 30% of our engineering salary on coding tools. It's actually gone down for the last two months, while tokens, the number of tokens that we've used, has actually gone up.
- HSHarry Stebbings
Today, we welcome Harrison Chase, the founder of LangChain, multi-billion dollar company. It is a leader in the ability to create and monitor your agents. And today we welcomed him to the hot seat to discuss why it is so hard to build agents today, what no one knows that everyone should know about what is coming with agents. And then my favorite, some really kooky predictions which no one else believes will happen, but he knows will definitely happen.
- HCHarrison Chase
I really underestimated the demand for inference. Everything just works. [laughs] This market is so big. There's just so much demand for it.
- HSHarry Stebbings
Ready to go? [upbeat music] Harrison, dude, I'm so looking forward to this. I've been looking forward to it for a while, 'cause I remember Miles telling me about you when he was at Benchmark and just singing your praises, so wanted to make it happen for a while. So thank you for joining me.
- HCHarrison Chase
I'm excited to be here.
- HSHarry Stebbings
Now,
- 1:05 – 3:17
What Is LangChain and Why Are Agents Still So Hard to Build?
- HSHarry Stebbings
for those that don't know, 60 seconds, what is LangChain and where do you, like, sit in the stack?
- HCHarrison Chase
We build tools to make it easy to build agents. So we build developer tools primarily. We have an open-source framework, which basically connects the model to everything else that you need to build a useful agent. So tools, data, computation, things like that. And then we also have LangSmith. LangSmith is observability and evals. Agents go wrong all the time. How do you know how they go wrong, and how do you measure them so that you can increase their performance over time? That's what LangSmith does.
- HSHarry Stebbings
Okay. I ask stupid questions for a living, um, that's why I'm a podcaster. Why is it difficult to build agents?
- HCHarrison Chase
The hard part about building agents is, uh, they don't behave how you want them to behave. So, so the models are these, like, non-deterministic black boxes, right? And they take natural language as input, and then they output natural language. And inside the black box, they're also non-deterministic. And so what that means is that the input you get is, like, infinite in dimension. It can be anything. You put a chatbot in front of someone, they can type anything that they want, right? And so that makes it really hard to know what they're gonna type. And then once it goes into this model... And by the way, agents are not just one model call. There's a s- they're a series of model calls, and they're interacting with other tools. And so you have these small block- black boxes that are then put inside this large system, and you get basically these cascading types of errors. So you have this, this, this very big kind of, like, input space. You have this very big output space because, uh, two similar inputs can go in, and they can arrive at completely different kind of like, uh, answers. And the thing that's, uh, making all this hard is the LLM itself. Yes, they're very, very smart, but you don't know exactly what they're going to do, and they change answers based on small changes to the input. And so you build this system, but it's really non-robust. And so getting the system to behave reliably, like, that is the hardest part. Like, when we talk about agents three years ago, four years ago to agents today, they've gotten a lot better, and the main thing that's happened is, one, the models have gotten better, and two, we've figured out how to build systems that basically harness these, these models better. And, and that second part, the, the harnessing of these models is, is what we focus on.
- 3:17 – 5:52
Why Better Models Force LangChain to Delete Parts of Its Product
- HSHarry Stebbings
How does your business change in a way that people don't see as the models get better?
- HCHarrison Chase
The main way our business changes as the model gets better is we basically have to erase big parts of our product. Um, so what I mean by that, and, and Cursor said this really well as well, I think this was one of their actual operating principles as a company is, like, erase the product or delete the product, I think is what they called it, where basically, like, these models get better, and things that you had to do, um, six months ago you no longer have to do. And so this is very relevant in our open source. So our open source is all about basically the, the harness and the framework for taking these models and helping them do things. And the right way to take these models and help them do things changes every month, three months, six months, cho- choose your time period, but it changes pretty rapidly.
- HSHarry Stebbings
How do you build for such an uncertain future? Do you just have to get in bed with the labs to know the roadmap, so you don't waste six months building something that then they erase?
- HCHarrison Chase
There, there, there's no easy answer. On, on one hand y- what are you, what are you gonna do? Not build? Like, that's, that's what I like to say to founders who ask us. Like, "What do I, what do I do?" Like, stuff is changing, um, you know, the models are changing. Uh, uh, if I build something now, it might not be relevant in six months. What, what's the alternative? You just don't build anything? And so I think you have to make kind of, like, educated guesses as to what parts are gonna be around and what parts aren't gonna be around in the future. So for example, early on, the way that we would do tool calling in LangChain was by having the model generate JSON, uh, DSL, and then parsing it. And, uh, that was something that I think we all thought or knew was, like, a little bit of a hack at, at, at the time, but we had to do it because that was the only way to get it to, to actually steer things, 'cause you have to get it to do this type of tool calling. And then, and then a few months later, OpenAI came out with tool calling. And I think we recognized pretty early... And this is, like, a more native version of tool calling that's directly in, in, in, in the model and in the APIs. And I think we recognized pretty early on that that was the future, and so we invested a lot of efforts and just completely switched from, from JSON parsing to this tool calling. Um, and, and so I think you have to basically recognize what things are short-term hacks and what things are long-term kind of, like, will always be there. The models will never know your proprietary data.
- HSHarry Stebbings
Mm.
- HCHarrison Chase
They will never be able to, to kind of, like, actually get the output of, of APIs. You have to take their, like, API call, pass it to the API, get the output, and then continue in the loop. And so I think the system around the model, connecting it to other places will always be there. But a lot of the hacks we do around, uh, chain of thought or planning or tool calling, things that can just be much more closely integrated into the model or into the model API will go away, and I think you just have to be quick to recognize that and, and go
- 5:52 – 8:59
Why Neutral Agent Harnesses Still Matter
- HCHarrison Chase
on.
- HSHarry Stebbings
Why do we need neutral harnesses? We see, you know, bluntly, very brilliant agent builders from the models themselves. Why does your layer not just get eaten by them?
- HCHarrison Chase
I think the reason we need neutral harnesses is that there is a lot of state that you build up either before you build the agent or as part of the agent running. And the state can include things like memory, uh, whether that's short-term memory or long-term memory. Memory also comes in different form factors. It can be from interacting with the agent versus from the agent learning from the environment. And the reason that this state gets built up is the agent interacts, it goes out, it, uh, learns, it chats, it, and, and it learns things to make it better. I think, I think one of the most interesting things right now is how to make these agents do better over time a- a- and learn. And some of that learning and some of the doing better over time will be in the model weights, but a lot of it is outside the model, and a lot of it is in this, in this context. And right now, when you build up this context, you really don't want it to be linked to a single kind of like model provider. Um, and the reason for that is that then it becomes linked there. You can't go to other models. Being able to change models is very offensive and defensive. So it's offensive in the fact that you wanna be able to take advantage of the best model when it comes out, no matter who it's coming from. It's defensive in the fact that you don't wanna get locked in, so they can't raise rates, things like that. So you wanna be able to switch models. The harness is, uh, can be very tied to the model. It can also be neutral. The context right now is really intricately linked to the harness. So the harness's job is to basically take the model and interact with all the other things. All the other things includes data and APIs, but also includes memory. And so if you build this memory in a way that is linked to the harness, and the harness is linked to the model, then you're kind of stuck. You're kind of trapped between this state.
- HSHarry Stebbings
So the true value of the harness structure, I'm sorry again, is the portability of data that it allows the user?
- HCHarrison Chase
Yeah. Yeah. You, you wanna, you wanna basically not be locked into models. And if, if, if memory of context starts building up in a way that is specific to one-- how one model wants to interact with memory and wants to interact with context, that's very bad for portability, and then that puts you at a disadvantage building your, your business going forward. We talk with a lot of people about owning their intelligence and what that means, and there's a lot of different versions of that. The version that, that, that we think like owning your intelligence means, we th- we think there's a few different components of that. But one key component is having kind of like this, this open and controllable agent. Like you wanna be able to adjust all the different pieces. You wanna be able to adjust the model. You wanna be able to adjust the harness as well, because sometimes the, these, these model labs are training harnesses to be really good at the things that are in distribution for them, which oftentimes is coding, and coding is great, and there's a ton of use cases around coding, but there are a ton of use cases that are- aren't coding. And maybe they'll write code as part of it, but they'll u- they're using that as, as a way to do sales, as a way to do research, as a way to do these, the law, as a way to do these other vertical specific things. And so when you get out of distribution of what the, the model labs are training on, you wanna have control over like what the, what the harness does. And then you also want control over the context as well. So model, the harness itself, and the context make up kind of like this open and controllable, uh, agent architecture, and we think that's a big part of what it means to own one intelligence.
- 8:59 – 10:42
Why Harrison Is Massively Bullish on Vertical Agents
- HSHarry Stebbings
I just wanna back up a little bit. When we think about like, um, agent value, so to speak, some people say that the value is actually in the verification, like whether it did well or not, because that unlocks the next step, which is automation. If value is derived through verification, who wins? The harness?
- HCHarrison Chase
If value is derived through verification, I think the people who win are the domain experts, the people who are able to kind of like q- quantify what that value means, which is very hard. Like building evals is a very hard job. And so I think the, the, the... a lot of the value will accrue to whoever can kind of like, uh, write those verifiers, capture that domain knowledge. I'm, I'm massively bullish on vertical agents for different domains for, for this reason, because I think the domain experts who are, who are best suited to, to write those verifiers or build those evals as, as, as we kind of talk about it, um, they will be able to build harnesses and models. And it, it won't be just the harness, it won't be just the model. It will be some combination of them. So, so very practically, if you're getting started building a, a vertical agent, I would highly suggest you start at the harness layer. You do context engineering. Uh, you, you, you kind of like tweak the harness, and then when you've got users, then you start tweaking the model in post-training. I think, I think it's, it's much higher ROI to start at the context or harness layer first. And so I think we'll have all these different verticals. They'll be able to quantify what good looks to them, um, in, in, in a very domain-specific way, and that'll let them build, uh, not only like vertical agents, but also vertical agent experiences. Uh, different agents might have different experiences. And so I think like a lot of the value will, will... I, I'm massively bullish on vertical agents, and so we want to enable a lot
- 10:42 – 12:14
Are Companies Really Building Their Own AI Models?
- HCHarrison Chase
of that.
- HSHarry Stebbings
You said like if I was building that, I would start at the harness layer. We're seeing a lot, whether it's, uh, uh, Harvey or I think it's Ramp, um, build their own models. To what extent are they actually building their own models? Mike Cannon-Brookes said the last thing said on a show we released today that it's largely kind of respectfully BS, and they're fine-tuning open-source Chinese models. Is that true? And, and how do you think about that?
- HCHarrison Chase
I think a lot of it is they are post-training open, open weight models, absolutely. Um, I, I, I still think that's a valuable thing to do. I don't think there's, I don't think there's kind of like anything wrong with that. I think that, I think there's different like dimensions that you can go in terms of owning your intelligence. I think the cheapest way is you just control the context. You use an off-the-shelf harness, off-the-shelf model. The next level down is you start tweaking the harness, and, and that gives you more control than just the context, but it's a lit- it's a little bit, uh, uh, more expensive to do.
- HSHarry Stebbings
Mm-hmm.
- HCHarrison Chase
The next level down is, is you post-train an open weight model. And so you might take an, an existing, uh, open weight Chinese model and post-train it for your domain. And then, and then the very kind of like bottom layer is they actually train a model from scratch. I don't think, I don't think many folks are training kind of like a model from scratch. I think most are post-training. I think if you're training a model from scratch, you're probably at NeoLab. I think those are one of the, the few that are doing that. Um, but yeah, the Ramps and the Harveys of the world, they, uh, they have a ton of talent. They have a ton of data. They have, uh, a, a bunch of usage and are trying to think about really optimizing it, and so they've gotten all the juice they can out of the context, all the juice they can out of the harness, and now they've arrived at basically post-training of these
- 12:14 – 14:15
Do Specialized Models Actually Beat Frontier Models?
- HCHarrison Chase
models.
- HSHarry Stebbings
Can I ask you, the jury's kind of out on the efficacy of these specialized models in the post-training realm like we mentioned there. Some reports I've seen say they're massively more efficient and better, and some less so. How do you think this will play out?
- HCHarrison Chase
I think what we've seen in regards to post-training is that a lot of the post-training that practically makes sense right now is probably for like simpler tasks that make up a larger harness. So what I mean by that is like when you have a harness, when you have an agent harness that's doing interesting work, it, it, it's actually very rarely just one model running in the loop. That might be the core of it, but there's a bunch of other model calls that happen either as like guardrails or routing or sub-agents or sub-workflows or things like that. And so a lot of the places where we see post-training being, uh, efficient now are for simpler tasks, like classification or extraction or summarization that might be a tool the agent uses or a guardrail or things like that. I, I, I think those types of post-training right now make a ton of sense and will continue going forward. I think the question, uh, that is more unanswered is for this core model that's driving the loop, because the, the main way an agent works is there's an LLM running in a loop calling tools. And so the question is for, for this model that's driving that loop, will those be kind of like post-trained models or, or will those be kind of like frontier models? And I think this is where we see, uh, uh, uh, more of like a back and forth, where like, um, every time a new frontier model comes out, the, the, the better they are at just running this kind of like core loop. We do see people like Harvey and Ramp and others kind of like experimenting with post-training models for this core loop. And so I think Harvey recently in the past month or so announced something e- exactly in that vein. Um, I, I, I think that's an open question. I think, um-
- HSHarry Stebbings
If you were building your own vertical application, would you post-train your own or not?
- HCHarrison Chase
Not to start. Absolutely not to start, but like if we're talking like five years in the future, assuming that I'm getting a ton of usage, um, I think it could be a realistic possibility.
- 14:15 – 20:05
Where Does the Value in Model Routing Really Sit?
- HSHarry Stebbings
One of the things that harnesses also needs to do, I imagine, is model routing depending on request. How much value is in the model routing layer?
- HCHarrison Chase
Model routing is so interesting, and I think it's one of the hottest and maybe most divisive topics right now. Um, so, so, so a few thoughts. First, there, there's, there's a question of where the model routing should even be. Should it be at the harness level, or should it be in the gateway level? And this is, this is like getting a little bit into the technical weeds, but there's-
- HSHarry Stebbings
Well, what is the difference? I'm sorry.
- HCHarrison Chase
The difference is in the harness level, it's in kind of like the code of, of, of how it runs, and in the gateway, it's just an API call. So the benefits of putting it in the gateway level, because it's just an API call, you can drop it into any harness. You could put this, this, this routing that's happening at the gateway level into Claude Code, into, into any harness without having to change the code. The downside, and why I think we're more bullish on routing happening in the harness, as are, I th- I think Factory and Cognition have also written things about this, is that in the harness, you just have more knowledge of what's going on. You have more awareness of the state. When you have an API, it's generally assuming that it's kind of like an OpenAI, uh, chat completions API, and so it only takes in a certain amount of state. You have more state in, in the harness to know, hey, uh, like what's going on with the prompt cache? You wanna try to keep the prompt cache, uh, warm and, and, and available. You can also change the prompt more easily. So a lot of these models have, uh, different prompting strategies. And so if you do the routing at the, at the, uh, harness layer, you can more easily change the, the, the prompt. And so I think a lot of what we've seen is, one, this debate between kind of like gateway and harness and where does routing go. I think we probably believe it belongs in the, um, in the harness.
- HSHarry Stebbings
And just so we understand, the Open Routers and all the other providers, which there are several-
- HCHarrison Chase
There are a lot
- HSHarry Stebbings
... there are, they are in the gateway.
- HCHarrison Chase
They are in the gateway. A lot of... I, I obviously Open Router, incredible company, incredible usage.
- HSHarry Stebbings
Had them on the show. Love Alex.
- HCHarrison Chase
Alex is great. I think, uh, I think a lot of why people use Open Router and Vercel and other gateways is largely for the model access, so I think there's also a difference between a router and a gateway. And so I would, I would probably say most of Open Router's usage, and Alex obviously knows the numbers a, a lot better than I do, but if I had to guess, I would guess a, a lot of it just comes from being a gateway, not a router. So people specify they m- the model they want up front, and Open Router just provides a single API with a single API key for accessing that. The ter- in terms of like smart routing, I think that's actually being done like way less than people think. I think it... And I think when it's done, it's, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done on terms of how they do, they do, uh, routing. But I think right now, I, I, I pretty strongly believe that, like routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific.
- HSHarry Stebbings
I'm sorry. When I listen to you, it seems so unwaveringly obvious that routing should be at the harness layer, given the context and memory that's is stored there. When I have such an obvious response, I like to understand why am I stupid. What is the bull case for it being at the gateway layer?
- HCHarrison Chase
I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So one, it makes it way easier to port. So you don't... You can, you can take Claude Code. People love Claude Code, right? If you, if you, if you tell people, "Hey, you can't use Claude Code, you have to use this other harness," um, and there are other good harnesses out there, but let's pretend for a second that you love Claude Code. The, uh... And you want, you want to let them keep on using that, but you want to do some semblance of, of kind of like routing to maybe like a DeepSeek or, or an open source model or something like that. Then the only way you can do that is at the gateway layer, 'cause you don't control the harness. Side note, this is another argument for open source harnesses that you can control and modify. Um, and then the second thing is when, when people talk about kind of like routing at the gateway layer, I think they're oftentimes implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for, and I actually think Open Router has, uh, has done this to some extent. You could argue coding. Coding is like the, the most popular domain. That's where all the usage is going. Let's build a router that works really good for coding. Maybe we won't be like harness aware, but we'll be like use case aware. So we know different models are good for different things in coding, and we can put it at the gateway. The other thing you could say is like, "Hey, we've got this like general chatbot Like we're not trying to be best in the class at law or best in the class at finance, but we're just gonna be like a general c- chatbot across the company. And like we can know well enough that if you say, "Hi, how are you doing?" You don't need like Opus for that. You can use a cheap model. And so I think when people talk about routing at the, at the gateway level, they're implicitly kind of assuming that you're doing either like general purpose chat or coding, um, and, and you're not doing like a, a domain-specific thing. As soon as you get into a domain-specific thing, I really think it has to be at the harness layer. You're not gonna use an open s- you're not gonna use an off-the-shelf, uh, gateway router for that.
- HSHarry Stebbings
So in two to three years' time, you think the majority is within the harness given the benefits that come from that context, but for very horizontal tasks it will sit in gateway?
- HCHarrison Chase
I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general purpose horizontal task, like just general chat or coding, it could, it could maybe be in the gateway.
- HSHarry Stebbings
Is it a commodity? Like, does everyone just build it?
- HCHarrison Chase
I don't think routing is a commodity. I, I, I do think people will build it, but I think building it is quite hard, uh, even today. And, and I think part of the reason for that is the models just change so rapidly.
- HSHarry Stebbings
Mm.
- HCHarrison Chase
So like you can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out, and it's just better at everything. And so then the value of routing kind of goes down.
- HSHarry Stebbings
Do you
- 20:05 – 21:15
Can You Trust AI Evals?
- HSHarry Stebbings
trust evals?
- HCHarrison Chase
I think evals are useful, but I wouldn't trust them fully. And so what I mean by that is I, I, I think, I think a lot of the academic evals that are out there that are used when a lot of the models come out are not really that useful anymore. I don't- I think all the models that come out are kind of Benchmark maxed against those evals, and so you don't learn a lot. I think internal evals that you have and that you build for your domain absolutely can still be useful. And so I would, uh, I mean, this is the main reason that building evals is such an important part of building a vertical specific agent. I think you have to have some way of, of measuring things. And, and I think you can generally trust tho- those evals. Um, but even then, one of the most important things to do is just look at the data. Just like look at the data and see what it's doing. Don't rely too much on kind of like a single number. I think this is why you wanna have traces, you wanna look at what's going on. Maybe your evals aren't perfect. Evals are really hard. You can have some sense, but maybe models are doing better, but in just a different way that you didn't know how to capture before. And so I do think, uh, yes, I think you can trust vertical specific benchmarks, but you have to evolve those as well, and that's one of the main things that we've seen because these models just get better in different ways.
- HSHarry Stebbings
Can
- 21:15 – 23:13
Where Does the Value Actually Accrue in the Agent Stack?
- HSHarry Stebbings
I ask you, you mentioned traces there. If like traces are the like exhaust of agent work [laughs] , where is the value then? Is it the app, the observability layer, the model layer? Where is it?
- HCHarrison Chase
If you're building an agent, I think the value is at what makes that agent good at the domain or vertical that you're, you're working in. So what I, what I mean by that is, again, I think there's different ways to, um, to, to control what an agent does. There's the context, there's the harness, and there's the model. Having, uh, specific proprietary things for, for any of those can be an advantage. I think controlling the h- the context is, is the easiest and, and the most, uh, obvious thing to do when starting-
- HSHarry Stebbings
And that would be data
- HCHarrison Chase
... then controlling the harness. And that would be data, that would be memory, um, that would be specific APIs, um, specific tools that you can use. Controlling the harness is like a little bit harder, uh, but still you can still put a lot of your domain knowledge around what should do both routing and things in there. And then the model is, is, uh, is the hardest to control, but potentially the highest ceiling. The, the, the, the thing that lets you iterate on each of those is evals and traces. So, so there is a ton of value in these, but there's only value in that you like capture insights that you bring back to the context or bring back to the harness or bring back to the model, and that exact process can look different. So post-training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably gonna wanna run like another agent over those traces and then just, and then have that agent output kind of like learnings for memory. And then for bringing the- these learnings in the traces back to the harness, you're probably gonna wanna run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called Engine. It's an agent in LangSmith that runs over traces and tries to update code. And then, and then when you update these things, you're gonna need to know that it's actually better, and this is where evals come in handy.
- 23:13 – 24:36
How Continual Learning Changes Agents
- HSHarry Stebbings
Can I ask, how does continual learning change agents and how they work?
- HCHarrison Chase
Continual learning I don't think so much changes how agents work, but I think it changes the systems that you wanna have running around the agent. So what I mean by that is, is, is I think the core loop for a, a continual learning agent, the core inner loop maybe to say, will, will still largely be the same. It will be an LLM running in a loop, calling tools, interacting with the model and, and an environment. I, I think what continual learning adds is an outer loop, um, an outer loop that looks at the exhaust of, of agents and then, and, and then does things with it. And again, what, what it does could be different depending on, on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe-
- HSHarry Stebbings
What would you define it as?
- HCHarrison Chase
I, I would define it as a system that gets better over time, um, a- a- as it runs. And again, there's different ways it could get better over time. It could update the context, it could update the code of the harness, or it could update the weights of the model. Um, and the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.
- 24:36 – 29:05
Why the Agent Memory Layer Is Still Wide Open
- HSHarry Stebbings
If we look at the, the context layer, we mentioned data being a part of that. We're seeing data companies scale to hundreds of millions of revenue in- The fastest time ever. I mean, like ever, ever, ever, and everyone is selling data.
- HCHarrison Chase
Mm-hmm.
- HSHarry Stebbings
How do you expect this market to play out given your neutrality on not being involved in it? But being adjacent to it, will this be a market of consolidation in years to come? How do you see it play out?
- HCHarrison Chase
I actually think the, the data market is still, like, pretty early in the grand scheme of things, insomuch as we're talking about kind of like a context layer for agents. So what I mean by that is, like, there are a, uh, there are a bunch of databases that are-- ClickHouse is maybe, like, a really good example of one that's doing incredibly well recently. Um, I th- I think they're still largely focused on kind of like their, their, their database, their good old-fashioned database. They're good at storing data and querying data, but it's still, um, in, in pretty kind of traditional ways. ClickHouse existed pre, pre-AI. When, when we talk about kind of like a context, uh, layer for agents, and again, like there will be a bunch of data produced from like software systems and great, like I think like existing databases, ClickHouse, Mongo, Elastic, whatever, are, are, are great for that. When we talk about a context layer for agents, I think a key part of what, what exists there is, yes, there's a database, but I actually think there's a system around the database. I th- I think like a context layer for agents will look more similar to like an agent around a database than just a database, and the reason for this is that taking all of these learnings and ingesting them into a database, you need some sort of like intelligence to do that, and I think that intelligence is actually really, really hard. The hardest part about memory is actually not the database at all. When we talk to a number of, uh, companies, the way that they do memory is they just store it in a single file. Very, very kind of simple. The, the hard part is taking all of the observations that happen, whether it's from the user interacting with like the, let's say it's a traditional SaaS interacting with the, the software, or the user interacting with the agent, and distilling that into information that's valuable enough to go into this context layer. That's the hard part in memory right now.
- HSHarry Stebbings
Why is that so hard?
- HCHarrison Chase
Because it's tricky to get LLMs to do what you want it to do, and this type of like, this type of like meta-reflection I think is something that LLMs actually still struggle at a lot. Like reflecting on, um, on, on, on kind of like prompts and improving prompts, reflecting on agent architecture and improving agent architecture. Uh, we should build a benchmark for this internally. We ac- we do have some benchmarks for this in terms of Engine, which is our agent that's running on things. But I think this type of meta-reflection is still really difficult for agents. I think it's also generally like a hard thing for humans to do as well. Like, I don't know if we actually understand... Some people probably do, so I shouldn't say this, but I don't know if we understand like, like what, like what, what goes on inside our brain. Like, I feel like our brains are still kind of like, uh, uh, uh, complex things, uh, that we don't fully understand. And so if we're trying to now replicate this kind of like learning process in, uh, in, in, in, in, in a system, I think there's a bunch of different pieces. And so I think you have to... I, yeah, I, I, I think models aren't great at kind of like reflection, and I don't think we know what this system should look like. What should the inputs look, be? What should the outputs be? What should the harness for this, this agent be? What should the different workflows be?
- HSHarry Stebbings
Will the models be meaningfully better at reflection in 12 months?
- HCHarrison Chase
Probably. I mean, it seems like they'll get better at everything.
- HSHarry Stebbings
Do you think the speed of model improvement will continue in the same manner of acceleration that it has done, or do you think we are approaching some form of asymptotic plateauing?
- HCHarrison Chase
I think the speed of model improvement is something that's probably really tough to say without being inside of a lab. I, I, I would say I think there are a lot of tasks for which the model improvements will, will, will kind of like, uh, flatline in the sense that like it's already good enough. Like, I think there are a bunch of tasks where the models are just good enough.
- HSHarry Stebbings
Which would be like coding.
- HCHarrison Chase
Yeah, coding's getting there. Um, uh, like summarization, uh, routing tasks. Um, th- there, there are probably more difficult tasks and/or tasks that we don't even know to think about, know to throw a model at. I, I would probably say like this type of like meta-learning is one of those tasks that the models aren't good at now. I would imagine it's very important to the labs in terms of kind of like cracking RSI or, or, or things like that. Um, and so I do think the models will get better at that over time, and it's possible that that progress actually looks, uh, much faster than, than, than, than previous. Um, I think it's really tough to have an educated opinion without being inside of a lab. There are some tasks where it's saturated. There are other tasks where I think it will probably just increase faster than before.
- 29:05 – 32:06
What About Agents Today Will Be Obvious in 3 Years?
- HSHarry Stebbings
What about agents today sounds crazy that you think will be incredibly common in two to three years?
- HCHarrison Chase
I mean, I think this memory layer for agents is still massively unsolved. I think one thing about agents that is, is rare today but will be incredibly common in the next few years is, is this memory layer. So what I mean by that is if you look at like Instinct or Muse, I think one of the things that they've nailed is this kind of like personalization and learning and seeming like it's, it's something for you, whether it's because it's speaking in the tone or because it's just following up, like that proactiveness. And, and so I think, I think memory is super, super early. We've been talking with folks about memory for like two years and trying to do things in the open source. We, we, we launched an alpha and then, and then killed a, a commercial solution for memory. It's just so early on in memory, and it's so domain specific. What memory looks like for Instinct or Muse is actually pretty different than what it probably looks like for a company or org harness, and it's pretty different than what it looks like for legal or finance or things like that. So I think memory i- is, is one thing that really makes a difference. You see this make a differ-- I think Instinct and Muse have done a great job at this, and so you see how it makes a difference there. Um, but I think it's still very, very early on, and I would expect, uh, all products in a few years to have some aspect of memory in it.
- HSHarry Stebbings
With respect, is that not quite obvious?
- HCHarrison Chase
[laughs] I think, I think it's... I, I think the fact that memory will have an impact is quite obvious, and that's why you see so many people interested in it and have- having it, be interested in it for, for the past few years. I still think it's non-obvious what the answer is. I still think it's incredibly non-obvious what the answer is. I don't think anyone knows what the answer for memory is.
- HSHarry Stebbings
Why did you kill the commercial solution for it, given the obviousness of it and the clear need for it in, in almost everything?
- HCHarrison Chase
We didn't-- We killed it because we didn't find product market fit, I think. Um, why didn't we find product market fit? I think we actually tried to be too general. I think memory's so early right now that I think a lot of the solutions that you wanna build are actually very, like, domain or agent specific.
- HSHarry Stebbings
And so you would build memory spec- like vertically specific memory- Solutions? Uh, what would you do differently?
- HCHarrison Chase
Going back to what the hard part of memory is, I think the hard part of memory is that process for learning from facts about, about what to remember, um, and that agent that, that, that does that remembering. Um, as I said, I'm, I'm massively bullish in kind of like vertical agents. And so, uh, a- and I think having those agents be good at learning things specific to their domain i- is, is a key part of, of that. And so I do think these kind of like learning agents, at least to start, probably have to be domain-specific. Um, and, and, and I don't know if there's like a general... Like what, what does the, what does the harness for these learning agents look like? It probably looks pretty similar to a lot of the harnesses we have now. It probably looks pretty-- We build deep agents. Deep agents is an agent harness. Probably looks similar to that, an LLM running the loop, but it's got specific instructions and specific tools, and maybe there's some specific sub-agents that are really good at, you know, d- uh, navigating existing memory structures or, or distilling kind of like facts. And so I think you'll have, you'll have agents for remembering things. I just think they'll be domain-specific.
- 32:06 – 33:37
Will Every Consumer App Have Its Own Vertical Agent?
- HSHarry Stebbings
Do you think you will have vertical agents for consumer applications? I understand and I agree with you when you look at finance, accounting, sales, marketing, legal. You mentioned Muse and Instinct, they would be horizontal agents. Do you think we'll have vertical consumer agents?
- HCHarrison Chase
I think my take on the consumer space is that you'll have both. You'll have both horizontal and vertical agents, and the horizontal agents will interact with the vertical agents.
- HSHarry Stebbings
Where does it make sense to have a vertical agent versus not?
- HCHarrison Chase
I think for most apps it probably makes sense to have a, a vertical agent, and I think the reason that having vertical agents makes sense at all is, is, is probably twofold. May- maybe more, but two that jump to mind. Um, one, a lot of these apps have, uh, knowledge about you and can learn things about how you interact with that, that app better than a general horizontal agent could. Um, so back to the point about domain specificity, how, how you interact with music for, say, like Spotify, um, is, is, is very different than how you might interact with Strava, right? And so tho- those apps are actually probably best at learning things about you, and I think the best way to expose those learnings is probably through an agent. The other thing is, a lot of these apps have a ton of APIs. And so you could say, "Hey, you know, Spotify, don't build an agent. Just expose a bunch of APIs." I actually think it's better for-- It's a better experience if there is an agent that kind of like wraps those APIs and exposes a single, or maybe multiple, but like probably a single like natural language interfa- face for a horizontal agent to interact with, with that, with that vertical agent.
- 33:37 – 35:49
What Happens When Agents Generate 1,000x More Requests Than Humans?
- HSHarry Stebbings
How does the infrastructure change when agents are the fundamental interface with sites? And what I mean by that is, you know, um, I wanted tickets to a specific concert, and I said, "Check every five minutes and buy any seat that comes available," da da da da. And for three days, it checked every single five minutes. The volume of queries was a, a thousand times what it would have been with humans. How does the infrastructure required change in a world of agent interaction not human interaction?
- HCHarrison Chase
When agents are interacting, uh, it, it, it, it makes the number of requests go way up. I think, I think you saw a story about how, I think it was Resy that blocked a, a user's, uh, account because they were using Instinct to, to pull it, uh, I don't know how frequently, but pretty frequently. A- and, and so I think, yeah, the, the infras-- There's just way more strain on, on the infrastructure. How, how does that percolate down to lower layers? I, I don't know if I have an opinion on that yet. I think you have to build systems that can be much more bursty, that can scale wa- way broader. I don't, I don't know if I have more specifics other than that.
- HSHarry Stebbings
Do you think providers will allow agent usage or not?
- HCHarrison Chase
I think the future is providers exposing their own agent and having people either interact with that agent or having people's horizontal agents interact with that agent.
- HSHarry Stebbings
So my horizontal agent, sorry, just so I understand this, would interact with Resy's vertical agent.
- HCHarrison Chase
I think that's exactly right.
- HSHarry Stebbings
Why would that work? 'Cause that still doesn't solve the two core problems of advertising for a lot of these businesses. They lose their ads model and then upsell. They lose the like anything else you might like also on the side, so you get reduced basket size. Why, why would agent-to-agent work?
- HCHarrison Chase
I think you could include some of those things in the agent response. Um, so you could, you could do advertising for specific things on, on Resy or Etsy or whatever and include that in the agent's response. Like, "Hey, I found these three things. You might also wanna consider this other thing." That's now a natural language response that goes back to kind of like the, the main agent. I think also to the point around the strain on the systems, if, if, if you have a vertical agent, it's gonna be better and more efficient at interacting with the underlying systems than a third-party agent will. So I think it will make it a lot more efficient for, for them to do that.
- HSHarry Stebbings
When we
- 35:49 – 38:44
Why Agent Costs Could Become a Huge Problem
- HSHarry Stebbings
look at the amount of queries and the volume that changes with agents, it, it takes a lot of tokens to do all of this work. How does the cost of tokens and cost of that intelligence change over time? 'Cause we always thought it would get cheaper. Hasn't got much cheaper. How do we actually think about that?
- HCHarrison Chase
Cost is one of the big things that we talk with people about right now. I think, I think there's a few different dimensions to this. One, I think like the, the frontier models are relatively getting cheaper over time. A- Astra is like price per performance fantastic compared to a lot of things that existed a year ago. Two, there's just way more demand. When you start talking about agents talking to agents or multi-agent systems, there are just so many more tokens that are getting burned, and so the costs are going up. Uh, we see this in coding primarily. Coding costs have skyrocketed, and I think a lot of organizations, including ours, are now thinking about ways to kind of like pare, pare those costs back. Um, how, how, how does this end up getting resolved? Uh, I mean, if you look at how we're tackling coding agents, I think there's a few hints there. So, so, um, when we're tackling coding agents, we think of it in three steps. One, we think of basically just, like, visibility, showing, showing kind of like how much things w- uh, will cost, how much things have cost to admins. Two is cost control, so you could say something like, "Hey, you know, go get me a reservation, but don't spend more than $5, like, doing it." Um, and then, uh, and then third is actually impacting the harness itself. So I, I, I think primarily a lot of harness engineering has been done around making the agent more efficient because that's the... Or sorry, a lot of the harness engineering has been done to make the agent just, like, better, more accurate, because that's the biggest blocker. Um, I think as cost becomes more a concern, we're seeing people care more about things like prompt caching, which are really good for kind of, like, reducing costs. We're seeing model providers actually make this easier by, and this is getting a little bit in the weeds, but they introduce things like where, where you can change the tools that are available to an agent mid-flight. Previously, this would have busted the cache and caused, like, a, a massive increase in, in cost. Now you can just add those dynamically, um, and, and keep the cost the same. So I think we're seeing more attention to both the model APIs and, and harnesses in terms of, like, controlling costs.
- HSHarry Stebbings
When you talk to large enterprises today, what do they matter most-- what matters most to them? Is it accuracy? Is it cost? Or is it security?
- HCHarrison Chase
Uh, another thing would be latency, by the way. So we run developer surveys every year where, where we ask developers what they care about. For the last two years, the number one response, uh, basically double the next two, has been accuracy. So that's still the, the primary thing. The next two are latency and cost, and security's, like, a, a little bit after that. Pretty close. Um, when w- when we talk to large enterprises recently, I'd say, like, cost has, has risen in terms of concern. I'd still s- probably say it's below, uh, below accuracy, but it's probably higher than, than latency. And, and then security is, for large enterprises, security is, is, is, is higher, but still probably below accuracy and cost.
- 38:44 – 41:41
LangChain Spends ~30% of Engineering Salary on Coding Agents
- HSHarry Stebbings
Can I ask you, what percent of your engineering salary, give or take, not specifically, but do you spend on, on coding tools?
- HCHarrison Chase
I think we spend probably roughly 30% of our engineering salary on coding tools. It's actually gone down the last, uh, two months. Um, while tokens, the number of tokens that we've used, has actually gone up. And so this has-- We've done this by doing, uh, three steps. First is basically just visibility. We tracked all of, all, all of our coding agent usage to LangSmith, so we could tell who was spending how much, what they were spending it on. We could do analytics over that. Uh, second is just basic cost controls. We, we set kind of like a budget, and, and if you, if you hit the budget, you can talk to our VP of engineering and, and he'll, he'll almost always, like, increase it. But it's just a safeguard against kind of like agents, you know, being used, uh, incorrectly or going in a, in a, in a bad loop. Um, and then third would be we control the harness. Um, so one of the things we've done is we actually, uh, built our own, uh, in-house cloud coding agent. So not, not a, not a, not a IDE, not a, not a terminal-based agent. We, we let people use whatever they want for that. But we built our own cloud-based agent. O-OpenSui is what we call it. Um, it's actually the only way I code anymore because I'm in Slack all the time, and so I just message it from Slack and it kicks off. But in that harness, we've done a bunch of work to control costs. Um, so things like prompt caching, but also things like routing we've built into this harness. And I actually think that's, like, a pretty repeatable three-step process for kind of like getting coding agent spend under control. Just first, visibility. Second, like, basic controls. Third, like, try to improve the harness itself.
- HSHarry Stebbings
Totally get that. I do just wanna unpack that. You mentioned that visibility, see who's spending what. Were the people spending the most the most effective?
- HCHarrison Chase
Sometimes the people spending most are the most effective, and other times they're not. So what I mean by that is we have a few senior engineers who spend a lot, and I, I 100% trust that what they're doing is well worth it. No questions there. We've also just had some cases where, um, due, due more to, like, kind of like, uh, silly mistakes or kind of like improper setup than anything. We've had, like, massive spikes in usage. So, so an example of this, um, we, we had, uh, one or two tasks where the, uh, some plug-in was being used that basically, like, would, would screenshot or, or, or take a snapshot of, of the DOM of a webpage and send that back every turn of the agent because they were doing some front-end engineering and they wanted to take what it looked like and just send it back. It would just run this massive loop. And so I think we spent, um, 40K in a week on, on that one thing. And so, you know, when it gets to that, like, again, like, you know, I, I think we're pretty kind of like, uh, liberal as a company and trust people, but I, I know that's probably not worth it. And so that's just, like, an example of, like, you know... A-again, it, it wasn't malicious. It wasn't, uh, it wasn't, um, it wasn't even intended usage, really. It was kind of just a mistake. And so I just think just, like, catching that is, is, is well worth it. So it depends a bit on, on the side of the spectrum.
- 41:41 – 43:41
Will AI Agents Make Engineering Teams Dramatically Smaller?
- HSHarry Stebbings
30% of salary, as you mentioned.
- HCHarrison Chase
That's a guesstimate, yeah.
- HSHarry Stebbings
A guesstimate. Totally fine. What is that in two years?
- HCHarrison Chase
I think the hard part about estimating coding agent spend is that orgs are just changing so rapidly. So what I mean by that is I, I, I think there's a future where orgs are much smaller and coding agent spend is, uh, you know, the same as, as human salary or, or, or more, or an order of magnitude more. And you have a small number of, of people that are ch- uh, driving and executing, uh, incredibly rapidly and incredibly performantly. And I th- and I think that's, uh, you know, a, a, an enticing vision in terms of, of, of what you think about. Um, I think-- I, I don't think many companies, including us, are, are probably there. I think we still see that, uh, we need a bunch of different people with a bunch of different backgrounds and a bunch of different- Expertise is collaborating on things, and there's a bunch of communication and overhead and review, and like yes, it would be ideal to have one person driving 100 agents and just like driving everything that like makes up, you know, a, a large part of the product. But I think we just see that it's still a very kind of like human, human... There's still a very human component to it. And so I spend a lot of time thinking about kind of like org design, and like is it possible to have like small teams, one-person teams, two-p-person teams that are kind of like driving agents. Um, I think it's very tempting, and in that situation, I would say like, you know, agents spend maybe 3X the cost of, of... But then also, you know, the engineers who can do that are incredibly valuable. And so maybe it actually will just stay the same because the people who can actually do that and who can drive all those agents, you're, you're, you're... There's fewer people, but you're paying them a lot more, and so the spend actually stays the same. So I think org design is, uh, very, very interesting right now. And so because of that, I think it's actually really hard to answer that question.
- HSHarry Stebbings
To me,
- 43:41 – 44:53
What Anthropic’s Valuation Depends On
- HSHarry Stebbings
that number is so central because it is the fundamental determinant of Anthropic's enterprise value. If that number goes to 5%, then the $2 trillion IPO might be a little bit overpriced. If that number actually goes to 300% like it is in McCall's case, then it's dramatically underpriced. And so that's why I focus so much on it. Do you agree that number underpins Anthropic's value?
- HCHarrison Chase
I think it's probably right that that number underpins Anthropic's value. I think the tricky part is that the only place we've seen agents really have effect is in software engineering so far though, and there are a ton of other places within an enterprise that I, I would argue have yet to kind of be affected by, by agents. And so I think it's possible. A-again, software engineering I think very difficult to, to, to predict, but, but I think there's basically two levers here. One is like what happens in software engineering. Is it the same as today? Is it more than? Is it less than? But I think, I think undoubtedly for the other domains, it will be a lot more than it is today. And so I think from that perspective, I think that's very bullish on Anthropic and OpenAI and other model providers.
- 44:53 – 46:46
Are Enterprises Afraid of Chinese Open-Source Models?
- HSHarry Stebbings
I think another question that a lot of people are asking is about kind of the proliferation of open models and how that changes the landscape. Um, you work with some very large enterprises. One of my dear friends, Jason Lemkin from SaaStr says that I'm not gonna do his accent because I'll butcher it, and I've done it before, and he got rather, um, he said it wasn't very good. Um, uh, do you find that large enterprises are fearful of working with open-source Chinese models, or actually they don't mind?
- HCHarrison Chase
We see that some enterprises are fearful of working with open-source Chinese models. We, we, we see others that don't, don't kind of like care.
- HSHarry Stebbings
Do you think that fear is justified?
- HCHarrison Chase
I don't really think that fear is justified. I, I think there's, um, I think there's different concerns. I think one concern, you know, is that, uh, the, the, y- the data that it runs on will, will somehow go back to the Chinese labs, and again, if you're, if you're using Fireworks or hosting it yourself, I think that's just ridiculous. I, I, I think there's other fears that, you know, the stuff that they've been trained on will cause it to respond in a way that's, uh, you know, not, um, consistent with the, the values of, of the enterprise or something like that. I think that's where evals come in handy, and I think that's largely unfounded. But like sure, maybe for some domains there's, there's some realness there. I, I think it's probably... So I think it's probably largely unfounded, but for some domains because of that kind of like response, like I could see there being some, some nuance there.
- HSHarry Stebbings
Do you think in the next 12 months we'll have comparable US open-source models?
- HCHarrison Chase
I think in the next 12 months we'll absolutely have comparable open-source, uh, US open-source models. I think you look at, uh, everything that NVIDIA is doing to kind of like, uh, work with the ecosystem. I think they, they care about it tremendously. I think, uh, you look at, uh, Reflection had a model launch recently. Um, I think it was still a little bit behind some of the Chinese models on some benchmarks, but I think there is a tremendous amount of attention in, in that domain. And so I think absolutely this will catch
- 46:46 – 48:05
What Most People Get Wrong About Open Models
- HCHarrison Chase
up.
- HSHarry Stebbings
Can I ask you, when you look at the state of landscape today in terms of open models, what does the world not see that you see very clearly given your position?
- HCHarrison Chase
I think when it comes to open models, I think there is a, a, a lot of usage of them, but I do think they're for more narrow tasks than people think. So, so I think we see a, a lot of people using more open-source models and more post-trained models than, than I would have maybe expected a, a year or year and a half ago. But I still think they're primarily for these simpler tasks like routing or guardrails or extraction or summarization. I don't think they're really driving the, the core agentic loops for a lot of, uh, the, the, these, these, these main, um, these, these main agents. And, and so I think open-source models are, are kind of divisive 'cause you have some people on, on one end who are like, "Great, they're, they're good enough. Let's use them for everything," and, and I don't think they're quite there. And then I think you have other people who are like, "You know, the frontier models are just gonna keep on getting better and better and cheaper and cheaper." And like, yes, that's true, but if you really know what you're gonna do, so you have to have a high-volume use case, and if there are a few components of that that are simple enough tasks, I think it's, I think it's, I think it's very good ROI to start kind of like post-training or y- or y- yeah, post-training models for, for those use cases. So I, I, I'd say, you know, unfortunately it's probably like a nuanced, uh, position that's somewhere in the middle of, of what I feel like the extremes on Twitter are.
- 48:05 – 49:31
Why Harrison Thinks the Inference Market Is Massive
- HSHarry Stebbings
How do you see the inference market playing out? You mentioned that Fireworks, uh, very openly we're a big investor in Fireworks, and I love Lynn and the team. But everyone's trying to do it. You, you ha- You know, we had Chase on the show from Crusoe who also have that as part of their business. You know, I'm friends with the Nebius guys. They also do it as part of it. How does this business play out, and how do you think about the need to do all of the stack versus inference-only providers like a Fireworks or a Base10?
- HCHarrison Chase
I think when it comes to how things will play out with respect to inference, one of the things that I... There's two things that I, I think I really underestimated, uh, over the, uh, from the start of LangChain to now. One is the demand for inference, and two is the demand for like coding agents. I think both of those markets are just tremendously bigger than, than I thought two, three, four years ago. And so I think, um, I think actually it's, it's Eric Vishria at Benchmark who, who said this on, uh, either a tweet or podcast recently where I think he basically said like everything just works [chuckles] . Like this, this market is so big. I th- I think, uh, Thomas Tunks as well, uh, had something around just like the market for inference is just so big that- You know, uh, uh, I think Eric said it about more broad stuff than just inference, but I think applying it to inference, I think any inference will work. That's like, you know, there's just so much demand for it. And, and, and I, I don't know if it's gonna get less anytime soon. And so, yeah, sure, Fireworks will work, AWS will work, Cursor will work. I just think there's so much demand for it.
- 49:31 – 54:31
Where Would Harrison Invest in AI Today?
- HSHarry Stebbings
Where would you not be investing today? Like, I, so I can give you s- some examples. I think the, the customer support space is, like, wildly uninteresting. 20 providers all funded with over 100 million in the last 18 months. You got Salesforce and Atlassian and big incumbents there too. You got kind of mildly old ones like Intercom and Talkdesk doing it too. [lip trills] Okay, cool. Or, like, robotics, like, uh, bold take, but there's so much money going into robotics. We're massively far away from having a ChatGPT moment. Humanoids, in particular, really struggle in form factor. I just think the funding and the excitement exceeds the capability significantly, would be my answer.
- HCHarrison Chase
I like to think I'm somewhat of an optimist, and so I, I, I think there-- I think, I do think this technology is transformative, and so I think there'll be ton of opportunity. Um, I, I think... A- and so I might answer that, and then through that we can back out what I wouldn't invest in. I, like, I, I, I think, uh, inference we already talked about. I think there's a tremendous, uh, there's tremendous-
- HSHarry Stebbings
Yeah, let's do this. You're, you're, you're-
- HCHarrison Chase
Yeah
- HSHarry Stebbings
... my newest partner-
- HCHarrison Chase
Yeah
- HSHarry Stebbings
... in the fund. Welcome.
- HCHarrison Chase
Yes.
- HSHarry Stebbings
It's so good to have you on your first day.
- HCHarrison Chase
Thank you.
- HSHarry Stebbings
Number one, we should have inference.
- HCHarrison Chase
We should have inference.
- HSHarry Stebbings
Okay.
- HCHarrison Chase
Um, there's gonna be way more software written, so we should also invest in, like, lower level, kind of like infrastructure primitives. Modal, incredible company. Uh, Temporal, incredible company. I think, uh, Supabase-
- HSHarry Stebbings
Do they not compete with Fireworks?
- HCHarrison Chase
... Convex. Um, Modal does on some things, yeah.
- HSHarry Stebbings
Okay.
- HCHarrison Chase
But they also do more. They do sandboxes, um-
- HSHarry Stebbings
So we've got inference layer, one, the coding layer.
- HCHarrison Chase
Co- coding, uh, or I, I would say inference and then, like, infrastructure. So, like, low-level infrastructure. Temporal, Modal, um, uh, Supabase, Convex. Um, vertical agents. I think vertical agents are, are, are going to be a good layer. I, I think, um-
- HSHarry Stebbings
Any specific verticals?
- HCHarrison Chase
I don't know if there's any kind of, like, verticals that I would overweight so much. I think the important parts are having, like, a domain-specific kind of, like, point of view and, and really diving deep there. I, I think Patrick Collison said this on a recent podcast, I think with Amjad from Replit when he was talking about, like, what the next, uh, what startup he would build. It, it, it's, it, it probably wouldn't be, like, a sexy area. It would be some area where, like, technology maybe, or at least AI penetration is low, but there's a lot of domain expertise built up there, and kind of going in there and, and really transforming that. I, I, I think the key, the key part is basically just having some domain expertise that can either be captured in data or logic that goes in the harness. Um, and, and eventually maybe the model weights, but, but I wouldn't start there. Having, having some domain expertise that can be captured there and, and then putting it to work.
- HSHarry Stebbings
Mm-hmm.
- HCHarrison Chase
Um, I think by the way, uh, there, there, there are maybe, like, three things that I would, uh, look for in all of these and I think are prereqs. If, if you look at all of the, uh, if you look at all of these teams that, that are succeeding, one, I think, like, the talent that they have is tremendously strong. Like, like Modal has an incredibly strong kind of, like, talent team. Lynn and Fireworks, right? They wrote PyTorch. Like, how cool is that, right? Um, two-
- HSHarry Stebbings
When we met them, you know, when you're kind of meeting founders, the last time you do the brief intro, it's like, "I built PyTorch."
- HCHarrison Chase
Yeah. [laughs]
- HSHarry Stebbings
Oh, well, that, that, that works. Yeah, yeah. Good tick in that box.
- HCHarrison Chase
So, so I think to build, like, especially this really deep infrastructure, you have to be, like, super technically strong, and there's just not that many people that, that can do that. Two, you have to go, like, really, really fast. Like, this, this, like, being able to iterate quickly I think is incredibly important. Um, uh, the market just changes so rapidly. I think, I think speed and execution speed... When, when I think about kind of like our advantages at LangChain, I, I think we iterate really, really quickly, and I think that's one of the, uh, advantages we, we have over other companies. And then the last thing would be brand. Um, I think, like, a lot of companies that, uh, uh, you know, made it in this AI area, AI era, basically caught lightning in a bottle and then, like, rode that wave and captured that. You look at, like, LangChain early on, like, we went very viral and, and have executed against that. Cursor, great example. They went very viral. Executed against that. Jev, uh, even Claude Code, ChatGPT. Like, it, it seems like there's just because-- I... It seems like the space is moving so fast, and because there are so many competitors, maybe this is even because of it. Because there are so many competitors, you know, there, there were, like, 100 different frameworks that came out in the, the two months after LangChain came out. But I think we had a, a brand advantage that was built up, and I think that was a durable advantage that we kind of, like, rode. And so Instinct, another example of this. There's probably a bunch of personal assistants. But, like, I think if there is a, a, like, in terms of, like, it'd probably be at, like, maybe higher valuations than is rational. But in terms of some of these funding rounds that are getting done where there's a, a breakout company with, like, a really strong technical roster that has proven that they can execute quickly, I think you just don't look at the price and you just invest.
- 54:31 – 55:32
Is Being First More Important Than Ever in AI?
- HSHarry Stebbings
Is being first more important than ever before?
- HCHarrison Chase
I think so, absolutely. Yeah. You look at-- I mean, it's possible to catch up. I think OpenAI does an incredible job of catching up with, with Codex. But being first, uh, Anthropic and Claude Code, that, wow, what, what an advantage that was.
- HSHarry Stebbings
But to your point, yeah, LangChain comes out, and then months after more and more come out. But Instinct, same, same. But actually the durable, um, brand monopoly that comes from being the haloed first is very real.
- HCHarrison Chase
I think that is, is, is super real. Um, I also think that you have to execute incredibly well after that. A c-counter example to what I just said, like just because you're first doesn't mean that you're guaranteed success. AutoGPT, I don't know if you remember the AutoGPT project, but it came out a, a few months after LangChain, was the fastest growing GitHub project of all time. Um, and, you know, a, I, I, uh, I, I haven't heard much, uh, from, from them since. And i- i- that's not a knock 'cause it's very hard to kind of like execute with speed, but I think like just because you're, you're first in an area doesn't mean that you'll, you'll kind of like survive.
- 55:32 – 56:18
The Bull Case for Jev at a $9BN Valuation
- HSHarry Stebbings
I never like to knock people, so I find it like helpful to like praise them instead in some respects. If you were to write a memo joining the firm for why we should invest a $9 billion Jev round, what would your bull case be for Jev?
- HCHarrison Chase
The bull case for Jev is that there's a new type of inference and new type of model, and they're, like, the, the model category is literally named after them. And so there's, uh, they're gonna ride that wave. Um, and they're, uh, uh, uh, they're a very talented team that can execute on that. Um, I don't know the team super well. I think that would have to be the assumption for, for a round at, at that value. From everything that I've seen, I, I, that seems to be the case, but, but that would be the bull case. They basically invented a new model category, are associated with it, and have a talented team that can execute on
- 56:18 – 58:15
How Benchmark and Sequoia Actually Help Founders
- HCHarrison Chase
that.
- HSHarry Stebbings
Does taking money from Benchmark change the trajectory of a company?
- HCHarrison Chase
100%. I, I, I think Benchmark when, when Miles and really the whole Benchmark partnership invested in us, uh, it was me and Ankush. Uh, we were two people. Um, we, we really thought of them as kind of like a, a, a third co-founder. I think there are, uh, you know, when, when I think about company building from, from two people, um, to where we are now today, which is like 400 people, they've been kind of like, uh, our, our most consistent kind of like partner in that, in that journey for sure. You know, I am, I am not a VC, I have never done VC. There's probably also other market dynamics where, great, Benchmark's involved in that Signals thing, and we're very lucky to have Sequoia join us after like two months, three months after the Benchmark round, and they've been a tremendous partner as well.
- HSHarry Stebbings
Who was it from Sequoia?
- HCHarrison Chase
Sonja from Sequoia, who's one of the most plugged in people in the spaces, I think.
- HSHarry Stebbings
How do their styles differ?
- HCHarrison Chase
So they're both very, very kind of like intelligent and sharp and, and ha- and have kind of like experience, real experience that they bring. I think Benchmark is maybe a little bit more like, um, what's the, uh, like what's the core nucleus of, of the, of the company and like that's, like we're, we're gonna be a sounding board for that and we're, we're gonna kind of like help with that. So, so what I mean by that is I feel like, uh, yeah, a lot of the company building things, positioning things, I would go, uh, I would go to d- them for and they would spend time there. Sequoia's style, I think, is they help with like everything. Like they are just a machine. So they help with recruiting of ICs, of executives. They help with marketing and launches, and they'll really operationalize.
- HSHarry Stebbings
Is it very well [beep]
- HCHarrison Chase
I, I, I think so, yeah. I mean, we've gotten a lot of our candidates from, from Sequoia references.
- HSHarry Stebbings
Really?
- HCHarrison Chase
Yeah. They ha- they have a talent network and we've gotten in a bunch of really, really strong candidates from there.
- HSHarry Stebbings
Would
- 58:15 – 1:00:45
Is It Worth Taking a Valuation Discount for a Top-Tier VC?
- HSHarry Stebbings
you say to founders it's worth taking a discount for Sequoia?
- HCHarrison Chase
I think absolutely it's worth taking a discount for Sequoia and Benchmark and, and probably other of these top one, uh, uh, tier or tier one VC firms, but those are the ones that we work with.
- HSHarry Stebbings
What do you know now about fundraising that you wish you could tell yourself when it was just the two of you?
- HCHarrison Chase
I think there is like this game to fundraising almost, and I, I, I definitely didn't know that going in. This m- this, I'm a first time founder. I also think I, I, I focus a lot, like wh- where I spend all my time is on product basically, and so I'm very focused on that. Um, I do think there is this game to fundraising where, you know, you, you, um, you wanna fundraise without fundraising. You wanna like, uh, once you get one term, you wanna get one term sheet and then, you know, tell others you have a term sheet and, uh-
- HSHarry Stebbings
Post a revenue graph on Twitter, and then it leads to all these inbound and-
- HCHarrison Chase
Yeah. It, it's-
- HSHarry Stebbings
Do you fucking like this? I fucking hate it. Maybe I'm too old for this game now, but like it's just posturing. It's like, "Harrison, do you really like me and wanna build a business with me? I'll give you the same price and the same money and whatever. Like let, let's just do this. Stop fucking posturing."
- HCHarrison Chase
I think, um, I, I don't like fundraising at all. I, I, I think, I mean, kind of what you were saying, like I, I wanna build something great with people I enjoy working with. I think like, uh, yeah, like we, we, especially for the scale of ambitions that we have, we need to fundraise to, to get that done. Um, I think, uh, you know, I, I think that was, uh, I think that the, that was the approach that, that Benchmark kind of took when, when we were fundraising from them as well. Like, "Hey, let's figure out like, you know, if, if we like each other and wanna work together." By the way, that was very good negotiating by them, and so I think they, you know, uh, they're very good negotiators as well. Um, but when we, when I look at like our most recent rounds, uh, that we've done, like we raised from Tom at, at IVP, our Series B.
- HSHarry Stebbings
Mm.
- HCHarrison Chase
I, I'd been chatting with Tom from like, uh, I think right around the Sequoia round for like over like two and a half years. And so that was like a relationship that we just built up over, over time, and I got to know him and he got to know me. And so when it, when it came time to, to fundraising, I feel like there was like less games in, in, involved there, and I really appreciated that.
- HSHarry Stebbings
I would love to do a quick fire round. So I say a short statement, and you give me your immediate thoughts. Does that sound okay?
- HCHarrison Chase
Let's do it.
- HSHarry Stebbings
Okay. What was your single favorite VC meeting?
- HCHarrison Chase
First pitch with Benchmark.
- HSHarry Stebbings
Why?
- HCHarrison Chase
So many legends in the room.
- HSHarry Stebbings
Who was the coolest to hang with?
- HCHarrison Chase
Uh, Bill Gurley.
- HSHarry Stebbings
[laughs] W- were you intimidated?
- HCHarrison Chase
I think I was too naive to be intimidated. [laughs]
- HSHarry Stebbings
[laughs] He's
- 1:00:45 – 1:01:47
The Most Important Thing People Get Wrong About Agents
- HSHarry Stebbings
also just like a big man. Like it's, it's an intimidating figure looming over you. Um, uh, what is the single most important thing you believe about agents that most of the ecosystem would disagree with?
- HCHarrison Chase
It's still really hard to build them, like really, really hard. Maybe people would agree with that, but I think it's just underrated how hard it is.
- HSHarry Stebbings
What have you changed your mind on most in the last 12 months?
- HCHarrison Chase
Market for inference is massive. I remember talking with the board about whether we should do a gateway a year and a half ago, and we didn't see what value we could add over something like Open Router, and I think that was 100% the wrong decision.
- HSHarry Stebbings
What's been the hardest technical thing for you and the team to build?
- HCHarrison Chase
We built our own database for storing agent traces, SmithDB.
- HSHarry Stebbings
How do you think about competing with ClickHouse and being a customer? Kinda weird or normal?
- HCHarrison Chase
I think there are 500 different companies that we are frenemies with, and so I think I've personally gotten very used to it.
- HSHarry Stebbings
[laughs]
- 1:01:47 – 1:02:55
OpenAI at $1.5TN vs Anthropic at $2TN
- HSHarry Stebbings
You can invest in one IPO, OpenAI at $1.5TN or Anthropic at $2TN.
- HCHarrison Chase
OpenAI.
- HSHarry Stebbings
Why?
- HCHarrison Chase
I think they have more diverse, different paths to really tremendous exits valuations.
- HSHarry Stebbings
Totally get that. What do you know now that you wish you'd known when you started in 2022?
- HCHarrison Chase
I wish I knew what harnesses today looked like so we could have built them earlier.
- HSHarry Stebbings
[laughs] Final one for you. What are you most excited for when you look forward five years?
- HCHarrison Chase
I'm really excited for what we, what creative things we can do with AI. I think a lot of AI, you know, and a lot of what we work on is, you know, helping enterprises be more efficient and, and delivering outcomes. I think creative things like Suno are, are tremendously interesting and, and I wish we'd see more of those.
- HSHarry Stebbings
Totally agree. I remember having Mikey on the show. That was a fun one actually. Um, and this has been so much fun to do, Harrison. Thank you so much for putting up with my waywardness and divergent schedules, but you've been an amazing guest.
- HCHarrison Chase
No, thank you so much for having me
Episode duration: 1:03:05
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode 1jDCPEHwQfI
