The Twenty Minute VCThe Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO
At a glance
WHAT IT’S REALLY ABOUT
Why agent value shifts to harnesses, routing, and domain evals
- LangChain positions itself as a neutral agent “harness” plus observability/evals (LangSmith), focused on making unreliable, nondeterministic multi-step agent systems measurable and improvable.
- As frontier models improve, parts of the agent framework become obsolete (e.g., hacks for tool calling), so the enduring value is in connecting models to proprietary context, tools, APIs, and portability of state across providers.
- Chase argues value in agent automation accrues to vertical, domain-specific builders who can define strong evals/verifiers and then iterate quickly from traces back into context, harness, and eventually model post-training.
- Model routing is not a commodity in practice: gateway routing is convenient for horizontal tasks, but high-performance routing for vertical agents must be harness-aware to use richer state (cache, prompts, tools, task context).
- Agent-driven demand will explode inference and token usage (especially in coding and agent-to-agent workflows), making cost control, prompt caching, and harness-level efficiency first-class concerns for enterprises.
IDEAS WORTH REMEMBERING
5 ideasAgents are hard mainly because reliability is hard, not because capability is missing.
LLMs are nondeterministic, and agents compound that uncertainty because they’re multiple model calls plus tools and external systems. This creates huge input/output spaces and cascading failures, making reliability (not raw intelligence) the core engineering challenge.
Better models don’t just improve products—they delete parts of them.
As models absorb formerly “hacky” scaffolding (e.g., JSON-parsing tool use replaced by native tool-calling), LangChain’s framework must rapidly evolve and sometimes remove features that are no longer necessary. The durable work is in connecting models to proprietary data, tools, APIs, and execution environments.
Neutral harnesses matter because long-lived agent state must stay portable across models.
State accumulates outside the model (memory, context, learned preferences, domain-specific behaviors) and becomes strategically valuable. If that state is tightly coupled to a single model provider’s harness or interfaces, customers lose offensive flexibility (adopting the best new model) and defensive leverage (avoiding lock-in and price hikes).
Vertical agents win because domain-specific evals/verifiers are the real moat.
Chase argues “verification” (evals/verifiers) is where domain knowledge crystallizes into automation, and domain experts are best positioned to define what “good” means. His playbook: start with context/harness engineering to reach ROI quickly, then post-train models later once usage and data justify it.
Model routing is higher-value (and harder) when it’s harness-aware and domain-specific.
Routing at the gateway (single API) is easy to drop into any harness, but lacks deep state awareness (prompt caching, tool availability mid-flight, prompting differences, and task context). For domain-specific agents, routing must be harness-aware; gateway routing can work for horizontal use cases like general chat or coding.
WORDS WORTH SAVING
5 quotesThe hard part about building agents is, uh, they don't behave how you want them to behave.
— Harrison Chase
The main way our business changes as the model gets better is we basically have to erase big parts of our product.
— Harrison Chase
The models will never know your proprietary data.
— Harrison Chase
I think evals are useful, but I wouldn't trust them fully.
— Harrison Chase
I really underestimated the demand for inference. Everything just works. This market is so big. There's just so much demand for it.
— Harrison Chase
High quality AI-generated summary created from speaker-labeled transcript.
