CHAPTERS
- 0:00 – 0:45
Why Braintrust exists: the persistent evals problem in AI products
Elad introduces Ankur Goyal and Braintrust, framing the company around helping teams ship reliable AI features. The conversation sets up evals, observability, and prompt development as the core pain points Braintrust targets for enterprise AI apps.
- •Ankur’s background (SingleStore, Impira acquired by Figma)
- •Braintrust positioned as end-to-end platform for AI application development
- •Core areas: evals, observability, prompt development for production AI
- •Examples of early customers shipping AI features (Notion, Airtable, Zapier, etc.)
- 0:45 – 2:43
From Impira to Figma to Braintrust: building the same tooling twice
Ankur recounts working on AI since 2017 and repeatedly running into the same problem: it’s hard to ship AI products that consistently work. The internal tooling he built at both Impira (pre-LLM) and Figma (post-LLM) became the seed for Braintrust.
- •AI product shipping remained hard both pre- and post-LLM
- •Internal tooling focused on evaluation and using real user data
- •Seeing the same needs at Figma validated longevity of the problem
- •Early user interviews strongly confirmed market pull for an eval solution
- 2:43 – 4:41
Market pull and the “don’t open source it” signal
Elad describes surprising customer feedback: teams wanted to pay for Braintrust and have it exist long-term rather than be open-sourced. Ankur explains that many early adopters had already tried building internal versions and discovered how deceptively hard evals are.
- •Customers had built in-house eval tooling and understood complexity
- •Evals aren’t “just a for-loop”—speed and analysis get complex fast (esp. with agents)
- •Standardization matters as AI becomes pervasive across an org
- •Braintrust aimed to encode best practices similarly to how Vercel docs shaped web dev norms
- 4:41 – 5:45
Demystifying AI quality: turning anxiety into an iteration loop
The discussion focuses on uncertainty about AI feature quality from prototype to production. Braintrust helps teams measure changes (models, prompts, architectures) and systematically improve outputs with an eval-driven workflow.
- •AI prototypes often look good on a few examples but fail in real usage
- •Evals help quantify how prompt/model changes propagate to user outcomes
- •Fast evaluation enables faster iteration and troubleshooting
- •Goal: a repeatable process to reach consistently high-quality outputs
- 5:45 – 7:57
What’s actually in production: RAG everywhere, fine-tuning less than expected
Elad asks what leading teams are truly using: RAG, fine-tuning, and agents. Ankur notes RAG is extremely common, while fine-tuning is better viewed as one optimization technique—and many teams have shifted away from it as frontier models got cheaper/better.
- •~50% of production use cases observed involve RAG
- •Fine-tuning is a technique; desired outcome is automated optimization
- •Fine-tuning was more useful when GPT-4 access/costs were limiting
- •Many customers moved from fine-tuned models to instruction-tuned/prompt-based approaches
- 7:57 – 8:57
Instruction tuning vs. fine-tuning: control, cost, and risk tradeoffs
Ankur breaks down instruction tuning as prompting with examples versus fine-tuning as weight updates. He emphasizes fine-tuning’s operational complexity and the risk of degrading real-world performance if done poorly.
- •Instruction tuning: behavior steered through prompts and exemplars
- •Fine-tuning: modifying/supplementing model weights
- •Fine-tuning is slower, more expensive, and easier to “injure” the model
- •Prompt/instruction approaches often deliver strong performance with faster iteration
- 8:57 – 10:39
Open-source models: interest rising, but production adoption still limited
The conversation turns to open-source versus proprietary model usage. Ankur sees an approaching “watershed moment” for open source (e.g., LLaMA 3.1), but notes enterprises prioritize UX and developer velocity over saving token costs—so adoption lags until open source wins on key axes.
- •Anthropic’s Claude 3/3.5 represented a major adoption step for proprietary models
- •Open-source adoption in production is limited, though interest is growing
- •Token costs often end up negligible or justified by ROI
- •Enterprises optimize for UX quality and iteration speed more than vendor ideology
- 10:39 – 14:43
Data infrastructure gets reshaped: from warehouses to embeddings-native workflows
Drawing on his database background, Ankur argues that the value and usage of enterprise data changes with LLMs. Instead of hoarding structured data for OLAP-style SQL exploration, advanced teams use embeddings and models to sift and curate text-heavy datasets for evals and improvement loops.
- •Internet-trained models can outperform enterprise-trained models on internal warehouse data
- •AI workloads are text-centric and not suited to ad hoc SQL exploration
- •Embeddings/models used to find underrepresented examples (e.g., support ticket edge cases)
- •Vector search indexing is solvable; the bigger disruption is traditional data-warehouse/OLAP architecture
- 14:43 – 17:44
Startup lessons carried forward: raising the technical bar and choosing the right customers
Ankur shares what he intentionally repeated from prior companies: demanding interviews and deep customer obsession. He describes selecting a specific early customer profile (leading AI product companies) so feedback generalizes and best practices can be built into the product for the next wave.
- •Hard technical interviews as a culture and capability filter
- •Hiring for engineers comfortable moving across the stack (even frontend candidates writing C++)
- •Customer obsession, but deliberate prioritization of “template” customers
- •Codifying best practices so later adopters can inherit proven workflows
- 17:44 – 19:28
Rethinking agents: from free-form loops to deterministic architectures with LLM calls
Elad probes common patterns among teams 12–18 months into LLM adoption. Ankur observes many are backing away from fully free-form agents due to compounding error rates and poor controllability, shifting toward deterministic control flow with LLM calls embedded throughout the system.
- •Early enthusiasm led teams deep into agent architectures
- •Free-form agents are hard to control; errors compound quickly
- •Trend toward deterministic orchestration with LLM calls as components
- •A ‘pervasive AI’ approach across the product codebase replaces the agent while-loop mindset
- 19:28 – 22:20
How AI teams are changing: from ML orgs to product-led AI platform teams
Ankur describes how early LLM adopters often moved fastest without legacy ML org structures. Over time, the winning pattern becomes a hybrid AI platform team combining product engineers with data/measurement rigor—starting with a flagship product and expanding to enable many teams.
- •Early LLM pioneers often had few/no ML staff at the start
- •Resistance from classical ML was partly emotional and identity-related
- •Modern AI teams blend product engineering with evaluation/measurement discipline
- •Teams evolve from building a marquee feature to an internal AI platform capability
- 22:20 – 23:35
Who to hire first for enterprise AI: product engineers before deep ML specialization
Responding to how an enterprise should staff AI adoption, Ankur recommends beginning with strong product engineers to identify what AI can change/cannibalize and to ship a usable experience quickly. More advanced investments (fine-tuning, open source for cost) come later after proving product value.
- •Start with product engineers to define AI-driven product transformation
- •Ship a good UI and simple AI engineering to validate the concept
- •Add rigor and advanced techniques after PMF signals emerge
- •Open source/fine-tuning become relevant later, not as day-one requirements
- 23:35 – 26:01
Why TypeScript is becoming the ‘language of AI’ (and frameworks are fading)
Ankur explains the shift he’s seeing in production teams: TypeScript increasingly dominates because AI is being built by product engineers and because types help sanitize uncertain model outputs. He also notes a trend away from AI-specific frameworks as AI becomes woven through general codebases.
- •Most Braintrust customers build AI features in TypeScript
- •Type systems help constrain and structure unpredictable model outputs
- •Python remains central for classic ML, but product engineering drives LLM app adoption
- •Teams are dropping heavy frameworks and integrating AI directly into core services
- 26:01 – 27:12
Enterprise vendor consolidation: AWS/Bedrock plus a few critical partners
Ankur observes enterprises reducing tool sprawl, often driven by AWS regaining momentum via Bedrock and Anthropic availability. Some stacks consolidate down to a small set of vendors (e.g., AWS + model provider + Braintrust) as procurement and operational simplicity become priorities.
- •AWS influence drives consolidation across infrastructure and AI tooling
- •Anthropic availability/quality on Bedrock accelerates standardization
- •Some enterprises narrow to just a few strategic AI vendors
- •Hyperscalers remain difficult to displace on infra-heavy layers
- 27:12 – 31:32
Operating Braintrust: CEO-as-builder, in-office speed, and interrupt-driven customer loops
Elad asks why Ankur still codes heavily as CEO; Ankur explains he architected the company to enable that while delegating sales/ops to strong partners. He also describes cultural choices—being in-office and reacting fast to customer feedback—to create a compounding product edge.
- •CEO time allocation shifts with modern enterprise expectations for UI polish
- •Company designed so Ankur can stay deep in code; early hire supports GTM/ops
- •Bias toward in-office collaboration and high-bandwidth iteration
- •Interrupt-driven culture: fix customer pain immediately as a differentiator
- 31:32 – 38:28
Go-to-market and the future of Braintrust: from evals to a universal AI developer platform
Ankur outlines an intentional early GTM: target ~50 leading AI builders as customers/investors, leveraging their influence. Looking forward, Braintrust expands from evals into observability and prompt/workflow development, with increasing use of LLM-based evaluators—potentially even evaluating sensitive data without exposing it to humans.
- •Deliberate targeting of influential early adopters to seed expansion
- •Evals led naturally into observability and prompt development inside Braintrust
- •Vision: hybrid GitHub-like developer platform for prompts + code + evals + logs
- •LLMs increasingly used as evaluators, including automated scoring on production logs and privacy-constrained contexts
