a16zHow OpenAI Builds for 800 Million Weekly Users: Model Specialization and Fine-Tuning
CHAPTERS
- 0:00 – 1:00
OpenAI Developer Platform at massive scale: API, ChatGPT, and why specialization is winning
Sherwin Wu opens with OpenAI’s dual focus: ChatGPT as a first-party product reaching ~800M weekly users and the developer platform/API as a broad distribution channel. He also sets the theme that the industry’s early “one model to rule them all” assumption is fading in favor of specialized models and deeper customization via fine-tuning and reinforcement fine-tuning (RFT).
- •ChatGPT as a first-party app for global distribution (~800M WAU)
- •The API/platform as another route to broad end-user reach
- •Shift away from belief in a single universal AGI model
- •Growing importance of specialized models and model portfolios
- •RFT as a key unlock for leveraging customer data more effectively
- 1:00 – 8:36
Sherwin Wu’s background: from Quora ML to Opendoor pricing to OpenAI’s API
Sherwin recounts his career path and how it shaped his product and ML intuition. He explains his current role leading engineering for OpenAI’s developer platform (primarily the API), plus exposure to government deployments like a local install at Los Alamos.
- •Leads engineering for OpenAI’s developer platform (API-centric)
- •Government/local deployments (e.g., Los Alamos) are part of the platform scope
- •Opendoor experience: pricing homes with ML and operational constraints
- •Quora experience: feed ranking, feature store work, strong ML culture
- •How prior pricing/ML experience influences thinking about platform products
- 8:36 – 12:12
Horizontal vs vertical OpenAI: how ChatGPT and the API coexist
Martin presses on the unusual structure of OpenAI as both a horizontal platform and a vertical app company. Sherwin describes internal principles from leadership: both surfaces matter, and growth plus mission alignment reduce the classic platform-vs-app conflict.
- •OpenAI runs both a developer platform (API) and a flagship consumer app (ChatGPT)
- •Potential platform tension: competing with API customers
- •Leadership principle: pursue both to maximize distribution of benefits
- •Growth reduces internal fear of enabling competitors
- •API reach can rival or exceed first-party reach depending on the moment
- 12:12 – 15:10
Why models resist disintermediation: ‘anti-abstraction’ and model stickiness
They explore why classic API disintermediation is harder in AI: users notice and care which model they’re using, and models don’t behave like swappable infrastructure. Sherwin notes unexpectedly high retention and increasing differentiation among models across use cases.
- •Models are difficult to abstract away behind software layers
- •Users can often tell when a model changes (capabilities/personality)
- •Industry moved from “interchangeable models” to acknowledging differentiation
- •Retention on a given model/API is higher than expected
- •Model divergence increases over time rather than converging
- 15:10 – 17:04
People build relationships with models—and products get tuned around them
Sherwin breaks down stickiness as both emotional and technical. End users acclimate to a model’s “feel,” while developers iterate deeply on harnesses, tools, and workflows that are uniquely optimized for a specific model’s behavior.
- •User familiarity and preference creates emotional/product stickiness
- •Developers build harnesses and tooling that align to a model’s quirks
- •Iteration loops make it costly to swap models later
- •Coding tools (e.g., Cursor) exemplify multi-model workflows
- •Specialized models fit different tasks: fast drafting vs deep reasoning
- 17:04 – 20:04
From ‘one AGI model’ to many: what proliferation means for OpenAI and the ecosystem
They revisit the once-common belief that AGI would imply a single dominant model and fine-tuning would be unnecessary. Sherwin argues the last few years have shown the opposite: model portfolios are emerging, and OpenAI benefits from a growing ecosystem—including supporting evaluation across multiple models.
- •OpenAI itself once expected one dominant model
- •Reality: multiple specialized models (e.g., Codex variants, GPT versions)
- •Proliferation isn’t necessarily bad; it reflects unexpected evolution
- •OpenAI benefits from ecosystem growth (“rising tide”)
- •Tools like Evals increasingly support multiple external models
- 20:04 – 22:26
Fine-tuning evolution: from SFT ‘tone changes’ to RFT for domain-leading performance
Sherwin explains why early supervised fine-tuning (SFT) felt limited—useful mostly for instruction-following and tone. Reinforcement fine-tuning (RFT) changes the value proposition by enabling substantial performance gains on targeted tasks, making customer data far more impactful.
- •Fine-tuning demand comes from enterprises sitting on valuable data
- •Early SFT was limited: incremental improvements, style/tone, instruction-following++
- •RFT enables running RL-style optimization for bigger capability jumps
- •RFT is more finicky and requires more investment, but yields higher payoff
- •Goal: achieve best-in-class performance for narrow domains (e.g., insurance coding)
- 22:26 – 24:44
Customer data choices and incentives: sharing data for discounts and free training
They discuss two practical fine-tuning patterns: offline ‘treasure trove’ datasets vs near-real-time product learning, with most customers currently using the former. Sherwin also describes an incentive model: customers can opt to share data in exchange for discounted inference and potentially free training.
- •Two data modes: offline corpora vs online/A-B-tested feedback loops
- •OpenAI’s ‘dream’ included scaling to many fine-tunes (e.g., LoRA)
- •In practice, offline expert-curated datasets dominate today
- •Optional data sharing can trade for discounted inference/free training
- •Data access is customer-controlled; OpenAI pilots pricing incentives
- 24:44 – 27:01
Prompt engineering isn’t the point anymore: the rise of context engineering
Sherwin contrasts early expectations that prompting would disappear with today’s reality: models are better at instruction-following, but success depends on assembling the right context. The focus shifts to tools, retrieval strategy, and orchestrating what information a model gets and when.
- •Old belief (2022): prompting will vanish as models ‘just know’
- •Reality: prompting remains important, but the work has shifted
- •Context engineering: choosing tools, data sources, and timing of retrieval
- •RAG as an example where ‘dumb retrieval’ feeds a powerful reasoner
- •Reasoning models can drive more intelligent tool use and multi-step retrieval
- 27:01 – 31:43
What an ‘agent’ really is: long-horizon action-taking as an interface to intelligence
Martin asks whether agents are a new modality; Sherwin reframes agents as one expression of OpenAI’s core ‘intelligence’ across multiple interfaces. He defines an agent as an AI that can take actions on your behalf over long time horizons, and explains why products like Codex became distinct surfaces (e.g., CLI) to better express agentic behavior.
- •Agent definition: action-taking AI operating over long time horizons
- •Agents aren’t treated as a single separate org/category internally
- •OpenAI products as interfaces: ChatGPT, API, Codex, Sora as manifestations of intelligence
- •Codex as a distinct interface where agentic behavior fits better than chat
- •Developers are encouraged to build businesses on agentic products/SDKs
- 31:43 – 36:46
Pricing intelligence: why usage-based persists, and why outcomes often correlate with compute
Sherwin explains OpenAI’s preference for usage-based API pricing as closest to how models are consumed and necessary given high COGS. They explore outcome-based pricing as an idea, but note it’s hard to measure for APIs and may approximate usage-based pricing anyway because test-time compute often correlates with value delivered.
- •Usage-based pricing matches consumption and helps manage high COGS
- •OpenAI prices from a cost-plus mindset to maintain responsible margins
- •Scaling billing systems for usage is technically demanding
- •Outcome-based pricing is appealing but difficult to implement broadly
- •Test-time compute/value correlation makes usage a proxy for outcomes
- 36:46 – 40:11
Open-weights strategy: why it doesn’t cannibalize the API and why inference is the moat
Sherwin shares OpenAI’s motivation for releasing open-weights (GPT OSS): ecosystem investment, brand, and enabling new use cases. He argues cannibalization has been minimal because open-weights serve different customers, and at frontier scale the hard part is reliable, optimized inference—where top labs retain an advantage.
- •OpenAI wanted to open-source earlier; compute and sequencing were constraints
- •Open-weights perception vs reality: not inherently anti-API
- •Observed cannibalization: essentially none; customers/use cases differ
- •Frontier revenue concentrates in a few large models that are hard to serve
- •Inference optimization and training-inference feedback loops are key moats
- 40:11 – 44:11
Different stacks for text, images, and video: why multimodal is operationally hard
They dig into the organizational and infrastructure realities of building both language models and pixel/video models. Sherwin notes the image/video orgs (e.g., Sora) run fairly separately with distinct inference stacks, while the API shares some common infrastructure until model-specific inference layers diverge.
- •Doing both language and diffusion/video is a known organizational anti-pattern
- •Sora/world-simulation teams operate with significant independence
- •Inference stacks differ materially between text and image/video workloads
- •API shares some general infrastructure, but inference becomes specialized
- •Separate roadmaps and ownership help parallelize progress across modalities
- 44:11 – 45:52
OpenAI’s multimodal API surface: DALL·E, Image Gen, and Sora in production
Sherwin clarifies that OpenAI’s API is not only text: it includes image and video generation products. He recounts DALL·E 2 as a personal turning point that drew him to OpenAI, and notes that Sora 2 in the API has seen strong demand and GPU pressure.
- •API includes DALL·E 2, Image Gen, and Sora 2 video generation
- •DALL·E 2’s release was a major catalyst for talent interest (including Sherwin)
- •Video generation is a ‘huge hit’ and drives significant GPU needs
- •Shared API surface, but separate inference approaches per modality
- •Continued expansion of multimodal capabilities in the developer platform
- 45:52 – 53:22
How the agent builder actually works: deterministic nodes for SOP and regulated workflows
Sherwin explains why OpenAI’s Agent Builder uses node-based, deterministic orchestration: it meets today’s reliability constraints and fits a major class of procedural work. He distinguishes exploratory knowledge work (e.g., engineering) from SOP-driven work (support, sales, marketing), where determinism and constrained outputs matter—especially in regulated settings.
- •Agent Builder’s node-based approach balances capability with practical reliability
- •Two work classes: undirected exploratory work vs procedural SOP-driven work
- •Many industries require strict adherence to policies and determinism
- •Regulated environments may require constrained response catalogs and validation
- •Structured prompts/logic (trees, pseudocode) help keep agents safe and predictable
