a16zHow Open Source Became AI's Backbone | Inferact with a16z
At a glance
WHAT IT’S REALLY ABOUT
How vLLM and open weights turned inference into AI infrastructure
- Serving LLMs is fundamentally different from traditional ML because inference must handle highly variable, non-deterministic workloads that demand sophisticated GPU scheduling, batching, and latency control.
- vLLM has become “critical infrastructure” by sitting at the junction of new model releases and new hardware, enabling day-zero support across many architectures while acting as a benchmark for chip vendors.
- Open weights are framed as essential for both control (latency SLAs, customization, security/compliance, guardrails) and, increasingly, cost management as token spend rises.
- Model licensing is evolving from permissive Apache-style norms toward commercial or threshold-based terms to create sustainable funding mechanisms for expensive training and repeated failed runs.
- The Hugging Face “guardrails” incident is used to argue that centralized proprietary moderation inevitably over-blocks legitimate work, pushing trusted users toward open-weight deployments where guardrails are configurable.
IDEAS WORTH REMEMBERING
5 ideasInference is a systems problem, not just an ML problem.
LLM serving must juggle variable prompt lengths, non-deterministic outputs, and tight latency expectations, which makes batching, scheduling, and GPU/TPU utilization the core engineering challenges.
vLLM’s defensibility comes from being the “meeting point” of models and hardware.
By supporting ~1,000+ architectures and collaborating with NVIDIA/AMD/Google/AWS/Intel, vLLM becomes both the day-zero launch surface for new open-weight models and a standard benchmark for new chips.
Open weights matter most when you need operational control, not ideology.
Users want to tune speed/latency tradeoffs, enforce data retention and compliance requirements, and meet SLAs (e.g., voice agents) without depending on opaque proprietary endpoints that can throttle or fail.
Performance tuning becomes a product surface with open inference.
Unlike proprietary “regular vs fast” switches, open deployments can expose many speed/cost configurations (including very high token/sec modes), letting providers optimize for different workloads and budgets.
Open-weight licensing is shifting to pay for the real cost center: training iteration.
As labs face huge up-front capex plus multiple failed runs, licenses increasingly include revenue/usage thresholds or commercial terms (e.g., Llama-style constraints, newer model-specific clauses) to sustain future R&D.
WORDS WORTH SAVING
5 quotesWe literally had this company called OpenAI, which- ... you know, it's become a little bit of a joke. It's not as open as it once was, or not nearly as open as it once was.
— Matt Bornstein
vLLM is a inference engine. That means its job is to turn available GPUs into a running endpoint for intelligence.
— Simon Mo
The world cannot just be controlled by proprietary APIs and where open weight, open development and research of these models are blocked or banned, right?
— Simon Mo
Right? Like, like I can't just like go home at night and like train a frontier open- source model with friends- for fun. Like we need millions or billions of dollars of computing resources enabled in order to do it.
— Matt Bornstein
If moderation is never solved, which is gonna be very, very hard, then there's always a place where you have a model where you know and trust that you are publishing to and to be able to use from.
— Simon Mo
High quality AI-generated summary created from speaker-labeled transcript.
