The Twenty Minute VCHow Many Will Actually Get Built & Is Energy AI's BIGGEST Bottleneck? | Positron AI Co-founder
At a glance
WHAT IT’S REALLY ABOUT
Why AI inference hits the memory wall, not compute limits
- Positron AI is building a full-stack inference hardware platform (chips through rack-scale systems) optimized for generative AI deployment rather than training.
- The episode’s core technical thesis is that inference is dominated by memory bandwidth/capacity constraints (the “memory wall”), whereas training is more compute/FLOPS-bound and parallelizable.
- Sohmers argues LLM token economics are misunderstood: KV caching makes many tokens extremely cheap to serve, enabling very high gross margins for frontier API providers if training spend is dialed back.
- He warns that “pace the frontier” regulation and anti–data center sentiment could unintentionally centralize AI capability into a small set of firms/governments, creating new monopolies and geopolitical risk if rivals do not slow down.
- Looking forward, he expects model capability and context windows to keep advancing via both hardware (more memory per device) and algorithms (e.g., sparse/linear attention), while pricing may eventually shift from per-token toward value- or agent-based models.
IDEAS WORTH REMEMBERING
5 ideasInference is a memory-bound, sequential workload—unlike training’s compute-bound parallelism.
Sohmers argues training is largely FLOPS/compute-bound because inputs can be massively parallelized, while inference is bottlenecked by repeatedly reading model weights and user-specific state for each next token in an autoregressive loop. That shift changes which chip architectures and system designs matter most (bandwidth, memory capacity, and data movement).
Hardware progress is increasingly gated by memory bandwidth, not raw compute.
He frames the “memory wall” as a decades-old structural issue: compute performance has improved far faster than memory bandwidth (he cites ~120× GPU FLOPS gain vs ~17× bandwidth over ~2014–2024). The result is widening inefficiency for memory-bound workloads like modern LLM inference.
KV caching drives enormous, underappreciated gross margin in LLM APIs.
Sohmers claims token pricing hides major economics: cached token processing is “essentially free” relative to recompute, so providers earn very high margins on cached reads even when they offer discounted cached rates. He uses this to argue API businesses can be highly profitable if frontier training spend is moderated.
At scale, user KV caches can rival or exceed model weights, reshaping memory hierarchy design.
KV caching reduces per-token compute by storing attention keys/values, trading quadratic attention recomputation for linear cache growth—but the cache becomes huge at long contexts and at scale across many users. He highlights a practical tipping point where aggregate user-session caches can exceed model-weight storage (especially for long-context, large-parameter models).
The biggest AI risk he emphasizes is regulatory-driven centralization and monopoly power.
He views calls to “pace the frontier” as risky because regulation may entrench a few labs and create durable monopolies—his core fear is concentration of capability, not sci‑fi extinction scenarios. He also argues the U.S. cannot effectively pace unilaterally if geopolitical competitors do not participate.
WORDS WORTH SAVING
5 quotesThe scariest thing to me on the political spectrum is that it's now become almost unifying issue on left and right about being anti-data centers, and I think that is almost entirely a Chinese PSYOP.
— Thomas Sohmers
It's absurd to me that the, the meme of these, you know, OpenAI, Anthropic, et cetera, are just burning cash and eventually they'll run out of cash that they can burn. Like, if they stop training, they'd be massively profitable overnight.
— Thomas Sohmers
The thing that I'm most worried about of any, you know, AI outcomes is that te- that, uh, technology and capability being concentrated to relatively few people.
— Thomas Sohmers
It's like the concentration of technological capability and, like, making legal to do matrix multiplications is, like, the thing that will set us back to, uh, you know, pre, uh, uh, uh, n- not just industrial revolution, it's like pre, um, uh, Enlightenment, you know, uh, capabilities.
— Thomas Sohmers
Fundamentally, that is a perfect track of our ability to produce and use energy, you know, discovery of fire up to, uh, you know, nuclear power plants. The, the simple, um, uh, tongue in cheek answer to your question is everything, uh, all, all progress is gated by, by energy.
— Thomas Sohmers
High quality AI-generated summary created from speaker-labeled transcript.
