By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Invest Like the BestEx-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Building a token factory: scavenging compute, power, and uptime for cheap AI
- Neil Movva describes Sail Research as a “token factory” offering an API for open-source LLM inference at extremely low cost and “Sailboxes” for long-running agent workloads.
- He argues the future of agentic inference is long-horizon background work, where throughput and economics matter more than ultra-low latency for interactive chat.
- Movva explains key hardware/software trade-offs—especially batching versus latency, NVLink’s role in latency optimization, and why memory (KV cache and HBM) is a persistent bottleneck.
- He outlines an infrastructure thesis built around heterogeneous chips, small distributed data centers with lower uptime expectations, and opportunistic power sourcing (including intermittent renewables) to drive token costs down.
- On data and model progress, he says the internet’s text is largely exhausted as a free training source and expects verifiable RL environments to become the main engine of self-improvement, while open-source capabilities diffuse inevitably despite closed-model premiums.
IDEAS WORTH REMEMBERING
5 ideasSail Research positions itself as a “token factory” focused on unbeatable cost per token and agent persistence.
He frames Sail Research as an API that serves open-source LLM inference as cheaply as possible, plus “Sailboxes,” which are long-running cloud VMs designed for agents that run for hours, days, or weeks.
Movva believes the center of gravity shifts from real-time chat to long-horizon background agents.
He argues low-latency chat has driven the market, but the better future is agents operating proactively in the background so users aren’t waiting; on that shape of workload, raw tokens-per-second matters less than cost and throughput.
Throughput-oriented inference is a different optimization target than latency-oriented inference, and it changes the whole stack.
He explains a fundamental trade-off: batching improves GPU utilization (throughput) but increases per-request latency, likening it to taking a bus vs. a private car; Sail optimizes for the “best bus,” not the fastest car.
NVLink is crucial for low latency, but not the main unlock for cheapest tokens.
He describes how NVIDIA’s NVLink enables splitting work across GPUs to reduce latency, but with sublinear scaling and communication overhead; he sees NVLink as mandatory for low-latency inference, but less central for his objectives.
Ultra-fast accelerators shine on weight access, but KV cache dynamics limit an all-in low-latency approach.
He contrasts SRAM-heavy accelerators (like Cerebras and Groq) with GPU+HBM approaches, noting SRAM offers extreme bandwidth but limited capacity, while the KV cache grows dynamically and becomes a constraint.
WORDS WORTH SAVING
5 quotesMy job is to make the tokens as cheap as humanly possible.
— Neil Movva
We think that whenever you make something ten times cheaper, it's a new product category.
— Neil Movva
One way to say it is like the best latency is no latency at all.
— Neil Movva
I like the phrase that internet was a one-time subsidy on data.
— Neil Movva
There's no bad chips. There's only bad pricing, and I will make any chip work at the right price.
— Neil Movva
SPEAKERS
Neil Movva
guestEx-NVIDIA engineer and founder of Sail Research, focused on GPU/kernel optimization and low-cost AI inference.
Patrick O'Shaughnessy
hostHost of Invest Like The Best, interviewing founders and experts about business, technology, and investing.
High quality AI-generated summary created from speaker-labeled transcript.
