No PriorsNo Priors Ep. 70 | With Cartesia Co-Founders Karan Goel & Albert Gu
CHAPTERS
- 0:05 – 0:27
Cartesia’s mission: a “rebellion” against Transformers with S4 & Mamba roots
Sarah and Elad introduce Cartesia co-founders Karan Goel and Albert Gu, known for S4 and Mamba. The framing sets up the core theme: alternative sequence architectures (SSMs) and how Cartesia is productizing them.
- •Cartesia’s founders are behind S4 and Mamba
- •Positioning SSMs as a serious alternative to Transformers
- •Episode focus: architecture trade-offs plus real products
- •Company context: research-to-production journey
- 0:27 – 1:32
Sonic today: ultra-low-latency text-to-speech for gaming and voice agents
Karan explains Cartesia’s first product, Sonic, a fast TTS engine optimized for interactive use. He highlights where latency matters most and sets a concrete goal of shaving hundreds of milliseconds off typical pipelines.
- •Sonic is designed for interactive, low-latency voice generation
- •Primary use cases: gaming NPCs/characters and voice agents
- •Early wins: ~150ms latency reduction vs typical stacks
- •Roadmap focus: pushing down end-to-end latency further
- 1:32 – 5:05
Founders’ backgrounds: Stanford PhD roots, DeepMind influence, and the S4 collaboration
Albert and Karan share how they came from the same Stanford lab and converged on sequence modeling research. They recount early collaboration stories and how S4 came together under deadline pressure.
- •Both founders from Stanford PhD group advised by Chris Ré (noted by Elad)
- •Albert’s path: DeepMind internship → sequence modeling alternatives to Transformers
- •Karan’s path: RL interests → exploration phase → S4 push
- •Origin stories: intense experimentation, infrastructure mishaps, and teamwork
- 5:05 – 6:41
State Space Models (SSMs) explained: sequential processing, compressed “state,” and fast updates
Albert describes SSMs as recurrent-style models that update a compact internal state as new data arrives. He emphasizes an intuition of “fuzzy compression” and connections to dynamical systems, contrasting with attention’s full-history access.
- •SSMs process sequences step-by-step, updating a maintained state
- •Intuition: compressed memory that strips irrelevant detail
- •Motivation from dynamical systems / mathematical elegance
- •High-level contrast vs Transformers’ explicit look-back over history
- 6:41 – 9:06
Which data favors which architecture: ‘pre-cooked’ text vs raw perceptual signals
The discussion shifts to modality-dependent trade-offs. Text is information-dense and tokenized, while audio/video are highly compressible and continuous—changing inductive biases and the best-fit architecture.
- •Text is dense and tokenized; each token carries meaning
- •Audio/video at raw sample/pixel level are continuous and compressible
- •Early SSM variants excelled on raw perceptual signals more than text
- •Transformers benefit from co-evolved pipelines (e.g., tokenization)
- 9:06 – 11:50
Efficiency and scaling: linear-time SSMs, quadratic attention, and why hybrids win
Albert highlights the efficiency motivation: SSM-style models scale linearly with sequence length, unlike attention’s quadratic cost. He then argues for hybrid architectures—SSM layers for most processing plus limited attention for exact retrieval—often outperforming either alone.
- •SSMs offer linear scaling; per-token processing can be constant-time
- •Attention cost grows with context length (quadratic scaling)
- •Transformers function like exact retrieval over all prior tokens
- •Empirical hybrid “sweet spot”: mostly SSM layers with sparse attention (e.g., ~10:1)
- 11:50 – 13:09
Early domain applications: hybrids on text, plus longer-sequence scientific data like DNA
Elad asks where these approaches are being applied first. Albert notes most experimentation remains in text due to demand, but mentions emerging work applying Mamba-like models to long DNA sequences for downstream predictive tasks.
- •Most hybrid adoption is in text modeling today
- •Example frontier: foundation-style pretraining over long DNA sequences
- •Likely downstream uses include classification and mutation-effect prediction
- •Acknowledgement: biology details are complex; focus is on long-sequence modeling fit
- 13:09 – 17:28
Why voice/TTS: streaming inference, real-time interaction, and edge deployment vision
Cartesia’s research direction meets a commercial wedge in audio. Albert and Karan explain why audio is a natural fit for efficient sequence models and why moving inference toward devices/sensors unlocks new applications and privacy/cost benefits.
- •Audio suits fast, streaming inference and large-context processing
- •Commercial pull: voice agents and real-time game interactions
- •Strategic thesis: push intelligence from cloud to edge/on-device
- •Edge compute enables filtering/understanding near sensors (e.g., cameras, microphones)
- 17:28 – 20:33
Second wave of AI: capability first, then efficiency—smaller models and new app assumptions
The hosts and Karan discuss industry trends toward on-device inference (e.g., Apple’s mention of ~3B-parameter models). The key argument: once models become cheap and ubiquitous, developers can assume far more “intelligent compute” per user interaction and build different products.
- •On-device constraints force smaller, efficient architectures
- •Two-wave pattern: discovery via scaling → optimization for efficiency
- •Running large Transformers locally hits memory/throughput/power limits
- •Cheaper inference changes product design (more calls, richer experiences)
- 20:33 – 24:04
Sonic launch and why TTS still has headroom: engagement, prosody, emotion, and role-based speech
Karan explains the product philosophy behind Sonic and argues TTS is not ‘solved.’ He introduces an engagement-based benchmark (“would you talk to it for more than 30 seconds?”) and points to missing nuance like emotion, intonation control, and profession/context-specific speaking styles.
- •Sonic showcases a general audio-generation stack conditioned on text
- •TTS progress is real, but engagement remains limited
- •Hard problems: prosody/intonation, emotion control, natural delivery
- •Nuance gap: speech style varies by role, geography, and social context
- 24:04 – 25:53
Audio needs language understanding: multimodality as a prerequisite, not a nice-to-have
Albert adds that even ‘basic’ TTS correctness can require deeper language understanding (e.g., pronunciations and context). This motivates building multimodal systems where modalities reinforce each other, using Sonic as a stepping stone toward a broader stack.
- •Pronunciation and correctness can require contextual language understanding
- •TTS isn’t fully modular; quality can depend on broader reasoning
- •Multimodal models may be necessary to do single modalities well
- •Sonic is positioned as an entry point to a wider multimodal platform
- 25:53 – 28:27
What’s next at Cartesia: more control, more languages, on-device Sonic, and multimodal assistants
Karan lays out near-term and longer-term roadmaps: improve Sonic, expand language coverage, add controllability, and bring performance to laptops and smaller devices. He also confirms plans to build large-scale multimodal models that reduce orchestration complexity and latency.
- •Short-term: improve Sonic quality/latency and add controls
- •Product breadth: multilingual support and better steerability
- •Deployment goal: real-time Sonic on-device (MacBook now, smaller later)
- •Long-term: multimodal conversational models with native audio in/out
- 28:27 – 29:30
Latency as a product constraint: reducing multi-model orchestration and ‘inelegant’ pipelines
Elad and Karan discuss how current voice stacks suffer from round trips across ASR → LLM → TTS and orchestration overhead. The ambition is to collapse complexity into fewer, more integrated models for responsiveness and cleaner system design.
- •Current voice systems incur latency from multi-stage pipelines
- •Orchestration adds engineering overhead and fragility
- •Integrated multimodal models can shrink inference time
- •Design principle: replace inelegant systems with unified modeling
- 29:30 – 31:24
Choosing research problems by ‘aesthetic’: elegance, ‘proofs from the book,’ and unification
Albert explains that an aesthetic sense of elegance guides his research direction—seeking simple, fundamental building blocks. The hosts connect this to mathematical ideals (“proofs from the book”), while noting ML often requires messy engineering trade-offs in practice.
- •Research taste driven by elegance and simplicity
- •SSMs originated partly from this ‘feels right’ intuition
- •Reference: Erdős ‘proofs from the book’ ideal
- •Tension: unify concepts, but accept real-world engineering complexity
- 31:24 – 32:48
Live demo: Sonic running in real time on a MacBook
Karan demonstrates on-device inference, showing Sonic generating speech instantly on a standard laptop. Elad prompts an Erdős quote to verify it’s live, reinforcing the edge-deployment and low-latency narrative.
- •On-device Sonic demo mirrors cloud feature set
- •Real-time streaming audio generation on a laptop
- •Ad-hoc prompt to validate live behavior (Erdős quote)
- •Edge readiness framed as a key differentiator
- 32:48 – 34:08
Team, hiring, and closing: building ‘Team SSM’
The episode wraps with Cartesia’s team size and hiring focus, especially for modeling roles. The founders invite candidates interested in SSM-based approaches and scaling research into production systems.
- •Cartesia at ~15 people plus many interns
- •Hiring across engineering, with emphasis on modeling talent
- •Culture theme: the SSM ‘rebellion’ / ‘overthrowing the empire’
- •Outro includes where to follow No Priors and find transcripts
