CHAPTERS
- 0:11 – 1:04
Cartesia’s origin: architecture research turned voice AI startup
Ankit introduces Karan Goel at NeurIPS, and Karan explains what Cartesia is and why it was started. He frames the company as a commercialization of their Stanford PhD work in “architecture research,” even though many people currently know Cartesia primarily for voice AI models used by developers.
- •Cartesia is ~2 years old and founded by former Stanford PhD researchers
- •Company roots are in “architecture research,” not just application-layer voice tooling
- •Cartesia is commonly perceived as a voice AI company building developer-facing models
- 1:04 – 1:33
What “architecture research” means (and why it still matters)
Karan clarifies architecture research as the search for better model “recipes,” not just scaling a single dominant design. He contrasts the industry’s transformer-centric scaling era with their grad-school focus on what breaks when models are pushed to their limits.
- •Modern AI is largely “data + architecture + compute”
- •Transformers became a powerful, widely standardized recipe
- •In 2019–2020 they focused on long-run bottlenecks as scaling continues
- •Motivation: new architectures for efficiency, long context, multimodality, interaction
- 1:33 – 3:33
Why transformers may hit a ceiling: efficiency, context, and human-like intelligence
Inspired by human efficiency and long-horizon behavior, Karan argues transformer limitations make them a poor endpoint for human-like intelligence. The goal is architectures that support long context, multimodal grounding, interaction, and actions over long timescales.
- •Humans are extremely compute/energy efficient (“intelligence per watt”)
- •Target capabilities: long context, multimodal integration, interactive use
- •Belief: transformer architecture has intrinsic limits for these goals
- •Cartesia’s work stems from exploring alternatives early in grad school
- 3:33 – 4:03
State Space Models (SSMs) as an alternative architecture family
Ankit asks how to place SSMs in the historical arc from RNNs/LSTMs to transformers. Karan positions SSMs as recurrent, state-based models that compress information differently, rooted in research pioneered by his co-founder Albert.
- •SSMs are recurrent models positioned as an architecture alternative
- •Framed in the lineage: RNNs/LSTMs → attention → transformers, then beyond
- •Co-founder Albert helped pioneer the SSM direction
- •Architecture research remains underexplored relative to scaling work
- 4:03 – 5:32
Intelligence as compression: abstraction vs. raw recall
Karan introduces compression as a core primitive for intelligence: to reason over massive information, models must form abstractions. He argues transformers skew toward retrieval over raw context, while compression pressure encourages consolidated representations spanning text, audio, vision, and physical concepts.
- •Compression enables abstraction and scalable reasoning over large information
- •Humans consolidate concepts across modalities (text, audio, physical world)
- •Transformers behave like “context window machines” optimized for recall/retrieval
- •Analogy: raw text file vs. zipped version—compression trades fidelity for structure
- 5:32 – 6:17
Retrieval vs. abstraction: transformer attention vs. SSM compression tradeoffs
Ankit probes whether attention’s query/key/value mechanism drives a retrieval prior, and Karan agrees. He contrasts transformers (raw access to all history) with SSMs (fuzzier compressed state), emphasizing the central tradeoff: fidelity for abstraction and long-range usability.
- •Transformers keep history in near-raw form and retrieve as needed
- •SSMs compress history into state, losing fidelity but gaining abstraction
- •Attention mechanisms naturally support retrieval behavior
- •The key tension: exact recall vs. compressed, reusable understanding
- 6:17 – 7:13
Hybrid architectures: combining extremes and searching for an “ultimate” design
Karan notes emerging hybrid models that blend transformer-like retrieval with state/compression-based components. He frames this as a step toward an eventual architecture suited for multimodal learning and long-timescale deployment—not just patchwork combinations.
- •Hybrids aim to integrate strengths of retrieval-heavy and compression-heavy designs
- •Many variants exist; implementation and inference details matter
- •Goal is an architecture that excels on multimodal data and long horizons
- •Focus on what ultimately works, not only incremental mixes of parts
- 7:13 – 9:21
Why voice AI is actually a multimodal beachhead (signal-to-symbol learning)
Ankit challenges the perception that voice is “single modality,” and Karan reframes multimodality as mapping continuous signals to discrete symbols. Even transcription becomes multimodal because it aligns audio signals with text tokens and meaning.
- •Multimodality isn’t just images/video—audio-to-text is inherently multimodal
- •Core multimodal problem: map a signal to discrete symbols
- •Audio-text tasks include generation, alignment, and semantic understanding
- •Voice is chosen as a grounded, tractable slice of a broader multimodal agenda
- 9:21 – 10:06
Audio as a recipe for other modalities: tokenization and representations as the common bottleneck
Karan explains why solving audio-text well could generalize to video and robotics: all are “signal → representation → model” problems. The shared challenge is how to represent continuous signals as tokens (or alternatives) for effective learning and prediction.
- •Across domains, the repeated problem is representing signals for training
- •Audio tokens, video tokens, and robotics trajectory representations share structure
- •Key research intersection: architectures + tokenization/representation learning
- •Belief: the right approach in one signal-text domain transfers to others
- 10:06 – 11:35
Beyond tokens: end-to-end learned representations and hierarchical abstraction
Cartesia’s research focus includes rethinking tokenization—moving away from heavily hand-engineered discretization toward models that learn internal representations directly. Karan argues this end-to-end approach is broadly transferable to many real-world signals.
- •Current pipelines often rely on hand-engineered tokenization/discretization
- •Goal: models learn representations internally, end-to-end
- •Emphasis on hierarchical abstraction learned within the model
- •Transferability: a general recipe for any continuous signal modality
- 11:35 – 13:54
Building for the “average human”: interactive agents, long context, long tenure
Karan connects the research agenda to concrete product goals: building highly interactive systems like call-center agents that improve over time. He distinguishes practical, context-rich competence from narrow “high-IQ” benchmark intelligence.
- •Product target: interactive agents with large context and action-taking
- •Call-center agent example: onboard quickly, operate and improve over years
- •Focus on everyday competence—people, systems, context—not just math/physics
- •Claim: limitations come from architectures/training/multimodality gaps, not inevitability
- 13:54 – 15:44
Research vs. startup reality: one vision, disciplined execution, and controlled exploration
Ankit asks about lessons moving from academia to a research-driven startup. Karan contrasts academia’s multi-vision exploration with a startup’s need for a single focused thesis, while still creating bounded room for exploration within that constraint.
- •Academia thrives on many independent visions and curiosity-driven work
- •Startups require one coherent vision to survive and ship
- •Challenge: preserve exploration without devolving into random research
- •High conviction plus sustained execution is treated as essential
- 15:44 – 17:21
Product as a truth serum: customers enforce intellectual honesty in research
Karan argues product requirements create rigor: customers don’t care about novelty or architecture branding, only outcomes. This pressure forces evidence-based choices—using new methods only when they materially improve user experience and performance.
- •They wouldn’t ship an SSM “just because”; it must win on metrics/users
- •Customers demand the best product, not the newest paper-friendly idea
- •Product constraints push experimental rigor and honesty about impact
- •Separates useful innovation from novelty for its own sake
- 17:21 – 18:14
Startup gravity applies to research companies too: YC lessons and execution focus
Karan closes by rejecting the idea that research startups are exempt from standard startup dynamics. He argues YC-style discipline and “startup gravity” still apply, even while selectively breaking rules when necessary.
- •Research companies still face core startup constraints and incentives
- •YC wisdom is applicable to research founders, not just traditional startups
- •Balance: disciplined execution with selective rule-breaking
- •Closing remarks and thanks
