Y CombinatorIan Fischer: How Stilts Beat a Frontier Model on ARC-AGI V2
Poetic's stilts pair self-improvement with an inference harness, not fine-tuning; it topped ARC-AGI V2 at lower cost than frontier deep-thinking modes.
CHAPTERS
- 0:00 – 0:40
Why engineers should “try things” with AI every day
Ian opens with practical advice: the speed of AI progress means the best way to learn is constant hands-on experimentation. He shares a personal example of using a frontier model to build an iPhone app in a weekend after years away from mobile development.
- •AI capability is moving so fast that last year’s workflow is already outdated
- •Daily experimentation beats waiting for a perfect plan or course
- •Personal story: using GPT to rapidly build an iPhone app after a long gap
- •Mindset: don’t self-limit—test what’s possible and iterate
- 0:40 – 1:07
What Poetic is building: recursively self-improving reasoning harnesses
Jared introduces Poetic and frames the central question: how it differs from RL, fine-tuning, and “context engineering.” Ian describes Poetic as a recursively self-improving system that produces higher-performing reasoning systems on top of existing LLMs.
- •Poetic builds “reasoning harnesses”/agentic systems for hard tasks
- •Core idea: recursive self-improvement without re-training base LLMs
- •Positioning: complements frontier models rather than competing with them
- •Goal: make LLM-based solutions reliably outperform out-of-the-box models
- 1:07 – 2:08
Recursive self-improvement vs. training new models from scratch
Ian explains why many self-improvement approaches are slow and expensive: they depend on training new models repeatedly. Startups can’t win a “train-from-scratch” arms race against frontier labs with massive budgets and frequent releases.
- •Common RSI approaches require training new LLMs for each improvement step
- •Training costs: hundreds of millions of dollars and months of work
- •Risk: a new frontier release can obsolete a startup’s training investment
- •Poetic’s claim: faster/cheaper self-improvement by staying above the base model
- 2:08 – 2:39
The fine-tuning trap: spending big only to get leapfrogged
Jared and Ian outline why fine-tuning often becomes a strategic dead end for startups. The time and cost to gather data and tune can be wiped out by the next generation of base models.
- •Fine-tuning requires sizable datasets (often tens of thousands of examples)
- •Compute and iteration costs can become prohibitive
- •Model releases can surpass your tuned result before you ship
- •Without an alternative, teams face re-tuning cycles or failure
- 2:39 – 5:05
“Stilts” for LLMs: a model-agnostic harness that upgrades with new releases
Ian introduces the “stilts” metaphor: Poetic builds a layer on top of one or more models that reliably performs better than the underlying LLM. When a better base model appears, the same harness can benefit immediately, with optional further optimization.
- •Harness sits above one/multiple LLMs and improves performance on tasks
- •Compatible with new model releases—no need to rebuild from scratch
- •Can be continually optimized for the latest model
- •Designed to be cheaper than fine-tuning while achieving better results
- 5:05 – 6:37
ARC-AGI V2: going public by taking the top spot (with lower cost)
Poetic’s emergence from stealth centered on demonstrating strong benchmark performance on ARC-AGI V2. Ian describes outperforming a leading result shortly after it was released, while using a cheaper base model and reducing per-problem cost.
- •Poetic used ARC-AGI V2 to showcase capability on “hard reasoning” tasks
- •Claimed a notable jump over a then-top leaderboard entry
- •Built on a cheaper model tier yet achieved higher verified performance
- •Reported improved cost efficiency (lower dollars per problem)
- 6:37 – 8:40
Humanity’s Last Exam: surpassing Claude and keeping optimization under $100K
Ian explains Poetic’s results on Humanity’s Last Exam—expert-written questions intended to challenge even PhDs. The team exceeded the prior state-of-the-art and emphasizes the comparatively small optimization spend versus massive foundation-model training costs.
- •Humanity’s Last Exam: 2,500 very hard expert-written questions
- •Poetic result: 55% vs prior SOTA ~53.1% (Claude)
- •Optimization cost stated as under $100K
- •Team size: seven researchers/engineers achieving SOTA-level benchmark results
- 8:40 – 10:26
How the meta-system works: automatically generating and optimizing harnesses
Ian describes the Poetic Meta-system as an automated, recursively self-improving engine that outputs systems (code, prompts, data, strategies) tailored to a task. It can build a harness from scratch or optimize parts of an existing agent a startup already has.
- •Harness ingredients: code + prompts + data + multi-model routing/structure
- •Meta-system outputs task-specific reasoning systems for reliability/robustness
- •Automation replaces much of the human “insight work” in agent building
- •Can optimize whole agents or components (prompts, strategies, modules)
- 10:26 – 11:32
Beyond RL: shifting S-curves and the path toward higher ceilings
Diana frames Poetic as a new paradigm beyond the recent RL-driven performance wave. Ian explains the idea of shifting S-curves: base models improve, the meta-system improves, and the combined system keeps lifting achievable performance until saturation (or more).
- •Performance viewed as multiple S-curves: per-model and for the meta-system
- •As both layers improve, the effective curve shifts upward repeatedly
- •Poetic aims to “hit the ceiling first” by stacking capability on top of models
- •Long-term framing: eventually saturate, or push toward AGI/superintelligence
- 11:32 – 13:19
Automating prompt/context engineering—and letting the system study the data
Jared compares manual context engineering to what a recursively improving system might do. Ian says Poetic minimizes human tinkering: the meta-system decides what to add to context, whether to synthesize examples, and how to address failure modes—sometimes producing prompts that look non-human yet work.
- •Meta-system decides when to “context stuff,” synthesize examples, or restructure inputs
- •Poetic avoids manual “monkeying” with outputs to preserve optimization integrity
- •Prompts may be surprising (even containing imperfect examples) but still effective
- •Shift from “know your dataset” to “AI learns the dataset and failure modes”
- 13:19 – 14:50
Why reasoning strategies (code) beat prompt tweaks: the 5% → 95% jump
Ian distinguishes prompt optimization from deeper reasoning strategies implemented in code. He cites prior work showing minimal gains from prompt tuning alone, but massive gains when adding structured reasoning and algorithmic scaffolding.
- •Prompt optimization yields incremental improvements, but often not the main win
- •Big gains come from reasoning strategies expressed in code, not just text prompts
- •Example result: moving from ~5% to ~95% on a hard task after adding strategies
- •Takeaway: “harness design” can dominate performance vs prompt tweaking alone
- 14:50 – 16:17
How startups can use Poetic: early access, ideal problems, and “bitter lesson” immunity
Ian shares how interested teams can engage: Poetic is not publicly released yet, but offers early access signups. The best fit is a company with a hard, high-stakes task that remains unreliable despite strong effort—where Poetic can push reasoning and knowledge extraction beyond the base model.
- •Call to action: sign up for early access at poetic.ai
- •Best fit: tasks that remain unreliable/fragile after extensive internal work
- •Claimed benefits: improved reasoning + deep knowledge extraction
- •Strategic upside: avoid repeated fine-tuning cycles (“vaccinated against the bitter lesson”)
- 16:17 – 19:45
From YC founder to DeepMind researcher: career pivot and lessons for the AI era
Harj asks about Ian’s transition from a YC startup (Apportable) to a decade in ML research at Google/DeepMind. Ian recounts exploring robotics, realizing hardware wasn’t for him, and fully committing to machine learning—then closes with advice to build and test relentlessly with AI tools.
- •Post-acquisition reflection led Ian toward AI/robotics interests at Google
- •Tried robotics; pivoted away after realizing hardware complexity
- •Committed to ML research for ~10 years across Google and DeepMind
- •Advice: experiment daily, find capability boundaries, and build what you want
