YC Root AccessImproving Small Language Model Reasoning With A* Search
At a glance
WHAT IT’S REALLY ABOUT
A* search uses self-critique to boost small-model reasoning accuracy
- The talk introduces a test-time scaling method that treats LLM reasoning as tree search and applies an A*-inspired algorithm to small language models.
- Instead of relying on external reward models or large teacher models, the approach uses the model’s own self-critique score as a heuristic to guide which reasoning branches to expand.
- The method is positioned as a practical alternative to distillation (which requires training and teacher access) and MCTS-style approaches (which often need separate evaluators).
- Experiments on math reasoning benchmarks (GSM8K and MATH500) with Qwen 3 4B show higher accuracy than other test-time methods, with ablations suggesting better gains per token and per unit time.
- The speaker also previews ongoing work on causal reinforcement learning for better credit assignment in LLM agents via counterfactual reasoning.
IDEAS WORTH REMEMBERING
5 ideasSelf-critique can replace external reward models for search-guided reasoning.
Rather than training/hosting a separate evaluator, the same model scores candidate answers (e.g., 0–100), and that score becomes the heuristic that steers search toward promising branches.
A*-style prioritization prunes hallucinated or low-quality reasoning paths early.
When two nodes are at similar depth, the self-critique differentiates them, letting the algorithm expand lower-cost (higher-quality) nodes and avoid wasting budget on paths the model itself rates poorly.
The method targets the small-model sweet spot: lower cost/latency without giving up as much reasoning quality.
Small models are often preferable operationally, but weaker at multi-step reasoning; test-time A* is framed as a way to buy back reasoning performance without retraining or bigger models.
Improvements are shown on standard math reasoning benchmarks using a single fixed base model.
Results are reported on GSM8K and MATH500 using Qwen 3 4B, comparing multiple test-time reasoning algorithms while holding the underlying model constant.
Efficiency matters: gains should be measured per token and per time, not just raw accuracy.
The ablation emphasizes that A* self-critique search yields a few percentage points better accuracy at matched token budgets and matched wall-clock time, arguing it’s not just ‘spending more’ to win.
WORDS WORTH SAVING
5 quotesIn a lot of cases, small language models are actually preferred to LLMs because of lower cost, lower latency, and also lower compute re-required.
— Alexander Braverman
This begs our, like, main research question: How can we provide high-quality reasoning to c-resource-constrained models?
— Alexander Braverman
However, instead of using a external reward model, we use the model's own self-critique as a heuristic.
— Alexander Braverman
So for example, right, say I ask an LLM to answer five plus five, right? And then it says ten. And I ask the model, the same model, "Okay, this is the question. This is the answer. What would you give it on a scale of zero to one hundred?" This is a self-critique, for example, right?
— Alexander Braverman
'Cause correlation does not equal causation, and to have true intelligence, true reasoning agents, you need, like, this ca-causation.
— Alexander Braverman
High quality AI-generated summary created from speaker-labeled transcript.
