Skip to content
YC Root AccessYC Root Access

Evaluating the Fine-Grained Planning Abilities of Web Agents

At our inaugural YCML at Startup School, YC Partner Ankit Gupta speaks with Surgan Jandial about evaluating how web agents plan, rather than only measuring whether they complete a task. The work breaks planning into individual skills such as temporal ordering, future-state prediction, action selection, and error correction. It then repurposes existing datasets to create inexpensive synthetic tests that measure each skill independently. These scores correlate with agents' performance on complete web tasks, providing a cheaper and more interpretable way to identify their weaknesses before running full trajectory evaluations. Apply to Y Combinator: https://www.ycombinator.com/apply Work at a startup: https://www.ycombinator.com/jobs

Ankit GuptahostSurgan Jandialguest
Aug 6, 20266mWatch on YouTube ↗

At a glance

WHAT IT’S REALLY ABOUT

Testing web-agent planning via cheap, interpretable skill-based evaluations framework

  1. Web agents repeatedly generate a plan/thought from UI context and then choose the next interaction, but evaluating the quality of that intermediate planning step is hard and expensive compared to scoring final outputs.
  2. The work proposes decomposing “planning” into smaller, measurable skills—such as temporal ordering of webpage states, future-state prediction, option evaluation/selection, and error correction—analogous to decomposing visual reasoning into sub-skills.
  3. These skills are discovered semi-automatically from agent traces across seed tasks, then operationalized as simple offline tests rather than requiring full multi-step trace evaluation.
  4. The approach repurposes existing datasets and synthetic question generation (e.g., ordering two screenshots by which comes first in a task flow) to keep evaluation cheap, scalable, and interpretable.
  5. Results show many (especially open-source) models score below 50% on these simple tests, and aggregate skill performance correlates strongly with real end-task performance, making the suite useful for early model filtering and diagnosis.

IDEAS WORTH REMEMBERING

5 ideas

Intermediate planning evaluation is a “needle-in-a-haystack” problem.

Judging a single plan step requires full past context and future outcomes across many steps, plus expensive multimodal inputs, making errors subtle and hard to locate.

Planning can be measured as a set of separable micro-skills.

Instead of scoring the exact chain-of-thought, the work isolates capabilities like temporal ordering (what comes first), future-state prediction (how to reach a goal state), option selection, and error correction.

Skill discovery can be driven by traces without heavy human labeling.

By running agents on seed tasks and analyzing what they repeatedly “reason about,” you can identify common planning operations the agent relies on.

Cheap offline tests provide interpretable diagnostics per capability.

Simple question formats—such as choosing which screenshot/state should occur earlier in a workflow—yield per-skill scores that help pinpoint where a model is weak.

Repurposing existing datasets reduces evaluation cost and friction.

The framework avoids new data collection and human involvement by generating synthetic evaluation items from existing web/task datasets and screenshots.

WORDS WORTH SAVING

5 quotes

So evaluating the final output, in my opinion, is slightly easier. Uh, so I'm, I'm concerned with what happens to evaluating this plan, this step.

— Surgan Jandial

Therefore, for this particular guy, you actually need the whole thing, right?

— Surgan Jandial

So therefore, finding a problem with this thing would be like a needle in the haystack problem, right?

— Surgan Jandial

So I'm thinking, how do I break down planning, which is the ability to come up with thoughts to solve this action into these fine-grained skills and put a number to the capability of the model for each of these skills, and then comment on the planning ability?

— Surgan Jandial

So the average of all, most of the models are bad, like below fifty percent.

— Surgan Jandial

Web agents and VLM-based UI interaction loopsChallenges of evaluating intermediate planning tracesDecomposing planning into fine-grained skillsSemi-automatic skill discovery from task tracesSynthetic, dataset-repurposed offline test constructionCorrelation between skill scores and end-task successComplementing trace-based evaluation with cheap screening

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome