YC Root AccessEvaluating the Fine-Grained Planning Abilities of Web Agents
CHAPTERS
- 0:07 – 1:13
Why web-agent planning needs its own evaluation (not just final answers)
Surgan introduces VLM web agents and highlights a gap in evaluation: the planning/thought stage that sits between observing a UI and taking actions. He argues that judging only the final output is easier, but misses whether the agent’s underlying planning is sound.
- •Web agents iterate: observe UI → plan/thought → take a specific interaction → repeat
- •Most evaluations focus on end outcomes, not the quality of intermediate plans
- •Planning quality matters even when final outputs look correct (or failures are hard to diagnose)
- 1:13 – 1:44
What makes plan evaluation hard: full-trace dependency and expensive context
Evaluating a plan at a single step requires both past context and future outcomes, effectively the whole execution trace. Because web-agent traces include images, tool calls, and text, auditing plans becomes costly and difficult to scale.
- •A plan depends on prior steps and anticipates future steps, so it can’t be judged in isolation
- •Traces are long (10–15 steps) and multimodal (screenshots + tool calls + text)
- •Context is expensive to process, making step-by-step plan grading impractical at scale
- 1:44 – 2:04
Subtle failure modes: ‘needle in a haystack’ planning bugs
As models improve, they make fewer obvious mistakes, making it harder to spot where a plan went wrong. Surgan frames plan inspection as searching for subtle errors hidden within long trajectories.
- •Better models fail in less conspicuous ways
- •Pinpointing a flawed thought within many steps is time-consuming
- •Motivation for complementary, cheaper evaluation methods
- 2:04 – 2:48
Analogy to visual reasoning: decompose planning into sub-skills
Using VQA as an analogy, Surgan argues that what looks like a single “planning” ability is actually a composition of smaller reasoning operations. The goal becomes measuring each fine-grained skill separately and using those measurements to characterize planning ability.
- •VQA tasks require grounding, spatial reasoning, and identification—multiple sub-steps
- •Planning similarly consists of multiple latent skills
- •Objective: assign scores to individual planning sub-skills, not just overall success
- 2:48 – 3:03
Semi-automatic discovery of planning skills from agent traces
They run agents on a set of seed tasks and analyze behavior to identify recurring planning operations. This yields a catalog of skills agents appear to deploy during web navigation and task completion.
- •Use seed tasks and trace analysis to surface common reasoning patterns
- •Skill discovery is semi-automatic rather than manually curated from scratch
- •Focus on repeatable, isolatable skills that show up across tasks
- 3:03 – 3:11
Skill 1: Temporal ordering of webpage states
A core planning component is reasoning about the correct sequence of states (e.g., whether checkout should come after adding to cart). The agent must understand what comes before/after in a task flow.
- •Agents check whether they’re progressing in the right direction
- •Requires ordering states like ‘add to cart’ vs ‘checkout’
- •Captures procedural understanding of typical web workflows
- 3:11 – 3:24
Skill 2: Future-state prediction and how to get there
Beyond ordering, agents must anticipate a desired future state and infer actions that move them from the current page to that target. This includes predicting the likely next screens and navigation paths.
- •Agents predict what the next relevant state/page should be
- •They infer transitions: which action will lead from current to desired future state
- •Measures foresight and transition modeling in web environments
- 3:24 – 3:48
Skill 3: Evaluating multiple action choices (option selection)
Agents often face several plausible actions and must choose the best one. This skill reflects comparing alternatives rather than executing a single deterministic step.
- •Real UIs present multiple clickable options or paths
- •Agents need to evaluate candidates and select the most promising action
- •Captures decision-making under ambiguity
- 3:48 – 4:07
From discovered skills to measurable tests (and including error correction)
After identifying skills, the next step is to operationalize them into targeted evaluations. Surgan notes that error correction is another important behavior when agents recognize mistakes and adjust course.
- •Separate ‘discovering skills’ from ‘measuring skill competence’
- •Create small, targeted tests for each skill (including counterfactual-style checks)
- •Error correction emerges as an additional deployed skill
- 4:07 – 4:37
Design constraints: keep evaluation cheap, scalable, and interpretable
The evaluation framework is designed to be inexpensive and to yield interpretable per-skill numbers. This is intended to complement costly full-task evaluations by quickly diagnosing where models are weak.
- •Primary constraints: low cost and high interpretability
- •Per-skill scores provide diagnostic signal beyond a single success rate
- •Goal is a practical screening tool before running full traces
- 4:37 – 5:27
Repurposing existing datasets with synthetic question generation
Instead of collecting new human-labeled data, the approach repurposes existing datasets and generates synthetic evaluation questions. An example for temporal ordering uses pairs of screenshots and asks which comes first in solving the task.
- •No new data collection or human annotation required
- •Generate simple Q&A-style probes from existing screenshots/tasks
- •Example: ask which of two pages would appear earlier in a successful flow
- 5:27 – 5:57
Results: simple probes reveal weak planning skills but predict end-task success
Across models, performance on these simple skill tests is often below 50%, indicating significant gaps. Importantly, aggregated skill scores correlate with end-task performance, supporting the tests’ usefulness as a planning proxy.
- •Many models perform poorly on supposedly simple skill tests (<50%)
- •Skill-test averages correlate with final task performance
- •Implication: improving these sub-skills should improve real web-agent outcomes
- 5:57 – 6:35
How to use the framework: cheap offline screening + targeted improvement
Surgan positions the method as complementary to trace-based evaluation: use quick, low-cost probes to filter models and pinpoint weaknesses before running expensive end-to-end tests. The per-skill breakdown can guide targeted training or optimization.
- •Run offline probes cheaply (on the order of dollars) for preliminary assessment
- •Use per-skill failures to diagnose where the agent is wrong
- •Then run full, complex trace evaluations after filtering/improving models
- 6:35 – 6:40
Wrap-up and closing remarks
Ankit and Surgan conclude with a brief recap and thanks. The conversation ends after reiterating the value of adding a complementary dimension to planning evaluation.
- •Method is presented as an addition, not a replacement, to existing evaluation
- •Emphasis on interpretability and practicality for practitioners
- •Closing acknowledgments
