Evaluating the Fine-Grained Planning Abilities of Web Agents

1. 0:00 – 0:07

   Intro

   1. AGAnkit Gupta0:00

      [upbeat music]

2. 0:07 – 1:13

   Why web-agent planning needs its own evaluation (not just final answers)

   1. AGAnkit Gupta0:07

      We're here with Surgan at YCML. Surgan's a master's student at CMU, and just presented this work at EMNLP last year. Surgan, tell us a little bit about what you worked on.

   2. SJSurgan Jandial0:16

      Yeah. Thanks a lot. Um, yeah. My work is titled On the Fine-Grained Planning Abilities of VLM Web Agents. Uh, so a little bit about web agents. Um, so as the web agents execute their task, for instance, you know, find this cheapest 4K monitor, they get input, you know, a UI screen, and then they first think about how to solve this task, which is like, you know, coming up with plan, strategy, and so on.

   3. AGAnkit Gupta0:38

      Yeah.

   4. SJSurgan Jandial0:38

      That's called planning stage. And based off of this thought, they predict a very specific interaction that they have to do on this particular screen. They keep doing this for like, you know, five steps, ten steps, or whatever is the limit, and produce a final output.

   5. AGAnkit Gupta0:50

      Yep.

   6. SJSurgan Jandial0:50

      Um, and then we have some rubrics to evaluate that, some verifiers or whatever the approach we wanna take. So evaluating the final output, in my opinion, is slightly easier. Uh, so I'm, I'm concerned with what happens to evaluating this plan, this step.

   7. AGAnkit Gupta1:04

      Yeah. So you're saying this is like an off-the-shelf VLM web agent-

   8. SJSurgan Jandial1:08

      Yeah

   9. AGAnkit Gupta1:08

      ... does this type of step.

   10. SJSurgan Jandial1:09

       Yeah.

   11. AGAnkit Gupta1:09

       And you're trying to evaluate this middle planning stage.

   12. SJSurgan Jandial1:11

       Middle planning step.

   13. AGAnkit Gupta1:12

       Yeah.

3. 1:13 – 1:44

   What makes plan evaluation hard: full-trace dependency and expensive context

   1. SJSurgan Jandial1:13

      Yeah. Uh, s-so let's think of what would an approach to evaluate this would be. Let's say you wanna evaluate the plan at this step. Since the plan is generated with the past context, you would need that, and the plan is about what you would do in the future, so you would also need that output. Therefore, for this particular guy, you actually need the whole thing, right? And this thing, it's like ten, fifteen of those, and the context is not cheap because for web agents, you actually need, like, an image, a bunch of tool calls, and a bunch of different, you know, text. In addition, uh,

4. 1:44 – 2:04

   Subtle failure modes: ‘needle in a haystack’ planning bugs

   1. SJSurgan Jandial1:44

      the models are slightly getting better, and they don't make obvious steps, so theirs will be subtle. So therefore, finding a problem with this thing would be like a needle in the haystack problem, right? So what we are saying is, evaluating each of the plan thought to evaluate the planning skill is a great way. You should do it. You know, you should run the traces-

   2. AGAnkit Gupta2:04

      Right

5. 2:04 – 2:48

   Analogy to visual reasoning: decompose planning into sub-skills

   1. SJSurgan Jandial2:04

      ... get this right, and comment on planning, but we should also focus and figure out alternatives, complementary things. Let's take an analogy from visual reasoning. Let's say there is a task, visual question answering task. What is the animal to the left of the lion?

   2. AGAnkit Gupta2:17

      Yeah.

   3. SJSurgan Jandial2:17

      It seems like a single task to all of, both of us-

   4. AGAnkit Gupta2:20

      Right

   5. SJSurgan Jandial2:20

      ... but it's not that. I mean, model is first going to ground the lion-

   6. AGAnkit Gupta2:23

      Right

   7. SJSurgan Jandial2:23

      ... you know, understand what's to the left and right, locate the animal, and then come up with the right answer.

   8. AGAnkit Gupta2:27

      And our brains are doing all of these things too.

   9. SJSurgan Jandial2:28

      Yeah, all of these things.

   10. AGAnkit Gupta2:29

       Yeah.

   11. SJSurgan Jandial2:30

       Yeah.

   12. AGAnkit Gupta2:30

       Yeah.

   13. SJSurgan Jandial2:30

       So, and we have to get all of that right to get this right, right? So I'm thinking, how do I break down planning, which is the ability to come up with thoughts to solve this action into these fine-grained skills and put a number to the capability of the model for each of these skills, and then comment on the planning ability?

   14. AGAnkit Gupta2:47

       Yeah.

6. 2:48 – 3:03

   Semi-automatic discovery of planning skills from agent traces

   1. SJSurgan Jandial2:48

      So to discover these skills, we have like a semi-automatic way. Uh, we run it for a bunch of, you know, uh, tasks, seed tasks. Um, yeah, for a bunch of seed tasks, and then we discover a bunch of these things. So what we figure out is agent is always th- trying to think about if it is going in the right direction.

   2. AGAnkit Gupta3:03

      Yeah.

7. 3:03 – 3:11

   Skill 1: Temporal ordering of webpage states

   1. SJSurgan Jandial3:03

      For instance, if it's at add to product, it's like, "Oh, I have to buy this product. Should the next state be checkout?" It's like, "Oh, does checkout come before add to product?"

   2. AGAnkit Gupta3:11

      Right.

8. 3:11 – 3:24

   Skill 2: Future-state prediction and how to get there

   1. SJSurgan Jandial3:11

      So it's always sort of doing the temporal ordering of the webpage states. Second, whenever it's at this add to product and it knows, like, I want to do checkout, it's always thinking about, oh, what will help me go from here to here? So it's like predicting the future-

   2. AGAnkit Gupta3:24

      Right

9. 3:24 – 3:48

   Skill 3: Evaluating multiple action choices (option selection)

   1. SJSurgan Jandial3:24

      ... and predicting how to go to future. So the skill would be future-state prediction. And often, agent is not just doing, you know, taking deterministic one action. It's like it has a bunch of options. It's trying to evaluate these options and then trying to figure out which is the best. So it has multiple plan choices, action choices, and then selecting the one.

   2. AGAnkit Gupta3:43

      And, and how are you implicitly assessing, let's say, plan assessment, how do you measure the agent's ability to do this?

10. 3:48 – 4:07

    From discovered skills to measurable tests (and including error correction)

    1. SJSurgan Jandial3:48

       Yeah. So this portion discovers the skill agents is deploying, and we isolate them. In the next one, we operationalize the skill discovery, actually evaluate their, you know, capability on that skill. We create some simple tests which evaluate-

    2. AGAnkit Gupta4:05

       Like, you create some counterfactuals to determine if it's-

11. 4:07 – 4:37

    Design constraints: keep evaluation cheap, scalable, and interpretable

    1. SJSurgan Jandial4:07

       Yeah, yeah. Very simple tests. Yeah. This is just telling you what skills they're deploying, and you should make the skill level evaluation. And once they can get wrong, the other skill they are also deploying is error correction. Now to your question, how am I making this evaluation very useful? So couple of constraints, I want to make this cheap because if it is as expensive as possible, then it's no use, and I want to make it more interpretable. You know, getting a good number for each of this is very important so that I can give you another, you know, detail in your evaluation. So

12. 4:37 – 5:27

    Repurposing existing datasets with synthetic question generation

    1. SJSurgan Jandial4:37

       what I do is we take existing datasets. It's like totally, you know, repurposing the existing datasets. We don't sort of generate new data. We don't have any human involvement. It's just having a very easy way to generate synthetic data. For instance, um, for temporal ordering, like whether this webpage comes before or not, you take two screenshots, like Google Scholar homepage and the final task, and then you're like, okay, if you want to solve this, which of the following come, will come first? So agent will answer this question.

    2. AGAnkit Gupta5:06

       Makes sense.

    3. SJSurgan Jandial5:07

       This will evaluate this skill, and for all of them, you will have these type of questions. All of them combined will give you a number for each of these skills, which will correlate to the final planning capability. So it's different. Earlier, you were evaluating their exact thoughts.

    4. AGAnkit Gupta5:21

       Yeah.

    5. SJSurgan Jandial5:21

       Now it's a generic skill.

    6. AGAnkit Gupta5:23

       And so what does that look like in terms of your ultimate results? Like, how, how should we think about how well this works?

13. 5:27 – 5:57

    Results: simple probes reveal weak planning skills but predict end-task success

    1. SJSurgan Jandial5:27

       So first result we have is, uh, these are like simple tests. So the average of all, most of the models are bad, like below fifty percent. So ideally, we would assume these are like simple tests, right? I mean, these are open source because of the compute constraints, but again, even them being be- less than fifty percent is a very bad result, very bad performance for them. So it's like we really need to improve on each of these dimensions. Second, we see that the average performance of this planning is very correlated to the end task performance. Models being good on these skills is actually

14. 5:57 – 6:35

    How to use the framework: cheap offline screening + targeted improvement

    1. SJSurgan Jandial5:57

       correlated to the end task performance. So now what we are saying is, you should definitely run the models on your tasks, get the traces, evaluate the thought, but this is like an offline test. You will take my simple tests, put it on your model, like for ten dollars, get a preliminary evaluation on these independent dimension, and you can filter out a lot of them, and then you should run and do this complex test. So now it's also like... it's, it's like you know where it is wrong. If this is wrong, then first do a GRPO or what you wanna do, and then you get to do this complex. That's how we add a dimension to this planning evaluation, and that's how we think it's going to be like a... It's not replacement. It's like complementary to the existing evaluation that we do. Yeah.

15. 6:35 – 6:40

    Wrap-up and closing remarks

    1. AGAnkit Gupta6:35

       Cool. Awesome, Surgan. Thanks so much for joining us. This is great. Thank you.

    2. SJSurgan Jandial6:39

       Thank you.

    3. AGAnkit Gupta6:39

       Appreciate it. Thanks.

Episode duration: 6:40