Any-Horizon Reasoning for Video Agents

1. 0:00 – 0:07

   Intro

   1. AGAnkit Gupta0:00

      [upbeat music]

2. 0:07 – 0:15

   Research goal: building “any-horizon” reasoning for video agents

   1. AGAnkit Gupta0:07

      We're here with Jitesh at YCML at Startup School. Jitesh is gonna tell us about his work that he published at CVPR this year. Jitesh, tell us what you did.

3. 0:15 – 0:45

   Human “skip vs. watch” behavior as motivation (TV/F1 vs. short reels)

   1. JJJitesh Jain0:15

      So this was done in collaboration with Georgia Tech and AI2. And, um, so if you watched, like, a TV show or a Formula 1 race, it's usually very long, and let's say you have a Codex agent running, a Claude code agent running. It's doing something. It's asking for information. You forgot to pause your show, and you went there, you missed some part of it. Now you come back, you missed, like, let's say thirty minutes of it. You don't re-watch the whole thing. You skip to parts to find out what you actually missed. Um, and otherwise, let's say

4. 0:45 – 1:16

   Why existing video agent reasoning systems struggle in the real world

   1. JJJitesh Jain0:45

      the situation reverses, and now you're looking at, like, an Instagram Reel ten seconds long. You watch the whole thing. You won't skip there. This is what we term as the natural any horizon ability of humans. Humans are very good at adapting how much time do they spend on what based on the task at hand. So based on this, our major motivation for this work was to figure out how are ex- how are existing video agent reasoning system at the time at, uh, such task, and we found they struggle. Uh, they, they underperform on open-ended questions, which is what,

5. 1:16 – 1:47

   Issue #1: over-reliance on temporal grounding models that aren’t robust for long videos

   1. JJJitesh Jain1:16

      uh, humans are interested in when they are interacting with videos in the real world, and they are very slow as well, so it's not really fruitful f-for them to be deployed. Uh, when we analyzed the things, we found three major issues. First was the system design itself. Um, we found that existing systems usually relied on temporal grounding, which is similar to grip for coding agents, and temporal grounding models for videos are not that accurate yet. So when humans look at something, when they're try-

6. 1:47 – 1:56

   Why temporal grounding is inaccurate: lack of long-video training data (a feedback loop)

   1. AGAnkit Gupta1:47

      So, so it's an accuracy problem, not... Is that because of how long they are that it, they're not good at that? Or why?

   2. JJJitesh Jain1:51

      They are not accurate because the models have not been trained on long videos-

   3. AGAnkit Gupta1:56

      Okay

7. 1:56 – 2:42

   Knowledge-grounded reasoning: adding tools like web search and transcripts to reduce reliance on vision-only localization

   1. JJJitesh Jain1:56

      ... in the first place, so it's, like, a cyclic thing. Um, so when humans look at long videos, they just don't look at the visual information. They do a very knowledge-grounded approach. You use your world knowledge. You use your s- context setting, like Formula 1. You know what Formula 1 is, how many drivers there are, so you try and find information based on that. Uh, so, like, if you're trying to find information about a Ferrari driver, you won't really look at a Red Bull car going on because you know Ferrari driver belongs to a red car. So we equip our system with tools like web search and transcribed speech, which usually have external signals that can help us alleviate the issue that arises from inaccuracy of the temporal grounding model, models, and that makes the system m-much more effective and efficient, as you can see here with the runtime thing. It's much faster than the previous systems.

   2. AGAnkit Gupta2:41

      Right.

8. 2:42 – 2:54

   Issue #2: long-video annotation is prohibitively expensive

   1. JJJitesh Jain2:42

      Then second with videos is videos are long, one hour, and if you use existing annotated, uh, platforms like Prolific, one hour of the annotated time can cost in ac- in excess of thirty dollars.

   2. AGAnkit Gupta2:54

      Yeah.

9. 2:54 – 3:05

   Synthetic training data pipeline: generating Q&A and tool trajectories using Gemini

   1. JJJitesh Jain2:54

      Very expensive if you want to train a model. We found that Gemini 2.5 Flash is actually good at generating questions. It's very slow, but it's like a one-time cost that you need to give.

   2. AGAnkit Gupta3:04

      And definitely not thirty dollars per hour.

10. 3:05 – 3:36

    Issue #3: teaching adaptive reasoning—multi-reward RL with an LLM judge for open-ended evaluation

    1. JJJitesh Jain3:05

       Yeah, exactly. So we generate question and answers pair for SFT and RL, and we generate synthetic tool called trajectories for goal-stradd SFT because you need to do that before RL. I'll come why we need to do RL, which is the third and most challenging part of this work, which is how do you actually equip the model with the adaptive reasoning ability? Um, and for that we have, like, a multi-reward RL recipe. We use LLM as a judge because we are working with open-ended, so string matching, anything like that doesn't work. Um, in this, for the rewards, there are two main

11. 3:36 – 4:06

    Reward shaping details: penalize wasteful tool calls; reward correct use of visual grounding when it helps

    1. JJJitesh Jain3:36

       things to look at. First, we penalize the model if the answer is wrong, but it calls tools, because that's not a good thing to do. Second, as I said, temporal grounding is not completely robust. So we want the model to adapt when to use temporal grounding, in which situations it's good. So we award the model with an extra point two five when it use, like, a visual tool like temporal grounding and it produce the right answer, so it can adapt around the inaccuracies of that. We have everything. Now we need to verify for recipe and everything. We collected data. Everything works.

12. 4:06 – 4:14

    Validation: behavior scales with video length and performance gains increase on longer videos

    1. JJJitesh Jain4:06

       Two ways to do that. First, as the duration of the video increases, the number of turns should increase-

    2. AGAnkit Gupta4:12

       Yeah

    3. JJJitesh Jain4:12

       ... which it does here, as you can see.

    4. AGAnkit Gupta4:14

       Mm-hmm.

13. 4:14 – 4:35

    Wrap-up: concluding that system design + synthetic data + RL recipe works

    1. JJJitesh Jain4:14

       Second, the boost of performance over, uh, the base model should also increase because base model has been trained for short videos, not long videos, which is also we observe here. So in conclusion, it proves that our whole recipe and our motivation and data and system design, everything works well.

    2. AGAnkit Gupta4:31

       Very cool. Awesome. Thanks so much, Jitesh. This is great. Thank you so much. Thanks.

Episode duration: 4:35