CHAPTERS
- 0:07 – 0:15
Research goal: building “any-horizon” reasoning for video agents
Ankit introduces Jitesh and his CVPR work, framing the core problem: video agents should reason effectively across both short and long videos. The conversation sets up the idea that the right amount of “watching” depends on the task and the video length.
- •Context: YCML at Startup School; work published at CVPR
- •Focus on video agents that can handle varying video lengths
- •Goal is practical navigation/reasoning rather than exhaustive viewing
- 0:15 – 0:45
Human “skip vs. watch” behavior as motivation (TV/F1 vs. short reels)
Jitesh explains the intuitive human capability to adjust attention: we skim long content to find what matters, but fully watch very short clips. This adaptable time-allocation is what the work calls “any-horizon” ability.
- •Example: missing 30 minutes of a long show—humans jump to relevant parts
- •Contrast: 10-second reel—humans just watch it all
- •Defines the “any-horizon” concept as adaptive allocation of time/effort
- 0:45 – 1:16
Why existing video agent reasoning systems struggle in the real world
The work starts by benchmarking/observing prior systems and finding they underperform on open-ended questions and are too slow to deploy. These weaknesses matter because real user interactions with video are often open-ended and time-sensitive.
- •Existing systems struggle with open-ended video Q&A
- •Runtime/latency is too high for practical deployment
- •Diagnosis begins by identifying root causes in design and training data
- 1:16 – 1:47
Issue #1: over-reliance on temporal grounding models that aren’t robust for long videos
Jitesh identifies system design as the first issue: many agents depend heavily on temporal grounding, but those models aren’t accurate enough—especially on long videos. This creates brittle behavior when the agent must locate relevant moments in extended footage.
- •Temporal grounding is analogous to “grep” for coding agents
- •Temporal grounding accuracy is a limiting factor in current systems
- •Long-video settings amplify localization errors
- 1:47 – 1:56
Why temporal grounding is inaccurate: lack of long-video training data (a feedback loop)
In response to Ankit’s question, Jitesh explains the accuracy problem stems from insufficient training on long videos. This produces a cyclical issue: models can’t handle long videos well because long-video supervision is scarce and expensive.
- •Inaccuracy is primarily due to limited long-video training
- •Long-video data scarcity contributes to a self-reinforcing problem
- •Sets up the need for alternative signals and cheaper supervision
- 1:56 – 2:42
Knowledge-grounded reasoning: adding tools like web search and transcripts to reduce reliance on vision-only localization
Humans use world knowledge and context cues to narrow down what to look for (e.g., knowing Ferrari is red in F1). The system mirrors this by incorporating external tools—web search and transcribed speech—to guide retrieval and reduce dependence on unreliable temporal grounding.
- •Humans use context/world knowledge to search efficiently in long videos
- •Example: use F1 knowledge to ignore irrelevant cars/teams
- •System adds web search + speech transcripts as external signals
- •Tooling improves effectiveness and runtime compared to prior systems
- 2:42 – 2:54
Issue #2: long-video annotation is prohibitively expensive
Jitesh highlights the data bottleneck: annotating long videos via crowd platforms can cost over $30 per hour of video, making scaling impractical. This motivates synthetic data generation to create training signals at far lower cost.
- •Long-video annotation costs can exceed ~$30/hour
- •Training at scale becomes economically infeasible
- •Necessitates automated/synthetic supervision pipelines
- 2:54 – 3:05
Synthetic training data pipeline: generating Q&A and tool trajectories using Gemini
To reduce annotation cost, the team uses Gemini 2.5 Flash to generate question–answer pairs for supervised fine-tuning (SFT) and reinforcement learning (RL). They also generate synthetic tool-use trajectories to enable goal-oriented SFT as a prerequisite before RL.
- •Gemini 2.5 Flash used to generate questions (slow but one-time cost)
- •Creates Q&A pairs for SFT and RL
- •Generates synthetic tool trajectories for goal-directed SFT
- •Positions SFT as the setup step before RL
- 3:05 – 3:36
Issue #3: teaching adaptive reasoning—multi-reward RL with an LLM judge for open-ended evaluation
The hardest part is instilling adaptive tool use and reasoning depth. Because questions are open-ended, they use an LLM-as-judge rather than string matching, and train with a multi-reward RL scheme to balance correctness with efficient, appropriate tool usage.
- •Core challenge: adaptive reasoning across horizons (when to watch, when to search)
- •Open-ended answers require LLM judging (not exact-match metrics)
- •Multi-reward RL recipe shapes behavior beyond raw accuracy
- 3:36 – 4:06
Reward shaping details: penalize wasteful tool calls; reward correct use of visual grounding when it helps
The reward design discourages calling tools when the model is wrong (wasted cost/latency). It also explicitly rewards correct answers that use visual tools like temporal grounding, helping the agent learn when grounding is reliable enough to use despite its imperfections.
- •Penalty: wrong answer + tool calls is explicitly discouraged
- •Temporal grounding is treated as useful but not always robust
- •Bonus reward (~+0.25) for correct answers that appropriately use visual grounding
- •Encourages selective, situation-dependent tool use
- 4:06 – 4:14
Validation: behavior scales with video length and performance gains increase on longer videos
They verify the approach in two main ways: as video duration grows, the agent takes more turns (indicating deeper interaction when needed), and the performance uplift over the base model grows with video length. This matches the hypothesis that base models are biased toward short-video training distributions.
- •Longer videos → more agent turns, indicating adaptive horizon behavior
- •Performance boost over base increases with video length
- •Supports the claim that the training recipe and tools address long-video weaknesses
- 4:14 – 4:35
Wrap-up: concluding that system design + synthetic data + RL recipe works
Jitesh concludes that the combined approach—tooling, synthetic supervision, and RL reward shaping—delivers the intended any-horizon reasoning improvements. Ankit closes by thanking him for the explanation.
- •End-to-end recipe validated: motivation, data, design, and training align
- •Emphasis on practical effectiveness and efficiency vs. prior systems
- •Conversation closes with brief acknowledgments
