Unified Temporal Video Grounding, Dense Captioning, and Object State-Change Understanding in Videos

This is a continuation of EHPC-AIF-2026SC01-171, whose goal is a unified video–language model for localising and describing object state changes (OSC) over time. That is, temporally localising the three phases of a state change (an object's initial state, its transitioning state, and its end state) together with their open-vocabulary description.

In execution, the team refined how to reach this goal: robust OSC understanding first requires the model to master two preliminary capabilities: temporal video grounding (TVG: localising when a queried event occurs) and dense video captioning (DVC: jointly localising and describing events).  Multi-phase OSC localisation can be seen as specialised instance of TVG and DVC. The researchers therefore built a new architecture coupling a large vision–language model (Qwen3-VL with LoRA) to a frozen text-aligned frame encoder. They enriched the encoder with spatio-temporal adapters and a temporal projection head that emits learned start/end ([S]/[E]) boundary predictions.

The full model is trained on a large multi-task pre-training (PT) corpus (~1.2 M samples) followed by per-benchmark fine-tuning (FT). This already yields strong results, e.g. Charades-STA test mIoU 56.11, with multi-benchmark results on ActivityNet-Captions, TACoS, QVHighlights and YouCook2. The original allocation, sized for a simplified plan, was exhausted in roughly the second month because the refined plan required extensive architecture R&D (an image-encoder bake-off, an adapter×pooling ablation, a Qwen size scaling experiment, data-mixture studies) on top of substantial operational overhead on shared HPC.

The team now request 90,000 node-hours to scale the TVG/DVC PT+FT and to deliver the original OSC objective on the same [S]/[E] machinery, with supervision scaled by automatic pseudo-labelling of narrated web videos from HowTo100M dataset using a 235B-parameter VLM. This contributes to ERC Advanced grant FRONTIER GA 101097822.