Geometry-Grounded Multi-Camera 4D World Models

This project aims to develop a geometry-grounded 4D world model tailored for urban driving scenarios. While current generative world models like GAIA-2 [Wayve, arXiv’25] and Cosmos [Nvidia, arXiv’25] excel at image synthesis, they frequently fail to enforce physical consistency, struggle with complex dynamic scenes involving occlusions, and are largely restricted to single-viewpoint generation. These limitations render them unsuitable for reliable, safety-critical autonomous vehicle (AV) simulations. To bridge this gap, we propose a large-scale, physically grounded 4D world model trained on massive multimodal driving datasets. Given the immense data volume (over 5,500 hours of multi-camera footage) and the complexity of multimodal 4D representations, this project necessitates the large-scale computational resources provided by EuroHPC.

The project is organized around three pillars. First, we establish empirical neural scaling laws for an RGB flow-matching latent video generation model for driving, following LTX-2 [Lightricks, arXiv’26]. We have already scaled this RGB backbone to 9 billion parameters on MareNostrum 5 H100 nodes, and we plan to extend it from single-viewpoint to multi-camera generation that supports arbitrary sensor rigs. Second, we ground the generation in 3D geometry: leveraging our recent urban visual geometry foundation model, OccAny [Cao et al., CVPR’26], for geometric pseudo-labels and adapting our multitask framework, StableMTL [Cao et al., CVPR’26], we augment the backbone with a decoder that jointly decodes depth (local geometry) and camera poses (global geometry), enforcing physical consistency at both local and global levels, which we plan to extend to cross-camera consistency across arbitrary sensor rigs. The third pillar, detailed below, turns the generator into a controllable neural simulator through feed-forward 4D Gaussian Splatting.

To enable unconstrained scene navigation and diverse camera control, we will introduce a feed-forward, Gaussian Splatting-based dynamic novel view synthesis method inspired by NeoVerse [Yang et al., CVPR’26]. By applying 4D feed-forward Gaussian splatting (AnySplat [Jiang et al., TOG’25]) on top of the generated video, we can render novel, physically plausible views in real-time, which in turn acts as a geometric grounding constraint for the video generation itself. Furthermore, we will explicitly condition the model on time and predict 3D flow between timesteps as in Any4D [Karhade et al., CVPR’26] to accurately capture and disentangle complex dynamic elements.

The resulting model will deliver physically consistent, time-aware video generation with controllability. This leap in capability is transformative for downstream AV applications: it enables closed-loop simulation, safety verification, and the generation of rare, complex "corner-case" scenarios that are too dangerous or costly to collect in the real world. Furthermore, it serves as a scalable synthetic data engine for training next-generation perception and planning systems. By open-sourcing the resulting models and methodologies, this project will directly bolster the global competitiveness of the European automotive sector and significantly advance the broader computer vision research community.