Streaming Autoregressive Diffusion for Real-Time Interactive Human Avatar Video Generation

This project develops a native streaming autoregressive diffusion framework for real-time, interactive human avatar video generation. Today's diffusion-based avatar models are bidirectional and non-causal: they synthesize a complete clip offline and therefore cannot support live, low-latency interaction. The researchers will instead generate synchronized audio-visual content chunk-by-chunk under a strictly causal regime, so the model produces and commits each short segment while attending only to its own committed past. This enables sub-second responsiveness and continuous, long-horizon streaming of a talking, gesturing avatar on a single GPU.

The scientific core of the project is a forcing-free training paradigm. Rather than distilling streaming behavior from a future-aware, non-causal teacher through teacher forcing, the avatar generator is trained directly under the same chunk-by-chunk causal rollout it will use at deployment. A self-resampling scheme exposes the model during training to the degraded, self-generated histories it will actually encounter over long rollouts, closing the train-test context gap that otherwise causes drift, identity loss, and audio-visual desynchronization.

The team's central contribution is forcing-free on-policy distillation. Quality criteria for social avatar video conflict across content domains (far-shot full-body stability, multi-person turn-taking, accurate lip-sync, expressive facial motion, and high-motion dance), so the team will first train a bank of lightweight domain-preference experts and then consolidate them into one deployable streaming policy. Crucially, this consolidation is performed entirely on student-visited denoising states: the student rolls out its own candidate trajectories, a domain verifier scores them, and a reward-adaptive expert prior steers the student in velocity space. This avoids teacher forcing, stochastic transition ratios, and PPO-style instability, yielding a single unified causal policy with a deterministic few-step sampler suited to real-time avatars.

The researchers will validate the approach on audio-driven multimodal avatar generation with synchronized speech, lip motion, gesture, and full-body emotive expression, targeting real-time frame rates and stable thousand-second rollouts. The work requires large-scale HPC to train a multi-billion-parameter causal audio-visual diffusion transformer and to run the on-policy distillation loop, which repeatedly rolls out, scores, and distills student trajectories across all domains.