One Diffusion Model to Rule Geometry: UGG Unifies Video, Depth, and Camera Pose in a Single Framework

UGG treats video generation and 3D geometry prediction as one fill-in-the-blanks problem, training a single diffusion model on heterogeneous data to beat specialist baselines across multiple tasks.

Imagine you're doing a jigsaw puzzle, but each box you bought is missing different pieces — one box has the sky but no ground, another has the ground but no trees. Most people would try to solve each box separately. UGG's insight is: dump all the incomplete boxes onto one table and let the shared edges teach you what every missing piece looks like. That's the core mechanism — conditional completion across modalities with different noise levels — and it's more powerful than it sounds. The committed claim: a single generative diffusion model can jointly handle RGB video generation, monocular depth estimation, and camera pose prediction — tasks that have historically required separate architectures — and outperform non-unified specialist baselines on each. This isn't a compromise where unification costs you accuracy; the authors report gains specifically because the model sees geometric and visual data together during training. The key architectural move is assigning different noise levels to different modalities and different timesteps. Clean RGB frames condition noisy depth maps; observed camera poses condition unobserved ones. This turns every task into a denoising problem within a single flow-matching diffusion transformer. The model trains on heterogeneous datasets — some with depth labels, some with camera poses, some with neither — because any missing modality is simply treated as fully noised. This is elegant: you don't need a single dataset with all annotations, which has historically been the bottleneck for unified 4D understanding. On the ladder, UGG competes against DUSt3R and MASt3R for pose estimation, DepthAnythingV2 for depth, and CamCo/MotionCtrl for camera-controlled video generation. The paper reports improvements in camera pose accuracy (lower rotation and translation error vs. DUSt3R in sparse-view settings) and competitive or superior depth estimation on standard benchmarks. Notably, they show that inference-time geometric search — sampling multiple hypotheses and selecting via geometric consistency with observed frames — provides free accuracy gains without ground-truth geometry at test time. This is the generative advantage over feedforward models: you can sample, score, and select. Integrity is mixed but honest. Evaluation uses standard community benchmarks (RealEstate10K, ScanNet, NYUv2), and the baselines are current — DUSt3R and DepthAnythingV2 are 2024 models, not straw men. However, this is all self-evaluated: no independent replication, no pre-registration, and the benchmark selection clearly favors sparse-view and multi-task settings where unification has the most advantage. The 31-page paper includes ablations and the project page promises code, which is good practice. The NeurIPS 2026 acceptance provides peer-review signal. The milestone question is about scaling. UGG currently operates on relatively short video sequences and moderate resolutions. The obvious next threshold is real-time or near-real-time 4D scene reconstruction from casual video — think AR glasses or autonomous driving — which requires both longer temporal horizons and higher geometric fidelity than demonstrated here. The gap is probably 2-3 years of compute scaling and architectural refinement. The experiment the authors didn't run is outdoor autonomous driving scenes with fast ego-motion and dynamic objects at scale — the KITTI/Waymo regime where geometric accuracy has immediate safety implications. My read: they're saving this for the next paper. The framework clearly could handle it, the datasets exist, and the industrial applications are obvious. Establishing the unified framework on indoor/static scenes first is strategic sequencing, not a limitation they're hiding.