DeWorldSG Icon DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors

Seok-Young Kim1 Abdelrahman Elskhawy2,3 Taewook Ha1 Dooyoung Kim1 Eunjae Shin1 Benjamin Busam2,3 † Woontack Woo1 †
1KAIST 2Technical University of Munich 3Munich Center for Machine Learning (MCML)
(† Co-corresponding author)
ECCV 2026
Paper Coming Soon

TL; DR

DeWorldSG generates spatio-temporally consistent 3D semantic scene graphs from RGB-D sequences by combining depth-aware probabilistic object modeling with world-model-guided relation reasoning.

Abstract

We present DeWorldSG, a novel framework that generates spatio-temporally robust 3D Semantic Scene Graphs from RGB-D sequences. Existing methods often struggle to construct reliable 3D scene graphs due to unstable 3D object representations and missing relations caused by frame-wise inference. DeWorldSG addresses these issues by estimating instance-level geometric 3D Gaussian distributions through depth-guided filtering and representing each object as a probabilistic 3D node rather than a single projected point. To mitigate relational sparsity from frame-wise inference, our framework further aggregates spatiotemporal evidence across object pairs and refines relations using contextual priors derived from a world model (V-JEPA 2). Experiments on the 3DSSG and ReplicaSSG datasets demonstrate state-of-the-art (SoTA) performance in both object and predicate prediction, while producing temporally consistent scene structures. In particular, our method improves triplet recall by 77.4% and predicate recall by 23.2% over prior SoTA approaches, making it suitable for robotic manipulation and AR applications.

Method

DeWorldSG first extracts frame-wise 2D scene graphs from RGB-D observations, then lifts each detected instance into a depth-aware probabilistic 3D Gaussian using SAM-guided masks and Dual-Domain Depth Refinement. Local 3D graphs are incrementally merged into a global scene graph via semantic consistency and Gaussian similarity, while uncertain object relations are refined by aggregating temporal evidence and fusing it with V-JEPA 2 world-model priors.

Results

Demo Video

Room
Office
Apartment

BibTeX

@article{kim2026deworldsg,
                title={DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors},
                author={Kim, Seok-Young and Elskhawy, Abdelrahman and Ha, Taewook and Kim, Dooyoung and Shin, Eunjae and Busam, Benjamin and Woo, Woontack},
                journal={arXiv preprint arXiv:2607.00889},
                year={2026}
              }