FROSS

Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images

Hao-Yu Hou1,3, Chun-Yi Lee2, Motoharu Sonogashira3, Yasutomo Kawanishi3,
1National Tsing Hua University , 2National Taiwan University, 3RIKEN
arXiv Video Code Dataset Poster

TL;DR

FROSS is an online 3D semantic scene graph (3D SSG) generation method, which

  • Can operate at faster-than-real-time speed
  • Supports incremental graph update in environments without predefined boundaries
  • Has superior performance
by representing 3D objects as Gaussian distributions. We also developed the ReplicaSSG dataset upon Replica , with additional inter-object relationship annotations.

Video

Abstract

The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods.

Framework Overview

FROSS takes RGB-D image sequences and their associated camera poses as input, and incrementally constructs a global 3D SSG in an online manner. The overall framework is illustrated in the figure below. FROSS incorporates each incoming RGB-D frame through the following steps:

  • (a) The process initiates with object detection via RT-DETR from an RGB-D image and its associated camera pose.
  • (b) The EGTR model extracts inter-object relationships through utilization of preserved self-attention features from RT-DETR.
  • (c) 2D Gaussian distributions derived from detected bounding boxes undergo spatial transformation into 3D space for local 3D SSG construction.
  • (d) The resultant local 3D SSG is subsequently incorporated into the global 3D SSG by merging Gaussian objects of the same class based on the Hellinger distance.

Framework Overview

Qualitative Results

The below figure presents the 3D SSG generation results from FROSS, demonstrating its ability to approximate object positions and shapes within a scene through Gaussian distributions, while simultaneously extracting their relationships. The unified objects highlight the effectiveness of the proposed merging algorithm, which consolidates predictions from multiple views, reducing redundant objects and processing time. This representation method offers a high-level scene understanding for downstream applications. In comparison, Wu et al.'s method1 frequently misclassifies objects with similar geometry (e.g., “curtain” vs. “door”, “table” vs. “sofa”) and confuses semantically close relationships (e.g., “standing on” vs. “supported by”). This performance gap shows FROSS's advantage in utilizing richer visual information, in contrast to Wu et al., which relies solely on cropped object image for visual feature extraction.

  1. Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. Incremental 3D semantic scene graph prediction from RGB sequences. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5064-5074, 2023
Qualitative Results

(Only representative objects are visualized. Errors are marked in red, with ground truth label shown in parentheses. Node colors correspond to the respective scene graphs.)

BibTeX


@InProceedings{hou2025fross,
    author    = {Hao-Yu Hou, Chun-Yi Lee, Motoharu Sonogashira, and Yasutomo Kawanishi},
    title     = {{FROSS}: {F}aster-than-{R}eal-{T}ime {O}nline 3{D} {S}emantic {S}cene {G}raph {G}eneration from {RGB-D} {I}mages},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025}
}