Panoptic NeRF: 3D-to-2D Label Transfer for
Panoptic Urban Scene Segmentation

3DV 2022


Xiao Fu1*, Shangzhan Zhang1*, Tianrun Chen1, Yichong Lu1, Lanyun Zhu2
Xiaowei Zhou1, Andreas Geiger3, Yiyi Liao1

1Zhejiang University    2Singapore University of Technology and Design    3University of Tübingen and MPI-IS

Abstract


Panoptic NeRF obtains per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives.
overall pipeline

Large-scale training data with high-quality annotations is critical for training semantic and instance segmentation models. Unfortunately, pixel-wise annotation is labor-intensive and costly, raising the demand for more efficient labeling strategies. In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives. Our method utilizes NeRF as a differentiable tool to unify coarse 3D annotations and 2D semantic cues transferred from existing datasets. We demonstrate that this combination allows for improved geometry guided by semantic information, enabling rendering of accurate semantic maps across multiple views. Furthermore, this fusion process resolves label ambiguity of the coarse 3D annotations and filters noise in the 2D predictions. By inferring in 3D space and rendering to 2D labels, our 2D semantic and instance labels are multi-view consistent by design. Experimental results show that Panoptic NeRF outperforms existing semantic and instance label transfer methods in terms of accuracy and multi-view consistency on challenging urban scenes of the KITTI-360 dataset.


Overview video



Semantic Label Transfer



Panoptic Label Transfer



Novel View Appearance & Label Synthesis



Geometry Reconstruction



Citation



@inproceedings{fu2022panoptic,
  title={Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation},
  author={Fu, Xiao and Zhang, Shangzhan and Chen, Tianrun and Lu, Yichong and Zhu, Lanyun and Zhou, Xiaowei and Geiger, Andreas and Liao, Yiyi},
  booktitle = {International Conference on 3D Vision (3DV)},
  year = {2022}
}