A Robust 3D-2D Interactive Tool for Scene Segmentation and Annotation

Duc Thanh Nguyen1       Binh-Son Hua2       Lap-Fai Yu3       Sai-Kit Yeung2

1Deakin University 2Singapore University of Technology and Design 3University of Massachusetts Boston (Equal contribution)


Recent advances of 3D acquisition devices have enabled large-scale acquisition of 3D scene data. Such data, if completely and well annotated, can serve as useful ingredients for a wide spectrum of computer vision and graphics works such as data-driven modeling and scene understanding, object detection and recognition. However, annotating a vast amount of 3D scene data remains challenging due to the lack of an effective tool and/or the complexity of 3D scenes (e.g. clutter, varying illumination conditions).

This paper aims to build a robust annotation tool that effectively and conveniently enables the segmentation and annotation of massive 3D data. Our tool works by coupling 2D and 3D information via an interactive framework, through which users can provide high-level semantic annotation for objects. We have experimented our tool and found that a typical indoor scene could be well segmented and annotated in less than 30 minutes by using the tool, as opposed to a few hours if done manually.

This tool has been used to annotate over 100 scenes in SceneNN, a RGB-D scene mesh dataset. More details about the dataset at www.scenenn.net.


                    
@article{anno-tvcg17,
    author = {Thanh Nguyen, Duc and Hua, Binh-Son and Yu, Lap-Fai and Yeung, Sai-Kit},
    title = {A Robust 3D-2D Interactive Tool for Scene Segmentation and Annotation},
    journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
    year = {2017}
}
                    
                    
Paper       Executable       WebGL

Acknowledgement

We thank Tan-Sang Ha and Quang-Trung Truong for helping with the development of the WebGL version of the tool.

We also thank Fangyu Lin and Quang-Hieu Pham for assisting with data capture and Guoxuan Zhang for the early version of the tool.

Binh-Son Hua is supported by the SUTD Digital Manufacturing and Design (DManD) Centre which is supported by the National Research Foundation (NRF) of Singapore.

Lap-Fai Yu is supported by the Joseph P. Healey Research Grant Program provided by the Office of the Vice Provost for Research and Strategic Initiatives & Dean of Graduate Studies of UMass Boston. This research is also supported by the National Science Foundation under award number 1565978. We acknowledge NVIDIA Corporation for graphics card donation.

Sai-Kit Yeung is supported by Singapore MOE Academic Research Fund MOE2016-T2-2-154 and Supported by Heritage Research Grant of the National Heritage Board, Singapore. We acknowledge the support of the SUTD Digital Manufacturing and Design (DManD) Centre which is supported by the National Research Foundation (NRF) of Singapore. This research is also supported by the National Research Foundation, Prime Minister's Office, Singapore under its IDM Futures Funding Initiative and its Environmental & Water Technologies Strategic Research Programme and administered by the PUB, Singapore’s National Water Agency. This material is based on research/work supported in part by the National Research Foundation under Virtual Singapore Award No. NRF2015VSG-AA3DCM001-014.