IROS 2018 Tutorial

Creating and Understanding 3D Annotated Scene Meshes

Zhiyuan Zhang1       Binh-Son Hua2       Duc Thanh Nguyen3       Lap-Fai Yu4       Sai-Kit Yeung5       Daniela Rus6

1Singapore University of Technology and Design 2The University of Tokyo 3Deakin University
4University of Massachusetts Boston 5The Hong Kong University of Science and Technology 6Massachusetts Institute of Technology

Capturing, reconstructing, and annotating 3D scenes from real world are often known as daunting tasks in preparing a high-quality dataset for 3D scene understanding despite recent advances in color and depth sensors. In contrast to 2D image datasets which have been readily and widely available, 3D scene mesh datasets for training and testing robotics algorithms have been scarce since creating such datasets often requires huge efforts in building a robust 3D scene reconstruction and annotation pipeline.

This tutorial aims to equip its audience with general knowledge about the state-of-the-art approaches in 3D scene reconstruction and annotation, as well as the technical and implementation knowledge about how to build a complete pipeline to reconstruct and annotate 3D scenes. Several topics for building the pipeline will be extensively discussed, including data capturing, real-time and offline reconstruction, automatic and interactive annotation, quality control and benchmarking metrics. A WebGL annotation tool for 3D scene segmentation will also be demonstrated during the tutorial. Finally, as an application of the annotated scene data, we will discuss the state-of-the-art techniques in 3D deep learning, particularly in deep learning for 3D point clouds.

Tentative Schedule

Monday, Oct 01, 2018

  • 14:30 Overview of Pipeline
  • 14:45 3D Scene Reconstruction
  • 15:15 3D Scene Annotation
  • 15:45 WebGL Demo
  • 16:00 Tea Break
  • 16:30 Datasets and Applications
  • 17:00 3D Deep Learning
  • 17:30 Panel Discussion, Q&A

Materials

looks_one
Overview
looks_two
3D Reconstruction
looks_3
3D Annotation
looks_4
Datasets
looks_5
3D Deep Learning
cloud
Data
create
WebGL annotation tool
create
C++ annotation tool
bookmarks
3D Machine Learning

Organizers

Zhiyuan Zhang is now a postdoctoral researcher at the Singapore University of Technology and Design. Before that, he worked as a staff researcher in Device Plus Lab of Lenovo from 2014 to 2017. He received his PhD degree from the National University of Singapore in 2015. His research interests include 3D Geometric understanding, Deep Learning on 3D Point Clouds, and 3D Biometrics. He has publications in Pattern Recognition, Computer Graphics Forum, Pacific Graphics and ICIP.

Binh-Son Hua is currently a postdoctoral researcher in The University of Tokyo. Before that, he was a postdoctoral researcher in Singapore University of Technology and Design. He received his PhD degree in Computer Science from National University of Singapore in 2015. His research interests are 3D reconstruction, 3D scene understanding, and physically based rendering. His recent works are published in both computer graphics and vision venues, including SIGGRAPH, Eurographics, TVCG, 3DV, and CVPR.

Duc Thanh Nguyen received his Ph.D. degree in Computer Science from the University of Wollongong, Australia, in 2012. Currently, he is a lecturer at the School of Information Technology, Deakin University, Australia. His research interests include Computer Vision and Pattern Recognition. Dr. Nguyen has published his work in highly-ranked publication venues in Computer Vision and Pattern Recognition such as the Journal of Pattern Recognition, CVPR, ICCV and ECCV. He also has served a technical program committee member of the IEEE Int. Conf. Image Process. (from 2012) and reviewers of the IEEE Trans. Intell. Transp. Syst., IEEE Trans. Image Process., IEEE Signal Processing Letters, Image and Vision Computing.

Lap-Fai (Craig) Yu is an assistant professor at the University of Massachusetts at Boston. He obtained his PhD degree in computer science from UCLA in 2013. His research interests are in computer graphics and vision, especially in the topics of synthesizing and analysing 3D models from the perspectives of functionality, physics, intentionality and causality. He is the recipient of the Cisco Outstanding Graduate Research Award, the UCLA Dissertation Year Fellowship, the Sir Edward Youde Memorial Fellowship and the Award of Excellence from Microsoft Research. His research has been featured in New Scientist, the UCLA Headlines and newspapers internationally. He regularly serves on the program committee of Eurographics, Pacific Graphics and IEEE Virtual Reality.

Sai-Kit Yeung is currently an Associate Professor at the Division of Integrative Systems and Design (ISD) at the Hong Kong University of Science and Technology (HKUST). Before joining HKUST, he was an Assistant Professor at the Singapore University of Technology an Design (SUTD) and founded the Vision, Graphics and Computational Design (VGD) Group. During his time at SUTD, he was also a Visiting Assistant Professor at Stanford University and MIT. Prior to that, he had been a Postdoctoral Scholar in the Department of Mathematics, University of California, Los Angeles (UCLA). He was a visiting student at the Image Processing Research Group at UCLA in 2008 and the Image Sciences Institute, University Medical Center Utrecht, the Netherlands in 2007. He received his PhD degree in Electronic and Computer Engineering, MPhil degree in Bioengineering, and BEng degree (First Class Honors) from HKUST, respectively in 2009, 2005 and 2003. Dr. Yeung's current research focus is primarily on 3D content reconstruction, understanding, creation, redesign and fabrication.

Daniela Rus is the Andrew (1956) and Erna Viterbi Professor of Electrical Engineering and Computer Science and Director of the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT. Rus’s research interests are in robotics, mobile computing, and data science. Rus is a Class of 2002 MacArthur Fellow, a fellow of ACM, AAAI, IEEE and RAS, and a member of the National Academy of Engineering, and the American Academy for Arts and Science. She earned her PhD in Computer Science from Cornell University. Prior to joining MIT, Rus was a professor in the Computer Science Department at Dartmouth College.

Contact

Please email us at scenenn@gmail.com if you have any comments and feedback. Thank you!

Last update: Aug 06, 2018