semantic segmentation 3d
5115938
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
C Qi Hao Su Kaichun Mo Leonidas J Guibas
201645228
Linking Points With Labels in 3D: A Review of Point Cloud Semantic Segmentation
Ripe with possibilities offered by deep learning techniques and useful in applications related to remote sensing, computer vision, and robotics, 3D point cloud semantic segmentation (PCSS) and point cloud segmentation (PCS) are attracting increasing interest. This article summarizes available data sets and relevant studies on recent developments in PCSS and PCS.
IEEE Geoscience and Remote Sensing Magazine
Yuxing Xie Jiaojiao Tian Xiaoxiang Zhu
52155993
Semantic Segmentation of 3D LiDAR Data in Dynamic Scene Using Semi Supervised Learning
This work studies the semantic segmentation of 3D LiDAR data in dynamic scenes for autonomous driving applications. A system of semantic segmentation using 3D LiDAR data, including range image segmentation, sample generation, inter frame data association, track level annotation, and semi supervised learning, is developed. To reduce the considerable requirement of fine annotations, a CNN based classifier is trained by considering both supervised samples with manually labeled object classes and pairwise constraints, where a data sample is composed of a segment as the foreground and neighborhood points as the background. A special loss function is designed to account for both annotations and constraints, where the constraint data are encouraged to be assigned to the same semantic class. A dataset containing 1838 frames of LiDAR data, 39 934 pairwise constraints and 57 927 human annotations is developed. The performance of the method is examined extensively. The qualitative and quantitative experiments show that the combination of a few annotations and large amount of constraint data significantly enhances the effectiveness and scene adaptability, resulting in greater than 10% improvement.
IEEE Transactions on Intelligent Transportation Systems
Jilin Mei Biao Gao Donghao Xu Wen Yao Xijun Zhao Huijing Zhao
208512862
xMUDA: Cross Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Unsupervised Domain Adaptation (UDA) is crucial to tackle the lack of annotations in a new domain. There are many multi modal datasets, but most UDA approaches are uni modal. In this work, we explore how to learn from multi modality and propose cross modal UDA (xMUDA) where we assume the presence of 2D images and 3D point clouds for 3D semantic segmentation. This is challenging as the two input spaces are heterogeneous and can be impacted differently by domain shift. In xMUDA, modalities learn from each other through mutual mimicking, disentangled from the segmentation objective, to prevent the stronger modality from adopting false predictions from the weaker one. We evaluate on new UDA scenarios including day to night, country to country and dataset to dataset, leveraging recent autonomous driving datasets. xMUDA brings large improvements over uni modal UDA on all tested scenarios, and is complementary to state of the art UDA techniques. Code is available at https:/github.com/valeoai/xmuda.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Maximilian Jaritz Tuan-Hung Vu Raoul de Charette Emilie Wirbel Patrick Perez
211296443
3D MiniNet: Learning a 2D Representation From Point Clouds for Fast and Efficient 3D LIDAR Semantic Segmentation
LIDAR semantic segmentation is an essential task that provides 3D semantic information about the environment to robots. Fast and efficient semantic segmentation methods are needed to match the strong computational and temporal restrictions of many real world robotic applications. This work presents 3D MiniNet, a novel approach for LIDAR semantic segmentation that combines 3D and 2D learning layers. It first learns a 2D representation from the raw points through a novel projection which extracts local and global information from the 3D data. This representation is fed to an efficient 2D Fully Convolutional Neural Network (FCNN) that produces a 2D semantic segmentation. These 2D semantic labels are re projected back to the 3D space and enhanced through a post processing module. The main novelty in our strategy relies on the projection learning module. Our detailed ablation study shows how each component contributes to the final performance of 3D MiniNet. We validate our approach on well known public benchmarks (SemanticKITTI and KITTI) where 3D MiniNet gets state of the art results while being faster and more parameter efficient than previous methods.
IEEE Robotics and Automation Letters
Inigo Alonso Luis Riazuelo Luis Montesano Ana Cristina Murillo
209439847
JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds
In this paper, we propose a novel joint instance and semantic segmentation approach, which is called JSNet, in order to address the instance and semantic segmentation of 3D point clouds simultaneously. Firstly, we build an effective backbone network to extract robust features from the raw point clouds. Secondly, to obtain more discriminative features, a point cloud feature fusion module is proposed to fuse the different layer features of the backbone network. Furthermore, a joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and then the transformed features are further fused with instance features to facilitate instance segmentation. Meanwhile, this module also aggregates instance features into semantic feature space to promote semantic segmentation. Finally, the instance predictions are generated by applying a simple mean shift clustering on instance embeddings. As a result, we evaluate the proposed JSNet on a large scale 3D indoor point cloud dataset S3DIS and a part dataset ShapeNet, and compare it with existing approaches. Experimental results demonstrate our approach outperforms the state of the art method in 3D instance segmentation with a significant improvement in 3D semantic prediction and our method is also beneficial for part segmentation. The source code for this work is available at this https URL.
AAAI
Lin Zhao Wenbing Tao
10154243
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
Convolutional networks are the de facto standard for analyzing spatio temporal data such as images, videos, and 3D shapes. Whilst some of this data is naturally dense (e.g. photos) many other data sources are inherently sparse. Examples include 3D point clouds that were obtained using a LiDAR scanner or RGB D camera. Standard "dense" implementations of convolutional networks are very inefficient when applied on such sparse data. We introduce new sparse convolutional operations that are designed to process spatially sparse data more efficiently, and use them to develop spatially sparse convolutional networks. We demonstrate the strong performance of the resulting models, called submanifold sparse convolutional networks (SS CNs) on two tasks involving semantic segmentation of 3D point clouds. In particular, our models outperform all prior state of the art on the test set of a recent semantic segmentation competition.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Benjamin Graham Martin Engelcke Laurens van der Maaten
203591957
Point Attention Network for Semantic Segmentation of 3D Point Clouds
Convolutional Neural Networks (CNNs) have performed extremely well on data represented by regularly arranged grids such as images. However, directly leveraging the classic convolution kernels or parameter sharing mechanisms on sparse 3D point clouds is inefficient due to their irregular and unordered nature. We propose a point attention network that learns rich local shape features and their contextual correlations for 3D point cloud semantic segmentation. Since the geometric distribution of the neighboring points is invariant to the point ordering, we propose a Local Attention Edge Convolution (LAE Conv) to construct a local graph based on the neighborhood points searched in multi directions. We assign attention coefficients to each edge and then aggregate the point features as a weighted sum of its neighbors. The learned LAE Conv layer features are then given to a point wise spatial attention module to generate an interdependency matrix of all points regardless of their distances, which captures long range spatial contextual features contributing to more precise semantic information. The proposed point attention network consists of an encoder and decoder which, together with the LAE Conv layers and the point wise spatial attention modules, make it an end to end trainable network for predicting dense labels for 3D point cloud segmentation. Experiments on challenging benchmarks of 3D point clouds show that our algorithm can perform at par or better than the existing state of the art methods.
Pattern Recognit.
Mingtao Feng Liang Zhang Xuefei Lin Syed Zulqarnain Gilani Ajmal S Mian
220961751
Cylinder3D: An Effective 3D Framework for Driving scene LiDAR Semantic Segmentation
State of the art methods for large scale driving scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. The projection methods includes spherical projection, bird eye view projection, etc. Although this process makes the point cloud suitable for the 2D CNN based networks, it inevitably alters and abandons the 3D topology and geometric relations. A straightforward solution to tackle the issue of 3D to 2D projection is to keep the 3D representation and process the points in the 3D space. In this work, we first perform an in depth analysis for different representations and backbones in 2D and 3D spaces, and reveal the effectiveness of 3D representations and networks on LiDAR segmentation. Then, we develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving scene point clouds. Moreover, a dimension decomposition based context modeling module is introduced to explore the high rank context information in point clouds in a progressive manner. We evaluate the proposed model on a large scale driving scene dataset, i.e. SematicKITTI. Our method achieves state of the art performance and outperforms existing methods by 6% in terms of mIoU.
ArXiv
Hui Zhou Xinge Zhu Xiao Song Yuexin Ma Zhe Wang Hongsheng Li Dahua Lin
220514614
JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds
Semantic segmentation and semantic edge detection can be seen as two dual problems with close relationships in computer vision. Despite the fast evolution of learning based 3D semantic segmentation methods, little attention has been drawn to the learning of 3D semantic edge detectors, even less to a joint learning method for the two tasks. In this paper, we tackle the 3D semantic edge detection task for the first time and present a new two stream fully convolutional network that jointly performs the two tasks. In particular, we design a joint refinement module that explicitly wires region information and edge information to improve the performances of both tasks. Further, we propose a novel loss function that encourages the network to produce semantic segmentation results with better boundaries. Extensive evaluations on S3DIS and ScanNet datasets show that our method achieves on par or better performance than the state of the art methods for semantic segmentation and outperforms the baseline methods for semantic edge detection. Code release: this https URL
ECCV
Zeyu Hu Mingmin Zhen Xuyang Bai Hongbo Fu Chiew-Lan Tai