fully convolutional networks for semantic segmentation.
1629541
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end to end, pixels to pixels, improve on the previous best result in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional networks achieve improved segmentation of PASCAL VOC (30% relative improvement to 67.2% mean IU on 2012) NYUDv2, SIFT Flow, and PASCAL Context, while inference takes one tenth of a second for a typical image.
IEEE Transactions on Pattern Analysis and Machine Intelligence
Evan Shelhamer Jonathan Long Trevor Darrell
56507745
Fully convolutional networks for semantic segmentation
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end to end, pixels to pixels, exceed the state of the art in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20] the VGG net [31] and GoogLeNet [32] into fully convolutional networks and transfer their learned representations by fine tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state of the art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012) NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Jonathan Long Evan Shelhamer Trevor Darrell
3545651
Fully Convolutional Networks for Semantic Segmentation of Very High Resolution Remotely Sensed Images Combined With DSM
Recently, approaches based on fully convolutional networks (FCN) have achieved state of the art performance in the semantic segmentation of very high resolution (VHR) remotely sensed images. One central issue in this method is the loss of detailed information due to downsampling operations in FCN. To solve this problem, we introduce the maximum fusion strategy that effectively combines semantic information from deep layers and detailed information from shallow layers. Furthermore, this letter develops a powerful backend to enhance the result of FCN by leveraging the digital surface model, which provides height information for VHR images. The proposed semantic segmentation scheme has achieved an overall accuracy of 90.6% on the ISPRS Vaihingen benchmark.
IEEE Geoscience and Remote Sensing Letters
Weiwei Sun Ruisheng Wang
13746462
Symmetrical Dense Shortcut Deep Fully Convolutional Networks for Semantic Segmentation of Very High Resolution Remote Sensing Images
Semantic segmentation has emerged as a mainstream method in very high resolution remote sensing land use/land cover applications. In this paper, we first review the state of the art semantic segmentation models in both computer vision and remote sensing fields. Subsequently, we introduce two semantic segmentation frameworks: SNFCN and SDFCN, both of which contain deep fully convolutional networks with shortcut blocks. We adopt an overlay strategy as the postprocessing method. Based on our frameworks, we conducted experiments on two online ISPRS datasets: Vaihingen and Potsdam. The results indicate that our frameworks achieve higher overall accuracy than the classic FCN 8s and SegNet models. In addition, our postprocessing method can increase the overall accuracy by about 1% 2% and help to eliminate "salt and pepper" phenomena and block effects.
IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Guanzhou Chen Xiaodong Zhang Qing Wang Fan Dai Yuanfu Gong Kun Zhu
20167786
An Adaptive Sampling Scheme to Efficiently Train Fully Convolutional Networks for Semantic Segmentation
Deep convolutional neural networks (CNNs) have shown excellent performance in object recognition tasks and dense classification problems such as semantic segmentation. However, training deep neural networks on large and sparse datasets is still challenging and can require large amounts of computation and memory. In this work, we address the task of performing semantic segmentation on large data sets, such as three dimensional medical images. We propose an adaptive sampling scheme that uses a posterior error maps, generated throughout training, to focus sampling on difficult regions, resulting in improved learning. Our contribution is threefold: 1) We give a detailed description of the proposed sampling algorithm to speed up and improve learning performance on large images. We propose a deep dual path CNN that captures information at fine and coarse scales, resulting in a network with a large field of view and high resolution outputs. We show that our method is able to attain new state of the art results on the VISCERAL Anatomy benchmark.
MIUA
Lorenz Berger Eoin R Hyde M Jorge Cardoso Sebastien Ourselin
3429309
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution' as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields of views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF) which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state of art at the PASCAL VOC 2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL Context, PASCAL Person Part, and Cityscapes. All of our code is made publicly available online.
IEEE Transactions on Pattern Analysis and Machine Intelligence
Liang-Chieh Chen George Papandreou Iasonas Kokkinos Kevin P Murphy Alan Loddon Yuille
5093704
Fully Convolutional Adaptation Networks for Semantic Segmentation
The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets. Nevertheless, collecting expert labeled datasets especially with pixel level annotations is an extremely expensive process. An appealing alternative is to render synthetic data (e.g. computer games) and generate ground truth automatically. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. In this paper, we facilitate this issue from the perspectives of both visual appearance level and representation level domain adaptation. The former adapts source domain images to appear as if drawn from the "style" in the target domain and the latter attempts to learn domain invariant representations. Specifically, we present Fully Convolutional Adaptation Networks (FCAN) a novel deep architecture for semantic segmentation which combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN) AAN learns a transformation from one domain to the other in the pixel space and RAN is optimized in an adversarial learning manner to maximally fool the domain discriminator with the learnt source and target representations. Extensive experiments are conducted on the transfer from GTA5 (game videos) to Cityscapes (urban street scenes) on semantic segmentation and our proposal achieves superior results when comparing to state of the art unsupervised adaptation techniques. More remarkably, we obtain a new record: mIoU of 47.5% on BDDS (drive cam videos) in an unsupervised setting.
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Yiheng Zhang Zhaofan Qiu Ting Yao Dong Liu Tao Mei
12916864
Multi class Semantic Segmentation of Skin Lesions via Fully Convolutional Networks
Early detection of skin cancer, particularly melanoma, is crucial to enable advanced treatment. Due to the rapid growth of skin cancers, there is a growing need of computerized analysis for skin lesions. These processes including detection, classification, and segmentation. There are three main types of skin lesions in common that are benign nevi, melanoma, and seborrhoeic keratoses which have huge intra class variations in terms of color, size, place and appearance for each class and high inter class visual similarities in dermoscopic images. The majority of current research is focusing on melanoma segmentation, but it is also very important to segment the seborrhoeic keratoses and benign nevi lesions as these regions potentially indicate the pre cancer stage. We propose a multiclass semantic segmentation for these three classes from publicly available ISBI 2017 challenge dataset which consists of 2750 dermoscopic images. We propose an end to end solution using fully convolutional networks (FCNs) for multi class semantic segmentation, which will automatically segment the melanoma, keratoses and benign lesions. To overcome the issue of data deficiency, we propose a transfer learning approach which uses both partial transfer learning and full transfer learning to train FCNs for multi class semantic segmentation of skin lesions. The results are presented in Dice Similarity Coefficient (Dice) to compare the performance of the deep learning segmentation methods on the dataset with 5 fold cross validation. The results showed that the two tier level transfer learning FCN 8s achieved the overall best result with Dice score of 0.785 in a benign category, 0.653 in melanoma segmentation, and 0.557 in seborrhoeic keratoses.
BIOINFORMATICS
Manu Goyal Moi Hoon Yap
18717559
PCA aided fully convolutional networks for semantic segmentation of multi channel fMRI
Semantic segmentation of functional magnetic resonance imaging (fMRI) makes great sense for pathology diagnosis and decision system of medical robots. The multi channel fMRI provides more information of the pathological features. But the increased amount of data causes complexity in feature detections. This paper proposes a principal component analysis (PCA) aided fully convolutional network to particularly deal with multi channel fMRI. We transfer the learned weights of contemporary classification networks to the segmentation task by fine tuning. The results of the convolutional network are compared with various methods e.g. k NN. A new labeling strategy is proposed to solve the semantic segmentation problem with unclear boundaries. Even with a small sized training dataset, the test results demonstrate that our model outperforms other pathological feature detection methods. Besides, its forward inference only takes 90 milliseconds for a single set of fMRI data. To our knowledge, this is the first time to realize pixel wise labeling of multi channel magnetic resonance image using FCN.
2017 18th International Conference on Advanced Robotics (ICAR)
Lei Tai Haoyang Ye Qiong Ye Ming Liu
215917123
Fully Convolutional Networks for Semantic Segmentation
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end to end, pixels to pixels, improve on the previou.