arXiv:1703.03372v3 [] 15 Mar 2017

LesionSeg: Semantic segmentation of skin lesions
using Deep Convolutional Neural Network

Dhanesh Ramachandram
School of Engineering
University of Guelph
Guelph, ON N1G 2W1, Canada
dramacha@uoguelph.ca

Terrance DeVries
School of Engineering
University of Guelph
Guelph, ON N1G 2W1, Canada
terrance@uoguelph.ca

Executive Summary
We present a method for skin lesion segmentation for the ISIC 2017 Skin Lesion Segmentation
Challenge. Our approach is based on a Fully Convolutional Neural Network architecture which is
trained end to end, from scratch, on a small dataset. Our semantic segmentation architecture utilizes
several recent innovations in deep learning particularly in the combined use of (i) atrous convolutions
to increase the effective field of view of the network’s receptive field without increasing the number
of parameters, (ii) network-in-network 1 × 1 convolution layers to increase network capacity and (iii)
state-of-art super-resolution upsampling of predictions using subpixel CNN layers. We achieved a
IOU score of 0.642 on the validation set provided by the organizers.

1

Background

One of the fundamental and challenging tasks in digital image analysis is segmentation, which is the
process of assigning pixel-wise labels to regions in an image that share some high-level semantics,
hence the term “semantic segmentation”. In skin lesion segmentation, the goal is to assign pixel-wise
labels to regions in dermoscopy images that represents skin lesions, such as melanoma, seborrhoeic
keratosis or benign nevus.
Skin lesion segmentation is challenging due to a variety of factors, such as variations in skin tone,
uneven illumination, partial obstruction due to the presence of hair, low contrast between lesion and
surrounding skin, and the presence of freckles or gauze in the image frame, which may be mistaken
for lesions. A successful lesion segmentation technique should be robust enough to accommodate
these variability.
Skin lesion segmentation is a widely researched topic in medical image analysis[1–3]. Until recently,
most skin lesion segmentation approaches were based on hand crafted algorithms[4–6]. Such
approaches require carefully designed pre-processing and post-processing approaches such as hair
removal, edge-preserving smoothing and morphological operations. The robustness of the such
approaches can be somewhat limited however, as each new scenario may require custom tuning.
An alternative approach to manually crafting segmentation algorithms is to instead leverage machine
learning techniques to learn a model capable of successfully dealing with the numerous factors of
variation. Specifically applicable to this application are artificial neural networks, which have made
an impressive resurgence in recent years. The active research area, now known as deep learning, is
currently enjoying interest and support not only from the academic community, but also from industry.
The advances in hardware performance and low costs involved have made it viable to analyse very
large data sizes using very deep neural network architectures in a reasonable amount of time. Deep
learning-based techniques have resulted in state-of-the-art performance for many practical problems,
especially in areas involving high-dimensional unstructured data such as in computer vision, speech
and natural language processing. Medical imaging problems have also been given much attention by
2017 ISIC Skin Lesion Segmentation Challenge

deep learning researchers[7] and has seen tremendous success for the related skin lesion classification
problem[8]. The notion of being able to train in an end-to end manner, without requiring any manual
feature engineering or complicated hand-crafted algorithms, is very attractive indeed.

2

Semantic Segmentation

The deep learning approach to image segmentation is known as semantic segmentation. In contrast to
low-level image segmentation, which operate purely on local image characteristics such as colour,
shape and texture, deep semantic segmentation algorithms are trained using thousands of examples
to recognize and delineate regions in an image corresponding to some high-level semantics. A
convolutional neural network can be adapted to perform perform semantic segmentation by replacing
the top layer1 of a classification network into a convolutional layer. As FCNN use downsampling
implemented via the max-pooling or strided convolutions to capture context, these architectures often
employ a single or several progressive upsampling layers which are used to upscale lower resolution
pixel-wise predictions to match the dimensions of the input image. Ground truth segmentation masks
provide pixel-wise labels for the segmentation task, now cast as a pixel-wise classification problem.
The fully convolutional neural network architecture was first proposed by Long et al.[9]. Subsequently,
a number of similar architectures have been reported in the literature[10, 11].
For the ISIC 2017 challenge, the skin lesion segmentation task is a binary segmentation task - the
goal is to produce accurate segmentation of various skin lesions, benign and malignant, against a
variety of background which may consist of skin, colored markers, or dark-vignetted region produced
by the dermoscope. Fig.?? shows an example lesion and its corresponding binary mask. The training
dataset consists of 2000 dermoscopy images and corresponding binary masks. The images consists of
3 types of skin lesions: nevus, seborrhoeic keratosis and melanoma; the latter lesion being malignant.
The images are also of various dimensions.

Figure 1: Left: An example of a skin lesion image with a blue marker in the background. Right: The
corresponding ground truth binary segmentation mask.

3

Our Approach

Our approach for lesion segmentation is primarily a fully convolutional neural network, trained from
scratch, in an end-to-end manner.
3.1

Architecture

The inputs to the network are images resized to 448 × 448. The first convolution layer uses a stride
of 2 to downsample the image by a factor of 2. This is followed by series of 2D convolution layers
interspersed with 1 × 1 convolutions. We also apply batch-normalization for the convolutional
layers and use ReLu activations. The 1 × 1 convolution layers add capacity to the network without
increasing the number of parameters.The last two convolutions are dilated convolutional layers or
1

typically the softmax layer

2

Layer

Filter Size

Stride/Rate

Padding

Non-linearity

Conv1-1
Conv1-2
Conv1-3
Conv2-1
Conv2-2
Conv2-3
Conv3-1
Conv3-2
Conv3-3
Subpixel

5x5x64
3x3x96
1x1x96
3x3x128
3x3x256
1x1x256
3x3x256
3x3x256
3x3x128
3x3x32

2
1
1
2
1
1
2
2
2
1

Mirror
Mirror
Same
Mirror
Mirror
Same
Mirror
Mirror
Mirror
Mirror

ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
ReLu
None
None

Type
2D Convolution
2D Convolution
2D Convolution
2D Convolution
2D Convolution
2D Convolution
Atrous Convolution
Atrous Convolution
Atrous Convolution
Subpixel CNN Layer

Table 1: LesionSeg Architecture

atrous convolutions[12] with a rate of 2. These two layers effectively enlarge the field of view of
filters to incorporate larger context without increasing the number of parameters or the amount of
computation. Before upsampling is performed, the final subpixel convolution layer is added. This
layer has number of filters equal to the upsampling factor and is applied with a stride of 1 and without
any non-linearities. The upsampling layer is a subpixel convolutional layer introduced by [13] which
produced state-of-art super-resolution reconstruction accuracy superior to the commonly used bilinear
upsampling as used by [9].
3.2

Preprocessing

The input images and the corresponding ground-truth masks are resized to 448 by 448 pixels. We
perform data augmentation on-the-fly by randomly rotating both the image and its mask by 90-degree
increments as well as flipping the images. In addition, we also perform per-image-standardization of
the input image.
3.3

Training

We trained the network using Adam[14] optimization using a per-pixel cross-entropy loss function.
During training, we performed randomly sampled images using a batch size of 32. The network is
trained until no improvement in the mean IOU is observed.
3.4

Post-processing

As the test and validation sets have images of different resolutions up to 6688 × 4439 pixels,
we resorted to upscaling the network output back to its original image dimensions using bicubic
interpolation and then binarizing the upsampled output mask using a threshold of 128. We also apply
morphological opening to eliminate small, spurious errors made by the semantic segmentation using
a 3 × 3 disk-shaped kernel.

4

Results and Discussion

We achieved a IOU score of 0.642 for the validation set using our approach. Example segmentation
outputs from our lesion segmentation architecture is shown in Fig.2.

3

(a) Sample 1

(b) Sample 2

(c) Sample 3

(d) Sample 4

Figure 2: Examples of segmentation output using our approach.

References
[1] M Emre Celebi, Quan Wen, Hitoshi Iyatomi, Kouhei Shimizu, Huiyu Zhou, and Gerald Schaefer.
A state-of-the-art survey on lesion border detection in dermoscopy images, 2015.
[2] Konstantin Korotkov and Rafael Garcia. Computerized analysis of pigmented skin lesions: a
review. Artificial intelligence in medicine, 56(2):69–90, 2012.
[3] M Emre Celebi, Hitoshi Iyatomi, Gerald Schaefer, and William V Stoecker. Lesion border
detection in dermoscopy images. Computerized medical imaging and graphics, 33(2):148–153,
2009.
[4] Huiyu Zhou, Gerald Schaefer, M Emre Celebi, Faquan Lin, and Tangwei Liu. Gradient vector
flow with mean shift for skin lesion segmentation. Computerized Medical Imaging and Graphics,
35(2):121–127, 2011.
[5] Xiaojing Yuan, Ning Situ, and George Zouridakis. A narrow band graph partitioning method
for skin lesion segmentation. Pattern Recognition, 42(6):1017–1028, 2009.
[6] Gerald Schaefer, Maher I Rajab, M Emre Celebi, and Hitoshi Iyatomi. Colour and contrast
enhancement for improved skin lesion segmentation. Computerized Medical Imaging and
Graphics, 35(2):99–104, 2011.
[7] Ge Wang. A perspective on deep imaging. IEEE Access, 4:8914–8924, 2016.
[8] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and
Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks.
Nature, 542(7639):115–118, 2017.
[9] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 3431–3440, 2015.
[10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv
preprint arXiv:1412.7062, 2014.
[11] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional
encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561, 2015.
4

[12] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and
fully connected crfs. arXiv preprint arXiv:1606.00915, 2016.
[13] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop,
Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an
efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages 1874–1883, 2016.
[14] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.

5

