Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 1
Bottom-up Instance Segmentation using
Deep Higher-Order CRFs
Anurag Arnab
anurag.arnab@eng.ox.ac.uk
Philip H.S. Torr
philip.torr@eng.ox.ac.ukDepartment of Engineering Science
University of Oxford
United Kingdom
Abstract
Traditional Scene Understanding problems such as Object Detection and Semantic
Segmentation have made breakthroughs in recent years due to the adoption of deep learn-
ing. However, the former task is not able to localise objects at a pixel level, and the latter
task has no notion of different instances of objects of the same class. We focus on the task
of Instance Segmentation which recognises and localises objects down to a pixel level.
Our model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable higher or-
der potentials based on object detector outputs. This allows us to reason about instances
from an initial, category-level semantic segmentation. Our simple method effectively
leverages the great progress recently made in semantic segmentation and object detec-
tion. The accurate instance-level segmentations that our network produces is reﬂected by
the considerable improvements obtained over previous work at high APrIoU thresholds.
1 Introduction
Object detection and semantic segmentation have been two of the most popular Scene Under-
standing problems within the Computer Vision community. Great advances have been made
in recent years due to the adoption of deep learning and Convolutional Neural Networks
[19, 22, 32]. In this paper, we focus on the problem of Instance Segmentation . Instance
Segmentation lies at the intersection of Object Detection – which localises different objects
at a bounding box level, but does not segment them – and Semantic Segmentation – which
determines the object-class label of each pixel in the image, but has no notion of different
instances of the same class. As shown in Figure 1, the task of instance segmentation localises
objects to a pixel level.
Many recent instance segmentation works have built on the “Simultaneous Detection
and Segmentation” (SDS) approach of Hariharan et al. [13]. These methods [4, 5, 13, 14]
generate object proposals [1], classify each proposal into an object category, and then reﬁne
the bounding box proposal into a segmentation of the primary object the proposal contains.
However, because these methods localise an object before segmenting it, they are restricted
by the quality of the initial proposal. Moreover, the proposal generator [1] these works used
takes about 40 seconds to process an image.
c/circlecopyrt2016. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
arXiv:1609.02583v1  [cs.CV]  8 Sep 2016
2 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Our method uses the results of an object detector and the original image (a) to pro-
duce a category-level semantic segmentation (b). The semantic segmentation result, object
detection bounding boxes, and recalibrated detection scores (§3.2) are then used to reﬁne
the semantic segmentation into an instance segmentation (c). The semantic segmentation,
in which a large number of pixels have been correctly labelled as “person”, does not tell us
how many people there are and where each person is in the image. The instance segmenta-
tion answers these questions. Also note how our method is not affected by the false-positive
“bottle” detections.
We present a different approach to instance segmentation, where we initially perform
a category-level, semantic segmentation of the image before reasoning about the different
instances in the scene. We can identify instances because our semantic segmentation network
incorporates a Higher Order Conditional Random Field (CRF) which uses cues from the
output of an object detector. This CRF is fully differentiable and can be inserted as a layer
of a neural network [2, 37]. As a result, it can be used as a simple extension to an existing
semantic segmentation network to perform the related task of instance segmentation.
Our simple, bottom-up method is able to effectively leverage the progress made by state-
of-the-art semantic segmentation and object detection networks to perform the related task
of instance segmentation. We show this by evaluating our approach on the PASCAL VOC
2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to
produce accurate instance segmentations is reﬂected by the considerable improvements that
we make over previous work at high overlap thresholds.
2 Related Work
Instance segmentation is closely related to the widely-studied problems of semantic segmen-
tation and object detection. We thus review these two tasks ﬁrst.
Semantic Segmentation: This task was traditionally performed by extracting dense fea-
tures from an image and classifying them on a per-pixel or per-region basis [16, 31]. The
individual predictions made by these classiﬁers were often noisy as they lacked global con-
text. As a result, they were post-processed with a CRF whose priors encourages nearby
pixels, and pixels of similar appearance, to share the same class label. The CRF of [31]
initially contained only unary terms (obtained from the classiﬁer) and 8-connected pairwise
terms. Subsequent improvements included introducing densely-connected pairwise poten-
tials [18], using higher order potentials encouraging consistency over cliques larger than two
pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of
object detectors [21, 33, 36].
Using CNNs for semantic segmentation can greatly improve the unary, per-pixel predic-
tions, as shown by [27]. Chen et al. showed further improvements by post-processing the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 3
CNN output with a CRF [3]. Current state-of-the-art methods [2, 25, 37] incorporate infer-
ence of a CRF as layers within a deep network which perform the differentiable mean ﬁeld
inference algorithm. In our work, we extend the end-to-end trainable, higher-order, object
detection potentials proposed by [2] to the task of instance segmentation. Additionally, un-
like the aforementioned works which always assume a ﬁxed number of labels in the CRF at
all times, our system includes a CRF where the number of labels varies per input image.
Object Detection: Part-based models [9] were popular before CNNs, which had been
shown to excel in the task of image classiﬁcation [19, 32], were adapted to object detec-
tion. Recent detection algorithms are based on using CNNs to classify object proposals
and then performing bounding box regression to reﬁne the initial proposals’ bounding boxes
[10, 11, 12, 28]. This pipeline can be accelerated by computing convolutional features over
the whole image, and then pooling features for each proposal [11, 15]. Object proposals
were initially hand-crafted [1, 34], but this step has now been integrated into CNNs [28].
Instance Segmentation: This task was popularised by Hariharan et al. [13]. Their ap-
proach was similar to object detection pipelines in that region proposals were ﬁrst gener-
ated and classiﬁed into different object categories before using bounding box regression as
post-processing. A class-speciﬁc segmentation was then performed to simultaneously detect
and segment an object. However, approaches that segment instances by reﬁning detections
[4, 5, 13, 14] are inherently limited by the quality of the initial proposals, and the systems
cannot recover from errors made by the initial detector. Furthermore, since these methods
generate and classify multiple overlapping region proposals per image, they cannot actually
produce segmentation maps of the form shown in Figures 1(b) and (c) [24]. Liang et al. [23]
address some of these shortcomings by iteratively reﬁning initial object proposals, whilst [7]
generates box-proposals, creates masks from these proposals and then classiﬁes these masks
in a single network trained with three loss functions summed together.
A proposal-free method was recently presented by [24] where a semantic segmenta-
tion of the image is ﬁrst performed using the network of [3]. Thereafter the category-level
segmentation output, along with CNN features, are used to predict instance-level bounding
boxes for the objects in the image. The number of instances of each object category are also
predicted in order to facilitate the ﬁnal clustering step. An alternate method was proposed
by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.
Our method also ﬁrst performs a bottom-up, semantic segmentation of the image. How-
ever, our semantic segmentation network uses the outputs of an object detector as an addi-
tional cue in its ﬁnal CRF layer. During inference, each object hypothesis from the detector
is evaluated in light of other energies in the CRF. Thus, the relative score of false positive
detections can be decreased, and correct detections increased. These rescored detections
are then used to identify instances from the initial semantic segmentation. The fact that our
system combines information from semantic segmentation and object detection networks al-
lows us to bypass the complexity of the “instance bounding box” and “instance number”
prediction steps used by [24].
3 Proposed Approach
Our proposed method ﬁrst performs a semantic segmentation of the input image, classifying
each pixel into one of K+1 categories where Kis the number of foreground classes. The
resulting semantic segmentation is then reﬁned into an instance-level segmentation, where
4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF  
Instance Identification  Instance CRF  
K+1  
Object Detector  
D+1  End-to-end object segmentation subnetwork  
End-to-end Instance segmentation subnetwork  
Input  
Output  Object  
Segmentation  
Identified  
Instances  
Figure 2: Overview of our end-to-end method. Our system ﬁrst computes a category-level
semantic segmentation of the image. A CRF with higher-order detection potentials is used
to obtain this result in a semantic segmentation subnetwork. This results in a W×H×(K+
1)dimensional volume where Kis the number of foreground classes. WandHare the
image’s width and height respectively. The original detections and the detector conﬁdences
recalibrated by the Higher Order CRF are used to identify instances in the image, producing
aW×H×(D+1)dimensional volume where Dis the number of detections (variable per
image). Thereafter, another Instance CRF is used to compute the ﬁnal result.
the object class of each instance segment is obtained from the previous semantic segmenta-
tion. Both of these stages, while conceptually different, are fully differentiable and the entire
system can be implemented as a neural network.
Figure 2 shows our category-level segmentation network, consisting of a pixelwise CNN
[27] followed by a fully differentiable CRF with object detection potentials, as described in
Section 3.2. The object detection results that our CRF takes in, as well as the recalibrated
detection scores that it outputs, allows us to identify object instances, and obtain a ﬁnal
instance-level segmentation in the next stage of our pipeline. This is described in Section
3.3. However, we ﬁrst review Conditional Random Fields and introduce the notation used in
this paper.
3.1 Conditional Random Fields
Assume an image Iwith Npixels, indexed 1 ,2...N, and deﬁne a set of random variables,
X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a
label from a predeﬁned set of labels Lsuch that each Xi∈L. Let X= [X1X2...XN]T. In
this case, any particular assignment xtoXis a solution to our labelling problem.
In the case of semantic segmentation, we assign each pixel, Xi, a label corresponding
to object categories such as “person” and “car”. In instance segmentation, the labels are
drawn from the product label space of object categories and instance numbers. Examples are
“person_1” and “person_2” as shown by the different colours in Figure 1.
Given a graph Gwhere the vertices are from {X}and connections between these vari-
ables are deﬁned by edges, the pair (I,X)is modelled as a CRF deﬁned by Pr (X=x|I) =
(1/Z(I))exp(−E(x|I)). The term, E(x|I)is the energy of the labelling xandZ(I)is the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5
data-dependent partition function. From here onwards, we drop the conditioning on Ito
keep the notation uncluttered. The energy of an assignment is deﬁned over the set of cliques
Cin the graph Gas
E(x) =∑
c∈Cψc(xc), (1)
where xcis the vector formed by selecting elements of xthat correspond to random variables
belonging to the clique c, andψc(.)is the cost function for a clique c.
The energy function considered by many semantic segmentation works, E(x) =∑iψU
i(xi)+
∑i<jψP
i j(xi,xj), consists of only unary (cliques of size one) and densely-connected pairwise
potentials (cliques of size two) [3, 18, 37]. When dense pairwise potentials are used in a
CRF to obtain higher accuracy, exact inference is not tractable and approximate inference
methods such as mean ﬁeld are used [18].
3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a
higher order term based on object detections. Intuitively, a state-of-the-art object detector
[11, 28] can help segmentation performance in cases where the detector correctly predicts an
object at a location in which the pixels have been misclassiﬁed by the unary classiﬁer, and
thus reduce the recognition error [6]. Arnab et al. [2] formulated object detection potentials
in a manner that is amenable to the differentiable mean ﬁeld inference algorithm so that it
could be integrated into a neural network, and showed that it does indeed improve semantic
segmentation performance. We review the detection potentials described in [2] as they also
help us to reason about instances in an input image, a problem not considered by [2].
Assume that we have Dobject detections for an input image (this number varies for every
image), and that the dthdetection is of the form (ld,sd,Fd,Bd)where ld∈Lis the object class
label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set
of indices of the pixels belonging to the foreground of the detection and Bd⊂{1,2,...N}is
the set of indices falling in the detection’s bounding box. The foreground is obtained from
a foreground/background segmentation method such as GrabCut [30] and indicates a rough
segmentation of the detected object. The detection potentials should encourage the set of
pixels represented by Fdto take the object class label, ld. However, this should be a soft
constraint, since the foreground segmentation could be inaccurate and the entire detection
itself could be a false-positive. The detection dshould also be identiﬁed as invalid if other
energies in the CRF strongly suggest that many pixels in Fddo not take the label ld.
This is formulated by introducing a latent binary random variable, Y1,Y2...YDfor every
detection. If the dthdetection has been found to be valid after inference, Ydwill be set to 1,
and 0 otherwise. The ﬁnal value of Ydis determined probabilistically by mean ﬁeld inference.
All latent Ydvariables are added to the CRF which previously only contained Xivariables.
Let each (Xd,Yd), where{Xd}={Xi∈{X}|i∈Fd}, form a clique in the CRF. An assignment
(xd,yd)to the clique (Xd,Yd)has the energy:
ψDet
d(Xd=xd,Yd=yd) =

wlsd
|Fd|∑|Fd|
i=1[x(i)
d=ld]ifyd=0,
wlsd
|Fd|∑|Fd|
i=1[x(i)
d/negationslash=ld]ifyd=1,(2)
where x(i)
dis the ithelement in vector xd,[.]is the Iverson bracket and wlis a class-speciﬁc,
learnable weight parameter. This potential encourages consistency among X(i)
dvariables and
6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Ydsince it encourages X(i)
dvariables to take the label ldwhen Ydis 1, and also encourages Yd
to be 0 when many X(i)
dvariables do not take the label ld.
A unary potential for the latent Yvariables is also included, which is initialised to the
conﬁdence score sdof the object detector (in addition to the joint potential with the Xvari-
ables). This initial conﬁdence then changes throughout CRF inference, depending on the
other potentials. The ﬁnal probability of the latent Yvariable can thus be viewed as a recal-
ibrated detection score, which gets adjusted during inference based on how much it agrees
with the segmentation unaries and pairwise potentials. Note that this “recalibration property”
of the latent Yvariables was not considered by [2]. The ﬁnal form of the energy function
(Eq 1), is thus
E(x) =∑
iψU
i(xi)+∑
i<jψP
i,j(xi,xj)+∑
dψDet
d(xd,yd)+∑
dψU
d(yd). (3)
3.3 Instance Identiﬁcation and Reﬁnement
Once we have a category-level segmentation of the image, each pixel still needs to be as-
signed to an object instance. We assume that each object detection represents a possible
instance. Since there are Dobject detections (where Dvaries for every image), and some
pixels do not belong to any object instance, but are part of the background, we have a la-
belling problem involving D+1 labels. Our set of labels, D, is thus{1,2,...D+1}.
A naïve solution to the problem of recovering instances from a category-level segmen-
tation would be as follows: If a pixel falls within the bounding box of a detection, and its
category label agrees with the class predicted by the object detector, then it is assigned to the
instance represented by that detection. However, this method cannot deal with overlapping
bounding boxes which are typical when objects occlude one another (as in Figs. 1 and 2).
Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to
that instance with a probability proportional to the rescored detection (obtained from the
probability of the latent Yvariable after inference) and the semantic segmentation conﬁdence
for that class, as shown in Equation 4
Pr(vi=k) =/braceleftBigg1
Z(Y,Q)Qi(lk)Pr(Yk=1)ifi∈Bk
0 otherwise .(4)
Here, viis a multinomial random variable indicating the “identiﬁed instance” at pixel iand
takes on labels from D,Qi(l)is the output of the initial category-level segmentation stage
of our network and denotes the probability of pixel itaking the label l∈L, and Z(Y,Q)
normalises the probability in case there are multiple bounding boxes overlapping the same
pixel. For this formulation to be valid, we also add another detection, d0, denoting back-
ground pixels that are not overlapped by any detection. This then acts as the unary potentials
of another CRF with the energy:
E(v) =∑
iψU
i(vi)+∑
i<jψP
i,j(vi,vj) ψU
i(vi) =−lnPr(vi). (5)
This Instance CRF contains unary, ψU
i(vi), and densely-connected pairwise terms, ψP
i,j(vi,vj),
encouraging appearance and spatial consistency [18]. These priors are valid in the case of
instance segmentation as well. We then perform mean ﬁeld inference on this ﬁnal Instance
CRF to get our ﬁnal output. Note that this ﬁnal, pairwise CRF is dynamic – the D+1 labels
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 7
that it deals with are different for every image, where Dis equal to the number of object de-
tections obtained after non-maximal suppression. This differs with the CRFs considered in
semantic segmentation literature which always have a ﬁxed number of labels for all images
in the dataset [17, 18, 35, 37]. Zheng et al. [37] showed how the iterative mean ﬁeld infer-
ence algorithm can be unrolled into a series of differentiable operations and be implemented
as a recurrent neural network. We employ a similar method so that our dynamic CRF is fully
differentiable and can thus be incorporated into a neural network. This is achieved by using
CRF weights that are not class-speciﬁc, and can thus generalise to any number of labels per
image. A label here refers to an instance associated with a detection and has no semantic
meaning. For example, in Fig. 1, label “1” was associated with a dining table, whilst in Fig.
2 it is a sheep. It therefore does not make sense to have class-speciﬁc weights in any case.
This approach to instance segmentation is bottom-up, in the sense that we ﬁrst perform
category-level segmentation of each pixel before predicting instances. However, the fact
that we used object detection cues in performing our semantic segmentation helps us to
subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be
regarded as being at the intersection of these two more common problems.
4 Experimental Results and Analysis
We ﬁrst describe the dataset and our experimental setup (§4.1), before examining the efﬁcacy
of our detection potentials (§4.2) and comparing our approach with other recent instance
segmentation methods (§4.3).
4.1 Experimental Setup
Following previous work, we evaluate our instance segmentation performance on the PAS-
CAL VOC 2012 [8] validation set which comprises of 1449 images with high-quality an-
notations. There is no test server for instance segmentation to evaluate on the test set. We
use the standard APrmeasure for evaluation [13], which computes the mean average preci-
sion under different Intersection over Union (IoU) scores between the predicted and ground
truth segmentations (rather than IoU between bounding boxes, as in object detection). In ob-
ject detection, a predicted bounding box is considered correct if the IoU between it and the
ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds
between the predicted and ground-truth segments, since higher overlap thresholds require the
segmentations to be more precise and are therefore better indicators of segmentation perfor-
mance. We also quote the APr
volmeasure, which is the mean APrscore for overlap thresholds
ranging from 0 .1 to 0 .9 in increments of 0 .1 [13]. The detailed annotations in the VOC 2012
dataset allows us to reliably evaluate the AProf our method at high overlap thresholds.
We ﬁrst trained a network for semantic segmentation, in a similar manner to [27] and
[37]. Although the trained models from these two authors have been made publicly available,
we could not use them since they had been trained on part of the VOC validation set. We
ﬁrst trained a fully convolutional network [27] (ﬁnetuned from VGG-16 trained on ImageNet
[32]) using VOC 2012 training data, augmented with images from the Semantic Boundaries
Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO
dataset [26].
To this pretrained network, we added the Higher Order CRF, and ﬁnetuned with a learn-
ing rate of 10−11using only VOC 2012 data which is ﬁnely annotated. The learning rate is
8 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Table 1: Comparison of instance segmentation performance against baselines on the VOC
2012 Validation Set of 1449 images. We report the APrmeasure at different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APr
vol
Baseline without detection potentials 54.6 48.5 41.8 50.0
Baseline with detection potentials,
butYvariables ignored57.5 51.6 44.5 52.4
Full system with detection potentials,
and recalibrated detection scores from
Yvariables58.3 52.4 45.4 53.1
low since the loss was not normalised by the number of pixels in the training image. When
training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a
mean IoU of 73.4% when using only pairwise potentials in the CRF. This rises to 75.3%
when using detection potentials as well. The authors of [2] observed a similar increase when
using detection potentials.
4.2 Effect of detection potentials
To analyse the effect of our detection potentials, we evaluate our system by disabling the
higher order potentials in our Semantic Segmentation CRF. Because our detection potentials
are disabled, we can no longer use the output of the latent Yvariables as our recalibrated
detection score, and use the original detector’s conﬁdence score instead. As shown in Table
1, the result is that the APrat 0.5 is 3.7% lower and the APr
volis 3.1% lower.
The detection potentials improve instance segmentation performance as they improve
both our initial semantic segmentation, and also recalibrate the detector’s initial scores. To
decouple these two effects, we then include detection potentials in our Segmentation CRF,
but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence
score instead as the input to the instance segmentation network. The second row of Table
1 shows that this second baseline has an APrat 0.5 and an APr
volthat are respectively 0.8%
and 0.7% lower than our ﬁnal method. The difference between the ﬁnal method and this
second baseline is due to the score recalibration performed by the detection potentials. The
two baselines differ in performance since the second baseline uses a semantic segmentation
(mean IoU of 75.3%) which is better than the other (73.4%) due to its use of detection
potentials. The large gap in their overall instance segmentation performance emphasises
the importance of a good initial semantic segmentation for our method, and underlines the
beneﬁts of a bottom-up approach to instance segmentation.
From the difference in performance between the baseline that does not fully make use of
detection potentials (Row 1 of Table 1), and our ﬁnal method (Row 3 of Table 1), we can
conclude that our higher order detection potentials are an integral part of our system.
4.3 Comparison to other current methods
In Table 2, we compare our methods to other approaches that have been evaluated on the
same dataset and reported APrresults at multiple IoU thresholds. The SDS method of [13]
was evaluated on the VOC Validation set by [4]. Our approach signiﬁcantly outperforms
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 9
Table 2: Comparison of instance segmentation performance to other methods on the VOC
2012 Validation Set. We report the APrmeasure at ﬁve different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APrat 0.8 APrat 0.9 APr
vol
SDS [13] 43.8 34.5 21.3 8.7 0.9 -
Chen et al. [4] 46.3 38.2 27.0 13.5 2.6 -
PFN [24] 58.7 51.3 42.5 31.2 15.7 52.3
Ours 58.3 52.4 45.4 34.9 20.1 53.1
both [13] and [4] which initially perform object detection, and reﬁne these detections into an
instance segmentation. In fact, even our baselines in Table 1 outperform them. The model
of [24] differs from the other two works in that the ﬁrst stage of their pipeline is to also
perform a category-level segmentation of the image. However, their method does not make
use of any information from object detectors, and our approach performs better, particularly
at high IoU thresholds where the difference in APrat an IoU threshold of 0.9 is 4.4%. This
indicates that our instance segmentations are more precise. The fact that our APr
volis only
0.8% greater, despite our APrbeing signiﬁcantly larger at high thresholds suggests that the
method of [24] has a higher APrat lower thresholds (below 0.5, which were not reported by
[24]). This indicates that their method identiﬁes more instances than ours, but the fact that
their APrat higher thresholds is lower than ours also suggests that their segmentations are
not as accurate.
Some success and failure cases of our method are shown in Figure 3. Our method has
trouble differentiating instances which are occluded and also very visually similar to each
other (Fig. 3, last two rows). The supplementary material includes a more detailed results
table, and more examples of success and failure cases.
5 Conclusion
We have presented a simple method that effectively leverages state-of-the-art semantic seg-
mentation and object detection networks to perform the less widely studied task of instance
segmentation. Our approach begins with a bottom-up semantic segmentation of the input
image, but is able to reason about instances since the ﬁnal CRF layer in our semantic seg-
mentation network incorporates information from an object detector in the form of a higher-
order, detection potential. Our method produces state-of-the-art instance segmentation re-
sults, achieving considerable improvements over existing work at high APrthresholds. Our
ﬁnal Instance CRF – which is dynamic in its number of labels – is fully differentiable. This
means that our neural network can be dynamically instantiated for every input image and
trained end-to-end.
Acknowledgements
We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful dis-
cussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC
grant ERC-2012-AdG 321162-HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and EP-
SRC/MURI grant EP/N019474/1.
10 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Input image Object Segmentation Instance Segmentation Ground truth
Figure 3: Some results of our system. Detector outputs are overlayed on the input image. Top
row: An example where our method performs well, handling occlusions and false-positive
detections. Second row: Our method performs well on this easy example which has no
occluding objects of the same class. The naïve method described in §3.3 would have pro-
duced the same result. Third row: A failure case where the system is not able to differentiate
visually similar instances. Bottom row: Heavily occluded objects are difﬁcult to identify.
References
[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.
Multiscale combinatorial grouping. In CVPR , pages 328–335. IEEE, 2014.
[2] Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip H. S. Torr. Higher order
conditional random ﬁelds in deep neural networks. In ECCV , 2016.
[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L
Yuille. Semantic image segmentation with deep convolutional nets and fully connected
crfs. ICLR , 2015.
[4] Yi-Ting Chen, Xiaokai Liu, and Ming-Hsuan Yang. Multi-instance object segmentation
with occlusion handling. In CVPR , pages 3470–3478, 2015.
[5] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object
and stuff segmentation. In CVPR , 2015.
[6] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise
convolutional networks for semantic segmentation. In ICCV , 2015.
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 11
[7] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via
multi-task network cascades. In CVPR , 2016.
[8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.
[9] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively
trained, multiscale, deformable part model. In CVPR , pages 1–8. IEEE, 2008.
[10] Spyros Gidaris and Nikos Komodakis. Object detection via a multi-region and semantic
segmentation-aware cnn model. In ICCV , pages 1134–1142, 2015.
[11] Ross Girshick. Fast r-cnn. In ICCV , 2015.
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar-
chies for accurate object detection and semantic segmentation. In CVPR , 2014.
[13] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous
detection and segmentation. In ECCV , pages 297–312. Springer, 2014.
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns
for object segmentation and ﬁne-grained localization. CVPR , 2015.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in
deep convolutional networks for visual recognition. In ECCV , 2014.
[16] Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpiñán. Multiscale conditional
random ﬁelds for image labeling. In CVPR , volume 2, pages II–695. IEEE, 2004.
[17] Pushmeet Kohli, Lubor Ladicky, and Philip H.S. Torr. Robust higher order potentials
for enforcing label consistency. IJCV , 82(3):302–324, 2009.
[18] P. Krähenbühl and V . Koltun. Efﬁcient inference in fully connected CRFs with Gaus-
sian edge potentials. In NIPS , 2011.
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classiﬁcation with
deep convolutional neural networks. In NIPS , pages 1097–1105. 2012.
[20] Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip HS Torr. Graph cut based
inference with co-occurrence statistics. In ECCV , pages 239–253. 2010.
[21] L’ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H. S. Torr.
What, where and how many? combining object detectors and crfs. In ECCV , pages
424–437, 2010.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learn-
ing applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324,
1998.
[23] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Zequn Jie, Jiashi Feng, Liang Lin, and
Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint
arXiv:1511.04517 , 2015.
12 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
[24] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, and Shuicheng
Yan. Proposal-free network for instance-level object segmentation. arXiv preprint
arXiv:1509.02636 , 2015.
[25] Guosheng Lin, Chunhua Shen, Ian Reid, et al. Efﬁcient piecewise training of deep
structured models for semantic segmentation. In CVPR , 2016.
[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra-
manan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in
context. In ECCV , pages 740–755. Springer, 2014.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for
semantic segmentation. In CVPR , 2015.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards
real-time object detection with region proposal networks. In NIPS , 2015.
[29] Bernardino Romera-Paredes and Philip HS Torr. Recurrent instance segmentation. In
ECCV , 2016.
[30] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive fore-
ground extraction using iterated graph cuts. ACM TOG , 2004.
[31] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for
image understanding: Multi-class object recognition and segmentation by jointly mod-
eling texture, layout, and context. IJCV , 2009.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-
scale image recognition. In ICLR , 2015.
[33] Min Sun, Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. Relating things and
stuff via object property interactions. PAMI , 36(7):1370–1383, 2014.
[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.
Selective search for object recognition. IJCV , 104(2):154–171, 2013.
[35] Vibhav Vineet, Jonathan Warrell, and Philip HS Torr. Filter-based mean-ﬁeld inference
for random ﬁelds with higher-order terms and product label-spaces. IJCV , 2014.
[36] Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the Scene as a Whole: Joint
Object Detection, Scene Classiﬁcation and Semantic Segmentation. In CVPR , pages
702–709, 2012.
[37] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet,
Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random ﬁelds as
recurrent neural networks. In ICCV , 2015.