We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr
Higher Order Conditional Random Fields in Deep
Neural Networks
Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip H.S. Torr
University of Oxford
{firstname.lastname }@eng.ox.ac.uk
Abstract. We address the problem of semantic segmentation using deep learning.
Most segmentation systems include a Conditional Random Field (CRF) to produce
a structured output that is consistent with the image’s visual features. Recent deep
learning approaches have incorporated CRFs into Convolutional Neural Networks
(CNNs), with some even training the CRF end-to-end with the rest of the network.
However, these approaches have not employed higher order potentials, which
have previously been shown to signiﬁcantly improve segmentation performance.
In this paper, we demonstrate that two types of higher order potential, based on
object detections and superpixels, can be included in a CRF embedded within a
deep network. We design these higher order potentials to allow inference with
the differentiable mean ﬁeld algorithm. As a result, all the parameters of our
richer CRF model can be learned end-to-end with our pixelwise CNN classiﬁer.
We achieve state-of-the-art segmentation performance on the PASCAL VOC
benchmark with these trainable higher order potentials.
Keywords: Semantic Segmentation, Conditional Random Fields, Deep Learning,
Convolutional Neural Networks
1 Introduction
Semantic segmentation involves assigning a visual object class label to every pixel in an
image, resulting in a segmentation with a semantic meaning for each segment. While a
strong pixel-level classiﬁer is critical for obtaining high accuracy in this task, it is also
important to enforce the consistency of the semantic segmentation output with visual
features of the image. For example, segmentation boundaries should usually coincide
with strong edges in the image, and regions in the image with similar appearance should
have the same label.
Recent advances in deep learning have enabled researchers to create stronger clas-
siﬁers, with automatically learned features, within a Convolutional Neural Network
(CNN) [1 –3]. This has resulted in large improvements in semantic segmentation accu-
racy on widely used benchmarks such as PASCAL VOC [4]. CNN classiﬁers are now
considered the standard choice for pixel-level classiﬁers used in semantic segmentation.
On the other hand, probabilistic graphical models have long been popular for struc-
tured prediction of labels, with constraints enforcing label consistency. Conditional
Random Fields (CRFs) have been the most common framework, and various rich and
arXiv:1511.08119v4  [cs.CV]  29 Jul 2016

Higher Order Conditional Random Fields in Deep Neural Networks 3
We evaluate our higher order potentials on the PASCAL VOC 2012 semantic seg-
mentation benchmark as well as the PASCAL Context dataset, to show signiﬁcant
improvements over our baseline and achieve state-of-the art results.
2 Related Work
Before deep learning became prominent, semantic segmentation was performed with
dense hand-crafted features which were fed into a per-pixel or region classiﬁer [14]. The
individual predictions made by these classiﬁers were often noisy as they lacked global
context, and were thus post-processed with a CRF, making use of prior knowledge such
as the fact that nearby pixels, as well as pixels of similar appearance, are likely to share
the same class label [14, 15].
The CRF model of [14] initially contained only unary and pairwise terms in an
8-neighbourhood, which [16] showed can result in shrinkage bias. Numerous improve-
ments to this model were subsequently proposed including: densely connected pairwise
potentials facilitating interactions between all pairs of image pixels [17], formulating
higher order potentials deﬁned over cliques larger than two nodes [5, 16] in order to
capture more context, modelling co-occurrence of object classes [18 –20], and utilising
the results of object detectors [6, 21, 22].
Recent advances in deep learning have allowed us to replace hand-crafted features
with features learned speciﬁcally for semantic segmentation. The strength of these
representations was illustrated by [3] who achieved signiﬁcant improvements over
previous hand-crafted methods without using any CRF post-processing. Chen et al. [12]
showed further improvements by post-processing the results of a CNN with a CRF.
Subsequent works [9 –11, 23] have taken this idea further by incorporating a CRF as
layers within a deep network and then learning parameters of both the CRF and CNN
together via backpropagation.
In terms of enhancements to conventional CRF models, Ladicky et al. [6] pro-
posed using an off-the-shelf object detector to provide additional cues for semantic
segmentation. Unlike other approaches that reﬁne a bounding-box detection to produce
a segmentation [8, 24], this method used detector outputs as a soft constraint and can
thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.
Object detectors have also been used by [21, 25], who also modelled variables that
describe the degree to which an object hypothesis is accepted.
We formulate the detection potential in a different manner to [6, 21, 25] so that it is
amenable to mean ﬁeld inference. Mean ﬁeld permits inference with dense pairwise con-
nections, which results in substantial accuracy improvements [10,12, 17]. Furthermore,
mean ﬁeld updates related to our potentials are differentiable and its parameters can thus
be learned in our end-to-end trainable architecture.
We also note that while the semantic segmentation problem has mostly been formu-
lated in terms of pixels [3,10,14], some have expressed it in terms of superpixels [26 –28].
Superpixels can capture more context than a single pixel and computational costs can
also be reduced if one considers pairwise interactions between superpixels rather than in-
dividual pixels [21]. However, such superpixel representations assume that the segments
4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several
authors [5, 7] have employed higher order potentials deﬁned over superpixels that en-
courage label consistency over regions, but do not strictly enforce it. This approach also
allows multiple, non-hierarchical layers of superpixels to be integrated. Our formulation
uses this kind of higher order potential, but in an end-to-end trainable CNN.
Graphical models have been used with CNNs in other areas besides semantic segmen-
tation, such as in pose-estimation [29] and group activity recognition [30]. Alternatively,
Ionescu et al. [31] incorporated structure into a deep network with structured matrix
layers and matrix backpropagation. However, the nature of models used in these works
is substantially different to ours. Some early works that advocated gradient backpropa-
gation through graphical model inference for parameter optimisation include [32, 33]
and [34].
Our work differentiates from the above works since, to our knowledge, we are the ﬁrst
to propose and conduct a thorough experimental investigation of higher order potentials
that are based on detection outputs and superpixel segmentation in a CRF which is
learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld
inference with higher order potentials, they did not consider object detection potentials
at all, nor were the parameters learned.
3 Conditional Random Fields
We now review conditional random ﬁelds used in semantic segmentation and introduce
the notation used in the paper. Take an image IwithNpixels, indexed 1,2,...,N . In
semantic segmentation, we attempt to assign every pixel a label from a predeﬁned set of
labels L={l1,l2,...,l L}. Deﬁne a set of random variables X1,X2,...,X N, one for
each pixel, where each Xi∈ L. LetX= [X1X2... X N]T. Any particular assignment
xtoXis thus a solution to the semantic segmentation problem.
We use notations {V}, andV(i)to represent the set of elements of a vector V, and
theithelement of V, respectively. Given a graph Gwhere the vertices are from {X}
and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a
CRF characterised by Pr(X=x|I) = (1/Z(I)) exp( −E(x|I)), whereE(x|I)is the
energy of the assignment xandZ(I)is the normalisation factor known as the partition
function. We drop the conditioning on Ihereafter to keep the notation uncluttered. The
energyE(x)of an assignment is deﬁned using the set of cliques Cin the graph G. More
speciﬁcally, E(x) =/summationtext
c∈Cψc(xc),where xcis a vector formed by selecting elements
ofxthat correspond to random variables belonging to the clique c, andψc(.)is the
cost function for the clique c. The function, ψc(.), usually uses prior knowledge about a
good segmentation, as well as information from the image, the observation the CRF is
conditioned on.
Minimising the energy yields the maximum a posteriori (MAP) labelling of the
image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference
is impracticable, and one has to resort to an approximate inference method such as mean
ﬁeld inference [17]. Mean ﬁeld inference is particularly appealing in a deep learning
setting since it is possible to formulate it as a Recurrent Neural Network [10].
Higher Order Conditional Random Fields in Deep Neural Networks 5
4 CRF with Higher Order Potentials
Many CRF models that have been incorporated into deep learning frameworks [10, 12]
have so far used only unary and pairwise potentials. However, potentials deﬁned on
higher order cliques have been shown to be useful in previous works such as [7, 16]. The
key contribution of this paper is to show that a number of explicit higher order potentials
can be added to CRFs to improve image segmentation, while staying compatible with
deep learning. We formulate these higher order potentials in a manner that mean ﬁeld
inference can still be used to solve the CRF. Advantages of mean ﬁeld inference are
twofold: First, it enables efﬁcient inference when using densely-connected pairwise
potentials. Multiple works, [10,33] have shown that dense pairwise connections result in
substantial accuracy improvements, particularly at image boundaries [12, 17]. Secondly,
we keep all our mean ﬁeld updates differentiable with respect to their inputs as well
as the CRF parameters introduced. This design enables us to use backpropagation to
automatically learn all the parameters in the introduced potentials.
We use two types of higher order potential, one based on object detections and the
other based on superpixels. These are detailed in Sections 4.1 and 4.2 respectively. Our
complete CRF model is represented by
E(x) =/summationdisplay
iψU
i(xi) +/summationdisplay
i < jψP
ij(xi,xj) +/summationdisplay
dψDet
d(xd) +/summationdisplay
sψSP
s(xs),(1)
where the ﬁrst two terms ψU
i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order
energies. Energies from the object detection take the form ψDet
d(xd), where vector xd
is formed by elements of xthat correspond to the foreground pixels of the dthobject
detection. Superpixel label consistency based energies take the form ψSP
s(xs), where xs
is formed by elements of xthat correspond to the pixels belonging to the sthsuperpixel.
4.1 Object Detection Based Potentials
Semantic segmentation errors can be classiﬁed into two broad categories [35]: recognition
and boundary errors. Boundary errors occur when semantic labels are incorrect at the
edges of objects, and it has been shown that densely connected CRFs with appearance-
consistency terms are effective at combating this problem [17]. On the other hand,
recognition errors occur when object categories are recognised incorrectly or not at
all. A CRF with only unary and pairwise potentials cannot effectively correct these
errors since they are caused by poor unary classiﬁcation. However, we propose that a
state-of-the-art object detector [36,37] capable of recognising and localising objects, can
provide important information in this situation and help reduce the recognition error, as
shown in Fig. 2.
A key challenge in feeding-in object-detection potentials to semantic segmentation
are false detections. A na ¨ıve approach of adding an object detector’s output to a CRF
formulated to solve the problem of semantic segmentation would confuse the CRF
due to the presence of the false positives in the detector’s output. Therefore, a robust
formulation, which can automatically reject object detection false positives when they

Higher Order Conditional Random Fields in Deep Neural Networks 7
positive: some of the detections obtained from the object detector may be identiﬁed to
be false following CRF inference.
AllYdvariables are added to the CRF which previously contained only Xivariables.
Let each (Xd,Yd), where {Xd}={Xi∈ {X}|i∈Fd}, form a clique cdin the
CRF. We deﬁne the detection-based higher order energy associated with a particular
assignment (xd,yd)to the clique (Xd,Yd)as follows:
ψDet
d(Xd=xd,Yd=yd) =

wDetsd
nd/summationtextnd
i=1[x(i)
d=ld]ifyd= 0,
wDetsd
nd/summationtextnd
i=1[x(i)
d/ne}ationslash=ld]ifyd= 1,(2)
wherend=|Fd|is the number of foreground pixels in the dthdetection,x(i)
dis theith
element of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson
bracket. Note that this potential encourages X(i)
ds to take the value ldwhenYdis1, and
at the same time encourages Ydto be 0when manyX(i)
ds do not take ld. In other words,
it enforces the consistency among X(i)
ds andYd.
An important property of the above deﬁnition of ψDet
d(.)is that it can be simpliﬁed
as a sum of pairwise potentials between Ydand eachX(i)
dfori= 1,2,...,n d. That is,
ψDet
d(Xd=xd,Yd=yd) =nd/summationdisplay
i=1fd(x(i)
d,yd), where,
fd(x(i)
d,yd) =

wDetsd
nd[x(i)
d=ld] ifyd= 0,
wDetsd
nd[x(i)
d/ne}ationslash=ld] ifyd= 1.(3)
We make use of this simpliﬁcation in Section 5 when deriving the mean ﬁeld updates
associated with this potential.
For the latent Yvariables, in addition to the joint potentials with Xvariables,
described in Eq. (2)and(3), we also include unary potentials, which are initialised
from the score sdof the object detection. The underlying idea is that if the object
detector detects an object with high conﬁdence, the CRF in turn starts with a high initial
conﬁdence about the validity of that detection. This conﬁdence can, of course, change
during the CRF inference depending on other information ( e.g.segmentation unary
potentials) available to the CRF.
Examples of input images with multiple detections and GrabCut foreground masks
are shown in Figure 3. Note how false detections are ignored and erroneous parts of the
foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels
obtained from oversegmentation [39, 40] quite often contain pixels from the same visual
object. It is therefore natural to encourage pixels inside a superpixel to have the same
semantic label. Once again, this should not be a hard constraint in order to keep the
8 Arnab et al.
(a) (b) (c) (d)
Fig. 3: Effects of imperfect foreground segmentation (a,b) Detected objects, as well
as the foreground masks obtained from GrabCut. (c,d) Output using detection potentials.
Incorrect parts of the foreground segmentation of the main aeroplane, and entire TV
detection have been ignored by CRF inference as they did not agree with the other energy
terms. The person is a failure case though as the detection has caused part of the sofa to
be erroneously labelled.
(a) (b) (c)
Fig. 4: Segmentation enhancement from superpixel based potentials (a) The output
of our system without any superpixel potentials. (b) Superpixels obtained from the image
using the method of [39]. Only one “layer” of superpixels is shown. In practice, we
used four. (c) The output using superpixel potentials. The result has improved as we
encourage consistency over superpixel regions. This removes some of the spurious noise
that was present previously.
algorithm robust to initial superpixel segmentation errors and to violations of this key
assumption.
We use two types of energies in the CRF to encourage superpixel consistency in
semantic segmentation. Firstly, we use the Pn-Potts model type energy [41], which is
described by,
ψSP
s(Xs=xs) =

wLow(l) if allx(i)
s=l,
wHigh otherwise,(4)
wherewLow(l)<w High for alll, and{Xs} ⊂ {X}is a clique deﬁned by a superpixel.
The primary idea is that assigning different labels to pixels in the same superpixel incurs
a higher cost, whereas one obtains a lower cost if the labelling is consistent throughout
the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.
Secondly, to make this potential stronger, we average initial unary potentials from the
classiﬁer (the CNN in our case), across all pixels in the superpixel and use the average
Higher Order Conditional Random Fields in Deep Neural Networks 9
as an additional unary potential for those pixels. During experiments, we observed that
superpixel based higher order energy helps in getting rid of small spurious regions of
wrong labels in the segmentation output, as shown in Fig. 4.
5 Mean Field Updates and Their Differentials
This section discusses the mean ﬁeld updates for the higher order potentials previously
introduced. These update operations are differentiable with respect to the Qi(Xi)distri-
bution inputs at each iteration, as well as the parameters of our higher order potentials.
This allows us to train our CRF end-to-end as another layer of a neural network.
Take a CRF with random variables V1,V2,...,V Nand a set of cliques C, which
includes unary, pairwise and higher order cliques. Mean ﬁeld inference approximates
the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We
useQ(Vc=vc)to denote the marginal probability mass for a subset {Vc}of these
variables. Where there is no ambiguity, we use the short-hand notation Q(vc)to represent
Q(Vc=vc). General mean ﬁeld updates of such a CRF take the form [13]
Qt+1(Vi=v) =1
Ziexp
−/summationdisplay
c∈C/summationdisplay
{vc|vi=v}Qt(vc−i)ψc(vc)
, (5)
whereQtis the marginal after the tthiteration, vcan assignment to all variables in
cliquec,vc−ian assignment to all variables in cexcept forVi,ψc(vc)is the cost of
assigning vcto the clique c, andZiis the normalisation constant that makes Q(Vi=v)
a probability mass function after the update.
Updates from Detection Based Potentials Following Eq. (3)above, we now use
Eq.(5)to derive the mean ﬁeld updates related to ψDet
d. The contribution from ψDet
dto
the update of Q(X(i)
d=l)takes the form
/summationdisplay
{(xd,yd)|x(i)
d=l}Q(xd−i,yd)ψDet
d(xd,yd) =

wDetsd
ndQ(Yd= 0) ifl=ld,
wDetsd
ndQ(Yd= 1) otherwise,(6)
where xd−iis an assignment to Xdwith theithelement deleted. Using the same equa-
tions, we derive the contribution from the energy ψDet
dto the update of Q(Yd=b)to
take the form
/summationdisplay
{(xd,yd)|yd=b}Q(xd)ψDet
d(xd,yd) =

wDetsd
nd/summationtextnd
i=1Q(X(i)
d=ld) ifb= 0,
wDetsd
nd/summationtextnd
i=1(1−Q(X(i)
d=ld))otherwise.
(7)
It is possible to increase the number of parameters in ψDet
d(.). Since we use backprop-
agation to learn these parameters automatically during end-to-end training, it is desirable
to have a high number of parameters to increase the ﬂexibility of the model. Following
10 Arnab et al.
this idea, we made the weight wDetclass speciﬁc, that is, a function wDet(ld)is used
instead ofwDetin Eqs. (2),(6)and (7). The underlying assumption is that detector
outputs can be very helpful for certain classes, while being not so useful for classes that
the detector performs poorly on, or classes for which foreground segmentation is often
inaccurate.
Note that due to the presence of detection potentials in the CRF, error differentials
calculated with respect to the Xvariable unary potentials and pairwise parameters will
no longer be valid in the forms described in [10]. The error differentials with respect to
theXandYvariables, as well as class-speciﬁc detection potential weights wDet(l)are
included in the supplementary material.
Updates for Superpixel Based Potentials The contribution from the Pn-Potts type
potential to the mean ﬁeld update of Q(xi=l), where pixel iis in the superpixel clique
s, was derived in [7] as
/summationdisplay
{xs|x(i)
s=l}Q(xs−i)ψSP
s(xs) =wLow(l)/productdisplay
j∈c,j/ne}ationslash=iQ(Xj=l)+wHigh
1−/productdisplay
j∈c−iQ(Xj=l)
.
(8)
This update operation is differentiable with respect to the parameters wLow(l)andwHigh,
allowing us to optimise them via backpropagation, and also with respect to the Q(X)
values enabling us to optimise previous layers in the network.
Convergence of parallel mean ﬁeld updates Mean ﬁeld with parallel updates, as
proposed in [17] for speed, does not have any convergence guarantees in the general case.
However, we usually empirically observed convergence with higher order potentials,
without damping the mean ﬁeld update as described in [7, 42]. This may be explained by
the fact that the unaries from the initial pixelwise-prediction part of our network provide
a good initialisation. In cases where the mean ﬁeld energy did not converge, we still
empirically observed good ﬁnal segmentations.
6 Experiments
We evaluate our new CRF formulation on two different datasets using the CRF-RNN
network [10] as the main baseline, since we are essentially enriching the CRF model
of [10]. We then present ablation studies on our models.
6.1 Experimental set-up and results
Our deep network consists of two conceptually different, but jointly trained stages. The
ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised
from the Imagenet-trained VGG-16 network [2], and then ﬁne-tuned with data from
the VOC 2012 training set [4], extra VOC annotations of [43] and the MS COCO [44]
dataset.
Higher Order Conditional Random Fields in Deep Neural Networks 11
Table 1: Comparison of each higher order
potential with baseline on VOC 2012 re-
duced validation set
MethodReduced
val set(%)
Baseline (unary + pairwise) [10] 72.9
Superpixels only 74.0
Detections only 74.9
Detections and Superpixels 75.8Table 2: Mean IoU accuracy on VOC 2012
test set. All methods are trained with MS
COCO [44] data
Method Test set(%)
Ours 77.9
DPN [9] 77.5
Centrale Super Boundaries [45] 75.7
Dilated Convolutions [46] 75.3
BoxSup [35] 75.2
DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9
DeepLab [12] 72.7
Table 3: Mean Intersection over Union (IoU) results on PASCAL Context validation set
compared to other current methods.
Method Ours BoxSup [35] ParseNet [49] CRF-RNN [10] FCN-8s [3] CFM [28]
Mean IoU (%) 41.3 40.5 40.4 39.3 37.8 34.4
The output of the ﬁrst stage is fed into our CRF inference network. This is im-
plemented using the mean ﬁeld update operations and their differentials described in
Section 5. Five iterations of mean ﬁeld inference were performed during training. Our
CRF network has two additional inputs in addition to segmentation unaries obtained from
the FCN-8s network: data from the object detector and superpixel oversegmentations of
the image.
We used the publicly available code and model of the Faster R-CNN [37] object de-
tector. The fully automated version of GrabCut [38] was then used to obtain foregrounds
from the detection bounding boxes. These choices were made after conducting prelimi-
nary experiments with alternate detection and foreground segmentation algorithms.
Four levels of superpixel oversegmentations were used, with increasing superpixel
size to deﬁne the cliques used in this potential. Four levels were used since performance
on the VOC validation set stopped increasing after this number. We used the superpixel
method of [39] as it was shown to adhere to object boundaries the best [40], but our
method generalises to any oversegmentation algorithm.
We trained the full network end-to-end, optimising the weights of the CNN classiﬁer
(FCN-8s) and CRF parameters jointly. We initialised our network using the publicly
available weights of [10], and trained with a learning rate of 10−10and momentum of
0.99. The learning rate is low because the loss was not normalised by the number of
pixels in the training image. This is to have a larger loss for images with more pixels.
When training our CRF, we only used VOC 2012 data [4] as it has the most accurate
labelling, particularly around boundaries.
12 Arnab et al.
PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential
was evaluated on the same reduced validation set [3] used by our baseline [10]. As Table
1 shows, each new higher order potential improves the mean IoU over the baseline. We
only report test set results for our best method since the VOC guidelines discourage
the use of the test set for ablation studies. On the test set (Table 2), we outperform our
baseline by 3.2% which equates to a 12.6% reduction in the error rate. This sets a new
state-of-the-art on the VOC dataset. Qualitative results highlighting success and failure
cases of our algorithm, as well as more detailed results, are shown in our supplementary
material.
PASCAL Context Table 3 shows our state-of-the-art results on the recently released
PASCAL Context dataset [50]. We trained on the provided training set of 4998 images,
and evaluated on the validation set of 5105 images. This dataset augments VOC with
annotations for all objects in the scene. As a result, there are 59 classes as opposed to
the 20 in the VOC dataset. Many of these new labels are “stuff” classes such as “grass”
and “sky”. Our object detectors are therefore only trained for 20 of the 59 labels in this
dataset. Nevertheless, we improve by 0.8% over the previous state-of-the-art [35] and
2% over our baseline [10].
6.2 Ablation Studies
We perform additional experiments to determine the errors made by our system, show
the beneﬁts of end-to-end training and compare our detection potentials to a simpler
baseline. Unless otherwise stated, these experiments are performed on the VOC 2012
reduced validation set.
Error Analysis To analyse the improvements made by our higher order potentials, we
separately evaluate the performance on the “boundary” and “interior” regions in a similar
manner to [35]. As shown in Fig. 5 c) and d), we consider a narrow band (trimap [16])
around the “void” labels annotated in the VOC 2012 reduced validation set. The mean
IoU of pixels lying within this band is termed the “Boundary IoU” whilst the “Interior
IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials
improves the Interior IoU over our baseline (only pairwise potentials [10]) as the object
detector may recognise objects in the image which the pixelwise classiﬁcation stage of
our network may have missed out. However, the detection potentials also improve the
Boundary IoU for all tested trimap widths as well. Improving the recognition of pixels
in the interior of an object also helps with delineating the boundaries since the strength
of the pairwise potentials exerted by the Qdistributions at each of the correctly-detected
pixels increase.
Our superpixel priors also increase the Interior IoU with respect to the baseline.
Encouraging consistency over regions helps to get rid of spurious regions of wrong
labels (as shown in Fig. 4). Fig. 5 suggests that most of this improvement occurs in the
interior of an object. The Boundary IoU is slightly lower than the baseline, and this may
Higher Order Conditional Random Fields in Deep Neural Networks 13
a) Image (b) Ground truth (c) Boundary (d) Interior
0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU
  
Pairwise only
Superpixels
Detections
Detections and Superpixels
0 5 10 15 20 25 30 35 40505560657075
Trimap WidthBoundary IoU
  
Pairwise only
Superpixels
Detections
Detections and Superpixels
(e) Interior IoU (f) Boundary IoU
Fig. 5: Error analysis on VOC 2012 reduced validation set The IoU is computed for
boundary and interior regions for various trimap widths. An example of the Boundary
and Interior regions for a sample image using a width of 9 pixels is shown in white in
the top row. Black regions are ignored in the IoU calculation.
be due to the fact that superpixels do not always align correctly with the edges of an
object (the “boundary recall” of various superpixel methods are evaluated in [40]).
We can see that the combination of detection and superpixel potentials results in a
substantial improvement in our Interior IoU. This is the primary reason our overall IoU
on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms
piecewise training. We trained the CRF piecewise by freezing the weights of the unary
part of the network, and only learning the CRF parameters.
Our results in Table 2 used the FCN-8s [3] architecture to generate unaries. To show
that our higher order potentials improve performance regardless of the underlying CNN
used for producing unaries, we also perform an experiment using our reimplementation
Table 4: Comparison of mean IoU (%) obtained on VOC 2012 reduced validation set
from end-to-end and piecewise training
Method FCN-8s DCN
Unary only, ﬁne-tuned on COCO 68.3 68.6
Pairwise CRF trained piecewise 69.5 70.7
Pairwise CRF trained end-to-end 72.9 72.5
Higher Order CRF trained piecewise 73.6 73.5
Higher Order CRF trained end-to-end 75.8 75.0
Test set performance of best model 77.9 76.9
14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]
instead of FCN-8s.
Table 4 shows that end-to-end training of the CRF yields considerable improvements
over piecewise training. This was the case when using either FCN-8s or DCN for
obtaining the initial unaries before performing CRF inference with higher order potentials.
This suggests that our CRF network module can be plugged into different architectures
and achieve performance improvements.
Baseline for detections To evaluate the efﬁcacy of our detection potentials, we formu-
late a simpler baseline since no other methods use detection information at inference
time (BoxSup [35] derives ground truth for training using ground-truth bounding boxes).
Our baseline is similar to CRF-RNN [10], but prior to CRF inference, we take the
segmentation mask from the object detection and add a unary potential proportional to
the detector’s conﬁdence to the unary potentials for those pixels. We then perform mean-
ﬁeld inference (with only pairwise terms) on these “augmented” unaries. Using this
method, the mean IoU increases from 72.9% to 73.6%, which is signiﬁcantly less than
the 74.9% which we obtained using only our detection potentials without superpixels
(Table 1).
Our detection potentials perform better since our latent Ydetection variables model
whether the detection hypothesis is accepted or not. Our CRF inference is able to evaluate
object detection inputs in light of other potentials. Inference increases the relative score
of detections which agree with the segmentation, and decreases the score of detections
which do not agree with other energies in the CRF. Figures 2 b) and d) show examples
of false-positive detections that have been ignored and correct detections that have been
used to reﬁne our segmentation. Our baseline, on the other hand, is far more sensitive to
erroneous detections as it cannot adjust the weight given to them during inference.
7 Conclusion
We presented a CRF model with two different higher order potentials to tackle the
semantic segmentation problem. The ﬁrst potential is based on the intuitive idea that
object detection can provide useful cues for semantic segmentation. Our formulation is
capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to
have consistent labelling. These two new potentials can co-exist with the usual unary
and pairwise potentials in a CRF.
Importantly, we showed that efﬁcient mean ﬁeld inference is still possible in the
presence of the new higher order potentials and derived the explicit forms of the mean
ﬁeld updates and their differentials. This enabled us to implement the new CRF model
as a stack of CNN layers and to train it end-to-end in a uniﬁed deep network with a
pixelwise CNN classiﬁer. We experimentally showed that the addition of higher order
potentials results in a signiﬁcant increase in semantic segmentation accuracy allowing us
to reach state-of-the-art performance.
This work was supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant
Seebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Clarendon Fund.
Higher Order Conditional Random Fields in Deep Neural Networks 15
References
1.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep convolutional
neural networks. In: NIPS. (2012) 1097–1105
2.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image
recognition. In: ICLR. (2015)
3.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation.
In: CVPR. (2015)
4.Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual
object classes (voc) challenge. IJCV (2010)
5.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical crfs for object class
image segmentation. In: ICCV . (2009) 739–746
6.Ladick ´y, L., Sturgess, P., Alahari, K., Russell, C., Torr, P.H.S.: What, where and how many?
combining object detectors and crfs. In: ECCV . (2010) 424–437
7.Vineet, V., Warrell, J., Torr, P.H.: Filter-based mean-ﬁeld inference for random ﬁelds with
higher-order terms and product label-spaces. IJCV (2014)
8.Hariharan, B., Arbel ´aez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation.
In: ECCV . Springer (2014) 297–312
9.Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Semantic image segmentation via deep parsing
network. In: ICCV . (2015)
10.Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr,
P.: Conditional random ﬁelds as recurrent neural networks. In: ICCV . (2015)
11.Lin, G., Shen, C., Reid, I.: Efﬁcient piecewise training of deep structured models for semantic
segmentation. In: CVPR. (2016)
12.Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmen-
tation with deep convolutional nets and fully connected crfs. ICLR (2015)
13.Koller, D., Friedman, N.: Probabilistic graphical models: principles and techniques. MIT
press (2009)
14.Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: Multi-
class object recognition and segmentation by jointly modeling texture, layout, and context.
IJCV (2009)
15.He, X., Zemel, R.S., Carreira-Perpi ˜n´an, M. ´A.: Multiscale conditional random ﬁelds for image
labeling. In: CVPR, IEEE (2004)
16.Kohli, P., Ladicky, L., Torr, P.: Robust higher order potentials for enforcing label consistency.
IJCV 82(3) (2009) 302–324
17.Kr¨ahenb ¨uhl, P., Koltun, V.: Efﬁcient inference in fully connected CRFs with Gaussian edge
potentials. In: NIPS. (2011)
18.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Graph cut based inference with co-occurrence
statistics. In: ECCV . (2010) 239–253
19.Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context.
In: ICCV . (2007) 1–8
20.Gonfaus, J.M., Boix, X., Van de Weijer, J., Bagdanov, A.D., Serrat, J., Gonzalez, J.: Harmony
potentials for joint classiﬁcation and segmentation. In: CVPR, IEEE (2010) 3280–3287
21.Yao, J., Fidler, S., Urtasun, R.: Describing the Scene as a Whole: Joint Object Detection,
Scene Classiﬁcation and Semantic Segmentation. In: CVPR. (2012) 702–709
22.Wojek, C., Schiele, B.: A dynamic conditional random ﬁeld model for joint labeling of object
and scene classes. In: ECCV , Springer (2008) 733–747
23.Lin, G., Shen, C., Reid, I., van den Hengel, A.: Deeply learning the messages in message
passing inference. In: NIPS. (2015) 361–369
16 Arnab et al.
24.Yang, Y., Hallman, S., Ramanan, D., Fowlkes, C.C.: Layered object models for image
segmentation. PAMI (2012)
25.Sun, M., Kim, B.s., Kohli, P., Savarese, S.: Relating things and stuff via objectproperty
interactions. PAMI 36(7) (2014) 1370–1383
26.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Semantic segmentation with second-
order pooling. In: ECCV . (2012) 430–443
27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene
labeling. PAMI (2013)
28.Dai, J., He, K., Sun, J.: Convolutional feature masking for joint object and stuff segmentation.
CVPR (2015)
29.Tompson, J.J., Jain, A., LeCun, Y ., Bregler, C.: Joint training of a convolutional network and
a graphical model for human pose estimation. In: NIPS. (2014) 1799–1807
30.Deng, Z., Zhai, M., Chen, L., Liu, Y., Muralidharan, S., Roshtkhari, M.J., Mori, G.: Deep
structured models for group activity recognition. In: BMVC. (2015)
31.Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with
structured layers. In: ICCV . (2015) 2965–2973
32.Domke, J.: Learning graphical model parameters with approximate marginal inference. PAMI
(2013)
33.Kr¨ahenb ¨uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random
ﬁelds. In: ICML. (2013)
34.Ross, S., Munoz, D., Hebert, M., Bagnell, J.A.: Learning message-passing inference machines
for structured prediction. In: CVPR. (2011)
35.Dai, J., He, K., Sun, J.: Boxsup: Exploiting bounding boxes to supervise convolutional
networks for semantic segmentation. In: ICCV . (2015)
36. Girshick, R.: Fast r-cnn. In: ICCV . (2015)
37.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with
region proposal networks. In: NIPS. (2015)
38.Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using
iterated graph cuts. ACM TOG (2004)
39.Felzenszwalb, P.F., Huttenlocher, D.P.: Efﬁcient graph-based image segmentation. IJCV
(2004)
40.Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared
to state-of-the-art superpixel methods. PAMI 34(11) (2012) 2274–2282
41. Kohli, P., Kumar, M.P., Torr, P.H.: P3 & beyond: Solving energies with higher order cliques.
In: CVPR. (2007)
42.Baqu, P., Bagautdinov, T., Fleuret, F., Fua, P.: Principled Parallel Mean-Field Inference for
Discrete Random Fields. In: CVPR. (2016)
43.Hariharan, B., Arbel ´aez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse
detectors. In: ICCV , IEEE (2011) 991–998
44.Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ´ar, P., Zitnick, C.L.:
Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740–755
45.Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. In: ICLR.
(2016)
46.Yu, F., Koltun, V .: Multi-scale context aggregation by dilated convolutions. In: ICLR. (2016)
47.Chen, L.C., Yang, Y ., Wang, J., Xu, W., Yuille, A.L.: Attention to scale: Scale-aware semantic
image segmentation. In: CVPR. (2016)
48.Papandreou, G., Chen, L., Murphy, K., Yuille, A.L.: Weakly- and semi-supervised learning of
a DCNN for semantic image segmentation. In: ICCV . (2015)
49.Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: Looking wider to see better. arXiv preprint
arXiv:1506.04579 (2015)
Higher Order Conditional Random Fields in Deep Neural Networks 17
50.Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., et al.: The
role of context for object detection and semantic segmentation in the wild. In: CVPR, IEEE
(2014) 891–898
51.Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In:
ICCV . (2015)
52.Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation
with zoom-out features. In: CVPR. (2015)
53.Dong, J., Chen, Q., Yan, S., Yuille, A.: Towards uniﬁed object detection and semantic
segmentation. In: ECCV . (2014) 299–314
54.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Free-form region description with
second-order pooling. PAMI (2014)
Appendix
Appendix A of this supplementary material presents the derivatives of the mean ﬁeld
updates which we use for inference in our Conditional Random Field (CRF). Appendix
B shows detailed qualitative results for the experiments described in our main paper.
A Derivatives of Mean Field Updates
The pseudocode for the mean ﬁeld inference algorithm with latent Ydetection variables
is shown below in Algorithm 1. We use the same notation used in the main paper.
Algorithm 1 Mean Field Inference
Q0(Xi=l)←1
Ziexp/parenleftbig
−ψU
i(l)/parenrightbig
,∀i,l
Q0(Yd=b)←sb
d(1−sd)(1−b),∀d,b
⊿Initialisation
fort = 0 :T−1do
Et(Xi=l)←UnaryUpdate + PairwiseUpdate +
DetectionUpdate + SuperpixelUpdate, ∀i,l
Et(Yd=b)←YUnaryUpdate + Y DetectionUpdate
⊿Mean ﬁeld updates
Qt+1(Xi=l)←1
Ziexp/parenleftbig
−Et(Xi=l)/parenrightbig
,∀i,l
Qt+1(Yd=b)←1
Zdexp/parenleftbig
−Et(Yd=b)/parenrightbig
,∀d,b
⊿Normalising
end for
For the explicit forms of the UnaryUpdate andPairwiseUpdate above, and their
differentials, we refer the reader to [10] and discuss the terms DetectionUpdate and
SuperpixelUpdate in detail below.
Let us assume that only one object detection of the form (ld,sd,Fd)is available for
the image under consideration. When multiple detections are present, simply a summa-
tion of the updates and differentials discussed below apply. Therefore, no generality
18 Arnab et al.
is lost with this assumption. Similarly, we can assume that only one superpixel clique
{Xs}is present, without a loss of generality.
Assuming that pixel iin Algorithm 1 belongs to Fd, Eq. (6) in the main paper
described the exact form of DetectionUpdate . Similarly, assuming that pixel ibelongs
to{Xs}Eq. (8) described the form of SuperpixelUpdate .
LetLdenote the value of the loss function calculated at the output of the deep
network. This could be the softmax loss or any other appropriate loss function. During
backpropagation, we get the error signal∂L
∂QTat the output of the mean ﬁeld inference.
Using this error information, we need to compute the derivative of the loss Lwith respect
to theXunaries and various CRF parameters. Note that, if we compute the relevent
differentials for only one iteration of the mean ﬁeld algorithm, it is possible to calculate
them for multiple iterations using the recurrent behaviour of the iterations.
Note that, by looking at Normalising step of Algorithm 1, it is trivial to calculate
∂Qt+1
∂Et. Therefore, we can then calculate∂L
∂Etusing the chain rule. This is same as
backpropagation of the usual softmax operation in a deep network (up to a negative sign).
Using this observation we can calculate the necessary differentials to take the forms
shown below:
∂L
∂wDet=sd
ndnd/summationdisplay
i=1/parenleftBig∂L
Et(X(i)
d=ld)Qt(Yd= 1) + (9)
/summationdisplay
l/prime/negationslash=ld∂L
∂Et(X(i)
d=l/prime)Qt(Yd= 1)/parenrightBig
+
∂L
∂Et(Yd= 0)sd
ndnd/summationdisplay
i=1Qt(X(i)
d=ld) +
∂L
∂Et(Yd= 1)sd
ndnd/summationdisplay
i=1/parenleftBig
1−Qt(X(i)
d=ld)/parenrightBig
∂L
∂Qt(X(i)
d=ld)=wDet∂L
∂Et(Yd= 0)−wDet∂L
∂Et(Yd= 1)(10)
∂L
∂Qt(Yd= 0)=wDetsd
ndnd/summationdisplay
i=1/parenleftBigg
∂L
Et(X(i)
d=ld)/parenrightBigg
(11)
∂L
∂Qt(Yd= 1)=wDetsd
ndnd/summationdisplay
i=1/summationdisplay
l/negationslash=ld/parenleftBigg
∂L
∂Et(X(i)
d=l/prime)/parenrightBigg
(12)
∂L
∂wLow(l)=/summationdisplay
i∈s
∂L
∂Et(X(i)
s=l)/productdisplay
j∈c,j/negationslash=iQt(Xj=l)
 (13)
Higher Order Conditional Random Fields in Deep Neural Networks 19
∂L
∂wHigh=/summationdisplay
i∈s/summationdisplay
l∈L
∂L
∂Et(X(i)
s=l)
1−/productdisplay
j∈c,j/negationslash=iQt(Xj=l)

 (14)
Effect of the superpixel potentials on the derivatives∂L
∂Qt(Xi=l)were negligible.
Therefore, we ignored them in our calculations.
B Additional Experimental Results
Table 5 presents more detailed results of our method, and that of other state-of-the-art
techniques, on the PASCAL VOC 2012 test set. In particular, we present the accuracy
for every class in the VOC test set. Note that our per-class accuracy improves over our
baseline, CRF-RNN [10], for all of the 20 classes in PASCAL VOC.
Figure 6 shows more sample results of our system, compared to our baseline, CRF-
as-RNN [10]. Figure 7 shows examples of failure cases of our method. Figure 8 examines
the effect of each of our potentials. Finally, Figure 9 shows a qualitative comparison
between the output of our system and other current methods on the PASCAL VOC 2012
test set.
20 Arnab et al.Table 5: Comparison of the mean Intersection over Union (IoU) accuracy of our approach and other state-of-the-art methods on the Pascal
VOC 2012 test set. Scores for other methods were taken from the original authors’ publications.
Methods trained with
COCOMean
IoU(%)aero-
planebike bird boat bottle bus car cat chair cow table dog horse mbikeper-
sonplant sheep sofa train tv
Our method 77.9 92.5 59.1 90.3 70.6 74.4 92.4 84.1 88.3 36.8 85.6 67.1 85.1 86.9 88.2 82.6 62.6 85.0 56.2 81.9 72.5
DPN [9] 77.5 89.0 61.6 87.7 66.8 74.7 91.2 84.3 87.6 36.5 86.3 66.1 84.4 87.8 85.6 85.4 63.6 87.3 61.3 79.4 66.4
Super Bound. [45] 75.7 90.3 37.9 89.6 67.8 74.6 89.3 84.1 89.1 35.8 83.6 66.2 82.9 81.7 85.6 84.6 60.3 84.8 60.7 78.3 68.3
Dilated Conv. [46] 75.3 91.7 39.6 87.8 63.1 71.8 89.7 82.9 89.8 37.2 84.0 63.0 83.3 89.0 83.8 85.1 56.8 87.6 56.0 80.2 64.7
BoxSup [35] 75.2 89.8 38.0 89.2 68.9 68.0 89.6 83.0 87.7 34.4 83.6 67.1 81.5 83.7 85.2 83.5 58.6 84.9 55.8 81.2 70.7
Attention [47] 75.1 92.0 41.2 87.8 57.2 72.7 92.8 85.9 90.5 30.5 78.0 62.8 85.8 85.3 87.2 85.6 57.7 85.1 56.5 83.0 65.0
CRF-as-RNN [10] 74.7 90.4 55.3 88.7 68.4 69.8 88.3 82.4 85.1 32.6 78.5 64.4 79.6 81.9 86.4 81.8 58.6 82.4 53.5 77.4 70.1
WSSL [48] 73.9 89.2 46.7 88.5 63.5 68.4 87.0 81.2 86.3 32.6 80.7 62.4 81.0 81.3 84.3 82.1 56.2 84.6 58.2 76.2 67.2
DeepLab [12] 72.7 89.1 38.3 88.1 63.3 69.7 87.1 83.1 85.0 29.3 76.5 56.5 79.8 77.9 85.8 82.4 57.4 84.3 54.9 80.5 64.1
Methods trained
without COCO
Our method 73.9 89.3 40.0 81.6 65.1 71.7 90.1 81.3 85.7 32.4 82.1 62.2 82.6 83.7 84.5 81.1 60.8 85.2 49.6 80.0 69.9
DPN [9] 74.1 87.7 59.4 78.4 64.9 70.3 89.3 83.5 86.1 31.7 79.9 62.6 81.9 80.0 83.5 82.3 60.5 83.2 53.4 77.9 65.0
DeconvNet [51] 72.5 89.9 39.3 79.7 63.9 68.2 87.4 81.2 86.1 28.5 77.0 62.0 79.0 80.3 83.6 80.2 58.8 83.4 54.3 80.7 65.0
CRF-as-RNN [10] 72.0 87.5 39.0 79.7 64.2 68.3 87.6 80.8 84.4 30.4 78.2 60.4 80.5 77.8 83.1 80.6 59.5 82.8 47.8 78.3 67.1
DeepLab [12] 71.6 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 74.1 59.8 79.0 76.1 83.2 80.8 59.7 82.2 50.4 73.1 63.7
Piecewise [11] 70.7 87.5 37.7 75.8 57.4 72.3 88.4 82.6 80.0 33.4 71.5 55.0 79.3 78.4 81.3 82.7 56.1 79.8 48.6 77.1 66.3
Zoomout [52] 69.6 85.6 37.3 83.2 62.5 66.0 85.1 80.7 84.9 27.2 73.2 57.5 78.1 79.2 81.1 77.1 53.6 74.0 49.2 71.7 63.3
FCN-8s [3] 62.2 76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 62.5 46.8 71.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 55.1
CFM [28] 61.8 75.7 26.7 69.5 48.8 65.6 81.0 69.2 73.3 30.0 68.7 51.5 69.1 68.1 71.7 67.5 50.4 66.5 44.4 58.9 53.5
NUS UDS [53] 50.0 67.0 24.5 47.2 45.0 47.9 65.3 60.6 58.5 15.5 50.8 37.4 45.8 59.9 62.0 52.7 40.8 48.2 36.8 53.1 45.6
O2P [54] 47.8 64.0 27.3 54.1 39.2 48.7 56.6 57.7 52.5 14.2 54.8 29.6 42.2 58.0 54.8 50.2 36.6 58.6 31.6 48.4 38.6
Higher Order Conditional Random Fields in Deep Neural Networks 21
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 6: Examples of images where our method has improved over our baseline,
CRF-as-RNN [10]. The input images have the detection bounding boxes overlaid on
them. Note that the method of [10] does not make use of this information. The improve-
ments from our method are due to our detection potentials, as well as our superpixel
based potentials. Note that all images are from the reduced validation set of VOC 2012
and have not been trained on at all. Best viewed in color.
22 Arnab et al.
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 7: Examples of failure cases where our method has performed poorly. The ﬁrst
row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our
superpixel potentials have a tendency to remove spurious noise by enforcing consistency
within regions. However, as shown in the second row, sometimes the “noise” being
removed is actually the correct label. In the other cases, we are limited by our pixelwise
classiﬁcation unaries which are poor. Our superpixel and detection potentials are not
always able to compensate for this. Note that all images are from the reduced validation
set of VOC 2012 and have not been trained on at all. The input images have the detection
bounding boxes overlaid on them. Note that the method of [10] does not make use of
this information. Best viewed in color.
Higher Order Conditional Random Fields in Deep Neural Networks 23
Input image Pairwise onlySuperpixels
onlyDetections
onlyDetections
and
SuperpixelsGround truth
Fig. 8: Comparison of pairwise potentials, superpixel and pairwise potentials, de-
tection and pairwise potentials, and a combination of all three (Row 1 and 2) These
are examples where superpixel potentials help to remove spurious noise in the output
but detection potentials do not affect the result. The ﬁnal result still improves when
all potentials are combined. (Row 3) Detection potentials greatly improve the result
by recognising the train correctly (the pixelwise unaries are largest for “bus”). And
superpixels, when combined with detections, slightly improve the output. (Row 4) An
example where both superpixel and detection potentials improve the ﬁnal output. (Row 5)
A case where the superpixel worsens the result as, although the output is more consistent
among superpixel regions, some pixels have had their correct labels removed. However,
the correct detection improves the result, and the output of combining superpixel and
detection potentials is actually better than either potential in isolation. (Row 6) Here, the
detection (although correct) worsens the output due to its imprecise foreground mask.
Superpixel potentials also exacerbate the result, since the legs of the chair and the chair’s
shadow are confused to be part of the same superpixel region. However, when the two
potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.
Input image FCN-8s [3] Deeplab [12] CRF-as-RNN [10] Our method Ground truth
Fig. 9: Qualitative comparison with other current methods. Sample results of our
method compared to other current techniques on VOC 2012. We reproduced the segmen-
tation results of Deeplab from their original publication, whilst we reproduced the results
of FCN-8s and CRF-as-RNN from their publicly-available source code. Best viewed in
colour.