The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani
Semantic Segmentation with Boundary Neural Fields
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
The state-of-the-art in semantic segmentation is cur-
rently represented by fully convolutional networks (FCNs).
However, FCNs use large receptive ﬁelds and many pool-
ing layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to pro-
duce segmentations that are poorly localized around object
boundaries. Prior work has attempted to address this issue
in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these ap-
proaches require additional parameters and low-level fea-
tures that are difﬁcult to tune and integrate into the original
network architecture. Additionally, most CRFs use color-
based pixel afﬁnities, which are not well suited for semantic
segmentation and lead to spatially disjoint predictions.
To overcome these problems, we introduce a Boundary
Neural Field (BNF), which is a global energy model inte-
grating FCN predictions with boundary cues. The bound-
ary information is used to enhance semantic segment co-
herence and to improve object localization. Speciﬁcally, we
ﬁrst show that the convolutional ﬁlters of semantic FCNs
provide good features for boundary detection. We then em-
ploy the predicted boundaries to deﬁne pairwise potentials
in our energy. Finally, we show that our energy decom-
poses semantic segmentation into multiple binary problems,
which can be relaxed for efﬁcient global optimization. We
report extensive experiments demonstrating that minimiza-
tion of our global boundary-based energy yields results su-
perior to prior globalization methods, both quantitatively
as well as qualitatively.
1. Introduction
The recent introduction of fully convolutional networks
(FCNs) [22] has led to signiﬁcant quantitative improve-
ments on the task of semantic segmentation. However, de-
spite their empirical success, FCNs suffer from some limita-
tions. Large receptive ﬁelds in the convolutional layers and
the presence of pooling layers lead to blurring and segmen-
tation predictions at a signiﬁcantly lower resolution than the
Figure 1: Examples illustrating shortcomings of prior se-
mantic segmentation methods: the second column shows
results obtained with a FCN [22], while the third column
shows the output of a Dense-CRF applied to FCN predic-
tions [19, 7]. Segments produced by FCN are blob-like and
are poorly localized around object boundaries. Dense-CRF
produces spatially disjoint object segments due to the use of
a color-based pixel afﬁnity function that is unable to mea-
sure semantic similarity between pixels.
original image. As a result, their predicted segments tend to
be blobby and lack ﬁne object boundary details. We report
in Fig. 1 some examples illustrating typical poor localiza-
tion of objects in the outputs of FCNs.
Recently, Chen at al. [7] addressed this issue by apply-
ing a Dense-CRF post-processing step [19] on top of coarse
FCN segmentations. However, such an approach introduces
several problems of its own. First, the Dense-CRF adds new
parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods
based on CRFs or MRFs use low-level pixel afﬁnity func-
tions, such as those based on color. These low-level afﬁni-
ties often fail to capture semantic relationships between ob-
jects and lead to poor segmentation results (see last column
in Fig. 1).
We propose to address these shortcomings by means of
a Boundary Neural Field (BNF), an architecture that em-
ploys a single semantic segmentation FCN to predict se-
mantic boundaries and then use them to produce semantic
segmentation maps via a global optimization. We demon-
strate that even though the semantic segmentation FCN has
not been optimized to detect boundaries, it provides good
1arXiv:1511.02674v2  [cs.CV]  24 May 2016
...
Convolutional Feature Maps RGB Input Image
Segmentation Mask in       Softmax Layer
   Minimize Global Energy
Boundaries
|{z}...
wn()++()w1
Xi✓i(zi)+Xij✓ij(zi,zj)
Linear Combination of Feature MapsFigure 2: The architecture of our system (best viewed in
color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries
for our global energy; 2) to compute object boundaries.
Speciﬁcally, we deﬁne semantic boundaries as a linear com-
bination of these feature maps (with a sigmoid function ap-
plied on top of the sum) and learn individual weights corre-
sponding to each convolutional feature map. We integrate
this boundary information in the form of pairwise potentials
(pixel afﬁnities) for our energy model.
features for boundary detection. Speciﬁcally, the contribu-
tions of our work are as follows:
We show that semantic boundaries can be expressed
as a linear combination of interpolated convolutional
feature maps inside an FCN. We introduce a boundary
detection method that exploits this intuition to predict
object boundaries with accuracy superior to the state-
the-of-art.
We demonstrate that boundary-based pixel afﬁnities
are better suited for semantic segmentation than the
commonly used color afﬁnity functions.
Finally, we introduce a new global energy that de-
composes semantic segmentation into multiple binary
problems and relaxes the integrality constraint. We
show that minimizing our proposed energy yields bet-
ter qualitative and quantitative results relative to tradi-
tional globalization models such as MRFs or CRFs.
2. Related Work
Boundary Detection. Spectral methods comprise one
of the most prominent categories for boundary detection.
In a typical spectral framework, one formulates a general-
ized eigenvalue system to solve a low-level pixel groupingproblem. The resulting eigenvectors are then used to pre-
dict the boundaries. Some of the most notable approaches
in this genre are MCG [2], gPb [1], PMI [17], and Normal-
ized Cuts [29]. A weakness of spectral approaches is that
they tend to be slow as they perform a global inference over
the entire image.
To address this issue, recent approaches cast boundary
detection as a classiﬁcation problem and predict the bound-
aries in a local manner with high efﬁciency. The most no-
table examples in this genre include sketch tokens (ST) [20]
and structured edges (SE) [9], which employ fast random
forests. However, many of these methods are based on
hand-constructed features, which are difﬁcult to tune.
The issue of hand-constructed features have been re-
cently addressed by several approaches based on deep learn-
ing, such as N4ﬁelds [11], DeepNet [18], DeepCon-
tour [27], DeepEdge [3], H FL [4] and HED [33]. All of
these methods use CNNs in some way to predict the bound-
aries. Whereas DeepNet and DeepContour optimize ordi-
nary CNNs to a boundary based optimization criterion from
scratch, DeepEdge and H FL employ pretrained models to
compute boundaries. The most recent of these methods is
HED [33], which shows the beneﬁt of deeply supervised
learning for boundary detection.
In comparison to prior deep learning approaches, our
method offers several contributions. First, we exploit the in-
herent relationship between boundary detection and seman-
tic segmentation to predict semantic boundaries. Specif-
ically, we show that even though the semantic FCN has
not been explicitly trained to predict boundaries, the con-
volutional ﬁlters inside the FCN provide good features for
boundary detection. Additionally, unlike DeepEdge [3] and
HFL [4], our method does not require a pre-processing step
to select candidate contour points, as we predict boundaries
on all pixels in the image. We demonstrate that our ap-
proach allows us to achieve state-of-the-art boundary detec-
tion results according to both F-score and Average Preci-
sion metrics. Additionally, due to the semantic nature of
our boundaries, we can successfully use them as pairwise
potentials for semantic segmentation in order to improve
object localization and recover ﬁne structural details, typ-
ically lost by pure FCN-based approaches.
Semantic Segmentation. We can group most seman-
tic segmentation methods into three broad categories. The
ﬁrst category can be described as “two-stage” approaches,
where an image is ﬁrst segmented and then each segment
is classiﬁed as belonging to a certain object class. Some
of the most notable methods that belong to this genre in-
clude [24, 6, 12, 14].
The primary weakness of the above methods is that they
are unable to recover from errors made by the segmentation
algorithm. Several recent papers [15, 10] address this issue
by proposing to use deep per-pixel CNN features and then
classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmenta-
tion problem, they perform predictions independently on
each pixel. This leads to extremely local predictions, where
the relationships between pixels are not exploited in any
way, and thus the resulting segmentations may be spatially
disjoint.
The third and ﬁnal group of semantic segmentation
methods can be viewed as front-to-end schemes where seg-
mentation maps are predicted directly from raw pixels with-
out any intermediate steps. One of the earliest examples of
such methods is the FCN introduced in [22]. This approach
gave rise to a number of subsequent related approaches
which have improved various aspects of the original seman-
tic segmentation [7, 34, 8, 16, 21]. There have also been at-
tempts at integrating the CRF mechanism into the network
architecture [7, 34]. Finally, it has been shown that semantic
segmentation can also be improved using additional training
data in the form of bounding boxes [8].
Our BNF offers several contributions over prior work. To
the best of our knowledge, we are the ﬁrst to present a model
that exploits the relationship between boundary detection
and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from seman-
tic boundaries inside an FCN, and use these boundaries to
predict the segmentations in a global fashion. Unlike [21],
which requires a large number of additional parameters to
learn for the pairwise potentials, our global model only
needs5Kextra parameters, which is about 3orders of
magnitudes less than the number of parameters in a typi-
cal deep convolutional network (e.g. VGG [30]). We em-
pirically show that our proposed boundary-based afﬁnities
are better suited for semantic segmentation than color-based
afﬁnities. Additionally, unlike in [7, 34, 21], the solution to
our proposed global energy can be obtained in closed-form,
which makes global inference easier. Finally we demon-
strate that our method produces better results than tradi-
tional globalization models such as CRFs or MRFs.
3. Boundary Neural Fields
In this section, we describe Boundary Neural Fields.
Similarly to traditional globalization methods, Boundary
Neural Fields are deﬁned by an energy including unary
and pairwise potentials. Minimization of the global en-
ergy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image
and then combine them in a global manner. More precisely,
the coarse segmentations predicted by a semantic FCN are
used to deﬁne the unary potentials of our BNF. Next, we
show that the convolutional feature maps of the FCN can
be used to accurately predict semantic boundaries. These
boundaries are then used to build pairwise pixel afﬁnities,
which are used as pairwise potentials by the BNF. Finally,we introduce a global energy function, which minimizes the
energy corresponding to the unary and pairwise terms and
improves the initial FCN segmentation. The detailed illus-
tration of our architecture is presented in Figure 2. We now
explain each of these steps in more detail.
3.1. FCN Unary Potentials
To predict semantic unary potentials we employ the
DeepLab model [7], which is a fully convolutional adapta-
tion of the VGG network [30]. The FCN consists of 16con-
volutional layers and 3fully convolutional layers. There are
more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].
Although these more advanced architectures could be in-
tegrated into our framework to improve our unary poten-
tials, in this work we focus on two aspects orthogonal to
this prior work: 1) demonstrating that our boundary-based
afﬁnity function is better suited for semantic segmentation
than the common color-based afﬁnities and 2) showing that
our proposed global energy achieves better qualitative and
quantitative semantic segmentation results in comparison to
prior globalization models.
3.2. Boundary Pairwise Potentials
In this section, we describe our approach for building
pairwise pixel afﬁnities using semantic boundaries. The ba-
sic idea behind our boundary detection approach is to ex-
press semantic boundaries as a function of convolutional
feature maps inside the FCN. Due to the close relationship
between the tasks of semantic segmentation and boundary
detection, we hypothesize that convolutional feature maps
from the semantic segmentation FCN can be employed as
features for boundary detection.
3.2.1 Learning to Predict Semantic Boundaries.
We propose to express semantic boundaries as a linear com-
bination of interpolated FCN feature maps with a non-linear
function applied on top of this sum. We note that interpo-
lation of feature maps has been successfully used in prior
work (see e.g. [15]) in order to obtain dense pixel-level fea-
tures from the low-resolution outputs of deep convolutional
layers. Here we adopt interpolation to produce pixel-level
boundary predictions. There are several advantages to our
proposed formulation. First, because we express boundaries
as a linear combination of feature maps, we only need to
learn a small number of parameters, corresponding to the
individual weight values of each feature map in the FCN.
This amounts to5Klearning parameters, which is much
smaller than the number of parameters in the entire network
(15M). In comparison, DeepEdge [3] and HFL [4] need
17M and 6M additional parameters to predict boundaries.
Furthermore, expressing semantic boundaries as a linear
combination of FCN feature maps allows us to efﬁciently
predict boundary probabilities for all pixels in the image
(we resize the FCN feature maps to the original image di-
mensions). This eliminates the need to select candidate
boundary points in a pre-processing stage, which was in-
stead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as
follows. First we use use SBD segmentations [13] to op-
timize our FCN for semantic segmentation task. We then
treat FCN convolutional maps as features for the bound-
ary detection task and use the boundary annotations from
BSDS 500 dataset [23] to learn the weights for each feature
map. BSDS 500 dataset contains 200training, 100valida-
tion, 200testing images, and ground truth annotations by 5
human labelers for each of these images.
To learn the weights corresponding to each convolutional
feature map we ﬁrst sample 80Kpoints from the dataset.
We deﬁne the target labels for each point as the fraction of
human annotators agreeing on that point being a boundary.
To ﬁx the issue of label imbalance (there are many more
non-boundaries than boundaries), we divide the label space
into four quartiles, and select an equal number of samples
for each quartile to balance the training dataset. Given these
sampled points, we then deﬁne our features as the values in
the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up
and apply a sigmoid function on top of it. We obtain the
weights corresponding to each convolutional feature map by
minimizing the cross-entropy loss using a stochastic batch
gradient descent for 50epochs. To obtain crisper bound-
aries at test-time we post-process the boundary probabilities
using non-maximum suppression.
To give some intuition on how FCN feature maps con-
tribute to boundary detection, in Fig. 3 we visualize the fea-
ture maps corresponding to the highest weight magnitudes.
It is clear that many of these maps contain highly localized
boundary information.
Boundary Detection Results Before discussing how
boundary information is integrated in our energy for seman-
tic segmentation, here we present experimental results as-
sessing the accuracy of our boundary detection scheme. We
tested our boundary detector on the BSDS500 dataset [23],
which is the standard benchmark for boundary detection.
The quality of the predicted boundaries is evaluated using
three standard measures: ﬁxed contour threshold (ODS),
per-image best threshold (OIS), and average precision (AP).
In Table 1 we show that our algorithm outperforms all
prior methods according to both F-score measures and the
Average Precision metric. In Fig. 4, we also visualize our
predicted boundaries. The second column shows the pixel-
level softmax output computed from the linear combina-
tion of feature maps, while the third column depicts our ﬁ-
Figure 3: An input image and convolutional feature maps
corresponding to the largest weight magnitude values. Intu-
itively these are the feature maps that contribute most heav-
ily to the task of boundary detection.
Method ODS OIS AP
SCG [25] 0.739 0.758 0.773
SE [9] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [11] 0.753 0.769 0.784
DeepEdge [3] 0.753 0.772 0.807
DeepContour [27] 0.756 0.773 0.797
HFL [4] 0.767 0.788 0.795
HED [33] 0.782 0.804 0.833
BNF 0.788 0.807 0.851
Table 1: Boundary detection results on BSDS500 bench-
mark. Our proposed method outperforms all prior algo-
rithms according to all three evaluation metrics.
nal boundaries after applying a non-maximum suppression
post-processing step.
We note that our predicted boundaries achieve high-
conﬁdence predictions around objects. This is important as
we employ these boundaries to improve semantic segmen-
tation results, as discussed in the next subsection.
3.2.2 Constructing Pairwise Pixel Afﬁnities.
We can use the predicted boundaries to build pairwise pixel
afﬁnities. Intuitively, we declare two pixels as similar (i.e.,
likely to belong to the same segment) if there is no bound-
ary crossing the straight path between these two pixels.
Conversely, two pixels are dissimilar if there is a bound-
ary crossing their connecting path. The larger the boundary
magnitude of the crossed path, the more dissimilar the two
pixels should be, since a strong boundary is likely to mark
the separation of two distinct segments. Similarly to [1], we
encode this intuition with a following formulation:
wsb
ij= exp (Mij
sb) (1)
Figure 4: A ﬁgure illustrating our boundary detection re-
sults. In the second column, we visualize the raw probabil-
ity output of our boundary detector. In the third column,
we present the ﬁnal boundary maps after non-maximum
suppression. While most prior methods predict the bound-
aries where the sharpest change in color occurs, our method
captures semantic object-level boundaries, which we subse-
quently use to aid semantic segmentation.
whereMijdenotes the maximum boundary value that
crosses the straight line path between pixels iandj,sbde-
picts the smoothing parameter and wsb
ijdenotes the semantic
boundary-based afﬁnity between pixels iandj.
Similarly, we want to exploit high-level object informa-
tion in the network to deﬁne another type of pixel similarity.
Speciﬁcally, we use object class probabilities from the soft-
max (SM) layer to achieve this goal. Intuitively, if pixels i
andjhave different hard segmentation labels from the soft-
max layer, we set their similarity ( wsm
ij) to0. Otherwise,
we compute their similarity using the following equation:
wsm
ij= exp (Dij
sm) (2)
whereDijdenotes the difference in softmax output val-
ues corresponding to the most likely object class for pixels
iandj, andsmis a smoothing parameter. Then we can
write the ﬁnal afﬁnity measure as:
wij= exp (wsm
ij)wsb
ij (3)
We exponentiate the term corresponding to the object-
level afﬁnity because our boundary-based afﬁnity may be
too aggressive in declaring two pixels as dissimilar. To ad-
dress this issue, we increase the importance of the object-
level afﬁnity in (3) using the exponential function. How-
ever, in the experimental results section, we demonstrate
that most of the beneﬁt from modeling pairwise potentials
comes from wsb
ijrather thanwsm
ij.We then use this pairwise pixel afﬁnity measure to build
a global afﬁnity matrix Wthat encodes relationships be-
tween pixels in the entire image. For a given pixel, we
sample10% of points in the neighborhood of radius 20
around that pixel, and store the resulting afﬁnities into W.
3.3. Global Inference
The last step in our proposed method is to combine se-
mantic boundary information with the coarse segmentation
from the FCN softmax layer to produce an improved seg-
mentation. We do this by introducing a global energy func-
tion that utilizes the afﬁnity matrix constructed in the pre-
vious section along with the segmentation from the FCN
softmax layer. Using this energy, we perform a global infer-
ence to get segmentations that are well localized around the
object boundaries and that are also spatially smooth.
Typical globalization models such as MRFs [31],
CRFs [19] or Graph Cuts [5] produce a discrete label as-
signment for the segmentation problem by jointly model-
ing a multi-label distribution and solving a non-convex op-
timization. The common problem in doing so is that the
optimization procedure may get stuck in local optima.
We introduce a new global energy function, which over-
comes this issue and achieves better segmentation in com-
parison to prior globalization models. Similarly to prior
globalization approaches, our goal is to minimize the energy
corresponding to the sum of unary and pairwise potentials.
However, the key difference in our approach comes from the
relaxation of some of the constraints. Speciﬁcally, instead
of modeling our problem as a joint multi-label distribution,
we propose to decompose it into multiple binary problems,
which can be solved concurrently. This decomposition can
be viewed as assigning pixels to foreground and background
labels for each of the different object classes. Additionally,
we relax the integrality constraint. Both of these relaxations
make our problem more manageable and allow us to formu-
late a global energy function that is differentiable, and has
a closed form solution.
In [35], the authors introduce the idea of learning
with global and local consistency in the context of semi-
supervised problems. Inspired by this work, we incorporate
some of these ideas in the context of semantic segmenta-
tion. Before deﬁning our proposed global energy function,
we introduce some relevant notation.
For the purpose of illustration, suppose that we only have
two classes: foreground and background. Then we can de-
note an optimal continuous solution to such a segmentation
problem with variable z. To denote similarity between pix-
elsiandjwe usewij. Then,diindicates the degree of a
pixeli. In graph theory, the degree of a node denotes the
number of edges incident to that node. Thus, we set the de-
gree of a pixel to di=Pn
j=1wijfor alljexcepti6=j.
Finally, with fiwe denote an initial segmentation proba-
Input
 Softmax
 Dense-CRF
 BNF Boundaries
 BNF Segmentation
Figure 5: A ﬁgure illustrating semantic segmentation results. Images in columns two and three represent FCN softmax and
Dense-CRF predictions, respectively. Note that all methods use the same FCN unary potentials. Additionally, observe that
unlike FCN and Dense-CRF, our methods predicts segmentation that are both well localized around object boundaries and
that are also spatially smooth.
bility, which in our case is obtained from the FCN softmax
layer.
Using this notation, we can then formulate our global
inference objective as:
z= argmin
z
2X
idi(zifi
di)2+1
2X
ijwij(zizj)2(4)
This energy consists of two different terms. Similar to
the general globalization framework, our ﬁrst term encodes
the unary energy while the second term includes the pair-
wise energy. We now explain the intuition behind each of
these terms. The unary term attempts to ﬁnd a segmentation
assignment ( zi) that deviates little from the initial candidate
segmentation computed from the softmax layer (denoted by
fi). Theziin the unary term is weighted by the degree di
of the pixel in order to produce larger unary costs for pixels
that have many similar pixels within the neighborhood. In-
stead, the pairwise term ensures that pixels that are similar
should be assigned similar zvalues. To balance the ener-
gies of the two terms we introduce a parameter and set it
to0:025throughout all our experiments.
We can also express the same global energy function in
matrix notation:
z= argmin
z
2D(zD1f)T(zD1f)+1
2zT(DW)z
(5)
where zis an1vector containing an optimal con-
tinuous assignment for all npixels, Dis a diagonal degree
matrix, and Wis thennpixel afﬁnity matrix. Finally, f
denotes an1vector containing the probabilities from the
softmax layer corresponding to a particular object class.
An advantage of our energy is that it is differentiable. Ifwe denote the above energy as E(z)then the derivative of
this energy can be written as follows:
@E(z)
@z=D(zD1f) + (DW)z=0 (6)
With simple algebraic manipulations we can then obtain
a closed form solution to this optimization:
z= (DW)1f (7)
where=1
1+and=
1+. In the general case where
we havekobject classes we can write the solution as:
Z= (DW)1F (8)
where Znow depicts a nkmatrix containing assign-
ments for all kobject classes, while Fdenotesnkmatrix
with object class probabilities from softmax layer. Due to
the large size of DWit is impractical to invert it. How-
ever, if we consider an image as a graph where each pixel
denotes a vertex in the graph, we can observe that the term
DWin our optimization is equivalent to a Laplacian ma-
trix of such graph. Since we know that a Laplacian matrix is
positive semi-deﬁnite, we can use the preconditioned con-
jugate gradient method [28] to solve the system in Eq. (9).
Alternatively, because our deﬁned global energy in Eq. (5)
is differentiable, we can efﬁciently solve this optimization
problem using stochastic gradient descent. We choose the
former option and solve the following system:
(DW)z=f (9)
To obtain the ﬁnal discrete segmentation, for each pixel
we assign the object class that corresponds to the largest
column value in the row of Z(note that each row in Zrep-
resents a single pixel in the image, and each column in Z
Metric Inference Method RGB Afﬁnity BNF Afﬁnity
PP-IOUBelief Propagation [31] 75.4 75.6
ICM 74.2 75.8
TRWS [32] 75.9 76.7
QPBO [26] 76.9 77.2
BNF 74.6 77.6
PI-IOUBelief Propagation [31] 45.9 46.2
ICM 45.7 48.8
TRWS [32] 51.5 52.0
QPBO [26] 55.3 57.2
BNF 53.0 58.5
Table 2: We compare semantic segmentation results
when using a color-based pixel afﬁnity and our proposed
boundary-based afﬁnity. We note that our proposed afﬁnity
improves the performance of all globalization techniques.
Note that all of the inference methods use the same FCN
unary potentials . This suggests that for every method our
boundary-based afﬁnity is more beneﬁcial for semantic seg-
mentation than the color-based afﬁnity.
represents one of the object classes). In the experimental
section, we show that this solution produces better quantita-
tive and qualitative results in comparison to commonly used
globalization techniques.
4. Experimental Results
In this section we present quantitative and qualitative re-
sults for semantic segmentation on the SBD [13] dataset,
which contains objects and their per-pixel annotations for
20Pascal VOC classes. We evaluate semantic segmenta-
tion results using two evaluation metrics. The ﬁrst metric
measures accuracy based on pixel intersection-over-union
averaged per pixels (PP-IOU) across the 20 classes. Ac-
cording to this metric, the accuracy is computed on a per-
pixel basis. As a result, the images that contain large ob-
ject regions are given more importance. However, for cer-
tain applications we may need to accurately segment small
objects. Therefore, similar to [4] we also consider the PI-
IOU metric (pixel intersection-over-union averaged per im-
age across the 20 classes), which gives equal weight to each
of the images.
We compare Boundary Neural Fields with other com-
monly used global inference methods. These methods in-
clude Belief Propagation [31], Iterated Conditional Mode
(ICM), Graph Cuts [5], and Dense-CRF [19]. Note that in
all of our evaluations we use the same FCN unary potentials
for every model.
Our evaluations provide evidence for three conclusions:
In Subsection 4.1, we show that our boundary-based
pixel afﬁnities are better suited for semantic segmenta-
tion than the traditional color-based afﬁnities.
In Subsection 4.2, we demonstrate that our global min-
imization leads to better results than those achieved byother inference schemes.
In Fig. 5, we qualitatively compare the outputs of FCN
and Dense-CRF to our predicted segmentations. This
comparison shows that the BNF segments are better
localized around the object boundaries and that they
are also spatially smooth.
4.1. Comparing Afﬁnity Functions for Semantic
Segmentation
In Table 2, we consider two global models. Both mod-
els use the same unary potentials obtained from the FCN
softmax layer. However, the ﬁrst model uses the popular
color-based pairwise afﬁnities, while the second employs
our boundary-based afﬁnities. Each of these two models
is optimized using several inference strategies. The table
shows that using our boundary based-afﬁnity function im-
proves the results of all global inference methods accord-
ing to both evaluation metrics. Note that we cannot in-
clude Dense-CRF [19] in this comparison because it em-
ploys an efﬁcient message-passing technique and integrat-
ing our afﬁnities into this technique is a non-trivial task.
However, we compare our method with Dense-CRF in Sub-
section 4.2.
The results in Table 2 suggest that our semantic bound-
ary based pixel afﬁnity function yields better semantic seg-
mentation results compared to the commonly-used color
based afﬁnities. We note that we also compared the results
of our inference technique using other edge detectors, no-
tably UCM [1] and H FL [4]. In comparison to UCM edges,
we observed that our boundaries provide 1:0%and6:0%
according to both evaluation metrics respectively. When
comparing our boundaries with H FL method, we observed
similar segmentation performance, which suggests that our
method works best with the high quality semantic bound-
aries.
4.2. Comparing Inference Methods for Semantic
Segmentation
Additionally, we also present semantic segmentation re-
sults for both of the metrics (PP-IOU and PI-IOU) in Ta-
ble 3. In this comparison, all the techniques use the same
FCN unary potentials. Additionally, all inference methods
except Dense-CRF use our afﬁnity measure (since the pre-
vious analysis suggested that our afﬁnities yield better per-
formance). We use BNF-SB to denote the variant of our
method that uses only semantic boundary based afﬁnities.
Additionally, we use BNF-SB-SM to indicate the version
of our method that uses both boundary and softmax -based
afﬁnities (see Eq. (3)).
Based on these results, we observe that our proposed
technique outperforms all the other globalization methods
according to both metrics, by 0:3%and1:3%respectively.
Metric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
PP-IOUFCN-Softmax 80.7 71.6 80.7 71.3 72.9 88.1 81.8 86.6 47.4 82.9 57.9 83.9 79.6 80.4 81.0 64.7 78.2 54.5 80.9 69.9 74.8
Belief Propagation [31] 81.4 72.2 82.4 72.2 74.3 88.8 82.4 87.2 48.4 83.8 58.4 84.6 80.5 80.9 81.5 65.1 79.5 55.5 81.5 71.2 75.6
ICM 81.7 72.2 82.8 72.1 75.3 89.6 83.4 87.7 46.3 83.3 58.4 84.6 80.6 81.4 81.5 65.8 79.5 56.0 80.7 74.1 75.8
TRWS [32] 81.6 70.9 83.8 72.0 75.1 89.5 82.5 88.0 51.7 86.6 61.9 85.8 83.3 80.8 81.1 65.3 81.5 58.8 77.6 75.9 76.7
Graph Cuts [5] 82.5 72.4 84.6 73.3 77.2 89.7 83.3 88.8 49.3 84.0 60.3 85.4 82.2 81.2 81.9 66.7 79.8 58.0 82.3 74.9 76.9
QPBO [26] 82.6 72.3 84.7 73.1 76.7 89.9 83.6 89.3 49.7 85.0 61.1 86.2 82.9 81.3 82.3 67.1 80.5 58.8 82.2 75.1 77.2
Dense-CRF [19] 83.4 71.5 84.9 72.6 76.2 89.5 83.3 89.1 50.4 86.7 61.0 86.8 83.5 81.8 82.3 66.9 82.2 58.2 81.9 75.1 77.3
BNF-SB 81.9 72.5 84.9 73.3 76.0 90.3 83.1 89.2 51.2 86.7 61.5 86.6 83.2 81.3 81.9 66.2 81.7 58.6 81.6 75.8 77.4
BNF-SB-SM 82.2 73.1 85.1 73.8 76.7 90.6 83.4 89.5 51.3 86.7 61.4 86.8 83.3 81.7 82.3 67.7 81.9 58.4 82.4 75.4 77.6
PI-IOUFCN-Softmax 56.9 35.1 47.8 41.1 27.4 51.1 43.4 52.7 22.2 43.1 29.2 54.2 40.5 45.6 59.1 24.2 43.6 24.8 55.9 37.2 41.8
Belief Propagation [31] 68.0 38.6 52.9 45.8 31.9 55.9 47.2 58.2 24.6 49.9 31.7 60.2 44.9 50.1 62.4 25.2 49.9 27.6 62.3 42.2 46.2
ICM 65.3 40.9 56.4 45.3 33.7 58.9 49.5 61.9 25.8 53.5 33.2 62.1 48.0 53.2 63.4 24.1 54.8 34.0 63.7 47.7 48.8
TRWS [32] 67.5 40.7 60.3 46.3 35.6 63.4 49.6 69.3 29.7 58.9 37.8 67.4 57.3 53.8 64.1 26.3 62.0 36.9 63.1 49.9 52.0
Graph Cuts [5] 72.1 47.8 64.5 50.8 36.0 70.8 51.4 71.6 31.7 65.8 34.4 71.8 62.0 59.4 64.8 29.0 60.9 38.7 70.3 51.6 55.3
QPBO [26] 71.6 46.8 65.6 49.6 38.0 72.6 52.7 76.7 32.5 69.6 38.9 74.4 61.4 61.0 66.2 30.3 68.7 41.4 72.2 52.8 57.2
Dense-CRF [19] 68.0 39.5 58.0 45.0 33.4 62.8 47.7 66.0 29.4 60.9 36.0 68.5 54.6 51.4 63.7 28.3 57.6 37.1 65.9 48.2 51.1
BNF-SB 71.6 48.1 67.2 52.3 37.8 79.5 52.9 80.8 33.3 71.5 39.5 75.1 65.7 63.4 65.1 31.1 67.5 39.6 73.2 54.7 58.5
BNF-SB-SM 72.0 48.9 66.5 52.9 39.1 79.0 53.4 78.6 32.9 72.2 39.4 74.6 65.9 64.2 65.8 31.7 66.9 39.0 73.1 53.9 58.5
Table 3: Semantic segmentation results on the SBD dataset according to PP-IOU (per pixel) and PI-IOU (per image) eval-
uation metrics. We use BNF-SB to denote the variant of our method that uses only semantic boundary based afﬁnities.
Additionally, we use BNF-SB-SM to indicate our method that uses boundary and softmax based afﬁnities (See Eq. (3)). We
observe that our proposed globalization method outperforms other globalization techniques according to both metrics by
at least 0:3%and1:3%respectively. Note that in this experiment, all of the inference methods use the same FCN unary
potentials . Additionally, for each method except Dense-CRF (it is challenging to incorporate boundary based afﬁnities into
the Dense-CRF framework) we use our boundary based afﬁnities, since those lead to better results.
Additionally, these results indicate that most beneﬁt comes
from the semantic boundary afﬁnity term rather than the
softmax afﬁnity term.
In Fig. 5, we also present qualitative semantic segmenta-
tion results. Note that, compared to the segmentation out-
put from the softmax layer, our segmentation is much bet-
ter localized around the object boundaries. Additionally,
in comparison to Dense-CRF predictions, our method pro-
duces segmentations that are much spatially smoother.
4.3. Semantic Boundary Classiﬁcation
We can also label our boundaries with a speciﬁc object
class, using the same classiﬁcation strategy as in the H FL
system [4]. Since the SBD dataset provides annotations for
semantic boundary classiﬁcation, we can test our results
against the state-of-the-art H FL [4] method for this task.
Due to the space limitation, we do not include full results for
each category. However, we observe that our produced re-
sults achieve mean Max F-Score of 54:5%(averaged across
all20classes) whereas H FL method obtains 51:7%.
5. Conclusions
In this work we introduced a Boundary Neural Field
(BNF), an architecture that employs a semantic segmenta-
tion FCN to predict semantic boundaries and then uses the
predicted boundaries and the FCN output to produce an im-
proved semantic segmentation maps a global optimization.
We showed that our predicted boundaries are better suited
for semantic segmentation than the commonly used low-level color based afﬁnities. Additionally, we introduced a
global energy function that decomposes semantic segmen-
tation into multiple binary problems and relaxes an inte-
grality constraint. We demonstrated that the minimization
of this global energy allows us to predict segmentations
that are better localized around the object boundaries and
that are spatially smoother compared to the segmentations
achieved by prior methods. We made the code of our global-
ization technique available at http://www.seas.upenn.
edu/~gberta/publications.html .
The main goal of this work was to show the effective-
ness of boundary-based afﬁnities for semantic segmenta-
tion. However, due to differentiability of our global energy,
it may be possible to add more parameters inside the BNFs
and learn them in a front-to-end fashion. We believe that
optimizing the entire architecture jointly could capture the
inherent relationship between semantic segmentation and
boundary detection even better and further improve the per-
formance of BNFs. We will investigate this possibility in
our future work.
6. Acknowledgements
This research was funded in part by NSF award CNS-
1205521.
References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Ma-
lik. Contour detection and hierarchical image segmentation. IEEE
Trans. Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2, 4, 7
[2] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqués, and Jitendra
Malik. Multiscale combinatorial grouping. In Computer Vision and
Pattern Recognition (CVPR) , 2014. 2, 4
[3] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A
multi-scale bifurcated deep network for top-down contour detection.
InThe IEEE Conference on Computer Vision and Pattern Recogni-
tion (CVPR) , June 2015. 2, 3, 4
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for-low
and low-for-high: Efﬁcient boundary detection from deep object fea-
tures and its applications to high-level vision. In The IEEE Interna-
tional Conference on Computer Vision (ICCV) , December 2015. 2,
3, 4, 7, 8
[5] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate en-
ergy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach.
Intell. , 23(11):1222–1239, November 2001. 5, 7, 8
[6] João Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu.
Semantic segmentation with second-order pooling. In Proceedings
of the 12th European Conference on Computer Vision - Volume Part
VII, ECCV’12, pages 430–443, Berlin, Heidelberg, 2012. Springer-
Verlag. 2
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin
Murphy, and Alan L. Yuille. Semantic image segmentation with deep
convolutional nets and fully. In ICLR , 2015. 1, 2, 3
[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bound-
ing boxes to supervise convolutional networks for semantic segmen-
tation. In The IEEE International Conference on Computer Vision
(ICCV) , December 2015. 3
[9] Piotr Dollár and C. Lawrence Zitnick. Fast edge detection using
structured forests. PAMI , 2015. 2, 4
[10] Clement Farabet, Camille Couprie, Laurent Najman, and Yann Le-
Cun. Learning hierarchical features for scene labeling. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence , August 2013.
2
[11] Yaroslav Ganin and Victor S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014. 2, 4
[12] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik.
Learning rich features from RGB-D images for object detection and
segmentation. In Proceedings of the European Conference on Com-
puter Vision (ECCV) , 2014. 2
[13] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu
Maji, and Jitendra Malik. Semantic contours from inverse detectors.
InInternational Conference on Computer Vision (ICCV) , 2011. 4, 7
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Ma-
lik. Simultaneous detection and segmentation. In European Confer-
ence on Computer Vision (ECCV) , 2014. 2
[15] Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and
Jitendra Malik. Hypercolumns for object segmentation and ﬁne-
grained localization. In The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , June 2015. 2, 3
[16] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled
deep neural network for semi-supervised semantic segmentation. In
NIPS) , December 2015. 3
[17] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson.
Crisp boundary detection using pointwise mutual information. In
ECCV , 2014. 2
[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual
boundary prediction: A deep neural prediction network and quality
dissection. AISTATS , 1(2):9, 2014. 2
[19] Philipp Krähenbühl and Vladlen Koltun. Efﬁcient inference in fully
connected crfs with gaussian edge potentials. In J. Shawe-Taylor,
R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors,
Advances in Neural Information Processing Systems 24 , pages 109–
117. Curran Associates, Inc., 2011. 1, 5, 7, 8[20] Joseph Lim, C. Lawrence Zitnick, and Piotr Dollár. Sketch tokens:
A learned mid-level representation for contour and object detection.
InCVPR , 2013. 2
[21] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton van den Hen-
gel. Efﬁcient piecewise training of deep structured models for se-
mantic segmentation. CoRR , abs/1504.01013, 2015. 3
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolu-
tional networks for semantic segmentation. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1,
3
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human
segmented natural images and its application to evaluating segmen-
tation algorithms and measuring ecological statistics. In Proc. 8th
Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4
[24] Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory
Shakhnarovich. Feedforward semantic segmentation with zoom-out
features. CoRR , abs/1412.0774, 2014. 2
[25] X. Ren and L. Bo. Discriminatively Trained Sparse Code Gradients
for Contour Detection. In Advances in Neural Information Process-
ing Systems , December 2012. 4
[26] Carsten Rother, Vladimir Kolmogorov, Victor Lempitsky, and Martin
Szummer. Optimizing binary mrfs via extended roof duality. In Proc
Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8
[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang
Zhang. Deepcontour: A deep convolutional feature learned by
positive-sharing loss for contour detection. June 2015. 2, 4
[28] Jonathan R Shewchuk. An introduction to the conjugate gradient
method without the agonizing pain. Technical report, Pittsburgh, PA,
USA, 1994. 6
[29] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmen-
tation. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence , 22:888–905, 1997. 2
[30] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 3
[31] Marshall F. Tappen and William T. Freeman. Comparison of graph
cuts with belief propagation for stereo, using identical mrf parame-
ters. In Proceedings of the Ninth IEEE International Conference on
Computer Vision - Volume 2 , ICCV ’03, pages 900–, Washington,
DC, USA, 2003. IEEE Computer Society. 5, 7, 8
[32] Martin Wainwright, Tommi Jaakkola, and Alan Willsky. Map esti-
mation via agreement on (hyper)trees: Message-passing and linear
programming approaches. IEEE Transactions on Information The-
ory, 51:3697–3717, 2002. 7, 8
[33] Saining Xie and Zhuowen Tu. Holistically-nested edge detection.
InThe IEEE International Conference on Computer Vision (ICCV) ,
December 2015. 2, 4
[34] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vib-
hav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr.
Conditional random ﬁelds as recurrent neural networks. In Interna-
tional Conference on Computer Vision (ICCV) , 2015. 3
[35] Dengyong Zhou, Olivier Bousquet, Thomas N. Lal, Jason Weston,
and Bernhard Schölkopf. Learning with local and global consistency.
In S. Thrun, L.K. Saul, and B. Schölkopf, editors, Advances in Neu-
ral Information Processing Systems 16 , pages 321–328. MIT Press,
2004. 5