SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes

arXiv:1709.07158v1 [] 21 Sep 2017

Trung T. Pham1,2 , Thanh-Toan Do1,2 , Niko Sünderhauf1,3 , Ian Reid1,2

Abstract— This paper presents SceneCut, a novel approach
to jointly discover previously unseen objects and non-object
surfaces using a single RGB-D image. SceneCut’s joint reasoning over scene semantics and geometry allows a robot to
detect and segment object instances in complex scenes where
modern deep learning-based methods either fail to separate
object instances, or fail to detect objects that were not seen
during training. SceneCut automatically decomposes a scene
into meaningful regions which either represent objects or scene
surfaces. The decomposition is qualified by an unified energy
function over objectness and geometric fitting. We show how
this energy function can be optimized efficiently by utilizing
hierarchical segmentation trees. Moreover, we leverage a pretrained convolutional oriented boundary network to predict
accurate boundaries from images, which are used to construct
high-quality region hierarchies. We evaluate SceneCut on several different indoor environments, and the results show that
SceneCut significantly outperforms all the existing methods.

(a) Input image

(b) RefineNet [1]

(c) Mask-RCNN [2]

(d) SceneCut

I. INTRODUCTION
In recent years, scene understanding driven by multiclass semantic segmentation [1], or object detection [2],
[3] has progressed significantly thanks to the power of
deep learning. However, a major limitation of these deep
learning based approaches is that they only work for a set
of known object classes that are used during training. In
contrast, autonomous robots often operate under open-set
conditions [4] in many application domains, i.e. they will
inevitably encounter objects that were not part of the training
dataset. State-of-the-art methods such as Mask-RCNN [2],
YOLO9000 [3], or RefineNet [1] fail to detect such unknown
objects. This behavior is detrimental to the goal of robotic
scene understanding that would ideally result in a semantically
meaningful map [5]–[7] comprising all objects, environmental
structures, and their various complex relations. The ability to
extract information about objects and the scene geometry in
complex environments under realistic, open-set conditions is
increasingly important for robotics.
We propose SceneCut, a novel approach to segment a
scene into class-agnostic objects without requiring semantic
class annotations for training. Using a single RGB-D image,
SceneCut can discover unseen objects in highly cluttered
indoor environments, thus allowing a higher degree of
completeness for robotic scene understanding or object-based
1 The

authors are with the Australian Centre for Robotic Vision.
T. Pham, Thanh-Toan Do, and Ian Reid are with the University
of Adelaide, SA 5005, Australia.
3 Niko Sünderhauf is with Queensland University of Technology (QUT),
Brisbane, QLD 4001, Australia.
Contact: trung.pham@adelaide.edu.au
The authors acknowledge the support of the Australian Research Council
through the Centre of Excellence for Robotic Vision (CE140100016) and
Laureate Fellowship (FL130100102) to IDR.
2 Trung

Fig. 1: SceneCut v.s. Mask-RCNN [2] v.s. RefineNet [1]. SceneCut
segments the input image into individual class-agnostic objects and
surfaces, encoded by different colors. RefineNet is unable to separate
object instances while Mask-RCNN is unable to detect surfaces and
unknown objects. Best viewed on-screen.

mapping or SLAM [6], [7]. In contrast to state-of-the-art
pixel-wise semantic segmentation [1], our approach can
differentiate between different object instances, e.g., chair A
and chair B. In addition, SceneCut reasons jointly over objects
and scene geometry and therefore produces a segmentation
of non-object surfaces such as ceilings and walls, as well
as supporting surfaces such as floor or tabletops. Unlike
methods [8]–[10] that deal with geometric segmentation and
object segmentation in isolation, our method simultaneously
segments objects and planes of a scene in an unified
formulation. SceneCut therefore ensures the segmentation of
geometric surfaces is consistent with the discovered objects.
It avoids typical segmentation errors and inconsistencies that
are often observed in other methods such as when pictures are
assigned to a wall, or mouse pads are assigned to a desk or
tabletop segment. Fig. 1 illustrates a typical cluttered indoor
scene and compares the segmentation produced by SceneCut
with RefineNet [1] and Mask-RCNN [2].
A key aspect of our approach is a novel unified energy
function that jointly quantifies geometric goodness-of-fit
and objectness measure. The function, however, is difficult
to optimize as its domain is over a continuous space of
plane parameters and a discrete collection of image regions.
For tractability, we utilize hierarchical segmentation trees

[11] to sample potential object and surface candidates. In
particular, the input image is converted to a region hierarchy,
where nodes of the tree represent meaningful regions —
potentially objects and surfaces. One interesting property
of the segmentation tree is that a cut through the tree will
result in a flat segmentation (i.e., one pixel belongs to a
unique region). The problem becomes finding a tree cut
that optimizes the geometric and object fitting function. As
the problem has a tree-structure, the solution can be found
efficiently and exactly using dynamic programming.
A vital ingredient in building a high-quality segmentation
tree is accurate boundaries. We make use of the boundary
map predicted by Convolutional Oriented Boundary (COB)
network of [12]. The COB network has shown its superior
performance over traditional boundary detection methods
(e.g., [13]) that use local features such as colors and depths.
Importantly, training such a boundary detection networks
does not rely on semantic class information, and thus the
model generalizes to unknown environments with unknown
objects, as we will show in Section IV.
We evaluate the effectiveness of our segmentation approach,
SceneCut, on the indoor scene segmentation task using
NYU [10] and TUM [14] datasets. Experimental results on
NYU dataset reveals that SceneCut greatly outperforms stateof-the-art image segmentation methods. In addition, using
the TUM dataset we show that SceneCut works well on
different environments without retraining or re-tuning hyperparameters.
Our paper is structured as follows. We begin by reviewing
related work in Section II. In Section III we describe
the details of SceneCut. In Section IV we investigate the
performance of the SceneCut method. Finally, we conclude
and highlight some limitations in Section V.
II. R ELATED W ORK
Simultaneous Localization and Mapping (SLAM) is a
key component in many service robots that need to map
new unstructured environments and navigate within them
automatically. Advanced robots, however, require more sophisticated interactions with the environments, thus need to
understand the world around them in terms of semantic entities. Noticeably, Salas-Moreno et. al. introduced SLAM++ [6]
which simultaneously performs object recognition and camera
tracking. Semantically-enriched maps with object entities are
then constructed. In [15], McCormac et. al. proposed SemanticFusion which labels 3D reconstructed maps with semantic
object classes. This method fuses semantic labels in realtime along with scene reconstruction. Recently, Sünderhauf
et. al. [7] developed a method for object-oriented semantic
mapping, where individual object instances are key entities
in the maps. Nevertheless, these methods require either 3D
object models available prior to execution (e.g., in [6]) or
object detectors trained previously (e.g., in [7]). These systems
face difficulties when deployed in unknown environments.
One possible solution is to jointly perform object discovery,
camera tracking and mapping, such as the method in [16],
which relies on the unsupervised segmentation method [8] to

segment planes and objects from incoming RGB-D frames.
However, the segmentation method [8] performs badly on
general messy and cluttered indoor environments, as shown
in Section IV. Inspired by this research trend, in this work,
we aim at providing a robust joint geometric and object
segmentation method, which will be useful for existing object
SLAM or semantic mapping systems such as [7], [16].
Our work is closely related to unsupervised segmentation
methods which are commonly used used for object discovery.
Felzenszwalb and Huttenlocher [17] proposed a graph-based
approach to color image segmentation. A graph structure is
imposed on the image, and edge weights are computed based
on color differences. A greedy merging algorithm is applied
to the graph to obtain final segmentations. Karpathy et. al.
[18] used this method to discover objects in 3D, where the
graph is constructed over the point cloud, and edge weights
are computed using colors and normals. Similarly, Trevor
et. al. [8] defined binary edge weights for plane extraction
using RGB-D images. To discover objects, the method in
[8] first removes points belonging to detected planes, then
segments remaining points into objects using Euclidean
clustering. Pham et. al. [9] also used planes to segment
objects in 3D. Unlike these methods which decouple plane
and object segmentation, SceneCut jointly segments surfaces
and discovers object instances in an unified formulation.
One of the limitations of unsupervised segmentation methods is that these methods often require tuning a parameter(s) to
control the number of regions in the segmentations. In practice,
there is no single threshold or parameter that works well for
all objects and different images. Hierarchical segmentation
methods [11], [19] tackle this issue by output, instead,
multiple segmentations nested in a tree so that hopefully
objects are correctly segmented at some level of the tree.
Finding correct levels for different objects is, however, nontrivial. The method in [20] attempted to align the scales so
that all objects are at the same scale/level. SceneCut greatly
makes use of hierarchical segmentation trees not only for
finding correct object regions, but also for sampling the space
of geometric plane models. Instead of realigning the object
scales as in [20], we find the best object regions and surface
models by searching for the optimal tree cut that maximizes
a joint geometric and object fitting function.
Our work also bears a resemblance to the works of Gould et.
at. [21] and Silberman et. al [10]. In [21], the authors proposed
to a learning based model to decompose an outdoor scene
into a ground plane and semantic-class regions. The method
in [10] tried to parse an indoor scene into semantic and
surface regions, as well as estimate support relations. Similar
to SceneCut, the inference also makes use of hierarchical
segmentation tree, but assumes that 3D scene structure has
been accurately estimated using RANSAC. Furthermore, the
inference is based on a linear programming formulation which
is expensive and does not scale well with the nodes of the
tree. In contrast, our geometric and object fitting function
can be optimized efficiently using dynamic programing.
Importantly, these two methods [10], [21] face difficulties
when encountering objects that were not seen during training.

Convolutional Oriented
Boundaries Network

Optimal Tree Cut

Output
geometry and object
segmentation

Hierarchical segmentation tree
Input RGB-D image

Fig. 2: An overview of SceneCut. The input image is first passed to a deep convolutional neural network [12] to predict a boundary map,
which is then used to construct a hierarchical segmentation tree. Nodes of the tree serve as potential object and surface candidates. The
optimal tree cut is optimized to output the final geometric and object segmentation.

III. S CENE C UT
A. Overview
Given an input image, our goal is to decompose the
image into meaningful non-overlapping regions, where each
region represents either an object (e.g., chair, computer) or
a planar surface (e.g., wall, floor, tabletop) instance. Figure
2 demonstrates the overview of our approach. The input
image is first passed to a deep convolutional neural network
to predict a boundary map, which is used to construct a
hierarchical segmentation tree. We then search for an optimal
tree cut that maximizes a geometric and object fitting function.
The found optimal tree cut yields the final segmentation.
B. Joint Geometry and Object Segmentation
We denote the input RGBD image as I = {xi }. The
task is to partition the image I into non-overlap regions
S = {S1 , S2 , . . . , SN } such that each region represents an
object or a planar surface instance. We also estimate the
corresponding parameters Θ = {θ1 , θ2 , . . . , θN }, where each
θi is used to describe the region properties. For instance, θ
can be either plane parameters of the surface or an integer
indicating object identity (not object class). θi can be seen
as the label of region Si . Note that multiple different regions
in S might have the same label. This property is useful
for handling occlusion — for instance the ground floor is
fragmented into several disconnected visible regions in the
image due to objects such as chairs, tables resting on it.
To find the optimal segmentation S and its corresponding
parameters Θ, we need a way to assess its quality. Assume
that a function ψ(Si , θi ) : [Si , θi ] → [0, +∞] is provided to
measure the score (energy) of the region Si with parameters
θi being an object or a surface. The quality of {S, Θ} can
be computed as:
E(S, Θ) =

N
X

ψ(Si , θi ).

(1)

i=1

The tricky part is to design the function ψ(Si , θi ) that properly
calibrates geometric goodness-of-fit and objectness quantity.
In this work, we use a “sum of likelihood” function:
(P
x∈Si g(x, θi ) if θi is plane
ψ(Si , θi ) =
(2)
|Si |f (Si )
otherwise,

where f (Si ) : Si → [0, 1] measures the likelihood of region
Si being an object, g(x, θi ) : x, θi → [0, 1] computes the
likelihood of assigning pixel x to the plane model θi , |Si | is
the size (number of pixels) of the region Si .
The problem of surface and object segmentation becomes
finding the optimal segmentation S ∗ and parameters Θ∗ such
that
{S ∗ , Θ∗ } = argmax E(S, Θ).

(3)

S,Θ

Nevertheless, solving the maximization (3) is non-trivial due
to the complex search spaces of S and Θ, because Θ is a
mix space of objects and continuous plane parameters. Next,
we will show how to approximate the optimization (3) using
hierarchical segmentation tree and tree cuts.
C. Hierarchical Segmentation Tree

(a)

(b)

(c)

Fig. 3: An example of a segmentation tree (a) and different cuts.
(b) a simple horizontal cut. (c) an optimal cut.

Hierarchical segmentation represents an image at multiple
scales, where pixels with similar characteristics (i.e., colors,
textures) are grouped into region hierarchies. By construction,
nodes of the tree present meaningful things appearing in
the image. In this work, we leverage region hierarchies to
approximate the space of scene surfaces and object candidates.
A common way to construct a segmentation tree for an
input image is based on a boundary map estimated from
the image. The boundary map can be converted to a region
hierarchy using the Ultrametric Contour Map (UCM) [19].
Traditionally, the boundaries is computed using local features
such as colors and depths. Recent works [12], [22] show that
high-quality boundary maps can be predicted directly from
images using deep convolutional networks. In this work, we
resort to the COB network [12] to predict boundary maps.
As we are interested in indoor scenes with RGBD data, we
select the COB model previously trained on NYU dataset

[10]. We will show that this model still works well on many
other datasets different from NYU, such as TUM [14].
Let T be a hierarchical segmentation tree obtained from
the image I, comprising K nodes, i.e., T = {R1 , . . . , RK }.
Each node represents a single image region, and the root
node is the whole image. Moreover, each node has one and
only one parent (other than the root) and has at least one
child (other than the leaves). We consider each image region
(node of the tree) as an object candidate. We also fit a plane
model to 3D points of each region, resulting in a set the
potential plane models P = {p1 , p2 , . . . , pK }.1
One interesting property of the segmentation tree is that a
“tree cut” will result in a flat image segmentation S. Figure 3
illustrates a segmentation tree and different tree cuts. Let C
be the space of all possible tree cuts, it is clear that S ∈ C. In
the next section, we will show to maximize the segmentation
function (1) over the spaces of C and P exactly using dynamic
programming.
D. Optimal Tree Cut using Dynamic Programming
We denote Y = {0, 1, 2, . . . , K, p1 , p2 , . . . , pK } as a joint
set of object and planar surface candidates. Finding the best
tree cut can be reformulated as optimizing label variables L =
[l1 , l2 , . . . , lK ], where li takes value in Y. More specifically,
for each node Ri of the tree T , we introduce a label li —
if li is zero, node Ri is not selected, otherwise node Ri is
included in the solution S. Furthermore, li will take values pi
if region Ri is a planar surface. Now the objective function
(1) becomes:
X
E(T , L) =
ψ(Ri , li ),
(4)

where

Ri ∈T

P

 x∈Ri g(x, li ) if li ∈ P
ψ(Ri , li ) = |Ri |f (Ri )
if li = i


0
otherwise.

(5)

Notice that multiple surface regions can take the same
label (plane parameters) in P, therefore disconnected surface
regions can be merged. However, the current formulation is
unable to merge isolated regions belonging to the same object
(which can arise due to occlusion).
Given L, we can obtain S = {Ri | li 6= 0}. However,
not any labeling L results in a valid tree cut. In order for
S (obtained from L) to be a valid tree cut, if region Ri has
label li 6= 0, its children and parent nodes will be assigned to
label 0. Assume the root of the tree is R1 , the maximum of
the function (4) with the “tree cut” constraint can be obtained
recursively as follow:
max E(T , L) = max(ψ(R1 , l1 ), E(TR1 , LTR1 )),

(6)

where TR1 is the subtree rooted at node R1 , and LTR1 is the
corresponding labels for the subtree nodes.
Thanks to the tree-structure property, we can find the
optimal L∗ using dynamic programming as follow. The
1 In

practice, plane models with less than 5000 inliers are discarded.

optimization includes one forward and one backward pass.
The forward pass, proceeding from the bottom to the top
of the tree, computes energies at each node and its subtree.
The backward pass will compare, from the top to bottom,
the energies of the current node and its subtree to obtain the
optimal labels. This dynamic programming method is highly
efficient with complexity O(K), where K is the total number
of nodes.
E. Objectness measure
For each region of the segmentation tree, we compute
its objectness f (.) — the likelihood of being an object. A
common way to measure objectness is to analyze object shape
and appearances [18]. In this work, we use simple compactness (in 3D) and convexity (in 2D) measures, although more
complicated learning based methods such as [23] can be
applied. Compactness prefers regions containing structure in
compact volume, which can be computed by the ratio the
total surface area of the region in 3D to the surface area
of its smallest bounding sphere. The convexity is the area
fraction of the region as compared to its convex hull in the
image space. Given an image region R, let A3d and A2d be
the surface areas of the region in 2D and 3D respectively,
Asph be the surface area of the smallest sphere, Acon be the
area of the convex hull, the objectness f can be computed
as:
A2d
A3d
+ (1 − λ)
(7)
f (R) = λ
Asph
Acon
where λ is a weighting parameter. We set λ = 0.5 in all our
experiments.
F. Goodness-of-fit
The likelihood of a pixel belonging to a planar surface is
computed using several criteria. First, its 3d location must
lie close to the surface. Second, its normal should agree with
the normal of the surface. Lastly, the pixel color should not
be much different from the mean color of the surface. The
last criterion is very useful to deal with flat objects, e.g.,
pictures attached to the wall. Given a pixel x with color xc ,
3d location xl (in homogeneous coordinate) and normal xn ,
a plane model with parameters p, normal pn and mean color
pc , the likelihood of assigning x to plane p is:
g(x, p) = exp(

−(xl · p)2
−(xn · pn )2
)
×
exp(
)
σd2
σn2
−kxc − pc k2
× exp(
),
σc2

(8)

where (xl · p)2 is the squared distance from 3D point xl
to plane p, (xn · pn )2 is the squared dot product between
normal vectors, σd , σc and σn are constant parameters. We
empirically set σd = 0.01, σn = 0.25 and σc = 10.
IV. E XPERIMENTS
A. Datasets and Baselines
We use indoor RGB-D NYU [10] and TUM [14] datasets to
evaluate the effectiveness of SceneCut. The NYU dataset con-

RefineNet [1]
Mask-RCNN [2]
Method of [8]
SceneCut

Fig. 4: SceneCut v.s. others using images from NYU dataset. Best viewed on-screen.

tains 654 RGB-D testing images with ground-truth semanticobject annotations, which is widely used for evaluating
indoor semantic instance segmentation. However, as our
goal is class-independent object segmentation, we do not
use semantic classes for evaluation. The TUM dataset, on
the other hand, has no ground-truth object annotations as
it was created for evaluating camera tracking and scene
reconstruction. Nevertheless, this dataset can be viewed as
unknown environments with unknown objects, which is useful
for testing the object discovery capability. We select three
sequences from the TUM repository, namely freiburg1 desk2,
freiburg2 desk, and fr3 household, and randomly sample 600
frames for testing. Since there is no ground-truth annotations,
we only provide qualitative evaluations on this dataset.
Since our method SceneCut aims at reasoning about
surfaces and discovering object instances in indoor scenes,
our closest competitors include: 1) the method of [8] which
first extracts planes using connected components, and then
discovers objects using Euclidean clustering, 2) the method
of Silberman et al. [10] which first infers the overall
3D scene structure (in terms of planes) using RANSAC,
then separates objects and reasons support relations via an
integer programming formulation. We also compare SceneCut
against popular unsupervised image segmentation approaches
including LCCP [24], graph-based segmentation [17]. Unlike
SceneCut, these unsupervised segmentation methods only
decompose images into meaningful regions without knowing
their object or non-object labels.
To demonstrate the deficiencies of the existing object detection and semantic segmentation approaches in recognizing

previously unseen objects and object instance separation, we
additionally test the state-of-the-art instance segmentation
Mask-RCNN2 [2] and semantic segmentation RefineNet3 [1]
on the considered datasets.
The NYU dataset does not provide accurate ground-truth
information for geometric surfaces although the dataset has
annotated ground-floors, walls and ceilings, but other objects
such as table tops or cabinet tops can be considered as planar
surfaces too. Therefore, we evaluate the performance using
class-independent segmentation accuracy without considering
geometric or object labels. We use two popular measures:
segmentation covering [11], [26] and F-region score [26].
B. Results
Table I reports the segmentation results of all the considered
methods using NYU dataset. It is clear that SceneCut
significantly outperforms its competitors that solve geometric
and object segmentation independently. SceneCut is also much
more accurate than the competing unsupervised segmentation
methods [17], [24]. The superiority of SceneCut over these
methods is due to two reasons: 1) joint reasoning about objects
and surfaces using optimal tree cut and 2) the use of deep
learning to predict high-quality boundaries. To demonstrate
that the improvement is not from the use of deep boundary
prediction only, we evaluate the performance of the greedy
horizontal cut (see Figure 3) with the best dataset scale
2 We implemented Mask-RCNN by ourself and trained on Pascal VOC
2012 dataset [25].
3 We took the RefineNet model trained on NYU dataset, downloaded from
https://github.com/guosheng/refinenet.

Method

Use depth

Learning

Surface

Object

Segmentation Covering (%)

F-Region (%)

Graph-based Segmentation [17]
LCCP [24]
Connected Components [8]
Silberman et al. [10]
SceneCut (proposed approach)

7
3
3
3
3

7
7
7
3
3

7
7
3
3
3

3
3
3
3
3

44.32
57.60
44.61
61.10
71.26

60.80
56.10
71.79

BDS horizontal cut (0.3)
RefineNet [1]
Mask-RCNN [2]

3
7
7

3
3
3

7
7
7

3
3
3

69.80
53.08
27.95

71.27
60.46
29.61

TABLE I: Quantitative comparison results on 654 NYU RGB-D testing images. Note that the results of RefineNet and Mask-RCNN are
just for reference as these methods do not aim at discovering unknown objects, scene geometry segmentation (i.e., Mask-RCNN) and
object instance segmentation (i.e., RefineNet). The results of LCCP and Silberman et al. [10] are taken from [8] and [10] respectively.

(BDS) that is computed using ground truth information.
Unsurprisingly, the optimal tree cut is more accurate than the
best horizontal cut. Note that the best estimated scale of the
NYU dataset might not translate well to different unknown
environments.
The result in Table I also shows low segmentation accuracies of Mask-RCNN and RefineNet as these methods can not
ether detect/segment unknown objects and scene surfaces or
separate object instances. Figure 4 visually demonstrates this
result. Observe that Mask-RCNN fails to detect novel objects
such as trash bin, cabinet or toilet. Mask-RCNN also does
not detect surfaces such as floor, ceiling. On the other hand,
RefineNet is unable to distinguish different objects of the
same class (e.g., trash bins, bottles). In contrast, SceneCut
achieves good surface and object segmentation. In comparison
with the method [8], which segments surfaces and objects
independently, SceneCut also shows clear improvement. For
example, [8] merges wall and picture into a surface, touching
objects (e.g., table and trash bins) are also not segmented
correctly.
Figure 5 visualizes the segmentation results of SceneCut
and its competitors using images from the TUM dataset,
where none of the methods was trained on. It can be seen
that SceneCut still achieves good results and performs better
than other methods on these unknown environments.
C. Runtime Analysis
Except for the boundary prediction and segmentation tree
construction making use of C++ and GPU implementation4 ,
other steps of SceneCut including computing objectness, plane
parameters and tree cut are run using unoptimized Matlab
implementation. Table II presents average runtimes for each
step. There are several ways to reduce the computation cost.
For instance, in the current implementation, features of image
regions (e.g., region areas) are computed independently. This
computation will be much cheaper if properly exploiting the
segmentation tree, i.e., propagating the features from leaf
nodes to parent nodes. Similar strategy can be applied for
fitting planes to depth regions. Moreover, a proper tree data
structure in C++ can significantly reduce the computation
cost of finding optimal tree cuts.
4 https://github.com/kmaninis/COB

Boundary

Seg. Tree

Objectness

Plane

Tree Cut

0.28

0.51

1.75

1.24

1.26

TABLE II: Average running times for each step of SceneCut (in
second).

V. CONCLUSIONS
We have presented SceneCut that has a capability to
simultaneously infer scene geometry and discover objects
of indoor scenes. Such a capability is useful for autonomous
robots working in open-set conditions [4], where the robots
will unavoidably encounter novel objects that were not part
of the training dataset.
SceneCut automatically estimates supporting planar surfaces and discovering objects using an unified formulation. In
particular, we proposed an unified energy function that jointly
encodes geometric fitting and objectness measures. We show
how efficiently maximize the energy by utilizing hierarchical
segmentation trees and tree cuts. Moreover, the method enjoys
the power of deep learning applied for predicting object
boundaries directly from input images. High-quality boundary
prediction has led to highly accurate geometric and object
segmentation. The experimental results demonstrate that our
proposed approach greatly outperforms many existing RGB-D
scene segmentation methods.
A limitation of the current approach is its inability to correctly segment occluded object instances that are fragmented
into disconnected regions in the image. Also the hierarchical
segmentation trees sometimes fail to merge regions belonging
to the same object instance due to badly estimated boundaries.
One way to work around is to use higher-order parametric
models such as ellipsoid or NURB to model objects rather
simple image regions considered in the current approach.
R EFERENCES
[1] G. Lin, A. Milan, C. Shen, and I. Reid, “RefineNet: Multi-path
refinement networks for high-resolution semantic segmentation,” in
CVPR, Jul. 2017.
[2] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” arXiv
preprint arXiv:1703.06870, 2017.
[3] J. Redmon and A. Farhadi, “YOLO9000: better, faster, stronger,” CoRR,
vol. abs/1612.08242, 2016.
[4] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult,
“Toward open set recognition,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 35, no. 7, pp. 1757–1772, 2013.

RefineNet [1]
Mask-RCNN [2]
Method of [8]
SceneCut

Fig. 5: SceneCut v.s. others using images from TUM dataset. Best viewed on-screen.
[5] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira,
I. Reid, and J. J. Leonard, “Past, present, and future of simultaneous
localization and mapping: Toward the robust-perception age,” IEEE
Transactions on Robotics, vol. 32, no. 6, pp. 1309–1332, 2016.
[6] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. J. Kelly, and
A. J. Davison, “Slam++: Simultaneous localisation and mapping at the
level of objects,” in 2013 IEEE Conference on Computer Vision and
Pattern Recognition, June 2013, pp. 1352–1359.
[7] N. Sünderhauf, T.-T. Pham, Y. Latif, M. Milford, and I. D. Reid,
“Meaningful maps with object-oriented semantic mapping,” in IEEE/RSJ
International Conference on Intelligent Robots and Systems, 2017.
[8] A. J. B. Trevor, S. Gedikli, R. B. Rusu, and H. I. Christensen, “Efficient
organized point cloud segmentation with connected components,” 2013.
[9] T. T. Pham, M. Eich, I. D. Reid, and G. Wyeth, “Geometrically
consistent plane extraction for dense indoor 3d maps segmentation,” in
IEEE/RSJ International Conference on Intelligent Robots and Systems,
2016, pp. 4199–4204.
[10] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation
and support inference from rgbd images,” in Proceedings of the
12th European Conference on Computer Vision - Volume Part V, ser.
ECCV’12. Berlin, Heidelberg: Springer-Verlag, 2012, pp. 746–760.
[11] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection
and hierarchical image segmentation,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, May
2011.
[12] K. Maninis, J. Pont-Tuset, P. Arbeláez, and L. V. Gool, “Convolutional
oriented boundaries: From image segmentation to high-level tasks,”
IEEE Transactions on Pattern Analysis and Machine Intelligence
(TPAMI), 2017.
[13] S. Gupta, P. Arbelez, and J. Malik, “Perceptual organization and
recognition of indoor scenes from rgb-d images,” in 2013 IEEE
Conference on Computer Vision and Pattern Recognition, June 2013,
pp. 564–571.
[14] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A
benchmark for the evaluation of rgb-d slam systems,” in Proc. of the
International Conference on Intelligent Robot Systems (IROS), Oct.
2012.
[15] J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “Semanticfusion: Dense 3d semantic mapping with convolutional neural networks,”

[16]

[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]

[26]

in 2017 IEEE International Conference on Robotics and Automation
(ICRA), May 2017, pp. 4628–4635.
S. Choudhary, A. J. B. Trevor, H. I. Christensen, and F. Dellaert,
“Slam with object discovery, modeling and mapping,” 2014 IEEE/RSJ
International Conference on Intelligent Robots and Systems, pp. 1018–
1025, 2014.
P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient graph-based image
segmentation,” International Journal of Computer Vision, vol. 59, no. 2,
pp. 167–181, Sep 2004.
A. Karpathy, S. Miller, and L. Fei-Fei, “Object discovery in 3d scenes
via shape analysis,” in 2013 IEEE International Conference on Robotics
and Automation, May 2013, pp. 2088–2095.
P. Arbelaez, “Boundary extraction in natural images using ultrametric
contour maps,” in 2006 Conference on Computer Vision and Pattern
Recognition Workshop (CVPRW’06), June 2006, pp. 182–182.
Y. Chen, D. Dai, J. Pont-Tuset, and L. Van Gool, “Scale-aware
alignment of hierarchical image segmentation,” in Computer Vision
and Pattern Recognition (CVPR), 2016.
S. Gould, R. Fulton, and D. Koller, “Decomposing a scene into
geometric and semantically consistent regions.” in ICCV. IEEE
Computer Society, 2009, pp. 1–8.
S. ”Xie and Z. Tu, “Holistically-nested edge detection,” in Proceedings
of IEEE International Conference on Computer Vision, 2015.
J. M. Weicheng Kuo, Bharath Hariharan, “Deepbox:learning objectness
with convolutional networks,” in International Conference on Computer
Vision (ICCV), 2015.
S. Stein, M. Schoeler, J. Papon, and F. Worgotter, “Object partitioning
using local convexity,” in Computer Vision and Pattern Recognition
(CVPR), 2014 IEEE Conference on, June 2014, pp. 304–311.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
Challenge
2012
(VOC2012)
Results,”
http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
J. Pont-Tuset and F. Marques, “Supervised evaluation of image
segmentation and object proposal techniques,” IEEE Transactions on
Pattern Analysis and Machine Intelligence (TPAMI), vol. 38, no. 7, pp.
1465–1478, 2016.

