IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

1

Hierarchical Scene Parsing by Weakly
Supervised Learning with Image Descriptions

arXiv:1709.09490v2 [] 27 Jan 2018

Ruimao Zhang, Liang Lin, Guangrun Wang, Meng Wang, and Wangmeng Zuo
Abstract—This paper investigates a fundamental problem of scene understanding: how to parse a scene image into a structured
configuration (i.e., a semantic object hierarchy with object interaction relations). We propose a deep architecture consisting of two
networks: i) a convolutional neural network (CNN) extracting the image representation for pixel-wise object labeling and ii) a recursive
neural network (RsNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative
annotations (e.g., manually labeled semantic maps and relations), we train our deep model in a weakly-supervised learning manner by
leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting
of nouns and verb phrases, and apply these tree structures to discover the configurations of the training images. Once these scene
configurations are determined, then the parameters of both the CNN and RsNN are updated accordingly by back propagation. The
entire model training is accomplished through an Expectation-Maximization method. Extensive experiments show that our model is
capable of producing meaningful scene configurations and achieving more favorable scene labeling results on two benchmarks (i.e.,
PASCAL VOC 2012 and SYSU-Scenes) compared with other state-of-the-art weakly-supervised deep learning methods. In particular,
SYSU-Scenes contains more than 5000 scene images with their semantic sentence descriptions, which is created by us for advancing
research on scene parsing.
Index Terms—Scene parsing, Deep learning, Cross-modal Learning, High-level understanding, Recursive structured prediction

F

1

I NTRODUCTION

Scene understanding started with the goal of creating systems that
can infer meaningful configurations (e.g., parts, objects and their
compositions with relations) from imagery like humans [1][2]. In
computer vision research, most of the scene understanding methods focus on semantic scene labeling / segmentation problems
(e.g., assigning semantic labels to each pixel) [3][4][5][6]. Yet
relatively few works attempt to explore how to automatically generate a structured and meaningful configuration of the input scene,
which is an essential task to human cognition [7]. In spite of some
acknowledged structured models beyond scene labeling, e.g., andor graph (AoG) [8], factor graph (FG) [9] and recursive neural
network (RsNN) [10], learning the hierarchical scene structure
remains a challenge due to the following difficulties.
•

•

•

•
•

The parsing configurations of nested hierarchical structure
in scene images are often ambiguous, e.g., a configuration
may have more than one parse. Moreover, making the
parsing result in accordance with human perception is also
intractable.
Training a scene parsing model usually relies on very
expensive manual annotations, e.g., labeling pixel-wise semantic maps, hierarchical representations and inter-object
relations.

R. Zhang, L. Lin and G. Wang are with the School of Data
and Computer Science, Sun Yat-sen University, Guangzhou, P.
R. China (E-mail: ruimao.zhang@ieee.org; linliang@ieee.org; wanggrun@mail2.sysu.edu.cn). Corresponding author is Liang Lin.
M. Wang is with the School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, P. R. China (E-mail:
eric.mengwang@gmail.com).
W. Zuo is with the School of Computer Science, Harbin Institute of
Technology, Harbin, P. R. China (E-mail: cswmzuo@gmail.com).

To address these above issues, we develop a novel deep neural
network architecture for hierarchical scene parsing. Fig. 1 shows
a parsing result generated by our framework, where a semantic
object hierarchy with object interaction relations is automatically
parsed from an input scene image. Our model is inspired by the
effectiveness of two widely successful deep learning techniques:
convolutional neural networks (CNN) [11][5] and recursive neural
network (RsNN) [10]. The former category of models is widely
applied for generating powerful feature representations in various
vision tasks such as image classification and object detection.
Meanwhile, the RsNN models (such as [10][6][12]) have been
demonstrated as an effective class of models for predicting hierarchical and compositional structures in image and natural language
understanding [13]. One important property of RsNN is the ability
to recursively learn the representations in a semantically and
structurally coherent way. In our deep CNN-RsNN architecture,
the CNN and RsNN models are collaboratively integrated for
accomplishing the scene parsing from complementary aspects. We
utilize the CNN to extract features from the input scene image and
generate the representations of semantic objects. Then, the RsNN
is sequentially stacked based on the CNN feature representations,
generating the structured configuration of the scene.
On the other hand, to avoid affording the elaborative annotations, we propose to train our CNN-RsNN model by leveraging the
image-level descriptive sentences. Our model training approach
is partially motivated but different from the recently proposed
methods for image-sentence embedding and mapping [14][15],
since we propose to transfer knowledge from sentence descriptions
to discover the scene configurations.
In the initial stage, we decompose each sentence into a semantic tree consisting of nouns and verb phrases with a standard
parser [16], WordNet [17] and a post-processing method. Then,
we develop an Expectation-Maximization-type learning method

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

in front of

car

r
ride

Image
person

b
bike

Structured Scene Configuration
Fig. 1. An example of structured scene parsing generated by our framework. An input scene image is automatically parsed into a structured
configuration that comprises hierarchical semantic objects (black labels)
and the interaction relations (red labels) of objects.

for model training based on these semantic trees and their associated scene images. Specifically, during the weakly-supervised
training, the semantic tree facilitators discover the latent scene
configuration in the two following aspects: 1) the objects (i.e.,
nouns) determine the object category labels existing in the scene,
and 2) the relations (i.e., verb phrases) among the entities help
produce the scene hierarchy and object interactions. Thus, the
learning algorithm iterates in three steps. (i) Based on the object
labels extracted from the sentence, it estimates an intermediate
label map by inferring the classification probability of each pixel.
Multi-scale information of the image is adopted to improve the
accuracy. (ii) With the label map, the model groups the pixels into
semantic objects and predicts the scene hierarchy and inter-object
relations through the RsNN. (iii) With the fixed scene labeling and
structure, it updates the parameters of the CNN and RsNN by back
propagation.
The main contributions of our work are summarized as follows. i) We present a novel CNN-RsNN framework for generating
meaningful and hierarchical scene representations, which helps
gain a deeper understanding of the objects in the scene compared
with traditional scene labeling. The integration of CNN and RsNN
models can be extended to other high-level computer vision tasks.
ii) We present a EM-type training method by leveraging descriptive sentences that associate with the training images. This method
is not only cost-effective but also beneficial to the introduction of
rich contexts and semantics. iii) The advantages of our method are
extensively evaluated under challenging scenarios. In particular,
on PASCAL VOC 2012, our generated semantic segmentations
are more favorable than those by other weakly-supervised scene
labeling methods. Moreover, we propose a dedicated dataset for
facilitating further research on scene parsing, which contains
more than 5000 scene images of 33 categories with elaborative
annotations for semantic object label maps, scene hierarchy and
inter-object relations.
The remainder of this paper is organized as follows. Section 2
provides a brief review of the related work. Then we introduce the
CNN-RsNN model in Section 3 and follow with the model training
algorithm in Section 4. The experimental results and comparisons
are presented in Section 5. Section 6 concludes the paper and
presents some outlook for future work.

2

2

R ELATED W ORK

Scene understanding has been approached through many recognition tasks such as image classification, object detection, and
semantic segmentation. In current research, a myriad of different
methods focus on what general scene type the image shows (classification) [18][19][20], what objects and their locations are in a
scene (semantic labeling or segmentation) [21][22][23][24]. These
methods, however, ignore or over simplified the compositional
representation of objects and fail to gain a deeper and structured
understanding on scene.
Meanwhile, as a higher-level task, structured scene parsing has
also attracted much attention. A pioneering work was proposed
by Tu et al. [25], in which they mainly focused on faces and
texture patterns by a Bayesian inference framework. In [1], Han
et al. proposed to hierarchically parse the indoor scene images
by developing a generative grammar model. An extended study
also explored the more complex outdoor environment in [26]. A
hierarchical model was proposed in [27] to represent the image
recursively by contextualized templates at multiple scales, and
rapid inference was realized based on dynamic programming.
Ahuja et al. [28] developed a connected segmentation tree for
object and scene parsing. Some other related works [29][30]
investigated the approaches for RGB-D scene understanding, and
achieved impressive results. Among these works, the hierarchical
space tiling (HST) proposed by Wang et al. [2], which was applied
to quantize the huge and continuous scene configuration space,
seemed to be the most related one to ours. It adopted the weakly
supervised learning associated the text (i.e. nouns and adjectives)
to optimize the structure of the parsing graph. But the authors
didn’t introduce the relations between objects into their method.
In terms of the model, HST used a quantized grammar, rather than
the neural networks which can adopt the transfer learning to obtain
better initialization for higher training efficiency.
With the resurgence of neural network models, the performances of scene understanding have been improved substantially.
The representative works, the fully convolutional network (FCN)
[5] and its extensions [31], have demonstrated effectiveness in
pixel-wise scene labeling. A recurrent neural network model was
proposed in [32], which improved the segmentation performance
by incorporating the mean-field approximate inference, and similar idea was also explored in [33]. For the problem of structured
scene parsing, recursive neural network (RsNN) was studied in
[10][12]. For example, Socher et al. [10] proposed to predict
hierarchical scene structures with a max-margin RsNN model.
Inspired by this work, Sharma et al. proposed the deep recursive
context propagation network (RCPN) in [6] and [12]. This deep
feed-forward neural network utilizes the contextual information
from the entire image to update the feature representation of
each superpixel to achieve better classification performance. The
differences between these existing RsNN-based parsing models
and our model are three folds. First, they mainly focused on
parsing the semantic entities (e.g., buildings, bikes, trees), while
the scene configurations generated by our method include not only
the objects but also the interaction relations of objects. Second,
we introduce a novel objective function to discover the scene
structure. Third, we incorporate convolutional feature learning into
our deep model for joint optimization.
Most of the existing scene labeling / parsing models are
studied in the context of supervised learning, and they rely on
expensive annotations. To overcome this issue, one can develop

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

3

Convolutional Neural Network

Recursive Neural Network
v1

person

car

bike

background

Fsem x1

person

r
ride

Fcom

…

…

…

…

…

…

v2

…

x12

Fsem x2

Fcom

bike

…

…

in front of

…

car

v3

Multiscale

Fsem

x3

Fig. 2. The proposed CNN-RsNN architecture for structured scene parsing. The input image is directly fed into the CNN to produce score feature
representation of each pixel and map of each semantic category. Then the model applies score maps to classify the pixels, and groups pixels with
same labels to obtain feature representation v of objects. After that v is fed into the RsNN, it is first mapped onto a transition space and then is used
to predict the tree structure and relations between objects. x denotes the mapped feature.

alternative methods that train the models from weakly annotated
training data, e.g., image-level tags and contexts [34][35][36][37].
Among these methods, the one that inspires us is [36], which
adopted an EM learning algorithm for training the model with
image-level semantic labels. This algorithm alternated between
predicting the latent pixel labels subject to the weak annotation
constraints and optimizing the neural network parameters. Different from this method, our model applies the sentence description to
label the salient semantic object in the image. By employing such
knowledge transfer, the model can deal with object labeling and
relation prediction simultaneously according to human perception.

3

CNN-R S NN A RCHITECTURE

3.1 CNN Model
The CNN model is designed to accomplish two tasks: semantic
labeling and generating feature representations for objects. For
semantic labeling, we adopt the fully convolutional network with
parameters WC to yield K + 1 score maps {s0 , ..., sk , ..., sK },
corresponding to one extra background category and K object
categories. Following the holistically-nested architecture in [39]
we adopt E = 3 multi-scale prediction streams, and each stream
is associated with K + 1 score maps with the specific scale. Let
st,e
j indicate the score value at pixel j in the t-th map of e-th scale.
t,e
We normalize sj in the e-th stream using softmax to obtain the
corresponding classification score:

σe (st,e
j )

exp(st,e
j )

= PK

k=0

This work aims to jointly solve three tasks: semantic labeling,
scene structure generation, and the inter-object relation prediction.
To achieve these goals, we propose a novel deep CNN-RsNN
architecture. The CNN model is introduced to perform semantic
segmentation by assigning an entity label (i.e. object category)
to each pixel, and the RsNN model is introduced to discover
hierarchical structure and interaction relations among entities.
Fig. 2 illustrates the proposed CNN-RsNN architecture for
structured scene parsing. First, the input image I is directly fed
into revised VGG-16 network [38] to produce different levels
of feature maps. According to these feature maps, multi-scale
prediction streams are combined to produce final score maps
S = {s0 , ..., sk , ..., sK } for object categories. Based on the
softmax normalization of score maps, the j -th pixel is assigned
with an object label cj . We further group the pixels with the
same label into an object, and obtain the feature representations
of objects. By feeding these feature representations of objects to
the RsNN, a greedy aggregation procedure is implemented for
constructing the parsing tree PI . In each recursive iteration, two
input objects (denoted by the child nodes) are merged into a
higher-level object (denoted by the parent node), and generated
root note represents the whole scene. Different from the RsNN
architecture in [10][12], our model also predicts the relation
between two objects when they are combined into a higher-level
object. Please refer to Fig. 2 for more information about the
proposed architecture. In the following, we discuss the CNN and
RsNN models in details.

exp(sk,e
j )

(1)

Then the final classification score σf (stj ) is further calculated by
P
t,e
σf (stj ) = E
e=1 αe σe (sj ), where αe > 0 is the fusion weight
PE
for the e-th stream, and e=1 αe = 1. The learning of this fusion
weight is equivalent to training 1 × 1 convolutional filters on the
concatenated score maps from all multi-scale streams. σf (stj )
denotes the probability
of j -th pixel belonging to t-th object
PK
category with t=1 σf (stj ) = 1. The set {cj }M
j=1 denotes the
predicted labels of pixels in the image I, where cj ∈ {0, ..., K}
and M is the number of pixels of image I. With σf (stj ), the label
of the j -th pixel can be predicted by:

cj = arg max σf (stj )
t

(2)

To generate feature representation for each entity category, we
group the pixels with the same label into one semantic category.
Considering that the pixel numbers vary with the semantic
entity categories, the pooling operation is generally required to
obtain fixed-length representation for any object category. Conventional sum-pooling treats feature representation from different
pixels equally, while max-pooling only considers the most representative one and ignores the contribution of the other. For the
tradeoff between sum-pooling and max-pooling, we use Log-SumExp (LSE), a convex approximation of the max function, as the
pooling operator to fuse the features of pixels,


X
1
exp(πv̄j )
(3)
vk = log 
π
c =k
j

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Wcat
…

Network Annotations. Following [10] and [12], object feature
vk produced by CNN is first mapped onto a transition space by the
Transition Mapper, which is a one-layer fully-connected neural
network.
xk = Ftran (vk ; Wtran )
(4)

Wscore
gkl

Wint
bkl

…

…

vk

xkl

Wcom

xk

…

Wtran

…

Wtran
…

4

…

xl
vl

where xk is the mapped feature, Ftran is the network transformation and Wtran indicates the network parameters. Then the
mapped features of two child nodes are fed into the Combiner
sub-network to generate the feature representation of the parent
node.

xkl = Fcom ([xk , xl ]; Wcom )
Fig. 3. An illustration of first layer of proposed recursive neural network
which is replicated for each pair of input feature representations. vk and
vl indicate the input feature vectors of two objects. xk and xl denote
the transition features mapped by one-layer fully-connected neural network. The feature representation after the merging operation is denoted
by xkl . Wtran , Wcom ,Wint ,Wcat and Wscore are parameters of
proposed RsNN model. This network is different to the RsNN model
proposed in [10] which only predicts a score for being a correct merging
decision. Our model can also be used to predict the interaction relation
between the merged objects.

where vk denotes the feature representation of the k -th entity
category, v̄j denotes the feature representation of the j -th pixel
by concatenating all feature maps at the layer before softmax at
position j into a vector, and π is a hyper-parameter to control
smootheness. One can see that LSE with π = 1 can serve as
convex and differentiable approximation of max-pooling [40].
While LSE with π → 0 degenerates to sum-pooling.
3.2

RsNN Model

With the feature representations of object categories produced by
CNN, the RsNN model is designed to generate the image parsing
tree for predicting hierarchical structure and interaction relations.
The inputs to scene configuration generation are a set Ψ of nodes,
where each node vk ∈ Ψ denotes the feature representation of an
object category. As illustrated in Fig. 3, the RsNN model takes
two nodes vk and vl and their contextual information as the
inputs. The output of RsNN includes three variables: (i) a single
real value hkl to denote the confidence score of merging vk and
vl , (ii) a relation probability vector ykl for predicting relation
label between the two nodes, and (iii) a feature vector xkl as
the combined representation. In each recursion step, the algorithm
considers all pairs of nodes, and choose the pair (e.g., vk and vl )
with the highest score to merge. After the merging, we add xkl and
remove vk and vl from Ψ. By this way, the nodes are recursively
combined to generate the hierarchical scene structure until all the
object categories in an image are combined into a root node.
Fig. 3 illustrates the process of RsNN in merging two nodes
vk and vl . In general, the RsNN model is composed of five
subnetworks: (i) transition mapper, (ii) combiner, (iii) interpreter,
(iv) categorizer, and (v) scorer. The transition mapper is a onelayer fully-connected neural network to generate xk and xl from
vk and vl . Based on xk and xl , the combiner is used to obtain
the feature representation xkl . Then, both xkl and their contextual
information bkl are considered in the interpreter to produce the
enhanced feature representation gkl . Finally, the categorizer and
scorer are used to predict the relation label and confidence score
for merging vk and vl . In the following, we further present more
detailed explanation on each subnetwork.

(5)

where Fcom is the network transformation and Wcom denotes the
corresponding parameters. Note that the parent node feature has
the same dimensionality as the child node feature, allowing the
procedure can be applied recursively.
Interpreter is the neural network that interprets the relation of
two nodes in the parsing tree. We note that the use of pooling operation in Eqn. (3) will cause the losing of spatial information which
is helpful to structure and relation prediction. As a remedy, we
design the context features to involve spatial context. Intuitively,
the interpreter network attempts to integrate the feature of two
nodes and their contextual information to represent the interaction
relation of two entities,

gkl = Fint ([xkl , bkl ]; Wint )

(6)

where Fint and Wint indicate the network and layer weights
respectively. bkl denotes the contextual information as follows,

b = [bang , bdis , bscal ]

(7)

where bang and bdis reflect the spatial relation between two
semantic entities, while bscal is employed to imply area relation of
semantic entities. As illustrated in Fig. 4, bang denotes the cosine
value of angle θ between the center of two semantic entities. bdis
indicates the distance γ of two centers (i.e. α1 and α2 ). bscal is the
area rate of such two entities, where bscal = β1 /β2 . In practice,
we normalize all of contextual information into a range of [−1, 1].
Categorizer sub-network determines the relation of two
merged nodes. Categorizer is a softmax classifier that takes relation feature gkl as input, and predicts the relation label ykl ,

ykl = sof tmax(Fcat (gkl ; Wcat ))

(8)

where ykl is the predicted relation probability vector, Fcat denotes the network transformation and Wcat denotes the network
parameters.
Scorer sub-network measures the confidence of a merging
operation between two nodes. It takes the enhanced feature gkl
as input and outputs a single real value hkl .

hkl = Fscore (gkl ; Wscore )
1
qkl =
1 + exp(−hkl )

(9)

where Fscore denotes the network transformation and Wscore
denotes the network parameters. qkl indicates the merging score
of node {kl}. Note such score is important to the configuration
discovery and is used to optimize the recursive structure in the
training phase, as described in Sec.4.2.2.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
motorcycle
person

semantic labelling and centers
contextual information

distance

relative angle

area ratio

Fig. 4. Incorporating the contextual representation into RsNN forward
process. The upper row shows the input image and the labeling results
of two entities, i.e, motorcycle and person. The center of each entity is
also given, i.e. α1 and α2 . Based on the centers and labeling results,
the bottom row illustrates three spatial relations, i.e., distance γ , relative
angle θ, and area ratio β1 /β2 , to characterize the contextual information
between the two entities.

4

M ODEL T RAINING

Fully supervised training of our CNN-RsNN model requires
expensive manual annotations on pixel-level semantic maps, interobject relations, and hierarchical structure configuration. To reduce the burden on annotations, we present a weakly-supervised
learning method to train our CNN-RsNN by leveraging a much
cheaper form of annotations, i.e., image-level sentence description.
To achieve this goal, the descriptive sentence is first converted to
the semantic tree to provide weak annotation information. Then
we formulate the overall loss function for structured scene parsing
based on the parsing results and the semantic trees. Finally, an
Expectation-Maximization (EM) algorithm is developed to train
CNN-RsNN by alternatively updating structure configuration and
network parameters. In the E-step, guided by the sentence description, we update scene configurations (i.e., intermediate label map
b , scene hierarchy and inter-object relations) together with the
C
intermediate CNN and RsNN losses. In the M-step, the model
parameters are updated via back-propagation by minimizing the
intermediate CNN and RsNN losses.
4.1

Sentence Preprocessing

For guiding semantic labeling and scene configuration, we convert
each sentence into a semantic tree by using some common techniques in natural language processing. As shown in the bottom
of Fig. 6, a semantic tree T only includes both entity labels
(i.e. nouns) and their interaction relations (i.e., verb/ prepositional
phrases). Therefore, in sentence preprocessing, we first generate
the constituency tree from the descriptive sentence, and then
remove the irrelevant leaf nodes and recognize the entities and
relations to construct the semantic tree.
The conversion process generally involves four steps. In the
first step, we adopt the Stanford Parser [16] to generate the constituency tree (i.e. the tree in the top of Fig. 6) from the descriptive
sentence. Constituency trees are two-way trees with each word in
a sentence as a leaf node and can serve as suitable alternative of
structured image tree annotation. However, such constituency trees
inevitably contain irrelevant words (e.g., adjectives and adverbs)
that do not denote semantic entities or interaction relations. Thus,
in the second step, we filter the leaf nodes by their part-ofspeech, preserving only nouns as object candidates, and verbs and
prepositions as relation candidates (i.e. the tree in the middle of
Fig. 6). In the third step, nouns are converted to object categories.
Note that sometimes different nouns (e.g. “cat” and “kitten”)

5

represent the same category. The lexical relation in WordNet [17]
is employed to unify the synonyms belonging to the same defined
category. The entities that are not in any defined object categories
(e.g. “grass” in “a sheep stands on the grass”) are also removed
from the trees. In the fourth step, relations are also recognized and
refined. Let R denote a set of defined relations. We provide the list
of relations we defined for different datasets in Table 10. Note that
R also includes an extra relation category, i.e. “others”, to denote
all the other relations that are not explicitly defined. Let T be the
set of triplets with the form of (entity1, verb/prep, entity2).
We construct a mapping T → R to recognize the relations and
construct the semantic tree (i.e., the tree in the bottom of Fig. 6).
4.2

Loss Functions

Before introducing the weakly supervised training algorithm, we
first define the loss function in the fully supervised setting. For
each image Ii , we assume that both the groundtruth semantic
map Ci and the groundtruth semantic tree Ti are known. Then,
the loss function is defined as the sum of three terms: semantic
label loss JC , scene structure loss JR , and regularizer R(W)
on model parameters. With a training set containing N images
{(I1 , C1 , T1 ), ..., (IN , CN , TN )}, the overall loss function can
be defined as,
N
1 X
(JC (WC ; Ii , Ci )+JR (W; Ii , Ti ))+λR(W)
N i=1
(10)
where Ii is the i-th image. Ti is the groundtruth semantic tree including both hierarchical scene structure and interobject relation. W = {WC , WR } denotes all model parameters. WC and WR are the model parameters of the
CNN and RsNN, respectively. Note that WR includes the
parameters of the five subnetworks defined in Sec.3.2, i.e.
WR = {Wtran , Wcom , Wint , Wcat , Wscore }. The regularization term is defined as R(W) = λ2 ||W||2 and λ is the
regularization parameter.

J (W) =

4.2.1 Semantic Label Loss
The goal of semantic labeling is to assign the category labels
to each pixel. Let Cf be the final predicted semantic map, Ce
the e-th semantic map of the multi-scale prediction streams. The
semantic label loss for an image I is defined as,
PE
Le (C, Ce )
(11)
+ Lf (C, Cf )
JC (WC ; I, C) = e=1
E
where Lf indicates the loss generated by the final predicted
semantic map Cf . Each element in Cf is calculated by Eqn. (1),
and we have Ct,f (j) = σf (stj ). C is the groundtruth label map.
By considering the multi-scale prediction streams, we also define
the loss Le , {e = 1, 2, ..., E} for multiple feature streams (i.e. the
red line in Fig. 3). Same as the Cf , each element in Ce is defined
t,e
by Ct,e (j) = σe (sj ). The cross entropy is adopted in Lf and
Le as the error measure.
4.2.2 Scene Structure Loss
The purpose of constructing scene structure is to generate the
meaningful configurations of the scene and predict the interaction
relations of the objects in the scene. To achieve this goal, the
scene structure loss can be divided into two parts: one for scene
hierarchy construction and the other for relation prediction,

JR (W; I, T ) = Jstruc (W1 ; I, T S ) + Jrel (W2 ; I, T R ) (12)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

6

Fig. 5. An illustration of the training process to our deep model architecture. The blue and green parts are corresponding to semantic labeling and
scene structure prediction, respectively. In practice, the input image is first fed into CNN to generate the predicted label map. Then we extract the
noun words from the semantic tree to refine the label map, and output intermediate label map. The semantic label loss (i.e. the blue dashed block)
is calculated by the difference between these two label maps. On the other hand, the feature representation of each object is also passed into
RsNN to predict the scene structure. We use scene hierarchy and inter-object relation, and the sematic tree to calculate the structure and relation
loss (i.e. the green dashed block). The red dotted lines represent the path of back propagation.

ROOT
S

constituency tree

VP
NP
NP
VP
VBZ
PP
DT NN VBG
NP
sits IN
NP
a man holding DT JJ NN
VP
on
NP
DT NN
VBG
a red bottle
a chair standing IN

POS tag filtering

by
CB
CB

CB

PP
NP
NP
DT NN

PP
IN

NP

a monitor on DT NN
the table

VBZ
CB
NN
CB
sits IN
CB
man VBG
NN
on NN
holding bottle
CB
chair VBG
CB
entity and relation
CB
standing IN
recognition
by NN
CB
monitor IN NN
sit on
hold
man

bottle

stand by
chair

support

on table

semantic tree

monitor table

Fig. 6. An illustration of the tree conversion process. The top is the
constituency tree generated by language parser, the middle is the constituency tree after POS tag filtering, and the bottom is the converted
semantic tree.

where T S and T R indicate the groundtruth of hierarchical scene structure and inter-object relations, respectively.
W1 = {WC , Wtran , Wcom , Wint , Wscore } and W2 =
{WC , Wtran , Wcom , Wint , Wcat }. The above two items are
jointly used to optimize the parameters of CNN and RsNN. The
difference is that Wscore in Eqn. (9) and Wcat in Eqn. (8) are
optimized by the structure loss and relation loss, respectively.
Scene Hierarchy Construction. Scene hierarchy construction
aims to learn a transformation I → PI . The predicted scene
hierarchy PI is said to be valid if the merging order between
regions is consistent with that in the groundtruth hierarchical

scene structure. Given the groundtruth hierarchical scene structure
T S , we extract a sequence of “correct” merging operations as
A(I, T S ) = {a1 , ..., aPS }, where PS is the total number of
merging operation. Given an operation a on the input image I, we
use q(a) to denote the merging score produced by the Scorer subnetwork. Based on the merging score q(a) calculated in Eqn. (9),
we define the loss to encourage the predicted scene hierarchy to
be consistent with the groundtruth. Specifically, the score of a
correct merging operation is required to be larger than that of
any incorrect merging operation b
a with a constant margin 4, i.e.,
q(a) ≥ q(b
a) + 4. Thus, we define the loss for scene hierarchy
construction as,
PS
1 X
ap )
[
max q(b
Jstruc (W; I, T S ) =
S)
PS p=1 abp ∈A(I,T
/
(13)

− q(ap ) + 4 ]
Intuitively, this loss intends to maximize the score of correct merging operation while minimizing the scores of incorrect merging
operations. To improve efficiency, only the highest score of the
incorrect merging operation is considered during training.
Relation Categorization. Denote by {kl} the combination of
two child nodes k and l. Let ykl be the softmax classification
result by the Categorizer sub-network in Eqn. (8), and ybkl be the
groundtruth relation from T R . The loss on relation categorization
is then defined as the cross entropy between ykl and ybkl ,
1 X
Jrel (W; I, T R ) =
Lr (ybkl , ykl )
(14)
|N |
R

{kl}

where ykl is the predicted relation probability in Eqn. (9). |NR |
denotes the number of relations in T R .
4.3

EM Method for Weakly Supervised Learning

In our weakly supervised learning setting, the only supervision
information is the descriptive sentence for each training image.
By converting the descriptive sentence to the semantic tree T , we
can obtain the entities T E (i.e., nouns), the relations T R (i.e.,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

7

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

Algorithm 1 EM Method for Weakly Supervised Training
Input:
Training samples (I1 , T1 ),(I2 , T2 ),...,(IZ , TZ ).
Output:
The parameters of our CNN-RsNN model W
Preparation:
Initialize the CNN model with the pre-trained networks on ImageNet
Initialize the RsNN model with Gaussian distribution
repeat
b i }Z according to
1. Estimate the intermediate semantic maps {C
i=1
Algorithm 2
2. Predict the scene hierarchy and inter-object relations for each image
Ii
3. Replace the groundtruth semantic maps {Ci }Z
i=1 in Eqn. (10) with
b i }Z .
intermediate semantic maps {C
i=1
4. Update the parameters W according to Eqn. (10)

pixel acc.
71.4
69.8
72.9
67.7
67.6
68.2
68.4

mean acc.
46.9
48.2
52.4
56.9
56.9
57.4
58.1

mean IoU
29.4
28.3
30.3
34.3
34.4
34.7
35.1

TABLE 1
Results on VOC 2012 val set under the weakly supervised learning.

on the predicted map the annotated entities T E ,
b = arg max log P (C|I; W0 ) + log P (T E |C).
C
C

until The optimization algorithm converges

C

(15)

0

Algorithm 2 Estimating Intermediate Label Map
Input:
Annotated entities T E in the semantic tree, normalized prediction score
k
σe (sk,e
j ) and final prediction score σf (sj ), where j ∈ {1, .., M }, k ∈
{0, .., K}, e ∈ {1, ..., E}.
Output:
b = {b
Intermediate label map C
cj }M
j=1
Preparation:
(1) To simplify, let f be the E + 1 scale.
(2) Set ψ k,e = 0 and Gej (k) = log σe (sk,e
j ) for all e ∈ {1, .., E + 1} and
k ∈ {0, ..., K};
(3) Let ρbg , ρf g indicate the number of pixels being assigned to background
and foreground. Set ρk = ρbg if k = 0, ρk = ρf g if k ∈ {1, ..., K}.
repeat
1. Compute the maximum score at each position j ,
[Gej ]max = maxk∈T E Gej (k)
2. repeat
if the k-th semantic category appears in annotated entities T E ,
a) Set δjk,e = [Gej ]max − Gej (k).
b) Rank {δjk,e }M
j=1 according to the ascending sorting and obtain
the ranking list.
c) Select δik,e in the ρk -th position of the ranking list, and let
ψ k,e = δik,e
else Set ψ k,e = −∞ to suppress the labels not present in T E .
k,e .
Update Gej (k) with Gej (k) = log σe (sk,e
j )+ψ

until Handling all of K + 1 semantic categories.
until Updating all of the prediction score in E + 1 scales.
Calculate the intermediate label of each pixel using Eqn. (15)

verbs or prepositional phrases) and the composite structure T S
between entities, but cannot directly get the semantic map C.
Therefore, we treat the semantic labeling map C as latent variable
and adopt a hard EM approximation for model training. In the
b based on
E-step, we estimate the intermediate semantic map C
the previous model parameters and the annotated entities T E ,
b i . In the Mand replace the Ci in Eqn. (10) with its estimate C
step, mini-batch SGD is deployed to update the CNN and RsNN
parameters by minimizing the overall loss function. The detail of
our EM algorithm is described as follows:
b . As illus(i) Estimate the intermediate semantic map C
trated in Fig. 5 (i.e. blue part), the input image I first goes through
the convolutional neural network to generate predicted semantic
b is estimated based
map. Then the intermediate semantic map C

The classification probability P (C|I; WC ) of each pixel can
be computed using Eqn. (1). Inspired by the effectiveness of
cardinality potentials [41][42], we define log P (T E |C) as entitydependent bias ψ k for the class label k , and set ψ k adaptively in
a manner similar to [36].
For multi-scale prediction streams, the score in the e-th stream
k,e
is calculated by Gej (k) = log σe (sj ) + ψ k,e . The fused score is
f
k
k,f
Gj (k) = log σf (sj ) + ψ . Then the intermediate label of pixel
j can be estimated by,
" E
#
X
f
e
(16)
cbj = arg max
Gj (k) + Gj (k)
k

e=1

Algorithm 2 summarizes our semantic map estimation method.
(ii) Predict the object hierarchy and inter-object relations.
Given the semantic labeling result, we group the pixels into
semantic objects and obtain the object feature representations
according to Eqn. (3) in Sec. 3.1. Then we use the RsNN model
to generate the scene structure recursively. In each recursion, the
model first calculates the context-aware feature representations
of two object regions ( object or the combination of objects )
according to Eqn. (4) ∼ Eqn. (6). Then it merges two object
regions with the largest confidence score by Eqn. (9) and predict
the interaction relation in the merged region by Eqn. (8). The green
part in Fig. 5 shows such process.
(iii) Update the CNN and RsNN parameters. Since the
ground truth label map is absent for the weakly supervision
manner, the model applies the intermediated label map estimated
in (i) as the pseudo ground truth, and calculates the semantic label
loss according to Eqn. (11). The blue dashed block in Fig. 5
shows this process. In contract, the structure and relation loss
is directly computed by the Eqn. (12), which uses the semantic
tree, scene hierarchy and inter-object relation as the inputs. The
green dashed block in Fig. 5 shows such process. With the minibatch BP algorithm, the gradients from the semantic label loss
propagate backward through all layers of CNN. The gradients
from the scene structure loss first propagate recursively through
the layers of RsNN, and then propagate through the object features
to the CNN. Thus, all the parameters (i.e., W) of our CNN-RsNN
model can be learned in an end-to-end manner (i.e. the red dotted
line in Fig. 5). Algorithm 1 summarizes the proposed EM method
for weakly supervised training.

5

E XPERIMENTS

In this section, we first apply our method for semantic scene
labeling and compare with existing weakly-supervised learning

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-MultiScale
Ours-Context
Ours-Full

pixel acc.
53.1
53.5
55.9
60.1
60.2
61.1
63.4

mean acc.
31.7
31.0
47.9
48.4
49.2
49.3
49.5

8

mean IoU
19.9
19.3
20.4
21.5
21.8
22.5
23.7

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

TABLE 2
Results on SYSU-Scenes under the weakly supervised learning.

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

#strong

#weak

208

1464

280

1464

1464

1464

1464

1464

pixel
acc.
82.7
82.2
81.8
78.1
78.0
78.2
78.2
86.4
86.3
85.7
83.1
83.3
83.3
83.5

mean
acc.
59.9
60.3
62.6
62.9
63.4
63.6
64.1
65.5
65.7
66.6
70.3
69.9
70.0
70.7

mean
IoU
39.3
38.4
42.5
43.2
43.3
43.5
43.7
46.2
45.7
46.2
50.9
51.1
51.2
51.7

TABLE 3
Results on VOC 2012 val set by ours and other semi-supervised
semantic segmentation methods.

based methods, and then evaluate the performance of our method
to generate scene structures. Extensive empirical studies for component analysis are also presented.
5.1

Experimental Setting

Datasets. We adopt PASCAL VOC 2012 segmentation benchmark [45] in our experiments, which includes 20 foreground
categories and one background category. And 1,464 annotated
images are used for training and 1,449 images for validation. Note
that we exclude the original testing subset on this benchmark due
to the lack of available ground-truth annotations.
We also introduce a new dataset created by us, i.e., SYSUScenes1 , especially for facilitating research on structured scene
parsing. SYSU-Scenes contains 5,046 images in 33 semantic categories, in which 3,000 images are selected from Visual Genome
dataset [46] and the rest are crawled from Google. For each
image, we provide the annotations including semantic object label
maps, scene structures and inter-object relations. We divide the
dataset into a training set of 3,793 images and a test set of 1,253
images. Compared with existing scene labeling / parsing datasets,
SYSU-Scenes includes more semantic categories (i.e., 33), detailed annotations for scene understanding, and more challenging
scenarios (e.g., ambiguous inter-object relations and large intraclass variations).
Sentence Annotation. We annotate one sentence description
for each image in both PASCAL VOC 2012 and SYSU-Scenes.
Since our work aims to learn a CNN-RsNN model for categorylevel scene parsing and structural configuration, in the supplementary materials, we explain the principles of sentence annotation in
more details, and provide representative examples and statistics of
1. http://www.sysu-hcp.net/SYSU-Scenes/

#strong

#weak

500

2552

500

2552

1241

2552

1241

2552

pixel
acc.
59.1
53.2
60.9
62.8
62.1
63.4
64.4
67.8
67.5
66.0
67.2
67.6
70.0
70.1

mean
acc.
54.3
58.1
56.8
57.2
57.4
58.5
57.6
49.4
50.9
53.1
53.4
51.9
54.8
52.3

mean
IoU
27.9
27.3
28.4
28.8
28.9
29.6
29.7
32.3
31.7
32.4
33.7
34.0
34.8
35.5

TABLE 4
Results on SYSU-Scenes by ours and other semi-supervised semantic
segmentation methods.

the sentence annotation. All the descriptive sentences on the VOC
2012 train and val sets are also given.
The sentence description of an image naturally provides a tree
structure to indicate the major objects along with their interaction
relations [47]. As introduced in Section 4.1, we use the Stanford
Parser [16] for sentence parsing and further convert the parsing
result into the regularized semantic tree. In this work, we see to it
that the semantic tree is generated from one sentence.
Network Architecture and Training. Our deep architecture
is composed of the stacked CNN and RsNN modules using the
Caffe [48] framework. We apply the VGG network [38] to build
the CNN module of 16 layers, and the RsNN is implemented by
four extra neural layers upon the CNN. Our network thus contains
20 layers.
All models in our experiment are trained and tested on a single
NVIDIA Tesla K40. The parameters of the VGG-16 network
are pre-trained on ImageNet [11], and the other parameters are
initialized with Gaussian distribution with standard deviation of
0.001. We train our network using stochastic gradient descent
(SGD) with the batch size of 9 images, momentum of 0.9, and
weight decay of 0.0005. The learning rate is initialized with 0.001.
We train the networks for roughly 15,000 iterations, which takes
8 to 10 hours.
5.2

Semantic Labeling

To evaluate the semantic scene labeling performance of our
method, we re-scale the output pixel-wise prediction back to the
size of original groundtruth annotations. The indicators, i.e., pixel
accuracy, mean class accuracy and mean intersection over
union (IoU) [5], are adopted for performance evaluation. We
consider two ways of training our CNN-RsNN model, i.e., weaklysupervised learning and semi-supervised learning.
Weakly-supervised Learning. We compare our method with
several state-of-the-art weakly-supervised semantic segmentation approaches, including MIL-ILP [43], MIL-FCN [35] and
DeepLab [36]. We perform experiments with the publicly available
code of DeepLab, and our own implementation of MIL-ILP
and MIL-FCN. In practice, we extract the multi-class labels of
each image from its groundtruth label map as the supervision
information to train the competing models. As for our method,
we apply the noun words in the semantic trees as the image-level
labels. Table 1 and Table 2 list the results of the three performance

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

bkg
72.2
69.9
71.8
62.4
62.4
65.3
68.1

aero
31.8
29.7
29.7
40.7
41.0
41.0
40.8

bike
19.6
16.5
17.0
20.1
20.3
20.2
20.8

bird
26.0
23.4
24.2
33.5
34.6
33.8
33.2

boat
27.3
23.5
27.1
31.3
31.8
31.1
32.1

bottle
33.4
30.3
32.2
25.2
25.0
25.1
25.8

bus
41.8
40.6
43.5
47.9
48.3
48.7
47.6

car
48.6
46.5
45.4
47.6
47.6
47.5
47.1

cat
42.8
40.8
38.7
42.9
41.4
42.8
43.7

9

chair
9.96
11.0
10.9
11.1
11.0
11.1
12.1

cow
24.8
28.5
30.0
40.4
40.1
40.4
41.5

table
13.7
12.0
21.1
22.9
21.7
22.7
23.1

dog
33.2
32.5
33.6
42.2
43.1
42.4
41.9

horse
21.4
22.9
27.7
40.8
41.6
41.2
40.8

mbike
30.7
29.8
32.5
40.6
41.2
41.3
42.6

person
22.4
22.8
32.2
27.8
27.2
27.6
27.4

plant
22.3
19.9
17.9
19.2
19.0
20.2
20.3

sheep
27.1
25.3
24.7
36.3
36.2
37.5
37.3

sofa
16.6
17.1
19.2
25.0
24.9
25.0
24.7

train
33.3
31.2
36.4
42.0
42.2
42.5
42.6

tv
19.3
20.1
19.9
21.4
21.6
21.5
22.3

mean
29.4
28.3
30.3
34.3
34.4
34.7
35.1

TABLE 5
Experimental results (IoU) on VOC 2012 val set under the weakly supervised learning.

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MulitScale
Ours-Full

bkg
41.2
39.4
42.4
46.2
45.6
46.1
48.4

aero
36.7
31.7
31.1
33.7
33.3
38.2
39.1

ball
1.93
2.13
2.72
3.26
3.42
3.48
4.08

bench
18.3
14.8
20.2
20.5
20.2
21.8
23.1

bike
28.6
25.9
21.9
21.9
22.6
25.0
26.8

bird
21.6
19.5
14.8
15.3
17.7
19.7
21.0

boat
14.4
11.3
14.7
20.0
19.9
20.9
20.4

bottle
9.71
8.15
10.1
13.8
13.9
13.4
13.7

bus
8.79
13.1
10.5
11.1
10.5
12.1
11.7

building
46.9
43.7
44.2
44.8
43.5
45.3
47.0

car
20.5
20.6
22.3
22.5
21.3
22.0
24.7

cat
28.2
34.5
34.7
34.2
34.3
35.8
36.1

chair
1.22
1.71
6.59
7.55
7.91
5.96
5.46

cow
17.7
17.7
20.3
21.0
22.5
23.3
24.8

cup
13.9
11.5
10.5
8.04
8.24
8.25
9.01

dog
30.3
32.3
22.8
23.3
23.3
24.1
25.1

grass
16.7
16.9
18.7
16.3
18.8
18.2
20.3

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

horse
20.2
21.1
28.4
32.5
33.4
35.0
36.1

laptop
28.3
29.2
26.5
29.8
20.1
28.0
30.1

mbike
47.7
48.2
39.5
42.5
42.6
44.7
48.7

person
25.9
27.0
26.3
24.6
26.8
26.7
32.9

racket
5.44
5.71
7.32
7.23
7.91
7.06
8.05

rail
12.1
10.6
17.4
17.5
18.6
19.0
19.3

sea
2.71
3.05
7.49
6.29
7.99
9.02
9.91

sheep
14.8
14.3
16.2
17.0
14.3
13.7
14.6

sky
10.9
19.7
16.9
18.1
18.5
18.4
18.4

sofa
18.2
11.1
17.3
17.4
18.3
19.8
19.7

street
10.7
9.43
19.4
20.9
22.5
22.3
23.2

table
14.0
11.5
14.3
14.6
14.4
13.6
14.1

train
33.8
27.1
34.5
36.4
36.7
38.8
41.2

tree
6.75
7.33
11.4
13.0
13.0
13.8
15.1

TV
25.8
24.5
19.1
20.9
21.3
21.9
22.1

umbrella
26.0
22.1
23.2
26.9
26.7
28.7
28.9

mean
19.9
19.3
20.4
21.5
21.8
22.5
23.7

TABLE 6
Experimental results (IoU) on SYSU-Scenes under the weakly supervised learning learning.

Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

#strong

#weak

280

1464

280

1464

1464

1464

1464

1464

bkg
82.4
81.9
83.0
74.9
74.8
75.1
75.0

aero
35.7
36.5
42.8
50.6
50.4
50.9
51.0

bike
23.5
22.9
22.6
22.9
23.1
23.2
23.7

bird
37.3
32.8
40.61
45.4
45.5
45.1
45.7

boat
30.7
29.2
37.5
41.9
41.6
42.2
42.0

bottle
40.9
39.0
36.9
36.9
37.4
37.2
37.2

bus
58.0
57.5
60.6
53.8
54.4
55.0
56.9

car
61.4
58.8
58.5
58.3
58.7
59.0
59.1

cat
56.9
57.9
60.3
62.4
62.5
62.7
62.9

chair
11.1
11.6
15.1
13.2
13.1
13.4
13.4

cow
32.0
31.4
38.5
49.0
49.4
49.1
48.5

table
13.9
13.5
26.0
20.9
21.1
21.1
22.0

dog
48.2
47.1
51.8
54.4
54.5
54.6
55.1

horse
39.2
36.1
43.6
50.4
50.4
50.4
50.3

mbike
44.3
43.9
47.5
49.1
49.2
49.4
48.9

person
58.2
57.1
58.4
56.3
56.5
56.7
57.0

plant
19.6
18.9
43.7
23.2
22.9
22.9
24.0

sheep
38.9
37.8
34.1
43.0
43.5
43.8
43.7

sofa
24.1
23.1
24.9
28.5
28.5
28.8
29.2

train
40.0
40.5
39.9
45.5
45.4
46.2
45.7

tv
29.1
29.4
26.5
25.5
25.7
26.1
26.5

mean
39.3
38.4
42.5
43.2
43.3
43.5
43.7

86.5
86.2
85.2
80.7
80.9
81.3
81.8

53.7
51.2
49.5
60.6
61.6
61.9
62.4

24.9
23.8
21.8
25.6
25.5
25.6
25.7

49.0
49.9
51.4
55.6
55.6
55.9
55.6

45.8
45.8
42.6
51.9
52.5
52.1
52.3

48.3
47.8
45.4
44.0
43.3
43.7
44.1

59.4
60.9
63.8
61.7
61.4
61.6
62.4

68.2
67.3
68.9
67.2
66.8
67.1
67.8

64.0
64.9
66.6
70.8
70.8
71.1
71.0

16.8
16.6
16.1
16.2
16.4
16.2
16.3

37.1
33.3
40.9
55.3
55.6
56.2
56.6

14.2
11.2
23.4
24.5
25.4
24.3
24.7

59.2
58.3
56.5
64.8
64.9
64.7
65.0

46.5
45.3
46.4
57.7
57.6
58.2
58.7

54.8
57.2
54.1
58.4
58.3
58.5
58.8

65.3
66.7
64.9
66.1
65.8
66.1
66.2

27.6
26.6
25.4
29.6
29.3
29.4
29.7

37.8
37.7
36.9
47.5
48.4
47.5
47.5

29.1
28.2
26.3
35.4
36.1
36.3
37.0

47.8
48.9
50.6
57.1
55.8
56.8
56.8

34.5
33.4
32.7
38.1
39.6
40.0
40.9

46.2
45.7
46.2
50.9
51.1
51.2
51.7

TABLE 7
Experimental results (IoU) on VOC 2012 val set under the semi-supervised learning.

metrics on PASCAL VOC 2012 and SYSU-Scenes. Table 5 and
Table 6 further report the breakdown IoU results with respect to
object category. Our method obtains the mean IoUs of 35.1% and
23.7% on the two datasets, outperforming DeepLab[36] by 4.8%
and 3.3%, respectively.
Semi-supervised Learning. Moreover, we evaluate our
method under the way of semi-supervised model learning. In
this setting, the groundtruth semantic labeling maps are available
for a part of images in the training set, and others still use
the image-level category labels as the supervision. Our CNNRsNN model can be easily trained on strongly-annotated images
without estimating their intermediate label maps. Following the
setting of existing semi-supervised learning based methods on
PASCAL VOC 2012, we employ part of images from the Semantic
Boundaries dataset (SBD) [49] to conduct the experiments: using
280 and 1464 strongly-annotated images from SBD, respectively,
in addition to the original 1464 weakly annotated (i.e., associated
sentences) images. We set the weight, i.e., 1 : 1, for combining the
loss scores that respectively computed on the strongly-annotated
images and weakly-annotated images. Table 3 reports the quan-

titative results generated by our method and other competing
approaches. Table 7 presents the breakdown IoU results on each
object category. We also conduct the experiments on SYSUScenes, and select 500 and 1241 images from the training set
as the strongly-annotated samples, respectively. And the overall
results are reported in Table 4 and the breakdown IoU results in
Table 8.
It can be observed that all methods benefit from the stronglyannotated supervision. On PASCAL VOC 2012, compared with
our weakly supervised CNN-RsNN baseline, the improvement
on IoU is 8.6% with 280 strongly annotated images (amount
of “strong” : “weak” samples = 1:5), and is 16.6% with 1464
strongly annotated images (amount of “strong” : “weak” samples = 1:1). Moreover, our method outperforms semi-supervised
DeepLab [36] by 1.2% with 280 strongly-annotated samples and
5.5% with 1464 strongly-annotated ones. On SYSU-Scenes, in
terms of IoU, our model outperforms the weakly-supervised CNNRsNN baseline by 6.0% with 500 strongly-annotated images
(amount of “strong” : “weak” samples = 1:5), and 11.8% with
1241 strongly annotated images (amount of “strong” : “weak”

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

#strong

#weak

500

2552

500

2552

1241

2552

1241

2552

#strong

#weak

500

2552

500

2552

1241

2552

1241

2552

MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full
Method
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full
MIL-ILP [43]
MIL-FCN [35]
DeepLab-EM-Adapt [36]
Ours-Basic [44]
Ours-Context
Ours-MultiScale
Ours-Full

10

bkg
59.8
56.0
56.9
56.8
56.0
58.9
59.8

aero
39.2
38.1
38.2
44.8
44.1
42.1
42.5

ball
3.51
3.16
6.85
4.34
4.06
4.76
4.06

bench
27.5
24.6
24.1
26.5
26.9
27.8
30.7

bike
34.1
32.0
31.6
35.7
36.0
36.2
39.7

bird
20.6
27.2
18.9
22.9
23.1
24.1
23.4

boat
21.6
20.1
24.0
23.0
24.0
22.6
24.4

bottle
14.1
14.9
13.1
21.4
21.1
21.4
21.9

building
17.7
15.5
13.2
10.3
10.9
11.7
11.3

bus
56.6
54.7
63.9
57.7
58.1
57.8
58.0

car
31.0
30.3
30.7
30.6
30.3
31.2
32.4

cat
25.9
24.8
41.6
37.1
37.1
38.3
38.5

chair
5.58
7.91
10.0
8.09
7.89
8.74
8.42

cow
27.6
27.8
26.9
32.1
32.4
32.6
32.2

cup
21.2
21.5
14.7
13.7
13.2
13.4
12.1

dog
39.4
39.5
30.4
30.5
29.8
31.0
30.2

grass
17.9
10.8
29.0
20.9
21.6
22.8
22.0

67.9
66.0
61.5
62.3
65.0
62.3
65.1

48.6
47.1
48.9
51.5
50.7
54.4
54.5

1.02
1.13
7.73
7.01
7.13
7.15
9.19

29.8
27.5
26.8
28.9
28.1
27.7
30.1

38.4
40.0
40.7
41.6
41.2
43.8
43.6

35.9
37.2
22.9
30.1
36.9
32.8
36.8

19.9
20.1
27.3
30.7
25.3
31.6
27.9

18.4
17.9
13.8
25.8
24.9
23.5
23.3

18.2
19.4
12.6
21.7
11.8
16.6
16.3

54.9
54.1
56.8
53.4
53.1
57.2
58.4

33.4
33.2
36.9
36.0
35.9
36.7
38.1

39.8
34.8
46.7
40.5
42.3
40.4
38.8

6.14
6.91
11.6
9.71
9.48
11.1
11.6

31.2
30.9
45.2
47.0
46.9
43.8
47.4

10.7
11.5
17.4
17.6
17.2
19.2
20.8

50.8
49.4
44.3
49.8
49.0
48.3
48.8

36.6
38.7
30.9
26.1
30.7
32.1
33.1

horse
39.5
37.6
36.3
37.0
36.6
38.3
40.3

laptop
41.5
39.7
34.1
35.6
36.3
39.1
37.6

mbike
45.5
47.2
50.1
53.4
51.3
54.9
55.8

person
45.0
41.5
41.0
45.6
44.4
48.7
55.0

racket
12.8
12.0
8.06
8.04
8.64
8.45
8.91

rail
15.2
15.1
23.9
13.7
19.6
20.8
12.7

sea
20.3
21.1
15.9
13.8
11.7
10.4
15.6

sheep
35.8
34.5
39.2
38.4
36.8
37.6
37.7

sky
15.1
18.8
25.2
26.3
24.7
27.0
26.8

sofa
25.9
28.3
20.8
20.5
20.3
21.2
21.8

street
24.7
26.7
32.3
29.8
29.7
30.6
29.9

table
17.6
17.7
14.7
19.1
20.4
19.4
19.8

train
43.1
42.8
48.2
47.5
48.8
49.9
48.7

tree
17.4
19.1
16.2
21.2
21.1
19.3
18.8

TV
21.1
19.7
25.4
26.6
26.2
27.0
23.1

umbrella
37.8
31.5
30.6
36.8
36.9
37.4
37.9

mean
27.9
27.3
28.4
28.9
28.9
29.4
29.7

43.5
47.6
42.1
51.2
52.4
51.5
51.5

39.6
39.7
41.1
43.5
42.5
42.2
44.3

53.7
55.2
56.5
57.8
58.8
58.6
60.9

56.2
51.5
49.4
56.6
57.5
62.3
62.0

15.2
12.2
10.6
9.83
10.7
11.0
11.1

13.2
14.1
14.5
8.44
12.3
10.0
12.6

21.4
21.5
26.2
28.5
31.9
34.8
36.2

24.7
24.5
29.8
28.0
28.3
30.3
29.1

37.0
35.8
29.9
28.3
24.9
29.8
32.1

29.7
28.3
31.0
33.7
32.9
33.8
34.1

23.2
16.7
36.7
27.8
33.2
25.6
23.2

18.8
17.7
17.9
20.1
20.8
22.6
21.6

40.6
42.8
49.0
50.5
51.3
51.7
52.8

23.6
21.0
14.7
18.5
16.3
24.5
24.4

35.5
29.7
30.5
26.7
27.0
30.2
30.0

48.7
51.5
39.4
41.6
46.6
43.2
44.2

32.3
31.7
32.4
33.7
34.0
34.8
35.5

TABLE 8
Experimental results (IoU) on SYSU-Scenes under the semi-supervised learning learning.

Method
MIL-ILP [43]
DeepLab-EM-Adapt [36]
Ours-Full
MIL-ILP [43]
DeepLab-EM-Adapt [36]
Ours-Full
MIL-ILP [43]
DeepLab-EM-Adapt [36]
Ours-Full

# strong

# weak

0

1464

280

1464

1464

1464

mean IoU
28.81
30.57
35.19
39.03
43.07
44.13
46.14
46.82
51.37

TABLE 9
Performance on PASCAL VOC 2012 test set.

Dataset

Amount

PASCAL VOC 2012

9

SYSU-Scenes

13

Relations
beside, lie, hold, ride, behind,
sit on, in front of, on and others.
behind, beside, fly, hold, play,
in front of, ride, sit on, stand,
under, walk, on, and others.

TABLE 10
The defined relations in PASCAL VOC 2012 and SYSU-Scenes.

samples = 1:2). Our model also outperforms semi-supervised
DeepLab [36] by 1.3% with 500 strongly-annotated images and
3.1% with 1241 strongly-annotated images. Finally, Fig. 7 presents
the visualized labeling results on SYSU-Scenes.
To follow the standard protocol for PASCAL VOC semantic
segmentation evaluation, we also report the performance of our
method on the VOC 2012 test dataset in Table 9, under both the
weakly-supervised and semi-supervised manners.
5.3

Scene Structure Generation

Since the problem of scene structure generation is rarely addressed
in literatures, we first introduce two metrics for evaluation: structure accuracy and mean relation accuracy. Let T be a semantic

Without
Context
With
Context

CNN
partial fixed
updated
partial fixed
updated

RsNN
updated
updated
updated
updated

struct.
61.7
64.2
62.8
67.4

mean rel.
27.9
28.6
27.4
32.1

TABLE 11
Results on PASCAL VOC 2012 with different learning strategies.

tree constructed by CNN-RsNN and P = {T, T1 , T2 , . . . , Tm }
be the set of enumerated sub-trees (including T ) of T . A leaf Ti
is considered to be correct if it is of the same object category as
the one in the ground truth semantic tree. A non-leaf Ti (with two
subtrees Tl and Tr ) is considered to be correct if and only if Tl and
Tr are both correct and the relation label is correct as well. Then,
(#of correctsubtrees)
and can be
the relation accuracy is defined as
m+1
computed recursively. The mean relation accuracy is the mean of
relation accuracies across relation categories. Note that the number
of sub-trees of each relation category is highly imbalanced in
both two datasets, where the relations of most sub-trees are from
several dominant categories. Taking this factor into account, the
mean relation accuracy metric should be more reasonable than the
relation accuracy metric used in our previous work [44].
Here we implement four variants of our CNN-RsNN model
for comparison, in order to reveal how the joint learning of CNNRsNN and the utility of context contribute to the overall performance. To train the CNN-RsNN model, we consider two learning
strategies: i) updating all parameters of the RsNN by fixing the
parameters of CNN; ii) joint updating the parameters of CNN and
RsNN in the whole process. For each strategy, we further evaluate
the effect of contextual information (i.e., distance, relative angle
and area ratio) by learning the interpreter sub-networks (i) with
contextual information and (ii) without contextual information.
Table 11 and Table 12 report the results on the PASCAL VOC
2012 validation set and the SYSU-Scenes testing set. Table 13 and

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

(a)

(b)

(c)

(d)

11

(e)

(f)

(g)

(h)

(i)

Fig. 7. Visualized semantic labeling results on SYSU-Scenes. (a) The input images; (b) The groundtruth labeling results; (c) Our proposed method
(weakly-supervised); (d) DeepLab (weakly-supervised) [36]; (e) MIL-ILP (weakly-supervised) [43]; (f) Our proposed method (semi-supervised with
500 strong training samples); (g) Our proposed method (semi-supervised with 1241 strong training samples); (h) DeepLab(semi-supervised with
500 strong training samples) [36]; (i) MIL-ILP (semi-supervised with 500 .strong training samples) [43].

in front of

others

car

ride

behind

in front of
car

ride
person

bike

table

sit on

sit on
person

person

bike

table

chair

person

chair

Fig. 8. Visualized scene parsing results on PASCAL VOC 2012 under the weakly-supervised setting. The left one is a successful case, and the right
is a failure one. In each case, the tree on the left is produced from descriptive sentence, and the tree on the right is predicted by our method.

Without
Context
With
Context

CNN
partial fixed
updated
partial fixed
updated

RsNN
updated
updated
updated
updated

struct.
38.0
44.3
41.7
48.2

mean rel.
19.6
24.1
21.8
24.5

TABLE 12
Results on SYSU-Scenes with different learning strategies.

Table 14 present the breakdown accuracy on relation categories.
Fig. 8 and Fig. 9 show several examples of visualized scene
parsing results on PASCAL VOC 2012 and SYSU-Scenes. The
experiment results show that: (i) the incorporation of contextual
information can benefit structure and relation prediction in terms
of all the three performance metrics; (ii) joint optimization is very
effective in improving structured scene parsing performance, no
matter contextual information is considered or not. Please refer
to the supplementary materials for more successful and failure
parsing results and our discussion on causes of failure.

5.4

Inter-task Correlation

Two groups of experiments are conducted to study the intertask correlation of the two tasks: semantic labeling and scene
structure generation (i.e., scene hierarchy construction and interobject relation prediction). In the first group, we report the results
with three different settings on the amount of strongly annotated
data in semi-supervised learning of CNN-RsNN: i) zero strongly
annotated image, ii) 280 strongly annotated images for PASCAL
VOC 2012, and 500 strongly annotated images for SYSU-Scenes,
and iii) 1464 strongly annotated images for PASCAL VOC 2012,
and 1241 strongly annotated images for SYSU-Scenes. Other
settings are the same with that described in Sec. 5.2.
In the second group, we report the results with three different configurations on the employment of relation information in
training CNN: i) zero relation, ii) relation category independent,
and iii) relations category aware. In Configuration i), we ignore
gradients from both the Scorer and the Categorizer sub-networks
(see Sec. 3.2) of the RsNN model. In Configuration ii), we
assume all relations are of the same class, and only back-propagate
the gradients from the Scorer sub-network. In Configuration iii),

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

12

others
on
street

beside

grass

ride

on

on
grass

ride
person

horse

behide
person

person

ride

car

ride

horse

car

street

bike
bike

person

Fig. 9. Visualized scene parsing results on SYSU-Scenes under the semi-supervised setting (i.e. with 500 strongly-annotated images). The left one
is a successful case, and the right is a failure one. In each case, the tree on the left is produced from descriptive sentence, and the tree on the right
is predicted by our method.

Without
Context
With
Context

CNN
partial fixed
updated
partial fixed
updated

RsNN
updated
updated
updated
updated

beside
20.7
23.6
18.3
19.7

lie
4.54
13.6
18.2
13.6

hold
3.57
14.3
17.8
21.4

ride
23.4
33.3
40.7
39.5

behind
14.3
17.8
10.7
21.4

sit
81.1
64.8
40.5
59.4

in front
2.77
7.93
4.36
8.33

on
59.0
46.3
55.4
61.4

other
34.9
35.6
40.9
43.6

mean
27.9
28.6
27.4
32.1

TABLE 13
The mean relation accuracy on the PASCAL VOC 2012 dataset.
Without
Context
With
Context

CNN
partial fixed
updated
partial fixed
updated

RsNN
updated
updated
updated
updated

behind
5.54
8.11
7.33
10.1

beside
9.24
11.3
13.4
14.8

fly
10.6
16.1
12.1
16.1

hold
27.3
37.0
28.6
33.5

play
60.8
66.9
61.6
64.1

in front
5.93
7.20
8.78
12.8

ride
17.6
25.6
23.0
25.7

sit
39.7
41.4
44.1
49.7

stand
4.81
7.13
4.41
3.19

under
17.4
23.0
22.4
20.8

walk
9.47
16.7
9.69
11.7

on
21.1
22.2
23.4
25.4

other
25.8
30.5
24.9
30.2

mean
19.6
24.1
21.8
24.5

TABLE 14
The mean relation accuracy on SYSU-Scenes.

wise classification loss is more effective than scene structure loss.

As shown in Fig. 10 and Fig. 11, the semantic labeling task
is strongly correlated with the scene structure generation task. Increasing the amount of strongly annotated data and employing relation information can benefit both the semantic labeling and scene
structure generation. As a result, the increase of relation/structure
accuracy can result in a near-linear growth of semantic labeling
accuracy.
We further study the correlation of two tasks under the full
pixel supervision setting. Different from the semi-supervised setting, we conduct the full pixel supervision without using extra data
from SBD [49]. Under this setting, we obtain two main observations as follows: (1) The introduction of full pixel supervision
does benefit structure and relation prediction. The accuracies of
structure and relation prediction are 71.3% and 39.5% under
the full pixel supervision, which are higher than the weaklysupervised setting with an obvious margin. (2) Under the full
pixel supervision, the further introduction of descriptive sentence
contributes little in semantic labeling accuracy. The mIoU of
segmentation achieves 53.67% on the PASCAL VOC val dataset
under the fully supervised setting, this value is improved only
0.13% when image description is introduced to calculate the scene
structure loss. The results is natural since structure and relation
prediction are performed after semantic labeling, and the pixel-

36

90
Structure Accuracy
Mean Relation Accuracy

80

35
34.3

70

34

69.1

67.2

Accuracy: %

64.2

33

60

IoU: %

we back-propagate the gradients from both the Scorer and the
Categorizer sub-networks.

50

32.7

32
31
30.3

40

30

30
28.6

29.8

30.2
29
28

20
34.3

43.2

IoU: %

50.9

8.67

14.5

28.6

Mean Relation Accuracy: %

Fig. 10. Results of the inter-task correlation experiments on PASCAL
2012. The figure shows how segmentation and structure prediction
task affect each other. Improving performance of one task results in
improvement of the other. The left shows the effect of segmentation
performance on relation and structure prediction based on the first
group of experiments. The right shows the effect of relation prediction
performance on semantic segmentation based on the second group of
experiments. In practice, the segmentation performance is improved by
adding more strongly annotated training data, while the performance of
structure and relation prediction is improved by considering more types
of relations.

6

C ONCLUSION AND F UTURE W ORK

In this paper, we have introduced a novel neural network model to
address a fundamental problem of scene understanding, i.e., parsing an input scene image into a structured configuration including

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
22

60
Structure Accuracy
Mean Relation Accuracy

55

21.8
53.1
21.6

49.4

50

21.5

21.4
21.2

IoU: %

Accuracy: %

45

44.3

40

21
20.8

35

20.6
20.6

30
25.7
25

26.9

13

entities in images, as it would result in sentences being excessively
long, complex and unnatural. Fig. 12 illustrates some pairs of
images and annotated sentences in VOC 2012 train and val set.
For most images, both the objects and their interaction relations
can be described with one sentence. In particular, we summarize
three significant annotation principles as follows:

20.4
20.4

•

24.1
20.2
20

20
21.5

28.9

33.7

IoU: %

6.81

10.3

24.1

Mean Relation Accuracy: %

Fig. 11. Results of the inter-task correlation experiments on SYSUScenes. The left shows the effect of segmentation performance on
relation and structure prediction experiments based on the first group
of experiments. The right shows the effect of relation prediction performance on semantic segmentation based on the second group of
experiments.

•

•

a semantic object hierarchy with object interaction relations. Our
CNN-RsNN architecture integrates the convolutional neural networks and recursive neural networks for joint end-to-end training,
and the two networks collaboratively handle the semantic object
labeling and scene structure generation. To avoid expensively
training our model with fully-supervised annotations, we have
developed a weakly-supervised model training method by leveraging the sentence descriptions of training images. In particular,
we distill rich knowledge from the sentence for discovering scene
configurations. Experimental results have demonstrated the effectiveness of our framework by producing meaningful and structured
scene configurations from scene images. We also release a new
dataset to facilitate research on structured scene parsing, which
includes elaborative annotations of scene configurations.
There are several directions in which we can do to extend
this work. The first is to improve our framework by adding a
component for recognizing object attributes in the scenes that corresponds the adjectives in the sentence descriptions. The second
is to incorporate some instance segmentation [50], [51], [52] or
object detection [53] model for instance level parsing. The third is
to deeply combine our framework with state-of-the-art language
processing techniques to improve the sentence parsing. Moreover,
how to deal with the ambiguities of multiple sentence descriptions
should be pursued.

ACKNOWLEDGEMENT
This work was supported by State Key Development Program
under Grant 2016YFB1001004, the National Natural Science
Foundation of China under Grant 61622214, and the Guangdong
Natural Science Foundation Project for Research Teams under
Grant 2017A030312006.

S UPPLEMENTARY M ATERIAL
6.1

Dataset

Sentence Annotation. We asked 5 annotators to provide one descriptive sentence for each image in the PASCAL VOC 2012 [45]
segmentation training and validation set. Images from two sets are
randomly partitioned into five subsets of equal size, each assigned
to one annotator. We provided annotators with a list of possible
entity categories, which is the 20 defined categories in PASCAL
VOC 2012 segmentation dataset.
We ask annotator to describe the main entities and their
relations in the images. We did not require them to describe all

For the image with only an instance of some object
category, e.g., the last image in the first row of Fig. 12,
the sentence describes the relation between the object (i.e.
airplane) and the background (i.e. runway);
For the instances from the same category with the same
state, we describe them as a whole. Such as the forth image
in the seconde row of Fig. 12, the annotation sentences is
“two motorbikes are parked beside the car”.
For the instances from the same category with the different
state, the annotator may only describe the most significant
one. As to the third image in the second row of Fig. 12,
the annotator describe the people sitting on the chairs but
ignore the baby sitting on the adult.

We did not prohibit describing entities that did not belong to
the defined categories, because they are necessary for natural
expression. But we will remove them in the process of generating
semantic trees.
We annotate one sentence for each image because our method
involves a language parser which produces one semantic tree for
each sentence. At this point, we are unable to generate one tree
structure from multiple sentences. Therefore, one sentence for
each images is sufficient for our study. To give more details of
the image descriptions, we provide our sentence annotations of
entire dataset in “train sentences.txt” and “val sentences.txt” as
supplementary materials.
As described in the main paper, we parse sentences and convert
them into semantic trees which consist of entities, scene structure
and relations between entities. Here we provide the list of 9
relation categories we defined: beside, lie, hold, ride, behind,
sit on, in front of, on and other. The label other is assigned in
the following two cases. (i) An entity has the relation with the
background, which often happens at the last layer of the parsing
structure. (ii) The other relation is used as placeholder for the
relation not identified as any of the 8 other relations
Annotation Statistics. Since the sentence annotations are not
a standard part of the PASCAL VOC dataset, we give some
statistical analysis of images and annotations in Fig. 13 and Fig. 14
to incorporate more information about our parsing task. Fig. 13
shows the number of object category of each image in VOC train
and val dataset. Obviously, for PASCAL VOC 2012 dataset, most
images only contain one object category. In order to construct
the tree structure, we combine the foreground object and the
background, and assign “other” as their relationship. Another kind
of images contain two or more object categories, and the number
of relations in these images is greater than one. As stated above,
we combine the merged foreground objects and the background
with the relation “other” at the last layer of the semantic tree.
According to the Fig. 13, the proportion of images with two or
more object categories in the entire dataset is greater than 1/3
(i.e. 39.21% for training set and 34.09% for validation set). Since
the the number of interaction relations usually increases with the
number of object growing, the total number of relations (except
“other”) in these images is more than 50% of the entire dataset
based on our sentence annotations.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

14

濈濜濙澔濫濣濡濕濢澔濖濙濧濝濘濙澔濕澔濖濠濕濗濟澔濘濣濛澔
濧濝濨濧澔濣濢澔濨濜濙澔濧濣濚濕

澵澔濖濝濦濘澔濧濨濕濢濘濧澔濣濢澔濨濜濙澔濖濣濭澔濠濭濝濢濛澔
濝濢澔濨濜濙澔濧濣濚濕

濈濫濣澔濖濝濦濘濧澔濕濦濙澔濙濕濨濝濢濛澔濖濦濙濕濘澔濣濢澔
濕澔濘濝濢濝濢濛澔濨濕濖濠濙

澵 濡濕濢澔濦濝濘濙濧澔濕澔濖濝濗濭濗濠濙澔濝濢澔濚濦濣濢濨澔
濣濚澔濕澔濛濦濙濙濢澔濖濩濧

澵澔濫濜濝濨濙澔濕濝濦濤濠濕濢濙澔濝濧澔濨濕濟濝濢濛澔濣濚濚澔濣濢澔
濨濜濙澔濦濩濢濫濕濭

濈濜濙澔濖濣濭澔濜濣濠濘濝濢濛澔濨濜濙澔濖濠濕濗濟澔濘濣濛澔
濝濧澔濧濝濨濨濝濢濛澔濣濢澔濨濜濙澔濧濣濚濕

濈濜濙澔濛濝濦濠澔濜濣濠濘濝濢濛澔濨濜濙澔濫濝濢濙澔濖濣濨濨濠濙澔
濝濧澔濝濢澔濚濦濣濢濨澔濣濚澔濨濜濙澔濨濕濖濠濙澔澔

濈濜濙澔濤濙濣濤濠濙澔濧濝濨濨濝濢濛澔濣濢澔濨濜濙澔濗濜濕濝濦濧澔
濕濦濙澔濖濙濜濝濢濘澔濨濜濙澔濘濝濢濝濢濛澔濨濕濖濠濙澔

濈濫濣澔濡濣濨濣濦濖濝濟濙濧澔濕濦濙澔濤濕濦濟濙濘澔
濖濙濧濝濘濙澔濨濜濙澔濗濕濦澔

濈濜濙澔濛濝濦濠澔濢濙濕濦澔濨濜濙澔濜濣濦濧濙澔濧濨濕濢濘濧澔
濖濙濜濝濢濘澔濨濜濦濙濙澔濘濣濛濧

Fig. 12. Some pairs of images and annotated descriptions in PASCAL VOC 2012 dataset. Images in the first row are sampled from training set,
while the second row’s images are collected from the validation set.
Subset
Num. of Image
mean IoU

(i)
766
35.94%

(ii)
266
33.19%

(ii)
417
34.70%

TABLE 15
Results on different subset of VOC 2012 val under the weakly
supervised learning.

Fig. 13. The number of object category of each image in VOC train and
val dataset. The abscissa indicates the number of object categories in
the image. The ordinate indicates the number of images. In each image,
the number of interaction relations usually increases with the number of
objects growing.
300

Training Set

Validation Set

250

200

150

100

50

0

beside

lie

hold

ride

behind

sit

in front

on

Fig. 14. The number of occurrences of each relation category in the train
and val dataset. Note that each image may contain multiply relations.

Fig. 14 reports the number of occurrences of each relation
category in VOC train and val dataset. The most common relation
label is “beside”, and the number of its occurrences is 236 in
training set and 245 in validation set. The label “lie” and “hold”
are two least common labels, and occurrences times are around 20
in both training and validation set.
6.2

Experiment Results

Analysis on Relation Loss. We note that the RsNN model in
previous works (e.g., Socher et al. [10]) only consider the structure
supervision, but our model takes both structure and relation

supervision during model training. To evaluate the performance
of our method with and without relation supervision, we add
some visualized results in Fig. 15. According to the figure, one
can see that both of two methods learn the correct combination
orders. However, our method can further predict the interaction
relation between two merged object regions. More importantly,
the relation loss can also regularize the training process of CNN,
which makes the segmentation model more effective to discover
the small objects and eliminate the ambiguity.
Analysis on Category Level Description. Instead of instancelevel parsing, this work aims to learn a CNN-RsNN model
for category-level scene parsing. When asking the annotator to
describe the image, some guidelines are introduced in Sec.6.1
to avoid instance-level descriptive sentences. Under such circumstances, it is interesting to ask whether such annotation strategy are
harmful to semantic labeling on images with multiple instances.
To answer this, we divide the VOC 2012 val set into three
subsets: (i) images with one instance from one object category,
(ii) images with instances from multiple object categories, but
only one instances from each category, and (iii) the others. The
mean IoU of our model on these three subsets are reported in
Table 15. Although the number of object categories per image, the
number of instances per category, and the number of images have
the obvious difference among three subsets, the changes of mIoU
remain in a small range. It demonstrates that our category-level
descriptions have little negative effect on semantic labeling results
of images with multiple instances.
Analysis on Parsing Results. To further investigate the performance of structure prediction, we provide some typical successful

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

and failure cases of scene structure prediction in Fig. 16 and
Fig. 17. All of them are generated under the weakly supervised
setting as described in the main paper.
We first show some successful parsing results in Fig. 16. It
is interesting to note that, our scene structure generation model
is robust to small degree of semantic labeling error. As in the
left image of the last row, even only a small part of the person
is correctly labeled, both structure and relation prediction can be
successfully predicted. The relation categories in these examples
cover most of the defined relations in this article. Then, the failure
cases are illustrated in Fig. 17. According to this figure, the failure
predictions usually happen in the following three cases. (i) All
of the structure and relation predictions are incorrect. Fig. 17(a) and Fig. 17-(c) illustrate such situation. (ii) The structure is
correct but the predicted relations are wrong. Fig. 17-(b) gives the
example like this. (iii) Both the structure and relation predictions
are partially correct. Fig. 17-(d) gives the example in such case.
According to the above discussion, one can see that the
main cause of failure is the semantic labeling error, including
seriously inaccurate labeling and complete failure in segmenting
some object category. Moreover, when the semantic labeling is
inaccurate, the relation tends to be wrongly predicted as others
(see Fig. 17-(a)(b)(c)). When some object category is completely
failed to be recognized, structure prediction is likely to be incorrect
or partially incorrect (see Fig. 17-(a)(d)).

R EFERENCES
[1]
[2]

[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]

F. Han and S. C. Zhu, “Bottom-up/top-down image parsing with attribute
grammar,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp.
59–73, 2009.
S. Wang, Y. Wang, and S.-C. Zhu, “Learning hierarchical space tiling
for scene modeling, parsing and attribute tagging,” IEEE transactions
on pattern analysis and machine intelligence, vol. 37, no. 12, pp. 2478–
2491, 2015.
V. S. Lempitsky, A. Vedaldi, and A. Zisserman, “Pylon model for
semantic segmentation,” in Proc. Adv. Neural Inf. Process. Syst., 2011,
pp. 1485–1493.
J. Tighe and S. Lazebnik, “Superparsing - scalable nonparametric image
parsing with superpixels,” Int. J. Comput. Vis., vol. 101, no. 2, pp. 329–
349, 2013.
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2015, pp. 3431–3440.
A. Sharma, O. Tuzel, and M. Liu, “Recursive context propagation
network for semantic scene labeling,” in Proc. Adv. Neural Inf. Process.
Syst., 2014, pp. 2447–2455.
J. Bobrow, “Representation and understanding: Studies in cognitive
science,” 2014.
Z. Si and S.-C. Zhu, “Learning and-or templates for object recognition
and detection,” IEEE transactions on pattern analysis and machine
intelligence, vol. 35, no. 9, pp. 2189–2205, 2013.
F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphs and
the sum-product algorithm,” IEEE Transactions on information theory,
vol. 47, no. 2, pp. 498–519, 2001.
R. Socher, C. C. Lin, A. Y. Ng, and C. D. Manning, “Parsing natural
scenes and natural language with recursive neural networks,” in Proc.
Int. Conf. Mach. Learn., 2011, pp. 129–136.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., 2012, pp. 1106–1114.
A. Sharma, O. Tuzel, and D. W. Jacobs, “Deep hierarchical parsing
for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2015, pp. 530–538.
R. Socher, C. D. Manning, and A. Y. Ng, “Learning continuous phrase
representations and syntactic parsing with recursive neural networks,” in
Deep Learning and Unsupervised Feature Learning Workshop, 2010.
A. Karpathy and F. Li, “Deep visual-semantic alignments for generating
image descriptions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
2015, pp. 3128–3137.

15

[15] J. Xu, A. G. Schwing, and R. Urtasun, “Tell me what you see and I
will show you where it is,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2014, pp. 3190–3197.
[16] R. Socher, J. Bauer, C. D. Manning, and A. Y. Ng, “Parsing with
compositional vector grammars,” in Proc. Ann. Meet. Assoc. Comput.
Linguis., 2013, pp. 455–465.
[17] G. A. Miler, R. Beckwith, C. Fellbuan, D. Gross, and K. Miller,
“Introduction to word net,” An Online Lexical Database, 1993.
[18] V. Ferrari and A. Zisserman, “Learning visual attributes,” in Proc. Adv.
Neural Inf. Process. Syst., 2007, pp. 433–440.
[19] C. Desai, D. Ramanan, and C. C. Fowlkes, “Discriminative models for
multi-class object layout,” Int. J. Comput. Vis., vol. 95, no. 1, pp. 1–12,
2011.
[20] B. Yao, G. R. Bradski, and F. Li, “A codebook-free and annotation-free
approach for fine-grained image categorization,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2012, pp. 3466–3473.
[21] X. Ren and J.Malik, “Learning a classification model for segmentation,”
in Proc. Int. Conf. Comput. Vis., 2003, pp. 10–17.
[22] B. Fulkerson, A. Vedaldi, and S. Soatto, “Class segmentation and object
localization with superpixel neighborhoods,” in Proc. Int. Conf. Comput.
Vis., 2009, pp. 670–677.
[23] M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich, “Feedforward
semantic segmentation with zoom-out features,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2015, pp. 3376–3385.
[24] J. Tighe, M. Niethammer, and S. Lazebnik, “Scene parsing with object
instances and occlusion ordering,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., 2014, pp. 3748–3755.
[25] Z. Tu, X. Chen, A. L. Yuille, and S.-C. Zhu, “Image parsing: Unifying
segmentation, detection, and recognition,” Int. J. Comput. Vis., vol. 63,
no. 2, pp. 113–140, 2005.
[26] X. Liu, Y. Zhao, and S. Zhu, “Single-view 3d scene parsing by attributed
grammar,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp.
684–691.
[27] L. Zhu, Y. Chen, Y. Lin, C. Lin, and A. Yuille, “Recursive segmentation
and recognition templates for image parsing,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 34, no. 2, pp. 359–371, 2012.
[28] N. Ahuja and S. Todorovic, “Connected segmentation tree—a joint
representation of region layout and hierarchy,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2008, pp. 1–8.
[29] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation
and support inference from rgbd images,” in Proc. Eur. Conf. Comput.
Vis., 2012, pp. 746–760.
[30] S. Gupta, P. Arbelaez, and J. Malik, “Perceptual organization and
recognition of indoor scenes from rgb-d images,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2013, pp. 564–571.
[31] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille,
“Semantic image segmentation with deep convolutional nets and fully
connected crfs,” Arxiv Preprint, 2015.
[32] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du,
C. Huang, and P. H. S. Torr, “Conditional random fields as recurrent
neural networks,” Arxiv Preprint, 2015.
[33] Z. Liu, X. Li, P. Luo, C. C. Loy, and X. Tang, “Semantic image
segmentation via deep parsing network,” in Proc. Int. Conf. Comput.
Vis., 2015, pp. 1377–1385.
[34] A. Vezhnevets, V. Ferrari, and J. M. Buhmann, “Weakly supervised
structured output learning for semantic segmentation,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., 2012, pp. 845–852.
[35] D. Pathak, E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional
multi-class multiple instance learning,” Arxiv Preprint, 2014.
[36] G. Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille, “Weakly- and
semi-supervised learning of a deep convolutional network for semantic
image segmentation,” in Proc. Int. Conf. Comput. Vis., 2015, pp. 1742–
1750.
[37] D. Pathak, P. Krähenbühl, and T. Darrell, “Constrained convolutional
neural networks for weakly supervised segmentation,” in Proc. Int. Conf.
Comput. Vis., 2015, pp. 1796–1804.
[38] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” Arxiv Preprint, 2014.
[39] S. Xie and Z. Tu, “Holistically-nested edge detection,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1395–1403.
[40] S. Boyd and L. Vandenberghe, Convex optimization.
Cambridge
university press, 2004.
[41] D. Tarlow, K. Swersky, R. S. Zemel, R. P. Adams, and B. J. Frey, “Fast
exact inference for recursive cardinality models,” in Proc. Conf. Uncert.
Artif. Intell., 2012, pp. 825–834.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

[42] Y. Li and R. S. Zemel, “High order regularization for semi-supervised
learning of structured output problems,” in Proc. Int. Conf. Mach. Learn.,,
2014, pp. 1368–1376.
[43] P. O. Pinheiro and R. Collobert, “From image-level to pixel-level labeling
with convolutional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2015, pp. 1713–1721.
[44] L. Lin, G. Wang, R. Zhang, R. Zhang, X. Liang, and W. Zuo, “Deep
structured scene parsing by learning with image descriptions,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., 2016.
[45] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams,
J. Winn, and A. Zisserman, “The pascal visual object classes challenge:
A retrospective,” Int. J. Comput. Vis., vol. 111, no. 1, pp. 98–136, 2015.
[46] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen,
Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and F. Li, “Visual
genome: Connecting language and vision using crowdsourced dense
image annotations,” Arxiv Preprint, 2016.
[47] J. L. Elman, “Distributed representations, simple recurrent networks, and
grammatical structure,” Machine Learning, vol. 7, pp. 195–225, 1991.
[48] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for
fast feature embedding,” in Proc. ACM Conf. on Multimedia, 2014, pp.
675–678.
[49] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik, “Semantic
contours from inverse detectors,” in Proc. Int. Conf. Comput. Vis., 2011,
pp. 991–998.
[50] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, “Simultaneous
detection and segmentation,” in Proc. Eur. Conf. Comput. Vis., 2014,
pp. 297–312.
[51] J. Dai, K. He, and J. Sun, “Instance-aware semantic segmentation via
multi-task network cascades,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2016, pp. 3150–3158.
[52] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask R-CNN,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017.
[53] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards
real-time object detection with region proposal networks,” IEEE Trans.
Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, 2017.

Ruimao Zhang is currently a postdoctoral fellow in the Department of Electronic Engineering, The Chinese University of Hong Kong
(CUHK), Hong Kong, China. He received the
B.E. and Ph.D. degrees from Sun Yat-sen University (SYSU), Guangzhou, China in 2011 and
2016, respectively. From 2013 to 2014, he was
a visiting Ph.D. student with the Department of
Computing, Hong Kong Polytechnic University
(PolyU). His research interests include computer
vision, deep learning and related multimedia applications. He currently serves as a reviewer of several academic journals, including IEEE Trans. on Neural Networks and Learning Systems,
IEEE Trans. on Image Processing, IEEE Trans. on Circuits and Systems
for Video Technology, Pattern Recognition and Neurocomputing.

Liang Lin (M’09, SM’15) is the Executive R&D
Director of SenseTime Group Limited and a full
Professor of Sun Yat-sen University. He is the
Excellent Young Scientist of the National Natural Science Foundation of China. From 2008
to 2010, he was a Post-Doctoral Fellow at University of California, Los Angeles. From 2014 to
2015, as a senior visiting scholar, he was with
The Hong Kong Polytechnic University and The
Chinese University of Hong Kong. He currently
leads the SenseTime R&D teams to develop
cutting-edges and deliverable solutions on computer vision, data analysis and mining, and intelligent robotic systems. He has authorized
and co-authorized on more than 100 papers in top-tier academic journals and conferences. He has been serving as an associate editor
of IEEE Trans. Human-Machine Systems, The Visual Computer and
Neurocomputing. He served as Area/Session Chairs for numerous conferences such as ICME, ACCV, ICMR. He was the recipient of Best
Paper Runners-Up Award in ACM NPAR 2010, Google Faculty Award
in 2012, Best Paper Diamond Award in IEEE ICME 2017, and Hong
Kong Scholars Award in 2014. He is a Fellow of IET.

16

Guangrun Wang is currently pursuing the Ph.D.
degree in the School of Data and Computer
Science, Sun Yat-sen University. He received
the B.E. degree from Sun Yat-sen University,
Guangzhou, China, in 2014. From Jan 2016
to Aug 2017, he was a visiting scholar with
the Department of Information Engineering, the
Chinese University of Hong Kong. His research
interests include computer vision and machine
learning.

Meng Wang is a professor at the Hefei University of Technology, China. He received his B.E.
degree and Ph.D. degree in the Special Class
for the Gifted Young and the Department of
Electronic Engineering and Information Science
from the University of Science and Technology
of China (USTC), Hefei, China, in 2003 and
2008, respectively. His current research interests
include multimedia content analysis, computer
vision, and pattern recognition. He has authored
more than 200 book chapters, journal and conference papers in these areas. He is the recipient of the ACM SIGMM
Rising Star Award 2014. He is an associate editor of IEEE Transactions
on Knowledge and Data Engineering (IEEE TKDE), IEEE Transactions
on Circuits and Systems for Video Technology (IEEE TCSVT), and IEEE
Transactions on Neural Networks and Learning Systems (IEEE TNNLS).

Wangmeng Zuo received the Ph.D. degree in
computer application technology from the Harbin
Institute of Technology, Harbin, China, in 2007.
He is currently a Professor in the School of Computer Science and Technology, Harbin Institute
of Technology. His current research interests include image enhancement and restoration, object detection, visual tracking, and image classification.He has published over 60 papers in toptier academic journals and conferences.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

17

Fig. 15. Some visualized semantic segmentation and scene structure prediction results with and without relation loss on PASCAL VOC 2012
val dataset. The first column shows the input images, the ground truth of semantic labeling and semantic trees. The second column gives the
segmentation and structure prediction results with the relation loss (our method). In contract, the results without the relation loss are illustrated in
the last column (like Socher et al. method).

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

18

Four men are standing
on the top of the plane

A woman is standing
beside a brown horse

on

beside

beside

on
plane

person
person

horse

person
horse

plane

person

The dining table is
in front of a boy
A man holds a
bottle of wine
behind
hold

hold

person

table

in front of

bottle

person

table
person

person

bottle

A cow is standing in
front of the white car
beside the people
beside

A girl rides a red
motorbike beside
some potted plants

beside

in front of
cow

person

beside
in front of

person

beside

car
ride
cow

car

plant

person motorbike

plant
ride

person

motorbike

Fig. 16. The visualized successful scene parsing results in PASCAL VOC 2012 dataset under the weakly supervised setting.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A brown dog and a
grey cat are lying on
the sofa

19

The sofa are placed
beside two chairs in
front of the tv
other

lie
beside

in front of

other

beside

sofa

monitor

other
monitor

cat

dog

dog

sofa

chair

sofa

chair

sofa
(a)

(b)

The baby holding a
bottle is standing
beside the chair in
front of the car

A cat is standing
on the table next
to the chair

beside

in front of
beside
beside

other
on

car

hold

chair
hold

cat

car

chair

table
person
cat
(c)

bottle

person

chair
(d)

Fig. 17. The visualized failure scene parsing results in PASCAL VOC 2012 dataset under the weakly supervised setting.

bottle

