1

Simultaneously Color-Depth Super-Resolution with
Conditional Generative Adversarial Network

arXiv:1708.09105v2 [] 8 Nov 2017

Lijun Zhao, Huihui Bai, Member, IEEE, Jie Liang, Senior Member, IEEE, Bing Zeng, Fellow, IEEE,
Anhong Wang, Member, IEEE, and Yao Zhao, Senior Member, IEEE

Abstract—Recently, Generative Adversarial Network (GAN)
has been found wide applications in style transfer, image-to-image
translation and image super-resolution. In this paper, a colordepth conditional GAN is proposed to concurrently resolve the
problems of depth super-resolution and color super-resolution
in 3D videos. Firstly, given the low-resolution depth image and
low-resolution color image, a generative network is proposed to
leverage mutual information of color image and depth image to
enhance each other in consideration of the geometry structural
dependency of color-depth image in the same scene. Secondly,
three loss functions, including data loss, total variation loss, and
8-connected gradient difference loss are introduced to train this
generative network in order to keep generated images close to the
real ones, in addition to the adversarial loss. Experimental results
demonstrate that the proposed approach produces high-quality
color image and depth image from low-quality image pair, and
it is superior to several other leading methods. Besides, we use
the same neural network framework to resolve the problem of
image smoothing and edge detection at the same time.
Index Terms—Depth image, color image, generative
adversarial network, super-resolution, image smoothing, edge
detection.

I. I NTRODUCTION

L

OW-RESOLUTION and noisy images are always
annoying for a variety of practical applications such as
image and video display, surveillance, to name a few. In
order to enlarge image’s resolution and enhance the quality
of super-resolution image, a tremendous amount of works
have been developed in the field of color super-resolution
(SR) for several decades [1, 2]. Recently several
convolutional neural network (CNN) based methods such as
[3–5] have reported better super-resolution results and have
an order of magnitude lower complexity than previous
methods.
One of the earliest CNN-based super-resolution works is
three-layer convolutional neural network, named by SRCNN
in [6]. Latter, the deconvolution operation is used in [3] to
directly learn the projection from low resolution (LR) image
L. Zhao, H. Bai, Y. Zhao are with Institute Information Science, Beijing
Jiaotong University, Beijing, 100044, P. R. China, e-mail: 15112084, hhbai,
yzhao@bjtu.edu.cn.
J. Liang is with School of Engineering Science, Simon Fraser University,
ASB 9843, 8888 University Drive, Burnaby, BC, V5A 1S6, Canada, email:jliang@sfu.ca
B. Zeng is with Institute of Image Processing, University of Electronic
Science and Technology of China, Chengdu, Sichuan 611731, China, email:eezeng@uestc.edu.cn
A. Wang is with Institute of Digital Media & Communication, Taiyuan
University of Science and Technology, Taiyuan, 030024, P. R. China, email:wah ty@163.com

to high-resolution (HR) image. In [4], an efficient sub-pixel
convolution layer is introduced to learn a series of filters to
project the final LR feature maps into HR image. Different
from the shallow neural network in [3, 4, 6], a very deep
neural network is presented in [5] to learn image’s residuals
with extremely high learning rates. However, these methods’
objective functions are always the mean squared SR errors,
so their SR output images usually fail to have
high-frequency details when the up-sampling factor is large.
In [7, 8], a generative adversarial network is proposed to
infer photo-realistic images in terms of the perceptual loss.
In addition to the single image SR, image SR aided with its
neighboring viewpoint’s high/low resolution image has also
been widely explored to further improve the quality. For
instance, high-frequency information from the neighboring
full-resolution views and corresponding depth image are used
to enhance the low-resolution view images in [9]. Except
mixed resolutions, the multiple LR stereo observations are
leveraged to increase image’s resolution [10].
Due to depth information’s facilities to many real-world
applications, depth SR problems have been widely explored
in recent years. When only LR depth image is given, this SR
problem is called single depth super-resolution. But, if the
LR depth image is available accompanied with corresponding
HR color image, researchers often name this kind problem
of SR after joint depth SR or color image-guided SR. In
[11], by searching a list of HR candidate patches from the
database to match with the LR patches, the problem of depth
SR is transformed into Markov random field (MRF) labeling
problem to reconstruct the full HR image. After that, single
depth SR is decomposed as two-step procedures: first the HR
edge map is synthesized with HR patches according to the
MRF optimization problem; and then a modified joint
bilateral filtering is employed to achieve image up-sampling
with this HR edge map [12]. Recently, depth
super-resolution is formulated as the boundary compensation
problem by learning multiple residual dictionaries and an
adaptive depth map refinement method is followed to remove
the ringing artifact around the edges after multiple residual
compensation [13].
Since the HR color image can be easily got by the
consumer camera sensors in most cases, the color image can
be used as an available prior information to upscaling the LR
depth image, under the assumption of structural similarity
between color image and depth image. Here, we just classify
joint depth SR approaches into three classes: filtering-based
methods, optimization methods and CNN-based SR methods.

2

For example, bilateral filtering and guided image filtering are
often used to get the interpolation weights to resolve the
problem of depth SR [14–16]. The joint bilateral filtering in
[14] use color image as a prior to guide the up-sampling
from low quality to high quality. Meanwhile, bilateral
filtering is iteratively used to refine the input low-resolution
depth image in [15] to improve the spatial resolution and
depth precision. In order to prevent texture-copy artifacts
from color image and against the inherent noisy nature of
real-time depth data, an adaptive multi-lateral up-sampling
filter in [16] is described to up-sample depth information. In
[17], a more advanced filtering is called guided filtering,
whose ambition is to transfer the structure from a guidance
image into a target image. From this paper, it can be seen
that depth super-resolution can be treated as structural
transfer filtering.
The second class of joint depth super-resolution methods
often build their model by converting SR problems into the
convex and non-convex optimization with different prior
knowledge to regularize the objective function. For example,
a MRF-based model [18], which consists of data term and
smoothness prior term, is built up to align the discontinuities
of depth image with color image’s boundaries. However, this
model always suffers from the texture-copy artifacts and
depth bleeding artifacts, when color image could not be used
sufficiently during depth image super-resolution. Thus, to
sharpen depth boundaries and to prevent depth bleeding, a
nonlocal means term is incorporated into the MRF model to
help local structure to be preserved [19]. To suppress
texture-copy artifacts and reduce computational cost, variable
bandwidth weighting scheme [20] is used into the MRF
model to adjust the guidance weights based on depth local
smoothness. These methods such as [19, 20] implicitly put
the inconsistency between the depth image and the color
image into the smoothness term of MRF model. Later, a
unified framework proposes to cast guided interpolation into
a global weighted least squares optimization framework [21].
In [22], the higher order regularization is used to formulate
depth image up-sampling as a convex optimization problem.
In [23], a static and dynamic filter (SDF) is designed to
address the problem of guided image filtering by using
structural information jointly from the guidance image and
input image.
Although these recent advanced techniques achieve some
appealing performances, they are built on the complex
optimization algorithms using hand-designed objective
functions, which always have high complexity of
computation and limit their widely practical applications.
Lately, deep joint image filtering framework based on
convolutional neural network is proposed in [24] to
adaptively transfer co-occurrence information from the
guidance image to the target images. Meanwhile, in order to
adaptively up-sample depth image’s small-scale and
large-scale structures, a multi-scale guided convolutional
network is trained in high-frequency domain for up-sampling
depth map [25].
In many cases, LR depth image can be available and
accompanied with HR color image in the dynamic scene, so

the majority of SR works put their emphasis on HR color
image aided depth super-resolution. But they often lose sight
of the significance of simultaneously depth image and color
image SR with deep learning. As a matter of fact, this task
is very significant for several 3D video applications, such as
3D video compression, 3D scene reconstruction. For
example, the 3D-HEVC [26] has leveraged the
full-resolution color video and depth video with multi-view
video plus depth (MVD) format to compress 3D video. If
the techniques of simultaneous depth and color SR can be
put into the 3D-HEVC framework, apparently their coding
efficiency can be greatly improved. From our investigation,
we find some works such as [27] have embedded the
CNN-based SR into HEVC coder to achieve significant bits
saving, so the research of simultaneous depth and color
image SR is a meaningful and important topic for both
industry and academia.
Recently, generative adversarial networks [28] is used to
generate high-quality image to achieve the tasks of
super-resolution
and
image
style
transfer,
and
image-to-image transfer [8]. In [8], perceptual loss function
is applied on the tasks of image transformation such as
image style transfer by training feed-forward networks. In
[29], a general solution to the problem of image-to-image
translation is proposed to finish a lot of tasks, such as
synthesizing a new image from the label map, reconstructing
a scene image from an edge map and so on.
Following the works of [8, 29], we propose to use
color-depth conditional generative adversarial network
(CDcGAN) to deal with challenging tasks of both color SR
and depth SR at the same time. Our generative network
consists of five components: color enhancement subnetwork,
depth enhancement subnetwork, color-depth feature merge
subnetwork, color image reconstruction subnetwork and
depth image reconstruction subnetwork. First, we
respectively enhance the quality of low-quality color-depth
image in the first two subnetworks and then these features
are incorporated by color-depth feature merge subnetwork,
which is inspired by the literature [24]. But, there are some
differences between them, which will be detailed later. After
that, color feature and/or depth feature feed into the last two
reconstruction subnetwork in addition to the merged
depth-color features in order to produce HR color-depth
images at the same time. Secondly, one discriminator is used
to distinguish real color image from the generated color
image. The reasons for training depth reconstruction
subnetwork without a discriminator to judge comes from that
depth image is not used to directly displayed on the screen,
but it is always used as scene’s geometry information to
direct the rendering of virtual images with each pixel of
depth image representing the distance between camera and
object. Thus, only three auxiliary losses are added to
regularize depth image SR. Thirdly, in our generative
network, three additional loss: data loss, Total Variation (TV)
loss, and 8-connected gradient difference loss are also used
for color image in addition to adversarial loss, so as to
ensure that image pairs produced by the generator are similar
to the true image pairs.

3

The rest of our paper is organized as follows. First, our
approach is given in Section 2. After experimental results are
evaluated in Section 3, we draw a conclusion in Section 4.
II. P ROPOSED M ETHOD
A. Networks Architecture
Given the LR color-depth image pair (c, d), we propose to
use the conditional GAN to generate the HR color-depth
image pair (x, y). To the best of our knowledge, this is the
first deep learning-based color-depth super-resolution
scheme. As illustrated in Fig. 1 and Fig. 2, our conditional
generative adversarial network consists of a generator
network (G) and a discriminator network(D). First, the
low-resolution color-depth images are initialized by Bicubic
interpolation to be full-resolution. Then, our proposed
generative network respectively feeds low-quality yet
full-resolution depth image d and color image c into color
enhancement subnetwork (S1) and depth enhancement
subnetwork (S2) to pre-filter the low-quality images, as
displayed in Fig. 1. In addition, the enhanced depth image
and enhanced color image are fed into color-depth feature
merge subnetwork (S3). Finally, the first two subnetworks
features and color-depth merged features are leveraged to
reconstruct HR color image and depth image with color
reconstruction subnetwork (S4) and depth reconstruction
subnetwork (S5) respectively. In particular, the generator G
has two subnetworks S4 and S5 to produce image pair
(x, y) from the low-quality image pair (c, d). In the
reconstruction subnetwork S4, the depth feature maps from
S1, S2, and S3 are chosen to generate HR color image.
However, HR depth images are convolved only with
enhanced depth image from S2 without enhanced color
image, in addition to the feature map from feature merge
subnetwork. In other words, the skip-connection is chosen
for both color and depth reconstruction, but for depth
image’s reconstruction, one skip-connection is used in order
to make the depth features only affected by depth features
and mutual features shared by color image and depth image
by considering depth super-resolution’s sensitivity to texture
details of color image. In [30], the skip-connection has been
successfully used for a semantic segmentation network.
Here, a similar idea about skip-connection is shared by
color-depth image super-resolution.
In each subnetwork, we use three convolutional layers
similar to [6]. The advantage of this network lies in the
middle convolutional network with kernel size of 1x1 in
spatial domain so that the parameters of the networks could
greatly decreased, while guaranteeing the nonlinear of this
neural network. The S1/S2 three convolutional layers are
presented as follows: 9x9x1x96; 1x1x96x48; and 5x5x48x1
respectively. The S3 convolutional layers are is listed as:
9x9x2x64; 1x1x64x32; and 5x5x64x2. In addition, the S4/S5
convolutional layers are 9x9x3x9; 1x1x96x48 and 5x5x48x1.
The convolutional layers of S1, S2, and S3 are operated with
stride of 1 and padding of 1, but the ones of S4 and S5 are
processed without padding in order to keep the output
image’s size same as the ground truth image. All the

convolutional layers are followed by the activation function
of ReLU, except each last convolutional layer of S1, S2, S3,
S4, and S5.
Except the SRCNN network [6], there are many other
choices for each component of our generative network. For
example, modified VGG deep networks [31], can be used for
S1 and S2 subnetworks. In [32], all the convolution kernel
size in spatial domain is 3x3 for image de-noising with
modified VGG deep networks. There are some alternative
networks for S1 and S2 subnetworks of our generator, such
as the Encoder-decoder network or U-net used in [29]. In
addition, two reconstruction subnetwork of our network can
alternatively choose the neural networks with deconvolution
or sub-pixel convolutional to reduce the parameter of
networks and corresponding computational costs [3, 4]. In
order to form a general neural network for image processing,
no deconvolution and sub-pixel convolutional is used in this
paper, because full-resolution image processing with deep
neural network is more general than low-resolution image
super-resolution and SR problem can be easily initialized to
the full-resolution image as the input of neural network.
Note that, the part network composed of S1, S2 and S3
subnetwork in our generator looks like network in [24], but
the functionality of this part network is different from [24],
where the CN NG is used to get the edge information of
corresponding color image and CN NT is used to initially
reconstruct the depth image. In our generator, both S1 and
S2 subnetworks are respectively regarded as a pre-filtering
network of low-quality color and depth image, which is
followed by feature merge subnetwork and reconstruction
subnetworks.
As depicted in Fig. 1, the discriminator is a three-layer
convolutional neural network. The parameters are
respectively 4x4x3x64 with stride=2, 4x4x64x64 with
stride=2, and 5x5x64x1 with stride=1. In the discriminator
network, the first two convolution layers are followed by
Leaky ReLU activation function, while the last layer is
activated by sigmoid function.
The generated color image x produced by G is used to fool
the discriminator D. The discriminator is trained to identify the
false images from the true image so that the generator tends to
create more realistic image. Note that only one discriminator
is used in our generative adversarial network. Actually, we
can use two discriminators to distinguish the true image pairs
from the false ones. However, the depth image is not watched
directly and keeping the accuracy of the depth image is the
major task for depth super-resolution, so there is no adversarial
loss for depth reconstruction subnetwork (S5).
B. Objective Function
In our objective function, there is no adversarial loss for
the depth image. Instead, three auxiliary losses are
considered to make the generated depth image close to the
truth image. Contrary to depth images, which only contain
sharp boundaries and some flat or piece-wise smooth
regions, color images usually have more informative textural
details. So it is important for color images to be more

4

Fig. 1. The diagram of color-depth conditional generative adversarial network(CDcGAN)

the probability of data from G(c, d, z) is represented as
D(c, d, G(c, d, z)).
In our model, the adversarial loss is expressed as follows:
LCDcGAN (G, D) = Ecg ∼pdata (cg ) log(D(c, d, cg ))+
Ex∼pdata (x),z∼pz [log(1 − D(c, d, G(c, d, z)))]

(2)

in which z is random noise.
D. Auxiliary Losses for Regularization

Fig. 2. The workflow of color-depth conditional generative adversarial

network(CDcGAN)

realistic compared to the true image, especially when the
up-sampling factor is very large.
In a summary, the objective of our model can be expressed
as follows:
G∗ = min max α · LCDcGAN (G, D)
D

G

(1)

+(Ldata (G) + LT V (G) + LGD (G)),
where LCDcGAN (G, D) is the adversarial loss and the others
are three auxiliary losses. They will be defined later. Here,
the parameter α is used to adjust the contribution for colordepth super-resolution between the GAN loss and three other
auxiliary losses.
C. Adversarial Loss
For brevity of latter description, we denote the true color
image data’s distribution and generated color image data’s
distribution as pdata (cg ) and pdata (x), while pz is input
noise’s distribution. As shown in Fig. 1, the generator
G(c, d, z) is used as a mapping from the LR image set (c, d)
to HR image one (cg , dg ). D(c, d, cg ) describes the
probability that cg comes from the true image data rather
than image data produced by the generator G(c, d, z), while

In our objective function, three auxiliary losses are included:
data loss, TV loss, and 8-connected gradient difference loss,
which are leveraged to make image pair (x, y) produced by
the generator G to be similar enough to the true image pair
(cg , dg ). The vectors of (x, y) and (cg , dg ) are represented as
(X, Y ), and (Cg , Dg ). Like the traditional TV model, the
data loss keeps the output’s value consistent to the ground truth
value, while the TV loss LT V (G) emphasizes the correlation
of the output’s values with its neighboring pixel’s value in
order to keep generated image to be smooth and robust against
noises. Our data loss function Ldata (G), including both color
image’s data loss and depth image’s data loss, is defined as
follows:
1 X
(||X(i) − Cg (i)||L
Ldata (G) =
M ·N i
(3)
+||Y (i) − Dg (i)||L ),
where || · ||L represents the L norm and the size of true image
pair is M · N .
Our TV loss function is defined as follows:
1 X
LT V (G) =
((||∇x X(i)||L +
M ·N i
(4)
||∇y X(i)||L ) + (||∇x Y (i)||L + ||∇y Y (i)||L ))
where ∇x , and ∇y are the gradients in the x-direction and
y-direction.
The 8-neighboring gradient difference (GD) loss is used to
make generated image pair’s gradient information to be similar

5

to that of the ground truth image in the gradient domain. The
8-neighboring GD loss is defined as follows:
1 X X
LGD (G) =
((
||∇k X(i)) − ∇k Cg (i)||L )+
M ·N i
k∈Ω
X
(
||∇k Y (i) − ∇k (Dg (i))||L ))
k∈Ω

(5)
where Ω is each pixel’s neighbourhood, and ∇k is the k-th
gradient between each pixel and k-th pixels among
8-neighbouring pixels.
In some literatures [33], it has been reported that L2 loss
always leads to image blurring. In [25, 29], the traditional loss,
such as L1 distance, has been added into the GAN objective
function, in which the generator aims to not only fool the
discriminator but also to make sure that generated samples
move towards the ground truth in an L1 sense. Thus, in the
proposed neural network, the L1 loss is used for three auxiliary
losses to keep generated sample close enough to the real one,
and make generated images to be sharp rather than blurring.
E. Other Applications
Our proposed network is not restricted to finish the task
of depth-color super-resolution. In fact, similar networks can
be specifically designed for different tasks, e. g. simultaneous
edge detection and semantic segmentation, concurrent edge
detection and image smoothing, and even finishing these three
tasks at the same time, when corresponding networks have one
input (e.g., color image) or two inputs (e.g. color image and
depth image).
For image smoothing and edge detection at the same time,
we change the output feature numbers of each subnetwork’s
last convolutional layer and each convolution layer is padded
to be the same size as the input features. For example, two
inputs in our network respectively have three channels and
six channels, then the last convolutional layer output feature
map numbers in the the S1 and S2 / in S4 and S5 will be 3
and 6 respectively. Here, one input image is color image,
while another one is composed of three gradient map of
color image in both vertical and horizonal direction. Note
that training image smoothing use adversarial loss, content
loss, TV loss, and gradient loss, but training edge detection
only employs content loss, because the edge information is
vulnerable to the regularization of adversarial loss, TV loss,
and gradient loss.
In fact, it can be extended into multiple inputs and
multiple outputs with one network. In many cases, several
images of a scene with different modalities or different
lighting conditions are observed at the same time, so one or
more images in other modalities are desire to be generated,
when some modal images are known.
III. E XPERIMETAL RESULTS AND ANALYSIS
To validate the efficiency of the proposed architecture for
super-resolution, we compare our method with the Bicubic
interpolation method and SRCNN [6] for color image SR. In
addition, for depth super-resolution, not only the results of

single depth SR with SRCNN [6] are given, we also compare
our approach with several joint depth super-resolution
methods such as GIF [17], FGS [21], RGIF [23], TGV [22],
RGDR [20], HQDU [22], MRF [18]. Three measurements of
image quality, e.g. Peak Signal-to-Noise Ratio (PSNR),
Structural SIMilarity (SSIM) index, and image sharpness
[33], are used for the comparison of different methods.
Finally, we also use our architecture to learn filters for
simultaneously image smoothing and edge detection.
A. Implementation details
Our architecture of simultaneous color-depth image superresolution is implemented by TensorFlow [34] including about
200 thousand parameters, but the generator only use 92358
parameters. We train our neural network with 100,000 image
color-depth patches with size 32 × 32 from 90 color-depth
image pairs. In our training dataset, 52 color-depth image pairs
come from the Middlebury dataset, and the remaining colordepth image pairs are got from the MPI Sintel color-depth
dataset. In our model, α equal to 0.002. We train our model
for 30 epochs using Adam, where the beta1=0.5, beta2=0.999,
and the learning rate is set to be 0.0002. Note that the hyperparameters of beta1 and beta2 of Adam control the exponential
decay rates of moving averages, whose details can be found in
[35]. During training, the the parameters of the discriminator
D are updated by Adam, which is followed by the updating
of generator’s ones. After alternative training the generator G
and discriminator D up to Nash equilibrium, the generator G
becomes powerful to produce high-quality image, as shown in
Fig. 2.
In order to further validate the efficiency of our
architecture, we use the same architecture in Fig. 1 to learn
filters for image smoothing and edge detection at the same
time. we use the BSDS500 dataset from Berkeley Computer
Vision Group and corresponding smoothed image with L0
gradient minimization in [36] as our training data for image
smoothing and edge detection. we augment data for training
by rotating image. Specifically, 100,000 patches with size
64 × 64 are extracted from these augmented data. We train
our model for image smoothing and edge detection with 10
epochs. The other training parameters are set the same as the
ones described above in simultaneous color and depth
super-resolution.
B. The objective
super-resolution

and

visual

quality

comparison

for

We use five standard 3D sequences to show the efficiency
of the proposed method. The five testing color-depth
sequences of first 100 frames contain Love Bird (denoted as
L), Book Arrival (B), and Newspaper (N) with resolution of
768x1024, Shark (S), and Undo dancer (U) with size of
1920x1088. Three objective quality of both color SR and
depth SR are evaluated in terms of PSNR, SSIM, and
sharpness. The comparative results are displayed in Table
1-3, where PSNR, SSIM, and sharpness are respectively
denoted as M1, M2, M3.

6

TABLE I
T HE OBJECTIVE COMPARISON OF COLOR IMAGE SUPER - RESOLUTION FOR 2 X AND 4 X UP - SAMPLING FACTOR
Seq

Bicubic

SRCNN [6]
2x

CDcGAN

B
L
N
U
S
Ave.

38.5
39.21
40.66
33.77
38.89
38.21

40.58
38.79
38.32
34.04
38.87
38.12

40.05
39.19
41.22
35.55
39.93
39.19

B
L
N
U
S
Ave.

0.926
0.97
0.968
0.881
0.958
0.941

0.934
0.967
0.965
0.89
0.958
0.943

0.928
0.969
0.97
0.906
0.967
0.948

B
L
N
U
S
Ave.

45.41
46.82
46.82
43.55
46.48
45.82

45.98
46.55
46.65
43.69
46.38
45.85

45.88
46.72
47.12
44.21
47.04
46.19

Seq

Bicubic

SRCNN [6]
4x

CDcGAN

B
L
N
U
S
Ave.

32.71
33.31
33.35
29.89
33.64
32.58

34.11
28.75
25.54
26.1
32.55
29.41

34.49
34.01
34.86
31.49
34.44
33.86

B
L
N
U
S
Ave.

0.833
0.882
0.894
0.732
0.875
0.843

0.849
0.85
0.869
0.729
0.859
0.831

0.86
0.891
0.906
0.77
0.889
0.863

B
L
N
U
S
Ave.

43.53
44.23
44.09
41.95
44.55
43.67

43.77
43.23
43.59
41.82
43.93
43.27

43.93
43.98
44.24
42.38
44.54
43.81

M1

M2

M3

TABLE II
T HE OBJECTIVE COMPARISON OF DEPTH SUPER - RESOLUTION FOR 2 X UP - SAMPLING FACTOR
Seq
B
L
N
U
S
Ave.
B
L
N
U
S
Ave.
B
L
N
U
S
Ave.

M

1

2

3

Bicubic
41.53
48.97
43.12
45.88
39.54
43.81
0.980
0.993
0.986
0.996
0.967
0.985
50.42
55.25
51.88
57.91
50.98
53.29

SRCNN
[6]
44.37
50.34
46.12
49.37
40.49
46.14
0.984
0.994
0.989
0.996
0.965
0.986
50.98
55.22
52.43
58.9
50.72
53.65

GIF
[17]
32.91
41.08
33.44
45.88
34.33
37.53
0.885
0.964
0.904
0.985
0.924
0.933
47.37
52.56
48.56
54.96
49.8
50.65

FGS
[21]
34.85
43.49
35.90
45.42
37.75
39.48
0.906
0.974
0.923
0.994
0.949
0.949
47.94
53.23
49.06
57.5
51.35
51.82

RGIF
[23]
39.54
47.13
40.52
45.72
37.21
42.02
0.952
0.984
0.961
0.992
0.944
0.967
49
53.82
50.17
56.34
50.28
51.92

TGV
[22]
38.37
46.09
38.91
44.26
37.95
41.11
0.968
0.987
0.965
0.993
0.955
0.974
49.33
54.05
50.32
56.23
50.69
52.12

RGDR
[20]
36.48
42.34
37.31
46.13
38.24
40.10
0.907
0.966
0.925
0.991
0.935
0.945
49.21
53.53
50.01
55.93
51.48
52.03

HQDU
[22]
37.74
46.24
38.74
43.36
37.43
40.70
0.958
0.988
0.967
0.990
0.940
0.968
49.05
54.05
50.25
56.02
50.79
52.03

MRF
[18]
36.70
44.92
37.61
43.51
36.77
39.90
0.944
0.983
0.956
0.993
0.946
0.964
48.44
53.49
49.63
56.46
50.43
51.69

CDcGAN
46.35
54.07
47.13
50.45
42.64
48.13
0.990
0.995
0.992
0.999
0.970
0.989
53
56.46
53.72
64.67
52.97
56.16

TABLE III
T HE OBJECTIVE COMPARISON OF DEPTH SUPER - RESOLUTION FOR 4 X UP - SAMPLING FACTOR
Seq
B
L
N
U
S
Ave.
B
L
N
U
S
Ave.
B
L
N
U
S
Ave.

M

1

2

3

Bicubic
37.91
45.54
39.22
42.33
36.42
40.28
0.951
0.984
0.964
0.991
0.943
0.967
48.22
53.35
49.58
55.30
49.57
51.21

SRCNN
[6]
39.70
46.42
40.80
38.59
27.66
38.63
0.957
0.985
0.967
0.971
0.657
0.908
48.43
53.24
49.75
53.74
49.62
50.95

GIF
[17]
32.82
40.98
33.35
40.82
34.00
36.39
0.883
0.964
0.903
0.985
0.920
0.931
47.37
52.55
48.54
54.89
49.79
50.63

FGS
[21]
31.19
40.83
32.23
42.77
35.23
36.45
0.871
0.964
0.892
0.991
0.928
0.929
47.58
52.95
48.59
55.94
50.65
51.14

From the Table I, it can be seen that PSNR, SSIM and
sharpness measurement of the proposed CDcGAN are better
than other approaches for 2x and 4x color super-resolution.

RGIF
[23]
37.24
45.01
38.46
42.55
35.83
39.82
0.937
0.979
0.950
0.989
0.935
0.958
48.06
53.10
49.36
54.92
49.74
51.04

TGV
[22]
35.14
43.58
35.42
42.67
35.44
38.45
0.933
0.976
0.927
0.989
0.938
0.953
47.85
52.95
48.76
55.34
50.30
51.04

RGDR
[20]
34.42
41.04
35.16
42.68
35.60
37.78
0.890
0.962
0.909
0.989
0.923
0.935
48.21
53.06
49.08
54.97
50.59
51.18

HQDU
[22]
33.92
42.54
34.37
40.35
34.56
37.15
0.913
0.973
0.927
0.987
0.928
0.945
47.60
52.73
48.66
54.33
49.79
50.62

MRF
[18]
33.01
41.36
33.56
40.07
33.93
36.39
0.900
0.969
0.917
0.986
0.918
0.938
47.64
52.70
48.73
54.50
49.78
50.67

CDcGAN
41.72
47.43
41.88
51.23
37.85
44.02
0.962
0.986
0.969
0.997
0.952
0.973
49.89
54.23
50.81
58.19
50.97
52.82

As displayed in Fig. 3, 4, 5, the proposed method keeps
generated image sharp enough and the visual quality is more
satisfactory as compared to SRCNN [6]. In a summary, our

7

Fig. 5. The SR results for color image with 4x scaling factor. (a) the
Fig. 3. The SR results for color image with 4x scaling factor. (a) the

first frame of Book Arrival, (b) the close-ups of (a), (c-e) the closeups of the results respectively with Bicubic interpolation, SRCNN
[6], and the proposed CDcGAN

Fig. 4. The SR results for color image with 4x scaling factor. (a) the
first frame of Shark, (b) the close-ups of (a), (c-e) the close-ups of
the results respectively with Bicubic interpolation, SRCNN [6] and
the proposed CDcGAN

method has better visual performance on image reconstruction
and is robust to noise, which benefits from that the TV loss
ensures the generated color image’s flat regions to be smooth,
and the gradient difference loss tends to keep the color image
similar enough to the ground truth color image in the gradient
domain. Thus, the generated color image better obey the real
sample’s distribution when conditional GAN is used for color
super-resolution.
We compare the proposed approach with nine methods

first frame of Undo Dancer, (b) the close-ups of (a), (c-e) the closeups of the results respectively with Bicubic interpolation, SRCNN
[6], and the proposed CDcGAN

including Bicubic interpolation, SRCNN [6], GIF [17], FGS
[21], RGIF [23], TGV [22], RGDR [20], HQDU [19], MRF
[18]. The methods of SRCNN [6] only take the LR-depth
image as input. For joint SR methods: GIF [17], FGS [21],
RGIF [23], TGV [22], RGDR [20], HQDU [19], MRF [18],
we use both low-resolution depth image and the ground-truth
HR color image to get the results of depth super-resolution
with the codes provided by the authors. As described above,
our CDcGAN uses the low-resolution depth image and
low-resolution color image as the input of our network. The
objective
quality
comparison
results
for
depth
super-resolution are presented in the Table II, III. From
these tables, it can be found the PSNR, SSIM, and sharpness
measurements of the proposed approach are better than
SRCNN [6]. The depth sharpness profits depth image’s
applications, such as depth-based image rendering, scene’s
foreground extraction. In addition, from the (d) of Fig. 6,
7, 8, it can be clearly seen that there are severe artifacts
existed in the SR image with SRCNN [6], but proposed
method does not suffer this problem. The objective and
subjective quality of the proposed method has better
performance than several novel joint methods such as
optimization and filtering including GIF [17], FGS [21],
RGIF [23], TGV [22], RGDR [20], HQDU [19], MRF [18],
although these methods use HR color image. From the (f-m)
of Fig. 6, 7, 8, it can be found that most of these methods
still has the problem of texture-copy and bleeding artifacts,
due to depth SR problem’s sensitivity to textural details and
the weak boundary of color image.
C. The visual comparison of architecture’s application on
image smoothing and edge detection
In [37], deep edge-aware filter is proposed to achieve the
tasks of learning different image smoothing approaches such
as L0 gradient minimization [36]. It uses a deep

8

on the tasks of simultaneously image smoothing and edge
detection have validated our architecture’s flexibility and
generality.

Fig. 6. The SR results for depth image with 4x scaling factor. (a) the
first frame of Book Arrival, (b) the close-ups of (a), (c-l) the closeups of the results respectively with Bicubic interpolation, SRCNN
[6], GIF [17], FGS [21], RGIF [23], TGV [22], RGDR [20], HQDU
[19], MRF [18], and the proposed CDcGAN

convolutional neural network to learn various filtering in the
gradient domain. Different from [37], we use the proposed
network to learn image smoothing filtering in both image
domain and gradient domain to finish the tasks of image
smoothing and edge detection. We use the learned gradient
information for image smoothing in the gradient domain
according to [37], in which you can find the detail
operations. As displayed in Fig 9(g-j) and Fig 10(g-j), we
can see that our image smoothing results in both gradient
domain and image domain are very close to the ones of L0
gradient minimization in the gradient domain [36]. It has
also reported that there is some problems for their deep edge
aware filters, such as unsatisfactory approximation to a few
edge-preserving operators, during learning the filters in the
image domain, but our architecture does not have this
problem, due to the usage of TV loss, and gradient
difference loss with L1 norm. So the extensive application

Fig. 7. The SR results for depth image with 4x scaling factor. (a) the

first frame of Shark, (b) the close-ups of (a), (c-l) the close-ups of
the results respectively with Bicubic interpolation, SRCNN [6], GIF
[17], FGS [21], RGIF [23], TGV [22], RGDR [20], HQDU [19],
MRF [18], and the proposed CDcGAN

IV. C ONCLUSION
In this paper, color-depth conditioned generative
adversarial network is trained to achieve color-depth
super-resolution concurrently. Three auxiliary losses are used
as complementary regularization terms to train our networks
in order to ensure the generated image close to the ground
truth images, in addition to the adversarial loss. More
importantly, we also apply our architecture to concurrently
resolving the problems of image smoothing and edge
detection.

9

Fig. 9. The results of image smoothing and edge detection. (a-i)

Fig. 8. The SR results for depth image with 4x scaling factor. (a) the
first frame of Undo Dancer, (b) the close-ups of (a), (c-l) the closeups of the results respectively with Bicubic interpolation, SRCNN
[6], GIF [17], FGS [21], RGIF [23], TGV [22], RGDR [20], HQDU
[19], MRF [18], and the proposed CDcGAN

R EFERENCES
[1] S. Park, M. Park, and M. Kang, “Superresolution image reconstruction: a technical
overview,” IEEE signal processing magazine,
vol. 20, no. 3, pp. 21–36, 2003.
[2] C. Yang, C. Ma, and M. Yang, “Single-image
super-resolution: A benchmark,” in European
Conference on Computer Vision, 2014, pp. 372–
386.
[3] C. Dong, C. C. Loy, and X. Tang, “Accelerating
the super-resolution convolutional neural
network,” in European Conference on Computer
Vision, 2016, pp. 391–407.
[4] W. Shi, J. Caballero, F. Huszr, J. Totz,
A. Aitken, R. Bishop, and Z. Wang, “Realtime single image and video super-resolution
using an efficient sub-pixel convolutional neural

the first rows are the full-resolution image; the second rows are
the close-ups located in the red rectangle; (a) the input image,
(b-c) input image’s edges of (a) in the horizonal and vertical
direction, (d) the smoothed image with L0 gradient minimization
approach [36], (e-f) the edges of (d) in the horizonal and vertical
direction, (g) the smoothed image using outputted edges generated
by the proposed CDcGAN, (h-i) the outputted edges generated by
the proposed CDcGAN, (j) the smoothed image generated by the
proposed CDcGAN

[5]

[6]

[7]

[8]

network,” in IEEE Conference on Computer
Vision and Pattern Recognition, 2016, pp. 1874–
1883.
J. Kim, L. Kwon, and L. Mu, “Accurate image
super-resolution using very deep convolutional
networks,” in IEEE Conference on Computer
Vision and Pattern Recognition, 2016, pp. 1646–
1654.
C. Dong, C. Loy, and X. Tang, “Image superresolution using deep convolutional networks,”
IEEE transactions on pattern analysis and
machine intelligence, vol. 38, no. 2, pp. 295–
307, 2016.
C. Ledig, L. Theis, F. Huszr, J. Caballero,
A. Cunningham, A. Acosta, and W. Shi, “Photorealistic single image super-resolution using
a generative adversarial network,” in arXiv:
1609.04802, 2016.
J. Johnson, A. Alahi, and F. Li, “Perceptual
losses for real-time style transfer and superresolution,” in European Conference on

10

[14]

[15]

[16]

[17]

[18]

[19]
Fig. 10. The results of image smoothing and edge detection. (a-i)
the first rows are the full-resolution image; the second rows are
the close-ups located in the red rectangle; (a) the input image,
(b-c) input image’s edges of (a) in the horizonal and vertical
direction, (d) the smoothed image with L0 gradient minimization
approach [36], (e-f) the edges of (d) in the horizonal and vertical
direction, (g) the smoothed image using outputted edges generated
by the proposed CDcGAN, (h-i) the outputted edges generated by
the proposed CDcGAN, (j) the smoothed image generated by the
proposed CDcGAN

Computer Vision, 2016, pp. 694–711.
[9] D. Garcia, C. Dorea, and R. de Queiroz, “Super
resolution for multiview images using depth
information,” IEEE Transactions on Circuits
and Systems for Video Technology, vol. 22,
no. 9, pp. 1249–1256, 2012.
[10] Z. Jin, T. Tillo, C. Yao, J. Xiao, and Y. Zhao,
“Virtual-view-assisted video super-resolution
and enhancement,” IEEE Transactions on
Circuits and Systems for Video Technology,
vol. 26, no. 3, pp. 467–478, 2016.
[11] A. Mac, N. Campbell, A. Nair, and G. Brostow,
“Patch based synthesis for single depth image
super-resolution,” in European Conference on
Computer Vision, 2012, pp. 71–84.
[12] J. Xie, R. Feris, and M. Sun, “Edge-guided
single depth image super resolution,” IEEE
Transactions on Image Processing, vol. 25,
no. 1, pp. 428–438, 2016.
[13] L. Zhao, H. Bai, J. Liang, A. Wang, and
Y. Zhao, “Single depth image super-resolution
with multiple residual dictionary learning and

[20]

[21]

[22]

[23]

[24]

[25]

[26]

refinement,” in IEEE International Conference
on Multimedia and Expo, 2017, pp. 739–744.
J. Kopf, M. Cohen, D. Lischinski, and
M. Uyttendaele, “Joint bilateral upsampling,”
ACM Transactions on Graphics, vol. 26, no. 3,
pp. 96–96, 2007.
Q. Yang, R. Yang, J. Davis, and D. Nistr,
“Spatial-depth super resolution for range
images,” in IEEE Conference on Computer
Vision and Pattern Recognition, 2007, pp. 1–8.
D. Chan, H. Buisman, C. Theobalt, and
S. Thrun, “A noise-aware filter for realtime depth upsampling,” in Workshop on
Multi-camera and Multi-modal Sensor Fusion
Algorithms and Applications, 2008.
K. He, J. Sun, and X. Tang, “Guided image
filtering,” IEEE transactions on Pattern Analysis
and Machine Intelligence, vol. 35, no. 6, pp.
1397–1409, 2013.
J. Diebel and S. Thrun, “An application of
markov random fields to range sensing,” in
Neural Information Processing Systems, 2005,
pp. 291–298.
J. Park, H. Kim, Y. Tai, M. Brown, and
I. Kweon, “High-quality depth map upsampling
and completion for rgb-d cameras,” IEEE
Transactions on Image Processing, vol. 23,
no. 12, pp. 5559–5572, 2014.
W. Liu, X. Chen, J. Yang, and Q. Wu, “Robust
color guided depth map restoration,” IEEE
Transactions on Image Processing, vol. 26,
no. 1, pp. 315–327, 2017.
Y. Li, D. Min, M. N. Do, and J. Lu, “Fast guided
global interpolation for depth and motion,”
in European Conference on Computer Vision,
2016, pp. 717–733.
D. Ferstl, C. Reinbacher, R. Ranftl, M. Rther,
and H. Bischof, “Image guided depth
upsampling using anisotropic total generalized
variationn,” in IEEE International Conference
on Computer Vision, 2013, pp. 993–1000.
B. Ham, M. Cho, and J. Ponce, “Robust guided
image filtering using nonconvex potentials,”
IEEE Transactions on Pattern Analysis and
Machine Intelligence, 2017.
Y. Li, J. Huang, N. Ahuja, and M. Yang, “Deep
joint image filtering,” in European Conference
on Computer Vision, 2016, pp. 154–169.
T. Hui, C. Loy, and X. Tang, “Depth map
super-resolution by deep multi-scale guidance,”
in European Conference on Computer Vision,
2016, pp. 353–369.
G. Tech, Y. Chen, K. Mller, J. Ohm, A. Vetro,
and Y. Wang, “Overview of the multiview and
3d extensions of high efficiency video coding,”
IEEE Transactions on Circuits and Systems for
Video Technology, vol. 26, no. 1, pp. 35–49,
2016.

11

[27] Y. Li, D. Liu, H. Li, L. Li, and F. Wu,
“Convolutional neural network-based block upsampling for intra frame coding,” in arXiv:
1702.06728, 2017.
[28] I. Goodfellow, J. Pouget, M. Mirza, B. Xu,
D. Warde, S. Ozair, and Y. Bengio, “Generative
adversarial nets,” in Neural Information
Processing Systems, 2014, pp. 2672–2680.
[29] P. Isola, J. Zhu, T. Zhou, and A. Efros, “Imageto-image translation with conditional adversarial
networks,” in arXiv: 1611.07004, 2016.
[30] E. Shelhamer, J. Long, and T. Darrell,
“Fully convolutional networks for semantic
segmentation,” IEEE transactions on pattern
analysis and machine intelligence, vol. 39, no. 4,
pp. 640–651, 2017.
[31] K. Simonyan and A. Zisserman, “Very deep
convolutional networks for large-scale image
recognition,” in arXiv: 1409.1556, 2014.
[32] K. Zhang, W. Zuo, Y. Chen, D. Meng, and
L. Zhang, “Beyond a gaussian denoiser:
Residual learning of deep cnn for image
denoising,” IEEE Transactions on Image
Processing, 2017.
[33] M. Mathieu, C. Couprie, and Y. LeCun,
“Deep multi-scale video prediction beyond
mean square error,” in arXiv: 1511.05440, 2015.
[34] M. Abadi, A. Agarwal, P. Barham, E. Brevdo,
Z. Chen, C. Citro, and et al., “Tensorflow:
Large-scale machine learning on heterogeneous
distributed systems,” in arXiv:1603.04467,
2016.
[35] D. Kingma and J. Ba, “Adam: A method for
stochastic optimization,” in arXiv:1412.6980,
2014.
[36] L. Xu, C. Lu, Y. Xu, and J. Jia, “Image
smoothing via l 0 gradient minimization,” ACM
Transactions on Graphics, vol. 30, no. 6, p. 174,
2011.
[37] L. Xu, J. Ren, Q. Yan, R. Liao, and J. Jia, “Deep
edge-aware filters,” in International Conference
on Machine Learningn, 2015, pp. 1669–1678.

