INTERACTIVE DEEP COLORIZATION WITH SIMULTANEOUS GLOBAL AND LOCAL
INPUTS

arXiv:1801.09083v1 [] 27 Jan 2018

Yi Xiao1 , Peiyao Zhou1 , Yan Zheng2
1 College of Computer Science and Electronic Engineering
2 College of Electric and Information Engineering
Hunan University
Changsha, China
ABSTRACT
Colorization methods using deep neural networks have become a recent trend. However, most of them do not allow user
inputs, or only allow limited user inputs (only global inputs
or only local inputs), to control the output colorful images.
The possible reason is that it’s difficult to differentiate the influence of different kind of user inputs in network training.
To solve this problem, we present a novel deep colorization
method, which allows simultaneous global and local inputs to
better control the output colorized images. The key step is to
design an appropriate loss function that can differentiate the
influence of input data, global inputs and local inputs. With
this design, our method accepts no inputs, or global inputs, or
local inputs, or both global and local inputs, which is not supported in previous deep colorization methods. In addition, we
propose a global color theme recommendation system to help
users determine global inputs. Experimental results shows
that our methods can better control the colorized images and
generate state-of-art results.
Index Terms— Colorization, Deep convolution Neural
networks, Color theme, User input
1. INTRODUCTION
Image colorization refers to the technique that adds colors to
monochrome images or videos. Generally speaking, colorization is a ill-posed problem, it does not have a unique solution.
To get satisfactory colorized results, two categories of method
have been proposed: user-guided edit propagation and datadriven automatic colorization.
The user-guided edit propagation methods [1, 2, 3, 4, 5, 6,
7, 8] require the user to draw colored strokes and propagate
the colors across the image by solving a global optimization
problem. These methods can achieve impressive colorized
images, but often require a very large number of scribbles for
images with complex textures. This is because each different
color region must be explicitly marked by a different colored
stroke, even regions with obvious semantic hints, such as a
blue sky or green trees, need to be specified by the user.

To address this problem, early data-driven colorization
methods proposed to automatically colorize a grayscale image by learning the color hints from one or several exemplar
color images with similar semantics [9, 10, 11, 12, 13]. Unfortunately, it may be time-consuming or hard to find a suitable exemplar image sometimes. With the popularity of deep
learning, recent data-driven colorization methods using deep
neural networks have become a recent trend [14, 15, 16]. Using a large number of grayscale and color image pairs, the
deep colorization methods learn a parametric mappings for
fully automatic colorization. These methods can generate
plausible colorful images in most of time. However, since an
semantic region can have multiple choices of colors, the results can contain colors or styles which users do not expected.
For example, users may want a green mountain in spring, but
get a yellow mountain in autumn.
The excellent recent work, user-guide deep colorization,
by Zhang et al [17] combines the advantages of user-guided
and data-driven methods. It provide better user controls by
taking either global inputs or local inputs in the deep network
training. The color of the colorized image can be controlled
by a global color histogram, or a few local color points. The
user-guide deep colorization [17] can generate plausible colorized images according to the users expectation with only
a few inputs. However, The user-guide deep colorization
method does not support simultaneous global inputs and local
inputs, it can only allow one kind of inputs at one time.
We argue that supporting simultaneous global inputs and
local inputs can provide better control on the output images.
The ideal case is that a user can control the overall color style
of the image with global color inputs, and meanwhile assign
local colors to certain regions with local inputs.
Supporting multiple kinds of inputs simultaneously in
deep networks is not straightforward. It’s difficult to differentiate the influence of different kind of user inputs in network
training. An example is shown in Figure 6, the influence of
local inputs is lost if the loss function is not designed appropriately. To solve this problem, we present a novel deep colorization method, which allows simultaneous global and local

inputs to better control the output colorized images. Our basic model neural network is a U-net network including three
parts, the feature extraction part, the fusion part, and the reconstruction part. The feature extraction part, consisting of
3 convolution layers and 3 pooling layers, extracts features
from the gray-scale images and the local input images. The
fusion part fuse the global inputs (color themes) and the extracted feature maps. The reconstruction part, consisting 3
deconvolution layers, then reconstructs the two chrominance
channels (Lab space). The key step is to design an appropriate
loss function that can differentiate the influence of input data,
global inputs and local inputs. With this design, our method
accepts no inputs, or global inputs, or local inputs, or both
global and local inputs, which is not supported in previous
deep colorization methods.
Different with previous works, which use exemplar images or histograms as global inputs, we choose to use the
color theme, a template of colors possibly associated verbal
description [18], as the global input. It is more easier for
users to assign a color theme than to choose exemplar images
or histograms. In our model, we accept color themes consist of 3 - 7 colors. More colors are possible but difficult to
use. To further save work time for users, we propose a global
color theme recommendation system to help users determine
the global input. Inspired by [18], we build a mapping between grayscale texture features and color histograms from
a large image data base, and use the mapping to predict the
color themes. Also, the user can define its own color theme.
Experimental results show that the color images generated by our method look real and natural, and the detail of
the image are preserved well. Using different color themes
and local inputs, we can output different style images (See
Figure 1). To summarize, our methods can better control the
colorized images and generate state-of-art results.
Our contributions in this paper are as follows:
• We propose a novel deep colorization methods that supports no inputs, or global inputs, or local inputs, or both
global and local inputs.
• Our methods enables global inputs using color themes
of variable color numbers.
• We propose a color theme recommendation system
which can suggest the color themes for users.

2. RELATED WORK
Image colorization is always a hot topic in the filed of image
process. After the advent of deep convolution neural network,
it is devoted to train images on network to achieve the goal of
great colorization. At present, there are two main ways of colorization. One is user-guided edit propagation colorization,
and the other is automatic colorization.

About user-guided coloring method, most of them are devoted to add scribbles on gray images. In the early paper, like
[1], it is necessary for users to draw some desired color graffitis on certain regions, through the color propagating among
pixels which are similar in intension to achieve the effect of
colorization. [3] and [19] worked on the issue of edge detection and enhancement, which can improve the quality of
color transmission. In [5], users select the area of interest
and then specify some color points within the area. [20] and
[21] are optimized for edit propagation and shading. Its also
popular to use color theme, such as using data driven color
theme enhancements in [18], which is used to change the
color style of color images through different color themes.
We take this method and using color theme as global input to
color gray images. [22] is a popular method to use the palette
to change the corresponding color on color image as same as
[18]. In [23], it is proposed to automatically learn the degree
of similarity between user strokes and input images to propagate graffiti color. [24] uses generative adversarial networks
to achieve transforming sketch to real images, which also supports to add user strokes. In [17], by adding global features
and local features on deep convolution neural network to train
a colorization model, separately, users can specify the color at
any location to render the color, or use a reference image to
transfer global color. Our method can achieve colorization
using both of them simultaneously. There are online applications developed mainly for line-drawing colorization, like
[25] and [26].
For more convenient to color grayscale images globally,
there are some method using reference images as global inputs early. [27] transfers colors between images with a simple
algorithm. [9] and [10] transmit the color emotion of reference images to grayscale images by matching the brightness
and texture information between images. [11] needs users
to provide the semantic information of the foreground image. The system searches for and downloads pictures with
the same foreground semantics on the network, and use them
to color the foreground and the background. [12] can quickly
transfer the color of a reference image to the target image by
using a fast cascade feature matching scheme to exploit multiple image features.
But using reference images which is similar with target
images to render the color needs a large image database.
Users also have to spend a lot of time to choose satisfactory reference image. So, the research of automatic colorization gets more and more attention. [28] uses algorithm to
reserve grayscale images or videos. In this method, some
representative color points are reserved and used to restore
these medias. [29] uses machine learning tools to extract as
much information as possible from the color sample dataset
and then estimates the probability color distribution for every pixel. [30] matches grayscale image with reference images downloaded automatically from web to transfer color.
[31] train the objective function based on image features on

Fig. 1. Our method colorizes two gray images by global input and local input simultaneously. The third column are the results
using only local or global inputs. The other results are obtained using simultaneously global input and local input.
the LEARCH framework and achieve the coloring effect by
minimizing the objective function. [14] introduces the concept of feature descriptor, and take the feature descriptors extracted from the grayscale image as input, and finally output
the corresponding color values of the UV channel. In [15], a
convolutional neural network (CNN) which accepts black and
white images as input is designed and constructed, and a statistical learning driven method is used to solve the problem of
grayscale colorization. The end-to-end network in [32] adds
global feature and uses classification label to optimize output
results. In addition, [33], [34], [16], [35], [36] also achieve
the effect in coloring gray image without users’ intervention.
[37] uses conditional adversarial networks and [25] proposes
a setup utilizing two networks in tandem to achieve colorization, which both are worth thinking about.
Neural network also has shown us some surprising results
in other fields of image processing. [38] reaches the target
of automatic style transferring between images. As well as
large-scale image recognition [39], automatic photo adjustment [40], sketch simplification [41], context encoders [42]
and cartoon colorization [43]. Accuracy and practicability of
neural network is the reason that we chose neural network to
research and construct satisfactory colorization method.
3. OUR METHOD
Our deep network model takes in a gray-scale image (L channel in CIE Lab space), a global input and a local input and
output the corresponding ab channels. The structure of the
model is shown in Figure 2. The global input includes a color
theme of several colors and a mask indicating the number of
colors (all black means no global inputs). The local input includes a image consisting of the assigned local color points
and a mask indicating the position of color points (all black

means no local inputs). Therefore, by setting the masks of
global inputs and local inputs, our model accepts four combination of user inputs: no inputs, global inputs, local inputs,
global inputs and local inputs.
3.1. Problem Formulation
The inputs of our model are a grayscale image X ∈
RH×W ×1 , a global input tensor Ug ∈ R1×K×3 , and a local input tensor Ul ∈ RH×W ×3 , where H, W is the width
and height of the input image, and K is the number of colors
in the color theme. The output is a tensor O ∈ RH×W ×2 . We
aims to train a convolution neural network (CNN), denoted by
F(X, Ug , Ul ; θ), to approximate the mapping between gray
and color, under the constraint of user inputs. Therefore, the
colorization problem can be formulated as
θ∗ = arg min EX,Ug ,Ul ,Y,D [L(F(X, Ug , Ul ; θ), Ug , Ul , Y )],
θ

(1)
where D denotes the training data set, L denotes the loss
function, and Y ∈ RH×W ×2 is ground truth image color.
More details of the loss function will be presented in Section 3.4. It should be noticed that we train a single model to
handle simultaneous global input and local input. Moreover,
our loss function is explicitly related to the global input and
the local input. These are the two key points which differ our
method from [17].
3.2. User inputs
Global inputs: To control the overall color style of the output, we design the global input as a color theme with K colors plus a one-channel mask indicating the number of colors.

Fig. 2. Our network model for colorization. The steps between layers without arrow instruction are convolution with 3 × 3
kernel.
inputs Ul = {Ulc , Ml } ∈ RH×W ×3 .
3.3. NetWork model
Fig. 3. Generation process of 5-color-map image
An example of color theme with 5 colors and the corresponding mask are shown in Figure 2. To prepare training data for
the global input, we use the K-mean clustering algorithm to
find the K representative colors for each color image in the
data set. The K representative colors form the global color
theme. To enable color theme with variable number of colors,
K randomly varies in the interval [3,7]. Color theme with
more colors are also supported by modifying the interval, but
may be harder to use. The ab channels of each color theme
Ugc ∈ R1×K×2 and its mask Mg ∈ R1×K×1 forms the global
input Ug = {Ugc , Mg } ∈ R1×K×3
For each color image, we generate a K-color map by decoding the color image with its representative colors. Figure 3 shows an example. The ab channels of K-color map,
I ∈ RH×W ×2 is used to calculate the loss function which
measures the similarity between the color theme and the output image. More details of loss functions will be described in
Section 3.4.1.
Local inputs: Similar to [17], we prepare the training data for
the local inputs colors Ulc ∈ RH×W ×2 by randomly selecting some points from the ab channels of every color image.
If there are no samples in certain position, the ab values are
set to zeros. A one-channel mask Ml ∈ RH×W ×1 is also
generated to indicate the positions of the local inputs. ”1”
in certain position means there is an input there, while ”0”
means there are no inputs in the position. Finally, the local

As shown in Figure 2, our network model uses a U-Net structure [44], which is already shown to works well in colorization [37, 17]. It is mainly composed of four parts: the feature
extraction module, the global input module, the fusion module and the reconstruction module.

3.3.1. Feature extraction module
The feature extraction module corresponds to the blue part in
Figure 2. The feature extraction module takes in a grayscale
image X ∈ RH×W ×1 and a local input Ul ∈ RH×W ×3 .
Initially, X and Ul are respectively convolved to a tensor of
size H × W × 32. These two tensors are merged to a single
tensor of size H × W × 32 by linear interpolation, whose
weight is treated as a parameter of the network to be trained.
The merged tensor is then feeded to the following convolution
blocks.
In convolution block 2 to 7, all the convolution kernels
are 3 × 3. Through every layer, only the tensor sizes of input
are halved spatially with the stride 2 × 2, or only the tensor
dimensions are doubled with the stride 1 × 1. After conv7,
W
the tensor is H
8 × 8 × 512. We use convolution layers with
stride 2 × 2 instead of using max-pooling layers to reduce the
tensor sizes, which is helpful in increasing the spatial support
of each layer [32]. Conv8 and conv9 process the tensor further
with convolution layers to reduce the 512-channel tensors to
256-channel tensor. The output of feature extraction module
is a tensor of size H8 × W
8 × 256, which will be fused with the
global input.

3.3.2. Global input module
The global input module, the pink part in Figure 2, is one
of the characteristics of our method. It takes in the global
inputs Ug ∈ R1×K×3 , which consists of the ab channels of
the color theme Ugc and the corresponding mask Mg . To to
unify its size with the size of the fusion module, we reshape
it to a tensor of size 1 × 1 × 3K, which is then processed by
three fully connected layers. This operation is similar to the
global feature network in [32], but we use it to handle user
global inputs instead of the features exacted from the input
gray-scale image.
3.3.3. Fusion module
The fusion layer, shown with yellow color in Figure 2, is an
important step to fuse the global input and the extracted features. Similar operations are also used in [32, 17]. The global
input is fused with the output of the feature extraction module by linear interpolation, whose weight is also treated as
a parameter in the network to be trained. Finally the fusion
W
module outputs a feature tensor of size H
8 × 8 × 256 .

4 Both inputs mean that the user wants to assign his/her
preferred color to certain region, but let the colors of
other regions be conditionally assigned with the ”experiences” of training data.
3.4.1. Loss function for no input or global input
For no inputs or only the global input, the straightforward
choice is to measure the differences between ab channels of
the output image O ∈ RH×W ×2 and the ground truth color
Y ∈ RH×W ×2 . There are many kind of loss functions as
discussed in [17]. We choose the Huber loss as it produces
relative high saturation effect. The Huber loss is given by

2
 21 (O − Y ) for |O − Y | ≤ δ
(2)
LH (O, Y ) =

δ |O − Y | − 12 δ 2 otherwise,
where δ is the parameter of the Huber loss. The value of δ
will slightly affect the results as shown in Figure 4. We set
δ = 0.5, as this value works well in our experiments.

3.3.4. Reconstruction module
After the fusion module, the feature tensor is processed by a
set of convolution layers and upsampling layers for ab channel reconstruction, shown with purple color in Figure 2. Convolution layers reduce tensor dimensions by half and upsampling layers doubles the wide and height of the tensor.
Conv17 is the last convolution layer with a Sigmoid transfer function, followed by an upsampling layer. The advantage of Sigmoid is that the output range is limited. The output tensor is of size H × W × 2, which combines the input
gray-scale image X ∈ RH×W ×1 to generate the final output
O ∈ RH×W ×3 .
3.4. Loss function
Designing an appropriate loss function is the most important
part of our work. It is not straightforward to design a loss
function which can differentiate the impact of different inputs.
We interpret the four combinations of inputs as follows:
1 No inputs means the user wants the colors to be assigned with the “experiences” of training data.
2 Only the global input means the user wants the colors
to be conditionally assigned with the ”experience” of
training data.
3 Only the local input means the user wants to assign
his/her preferred color to certain region, but let the colors of other regions be assigned with the “experiences”
of training data.

δ = 0.5

δ = 1.0
Fig. 4. Comparison of different delta parameters. The parts
pointed out by red rectangle indicate the differences when use
different delta parameters.
However, in some cases, the impact of the global input
(color theme ) is not so obvious. Therefore, we add an additional part which calculates the Huber loss of the output
image and the K-color map I, which is defined by decoding
the ground truth with the color theme (see Section 3.2 and
Figure 3).
Therefore, the loss function for this part is given by
Lg = α1 ∗ LH (O, Y ) + α2 ∗ LH (O, I),

(3)

where α1 and α2 are two parameters to balance the influence
of two parts. The influence of the two parameters is shown
in Figure 5. As we expect that output images not only keep
natural color but also reflect colors of the color theme, we set
α1 = 0.7 and α2 = 0.3, with which the output can preserve
the characteristics of the color theme with almost no color
overflows, as showed in Figure 5. When there is no color
theme input, the K-color map image will be replaced by the
ground truth. In this case, Lg = Ly .

(a) 0.1:0.9

(b) 0.3:0.7

(c) 0.5:0.5

(d) 0.7:0.3

(e) 0.9:0.1

Fig. 5. Comparison of different fusion times and comparison of different ratios of loss. (a) to (e) show the results with
different loss ratios. (d) is the ratio we finally chosen for our method, which meets our demands for bright and well-distributed
colorization and avoids color overflow.
3.4.2. Loss function for local input
The local input in our method is Ul = {Ulc , Ml } ∈ RH×W ×3 ,
which is combined by the ab channels of the user input and
the corresponding mask, as introduced in Section 3.2. If a
user provides a local input in a certain position, we think the
user wants the preferred colors instead of colors assigned by
the “experience” of training data. In there is only the local
input, the Hubber loss LH (O, Y ) defined in Equation (2) is
already shown to works well [17], since the differences of
the output image and the local inputs at positions with local
inputs are already contained in Equation (3).
However, if there are simultaneous global input and local
input, the case becomes more complex. The Hubber loss may
seems to work sometimes, when the local inputs are consistent or close with the global color theme, as shown in Figure 6(f). For example, the input brown color in the back and
swing is close to one color in the color theme (Figure 6(a)).
In this case, the local input can enhance the colors of local regions compared to results with only global inputs (Figure 6(b)). But there exist some colors which overflow from
the bird body to the background (TODO, indicate in the figure).
The color overflow can be removed by adding the gradient
of the output image and the ground truth in the loss function,
given by
Ls = MSE (OSobel , YSobel ) ,
(4)
where MSE denotes the mean squared error, Sobel denotes
the Sobel gradient operator. As shown in Figure 6(g), the
color overflows are removed at the cost of removing the local
color impacts on the head and abdomen.
When the local input colors are quite different with the
colors in the global color theme, the phenomenon is more obvious, as shown in Figure 6(j)(k). Using only the loss Lg
cannot guarantee the local color impact, and results in more
obvious color overflows (Figure 6(j)). The gradient loss Ls almost remove the color overflows, but also remove the impact
of local color inputs (Figure 6(k)). Since we expect the colorized image preserves not only the global input color theme,

but also the local input colors, without few color overflows.
We emphasize the impact the local inputs by adding the loss
of the output image and the local inputs at positions with local
inputs, given by
Lp = M SE ((O ∗ Ml ) , (Ulc ∗ Ml ))

(5)

where MSE denotes the mean squared error, Ml is the mask
of the local input. With Lp , we can see from Figure 6(h)(i)
that the aforementioned problems are solved.

(a) Inputs

(b) only Lg

(c) Lg +Ls

(d) Lg +Ls +Lp

(e) Inputs

(f) only Lg

(g) Lg +Ls

(h) Lg +Ls +Lp

(i) Inputs

(j) only Lg

(k) Lg +Ls

(l) Lg +Ls +Lp

Fig. 6. Comparison of different combination of loss functions. (a) is a grayscale image. We use the color theme above
(a) as global input. (b) to (d) are results of using the color
theme only. The last two rows show results of using both the
color theme and local inputs, which are different in color of
some points. Subscripts of images show the composition of
loss function used. For example, (b) is the result of only using
Lg as loss function. (c) add Ls . (d) add Ls and Lp , which is
our final selection loss function.
To summarize, our final loss function L is
L(F(X, Ug , Ul ; θ), Ug , Ul , Y )
= Lg + Ls + Lp .

(6)

3.5. Color theme recommendation system
To help user determine the possible color theme of a grayscale image, we propose a color theme recommending system. Our system is inspired by the data driven method in
[18], which consists of an off-line process to builds a texturecolor distribution mapping and an on-line process to lookup
the color distribution for the segmented input color image.
Different with the off-line process in [18], which built its
texture-color distribution mapping on color images to constrain the color distribution of textures, we build the mapping
on the gray-color image pairs to predict the colors themes.
The framework of color theme commended system is illustrated in Figure 7. It consists of an off-line phase(lower
row) and an online phase(upper row). The mapping of
texture-color distribution is obtained in the off-line stage. The
color themes of gray-scale image that input by users can be
predicted by computing the main texture of gray-scale image
in online phase.
During the off-line phase, we start with building an image database, by randomly choosing thousands of colorful
images from Imagenet, and then convert the colorful images
to grayscale images to form a grayscale image database. Next
we adapt the graph-based method [11] to segment each grayscale image into segmentations. We use the mean and standard deviation of the pixelwise Gabor filter (scale=4, rotation=6) responses within every segmentation, resulting a 24
dimensional texture descriptor for every segmentation. We
then build a texture library by clustering all the texture features into 120 textures with k-means algorithm. For each cluster, we accumulate a 2D color distribution with 100 bins by
quantizing the corresponding ab channels of the segmentations. Hence, we build a texture-color distribution mapping.
In the online-phase, we segment the input gray-scale image, extract the texture feature vectors, find the closet clusters
in the texture library, and lookup the corresponding color distributions. We choose the most frequent color of the K largest
segmentations (with more number of pixels) as our recommended color theme. We display some images colored by
color themes which are generated by this system in Figure 8.
4. EXPERIMENT
We implemented the network model on the NIVIDIA
GTX1080Ti GPU, using Google’s TensorFlow architecture.
We use the Place dataset to train our model. Figure 9 shows
some colored results of this dataset. The K-color map images,
color themes, local inputs and their corresponding masks are
generated on Matlab. Then, we randomly adjust the order of
training sets as input of our model. We train the model using
a batch size of 50 for 40,000 iterations, which takes about two
days on the GPU. To enable the four combinations of input:
no inputs, only global input, only local input, and both global
and local input, we randomly select one kind of combination

in the training. The ratio to select each combination is equal
to 25%.
In the previous works, the normalization of input has been
shown to speed up the learning speed. In Lab space, the range
of L is [0 − 100], and the range of ab channel is [127, −128].
In order to accelerate the learning speed, we normalize the
range of L and ab to [0,1] as
L = L/100
ab = (ab + 128)/255

(7)

4.1. Automatic Colorization
In these section, we compare our automatic colorization
method with several state-of-art methods, including [17],
which train coloring with no inputs or local inputs , [32]
and [33], which focus on automatic colorization. The colorized results of these four methods are shown in Figure 10.
As shown in the figure, all the automatic methods can colorize the gray-scale images. But due to the different training
sets, the colorized images may have different color styles. In
terms of numerical evaluation, our method and [32] get better PSNRs without compared with the other two methods. All
PSNRs of our automatic results are higher than [17]. Note
that, although automatic colorization is only our by-product,
it still has good performances.
4.2. Interactive Colorization
In this section, we compare our interactive colorization
method with the state-of-art user guide method [17], and the
classical optimization method [1]. The results are shown in
Figure 11. The Butterfly images in the first three columns
show results colorized by different local input points. With
only local inputs, the results of our method are similar to those
of the user-guide method [17]. In contrast, the results of [1]
fail to diffuse the color to the whole butterfly.
For the next five images, our method uses global input
(color theme) and local input together. The other two methods use local inputs only. For images with less clear boundaries, like the fourth column, our method can faithfully assign
green to the trees, where the other methods cause color overflows to the background. Using our method, users can control
the colorized image by choosing a color theme, and add a
small number of local input points to certain regions, while
other methods maybe need to add more local input points to
produce the same results. For example, in the fourth column, users maybe need to add several extra color points on
the house and sky to avoid color overflow and assign certain
colors for the user guide method [17]. [1] needs to assign
more color points for everywhere in every image. Otherwise,
the whole image is going to be one color as shown in the figure. Besides, our method also shows good performance on
other type images, like human and natural pictures showed in
the last three columns of the Figure 11.

Fig. 7. The overall pipeline of color theme commended system framework. The (c) is a texture library which stores the texture
center of 120 textures. The image in (d) is a mapping between each pixel and texture, so the value of the pixel is the texture
type of the pixel belongs. In the same way, (f) can be converted into (g) by calculating the nearest texture center (from (c)) of
each segmentation. So the main texture in (f) can be counted. Combine with the main texture, the desired color theme into
consideration to generate final recolored image in (h) can be found from (e).

Fig. 8. Examples of recommendation system.

Fig. 9. Examples of colorization results in the training set
Place.
In general, our method provides a more convenient and
efficient tool for the users by supporting four combinations of
input in a single network: no inputs, only global inputs, only
local inputs, and simultaneous global input and local input.
4.3. Numerical Comparisons
Besides visual comparisons, we also evaluate the PSNRs of
the related methods. We randomly choose another 20 images
and colorize them using the automatic methods and interactive methods mentioned above. We use original images as
global inputs for [17], and use color themes extracted from
original images as global inputs for our method. Local inputs are 3 to 20 colorful points randomly extracted from original images. Then we calculate the average PSNR of them
as showed in Table 1. Through Table 1, we can see that results of automatic methods have similar performances, which
have almost identical PSNRs. When we add global inputs,
our method works much better and gets a higher PSNR.
For method with global inputs, the user guide method [17]
shows good performance and get the highest PSNR. Our
method also works well. But the PSNR of method is a bit
lower than the user guide method [17]. This is possibly because we train our network to support four combination of inputs , the effect of local inputs is not optimized in our method.
For method with local inputs, the user guide method [17]
shows good performance and get the highest PSNR. Our
method also works well. But the PSNR of method is a bit
lower than the user guide method [17]. This is possibly also
because we train our network to support four combination
of inputs , the effect of local inputs is not optimized in our
method. The optimization method [1] gets a lower PSNR,
probably because it was designed for strokes not point inputs. When both global inputs and local inputs are used, our
method get a higher PSNR, which is very close to the highest
in Table 1.
4.4. Colorization for past
We test our method on some historic images, which are whiteand-black images. Although these images are different from

Table 1. The average PSNR of 20 images.
Method
Added Inputs
PSNR(dB)
[33]
automatic
23.8907
[32]
automatic
24.4463
[17]
automatic
24.4997
Ours
automatic
24.4691
[17]
global inputs
29.2105
Ours
global inputs
27.8543
[1]
local inputs
24.075
[17]
local inputs
29.0978
Ours
local inputs
27.2619
Ours
global + local
28.5375

Table 2. The average computation time.
Image Size
Pixels
Time
256 × 256
65,536 0.00750 s
512 × 512 262,144 0.0228 s

our dataset, as that they may have rough edges and their picture quality may be unclear, they also can be colorized by our
method. Compared with our method, other methods, which
use local input only, maybe need a lot of local input points
to get the same results. Some examples of coloring historic
images are shown in Figure 12.

4.5. Computation time
We test computation time of two image sizes on the NIVIDIA
GTX1080Ti GPU. We take the average time of 100 computations on 50 images to get a reliable testing value, as shown
in Table 2. The data shows that our method is comparable to
real-time colorization.

4.6. Limitations and discussion
Our model is trained by the Place dataset, which can not include everything in this world. If an gray-scale image contains certain contents, which is not learned from the data, the
colors may be not assigned appropriately. In this case, the
user may need to give more inputs to the system. For some
images with unclear edges, the network may produce unexpected results, like color overflow on background. Users can
add another local input points in the background region. In
addition, When users add local input points with unusual colors, they may have to add more points to achieve the desired
effect or directly change to use an appropriate color theme.

Gray Image

(a)

[17]

[32]

[33]

Ours

16.5179

21.6247

18.28

16.7994

28.4547

23.7707

22.6357

29.0915

19.4188

21.2468

20.3247

19.8622

13.931

14.9813

10.6654

16.6442

25.0427

23.4359

19.9974

26.0092

(b)

(c)

(d)

(e)

Ground Truth

(f)

Fig. 10. Comparison with automatic colorization methods. (a) are grayscale images. (b) to (e) are results of automatic colorization methods. (f) are ground truth images. The digits above every image are PSNR, in which the bold one is the best.

Input

Local

Ours

[17]

[1]
Fig. 11. Comparisons of interactive colorization methods. We only add local inputs on the first grayscale image and change the
color of some points on the Butterfly wings, without changing their position. For the last five images, we add color theme and
local inputs simultaneously, while other methods add local inputs only.
5. CONCLUSION AND FUTURE WORK
In this paper, we propose a novel interactive deep colorization
method. By designing a suitable loss function, our method allows four combinations of input in a single network model,
including no inputs, only global input, only local input, and
both global input and local input. Furthermore, our global input is a color theme with variable number of colors, which
is more easy and straightforward for user to use. Together
with recommendation system, our method can reduce time
and cost of coloring images for users. In addition, we evaluate our method on images of outdoor, human picture, past
pictures, and show that it can produce satisfactory results for
all them. In the future, we would like to extend our method to
gray-scale video sequence.
6. REFERENCES
[1] Anat Levin, Dani Lischinski, and Yair Weiss, “Colorization using optimization,” in SIGGRAPH, 2004, pp. 689–
694.
[2] T. Horiuchi and H. Kotera,
“Colorization for
monochrome image with texture,” in Color Imaging
Conference, 2005, pp. 245–250.

[3] Yi Chin Huang, Yi Shin Tung, Jun Cheng Chen,
Sung Wen Wang, and Ja Ling Wu, “An adaptive edge
detection based colorization algorithm and its applications,” in ACM MM, 2005, pp. 351–354.
[4] Liron Yatziv and Guillermo Sapiro, “Fast image and
video colorization using chrominance blending,” Image
Processing, IEEE Transactions on, vol. 15, no. 5, pp.
1120–1129, 2006.
[5] Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang,
Ying Qing Xu, and Heung Yeung Shum, “Natural image
colorization,” in Eurographics Conference on Rendering Techniques, 2007, pp. 309–320.
[6] M. Kawulok and B. Smolka, “Competitive image colorisation,” in Proceedings of 17th International Conference on Image Processing, 2010, pp. 405–408.
[7] Chen Yao, Xiaokang Yang, Li Chen, and Yi Xu, “Image
colorization using bayesian nonlocal inference,” Journal of Electronic Imaging, vol. 20, no. 2, pp. 023008–1–
023008–6, 2011.
[8] Michal Kawulok, Jolanta Kawulok, and Bogdan
Smolka, “Textural features for scribble-based image
colorization,” Computer Recognition Systems 4, vol. 95,
pp. 269–278, 2011.

[14] Zezhou Cheng, Qingxiong Yang, and Bin Sheng, “Deep
colorization,” in ICCV, 2015, pp. 415–423.
[15] Tung Duc Nguyen, Kazuki Mori, and Ruck Thawonmas, “Image colorization using a deep convolutional
neural network,” CoRR, vol. abs/1604.07904, 2016.
[16] Matthias Limmer and Hendrik P. A. Lensch, “Infrared
colorization using deep convolutional neural networks,”
in ICMLA, 2017, pp. 61–68.
[17] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang
Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros,
“Real-time user-guided image colorization with learned
deep priors,” ACM TOG, vol. 9, no. 4, 2017.
[18] Baoyuan Wang, Yizhou Yu, Tien-Tsin Wong, Chun
Chen, and Ying-Qing Xu, “Data-driven image color
theme enhancement,” TOG, vol. 29, no. 6, pp. 146:1–
146:10, December 2010.
[19] Y. Li, E. Adelson, and A. Agarwala, “Scribbleboost:
adding classification to edge-aware interpolation of local image and video adjustments,” in Nineteenth Eurographics Conference on Rendering, 2008, pp. 1255–
1264.

Fig. 12. Examples for coloring historic images. The first two
images are colored by color themes and local inputs. The last
image is colored by local inputs only.

[20] Kun Xu, Yong Li, Tao Ju, Shi-Min Hu, and Tian-Qiang
Liu, “Efficient affinity-based edit propagation using k-d
tree,” ACM TOG, vol. 28, no. 5, pp. 118:1–118:6, 2009.
[21] Xiaowu Chen, Dongqing Zou, Qinping Zhao, and Ping
Tan, “Manifold preserving edit propagation,” ACM
TOG, vol. 31, no. 6, pp. 132, 2012.

[9] Tomihisa Welsh, Michael Ashikhmin, and Klaus
Mueller, “Transferring color to greyscale images,”
in Conference on Computer Graphics and Interactive
Techniques, 2002, pp. 277–280.

[22] Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein, “Palette-based photo recoloring,” ACM TOG, vol. 34, no. 4, jul 2015.

[10] Youngha Chang, Suguru Saito, Keiji Uchikawa, and
Masayuki Nakajima, “Example-based color stylization
of images,” Acm Transactions on Applied Perception,
vol. 2, no. 3, pp. 322–345, 2005.

[23] Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun
Mitani, “Deepprop: Extracting deep features from a single image for edit propagation,” Computer Graphics Forum, vol. 35, no. 2, pp. 189–201, 2016.

[11] Yong Sang Chia, Shaojie Zhuo, Raj Kumar Gupta,
Yu Wing Tai, Siu Yeung Cho, Ping Tan, and Stephen
Lin, “Semantic colorization with internet images,” ACM
TOG, vol. 30, no. 6, pp. 1–8, 2011.

[24] Patsorn Sangkloy, Jingwan Lu, Chen Fang, FIsher Yu,
and James Hays, “Scribbler: Controlling deep image
synthesis with sketch and color,” CVPR, 2017.

[12] Raj Kumar Gupta, Yong Sang Chia, Deepu Rajan,
Ee Sin Ng, and Zhiyong Huang, “Image colorization
using similar images,” in ACM MM, 2012, pp. 369–378.
[13] Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong,
Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng,
“Intrinsic colorization,” ACM Transactions on Graphics (SIGGRAPH Asia 2008 issue), vol. 27, no. 5, pp.
152:1–152:9, December 2008.

[25] Kevin Frans, “Outline colorization through tandem adversarial networks,” CoRR, vol. abs/1704.08834, 2017.
[26] Preferred Networks,
“Paints chainer,” http:
//paintschainer.preferred.tech/webui/
howto/, Accessed 2017.
[27] Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and
Peter Shirley, “Color transfer between images,” IEEE
Computer Graphics and Applications, vol. 21, no. 5, pp.
34–41, 2001.

[28] Cheng Li and S. V. N. Vishwanathan, “Learning to compress images and videos,” in ICML, 2007, pp. 161–168.
[29] Guillaume Charpiat, Matthias Hofmann, and Bernhard
Sch?lkopf, “Automatic image colorization via multimodal predictions,” in ECCV, 2008, pp. 126–139.
[30] Yuji Morimoto, Yuichi Taguchi, and Takeshi Naemura,
“Automatic colorization of grayscale images using multiple images on the web,” in SIGGRAPH, 2009, pp. 1–1.
[31] Aditya Deshpande, Jason Rock, and David Forsyth,
“Learning large-scale automatic image colorization,” in
ICCV, 2015, pp. 567–575.
[32] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi
Ishikawa, “Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic
Image Colorization with Simultaneous Classification,”
TOG, vol. 35, no. 4, 2016.
[33] Richard Zhang, Phillip Isola, and Alexei A. Efros, “Colorful image colorization,” in ECCV, 2016, pp. 649–666.
[34] Domonkos Varga and Tams Szirnyi, “Fully automatic
image colorization based on convolutional neural network,” in ICPR, 2017, pp. 3691–3696.
[35] Yili Zhao, Dan Xu, and Yan Zhang, “Retracted chapter:
Image colorization using convolutional neural network,”
in IGTA, 2016, pp. 238–244.
[36] Gustav Larsson, Michael Maire, and Gregory
Shakhnarovich,
“Learning representations for automatic colorization,” in ECCV, 2016.
[37] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A.
Efros, “Image-to-image translation with conditional adversarial networks,” CoRR, vol. abs/1611.07004, 2016.
[38] Yiming Liu, Cohen Michael, Uyttendaele Matt, and
Rusinkiewicz Szymon, “Autostyle: Automatic style
transfer from image collections to users’ images,” in
EGSR, 2014, pp. 21–31.
[39] Karen Simonyan and Andrew Zisserman, “Very deep
convolutional networks for large-scale image recognition,” Computer Science, 2014.
[40] Zhicheng Yan, Hao Zhang, Sylvain Paris, Sylvain Paris,
and Yizhou Yu, “Automatic photo adjustment using
deep neural networks,” ACM TOG, vol. 35, no. 2, pp.
11, 2014.
[41] Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and
Hiroshi Ishikawa, “Learning to simplify: fully convolutional networks for rough sketch cleanup,” ACM TOG,
vol. 35, no. 4, pp. 121, 2016.

[42] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue,
Trevor Darrell, and Alexei Efros, “Context encoders:
Feature learning by inpainting,” 2016.
[43] Domonkos Varga, “Automatic cartoon colorization
based on convolutional neural network,” in International Workshop on Content-Based Multimedia Indexing, 2017, p. 28.
[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,
“U-net: Convolutional networks for biomedical image
segmentation,” in MICCAI, 2015, pp. 234–241.

