Capturing Localized Image Artifacts through a CNN-based Hyper-image
Representation
Parag Shridhar Chandakkar and Baoxin Li, Senior member, IEEE

arXiv:1711.04945v1 [] 14 Nov 2017

School of Computing, Informatics and Decision Systems Engineering
Arizona State University, Tempe, USA
{pchandak, baoxin.li}@asu.edu
Abstract
Training deep CNNs to capture localized image artifacts on
a relatively small dataset is a challenging task. With enough
images at hand, one can hope that a deep CNN characterizes
localized artifacts over the entire data and their effect on the
output. However, on smaller datasets, such deep CNNs may
overfit and shallow ones find it hard to capture local artifacts.
Thus some image-based small-data applications first train their
framework on a collection of patches (instead of the entire
image) to better learn the representation of localized artifacts.
Then the output is obtained by averaging the patch-level results. Such an approach ignores the spatial correlation among
patches and how various patch locations affect the output. It
also fails in cases where only few patches mainly contribute
to the image label. To combat these scenarios, we develop
the notion of hyper-image representations. Our CNN has two
stages. The first stage is trained on patches. The second stage
utilizes the last layer representation developed in the first stage
to form a hyper-image, which is used to train the second stage.
We show that this approach is able to develop a better mapping between the image and its output. We analyze additional
properties of our approach and show its effectiveness on one
synthetic and two real-world vision tasks - no-reference image quality estimation and image tampering detection - by its
performance improvement over existing strong baselines.

(a)

(b)

Introduction
Convolutional neural networks (CNN) have recently enjoyed
tremendous success in many areas. In various fields, CNNs
have surpassed the performance of previous state-of-art approaches by significant margins. For example, CNNs have
shown considerable superiority in object recognition (Simonyan and Zisserman 2014; He et al. 2015), face recognition (Taigman et al. 2014), semantic segmentation (Dai et al.
2016) etc. as well as in some understudied but long-standing
and difficult problems such as gaze-following (Recasens et al.
2015). Since the advent of AlexNet (Krizhevsky, Sutskever,
and Hinton 2012), deep networks have also been continuously
evolving for tasks such as segmentation (Zheng et al. 2015),
object detection and image classification (He et al. 2015;
Goodfellow et al. 2014; Salimans et al. 2016).
However, training CNNs to characterize localized image
artifacts and label the image accordingly on a relatively small
dataset remains a challenging task. With large amounts of
data, deep CNNs may be able to learn a good representation

(c)

Figure 1: (a) and (b) Clean (left) and distorted (right) image
pairs in the TID 2013 dataset. The images on the right in
(a) and (b) are distorted by non-uniform and uniform noise
respectively. (c) Authetic and forged image pair from CASIA2 dataset. Red overlay shows distorted/forged regions in an
image. Please zoom in for details and view the online version.

for localized artifacts using a conventional pipeline (i.e. endto-end training on images). Unfortunately, there are many
applications where the labeled data is scarce, and the only
way to obtain more data is by employing human experts,

which is expensive and subject to their availability. This realworld constraint hinders the widespread use of advanced
CNN architectures in such problems. On the other hand, the
nature of some of these problems may exhibit properties that
can be leveraged to increase the localization power as well as
the volume of useful training data. For example, the images
can be divided into smaller patches, and if the labels of these
patches could be derived from the original image, the number
of labeled training samples could be increased potentially by
a factor of, say 10-100, depending on how the patches are
formed. Then CNNs could be trained on the augmented data,
and image-level results could be derived by averaging patchlevel results. Such patch-level training followed by averaging
currently gives the state-of-art results for the problem of noreference image quality estimation (NR-IQA) (Kang et al.
2014) (see Experiments section for details on NR-IQA).
The effectiveness of this patch-level training technique is
only observed if one would assume that the artifacts are uniformly spread over the entire image, which is unfortunately
too strong a constraint in practice. Certain real-world problems such as NR-IQA and image forgery classification are
good examples where these strong constraints are often violated. Fig. 1(a) and 1(b) shows NR-IQA examples containing
original (left) and distorted (right) images. The red overlay
shows the distorted region in both the images. The distortions
are localized and thus only few image patches are responsible
for degrading its quality. Note that in the bottom image, the
flower in the central salient region is distorted whereas in the
upper image, some parts towards the lower non-salient region
are distorted, preserving the quality of salient face. This affects the perceived quality as the top image (score = 5.56) has
been judged to be of higher quality than the bottom one (score
= 3.53) based on the extensive subjective tests conducted in
(Ponomarenko et al. 2015). Interestingly, the type and the
severity of the distortion added are identical for both images.
Thus the image quality is dependent on various factors such
as distortion type, severity, affected patch locations and their
texture etc. This cannot be handled by the aforementioned
patch-level training technique. Similar observations can be
made about Fig. 1(c). The image needs to be categorized into
authentic or forged. The forgery is localized (shown by the
red bounding box) and may not be effectively captured by a
conventional deep CNN-pipeline or independent patch-level
training, especially when the training data is only in the order
of thousands. Patch-level training is only effective in case of
uniform distortion as shown in Fig. 1(b).
To combat these scenarios, we present a novel CNN-based
approach focused towards such type of data. Our approach
does not require the image and its patches to share a similar
distribution. It works well even if there is only one patch
per image which plays an important role in determining the
image label (relaxing the requirement that all patches from
an image should equally contribute to the decision, e.g. by
means of averaging patch-results). We evaluate our approach
on one synthetic data and two real-world, challenging vision
tasks - NR-IQA and image forgery classification. We will
demonstrate that the proposed method produces a state-of-art
performance for both the applications. We now explain the
problem setup followed by our approach.

Problem Setup
We consider a problem where our data has some localized information embedded in it that is crucial to getting the desired
output. Additionally, the quantity of available data is limited
and its artificial generation is non-trivial or even impossible.
We assume that inferring important patches is possible or this
information is provided as ground-truth.
Consider a database of images X containing N images,
and their labels, denoted by (Xi , Yi ) ∀ i. An image has m
j
th
th
patches, x1i , ..., xm
i . Here, xi denotes j patch in the i im1
m
age. We denote patch-level labels by yi , . . . , yi . Patch-level
labels can be inferred from the image or they can be a part of
ground-truth labels. Thus training on patches and then averaging all the patch scores to obtain the image label is a naive
way which, actually, works well in practice. However, we do
not allow the learner to understand the relation between patch
scores and the image score. In other words, the network cannot learn an optimal weighing strategy for all image regions,
especially when only few regions contain the important localized artifacts. Training on the entire image with a deep
stack of convolutional filters may achieve the desired result,
but then limited amounts of data prevents CNN from generalizing well. Thus, our problem becomes, given an image
Xi and its patches x1i , . . . , xm
i , first infer patch-level labels
- yi1 , . . . , yim . Subsequently, develop a CNN framework that
aggregates the information from the collection (xji , yij ) ∀i, ∀j
and forms a mapping function f : Xi → Yi , ∀ i.

Proposed Approach
Our approach has two CNN stages out of which the first one
is trained on a collection of labeled patches. Given a database
of labeled images - X , Y - we first extract all the patches and
derive their corresponding labels. As mentioned earlier, the
patch-level labels are either inferred from the image label or
are provided as ground-truth, if available.

Training the First Stage
The first stage CNN follows a conventional training pipeline
and is trained on a collection of labeled patches. We detail
the pre-processing steps and the training procedure in Experiments section as they vary by application.
It is well-known and empirically verified that deeper layers encode increasingly powerful features (Zeiler and Fergus
2014; Yosinski et al. 2015) that are semantically more meaningful (Zhou et al. 2014). Thus after training the first stage,
we extract the ReLU-activated responses of the last but one
layer for all the patches in the train and validation set. During
our experiments, we observed that ReLU-activated sparse
response provides better validation loss than the pre-ReLU
responses. This will be used as the input representation for
the second stage described as follows.

Training the Second Stage
A trained first stage CNN provides a D-dimensional representation obtained from the last but one layer for any given
image patch. Let us denote the m overlapped patches for
the ith image by x1i , . . . , xm
i . We extract the D-dimensional
representations from these m patches and arrange them in a

U × V × D hyper-image denoted by Hi . The arrangement is
done such that the (u, v) element of the hyper-image - Hiuv corresponds to the representation of (u, v) patch in the original image. In other words, each patch in the original image
now corresponds to a point in D-dimensions in the hyperimage. Note that U and V are smaller as compared with the
image height and width respectively, by a factor proportional
to the patch size and overlap factor.
We now train a second stage CNN on the hyper-images
and their labels, which shares architectural similarities with
its counterpart - the first stage CNN. We do not perform
mean-centering or any other pre-processing since these representations are obtained from normalized images.
Each pixel in the hyper-image being inputted to
the second stage CNN is of the form Hiuv =
f1 (f2 (. . . (fn (xuv
i )) . . .)) ∀ u, v, where f1 , . . . , fn represent
the non-linear operations (i.e. max-pooling, convolutions, ReLUs etc.) on an image as it makes a forward pass through the
first stage. Then in the second stage, the label is inferred as,
yi = g1 (g2 (. . . (gn (Hi )) . . .)), where Hi denotes the hyperimage being inputted to the second stage corresponding to
the ith image and g1 , . . . , gn denote the non-linear operations
in the second stage. The following equation expresses yi as a
highly nonlinear function of xuv
i ∀ u, v.
yi = g1 (g2 (. . . (gn ({Hi11 , . . . , Hiuv })) . . .)),
where, Hiuv = f1 (f2 (. . . (fn (xuv
i )) . . .)) ∀ u, v

(1)

This allows the multi-stage CNN to take decisions based on
context and by jointly considering all the image patches. Both
the stages combined also provide us with higher representational capacity than had we trained it simply with patches
followed by averaging.
Note that the convolutional filters of the second stage CNN
only operate on a small neighborhood at a time (usually
3 × 3), where each point in this neighborhood corresponds
to the D-dimensional representation of a patch. So if the
receptive field of a neuron in the first layer is 3 × 3, it is
looking at 9 patches arranged in a square grid. Thus filters
in the early layers learn simple relations between adjacent
patches whereas the deeper ones will start pooling in patch
statistics from all over the image to build a complex model
which relates the image label and all patches in an image.
This two-stage training scheme raises an issue that it needs
the first stage to produce a good representation of the data,
since the second stage is solely dependent on it. Any errors
in the initial stage may be propagated. However, in our experiments we find that the accuracy of the entire architecture is
always more than the accuracy obtained by:
1. Training end-to-end with a deep CNN.
2. Training on patches and averaging patch scores over the
entire image.
3. End-to-end fine-tuning of a pre-trained network.
This points to two possibilities. Firstly, the second stage is
powerful enough to handle slight irregularities in the first
stage representation. This is expected since CNNs are resilient even in the presence of noise or with jittered/occluded
images to some extent (Dodge and Karam 2016). Secondly,

inputting a hyper-image containing D-dimensional points to a
CNN results in a performance boost. For example, predicting
a quality score for the distorted images shown in Fig. 1(a)
and Fig. 1(b) can be viewed as a regression task with multiple
instances. A single patch having lower quality will not necessarily cause the entire image to have lower quality score.
Similarly, an image with multiple mildly distorted patches
at corners may have higher quality score than an image with
a single severely distorted patch placed in a salient position.
Thus quality score is related to the appearance/texture of all
patches, their locations, distortion severity etc. Our network
acquires location sensitivity to a certain extent (shown in experiments on synthetic data) as it learns on a U × V × D grid.
Distortion strengths of the individual patches are encoded in
the D-dimensions, which are unified by the second stage to
get desired output for a given image. On the other hand, an
end-to-end conventional network pipeline will need to learn
both - 1. the discriminative feature space and 2. the patches
which contribute to the image label. The patch-averaging
technique will fail as it will assign equal weight to all the
individual patches potentially containing distortions of different strengths. To summarize, the proposed architecture
attempts to learn the optimal mapping between image label
and spatially correlated patches of that image.

Testing
Firstly, we extract a fixed number of patches from an image,
extract their representations using the first stage and arrange
it in a U × V × D grid to form a hyper-image. Our approach
does not require resizing of input images. Instead, we change
the strides between patches at run-time to obtain the desired
shape. In applications where resizing images could change
the underlying meaning of images or introduce additional
artifacts (e.g. image quality, image forgery classification),
this approach could be useful. The procedure to compute
the strides at run-time is as follows. we first compute (or
assume) the maximum height and width among all the images in our dataset. If a test image exceeds those maximum
dimensions, we scale it isotropically so that both its dimensions meet the size constraints. Lets denote the maximum
height and width by M and N respectively. Thus the number
of patches required in the x and y direction of the grid are
M
N
npx = d sz
e, npy = d sz
e. Here, szy × szx is the patch size
x
y
that is used in the first stage. For any new M̂ × N̂ image,
the required strides (denoted by sx and sy ) to obtain a fixed
number of patches in the grid are as follows:

sx = rnd

N̂ − szx
npx − 1

!
, sy = rnd

M̂ − szy
n py − 1

!
(2)

After obtaining the hyper-image ∈ RU ×V ×D from the first
stage, we make a forward pass with it through the second
stage to obtain the desired output. In the upcoming sections,
we apply the proposed approach to a synthetic problem and
two real-world challenging vision tasks, review relevant literature and discuss about other aspects of our approach.

Experiments and Results
We evaluate our approach on a synthetic task and two challenging real-word problems - 1. no-reference image quality
assessment (NR-IQA) and 2. image forgery classification.
Apart from the dependence on localized image artifacts, an
additional common factor that aggravates the difficulty level
of both these problems is that the amount of data available is
scarce. Manual labeling is expensive as subject experts need
to be appointed to evaluate the image quality or to detect
forgery. Additionally, artificial generation of data samples is
non-trivial in both these cases. We will now begin by describing the setup for our synthetic task.
Synthetic task: While this task is constructed to be simple, we have included certain features that will examine the
effectiveness of the proposed approach. The task is to map
an image to a real-valued score that quantifies the artifacts
introduced in that image. The dataset contains 128 × 128
gray-scale images that have a constant colored background.
The color is chosen randomly from [0, 1]. We overlay between one to five patches on that image. Each patch contains
only two shades - a dark one ∈ [0, 0.5) and a bright one
∈ [0.5, 1]. A random percentage of pixels in a patch is made
bright and the others are set to dark. Finally, all the pixels are
scrambled. Size of each patch is 16 × 16.
Let us denote the synthetic image by S and the ith patch
by pi . The number of patches overlaid on S is η (≤ 5). Let
s0 denote the constant background value of S. pjk
i denotes
the (j, k) pixel of pi . The center of the patch pi is denoted by
ci . The image center is denoted
ĉ. The score of this image
pPby
η
2
can now be defined as 5 −
i=1 αi , where αi is:

! 

16
16 X
jk
X
|s
−
p
|
0
i
 + 1 − ||ci − ĉ||2 (3)
αi = 
16 × 16
distN
j=1
k=1

The first term computes the Manhattan distance between a
background pixel and all the pixels of a patch. This can be
viewed as a dissimilarity metric between the background and
a patch. The second term imposes a penalty if the patch is too
close to the image center. The term distN is a normalization
factor. If the patch lies at any of the four corners, then ||ci −
ĉ||2 = distN and the penalty reduces to zero. Higher score
indicates presence of lower artifacts in an image.
We perform another experiment to test the sensitivity of
our approach to the patch locations. To this end, we cre-

Table 1: Results of experiments on synthetic data
Experiment 1 on synthetic data
Approach
SROCC
PLCC
Patch-averaging
0.9132
0.8982
Proposed
0.9611 0.9586
Experiment 2 on synthetic data
Approach
Mean of SROCC and PLCC
Patch-averaging
0.665, 0.7, 0.544, 0.5, 0.439
Proposed
0.886,0.811,0.738,0.744,0.718

ated 1K images that all had identical background. The number of patches and their content (i.e. two shades and pixel
scrambling) were also fixed across all 1K images. The only
variable was their positions in the image. we compared our
two-stage approach and patch-averaging. Patch-averaging assigns nearly identical scores to all 1K images whereas our approach gives higher correlation with the ground truth. When
everything except patch positions was fixed, our approach
had higher correlation and slow degradation with increasing
number of patches. See Table 1 for results. Synthetic images
are provided in supplementary.
No-reference image quality assessment (NR-IQA): Images may get distorted due to defects in acquisition devices,
transmission-based errors etc. The task of IQA requires an
automated method to assess the visual quality of an image.
Conventional error metrics such as RMSE/PSNR cannot capture the correlation between image appearance and humanperception of the image quality. Two variants of this problem
exist - full-reference IQA and no-reference IQA (NR-IQA).
In the former task, an original image and its distorted counterpart is given. A quality score has to be assigned to the
distorted one with respect to the original image. Few representative approaches that try to solve this problem are SSIM
(Wang et al. 2004), MSSIM (Wang, Simoncelli, and Bovik
2003), FSIM (Zhang et al. 2011), VSI (Zhang, Shen, and Li
2014) etc. However, in real-world scenarios, we may not have
a perfect, non-distorted image available for comparison. Thus
NR-IQA variant has been proposed. In NR-IQA, a single image needs to be assigned a quality score with respect to a nondistorted, unobserved version of that image. This score must
correlate well with human perception of image quality. While
creating ground-truth for this problem, a constant value is
associated with a non-distorted image. This serves as a reference on the quality score scale. This problem involves developing a discriminative feature space to different kinds and degrees of distortions. Such setting is more suitable for learning
schemes, which is reflected by the fact that most approaches
used to tackle this problem belong to the learning paradigm.
Few of the representative approaches include BRISQUE (Mittal, Moorthy, and Bovik 2012), CORNIA (Ye et al. 2012;
Ye et al. 2013), DIIVINE (Moorthy and Bovik 2011), BLIINDS (Saad, Bovik, and Charrier 2012), CBIQ (Mittal,
Soundararajan, and Bovik 2013), LBIQ (Tang, Joshi, and
Kapoor 2011) and the current CNN-based state-of-art (Kang
et al. 2014).
We perform all our NR-IQA experiments on two widely
used datasets - 1. LIVE (Sheikh et al. 2005) containing 29
reference images, 779 distorted images, 5 distortion types
and 5-7 distortion levels. The images are gray-scale. Through
subjective evaluation tests, each user has assigned a score to
an image according to its perceptual quality. A metric named
difference of mean opinion scores (DMOS) ∈ [0, 100] was
then developed, where 0 indicates highest quality image. 2.
TID 2013 (Ponomarenko et al. 2015) has 25 reference RGB
images, 3000 distorted images, 24 distortion types and 5 distortion levels. In subjective tests, each user was told to assign
a score between 0-9 where 9 indicates best visual quality.
Mean opinion scores (MOS) were calculated and were provided as ground truth scores/labels for each image. Four of

the 24 distortions are common to LIVE and TID datasets.
LIVE has all uniform distortions whereas 12 distortions out
of total 24 in TID 2013 are non-uniform. See Fig. 1 for an
example of uniform/non-uniform distortion. The aim is to
learn a mapping between the images and the scores which
maximizes Spearman’s (SROCC), Pearson’s correlation coefficient (PLCC) and Kendall’s correlation as those are the
standard metrics used for IQA. Since the number of images
are so small, we run ours and competing algorithms for 100
splits to remove any data-split bias in all four experiments.
As a widely followed convention in IQA, we use 60% of
reference and distorted images for training, 20% each for
validation and testing everywhere.
The subjective tests conducted for these datasets are extensive. Extreme care was taken to make them statistically
unbiased and it is non-trivial to reproduce them. LIVE data
creation needed more than 25K human comparisons for a
total of 779 images. TID 2013 data creation had over 1M visual quality evaluations and 524, 340 comparisons. To avoid
geographical bias, the participants came from five countries.
In summary, the constraint of small data in such applications
comes from the real-world and it is difficult to generate new
data or conduct additional tests. The various experiments we
perform to evaluate our approach are as follows.
In all the experiments, before we feed training images to
the first stage CNN, we preprocess it following the approach
of (Mittal, Moorthy, and Bovik 2012) that performs local
contrast normalization as follows.
3
X
ˆ j) = I(i, j) − µ(i, j) , µ(i, j) =
I(i,
wk,l Ik,l (i, j),
σ(i, j) + 
k,l=−3
v
u 3
u X
2
wk,l (Ik,l (i, j) − µ(i, j)) .
and σ(i, j) = t
k,l=−3

(4)
Here, w is a 2-D circular symmetric Gaussian with 3 standard
deviations and normalized to unit volume.
Data generation: The data preparation method of (Kang
et al. 2014) is common for both datasets. It extracts overlapping 32 × 32 patches and then assigns them equal score as
that of the image. This strategy works well for (Kang et al.
2014) as they only handle LIVE and specific TID distortions
that are shared by LIVE. However, to handle non-uniform
distortions, we make a small but important modification to
their method. We compare the corresponding patches of the
original image and its distorted counterpart with the help of
SSIM (Wang et al. 2004). SSIM is used to measure structural similarity between two images and is robust to slight
alterations unlike RMSE. We keep all the patches belonging
to a distorted image that have low SSIM scores with their
original counterparts. This indicates low similarity between
patches which could only point to distortion. We select all
patches belonging to reference images. Finally, we make the
number of reference and distorted patches equal. We call this
method as selective patch training. We now describe certain
protocols common to all our NR-IQA experiments.
Training/testing pipeline: The method of (Kang et al.

2014) trains on 32 × 32 patches and averages patch-level
scores to get a score for an entire image. We train our first
stage on 32 × 32 patches obtained by our selection method.
The second stage is then trained by hyper-images that are
formed using the first-stage patch representations. In our first
three NR-IQA experiments, we use the same first stage as that
of (Kang et al. 2014) to be able to clearly assess the impact
of adding a second stage. Addition of a second stage entails
little overhead as on LIVE (TID) data, one epoch requires 23
(106) and 3 (43) seconds for both stages respectively. Both
the stages as well as (Kang et al. 2014) uses mean absolute
error as the loss function.
All the networks are trained using SGD with initial learning rate 0.005 and decaying at a rate of 10−5 with each update.
Nesterov momentum of 0.9 was used. The learning rate was
reduced by 10 when the validation error reached a plateau.
The first stage was trained for 80 epochs and training was
terminated if no improvement in validation error was seen
for 20 epochs. The second stage was trained for 200 epochs
with the termination criterion set at 40 epochs. Implementations were done in Theano on an Nvidia Tesla K40. We now
describe individual experiments. All the architectures used
for various experiments are listed in the supplementary.
Experiment 1: We evaluate ours and the competing approaches on LIVE. The input sizes for both stages are 32×32
and 24 × 23 × 800 respectively. Even though LIVE contains
only uniform distortions, our approach marginally improves
over (Kang et al. 2014) 100 splits. This could be due to the
better representational capacity of our two stage network as
all the image patches contain the same kind of distortion. The
results obtained for all the approaches are given in Table 2.
Experiment 2: Intuitively, our selective patch training
should give us a significant boost in case of TID 2013 data.
Since only few patches are noisy, assigning same score to
all patches will corrupt the feature space during training. To
verify this, we train on TID 2013 data using the approach of
(Kang et al. 2014) - with and without our selection strategy.
Input sizes for both stages are 32 × 32 × 3 and 23 × 31 ×
800. We also evaluate our two-stage network to show its
superiority over both these approaches. We find that selective
patch training boosts Spearman (SROCC) by 0.0992 and
Pearson correlation coefficient (PLCC) by 0.072. Two-stage
training further improves SROCC by 0.0265 and PLCC by
0.0111. The results are given in Table 2. Hereinafter, we
compare with (Kang et al. 2014) assisted with our selective
patch training to understand the benefits of the second stage.
Experiment 3: First, we take four distortions from TID
that are shared with LIVE (thus these are uniform). We
observe a marginal improvement here for similar reasons
as the first experiment. The second part is designed to
clearly show the adverse effects of non-uniform, localized
distortions on the correlations. Out of 24 distortions, we
already have four common ones. We add just two most
non-uniform distortions - 1. Non eccentricity pattern noise
and 2. Local block-wise distortions. On these six distortions, our approach significantly outperforms that of (Kang
et al. 2014) with patch selection. Thus to characterize nonuniform distortions, one needs to weight every patch differently, which is exactly what our second stage achieves.

Table 2: Results of NR-IQA experiments
Experiment 1 on LIVE data - 100 Splits
Approach
SROCC
PLCC
DIIVINE
0.916
0.917
BLIINDS-II
0.9306
0.9302
BRISQUE
0.9395
0.9424
CORNIA
0.942
0.935
CNN + Selective patch training
0.956
0.953
Proposed
0.9581 0.9585
Experiment 2 on TID data - 100 Splits
CNN
0.6618
0.6907
CNN + Selective patch training
0.761
0.7627
CNN + Selective patch training
0.7875 0.7738
+ Stage 2 CNN (proposed)
Experiment 3 on select distortions of TID - 100 splits
CNN + Selective
Proposed
patch training
# distortions
(SROCC,PLCC)
(SROCC, PLCC)
Four
0.92,0.921
0.932,0.932
Six
0.625,0.653
0.76,0.755
All (24)
0.761,0.763
0.788,0.774
Experiment 4 on TID data - 10 splits
Approach
SROCC
PLCC
Shallow end-to-end CNN
0.2392
0.4082
Deep end-to-end CNN
0.3952
0.52
Experiment 5 on TID using pre-trained VGG - 10 splits
VGG + patch-averaging
0.6236
0.6843
VGG + second stage CNN 0.6878
0.713

Finally, in the third part, we test on the entire TID 2013.
To the best of our knowledge, no other learning-based approach has attempted the entire data. The only approach we
are aware of that tested on TID is CORNIA (Ye et al. 2012;
Ye et al. 2013). However, even they skip two kinds of block
distortions. The reasons could be lack of training samples
or severe degradation in performance as observed here. We
compare our approach with (Kang et al. 2014) plus selective
patch training. The detailed results are listed in Table 2. The
architecture used was identical to that used in the second
experiment.
Experiment 4: We verify that training networks end-toend from scratch gives poor performance with such low
amounts of training data. We define a shallow and a deep
CNN of 8 and 14 layers respectively and train them end-toend on 384 × 512 images from TID 2013. Out of all the
experiments, this produces the worst performance, making
it clear that end-to-end training on such small data is not an
option. See Table 2 for results.
Experiment 5: A popular alternative when the training
data is scarce is to fine-tune a pre-trained network. We took
VGG-16, pre-trained on ILSVRC 2014. We used it as the first
stage to get patch representations. VGG-16 takes 224 × 224
RGB images whereas we have 32 × 32 RGB patches. Thus

we only consider layers till “conv4 3” and get its ReLUactivated responses. All the layers till “conv4 3” reduce a
patch to a size of 2 × 2 × 512. We append two dense layers
of 800 neurons each and train them from scratch. Rest of the
layers are frozen. Please refer to the Caffe VGG prototxt for
further architectural details. To train this network, we use
a batch size of 256 and a learning rate of 0.01. We average
the patch scores obtained from fine-tuned VGG and compute
the correlations over 5 splits. In principle, we should get a
performance boost by appending a second stage after VGG,
since it would pool in VGG features for all patches and
regress them jointly. We use a second stage CNN identical
to the one used in experiment 2. We observe that SROCC
and PLCC improves by 0.06 and 0.0287 respectively. For
detailed results, see Table 2. On the other hand, we see sharp
drop in performance for VGG despite it being deep and pretrained on ImageNet. The reasons for this could be two-fold.
As also observed in (Kang et al. 2014), the filters learned
on NR-IQA datasets turn out to be quite different than those
learned on ImageNet. Thus the semantic concepts represented
by the deeper convolutional layers of pre-trained VGG may
not be relevant for NR-IQA. Secondly, VGG performs a
simple mean subtraction on input images versus we do local
contrast normalization (LCN). The latter helps in enhancing
the discontinuities (e.g. edges, noise etc.) and suppresses
smooth regions, making LCN more suitable for NR-IQA.
Our extensive evaluations on NR-IQA shows that our approach is better at characterizing local distortions present in
an image. It improves on the current state-of-art (Kang et al.
2014) and other approaches, such as training a shallow/deep
network from scratch, or fine-tuning a pre-trained network.
Image forgery classification: In today’s age of social media, fake multimedia has become an issue of extreme importance. To combat this, it is necessary to improve detection systems to categorize fake posts. Here, we focus on
image forgery/tampering, which is defined as altering an
image by various means and then applying post-processing
(e.g. blurring) to conceal the effects of forging. Image tampering comes in various forms, for example, copy-move
forgery (Bayram, Sencar, and Memon 2009; Sutthiwan et
al. 2011) and manipulating JPEG headers (Farid 2009a;
He et al. 2006). Some other techniques have also been developed to detect forgeries from inconsistent lighting, shadows
(Fan et al. 2012; Kee and Farid 2010) etc. See the surveys
for more information (Bayram, Sencar, and Memon 2008;
Farid 2009b). However, the problem of tampering detection
from a single image without any additional information is
still eluding researchers. The current state-of-art uses a blockbased approach (Sutthiwan et al. 2011) which uses blockDCT features. It forms a Markov transition matrix from these
features and finally feeds them into a linear SVM. They carry
out their experiments on the CASIA-2 tampered image detection database 1 . It contains 7491 authentic and 5123 tampered
images of varying sizes as well as types.
Data generation: Given a database of authentic and (corresponding) tampered images, we first focus on getting the
contour of tampered region(s) by doing image subtraction
1

http://forensics.idealtest.org/casiav2/

500-D
Patch from
first image

CNN1

C1 ∈ R500

Figure 2: Authentic (left) and tampered (middle) image. The
resultant contour of the tampered region (right). Please zoomin and view in color.

Approach

Classification accuracy

End-to-end CNN
Current State-of-art
(Sutthiwan et al. 2011)
Proposed two stage CNN

75.22%
79.20%
83.11%

followed by basic morphological operations. The resultant
contour is shown in Fig. 2. We sample 15 pixels along this
contour and crop 64 × 64 patches by keeping the sampled
points as the patch-centroids. Similar to (Sutthiwan et al.
rd
th
2011), we train on 23 of the data and we use 16 data
each for validation and testing. We only subtract the mean
of training patches from each patch and do on-the-fly data
augmentation by flipping the images. Instead of categorizing
patches as authentic/tampered, we develop a ranking-based
formulation, where the rank of an authentic patch is higher
than its counterpart. Note that during testing, we are only
given a single image to be classified as authentic or forged
and thus contour of forged region cannot be found (or used).
Experiment 1: We train an end-to-end deep CNN that
takes an entire image as input and categorizes it as authentic
or tampered. The architecture used is given in supplementary.
It takes 256 × 384 RGB images as input. This size is chosen
since it needs minimum number of resize operations over the
dataset. The classification accuracy is shown in Table 3.
Experiment 2: Our first stage CNN learns to rank authentic patches higher than tampered ones. We propose ranking because every patch may contain different amount of
forged region or blur. This CNN has two identical channels that share weights. Its architecture is given in supplementary. Let us denote the last dense layer features obtained from an authentic and a tampered patch by CAu and
CT p respectively. We need to learn a weight vector such
that wT CAu − wT CT p > 0. However, the network trains
poorly if we keep feeding the first channel with authentic
patches and the second with the tampered ones. Shuffling
of patches is necessary in order to achieve convergence.
We assign a label of 1 if two channels get authentic and
tampered patches (in that order), else -1. Thus we need
d(C1 , C2 ) = w2T y · (f (w1T · (C1 − C2 )) > 0, where Ci
is the feature from the ith channel, whereas y ∈ {−1, 1}
denotes the label. The transformations achieved through two
dense layers and an ReLU are denoted by w2 (·), w1 (·) and
f (·) respectively, as shown in Fig. 3. The loss function be-

−

d

d = w2T · (f (w1T · (C1 − C2 )))
Patch from
second
image

CNN2

C2 ∈ R500

Table 3: Results of image forgery classification

500-D

C1 − C2

f (w1T ·(C1 −C2 ))

Figure 3: Proposed channel architecture. Weight sharing occurs between both channels. Please zoom in to see details.
comes, L = max(0, δ − y · d(C1 , C2 )). The term max(0, ·)
is necessary to ensure that only non-negative loss gets backpropagated. The δ(= 3) is a user-defined parameter that
avoids trivial solutions and introduces class separation.
Stage 1 representation should discriminate between neighborhood patterns along a tampered and an authentic edge
(since we chose patches centered on the contour of tampered
region). Given an image, we extract patches and form the
hyper-image required to train the second stage. We use binary
labels, where 1 denotes authentic image and vice-versa along
with a binary cross-entropy loss. Architecture of the second
stage is provided in supplementary. To overcome class imbalance, we weigh the samples accordingly. We compare our
approach with an end-to-end CNN network (experiment 1)
and the current state-of-art in passive, generic image forgery
classification (Sutthiwan et al. 2011). CNN-baseline gives
the worst performance followed by the current state-of-art.
This is expected since the latter extracts block-level DCT features whereas the CNN-baseline tries to learn from an entire
image - a considerably difficult task especially when the tampered/forged parts are localized and well camouflaged. Our
hybrid approach beats the CNN-baseline and the state-of-art
by 8% and 4% respectively. All these experiments underlines
the importance of collectively learning from image patches
when the data is scarce and shows flexibility of our approach.

Conclusion
We presented the notion of CNN-based hyper-image representations. Our training scheme involving these hyper-images
excels in scenarios where the label is dependent on the localized artifacts in an image. In these networks, the first stage is
only responsible for learning the discriminative representations of small image patches. The second stage collectively
considers all the patches of an image unlike many other previous approaches. It optimally weighs and pools all the patches,
and develops a mapping between them and the image label.
Our approach enables us to train networks with greater representational capacity than their conventional counterparts.
We observe in all our experiments that the second stage always provides us with a significant improvement. We apply
our approach to a synthetic and two challenging vision tasks
- NR-IQA and image forgery classification. Our approach
comfortably outperforms other CNN-baselines as well as the

existing state-of-art approaches.

Table 4: Architectures of deep networks used in this paper. The term C(n, r, s) denotes r × r, n “same” convolutional filters
with stride s. We omit r and s when r = 3 and s = 1. MP(n) is a max-pooling reduce the image size by a factor of n. FC(n) and
Drop(n) denote dense layer with n neurons and dropout rate of n respectively.
Architecture
Synthetic stage 1
Synthetic stage 2
LIVE/TID stage 1

Layer descriptions
Input(128, 128) – C(16) – MP(2) – C(32) – MP(2) – 2 × C(48) – MP(2) – 2 × C(64) –
MP(2) – 2 × C(128) – MP(2) – FC(400) – Drop(0.5) – FC(400) – Drop(0.5) – FC(1,‘linear’)
Input(10,10,400) – 2 × C(16) – MP(2) – 2 × C(32) – MP(2) – 2 × C(64) –
MP(2) – FC(400) – Drop(0.5) – FC(400,‘tanh’) – Drop(0.5) – FC(1,‘linear’)
Please refer to Table 5 on the next page

Input(24,23,800) – 2 × C(32) – MP(2) – 2 × C(48) – MP(2) – 2 × C(64) – MP(2) –
2 × C(128) – MP(2) – FC(500) – Drop(0.5) – FC(500,‘tanh’) – Drop(0.5) – FC(1,‘linear’)
Input(23,31,800) – 2 × C(64) – MP(2) – 2 × C(64) – MP(2) – 2 × C(128) – MP(2) –
TID stage 2
2 × C(128) – MP(2) – FC(500) – Drop(0.5) – FC(500,‘tanh’) – Drop(0.5) – FC(1,‘linear’)
Shallow end-to-end
Input(384, 512, 3) – C(32, 7, 2) – MP(2) – C(64, 7, 1) – MP(2) – 2 * ( C(128) – MP(2) )
network for TID (Expt 4)
– 2 * ( C(256) ) – MP(2) – FC(400) – Drop(0.25) – FC(400, ’tanh’) – Drop(0.25) – FC(1,’linear’)
Input(384, 512, 3) – C(32) – MP(2) – C(64) – MP(2) – 2 * ( C(128) – MP(2) ) Deep end-to-end network
2 * ( C(256) - C(256) - MP(2) ) – 2 * (C(512) – C(512) – MP(2) ) –
for TID (Expt 4)
FC(400) - Drop(0.25) – FC(400, ’tanh’) – Drop(0.25) – FC(1,’linear’)
Input(64,64,3) – 2 × C(64) – MP(2) – 2 × C(128) – MP(2) – 2 × C(128) –
Forgery channel
MP(2) – 2 × C(256) – MP(2) – FC(500) – Drop(0.5) – FC(500) – Drop(0.5)
Input(15,15,500) – 3 × C(64) – MP(2) – 3 × C(128) – MP(2) – 3 × C(256) –
Forgery stage 2
MP(2) – FC(800) – Drop(0.5) – FC(800) – Drop(0.5) – FC(1,‘sigmoid’)
Forgery endInput(256,384,3) – C(32) – MP(2) – C(64) – MP(2) – 2 × C(64) – MP(2) – 2 × C(128) – MP(2) – 2 ×
to-end CNN
C(128) – MP(2) – 2 × C(256) – MP(2) –FC(500) – Drop(0.5) – FC(500) – Drop(0.5) – FC(1,‘sigmoid’)
LIVE stage 2

Table 5: First stage architecture for LIVE/TID data used in this paper.
Layer Details
Input – 32 × 32 × 3 (for TID) and 32 × 32 (for LIVE)

Connected to
–

Convolutional layer – 50 × 7 × 7, name = ’Conv1’

Input

Max-pooling: size = (26,26), stride = (26,26) , name = ’MaxPool’

Conv1

Min-pooling: size = (26,26), stride = (26,26), name = ’MinPool’

Conv1

Concatenation layer: inputs = (MinPool, MaxPool), name = ’Concat’

–

Fully-connected layer: 800 nodes, name = ’FC1’

Concat

Fully-connected layer: 800 nodes, name = ’FC2’

FC1

Output node, name = ’output’

FC2

Loss = ’Mean absolute error’

–

References
[Bayram, Sencar, and Memon 2008] Bayram, S.; Sencar, H. T.; and
Memon, N. 2008. A survey of copy-move forgery detection techniques. In IEEE Western New York Image Processing Workshop,
538–542. Citeseer.
[Bayram, Sencar, and Memon 2009] Bayram, S.; Sencar, H. T.; and
Memon, N. 2009. An efficient and robust method for detecting
copy-move forgery. In 2009 IEEE International Conference on
Acoustics, Speech and Signal Processing, 1053–1056. IEEE.
[Dai et al. 2016] Dai, J.; He, K.; ; and Sun, J. 2016. Instance-aware
semantic segmentation via multi-task network cascades.
[Dodge and Karam 2016] Dodge, S., and Karam, L. 2016. Understanding how image quality affects deep neural networks. arXiv
preprint arXiv:1604.04004.
[Fan et al. 2012] Fan, W.; Wang, K.; Cayre, F.; and Xiong, Z.
2012. 3d lighting-based image forgery detection using shape-fromshading. In Signal Processing Conference (EUSIPCO), 2012 Proceedings of the 20th European, 1777–1781. IEEE.
[Farid 2009a] Farid, H. 2009a. Exposing digital forgeries from jpeg
ghosts. IEEE Transactions on information forensics and security
4(1):154–160.
[Farid 2009b] Farid, H. 2009b. Image forgery detection–a survey.
[Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza,
M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio,
Y. 2014. Generative adversarial nets. In Advances in Neural
Information Processing Systems, 2672–2680.
[He et al. 2006] He, J.; Lin, Z.; Wang, L.; and Tang, X. 2006. Detecting doctored jpeg images via dct coefficient analysis. In European
conference on computer vision, 423–435. Springer.
[He et al. 2015] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015.
Deep residual learning for image recognition. arXiv preprint
arXiv:1512.03385.
[Kang et al. 2014] Kang, L.; Ye, P.; Li, Y.; and Doermann, D. 2014.
Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 1733–1740.
[Kee and Farid 2010] Kee, E., and Farid, H. 2010. Exposing digital
forgeries from 3-d lighting environments. In 2010 IEEE International Workshop on Information Forensics and Security, 1–6. IEEE.
[Krizhevsky, Sutskever, and Hinton 2012] Krizhevsky,
A.;
Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in
neural information processing systems, 1097–1105.
[Mittal, Moorthy, and Bovik 2012] Mittal, A.; Moorthy, A. K.; and
Bovik, A. C. 2012. No-reference image quality assessment in
the spatial domain. Image Processing, IEEE Transactions on
21(12):4695–4708.
[Mittal, Soundararajan, and Bovik 2013] Mittal, A.; Soundararajan,
R.; and Bovik, A. C. 2013. Making a completely blind image
quality analyzer. IEEE Signal Processing Letters 20(3):209–212.
[Moorthy and Bovik 2011] Moorthy, A. K., and Bovik, A. C. 2011.
Blind image quality assessment: From natural scene statistics
to perceptual quality. IEEE Transactions on Image Processing
20(12):3350–3364.
[Ponomarenko et al. 2015] Ponomarenko, N.; Jin, L.; Ieremeiev, O.;
Lukin, V.; Egiazarian, K.; Astola, J.; Vozel, B.; Chehdi, K.; Carli,
M.; Battisti, F.; et al. 2015. Image database tid2013: Peculiarities,
results and perspectives. Signal Processing: Image Communication
30:57–77.

[Recasens et al. 2015] Recasens, A.; Khosla, A.; Vondrick, C.; and
Torralba, A. 2015. Where are they looking? In Advances in Neural
Information Processing Systems, 199–207.
[Saad, Bovik, and Charrier 2012] Saad, M. A.; Bovik, A. C.; and
Charrier, C. 2012. Blind image quality assessment: A natural scene
statistics approach in the dct domain. Image Processing, IEEE
Transactions on 21(8):3339–3352.
[Salimans et al. 2016] Salimans, T.; Goodfellow, I.; Zaremba, W.;
Cheung, V.; Radford, A.; and Chen, X. 2016. Improved techniques
for training gans. arXiv preprint arXiv:1606.03498.
[Sheikh et al. 2005] Sheikh, H. R.; Wang, Z.; Cormack, L.; and
Bovik, A. C. 2005. Live image quality assessment database release
2.
[Simonyan and Zisserman 2014] Simonyan, K., and Zisserman, A.
2014. Very deep convolutional networks for large-scale image
recognition. arXiv preprint arXiv:1409.1556.
[Sutthiwan et al. 2011] Sutthiwan, P.; Shi, Y. Q.; Zhao, H.; Ng, T.T.; and Su, W. 2011. Markovian rake transform for digital image
tampering detection. In Transactions on data hiding and multimedia
security VI. Springer. 1–17.
[Taigman et al. 2014] Taigman, Y.; Yang, M.; Ranzato, M.; and
Wolf, L. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 1701–1708.
[Tang, Joshi, and Kapoor 2011] Tang, H.; Joshi, N.; and Kapoor, A.
2011. Learning a blind measure of perceptual image quality. In
Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 305–312. IEEE.
[Wang et al. 2004] Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility
to structural similarity. Image Processing, IEEE Transactions on
13(4):600–612.
[Wang, Simoncelli, and Bovik 2003] Wang, Z.; Simoncelli, E. P.;
and Bovik, A. C. 2003. Multiscale structural similarity for image quality assessment. In Signals, Systems and Computers, 2004.
Conference Record of the Thirty-Seventh Asilomar Conference on,
volume 2, 1398–1402. Ieee.
[Ye et al. 2012] Ye, P.; Kumar, J.; Kang, L.; and Doermann, D. 2012.
Unsupervised feature learning framework for no-reference image
quality assessment. In Computer Vision and Pattern Recognition
(CVPR), 2012 IEEE Conference on, 1098–1105. IEEE.
[Ye et al. 2013] Ye, P.; Kumar, J.; Kang, L.; and Doermann, D. 2013.
Real-time no-reference image quality assessment based on filter
learning. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 987–994.
[Yosinski et al. 2015] Yosinski, J.; Clune, J.; Nguyen, A.; Fuchs, T.;
and Lipson, H. 2015. Understanding neural networks through deep
visualization. arXiv preprint arXiv:1506.06579.
[Zeiler and Fergus 2014] Zeiler, M. D., and Fergus, R. 2014. Visualizing and understanding convolutional networks. In Computer
vision–ECCV 2014. Springer. 818–833.
[Zhang et al. 2011] Zhang, L.; Zhang, L.; Mou, X.; and Zhang, D.
2011. Fsim: A feature similarity index for image quality assessment.
IEEE Transactions on Image Processing 20(8):2378–2386.
[Zhang, Shen, and Li 2014] Zhang, L.; Shen, Y.; and Li, H. 2014.
Vsi: A visual saliency-induced index for perceptual image quality
assessment. Image Processing, IEEE Transactions on 23(10):4270–
4281.
[Zheng et al. 2015] Zheng, S.; Jayasumana, S.; Romera-Paredes, B.;
Vineet, V.; Su, Z.; Du, D.; Huang, C.; and Torr, P. H. 2015. Condi-

tional random fields as recurrent neural networks. In Proceedings of
the IEEE International Conference on Computer Vision, 1529–1537.
[Zhou et al. 2014] Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.;
and Torralba, A. 2014. Object detectors emerge in deep scene cnns.
arXiv preprint arXiv:1412.6856.

