The latest generation of Convolutional Neural Networks (CNN) have achieved
impressive results in challenging benchmarks on image recognition and object
detection, significantly raising the interest of the community in these
methods. Nevertheless, it is still unclear how different CNN methods compare
with each other and with previous state-of-the-art shallow representations such
as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts
a rigorous evaluation of these new techniques, exploring different deep
architectures and comparing them on a common ground, identifying and disclosing
important implementation details. We identify several useful properties of
CNN-based representations, including the fact that the dimensionality of the
CNN output layer can be reduced significantly without having an adverse effect
on performance. We also identify aspects of deep and shallow methods that can
be successfully shared. In particular, we show that the data augmentation
techniques commonly applied to CNN-based methods can also be applied to shallow
methods, and result in an analogous performance boost. Source code and models
to reproduce the experiments in the paper is made publicly available.
Ken Chatfield Karen Simonyan Andrea Vedaldi Andrew Zisserman
1
Return of the Devil in the Details:
Delving Deep into Convolutional Nets
Ken Chatﬁeld, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford
fken,karen,vedaldi,az g@robots.ox.ac.uk
Abstract —The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in chal-
lenging benchmarks on image recognition and object detection, signiﬁcantly raising the interest of the community in
these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous
state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper
conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on
a common ground, identifying and disclosing important implementation details. We identify several useful properties
of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced
signiﬁcantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods
that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to
CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source
code and models to reproduce the experiments in the paper is made publicly available.
F
1 I NTRODUCTION
PERHAPS the single most important design
choice in current state-of-the-art image classiﬁ-
cation and object recognition systems is the choice
of visual features, or image representation. In fact,
most of the quantitative improvements to image
understanding obtained in the past dozen years can
be ascribed to the introduction of improved repre-
sentations, from the Bag-of-Visual-Words (BoVW) [1],
[2] to the (Improved) Fisher Vector (IFV) [3]. A
common characteristic of these methods is that
they are largely handcrafted . They are also relatively
simple, comprising dense sampling of local image
patches, describing them by means of visual de-
scriptors such as SIFT, encoding them into a high-
dimensional representation, and then pooling over
the image. Recently, these handcrafted approaches
have been substantially outperformed by the in-
troduction of the latest generation of Convolutional
Neural Networks (CNNs) [4] to the computer vision
ﬁeld. These networks have a substantially more
sophisticated structure than standard representa-
tions, comprising several layers of non-linear fea-
ture extractors, and are therefore said to be deep
(in contrast, classical representation will be referred
to as shallow ). Furthermore, while their structureis handcrafted, they contain a very large number
of parameters learnt from data. When applied to
standard image classiﬁcation and object detection
benchmark datasets such as ImageNet ILSVRC [5]
and PASCAL VOC [6] such networks have demon-
strated excellent performance [7], [8], [9], [10], [11],
signiﬁcantly better than standard image encod-
ings [12].
Despite these impressive results, it remains un-
clear how different deep architectures compare to
each other and to shallow computer vision meth-
ods such as IFV . Most papers did not test these
representations extensively on a common ground,
so a systematic evaluation of the effect of differ-
ent design and implementation choices remains
largely missing. As noted in our previous work [12],
which compared the performance of various shal-
low visual encodings, the performance of computer
vision systems depends signiﬁcantly on implementation
details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use
of very large scale datasets, GPU computation, and
data augmentation (also known as data jittering or
virtual sampling). These improvements could also
transfer to shallow representations such as the IFV ,
potentially explaining a part of the performancearXiv:1405.3531v4  [cs.CV]  5 Nov 2014
2
gap [14].
In this study we analyse and empirically clarify
these issues, conducting a large set of rigorous
experiments (Sect. 4), in many ways picking up
the story where it last ended in [12] with the
comparison of shallow encoders. We focus on meth-
ods to construct image representations ,i.e. encod-
ing functions mapping an image Ito a vector
(I)2Rdsuitable for analysis with a linear classi-
ﬁer, such as an SVM. We consider three scenarios
(Sect. 2, Sect. 3): shallow image representations,
deep representations pre-trained on outside data,
and deep representation pre-trained and then ﬁne-
tuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that
are nevertheless more often found in combina-
tion with CNNs [13] or, alternatively, with shallow
encoders [12], porting them with mutual beneﬁt.
These are (Sect. 2): the use of colour information ,
feature normalisation , and, most importantly, the
use of substantial data augmentation . We also de-
termine scenario-speciﬁc best-practices , improving
the ones in [12], [15] and others, including dimen-
sionality reduction for deep features. Finally, we
achieve performance competitive with the state
of the art [16], [17] on PASCAL VOC classiﬁcation
using less additional training data and signiﬁcantly
simpler techniques. As in [12], the source code and
models to reproduce all experiments in this paper
is available on the project website1.
2 S CENARIOS
This section introduces the three types of image
representation (I)considered in this paper, de-
scribing them within the context of three different
scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation
and feature normalisation, together with the linear
classiﬁer (trained with a standard hinge loss). We
also specify here the benchmark datasets used in
the evaluation.
2.1 Scenario 1: Shallow representation (IFV)
Our reference shallow image representation is the
IFV [3]. Our choice is motivated by the fact that
IFV usually outperforms related encoding methods
such as BoVW, LLC [12], and VLAD [18]. Given an
imageI, the IFVFV(I)is obtained by extracting
1. http://www.robots.ox.ac.uk/vgg/research/deep eval/a dense collection of patches and corresponding
local descriptors xi2RD(e.g. SIFT) from the image
at multiple scales. Each descriptor xiis then soft-
quantized using a Gaussian Mixture Model with
Kcomponents. First and second order differences
between each descriptor xiand its Gaussian cluster
meankare accumulated in corresponding blocks
uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and
covariance, leading to a 2KD-dimensional image
representation FV(I) = [u>
1;v>
1;:::u>
K;v>
K]>:The
improved version of the Fisher vector involves post-
processing FVby computing the signed square-
root of its scalar components and normalising the
result to a unit `2norm. The details of this con-
struction can be found in [3]; here we follow the
notation of [12].
2.2 Scenario 2: Deep representation (CNN) with
pre-training
Our deep representations are inspired by the suc-
cess of the CNN of Krizhevsky et al. [13]. As shown
in [7], [19], the vector of activities CNN(I)of the
penultimate layer of a deep CNN, learnt on a large
dataset such as ImageNet [5], can be used as a pow-
erful image descriptor applicable to other datasets.
Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing
the best one remains an open question. Many are in-
spired by [13]: DeCAF [7], [11], Caffe [20], Oquab et
al. [8]. Others use larger networks with a smaller
stride of the ﬁrst convolutional layer: Zeiler and
Fergus [19] and OverFeat [10], [9]. Other differences
include the CNN pre-training protocols. Here we
adopt a single learning framework and experiment
with architectures of different complexity exploring
their performance-speed trade-off.
2.3 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
In Scenario 2 features are trained on one (large)
dataset and applied to another (usually smaller).
However, it was demonstrated [11] that ﬁne-tuning
a pre-trained CNN on the target data can sig-
niﬁcantly improve the performance. We consider
this scenario separately from that of Scenario 2, as
the image features become dataset-speciﬁc after the
ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.
2.4.1 Data augmentation
Data augmentation is a method applicable to shal-
low and deep representations, but that has been
so far mostly applied to the latter [13], [19]. By
augmentation we mean perturbing an image Iby
transformations that leave the underlying class un-
changed ( e.g. cropping and ﬂipping) in order to
generate additional examples of the class. Augmen-
tation can be applied at training time, at test time,
or both. The augmented samples can either be taken
as-is or combined to form a single feature, e.g. using
sum/max-pooling or stacking.
2.4.2 Linear predictors
All the representations (I)in the three scenarios
are used to construct linear predictorshw;(I)ifor
each class to be recognized. These predictors are
learnt using Support Vector Machines (SVM) by
ﬁtting wto the available training data by mini-
mizing an objective function balancing a quadratic
regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of
the data. Here we use the same learning framework
with all representations. It is common experience
that linear classiﬁers are particularly sensitive to
thenormalisation of the data and that, in particular,
SVMs tend to beneﬁt from `2normalisation [3] (an
interpretation is that after normalisation the inner
product corresponds to the cosine similarly).
2.5 Benchmark data
As reference benchmark we use the PASCAL
VOC [6] data as already done in [12]. The VOC-
2007 edition contains about 10,000 images split into
train, validation, and test sets, and labelled with
twenty object classes. A one-vs-rest SVM classiﬁer
for each class is learnt and evaluated independently
and the performance is measured as mean Average
Precision (mAP) across all classes. The VOC-2012
edition contains roughly twice as many images
and does not include test labels; instead, evaluation
uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 chal-
lenge dataset. This contains 1,000 object categories
from ImageNet [5] with roughly 1.2M training im-
ages, 50,000 validation images, and 100,000 testimages. Performance is evaluated using the top-5
classiﬁcation error. Finally, we also evaluate over
theCaltech-101 and Caltech-256 image classiﬁca-
tion benchmarks [21], [22]. For Caltech-101, we
followed the protocol of [12], and considered three
random splits into training and testing data, each
of which comprises 30 training and up to 30 testing
images per class. For Caltech-256, two random
splits were generated, each of which contains 60
training images per class, and the rest are used for
testing. On both Caltech datasets, performance is
measured using mean class accuracy.
3 D ETAILS
This section gives the implementation details of the
methods introduced in Sect. 2.
3.1 Improved Fisher Vector details
Our IFV representation uses a slightly improved
setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction
with a stride of 3 pixels at 7 different scales withp
2
scale increments. These features are square-rooted
as suggested by [24], and decorrelated and reduced
in dimension from 128Dto80Dusing PCA. A
GMM with K= 256 components is learnt from fea-
tures sampled from the training images. Hence the
Fisher Vector FV(I)has dimension 2KD= 40;960.
Before use in classiﬁcation, the vector is signed-
square-rooted and l2-normalised (square rooting
correspond to the Hellinger’s kernel map [25]). As
in [12], square-rooting is applied twice, once to the
raw encodings, and once again after sum pooling
and normalisation. In order to capture weak geo-
metrical information, the IFV representation is used
in a spatial pyramid [26]. As in [12], the image is
divided into 11,31, and 22spatial subdivisions
and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.
In addition to this standard formulation, we ex-
periment with a few modiﬁcations. The ﬁrst one
is the use of intra-normalisation of the descriptor
blocks, an idea recently proposed for the VLAD
descriptor [27]. In this case, the `2normalisation
is applied to the individual sub-blocks (uk;vk)of
the vector FV(I), which helps to alleviate the
local feature burstiness [28]. In the case of the
improved intra-normalised features, it was found
4
that applying the square-rooting only once to the
ﬁnal encoding produced the best results.
The second modiﬁcation is the use of spatially-
extended local descriptors [23] instead of a spatial
pyramid. Here descriptors xiare appended with
their image location (xi;yi)before quantization
with the GMM. Formally, xiis extended, after PCA
projection, with its normalised spatial coordinates:
[x>
i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes
both appearance and location, this allows for spatial
information to be captured directly by the soft-
quantization process. This method is signiﬁcantly
more memory-efﬁcient than using a spatial pyra-
mid. Speciﬁcally, the PCA-reduced SIFT features
are spatially augmented by appending (x;y)yield-
ingD= 82 dimensional descriptors pooled in a
2KD= 41;984dimensional IFV .
The third modiﬁcation is the use of colour fea-
tures in addition to SIFT descriptors. While colour
information is used in CNNs [13] and by the orig-
inal FV paper [3], it was not explored in our pre-
vious comparison [12]. We do so here by adopting
the same Local Colour Statistics (LCS) features as
used by [3]. LCS is computed by dividing an input
patch into a 44spatial grid (akin to SIFT), and
computing the mean and variance of each of the
Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.
3.2 Convolutional neural networks details
The CNN-based features are based on three CNN
architectures representative of the state of the art
(shown in Table 1) each exploring a different accu-
racy/speed trade-off. To ensure a fair comparison
between them, these networks are trained using the
same training protocol and the same implemen-
tation, which we developed based on the open-
source Caffe framework [20]. `2-normalising the
CNN features CNN(I)before use in the SVM was
found to be important for performance.
Our Fast (CNN-F) architecture is similar to the
one used by Krizhevsky et al . [13]. It comprises
8 learnable layers, 5 of which are convolutional,
and the last 3 are fully-connected. The input image
size is 224224. Fast processing is ensured by the
4 pixel stride in the ﬁrst convolutional layer. The
main differences between our architecture and that
of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).
Our Medium (CNN-M) architecture is similar
to the one used by Zeiler and Fergus [19]. It is
characterised by the decreased stride and smaller
receptive ﬁeld of the ﬁrst convolutional layer, which
was shown to be beneﬁcial on the ILSVRC dataset.
At the same time, conv2 uses larger stride (2 instead
of 1) to keep the computation time reasonable. The
main difference between our net and that of [19] is
we use less ﬁlters in the conv4 layer (512 vs. 1024).
Our Slow (CNN-S) architecture is related to the
‘accurate’ network from the OverFeat package [10].
It also uses 77ﬁlters with stride 2in conv1.
Unlike CNN-M and [19], the stride in conv2 is
smaller (1 pixel), but the max-pooling window in
conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous
architectures ([10] used 6), and less ﬁlters in conv5
(512 instead of 1024); we also incorporate an LRN
layer after conv1 ([10] did not use contrast normal-
isation).
3.2.1 CNN training
In general, our CNN training procedure fol-
lows that of [13], learning on ILSVRC-2012 us-
ing gradient descent with momentum. The hyper-
parameters are the same as used by [13]: momen-
tum0:9; weight decay 5104; initial learning rate
102, which is decreased by a factor of 10, when
the validation error stop decreasing. The layers are
initialised from a Gaussian distribution with a zero
mean and variance equal to 102. We also employ
similar data augmentation in the form of random
crops, horizontal ﬂips, and RGB colour jittering.
Test time crop sampling is discussed in Sect. 3.3;
at training time, 224224 crops are sampled ran-
domly, rather than deterministically. Thus, the only
notable difference to [13] is that the crops are taken
from the whole training image P256;P256,
rather than its 256256 centre. Training was per-
formed on a single NVIDIA GTX Titan GPU and
the training time varied from 5 days for CNN-F to
3 weeks for CNN-S.
3.2.2 CNN ﬁne-tuning on the target dataset
In our experiments, we ﬁne-tuned CNN-S using
VOC-2007, VOC-2012, or Caltech-101 as the target
data. Fine-tuning was carried out using the same
framework (and the same data augmentation), as
we used for CNN training on ILSVRC. The last
5
Arch. conv1 conv2 conv3 conv4 conv5 full6 full7 full8
CNN-F64x11x11 256x5x5 256x3x3 256x3x3 256x3x3 4096 4096 1000
st. 4, pad 0 st. 1, pad 2 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-M96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 2, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-S96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- soft-
LRN, x3 pool x2 pool - - x3 pool out out max
TABLE 1
CNN architectures. Each architecture contains 5 convolutional layers (conv 1–5) and three
fully-connected layers (full 1–3). The details of each of the convolutional layers are given in three sub-rows:
the ﬁrst speciﬁes the number of convolution ﬁlters and their receptive ﬁeld size as “num x size x size”; the
second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local
Response Normalisation (LRN) [13] is applied, and the max-pooling downsampling factor. For full 1–3, we
specify their dimensionality, which is the same for all three architectures. Full6 and full7 are regularised
using dropout [13], while the last layer acts as a multi-way soft-max classiﬁer. The activation function for all
weight layers (except for full8) is the REctiﬁcation Linear Unit (RELU) [13].
fully-connected layer (conv8) has output dimen-
sionality equal to the number of classes, which
differs between datasets, so we initialised it from
a Gaussian distribution (as used for CNN training
above). Now we turn to dataset-speciﬁc ﬁne-tuning
details.
VOC-2007 and VOC-2012. Considering that PAS-
CAL VOC is a multi-label dataset ( i.e. a single
image might have multiple labels), we replaced the
softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss
as used in the SVM experiments) and ranking hinge
loss. Both losses deﬁne constraints on the scores of
positive (Ipos) and negative ( Ineg) images for each
class:wc(Ipos)>1;wc(Ineg)<1 +for the
classiﬁcation loss, wc(Ipos)> w c(Ineg) + 1
for the ranking loss ( wcis thec-th row of the
last fully-connected layer, which can be seen as a
linear classiﬁer on deep features (I);is a slack
variable). Our ﬁne-tuned networks are denoted as
“CNN S TUNE-CLS” (for the classiﬁcation loss)
and “CNN S TUNE-RNK” (for the ranking loss).
In the case of both VOC datasets, the training
and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we
controlled for over-ﬁtting by using lower initial
learning rates for the ﬁne-tuned hidden layers.The learning rate schedule for the last layer /
hidden layers was: 102=104!103=104!
104=104!105=105.
Caltech-101 dataset contains a single class label
per image, so ﬁne-tuning was performed using the
softmax regression loss. Other settings (including
the learning rate schedule) were the same as used
for the VOC ﬁne-tuning experiments.
3.2.3 Low-dimensional CNN feature training
Our baseline networks (Table 1) have the same
dimensionality of the last hidden layer (full7): 4096.
This design choice is in accordance with the state-
of-the-art architectures [13], [19], [10], and leads to
a 4096-D dimensional image representation, which
is already rather compact compared to IFV . We
further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The
networks were learnt on ILSVRC-2012. To speed-
up training, all layers aside from full7/full8 were
set to those of the CNN-M net and a lower initial
learning rate of 103was used. The initial learning
rate of full7/full8 was set to 102.
3.3 Data augmentation details
We explore three data augmentation strategies. The
ﬁrst strategy is to use no augmentation . In con-
trast to IFV , however, CNNs require images to
6
be transformed to a ﬁxed size ( 224224) even
when no augmentation is used. Hence the image is
downsized so that the smallest dimension is equal
to224 pixels and a 224224 crop is extracted
from the centre.2The second strategy is to use
ﬂip augmentation , mirroring images about the y-
axis producing two samples from each image. The
third strategy, termed C+F augmentation , combines
cropping and ﬂipping. For CNN-based representa-
tions, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224
crops are extracted from the four corners and the
centre of the image. Note that the crops are sampled
from the whole image, rather than its 256256
centre, as done by [13]. These crops are then ﬂipped
about they-axis, producing 10perturbed samples
per input image. In the case of the IFV encoding, the
same crops are extracted, but at the original image
resolution.
4 A NALYSIS
This section describes the experimental results,
comparing different features and data augmenta-
tion schemes. The results are given in Table 2 for
VOC-2007 and analysed next, starting from gener-
ally applicable methods such as augmentation and
then discussing the speciﬁcs of each scenario. We
then move onto other datasets and the state of the
art in Sect. 4.7.
4.1 Data augmentation
We experiment with no data augmentation (de-
noted Image Aug=– in Tab. 2), ﬂip augmenta-
tion ( Image Aug=F ), and C+F augmentation ( Image
Aug=C ). Augmented images are used as stand-
alone samples ( f), or by fusing the corresponding
descriptors using sum ( s) or max ( m) pooling or
stacking ( t). So for example Image Aug=(C) f s in
row [f] of Tab. 2 means that C+F augmentation is
used to generate additional samples in training ( f),
and is combined with sum-pooling in testing ( s).
Augmentation consistently improves perfor-
mance by3%for both IFV ( e.g. [d] vs. [f]) and
CNN ( e.g. [o] vs. [p]). Using additional samples for
training and sum-pooling for testing works best
([p]) followed by sum-pooling [r], max pooling [q],
and stacking [s]. In terms of the choice of transfor-
mations, ﬂipping improves only marginally ([o] vs.
2. Extracting a 224224 centre crop from a 256256
image [13] resulted in worse performance.[u]), but using the more expensive C+F sampling
improves, as seen, by about 23%([o]vs. [p]). We
experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour
Colour information can be added and subtracted in
CNN and IFV . In IFV replacing SIFT with the colour
descriptors of [3] (denoted COL inMethod ) yields
signiﬁcantly worse performance ([j] vs. [h]). How-
ever, when SIFT and colour descriptors are com-
bined by stacking the corresponding IFVs ( COL+ )
there is a small but signiﬁcant improvement of
around1%in the non-augmented case ( e.g. [h]
vs. [k]) but little impact in the augmented case
(e.g. [i] vs. [l]). For CNNs, retraining the network
after converting all the input images to grayscale
(denoted GSinMethods ) has a more signiﬁcant
impact, resulting in a performance drop of 3%
([w] vs. [p], [v] vs. [o]).
4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken
from Chatﬁeld et al. [12], primarily due to a larger
number of spatial scales being used during SIFT
feature extraction, and the resultant SIFT features
being square-rooted. Intra-normalisation , denoted as
INin the Method column of the table, improves the
performance by 1%(e.g. [c] vs. [d]). More inter-
estingly, switching from spatial pooling (denoted
spm in the SPool column) to feature spatial aug-
mentation ( SPool=(x,y) ) has either little effect on the
performance or results in a marginal increase ([a] vs.
[c], [b] vs. [d]), whilst resulting in a representation
which is over 10 smaller. We also experimented
with augmenting with scale in addition to position
as in [23] but observed no improvement. Finally,
we investigate pushing the parameters of the repre-
sentation setting K= 512 (rows [h]-[l]). Increasing
the number of GMM centres in the model from
K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of
higher-dimensional codes (125K dimensional).
4.4 Scenario 2: Deep representation (CNN) with
pre-training
CNN-based methods consistently outperform the
shallow encodings, even after the improvements
7
Method SPool Image Aug. Dim mAP
(I) FK BL spm – 327K 61.69 79.0 67.4 51.9 70.9 30.8 72.2
(II) DECAF – (C) t t 327K 73.41 87.4 79.3 84.1 78.4 42.3 73.7
(a) FK spm – 327K 63.66 83.4 68.8 59.6 74.1 35.7 71.2
(b) FK IN spm – 327K 64.18 82.1 69.7 59.7 75.2 35.7 71.3
(c) FK (x,y) – 42K 63.51 83.2 69.4 60.6 73.9 36.3 68.6
(d) FK IN (x,y) – 42K 64.36 83.1 70.4 62.4 75.2 37.1 69.1
(e) FK IN (x,y) (F) f - 42K 64.35 83.1 70.5 62.3 75.4 37.1 69.1
(f) FK IN (x,y) (C) f s 42K 67.17 85.5 71.6 64.6 77.2 39.0 70.8
(g) FK IN (x,y) (C) s s 42K 66.68 84.9 70.1 64.7 76.3 39.2 69.8
(h) FK IN 512 (x,y) – 84K 65.36 84.1 70.4 65.0 76.7 37.2 71.3
(i) FK IN 512 (x,y) (C) f s 84K 68.02 85.9 71.8 67.1 77.1 38.8 72.3
(j) FK IN COL 512 – – 82K 52.18 69.5 52.1 47.5 64.0 24.6 49.8
(k) FK IN 512 COL+ (x,y) – 166K 66.37 82.9 70.1 67.0 77.0 36.1 70.0
(l) FK IN 512 COL+ (x,y) (C) f s 166K 67.93 85.1 70.5 67.5 77.4 35.7 71.2
(m) CNN F – (C) f s 4K 77.38 88.7 83.9 87.0 84.7 46.9 77.5
(n) CNN S – (C) f s 4K 79.74 90.7 85.7 88.9 86.6 50.5 80.1
(o) CNN M – – 4K 76.97 89.5 84.3 88.8 83.2 48.4 77.0
(p) CNN M – (C) f s 4K 79.89 91.7 85.4 89.5 86.6 51.6 79.3
(q) CNN M – (C) f m 4K 79.50 90.9 84.6 89.4 85.8 50.3 78.4
(r) CNN M – (C) s s 4K 79.44 91.4 85.2 89.1 86.1 52.1 78.0
(s) CNN M – (C) t t 41K 78.77 90.7 85.0 89.2 85.8 51.0 77.8
(t) CNN M – (C) f - 4K 77.78 90.5 84.3 88.8 84.5 47.9 78.0
(u) CNN M – (F) f - 4K 76.99 90.1 84.2 89.0 83.5 48.1 77.2
(v) CNN M GS – – 4K 73.59 87.4 80.8 82.4 82.1 44.5 73.5
(w) CNN M GS – (C) f s 4K 77.00 89.4 83.8 85.1 84.4 49.4 77.6
(x) CNN M 2048 – (C) f s 2K 80.10 91.3 85.8 89.9 86.7 52.4 79.7
(y) CNN M 1024 – (C) f s 1K 79.91 91.4 86.9 89.3 85.8 53.3 79.8
(z) CNN M 128 – (C) f s 128 78.60 91.3 83.9 89.2 86.9 52.1 81.0
(a) FK+CNN F (x,y) (C) f s 88K 77.95 89.6 83.1 87.1 84.5 48.0 79.4
(b) FK+CNN M 2048 (x,y) (C) f s 86K 80.14 90.9 85.9 88.8 85.5 52.3 81.4
(g) CNN S TUNE-RNK – (C) f s 4K 82.42 95.3 90.4 92.5 89.6 54.4 81.9
TABLE 2
VOC 2007 results (continued overleaf) . See Sect. 4 for details.
discussed above, by a large 10% mAP margin
([i]vs. [p]). Our small architecture CNN-F, which is
similar to DeCAF [7], performs signiﬁcantly better
than the latter ([II] vs. [s]), validating our imple-
mentation. Both medium CNN-M [m] and slow
CNN-S [p] outperform the fast CNN-F [m] by a
signiﬁcant 23% margin. Since the accuracy
of CNN-S and CNN-M is nearly the same, we
focus on the latter as it is simpler and marginally
(25%) faster. Remarkably, these good networks
work very well even with no augmentation [o].
Another advantage of CNNs compared to IFV is
the small dimensionality of the output features,although IFV can be compressed to an extent. We
explored retraining the CNNs such that the ﬁnal
layer was of a lower dimensionality, and reducing
from 4096 to 2048 actually resulted in a marginal
performance boost ([x] vs. [p]). What is surprising
is that we can reduce the output dimensionality
further to 1024D [y] and even 128D [z] with only
a drop of2%for codes that are 32smaller
(650smaller than our best performing IFV [i]).
Note,`2-normalising the features accounted for up
to5%of their performance over VOC 2007; it
should be applied before input to the SVM and
after pooling the augmented descriptors (where
8
(I) 79.9 61.4 56.0 49.6 58.4 44.8 78.8 70.8 85.0 31.7 51.0 56.4 80.2 57.5
(II) 83.7 83.7 54.3 61.9 70.2 79.5 85.3 77.2 90.9 51.1 73.8 57.0 86.4 68.0
(a) 80.7 64.4 53.8 53.8 60.2 47.8 79.9 68.9 86.1 37.3 51.1 55.8 83.7 56.9
(b) 80.6 64.8 53.9 54.9 60.7 50.5 80.4 69.5 86.2 38.3 54.4 56.3 82.7 56.7
(c) 81.1 64.2 51.1 53.4 61.9 50.0 80.0 67.5 85.3 35.7 51.9 53.8 83.5 58.9
(d) 80.5 66.9 50.9 53.9 62.1 51.5 80.5 68.5 85.9 37.2 55.2 54.3 83.3 59.2
(e) 80.5 66.8 51.0 54.1 62.2 51.5 80.4 68.2 86.0 37.3 55.1 54.2 83.3 59.2
(f) 82.4 71.6 52.8 62.4 63.4 57.1 81.6 70.9 86.9 41.2 61.2 56.9 85.2 61.5
(g) 81.9 71.0 52.8 61.6 62.2 56.8 81.8 70.0 86.5 41.5 61.0 56.5 84.3 60.9
(h) 81.1 67.9 52.6 55.4 61.4 51.2 80.5 69.1 86.4 41.2 56.0 56.2 83.7 59.9
(i) 82.5 73.2 54.7 62.7 64.5 56.6 82.2 71.3 87.5 43.0 62.0 59.3 85.7 62.4
(j) 66.1 46.6 42.5 35.8 41.1 45.5 75.4 58.3 83.9 39.8 47.3 35.6 69.2 49.0
(k) 80.0 65.9 52.8 56.1 61.0 56.9 81.4 69.6 88.4 49.0 59.2 56.4 84.7 62.8
(l) 81.6 70.8 52.9 59.6 63.1 59.9 82.1 70.5 88.9 50.6 63.7 57.5 86.1 64.1
(m) 86.3 85.4 58.6 71.0 72.6 82.0 87.9 80.7 91.8 58.5 77.4 66.3 89.1 71.3
(n) 87.8 88.3 61.3 74.8 74.7 87.2 89.0 83.7 92.3 58.8 80.5 69.4 90.5 74.0
(o) 85.1 87.4 58.1 70.4 73.1 83.5 85.5 80.9 90.8 54.1 78.9 61.1 89.0 70.4
(p) 87.7 88.6 60.3 80.1 74.4 85.9 88.2 84.6 92.1 60.3 80.5 66.2 91.3 73.5
(q) 87.6 88.6 60.7 78.2 73.6 86.0 87.4 83.8 92.3 59.3 81.0 66.8 91.3 74.0
(r) 87.5 88.1 60.4 76.9 74.8 85.8 88.1 84.3 92.2 59.5 79.3 65.8 90.8 73.5
(s) 87.3 87.6 60.1 72.3 75.3 85.2 86.9 82.6 91.9 58.5 77.9 66.5 90.5 73.4
(t) 85.7 87.9 58.3 74.2 73.9 84.7 86.6 82.0 91.0 55.8 79.2 62.1 89.3 71.0
(u) 85.3 87.3 58.1 70.0 73.4 83.5 86.0 80.8 90.9 53.9 78.1 61.2 88.8 70.6
(v) 85.0 84.9 57.8 65.9 69.8 79.5 82.9 77.4 89.2 42.8 71.7 60.2 86.3 67.8
(w) 87.2 86.5 59.5 72.4 74.1 81.7 86.0 82.3 90.8 48.9 73.7 66.8 89.6 71.0
(x) 87.6 88.4 60.2 76.9 75.4 85.5 88.0 83.4 92.1 61.1 83.1 68.5 91.9 74.2
(y) 87.8 88.6 59.0 77.2 73.1 85.9 88.3 83.5 91.8 59.9 81.4 68.3 93.0 74.1
(z) 86.6 87.5 59.1 70.0 72.9 84.6 86.7 83.6 89.4 57.0 81.5 64.8 90.4 73.4
(a) 86.8 85.6 59.9 72.0 73.4 81.4 88.6 80.5 92.1 60.6 77.3 66.4 89.3 73.3
(b) 87.7 88.4 61.2 76.9 76.6 84.9 89.1 82.9 92.4 61.9 80.9 68.7 91.5 75.1
(g) 91.5 91.9 64.1 76.3 74.9 89.7 92.2 86.9 95.2 60.7 82.9 68.0 95.5 74.4
TABLE 2
VOC 2007 results (continued from previous page)
applicable).
4.5 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
We ﬁne-tuned our CNN-S architecture on VOC-
2007 using the ranking hinge loss, and achieved
a signiﬁcant improvement: 2:7%([g]vs. [n]). This
demonstrates that in spite of the small amount
of VOC training data (5,011 images), ﬁne-tuning
is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking
deep and shallow representations to form a higher-
dimensional descriptor makes little difference ([x]
vs. [b]). For the weaker CNN-F it results in a small
boost of0:8%([m] vs. [a]).
4.7 Comparison with the state of the art
In Table 3 we report our results on ILSVRC-2012,
VOC-2007, VOC-2012, Caltech-101, and Caltech-256
datasets, and compare them to the state of the art.
First, we note that the ILSVRC error rates of our
CNN-F, CNN-M, and CNN-S networks are better
9
ILSVRC-2012 VOC-2007 VOC-2012 Caltech-101 Caltech-256
(top-5 error) (mAP) (mAP) (accuracy) (accuracy)
(a) FK IN 512 - 68.0 – – –
(b) CNN F 16.7 77.4 79.9 – –
(c) CNN M 13.7 79.9 82.5 87.15 0.80 77.030.46
(d) CNN M 2048 13.5 80.1 82.4 86.64 0.53 76.880.35
(e) CNN S 13.1 79.7 82.9 87.76 0.66 77.610.12
(f) CNN S TUNE-CLS 13.1 - 83.0 88.350.56 77.330.56
(g) CNN S TUNE-RNK 13.1 82.4 83.2 – –
(h) Zeiler & Fergus [19] 16.1 - 79.0 86.5 0.5 74.20.3
(i) Razavian et al. [9], [10] 14.7 77.2 – – –
(j) Oquab et al. [8] 18 77.7 78.7 (82.8*) – –
(k) Oquab et al. [16] - - 86.3*– –
(l) Wei et al. [17] - 81.5 ( 85.2*) 81.7 ( 90.3*) – –
(m) He et al. [29] 13.6 80.1 - 91.40.7 –
TABLE 3
Comparison with the state of the art on ILSVRC2012, VOC2007, VOC2012, Caltech-101, and
Caltech-256. Results marked with * were achieved using models pre-trained on the extended ILSVRC
datasets (1512 classes in [8], [16], 2000 classes in [17]). All other results were achieved using CNNs
pre-trained on ILSVRC-2012 (1000 classes).
than those reported by [13], [19], and [10] for the
related conﬁgurations. This validates our imple-
mentation, and the difference is likely to be due to
the sampling of image crops from the uncropped
image plane (instead of the centre). When using
our CNN features on other datasets, the relative
performance generally follows the same pattern
as on ILSVRC, where the nets are trained – the
CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC
datasets turns out to be beneﬁcial; on VOC-2012,
using the ranking loss is marginally better than
the classiﬁcation loss ([g] vs. [f]), which can be
explained by the ranking-based VOC evaluation
criterion. Fine-tuning on Caltech-101 also yields a
small improvement, but no gain is observed over
Caltech-256.
Our CNN-S net is competitive with recent CNN-
based approaches [19], [9], [8], [16], [17], [29] and
on a number of datasets (VOC-2007, VOC-2012,
Caltech-101, Caltech-256) and sets the state of the
art on VOC-2007 and VOC-2012 across methods
pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better
performance on VOC (86.3% and 90.3% respec-
tively), they were trained using extended ILSVRCdatasets, enriched with additional categories se-
mantically close to the ones in VOC. Addition-
ally, [17] used a signiﬁcantly more complex clas-
siﬁcation pipeline, driven by bounding box pro-
posals [30], pre-trained on ILSVRC-2013 detection
dataset. Their best reported result on VOC-2012
(90.3%) was achieved by the late fusion with a
complex hand-crafted method of [31]; without fu-
sion, they get 84.2%. On Caltech-101, [29] achieves
the state of the art using spatial pyramid pooling
of conv5 layer features, while we used full7 layer
features consistently across all datasets (for full7
features, they report 87:08%).
In addition to achieving performance comparable
to the state of the art with a very simple approach
(but powerful CNN-based features), with the mod-
iﬁcations outlined in the paper (primarily the use
of data augmentation similar to the CNN-based
methods) we are able to improve the performance
of shallow IFV to 68.02% (Table 2, [i]).
4.8 Performance Evolution on VOC-2007
A comparative plot of the evolution in the per-
formance of the methods evaluated in this paper,
along with a selection from our earlier review of
shallow methods [12] is presented in Fig. 1. Clas-
siﬁcation accuracy over PASCAL VOC was 54.48%
mAP for the BoVW model in 2008, 61.7% for the
10
      BOW
32K
–FK-BL
327K
–
[I]FK
327K
–
[a]FK-IN
84K
f s
[i]DeCAF
4K
t t
[II]CNN-F
4K
f s
[m]CNN-M 2K
2K
f s
[y]CNN-S
4K (TN)
f s
[γ]   
                           
545658606264666870727476788082mAP 68.02
54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.
Fig. 1. Evolution of Performance on PASCAL VOC-2007 over the recent years. Please refer to Table 2
for details and references.
IFV in 2010 [12], and 73.41% for DeCAF [7] and
similar [8], [9] CNN-based methods introduced in
late 2013. Our best performing CNN-based method
(CNN-S with ﬁne-tuning) achieves 82.42%, compa-
rable to the most recent state-of-the-art.
4.9 Timings and dimensionality
One of our best-performing CNN representations
CNN-M-2048 [x] is 42more compact than the
best performing IFV [i] (84K vs. 2K) and CNN-M
features are also 50faster to compute ( 120s
vs.2:4sper image with augmentation enabled,
over a single CPU core). Non-augmented CNN-M
features [o] take around 0:3sper image, compared
to0:4sfor CNN-S features and 0:13sfor CNN-
F features.
5 C ONCLUSION
In this paper we presented a rigorous empiri-
cal evaluation of CNN-based methods for image
classiﬁcation, along with a comparison with more
traditional shallow feature encoding methods. We
have demonstrated that the performance of shal-
low representations can be signiﬁcantly improved
by adopting data augmentation, typically used indeep learning. In spite of this improvement, deep
architectures still outperform the shallow methods
by a large margin. We have shown that the per-
formance of deep representations on the ILSVRC
dataset is a good indicator of their performance
on other datasets, and that ﬁne-tuning can further
improve on already very strong results achieved
using the combination of deep representations and
a linear SVM. Source code and CNN models to
reproduce the experiments presented in the paper
are available on the project website [32] in the
hope that it would provide common ground for
future comparisons, and good baselines for image
representation research.
ACKNOWLEDGEMENTS
This work was supported by the EPSRC and ERC
grant VisRec no. 228180. We gratefully acknowl-
edge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES
[1] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual catego-
rization with bags of keypoints,” in Workshop on Statistical
Learning in Computer Vision, ECCV , 2004, pp. 1–22.
11
[2] J. Sivic and A. Zisserman, “Video Google: A text retrieval
approach to object matching in videos,” in Proc. ICCV ,
vol. 2, 2003, pp. 1470–1477.
[3] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the
Fisher kernel for large-scale image classiﬁcation,” in Proc.
ECCV , 2010.
[4] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel, “Backpropaga-
tion applied to handwritten zip code recognition,” Neural
Computation , vol. 1, no. 4, pp. 541–551, 1989.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“Imagenet: A large-scale hierarchical image database,” in
Proc. CVPR , 2009.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
(VOC) challenge,” IJCV , vol. 88, no. 2, pp. 303–338, 2010.
[7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell, “Decaf: A deep convolutional
activation feature for generic visual recognition,” CoRR ,
vol. abs/1310.1531, 2013.
[8] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning
and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.
[9] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson,
“CNN Features off-the-shelf: an Astounding Baseline for
Recognition,” CoRR , vol. abs/1403.6382, 2014.
[10] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “OverFeat: Integrated Recognition, Local-
ization and Detection using Convolutional Networks,” in
Proc. ICLR , 2014.
[11] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich
feature hierarchies for accurate object detection and se-
mantic segmentation,” in Proc. CVPR , 2014.
[12] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman,
“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
classiﬁcation with deep convolutional neural networks,”
inNIPS , 2012, pp. 1106–1114.
[14] M. Paulin, J. Revaud, Z. Harchaoui, F. Perronnin, and
C. Schmid, “Transformation Pursuit for Image Classiﬁca-
tion,” in Proc. CVPR , 2014.
[15] F. Perronnin, Z. Akata, Z. Harchaoui, and C. Schmid,
“Towards good practice in large-scale learning for image
classiﬁcation,” in Proc. CVPR , 2012, pp. 3482–3489.
[16] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Weakly
supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.
[17] Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and
S. Yan, “CNN: Single-label to multi-label,” CoRR , vol.
abs/1406.5726, 2014.
[18] H. J ´egou, F. Perronnin, M. Douze, J. S ´anchez, P . P ´erez, and
C. Schmid, “Aggregating local images descriptors into
compact codes,” IEEE P AMI , 2012.
[19] M. D. Zeiler and R. Fergus, “Visualizing and understand-
ing convolutional networks,” CoRR , vol. abs/1311.2901,
2013.
[20] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.
org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremen-
tal bayesian approach tested on 101 object categories,”
inIEEE CVPR Workshop of Generative Model Based Vision ,
2004.[22] G. Grifﬁn, A. Holub, and P . Perona, “Caltech-256 object
category dataset,” California Institute of Technology,
Tech. Rep. 7694, 2007. [Online]. Available: http://
authors.library.caltech.edu/7694
[23] J. S ´anchez, F. Perronnin, and T. Em ´ıdio de Campos,
“Modeling the spatial layout of images beyond spatial
pyramids,” Pattern Recognition Letters , vol. 33, no. 16, pp.
2216–2223, 2012.
[24] R. Arandjelovi ´c and A. Zisserman, “Three things every-
one should know to improve object retrieval,” in Proc.
CVPR , 2012.
[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.
[26] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags
of Features: Spatial Pyramid Matching for Recognizing
Natural Scene Categories,” in Proc. CVPR , 2006.
[27] R. Arandjelovi ´c and A. Zisserman, “All about VLAD,” in
Proc. CVPR , 2013.
[28] H. J ´egou, M. Douze, and C. Schmid, “On the burstiness
of visual elements,” in Proc. CVPR , Jun 2009.
[29] K. He, A. Zhang, S. Ren, and J. Sun, “Spatial pyramid
pooling in deep convolutional networks for visual recog-
nition,” in Proc. ECCV , 2014.
[30] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr,
“BING: Binarized normed gradients for objectness esti-
mation at 300fps,” in Proc. CVPR , 2014.
[31] S. Yan, J. Dong, Q. Chen, Z. Song, Y. Pan, W. Xia,
H. Zhongyang, Y. Hua, and S. Shen, “Generalized hi-
erarchical matching for subcategory aware object classi-
ﬁcation,” in The P ASCAL Visual Object Classes Challenge
Workshop , 2012.
[32] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman,
“Return of the devil in the details: delving deep into
convolutional nets webpage,” 2014. [Online]. Available:
http://www.robots.ox.ac.uk/vgg/research/deep eval