Deep Hashing with Category Mask for Fast Video Retrieval
Dajun Ding1
Yajiao Dong1,2
1
MTdata, Meitu
2
School of Information and Electronics, Beijing Institution of Technology, China

arXiv:1712.08315v1 [] 22 Dec 2017

Xu Liu1

Lili Zhao1

Abstract
This paper proposes an end-to-end deep hashing framework with category mask for fast video retrieval. We train
our network in a supervised way by fully exploiting interclass diversity and intra-class identity. Classification loss is
optimized to maximize inter-class diversity, while intra-pair
is introduced to learn representative intra-class identity. We
investigate the binary bits distribution related to categories
and find out that the effectiveness of binary bits is highly
related to categories, and certain bits may degrade classification performance of some categories. We then design
hash code generation scheme with category mask to filter
out bits with negative contribution. Experimental results
demonstrate the proposed method outperforms state-of-thearts under various evaluation metrics on public datasets.
We are making our code and models public online.

1. Introduction

Figure 1. t-SNE [29] visualization of 128-bit binary hash code generated by the proposed method on UCF101 [34]. Due to space limitation, clustering results of only 50 categories are demonstrated
here. Videos belonging to the same category are marked with the
same color. As is illustrated, the hash code produced by the proposed method successfully preserves semantic information as well
as the class distinguishability.

Over recent years, industry has been witnessing the
booming of short-video sharing apps and platforms, through
which people record and share their daily moments in the
form of short videos of seconds-length. This has encouraged the development of advanced techniques for a wide
range of multimedia understanding applications. One open
question is how to efficiently retrieve the relevant video
from the large-scale video database, which requires efficient video representation learning. In industrial applications, efficiently learned video representations should satisfy three qualities: content representative, storage effective, and computation low-complexity. One approach that
qualifies these requirements is learning based video hashing [40, 25].
Existing learning to hash methods can be classified into
two approaches: non-deep hash learning [28, 9, 38, 15, 44,
27] and deep hash learning [41, 19, 7, 21, 26, 22, 37, 5,
6, 14]. The first approach uses various statistical learning
techniques to learn hash functions which map samples into
binary codes. In the past few years, deep convolutional neural networks(CNN) [17, 32, 12] have demonstrated state-

of-the-art performance on various visual tasks [8, 24, 31].
Inspired from the advancement of deep CNN techniques,
many deep hashing methods have been proposed [41, 19,
7, 21, 26, 22, 37, 5, 6, 14]. By training an end-to-end CNN
model, existing deep hashing techniques manage to simultaneously learn image representations as well as binary codes
either in supervised [41, 19, 7, 21, 26, 6, 14] or in unsupervised [22, 37, 5] way.
Although existing deep hashing approaches have
achieved remarkable performance, they were mostly designed for image based binary code learning. By contrast, there are relatively fewer deep hashing methods specially designed for videos in literature. Learning to hash
for videos is much more challenging than that for images
as videos provide far more diverse and complex visual information than images provide [13]. Existing video hashing
approaches mostly focus on first extracting statistical or perceptual features from videos and then applying image hashing methods on those features to obtain the binary codes
[4, 39, 2, 42, 33, 35], which is unidirectional. As a result,
1

Figure 2. Overview of the proposed deep hashing framework. At training phase, model is trained with batches of intra-pair data by
optimizing cross-entropy loss for classification and l2 loss within an intra-pair. At retrieval phase, shown in the bottom part, Hamming
distance between the query video and the target is calculated. First, the query video is forwarded through the network and binary code
bq is produced by thresholding on the outputs of the binary encoding module. Second, bq is XOR operated with hash code bd of target
video, followed by category masking on the outputs of XOR operation to filter out bits that degrade classification accuracy, producing a
binary code for counting Hamming distance. The category mask is a K × L matrix, where K is the number of categories and L denotes
the length of binary vector b.

the quality of produced hash code heavily depends on the
quality of obtained features, while the hash code was not
utilized to guide the learning of features. To address this
problem, Wu et al. [40] integrated video feature learning
and hash value learning into a joint learning model, where
offline processing such as K-means clustering and Canonical Correlation Analysis (CCA) need to be employed on
the learned video features to learn binary codes and hash
functions. Liong et al. [25] simplified the learning pipeline
by integrating the learnings of video feature, binary code
and hash function into a single deep neural network. Model
parameters are learned by employing siamese network. In
this work we propose an end-to-end deep video hashing network by learning intra-class identity and maximizing inter
class diversity. Furthermore, inspired from [20, 30] which
demonstrated that convolutional neural networks involve
massive redundant parameters, we study binary bits distribution related to categories and present a category mask related binary code generation approach. The contributions
of the present work can be summarized as follows:
We present an end-to-end deep video hashing framework which simultaneously learns feature representation
and hash code.
We propose to employ inter-class diversity and intraclass identity as training objectives to learn discriminative
yet representative binary descriptor. Video intra-pair is introduced by this work to learn intra-class identity.
We investigate the video category influence on hash code
bits distribution and propose a category mask based hash
code generation for efficient video retrieval. To our best
knowledge, none of existing works ever investigated the influence of category on hash code bits distribution for effi-

cient video retrieval.

2. Related Work
Learning to Hash. Typical works on statistical hash
learning [28, 9, 38, 15, 44, 27] include supervised hashing
with kernels (KSH)[27], PCA-random rotation (PCA-RR)
[9], spectral hashing (SH) [38], iterative quantization (ITQ)
[9], scalable graph hashing (SGH) [15], sparse embedding
and least variance encoding (SELVE)[44]. All these hashing methods take a vector of hand-crafted visual features
extracted from an image as input. In the last few years,
many deep hashing methods have been proposed to simultaneously learn image representations and binary codes in either supervised [41, 19, 7, 21, 26, 6, 14]or unsupervised way
[22, 37, 5]. Xia et al. [41] presented a two-stage supervised
hashing method via image representation learning, where
the learned approximate hash codes are used to guide the
learning of the image representation, but the learned image
representation cannot give feedback for learning better approximate hash codes. To address this issue, simultaneous
feature learning and hashing techniques in a single neural
network were proposed [19, 23]. Jain et al. [14] presented
a structured hash code learning framework by introducing
block-softmax nonlinearity. Venkateswara et al. [37] proposed a supervised deep hashing framework that tries to
address the domain adaption problem. Regarding training
objectives, Long et al. in [7] incorporated pair-wise supervision to train the deep hashing model. Similar pair-wise
based works can be found in [21, 26]. Triplet ranking was
employed to learn parameters in [19] which achieved improved performance. Lin et al. [22] learned deep hashing
function in an unsupervised way with three training objec-

tives: image rotation invariant, quantization loss minimization and learned bits evenly distributed. Duan et al. [5]
also proposed an unsupervised binary descriptor learning
framework, where K-Auto Encoders were used to minimize
multi-quantization loss.
Video Hashing. Convenient approaches for video hashing usually select frames from a video, treat the selected
frames as images and then employ image hashing techniques on them [4, 39, 2, 42, 33, 11]. For example, Li et al.
[39] proposed to extract feature from each frame and then
generate hash code based on the extracted statistical feature
vector, while Cao et al. [2] and Hao et al. [11] utilized multiple frame sets and multiple key frames to learn hash functions. Sun et al. in [35] proposed a hash learning method via
deep belief network, where a fusion of visual-appearance
and visual-attention features are used as inputs. All the
above methods employed hand-crafted features which were
fixed during hash learning process. In [10], CNN features
were extracted to learn hash function. Zhang et al. [43] presented an unsupervised video hash learning framework, by
using a binary LSTM module and a normal LSTM module
as the encoder and the decoder respectively. Frame-level
features are extracted via a deep CNN. Still, the feature
generation and the hash code generation are processed separately. Wu et al. [40] proposed an integrated framework
in which feature extraction, binary code learning and hash
function learning are optimized in a self-taught manner. Yet
this method doesn’t learn binary code and hash function as
part of the deep architecture.
We propose to learn both video feature representation
and hash functions within a deep learning pipeline. As far as
we know, only one other approach has provided an end-toend deep hash learning pipeline [25]. Our approach differs
from that work both in neural network structure and in supervision learning metric. We also investigate the influence
of category on hash code bits distribution and devise a category mask based hash code generation method for efficient
video retrieval.

3. The Proposed Approach
In this section, we present the design of our deep video
hashing architecture, its supervised learning and the category mask for fast retrieval.

3.1. Architecture
The proposed deep video hashing architecture is shown
in Fig 2, which consists of the input module, the backbone
network, the binary encoding module and the output module.
The input module selects frames from video. We introduce intra-pairs for training purpose. An intra-pair I ∗ ,
{Ia , Ib }, is defined as a pair of frame sets extracted from
the same video, with each frame set Ia and Ib consisting of

a group of video frames randomly selected at even interval,
Ia , (xa1 , xa2 , · · ·, xaN ) and Ib , (xb1 , xb2 , · · ·, xbN ).
There are no overlap frames between two frame sets within
an intra-pair, i.e. Ia ∩ Ib ≡ ∅. At training phase, the input is an intra-pair I ∗ , while at retrieval phase it is a single
frameset Iq , (xq1 , xq2 , · · ·, xqN ).
The backbone network is a deep CNN involving multiple convolutional layers followed by a fully-connected
layer. The deep CNN is used to learn video representation:
N frames (x1 , x2 , · · ·, xN ) selected evenly from the input
video are forwarded to the backbone CNN module which
extracts feature maps for each selected frame and fuse them
to generate a single feature map set. The fused feature
maps are connected to a full connection layer to generate
the video representation. To better capture temporal evolution across consecutive frames, we fuse the feature maps in
a weighted way, as illustrated in Equation 1.
Fe , [F (x1 ), F (x2 ), · · ·, F (xN )] · [ω1 , ω2 , · · ·, ωN ]T

(1)

where F̃ represents the fusion output from feature maps of
input frames, and F (xi ) is a combination of feature maps
in a layer for frame xi with F (xi ) , [f1 (xi ), f2 (xi ), · ·
·, fC (xi )], and fj (xi ) is the feature map output for the j th
channel, j ∈ [1, C]. [ω1 , ω2 , · · ·, ωN ] is the fusion weights
that are learned during training process.
The backbone network is loosely coupled with the other
modules in the proposed architecture, thus can be replaced
with any efficient CNN modules, such as Alexnet [17],
VGG [32] and ResNet [12], or other LSTM modules employed to extract video representations.
The binary encoding module consists of a fullyconnected layer which encodes the video representation
vector into binary-like outputs by employing sigmoid operations.
The output module outputs both the class-probability
estimates and the binary-like vector which is then thresholded to produce the binary hash code.

3.2. Supervised Learning with Intra-Pair Loss and
Classification Loss
The proposed deep hashing model is learned in a supervised way. We aim to learn a set of parameters W that quantize the input video into a compact binary vector while preserving high level semantic information. We enforce two
criterions on a compact yet discriminative binary descriptor. First, the learned binary descriptor should maximize
inter-class diversity. Second, the learned binary descriptor
should be highly representative of intra-class identity. To
achieve the above two objectives, we formulate the following optimization problem to learn W using the proposed
deep hashing network:
minL (W) = α · L (W) + β · L (W)
W

inter

intra

(2)

where L (W) defines the loss to maximize inter-class diver-

3.3.1

Observation

inter

sity, L (W) is minimized to learn intra-class identity, and α
intra

and β are parameters to balance different objectives.
Inter-class diversity. As classification information describes high-level semantic category of video data, it is a
simple yet effective method to represent inter-class diversity. As such, we employ cross-entropy loss to define L(W)
inter

in order to maximize inter-class diversity, as interpreted in
Equation 3.
L(W) , −
inter

X

yi log(b
yi )

(3)

i

Intra-class identity. Ideally, the intra-class identity is
the unique identifier for a specific class. In real scenarios,
training data usually owns two characteristics: 1) the intraclass sample distances vary for different classes. 2) large
amount of noisy data exists especially in complex datasets.
As a result, it involves much difficulty to train a representative class identity directly from video pairs within the same
class. Instead, highly identical sample pairs with minimized
noise yet with maximized low-level visual discrimination
are required to learn high-quality intra-class identity.
To address this issue, we introduce intra-pair I ∗ which
is defined as a pair of frame sets extracted from the same
video, as described in section 3.1. It is supposed that for any
frame set extracted over the whole lifespan from a video, a
perfectly learned hash function would output the same binary code, as frame sets extracted from the same video share
the same semantic information. Consequently, we learn
intra-class identity by minimizing the distance between the
two frame sets within an intra-pair. In practice, the intrapair loss is defined as the l2 loss between the binary-like
outputs of an intra-pair, as Equation 4 shows.
L(W) , max(||ip1 − ip2 ||22 − M, 0)
intra

For a dataset with K categories, we define a binary vector
for each category vk ∈ {0, 1}L of dimensionL, k = 1, 2, · ·
·, K. Given a ratio r, binary vector vk for the kth category
is set as in Equation 5.

1, if the absolute value of ith weight is bigger




than the (r ∗ L)th largest absolute weight
vk [i] =

in the softmax layer;



0, otherwise .
(5)
The sum of contributed categories is then represented as
a natural-number vector of dimension L, s ∈ NL
+ , calculated as:
K
X
s[i] =
vk [i]
(6)
k=1

where vk [i] and s[i] denotes the i-th entry of vector vk and
of vector s, respectively.
Obviously, for an evenly distributed hash code of length
L, the value of s[i], i = 1, 2, · · ·, L, should be r ∗ L, given
ratio r. We learn hash code @64 bits on UCF101 dataset
and plot the values of vector s under ratios 0.3, 0.5 and 0.7
as shown in Fig 3, where the solid curves represent the experimental results and the dashed lines represent the ideal
distribution. It can be observed that the deviation from the
ideal value for each bit is within a small range and on average the number of contributed categories for each bit equals
the ideal value, proving that every bit in the learned binary
code contributes evenly to the category classification.

(4)

where ip1 and ip2 are binary-like vector respectively
learned to represent Ia and Ib , and ||.||2 defines the L2 norm distance. M is the positive margin value as defined
in [26]. Without M , minimizing the loss function will be
restrained in a representation tending to be all 0s, seriously
affecting the system performance.

3.3. Category Mask
Previous works have shown that convolutional neural
networks involve massive redundant parameters [20, 30].
Inspired from that observation, we investigated the distribution of hash bits produced by thresholding on the outputs of
the binary encoding module in the proposed method.

Figure 3. Sum of contributed categories for each bit, where the xaxis denotes the bit order with value starting from 0 to 63, and the
y-axis is the total number of categories calculated using Equation
6.

To illustrate the influence of different bits on the classification output, we further visualize the mapping relationship
between each bit and each category, as shown in Fig 4. Due
to page limitation, the results of only 10 categories are presented here. We can see that every single bit behaves differently on different categories, and for each category only
specific bits contribute to its classification outputs.
From the above analysis, we can see that the learned hash
bits are evenly distributed for all categories while each bit

Figure 4. Visualization of mapping relationship between bits and categories under ratio 0.3. The horizontal axis is the bit order starting
from 0 to 63, and the vertical axis illustrates 10 categories from UCF101. Each block defines a mapping relationship, and a green block at
(i, k) means for category k, the value of bk [i] is set to 1 according to Equation 5, while a white block says bk [i] = 0.

contributes distinguishably on different categories. In other
words, every bit of hash code is essential for the classification task, but there is indeed redundancy existing locally
with respect to categories.
Consequently, we consider that the hash code generation should be category related and propose a category mask
based hash code generation method for fast video retrieval.
3.3.2

Figure 5. Precision curve w.r.t. top-N @64 bits under different
mask ratios on UCF101 and HMDB51.

Methodology

The category mask is defined as a binary matrix Mr of dimension K×L with respect to a ratio value r, where K is the
number of classes in the database and L is length of the hash
code. Each row vector of category mask Mr is assigned by
a binary vector for that class, as defined in Equation 7.
Mr [k] = vk , k = 0, 1, · · ·, K − 1

(7)

where vk is calculated as Equation 5.
Category mask is calculated after training is complete.
Let KK represent the hash code collection of retrieval
dataset with K classes, and bd ∈ KK denote a binary hash
code. At retrieval phase, the classification output k indicating the category of the query video is used to index the
binary mask, which is then employed on the output of XOR
operation between the query and the target.
d = (bq ⊕ bd ) ∧ Mr [k]

(a) UCF101

(8)

where bq denotes the query hash code and the output d is a
binary vector of length L. The Hamming distance
between
PL−1
the query and the target is then calculated with i=0 d[i].
As can be seen from Equation 7, mask Mr is actually a
filter on the hash code bit with r as the adjusting factor to
control the strength. When r is set to 1.0, the mask is filled
with all 1s and Equation 7 turns into a general Hamming
distance computation between the query and the target.
The overall hash code generation process with category
mask is illustrated in Fig 2. Experimental section demonstrates the efficiency of the proposed category masking
scheme.

4. Experimental Results
In this section, we verify the efficacy of the proposed
category mask based deep video hashing method named

DVHCM. The experimental settings are described in subsection 4.1, including the benchmark datasets used for evaluation and the network layer setup for the proposed method.
For the performance verification, we first demonstrate in
subsection 4.2 the efficacy of the proposed category mask
scheme by employing various masks generated under different ratios, and then in subsection 4.3 we provide extensive evaluations on the proposed DVHCM by comparing it
with state-of-the-arts.

4.1. Experimental settings
We evaluate our approach on two benchmark datasets for
action recognition: UCF101[34] and HMDB51[18].
• The UCF101 dataset consists of 101 action categories
from 13320 realistic action videos, covering five activity types: human-object interaction, body-motion
only, human-human interaction, playing musical instruments, and sports. The clip duration for most
videos in UCF101 is less than 10 seconds.
• The HMDB51 dataset contains 6766 videos with 51
distinct action categories, covering various facial actions and body movements.The clip duration for most
videos in HMDB51 is less than 5 seconds.
Training datasets for UCF101 and HMDB51 consist of
respectively 9624 and 5115 videos, and the rest videos comprise the test datasets. The retrieval is performed by using
the videos from the testing test as the queries for the system
to retrieve relevant ones from the training set. Semanticlevel labels define the similarity labels, i.e., the queried

Table 1. mAP performance by Hamming ranking for different category mask ratios under various binary code length.
Ratio
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0

512 bits
0.934
0.953
0.953
0.951
0.946
0.937
0.924
0.905
0.881
0.853

256 bits
0.909
0.938
0.956
0.959
0.950
0.946
0.936
0.921
0.906
0.886

UCF101
128 bits
0.773
0.925
0.932
0.949
0.942
0.949
0.945
0.935
0.921
0.901

64 bits
0.333
0.778
0.854
0.893
0.898
0.901
0.900
0.895
0.881
0.867

32 bits
0.063
0.372
0.624
0.784
0.825
0.827
0.857
0.855
0.849
0.838

512 bits
0.660
0.672
0.660
0.647
0.619
0.585
0.545
0.500
0.454
0.420

256 bits
0.512
0.573
0.588
0.582
0.558
0.540
0.507
0.480
0.447
0.422

HMDB51
128 bits
0.417
0.555
0.588
0.580
0.546
0.528
0.514
0.482
0.448
0.427

64 bits
0.240
0.457
0.514
0.584
0.600
0.605
0.586
0.563
0.507
0.457

32 bits
0.083
0.206
0.304
0.386
0.434
0.441
0.487
0.461
0.442
0.411

Table 2. mAP performance by Hamming ranking of the proposed methods and the baselines. Note that deep features are used for non-deep
hashing baselines.
Method
AGH [28]
ITQ [42]
KSH [36]
PCA-RR [9]
SELVE [27]
SGH [27]
SH [23]
DBH [26]
DNNH [19]
Proposed w/o CM
Proposed

512 bits
0.393
0.790
0.848
0.748
0.630
0.444
0.417
0.785
0.835
0.853
0.953

UCF101
256 bits
128 bits
0.492
0.600
0.778
0.754
0.810
0.786
0.720
0.696
0.605
0.685
0.389
0.159
0.486
0.439
0.766
0.736
0.817
0.789
0.886
0.901
0.959
0.949

64 bits
0.572
0.706
0.716
0.619
0.692
0.115
0.406
0.681
0.740
0.867
0.901

512 bits
0.127
0.282
0.473
0.278
0.182
0.145
0.165
0.346
0.480
0.420
0.672

HMDB51
256 bits
128 bits
0.130
0.198
0.281
0.301
0.450
0.464
0.261
0.270
0.183
0.278
0.115
0.126
0.107
0.164
0.386
0.391
0.493
0.503
0.422
0.427
0.588
0.588

64 bits
0.157
0.332
0.431
0.264
0.282
0.028
0.165
0.389
0.487
0.457
0.605

Table 3. mAP performance of the proposed method and SUBIC calculated on top returned samples.
Method
SUBIC [14]
Proposed w/o CM
Proposed

512 bits
0.528
0.730
0.870

UCF101
256 bits
128 bits
0.449
0.432
0.744
0.755
0.843
0.817

video is relevant to the query if they share the same semantic label.
Following previous hashing works, three standard evaluation metrics are used to measure the accuracy of our proposed method and other baselines: mean Average Precision
(mAP), Precision-Recall curves, and Precision curves w.r.t.
different numbers of top returned samples.
For the proposed method, we employ ResNet-50 [12] as
the backbone module in the proposed deep hashing architecture. Model is trained in supervised way by optimizing
the proposed intra-pair loss and category loss. At training phase, the input to the network is intra-pairs consisting of two frame sets, while at retrieval phase, the input
is a single frameset. Each frameset contains a number K
of randomly selected frames from a video. In the experiments, we set K to 5. Each frame in a frame-set is resized to 224 × 224 and then input to the ResNet-50 module
to generate a 2048-d image representation. As described
in section 3.1, image representations from the same frame
set which is 5 × 2048 are fused with weights to produce a

64 bits
0.324
0.720
0.759

512 bits
0.286
0.307
0.372

HMDB51
256 bits
128 bits
0.286
0.247
0.295
0.301
0.368
0.367

64 bits
0.192
0.304
0.356

single output of 1 × 2048, which is regarded as the video
representation. The weights for fusion is learned at training phase. The binary encoding module consists of a fully
connected layer followed by a sigmoid activation layer to
produce binary-like vectors. To evaluate the performance
of hash codes with various lengths, we set the length of
binary-like vectors to 32, 64, 128, 256 and 512, respectively.
Following the binary encoding module, the output module
employs a SoftMax layer to do classification.

4.2. Evaluation on category mask
To explore how well the proposed category mask
scheme improves retrieval performance, we generate category masks with various top weight ratios ranging from 0.1
to 1.0 with different binary code lengths.
Table 1 presents the mAP values of different mask ratios @32, 64, 128, 256 and 512 bits. Items in bold show
the best mAP values under corresponding code length in
each column. We can observe that in general longer binary
code shows better mAP performance. And as the binary

(a) Precision-Recall curve

(b) Precision-Recall curve

(c) Precision curve w.r.t. the top-N

Figure 6. Performance of DVHCM and baselines @64 bits on UCF101 dataset in terms of precision-recall curves and precision curves
w.r.t. top-N samples.

(a) Precision-Recall curve

(b) Precision-Recall curve

(c) Precision curve w.r.t. the top-N

Figure 7. Performance of DVHCM and baselines @64 bits on HMDB51 dataset in terms of precision-recall curves and precision curves
w.r.t. top-N samples.

code length becomes larger, the value of best mask ratio
increases. Hence at retrieval phase, we set larger ratio to
produce masks for longer binary code, and use smaller ratio for short binary code. For masks with very low ratios,
the retrieval accuracy degrades at short bits. For example,
curves of ratio 0.1 @64 bits degrade the precision-recall
performance. We consider this degradation is due to lacks
of enough bits to represent the video, which is the opposite
of bits redundancy.
Fig 5 demonstrates the precision curves w.r.t. top-N retrieved videos @64 bits under various mask ratios. Here
precision values of maximum 60 returned samples are presented as each category in UCF101 and HMDB51 training set consists of more than 60 videos on average. We
can find that retrieval performance with mask ratios larger
0.4 clearly outperforms the one without category masking
shown by the black curve in the figure.

4.3. Comparison with the state-of-the-arts
We compare the proposed DVHCM method with stateof-the-art baselines on video retrieval tasks, including seven
non-deep approaches: SH [23], ITQ [42], AGH [28], PCARR [9], SGH [27], SELVE [27] and KSH [36], and three
deep approaches: DBH [26], DNNH[19] and SUBIC[14].
All of the methods use identical training and query sets.
For the proposed method, we present results with the best
mask ratios as well as results without using category mask.

For dataset UCF101, ratios are set to 0.6, 0.4, 0.4 and 0.3
respectively at 64 bits, 128 bits, 256 bits and 512 bits, while
for dataset HMDB51, ratios are set to 0.5, 0.3, 0.3 and 0.2.
The network setting is described in the experimental setting
section.
For a fair comparison, the deep baselines use the same
backbone network as the proposed method, i.e. ResNet50 initialized with the model pre-trained on the ImageNet
dataset. The rest of layers for the baselines are set according to the reference paper. The triplets to train DNNH are
generated using the same method presented in the original
paper by replacing images with videos. For SUBIC [14],
the dimension of each one-hot vector is set to 8 . We implemented the proposed method and deep baselines on MXNet
[3], and as the baselines have been originally designed for
image retrieval, we first trained and tested hash models for
image retrieval on Cifar-10 dataset [16] to make sure our
implementation reproduce similar results with those presented in the reference paper.
For non-deep baselines, instead of using hand-crafted visual features as most of previous hash works did, we represent each video by a 2048-dimensional deep feature extracted from the DPH network to minimize the retrieval performance gap caused by feature representations. All the
non-deep baselines are evaluated using the implementions
from HABIR toolkit [1].
As the original SUBIC paper proposes to use floating

Figure 8. Top 10 retrieved videos from UCF101 dataset by DVHCM @128 bits

vector instead of hash code to do retrieval, we compare our
method with SUBIC using mAP calculated based on the top
returned samples. While with the rest of baselines, we do
comparison using mAP by Hamming ranking.
Table 2 and Table 3 compare the mAP performance of
the proposed method and the baselines at 64, 128, 256 and
512 bits. For the proposed method, results both with and
without using category masks are reported in the tables,
represented respectively as ’Proposed’ and ’Proposed w/o
CM’. We can find that on both datasets the proposed method
DVHCM dramtically outperforms the other baselines at every bit length.
To be specific, on dataset UCF101, the proposed
DVHCM achieves accuracy of 0.901, 0.949, 0.959 and
0.953 respectively at 64 to 512 bits, yielding 10.5% to
16.1% retrieval improvement over the best baselines DNNH
and KSH with deep features. On dataset HMDB51, the
DVHCM achieves around 8.5% to 19.2% improvements
over the best baseline. In contrast to DVHCM, the proposed
without category masking shows inferior performance, but
still it outperforms all the baselines on dataset UCF101
with a maximum improvements of 12.7%, proving the proposed supervised training with intra-pair loss and classification loss does have positive effects on hash code learning. On HMDB51, the proposed w/o CM exceeds the
other baselines except for the DNNH which is trained with
triplet ranking. We consider this performance degradation
is caused by training using videos in short duration length,
as videos from HMDB51 are all less than 5 seconds and
some of them last event less than 2 seconds. With videos
in such short duration, the extracted two frame sets in an
intra-pair are usually very close to each other in low-level
or even pixel-level features, thus the intra-pair loss will be
very small and contributes very little during training, compared with category-loss.

Figure 6 and Figure 7 demonstrate the retrieval performance of the proposed method and the baselines in terms
of precision-recall curve and precision curve with respect
to top returned samples at 64 bits on dataset UCF101 and
HMDB51. We can find that the proposed DVHCM outperforms baselines by large margins under both evaluation
terms.
Figure 8 presents the retrieved relevant videos by the proposed DVHCM approach on dataset UCF101, where we can
see that most of matched videos are both visually and semantically belonging to the the same category.

5. Conclusion
In this work we presented DVHCM, an end-to-end deep
hashing approach with category mask for fast video retrieval. We introduced intra-pair and proposed to learn
hash model by optimizing the classification loss and the
intra-pair loss. The binary bits distribution related to categories was investigated and category masking scheme was
proposed to improve retrieval accuracy. Experimental results show that the proposed method achieves superior performance under various evaluation metrics, compared with
both deep and non-deep state-of-the-arts.

References
[1] Habir: Hashing baseline for image retrieval. 7
[2] L. Cao, Z. Li, Y. Mu, and S.-F. Chang. Submodular video
hashing: a unified framework towards video pooling and indexing. In Proceedings of the 20th ACM international conference on Multimedia, pages 299–308. ACM, 2012. 1, 3
[3] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao,
B. Xu, C. Zhang, and Z. Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed
systems. arXiv preprint arXiv:1512.01274, 2015. 7

[4] B. Coskun, B. Sankur, and N. Memon. Spatio–temporal
transform based video hashing. 8(6):1190–1208, 2006. 1,
3
[5] Y. Duan, J. Lu, Z. Wang, J. Feng, and J. Zhou. Learning
deep binary descriptor with multi-quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 1183–1192, 2017. 1, 2
[6] V. Erin Liong, J. Lu, Y.-P. Tan, and J. Zhou. Cross-modal
deep variational hashing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
4077–4085, 2017. 1, 2
[7] V. Erin Liong, J. Lu, G. Wang, P. Moulin, and J. Zhou. Deep
hashing for compact binary codes learning. In Proceedings
of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 2475–2483, 2015. 1, 2
[8] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–
338, 2010. 1
[9] Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin. Iterative quantization: A procrustean approach to learning binary
codes for large-scale image retrieval. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 35(12):2916–
2929, 2013. 1, 2, 6, 7
[10] Y. Hao, T. Mu, J. Y. Goulermas, J. Jiang, R. Hong, and
M. Wang. Unsupervised t-distributed video hashing and its
deep hashing extension. IEEE Transactions on Image Processing, 26(11):5531–5544, 2017. 3
[11] Y. Hao, T. Mu, R. Hong, M. Wang, N. An, and J. Y. Goulermas. Stochastic multiview hashing for large-scale nearduplicate video retrieval. IEEE Transactions on Multimedia,
19(1):1–14, 2017. 3
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770–778, 2016. 1, 3, 6
[13] W. Hu, N. Xie, L. Li, X. Zeng, and S. Maybank. A survey
on visual content-based video indexing and retrieval. IEEE
Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6):797–819, 2011. 1
[14] H. Jain, J. Zepeda, P. Pérez, and R. Gribonval. Subic: A
supervised, structured binary code for image search. arXiv
preprint arXiv:1708.02932, 2017. 1, 2, 6, 7
[15] Q.-Y. Jiang and W.-J. Li. Scalable graph hashing with feature
transformation. In IJCAI, pages 2248–2254, 2015. 1, 2
[16] A. Krizhevsky. Learning multiple layers of features from
tiny images. Computer Science Department, University of
Toronto, Tech. Report, 2009. 7
[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in neural information processing systems, pages
1097–1105, 2012. 1, 3
[18] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre.
Hmdb: a large video database for human motion recognition. In Computer Vision (ICCV), 2011 IEEE International
Conference on, pages 2556–2563. IEEE, 2011. 5

[19] H. Lai, Y. Pan, Y. Liu, and S. Yan. Simultaneous feature
learning and hash coding with deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3270–3278, 2015. 1, 2, 6, 7
[20] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P.
Graf. Pruning filters for efficient convnets. arXiv preprint
arXiv:1608.08710, 2016. 2, 4
[21] W.-J. Li, S. Wang, and W.-C. Kang. Feature learning based
deep supervised hashing with pairwise labels. arXiv preprint
arXiv:1511.03855, 2015. 1, 2
[22] K. Lin, J. Lu, C.-S. Chen, and J. Zhou. Learning compact
binary descriptors with unsupervised deep neural networks.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 1183–1192, 2016. 1, 2
[23] K. Lin, H.-F. Yang, J.-H. Hsiao, and C.-S. Chen. Deep learning of binary hash codes for fast image retrieval. In Proceedings of the IEEE conference on computer vision and pattern
recognition workshops, pages 27–35, 2015. 2, 6, 7
[24] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer
vision, pages 740–755. Springer, 2014. 1
[25] V. E. Liong, J. Lu, Y.-P. Tan, and J. Zhou. Deep video hashing. IEEE Transactions on Multimedia, 19(6):1209–1219,
2017. 1, 2, 3
[26] H. Liu, R. Wang, S. Shan, and X. Chen. Deep supervised
hashing for fast image retrieval. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
pages 2064–2072, 2016. 1, 2, 4, 6, 7
[27] W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang. Supervised hashing with kernels. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages
2074–2081. IEEE, 2012. 1, 2, 6, 7
[28] W. Liu, J. Wang, S. Kumar, and S.-F. Chang. Hashing with
graphs. In Proceedings of the 28th international conference
on machine learning (ICML-11), pages 1–8, 2011. 1, 2, 6, 7
[29] L. v. d. Maaten and G. Hinton. Visualizing data using t-sne.
Journal of Machine Learning Research, 9(Nov):2579–2605,
2008. 1
[30] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz.
Pruning convolutional neural networks for resource efficient
transfer learning. arXiv preprint arXiv:1611.06440, 2016. 2,
4
[31] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211–252,
2015. 1
[32] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014. 1, 3
[33] J. Song, Y. Yang, Z. Huang, H. T. Shen, and J. Luo. Effective
multiple feature hashing for large-scale near-duplicate video
retrieval. IEEE Transactions on Multimedia, 15(8):1997–
2008, 2013. 1, 3
[34] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset
of 101 human actions classes from videos in the wild. arXiv
preprint arXiv:1212.0402, 2012. 1, 5

[35] J. Sun, X. Liu, W. Wan, J. Li, D. Zhao, and H. Zhang. Video
hashing based on appearance and attention features fusion
via dbn. Neurocomputing, 213:84–94, 2016. 1, 3
[36] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 1–9, 2015. 6, 7
[37] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain
adaptation. arXiv preprint arXiv:1706.07522, 2017. 1, 2
[38] Y. Weiss, A. Torralba, and R. Fergus. Spectral hashing. In
Advances in neural information processing systems, pages
1753–1760, 2009. 1, 2
[39] L. Weng and B. Preneel. From image hashing to video hashing. In International Conference on Multimedia Modeling,
pages 662–668. Springer, 2010. 1, 3
[40] G. Wu, L. Liu, Y. Guo, G. Ding, J. Han, J. Shen, and L. Shao.
Unsupervised deep video hashing with balanced rotation. IJCAI, 2017. 1, 2, 3
[41] R. Xia, Y. Pan, H. Lai, C. Liu, and S. Yan. Supervised hashing for image retrieval via image representation learning. In
AAAI, volume 1, pages 2156–2162, 2014. 1, 2
[42] G. Ye, D. Liu, J. Wang, and S.-F. Chang. Large-scale video
hashing via structure learning. In Proceedings of the IEEE
International Conference on Computer Vision, pages 2272–
2279, 2013. 1, 3, 6, 7
[43] H. Zhang, M. Wang, R. Hong, and T.-S. Chua. Play and
rewind: Optimizing binary representations of videos by selfsupervised temporal hashing. In Proceedings of the 2016
ACM on Multimedia Conference, pages 781–790. ACM,
2016. 3
[44] X. Zhu, L. Zhang, and Z. Huang. A sparse embedding and
least variance encoding approach to hashing. IEEE transactions on image processing, 23(9):3737–3750, 2014. 1, 2

