1

Tracking in Aerial Hyperspectral Videos using Deep
Kernelized Correlation Filters
Burak Uzkent* , Aneesh Rangnekar* , and Matthew J. Hoffman**

arXiv:1711.07235v2 [] 27 Dec 2017

*

Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology
**
School of Mathematical Sciences, Rochester Institute of Technology
{bxu2522, apr2635, mjhsma}@rit.edu

Abstract—Hyperspectral imaging holds enormous potential to
improve the state-of-the-art in aerial vehicle tracking with low
spatial and temporal resolutions. Recently, adaptive multi-modal
hyperspectral sensors, controlled by Dynamic Data Driven Applications Systems (DDDAS) methodology, have attracted growing
interest due to their ability to record extended data quickly from
the aerial platforms. In this study, we apply popular concepts
from traditional object tracking—(1) Kernelized Correlation
Filters (KCF) and (2) Deep Convolutional Neural Network (CNN)
features—to the hyperspectral aerial tracking domain. Specifically, we propose the Deep Hyperspectral Kernelized Correlation
Filter based tracker (DeepHKCF) to efficiently track aerial
vehicles using an adaptive multi-modal hyperspectral sensor. We
address low temporal resolution by designing a single KCF-inmultiple Regions-of-Interest (ROIs) approach to cover a reasonable
large area. To increase the speed of deep convolutional features
extraction from multiple ROIs, we design an effective ROI
mapping strategy. The proposed tracker also provides flexibility
to couple it to the more advanced correlation filter trackers. The
DeepHKCF tracker performs exceptionally with deep features
set up in a synthetic hyperspectral video generated by the
Digital Imaging and Remote Sensing Image Generation (DIRSIG)
software. Additionally, we generate a large, synthetic, singlechannel dataset using DIRSIG to perform vehicle classification
in the Wide Area Motion Imagery (WAMI) platform . This way,
the high-fidelity of the DIRSIG software is proved and a large
scale aerial vehicle classification dataset is released to support
studies on vehicle detection and tracking in the WAMI platform.
Index Terms—vehicle tracking, discriminative tracking, hyperspectral sensing, deep features.

I. I NTRODUCTION
ERIAL object tracking is a popular topic due to its large
range of applications in security, traffic surveillance,
autonomous driving, and UAV monitoring. Tracking from
aerial platforms can be performed with a number of data
modalities including, but not limited to, grayscale [1], thermal
[2], color [3] and recently, hyperspectral imagery [4], [5].
Each modality has been exploited for a unique application
and comes with its own advantages and disadvantages. In this
paper, we will focus on two specific types of sensor modalities
: (1) Wide Area Motion Imagery (WAMI), and (2) adaptive
hyperspectral imagery.
The WAMI platform has enjoyed much attention recently,
due to its large area coverage and reasonable spatial resolution.
It scans up to a 5km x 5km area at 2 fps where vehicles occupy
roughly 100-200 pixels. Such large area coverage becomes
possible by employing an array of 6 cameras. Each camera

A

captures a large field of view single-channel image with slight
overlap to the neighboring frame. One can perform persistent
tracking utilizing this large field of view image. On the other
hand, the low spatial resolution has good performance in vehicle tracking under certain circumstances but is not super helpful when it comes to tackling background clutter, occlusions,
and low contrast objects. To counter this, multiple appearancebased features like textures, color histograms, histogram of
gradients, and motion cues are used in combination, leading
to multiple heat maps for the search area. This is not feasible in
real-time tracking as the methods can become computationally
expensive and hence there exists a balance between designing
complex models vs real-time implementation. Using deep
learning techniques in an end-to-end fashion, on the other
hand, is not feasible due to availability of the small size
datasets captured by WAMI. The datasets recorded by WAMI
includes : (1) WPAFB 2009 [6] and (2) CLIF [7] dataset where
each one includes a single video with less than 100 frames.
Deep learning can be feasible to perform tracking from WAMI
with the availability of many more datasets which can take
a long time and be very costly. Since data collection from
an aerial platform is lengthy and costly, one can utilize deep
learning architectures as feature extractors rather than an endto-end tracking system. In this case, however, WAMI is not a
practical platform since it provides single-channel imagery and
the deep learning architectures are trained on the large-scale
image recognition dataset, ImageNet [8], which uses RGB
images.
Due to the availability of rich spectral information that
can be recorded by full or adaptive hyperspectral sensors,
hyperspectral target tracking from aerial platforms has recently seen increasing popularity [4], [9], [5] as a potential
improvement over the WAMI platform. Using this extended
spectral information, one can better utilize pre-trained deep
architecture in a tracking framework. Also, the use of spectral
information can reduce the need to collect higher spatial
resolution information. On the other hand, accessibility to a
wide spectral range comes with its unique set of challenges,
the most important ones being: (1) transmission of massive
amount of data on-board the collection platform and, (2)
limited temporal resolution. For example, full hyperspectral
sensors record hyperspectral channels for the overlooking
scene individually. Although very informative, the extremely
low temporal resolution and distortion in spectral dimension

2

prohibits using them for tracking objects. Adaptive hyperspectral sensors are more useful since they they pose a number of
advantages over full hyperspectral sensors and can be easily
controlled by the DDDAS framework. The Dynamic DataDriven Applications Systems (DDDAS) framework is very
suitable for adaptive sensor control in object tracking to collect
the most beneficial data by switching between the sensor
modalities. For instance, the Rochester Institute of Technology Multi-object Spectrometer (RITMOS) can record data in
multiple modalities like panchromatic and hyperspectral data,
at comparatively reasonable frame rates by rapidly switching
from single-band to hyperspectral modality. This allows the
sensor to acquire a wide field-of-view (FOV) panchromatic
image followed by a narrow FOV hyperspectral data capture
[4], [5]. For the purpose of this study which is more tracking
oriented and less control oriented, we refer the readers to [9],
[10] for more information on DDDAS inspired sensor control.
A. Motivations
The rich sensory information from hyperspectral imagery
has been utilized by generative trackers [4], [5]. The discriminative and deep learning driven trackers on the other
hand have recently improved the traditional object tracking
dramatically. The main challenges behind the application of
discriminative and deep learning trackers in aerial hyperspectral images are:
• Well-established discriminative algorithms such as Efficient Convolutional Operators (ECO) [3], Kernelized
Correlation filters (KCF) [11], Struck [12], and Trackinglearning-detection (TLD) [13] are mostly associated with
close-angle color image single/multi object tracking at
high video frame rates and thus, consider a small ROI.
• The cost of collecting data from aerial platforms makes
it hard to find/collect large samples of aerial data, leaving
the community to split a single video into training and
validation sets [14], [15]. This leads to very optimistic
results during off-line tracking due to minimal variance
in the training dataset, which results poor performance
during online tracking.
B. Contributions
This study addresses the unique challenges posed by the
application of discriminative trackers to aerial platforms. We
propose a novel method that employs a discriminative tracker
that is robust to low temporal (around 1.42 fps) and spatial
resolutions (0.3 m). First, we design a method to enlarge the
area considered by the tracker to handle the low temporal
resolution. Given the rich hyperspectral imagery, we utilize
pre-trained deep convolutional networks as feature encoders
to boost tracking performance. To accommodate deep features
in a near real-time tracking system, we design an ROI mapping
strategy that only forward-passes the large ROI and projects
the individual ROIs to the large ROI feature maps. To the best
of our knowledge, this is the first time an adaptive hyperspectral sensor-inspired discriminative tracker (DeepHKCF)
has been proposed to perform robust single target tracking
in spectral aerial imagery that can be generalized to the

WAMI platform. Finally, we evaluate the proposed tracker on a
synthetic hyperspectral video generated by the Digital Imaging
and Remote Sensing (DIRSIG) software [16]. To prove the
high-fidelity of this video, we synthesize a large single-channel
aerial dataset using DIRSIG and train a CNN on it to classify
images from the real dataset (WAMI). We refer the readers
to following link to access our synthetic vehicle classification
dataset (https://buzkent86.github.io/datasets/).
II. R ELATED W ORK
Our work builds upon the previous work of Henriques
et al.[17], [11] with Kernelized Correlation Filters (KCF).
Having already discussed the lack of development in using
discriminative trackers for aerial tracking, we limit our review
of related work to the datasets available for aerial vehicle
tracking and recent research in WAMI and spectral vehicle
tracking.
A review of the related papers indicates that there is
limited availability of annotated datasets for aerial tracking
in which trackers can be evaluated for their accuracy and
robustness. The VIVID [18] is the one of the few publicly
available datasets that comes with RGB images, however, it
has application limitations due to its relatively small sequence
size, low temporal resolution (6 fps), and few occlusions.
An upgraded version of this dataset, UAV123, was recently
released for research purposes by Meuller et al.[19], but the
ground sampling distance (GSD) is significantly lower than
the high-altitude aerial platforms - thus resulting in objects
occupying more than 500-1000 pixels whereas in WAMI
platform vehicles are represented by 100-200 pixels. At the
same time, the dataset has sequences at 30 fps, drastically
higher than standard WAMI/Spectral sequences which are
generally in the 1.42 fps - 2 fps range. Since flying RITMOS
on an aerial platform is still an ongoing area of development
and due to lack of any other real dataset in this area, we use
tracking in real world WAMI as a baseline for our approach
and then port to using synthetically generated RITMOS-like
data to evaluate the performance of our proposed tracker.
A recent survey by Sommer et al.[20] showed that a major
share of state-of-the-art methods in WAMI moving object
detection broadly fall under two categories: 2-frame differencing and background subtraction. Teutsch and Grinberg [21]
attempted to track vehicles in WAMI using the former with
promising results, however a large number of false negatives
still occur during the initial 2-frame differencing for making
object proposals. Cormier et al.[22] showed that combining
Local Binary Patterns (LBP) and Local Variance Measure
(VAR) histograms as descriptors, with the usage of Hellinger
distance as a measure for similarity, can significantly improve
vehicle detection in WAMI data, which can result in superior
tracking. Recently, LaLonde et al.[14] used a multiple-frame
fully convolutional deep model for single and multi-object
detection in WAMI videos with great results. On the other
hand, there have been few tracking studies using hyperspectral
imagery. Uzkent et al.[4] considered the RITMOS platform
to perform single target tracking by sampling a narrow FOV
region of interest (ROI) hyperspectrally to search for the

3

target. Using the full frame single-channel image, they remove the global camera motion and couple the narrow-ROI
detection and Bayes Filter based tracking modules. Many
remote-sensing based processing stages are applied before an
HoG-SVM classifier is finally trained to detect vehicles. The
proposed approach showed promising results, however they
relied on classifiers trained on the same scene to the test video.
A more recent approach [5] proposed the adaptive mixture of
hyperspectral likelihood maps into a single distance map, thus
removing the need for off-line learned classifiers.
III. A DAPTIVE H YPERSPECTRAL S ENSING C ONCEPT
Single modality electro-optical sensors such as WAMI are
commonly used for aerial tracking, however, the unique challenges posed by aerial platforms can be better addressed by
smarter multi-modal data acquisition. In this direction, the
Rochester Institute of Technology Multi-object Spectrometer
(RITMOS) concept is utilized by a number of trackers [23],
[24], [4], [5] as an example of a sensor that can collect a
small, targeted amount of hyperspectral data. The RITMOS
captures data in two different modalities : (1) full frame
single channel image, and (2) limited hyperspectral data from
the desired pixel locations. It can acquire a full-frame single
channel image in about 0.1 sec and scan a row of pixels
hyperspectrally in 1 ms. Such an adaptive and multi-modal
data concept provides more freedom to address aerial tracking
challenges. Driven by this freedom and specifications, we
design a discriminative tracker to operate on this platform as
shown in Fig. 1. For more information on RITMOS see [25],
[26].
1500x1500 px

96 x 96 px

A single KCF - Multiple ROIs approach
48 px 48 px 48 px 48 px

1000 nm
990 nm
980 nm
440 nm
430 nm
420 nm
410 nm
400 nm

t s.

wide field-of-view
single channel image
acquisition for
image registration

t + 0.1 s

narrow field-of-view
hyperspectral image
acquisition for
tracking

48 px
48 px
48 px
48 px

t + 0.2 s

DeepHKCF Tracker
to operate on
hyperspectral image

t + 0.7 s

Fig. 1: The proposed Kernelized Correlation Filter driven
tracker inspired by RITMOS.

A. Synthetic Scenarios
The Digital Imaging and Remote Sensing (DIRSIG) software has been used before to generate spectral scenarios
for varied applications that use conventional computer vision
techniques and deep learning based models [9], [5], [27], [28].
Since flying spectral sensors on an aerial platform is still an
ongoing area of development due to the high costs involved,
we evaluate our tracker on synthetic scenarios generated using
DIRSIG by [29], [30], [26]. In particular, we will focus on
two scenarios : (1) with trees and (2) without trees. We track
43 vehicles in both scenarios similar to the previous studies.
One frame from each scenario can be visualized in fig. 2. In
addition, we generate a synthetic single-channel aerial dataset
for training a CNN and use it to perform vehicle classification
on the real WAMI platform, similar to [27] (Sect. V).

(a) 130th frame (with trees)

(b) 130th frame (without trees)

Fig. 2: Part of the two frames from the synthetic scenarios
generated by DIRSIG. The scene comes from the Mega-Scene
1 area available in DIRSIG. The Mega-Scene 1 area represents
part of Rochester, NY.

IV. P ROPOSED A LGORITHM
The Kernelized Correlation Filter (KCF) is a well-known
single target tracking method that has emerged as a state-ofthe-art tracker in the VOC15 tracking benchmark. Its work
flow has been detailed in other papers [17], [11]. This section,
first, briefly goes over the KCF and document the parts of
proposed DeepHKCF tracker later.
The KCF has emerged as a high accuracy tracker that
can operate at hundreds of frame rate. Its computational
efficiency is derived from the correlation filter framework that
represents training examples using a circulant matrix. The
fact that a circulant matrix can be diagonalized by Discrete
Fourier transform is the key to reducing the complexity of
any tracking method based on the correlation filter. The offdiagonal elements become zero whereas the diagonal elements
represent the eigenvalues of the circulant matrix. The eigenvalues are equal to the DFT transformation of the base sample
(x) elements. The Kernelized Correlation Filter, in particular,
applies a kernel to x to transform the feature channels to
a more discriminative domain. The circulant matrix is then
formed by applying cyclic shifts on the kernelized x. This
kernelization operation maintains O(nlog(n)) complexity unlike other kernel algorithms which are O(n2 ).
Essentially, the KCF solves the problem of regression in the
form of the regularization (ridge regression):
E(h) =

C
C
X
1
λX
||y −
hc ∗ xc ||2 +
||hc ||2
2
2
c=1
c=1

(1)

where y represents the desired continuous response whereas h
and xc represent the learned correlation filter and the training
template for the given channel. The parameter c enables one to
integrate features in multiple channels, such as HoG and color,
in this setup [11], [31]. A closed-form solution for Equation 1
exists. To reduce the complexity of the closed-form solution,
an element-wise multiplication in the frequency domain was
proposed for ŵ:
ŵ = x̂∗ ∗ ŷ(x̂∗ ∗ x̂ + λ)−1 .

(2)

A non-linear version of this closed-form solution addresses
further geometric and photometric variations [11]. The diago-

4

nalized Fourier domain dual form solution (non-linear version)
is expressed as
α̂ = ŷ(k̂ xx + λ)−1 ,
(3)
where k̂ xx represents the first row of the kernel matrix K
known as gram matrix and is formulated as
k xx

0

ROI11 ROI12
48 px
Full ROI
( 96 x 96 px )

ROI21 ROI22 ROI23 ROI24
48 px

C

X
0
1
= exp(− 2 (||x||2 + ||x ||2 − 2F −1 (
x̂∗c
α
c

0

x̂c ))). (4)

ROI31 ROI32 ROI33 ROI34
48 px

An earlier version based on this formulation employed
grayscale feature (C = 1) to learn the solution vector w
(MOSSE). Later, multi-channel features such as HoG, Color,
and a concatenation of them showed improved accuracy [11],
[31], [32], [33], [34]. Once the kernel is learned, one can
execute an object detection with multi-channel features as
r(z) = F −1 (k̂ xz

α̂),

(5)

where r denotes the correlation response at all cyclic shifts of
the first row of the kernel matrix.
The temporal information can be further integrated into the
KCF by updating the filter at every frame as follows.
α̂t = β α̂t + (1 − β)αt−1
ˆ .

ROI13 ROI14

(6)

This correlation filter framework only estimates the translation of the object whereas the scale of the object can be
updated by running a correlation filter on different size ROIs
with same centroids [35]. By correlating a filter with different
ROIs, we can get multiple response maps and choose the one
with highest confidence to estimate the new scale of the target.
In this study, we do not estimate the scale of the target as the
scenarios are captured from a fixed altitude platform.

ROI41 ROI42 ROI43 ROI44
48 px
48 px

48 px

48 px

48 px

Fig. 3: The proposed single KCF - multiple ROI approach to
enlarge the ROI to overcome large displacement of vehicles in
low resolution data. The vehicle of interest is shown in yellow
rectangles.

at boundaries due to FFT operation. This approach can be
formulated as
r(zij ) = F −1 (k̂ xzij

α̂)

(7)

where i and j represent the indexes for different ROIs. A
simple way to estimate the new position of the target in this
framework would be using the peak-to-side-lobe (PSR) values
in ROIs and finding the position of the pixel with maximum
confidence in all ROIs as
r(zf inal ) = argmax
(PSR(r(zij )))
√

(8)

i,j∈ m

A. Single KCF-Multiple ROIs Approach
Discriminative trackers like KCF learn to function in an
online manner by collecting positive and negative samples and
then detecting the target of interest in a ROI to update the
classifier. Unfortunately, the discriminative features required
for KCF to function well are hard to collect from aerial
imaging platforms due to their low spatial resolution.
The vanilla-form KCF requires relatively small ROIs, as the
appearance-based features deteriorate with larger background
context in the ROI. An area as big as three times the size
of the target is considered in vanilla-form KCF. Increasing the
context size leads to background dominated features, resulting
in confusion between different objects. Unfortunately, the
platform we consider has lower temporal resolution (1.4 fps)
than the other aerial platforms, leading to large displacement
of objects in successive frames. Adding the platform motion
into this picture makes the application of vanilla-form KCF in
aerial platforms extremely difficult.
To handle these challenges, we propose a single KCF-inmultiple ROIs approach as shown in fig. 3. Our approach
applies the same KCF to different ROIs overlapping each other
to minimize the likelihood of target loss. It is essential to
have reasonable overlap between the ROIs (sect. VI-F) as we
filter each ROI with a hanning window to avoid distortion

where m represents the number of ROIs in full ROI. The
PSR, on the other hand, denotes the margin between the peak
value in the response map and the mean of the sidelobe
corresponding to the area excluding the 11x11 pixels around
the peak. The result is normalized by the standard deviation
of the sidelobe as follows.
max(r(zf inal )) − µsidelobe
P SR(r(zf inal )) =
(9)
σsidelobe
This position estimation approach can be softened by considering all the ROIs with PSR values larger than a predetermined threshold. In this case, the formulation can be
reorganized as follows.
r(zf inal ) =

βij =

√

√

i

j

m m
X
X

βij ∗ r(zij ),

0, if P SR(r(zij )) > T
P SR(r(zij )), otherwise.

(10)
(11)

By softening our decision, we perform low-pass filtering
and avoid jumps to other objects that has a high PSR value in
only one ROI.
As mentioned earlier, the single KCF-multiple ROIs approach can better handle the low temporal resolution than
the traditional KCF. On the other hand, it increases the

5

complexity linearly to O(m∗nlog(n)) from O(nlog(n)) where
m represents the number of ROIs in the full ROI. The low
temporal frame rate of the scenario helps us accommodate this
approach in the DeepHKCF tracker. One can further increase
the frame rate by running the KCF on the multiple ROIs in
parallel as ROI operations are independent.
B. Traditional Low-level Features
Tracking-by-detection algorithms exploit low-level features
such as Histogram of Oriented Gradients and Color-naming
features [36], [35], [37] to perform discriminative tracking
until the emergence of deep CNN architectures in the computer
vision field. The first correlation filter tracker, MOSSE [38],
used a grayscale channel to learn the classifier vector. Following the MOSSE tracker, a correlation filter accommodating
multi-channel features was proposed to boost tracking. Later,
the SAMF tracker [35] was proposed to concatenate multichannel HoG and color-naming features. Finally, a kernelized
version of correlation filter (KCF) using multi-channel features
was proposed to further improve tracking without drastically
increasing the computational complexity. In this study, we
follow the KCF tracker and concatenate multiple features as
in the SAMF tracker. More specifically, we concatenate the
HoG feature channels and pure hyperspectral channels and
apply Gaussian kernel operation to learn a more discriminative
model as follows
0
0
1
(12)
k xz = exp(− 2 (||x||2 + ||x ||2 −
α
CX
CX
HSI
HoG
0
0
2F −1 (
x̂∗cHoG ẑcHoG +
x̂∗cHSI ẑcHSI )
cHoG =1

cHSI =1

where CHoG and CHSI represent the number of channels
in HoG features and hyperspectral data. For hyperspectral
channel features, we use all 61 bands provided by the RITMOS
platform in the ROI. To complement spectral features with
shape features, we utilize the fHoG features proposed by
[37]. The fHoG features provide 31 feature channels where
channels specialize in different orientation bins computed over
the gradients in red, green and blue channels. To sum up, the
KCF tracker with traditional features learns a model over the
Gaussian-kernelized space of 91 dimensional feature channels.
C. Deep Convolutional Features
Convolutional Neural Networks have enjoyed enormous
popularity in the computer vision/machine learning community following the emergence of large-scale datasets such as
ImageNet. They have surpassed the state-of-the-art methods
in all the computer vision/machine learning domains, thanks
to their capability to learn varying levels of feature abstractions. Similar to image classification and object detection, the
benchmark performances for object tracking have improved
dramatically over the last few years.
The first studies utilizing CNN architectures in object tracking focused on employing the features learned in architectures such as AlexNet [39], VGGNet [40] and GoogleNet
[41] trained on the ImageNet large scale image classification

dataset [8]. [42] extracted low-level features from VGGNet to
learn a more discriminative correlation filter. Specifically, they
encoded objects with the activations of the first several convolutional layers from VGGNet. This setting provided them
with a 64x64x96 dimensional low-level feature set that can be
interpreted as a more advanced version of HoG features. They
reported slight improvement in the VOC15 object tracking
challenge with deep CNN features over the HoG features.
Going deeper is a major key to achieving the state-of-the-art in
most computer vision challenges, however, the nature of deep
CNN architectures prohibits us from applying high-level features in tracking-by-detection algorithms. This is mainly due
to increasing translation invariance in deeper layers resulting
from spatial pooling operations. [43] proposed a hierarchical
multiple KCFs tracking framework accommodating a higher
level of feature encodings with CNNs. Their method first
employs the activations of the fifth convolutional layer to
perform a coarse target search. In the next stage, they use
mid-level features computed over the position estimated in the
first step to fine-tune the estimated position. In the final step,
low level features are extracted from the previously estimated
position to further fine-tune the estimate.
The object tracking community later migrated to training
architectures to perform object tracking in an end-to-end
framework [44], [45], [46]. In this direction, Siamese Networks
have gained the reputation of the most efficient and effective
architecture in tracking. Two branches consisting of the same
architecture layers are used in a typical Siamese Network. The
bottom branch is provided the ground truth of an object of
interest in an ROI, whereas the top branch is assigned the task
of estimating the position of the object given the new ROI.
Late fusion of the branches is performed and the new position
is predicted. The Siamese Networks have surpassed all the
other deep tracking-by-detection algorithms in the VOC15
challenge, however, the lack of aerial datasets prohibits us
from applying these networks to the aerial tracking domain.
For this reason, we believe that, at this stage, it is more
practical to use the CNN architectures as feature extractors
in aerial platforms where we have RGB data.
In this study, we follow an approach similar to [42] to learn
a discriminative model for the KCF as shown in fig. 4. Thanks
to low spatial resolution in our scenario, we can pursue a
slightly higher level of abstraction of objects. In particular, we
apply the activations of the fifth convolutional layer learned
in VGGNet [40] trained over ImageNet [8]. Additionally, we
experiment with different levels of object abstractions in the
results section.
3x3 Conv +
ReLU

3x3 Conv +
ReLU

Max
Pooling

3x3 Conv +
ReLU

Max
3x3 Conv + Pooling
ReLU

3x3 Conv +
ReLU

650 nm
550 nm
450 nm

ROI
(48 x 48 px)

224x224x64

224x224x64

112x112x128

112x112x128

112x112x128

56x56x256

56x56x256

Fig. 4: The proposed feature extraction scheme in the deep
hyperspectral kernelized correlation filter tracker. In this figure,
the first 5 convolutional layers of the VGGNet [40] are
considered.

6

DIRSIG imagery provides us with a full-frame grayscale
image as well as a narrow field of view hyperspectral image
at 1.42 fps. Unlike other aerial platforms, it provides hyperspectral data in the visible wavelength range, enabling the use
of deep CNN architectures trained on ImageNet consisting
of RGB images. One can pick the central red, green and
blue channels and forward-pass them through the layers of
interest. Another approach could be computing the average of
red, green and blue channels in their respective range to come
up with the representative red, green and blue channel images
to feed the CNN. Our experiments favor the first approach as
the latter approach could introduce undesired noise due to the
averaging operation.
1) Fast Convolutional Features with ROI Mapping: The
single KCF-multiple ROIs approach treats each ROI independently to compute the filter response. This requires forwardpassing individual ROIs through the CNN architecture as
shown in fig. 4. Such an inefficient approach leads to a slower
tracker. To increase the run-time performance and perform
near real-time tracking at the platform frame-rate, we use the
ROI mapping strategy commonly used in convolutional object
detectors such as Fast R-CNN [47], Faster R-CNN [48], and
R-FCN [49]. This way, we only forward-pass the full ROI and
project the individual ROIs to the feature maps extracted from
the full ROI as shown in fig. 5.
Full ROI
(96x96
px)
Full ROI
(96x96 px)

ROI Mapping
ROI Mapping

roi14

roi14

f(roi14)

roi41

f(roi14)

roi41
f(roi41)

f(roi41)

ROI Mapping
Mappingin FastDeepHKCF tracker.
(a) ROI mapping forROI
detection
Full ROI
(96x96 px)
Full ROI
(96x96 px)

ROI Mapping

ROI Mapping
roi

roi

f(roi)

f(roi)

(b) ROI mapping for training in FastDeepHKCF tracker.

Fig. 5: Proposed ROI mapping strategy to avoid individual
ROIs forward-passing through the convolutional network.
With the ROI mapping method, eqns. 7 and 10 can be
replaced with the following formulations to perform detection
in the FastDeepHKCF tracker.
r(f (dROI, droiij , z)) = F −1 (k̂ xf (dROI,droiij ,z)
√

r(zf inal ) =

m

√

XX
i

α̂), (13)

m

βij ∗ r(f (dROI, droiij , z))

Once we estimate the translation of the target, we update the
filter using the Fourier domain solution as in eqn. 3. First, the
96x96 px neighborhood around the target is considered and
forward-passed through the convolutional network as shown
in fig. 5b. To match the detection ROI size (fig. 5a), we then
project the central 48x48 px area to the feature maps and
reformulate the solution as
α̂ = ŷ(k̂ f (tROI,troi,x)f (tROI,troi,x) + λ)−1

(15)

where tROI and troi represent the full training ROI and actual
training ROI mapped to feature maps of the tROI using the
function f . On the other hand, we can avoid forward-passing
the training ROI if the actual training ROI, troi, is a subset
of the detection ROI, dROI. In this case, the Fourier domain
solution can be reformulated as
α̂ = ŷ(k̂ f (dROI,troi,z)f (dROI,troi,z) + λ)−1 .

(16)

V. V EHICLE C LASSIFICATION IN WAMI P LATFORM BY
USING A S YNTHETIC DATASET
As discussed in Sect. I, detecting cars with high accuracy
is a major component of tracking algorithms utilizing the
WAMI platform. Unsurprisingly, object detection in traditional
images has improved rapidly over the last five years with
the emergence of deep learning. To prove this, the state-ofthe-art detection accuracy in the PASCAL VOC07 dataset
increased from 35% accuracy to over 85% accuracy. However,
this dramatic increase is not the case in vehicle detection from
the aerial platforms (WAMI). This is due to two major reasons
: (1) the lack of a large dataset captured from the WAMI
platform and (2) the lack of color channels prevents smooth
transfer learning from the networks trained on the ImageNet.
In this study, we build a synthetic single-channel vehicle
classification dataset using DIRSIG and fine-tune a CNN to
perform vehicle classification on the real platform (WAMI).
By building this dataset, the car detection/classification from
the WAMI platform will require a smaller number of samples
from WAMI. This also proves the high fidelity of hyperspectral
videos used to evaluate the DeepHKCF tracker.
To build such a large-scale dataset, we generate fullframe hyperspectral images captured from the Mega-Scene-I
scene available in DIRSIG [16] with different settings. The
simulation setting is designed as a function of time. We
keep the other parameters similar to the simulation used to
generate the RITMOS-like scenario. Overall, nine different
simulations are generated from the same scene with the same
vehicular traffic and platform motion to the tracking video.
One can generate as many simulations as desired, however, we
believe that given the aforementioned settings we can generate
reasonable number of vehicle and background samples in nine
simulations.

(14)

j

where dROI represents the full detection ROI used to get
the convolutional features z. The individual detection ROIs,
droiij , are then projected to the feature map, z, through the
projection function f .

A. Temporal Data Augmentation
To keep things manageable, one can generate one frame
per simulation mentioned in the previous section. However,
then, we need to increase the number of simulations by adding
more temporal-variance and changing the initial platform

7

location. Changing the platform location in a large number
of simulations can be a tedious task, thus, to avoid that, we
perform temporal data augmentation by generating low frame
rate videos on a moving platform. More specifically, the frame
rate for each simulation is set to 0.2 fps, resulting in 20 images
per simulation. This way, we can capture cars from different
angles with different backgrounds.
B. Hyperspectral Data Augmentation
The data augmentation is highly important in our case as
we mimic the WAMI platform in a dataset consisting of fully
synthetic images. In particular, it is difficult to approximate
the spectral sensitivity curve of a real platform synthetically. In
this direction, the same car samples from different wavelengths
are augmented to better approximate the WAMI platform
internal mechanics.
Similar to our tracking video simulation, we stick with 61
channels in the visible (400 nm) to near infrared (1000 nm)
wavelength range. In a single-band image setting with 0.2
fps, we produce about 180 images leading to small spectral
variance in the dataset. By using all 61 channels, we generate
over 1000 images over the 9 simulations, considering time and
spectral depth. This approach has the potential downside of
generating to a dataset dominated by highly similar images. To
address this, we sample 6 bands from 6 uniform distributions
covering the 61 channels as shown in fig. 6. This increases
the spectral variance while ensuring a reasonably large gap
between the augmented images at different wavelengths.
400 - 500 nm

500 - 600 nm

randomly sampled
bands from
each interval

490 nm

600 - 700 nm

700 - 800 nm

800 - 900 nm

900 - 1000 nm

530 nm

670 nm

750 nm

830 nm

960 nm

Fig. 6: The proposed hyperspectral data augmentation to
increase spectral variance in the dataset. One can observe
the variance in the same sample at reasonably distinct wavelengths.

C. Positive and Negative Samples Collection
The procedure described above produces about 27613 vehicle chips, where each chip is cropped from 48x48 pixels area.
The vehicles are located in the central position of the positive
chips. In our scenarios, similar to the WAMI platform, a vehicle is on average represented by 20x10 pixels. Adding context
in positive samples seems to improve the learned weights in
a CNN [15]. To collect negative samples, we perform hardnegative mining by considering areas surrounding the positive
samples. We randomly sample from an area whose center is
T=10-30 pixels away from the center of the positive sample.
Our final dataset consists of 55226 chips captured from
different positions of Mega-Scene 1 at different times. To

(a) DIRSIG positive samples

(b) WAMI positive samples

Fig. 7: Positive training samples collected in the DIRSIG vehicle classification dataset and real platform validation samples
collected from the WAMI platform (CLIFF06 and CLIFF07)
videos.

validate the performance on the WAMI platform, we annotate 600 positive and negative chips from the CLIFF06 and
CLIFF07 videos captured from the WAMI platform. Some of
the positive samples from the training and validation dataset
can be visualized in fig. 7. Finally, we train a well-known CNN
architecture to perform vehicle classification on the WAMI
platform.
The architecture used in this study is the ZFNet, an optimized version of AlexNet. We adapt two different training
strategies : (1) training from scratch and (2) fine-tuning the
weights learned on the ImageNet using the synthetic aerial
vehicle detection dataset. In the latter approach, the learning
rate is set to 0.0001 other than the classification layer. The
classification layer is assigned 0.0005 learning rate. On the
other hand, in the former method, we tune the learning rate to
0.1. The ZFNet from scratch is trained for 4200 iterations
with the batch size of 64 whereas the one pre-trained on
ImageNet is trained for 400 iterations with the same batch
size. In this two experiments, the networks are validated on
the 600 samples from WAMI (CLIFF06 and CLIFF07). To
integrate further context information into the learned weights,
we introduce dilated convolutions with hole size 2 and 1 in
the 1st and 2nd convolutional layers [50]. Finally, we follow
a two-stage training strategy that uses the 200 WAMI samples
to further update the weights from the fine-tuned ZFNet. The
model is validated on the remaining 400 WAMI samples. This,
as expected, boosts the classification accuracy on the WAMI
platform. Training is performed on the NVIDIA Tesla K80
GPU in Caffe framework [51].
Method
Accuracy

ZFNet from Scratch
93.20

ZFNet (ImageNet)
92.230

ZFNet*
97.0

AlexNet - WAMI [15]
97.1

TABLE I: Performance of the trained neural networks on the
WAMI vehicle classification task. In the ”*” case, 200 WAMI
samples are used to further fine-tune and validate the network
on the 400 WAMI samples. [15] splits the WPAFB2009 video
(WAMI) into training and validation.
As seen in table I, over 93% accuracy is achieved by only
using our synthetic dataset to train the ZFNet. This proves the

8

— 2nd Best

— Best
Method
Pr (20 px)
Pr (50 px)
CLE
FPS

— 3th Best

DeepHKCF - ZFNet-2
(4x4 ROI)

DeepHKCF - VGG16-5
(4x4 ROI)

FastDeepHKCF - VGG16-5
(4x4 ROI)

HKCF - fHoG + HSI
(4x4 ROI)

HKCF - HSI
(4x4 ROI)

HKCF - fHoG
(4x4 ROI)

KCF - fHoG
(1x1 ROI)

ECO - fHoG

ECO - VGG16-5

70.13
81.05
48.97
0.51

68.45
80.27
51.71
0.22

66.26
80.65
51.15
1.11

38.79
57.56
118.73
3.01

39.53
54.30
146.30
2.32

39.30
58.58
119.04
2.98

38.74
42.12
179.76
25.11

39.86
43.24
168.43
2.70

64.15
67.61
113.46
1.19

TABLE II: A detailed analysis of the DeepHKCF tracker, its variants and original KCF and ECO trackers on the DIRSIG notrees video. The DeepHKCF tracker with ROI mapping delivers the optimal results considering the trade-off between run-time
and tracking performance. The experiments are carried out on a CPU with 8GB RAM and 2.9GHz i5 processor and the CLE
represents the average central location error of the tracker.

high fidelity of the hyperspectral scenario used to evaluate
the deep hyperspectral kernelized correlation filter tracker.
To further improve the accuracy, a small amount of WAMI
samples are used to train a more WAMI domain-specific
model. This improves the accuracy up to 97% reaching the
state-of-the-art in vehicle classification in WAMI platform.
Finally, with the availability of this dataset, the need to collect
large amount of training samples from the WAMI platform
is removed. To support the aerial vehicle detection related
studies, we plan on releasing the full images with ground truth
locations of the vehicles. This will give more freedom to train
detection-domain architectures such as Faster-RCNN [48], RFCNN [49], YOLO9000 [52], and SSD [53].
VI. T RACKING E XPERIMENTS
DIRSIG is a very useful program for generating remote
sensing images with high fidelity. This is proved in the
previous section where we generated a large synthetic dataset
representative of the WAMI platform and trained a convolutional network to classify real images from the WAMI platform. To further emphasize this, [54] performed segmentation
on real aerial multi-spectral images using a CNN, trained
on a multispectral dataset consisting of DIRSIG images. A
hyperspectral tracking video representative of the RITMOS
sensor was generated in previous studies [55], [5], [9], [23].
The hyperspectral tracking scenario has two different videos :
(1) without trees and (2) with dense trees (25% full occlusion
by trees). Both videos have 1.42 fps and 157 frames with same
vehicular traffic, and platform position. We used both videos to
evaluate the performance of the proposed DeepHKCF tracker,
its variants, and other hyperspectral state-of-the-art trackers.
A. Hyperparameter Tuning
Before reviewing the DeepHKCF’s performance, we will go
through the hyperparameters that need to be tuned to perform
optimal tracking considering both run-time performance and
accuracy. The KCF has a number of hyperparameters including
padding size, desired Gaussian response width, learning rate
and Gaussian kernel bandwith. The proposed single KCFmultiple ROIs approach, on the other hand, includes three
hyperparameters, full ROI size, overlap between ROIs, and
PSR threshold to remove the contribution of noisy response
maps. We set the size of a single ROI to (48x48) pixels since
that includes reasonable context as each vehicle occupies about
20 x 10 pixels. This removes the need to have a padding

size hyperparameter in the DeepHKCF. The other KCF hyperparameters are tuned to similar values as the original KCF
paper [23]. The overlap between ROIs is set to 64% in each
dimension (sect. VI-F) whereas the full ROI size and PSR
threshold are set to 96x96 px (sect. VI-G) and 7 respectively.
B. Results on the No-Trees Scenario
After tuning the hyperparameters of the DeepHKCF, we first
test it on the 43 vehicles in the no-trees scenario. We compare
the DeepHKCF to a number of variants of the proposed
single KCF multiple ROIs approach. They include HKCF with
fHoG features, HKCF with hyperspectral feature channels,
and HKCF with concatenated fHoG and hyperspectral feature
channels. Also, we perform experiments on the original KCF
algorithm (single KCF-single ROI approach) with the same
ROI size (48x48 px). Finally, in addition to the original KCF,
we compare the proposed tracker to Efficient Convolutional
Operator tracker (ECO) that is ranked as the first tracker in the
VOT16 tracking benchmark [3]. ECO claims that correlation
filters are overfit to the last few frames by doing model
update in every frame and addresses this problem to learn
a more general filter. Also, it employs factorized convolutional operators to increase the speed. For the sake of fair
comparison, we increase the learning rate of the ECO tracker
to match our scenarios and use the same features. Similar
to DeepHKCF and HKCF, we use the activations of the 5th
convolutional layer of the VGGNet (fig. 4) and fHoG features.
To determine the ROI area, ECO considers the padding size
of 3.5, larger than the optimal padding size (2.0) of the
KCF. We keep this hyper-parameter same as including further
background deteriorates the features. Fig. 8a and table II show
the performances of the proposed DeepHKCF, its variants and
the baseline KCF and ECO trackers.
The DeepHKCF performs exceptionally well in the notrees scenario, achieving 70% precision at 20 px threshold
and outperforming all the baseline methods by a large margin.
Meanwhile, the proposed HKCF with fHoG features performs
substantially worse than the one with deep features. However,
it outruns the original KCF with fHoG features by large margin
at 50 px precision as shown in table II, proving the contribution
of the proposed single KCF multiple ROIs approach in low
frame rate tracking. Concatenating hyperspectral features with
fHoG slightly degrades the precision whereas hyperspectral
feature channels alone performs worse than the former methods. The ECO tracker, on the other hand, delivers 10-15%
lower accuracy than the DeepHKCF trackers at 20 px precision

9

0.8

0.6
0.5
0.4

0.7
0.6
Precision

0.7

Precision

0.8

DeepHKCF - ZFNet-2
DeepHKCF - VGG16-5
FastDeepHKCF - VGG16-5
HKCF - fHoG
HKCF - fHoG + HSI
HKCF - HSI
KCF - fHoG
ECO - fHoG
ECO - VGG16-5

0.5

0.3

0.3

0.2

0.2

0.1

0.1

0

DeepHKCF - ZFNet-2
DeepHKCF - VGG16-5
FastDeepHKCF - VGG16-5
HKCF - fHoG
HKCF - fHoG + HSI
HKCF - HSI
KCF - fHoG
ECO - fHoG
ECO - VGG16-5

0.4

0
0

10

20

30

Distance Threshold

40

50

0

10

20

30

40

50

Distance Threshold

(a) Precision curve - with trees sce- (b) Precision curve - without trees
nario
scenario

Fig. 8: Precision curves of the DeepHCKF tracker and its
variants on the DIRSIG scenario with trees and without trees.
The ZFNet-2 and VGGNet-5 refer to the activations of the
2nd and 5th convolutional layers.

and about 20% worse in terms of the precision at 50 px and
central location error. All in all, the DeepHKCF tracker with
ROI mapping (FastDeepHKCF) provides the best tracking
results due to its reasonably high tracking accuracy and highest
operation rate among the DeepHKCF trackers. What is more,
the proposed tracking strategy can be coupled to an advanced
KCF tracker such as ECO to further improve tracking.
C. Results on the Scenario with Dense Trees
In addition to conducting experiments on the no-trees scenario, we run the DeepHKCF tracker on the same scenario
with dense trees. This is an extremely challenging scene
dominated by large trees and their shadows as shown in
fig. 2. On average, 1 out of 4 frames a vehicle is fully
occluded in this video. The occlusions combined with the
low frame rate makes this a very challenging scene. The
DeepHKCF trackers outperform the other baseline methods
other than ECO tracker by a large margin as in the no-trees
scenario (fig. 8a). At 20 px precision, the DeepHKCF tracker
achieves about 39% accuracy whereas others perform 10-20%
worse. On the other hand, among the DeepHKCF trackers, the
FastDeepHKCF, delivers similar precision at 50 px and higher
frame rate. Interestingly, the combination of hyperspectral
feature channels with fHoG degrades the performance with
respect to the fHoG-only features. We believe that this could
be due to more frequent switching to non-vehicle objects with
similar hyperspectral features to target the through occlusions.
By using fHoG-only features, it is less likely to switch to an
object that does not appear like a vehicle.
The dramatic drop in precision rates between the no-trees
and dense trees scenarios is easily seen in fig. 8. This is
likely due to three major reasons : (1) high frequency of
severe occlusions, (2) low video frame rate, and (3) relatively
smaller search area considered by our single KCF-multiple
ROIs approach. The combination of the first two reasons
leads to dramatically large displacement of objects in between
the frames where they are visible. This results in the targets
located out of the search area of the DeepHKCF tracker.

There are two solutions to address the challenge of tracking
through severe occlusions. The first and less practical solution,
is increasing the full ROI size in each dimension. This way, we
increase the likelihood of keeping the target in our search area
traveling through severe occlusions. However, this will also
reduce the run-time performance. A more practical solution
could be delivered by leveraging a Bayes Filter. For instance,
[5], [55] uses a Bayes Filter in a Multi-dimensional Assignment algorithm to update the measurements in light of the later
measurements in the same scenario. This way, we can low-pass
the unlikely jumps that occurs during severe occlusions. On
the other hand, we believe that, increasing the search area in a
practical manner might be the key to achieving state-of-the-art
performance in scenarios dominated by trees (see sect. VI-G).

D. Experiments on Temporally Down-sampled Video
In the previous section, we evaluated the proposed trackers
and the baseline methods on the 1.42 fps videos with dense
trees and without trees. The ECO tracker, the winner of the
VOT16 object tracking challenge, performs only slightly worse
to the DeepHKCF trackers at 20 px precision. We believe
that this might be due to slowly moving or stopped vehicles.
To further test the ECO and FastDeepHKCF’s performance
with respect to more drastic target motion, we down-sample
the video without trees temporally by two, resulting in 0.7
fps video. All the hyperparameters in the FastDeepHKCF and
ECO are kept same as before for fair comparison. As seen in
table III, the DeepHKCF trackers outperform the ECO tracker
by a large margin in terms of precision and central location
error, showing its robustness to extreme target displacement in
successive frames. The ECO, on the other hand, misses more
targets due to the smaller ROI considered in the detection
operation.
E. Comparison with Hyperspectral Trackers
Finally, we compare the DeepHKCF tracker, a trackingby-detection algorithm, to the template-matching based hyperspectral video trackers [5], [4], [55]. These trackers are the
state-of-the-art trackers for the DIRSIG scenarios. [55] (HFAT)
performs median filtering based background subtraction to
detect motion, and uses a Bayes Filter to perform predictive
hyperspectral sampling to detect stopped vehicles. They assign
hyperspectral and kinematic scores to each detected blob and
uses the multi-dimensional assignment algorithm to optimally
assign the blobs to target. The proposed idea is better suited to
fixed camera platforms due to the parallax effect introduced by
registration in the moving platforms. [5] propose a generative
tracker (HLT) consisting of detection and tracking modules.
Method
Pr. (20px)
Pr. (50px)
CLE
FPS

DeepHCKF - VGGNet-5
44.34
53.30
156.98
0.51

FastDeepHKCF - VGGNet-5
39.11
47.67
174.53
1.11

ECO - VGGNet-5
29.79
34.29
204.85
1.12

TABLE III: Performance of the DeepHKCF and ECO trackers
on the temporally down-sampled no-trees video (0.7 fps).

10

Method
Pr. (20px) - Trees
Pr. (20px) - No-trees
Pr. (50px) - Trees
Pr. (50px) - No-trees
CLE - Trees
CLE - No-trees
FPS

— Best
DeepHKCF - ZFNet-2
38.08
70.13
43.83
81.05
156.74
48.97
0.51

— 2nd Best
FastDeepHKCF - VGGNet-5
HFAT [55](6D)*
31.71
57.63
66.26
N/A
44.01
N/A
80.27
N/A
143.66
N/A
51.71
N/A
1.11
1.42

— 3th Best
HFAT [55] (2D)*
39.12
N/A
N/A
N/A
N/A
N/A
2.32

HLT [5] (5D)
51.69
64.42
55.12
71.27
135.03
65.36
1.01

HLT [5] (2D)
41.86
57.25
46.72
68.31
158.12
91.97
1.09

TABLE IV: Comparison of the DeepHKCF tracker to the other hyperspectral template-matching based trackers. The ”*” denotes
the tracker tested on the same scene from a fixed platform. The 6D, 5D and 2D refer to the number of past frames considered
in the multi-dimensional assignment algorithm [56]. The use of a multi-dimensional assignment algorithm and a non-linear
Bayes Filter is the key to achieve persistent tracking in the occlusion-dominated scenes. The experiments are carried out on a
CPU with 8GB RAM and 2.9GHz i5 processor.

As seen in table IV, the use of a Bayes Filter and the
multi-dimensional assignment algorithm (MDA) is crucial in a
scenario largely dominated by occlusions where discriminative
trackers-alone are more likely to fail. We believe that the
integration of those modules into the DeepHKCF tracker can
improve tracking in similar scenarios. One can see the effect
of reducing the length of the time window in MDA in both
HFAT and HFT tracker. Going from 6-D to 2-D window
decreases the precision by 40% for the HFAT. Interestingly,
in the fixed platform scenario, the HFAT tracker with 2D
assignment algorithm performs similarly to the DeepHKCF
trackers tested on the moving platform scenario. Similarly, the
HFT’s performance drops drastically by reducing the width
of the time window considered in MDA from 5-D to 2-D,
especially for the scenario with trees. On the other hand, the
proposed DeepHKCF trackers outperform HLT in the no-trees
scenario by about 30% in terms of central location error.
Also, the FastDeepHKCF delivers a slightly faster tracker
than the HLT and its individual ROI operations can be easily
parallelized whereas the HLT is a more serialized algorithm.

F. Effect of Overlap Ratio in Single KCF-Multiple ROIs
Approach
We experiment on the DeepHKCF tracker with ROI mapping (FastDeepHKCF) as a function of the overlap ratio between the adjacent ROIs. As mentioned before, it is necessary
to have overlap between the adjacent ROIs as the hanning
window is applied to the features before the FFT operation.
The hanning window filters the noise at the boundaries resulting from FFT operation. Increasing overlap ratio, at the same
time, leads to increased complexity (O(m ∗ nlog(n))) due to
a larger number of ROIs (m) in the full ROI (96x96 px).
Fig. 9a shows the precision rates of FastDeepHKCF tracker
with different overlap ratios between the adjacent ROIs.

0.8

0.8

0.7

0.7

0.6

0.6
FastDeepHKCF - 2x2 ROIs %0 Overlap
FastDeepHKCF - 3x3 ROIs %50 Overlap
FastDeepHKCF - 4x4 ROIs %64 Overlap
FastDeepHKCF - 5x5 ROIs %75 Overlap

0.5
0.4
0.3

Precision

Precision

The detection module uses every five neighboring channels
to get the heat map of the ROI (200x200 px), resulting in
12 heat maps (61 channels). The heat maps are fused with
adaptive coefficients to get a final smoothened heat map. In
the final stage, they extract the blobs from the final map
and update the track statistics in the past N frames using
the multi-dimensional assignment algorithm [56]. The tracker
module updates the state space matrix components of the
Gaussian Mixture Filter using the new observations in the
past N frames. Finally, [4] (HFT) relies on off-line trained
road and car classifiers to optimize the search space (200x200
px). The hyperspectral histograms are then computed in a
sliding window to assign similarity scores. To optimize the
feature extraction step, the integral image theorem [57] is
used to compute histogram for each channel in O(3). The
heat map is then thresholded and post-processed to find the
blobs. The assignment and Bayes Filter modules are similar
to [5]. The comparison of the DeepHKCF tracker with those
trackers is shown in the table IV. We want to highlight that
we exclude the HFT in table IV as it relies on vehicle and
asphalt classifiers trained on the samples from the same scene.
Such frameworks are limited to similar scenes and may deliver
optimistic results.

0.5
DeepHKCF - ZFNet-2
48x48 ROI
DeepHKCF - ZFNet-2
64x64 ROI
DeepHKCF - ZFNet-2
96x96 ROI
DeepHKCF - ZFNet-2
144x144 ROI
DeepHKCF - ZFNet-2
192x192 ROI

0.4
0.3

0.2

0.2

0.1

0.1

0

0
0

10

20

30

40

50

0

10

Distance Threshold

(a) Varying overlap ratio between (b) Varying
ROIs in FastDeepHKCF
DeepHKCF

20

30

40

50

size

in

Distance Threshold

full

ROI

Fig. 9: The FastDeepHKCF tracker is experimented on the notrees scenario with varying overlap ratio in the single KCFmultiple ROIs approach (a) whereas (b) shows the results on
the effect of different full ROI sizes in DeepHKCF. In (b), the
trackers are run through the first 25 frames of each target on
the scenario with trees.
The 64% and 75% overlap ratios lead to drastically better
results than the lower ones (fig. 9a), however, considering the
accuracy/speed trade-off, the 64% overlap (m = 16) setting
delivers the optimal tracker.
G. Effect of ROI Size
The overlap between the adjacent ROIs is an essential
part of the DeepHKCF tracker as it ensures consideration

11

of the every single point in the full ROI by the correlation
filter. Another key parameter in this direction is the full ROI
size since we have a low temporal resolution and occlusiondominated scene. We enlarge the ROI size of the optimal
DeepHCKF tracker and observe the performance in the scene
with trees. The results are shown in fig. 9b. In this experiment,
the run-time performance of the tracker is ignored as the goal
is to measure the contribution of full ROI size.
As shown in fig. 9b, the larger ROI size with the same
overlap ratio does not necessarily lead to better performance
while quadratically reducing the speed. We believe this is
mainly due to growing confusion as the larger ROIs contain a higher number of similar objects. Additionally, these
results demonstrate the obvious need to couple the trackingby-detection algorithms to a Multi-dimensional Assignment
algorithm in a Bayes Filter framework in occlusion-dominated
scenes [5], [4], [55].

VII. C ONCLUSION
Adaptive multi-modal sensors are becoming increasingly
important in the aerial tracking domain due to the unique
challenges posed by this platforms. In this study, we propose
a tracking-by-detection algorithm driven tracker inspired by a
multi-modal sensor and deep features. This approach replaces
the traditional template-matching based hyperspectral trackers
with a new state-of-the-art tracker becoming increasingly popular in traditional visual object tracking. More specifically, we
delivered a new framework to handle low temporal resolution
in aerial platforms in KCF tracker, called single KCF-multiple
ROIs approach. To further boost the tracking accuracy, we
replaced the traditional features with deep CNN features.
Finally, an ROI mapping approach was proposed to speed up
extracting features in a single KCF-multiple ROIs approach.
The proposed DeepHKCF tracker was evaluated on synthetic
scenarios generated by DIRSIG software. In the scenario with
no-trees, the DeepHKCF tracker performs exceptionally well
with 80% precision at 50 px, outperforming other templatematching based trackers. In the same scenario but dominated
by occlusions, it is outperformed by the trackers employing
a multi-dimensional assignment algorithm and Bayes Filter.
To prove the high-fidelity of the DIRSIG generated scenarios, we build a synthetic, aerial vehicle classification dataset
to perform classification on the real-platform (WAMI). Our
dataset, consisting of 55226 samples, was used to train CNNs
to perform binary classification. We achieve about 93.2% on
the WAMI samples by only training on synthetic dataset. This
dataset can be highly beneficial in aerial detection and tracking
due to limited amount of publicly available data in those
domains.
In future work, we plan on supporting the DeepHKCF
tracker by integrating a multi-dimensional assignment algorithm and Bayes Filter to better handle severe occlusions. Also,
we want to extend the proposed single KCF-multiple ROIs
tracking strategy to adopt a more advanced correlation filter
tracker such as ECO.

R EFERENCES
[1] R. Pelapur, S. Candemir, F. Bunyak, M. Poostchi, G. Seetharaman,
and K. Palaniappan, “Persistent target tracking using likelihood fusion
in wide-area and full motion video sequences,” in Information Fusion
(FUSION), 2012 15th International Conference on. IEEE, 2012, pp.
2420–2427.
[2] J. Portmann, S. Lynen, M. Chli, and R. Siegwart, “People detection and
tracking from aerial thermal views,” in Robotics and Automation (ICRA),
2014 IEEE International Conference on. IEEE, 2014, pp. 1794–1800.
[3] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “Eco: Efficient
convolution operators for tracking,” arXiv preprint arXiv:1611.09224,
2016.
[4] B. Uzkent, M. J. Hoffman, and A. Vodacek, “Real-time vehicle tracking
in aerial video using hyperspectral features,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition Workshops,
2016, pp. 36–44.
[5] B. Uzkent, A. Rangnekar, and M. J. Hoffman, “Aerial vehicle tracking by
adaptive fusion of hyperspectral likelihood maps,” in Computer Vision
and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference
on. IEEE, 2017, pp. 233–242.
[6] AFRL, “Wright-patterson air force base (wpafb) dataset,” https://www.
sdms.afrl.af.mil/index.php?collection=wpafb2009, 2009.
[7] ——, “Wami columbus large image format (clif) dataset,” https://www.
sdms.afrl.af.mil/index.php?collection=clif2007, 2007.
[8] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large
scale visual recognition challenge,” International Journal of Computer
Vision, vol. 115, no. 3, pp. 211–252, 2015.
[9] B. Uzkent, M. J. Hoffman, and A. Vodacek, “Integrating Hyperspectral
Likelihoods in a Multidimensional Assignment Algorithm for Aerial
Vehicle Tracking,” IEEE Journal of Selected Topics in Applied Earth
Observations and Remote Sensing, vol. 9, no. 9, pp. 4325–4333, 2016.
[10] F. Darema, “Dynamic data driven applications systems: A new paradigm
for application simulations and measurements,” Computational ScienceICCS 2004, pp. 662–669, 2004.
[11] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed
tracking with kernelized correlation filters,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583–596,
2015.
[12] S. Hare, S. Golodetz, A. Saffari, V. Vineet, M.-M. Cheng, S. L. Hicks,
and P. H. Torr, “Struck: Structured output tracking with kernels,” IEEE
transactions on pattern analysis and machine intelligence, vol. 38,
no. 10, pp. 2096–2109, 2016.
[13] Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,”
IEEE transactions on pattern analysis and machine intelligence, vol. 34,
no. 7, pp. 1409–1422, 2012.
[14] R. LaLonde, D. Zhang, and M. Shah, “Fully convolutional deep neural
networks for persistent multi-frame multi-object detection in wide area
aerial videos,” arXiv preprint arXiv:1704.02694, 2017.
[15] M. Yi, F. Yang, E. Blasch, C. Sheaff, K. Liu, G. Chen, and H. Ling,
“Vehicle classification in wami imagery using deep network,” in SPIE
Defense+ Security. International Society for Optics and Photonics,
2016, pp. 98 380E–98 380E.
[16] E. J. Ientilucci and S. D. Brown, “Advances in wide-area hyperspectral
image simulation,” in AeroSense 2003. International Society for Optics
and Photonics, 2003, pp. 110–121.
[17] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the
circulant structure of tracking-by-detection with kernels,” in Proceedings
on European Conference on Computer Vision, 2012, pp. 702–715.
[18] R. Collins, X. Zhou, and S. K. Teh, “An open source tracking testbed
and evaluation web site.”
[19] M. Mueller, N. Smith, and B. Ghanem, “A benchmark and simulator
for uav tracking,” in Proceedings of European Conference on Computer
Vision, 2016, pp. 445–461.
[20] L. W. Sommer, M. Teutsch, T. Schuchert, and J. Beyerer, “A survey on
moving object detection for wide area motion imagery,” in Applications
of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE,
2016, pp. 1–9.
[21] M. Teutsch and M. Grinberg, “Robust detection of moving vehicles in
wide area motion imagery,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Workshops, 2016, pp. 27–35.
[22] M. Cormier, L. W. Sommer, and M. Teutsch, “Low resolution vehicle
re-identification based on appearance features for wide area motion imagery,” in 2016 IEEE Winter Applications of Computer Vision Workshops
(WACVW). IEEE, 2016, pp. 1–7.

12

[23] B. Uzkent, M. J. Hoffman, A. Vodacek, J. P. Kerekes, and B. Chen,
“Feature matching and adaptive prediction models in an object tracking
dddas,” Procedia Computer Science, vol. 18, pp. 1939–1948, 2013.
[24] B. Uzkent, M. J. Hoffman, A. Vodacek, and B. Chen, “Feature matching
with an adaptive optical sensor in a ground target tracking system,” IEEE
Sensors Journal, vol. 15, no. 1, pp. 510–519, 2015.
[25] R. D. Meyer, K. J. Kearney, Z. Ninkov, C. T. Cotton, P. Hammond, and
B. D. Statt, “RITMOS: a micromirror-based multi-object spectrometer,”
in SPIE Astronomical Telescopes+ Instrumentation.
International
Society for Optics and Photonics, 2004, pp. 200–219.
[26] B. Uzkent, Real-time Aerial Vehicle Detection and Tracking using a
Multi-modal Optical Sensor. Rochester Institute of Technology, 2016.
[27] S. Han, A. Fafard, J. Kerekes, M. Gartley, E. Ientilucci, A. Savakis,
C. Law, J. Parhan, M. Turek, K. Fieldhouse et al., “Efficient generation
of image chips for training deep learning algorithms,” in Automatic
Target Recognition XXVII, vol. 10202. International Society for Optics
and Photonics, 2017, p. 1020203.
[28] S. Han and J. P. Kerekes, “Overview of passive optical multispectral and
hyperspectral image simulation techniques,” IEEE Journal of Selected
Topics in Applied Earth Observations and Remote Sensing, 2017.
[29] B. Uzkent, M. J. Hoffman, and A. Vodacek, “Spectral validation of
measurements in a vehicle tracking dddas,” Procedia Computer Science,
vol. 51, pp. 2493–2502, 2015.
[30] B. Uzkent, M. J. Hoffman, A. Vodacek, and B. Chen, “Background
image understanding and adaptive imaging for vehicle tracking,” in SPIE
Defense+ Security. International Society for Optics and Photonics,
2015, pp. 94 600F–94 600F.
[31] H. K. Galoogahi, T. Sim, and S. Lucey, “Multi-channel correlation
filters,” in Proceedings of International Conference on Computer Vision,
2013.
[32] M. Tang and J. Feng, “Multi-kernel correlation filter for visual tracking,”
in Proceedings of International Conference on Computer Vision, 2015,
pp. 3038–3046.
[33] C. Ma, X. Yang, C. Zhang, and M.-H. Yang, “Long-term correlation
tracking,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2015, pp. 5388–5396.
[34] A. Bibi and B. Ghanem, “Multi-template scale-adaptive kernelized
correlation filters,” in Proceedings of the IEEE International Conference
on Computer Vision Workshops, 2015, pp. 50–57.
[35] Y. Li and J. Zhu, “A scale adaptive kernel correlation filter tracker
with feature integration,” in European Conference on Computer Vision.
Springer, 2014, pp. 254–265.
[36] N. Dalal and B. Triggs, “Histograms of oriented gradients for human
detection,” in Computer Vision and Pattern Recognition, 2005. CVPR
2005. IEEE Computer Society Conference on, vol. 1. IEEE, 2005, pp.
886–893.
[37] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan,
“Object detection with discriminatively trained part-based models,”
IEEE transactions on pattern analysis and machine intelligence, vol. 32,
no. 9, pp. 1627–1645, 2010.
[38] D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual
object tracking using adaptive correlation filters,” in Computer Vision
and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE,
2010, pp. 2544–2550.
[39] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[40] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[41] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
in Proceedings of the IEEE conference on computer vision and pattern
recognition, 2015, pp. 1–9.
[42] M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg, “Convolutional features for correlation filter based visual tracking,” in Proceedings
of the IEEE International Conference on Computer Vision Workshops,
2015, pp. 58–66.
[43] L. Wang, T. Liu, G. Wang, K. L. Chan, and Q. Yang, “Video tracking using learned hierarchical features,” IEEE Transactions on Image
Processing, vol. 24, no. 4, pp. 1424–1435, 2015.
[44] D. Held, S. Thrun, and S. Savarese, “Learning to track at 100 fps with
deep regression networks,” in European Conference on Computer Vision.
Springer, 2016, pp. 749–765.
[45] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr,
“Fully-convolutional siamese networks for object tracking,” in European
Conference on Computer Vision. Springer, 2016, pp. 850–865.

[46] L. Leal-Taixé, C. Canton-Ferrer, and K. Schindler, “Learning by tracking: Siamese cnn for robust target association,” in Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition
Workshops, 2016, pp. 33–40.
[47] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international
conference on computer vision, 2015, pp. 1440–1448.
[48] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
object detection with region proposal networks,” in Advances in neural
information processing systems, 2015, pp. 91–99.
[49] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via regionbased fully convolutional networks,” in Advances in neural information
processing systems, 2016, pp. 379–387.
[50] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated
convolutions,” arXiv preprint arXiv:1511.07122, 2015.
[51] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for
fast feature embedding,” arXiv preprint arXiv:1408.5093, 2014.
[52] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” arXiv
preprint arXiv:1612.08242, 2016.
[53] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.
Berg, “Ssd: Single shot multibox detector,” in European conference on
computer vision. Springer, 2016, pp. 21–37.
[54] R. Kemker and C. Kanan, “Deep neural networks for semantic segmentation of multispectral remote sensing imagery,” arXiv preprint
arXiv:1703.06452, 2017.
[55] B. Uzkent, M. J. Hoffman, and A. Vodacek, “Efficient integration of
spectral features for vehicle tracking utilizing an adaptive sensor,” in
IS&T/SPIE Electronic Imaging, 2015, pp. 940 707–940 707.
[56] A. B. Poore, “Multidimensional assignment formulation of data association problems arising from multitarget and multisensor tracking,”
Computational Optimization and Applications, vol. 3, no. 1, pp. 27–57,
1994.
[57] F. Porikli, “Integral histogram: A fast way to extract histograms in
cartesian spaces,” in Computer Vision and Pattern Recognition, 2005.
CVPR 2005. IEEE Computer Society Conference on, vol. 1. IEEE,
2005, pp. 829–836.

