1

No Blind Spots: Full-Surround Multi-Object
Tracking for Autonomous Vehicles using
Cameras & LiDARs

arXiv:1802.08755v1 [] 23 Feb 2018

Akshay Rangesh, Member, IEEE, and Mohan M. Trivedi, Fellow, IEEE

Abstract—Online multi-object tracking (MOT) is extremely
important for high-level spatial reasoning and path planning
for autonomous and highly-automated vehicles. In this paper,
we present a modular framework for tracking multiple objects
(vehicles), capable of accepting object proposals from different
sensor modalities (vision and range) and a variable number
of sensors, to produce continuous object tracks. This work
is inspired by traditional tracking-by-detection approaches in
computer vision, with some key differences - First, we track
objects across multiple cameras and across different sensor
modalities. This is done by fusing object proposals across sensors
accurately and efficiently. Second, the objects of interest (targets)
are tracked directly in the real world. This is a departure from
traditional techniques where objects are simply tracked in the
image plane. Doing so allows the tracks to be readily used by an
autonomous agent for navigation and related tasks.
To verify the effectiveness of our approach, we test it on real
world highway data collected from a heavily sensorized testbed
capable of capturing full-surround information. We demonstrate
that our framework is well-suited to track objects through entire
maneuvers around the ego-vehicle, some of which take more than
a few minutes to complete. We also leverage the modularity of
our approach by comparing the effects of including/excluding
different sensors, changing the total number of sensors, and the
quality of object proposals on the final tracking result.
Index Terms—Multi-object tracking (MOT), panoramic surround behavior analysis, highly autonomous vehicles, computer
vision, sensor fusion, collision avoidance, path planning.

Y

X

(a)

I. I NTRODUCTION

T

RACKING for autonomous vehicles involves accurately
identifying and localizing dynamic objects in the environment surrounding the vehicle. Tracking of surround vehicles is
essential for many tasks crucial to truly autonomous driving,
such as obstacle avoidance, path planning, and intent recognition. To be useful for such high-level reasoning, the generated
tracks should be accurate, long and robust to sensor noise.
In this study, we propose a full-surround MOT framework to
create such desirable tracks.
Traditional MOT techniques for autonomous vehicles can
roughly be categorized into 3 groups based on the sensory
inputs they use - 1) dense point clouds from range sensors,
2) stereo vision, and 3) a fusion of range and vision sensors.
Studies like [1]–[4] make use of dense point clouds created
by 3D LiDARs like the Velodyne HDL-64E. Such sensors,
although bulky and expensive, are capable of capturing finer
The authors are with the Laboratory for Intelligent and Safe Automobiles,
University of California, San Diego, CA 92092, USA.
email - arangesh, mtrivedi@ucsd.edu

(b)

Fig. 1: (a) Illustration of online MOT for autonomous vehicles.
The surround vehicles (in red) are tracked in a coordinate
system centered on the ego-vehicle (center). The ego-vehicle
has full-surround coverage from vision and range sensors, and
must fuse proposals from each of them to generate continuous
tracks (dotted lines) in the real world.
(b) An example of images captured from a full-surround
camera array mounted on our testbed, along with color coded
vehicle annotations.

2

details of the surroundings owing to its high vertical resolution.
Trackers can therefore create suitable mid-level representations
like 2.5D grids, voxels etc. that retain unique statistics of
the volume they enclose, and group such units together to
form coherent objects that can be tracked. It must be noted
however, that these approaches are reliant on having dense
point representations of the scene, and would not scale well
to LiDAR sensors that have much fewer scan layers. On
the other hand, studies such as [5]–[8] make use of stereo
vision alone to perform tracking. The pipeline usually involves
estimating the disparity image and optionally creating a 3D
point cloud, followed by similar mid-level representations like
stixels, voxels etc. which are then tracked from frame to frame.
These sensors are limited by the quality of disparity estimates
and the field of view (FoV) of the stereo pair. Unlike 3D
LiDAR based systems, they are unable to track objects in
full-surround. There are other single camera approaches to
surround vehicle behavior analysis [9], [10], but they too are
limited in their FoVs and localization capabilities. Finally,
there are fusion based approaches like [11]–[13], that make
use of LiDARs, stereo pairs, monocular cameras, and Radars
in a variety of configurations. These techniques either perform
early or late fusion based on their sensor setup and algorithmic
needs. However, none of them seem to offer full-surround
solutions for vision sensors, and are ultimately limited to
fusion only in the FoV of the vision sensors.
In this study, we take a different approach to full-surround
MOT and try to overcome some of the limitations in previous
approaches. Specifically, we propose a framework to perform full-surround MOT using calibrated camera arrays, with
varying degrees of overlapping FoVs, and with an option to
include low resolution range sensors for accurate localization
of objects in 3D. We term this the M3 OT framework, which
stands for multi-perspective, multi-modal, multi-object tracking framework. To train and validate the M3 OT framework, we
make use of naturalistic driving data collected from our testbed
(illustrated in Figure 1) that has full-surround coverage from
vision and range modalities.
Since we use vision as our primary perception modality,
we leverage recent approaches from the 2D MOT community
which studies tracking of multiple objects in the 2D image
plane. Recent progress in 2D MOT has focused on the
tracking-by-detection strategy, where object detections from
a category detector are linked to form trajectories of the
targets. To perform tracking-by-detection online (i.e. in a
causal fashion), the major challenge is to correctly associate
noisy object detections in the current video frame with previously tracked objects. The basis for any data association
algorithm is a similarity function between object detections
and targets. To handle ambiguities in association, it is useful
to combine different cues in computing the similarity, and
learn an association based on these cues. Many recent 2D
MOT methods such as [14]–[18] use some form of learning
(online or offline) to accomplish data association. Similar to
these studies, we formulate the online multi-object tracking
problem using Markov Decision Processes (MDPs) proposed
in [19], where the lifetime of an object is modeled with a
MDP (see Figure 2), and multiple MDPs are assembled for

multi-object tracking. In this method, learning a similarity
function for data association is equivalent to learning a policy
for the MDP. The policy learning is approached in a reinforcement learning fashion which benefits from advantages of
both offline-learning and online-learning in data association.
The M3 OT framework is also capable to naturally handle the
birth/death and appearance/disappearance of targets by treating
them as state transitions in the MDP, and also benefits from
the strengths of online learning approaches in single object
tracking ( [20]–[23]).
Our main contributions in this work can be summarized
as follows - 1) We extend and improve the MDP formulation
originally proposed for 2D MOT, and modify it to track objects
in 3D (real world). 2) We make the M3 OT framework capable
of tracking objects across multiple vision sensors in calibrated
camera arrays by carrying out efficient and accurate fusion of
object proposals. 3) The M3 OT framework is made highly
modular, capable of working with any number of cameras,
with varying degrees of overlapping FoVs, and with the option
to include range sensors for improved localization and fusion
in 3D. 4) Finally, we carry out experiments using naturalistic
driving data collected on highways using full-surround sensory
modalities, and validate the accuracy, robustness and modularity of our framework.
II. R ELATED R ESEARCH
We highlight some representative works in 2D and 3D MOT
below. We also summarize the some key aspects of related 3D
MOT studies in Table I. For a recent survey on detection and
tracking for intelligent vehicles, we refer the reader to [25].
2D MOT: Recent research in MOT has focused on trackingby-detection, where the main challenge is data association
for linking object detections to targets. Majority of batch
(offline) methods ( [17], [26]–[31]) formulate MOT as a
global optimization problem in a graph-based representation,
while online methods solve the data association problem
either probabilistically [32]–[34] or deterministically (e.g.,
Hungarian algorithm [35] in [14], [15] or greedy association
[36]). A core component in any data association algorithm
is a similarity function between objects. Both batch methods
[16], [17] and online methods [14], [15], [18] have explored
the idea of learning to track, where the goal is to learn a
similarity function for data association from training data. In
this work, we extend and improve the MDP framework for 2D
MOT proposed in [19], which is an online method that uses
reinforcement learning to learn associations.
3D MOT for autonomous vehicles: In [5], the authors use a
stereo rig to first calculate the disparity using SGM, followed
by height based segmentation and free-space calculation to
create a mid-level representation using stixels that encode the
height within a cell. Each stixel is then represented by a 6D
state vector, which is tracked using the Extended Kalman Filter
(EKF). In [6], the authors use a voxel based representation
instead, and cluster neighboring voxels based on color to
create objects that are then tracked using a greedy association
model. The authors in [7] use a grid based representation of
the scene, where cells are grouped to create objects, each

3

TABLE I: Relevant research in 3D MOT for intelligent vehicles.
Sensors used
Study

Tracker Type

Monocular
camera

Stereo
pair

Full-surround
camera array

LiDAR

Choi et al. [1]

-

-

-

Broggi et al. [6]

-

3

Song et al. [2]

-

-

Experimental Analysis

Radar

Single object
tracker

Multi-object
tracker

Online
(causal)

Dataset

Evaluation metrics

3

-

-

3

3

Proposed

Distance and velocity errors

-

-

-

-

3

3

Proposed

True positives, false positives,
false negatives

-

3

-

3

7

3

KITTI

Position error, intersection ratio

Cho et al. [11]

3

-

-

3

3

-

3

3

Proposed

Correctly tracked, falsely tracked,
true and false positive rates

Asvadi et al. [3]
Vatavu et al. [7]

-

3

-

3
-

-

-

3
3

3
3

KITTI
Proposed

-

Asvadi et al. [4]

-

-

-

3

-

-

3

3

KITTI

Number of missed and false
obstacles

Asvadi et al. [12]

3

-

-

3

-

3

7

3

KITTI

Average center location errors in
2D and 3D, orientation errors

Ošep et al. [8]

-

3

-

-

-

-

3

3

KITTI

Class accuracy, GOP recall,
tracking recall

Allodi et al. [13]

-

3

-

3

-

-

3

3

KITTI

MOTP, IDS, Frag, average
localization error

Dueholm et al. [24]

-

-

3

-

-

-

3

7

VIVA Surround

MOTA, MOTP, IDS, Frag,
Precision, Recall

This work1
(M3 OT)

-

-

3

3

-

-

3

3

Proposed

MOTA, MOTP, MT, ML, IDS for a
variety of sensor configurations

1

this framework can work with or without LiDAR sensors, and with any subset of camera sensors.

of which is represented by a set of control points on the
object surface. This creates a high dimensional state-space
representation, which is accounted for by a Rao-Blackwellized
particle filter. More recently, the authors of [8] propose to carry
out semantic segmentation on the disparity image, which is
then used to generate generic object proposals by creating a
scale-space representation of the density, followed by multiscale clustering. The proposed clusters are then tracked using
a QPBO framework. The work in [24] use a camera setup
similar to ours, but the authors propose an offline framework
for tracking, hence limiting their use to surveillance related
applications.
Alternatively, there are approaches that make use of dense
point clouds generated by LiDARs as opposed to creating
point clouds from a disparity image. In [1], the authors first
carry out ground classification based on the variance in the
radius of each scan layer, followed by a 2.5D occupancy grid
representation of the scene. The grid is then segmented, and
regions of interest (RoIs) identified within, each of which is
tracked by a standard Kalman Filter (KF). Data association
is achieved by simple global nearest neighbor. Similar to
this, the authors in [3] use a 2.5D occupancy grid based
representation, but augment this with an occupancy grid map
which accumulates past grids to create a coherent global map
of occupancy by accounting for ego-motion. Using these two
representations, a 2.5D motion grid is created by comparing
the map with the latest occupancy grid, which isolates and
identifies dynamic objects in the scene. Although the work in
[4] follows the same general idea, the authors propose a piecewise ground plane estimation scheme capable of handling nonplanar surfaces. In a departure from grid based methods, the
authors in [2] project the 3D point cloud onto a virtual image
plane, creating an object appearance model based on 4 image-

based cues for each template of the desired target. A particle
filtering framework is implemented, where the particle with
least reconstruction error with respect to the stored template
is chosen to update the tracker. Background filtering and
occlusion detection are implemented to improve performance.
Finally, we list recent methods that rely on fusion of
different sensor modalities to function. In [11], the authors
propose an EKF based fusion scheme, where measurements
from each modality are fed sequentially. Vision is used to
classify the object category, which is then used to choose
appropriate motion and observation models for the object.
Once again, observations are associated based on a global
nearest neighbor policy. This is somewhat similar to the work
in [12], where given an initial 3D detection box, the authors
propose to project 3D points within the box to the image
plane and calculate the convex hull of projected points. In this
case, a KF is used to perform both fusion and tracking, where
fusion is carried out by projecting the 2D hull to a sparse
3D cloud, and using both 3D cues to perform the update.
In contrast, the authors of [13] propose using the Hungarian
algorithm (for bipartite matching) for both data association and
fusion of object proposals from different sensors. The scores
for association are obtained from Adaboost classifiers trained
on high-level features. The objects are then tracked using an
Unscented Kalman Filter (UKF).
III. F USION OF O BJECT P ROPOSALS
In this study, we make use of full-surround camera arrays comprising of sensors with varying FoVs. The M3 OT
framework, however, is capable of working with any type and
number of cameras, as long as they are calibrated. In addition
to this, we also propose a variant of the framework for cases
where LiDAR point clouds are available. To effectively utilize

4

a3

fused
proposals
a1

Object Proposal

(u, v)

w

a4

Tracked

h

a6
Active

Lost

P

a5
a2

Inactive

a7

(a)
Flattened LiDAR
Point Cloud

Object Proposal

l1

Fig. 2: The Markov Decision Process (MDP) framework
proposed in [19]. In this work, we retain the structure of the
MDP, and modify the actions, rewards and inputs to enable
multi-sensory tracking.

all available sensors, we propose an early fusion of object
proposals obtained from each of them. At the very start of each
time step during tracking, we identify and fuse all proposals
belonging to the same object. These proposals are then utilized
by the M3 OT framework to carry out tracking.
a) Projection & Back-projection: It is essential to have
a way of associating measurements from different sensors to
track objects across different camera views, and to carry out
efficient fusion across sensor modalities. This is achieved by
defining a set of projection mappings, one from each sensor’s
unique coordinate system to the global coordinate system, and
a set of back-projection mappings that take measurements in
the global coordinate system to individual coordinate systems.
In our case, the global coordinate system is centered at the
mid-point of the rear axle of the ego-vehicle. The axes are
oriented as shown in Figure 1.
The LiDAR sensors output a 3D point cloud in a common
coordinate system at every instant. This coordinate frame may
either be centered about a single LiDAR sensor, or elsewhere
depending on the configuration. In this case, the projection and
back-projection mappings are simple 3D coordinate transformations:
Prange→G (xrange ) = Rrange · xrange + trange l,

(1)

and,
Prange←G (xG ) = RTrange · xG − RTrange trange ,

(2)

where Prange→G (·) and Prange←G (·) are the projection and
back-projection mappings from the LiDAR (range) coordinate
system to the global (G) coordinate system and vice-versa.
The vectors xrange and xG are the corresponding coordinates
in the LiDAR and global coordinate frames. The 3 × 3

l2
(b)

Fig. 3: Projection of object proposals using: (a) IPM: The
bottom center of the bounding boxes are projected into the
global coordinate frame (right), (b) LiDAR point clouds:
LiDAR points that fall within a detection window are flattened
and lines are fit to identify the vehicle center (right).
orthonormal rotation matrix Rrange and translation vector
trange are obtained through calibration.
Similarly, the back-projection mappings for each camera
k ∈ {1, · · · , K} can be defined as:
Pcamk ←G (xG ) = (u, v)T ,
 
u
s.t v  = C k · [Rk |tk ] · [(xG )T |1]T ,
1

(3)
(4)

where the set of camera calibration matrices {C k }K
k=1 are
obtained after the intrinsic calibration of cameras, and the set
of tuples {(Rk , tk )}K
k=1 obtained after extrinsic calibration.
Unlike the back-projection mappings, the projection mappings for camera sensors are not well defined. In fact, the mappings are one-to-many due to the depth ambiguity of single
camera images. To find a good estimate of the projection, we
use two different approaches. In case of a vision-only system,
we use the inverse perspective mapping (IPM) approach:
Pcamk →G (xk ) = (x, y)T ,
 
x
s.t. y  = H k · [(xk )T |1]T ,
1

(5)
(6)

where {H k }K
k=1 are the set of homographies obtained after
IPM calibration. Since we are only concerned with lateral and

5

longitudinal displacements of vehicles in the global coordinate
system, we only require the (x, y)T coordinates, and set the
altitude coordinate to a fixed number.
b) Sensor Measurements & Object Proposals: As we
adopt a tracking-by-detection approach, each sensor is used
to produce object proposals to track and associate. In case of
vision sensors, a vehicle detector is run on each individual
camera’s image to obtain multiple detections d, each of which
is defined by a bounding box in the corresponding image. Let
(u, v) denote the top left corner and (w, h) denote the width
and height of a detection d respectively.
In case of a vision-only system, the corresponding location
of d in the global coordinate system is obtained using the
mapping Pcamk →G ((u + w2 , v + h)T ), where k denotes the
camera from which the proposal was generated. This procedure is illustrated in Figure 3a.
In cases where LiDAR sensors are available, an alternative
is considered (shown in Figure 3b). First, the back-projected
LiDAR points that fall within a detection box d are identified
using a look-up table with pixel coordinates as the key, and
the corresponding global coordinates as the value. These points
are then flattened by ignoring the altitude component of their
global coordinates. Next, a line l1 is fitted to these points
using RANSAC with a small inlier ratio (0.3). This line aligns
with the dominant side of the detected vehicle. The other
side of the vehicle corresponding to line l2 is then identified
by removing all inliers from the previous step and repeating
a RANSAC line fit with a slightly higher inlier ratio (0.4).
Finally, the intersection of lines l1 and l2 along with the
vehicle dimensions yield the global coordinates of the vehicle
center.
Depending on the type of LiDAR sensors used, object
proposals along with their dimensions in the real world can
be obtained. However, we decide not to make use of LiDAR
proposals, but rather use vision-only proposals with high recall
by trading off some of the precision. This was seen to provide
sufficient proposals to track all surrounding vehicles, at the
expense of more false positives which the tracker is capable
of handling.
c) Early Fusion of Proposals: Since we operate with
camera arrays with overlapping FoVs, the same vehicle may
be detected in two adjacent views. It is important to identify
and fuse such proposals to track objects across camera views.
Once again, we propose two different approaches to carry out
this fusion. For vision-only systems, the fusion of proposals is
carried out in 4 steps: i) Project proposals from all cameras to
the global coordinate system using proposed mappings, ii) Sort
all proposals in descending order based on their confidence
scores (obtained from the vehicle detector), iii) Starting with
the highest scoring proposal, find the subset of proposals
whose euclidean distance in the global coordinate system falls
within a predefined threshold. These proposals are considered
to belong to the same object and removed from the original
set of proposals. iv) The projection of each proposal within
this subset is set to the mean of projections of all proposals
within the subset. This process is repeated for the remaining
proposals until no proposals remain.
Alternatively, for a system consisting of LiDAR sensors,

Fig. 4: Fusion of object proposals using LiDAR point clouds:
Points common to both detections are drawn in green, and the
rest are drawn red.
we project the 3D point cloud onto each individual camera
image. Next, for each pair of proposals, we make a decision
as to whether or not they belong to the same object. This is
done by considering the back-projected LiDAR points that fall
within the bounding box of each proposal (see Figure 4). Let
P1 and P2 denote the index set of LiDAR points falling within
each bounding box. Then, two proposals are said to belong to
the same object if:


P1 ∩ P2 P1 ∩ P2
,
≥ 0.8,
(7)
max
|P1 |
|P2 |
where |P1 | and |P2 | denote the cardinalities of sets P1
and P2 respectively. It should be noted that after fusion is
completed, the union of LiDAR point sets that are backprojected into fused proposals can be used to obtain better
projections.
IV. M3 OT F RAMEWORK
Once we have a set of fused object proposals, we feed
them into the MDP as illustrated in Figure 2. This forms a
crucial component of the M3 OT framework, details of which
are presented in this section.
A. Markov Decision Process
As detailed in [19], we model the lifetime of a target with
a Markov Decision Process (MDP). The MDP consists of the
tuple (S, A, T (·, ·), R(·, ·)), where:
• States s ∈ S encode the status of the target.
• Actions a ∈ A define the actions that can be taken.
• The state transition function T : S ×A 7→ S dictates how
the target transitions from one state to another, given an
action.
• The real-valued reward function R : S × A 7→ R assigns
the immediate reward received after executing action a
in state s.
In this study, we retain the states S, actions A and the state
transition function T (·, ·) from the MDP framework for 2D
MOT [19], while changing only the reward function R(·, ·).
States: The state space is partitioned into four subspaces,
i.e., S = SActive ∪ ST racked ∪ SLost ∪ SInactive , where each
subspace contains infinite number of states which encode the
information of the target depending on the feature representation, such as appearance, location, size and history of the
target. Figure 2 illustrates the transitions between the four
subspaces. Active is the initial state for any target. Whenever

6

an object is detected by the object detector, it enters an
Active state. An active target can transition to Tracked or
Inactive. Ideally, a true positive from the object detector
should transition to a Tracked state, while a false alarm should
enter an Inactive state. A tracked target can stay tracked, or
transition to a Lost state if the target is not visible due to some
reason, e.g. occlusion, or disappearance from sensor range.
Likewise, a lost target can stay Lost, or go back to a Tracked
state if it appears again, or transition to an Inactive state if it
has been lost for a sufficiently long time. Finally, Inactive is
the terminal state for any target, i.e., an inactive target stays
inactive forever.
Actions and Transition Function: Seven possible transitions are designed between the state subspaces, which correspond to seven actions in our target MDP. Figure 2 illustrates
these transitions and actions. In the MDP, all the actions are
deterministic, i.e., given the current state and an action, we
specify a new state for the target. For example, executing
action a6 on a Lost target would transfer the target into a
Tracked state, i.e., T (sLost , a6 ) = sT racked .
Reward Function: As in the original study [19], we
learn the reward function from training data, i.e., an inverse
reinforcement learning problem, where we use ground truth
trajectories of the targets as supervision.
B. Policy
In the context of our MDP, a policy π is a mapping from
the state space S to the action space A, i.e., π : S 7→ A.
Given the current state of the target, a policy determines which
action to take. Ideally, the policy is chosen to maximize the
total reward obtained. In this subsection, we list the policies
chosen for the Active and Tracked subspaces, after which we
describe the reinforcement learning algorithm used to generate
a good policy for data association in the Lost subspace.
a) Policy in Active States: In an Active state s, the MDP
makes the decision between transferring an object proposal
into a Tracked or Inactive state based on whether the detection
is true or noisy. To do this, we train a set of binary Support
Vector Machines (SVM) offline, one for each camera view, to
classify a detection belonging to that view into Tracked or Inactive states using a normalized 5D feature vector φActive (s),
i.e., 2D image plane coordinates, width, height and score of the
detection, where training examples are collected from training
video sequences.
This is equivalent to learning the reward function:
k
RActive (s, a) = y(a)((wActive
)T · φActive (s) + bkActive ), (8)

for an object proposal belonging to camera k ∈ {1, · · · , K}.
k
(wActive
, bkActive ) defines the learned weights and bias of
the SVM for camera k, y(a) = +1 if action a = a1 , and
y(a) = −1 if a = a2 (see Figure 2). Training a separate SVM
for each camera view allows weights to be learned based on
object dimensions and locations in that particular view, and
thus works better than training a single SVM for all views.
Since a single object can result in multiple proposals, we
initialize a tracker for that object if any of the fused proposals

result in a positive reward. Note that a false positive from the
object detector can still be misclassified and transferred to a
Tracked state, which we then leave to be handled by the MDP
in Tracked and Lost states.
b) Policy in Tracked States: In a Tracked state, the MDP
needs to decide whether to keep tracking the target or to
transfer it to a Lost state. As long as the target is visible, we
should keep tracking it. Else, it should be marked ”lost”. We
build an appearance model for the target online and use it to
track the target. If the appearance model is able to successfully
track the target in the next video frame, the MDP leaves the
target in a Tracked state. Otherwise, the target is transferred
to a Lost state.
Template Representation: The appearance of the target is
simply represented by a template that is an image patch of
the target in a video frame. Whenever an object detection is
transferred to a Tracked state, we initialize the target template
with the detection bounding box. If the target is initialized
with multiple fused proposals, then each detection is stored as
a template. We make note of detections obtained from different
camera views, and use these to model the appearance of the
target in that view. This is crucial to track objects across
camera views under varying perspective changes. When the
target is being tracked, the MDP collects its templates in the
tracked frames to represent the history of the target, which
will be used in the Lost state for decision making.
Template Tracking: Tracking of templates is carried out
by performing dense optical flow as described in [19]. The
stability of the tracking is measured using the median of the FB
n
errors of all sampled points: emedF B = median(e(ui )i=1 ),
where n is the number of points. If emedF B is larger than
some threshold, the tracking is considered to be unstable.
Moreover, after filtering out unstable matches whose FB error
is larger than the threshold, a new bounding box of the target
is predicted using the remaining matches by measuring scale
change between points before and after. This process is carried
out for all camera views in which a target template has been
initialized and tracking is in progress.
Similar to the original MDP framework, we use the optical
flow information in addition to the object proposals history
to prevent drifting of the tracker. To do this, we compute the
bounding box overlap between the target box for l past frames,
and the corresponding detections in each of those frames. Then
we compute the mean bounding box overlap for the past L
tracked frames omean as another cue to make the decision.
Once again, this process is repeated for each camera view the
target is being tracked in. In addition to the above features, we
also gate the target track. This involves introducing a check
to see if the current global position of the tracked target falls
within a window (gate) of it’s last known global position.
This forbids the target track from latching onto objects that
appear close on the image plane, yet are much farther away in
the global coordinate frame. We denote the last know global
position and the currently tracked global position of the target
as xG (t − 1) and x̂G (t) respectively.
Finally, we define the reward function in a
Tracked state s using the feature set φT racked (s) =

7

0

0

0

0

k
K
G
G
({ekmedF B }K
k0 =1 , {omean }k0 =1 , x (t − 1), x̂ (t)) as:

y(a),
if ∃k 0 ∈ {1, · · · , K 0 } s.t.



0
0

(ekmedF B < e0 ) ∧ (okmean > o0 )
RT racked (s, a) =

∧(|xG (t − 1) − x̂G (t)| ≤ tgate ),



−y(a), otherwise,
(9)
where e0 and o0 are fixed thresholds, y(a) = +1 if action
a = a3 , and y(a) = −1 if a = a4 (see Figure 2). k 0 above
indexes camera views in which the target is currently being
tracked and tgate denotes the gating threshold. So the MDP
keeps the target in a Tracked state if emedF B is smaller and
omean is larger than their respective thresholds for any one of
K 0 camera views in addition to satisfying the gating check.
Otherwise, the target is transferred to a Lost state.
Template Updating: The appearance model of the target
needs to be regularly updated in order to accommodate appearance changes. As in the original work, we adopt a ”lazy”
updating rule and resort to the object detector in preventing
tracking drift. This is done so that we don’t accumulate
tracking errors, but rather rely on data association to handle
appearance changes and continue tracking. In addition to this,
templates are initialized in views where the target is yet to
be tracked by using proposals that are fused with detections
corresponding to the tracked location in an adjacent camera
view. This helps track objects that move across adjacent
camera views, by creating target templates in the new view
as soon as they are made available.
c) Policy in Lost States: In a Lost state, the MDP needs
to decide whether to keep the target in a Lost state, or transition
it to a Tracked state, or mark it as Inactive. We simply mark
a lost target as Inactive and terminate the tracking if the
target has been lost for more than LLost frames. The more
challenging task is to make the decision between tracking
the target and keeping it as lost. This is treated as a data
association problem where, in order to transfer a lost target
into a Tracked state, the target needs to be associated with an
object proposal, else, the target retains its Lost state.
Data Association: Let t denote a lost target, and d be an
object detection. The goal of data association is to predict the
label y ∈ {+1, −1} of the pair (t, d) indicating that the target
is linked (y = +1) or not linked (y = −1) to the detection.
Assuming that the detection d belongs to camera view k,
this binary classification is performed using the real-valued
k
linear function f k (t, d) = (wLost
)T ·φLost (t, d)+bkLost , where
k
k
(wLost , bLost ) are the parameters that control the function (for
camera view k), and φLost (t, d) is the feature vector which
captures the similarity between the target and the detection.
The decision rule is given by y = +1 if f k (t, d) ≥ 0, else
y = −1. Consequently, the reward function for data association in a lost state s given the feature set {φLost (t, dj )}M
m=1
is defined as



M
km T
km
RLost (s, a) = y(a) max ((wLost ) · φLost (t, dm ) + bLost ) ,
m=1

(10)
where y(a) = +1 if action a = a6 , y(a) = −1 if a = a5
(see Figure 2), and m indexes M potential detections for

input : Set of multi-video sequences V = {(vi1 , · · · , viK )}N
i=1 ,
Ni
ground truth trajectories Ti = {tij }j=1
, object proposals
Mi
Di = {dim }m=1
and their corresponding projections for
each multi-video sequence (vi1 , · · · , viK )
k
output: Binary classifiers {(wLost
, bkLost )}K
k=1 for data
association
1
2
3
4
5
6

7
8
9
10
11
12
13

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

repeat
foreach multi-video sequence (vi1 , · · · , viK ) in V do
foreach target tij do
Initialize the MDP in an Active state;
l ← index of the first frame in which tij is correctly
detected;
Transfer the MDP to a Tracked state and initialize
the target template for each camera view in which
target is observed;
while l ≤ index of last frame of tij do
Fuse object proposals as described in III;
Follow the current policy and choose an action
a;
Compute the action agt according to the
ground truth;
if current state is lost and a 6= agt then
foreach camera view k in which the target
has been seen do
k
Decide the label ym
of the pair
k
(tkij , dkimk );
S
Sk ← Sk
{(φ(tkij , dkimk ), ymk )};
k
(wLost
, bkLost ) ← solution of Eq.11
on S k ;
end
break;
else
Execute action a;
l ← l + 1;
end
if l > index of last frame of tij then
Mark target tij as successfully tracked;
end
end
end
end
until all targets are successfully tracked;

Algorithm 1: Reinforcement learning of the binary
classifier for data association.

association. Potential detections for association with a target
are simply obtained by applying a gating function around
the last known location of the target in the global coordinate
system. Note that based on which camera view each detection
km
m
dm originates from, the appropriate weights (wLost
, bkLost
)
associated with that view are used. As a result, the task of
policy learning in the Lost state reduces to learning the set
k
of parameters {(wLost
, bkLost )}K
k=1 for the decision functions
k
K
{f (t, d)}k=1 .
Reinforcement Learning: We train the binary classifiers
described above using the reinforcement learning paradigm.
Let V = {(vi1 , · · · , viK )}N
i=1 denote a set of multi-video
sequences for training, where N is the number of sequences
and K is the total number of camera views. Suppose there
th
i
are Ni ground truth targets Ti = {tij }N
multij=1 in the i
1
K
video sequence (vi , · · · , vi ). Our goal is to train the MDP
to successfully track all these targets across all camera views
they appear in. We start training with initial weights (w0k , bk0 )
and an empty training set S0k = ∅ for the binary classifier
corresponding to each camera view k. Note that when the

8

TABLE II: Features used for data association [19]. We introduce two new features (highlighted in bold) based on the
global coordinate positions of targets and detections.
Type

Notation

FB error

φ1 , · · · , φ 5

φ6

Feature Description
Mean of the median forward-backward errors from the entire, left half, right half,
upper half and lower half of the templates
obtained from optical flow
Mean of the median Normalized Correlation
Coefficients (NCC) between image patches
around the matched points in optical flow

input : A multi-video sequence (v 1 , · · · , v K ), corresponding object
proposals D = {dm }M
m=1 and their projections, learned binary
k
classifier weights {(wActive
, bkActive )}K
k=1 and
k
{(wLost
, bkLost )}K
k=1
output: Trajectories of targets T = {tj }N
j=1 in the sequence
1
2
3
4
5
6
7

NCC
φ7

φ8
Height ratio
φ9
Overlap

φ10

Score

φ11
φ12

Distance

φ13
φ14

Mean of the median Normalized Correlation
Coefficients (NCC) between image patches
around the matched points obtained from
optical flow
Mean of ratios of the bounding box height
of the detection to that of the predicted
bounding boxes obtained from optical flow
Ratio of the bounding box height of the
target to that of the detection
Mean of the bounding box overlaps between
the detection and the predicted bounding
boxes from optical flow
Normalized detection score
Euclidean distance between the centers of
the target and the detection after motion
prediction of the target with a linear velocity
model
Lateral offset between last known global
coordinate position of the target and that
of the detection
Longitudinal offset between last known
global coordinate position of the target
and that of the detection

8
9
10

11
12
13
14
15
16
17
18
19
20
21
22

23
24
25
26
27
28

foreach frame l in (v 1 , · · · , v K ) do
Fuse object proposals as described in III;
/* process targets in tracked states
*/
foreach tracked target tj in T do
Follow the policy, move the MDP of tj to the next state;
end
/* process targets in lost states
*/
foreach lost target tj in T do
foreach proposal dm not covered by any tracked target do
Compute
km
f km (tj , dm ) = (wkm )T
Lost · φ(tj , dm ) + bLost ;
end
end
Data association with Hungarian algorithm for the lost targets;
Initialize target templates for uninitialized camera views using
matched (fused) proposals;
foreach lost target tj in T do
Follow the assignment, move the MDP of tj to the next state;
end
/* initialize new targets
*/
foreach proposal dm not covered by any tracked target in T do
Initialize a MDP for a new target t with proposal dm ;
if action a1 is taken following the policy then
Transfer t to the tracked state and initialize the target
template for each camera view in which target is
observed;
S
T ←T
{t};
else
Transfer t to the Inactive state;
end
end
end

Algorithm 2: Multi-object tracking with MDPs.

weights of the binary classifiers are specified, we have a
complete policy for the MDP to follow. So the training
algorithm loops over all the multi-video sequences and all
the targets, follows the current policy of the MDP to track
the targets. The binary classifier or the policy is updated only
when the MDP makes a mistake in data association. In this
case, the MDP takes a different action than what is indicated
by the ground truth trajectory. Suppose the MDP is tracking
the j th target tij in the video vik , and on the lth frame
of the video, the MDP is in a lost state. Consider the two
types of mistakes that can happen: i) The MDP associates
the target tkij (l) to an object detection dkm which disagrees
with the ground truth, i.e., the target is incorrectly associated
to a detection. Then φ(tkij (l), dkm ) is added to the training
set S k of the binary classifier for camera k as a negative
example. ii) The MDP decides to not associate the target to
any detection, but the target is visible and correctly detected
by a detection dkm based on the ground truth, i.e., the MDP
missed the correct association. Then φ(tkij (l), dkm ) is added
to the training set as a positive example. After the training
set has been augmented, we update the binary classifier by
re-training it on the new training set. Specifically, given the
k
current training set S k = {(φ(tkm , dkm ), ym
)}M
m=1 , we solve
the following soft-margin optimization problem to obtain a
max-margin classifier for data association in camera view k:

M
X
1
k
||wLost
||2 + C
ξm
k
2
wLost
,bk
Lost ,ξ
m=1

k
k
s.t. ym
(wLost
)T ·φ(tkm , dkm )+bkLost ≥ 1−ξm , ξm ≥ 0, ∀m,
(11)

min

where ξm , m = 1, · · · , M are the slack variables, and C
is a regularization parameter. Once the classifier has been
updated, we obtain a new policy which is used in the next
iteration of the training process. Note that based on which
view the data association is carried out in, the weights of
the classifier in that view are updated in each iteration. We
keep iterating and updating the policy until all the targets
are successfully tracked. Algorithm 1 summarizes the policy
learning algorithm.
Feature Representation: We retain the same feature representation described in [19], but add two features based on
the lateral and longitudinal displacements of the last known
target location and the object proposal location in the global
coordinate system. This leverages 3D information that is
otherwise unavailable in 2D MOT. Table II summarizes our
feature representation.

9

TABLE III: Quantitative results showing ablative analysis of our proposed tracker.
Criteria for Comparison

Tracker Variant

Sensor Configuration

MOT Metrics [38], [39]

# of Cameras

Range Sensors

MOTA (↑)

MOTP (↓)

MT (↑)

ML (↓)

IDS (↓)

-

2
3
4
4†
6
8

3
3
3
3
3
3

73.38
77.26
72.81
74.18
79.06
75.10

0.03
0.03
0.05
0.05
0.04
0.04

71.36%
77.34%
72.48%
74.10%
79.66%
70.37%

16.13%
14.49%
20.76%
18.18%
11.93%
14.07%

16
38
49
45
51
59

Projection Scheme
(Section V-B)

Point cloud based projection
IPM projection

8
8

3
3(for fusion)

75.10
47.45

0.04
0.39

70.37%
53.70%

14.07%
19.26%

59
152

Fusion Scheme
(Section V-C)

Point cloud based fusion
Distance based fusion

8
8

3
3(for projection)

75.10
72.20

0.04
0.04

70.37%
68.23%

14.07%
12.23%

59
65

Sensor Modality
(Sections V-B,V-C)

Cameras+LiDAR
Cameras

8
8

3
7

75.10
40.98

0.04
0.40

70.37%
50.00%

14.07%
27.40%

59
171

Vehicle Detector
(Section V-D)

RefineNet [40]
RetinaNet [41]
SubCat [42]

8
8
8

3
3
3

75.10
73.89
69.93

0.04
0.05
0.04

70.37%
68.37%
66.67%

14.07%
17.07%
22.22%

59
72
81

Global Position
based Features
(Section V-E)

with {φ13 , φ14 }
without {φ13 , φ14 }

8
8

3
3

75.10
71.32

0.04
0.05

70.37%
64.81%

14.07%
17.78%

59
88

Number of
Cameras Used
(Section V-A)

C. Multi-Object Tracking with MDPs
After learning the policy/reward of the MDP, we apply it to
the multi-object tracking problem. We dedicate one MDP for
each target, and the MDP follows the learned policy to track
the object. Given a new input video frame, targets in tracked
states are processed first to determine whether they should stay
as tracked or transfer to lost states. Then we compute pairwise
similarity between lost targets and object detections which
are not covered by the tracked targets, where non-maximum
suppression based on bounding box overlap is employed
to suppress covered detections, and the similarity score is
computed by the binary classifier for data association. After
that, the similarity scores are used in the Hungarian algorithm
[35] to obtain the assignment between detections and lost
targets. According to the assignment, lost targets which are
linked to some object detections are transferred to tracked
states. Otherwise, they stay as lost. Finally, we initialize a
MDP for each object detection which is not covered by any
tracked target. Algorithm 2 describes our 3D MOT algorithm
using MDPs in detail. Note that, tracked targets have higher
priority than lost targets in tracking, and detections covered
by tracked targets are suppressed to reduce ambiguities in data
association.
V. E XPERIMENTAL A NALYSIS
Testbed: Since we propose full-surround MOT using vision
sensors, we use a testbed comprising of 8 outside looking RGB
cameras (seen in Figure 1). This setup ensures full surround
coverage of the scene around the vehicle, while retaining
a sufficient overlap between adjacent camera views. Frames
captured from these cameras along with annotated surround
vehicles are shown in Figure 1. In addition to full vision
coverage, the testbed has full-surround Radar and LiDAR
FoVs. Despite the final goal of this study being full-surround
MOT, we additionally consider cases where only a subset of

the vision sensors are used to illustrate the modularity of the
approach. More details on the sensors, their synchronization
and calibration, and the testbed can be found in [43].
Dataset: To train and test our 3D MOT system, we collect
a set of four sequences, each 3-4 minutes long, comprising of
multi-camera videos and LiDAR point clouds using our testbed
described above. The sequences are chosen much longer than
traditional MOT sequences so that long range maneuvers of
surround vehicles can be tracked. This is very crucial to
autonomous driving. We also annotate all vehicles in the 8
camera videos for each sequence with their bounding box, as
well as track IDs. It should be noted that each unique vehicle
in the scene is assigned the same ID in all camera views. With
these sequences set up, we use one sequence for training our
tracker, and reserve the rest for testing. All our results are
reported on the entire test set.
Evaluation Metrics: We use multiple metrics to evaluate
the multiple object tracking performance as suggested by the
MOT Benchmark [38]. Specifically, we use the 3D MOT
metrics described in [39]. These include Multiple Object
Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), Mostly Track targets (MT, percentage of ground
truth objects who trajectories are covered by the tracking
output for at least 80%), Mostly Lost targets (ML, percentage
of ground truth objects who trajectories are covered by the
tracking output less than 20%), and the total number of ID
Switches (IDS). In addition to listing the metrics in Table III,
we also draw arrows next to each of them indicating if a high
(↑) or low (↓) value is desirable. Finally, we provide top-down
visualizations of the tracking results in a global coordinate
system centered on the ego-vehicle for qualitative evaluation.
A. Experimenting with Number of Cameras
As our approach to tracking is designed to be extremely
modular, we test our tracker with different camera configurations. We experiment with 2, 3, 4, 6 and 8 cameras

10

(a) Ground truth

(b) 8 cameras

(c) 6 cameras

(d) 4 cameras

(e) 4† cameras

(f) 3 cameras

(g) 2 cameras

Fig. 5: Tracking results with different number of cameras. The camera configuration used is depicted above each result.

respectively. Top-down visualizations of the generated tracks
for a test sequence are depicted in Figure 5. The ground truth
tracks are provided for visual comparison. As can be seen,
the tracker provides consistent results in its FoV irrespective
of the camera configuration used, even if the cameras have no
overlap between them.
The quantitative results on the test set for each camera
configuration are listed in Table III. It must be noted that
the tracker for each configuration is scored only based on the
ground truth tracks visible in that camera configuration. The
tracker is seen to score very well on each metric, irrespective
of the number of cameras used. This illustrates the robustness
of the M3 OT framework. More importantly, it is seen that
our tracker performs exceptionally well in the MT and ML
metrics, especially in camera configurations with overalapping
FoVs. Even though our test sequences are about 3 minutes long
in duration, the tracker mostly tracks more than 70% of the
targets, while mostly losing only a few. This demonstrates that
our M3 OT framework is capable of long-term target tracking.
B. Effect of Projection Scheme
Figure 6 depicts the tracking results for a test sequence
using the two projection schemes proposed. It is obvious that
LiDAR based projection results in much better localization in
3D, which leads to more stable tracks and fewer fragments.
The IPM based projection scheme is very sensitive to small
changes in the input domain, and this leads to considerable
errors during gating and data association. This phenomenon
is verified by the high MOTP value obtained for IPM based
projection as listed in Table III.
C. Effect of Fusion Scheme
Once again, we see that the LiDAR point cloud based fusion
scheme is more reliable in comparison to the distance based
approach, albeit this difference is much less noticeable when
proposals are projected using LiDAR point clouds. The LiDAR
based fusion scheme results in objects being tracked longer
(across camera views), and more accurately. The distance
based fusion approach on the other hand fails to associate

(a) Ground truth

(b) LiDAR
based projection

(c) IPM based
projection

Fig. 6: Tracking results on a test sequence with different
projection schemes.
certain proposals, which results in templates not being stored
for new camera views, thereby cutting short the track as soon
as the target exits the current view. This superiority is reflected
in the quantitative results shown in Table III. The drawbacks of
the distance based fusion scheme are exacerbated when using
IPM to project proposals, reflected by the large drop in MOTA
for a purely vision based system. This drop in performance is
to be expected in the absence of LiDAR sensors. However, it
must be noted that half the targets are still tracked for most
of their lifetime, while only a quarter of the targets are mostly
lost.
D. Effect of using Different Vehicle Detectors
Ideally, a tracking-by-detection approach should be detector
agnostic. To observe how the tracking results change for
different vehicle detectors, we ran the proposed tracker on
vehicle detections obtained from three commonly used object
detectors [40]–[42]. All three detectors were trained on the
KITTI dataset [44] and have not seen examples from the
proposed multi-camera dataset. The ROC curves for the detectors on the proposed dataset are shown in Figure 7. The
corresponding tracking results for each detector are listed in
Table III. Despite the sub-optimal performance of all three

11

our colleagues Kevan Yuen, Nachiket Deo, Ishan Gupta and
Borhan Vasli at the Laboratory for Intelligent and Safe Automobiles (LISA), UC San Diego for their useful inputs and
help in collecting and annotating the dataset.
R EFERENCES

Fig. 7: ROC curves for different vehicle detectors on the 4 test
sequences.
detectors in addition to significant differences in their ROC
curves, the tracking results are seen to be relatively unaffected.
This indicates that the tracker is less sensitive to errors made
by the detector, and consistently manages to correct for it.
E. Effect of Global Position based Features
Table III indicates a clear benefit in incorporating features
{φ13 , φ14 } for data association in Lost states. These features
express how near/far a proposal is from the last know location
of a target. This helps the tracker disregard proposals that
are unreasonably far away from the latest target location.
Introduction of these features leads to an improvement in all
metrics and therefore justifies their inclusion.
VI. C ONCLUDING R EMARKS
In this work, we have described a full-surround camera
and LiDAR based approach to multi-object tracking for autonomous vehicles. To do so, we extend a 2D MOT approach
based on the tracking-by-detection framework, and make it
capable of tracking objects in the real world. The proposed
M3 OT framework is also made highly modular so that it
is capable of working with any camera configuration with
varying FoVs, and also with or without LiDAR sensors. An
efficient and fast early fusion scheme is adopted to handle
object proposals from different sensors within a calibrated
camera array. We conduct extensive testing on naturalistic
full-surround vision and LiDAR data collected on highways,
and illustrate the effects of different camera setups, fusion
schemes and 2D-to-3D projection schemes, both qualitatively
and quantitatively. Results obtained on the dataset support the
modular nature of our framework, as well as its ability to track
objects for a long duration. In addition to this, we believe
that the M3 OT framework can be used to test the utility of
any camera setup, and make suitable modifications thereof to
ensure optimum coverage from vision and range sensors.
VII. ACKNOWLEDGMENTS
We gratefully acknowledge the continued support of our
industrial and other sponsors. We would also like to thank

[1] J. Choi, S. Ulbrich, B. Lichte, and M. Maurer, “Multi-target tracking
using a 3d-lidar sensor for autonomous vehicles,” in Intelligent Transportation Systems-(ITSC), 2013 16th International IEEE Conference on.
IEEE, 2013, pp. 881–886.
[2] S. Song, Z. Xiang, and J. Liu, “Object tracking with 3d lidar via multitask sparse learning,” in Mechatronics and Automation (ICMA), 2015
IEEE International Conference on. IEEE, 2015, pp. 2603–2608.
[3] A. Asvadi, P. Peixoto, and U. Nunes, “Detection and tracking of moving
objects using 2.5 d motion grids,” in Intelligent Transportation Systems
(ITSC), 2015 IEEE 18th International Conference on. IEEE, 2015, pp.
788–793.
[4] A. Asvadi, C. Premebida, P. Peixoto, and U. Nunes, “3d lidar-based
static and moving obstacle detection in driving environments: An
approach based on voxels and multi-region ground planes,” Robotics
and Autonomous Systems, vol. 83, pp. 299–311, 2016.
[5] D. Pfeiffer and U. Franke, “Efficient representation of traffic scenes by
means of dynamic stixels,” in Intelligent Vehicles Symposium (IV), 2010
IEEE. IEEE, 2010, pp. 217–224.
[6] A. Broggi, S. Cattani, M. Patander, M. Sabbatelli, and P. Zani, “A full3d voxel-based dynamic obstacle detection for urban scenario using
stereo vision,” in Intelligent Transportation Systems-(ITSC), 2013 16th
International IEEE Conference on. IEEE, 2013, pp. 71–76.
[7] A. Vatavu, R. Danescu, and S. Nedevschi, “Stereovision-based multiple
object tracking in traffic scenarios using free-form obstacle delimiters
and particle filters,” IEEE Transactions on Intelligent Transportation
Systems, vol. 16, no. 1, pp. 498–511, 2015.
[8] A. Ošep, A. Hermans, F. Engelmann, D. Klostermann, M. Mathias,
and B. Leibe, “Multi-scale object candidates for generic object tracking
in street scenes,” in Robotics and Automation (ICRA), 2016 IEEE
International Conference on. IEEE, 2016, pp. 3180–3187.
[9] R. K. Satzoda, S. Lee, F. Lu, and M. M. Trivedi, “Vision based
front & rear surround understanding using embedded processors,” IEEE
Transactions on Intelligent Vehicles, 2017.
[10] S. Sivaraman and M. M. Trivedi, “Dynamic probabilistic drivability
maps for lane change and merge driver assistance,” IEEE Transactions
on Intelligent Transportation Systems, vol. 15, no. 5, pp. 2063–2073,
2014.
[11] H. Cho, Y.-W. Seo, B. V. Kumar, and R. R. Rajkumar, “A multisensor fusion system for moving object detection and tracking in urban
driving environments,” in Robotics and Automation (ICRA), 2014 IEEE
International Conference on. IEEE, 2014, pp. 1836–1843.
[12] A. Asvadi, P. Girão, P. Peixoto, and U. Nunes, “3d object tracking using
rgb and lidar data,” in Intelligent Transportation Systems (ITSC), 2016
IEEE 19th International Conference on. IEEE, 2016, pp. 1255–1260.
[13] M. Allodi, A. Broggi, D. Giaquinto, M. Patander, and A. Prioletti,
“Machine learning in tracking associations with stereo vision and
lidar observations for an autonomous vehicle,” in Intelligent Vehicles
Symposium (IV), 2016 IEEE. IEEE, 2016, pp. 648–653.
[14] S.-H. Bae and K.-J. Yoon, “Robust online multi-object tracking based
on tracklet confidence and online discriminative appearance learning,”
in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2014, pp. 1218–1225.
[15] S. Kim, S. Kwak, J. Feyereisl, and B. Han, “Online multi-target tracking
by large margin structured learning,” in Asian Conference on Computer
Vision. Springer, 2012, pp. 98–111.
[16] C.-H. Kuo, C. Huang, and R. Nevatia, “Multi-target tracking by online learned discriminative appearance models,” in Computer Vision and
Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010,
pp. 685–692.
[17] Y. Li, C. Huang, and R. Nevatia, “Learning to associate: Hybridboosted
multi-target tracker for crowded scene,” in Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009, pp.
2953–2960.
[18] X. Song, J. Cui, H. Zha, and H. Zhao, “Vision-based multiple interacting
targets tracking via on-line supervised learning,” in European Conference on Computer Vision. Springer, 2008, pp. 642–655.
[19] Y. Xiang, A. Alahi, and S. Savarese, “Learning to track: Online multiobject tracking by decision making,” in Proceedings of the IEEE
International Conference on Computer Vision, 2015, pp. 4705–4713.

12

[20] B. Babenko, M.-H. Yang, and S. Belongie, “Robust object tracking with
online multiple instance learning,” IEEE transactions on pattern analysis
and machine intelligence, vol. 33, no. 8, pp. 1619–1632, 2011.
[21] C. Bao, Y. Wu, H. Ling, and H. Ji, “Real time robust l1 tracker using
accelerated proximal gradient approach,” in Computer Vision and Pattern
Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 1830–
1837.
[22] S. Hare, S. Golodetz, A. Saffari, V. Vineet, M.-M. Cheng, S. L. Hicks,
and P. H. Torr, “Struck: Structured output tracking with kernels,” IEEE
transactions on pattern analysis and machine intelligence, vol. 38,
no. 10, pp. 2096–2109, 2016.
[23] Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,”
IEEE transactions on pattern analysis and machine intelligence, vol. 34,
no. 7, pp. 1409–1422, 2012.
[24] J. V. Dueholm, M. S. Kristoffersen, R. K. Satzoda, T. B. Moeslund, and
M. M. Trivedi, “Trajectories and maneuvers of surrounding vehicles with
panoramic camera arrays,” IEEE Transactions on Intelligent Vehicles,
vol. 1, no. 2, pp. 203–214, 2016.
[25] S. Sivaraman and M. M. Trivedi, “Looking at vehicles on the road:
A survey of vision-based vehicle detection, tracking, and behavior
analysis,” IEEE Transactions on Intelligent Transportation Systems,
vol. 14, no. 4, pp. 1773–1795, 2013.
[26] J. Berclaz, F. Fleuret, E. Turetken, and P. Fua, “Multiple object tracking
using k-shortest paths optimization,” IEEE transactions on pattern
analysis and machine intelligence, vol. 33, no. 9, pp. 1806–1819, 2011.
[27] A. A. Butt and R. T. Collins, “Multi-target tracking by lagrangian
relaxation to min-cost network flow,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2013, pp.
1846–1853.
[28] A. Milan, S. Roth, and K. Schindler, “Continuous energy minimization
for multitarget tracking,” IEEE transactions on pattern analysis and
machine intelligence, vol. 36, no. 1, pp. 58–72, 2014.
[29] J. C. Niebles, B. Han, and L. Fei-Fei, “Efficient extraction of human motion volumes by tracking,” in Computer Vision and Pattern Recognition
(CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 655–662.
[30] H. Pirsiavash, D. Ramanan, and C. C. Fowlkes, “Globally-optimal
greedy algorithms for tracking a variable number of objects,” in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference
on. IEEE, 2011, pp. 1201–1208.
[31] L. Zhang, Y. Li, and R. Nevatia, “Global data association for multiobject tracking using network flows,” in Computer Vision and Pattern
Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008, pp.
1–8.
[32] Z. Khan, T. Balch, and F. Dellaert, “Mcmc-based particle filtering for
tracking a variable number of interacting targets,” IEEE transactions on
pattern analysis and machine intelligence, vol. 27, no. 11, pp. 1805–
1819, 2005.
[33] S. Oh, S. Russell, and S. Sastry, “Markov chain monte carlo data
association for multi-target tracking,” IEEE Transactions on Automatic
Control, vol. 54, no. 3, pp. 481–497, 2009.
[34] K. Okuma, A. Taleghani, N. d. Freitas, J. J. Little, and D. G. Lowe,
“A boosted particle filter: Multitarget detection and tracking,” Computer
Vision-ECCV 2004, pp. 28–39, 2004.
[35] J. Munkres, “Algorithms for the assignment and transportation problems,” Journal of the society for industrial and applied mathematics,
vol. 5, no. 1, pp. 32–38, 1957.
[36] M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and
L. Van Gool, “Online multiperson tracking-by-detection from a single,
uncalibrated camera,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 9, pp. 1820–1833, 2011.
[37] J.-Y. Bouguet, “Pyramidal implementation of the affine lucas kanade
feature tracker description of the algorithm,” Intel Corporation, vol. 5,
no. 1-10, p. 4, 2001.
[38] K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking
performance: the clear mot metrics,” EURASIP Journal on Image and
Video Processing, vol. 2008, no. 1, pp. 1–10, 2008.
[39] A. Milan, K. Schindler, and S. Roth, “Challenges of ground truth evaluation of multi-target tracking,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Workshops, 2013, pp. 735–
742.
[40] R. N. Rajaram, E. Ohn-Bar, and M. M. Trivedi, “Refinenet: Refining object detectors for autonomous driving,” IEEE Transactions on Intelligent
Vehicles, vol. 1, no. 4, pp. 358–368, 2016.
[41] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for
dense object detection,” arXiv preprint arXiv:1708.02002, 2017.
[42] E. Ohn-Bar and M. M. Trivedi, “Fast and robust object detection
using visual subcategories,” in Computer Vision and Pattern Recognition

Workshops (CVPRW), 2014 IEEE Conference on. IEEE, 2014, pp. 179–
184.
[43] A. Rangesh, K. Yuen, R. K. Satzoda, R. N. Rajaram, P. Gunaratne, and
M. M. Trivedi, “A multimodal, full-surround vehicular testbed for naturalistic studies and benchmarking: Design, calibration and deployment,”
arXiv preprint arXiv:1709.07502, 2017.
[44] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics:
The kitti dataset,” The International Journal of Robotics Research,
vol. 32, no. 11, pp. 1231–1237, 2013.

Akshay Rangesh is currently working towards his
PhD in electrical engineering from the University
of California at San Diego (UCSD), with a focus
on intelligent systems, robotics, and control. His
research interests span computer vision and machine
learning, with a focus on object detection and tracking, human activity recognition, and driver safety
systems in general. He is also particularly interested
in sensor fusion and multi-modal approaches for real
time algorithms.

Mohan Manubhai Trivedi is a Distinguished
Professor at University of California, San Diego
(UCSD) and the founding director of the UCSD
LISA: Laboratory for Intelligent and Safe Automobiles, winner of the IEEE ITSS Lead Institution
Award (2015). Currently, Trivedi and his team are
pursuing research in intelligent vehicles, machine
perception, machine learning, human-robot interactivity, driver assistance, active safety systems. Three
of his students have received ”best dissertation”
recognitions. Trivedi is a Fellow of IEEE, ICPR and
SPIE. He received the IEEE ITS Society’s highest accolade ”Outstanding
Research Award” in 2013. Trivedi serves frequently as a consultant to industry
and government agencies in the USA and abroad.

