3D Trajectory Reconstruction of Dynamic Objects Using Planarity Constraints
Sebastian Bullinger1 , Christoph Bodensteiner1 , Michael Arens1 and Rainer Stiefelhagen2
1

arXiv:1711.06136v1 [] 16 Nov 2017

2

Fraunhofer IOSB
Karslruhe Institute of Technology

{sebastian.bullinger,christoph.bodensteiner,michael.arens}@iosb.fraunhofer.de
rainer.stiefelhagen@kit.edu

Abstract

mented reality applications. There are different platforms
like drones or wearable systems where one wants to achieve
this task with a minimal number of devices in order to reduce weight or lower production costs. We propose an approach to reconstruct three-dimensional object motion trajectories using a single camera as sensor.
The reconstruction of object motion trajectories in monocular video data captured by moving cameras is a challenging
task, since in general it cannot be solely solved exploiting
image observations. Each observed object motion trajectory
is scale ambiguous. Additional constraints are required to
identify a motion trajectory consistent to background structures. [23, 13, 3] assume that the camera is mounted on
a driving vehicle, i.e. the camera has specific height and
a known pose. [16, 28, 17] solve the scale ambiguity by
making assumptions about object and camera motion trajectories. We follow Ozden’s principle of non-accidental
motion trajectories [16] and introduce a new object motion
constraint exploiting semantic segmentation and terrain geometry to compute consistent object motion trajectories.
In many scenarios objects cover only a minority of pixels in
video frames. This increases the difficulty of reconstructing object motion trajectories using image data. In such
cases current state-of-the-art Structure from Motion (SfM)
approaches treat moving object observations most likely as
outliers and reconstruct background structures instead. Previous works, e.g. [11, 12], tackle this problem by considering multiple video frames to determine moving parts in the
video. They apply motion segmentation or keypoint tracking to detect moving objects. These kind of approaches are
vulnerable to occlusion and require objects to move in order
to separate them from background structures.
Our method exploits recent results in instance-aware semantic segmentation and rigid Structure from Motion techniques. Thus, our approach extends naturally to stationary objects. In addition, we do not exploit specific camera
pose constraints like a fixed camera-ground-angle or a fixed

We present a method to reconstruct the threedimensional trajectory of a moving instance of a known
object category in monocular video data. We track the
two-dimensional shape of objects on pixel level exploiting
instance-aware semantic segmentation techniques and optical flow cues. We apply Structure from Motion techniques
to object and background images to determine for each
frame camera poses relative to object instances and background structures. By combining object and background
camera pose information, we restrict the object trajectory to
a one-parameter family of possible solutions. We compute a
ground representation by fusing background structures and
corresponding semantic segmentations. This allows us to
determine an object trajectory consistent to image observations and reconstructed environment model. Our method
is robust to occlusion and handles temporarily stationary
objects. We show qualitative results using drone imagery.
Due to the lack of suitable benchmark datasets we present a
new dataset to evaluate the quality of reconstructed threedimensional object trajectories. The video sequences contain vehicles in urban areas and are rendered using the
path-tracing render engine Cycles to achieve realistic results. We perform a quantitative evaluation of the presented
approach using this dataset. Our algorithm achieves an average reconstruction-to-ground-truth distance of 0.31 meter. The dataset will be publicly available on our website1 .

1. Introduction
1.1. Trajectory Reconstruction
The reconstruction of three-dimensional object motion
trajectories is important for autonomous systems and aug1 Project

page: URL

1

camera-ground-distance. We evaluate the presented object
motion trajectory reconstruction algorithm in UAV scenarios, where such constraints are not valid.

struction related research.

1.4. Paper Overview
The paper is organized as follows. Section 2 describes
the structure and the components of the proposed pipeline.
In section 2.1 we derive an expression for a one-parameter
family of possible object motion trajectories combining object and background reconstruction results. Section 2.2 describes a method to approximate the ground locally. In section 2.3 we describe a method to compute consistent object
motion trajectories. In section 4 we provide an qualitative
and quantitative evaluation of the presented algorithms using drone imagery and rendered video data. Section 5 concludes the paper.

1.2. Related Work
Semantic segmentation or scene parsing is the task of
providing semantic information at pixel-level. Early semantic segmentation approaches using ConvNets, e.g. Farabet
et al. [5], exploit patchwise training. Long et al. [21] applied Fully Convolutinal Networks for semantic segmentation, which are trained end-to-end. Recently, [4, 14, 9] proposed instance-aware semantic segmentation approaches.
The field of Structure from Motion (SfM) can be divided
into iterative and global approaches. Iterative or sequential
SfM methods [22, 27, 15, 24, 20] are more likely to find
reasonable solutions than global SfM approaches [15, 24].
However, the latter are less prone to drift.
The determination of the correct scale ratio between object and background reconstruction requires additional constraints. Ozden et al. [16] exploit the non-accidentalness
principle in the context of independently moving objects.
Yuan et al. [28] propose to reconstruct the 3D object trajectory by assuming that the object motion is perpendicular to the normal vector of the ground plane. Kundu et al.
[11] exploit motion segmentation with multibody VSLAM
to reconstruct the trajectory of moving cars. They use an
instantaneous constant velocity model in combination with
Bearing only Tracking to estimate consistent object scales.
Park et al. propose an approach in [17] to reconstruct the
trajectory of a single 3D point tracked over time by approximating the motion using a linear combination of trajectory
basis vectors. Previous works, like [16, 28, 11, 17] show
only qualitative results.

2. Object Motion Trajectory Reconstruction
The pipeline of our approach is shown in Fig. 1. The
input is an ordered image sequence. We track twodimensional object shapes on pixel level across video sequences following the approach presented in [2]. In contrast
to [2], we identify object shapes exploiting the instanceaware semantic segmentation method presented in [14] and
associate extracted object shapes of subsequent frames using the optical flow approach described in [10]. Without
the loss of generality, we describe motion trajectory reconstructions of single objects. We apply SfM [15, 20] to object and background images as shown in Fig. 1. Object
images denote images containing only color information of
single object instance. Similarly, background images show
only background structures. We combine object and background reconstructions to determine possible, visually identical, object motion trajectories. We compute a consistent
object motion trajectory exploiting constraints derived from
reconstructed terrain ground geometry.

1.3. Contribution

2.1. Object Trajectory Representation

The core contributions of this work are as follows. (1)
We present a new framework to reconstruct the threedimensional trajectory of moving instances of known object categories in monocular video data leveraging sateof-the-art semantic segmentation and structure from motion approaches. (2) We propose a novel method to compute object motion trajectories consistent to image observations and background structures. (3) In contrast to previous work, we quantitatively evaluate the reconstructed object motion trajectories. (4) We created a new object motion trajectory benchmark dataset due to the lack of publicly
available video data of moving objects with suitable ground
truth data. The dataset consists of photo-realistic rendered
videos of urban environments. It includes animated vehicles
as well as set of predefined camera and object motion trajectories. 3D vehicle and environmental models used for rendering serve as ground truth. (5) We will publish the dataset
and evaluation scripts to foster future object motion recon-

In order to estimate a consistent object motion trajectory
we apply SfM simultaneously to object and background images as shown in Fig. 1. We denote the corresponding SfM
(o)
results with sf m(o) and sf m(b) . Let oj ∈ P (o) and
(b)

bk ∈ P (b) denote the 3D points contained in sf m(o) or
(o)
sf m(b) , respectively. The superscripts o and b in oj and
(b)

bk describe the corresponding coordinate frame. The variables j and k are the indices of points in the object or the
background point cloud, respectively. We denote the reconstructed intrinsic and extrinsic parameters of each registered input image as virtual camera. Each virtual camera
in sf m(o) and sf m(b) corresponds to a certain frame from
which object and background images are extracted. We associate virtual cameras in sf m(o) with the corresponding
virtual cameras in sf m(b) and vice versa. In the following, we consider only camera pairs, whose virtual cameras
2

Input Frames

Semantic
Segmentation and
Object Tracking

Object Segmentations

Background
Segmentations

SfM

Ground Segmentations

SfM

Background
SfM Result

Object SfM Result

Object Trajectory Family

Trajectory Family
Computation

Background
SfM Result

Scale Estimation
and Trajectory
Computation

Ground Representation

Consistent Object
Motion Trajectory

Ground
Computation

Figure 1: Overview of the Trajectory Reconstruction Pipeline.
are contained in sf m(o) and sf m(b) . Because of missing
image registrations this may not be the case for all virtual
cameras.
We reconstruct the object motion trajectory by combining
information of corresponding virtual cameras. For any virtual camera pair of an image with index i the object SfM
result sf m(o) contains information of object point positions
(o)
(o)
oj relative to virtual cameras with camera centers ci and
(o)
(o)
rotations Ri . We express each object point oj
(i)
coordinates oj of camera i using equation (1)
(i)

(o)

oj = Ri

(o)

(o)

· (oj − ci ).

described according to equation (3).
(b)

(b)

(b)

T (b)

(i)

· oj .

T (b)

(o)

· Ri

(o)

(o)

· (oj − ci )

:= ci(b) + r · v(b)
j,i

(3)

with
(b)

T (b)

vj,i = Ri

in camera

(o)

· Ri

(o)

(o)

(b)

(b)

· (oj − ci ) = oj,i − ci .

(4)

Given the scale ratio r, we can recover the full object motion
trajectory computing equation (4) for each virtual camera
(b)
pair. We use oj,i of all cameras and object points as object
motion trajectory representation. The ambiguity mentioned
in section 1 is expressed by the unknown scale ratio r.

(1)

The background SfM result sf m(b) contains the camera
(b)
(b)
center ci and the corresponding rotation Ri , which provide pose information of the camera with respect to the reconstructed background. Note, that the camera coordinate
systems of virtual cameras in sf m(o) and sf m(b) are equiv(b)
(b)
alent. We use ci and Ri to transform object points to the
background coordinate system using equation (2)
oj,i = ci + Ri

(b)

oj,i = ci + r · Ri

2.2. Terrain Ground Approximation
Further camera or object motion constraints are required
to determine the scale ratio r introduced in equation (4).
In contrast to previous work [16, 28, 17, 13, 23, 3] we assume that the object category of interest moves on top of
the terrain. We exploit semantic segmentation techniques
to estimate an approximation of the ground surface of the
scene. We apply the ConvNet presented in [21] to determine ground categories like street or grass for all input images on pixel level. We consider only stable background
points, i.e. 3D points that are observed at least four times.
We determine for each 3D point a ground or non-ground
label by accumulating the semantic labels of corresponding
keypoint measurement pixel positions. This allows us to determine a subset of background points, which represent the
ground of the scene. We approximate the ground surface
locally using plane representations. For each frame i we

(2)

In general, the scale ratio of object and background reconstruction does not match due to the scale ambiguity of
SfM reconstructions [8]. We tackle this problem by treating the scale of the background as reference scale and by
introducing a scale ratio factor r to adjust the scale of object point coordinates. The overall transformation of object
(o)
points given in object coordinates oj to object points in
(b)

the background coordinate frame system oj,i of camera i is
3

Equation (8) allows us to determine the scale ratio r
between object and background reconstruction using the
extrinsic parameters of two cameras and corresponding
ground approximations.

use corresponding estimated camera parameters and object
point observations to determine a set of ground points Pi
close to the object. We build a kd-tree containing all ground
measurement positions of the current frame. For each object point observation, we determine the numb closest background measurements. In our experiments, we set numb to
50. Let cardi be the cardinality of Pi . While cardi is less
than numb , we add the next background observation of each
point measurement. This results in an equal distribution of
local ground points around the vehicle. We apply RANSAC
[6] to compute a local approximation of the ground surface
using Pi . Each plane is defined by a corresponding normal
vector ni and an arbitrary point pi lying on the plane.

2.3.2

The accuracy of the estimated scale ratio r in equation (8)
is subject to the condition of the parameters of the particular view pair. For instance, if the numerator or denominator is close to zero, small errors in the camera poses or
ground approximations may result in negative scale ratios.
In addition, wrongly estimated local plane normal vectors
may disturb camera-plane distances. We tackle these problems by combining two different view pair rankings. The
first ranking uses for each view pair the difference of the
camera-plane distances, i.e. the numerator in equation (8).
The second ranking reflects the quality of the local ground
approximation w.r.t. the object reconstruction. For a view
pair with well reconstructed local planes the variance of the
corresponding scale ratios is small. This allows for the determination of ill conditioned view pairs. The second ranking uses the scale ratio difference to order the view pairs.
We sort the view pairs by weighting both ranks equally. Let
vp denote the view pair with the lowest overall rank. The final scale ratio is determined by using a least squares method
w.r.t. all equations of vp.

2.3. Scale Estimation using Constant Distance Constraints
In section 2.3, we exploit priors of object motion to improve the robustness of the reconstructed object trajectory.
We assume that the object of interest moves on a locally
planar surface. In this case the distance of each object point
(b)
oj,i to the ground is constant for all cameras i. The reconstructed trajectory shows this property only for the true
scale ratio and non-degenerated camera motion. For example, a degenerate case occurs when the camera moves exactly parallel to a planar object motion. For a more detailed
discussion of degenerated camera motions see [16].
2.3.1

Scale Ratio Estimation using a Single View Pair

We use the term view to denote cameras and corresponding local ground planes. The signed distance of an object
(b)
point oj,i to the ground plane can be computed according
to equation (5)
(b)

dj,i = ni · (oj,i − pi ),

3. Virtual Object Motion Trajectory Dataset
To quantitatively evaluate the quality of the reconstructed object motion trajectory we require accurate object
and environment models as well as object and camera poses
at each time step. The simultaneous capturing of corresponding ground truth data with sufficient quality is difficult
to achieve. For example, one could capture the environment
geometry with LIDAR sensors and the camera / object pose
with an additional system. However, the registration and
synchronization of all these different modalities is a complex and cumbersome process. The result will contain noise
and other artifacts like drift. To tackle these issues we exploit virtual models. Previously published virtually generated and virtually augmented datasets, like [18, 19, 7, 25],
provide data for different application domains and do not include three-dimensional ground truth information. We build
a virtual world including an urban environment, animated
vehicles as well as predefined vehicle and camera motion
trajectories. This allows us to compute spatial and temporal
error free ground truth data. We exploit procedural generation of textures to avoid artificial repetitions. Thus, our
dataset is suitable for evaluating SfM algorithms.

(5)

where pi is a point on the local ground plane and ni is the
corresponding normal vector. If the object moves on top of
the approximated terrain ground the distance dj,i should be
independent of a specific camera i. We substitute dj,i with
dj in equation (5). This allows us to combine equation (5)
of the same point and different cameras.
(b)

(b)

ni · (oj,i − pi ) = ni0 · (oj,i0 − pi0 ).

(6)

Substituting equation (3) in equation (6) results in (7)
(b)

(b)

(b)

(b)

ni · (ci + r · vj,i − pi ) = ni0 · (ci0 + r · vj,i0 − pi0 ) (7)
Solving equation (7) for r yields equation (8)
(b)

r=

(b)

ni0 · (ci0 − pi0 ) − ni · (ci − pi )
(b)

(b)

(ni · vj,i − ni0 · vj,i0 )

.

Scale Ratio Estimation using View Pair Ranking

(8)

4

(a) Environment Model from (b) Environment Rendered from (c) Car Model with Motion Path
a Bird’s Eye Perspective (in a Bird’s Eye Perspective (Ren- in Street Scene (in Blender)
Blender)
dered)

(d) Rendered Street Scene

Figure 2: Example Scenes from the Virtual Object Trajectory Dataset.

(a) Input Frame 0

(b) Input Frame 100

(c) Input Frame 200

(d) Reconstructed
(Top View)

Trajectory

Figure 3: Car Trajectory Reconstruction using 250 frames with a resolution of 1920 x 1080 pixels captured by a DJI drone.

3.1. Virtual World

quences cover a high variety of object and camera poses.
The object trajectories reflect common vehicle motions include vehicle acceleration, different curve types and motion
on changing slopes. We use the path-tracing render engine
Cycles [1] to achieve photo realistic rendering results. We
observed that the removal of artificial path-tracing artifacts
using denoising is crucial to avoid degenerated SfM reconstructions.
The dataset includes 6D object and camera poses for each
frame as well as ground truth meshes of corresponding vehicle models. In contrast to measured ground truth data, virtual ground truth data is free of noise and shows no spatial
registration or temporal synchronization inaccuracies. The
dataset contains semantic segmentations of objects, ground
and background to separate the reconstruction task from
specific semantic segmentation and tracking approaches. In
addition to the virtual data, the dataset also includes the
computed reconstruction results. We will make our evaluation scripts publicly available to foster future analysis of
object trajectory estimation.

We used Blender [1] to create a virtual world consisting of a city surrounded by a countryside. We exploit procedural generation to compute textures of large surfaces,
like streets and sidewalks, to avoid degenerated Structure
from Motion results caused by artificial texture repetitions.
The virtual world includes different assets like trees, traffic lights, streetlights, phone booths, bus stops and benches.
We collected a set of publicly available vehicle assets to
populate the scenes. We used skeletal animation, also referred to as rigging, for vehicle animation. This includes
wheel rotation and steering w.r.t. the motion trajectory as
well as consistent vehicle placement on uneven ground surfaces. The animation of wheels is important to avoid unrealistic wheel point triangulations. We adjusted the scale of
vehicles and virtual environment using Blender’s unit system. This allows us to set the virtual space in relation to the
real world. The extent of the generated virtual world corresponds to one square kilometer. We exploit environment
mapping to achieve realistic illumination. With Blender’s
built-in tools, we defined a set of camera and object motion
trajectories. This allows us to determine the exact 3D pose
of cameras and vehicles at each time step.

4. Experiments and Evaluation
We show qualitative results and quantitative evaluations
using real drone footage and virtual generated drone imagery, respectively. Fig. 3 shows the reconstruction of a car
motion trajectory using images captured by a DJI drone.
Fig. 4 shows an example of the virtual object trajectory
dataset. Fig. 4(j) and Fig. 4(i) show the object point cloud
transformed into the virtual world coordinate frame system.

3.2. Trajectory Dataset
We use the previously created virtual world to build a
new object trajectory dataset1 . The dataset consists of 35
sequences capturing five vehicles in different urban scenes.
Fig. 2 shows some example images. The virtual video se5

(a) Ground Truth Object Path

(b) Object Model in Blender

(c) Rendered Scene

(d) Object Segmentation

(e) Background Segmentation

(f) Ground Segmentation

(g) Object Reconstruction

(h) Background Reconstruction

(i) Reconstructed Object Motion (j) Reconstructed Object Motion (k) Ground Truth Model at Dif- (l) Ground Truth Model at DifferTrajectory
Trajectory
ferent Frames Overlayed With ent Frames Overlayed With ReReconstruction
construction

Figure 4: Car trajectory reconstruction using 60 virtual frames with a resolution of 1920 x 1080 pixels.
truth coordinates. For each sequence we define the trajectory error as the average trajectory-point-mesh distance.
Fig. 5 shows for each sequence the trajectory error in meter. The average trajectory error per vehicle using the full
dataset is shown in table 1. Overall, we achieve a trajectory
error of 0.31 meter. The error of the object trajectory reconstructions reflects four types of computational inaccuracies:
deviations of camera poses w.r.t. object and background
point clouds, wrong triangulated object points as well as
scale ratio discrepancies. Fig. 6 compares the estimated
scale ratios of the proposed and the baseline method w.r.t.
the reference scale ratio. The reference scale ratio computation is described in section 4.4. The overall estimated scale
ratio deviation w.r.t. the reference scale per vehicle is shown
in table 1. The provided reference scale ratios are subject to
the registration described in section 4.3. Wrongly reconstructed background camera poses may influence the reference scale ratio. The van object reconstruction was only
partial successful on the sequences crossing, overtaking and
steep street. The SfM algorithm registered 19%, 60% and
98% of the images, respectively. The object reconstruction
of the smart model contained 74% of the crossing input object images. Here, we use the subset of registered images to
perform the evaluation. The camera and the object motion
in bumpy road simulate a sequence close to a degenerated
case, i.e. equation (8) is ill conditioned for all view pairs.

Fig. 4(k) and Fig. 4(l) show the overlay result of transformed points and the corresponding virtual ground truth
model. To segment the two-dimensional shape of objects of
interest we follow the approach presented in [2]. In contrast to [2], we used [14] and [10] to segment and track
visible objects, respectively. We considered the following SfM pipelines for object and background reconstructions: Colmap [20], OpenMVG [15], Theia [24] and VisualSfM [27]. Our object trajectory reconstruction pipeline
uses Colmap for object and OpenMVG for background reconstructions, since Colmap and OpenMVG created in our
experiments the most reliable object and background reconstructions.

4.1. Virtual Object Motion Trajectory Evaluation
We use the dataset presented in section 3 to quantitatively evaluate the proposed object motion trajectory reconstruction approach. The evaluation is based on object, background and ground segmentations included in the dataset.
This allows us to show results independent from the performance of specific instance segmentation and tracking approaches. We compare the proposed method with the baseline presented in section 4.2 using 35 sequences contained
in the dataset. We automatically register the reconstructed
object trajectory to the ground truth using the method described in section 4.3. We compute the shortest distance of
each object trajectory point to the object mesh in ground
6

Trajectory Error in meter

Lancer (Baseline)
Lancer (Ours)

2

Lincoln (B.)
Lincoln (O.)

Smart (B.)
Smart (O.)

Golf (B.)
Golf (O.)

Van (B.)
Van (O.)

1

0
Right Curves

Left Curves

Crossing

Overtaking

Bridge

Steep Street

Bumpy Road

Deviation w.r.t. Reference

Figure 5: Quantitative evaluation of the trajectory error in meter computed by the methods constant distance (ours) and
intersection (baseline). We evaluate seven different vehicle trajectories (right curves, left curves, crossing, overtaking, bridge,
steep street and bumpy road) and five different vehicle models (Lancer, Lincoln Navigator, Smart, Golf and Van). The
cropped values of the baseline are 6.3m, 4m and 4.8m.

Lancer (Baseline)
Lancer (Ours)

0.4

Lincoln (B.)
Lincoln (O.)

Smart (B.)
Smart (O.)

Golf (B.)
Golf (O.)

Van (B.)
Van (O.)

0.2

0
Right Curves

Left Curves

Crossing

Overtaking

Bridge

Steep Street

Bumpy Road

Figure 6: Quantitative evaluation of the scale ratio of the methods constant distance (ours) and intersection (baseline). The
provided values are the deviations w.r.t. reference scales.

4.2. Scale Estimation Baseline: Intersection Constraints

The intersection parameter ri corresponds to the point being
closest to the ground surface, i.e. a point at the bottom of the
object. Plugging ri in equation (3) for camera i places the
object point cloud on top of the ground surface. Thus, the
smallest ray-plane-intersection-parameter ri is a value close
to the real object-to-background-scale. Finally, we use the
median s of all image scale ratio factors as scale ratio factor
to reconstruct the trajectory as computed in equation (11)

The baseline is motivated by the fact, that some reconstructed points of the bottom of an object should lie in the
proximity of the ground surface of the environment. Consider for example the reconstructed 3D points corresponding to the wheels of a car. This approach works only if
at least one camera-point-ray of an arbitrary point in the
object point cloud intersects the ground surface. For each
(b)
camera we generate a set of vectors vj,i pointing from the
(b)

r = med({ri |i ∈ {1, . . . , nI }}),

where med denotes the median and nI the number of images. We do not consider invalid image scale ratios ri , i.e.
cameras which have no camera-object-point-rays intersecting the ground representation.

(b)

camera center ci towards each object point oj,i . For non(b)

orthogonal direction vectors vj,i and normal vectors ni
we compute the ray plane intersection parameter for each
camera-object-point-pair according to equation (9)

4.3. Registration of Background Reconstruction
and Virtual Environment

(b)

rj,i =

(pi − ci ) · ni
(b)

vj,i · ni

.

(9)

A common approach to register different coordinate systems is to exploit 3D-3D correspondences. To determine
points in the virtual environment corresponding to background reconstruction points one could create a set of rays
from each camera center to all visible reconstructed back-

We compute the smallest ray-plane-intersection parameter
for each image i.
ri = min({rj,i |j ∈ {1, . . . , |P (o) |}})

(11)

(10)
7

Scale Ratio
Est. Type
Intersection (Baseline)
Constant Distance (Ours)

Average Scale Ratio Deviation w.r.t. Reference
Lancer Lincoln Smart Golf
Van
0.05
0.07
0.01
0.08
0.13
0.04
0.04
0.04
0.06
0.08

Lancer
0.42
0.20

Average Trajectory Error
Lincoln Smart Golf
0.53
0.25
0.95
0.23
0.33
0.33

Van
1.68
0.47

Table 1: Summary of the conducted evaluation. The second column shows the deviation of the estimated scale ratio w.r.t to
the reference scale ratio per vehicle. The third column contains the average distances of the full dataset in meter. Overall, the
trajectory error of the baseline and our approach is 0.77m and 0.31m.
ground points. The corresponding environment points are
defined by the intersection of these rays with the mesh of
the virtual environment. Due to the complexity of our environment model this computation is in terms of memory and
computational effort quite expensive. Instead, we use the
algorithm presented in [26] to estimate a similarity transformation Ts between the cameras contained in the background reconstruction and the virtual cameras used to render the corresponding video sequence. This allows us to
perform 3D-3D-registrations of background reconstructions
and the virtual environment as well as to quantitatively evaluate the quality of the reconstructed object motion trajectory. We use the camera centers as input for [26] to compute an initial reconstruction-to-virtual-environment transformation. Depending on the shape of the camera trajectory
there may be multiple valid similarity transformations using camera center positions. In order to find the semantically correct solution we enhance the original point set
with camera pose information, i.e. we add points reflect(b)
T (b)
ing up vectors ui = Ri · (0, 1, 0)T and forward vectors
(b)
T (b)
f i = Ri · (0, 0, 1)T . For the reconstructed cameras, we
adjust the magnitude of these vectors using the scale computed during the initial similarity transformation. We add
(b)
(b)
the corresponding end points of up ci + m · ui as well
(b)
(b)
as viewing vectors ci + m · f i to the camera center point
set. Here, m denotes the corresponding magnitude.

plicitly contains information about the scale ratio r(bv) between background reconstruction and virtual environment.
To compute r(ov) we use corresponding pairs of object reconstruction and virtual cameras. We use the extrinsic parameters of the object reconstruction camera to transform
all 3D points in the object reconstruction into camera coordinates. Similarly, the object mesh with the pose of the
corresponding frame is transformed into the camera coordinates leveraging the extrinsic camera parameters of the
corresponding virtual camera. The ground truth pose and
shape of the object mesh is part of the dataset. In camera
coordinates we generate rays from the camera center (i.e.
(i)
the origin) to each 3D point oj in the object reconstruc-

4.4. Reference Scale Ratio Computation

ref
Equation (14) shows that r(ob)
depends on the quality of the
cameras in the background reconstruction and may slightly
differ from the true scale ratio.

(i)

tion. We determine the shortest intersection mj of each
ray with the object mesh in camera coordinates. This allows us to compute r(ov) according to equation (13)
(i)

r(ov) = med({med({

kmj k
(i)

koj k

|j ∈ {1, . . . , nJ }}

(13)

|i ∈ {1, . . . , nI }}).
ref
This allows us to compute the reference scale ratios r(ob)
using equation (14)
ref
r(ob)
= r(ov) · r(bv) −1 .

As explained in section 4.1 the presented average trajectory errors in Fig. 5 are subject to four different error
sources. To evaluate the quality of the scale ratio estimation between object and background reconstruction we provide corresponding reference scale ratios. The scale ratios
between object reconstruction, background reconstruction
and virtual environment are linked via the relation shown in
equation (12)
r(ov) = r(ob) · r(bv) ,
(12)

(14)

5. Conclusions
This paper presents a pipeline to reconstruct the threedimensional trajectory of moving objects using monocular
video data. We propose a novel constraint to estimate consistent object motion trajectories. In contrast to previous
approaches, the presented scale estimation constraint is robust to occlusion and extents naturally to stationary objects.
Due to the lack of 3D object motion trajectory benchmark
datasets with suitable ground truth data, we present a new
virtual dataset to quantitatively evaluate object motion trajectories. The dataset contains rendered videos of urban en-

where r(ov) and r(bv) are the scale ratios between object
and background reconstructions and virtual environment,
respectively. The scale ratios r(ob) in Fig. 6 express the
spatial relation of object and background reconstructions.
The similarity transformation Ts defined in section 4.3 im8

vironments and accurate ground truth data including semantic segmentations, object meshes as well as object and camera poses for each frame. The proposed algorithm achieves
an average reconstruction-to-ground-truth distance of 0.31
m evaluating 35 trajectories. In future work, we will analyze
breakdown points of the proposed pipeline in more detail.
This includes minimal object sizes, object occlusions and
degeneracy cases. In addition, we intend to integrate previously published scale estimation approaches. These will
serve together with the dataset1 as benchmark references for
future object motion trajectory reconstruction algorithms.

[14]

[15]
[16]

[17]

References
[1] Blender Online Community. Blender - a 3d modelling and
rendering package, 2016.
[2] S. Bullinger, C. Bodensteiner, and M. Arens. Instance flow
based online multiple object tracking. In IEEE International
Conference on Image Processing (ICIP), 2017.
[3] F. Chhaya, N. D. Reddy, S. Upadhyay, V. Chari, M. Z. Zia,
and K. M. Krishna. Monocular reconstruction of vehicles:
Combining SLAM with shape priors. In 2016 IEEE International Conference on Robotics and Automation, ICRA
2016, Stockholm, Sweden, May 16-21, 2016, pages 5758–
5765, 2016.
[4] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2016.
[5] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning
hierarchical features for scene labeling. IEEE Transactions
on Pattern Analysis and Machine Intelligence, August 2013.
[6] M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM,
24(6):381–395, June 1981.
[7] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig. Virtual worlds as
proxy for multi-object tracking analysis. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2016.
[8] R. I. Hartley and A. Zisserman. Multiple View Geometry
in Computer Vision. Cambridge University Press, ISBN:
0521540518, second edition, 2004.
[9] K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask r-cnn.
In The IEEE International Conference on Computer Vision
(ICCV), Oct 2017.
[10] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and
T. Brox. Flownet 2.0: Evolution of optical flow estimation
with deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[11] A. Kundu, K. M. Krishna, and C. V. Jawahar. Realtime multibody visual slam with a smoothly moving monocular camera. In ICCV, 2011.
[12] K. Lebeda, S. Hadfield, and R. Bowden. 2D or not 2D:
Bridging the gap between tracking and structure from motion. In Proceedings, Asian Conference on Computer Vision
(ACCV), 2014.
[13] B. Lee, K. Daniilidis, and D. D. Lee. Online self-supervised
monocular visual odometry for ground vehicles. In 2015

[18]

[19]

[20]

[21]

[22]

[23]

[24]
[25]

[26]

[27]
[28]

9

IEEE International Conference on Robotics and Automation
(ICRA), pages 5232–5238, May 2015.
Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional
instance-aware semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
July 2017.
P. Moulon, P. Monasse, R. Marlet, and Others. Openmvg. an
open multiple view geometry library., 2013.
K. E. Ozden, K. Cornelis, L. V. Eycken, and L. J. V. Gool.
Reconstructing 3d trajectories of independently moving objects using generic constraints. Computer Vision and Image
Understanding, 96(3):453–471, 2004.
H. S. Park, T. Shiratori, I. Matthews, and Y. Sheikh. 3d trajectory reconstruction under perspective projection. International Journal of Computer Vision, 115(2):115–135, 2015.
S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing
for data: Ground truth from computer games. In B. Leibe,
J. Matas, N. Sebe, and M. Welling, editors, European Conference on Computer Vision (ECCV), volume 9906 of LNCS,
pages 102–118. Springer International Publishing, 2016.
G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M.
Lopez. The synthia dataset: A large collection of synthetic
images for semantic segmentation of urban scenes. In 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3234–3243, June 2016.
J. L. Schönberger and J.-M. Frahm. Structure-from-motion
revisited. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
E. Shelhamer, J. Long, and T. Darrell. Fully convolutional
networks for semantic segmentation. IEEE Trans. Pattern
Anal. Mach. Intell., 39(4):640–651, 2017.
N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism:
Exploring photo collections in 3d. ACM Trans. Graph.,
25(3):835–846, July 2006.
S. Song, M. Chandraker, and C. C. Guest. High accuracy
monocular SFM and scale correction for autonomous driving. IEEE Trans. Pattern Anal. Mach. Intell., 38(4):730–743,
2016.
C. Sweeney. Theia Multiview Geometry Library: Tutorial &
Reference. University of California Santa Barbara., 2014.
A. Tsirikoglou, J. Kronander, M. Wrenninge, and J. Unger.
Procedural modeling and physically based rendering for synthetic data generation in automotive applications. CoRR,
abs/1710.06270, 2017.
S. Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Trans. Pattern
Anal. Mach. Intell., 13(4):376–380, 1991.
C. Wu. Visualsfm: A visual structure from motion system,
2011.
C. Yuan and G. G. Medioni. 3d reconstruction of background
and objects moving on ground plane viewed from a moving camera. In 2006 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR 2006), 1722 June 2006, New York, NY, USA, pages 2261–2268, 2006.

