---
abstract: |
    In this work we present a lightweight, unsupervised learning pipeline for *dense* depth, optical flow and egomotion estimation from sparse event output of the Dynamic Vision Sensor (DVS). To tackle this low level vision task, we use a novel encoder-decoder neural network architecture - ECN.

    Our work is the first monocular pipeline that generates dense depth and optical flow from sparse event data only. The network works in self-supervised mode and has just 150k parameters. We evaluate our pipeline on the MVSEC self driving dataset and present results for depth, optical flow and and egomotion estimation. Due to the lightweight design, the inference part of the network runs at 250 FPS on a single GPU, making the pipeline ready for realtime robotics applications. Our experiments demonstrate significant improvements upon previous works that used deep learning on event data, as well as the ability of our pipeline to perform well during both day and night.

    *Keywords – event-based learning, neuromorphic sensors, DVS, autonomous driving, low-parameter neural networks, structure form motion, unsupervised learning*
author:
- |
    Chengxi Ye$^{1*}$, Anton Mitrokhin$^{1*}$,\
    Cornelia Fermüller$^{1}$, James A. Yorke$^{2}$, Yiannis Aloimonos$^{1}$[^1] [^2][^3]
bibliography:
- 'references.bib'
title: '**Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data** '
---

Supplementary Material {#supplementary-material .unnumbered}
======================

The supplementary video, code, trained models, appendix and a preprocessed dataset will be made available at <http://prg.cs.umd.edu/ECN.html>.

INTRODUCTION
============

With recent advances in the field of autonomous driving, autonomous platforms are no longer restricted to research laboratories and testing facilities - they are designed to operate in an open world, where reliability and safety are key factors. Modern self-driving cars are often fitted with a sophisticated sensor rig, featuring a number of LiDARs, cameras and radars, but even those undoubtedly expensive setups are prone to misperform in difficult conditions - snow, fog, rain or at night.

Recently, there has been much progress in imaging sensor technology, offering alternative solutions to scene perception. A neuromorphic imaging device, called Dynamic Vision Sensor (DVS) [@lichtsteiner_latency_2008], inspired by the transient pathway of mammalian vision, can offer exciting alternatives for visual motion perception. The DVS does not record image frames, but instead - the changes of lighting occurring independently at every camera pixel. Each of these changes is transmitted asynchronously and is called an *event*. By its design, this sensor accommodates a large dynamic range, provides high temporal resolution and low latency – ideal properties for applications where high quality motion estimation and tolerance towards challenging lighting conditions are desirable. The DVS offers new opportunities for robust visual perception so much needed in autonomous robotics, but challenges associated with the sensor output, such as high noise, relatively low spatial resolution and sparsity, ask for different visual processing approaches.

![**](img/teaser.png){width="0.95\columnwidth"}

\[fig:teaser\]

In this work we introduce a novel lightweight encoding-decoding neural network architecture - the Evenly-Cascaded convolutional Network (ECN) to address the problems of event data sparsity for depth, optical flow and egomotion estimation in a self driving car setting. Despite having just 150k parameters, our network is able to generalize well to different types of sequences. The simple nature of our pipeline allows it to run at more than 250 inferences per second on a single NVIDIA 1080 Ti GPU. We perform ablation studies using the SfMlearner architecture [@zhou2017unsupervised] as a baseline and evaluate different normalization techniques (including our novel *feature decorrelation*) to show that our model is well suited for event data.

We demonstrate superior generalization to low-light scenes. Fig. \[fig:teaser\] shows an example featuring night driving - the network trained on a day light scene was able to predict both depth and flow even with a low event rate and abundance of noise. This is facilitated by our event-image representation: instead of the latest event timestamps, we use the average timestamp of the events generated at a given pixel. The averaging helps to reduce the noise without losing the timestamp information. Moreover, we use multiple slices as input to our model to better preserve the 3D structure of the event cloud and more robustly estimate egomotion. The main contributions of our work can be summarized as:

-   The first unsupervised learning-based approach to structure from motion recovery using monocular DVS input.

-   Demonstrating that dense, meaningful scene and motion information can be reliably recovered from sparse event data.

-   A new lightweight high-performance network architecture – $ECN$.

-   A new normalization technique, called *feature decorrelation*, which significantly improves training time and inference quality.

-   Quantitative evaluation on the MVSEC dataset [@MVSEC] of dense and sparse depth, optical flow and egomotion.

-   A pre-processesed MVSEC [@MVSEC] dataset facilitating further research on event-based SfM.

Related Work
============

![*A three-channel DVS data representation. The first channel represents the time image described in [@iROSBetterFlow]. The second and third channels represent the per-pixel positive and negative event counts. Best viewed in color.*](img/representation.png){width="1.0\columnwidth"}

\[fig:representation\]

Event-based Depth Estimation
----------------------------

The majority of event-based depth estimation methods [@zhou2018semi; @UpennSlices] use two or more event cameras. As our proposed approach uses only one event camera, we focus our discussion on monocular depth estimation methods. The first works on event-based monocular depth estimation were presented in [@BMVC2016_63] and [@10.1007/978-3-319-46466-4_21]. Rebecq [*et al*. ]{}[@BMVC2016_63] used a space-sweep voting mechanism and maximization strategy to estimate semi-dense depth maps where the trajectory is known. Kim [*et al*. ]{}[@10.1007/978-3-319-46466-4_21] used probabilistic filters to jointly estimate the motion of the event camera, a 3D map of the scene, and the intensity image. More recently, Gallego [*et al*. ]{}[@Gallego2018AUC] proposed a unified framework for joint estimation of depth, motion and optical flow. So far there has been no deep learning framework to predict depths from a monocular event camera.

Event-based Optical Flow
------------------------

Previous approaches to image motion estimation used local information in event-space. The different methods adapt in smart ways one of the three principles known from frame-based vision, namely correlation [@delbruck_frame_2008; @liu2017block], gradient [@benosman_asynchronous_2012] and local frequency estimation [@tschechne2014event] The most popular approaches are gradient based - namely, to fit local planes to event clouds [@benosman_event_2014] As discussed in [@barranco2014contour], local event information is inherently ambiguous. To resolve the ambiguity Barranco et al. [@barranco2014contour] proposed to collect events over a longer time intervals and compute the motion from the trace of events at contours. Recently, neural network approaches have shown promising results in various estimation problems without explicit feature engineering. Orchard and Etienne-Cummings [@6891162] used a spiking neural network to estimate flow. Most recently, Zhu [*et al*. ]{}[@zhu2018ev] released the MVSEC dataset [@MVSEC] and proposed self-supervised learning algorithm to estimate optical flow. Unlike [@zhu2018ev], which uses grayscale information as a supervision signal, our proposed framework uses only events and thus can work in challenging lighting conditions.

Self-supervised Structure from Motion
-------------------------------------

The unsupervised learning framework for 3D scene understanding has recently gained popularity in frame-based vision research. Zhou et. al [@zhou2017unsupervised] pioneered this line of work. The authors followed a traditional geometric modeling approach and built two neural networks, one for learning pose from single image frames, and one for pose from consecutive frames, which were self-supervised by aligning the frames via the flow. Follow-up works  (e.g. [@DBLP:journals/corr/abs-1802-05522] have used similar formulations with better loss functions and networks, and recently [@UpennSlices] proposed SfM learning from stereo DVS data.

Methods
=======

Ego-motion Model
----------------

We assume that the camera is moving with rigid motion with translational velocity $v = (v_x,v_y,v_z)^T$ and rotational velocity $\omega = (\omega_x, \omega_y,\omega_z)$, and that the camera intrinsic parameters are provided. Let $\mathbf{X}=(X,Y,Z)^T$ be the world coordinates of a point, and $\mathbf{x}= (x,y)^T$ be the corresponding pixel coordinates in the calibrated camera. Under the assumption of rigid motion, the image velocity $\mathbf{u} = (u,v)^T$ at $(x,y)^T$ is given as: $$\resizebox{.9\hsize}{!}{$
\begin{pmatrix}
u\\v
\end{pmatrix}\\
=\frac{1}{Z}
\begin{pmatrix}
-1 &0 &x\\
0 &-1 &y
\end{pmatrix}
\begin{pmatrix}
v_x\\
v_y\\
v_z
\end{pmatrix}+
\begin{pmatrix}
xy &-1-x^2 &y\\
1+y^2 &-xy &-x
\end{pmatrix}
\begin{pmatrix}
\omega_x\\
\omega_y\\
\omega_z
\end{pmatrix}\\
= A \mathbf{p}
$}
\label{uv2}$$ In words, given the inverse depth and the ego-motion velocities $\mathbf{v},\omega$, we can calculate the optical flow or pixel velocity at a point using a simple matrix multiplication (Equation \[uv2\]) Here $\mathbf{p}$ is used to denote the pose vector $(\mathbf{v},\omega)^T$, and $A$ is a $2\times6$ matrix. Due to scaling ambiguity in this formulation, depth $Z$ and translation $(v_x, v_y, v_z)$ are computed up to a scaling factor.

Input Data Representation
-------------------------

The raw data from the DVS sensor is a stream of events, which we treat as data of 3 dimensions. Each event encodes the pixel coordinate $(x, y)$ and the *timestamp* $t$. In addition, it also carries information about its *polarity* - a binary value that disambiguates events generated on rising light intensity (positive polarity) and events generated on falling light intensity (negative polarity).

The 3D $(x, y, t)$ event cloud (within a small time slice), called event slice, is projected onto a plane and converted to a 3-channel image. An example of such image can be seen in Fig. \[fig:representation\]. The two channels of the image are the per-pixel counts of positive and negative events. The third channel is the *time image* as described in [@iROSBetterFlow] - each pixel consists of the average timestamp of the events generated on this pixel, because the averaging of timestamps provides better noise tolerance. The neural network input consists of up to 5 such consecutive slice images to better preserve the timestamp information of the event cloud.

The Pipeline
------------

![**](img/pipeline.png){width=".95\columnwidth"}

\[fig:pipeline\]

The global network structure is similar to the one proposed in [@zhou2017unsupervised]. It consists of a depth prediction component and a parallel pose prediction component, which both feed into a optical flow component to warp successive event slices. The loss from the warping error is backpropagated to train flow, inverse depth, and pose. Our network components, instead of the standard CNNs, are based on our ECN network structure. An (ECN based) encoding-decoding architecture is used to estimate scaled inverse depth $\frac{1}{Z}$ from a single slice of events. To address the data sparsity, we use bilinear interpolation, which propagates local information and fills in the gaps between events. A second network, which takes consecutive slices of signals, is used to derive $v$ and $\omega$. Then, using the rigid motion and inverse depth to predict the optical flow, neighboring slices at neighboring time instances $T+1, T+2$ and $T-1, T-2$ are warped to the slice at $T$ (Fig. \[fig:pipeline\]). The $l^1$ loss is used to measure the difference between the warped events and the middle slice as the supervision signal.

Evenly Cascaded Network Architecture
------------------------------------

We use an encoder network to estimate pose from consecutive frames and an encoder-decoder network [@DBLP:conf/miccai/RonnebergerFB15] (Fig. \[fig:ecn\]) to estimate the scaled depth. Next, we describe its main novelties.

![**](img/ecn_encoder.png){width=".8\columnwidth"}

\[fig:ecn\_encoder\]

![**](img/ecn.png){width="1.\columnwidth"}

\[fig:ecn\]

First, our transform of features takes biological inspiration from multi-stage information distillation, and incorporates feedback. Our encoding layers [@DBLP:conf/bigdataconf/YeDMFA18] split the (layer) input into two streams of features (Fig. \[fig:ecn\]): one incorporates ‘feedback’ via residual learning [@DBLP:journals/corr/HeZRS15] (convolution outputs are added to *downsampled* existing features as modulation signals); the other directly generates a set of higher level features (convolution outputs are directly utilized). At the end of the encoding stage, the network has a multi-scale feature representation. This representation is used in our pose prediction. In each decoding layer, the highest level features, together with the corresponding features in the encoding layers are convolved and added back to the *upsampled* lower level features as modulation. At the end of the decoding stage, the network acquires a set of modulated high resolution low-level features. Our evenly-cascaded (EC) structure facilitates training by providing easily-trainable shortcuts in the architecture. The mapping $f$ from network input to output is decomposed into a series of progressively deeper, and therefore harder to train functions: $f=f_1+f_2+...+f_N$. The leftmost pathway (the green blocks) in Fig. \[fig:ecn\] contains the easiest to train, lowest-level features. A backbone pathway remains ***unblocked*** by only going through downsampling and upsampling, promises the final gradient can be propagated to the first network layer. This construction alleviates the vanishing gradient problem in neural network training. More difficult-to-learn, modulation signals are added to this backbone pathway, allowing the network to selectively enhance and utilize multiple levels of features.

Second, to tackle the challenges raised by sparse event data and evenly resize the features, we use bilinear interpolation. In the encoding layers, our network evenly downscales the feature maps by a scaling factor of $(s < 1)$ to get coarser and coarser features until the feature size falls below a predefined threshold. In the decoding layers, the feature maps are reversely upscaled by a scaling factor of $1/s$. The network construction is automatic and is controlled by the scaling factor. Bilinear interpolation propagates the sparse data spatially, facilitating dense prediction of depth and optical flow.

![**](img/big_pic.png){width="47.50000%"}

\[fig:big\_pic\]

Depth Predictions {#sec:depth_prediction}
-----------------

In the decoding stage, we make predictions from features at different resolutions and levels (Fig. \[fig:ecn\]). Initially, both high and low-level coarse features are used to predict a backbone depth map. The depth map is then upsampled with bilinear interpolation for refinement. Then the enhanced lower level features are used to estimate the prediction residue, which are usually also low-level structures. The residue is added to the backbone estimation to refine it. The final prediction map is therefore obtained through successive upsamplings and refinements. We also apply a smoothness constraint on the depth prediction.

Our pipeline is based on monocular vision and predicts depth up to a scale. In real world driving scenes, the mean depth value stays relatively stable. Taking advantage of this observation, we use batch normalization before making the depth prediction, and this way the predicted depths have similar range. Optionally, we also apply depth normalization [@DBLP:journals/corr/abs-1712-00175] to strictly enforce that the mean depth remains a constant.

Feature Decorrelation
---------------------

Gradient descent training of neural networks can be challenging if the features are not properly conditioned, and features are correlated. Researchers have proposed normalization strategies [@DBLP:journals/corr/IoffeS15; @DBLP:journals/corr/abs-1803-08494] to account for the scale inconsistency problem. We proceed one step further with a decorrelation algorithm to combat the feature collinearity problem. We apply Denman-Beavers iterations [@Denman:1976:MSF:2612816.2613024] to decorrelate the feature channels in a simple and forward fashion. Given a symmetric positive definite covariance matrix $C$, Denman-Beavers iterations start with initial values $Y_0=C$, $Z_0=I$. Then we iteratively compute: $Y_{k+1}=\frac{1}{2}Y_{k}(3I-Z_{k}Y_{k}),Z_{k+1}=\frac{1}{2}(3I-Z_{k}Y_{k})Z_{k}$. We then have $Z_{k}\xrightarrow{} C^{-\frac{1}{2}}$ [@DBLP:journals/corr/LinM17]. In our implementation, we evenly divide the features into 16 groups as proposed in group normalization [@DBLP:journals/corr/abs-1803-08494], and reduce the correlation between the groups by performing a few (1-10) Denman-Beavers iterations. We notice that a few iterations lead to significantly faster convergence and better results.

\[table:main\_table\]

Experimental Evaluation
=======================

Our our self-supervised learning framework can infer both dense optical flow and depth from sparse event data. We evaluate our work on the MVSEC [@MVSEC] event camera dataset which, given a ground truth frequency of 20 Hz, contains over 40000 ground truth images.

The MVSEC dataset, inspired by KITTI [@KITTI1], features 5 sequences of a car on the street (2 during the day and 3 during the night), as well as 4 short indoor sequences shot from a flying quadrotor. MVSEC was shot in a variety of lighting conditions and features low-light and high dynamic range frames which are often challenging for an analysis with classical cameras.

![**](img/norm_plot.png){width="1.0\columnwidth"}

\[fig:normalizations\]

Implementation Details
----------------------

Our standard network architecture has scaling rates of $0.5$ and $2.0$ respectively for the encoding and decoding layers, and results in $5$ encoding/decoding layers. Our depth network has $8$ initial hidden channels and expands with a growth rate of $8$. We halve these settings to $4$ for our pose network. The pose network takes $5$ consecutive event slices and predicts $6d$ pose vectors. We use $3\times 3$ convolutions, and the combined network has $150k$ parameters. We train the network with a batch size of $32$ and use the Adam optimizer with a learning rate of $0.01$. Interestingly, we notice that compared to the standard architecture of the SfMlearner, the learning rate is higher. Thus, the new design allows us to learn at a faster rate. The learning rate is annealed using cosine scheduling, and we let the training run for $30$ epochs. Our training takes $7$-minutes for each epoch using a single Nvidia GTX 1080Ti GPU. Our model using batch normalization runs at 250 FPS at inference as it has been heavily GPU optimized. The model using feature decorrelation runs at 40 FPS. The slow down is mainly due to matrix multiplications in our customized layer which are less optimized for the GPU.

In the experiments with the outdoor sequences, we trained the network using only the *outdoor day 2* sequence with the hood of the car cropped. Our experiments demonstrate that our training generalizes well to the notably different *outdoor day 1* sequence, as well as to the *night* sequences. For the *indoor* sequences, since they were too short to create a representative training set, we used $80\%$ of each sequence for training and evaluated on the remaining $20\%$. We would like to note that the *outdoor night* sequences have occasional errors in the ground truth (see for example Fig. \[fig:big\_pic\] last three rows, or Fig. \[fig:failure\_1\]). All incorrect frames had to be manually removed for the evaluation. In our experiments, we use fixed-width time slices of $1/40$-th of a second.

Ablation Studies
----------------

### Testing on the *SfMlearner*

As baseline we use the state-of-the-art *SfMlearner* [@zhou2017unsupervised] on our data (event images). *SfMlearner* has a fixed structure of 7 encoding and 7 decoding layers. It has 32 initial hidden channels and expands to 512 channels. Overall the model contains 33M parameters. SfMlearner is trained using Adam optimizer with a learning rate of $2e-4$ and a batch size of $4$. We replace the inputs with our event slices, and we include the evaluation results for flow and egomotion in tables \[table:main\_table\] and \[table:main\_table\_egomotion\].

### Normalization Methods

We compare two normalization methods and our decorrelation method on the validation set portion of the outdoor day 2 sequence. We apply $5$ Denman-Beavers iterations in the decorrelation procedure. Compared with normalization methods, decorrelation leads to more thorough data whitening, and we have noticed this layer-wise whitening lead to faster convergence and lower evaluation loss (Fig. \[fig:normalizations\]).

### Visualizing a Shallow and Tiny Network

Our lightweight multi-level, multi-resolution design allows us to construct networks of any depth and size. As a preliminary attempt, we set the scaling rate to $1/3$ and $3.0$ for encoding and decoding layers respectively, so the network has only 3 encoding/decoding layers. As the network is small, we can directly visualize all its internal feature maps. A deeper and wider network would produce a higher quality output but also more feature maps, which we do not list here. In Fig. \[fig:depthnet\_maps\] we have listed all the feature maps in the small depth network. The row number corresponds to the level number of the features for each figure. We notice the encoder seems to play a feature extraction role in the network and the decoder starts to produce semantically meaningful representation corresponding to the desired output (depth). By scrutinizing the pose network outputs, we notice the network is intelligent enough to aggregate information corresponding to different time periods of the events in the first layer (Fig. \[fig:posenet\_maps\]). Otherwise, mixing up the events at different time period would make the pose estimation harder.

\
\
\

### Performance Versus Event Rate

![**](img/plot_aee_ecount.png){width="1.0\columnwidth"}

\[fig:plot\_aee\_ecount\]

Since the event data is inherently sparse, we investigate the performance of the pipeline in relation to the data sparsity.

We measure the data sparsity as a percentage of the pixels on the input images with at least one event. Fig. \[fig:plot\_aee\_ecount\] demonstrates how the event rate is inversely proportional to the average endpoint error for the optical flow (we have observed similar behavior for depth and egomotion). The *outdoor day 1* sequence is used to minimize the influence of the noise.

We find the inverse correlation between event rate and inference quality to be a useful observation, as this property could be efficiently used in sensor fusion in a robotic system. Motivated by that, we provide an additional row to the Table \[table:main\_table\]: $ECN_{erate}$, and report our error metrics once again only for the frames with *higher than average* number of event pixels across the datasets.

Qualitative Results
-------------------

In addition to the quantitative evaluation, we present a number of samples for qualitative analysis in Fig. \[fig:big\_pic\]. The last three rows of the table show the night sequences, and how the pipeline performs well even when only a few events are available. The third and the fourth rows show indoor scenes. The indoor sequences were relatively small and it is highly possible that the quality of the output would increase given a larger dataset.

Optical Flow Evaluation
-----------------------

We evaluate our optical flow results in terms of Average Endpoint Error ($AEE = \frac{1}{n}\sum {\left\lVert\vec{y} - \vec{y}^*\right\rVert}_2$ with $y^*$ and $y$ the estimated and ground truth value, and $n$ the number of events) and compare our results against two state-of-the-art optical flow methods for event-based cameras: $EV-FlowNet$ [@zhu2018ev] and a recent stereo method [@UpennSlices] (in the tables - $Zhu18$).

Because our network produces flow and depth values for every image pixel, our evaluation is not constrained by pixels which did not trigger a DVS event. Still, for consistency reasons, we report both numbers for each of our experiments (for example, $ECN$ and $ECN_{masked}$, where the latter has errors computed only on the pixels with at least one event). Similar to KITTI and EV-FlowNet, we report the percentage of outliers - values with error more than 3 pixels or 5% of the flow vector magnitude.

To compare against [@zhu2018ev] and [@UpennSlices], we account for the difference in the frame rates (for example, EV-FlowNet uses the frame rate of the DAVIS classical frames) by scaling our optical flow. We also provide aggregated results for the indoor scenes (split on a train and a test set 80/20 as described above), although these are not the main focus of our study. Our main results are presented in the Table \[table:main\_table\].

We show that our optical flow performs well during both day and night, all on unseen sequences. The results are typically better for the experiments with event masks except for the *outdoor night*. One possible explanation for that is that this sequence is much noisier with events being generated not only on the edges, which leads to suboptimal masking.

Depth Evaluation
----------------

Since there are currently no monocular event-based methods for the depth estimation based on unsupervised learning, we provide the classical scale-invariant depth metrics, used in many works such as [@DEPTH_COOLGUY], [@zhou2017unsupervised], [@MetricUser]: $$Accuracy:  \% of  y_i\ s.t.\ max(\frac{y_i}{y_i^*}, \frac{y_i^*}{y_i}) = \delta < th
\label{eq:delta}
\vspace{-0.5\baselineskip}$$ $$SILog: \frac{1}{n}\sum d_i^2 - \frac{1}{n^2}(\sum d_i)^2, d_i = \log y_i - \log y_i^*
\label{eq:SILog}
\vspace{-0.5\baselineskip}$$ $$Absolute Relative Difference: \frac{1}{n}\sum \frac{|y - y^*|}{y^*}
\label{eq:AbsRelDiff}
\vspace{-0.5\baselineskip}$$ $$Logarithmic RMSE: \sqrt{\frac{1}{n}\sum {\left\lVert\log y - \log y^*\right\rVert}^2}
\label{eq:RMSElog}$$

Our results are presented in Table \[table:main\_table\_depth\] for both event count-masked depth values and full, dense depth. Since the night driving scenes have similar depth geometries, we aggregate all 3 sequences in one table entry.

Applying an event mask during the evaluation increases accuracy for all scenes - this is expected, as the inference is indeed more accurate on the pixels with event data. On the contrary, the error rate increases on the outdoor scenes and decreases on the indoor scenes. This is probably due to higher variation of the outdoor scenes and also faster motion of the car.

        outdoor day 1   outdoor night   indoor flying
  -- -- --------------- --------------- ---------------
        0.29 (0.33)     0.34 (0.39)     0.28 (0.22)
        0.29 (0.33)     0.38 (0.42)     0.29 (0.25)
        0.12 (0.14)     0.15 (0.18)     0.11 (0.11)
        0.80 (0.97)     0.67 (0.95)     0.75 (0.98)
        0.91 (0.98)     0.85 (0.98)     0.91 (0.99)
        0.96 (0.99)     0.93 (0.99)     0.96 (1.0)

\[table:main\_table\_depth\]

Egomotion Estimation {#sec:egomotion}
--------------------

Our pipeline is capable of inferring egomotion on both day and night sequences, and transfers well from *outdoor day 2* onto *outdoor day 1* and *outdoor night 1,2,3*. Since our pipeline is monocular, we predict the translational component of the velocity up to a scaling factor, while the rotational velocity does not need scaling. Despite our network outputs full 6 degree of freedom velocity, we did not achieve high quality on indoor sequences. This is likely due to highly more complicated motion types and a small size of the indoor dataset. We further discuss this in sec. \[sec:discussion\].

For the driving scenarios we can *make an important observation* - the mean distance of the road in respect to the camera is often a constant. We crop the lower middle part of the inferred depth image and apply a scaling factor such that the mean depth value (corresponding to the road location) is constant. Consequently, only a single extrinsic value (camera height on the car) is needed to reconstruct the scaled motion. In our experiments, we report egomotion with translational scales taken both from ground truth ($AEE_{tr}^{gt}$) and using the depth constancy constraint ($AEE_{tr}^{depth}$), with a global scale taken from ground truth. The qualitative results can be seen in Fig. \[fig:trajectories\].

Unlike [@UpennSlices], we train *SfMlearner* on the event images, and not on the classical frames to allow for evaluation on the night sequences. We provide comparison to the work in [@UpennSlices], although it uses a stereo pipeline and reports results only on the *outdoor day 1* sequence.

To be consistent with [@UpennSlices], we report our trajectory estimation relative pose and relative rotation errors as $RPE = arccos(\frac{t_{pred} \cdot t_{gt}}{\|t_{pred}\|_2 \cdot \|t_{gt}\|_2})$ and $RRE = \|logm(R_{pred}^TR_{gt})\|_2$. Here $logm$ is matrix logarithm and $R$ are Euler rotation matrices. The $RPE$ essentially amounts to the angular error between translational vectors (ignoring the scale), and $RRE$ amounts to the total 3-dimensional angular rotation error. To account for translational scale errors, we report classical Endpoint Errors - computed as a magnitude of the differences in translational component of the velocities. Our quantitative results are presented in Table \[table:main\_table\_egomotion\].

\

[cccccc]{} & & & & &\

& & 3.98 & 0.00267 & 0.70 & 1.29\
& & 7.74 & 0.00867 & - & -\
& & 16.99 & 0.00916 & 1.59 & 2.39\

\

& & 3.90 & 0.00139 & 0.42 & 1.26\
& & 9.95 & 0.00433 & 1.04 & 2.47\
& & 3.44 & 0.00202 & 0.80 & 1.34\
& & 13.51 & 0.00499 & 1.66 & 3.15\
& & 3.28 & 0.00202 & 0.49 & 1.03\
& & 1.053 & 0.00482 & 1.42 & 2.74

Discussion and Failure Cases {#sec:discussion}
----------------------------

A monocular pipeline tends infer more information from the shape of the contours on depth estimation and hence would transfer poorly on completely different scenarios. Nevertheless, we were able to achieve good generalization on night sequences and demonstrate promising results for depth and flow for indoor scenes (trained separately on parts of indoor sequences).

We observe a relatively small angular drift on trajectory estimation (Fig. \[fig:trajectories\]). Despite our model predicting a full 6 degree of freedom motion we admit that in the car scenario only 2 motion parameters play meaningful role and the network may tend to overfit. For this reason, training on the indoor scenes, featuring more complicated motion would require a notably larger dataset than presented in MVSEC. We still achieve results superior to *SfMlearner* and the stereo method [@UpennSlices], while for the comparison with the latter we must attribute some of our success to the fact that our translational velocity prediction is only up to scale. A common shortcoming of event-based sensors in the lack of data when the relative motion is not present. Fig. \[fig:failure\_1\] shows such an example. This issue (although it does not affect egomotion) can be solved only by fusing data from other visual sensors or by moving the event-based sensor continuously. Because of the smoothness constraint used to combat data sparsity, the network tends to blur object boundaries. Still, for the driving environment the contours of obstacles, people and cars are clearly visible, as can be seen in Fig. \[fig:big\_pic\].

![**](img/failure_1.png){width="1.0\columnwidth"}

\[fig:failure\_1\]

![**](img/fast_motion.png){width="1.0\columnwidth"}

\[fig:fast\_motion\]

CONCLUSION
==========

We have presented a novel low-parameter pipeline for generating dense optical flow, depth and egomotion from sparse event camera data. We also have shown experimentally that our new neural network architecture using multi-level features improves upon existing work. Future work will investigate the estimation of moving objects as part of the pipeline, using event cloud representations instead of accumulated events, and the use of space-time frequency representations in the learning.

[^1]: $^{*}$ The authors contribute equally to this work.

[^2]: $^{1}$University of Maryland Institute for Advanced Computer Studies, College Park, MD 20742, USA. [E-mails: cxy@umd.edu, amitrokh@umd.edu, fer@umiacs.umd.edu, yiannis@cs.umd.edu]{}

[^3]: $^{2}$Institute for Physical Science and Technology, University of Maryland, College Park, MD 20742, USA. [E-mail:yorke@umd.edu]{}
