\section{Method: additional details}

\subsection{Scale ambiguity in SFM}\label{s:sfm}

In Sec. 3.2 in the paper, we explain that the scale ambiguity of structure from motion (SFM) causes each reconstruction of a sequence $S^i$ to be known only up to
a global sequence specific scaling factor $\lambda^i$.
Since $\lambda^i$ is not required to learn \ensuremath{\Phi_\text{vp}}, but it is important for depth prediction (as discussed in Sec. 3.3 from the paper), we estimate it as well.

To do so, we note that, given a pair of frames $(t,t^\prime)$ from sequence $S^i$, one can estimate the sequence scale as
$
\lambda^i_{t,t^\prime}
=
\frac
{\| T^i_{t^\prime} - R^i_{t^\prime t} T^i_{t} \|}
{\| \hat T^i_{t^\prime} - R^i_{t^\prime t} \hat T^i_{t} \|}.
$
This expression allows us to conveniently estimate $\lambda^i$ on the fly as a moving average during the SGD iterations used to learn \ensuremath{\Phi_\text{vp}}, as samples $\lambda^i_{t,t^\prime}$ can be computed essentially for free during this process.

\subsection{The VpDR-Net\xspace architecture: further details}\label{s:arch}

\begin{figure*}
\centering
\includegraphics[width=\linewidth]{figures/network-hc.pdf}
\caption{\textbf{The core architecture of VpDR-Net\xspace.} This figure extends the Viewpoint \& Depth estimation block from Figure 2 in the paper and describes the architecture of the hypercolumn (HC) module. \label{fig:overview-hc}}
\end{figure*}

This section contains additional details about the layers that compose the VpDR-Net\xspace architecture.

\myparagraph{The core architecture}
The architecture of the VpDR-Net\xspace (introduced in Sec. 3.2 from the paper) is a variant of the ResNet-50 architecture~\cite{he16resnet} with some modifications to improve its performance as a viewpoint and depth predictor that we detail below.

In order to decrease the degree of geometrical invariance of the network, we first replace all $1\times 1$ downsampling filters with full $2\times 2$ convolutions. We then attach bilinear upsampling layers that first resize features from 3 different layers of the architecture (res2d, res3d, res4d) into fixed-size tensors and then sum them in order to create a multiscale intermediate image representation which resembles hypercolumns (HC) \cite{hariharan2015hypercolumns}. An extension of Fig. 2 from the paper that contains the diagram of this HC module can be found in \Cref{fig:overview-hc}.

\myparagraph{Architecture of the viewpoint factorization network \ensuremath{\Phi_\text{vp}}}
HC is followed by 3 modified $3\times 3$ downsampling residual layers that produce the final viewpoint prediction. While the standard downsampling residual layers do not contain the residual skip connection due to different sizes of the input and output tensors, here we retain the skip connection by performing $3\times 3$ average pooling over the input tensor and summing the result with the result of the second $3\times 3$ downsampling convolution branch. We further remove the ReLU after the final residual summation layer. \Cref{fig:overview-vp} contains an overview of the viewpoint estimation module together with a detailed illustration of the modified downsampling residual blocks.

\myparagraph{Architecture of the depth prediction  \ensuremath{\Phi_\text{depth}}}
The depth prediction network (introduced in Sec. 3.3 from the paper) shares the early HC layers with the viewpoint factorization network \ensuremath{\Phi_\text{vp}}. The remainder of the pipeline is based on the state-of-the-art depth estimation method of \cite{laina2016deeper}. More precisely, after attaching 2 standard residual blocks to the HC layers, the network also contains
two 2x2 up-projection layers from \cite{laina2016deeper} leading to a 64-dimensional representation of the same size as the input image. This is followed by 1x1 convolutional filters that predict the depth and confidence maps $\hat D_t$ and $\hat \sigma_{d_j}$ respectively.
\Cref{fig:overview-depth} contains an illustration of \ensuremath{\Phi_\text{depth}}.

\myparagraph{Architecture of the point cloud completion network \ensuremath{\Phi_\text{pcl}}}
Differently from the two previous networks, the point cloud completion network $\ensuremath{\Phi_\text{pcl}}$ (introduced in Sec. 3.4 from the paper) is not convolutional but uses a residual multi-layer perceptron (MLP), \ie a sequence of residual fully connected layers.

In more details, the network starts by appending to each 3D point $\hat p_i \in \hat P^G_f \subset \mathbb{R}^3$ an appearance descriptor $a_i$ and processes
this input with an MLP with an intermediate pooling operator:
$$
(\hat S, \hat \delta) =
\ensuremath{\Phi_\text{pcl}}(\hat P^G_f) =
  \operatorname{MLP}_2
  \left(
  \operatornamewithlimits{pool}_{1\leq i \leq |\hat P^G_f|}
  \operatorname{MLP}_1(\hat p_i, a_i)
  \right).
$$
The intermediate pooling operator, which is permutation invariant, removes the dependency on the number and order of input points $\hat P_f^G$. In practice, the pooling operator uses both max and sum pooling, stacking the results of the two.

For the appearance descriptors, recall that each point $\hat p_i$ is the back-projection of a certain pixel $(u_i,v_i)$ in image $f$. To obtain the appearance descriptor $a_i$ we reuse the HC features from the core architecture and sample a column of feature channels at location $(u_i,v_i)$ using differentiable bilinear sampling. Note that, following \cite{tatarchenko16multi}, the fully connected residual blocks contain leaky-ReLUs with the leak factor set to 0.2. A diagram depicting \ensuremath{\Phi_\text{pcl}}{} can be found in \Cref{fig:overview-pcl}.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/figureB_supp.pdf}
\caption{\textbf{The architecture of \ensuremath{\Phi_\text{depth}}.} \label{fig:overview-depth}}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/figureC_supp.pdf}
\caption{\textbf{The architecture of \ensuremath{\Phi_\text{vp}}.} Top: the layers of \ensuremath{\Phi_\text{vp}}, bottom: A detail of the 3x3 downsampling residual block.\label{fig:overview-vp}}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{figures/figureD_supp.pdf}
\caption{\textbf{The architecture of \ensuremath{\Phi_\text{pcl}}.} Top: The overview of the point cloud completion network, bottom: A detail of the fully connected residual block.
Orange boxes denote the sizes of the layer outputs. \label{fig:overview-pcl}}
\end{figure}

\section{Experimental evaluation}

In this section we provide additional details about the learning procedures of the baseline networks
and about the experimental evaluation.

\subsection{Learning details of BerHu-Net\xspace and VPNet}
In this section we provide learning details for the BerHu-Net and VPNet baselines.
The learning rates and batch sizes were in all cases adjusted empirically such that the convergence is achieved on the
respective training sets.

\textbf{BerHu-Net\xspace} is trained with stochastic gradient descent with a momentum of 0.0005, initial learning rate $10^{-3}$ and a batch size of 16.
The learning rate was lowered tenfold when no further improvement
in the training losses was observed. The BerHu loss
uses the adaptive adjustment of the loss cut-off threshold as explained in \cite{laina2016deeper}.
For the 2x2 up-projection layers we used the implementation of \cite{laina2016deeper}.
For each test image, we repeat the depth map extraction 70 times\footnote{We empirically verified that 70 repetitions are enough for convergence of the variance estimates.}
with the dropout layer turned on and compute the variance of the predictions in order to obtain the per-pixel depth confidence values.
The final feed-forward pass turns off the dropout layer and produces the actual depth predictions.

\textbf{VPNet} is trained with stochastic gradient descent with a momentum of 0.0005, initial learning rate $10^{-2}$ and a batch size of 128.
The learning rate was lowered tenfold when no further improvement
in the training losses was observed.
For VPNet trained on aligned FrC, we adjusted the produced bounding box and viewpoint annotations in the same fashion as
done for adjusting the Pascal3D annotations in sec. 5.1. in the paper,
ensuring that the aligned FrC dataset is as compatible as possible with the target Pascal3D dataset.
For LDOS, the produced dataset was adjusted in the same way except that we did not use the bounding boxes predicted by \cite{sedaghat15unsupervised}
because the input video frames already focus on full/truncated views of the object category.

\subsection{Additional results}
In sec. 5.1. in the paper we compared VpDR-Net\xspace to
\cite{sedaghat15unsupervised} on an adjusted version of the Pascal3D dataset. In this section, we additionally report the standard
AVP measure \cite{xiang2014beyond} on the original Pascal3D dataset in order to present a better comparison
with fully supervised state-of-the-art on this dataset. Because the AVP measure requires an object detector,
we extract viewpoints from the same set of RCNN detections as in \cite{tulsiani2015viewpoints}.
Due to the fact that the AVP measure, as well as most other measures from sec. 5.1. in the paper, depends on the dataset-specific
global alignment transformation $\mathcal{T}_G$, we estimate it from the ground truth annotations
of the training set of \cite{xiang2014beyond} using the same method as described in sec. 5.1. in the paper.

Due to the additional measurement noise brought by the estimation of $\mathcal{T}_G$, we report results only
for the coarsest resolution of 4 azimuth bins. Our VpDR-Net\xspace obtained 33.4 and 14.7 AVP for the car and chair classes
vs. 29.4 and 14.3 AVP obtained by \cite{sedaghat15unsupervised} using the same detections from \cite{tulsiani2015viewpoints}.
Our approach performs on par with some fully supervised approaches such as 3D DPM \cite{pepik2012teaching},
while being inferior to the fully supervised state-of-the-art by the same margin as for the other metrics reported in table 1 in the paper.

\subsection{Absolute pose evaluation protocol}
As noted in the paper, the absolute pose error metrics $e_R$ and $e_C$
can be computed only after aligning the implicit global coordinate frames of the benchmarked network
and of the ground truth annotations. This procedure is explained in detail below.

Given a set of ground truth camera poses $g_i^\ast\ = (R_i^\ast,T_i^\ast)$
and the corresponding predictions  $\hat g_i = ( \hat R_i,\hat T_i)$,
we want to estimate a global similarity transform $\mathcal{T}_G = (R_G,T_G,s_G)$, parametrized by a
scale $s_G \in \mathbb{R}$, translation $T_G \in \mathbb{R}^3$ and rotation $R_G \in SO(3)$, such that
the coordinate frames of $g_i^\ast$ and $\hat g_i$ become aligned.

In more detail, the desired global similarity transform
satisfies the following equation:
\begin{equation}
\hat R_i ( R_G X + T_G ) + s_G \hat T_i = R_i^\ast X + T_i^\ast ~ ; ~ \forall X
\label{eq:globadjust}
\end{equation}
\ie given an arbitrary world-coordinate point $X \in \mathbb{R}^3$, its projection into the coordinate frame of $g_i^\ast$ (the right part of \cref{eq:globadjust})
should be equal to the projection of $X$ into the coordinate frame of $\hat g_i$ after transforming $X$ with $R_G$, $T_G$ and
scaling the corresponding camera translation vector $\hat T_i$ with $s_G$ (the left side of \cref{eq:globadjust}).
Note that for LDOS data $\mathcal{T}_G$ corresponds to a rigid motion and $s_G=1$.
Given $\mathcal{T}_G$, the adjusted camera matrices $\hat g_i^{ADJUST}$ for which $\hat g_i^{ADJUST} \approx g_i^\ast$
are then computed with
$$
\hat g_i^{ADJUST} = ( ~~ \hat R_i R_G ~~ ,  ~~\hat R_i T_G + s_G \hat T_i ~~ )
$$

In order to estimate $\mathcal{T}_G$, $X$ is substituted in \cref{eq:globadjust} with $X = C_i^\ast = -{R_i^\ast}^T T_i^\ast$,
\ie $X$ is set to be the center of the ground truth camera $g_i^\ast$ which is a valid point of the world coordinate frame.
After performing some additional manipulations, we end up with the following constraint:
\begin{equation}
\frac{1}{s_G} R_G C_i^\ast + \frac{1}{s_G} T_G = \hat C_i ~ ; ~ \forall i
\label{eq:umeyama}
\end{equation}
where $\hat C_i = - \hat R_i^T \hat T_i$ is the center of the predicted camera $\hat g_i$.
Given the corresponding camera pairs $ \{ (g_i^\ast,\hat g_i) \}_{i=1}^N $
the constraint in \cref{eq:umeyama} is converted to a least squares minimization problem:
\begin{equation}
\arg\min_{R_G,T_G,s_G} \sum_{i=1}^N || \frac{1}{s_G} R_G C_i + \frac{1}{s_G} T_G - \hat C_i ||^2
\label{eq:umeyama2}
\end{equation}
and solved using the UMEYAMA algorithm \cite{umeyama1991least}.

For Pascal3D we estimate $\mathcal{T}_G$ from the held-out training set
and later use it for evaluation on the test set.
For LDOS, due to the absence of a held-out annotated training set,
we estimate $\mathcal{T}_G$ on the test set.

\subsection{Point cloud prediction}
The normalized point cloud distance of~\cite{rock2015completing} is computed as
$
D_\text{pcl}(C,\hat C) =
\frac{1}{|C|}
\sum_{c \in C}
\min_{\hat c \in \hat C}
\| \hat c - c \|
+
\frac{1}{|\hat C|}
\sum_{\hat c \in \hat C}
\min_{c \in C} \|\hat c - c\|.
$
For the VIoU measure, a voxel grid is setup around each ground truth point-cloud $C$ by uniformly subdividing $C$'s  bounding volume into $30^3$ voxels.

The point clouds are compared within the local coordinate frames of each frame's camera (whose focal length is assumed to be known). Furthermore, since the
SFM reconstructions are known only up to a global scaling factor, we adjust each point cloud prediction $\hat C$ from the FrC dataset by multiplying it with
a scaling factor $\zeta$ that aligns the means of $\hat C$ and $C$. Note that $\zeta$ can be computed analytically with:
$$
\zeta = \frac{\mu_{C}^T \mu_{\hat C} } { \mu_{\hat C}^T \mu_{\hat C} },
$$
where $\mu_{C} = \frac{1}{|C|} \sum_{c_m \in C} c_m $ is the centroid of the point cloud $C$.

\myparagraph{Ablative study} In table 2 in the paper, we have presented a comparison of VpDR-Net\xspace to the baseline approach from \cite{aubry14seeing}.
Here we provide an additional ablative study that evaluates the contribution of the components of $\ensuremath{\Phi_\text{pcl}}$.
More exactly, \cref{tab:completion} extends table 2 from the paper with the following flavours of VpDR-Net\xspace:
(1) VpDR-Net\xspace-$\hat P_f$ which only predicts the partial point cloud $P_f$,
(2) VpDR-Net\xspace-Chamfer which removes the density predictions $\hat \delta$ and replaces $l_{pcl}(\hat S)$ with a Chamfer distance loss and
(3) VpDR-Net\xspace-$\hat S$ that predicts the raw unfiltered and untruncated point cloud $\hat S$.

The drops in performance by predicting solely the raw and partial point clouds $\hat P_f$ and $\hat S$ emphasize the importance of the point cloud completion and density prediction components respectively. The Chamfer distance loss brings marginal improvements in $D_{pcl}$ but a significant decrease of VIoU due to the inability of the network to represent and discard outliers.

\begin{table}
\centering
\footnotesize
\setlength\tabcolsep{2pt}
\begin{tabular}{lrrrr}
\toprule
Test set                          & \multicolumn{2}{c}{\textbf{LDOS}} & \multicolumn{2}{c}{\textbf{FrC}} \\
\midrule
Metric                            &  $\uparrow$ mVIoU    & $\downarrow$ m$D_{pcl}$  & $\uparrow$ mVIoU     & $\downarrow$ m$D_{pcl}$   \\
\midrule
Aubry \cite{aubry14seeing}       & 0.06      & 1.30        & 0.21      & 0.41                 \\
VpDR-Net\xspace-$\hat P_f$     & 0.10      & 0.37        & 0.11      & 0.56 \\
VpDR-Net\xspace-Chamfer        & 0.09	     & \textbf{0.18}	       & 0.20	   &\textbf{0.24} \\
VpDR-Net\xspace-$\hat S$       & 0.12      & 0.27        & 0.18      & 0.50                 \\
\textbf{VpDR-Net\xspace (ours) }         & \textbf{0.13}      & 0.20        & 0.24      & 0.28                 \\
\textbf{VpDR-Net\xspace-Fuse (ours)}    & \textbf{0.13}      & 0.19        & \textbf{0.26}      & 0.26                 \\
\bottomrule
\end{tabular}
\caption{ \textbf{Point cloud prediction ablative study}.
Comparison between VpDR-Net\xspace and the method of Aubry \etal \cite{aubry14seeing} and an additional ablative study.}
\label{tab:completion}
\vspace{-2em}
\end{table}

\myparagraph{Related methods} Note that apart from \cite{aubry14seeing}, there exist newer works that tackle the problem of single-view
3D reconstruction \cite{huang2015single,massa2016deep}, however these were not considered due to their requirement of renderable
mesh models which are not available in our supervision setting.

\section{Conclusion}\label{s:conclusions}

We have demonstrated the power of motion cues in replacing manual annotations and synthetic data in learning 3D object categories. We have done so by proposing a single neural network that simultaneously performs monocular viewpoint estimation, depth estimation, and shape reconstruction. This network is based on two innovations, a new image-based viewpoint factorization method and a new probabilistic shape representation. The contribution of each component was assessed against suitable baselines.
\section{Experiments}\label{s:exp}

We assess viewpoint estimation in~\cref{s:exp-pose}, depth prediction in~\cref{s:exp-depth}, and point cloud prediction in~\cref{s:exp-pcc}.

\myparagraph{Datasets} Throughout the experimental section, we consider three datasets for training and benchmarking our network: (1) \textbf{FreiburgCars (FrC)}~\cite{sedaghat15unsupervised}  which consists of RGB video sequences with the camera circling around various types of cars; (2) the \textbf{Large Dataset of Object Scans (LDOS)}~\cite{choi2016large} containing RGBD sequences of man-made objects; and (3) \textbf{Pascal3D}~\cite{xiang2014beyond}, a standard benchmark for pose estimation~\cite{tulsiani2015viewpoints,su2015render}.

For viewpoint estimation, Pascal3D already contains viewpoint annotations. For LDOS, experiments focus on the \textit{chair} class. In order to generate ground truth pose annotations for evaluation, we manually aligned 3D reconstructions of 10 randomly-selected chair videos and used 50 randomly-selected frames for each video as a test set.

For depth estimation, we evaluate on LDOS as it provides high quality depth maps one can use as ground truth.

For point cloud reconstruction, we use FrC and LDOS. Ground truth point clouds for evaluation are obtained by merging the SFM or RGBD depth maps from all frames of a given test video sequence, sampling $3\cdot10^4$ points and post-processing those using a 3D Laplacian filter. For FrC, five videos were randomly selected and removed from the train set, picking 60 random frames per video for evaluation. For LDOS the pose estimation test frames are used.

\myparagraph{Learning details} VpDR-Net\xspace is trained with stochastic gradient descent with a momentum of 0.0005 and an initial learning rate of $10^{-2}$. The weights of the losses were empirically set to achieve convergence on the training set.
Better convergence was observed by training VpDR-Net\xspace in two stages. First, $\ensuremath{\Phi_\text{depth}}$ and $\ensuremath{\Phi_\text{vp}}$ were optimized jointly, lowering the learning rate tenfold when no further improvement in the training losses was observed. Then, $\ensuremath{\Phi_\text{pcl}}$ is optimized after initializing the bias of its last layer, which corresponds to an average point cloud of the object category, by randomly sampling points from the ground truth models.

\subsection{Pose estimation}\label{s:exp-pose}

\myparagraph{Pascal3D} First, we evaluate the VpDR-Net\xspace viewpoint predictor on the Pascal3D benchmark \cite{xiang2014beyond}. Unlike previous works \cite{su2015render,tulsiani2015viewpoints} that focus on estimating the object/camera viewpoint represented by a 3 DoF rotation matrix, we evaluate the full 6 DoF camera pose represented by the rotation matrix $R$ together with the translation vector $T$.

In Pascal3D, the camera poses are expressed relatively to the whole scenes instead of the objects themselves, so we adjust the dataset annotations.
We crop every object using bounding box annotations after reshaping the box to a fixed aspect ratio, and resize the crop to $240\times 320$ pixels. The camera pose is adjusted to the cropped object using the P3P algorithm to minimize the reprojection error between the camera-projected vertices of the ground truth CAD model and the original projection after cropping and resizing.

\myparagraph{Absolute pose evaluation} We first evaluate absolute camera pose estimation using two standard measures: the angular error
$
e_R = 2^{-\frac{1}{2}}
\|\ln R^\ast \hat R^\top \|_F
$
between the ground truth camera pose $R^\ast$ and the prediction $\hat R$ \cite{tulsiani2015viewpoints,su2015render}, as well as the camera-center distance
$
e_C = \|\hat C - C^\ast\|_2
$
between the predicted camera center $\hat C$ and the ground truth $C^\ast$. Following the common practice \cite{tulsiani2015viewpoints,su2015render} we report median $e_R$ and $e_C$ over all pose predictions on each test set.

Note that, while object viewpoints in Pascal3D and our method are internally consistent for a whole category, they may still differ between them by an arbitrary global 3D similarity transformation. Thus, as detailed in the supplementary material, the two sets of annotations are aligned by a single global similarity $\mathcal{T}_G$ before assessment.

\myparagraph{Relative pose evaluation} To assess methods with measures independent of $\mathcal{T}_G$ we also evaluate: (1) the relative rotation error between pairs of ground truth relative camera motions $R_{t t^\prime}^\ast$ and the corresponding predicted relative motions $\hat R_{t t^\prime}$ given by
$
e_R^\text{rel} =
2^{-\frac{1}{2}}
\|\ln
R_{t t^\prime}^\ast
\hat R_{t t^\prime}^\top
\|_F
$
and (2) the normalized relative translation error
$
e_T^\text{rel}
= \|
\hat T_{t t^\prime} - T_{t t^\prime}^\ast
\|_2
$%
, where both $\hat T_{t t^\prime}$ and $T_{ t t^\prime}^\ast$ are $\ell_2$-normalized so the measure is invariant to the scaling component of $\mathcal{T}_G$. We report the median errors over all possible image pairs in each test set.

\begin{figure}[t]
	\centering
   \pbox{\textwidth}{
   \includegraphics [width=0.47\linewidth] {figures/depthRMSDepthSortchart.pdf}
   \includegraphics [width=0.48\linewidth] {figures/depthRMSConfSortchart.pdf}\\
   \includegraphics [width=1\linewidth] {figures/depthLegend.pdf}
   }
  \caption{\textbf{Monocular depth prediction.} Cumulative RMS depth reconstruction error for the LDOS data, when pixels are ranked by ground truth depth (left) and by confidence (right).}
  \label{fig:depthest}
\end{figure}

\begin{figure}[t]
\newcommand{0.84cm}{0.84cm}
\centering
\includegraphics[height=0.84cm]{figures/depth/car/00002_image_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00001_image_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00006_image_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_image_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_image_2_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_image_3_small.png}\\
\includegraphics[height=0.84cm]{figures/depth/car/00002_depth_pred_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00001_depth_pred_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00006_depth_pred_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_pred_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_pred_2_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_pred_3_small.png}\\
\includegraphics[height=0.84cm]{figures/depth/car/00002_depth_confidence_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00001_depth_confidence_small.png}
\includegraphics[height=0.84cm]{figures/depth/car/00006_depth_confidence_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_confidence_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_confidence_2_small.png}
\includegraphics[height=0.84cm]{figures/depth/chair/00001_depth_confidence_3_small.png}\\
\caption{\textbf{Monocular depth prediction.} Top: input image; middle: predicted depth; bottom: predicted depth confidence.
Depth maps are filtered by removing low confidence pixels.
\label{fig:depth_qual}}
\vspace{-1em}
\end{figure}

\myparagraph{Pose prediction confidence evaluation} A feature of our model is to produce confidence scores with its viewpoint estimates. We evaluate the reliability of these scores by correlating them with viewpoint prediction accuracy. In order to do so, predictions are divided into ``accurate'' and ``inaccurate'' by comparing their errors $e_R$ and $e_C$ to thresholds (set to $e_R=\frac{\pi}{6}$ following~\cite{su2015render,tulsiani2015viewpoints} and $e_C=15$ and $0.5$ for Pascal3D or LDOS respectively). Predictions are then ranked by decreasing confidence scores and the average precisions $AP_{e_R}$ and $AP_{e_C}$ of the two ranked lists are computed.

\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figures/pcls.pdf}
\caption{\textbf{Point cloud prediction.} From a single input image of an unseen object instance (top row), VpDR-Net\xspace predicts the 3D geometry of that instance in the form of a 3D point cloud (seen from two different angles, middle and bottom rows).\label{fig:pcl_qual}}
\vspace{-1em}
\end{figure*}

\myparagraph{Baselines} We compare our viewpoint predictor to a strong baseline, called~\textbf{VPNet}, trained using absolute viewpoint labels. VPNet is a ResNet50 architecture \cite{he16resnet} with the final softmax classifier replaced by a viewpoint estimation layer that predicts the 6 DoF pose $\hat g_t^i$. Following \cite{tulsiani2015viewpoints}, rotation matrices are decomposed in Euler angles, each discretized in 24 equal bins.
This network is trained to predict a softmax distribution over the angular bins and to regress a 3D vector corresponding to the camera translation $T$. The average softmax value across the three max-scoring Euler angles is used as a prediction confidence score.

We test both an unsupervised and a fully-supervised variant of VPNet. VPNet-unsupervised is comparable to our setting and is trained on the output of the global camera poses estimated from the videos by the state-of-the-art sequence-alignment method of \cite{sedaghat15unsupervised}. In the fully-supervised setting, VPNet is trained instead by using ground-truth global camera poses provided by the Pascal3D training set.

\myparagraph{Results} \Cref{tab:posest} compares VpDR-Net\xspace to the VPNet baselines. First, we observe that our baseline VPNet-unsupervised is very strong, as we report $e_R=49.6$ error for the full rotation matrix, while the original method of \cite{sedaghat15unsupervised} reports an error of 61.5 just for the azimuth component. Nevertheless, VpDR-Net\xspace outperforms VPNet in all performance metrics except for a single case ($e_R$ for LDOS chairs). Furthermore, the advantage is generally substantial,  and the unsupervised VpDR-Net\xspace reduces the gap with fully-supervised VPNet by 20 \% or better in the vast majority of the cases. This shows the advantage of the proposed viewpoint factorization method compared  to aligning 3D shapes as in~\cite{sedaghat15unsupervised}. Second, we observe that the confidence scores estimated by VpDR-Net\xspace are significantly more correlated with the accuracy of the predictions than the softmax scores in VPNet, providing a reliable self-assessment mechanism. The most confident viewpoint predictions of VpDR-Net\xspace are shown in \cref{fig:vp_qual}.

\myparagraph{Ablation study} We evaluate the importance of the different components of VpDR-Net\xspace by turning them off and measuring performance on the \textit{chair} class. In \cref{tab:ablation}, \textbf{VpDR-Net\xspace{}-NoProb} replaces the robust probabilistic losses $\mathcal{L}_R$ and $\mathcal{L}_T$ with their non-probabilistic counterparts $\ell_R$ and $\ell_T$, and confidence predictions are replaced with random scores for AP evaluation. \textbf{VpDR-Net\xspace{}-NoDepth} removes the depth prediction and point cloud prediction branches during training, retaining only the $\ensuremath{\Phi_\text{vp}}$ subnetwork. \textbf{VpDR-Net\xspace{}-NoAug} does not use the data augmentation mechanism of \cref{s:augmentation}.

We observe a significant performance drop when each of the components is removed. This confirms the importance of all contributions in the network design.
Interestingly, we observe that the depth prediction branch $\ensuremath{\Phi_\text{depth}}$ is  crucial for pose estimation (\eg~-34.27 $e_R$ on LDOS).

\input{tab_completion}

\subsection{Depth prediction}\label{s:exp-depth}

The monocular depth prediction module of VpDR-Net\xspace is compared against three baselines: \textbf{VpDR-Net\xspace-Rand} uses VpDR-Net\xspace to estimate depth but predicts random confidence scores. \textbf{BerHu-Net\xspace} is a variant of the state-of-the-art depth prediction network from \cite{laina2016deeper} based on the same $\ensuremath{\Phi_\text{depth}}$ subnetwork as VpDR-Net\xspace (but dropping $\ensuremath{\Phi_\text{pcl}}$ and $\ensuremath{\Phi_\text{vp}}$). Following \cite{laina2016deeper}, for training it uses the BerHu depth loss and a dropout layer, which allows it to produce a confidence score of the depth measurements at test time using the sampling technique of~\cite{kendall2015bayesian,gal2016Bayesian}. Finally, \textbf{BerHu-Net\xspace-Rand} is the same network, but predicting random confidence scores.

\myparagraph{Results} Fig.~\ref{fig:depthest} (right) shows the cumulative root-mean-squared (RMS) depth reconstruction error for LDOS after sorting pixels by their confidence as estimated by the network. By fitting better to inlier pixels and giving up on outliers, VpDR-Net\xspace produces a much better estimate than alternatives for the vast majority of pixels. Furthermore, accuracy is well predicted by the confidence scores. Fig.~\ref{fig:depthest} (left) shows the cumulative RMS by depth, demonstrating that accuracy is better for pixels closer to the camera, which are more likely to be labeled with correct depth. Qualitative results are shown in \cref{fig:depth_qual}.

\subsection{Point cloud prediction}\label{s:exp-pcc}

We evaluate the point cloud completion module of VpDR-Net\xspace by comparing ground truth point clouds $C$ to the point clouds $\hat C$ predicted by $\ensuremath{\Phi_\text{pcl}}$ using: (1) the voxel intersection-over-union (VIoU) measure that computes the Jaccard similarity between the volumetric representations of $\hat C$ and $C$, and (2) the normalized point cloud distance of~\cite{rock2015completing}. We average these measures over the test set leading to mVIoU and m$D_{pcl}$ (see supp. material for details).

VpDR-Net\xspace is compared against the approach of Aubry~\etal~\cite{aubry14seeing} using their code. \cite{aubry14seeing}
is a 3D CAD model retrieval method which first trains a large number of exemplar models which, in our case, are represented by individual video frames with their corresponding ground truth 3D point clouds.
Then, given a testing image, \cite{aubry14seeing} detects the object instance
and retrieves the best matching model from the database. We align the retrieved point cloud to the object location in the testing image using the P3P algorithm.
For VpDR-Net\xspace, we evaluate two flavors. The original VpDR-Net\xspace that predicts the point cloud $\hat C$ and VpDR-Net\xspace-Fuse which further merges $\hat C$ with the predicted partial depth map point cloud $\hat P$.

\Cref{tab:completion} shows that our reconstructions are significantly better on both metrics for both LDOS chairs and FrC cars. Fusing the results with the original depth map produces a denser point cloud estimate and marginally improves the results.
Qualitative results are shown in \cref{fig:pcl_qual}.

\section{Introduction}

Despite their tremendous effectiveness in tasks such as object category detection, most deep neural networks do not understand the 3D nature of object categories. Reasoning about objects in 3D is necessary in many applications, for physical reasoning, or to understand the geometric relationships between different objects or scene elements.

The typical approach to learn 3D objects is to make use of large collections of high quality CAD models such as \cite{shapenet2015} or \cite{xiang2016objectnet3d}, which can be used to fully supervise models to recognize the objects' viewpoint and 3D shape. Alternatively, one can start from standard image datasets such as PASCAL VOC
\cite{Everingham10}, augmented with other types of supervision, such as object segmentations and keypoint annotations~\cite{carreira16lifting}. Whether synthetically generated or manually collected, annotations have so far been required in order to overcome the significant challenges of learning 3D object categories, where both viewpoint and geometry are variable.

In this paper, we develop an alternative approach that can learn 3D object categories in an \emph{unsupervised manner} (\cref{fig:front}), replacing synthetic or manual supervision with \emph{motion}. Humans learn about the visual word by experiencing it continuously, through a variable viewpoint, which provides very strong cues on its 3D structure. Our goal is to build on such cues in order to learn the 3D geometry of object categories, using videos rather than images of objects. We are motivated by the fact that videos are almost as cheap as images to capture, and do not require annotations

\begin{figure}[t]
\vspace{-1em}
\includegraphics[width=\linewidth,trim=0 1em 0 1em]{figures/splash/splashv2.pdf}
\caption{We propose a convolutional neural network architecture to learn the 3D geometry of object categories from videos only, without manual supervision. Once learned, the network can predict i)~viewpoint, ii) depth, and iii) a point cloud, all from a single image of a new object instance.\label{fig:front}}
\end{figure}

\begin{figure*}
\includegraphics[width=\linewidth]{figures/network.png}
\caption{\textbf{Overview of our architecture.} As a preprocessing, structure from motion (SFM) extracts egomotion and a depth map for every frame. For training, our architecture takes pairs of frames $f_t$, $f_{t'}$ and produces a viewpoint estimate, a depth estimate, and a 3D geometry estimate. At test time, viewpoint, depth, and 3D geometry are predicted from single images.\label{fig:overview}}
\end{figure*}

We build on mature structure-from-motion (SFM) technology to extract 3D information from individual video sequences. However, these cues are specific to each object instance as contained in different videos. The challenge is to integrate this information in a global 3D model of the object category, as well as to work with noisy and incomplete reconstructions from SFM.

We propose a new deep architecture composed of three modules (\cref{fig:overview}). The first module estimates the \emph{absolute viewpoint} of objects in all video sequences (\cref{s:viewpoint}). This aligns different object instances to a common reference frame where geometric relationships can be modeled more easily. The second estimates the 3D shape of an object from a given viewpoint, producing a \emph{depth map} (\cref{s:depth}). The third \emph{completes the depth map to a full 3D reconstruction} in the globally-aligned reference frame (\cref{s:recon}).
Combined and trained end-to-end without supervision, from videos alone, these components constitute VpDR-Net\xspace,
a network for \textbf{v}iew\textbf{p}oint, \textbf{d}epth and \textbf{r}econstruction,
capable of extracting viewpoint and shape of a new object instance from a single image.

One of our main contributions is thus to demonstrate the utility of using motion cues in learning 3D categories. We also introduce two significant technical innovations in the viewpoint and shape estimation modules as well as design guidelines and training strategies for 3D estimation tasks.

The first innovation (\cref{s:viewpoint}) is a new approach to align video sequences of different 3D objects based on a \emph{Siamese viewpoint factorization network}. While existing methods~\cite{sun09multi,sedaghat15unsupervised} align shapes by looking at 3D features, we propose to train VpDR-Net\xspace to directly estimate the absolute viewpoint of an object. We train our network to reconstruct \emph{relative camera motions}  and we show that this implicitly aligns different objects instances together. By avoiding explicit shape comparisons in 3D space, this method is simpler and more robust than alternatives.

The second innovation (\cref{s:recon}) is a new network architecture that can generate a complete point cloud for the object from a partial reconstruction obtained from monocular depth estimation. This is based on a shape representation that predicts the support of a point probability distribution in 3D space, akin to a flexible voxelization, and a corresponding space occupancy map.

As a general design guideline, we demonstrate throughout the paper the utility of allowing deep networks to \emph{express uncertainty} in their estimate by predicting probability distributions over outputs (\cref{s:method}), yielding more robust training and useful cues (such as separating foreground and background in a depth map). We also demonstrate the significant power of \emph{geometry-aware data augmentation}, where a deep network is used to predict the geometry of an image and the latter is used to generate new realistic views to train other components of the system (\cref{s:augmentation}). Each component and design choice is thoroughly evaluated in~\cref{s:exp}, with significant improvements over the state-of-the-art.
\section{Geometry-aware data augmentation}\label{s:augmentation}

As viewpoint prediction with deep networks benefits significantly from large training sets~\cite{su2015render}, we increase the effective size of the training videos by \emph{data augmentation}. This is trivial for tasks such as classification, where one can translate or scale an image without changing its identity. The same is true for viewpoint recognition if the task is to only estimate the viewpoint orientation as in~\cite{su2015render,tulsiani2015viewpoints}, as images can be scaled and translated without changing the equivalent viewpoint orientation. However, this assumption is not satisfied if, as in our case, the goal is to estimate all 6 DoF of the camera pose.

Inspired by the approach of~\cite{gupta2016synthetic}, we propose to solve this problem by using the estimated scene geometry to \emph{generate new realistic viewpoints} (\cref{f:aug}). Given a sample $(f^i_t, g^i_t, D^i_t)$, we apply a random perturbation to the viewpoint (with a forward bias to avoid unoccluding too many pixels) and use depth-image-based rendering (DIBR)~\cite{morvan2009acquisition} to generate a new sample $(f^i_*, g^i_*, D^i_*)$, warping both the image and the depth map.

Sometimes the depth map $D^i_t$ from KF contains too many holes to yield satisfactory DIBR results (\cref{f:aug}, bottom); we found preferable to use the depth $\hat D^i_t = \ensuremath{\Phi_\text{depth}}(f_t)$ estimated by the network which is less accurate but more robust, containing almost no missing pixels (\cref{f:aug}, top).

\section{Method}\label{s:method}

We propose a single Convolutional Neural Network (CNN), VpDR-Net\xspace, that learns a \emph{3D object category} by observing it from a~\emph{variable viewpoint} in videos and no supervision (\cref{fig:overview}). Videos do not solve the problem of modeling intra-class shape variations, but they provide powerful yet noisy cues about the 3D shape of individual objects.

VpDR-Net\xspace takes as an input a set of $K$ video sequences $S^1, ..., S^K$ of an object category (such as cars or chairs), where a video $S^i = (f_1^i, ... , f_{N^i}^i)$ contains $N^i$ RGB or RGBD frames $f_t^i \in \mathbb{R}^{H \times W \times \mathcal{C} }$ (where $\mathcal{C}=3$ for RGB and $\mathcal{C}=4$ for RGBD data)
and learns a model of the 3D category. This model has three components: i) a predictor $\ensuremath{\Phi_\text{vp}}(f_i^t)$ of the \emph{absolute viewpoint} of the object (implicitly aligning the different object instances to a common reference frame; \cref{s:viewpoint}), ii) a \textit{monocular depth} predictor $\ensuremath{\Phi_\text{depth}}(f_i^t)$
(\cref{s:depth}) and iii) and a \textit{shape} predictor $\ensuremath{\Phi_\text{pcl}}(f_i^t)$ that extends the depth map to a point cloud capturing the complete shape of the object (\cref{s:recon}). Learning starts by preprocessing videos to extract instance-specific egomotion and shape information (\cref{s:sfm}).

\subsection{Sequence-specific structure and pose}\label{s:sfm}

Video sequences are pre-processed to extract from each frame $f_t^i$ a tuple $(K_t^i,g_t^i,D_t^i)$ consisting of: (i) the camera calibration parameters $K_t^i$, (ii) its pose $g_t^i\in SE(3)$, and (iii) a depth map $D_t^i\in \mathbb{R}^{H \times W}$ associating a depth value to each pixel of $f_t^i$. The camera pose $g_t^i = (R_t^i,T_t^i)$ consists of a rotation matrix $R_t^i \in SO(3)$ and a translation vector $T_t^i \in \mathbb{R}^3$.\footnote{We use the convention that $g_t^i$ transforms world-relative coordinates $p_\text{world}$ to camera-relative coordinates $p_\text{camera} = g_t^i p_\text{world}$.}
We extract this information using off-the-shelf methods: the structure-from-motion (SFM) algorithm COLMAP for RGB sequences \cite{schoenberger2016sfm,schoenberger2016mvs}, and an open-source implementation~\cite{rusu2011pcl} of KinectFusion (KF) \cite{newcombe2011kinectfusion} for RGBD sequences. The information extracted from RGB or RGBD data is qualitatively similar, except that the scale of SFM reconstructions is arbitrary.

\subsection{Intra-sequence alignment}\label{s:viewpoint}

Methods such as SFM or KF can reliably estimate camera pose and depth information for single objects and individual video sequences, but are not applicable to~\emph{different instances and sequences}. In fact, their underlying assumption is that geometry is fixed, which is true for single (rigid) objects, but false when the geometry and appearance differ due to intra-class variations.

Learning 3D object categories requires to relate their variable 3D shapes by identifying and putting in correspondence analogous geometric features, such as the object front and rear. For rigid objects, such correspondences can be expressed by rigid transformations that \emph{align} occurrences of analogous geometric features.

The most common approach for aligning 3D shapes, also adopted by~\cite{sedaghat15unsupervised} for video sequences, is to extract and match 3D feature descriptors. Once objects in images or videos are aligned, the data can be used to supervise other tasks, such as learning a monocular predictor of the absolute viewpoint of an object~\cite{sedaghat15unsupervised}.

One of our main contributions, described below, is to reverse this process by learning a viewpoint predictor \emph{without} explicitly matching 3D shapes. Empirically (\cref{s:exp}), we show that, by skipping the intermediate 3D analysis, our method is often more effective and robust than alternatives.

\myparagraph{Siamese network for viewpoint factorization} Geometric analogies between 3D shapes can often be detected in image space directly, based on visual similarity. Thus, we propose to train a CNN $\ensuremath{\Phi_\text{vp}}$ that maps a single frame $f_t^i$ to its \emph{absolute viewpoint} $\hat g_t^i = \ensuremath{\Phi_\text{vp}}(f_t^i)$ in the globally-aligned reference frame. We wish to learn this CNN from the viewpoints estimated by the algorithms of~\cref{s:sfm} for each video sequence. However, these estimated viewpoints are \emph{not} absolute, but valid only within each sequence; formally, there are unknown sequence-specific motions $h^i = (R^i,T^i) \in SE(3)$ that map the sequence-specific camera poses $g^i_t$ to global poses $\hat g_t^i = g_t^i h^i$.\footnote{$h^i$ composes to the right: it transforms the world reference frame and then moves it to the camera reference frame.}

To address this issue, we propose to supervise the network using \emph{relative pose changes within each sequence}, which are invariant to the alignment transformation $h^i$. Formally, the transformation $h^i$ is eliminated by computing the relative pose change of the camera from frame $t$ to frame $t^\prime$:
\begin{equation}\label{e:fund}
   \hat g^i_{t^\prime} (\hat g^i_{t})^{-1}
   =
   g^i_{t^\prime} h^i (h^i)^{-1} (g^i_{t})^{-1}
   =
   g^i_{t^\prime} (g^i_{t})^{-1}.
\end{equation}
Expanding the expression with $\hat g_t^i = (\hat R_t^i,\hat T_t^i)$, we find equations
expressing the relative rotation and translation
\begin{align}
\hat R_{t^\prime}^i (\hat R_t^i)^\top
&=
R_{t^\prime}^i (R_t^i)^\top,
\label{e:fund1}
\\
\hat T^i_{t^\prime} - \hat R_{t^\prime}^i (\hat R_t^i)^\top \hat T^i_{t}
&=
T^i_{t^\prime} - R_{t^\prime}^i (R_t^i)^\top T^i_{t}.
\label{e:fund2}
\end{align}
Eqs.~\eqref{e:fund1} and \eqref{e:fund2} are used to constrain the training of a \emph{Siamese architecture}, which, given two frames $t$ and $t'$, evaluates the CNN twice to obtain estimates $(\hat R^i_t, \hat T^i_t) = \ensuremath{\Phi_\text{vp}}(f^i_t)$ and $(\hat R^i_{t^\prime}, \hat T^i_{t^\prime}) = \ensuremath{\Phi_\text{vp}}(f^i_{t^\prime})$. The estimated poses are then compared to the ground truth ones, $(R^i_t,T^i_t)$ and $(R^i_{t^\prime},T^i_{t^\prime})$, in a relative manner by using losses that enforce the estimated poses to satisfy \cref{e:fund1,e:fund2}:
\begin{align}
\ell_R
(\hat R^i_t, \hat T^i_t, \hat R^i_{t^\prime}, \hat T^i_{t^\prime})
&\ensuremath{\overset{\cdot}{=}}
\|
\ln \hat R_{ t t^\prime}^i (R_{t t^\prime}^i)^\top
\|_F
\label{e:loss1}
\\
\ell_T (\hat R^i_t, \hat T^i_t, \hat R^i_{t^\prime}, \hat T^i_{t^\prime})
&\ensuremath{\overset{\cdot}{=}}
\|
\hat T_{t t^\prime}^i - T_{t t^\prime}^i
\|_2
\label{e:loss2}
\end{align}
where $\ln$ is the principal matrix logarithm and
\begin{align*}
R^i_{t^\prime t}
&\ensuremath{\overset{\cdot}{=}}
R_{t^\prime}^i (R_t^i)^\top,
&
\hat R^i_{t^\prime t}
&\ensuremath{\overset{\cdot}{=}}
\hat R_{t^\prime}^i (\hat R_t^i)^\top,
\\
T^i_{t^\prime t}
&\ensuremath{\overset{\cdot}{=}}
T^i_{t^\prime} - R^i_{t^\prime t} T^i_{t},
&
\hat T^i_{t^\prime t}
&\ensuremath{\overset{\cdot}{=}}
\hat T^i_{t^\prime} - \hat R^i_{t^\prime t} \hat T^i_{t}.
\end{align*}
While this CNN is only required to correctly predict relative viewpoint changes \emph{within each sequence}, since the \emph{same CNN} is used for all videos, the most plausible/regular solution for the network is to assign similar viewpoint predictions $(\hat R_t^i$, $\hat T_t^i)$ to images viewed from the same viewpoint, leading to a globally consistent alignment of the input sequences.  Furthermore, in a large family of 3D objects, different ones (e.g. SUVs and sedans) tend to be mediated by intermediate cases.
This is shown empirically in \cref{s:exp}.

\myparagraph{Scale ambiguity in SFM} \label{s:sfmambiguity} For methods such as SFM, there is an additional ambiguity: reconstructions are known only up to sequence-specific scaling factors $\lambda^i > 0$, so that the camera pose is parametrized as
$
 g^i_t(\lambda^i) = (R_t^i, \lambda^i T_t^i).
$
This ambiguity leaves~\cref{e:fund1} unchanged, but~\cref{e:fund2} becomes:
$$
\hat T^i_{t^\prime} - \hat R^i_{t^\prime t} \hat T^i_{t}
=
\lambda^i (T^i_{t^\prime} - R^i_{t^\prime t} T^i_{t})
\quad\Rightarrow\quad
\hat T^i_{t^\prime t} = \lambda^i T^i_{t^\prime t}
$$
During training, the ambiguity can be removed from loss~\eqref{e:loss2} by dividing vectors $T^i_{t^\prime t}$ and $\hat T^i_{t^\prime t}$ by their Euclidean norm. Note that for KF sequences $\lambda^i = 1$.
As the viewpoints are learned, an estimate of $\hat \lambda^i$ is computed using a moving average over training iterations for the other network modules to use (see supplementary material for details).

\myparagraph{Probabilistic predictions} Due to intrinsic ambiguities in the images or to errors in the SFM supervision (caused for example by reflective or textureless surfaces), $\ensuremath{\Phi_\text{vp}}$ is occasionally unable to predict the ground truth viewpoint accurately. We found beneficial to allow the network to explicitly learn these cases and express uncertainty as an additional input-dependent prediction. For the translation component, we modify the network to predict the absolute pose $\hat T^i_t$ as well as its confidence score $\sigma_{\hat T^i_t}$ (predicted as the output of a soft ReLU units to ensure positivity). We then model the relative translation as a Gaussian distribution with standard deviation $\sigma_T = \sigma_{\hat T_{t^\prime}^i} + \sigma_{\hat T_{t}^i}$ and our model is now learned by minimizing the  negative log-likelihood $\mathcal{L}_T$ which replaces the loss $\ell_T$:
\begin{equation}\label{e:robust1}
\mathcal{L}_T =
-\ln \frac{1}{(2\pi\sigma^2_T)^\frac{3}{2}}
\exp\left(
-\frac{1}{2} \frac{\ell_T^2}{\sigma_T^2}
\right).
\end{equation}

The rotation component is more complex due to the non-Euclidean geometry of $SO(3)$, but it was found sufficient to assume that the error term~\eqref{e:loss1} has Laplace distribution and optimize
$
\mathcal{L}_R = - \ln \frac{1}{C_R}
\exp\left(
-\frac{\sqrt{2} \ell_R}{\sigma_R}
\right),$
$\sigma_R =
\sigma_{\hat R_{t^\prime}^i}
+
\sigma_{\hat R_{t}^i},
$ where $C_R$ is a normalization term ensuring that the probability distribution integrates to one.
During training, by optimizing the losses $\mathcal{L}_R$ and $\mathcal{L}_T$ instead of $\ell_R$ and $\ell_T$, the network can discount gross errors by dividing the losses by a large predicted variance.

\myparagraph{Architecture} The architecture of $\ensuremath{\Phi_\text{vp}}$ is a variant of ResNet-50~\cite{he16resnet} with some modifications to improve its performance as viewpoint predictor. The lower layers of $\ensuremath{\Phi_\text{vp}}$ are used to extract a multiscale intermediate representation (denoted HC for hypercolumn \cite{hariharan2015hypercolumns} in \cref{fig:overview}). The upper layers consist of $2\times 2$ downsampling residual blocks that predict the viewpoint (see supp. material for details).

\input{fig_aug}

\subsection{Depth prediction}\label{s:depth}

The depth predictor module $\ensuremath{\Phi_\text{depth}}$ of VpDR-Net\xspace takes individual frames $f^i_t$ and outputs a corresponding depth map $\hat D_t = \ensuremath{\Phi_\text{depth}}(f^i_t)$, performing monocular depth estimation.

Estimating depth from a single image is inherently ambiguous and requires comparing the image to internal priors of the object shape. Similar to pose, we allow the network to explicitly \emph{learn and express uncertainty} about depth estimates by predicting a posterior distribution over possible pixel depths.
For robustness to outliers from COLMAP and KF, we assume a Laplace distribution
with negative log-likelihood loss
\begin{equation}
\mathcal{L}_D = \sum_{j=1}^{WH}
-\ln \frac{\sqrt{2}}{2\hat\sigma_{d_j}} ~
\exp\left(
-\frac{\sqrt{2} ~ |d_j - \hat {\lambda^i}^{-1} \hat d_j|}{\hat \sigma_{d_j} }
\right),
\end{equation}
where $d_j$ is the noisy ground truth depth output by SFM or KF for a given pixel $j$, and $\hat d_j$ and $\hat \sigma_{d_j}$ are respectively the corresponding predicted depth mean and standard deviation. The relative scale $\hat \lambda^i$ is 1 for KF and is estimated as explained in~\cref{s:sfmambiguity} for SFM.

\input{viewpoint_qualitative.tex}

\subsection{Point-cloud completion}\label{s:recon}

Given any image $f$ of an object instance, its \textit{aligned 3D shape} can be reconstructed by estimating and aligning its depth map using the output of the viewpoint and depth predictors of~\cref{s:depth,s:viewpoint}. However, since a depth map cannot represent the occluded portions of the object, such a reconstruction can only be partial. In this section, we describe the third and last component of~VpDR-Net\xspace, whose goal is to generate a full reconstruction of the object, beyond what is visible in the given view.

\myparagraph{Partial point cloud} The first step is to convert the predicted depth map $\hat D_f = \ensuremath{\Phi_\text{depth}}(f)$ into a partial point cloud
$
\hat P_f
\ensuremath{\overset{\cdot}{=}}
\{ \hat p_j :  j=1,\dots,HW \},
$
$
\hat p_j
\ensuremath{\overset{\cdot}{=}}
K^{-1}
\begin{bmatrix} u_j & v_j & \hat d_i \end{bmatrix}^\top,
$
where $(u_j,v_j)$ are the coordinates of a pixel $j$ in the depth map $\hat D_f$ and $K$ is the camera calibration matrix. Empirically, we have found that the reconstruction problem is much easier if the data is aligned in the global reference frame established by VpDR-Net\xspace. Thus, we transform $\hat P_f$ into a globally-aligned point cloud as $\hat P^G_f = \hat g^{-1} \hat P_f$, where $\hat g=\ensuremath{\Phi_\text{vp}}(f)$ is the camera pose estimated by the viewpoint-prediction network.

\myparagraph{Point cloud completion network} Next, our goal is to learn the point cloud completion part of our network $\ensuremath{\Phi_\text{pcl}}$ that takes the aligned but incomplete point could $\hat P^G_f$ and produces a complete object reconstruction $\hat C$. We do so by predicting a 3D occupancy probability field. However, rather than using a volumetric method that may require a discrete and fixed voxelization of space, we propose a simple and efficient alternative. First, the network $\ensuremath{\Phi_\text{pcl}}$ predicts a  set of $M$ 3D points $\hat S = (\hat s_1,\dots, \hat s_M) \in \mathbb{R}^{3\times M}$ that, during training, closely fit the ground truth 3D point cloud $C$. This step minimizes the fitting error:
\begin{equation}
 \ensuremath{\ell_\text{pcl}}(\hat S) = \frac{1}{|C|}\sum_{c \in C}
 \min_{m=1,\dots,M} \left\| c -  \hat s_m \right\|_2.
\end{equation}
The 3D point cloud $\hat S$ provides a good coverage of the ground truth object shape. However, this point cloud is conservative and distributed \emph{in the vicinity} of the ground truth object. Thus, while this is not a precise representation of the object shape, it works well as a support of a probability distribution of space occupancy. In order to estimate the occupancy probability values, the network $\ensuremath{\Phi_\text{pcl}}(\hat P^G_f)$ predicts additional scalar outputs
$$
\delta_m = | \{ c \in C : \forall m^\prime: \| \hat s_m - c \|_2 \leq \| \hat s_{m^\prime} - c \|_2 \} | / |C|
$$
proportional to the number of  ground truth surface points $c \in C$ for which the support point $\hat s_m$ is the nearest neighbor. The network is trained to compute a prediction $\hat\delta_m$ of the occupancy masses $\delta_m$ by minimizing the squared error loss
$
\ell_\delta(\hat \delta, \delta) = \sum_{m=1}^M (\hat \delta_m - \delta_m)^2.
$

Given the network prediction $(\hat S, \hat \delta) = \ensuremath{\Phi_\text{pcl}}(\hat P^G_f)$, the completed point cloud is then defined as the subset of points $\hat C$ that have sufficiently high occupancy, defined as:
$
\hat C_\tau = \{ \hat s_m \in \hat S : \delta_m \geq \tau \}
$
where $\tau$ is a confidence parameter. The set $\hat C_\tau$ can be further refined by using e.g.\ a 3D Laplacian filter to smooth out noise.

\myparagraph{Architecture} The point cloud completion network $\ensuremath{\Phi_\text{pcl}}$ is modeled after PointNet \cite{qi16pointnet}, originally proposed to semantically \emph{segment} a point clouds. Here we adapt it to perform a completely different task, namely 3D shape reconstruction. This is made possible by our model where shape is represented as a cloud of 3D support points $\hat S$ and their occupancy masses $\hat \delta$.
Differently from $\ensuremath{\Phi_\text{vp}}$ and $\ensuremath{\Phi_\text{depth}}$, $\ensuremath{\Phi_\text{pcl}}$ is \emph{not} convolutional but uses a sequence of fully connected layers to process the 3D points in $\hat P_f^G$, after appending an appearance descriptor to each of them. A key step is to add an intermediate orderless pooling operator to remove the dependency on the order and number of input points (see the supplementary material for details). The architecture is configured to predict $M=10^4$ points $\hat S$.
\input{tab_posest}

\myparagraph{Leave out} During training the incomplete point cloud $\hat P_f^G$ is downsampled by randomly selecting between $M = 10^3$ and $10^4$ points based on their depth prediction confidence as estimated by $\ensuremath{\Phi_\text{depth}}$. Similar to dropout, dropping points allows the network to overfit less, to become less sensitive to the size of the input point cloud, and to implicitly discard background points (as these are assigned low confidence by depth prediction). For the latter reason, leave out is maintained at test time too with $M=10^4$.
\section{Related work}

\myparagraph{Viewpoint estimation} The vast majority of methods for learning the viewpoint of object categories use manual supervision~\cite{savarese073d,Ozuysal09pose,glasner11viewpoint,pepik14multi,xiang2014beyond,mottaghi2015coarse,tulsiani2015viewpoints} or synthetic~\cite{su2015render} data. In \cite{ummenhofer16demon}, a deep architecture predicts a relative camera pose and depth for a pair of images. Only a few works have used videos~\cite{sun09multi,sedaghat15unsupervised}. \cite{sedaghat15unsupervised} solves the shape alignment problem using a global search strategy based on the pairwise alignment of point clouds, a step we avoid by means of our Siamese viewpoint factorization network.

\myparagraph{3D shape prediction}
A traditional approach to 3D reconstruction is to use handcrafted 3D models~\cite{lawrence63machine, lowe87three}, and more recently 3D CAD models~\cite{shapenet2015,xiang2014beyond}. Often the  idea is to search for the 3D model in a CAD library that best fits the image~\cite{lim13parsing,aubry14seeing,gupta15aligning,bansal16marr}. Alternatively, CAD models can be used to train a network to directly predict the 3D shape of an object~\cite{girdhar16learning,wu16learning,tatarchenko16multi,choy163d}. Morphable models have sometimes been used ~\cite{zia13detailed,kar15category}, particularly for modeling faces~\cite{blanz03face,liu16joint}. All these methods require 3D models at train time.

\myparagraph{Data-driven approaches for geometry} Structure from motion (SFM) generally assumes fixed geometry between views and is difficult to apply directly to object categories due to intra-class variations. Starting from datasets of unordered images, methods such as~\cite{zhu10model} and \cite{prasad10finding} use SFM and manual annotations, such as keypoints in \cite{carreira16lifting,kar15category}, to estimate a rough 3D geometry of objects. Here, we leverage motion cues and do not need extra annotations.


\section{Introduction} \label{sec:intro}

The total solar flux\footnote{We have observed the total solar 'flux density' practically. It is commonly, and perhaps inaccurately, referred to as 'flux' in the literature. In the paper, we will use the common term 'flux' rather than 'flux density'.} in microwaves, especially the F10.7 index (total solar flux at 2.8 GHz), is widely used as an indicator of solar activity in the fields of heliophysics and geophysics, because the F10.7 index indicates the variation of solar UV emission in comparison to sunspot number. The time variation in microwaves is traditionally classified into three components based on the timescale of enhancements. The components are a background component from the quiet Sun, a slowly varying component, and a sporadic (burst) component \citep{1965sra..book.....K}. The background component is considered to originate from optically thick thermal bremsstrahlung emission from the atmosphere above the upper chromosphere, but the emission mechanism of the background component is not so simple at lower frequencies (1$\sim$10 GHz). Actually, a simple calculation of thermal bremsstrahlung emission from optically thick layers cannot explain the quiet Sun spectrum measured by \cite{1991ApJ...370..779Z}. The reason for this discrepancy is that the transition region, which is an important contributor in this frequency range, is optically thin \citep[e.g.][]{2011SoPh..273..309S}. The contribution of coronal emission is also a reason for the discrepancy \citep{1991ApJ...370..779Z, 1996ApJ...473..539B}. To reproduce the spectrum of the background component at low frequencies, we need to consider the vertical thermal structure of the quiet Sun from the chromosphere to the corona. Therefore, modeling of the background component is needed, such as those based on recent Radiative-MHD simulations \citep{2004A&A...419..747L}.

The emission sources of a slowly varying component were investigated vigorously using the Westerbork Synthesis Radio Telescope and Very Large Array \citep{1959AnAp...22....1K, 1977A&A....61...79C, 1980A&A....82...30A, 1982ApJ...258..384L, 1982SoPh...80...71C, 1983ApJ...269..698M, 1983A&A...124..103D, 1984ApJ...283..413S, 1987ApJ...315..716W, 1991ApJ...379..366G, 1992ApJS...78..599W, 1994ApJ...426..434A}. From these studies, it is widely accepted that the slowly varying component originates in the thermal bremsstrahlung emission of coronal loops above active regions and thermal gyro-resonance emission from strong magnetic field regions, such as sunspots. A sporadic component is related to flares and coronal mass ejections, and is emitted from the non-thermal electrons accelerated in these phenomena.

The total solar flux at 2.8 GHz has been observed from 1947 \citep{1697669}. Nevertheless, the long-term variation of solar microwave emission itself has not been well investigated except in a few studies \citep[e.g.][]{1969JRASC..63..125C}, because it is hard to study the properties of microwave emission from the single-frequency data. In Japan, the total solar fluxes at 1, 2, 3.75, and 9.4 GHz have been observed continuously from 1957 \citep{1953PRIAN..1..71T, 1957PRIAN..4..60T}, and the dataset is calibrated well with the established method \citep{1973SoPh...29..243T}. \cite{1994SoPh..152..167S, 1995JGR...10019851S} investigated the microwave spectrum of 1980s data obtained in Japan. Because the main purpose of their investigation is the comparison between the solar irradiance measured with the Active Cavity Radiometer Irradiance Monitor (ACRIM) aboard the Solar Maximum Mission (SMM) and microwave spectrum, the period of their investigation is just only 10 years. Hence, there is no studies of  microwave spectrum variation in the period that is longer than one solar cycle yet.
In the paper, we examine the long-term variation of solar microwave emission using the dataset of the last half century, and report the properties of the solar cycle variation in microwaves.

\section{Observation \& Dataset} \label{sec:obs}

In November 1951, monitoring of the total solar flux density at 3.75 GHz started at Toyokawa Observatory, the Research Institute of Atmospherics, Nagoya University \citep{1953PRIAN..1..71T}. Observations at 9.4 GHz started in May 1956, and the 1 and 2 GHz observations started in May and June 1957, respectively at the same site \citep{1957PRIAN..4..60T}. The solar radio telescopes for the 1, 2, and 9.4 GHz observations were prepared for the coordinated observations in the International Geophysical Year lasting from July 1957 to December 1958 \citep{1958BRIA....67T}. For the analysis described in the paper, we need data at all four frequencies, 1, 2, 3.75, and 9.4 GHz. Therefore, we analyzed the data obtained from June 1957.

In March 1994, the telescopes for the 1, 2, and 9.4 GHz observations were moved to Nobeyama Solar Radio Observatory (NSRO), a branch of the National Astronomical Observatory of Japan (NAOJ), and their operation resumed late in May 1994. At the same time, a new telescope for a 3.75 GHz observation was constructed in Nobeyama. Before the transfer and construction of the telescopes, the 17, 35, and 80 GHz monitoring observations were carried out by NSRO \citep{1985PASJ...37..163N}. Thus, the monitoring observations of the total solar fluxes at 1, 2, 3.75, 9.4, 17, 35, and 80 GHz started at the same site from 1994 May. The monitoring system was named Nobeyama Radio Polarimeters (NoRP). Due to the change of the observing site, the total solar fluxes at 1, 2, and 9.4 GHz were not measured from March to May 1994. Although there is no data loss at 3.75 GHz, we did not analyze the data of this period because no spectrum can be obtained for this study.

NSRO closed on 31 March 2015. Since 1 April 2015, Nobeyama Radio Observatory (NRO), a branch of NAOJ, has continued the operation of NoRP. The dataset used in the paper covers until December 2016.

NSRO released calibrated daily flux densities at each observing frequency in tabulated form every month at its FTP site\footnote{ftp://solar-pub.nao.ac.jp/pub/nsro/norp/data/daily/}. The calibration method is described in \cite{1973SoPh...29..243T}. In cooperation with the consortium for NoRP scientific operations\footnote{http://solar.nro.nao.ac.jp/norp/html/policy\_new.html}, NRO continues to release the tabulated data using the same FTP site. We used the daily fluxes in the tables for our analysis. The value in the table is not the same as the value calculated using the analysis package of NoRP that is included in the SolarSoftWare \citep[SSW:][]{1998SoPh..182..497F}. The value in the tables is derived after manually removing the effects of radio bursts, bad weather conditions, and instrumental problems. In contrast, when the SSW package is used, the daily flux is automatically calculated from the raw data without such treatment. The difference can be neglected for a study of flares, but the values in the tables are well calibrated and much more reliable for a study of the long-term variation. The total solar flux used in the paper is corrected based on the seasonal variation of the distance between the Sun and the Earth. To compare the solar cycle variation in microwaves with that of the sunspot number, the monthly mean total sunspot number and 13-month smoothed monthly total sunspot number provided from WDC-SILSO, Royal Observatory of Belgium are used \citep{sidc}.

\section{Solar Cycle Variation in Microwave} \label{sec:flux}

Panels (a) and (b) in Figure \ref{fig:fig1} show the long-term variations of the total solar fluxes in microwaves from June 1957 to December 2016. As mentioned in the previous section, the effect of radio bursts is carefully removed. Therefore, the total solar fluxes in panels (a) and (b) in Figure \ref{fig:fig1} are composed of a background component and a slowly varying component, and we can consider that the enhancement around the solar maxima (vertical dashed lines) is caused by the increasing of coronal plasma, and the rising of the number, size, and magnetic field strength of sunspots.

\begin{figure}[h]
\epsscale{.70}
\plotone{fig1.eps}
\caption{The variation of the total solar flux in microwaves. (a) The daily total solar fluxes in microwaves and the monthly mean total sunspot number. (b) The monthly mean microwave fluxes, the monthly mean total sunspot number (solid lines) and the monthly standard deviations (MSD: asterisk). In the two panels, Blue, Red, Orange, Green, and Black indicate 1 GHz, 2 GHz, 3.75 GHz, 9.4 GHz and sunspot number, respectively. (c) The black asterisks indicate the monthly mean total sunspot number, and the red line is the 13-month smoothed monthly total sunspot number. The green asterisks indicate the AMSD, and the blue line is the 13-month smoothed AMSD. (d) The parameters of the model function. The red indicates a: intercept, and the black indicates b: slope. The vertical dashed lines and dotted-dashed lines indicate the solar maxima and solar minima defined from the monthly mean total sunspot number, as shown in Table 1. \label{fig:fig1}}
\end{figure}

Comparing the daily and monthly total solar fluxes in microwaves and the monthly mean total sunspot number, their variations are very similar at the solar maxima. We can find the counterparts of almost all microwave peaks in the time profile of sunspot number. On the other hand, there are large gaps between them at the solar minima. The total solar fluxes in microwaves do not decrease as significantly as the sunspot number does, because there is a contribution of the emission from the quiet Sun even at solar minima.

To characterize the microwave spectrum in each month, we calculated not only the monthly mean flux but also the monthly standard deviation (MSD) from the daily total solar flux in each frequency (panel (b) in Figure \ref{fig:fig1}). We found that the MSD also shows the solar cycle variation. The MSD around the solar maxima becomes large because the flux from active regions can change up to 40 \% from one day to the next \citep{1978SoPh...56..335F, 1982SoPh...80...71C, 1983A&A...124..103D, 1991ApJ...379..366G}. The emission that causes the large MSD is a slowly varying component, and the variation of the emission strongly depends on the coronal activity and magnetic field strength. At solar minima, no magnetic field is strong enough for gyro-resonance emission, and the coronal activity that enhances the fluctuation of coronal plasma is very low. Consequently, the MSD at the solar minima becomes very small.

Although the four MSDs can be calculated from the total solar fluxes at the four observing frequencies (1, 2, 3.75, and 9.4 GHz), their variations are quite similar. To simplify our discussion, the averaged value of the four MSDs is used hereafter, and we call the averaged value ``Averaged Monthly Standard Deviation" (AMSD). Panel (c) in Figure \ref{fig:fig1} shows the long-term variation of the AMSD, 13-month smoothed AMSD, monthly mean total sunspot number, and 13-month smoothed monthly total sunspot number. The variation of the AMSD is very similar to that of the monthly mean total sunspot number, and the correlation coefficient between them is 0.71. When we calculate the correlation coefficient between the 13-month smoothed values of the monthly mean total sunspot number and AMSD, the coefficient increases to 0.91. Considering the emission mechanisms at the observing frequencies reviewed at Section \ref{sec:intro}, the AMSD directly reveals the variation of the strong magnetic field regions and the activities in the atmosphere higher than the upper chromosphere.

Table \ref{tab:tab1} presents the month when the monthly mean total sunspot number or AMSD becomes the maximum/minimum value in each solar cycle. In the table, we can see the difference between the months derived from the monthly mean total sunspot number and that from the AMSD. The difference no doubt results from the differing atmospheric layers that dominate the two indexes, as mentioned above.

\begin{table}[h]
\begin{center}
\begin{tabular}{c c  c c || c c  c c}
\hline \hline
Cycle & Max/Min & SN & AMSD & Cycle & Max/Min & SN & AMSD\\
\hline
19 & Min. &           &                & 22 & Min. & Jun. 1986 & \bf{Sep. 1986} \\
   & Max. & Oct. 1957 & \bf{Jan. 1959} &    & Max. & Jun. 1989 & \bf{Mar. 1989} \\
\hline
20 & Min. & Jul. 1964 & \bf{Sep. 1964} & 23 & Min. & Oct. 1996 & \bf{Jan. 1997} \\
   & Max. & Mar. 1969 & \bf{Jun. 1969} &    & Max. & Jul. 2000 & \bf{Oct. 2003} \\
\hline
21 & Min. & Jul. 1976 & \bf{Jul. 1976} & 24 & Min. & Aug. 2009 & \bf{Jul. 2008} \\
   & Max. & May  1980 & \bf{Jun. 1982} &    & Max. & Feb. 2014 & \bf{Jul. 2014} \\
\hline \hline
\end{tabular}
\end{center}
\caption{The months when the monthly mean total sunspot number (SN) or averaged monthly standard deviation (AMSD) becomes the maximum/minimum value in each solar cycle. \label{tab:tab1}}
\end{table}

\section{Microwave Spectra at Solar Maxima and Minima} \label{sec:spec}

To investigate the solar cycle variation of the microwave spectrum, we carried out the fitting of monthly mean microwave spectra using several simple functions. Finally, we found that an exponential function can give a satisfactory fit to most of the spectra: log$_{10}$(Flux) = a + b $\times ~\nu$, $\nu$ is frequency in Hz. Panel (d) in Figure \ref{fig:fig1} shows the time variation of the fitting model parameters (a: the intercept, b: the slope of the spectrum). The intercept correlates weakly with the solar cycle variation. On the other hand, the slope of the spectrum indicates the anti-correlation. Gyro-resonance emission in lower frequency is stronger than in higher frequency when the emission regions of both frequencies are the same. Because the property of gyro-resonance emission tends to flatten the spectrum at solar maxima, the anti-correlation is a matter of course. We mention that the maximum values of the spectral slopes at the solar minima are about 9.3$\pm$0.3$\times$10$^{-8}$ (3 \% variation) and that the difference is so small though the minimum value of the spectral slope at the solar maximum significantly differs from that in the other solar maxima (5.3$\pm$1.0$\times$10$^{-8}$, $\sim$19 \% variation).

To compare the microwave spectra in the solar cycles, we plot the monthly mean solar microwave spectra at the solar maxima and solar minima (Figure \ref{fig:fig2}). To investigate the states where the activity in the upper chromosphere -- corona is highest/lowest, from here, we define ``solar maximum" and ``solar minimum" as the months when the AMSD becomes the maximum/minimum value in each solar cycle. The months are written in bold, Table \ref{tab:tab1}. As shown in the time variation of the fitting model parameter, the spectrum at a solar maximum significantly differs among the cycles. The absolute values of the total solar fluxes at the solar maxima also vary by over 100 \% between the cycles. The difference in the spectra tends to be related to the strength of the solar cycle variation.

\begin{figure}[h]
\plotone{fig2.eps}
\caption{ The monthly mean microwave spectra in the months when the AMSD becomes the maximum value (Left panel) or the minimum value (Right panel) in each solar cycle. Color indicates the number of the solar cycle. The months of the solar maxima and minima are shown in bold, Table \ref{tab:tab1}. \label{fig:fig2}}
\end{figure}

A solar minimum is a suitable moment for investigating the spectrum of a background component, because there is no object that emits a slowly varying component and a sporadic component. To obtain the pure spectrum of a background component, we need to select the moment when the coronal activity is extremely low. For this reason, the monthly mean spectrum when the AMSD is lowest in its cycle is suitable for this study. The right panel in Figure \ref{fig:fig2} displays the five monthly mean spectra at the solar minima defined from the AMSD, and the months of the spectra are written in bold, Table \ref{tab:tab1}. The slopes of these spectra are very similar because the variance of the total solar fluxes at the solar minima is only 5.6 SFU (12 \%) at 1 GHz, 4.72 SFU (9 \%) at 2 GHz, 2.23 SFU (3 \%) at 3.75 GHz, and 11.2 SFU (4 \%) at 9 GHz. The values are similar to the precisions of the instruments.

\section{Summary \& Discussion} \label{sec:sum}

We investigated the total solar fluxes at 1, 2, 3.75, and 9.4 GHz obtained from June 1957 to December 2016, and found that the averaged monthly standard deviation (AMSD) of microwave solar fluxes can reveal the long-term variation of solar activity well. Because of the high sensitivity to the activities in the transition region and corona, AMSD can be very useful for investigating the solar-terrestrial environment.

Although an exponential function can give a satisfactory fit to most of the spectra, there is no physical background of the model function. When we execute the fitting of the daily spectra, the chi-square value of the fitting becomes very large around the solar maxima. The tendency can be also seen in the spectra at the solar maxima in Figure \ref{fig:fig2}. The result is consistent with \cite{1994SoPh..152..167S}, and it can be understood as the effect of gyro-resonance emission from sunspots, as mentioned by them, because the spectrum of the emission has a power-law distribution \citep[e.g.][]{1985ARA&A..23..169D}.

To investigate the spectrum of a background component in each cycle, we compared the monthly mean spectra at the solar minima in Cycle 20$\sim$24 defined from the AMSD. Based on the absolute values of the fluxes and the slopes of the spectra, we found that the difference in the spectra is very small. As mentioned in Section \ref{sec:intro}, a background component should reveal the average atmospheric structure of the transition region and lower corona in the quiet Sun. Therefore, the results indicate that the average atmospheric structure above the upper chromosphere in the quiet Sun at solar minima, which may be related to the energy input for atmospheric heating from the sub-photosphere to the corona, has not varied for half a century. The peaks of sunspot numbers at the solar maxima varied during the period. In other words, the Sun’s magnetic activity has been changing significantly \citep[e.g.][]{2012ApJ...757L...8L, 2012SoPh..281..577P, 2012ApJ...750L..42G, 2012ApJ...753..157S, 2013ApJ...763...23S, 2013PASJ...65S..16S}. Nevertheless, our observing results suggest that the energy input for atmospheric heating from the sub-photosphere to the corona has not changed in the quiet Sun. It is commonly supposed that the energy input in the quiet Sun is caused by convection motion and the local dynamo mechanism in the sub-photosphere. Therefore, our results may indicate that the properties of convection motion and the local dynamo mechanism have not changed for half a century, and the solar cycle differences in strong magnetic field generated by the global dynamo mechanism.

\acknowledgments
The authors would like to express thank to the people who have participated in the operation of the solar radio telescopes at Toyokawa and Nobeyama Radio Polarimeters for over half a century.  The Nobeyama Radio Polarimeters (NoRP) are operated by Nobeyama Radio Observatory, a branch of National Astronomical Observatory of Japan, and their observing data are verified scientifically by the consortium for NoRP scientific operations. This work was carried out on the Solar Data Analysis System operated by the Astronomy Data Center of the National Astronomical Observatory of Japan. M.S. was supported by JSPS KAKENHI Grant Number JP17K05397. A.A. was supported by JSPS KAKENHI Grant Numbers JP15H05816 and JP15K17772.

\facility{NoRP}


\section{Hypergraphs and quantum states}
Given a hypergraph $G=(V,E)$ and non-negative edge weights $w\colon E\to\mathbb R_+$, the \emph{cut function} $\delta\colon 2^V\to\mathbb R_{\geq0}$ is defined as $c(S) = \sum_{e\in\delta(S)} w(e)$, where $\delta(S)$ is the set of hyperedges that contain vertices both in~$S$ and~$V\setminus S$.
Fixing a subset of terminals~$T\subseteq V$, the \emph{min-cut function} $m\colon 2^T \to \mathbb R_{\geq0}$ is then given by $m(A) = \min_{S : S \cap T = A} c(S)$.
It is well-known that hypergraph cut functions are symmetric and submodular.
This property extends directly to min-cut functions.

We now turn to quantum states.
Given a quantum state~$\rho$ on a finite-dimensional tensor product Hilbert space $\mathcal H = \bigotimes_{t\in T} \mathcal H_t$, the \emph{entropy function} $S\colon 2^T\to\mathbb R_{\geq0}$ assigns to each subset $A\subseteq T$ the von Neumann entropy $S(A) = -\tr[\rho_A \log \rho_A]$ of the reduced state~$\rho_A = \tr_{T\setminus A}[\rho]$.
By a celebrated theorem of Lieb-Ruskai, the entropy function is submodular, and it is also symmetric if we restrict to pure states.

This coincidence begs the question if hypergraph min-cuts can always be realized by quantum entropies.
Such a result has been proved for graphs~\cite{hayden2016holographic,nezami2016multipartite}, motivated by research on the holographic principle in theoretical high-energy physics~\cite{bao2015holographic}.

To state the question more precisely, define the \emph{hypergraph cone}~$\Chyper n \subseteq \mathbb R_{\geq0}^{2^n}$ as the set of min-cut functions obtained from arbitrary weighted hypergraphs with terminals~$T=[n]$~\cite{bao2020quantum}.
Similarly, define the \emph{quantum entropy cone}~$\Cquantum n \subseteq \mathbb R_{\geq0}^{2^n}$ as the closure of the set of entropy functions obtained by varying over all pure quantum states on finite-dimensional tensor product Hilbert spaces as above~\cite{pippenger2003inequalities}.
Both sets are convex cones.
It is an well-known open problem in quantum information theory to determine the cones~$\Cquantum n$ for~$n \geq 4$~\cite{pippenger2003inequalities,linden2005new}.

Due to its import, this problem has also been studied for the class of stabilizer states, which are a versatile family of quantum states that have many applications in quantum information theory~\cite{gottesman1997stabilizer,gross2017schur}.
Thus, let $\Cstab n$ denote the closed convex cone generated by the set of entropy functions of pure stabilizer states (over any fixed prime).
In general, $\Cstab n \subseteq \Cquantum n$ is a proper subcone~\cite{linden2013quantum,gross2013stabilizer}.
Our main result is as follows:

\begin{thm}\label{thm:main}
  For any $n\in\mathbb N$, we have that $\Chyper n \subseteq \Cstab n \subseteq \Cquantum n$.
\end{thm}

\noindent
\Cref{thm:main} proves a conjecture made in \cite{bao2020quantum}, where the authors verified the inclusion for $n \leq 4$.
However, not even the inclusion $\Chyper n \subseteq \Cquantum n$ was known before this work.

To prove \cref{thm:main}, we start by constructing for any weighted hypergraph a quantum state whose entropies realize the cut-function.
We then apply a general construction for transforming entropies into minimized entropies, and show that it succeeds with high probability.

\section{Tensor network states for hypergraph cuts}
Fix a hypergraph $G=(V,E)$ with integral edge weights~$w\colon E \to \mathbb N$.
For each vertex $x\in V$, define the Hilbert space $\mathcal H_x = \bigotimes_{e \in E : x \in e} \mathcal H_{x,e}^{\otimes w(e)}$, where $\mathcal H_{x,e} = \mathbb C^D$.
The dimension $D$ will later be taken to be large.
Now let $\ket\Omega$ be the state given by
\begin{align}\label{eq:omega}
    \ket\Omega = \bigotimes_{e\in E} \ket{\GHZ(e)}^{\otimes w(e)} \in \bigotimes_{x\in V} \mathcal H_x.
\end{align}
Here, $\ket{\GHZ(e)} = \frac1{\sqrt D} \sum_{i=1}^D \ket i^{\otimes \abs e} \in \bigotimes_{x \in e} \mathcal H_{x,e}$ denotes an $\abs e$-partite GHZ state of local dimension~$D$.
In general, one can also develop the following theory for other multipartite entangled states than the GHZ state, but for our purposes this construction will suffice.

The GHZ state has the property that it is a pure state whose reduced states have~$D$ nonzero eigenvalues which are all equal to~$1/D$.
This implies that not only the von Neumann entropy of any subsystem $S \subseteq V$, but in fact any R\'enyi-$\alpha$ entropy can be calculated by
\begin{align}\label{eq:cut entropies}
    S_\alpha(\Omega_S)
= \sum_{e\in E} w(e) \, S_\alpha\bigl(\GHZ(e)_{S \cap e}\bigr)
= \log(D) \sum_{e\in \delta(S)} w(e) = \log(D) \, c(S).
\end{align}
We have thus found a family of quantum states, parameterized by $D$, whose entropy function is exactly proportional to the cut function~$c\colon 2^V\to\mathbb R_+$ of the given hypergraph.

\section{Random tensor network states for hypergraphs min-cuts}
We will now explain how to turn the state~\eqref{eq:omega} into one that approximates the min-cut function~$c\colon 2^T\to\mathbb R_+$ of the hypergraph $G=(V,E)$ for any fixed choice of terminals~$T\subseteq V$.
We may assume that any connected component of $G$ touches $T$ (otherwise we can remove this component without impacting the min-cut function).

Our main tool is the following construction, which is implicit in~\cite{hayden2016holographic} (cf.~\cite{horodecki2007quantum,dutil2010one}).
Define the (not necessarily normalized) pure state
\begin{align}\label{eq:projected state}
    \ket{\Psi} = \Bigl( \bigotimes_{x \in V \setminus T} \bra{\phi_x} \Bigr) \ket\Omega \in \mathcal H = \bigotimes_{x\in T} \mathcal H_x,
\end{align}
obtained by projecting the tensor factors for each non-terminal vertex~$x\not\in T$ onto pure states~$\ket{\phi_x}$ (which we will below choose at random).
We note that~$\Psi$ can be understood as a tensor network state of bond dimension~$D$.

We now relate the entropies of~$\Psi$ to those of the state~$\Omega$.
For this, recall that for any quantum state~$\rho$,
\begin{align}\label{eq:renyi mono}
    S_2(\rho) \leq S(\rho) \leq S_0(\rho),
\end{align}
where $S_2(\rho) = -\log\tr[\rho^2]$ the \emph{R\'enyi-2 entropy}, $S(\rho) = -\tr[\rho\log\rho]$ the \emph{von Neumann entropy}, and $S_0(\rho) = \log\rk[\rho]$ the \emph{log-rank}.
If $\rho$ is not normalized then we define $S_\alpha(\rho)$ in terms of the normalization (this makes no difference for the log-rank).
It is easy to see that for any state of the form~\eqref{eq:projected state} and any $A \subseteq T$,
\begin{align}\label{eq:rank upper}
    S_0(\Psi_A) \leq \min_{S : S \cap T = A} S_0(\Omega_S) = \log(D) m(A).
\end{align}
Indeed, any cut $S$ for $A$ gives us an upper bound on the rank of $\Psi_A$ in terms of the rank of $\Omega_S$, which can be calculated using~\eqref{eq:cut entropies} for $\alpha=0$.

In view of~\eqref{eq:renyi mono}, we would like to complement this upper bound by a similar lower bound on the R\'eniy-2 entropy.
For this, we choose each $\phi_x$ independently at random from a \emph{projective 2-design}.
This means that the first two moments agree with the unitarily invariant probability measure on pure states, i.e., $\mathbb E[\phi_x] = \tfrac I {D_x}$ and $\mathbb E[\phi_x^{\otimes 2}] = \tfrac{I + F}{D_x(D_x+1)}$, where~$D_x = \dim \mathcal H_x$ and~$F$ denotes the swap operator on $\mathcal H_x^{\otimes 2}$.
For example, we may choose~$\phi_x$ to be a uniformly random stabilizer state~\cite{klappenecker2005mutually,gross2007evenly}, in which case~$\Psi$ is again a stabilizer state~\cite{hayden2016holographic}.
We now compute the expected trace and purity of any subsystem.

\begin{lem}\label{lem:moments}
Let the $\phi_x$ be chosen independently at random from a 2-design.
Then the state~$\Psi$ in~\eqref{eq:projected state} satisfies
\begin{align*}
    \mathbb E[\tr[\Psi]] = \frac1{D_b}
\quad\text{and}\quad
    \mathbb E[\tr[\Psi_A^2]] = \frac1{D_b^2} D^{-m(A)} \left( k_A + O(D^{-1}) \right)
\end{align*}
for all $A \subseteq T$, where $D_b = \prod_{x\in V\setminus T} D_x$ and $k_A$ denotes the number of minimal cuts for~$A$.
\end{lem}
\begin{proof}
From the formula for the first moment of a 2-design,
\begin{align*}
  \mathbb E[\tr[\Psi]]
= \tr\Bigl[\Omega \bigl( \bigotimes_{x \in V \setminus T} \mathbb E[\phi_x] \bigr) \Bigr]
= \frac 1 {D_b},
\end{align*}
(as is common we suppress tensor products with identity operators).
For $\tr[\Psi_A^2]$, we first use the swap trick,
\begin{align*}
  \tr[\Psi_A^2]
= \tr[\Psi^{\otimes 2} F_A]
= \tr\Bigl[\bigl( \bigotimes_{x \in V \setminus T} \bra{\phi_x}^{\otimes 2} \bigr) \Omega^{\otimes 2} \bigl( \bigotimes_{x \in V \setminus T} \ket{\phi_x}^{\otimes 2} \bigr) F_A\Bigr]
= \tr\Bigl[\Omega^{\otimes 2} \bigl( \bigotimes_{x \in V \setminus T} \phi_x^{\otimes 2} \bigr) F_A\Bigr].
\end{align*}
Here, $F_A$ denotes the product of the swap operators on~$\mathcal H_x^{\ot2}$ for $x\in A$.
In the last step, we used the cyclicity of the trace and the fact that the swap operator $F_A$ `commutes' with $\bra{\phi_x}^{\otimes 2}$ for~$x\in V\setminus T$.
We can then use the compute the average by using the formula for the second moment of a 2-design:
\begin{align*}
  \mathbb E[\tr[\Psi_A^2]]
&= \tr\Bigl[\Omega^{\otimes 2} \bigl( \bigotimes_{x \in V \setminus T} \mathbb E[\phi_x^{\otimes 2}] \bigr) F_A\Bigr]
= \tr\Bigl[\Omega^{\otimes 2} \bigl( \prod_{x \in V \setminus T} \frac {I_x + F_x} {D_x(D_x+1)} \bigr) F_A\Bigr] \\
&= \prod_{x\in V\setminus T} \frac1{D_x(D_x+1)} \sum_{S : S \cap T = A} \tr\bigl[\Omega^{\otimes 2} F_S \bigr]
\end{align*}
where the last step is again by the swap trick.
The prefactor is $D_b^{-2} (1 + O(D^{-1}))$, while the sum can be estimated using~\eqref{eq:cut entropies},
\begin{align*}
  \sum_{S : S \cap T = A} \tr\bigl[\Omega^{\otimes 2} F_S \bigr]
= \sum_{S : S \cap T = A} 2^{-S_2(\Omega_S)}
= \sum_{S : S \cap T = A} D^{-c(S)}
= D^{-m(A)} \left( k_A + O(D^{-1}) \right),
\end{align*}
where we recall that~$k_A$ denotes the number of minimal cuts for $A$.
Together we obtain the desired bound.
\end{proof}

For $A=\emptyset$, the min-cut is empty and nondegenerate by our assumption that any connected component of~$G$ touches~$T$, so \cref{lem:moments} states that $\mathbb E[\tr[\Psi]^2] = D_b^{-2} (1 + O(D^{-1}))$.
Thus, $\tr[\Psi]$ is concentrated around its mean, suggesting that, with high probability, $\Psi\neq0$ and $S_2(\Psi_A) / \! \log(D) \approx m(A)$ for large~$D$.
The following lemma which follows the proof strategy of~\cite{hayden2016holographic} makes this intuition precise.

\begin{lem}\label{lem:concentration}
Let~$\Psi$ be defined as in~\eqref{eq:projected state}, with each $\phi_x$ chosen independently at random from a 2-design. Then the following properties hold for large~$D$:
\begin{enumerate}[label={(\alph*)}]
\item\label{it:nonzero}
$\mathbb P(\Psi\neq0) = 1 - O(D^{-1})$.
\item\label{it:expected renyi}
$\mathbb E[S_2(\Psi_A) | \Psi\neq0] \geq \log(D) \, m(A) - \log(k_A) - O(D^{-1/4})$, with $k_A$ the number of minimal cuts for~$A$.
\item\label{it:high probability entropy}
For any~$\delta>0$, it holds that
\begin{align*}
  \mathbb P\Bigl(\Psi\neq0 \text{ and } \abs[\Big]{ \frac{S_2(\Psi_A)}{\log(D)} - m(A) } \leq \delta \text{ for all } A \subseteq T\Bigr) = 1 - O\Bigl(\frac1{\delta\log(D)}\Bigr)
\end{align*}
The same statement holds for the the von Neumann entropy~$S(\Psi_A)$ instead of the R\'enyi entropy~$S_2(\Psi_A)$.
\end{enumerate}
\end{lem}
\begin{proof}
As just noted, $\mathbb E[\tr[\Psi]^2] = \mathbb E[\tr[\Psi]]^2 (1 + O(D^{-1}))$, so Chebyshev's inequality shows that, for any~$\varepsilon>0$,
\begin{align}\label{eq:cheby}
  \mathbb P\bigl(\lvert D_b\tr[\Psi] - 1 \rvert \leq \varepsilon \bigr)
\geq 1 - O\Bigl(\frac1{\varepsilon^2 D}\Bigr).
\end{align}
This establishes~\ref{it:nonzero}.

Next we prove~\ref{it:expected renyi}.
Let $E$ denote the event that $\lvert D_b\tr[\Psi] - 1 \rvert \leq D^{-1/4}$.
By~\eqref{eq:cheby},
\begin{align}\label{eq:p_E}
    p_E := \mathbb P(E) = 1 - O(D^{-1/2}).
\end{align}
Since~$E$ implies that~$\Psi\neq0$, we have
\begin{align}\label{eq:E[S_2|nonzero]}
  \mathbb E[S_2(\Psi_A) | \Psi\neq0]
\geq \mathbb P(\Psi\neq0) \, \mathbb E[S_2(\Psi_A) | \Psi\neq0]
\geq p_E \, \mathbb E[S_2(\Psi_A) | E]
\end{align}
We now bound
\begin{equation}\label{eq:E[S_2|E]}
\begin{aligned}
  \mathbb E[S_2(\Psi_A) | E]
&= -\mathbb E\bigl[\log(D_b^2 \tr[\Psi_A^2]) \big| E \bigr] + 2 \mathbb E\bigl[ \log(D_b\tr[\Psi]) \big| E \bigr] \\
&\geq -\log\mathbb E\bigl[D_b^2 \tr[\Psi_A^2] \big| E \bigr] + 2 \log\bigl(1-D^{-1/4}\bigr) \\
&= -\log\mathbb E\bigl[D_b^2 \tr[\Psi_A^2] \big| E \bigr] - O(D^{-1/4})
\end{aligned}
\end{equation}
where we used Jensen's inequality to lower-bound the first term.
Using~$p_E \, \mathbb E[\tr[\Psi_A^2] | E] \leq \mathbb E[\tr[\Psi_A^2]]$, we obtain
\begin{align*}
  -\log\mathbb E\bigl[D_b^2 \tr[\Psi_A^2] \big| E \bigr]
&\geq -\log \mathbb E[D_b^2 \tr[\Psi_A^2]] + \log(p_E) \\
&= \log(D) \, m(A) - \log \bigl( k_A + O(D^{-1}) \bigr) + \log\bigl( 1 - O(D^{-1/2}) \bigr) \\
&= \log(D) \, m(A) - \log(k_A) - O(D^{-1/2})
\end{align*}
by \cref{lem:moments} and~\eqref{eq:p_E}.
Together with~\eqref{eq:E[S_2|nonzero]}, \eqref{eq:E[S_2|E]}, and~$S_2(\Psi_A) \leq \log(D)m(A)$, which holds by~\eqref{eq:renyi mono} and \eqref{eq:rank upper}, we find
\begin{align*}
  \mathbb E[S_2(\Psi_A) | \Psi\neq0]
&\geq \log(D) \, m(A) - \log(k_A) - O(D^{-1/4}),
\end{align*}
proving~\ref{it:expected renyi}.

To prove \ref{it:high probability entropy}, we note that $\log(D)m(A) - S_2(\Psi_A)$ is a nonnegative random variable. Thus, for any fixed~$A\subset T$,
\begin{align*}
\mathbb P\Bigl(m(A) - \frac{S_2(\Psi_A)}{\log(D)} > \delta \Big| \Psi\neq0 \Bigr)
\leq \frac{\log(D) m(A) - \mathbb E[S_2(\Psi_A) | \Psi\neq0 ]}{\delta \log(D)}
\leq \frac{\log(k_A) + O(D^{-1/4})}{\delta \log(D)} = O\left(\frac1{\delta\log(D)}\right)
\end{align*}
where we first used the Markov inequality and then part~\ref{it:expected renyi}.
By taking the union bound over all the finitely many subsets~$A \subseteq T$ and using part~\ref{it:nonzero}, we obtain
\begin{align*}
  \mathbb P\Bigl(\Psi\neq0 \text{ and } m(A) - \frac{S_2(\Psi_A)}{\log(D)} \leq \delta \text{ for all } A \subseteq T\Bigr) = 1 - O\Bigl(\frac1{\delta\log(D)}\Bigr).
\end{align*}
In view of $S_2(\Psi_A) \leq S(\Psi_A) \leq \log(D)m(A)$, this proves part~\ref{it:high probability entropy}.
\end{proof}

\noindent
\Cref{lem:concentration} readily implies our main result.

\begin{proof}[Proof of \cref{thm:main}]
We only need to show that $\Chyper n \subseteq \Cstab n$.
Since $\Cstab n$ is a closed cone, it suffices to show that for any $\delta>0$ and any hypergraph with integral edge weights, terminal set~$T=[n]$, and min-cut function~$m$, there exists a number~$c>0$ and a stabilizer state $\Psi$ on an $n$-partite Hilbert space such that
\begin{align*}
  \abs[\Big]{ \frac{S(\Psi_A)}{c} - m(A) } \leq \delta
\end{align*}
for all~$A \subseteq T$.
This follows from \cref{lem:concentration} if we use the set of stabilizer states as the 2-design and choose~$D$ to be sufficiently large.
Indeed, if each~$\phi_x$ is a stabilizer state then so is $\Psi$.
\end{proof}

\small
\bibliographystyle{alpha}