\section{Introduction}
Feature matching is a key component in many 3D vision applications such as structure from motion (SfM) or simultaneous localization and mapping (SLAM).
Conventional pose estimation is a multi-step process: feature detection finds interest points, for which local descriptors are computed. Based on the descriptors, pairs of keypoints from different images are matched,
which defines constraints in the pose optimization.
A major challenge lies in the ambiguity of matching local descriptors by nearest-neighbor search, which is error-prone, particularly in texture-less areas or in presence of repetitive patterns.
Hand-crafted heuristics or outlier filters become necessary to circumvent this problem to some degree.

Recent learning-based approaches~\cite{Sarlin2020SuperGlueLF,Sun2021LoFTRDL,Jiang2021COTRCT}
instead leverage the greater image context to address the matching difficulty, e.g.,
SuperGlue~\cite{Sarlin2020SuperGlueLF} introduces a graph neural network (GNN) for descriptor matching on an image pair.
Graph edges connect keypoints from arbitrary locations
and enable reasoning in a broad context, leading to globally well
informed solutions compared to convolutional neural networks (CNN) with limited receptive field.
The receptive field in SuperGlue, however, remains limited by the two-view setup, despite that more images are typically available in pose estimation tasks.
Our idea is to further facilitate information flow by joining
multiple views into the matching process. This way, we allow multi-view correlation to strengthen geometric reasoning and confidence prediction.
Joint matching of multiple images integrates well into pose estimation pipelines, as they
 typically solve for more than two cameras.

Additionally, we note that accurate feature matching, in and of itself, does not necessarily give rise to accurate pose estimation, as the spatial distribution of feature matches is essential for robust pose optimization.
For instance, perfectly precise matches may form a degenerate case (e.g., lying on a line) and thus have no value for pose optimization.
In addition, confidence scores predicted by matching networks do not necessarily reflect the value of matches towards pose optimization.
Feature matching and pose estimation are thus tightly coupled problems, for which we propose a joint solution:

We encode
keypoints and descriptors from multiple images to construct a graph, where self-attention provides context awareness within the same image and cross-attention enables reasoning with respect to all other images. A GNN predicts matches along with confidence weights, which define constraints on the camera poses that we optimize with a differentiable Gauss-Newton solver. The GNN is trained end-to-end using gradients from the pose optimization. From this feedback, the network learns to produce valuable matches for pose estimation and thereby
learns effective outlier rejection.
We evaluate our method on the ScanNet~\cite{Dai2017ScanNetR3}, Matterport3D~\cite{Chang2017Matterport3DLF} and MegaDepth~\cite{Li2018MegaDepthLS} datasets and show that it improves over prior work on learned feature matching.

In summary, we demonstrate that a joint approach to feature matching and pose estimation benefits both matching and pose accuracy, enabled by the following contributions:
\begin{itemize}
    \item We propose a multi-view graph attention network to learn feature matches simultaneously across multiple frames.
    \item We introduce an end-to-end trainable pose estimation that both guides confidence weights of feature matches in an unsupervised fashion and backpropagates gradients to inform the graph-matching network.
\end{itemize}

\section{Related Work}
\subsubsection{Conventional Feature Matching.}
The classical feature matching pipeline comprises the following steps: 1) interest point detection, 2) feature description, 3) matching through nearest neighbor search in descriptor space, and 4) outlier filtering. In this pipeline, hand-crafted features like SIFT~\cite{LoweDavid2004DistinctiveIF} and ORB~\cite{Rublee2011ORBAE} are very successful and have been widely used for many years. However, they tend to struggle with appearance or viewpoint changes.
Starting with LIFT~\cite{Yi2016LIFTLI}, learning-based descriptors have been developed to tackle these challenges~\cite{Ono2018LFNetLL,Dusmanu2019D2NetAT,Revaud2019R2D2RA,Bhowmik2020ReinforcedFP,Tyszkiewicz2020DISKLL}. They often combine interest point detection and description, such as SuperPoint \cite{DeTone2018SuperPointSI}, which we use for our method.
Nearest neighbor feature matching is prone to outliers, making post-processing methods indispensable. This includes mutual check, ratio test \cite{LoweDavid2004DistinctiveIF}, neighborhood consensus \cite{Tuytelaars2000WideBS,Cech2008EfficientSC,Cavalli2020HandcraftedOD,Bian2017GMSGM,Ma2018LocalityPM} and sampling based outlier rejection~\cite{Fischler1981RandomSC,Barth2019MAGSACMS,Raguram2008ACA}.
Learning-based approaches have also addressed outlier detection as a classification task~\cite{Yi2018LearningTF,Ranftl2018DeepFM,Brachmann2019NeuralGuidedRL,Zhang2019LearningTC}. These methods rely on reasonable matching proposals and lack visual information in their decision process.

\subsubsection{Learning-based Feature Matching.}
Recent approaches employ neural networks for feature matching on image pairs. There are methods that determine dense, pixel-wise correspondences
with confidence estimates for filtering~\cite{Rocco2018NeighbourhoodCN,Rocco2020EfficientNC,Li2020DualResolutionCN}. This effectively combines steps (1)-(3) from the classical matching pipeline. However, it suffers from the limited receptive field of CNNs and fails to distinguish regions of little texture or repetitive structure, due to missing global context.
In contrast, SuperGlue \cite{Sarlin2020SuperGlueLF} represents a sparse matching network that operates on keypoints with descriptors. Using an attention-based GNN~\cite{Vaswani2017AttentionIA} all keypoints interact, hence the receptive field spans across both images, leading to accurate matches in wide-baseline settings. Inspired by the success of GNN-based feature matching, we build up on SuperGlue by further extending its receptive field through multi-view matching and by improving
outlier filtering through end-to-end training with pose optimization.
LoFTR \cite{Sun2021LoFTRDL} recently proposed a detector-free approach, that processes CNN features in a coarse-to-fine manner. Combined with attention it equally achieves a receptive field across the image pair and high quality matches.
COTR \cite{Jiang2021COTRCT}, like LoFTR, operates on images directly
in a coarse-to-fine fashion. It is a transformer network that predicts for a query point in one image the correspondence in a second image. This way, it considers the global context; however, inference for a complete image pair takes tens of seconds.
We show that our multi-view, end-to-end approach performs better than
SuperGlue and the detector-free methods LoFTR and COTR.

\subsubsection{Pose Optimization.}

Once matches between a set of images are found, poses can be optimized using a bundle adjustment formulation~\cite{triggs1999bundle}.
The optimization can be applied to a set of RGB images~\cite{agarwal2011building} or lifted to the RGB-D case, if depth data is available from range sensors~\cite{dai2017bundlefusion}.
The resulting optimization problems typically lead to non-linear least squares formulations which are optimized using non-linear solvers such as Gauss-Newton or Levenberg-Marquardt.
The pipeline in these methods usually performs feature matching as a pre-process; i.e., correspondences are established first and then filtered with a combination of RANSAC and robust optimization techniques~\cite{zach2014robust,Choi_2015_CVPR}.
However, feature matching and pose optimization largely remain separate steps and cannot inform each other.

To this end, differentiable optimization techniques have been proposed for pose estimation, such as DeMoN~\cite{ummenhofer2017demon}, BA-Net~\cite{tang2018ba}, RegNet~\cite{han2018regnet}, or 3DRegNet~\cite{pais20203dregnet}.
The core idea of these methods is to obtain gradients through the pose optimizations that in turn guide the construction of learned feature descriptors.
In comparison to treating feature extraction as a separate step, feature descriptors are now learned with the objective to obtain well-aligned global poses instead of just trying to get good pair-wise matches.
In our work, we go a step further and focus on learning how to match features rather than using a pre-defined matching method.

As a result, we can leverage differentiable pose optimization to provide gradients for our newly-proposed multi-view graph attention network for feature matching, and achieve significantly improved pose estimation results.

\section{Method}
Our method associates keypoints from $N$ images $\{I_n\}^{N}_{n=1}$, such that resulting matches and confidence weights are particularly valuable for estimating the corresponding camera poses $\{\mathbf{p}_n\}^{N}_{n=1}$, $\mathbf{p}_n \in \mathbb{R}^6$.
Keypoints are represented by their image coordinates $\mathbf{x} \in \mathbb{R}^2$, visual descriptors $\mathbf{d} \in \mathbb{R}^D$ and a confidence score $c \in [0, 1]$.
We use the SuperPoint network for feature detection and description, as it has shown to perform well in combination with learned feature matching \cite{DeTone2018SuperPointSI,Sarlin2020SuperGlueLF}. The source of input descriptors, however, is flexible; for instance, the use of conventional descriptors, such as SIFT \cite{LoweDavid2004DistinctiveIF}, is also possible.

Our pipeline, as shown in \cref{fig:pipeline}, ties together feature matching and pose optimization: we employ a GNN to associate keypoints across multiple images (\cref{ssec:multi_view_matching}). The resulting matches and confidence weights define constraints in the subsequent pose optimization (\cref{ssec:pose_optimization}), which is differentiable, thus enabling end-to-end training (\cref{ssec:end2end_training}).

\subsection{Graph Attention Network for Multi-View Matching}
\label{ssec:multi_view_matching}
\subsubsection{Motivation.}
In the multi-view matching problem of $N$ images, each keypoint matches to at most $N - 1$ other keypoints, where each of the matching keypoints has to come from a different input image.
Without knowing the transformations between images, one keypoint can match to any keypoint location in the other images. Hence, all keypoints in the other images need to be considered as matching candidates. Although keypoints from the same image are not matching candidates, they contribute valuable constraints in the assignment problem, e.g., their projection into other images must follow consistent transformations. The matching problem can be represented as a graph, where nodes model keypoints and edges their relationships.
A GNN architecture reflects this structure and enables learning the complex relations between keypoints to determine feature matches. The iterative message passing process enables the search for globally optimal matches as opposed to a greedy local assignment.
On top of that, attention-based message aggregation allows each keypoint to focus on information from the keypoints that provide the most  insight for its assignment.
We build upon the SuperGlue architecture, which introduces an attention-based GNN for descriptor matching between image pairs \cite{Sarlin2020SuperGlueLF}. Our extension to multi-image matching is motivated by the following considerations: first,  graph-based reasoning can benefit from tracks that are longer than two keypoints---i.e., a match becomes more confident, if multiple views agree on the keypoint similarity and its coherent location with respect to the other keypoints in each frame.
In particular, with regards to robust pose optimization, it is crucial to facilitate this information flow and boost the confidence prediction.
Second, pose estimation or SLAM systems generally consider multiple input views.
With the described graph structure, jointly
matching $N$ images is more efficient in terms of GNN messages than matching the corresponding image  pairs individually, as detailed in the following paragraph.
\subsubsection{Graph Construction.}
Each keypoint represents a graph node. The initial node embedding ${}^{(1)}\mathbf{f}_i$ of keypoint $i$ is computed from its image coordinates $\mathbf{x}_i$, confidence $c_i$ and descriptor $\mathbf{d}_i$ (\cref{eq:node_embedding}). This allows the GNN to consider spatial location, certainty and visual appearance in the matching process:
\begin{equation}
    {}^{(1)}\mathbf{f}_i = \mathbf{d}_i + F_\mathrm{encode}\left(\left[\mathbf{x}_i \mathbin\Vert c_i\right]\right),
    \label{eq:node_embedding}
\end{equation}
where $\mathbin\Vert$ denotes concatenation and $F_{\mathrm{encode}}$ is a multilayer perceptron (MLP) that lifts the image point and its confidence into the high-dimensional space of the descriptor. Such positional encoding helps the spatial learning \cite{Sarlin2020SuperGlueLF,Gehring2017ConvolutionalST,Vaswani2017AttentionIA}.
The graph nodes are connected by two kinds of edges: self-edges connect keypoints within the same image. Cross-edges connect keypoints from different images (\cref{fig:graph_edges}). The edges are undirected, i.e., information flows in both directions. \cref{tab:gnn_messages} shows that jointly matching $N$ images reduces the number of GNN messages compared to separately matching the corresponding $P=\sum_{n=1}^{N-1}n$ pairs. The savings result from fewer intra-frame messages between keypoints of the same image, e.g., for five images with $K$ keypoints each, pairwise matching involves $20K^2$ messages on a self-layer and $20K^2$ on a cross-layer---joint matching requires only $5K^2$ and $20K^2$, respectively.
\begin{figure}[tbp]
\centering
\includegraphics[width=0.9\textwidth,trim={4.8cm 9.9cm 4.3cm 5.7cm},clip]{figures/graph_edges.pdf}
\caption{Self- and cross-edges connected to a node $i$.}
\vspace{-0.4cm}
\label{fig:graph_edges}
\end{figure}
\setlength{\tabcolsep}{4pt}
\begin{table}[tb]
\begin{center}
\caption{Number of GNN messages per layer for matching $N$ images, each  with $K$ keypoints, as $P$ individual image pairs versus joint matching in a single graph.}
\label{tab:gnn_messages}
\begin{tabular}{lccc}
\toprule
& Messages along self-edges & Messages along cross-edges \\
\midrule
Pairwise matching & $2PK^2$ & $N(N-1)K^2$ \\
Joint matching & $NK^2$ & $N(N-1)K^2$ \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.6cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\subsubsection{Message Passing.}
Interaction between keypoints---the graph nodes---is realized through message passing \cite{Duvenaud2015ConvolutionalNO,Gilmer2017NeuralMP}. The goal is to achieve a state where node descriptors of matching keypoints are close in descriptor space, whereas unrelated keypoints are far apart. The GNN has $L$ layers, where each layer $\ell$ corresponds to a message exchange between keypoints. The layers alternate between updates along self-edges $\mathcal{E}_{\mathrm{self}}$ and cross-edges $\mathcal{E}_{\mathrm{cross}}$---starting with an exchange along self-edges in layer $\ell=1$ \cite{Sarlin2020SuperGlueLF}. \cref{eq:node_update} describes the iterative node descriptor update, where ${}^{(\ell)}\mathbf{m}_{\mathcal{E}\rightarrow i}$ is the aggregated message from all keypoints that are connected to keypoint $i$ by an edge in $\mathcal{E} \in \{\mathcal{E}_{\mathrm{self}}, \mathcal{E}_{\mathrm{cross}}\}$. ${}^{(\ell)}F_{\mathrm{update}}$ is a MLP, where each GNN layer $\ell$ has a separate set of network weights.
\begin{equation}
    {}^{(\ell+1)}\mathbf{f}_i = {}^{(\ell)}\mathbf{f}_i + {}^{(\ell)}F_{\mathrm{update}}\left(\left[{}^{(\ell)}\mathbf{f}_i \mathbin\Vert {}^{(\ell)}\mathbf{m}_{\mathcal{E}\rightarrow i}\right]\right)
    \label{eq:node_update}
\end{equation}
Multi-head attention \cite{Vaswani2017AttentionIA} is used to merge all incoming information for keypoint $i$ into a single message ${}^{(\ell)}\mathbf{m}_{\mathcal{E}\rightarrow i}$ \cite{Sarlin2020SuperGlueLF}.
Messages along self-edges are combined by self-attention between the keypoints of the same image, messages along cross-edges by cross-attention between the keypoints from all other images.
Linear projection of node descriptors is used to compute the query ${}^{(\ell)}\mathbf{q}_i$ of query keypoint $i$, as well as the keys ${}^{(\ell)}\mathbf{k}_j$ and values ${}^{(\ell)}\mathbf{v}_j$ of its source keypoints $j$:
\begin{align}
    {}^{(\ell)}\mathbf{q}_i &= {}^{(\ell)}\mathbf{W}_1 {}^{(\ell)}\mathbf{f}_i + {}^{(\ell)}\mathbf{b}_1 ,
    \label{eq:query} \\
    \begin{bmatrix}
    {}^{(\ell)}\mathbf{k}_j \\ {}^{(\ell)}\mathbf{v}_j
    \end{bmatrix}
    &=
    \begin{bmatrix}
    {}^{(\ell)}\mathbf{W}_2 \\ {}^{(\ell)}\mathbf{W}_3
    \end{bmatrix}
    {}^{(\ell)}\mathbf{f}_j +
    \begin{bmatrix}
    {}^{(\ell)}\mathbf{b}_2 \\ {}^{(\ell)}\mathbf{b}_3
    \end{bmatrix}.
    \label{eq:key_val}
\end{align}
The set of source keypoints $\{j : (i, j) \in \mathcal{E}\}$ comprises all keypoints connected to $i$ by an edge of the type, that is relevant to the current layer. $\mathbf{W}$ and $\mathbf{b}$ are per-layer weight matrices and bias vectors, respectively.
For each source keypoint the similarity to the query is computed by the dot product ${}^{(\ell)}\mathbf{q}_i\cdot{}^{(\ell)}\mathbf{k}_j$. The softmax over the similarity scores determines the attention weight $\alpha_{ij}$ of each source keypoint $j$ in the aggregated message to $i$:
\begin{equation}
    {}^{(\ell)}\mathbf{m}_{\mathcal{E}\rightarrow i} = \sum_{j : (i, j) \in \mathcal{E}} {}^{(\ell)}\alpha_{ij}{}^{(\ell)}\mathbf{v}_j .
\end{equation}
It is important to note that in layers, which update along cross-edges, the source keypoints $j$ to a  query keypoint $i$ come from multiple images. The softmax-based weighting is robust to variable number of input views and therewith variable number of keypoints.
After $L$ message passing iterations the node descriptors for subsequent assignment are retrieved by linear projection:
\begin{equation}
    \mathbf{f}_i = \mathbf{W}_4 {}^{(L+1)}\mathbf{f}_i + \mathbf{b}_4 .
    \label{eq:final_proj}
\end{equation}
\subsubsection{Partial Assignment.}
SuperGlue \cite{Sarlin2020SuperGlueLF} addresses the partial assignment problem between keypoints of two images, $I_1$ and $I_2$, where each keypoint either obtains a match in the other image or remains unmatched. A score matrix $\mathbf{S} \in \mathbb{R}^{(K_1+1)\times (K_2+1)}$ is defined, where $K_1$ and $K_2$ are the number of keypoints in the images, hence all potential matches and the unmatched option are represented. The elements $\mathbf{S}_{i,j}$ are filled with the dot-product similarity of the final node descriptors $\mathbf{f}_{1,i} \cdot \mathbf{f}_{2,j}$, where $\mathbf{f}_{1,i}$ is from $I_1$ and $\mathbf{f}_{2,j}$ from $I_2$.
The last row and column of $\mathbf{S}$, representing unmatched, are initialized with a trainable parameter $q \in \mathbb{R}$. The differentiable Sinkhorn algorithm \cite{Sinkhorn1967ConcerningNM,Cuturi2013SinkhornDL} optimizes for a soft assignment matrix $\mathbf{P} \in [0, 1]^{(K_1+1)\times (K_2+1)}$ that maximizes the sum of scores $\sum_{r,c}\mathbf{S}_{r,c}\mathbf{P}_{r,c}$ while obeying constraints on the number of matches:
\begin{equation}
    \mathbf{P}\mathbf{1}_{K_2+1} =
    \begin{bmatrix}
        \mathbf{1}_{K_1 + 1}^{\top} & K_2
    \end{bmatrix}^{\top} \quad \text{and} \quad
    \mathbf{P}^\top\mathbf{1}_{K_1+1} =
    \begin{bmatrix}
        \mathbf{1}_{K_2 + 1}^{\top} & K_1
    \end{bmatrix}^{\top}.
    \label{eq:sinkhorn_constraints}
\end{equation}
We adopt this approach and apply it pairwise to the images in the multi-view setting. $\mathcal{P}$ is the set of all possible image pairs from $\{I_n\}_{n=1}^N$, excluding pairs between identical images, as well as pairs that are a permutation of an existing pair. For each pair $(a,b) \in \mathcal{P}$, where $a,b \in \{1,2,\dots,N\}$, a score matrix $\mathbf{S}_{ab}$ is created and the assignment $\mathbf{P}_{ab}$ is computed by means of Sinkhorn algorithm. From $\mathbf{P}_{ab}$ the set of matches $\mathcal{M}_{ab}$ is derived: first, a candidate match for each keypoint in $I_a$ and $I_b$ is determined by the row-wise and column-wise maximal elements of $\mathbf{P}_{ab}$. Second, mutual agreement of matching keypoints is enforced.
\subsection{Differentiable Pose Optimization}
\label{ssec:pose_optimization}
We introduce a differentiable optimizer $\Omega$ that jointly estimates all camera poses from the matches determined by the partial assignment:
\begin{equation}
    \{\mathbf{p}_n\}_{n=1}^{N} = \Omega(\{\mathcal{M}_{ab}:(a,b) \in \mathcal{P}\}, \{Z_n\}_{n=1}^{N}) .
    \label{eq:optimizer}
\end{equation}
To stabilize the optimization, we use the depth maps $\{Z_n\}_{n=1}^{N}$ as additional input. Without depth measurements or good pose initialization, the optimization of bundle adjustment formulations is prone to fall into local minima.

We define the energy as weighted sum of squared errors between matches in world coordinates (\cref{eq:energy}). A match consists of the image coordinates, $\mathbf{x}_a$ in $I_a$ and $\mathbf{x}_b$ in $I_b$, as well as the matching confidence $w$, i.e., the corresponding element from the assignment $\mathbf{P}_{ab}$. The function $\pi_n^{-1}(\mathbf{x}_n, Z_n)$ unprojects an image point $\mathbf{x}_n$ in $I_n$ to homogeneous camera coordinates using its depth from $Z_n$. ${\mathbf{T}_{\mathbf{p}_n} \in \mathbb{R}^{3\times4}}$ defines the transformation from camera pose $\mathbf{p}_n$ to world coordinates. $\mathbf{p} \in \mathbb{R}^{6N}$ refers to the concatenation of all pose vectors, which are in $\mathfrak{se}(3)$ coordinates, i.e., three translation elements followed by three rotation elements.
\begin{align}
    E(\mathbf{p}) = \sum_{(a,b) \in \mathcal{P}} \; \sum_{(\mathbf{x}_a,\mathbf{x}_b,w) \in \mathcal{M}_{ab}} w^2 \left\Vert \mathbf{T}_{\mathbf{p}_a}\mathbf{y}_a - \mathbf{T}_{\mathbf{p}_b}\mathbf{y}_b \right\Vert_2^2&,
    \label{eq:energy} \\
    \text{where} \quad \mathbf{y}_a=\pi_a^{-1}(\mathbf{x}_a, Z_a) \quad \text{and} \quad \mathbf{y}_b=\pi_b^{-1}(\mathbf{x}_b, Z_b)&.
\end{align}
Gauss-Newton is used to minimize the energy with respect to the camera poses. For this purpose, a residual vector $\mathbf{r} \in \mathbb{R}^{3M}$ is created from the energy terms, where $M$ is the total number of matches between all images. Each match $m$ fills its corresponding subvector $\mathbf{r}_m \in \mathbb{R}^3$:
\begin{equation}
    \mathbf{r}_m = w(\mathbf{T}_{\mathbf{p}_a}\mathbf{y}_a - \mathbf{T}_{\mathbf{p}_b}\mathbf{y}_b).
\end{equation}
All poses are initialized to $\mathbf{0}$. We keep one pose fixed, which defines the world frame, and optimize for the remaining poses $\bar{\mathbf{p}}\in \mathbb{R}^{6(N-1)}$. The Jacobian matrix $\mathbf{J}\in \mathbb{R}^{3M\times6(N-1)}$ is initialized to $\mathbf{0}$ and filled with the partial derivatives with respect to the pose parameters: for each match $m$ the corresponding blocks $\mathbf{J}_{ma}, \mathbf{J}_{mb} \in \mathbb{R}^{3\times6}$ are assigned \cite{Blanco}:
\begin{equation}
    \mathbf{J}_{ma}=\frac{\partial \mathbf{r}_m}{\partial \mathbf{p}_a}=w\begin{bmatrix}
        \mathbf{I}_3 & \: & -\left(\mathbf{T}_{\mathbf{p}_a}\mathbf{y}_a\right)^\wedge
    \end{bmatrix} \enspace , \enspace
    \mathbf{J}_{mb}=\frac{\partial \mathbf{r}_m}{\partial \mathbf{p}_b}=w\begin{bmatrix}
        -\mathbf{I}_3 & \: & \left(\mathbf{T}_{\mathbf{p}_b}\mathbf{y}_b\right)^\wedge
    \end{bmatrix}.
    \label{eq:jac}
\end{equation}
$\mathbf{I}_3$ is a $3\times3$ identity matrix and $(\cdot)^\wedge$ maps a vector $\in \mathbb{R}^3$ to its skew-symmetric matrix: $\scriptsize \begin{bmatrix}
    x \\ y \\ z
\end{bmatrix}\rightarrow\begin{bmatrix}
    0&-z&y\\
    z&0&-x\\
    -y&x&0
\end{bmatrix}$.
If $a$ or $b$ identify the fixed pose, the corresponding assignment to $\mathbf{J}$ is skipped.
Using the current state of the camera poses, each Gauss-Newton iteration establishes a linear system, that is solved for the pose update $\mathrm{\Delta} \bar{\mathbf{p}}$ using LU decomposition:
\begin{equation}
    \mathbf{J}^\top\mathbf{J} \mathrm{\Delta} \bar{\mathbf{p}}=-\mathbf{J}^\top\mathbf{r}.
    \label{eq:gn_update}
\end{equation}
We update the poses in $T=10$ Gauss-Newton iterations, from which the set of poses with minimal energy is used for end-to-end training in \cref{ssec:end2end_training}.
\subsection{End-to-End Training}
\label{ssec:end2end_training}
The learnable parameters include the GNN parameters and the parameter $q$ of the partial assignment module.
The whole pipeline, from the matching network to the pose optimization, is differentiable, which allows for a pose loss that guides the matching network to produce valuable matches and accurate confidences for robust pose optimization. The training objective $\mathcal{L}$ consists of a matching term $\mathcal{L}_{\mathrm{match}}$ \cite{Sarlin2020SuperGlueLF} and a pose term $\mathcal{L}_{\mathrm{pose}}$, which are balanced by the factor $\lambda$:
\begin{align}
    \mathcal{L}&=\sum_{(a,b)\in \mathcal{P}}\mathcal{L}_{\mathrm{match}}(a,b)+\lambda \mathcal{L}_{\mathrm{pose}}(a,b), \quad \text{where}
    \label{eq:total_loss} \\
    \mathcal{L}_{\mathrm{match}}(a,b)&=-\sum_{(i,j)\in \mathcal{T}_{ab}}\log \mathbf{P}_{ab,i,j}-\sum_{i\in \mathcal{U}_{ab}}\log \mathbf{P}_{ab,i,K_b+1}-\sum_{j\in \mathcal{V}_{ab}}\log \mathbf{P}_{ab,K_a+1,j}, \nonumber \\
    \mathcal{L}_{\mathrm{pose}}(a,b)&=\left\Vert\hat{\mathbf{t}}_{a\rightarrow b}-\mathbf{t}_{a\rightarrow b}\right\Vert_2+\lambda_{\mathrm{rot}}\cos^{-1}\left(\frac{\mathrm{tr}(\mathbf{R}_{a\rightarrow b}^\top\hat{\mathbf{R}}_{a\rightarrow b})-1}{2}\right). \nonumber
\end{align}
$\mathcal{L}_{\mathrm{match}}$ computes the negative log-likelihood of the assignment between an image pair. The labels are computed using the ground truth depth maps, camera poses and intrinsic parameters: $\mathcal{T}_{ab}$ is the set of matching keypoints, $\mathcal{U}_{ab}$ and $\mathcal{V}_{ab}$ identify unmatched keypoints from $I_a$ and $I_b$, respectively.
$\mathcal{L}_{\mathrm{pose}}$ computes a transformation error between a pair of camera poses, where the translational and rotational components are balanced by $\lambda_{\mathrm{rot}}$. $\hat{\mathbf{R}}_{a\rightarrow b}$ and  $\hat{\mathbf{t}}_{a\rightarrow b}$ are a rotation matrix and translation vector computed from the pose optimization result (\cref{ssec:pose_optimization}). Rodrigues' formula is used to convert from axis-angle representation to rotation matrix.  $\mathbf{R}_{a\rightarrow b}$ and  $\mathbf{t}_{a\rightarrow b}$ define the ground truth transformation.
We use the Adam optimizer \cite{Kingma2015AdamAM}. Further details on the network architecture and training setup are provided in the supplementary material.
\section{Results}
We compare our method to baselines by evaluating indoor and outdoor pose estimation (\cref{ssec:pose_estimation}) and matching accuracy (\cref{ssec:matching}). \cref{ssec:ablation} shows the effectiveness of the added components in an ablation study. Runtime considerations are part of the supplementary material.
\subsection{Datasets}
\label{ssec:datasets}
\subsubsection{ScanNet \cite{Dai2017ScanNetR3}.}
Following the data generation in previous works \cite{Sarlin2020SuperGlueLF,Sun2021LoFTRDL}, we sample images from the video sequence, such that the overlap to the previous image lies in $[0.4, 0.8]$.
Instead of sampling a pair, we append three more images according to this overlap criterion. The resulting 5-tuples enable multi-view evaluation and provide a more realistic pose estimation scenario. The overlap is computed from ground truth poses, depth maps and intrinsic parameters.
\subsubsection{Matterport3D \cite{Chang2017Matterport3DLF}.}
Compared to ScanNet, Matterport3D view captures are much more sparse, i.e., neighboring images are $60\degree$ horizontally and $30\degree$ vertically apart. Hence, Matterport3D is a challenging dataset for the matching task.
To obtain a sufficient dataset size, we relax the overlap criterion to $[0.25, 0.8]$.
This challenging dataset, serves to measure robustness on the pose estimation task.
\subsubsection{MegaDepth \cite{Li2018MegaDepthLS}.}
As in prior work \cite{Sarlin2020SuperGlueLF,Dusmanu2019D2NetAT}, the overlap between images is the portion of co-visible 3D points of the sparse reconstruction, thus the overlap definition is different from the indoor datasets and not comparable. Overlap ranges $[0.1, 0.7]$ and $[0.1, 0.4]$ are used at train and test time, respectively \cite{Sarlin2020SuperGlueLF}.

\subsection{Pose Estimation}
\label{ssec:pose_estimation}
\setlength{\tabcolsep}{4pt}
\begin{table}[tb]
\centering
\caption{Baseline comparison and ablation study on wide-baseline indoor pose estimation on ScanNet; ``cross-dataset'' indicates that COTR was trained on MegaDepth.}
\label{tab:pose_scannet}
\resizebox{\linewidth}{!}{
  \begin{tabular}{l >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm}}
    \toprule
    & \multicolumn{3}{c}{Rotation error AUC [\%] $\uparrow$} & \multicolumn{3}{c}{Translation error AUC [\%] $\uparrow$} \\
    \cmidrule(lr){2-4}
    \cmidrule(lr){5-7}
    & @5\degree & @10\degree & @20\degree & @5cm & @10cm & @20cm \\
    \midrule
    Mutual nearest neighbor & 14.5 & 25.9 & 40.7 & 3.7 & 8.9 & 17.9 \\
    SuperGlue \cite{Sarlin2020SuperGlueLF} & 63.4 & 78.9 & 88.2 & 28.9 & 49.0 & 67.8 \\
    LoFTR \cite{Sun2021LoFTRDL} & 72.2 & 83.9 & 90.4 & 40.2 & 59.7 & 75.4  \\
    COTR \cite{Jiang2021COTRCT} cross-dataset & 46.2 & 60.5 & 72.0 & 20.9 & 36.1 & 51.8 \\
    Ours w/o multi-view & 68.7 & 81.9 & 89.6 & 35.8 & 56.7 & 73.6 \\
    Ours w/o end-to-end & 66.0 & 80.6 & 89.2 & 31.0 & 51.8 & 70.3 \\
    Ours & \textbf{72.5} & \textbf{84.6} & \textbf{91.5} & \textbf{41.5} & \textbf{61.8} & \textbf{77.5} \\
    \bottomrule
  \end{tabular}
  }
\vspace{0.4cm}
\caption{Baseline comparison and ablation study on wide-baseline indoor pose estimation on Matterport3D.}
\label{tab:pose_matterport}
\resizebox{\linewidth}{!}{
  \begin{tabular}{l >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm}}
    \toprule
    & \multicolumn{3}{c}{Rotation error AUC [\%] $\uparrow$} & \multicolumn{3}{c}{Translation error AUC [\%] $\uparrow$} \\
    \cmidrule(lr){2-4}
    \cmidrule(lr){5-7}
    & @5\degree & @10\degree & @20\degree & @5cm & @10cm & @20cm \\
    \midrule
    Mutual nearest neighbor & 0.6 & 2.0 & 5.4 & 0.0 & 0.1 & 0.3 \\
    SuperGlue \cite{Sarlin2020SuperGlueLF} & 18.5 & 29.6 & 41.7 & 3.4 & 8.5 & 16.9 \\
    Ours w/o multi-view & 27.2 & 38.0 & 49.1 & 6.5 & 14.2 & 24.6 \\
    Ours w/o end-to-end & 30.5 & 42.3 & 53.5 & 7.1 & 16.3 & 28.2  \\
    Ours & \textbf{42.4} & \textbf{55.4} & \textbf{66.2} & \textbf{12.2} & \textbf{24.9} & \textbf{39.8} \\
    \bottomrule
  \end{tabular}
  }
  \vspace{0.4cm}
  \caption{Baseline comparison and ablation study on wide-baseline outdoor pose estimation on MegaDepth. For comparison to LoFTR, we retrain and test our model on the LoFTR data split (bottom section of the table).}
\label{tab:pose_megadepth}
\resizebox{\linewidth}{!}{
\begin{tikzpicture}[very thick,squarednode/.style={rectangle, draw=none, fill=white, very thin, minimum size=2mm, text opacity=1,fill opacity=0}]
        \node[anchor=south west,inner sep=0] (image) at (0,0)
  {\begin{tabular}{p{0.3cm}l >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm} >{\centering\arraybackslash}p{1.2cm}}
    \toprule
    & & \multicolumn{3}{c}{Rotation error AUC [\%] $\uparrow$} & \multicolumn{3}{c}{Translation error AUC [\%] $\uparrow$} \\
    \cmidrule(lr){3-5}
    \cmidrule(lr){6-8}
    & & @5\degree & @10\degree & @20\degree & @5\degree & @10\degree & @20\degree \\
    \midrule
    & Mutual nearest neighbor & 14.3 & 27.8 & 44.2 & 6.6 & 14.6 & 26.5 \\
    & SuperGlue \cite{Sarlin2020SuperGlueLF} & 70.3 & 77.8 & 83.7 & 53.3 & 64.1 & 73.6 \\
    & COTR \cite{Jiang2021COTRCT} & 61.4 & 69.7 & 77.5 & 45.7 & 56.7 & 66.9 \\
    & Ours w/o multi-view & 74.4 & 80.8 & 86.1 & 58.5 & 68.8 & 77.4 \\
    & Ours w/o end-to-end & 74.5 & 81.6 & 87.0 & 57.8 & 68.9 & 77.8 \\
    & Ours & \textbf{81.1} & \textbf{86.8} & \textbf{91.2} & \textbf{67.7} & \textbf{76.6} & \textbf{83.6} \\
    \cmidrule(l){2-8}
    & LoFTR \cite{Sun2021LoFTRDL} & 75.2 & 83.0 & 88.6 & 60.5 & 71.3 & 79.7 \\
    & Ours & \textbf{89.6} & \textbf{93.6} & \textbf{95.9} & \textbf{74.1} & \textbf{82.3} & \textbf{88.3} \\
    \bottomrule
  \end{tabular}};
    \begin{scope}[x=(image.south east),y=(image.north west)]
        \node[squarednode,rotate=90] at (0.02, 0.5) (a) {\scriptsize Split A};
        \draw [decorate, decoration = {calligraphic brace}] (0.045,0.23) --  (0.045,0.74);
        \node[squarednode,rotate=90] at (0.02, 0.12) (a) {\scriptsize Split B};
        \draw [decorate, decoration = {calligraphic brace}] (0.045,0.02) --  (0.045,0.2);
        \end{scope}
    \end{tikzpicture}
  }
  \vspace{-0.4cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\begin{figure*}[tb]
\centering
    \begin{tikzpicture}[squarednode/.style={rectangle, draw=none, fill=white, very thin, minimum size=2mm, text opacity=1,fill opacity=0.}]
        \node[anchor=south west,inner sep=0] (image) at (0,0)
        {\includegraphics[width=\linewidth]{figures/scannet_results.jpg}};
        \begin{scope}[x=(image.south east),y=(image.north west)]
        \node[squarednode] at (0.05, 69/72) (a) {\bf 1};
        \node[squarednode] at (0.05, 57/72) (b) {\bf 2};
        \node[squarednode] at (0.05, 45/72) (c) {\bf 3};
        \node[squarednode] at (0.05, 33/72) (d) {\bf 4};
        \node[squarednode] at (0.05, 21/72) (e) {\bf 5};
        \node[squarednode] at (0.05, 9/72) (f) {\bf 6};
        \node[squarednode] at (0.165, -0.025) (g) {Input 5-tuples};
        \node[squarednode] at (0.4215, -0.025) (g) {SuperGlue \cite{Sarlin2020SuperGlueLF}};
        \node[squarednode] at (0.59, -0.025) (g) {LoFTR \cite{Sun2021LoFTRDL}};
        \node[squarednode] at (0.75, -0.025) (g) {COTR \cite{Jiang2021COTRCT}};
        \node[squarednode] at (0.915, -0.025) (g) {Ours};
        \end{scope}
    \end{tikzpicture}
\vspace{-0.6cm}
\caption{Reconstructions (right) from estimated camera poses on ScanNet 5-tuples (left). With multi-view matching and end-to-end training, our method successfully handles challenging pose estimation scenarios, while baselines show severe camera pose errors.}
\label{fig:scannet_results}
\vspace{-0.4cm}
\end{figure*}
\begin{figure*}[tb]
\centering
    \begin{tikzpicture}[squarednode/.style={rectangle, draw=none, fill=white, very thin, minimum size=2mm, text opacity=1,fill opacity=0}]
        \node[anchor=south west,inner sep=0] (image) at (0,0)
        {\includegraphics[width=\linewidth]{figures/matterport_results_short.jpg}};
        \begin{scope}[x=(image.south east),y=(image.north west)]
        \node[squarednode] at (0.05, 19/20) (a) {\bf 1};
        \node[squarednode] at (0.05, 15/20) (b) {\bf 2};
        \node[squarednode] at (0.05, 11/20) (c) {\bf 3};
        \node[squarednode] at (0.05, 7/20) (d) {\bf 4};
        \node[squarednode] at (0.05, 3/20) (e) {\bf 5};
        \node[squarednode] at (0.16, -0.026) (g) {Input 5-tuples};
        \node[squarednode] at (0.41, -0.026) (g) {SuperGlue \cite{Sarlin2020SuperGlueLF}};
        \node[squarednode] at (0.58, -0.026) (g) {Ours w/o};
        \node[squarednode] at (0.58, -0.057) (g) {multi-view};
        \node[squarednode] at (0.745, -0.026) (g) {Ours w/o};
        \node[squarednode] at (0.745, -0.057) (g) {end-to-end};
        \node[squarednode] at (0.915, -0.026) (g) {Ours};
        \end{scope}
    \end{tikzpicture}
\vspace{-0.6cm}
\caption{Reconstructions (right) from estimated camera poses on Matterport3D 5-tuples (left). Our complete method improves camera alignment over the ablated versions and SuperGlue, showing the importance of multi-view matching and end-to-end training.}
\label{fig:matterport_results}
\vspace{-0.4cm}
\end{figure*}
Prior work, in particular SuperGlue~\cite{Sarlin2020SuperGlueLF}, has extensively demonstrated the superiority of the GNN approach over conventional matching. Hence, we focus on comparisons to recent feature matching networks: SuperGlue~\cite{Sarlin2020SuperGlueLF}, LoFTR~\cite{Sun2021LoFTRDL} and COTR~\cite{Jiang2021COTRCT}. We additionally compare to a non-learning-based matcher, i.e., mutual nearest neighbor search on the SuperPoint \cite{DeTone2018SuperPointSI} descriptors. This serves to confirm the effectiveness of SuperGlue and our method, which both use SuperPoint descriptors.

For each method, the matches and confidences are used to optimize for the camera poses according to \cref{ssec:pose_optimization}. As the baselines are designed for matching image pairs, we run them repeatedly on all 10 possible pairs of the 5-tuples and use all resulting matches in the pose optimization.
The pose accuracy is evaluated based on the area under the curve (AUC) in \% at the thresholds $[5\degree, 10\degree, 20\degree]$ for rotation error and $[5\mathrm{cm}, 10\mathrm{cm}, 20\mathrm{cm}]$ for translation error on ScanNet and Matterport3D. As MegaDepth reconstructions are up to an unknown scale factor, the translation error is measured by the angle between translation vectors using thresholds $[5\degree, 10\degree, 20\degree]$ for the AUC. For qualitative comparison we use the computed poses to fuse the 5 depth maps in a truncated signed distance field (TSDF), which is then converted into a mesh using marching cubes \cite{Lorensen1987MarchingCA}.

Quantitative results on ScanNet are shown in \cref{tab:pose_scannet}, demonstrating that our method achieves higher accuracy than baselines.
The misalignments in the reconstructions (\cref{fig:scannet_results}) reveal that the baselines struggle in presence of repetitive patterns such as the washing machines (sample 1), the pictures on the wall (sample 5) or the patterned couches (sample 6). With multi-view reasoning during matching and learned outlier rejection through end-to-end training, our approach is more robust in these situations.

\cref{tab:pose_matterport} evaluates pose estimation on Matterport3D. The pose accuracy on Matterport3D is overall lower than on ScanNet, due to the smaller overlap between images and possibly amplified by the smaller training dataset.
In this scenario, our method outperforms SuperGlue with a larger gap than on ScanNet, which shows that our approach copes better with the more challenging setting in Matterport3D.
We show additional analysis in the ablation study.

Quantitative results on MegaDepth demonstrate the gain from multi-view matching and end-to-end training in the outdoor setting, leading to higher accuracy than baselines (\cref{tab:pose_megadepth}). Qualitative results are provided in the supplement.

\paragraphNoSpace{Implementation Details.}
COTR does not predict confidences, hence we use equal weights for all matches. For SuperGlue and LoFTR, the predicted confidences are used in the pose optimization, which we empirically found to perform better than thresholding. Further implementation detail is available in the supplemental.
\subsection{Matching}
\label{ssec:matching}
To avoid manual setting of confidence thresholds, the matching accuracy is evaluated by computing the weighted mean of the epipolar error $e$ on image pairs:
\begin{equation}
e = \frac{\sum_{m=1}^{M} w_m e_m}{\sum_{m=1}^{M} w_m},
\label{eq:epipolar_error}
\end{equation}
where $M$ is the number of matches between an image pair, $e_m$ the symmetric epipolar error of a match and $w_m$ its confidence.
The epipolar error and the average number of detected matches on ScanNet are listed in \cref{tab:match_scannet}.
As SuperGlue explicitly proposes a confidence threshold at 0.2 to determine valid matches, we also report this version of the baseline.
While our method achieves the lowest epipolar error, LoFTR produces a much higher number of matches.
This  shows that the number of matches is not a reliable indicator for pose accuracy, but rather accurate matches and confidences are beneficial.
\setlength{\tabcolsep}{4pt}
\begin{table}[tb]
\centering
\caption{Baseline comparison on wide-baseline matching accuracy on ScanNet.}
\label{tab:match_scannet}
  \begin{tabular}{lcc}
    \toprule
    & Number of matches & Epipolar error [m] $\downarrow$ \\
    \midrule
    Mutual nearest neighbor & 192 & 0.373 \\
    SuperGlue \cite{Sarlin2020SuperGlueLF} & 207 & 0.158 \\
    SuperGlue \cite{Sarlin2020SuperGlueLF} w/ threshold 0.2 & 189 & 0.032 \\
    LoFTR \cite{Sun2021LoFTRDL} & \textbf{1304} & 0.034 \\
    COTR \cite{Jiang2021COTRCT} & 96 & 0.069 \\
    Ours & 186 & \textbf{0.020} \\
    \bottomrule
  \end{tabular}
  \vspace{-0.4cm}
\end{table}
\setlength{\tabcolsep}{1.4pt}
\subsection{Ablation Study}
\label{ssec:ablation}
The quantitative results on ScanNet, Matterport3D and MegaDepth (\cref{tab:pose_scannet,tab:pose_matterport,tab:pose_megadepth}), show that the full version of our method achieves the best performance. This is consistent with the qualitative results in \cref{fig:matterport_results}.

\paragraphNoSpace{Without Multi-View.}
Omitting multi-view in the GNN causes an average performance drop of 3.9\% on ScanNet and 13.6\% on Matterport3D.
This suggests that the importance of multi-view increases with decreasing overlap between images. Intuitively, the multi-view receptive field supports information flow from other views to bridge gaps, where the overlap is small. \cref{fig:matterport_results} shows the notably improved camera alignment through multi-view input.

\paragraphNoSpace{Without End-to-End.}
Omitting end-to-end training drops the average performance by  6.8\% on ScanNet and 10.5\% on Matterport3D.
This shows that end-to-end training enables the learning of reliable outlier down-weighting, which is even more beneficial in the difficult Matterport3D scenarios.
Lack of end-to-end training is visible in the reconstructions (\cref{fig:matterport_results}), e.g., the misaligned pattern on the floor (sample 3) or the failure to reconstruct thin objects (sample 5).

\paragraphNoSpace{Variable Number of Input Views.} In \cref{fig:number_images}, we investigate the impact of the number of images used for matching, both in pairwise (ours w/o multi-view) and joint (our full version) fashion.
The experiment is conducted on sequences of 9 images which are generated on ScanNet as described in \cref{ssec:datasets}.
The results show that pose accuracy improves, when matching across a larger span of neighboring images. The curves, however, plateau when a larger window size does not bring any more relevant images into the matching.
Additionally, the results show the benefit of joint matching in a single graph as opposed to matching all possible image pairs individually.
\begin{figure}[tb]
  \centering
\begin{tikzpicture}[squarednode/.style={rectangle, draw=white, fill=white, very thin, minimum size=2mm, text opacity=1,fill opacity=0,draw opacity=0}]
        \node[anchor=south west,inner sep=0] (image) at (0,0)
        {\includegraphics[width=0.85\textwidth,trim={0.2cm 0.3cm -3.cm 0.2cm},clip]{figures/number_images.pdf}};
        \begin{scope}[x=(image.south east),y=(image.north west)]
        \scriptsize
        \node[squarednode] at (0.21, -0.06) (a) {Number of images};
        \node[squarednode] at (0.66, -0.06) (a) {Number of images};
        \node[squarednode]  at (0.97, 0.63) (a) {Pairwise matching};
        \node[squarednode] at (0.95, 0.47) (a) {Joint matching};
        \node[squarednode] at (0.215, 1.07) (b) {Rotation error AUC @10\degree [\%] $\uparrow$};
        \node[squarednode] at (0.665, 1.07) (b) {Translation error AUC @10cm [\%] $\uparrow$};
        \end{scope}
\end{tikzpicture}
   \vspace{-.2cm}
   \caption{Pose error AUC on sequences of 9 images on ScanNet using variable number of images in pairwise or joint matching.}
   \label{fig:number_images}
   \vspace{-.4cm}
\end{figure}

\subsection{Limitations}
Our method builds on SuperGlue~\cite{Sarlin2020SuperGlueLF} and improves pose estimation accuracy and robustness to small image overlap.
Here, one of our contributions is the end-to-end differentiablity of the pose optimization that guides the matching network.
While this significantly improves matching quality, we currently only backpropgate gradients to the matching network but do not update keypoint descriptors; i.e., we use existing SuperPoint~\cite{DeTone2018SuperPointSI}.
However, we believe that jointly training feature descriptors is a promising avenue to even further improve performance.

\section{Conclusion}
We have presented a method that couples multi-view feature matching and pose optimization into an end-to-end trainable pipeline.
Using a graph neural network, we match features across multiple views in a joint fashion, which increases global awareness of the matching process.
Combined with differentiable pose optimization, gradients inform the matching network, which learns to produce valuable, outlier-free matches for pose estimation.
The experiments show that our method improves both pose and matching accuracy compared to prior work. In particular, we observe increased robustness in challenging settings, such as in presence of repetitive structure or small image overlap.
Overall, we believe that our end-to-end approach is an important stepping stone towards an end-to-end trained SLAM method.
\section*{Acknowledgements}
This project is funded by a TUM-IAS Rudolf Mößbauer Fellowship, the ERC Starting Grant Scan2CAD (804724), and the German Research Foundation (DFG) Grant Making Machine Learning on Static and Dynamic 3D Data Practical.
We thank Angela Dai for the video voice-over.
\section{Qualitative Results on MegaDepth}
\cref{fig:megadepth_results} shows qualitative results from the ablation study and baseline comparison on MegaDepth dataset. The full version of our method accurately estimates camera poses even across large viewpoint changes (e.g., sample 4), and strong appearance variations (e.g., samples 1 and 2).
\begin{figure*}[b]
\centering
    \begin{tikzpicture}[squarednode/.style={rectangle, draw=none, fill=white, very thin, minimum size=2mm, text opacity=1,fill opacity=0.}]
        \node[anchor=south west,inner sep=0] (image) at (0,0)
        {\includegraphics[width=\linewidth]{figures/megadepth_results.jpg}};
        \begin{scope}[x=(image.south east),y=(image.north west)]
        \node[squarednode] at (0.037, 0.935) (a) {\bf 1};
        \node[squarednode] at (0.037, 0.685) (b) {\bf 2};
        \node[squarednode] at (0.037, 0.435) (c) {\bf 3};
        \node[squarednode] at (0.037, 0.185) (d) {\bf 4};
        \node[squarednode] at (0.115, -0.035) (g) {\footnotesize Input 5-tuples};
        \node[squarednode] at (0.31, -0.035) (g) {\footnotesize SuperGlue \cite{Sarlin2020SuperGlueLF}};
        \node[squarednode] at (0.47, -0.035) (g) {\footnotesize COTR \cite{Jiang2021COTRCT}};
        \node[squarednode] at (0.615, -0.035) (g) {\footnotesize Ours w/o};
        \node[squarednode] at (0.615, -0.075) (g) {\footnotesize multi-view};
        \node[squarednode] at (0.77, -0.035) (g) {\footnotesize Ours w/o};
        \node[squarednode] at (0.77, -0.075) (g) {\footnotesize end-to-end};
        \node[squarednode] at (0.925, -0.035) (g) {\footnotesize Ours};
        \end{scope}
    \end{tikzpicture}
\vspace{-0.6cm}
\caption{Reconstructions (right) from estimated camera poses on MegaDepth 5-tuples (left). With multi-view matching and end-to-end training, our method successfully estimates camera poses in challenging outdoor scenarios, while baselines show misalignment.}
\label{fig:megadepth_results}
\vspace{-0.4cm}
\end{figure*}

\section{Runtime}
Our method takes on average 207ms for matching a 5-tuple, which corresponds to matching 10 image pairs. SuperGlue requires on average 40ms for matching one pair, which shows that inference time correlates well with the number of GNN messages (\cref{tab:gnn_messages}). LoFTR takes on average 89ms for a pair and COTR is much slower with 35s.
Although we did not optimize our implementation for speed, the measurement shows that it is suited for real-time application.
We believe that the coupling of multi-view matching with pose optimization fits particularly well for keyframe alignment in reconstruction or SLAM. An alternative pose optimization module using relative poses, e.g., from inertial sensors, instead of depth measurements can also be realized.
All runtime is measured on a Nvidia GeForce RTX 2080.

\section{Architecture Details}
Our multi-view matching network builds up on the SuperGlue \cite{Sarlin2020SuperGlueLF} architecture and uses the following parameters:

\paragraphNoSpace{Keypoint Encoder.}
The input visual descriptors from SuperPoint \cite{DeTone2018SuperPointSI} have size $D = 256$. The graph nodes equally have an embedding size of $D$. Hence, the keypoint encoder $F_{\mathrm{encode}}$ maps a keypoint's image coordinates and confidence score to $D$ dimensions. It is a MLP,  composed of five layers with 32, 64, 128, 256 and $D$ channels. Each layer, except the last, uses batch normalization and ReLU activation.

\paragraphNoSpace{Graph Attention Network.}
The GNN has $L=9$ layers. The layers alternate between message exchange along self-edges and message exchange along cross-edges, such that the first and last layers perform updates along self-edges.
The attentional aggregation of incoming messages from other nodes uses multi-head attention with four heads. The resulting messages have size $D$, like the node embeddings.
The MLP $F_{\mathrm{update}}$, which computes the update to the receiving node, operates on the concatenation of the current node embedding with the incoming message. It has two layers with $2D$ and $D$ channels. Batch normalization and ReLU activation are employed between the two layers.

\paragraphNoSpace{Partial Assignment.}
We use 100 iterations of the Sinkhorn algorithm to determine the partial assignment matrices.

\paragraphNoSpace{Pose Optimization.}
The camera poses are optimized by conducting \mbox{$T=10$} Gauss-Newton updates.

\section{Training Details}
\subsubsection{Two-Stage Training.}
Our end-to-end pipeline is trained in two stages. The first stage uses the loss term on the matching result $\mathcal{L}_{\mathrm{match}}$. The second stage additionally employs the pose loss $\mathcal{L}_{\mathrm{pose}}$. Stage 1 is trained until the validation match loss converges, stage 2 until the validation pose loss converges. On ScanNet~\cite{Dai2017ScanNetR3}/ Matterport3D~\cite{Chang2017Matterport3DLF}/ MegaDepth~\cite{Li2018MegaDepthLS} the training takes 25/ 228/ 71 epochs for stage 1 and 4/ 7/ 11 epochs for stage 2. We found that the training on MegaDepth benefits from initializing the network parameters to the network parameters after the first ScanNet training stage. During stage 2 we linearly increase the weight of $\mathcal{L}_{\mathrm{pose}}$ from 0 to 2000 on the indoor datasets, and from 0 to 685 on MegaDepth, while linearly decreasing the weight of $\mathcal{L}_{\mathrm{match}}$ from 1 to 0, over a course of 40000 iterations. The balancing factor of the rotation term in $\mathcal{L}_{\mathrm{pose}}$ is set to $\lambda_{\mathrm{rot}}=2$ on the indoor datasets and $\lambda_{\mathrm{rot}}=6.75$ on MegaDepth.
We use the Adam optimizer with learning rate 0.0001.
The learning rate is exponentially decayed with a factor of 0.999992 starting after 100k iterations.

\subsubsection{Ground Truth Generation.}
The ground truth matches $\mathcal{T}_{ab}$ and sets of unmatched keypoints $\mathcal{U}_{ab}$, $\mathcal{V}_{ab}$ of an image pair are computed by projecting the detected keypoints from each image to the other, resulting in a reprojection error matrix. Keypoint pairs where the reprojection error is both minimal and smaller than 5 pixels in both directions are considered matches. Unmatched keypoints must have a minimum reprojection error greater than 15 pixels on the indoor datasets and greater than 10 pixels on MegaDepth.

\subsubsection{Input Data.}
We train on 5-tuples with a batch size of 24 on indoor data and with a batch size of 4 on outdoor data.
The image size is 480$\times$640 on ScanNet, 512$\times$640 on Matterport3D and 640$\times$640 on MegaDepth.
The SuperPoint network is configured to detect keypoints with a non-maximum suppression radius of 4/ 3 on indoor/ outdoor data.
On the indoor datasets we use 400 keypoints per image during training time: first, keypoints above a confidence threshold of 0.001 are sampled, second, if there are fewer than 400, the remainder is filled with random image points and confidence 0 as a data augmentation. On MegaDepth the same procedure is applied to sample 1024 keypoints using confidence threshold 0.005. At test time on indoor/ outdoor data, we use up to 1024/ 2048 keypoints above the mentioned confidence thresholds.

\subsubsection{Dataset Split.}
On ScanNet and Matterport3D we use the official dataset split. On Mega\-Depth we use scenes 0016, 0047, 0058, 0064, 0121, 0129, 0133, 0168, 0175, 0178, 0181, 0185, 0186, 0204, 0205, 0212, 0217, 0223, 0229 for validation, 0271, 0285, 0286, 0294, 0303, 0349, 0387, 0412, 0443, 0482, 0768, 1001, 3346, 5014, 5015, 5016, 5018 for testing and the remaining scenes for training.
This way on ScanNet/ Matterport3D/ MegaDepth we obtain 240k/ 20k/ 15k 5-tuples for training, 62k/ 2200/ 1200 for validation and 1500/ 1500/ 1000 for testing.

\section{Baseline Comparison}
In the baseline comparison we use the network weights trained by the authors of SuperGlue~\cite{Sarlin2020SuperGlueLF}, LoFTR~\cite{Sun2021LoFTRDL} and COTR~\cite{Jiang2021COTRCT}.
There are SuperGlue and LoFTR models trained on ScanNet and on MegaDepth, as well as a COTR model trained on MegaDepth. We additionally train a SuperGlue model on Matterport3D~\cite{Chang2017Matterport3DLF}.


\section{0pt}{12pt plus 4pt minus 2pt}{8pt plus 2pt minus 2pt}

\usepackage{amsthm,amsmath}
\RequirePackage{hyperref}
\usepackage[utf8]{inputenc}
\usepackage{nameref}

\def{}
\def{}
\usepackage{graphicx}
\graphicspath{{./figures/}}

\usepackage{subcaption}
\captionsetup{compatibility=false}

\usepackage{amsfonts}
\usepackage{dsfont}
\usepackage{xcolor}
\usepackage{lipsum}
\usepackage[nameinlink]{cleveref}

\crefname{equation}{Eq.}{Eqs.}
\Crefname{equation}{Equation}{Equations}
\crefname{paragraph}{Paragraph}{Paragraphs}

 \usepackage[multidot]{grffile}

\newcommand{\vv} {{\bm v}}
\newcommand{\begin{equation}}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\begin{eqnarray}}{\begin{eqnarray}}
\newcommand{\end{eqnarray}}{\end{eqnarray}}

\newcommand{\mathrm{e}}{\mathrm{e}}
\renewcommand{\L}{\mathcal{L}}
\renewcommand{\vec}[1]{#1}

\newcommand{\mbeq}{\stackrel{!}{=}}

\newcommand{\mathbf{s}}{\mathbf{s}}
\newcommand{\grad}{{\nabla}}

\let\vaccent=\v
\renewcommand{\v}[1]{\mathbf{#1}}
\renewcommand{\bf}[1]{\textbf{#1}}

\newcommand{\rightarrow}{\rightarrow}
\newcommand{\f}[2]{\frac{#1}{#2}}
\newcommand{\avg}[1]{\left\langle #1 \right\rangle}
\let\setunion=\ccup
\newcommand{\ccup}[1]{\left\{#1\right\}}
\let\setunion=\bup
\let\setunion=\rup
\newcommand{\bup}[1]{\left(#1\right)}
\newcommand{\rup}[1]{\left[#1\right]}

\let\oldt=\t
\renewcommand{\t}[1]{\tilde{#1}}
\newcommand{\omega}{\omega}
\newcommand{\bb}[1]{\mathbf{#1}}
\newcommand{\sigma}{\sigma}
\newcommand{\times}{\times}
\newcommand{\alpha}{\alpha}
\renewcommand{\b}[1]{\bar{#1}}
\newcommand{\h}[1]{\hat{#1}}

\newcommand{\mathbb{I}}{\mathbb{I}}
\newcommand{\mathbb{E}}{\mathbb{E}}
\newcommand{\mathbf{1}}{\mathbf{1}}

\usepackage[normalem]{ulem}
\newcommand{\CDB}[1]{\textcolor{blue}{#1}}
\newcommand{\CDBcom}[1]{[\textcolor{blue}{CDB: #1}]}
\newcommand{\DBT}[1]{\textcolor{red}{#1}}
\newcommand{\DBTcom}[1]{[\textcolor{red}{DT: #1}]}

\newcommand{\nix}[1]{\sout{#1}}

\newcommand{\mbox{{\small \textit{DMK-Solver}}}}{\mbox{{\small \textit{DMK-Solver}}}}
\newcommand{\mbox{{\small DMK}}}{\mbox{{\small DMK}}}
\newcommand{\mbox{{\small \textit{discrete-DMK-Solver}}}}{\mbox{{\small \textit{discrete-DMK-Solver}}}}

\begin{document}
\title{Convergence properties of optimal transport-based temporal hypernetworks}
\author{Diego Baptista}
\affiliation{ Max Planck Institute for Intelligent Systems, Cyber Valley, Tuebingen, 72076, Germany}
\author{Caterina De Bacco}
\affiliation{ Max Planck Institute for Intelligent Systems, Cyber Valley, Tuebingen, 72076, Germany}

\begin{abstract}

We present a method to extract temporal hypergraphs from sequences of 2-dimensional  functions obtained as solutions to Optimal Transport problems.  We investigate optimality principles exhibited by these solutions from the point of view of hypergraph structures. Discrete properties follow patterns that differ from those characterizing their continuous counterparts. Analyzing these patterns can bring new insights into the studied transportation principles. We also compare these higher-order structures to their network counterparts in terms of standard graph properties. We give evidence that some transportation schemes might benefit from hypernetwork representations. We demonstrate our method on real data by analyzing the properties of hypernetworks extracted from images of real systems.
\end{abstract}

\maketitle

\section*{Introduction}
Optimal Transport (OT) is a principled theory to compare probability distributions \cite{kantorovich1942transfer, villaniot, santambrogio2015optimal, peyre2019computational}. Although this task is usually framed as an  optimization problem, recent studies have mapped  it  within the framework of dynamic partial differential equations \cite{evans1999differential, facca2016towards, facca2019numerics, facca2021branching,tero2007mathematical, tero2010rules}. In this context, solutions to a transportation problem are often found as the convergent state of evolving families of functions.
\\
In some scenarios, the steady states of these evolving families are supported in network-shaped structures \cite{xia2003optimal, xia2014landscape, Xia2015}. Recently, this fact has called the attention of network scientists and graph theorists leading to the development of methods that convert the solutions of OT problems into actual graph structures \cite{baptista2020network,leite2022revealing}. This has broadened the available set of tools to understand and solve these transportation problems. Recent studies have shown that common patterns can be unveiled in both the original mathematical setting and  in the converted graph structures \cite{baptista2021temporal}.

Representations of these functions as sets of dyadic relations have been proven meaningful in various applications \cite{baptista2020principlednet,facca2021branching}.  Nonetheless, traditional dyadic representations may be limited in representing flows of quantities like mass or information as observed in real systems. Various examples of systems where interactions happen between 3 individuals or more are observed in applications as social contagion \cite{PhysRevResearch.2.023032,chowdhary2021simplicial}, random walks \cite{PhysRevE.101.022308,schaub2020random} or non-linear consensus \cite{PhysRevE.101.032310}. Understanding the relation between the structure and
 dynamics taking place on higher-order structures is an active field of research \cite{taylor2015topological,patania2017topological}. For instance, key elements controlling dynamics are linked to the heterogeneity of hyperedges' sizes present in their higher-order representations \cite{patania2017topological}. These systems are hence best described by hypergraphs, generalizations of networks that encode structured relations among any number of individuals. With this in mind, a natural question to ask is how do OT-based structures perform in terms of higher-order representations?
\\
To help bridge this knowledge gap about higher-order properties of structures derived from OT solutions, we elaborate on the results observed in \cite{baptista2021temporal}. Specifically, we propose a method to convert the families of 2-dimensional functions into temporal hypernetworks. We enrich the existing network structures associated with these functions by encoding the  observed interactions into hyperedges. We study classic hypergraph properties and compare them to the predefined cost functional linked to the transportation problems.  Finally, we extend this method and the analysis to study systems coming from real data. We build hypergraph representations of \textit{P. polycephalum} \cite{westendorf2016quantitative} and analyze their topological features.

\section*{Methods}\label{section:methods}
\subsection*{The Dynamical Monge-Kantorovich Method}
\paragraph*{The Dynamical Monge-Kantorovich set of equations.} We start by reviewing the basic elements of the mechanism chosen to solve the OT problems. As opposed to other standard optimization methods used to solve this \cite{cuturi2013sinkhorn}, we use an approach that turns the problem into a dynamical set of partial differential equations. In this way, initial conditions are updated until a convergent state is reached. The dynamical system of equations as proposed by Facca et al. \cite{facca2016towards,facca2019numerics,facca2021branching}, is presented as follows.  We assume that the OT problem is set on a continuous 2-dimensional space $\Omega  \in \mathbb{R}^{2}$, and at the beginning,  no  underlying network structure is observed. This gives us the freedom of exploring the whole space to design an optimal network topology, solution of the transportation problem. The main quantities that need to be specified in input are \textit{source} and \textit{target} distributions. We refer to them as sources and sinks, where a certain mass (e.g. passengers in a transportation network, water in a water distribution network) is injected and then extracted. We denote these with a ``forcing'' function $f(x)=f^+(x)-f^-(x)\in \mathbb{R}$,  describing the flow-generating sources $f^+(x)$ and sinks $f^-(x)$. To ensure mass balance it is imposed $\int_\Omega f(x)dx  = 0$. We assume that the flow is governed by a transient Fick-Poiseuille  flux $q=- \mu \grad u$, where $\mu,u$ and $q$ are called \textit{conductivity} (or \textit{transport density}), \textit{transport potential} and \textit{flux}, respectively. Intuitively, mass is injected through the source, moved based on the conductivity across space, and then extracted through the sink. The way mass moves determines a flux that depends on the pressure exerted on the different points in space; this pressure is described by a potential function.

The set of \textit{Dynamical Monge-Kantorovich} (DMK) equations is given by:
\begin{align}
-\nabla \cdot (\mu(t,x)\nabla u(t,x)) &= f^+(x)-f^-(x) \,, \label{eqn:ddmk1}\\
\frac{\partial \mu(t,x)}{\partial t}  &= \rup{\mu(t,x)\nabla u(t,x)}^{\beta} - \mu(t,x) \,, \label{eqn:ddmk2}\\
\mu(0,x)  &= \mu_0(x) > 0  \label{eqn:ddmk3} \,,
\end{align}
where $\nabla=\nabla_{x}$. \Cref{eqn:ddmk1} states the spatial balance of the Fick-Poiseuille flux and is complemented by no-flow Neumann boundary conditions. \Cref{eqn:ddmk2} enforces the dynamics of this system, and it is controlled by the so-called \textit{traffic rate} $\beta$. It determines the transportation scheme, and it shapes the topology of the solution: for $\beta<1$ we have congested transportation where traffic is minimized, whereas  $\beta>1$ induces branched transportation where traffic is consolidated into a smaller amount of space. The case $\beta=1$ recovers shortest path-like structures. Finally, \Cref{eqn:ddmk3} constitutes the initialization of the system and can be thought of as an initial guess of the solution.

Solutions $(\mu^*, u^*)$ of \crefrange{eqn:ddmk1}{eqn:ddmk3} minimize the transportation cost function $\mathcal{L}(\mu,u)$ \cite{facca2016towards,facca2019numerics,facca2021branching}, defined as:
\begin{align}\label{eqn:L}
& \mathcal{L}(\mu,u) := \mathcal{E}(\mu,u)+ \mathcal{M}(\mu,u) \\
& \mathcal{E}(\mu,u) := \dfrac{1}{2}\int_{\Omega} \mu |\grad u|^2 dx, \  \ \mathcal{M}(\mu,u) :=  \dfrac{1}{2}\int_{\Omega} \dfrac{\mu^{\frac{(2-\beta)}{\beta}}}{2-\beta} dx \quad.
\end{align}

$\mathcal{L}$ can be thought of as a combination of $\mathcal{M}$, the total energy dissipated during  transport (or network operating cost) and $\mathcal{E}$, the cost to build the network infrastructure (or infrastructural cost). It is known that this functional's convexity changes as a function of $\beta$. Non-convex cases arise in the branched schemes, inducing fractal-like structures \cite{facca2021branching,  santambrogio2007optimal}. This is the case that we considered in this work, and it is the only one where meaningful network  structures, and thus, hypergraphs, can be extracted \cite{baptista2020network}.

\subsection*{Hypergraph sequences}
\paragraph*{Hypergraph construction.} We define a hypergraph (also, hypernetwork) as follows \cite{battiston2020networks}: a \textit{hypergraph} is a tuple $H = (V, E),$ where $V = \{v_1, ... ,v_n\}$ is the set of \textit{vertices} and $E = \{ e_1, e_2, ... , e_m\}$ is the set of \textit{hyperedges} in which $e_i\subset V, \forall i = 1,...,m,$ and $|e_i|>1$.   If $|e_i|=2,\forall i$ then $H$ is simply a graph.  We call \textit{edges} those hyperedges $e_i$ with $|e_i|=2$ and \textit{triangles}, those with $|e_i|=3$.   We refer to the \textit{1-skeleton} of $H$ as the \textit{clique expansion} of $H$. This is the graph $G=(V,E_{G})$ made of the vertices $V$ of $H$, and of the pairwise edges built considering all the possible combinations of pairs that can be built from each set of nodes defining each hyperedge in $E$.

Let $\mu$ be the conductivity found as a solution of \crefrange{eqn:ddmk1}{eqn:ddmk3}. As previously mentioned, $\mu$  at convergence regulates where the mass should travel for optimal transportation. Similar  to \cite{baptista2021temporal}, we turn this 2-dimensional function into a different data structure, namely, a hypergraph.  This is done as follows: consider $G(\mu) = (V_G,E_G)$ the network extracted using the method proposed in \cite{baptista2020network}. We define $H(\mu)$ as the tuple $(V_H,E_H)$ where $V_H = V_G$ and $E_H = E_G \cup T_G,$ s.t., $T_G = \{(u,v,w):  (u,v),(v,w),(w,u) \in E_G, \}.$ In words, $H(\mu)$ is the graph $G(\mu)$ together with all of its triangles.
This choice is motivated by the fact that the graph-extraction method proposed in \cite{baptista2020network} uses triangles to discretize the continuous space $\Omega$, which can have a relevant impact on the extracted graph or hypergraph structures. Hence, triangles are the natural sub-structure for hypergraph constructions. The method proposed in this work is valid for higher-order structures beyond triangles. Exploring how these additional structures impact the properties of the resulting hypergraphs is left for future work.

\Cref{fig:image1} shows an example of one of the studied hypergraphs. The red shapes represent the different triangles of $H(\mu)$. Notice that, although we consider here the case where $|e|\leq 3$ for each hyperedge $e$---for the sake of simplicity---higher-order structures are also well represented by the union of these elements, as shown in the right panels of the figure.

\begin{figure}[!ht]
    \centering
\begin{subfigure}[b]{0.95\textwidth}
[width=\textwidth]{./fig1_family_of_subgraphs.jpg}
\end{subfigure}
\caption{\textbf{Hypernetwork construction.} Higher order structures are built using edges and triangles as hyperedges. The leftmost panel shows one of the studied graphs together with the triangles (in red) used. The subsequent panels highlight different clusters of triangles that can be seen in the main hypergraph.} \label{fig:image1}
\end{figure}

Since this hypergraph construction method is valid for any 2-dimensional transport density, we can extract a hypergraph not only from the convergent $\mu$ but also at any time step before convergence. This then allows us to represent optimal transport sequences as hypergraphs evolving in time, i.e. temporal hypernetworks.

\paragraph*{Hypergraph sequences.} Formally, let $\mu(x,t)$ be a \textit{transport density} (or \textit{conductivity}) function of both time and space obtained as a solution of the DMK model. We denote it as the sequence $\{\mu_t\}_{t=0}^T$, for some index $T$ (usually taken to be that of the convergent state). Each $\mu_{t}$ is the $t$-th update of our initial guess $\mu_0$, computed by following the rules described in \crefrange{eqn:ddmk1}{eqn:ddmk3}. This determines a sequence of hypernetworks $\{ H(\mu_t)\}_{t=0}^T$ extracted from $\{\mu_t\}_{t=0}^T$ with the extraction method proposed in \cite{baptista2020network}. \Cref{fig:image2} shows three hypergraphs built from one of the studied sequences $\{\mu_t\}$ using this method at different time steps. The corresponding OT problem is that defined by the (filled and empty) circles: mass is injected in the bottom left circle and must be extracted at the highlighted destinations. On the top row, different updates (namely, $t=12, 18, 26$) of the solution are shown. They are defined on a discretization of $[0,1]^2.$ Darkest colors represent their support. Hypergraphs extracted from these functions are displayed at the bottom row. As can be seen, only edges (in gray) and triangles (in red) are considered as part of $H(\mu_t)$. Notice that the larger the $t$ is, the less dense the hypergraphs are, which is expected for a uniform initial distribution $\mu_0$ and branched OT ($\beta>1$) \cite{facca2021branching}.

\begin{figure}[!ht]
    \centering
\begin{subfigure}[b]{0.9\textwidth}
[width=\textwidth]{./fig2_hypernet_sequence.jpg}
\end{subfigure}
\caption{\textbf{Temporal hypergraphs.}  Top row: different timestamps of the sequence $\{\mu_t\}$; triangles are a  discretization of $[0,1]^2$. Bottom row: hypergraphs extracted for $\mu_t$ at the time steps displayed on the top row; triangles are highlighted in red. In both rows, filled and empty circles correspond to the support of $f^+$ and $f^-$, i.e. sources and sinks, respectively. This sequence is obtained for $\beta = 1.5$.} \label{fig:image2}
\end{figure}

\subsection*{Graph and hypergraph properties}
We compare hypergraph sequences to their correspoding network counterparts (defined as described in the previous paragraph). We analyze the following main network  and hypergraph properties for the different elements in the sequences and for different sequences. Denote with $G = (V_G,E_G)$ and $H = (V_H, E_H)$ one of the studied graphs and hypergraphs belonging to some sequence $\{ G(\mu_t)\}_{t=0}^T$ and $\{ H(\mu_t)\}_{t=0}^T$, respectively. We consider the following network properties:
\\
\begin{enumerate}
\item $|E_G|$, total number of edges;
\item Average degree $d(G)$, the mean number of neighbors per node;
\item Average closeness centrality $c(G)$: let $v\in V_G$, the closeness centrality of $v$ is defined as $
\sum_{u\in V_G} 1/d(u,v),$ where $d(u,v)$ is the shortest path distance between $u$ and $v$.
\end{enumerate}

Hypernetwork properties can be easily adapted from the previous definitions with the help of generalized adjacency matrices and line graphs \cite{aksoy2020hypernetwork}.  Let $H$ be a hypergraph with vertex set $V = \{1,..,n\}$ and edge set $E = \{e_1, ... ,e_m\}$.  We define the generalized \textit{node} $s$-\textit{adjacency matrix} $A_s$ of $H$ as the binary matrix of size $n\times n$, s.t., $A_s[i][j]=1$ if $i$ and $j$ are part of at least $s$ shared hyperedges; $A_s[i][j]=0,$ otherwise. We define the $s$-\textit{line graph} $L_s$ as the graph generated by the adjacency matrix $A_s$.  Notice that $A_1$ corresponds to the adjacency matrix of $H$'s skeleton (which is $L_1$). \Cref{fig:image3} shows a family of adjacency matrices together with the line graphs generated using them.  We can then define hypergraphs properties in the following way:
\\
\begin{enumerate}
\item $|E_H|$, total number of hyperedges;
\item $|T| = |\{e \in E_H: |e|= 3\}|,$ total number of triangles;
\item $S = \sum_{t\in T} a(t),$ \textit{covered area}, where $a(t)$ is the area of the triangle $t;$
\item Average degree $d_s(H)$, the mean number of incident hyperedges of size greater or equal than $s$ per node;
\item Average closeness centrality $c_s(H)$: let $v\in V_H$, the closeness centrality of $v$ is defined as its closeness centrality in $L_s$.
\end{enumerate}

\begin{figure}[!ht]
    \centering
\begin{subfigure}[b]{\textwidth}
[width=0.98\textwidth]{./fig3_adj_matrices.jpg}
\end{subfigure}
\caption{\textbf{Adjacency matrices and line graphs.} Top: generalized node $s$-adjacency matrices for different values of $s$ from a given toy graph $G$. Bottom, from left to right: reference network $G$, and $s$-line graphs for $s=2,3,$ and $4$. } \label{fig:image3}
\end{figure}

$S$ can be defined in terms of any other property of a hyperedge, e.g. a function of its size $|e|$. Here we consider the area covered by a hyperedge to keep a geometrical perspective.  On the other hand, this area $S$ can be easily generalized to hyperedges with $|e_{i}|>3$ by suitably changing the set $T$ in the summation, e.g. by considering structures containing four nodes.  As for the centrality measures, we focus our attention to compare the case $s>1$ against $s=1$, as the latter traces back to standard graph properties and we are interested instead to investigate what properties are inherent to hypergraps. \Cref{fig:image4} shows values of the $d_s(H)$ and $c_s(H)$ for convergent hypergraphs $H$ (obtained from different values of $\beta$) together with the degree and closeness centrality of their correspondent graph versions. The considered hypergraphs are displayed in the top row of the figure.   As can be seen in the figure, patterns differ considerably for different values of $\beta$. As $s$ controls the minimum number of shared connections for different nodes in the networks, the higher this number, the more restrictive this condition becomes, thus leading to more disconnected line graphs. In the case of the $s$-degree centrality, we observe decreasing values for increasing $s$, with nodes with the highest centrality having much higher values than nodes less central. For both $s=2,3$ we observe higher values than nodes in $G$. This follows from the fact that once hyperedges are added to $G$, the number of incidences per node can only increase.  Centrality distributions strongly depend on $\beta$. For small values---more distributed traffic ($\beta=1.1$)---the number of hyperedges per node remains larger than the number of regular edges connected to it. But if traffic is consolidated on less space ($\beta=1.9$), then very few hyperedges are found. This suggests that the information learned from hypergraphs that is distinct to that contained in the graph skeleton is influenced by the chosen traffic regime.

As for the closeness centrality distribution, this resembles that of $G$  for small values of $\beta$, regardless $s$. For higher $\beta$ it switches towards an almost binary signal. Thus, nodes tend to become more central as $\beta$ increases, suggesting that adding hyperedges to networks $G$ leads to shorter distances between nodes.  The loss of information seen for the highest values of $s$ is due to the fact that the line graphs $L_s$ become disconnected with many small connected components. In these cases, the closeness centrality of a node is either 0 if it is isolated, or proportional to the diameter of the small connected component where it lives in.

\begin{figure}[!ht]
    \centering
[width=0.98\textwidth]{./fig4_side-by-side-hnx-deg-close.jpg}
\caption{\textbf{Graph and Hypergraph properties.}  Top row: optimal hypernetworks obtained with different traffic rates. Center and bottom rows: degree distributions and closeness distributions for the hypernetworks shown on the top row, and their 1-skeletons. The node labels in the $x$-axis of the center and bottom rows are sorted by their degree of centrality values.} \label{fig:image4}
\end{figure}

\paragraph{Convergence criteria.} Numerical convergence of the DMK \crefrange{eqn:ddmk1}{eqn:ddmk3}  is usually defined by fixing a threshold $\tau$. The updates are considered enough once the cost associated to them does not change more ($\leq \tau$) than that of the previous time step. As it is usually the case when this threshold is too small ($\tau=10^{-12}$ in our experiments), the cost or the network structure may consolidate to a constant value earlier than algorithmic convergence. Similar to \cite{baptista2021temporal}, to meaningfully establish when is hypergraph optimality reached, we consider as convergence time the first time step when the transport cost, or a given network property, reaches a value that is smaller or equal to a certain fraction $p$ of the value reached by the same quantity at algorithmic convergence (in the experiments here we use $p=1.05$). We refer to $t_\mathcal{L}$ and  $t_P$ for the convergence in times in terms of cost function or a network property, respectively.

\section*{Results}
To test the properties presented in the previous section and understand their connection to transportation optimality, we synthetically generate a set of optimal transport problems, determined by the configuration of sources and sinks.  As done in \cite{baptista2021temporal}, we fix a source's location and sample several points in the set $[0,1]^2$ to be used as sinks' locations. Let  $S = \{s_0,s_1,...,s_M\}$ be the set of locations in the space $[0,1]^2,$ and fix a positive number $0<r$. We define the distributions $f^+$ and $f^-$ as $
f^+(x) \propto  \mathds{1}_{R_0}(x),$ and   $f^-(x) \propto  \sum_{i>0} \mathds{1}_{R_i}(x),$  where $\mathds{1}_{R_i}(x) := 1,$ if $x\in R_i$, and $\mathds{1}_{R_i}(x) := 0$, otherwise; $R_i = C(s_i,r)$ is the circle of center $s_i$ and radius $r$. The value of $r$ is chosen based on the used discretization, and as mentioned before, the centers are sampled uniformly at random. The   symbol $\propto$ stands for proportionality and is used to ensure that $f^+$ and $f^-$ are both probability distributions.  The transportation cost is that of \cref{eqn:L}.

\paragraph{Synthetic OT problems.}\label{sec:synthetic}
The set of transportation problems considered in our experiments consists of 100 source-sink configurations. We place the location of the source $s_0=(0,0)$ (i.e. the support of $f^+$ at $(0,0)$), and sample 15 points $s_1,s_2,...,s_M$  uniformly at random from a regular grid. By sampling them from the nodes of the grid, we ensure that two different locations are at a safe distance so they are considered different once the space is discretized.  We initialize $\mu_0(x)=1, \forall x$ to be a uniform distribution on $[0,1]^2$. This can be interpreted as a non-informative initial guess for the solution. Starting from $\mu_0,$ we compute a maximum of 300 updates. Depending on the chosen traffic rate $\beta$ more or fewer iterations can be needed.  We claim that the sequence $\{\mu_t\}_{t=0}^T$ \textit{converges} to a certain function $\mu^*$ at iteration $T$ if either $|\mu_T-\mu_{T-1} |<\tau,$  for a \textit{tolerance} $\tau\in (0,1],$ or $T$ reaches the mentioned maximum.  For the experiments reported in this manuscript, the tolerance $\tau$ is set to be $10^{-12}$. Given the dependence of the solution of traffic constraints, a wide range of values of $\beta$ is considered. Namely, we study solutions obtained from low traffic cases ($\beta=1.1$, and thus, less traffic penalization) to large ones ($\beta=1.9$), all of them generating branched transportation schemes. Our 100 problems are linked to a total of 900 hypergraph sequences, each of them containing between 50 and 80 higher-order structures.

\begin{figure}[!h]
    \centering
[width=\textwidth]{./fig5_surface_decay.jpg}
\caption{\textbf{Covered area and Lyapunov cost.} Mean (markers) and standard deviations (shades around the markers) of the covered area $S$ (top plots) and  of the Lyapunov cost, energy dissipation $\mathcal{E}$ and structural cost $\mathcal{M}$ (bottom plots), as functions of time $t$. Means and standard deviations are computed on the set described in Paragraph \textit{Synthetic OT problems}. From left to right: $\beta=1.2, 1.5$ and $1.8$. Red and blue lines denote $t_P$ and $t_\mathcal{L}$.}  \label{fig:image5}
\end{figure}

\paragraph{Convergence: transport cost vs hypernetwork properties.}

As presented in \cite{baptista2021temporal}, we show a comparison between hypernetwork properties and the cost function minimized by the dynamics, where convergence times are highlighted (\Cref{fig:image5}). We focus on the property $S$, the area of the surface covered by the triangles in $H$. This quantity is influenced by both the amount of triangles (hence of hyperedges) and their distribution in space. Hence, it is a good proxy for how hypergraph properties change both in terms of iteration time and as we tune $\beta$.
We observe that $t_P>t_\mathcal{L}$ in all the cases, i.e. convergence in terms of transportation cost is reached earlier than the convergence of the topological property.  Similar behaviors are seen for other values of $\beta\in[1.1,1.9]$ and  other network properties (see Appendix). Similar to DMK-based network properties, the covered area's decay is faster for the smallest values of $\beta$. This is expected, given the convexity properties of $\mathcal{L}$ \cite{facca2016towards,facca2019numerics,facca2021branching}. However, the transport cost decays even faster, in a way that the value of $S$ is still far away from convergence in the congested transportation case (small $\beta$).
\\
 Notice that  $S$ remains stable after the first few iterations, and then it starts decreasing at different rates (depending on $\beta$) until reaching the converged value. This suggests that the dynamics tend to develop thick branches---covering a large area--- at the beginning of the evolution, and then it slowly compresses them until reaching the optimal topologies.
\\
These different convergence rates for $S$ and $\mathcal{L}$ may prevent construction of converged hypernetwork topologies: if the solver is stopped at $t_\mathcal{L}< t_{P}$, the resulting hypergraphs $H(\mu_t), \ t=t_\mathcal{L}$ would mistakenly cover a surface larger than that covered by the convergent counterpart ($H(\mu_t),$ for $t\geq t_P$). This scenario is less impactful for larger values of $\beta$, although in these scenarios $H$ is much more similar to a regular graph, because of the small number of higher-order structures. Topological differences between converged hypernetworks can be seen in \Cref{fig:image4}.
\\
Finally, we observe that both $t_\mathcal{L}(\beta)$ and $t_P(\beta)$ are increasing functions on $\beta$. This is expected since the larger the traffic rate is, the longer it takes for the sequences to converge. This particular behavior matches what is shown in \cite{baptista2021temporal} in the case of $t_\mathcal{L}$, but this is not the case for $t_P(\beta)$:  it was observed a non-monotonic behavior in the network case.

\paragraph{Convergence behavior of hypernetwork properties.}
\Cref{fig:image6} shows how the various network properties change depending on the traffic rate. Mean values and standard deviations are computed across times, for a fixed value of $\beta$. As shown, the number of hyperedges, number of triangles, covered area, and average 1-degree exhibit decreasing patterns as functions of $t$. As a consequence, transport optimality can be thought of as reaching minimum states on the mentioned hypernetwork properties. Another clear feature of these functions is related to the actual converged values: the larger the $\beta$ is, the smaller these metrics become. This is explained by a cost function increasingly encouraging consolidations of paths on fewer edges. Notice also that the gap between these converged values signals a non-linear dependence on the outputs of the dynamics; e.g., a converged hypernetwork obtained for $\beta=1.1.$ loses many more hyperedges if the traffic rate is then set to 1.2, whereas this loss would not be that large if $\beta=1.2$ is increased to 1.3. The nature of these gaps is substantially different depending on the property itself. This also shows that certain properties better reveal  the distinction between different optimal traffic regimes.

The behavior of the closeness centralities is distinctly different than that of the other properties. While its initial values are the same for all values of $\beta$ (similar to the previous properties), no clear trend can be found as time increases. For  $s=1$, on average $\beta=1.1$ generates sequences that tend to recover initial values after increasing and then decreasing behavior. For the other traffic rates, we observe different patterns. Notice that $s-$closeness centrality on the hypergraph for $s=1$ is the same as the classic closeness centrality on the skeleton of it. Thus, these rather noisy patterns are not due to the addition of hyperedges. On the other hand, for $s=2$ the average centrality  shows increasing curves. This may be due to $L_s$ getting increasingly disconnected with small connected components. Therefore, the larger $s$, the closer the nodes are seen (see \Cref{fig:image3}).  Moreover, in this case small values of $\beta$ lead to more stable closeness centrality values, showing the impact of $\beta$ in  building higher-order structures. While different values of $\beta$ lead to different behaviors of the hypergraph properties (e.g. decreasing degrees and amount of hyperedges for increasing $\beta$) we remark that choosing the value of $\beta$ should depend on the application at hand. The analysis performed here showcases how this choice may impact the resulting topologies. This can help practitioners to visualize possible consequences in terms of downstream analysis on the transportation properties of the underlying infrastructure.

\begin{figure}[!h]
    \centering
        [width=0.8\textwidth]{./fig6_single_fig_dpp.jpg}
    \caption{\textbf{Evolution of hypernetwork properties}. Mean (markers) and standard deviations (shades around the markers) of number of hyperedges $|E_H|$ (upper left), number of triangles  $|T|$ (upper center), covered area $S(H)$ (upper right), average $2$-degree $d_2(H)$ (lower left), average $1$-closeness centrality $c_1(H)$(lower center) and $2$-closeness centrality $c_2(H)$(lower right), computed for different values of $\beta$ as a function of time.}\label{fig:image6}
\end{figure}

\section*{\textit{P. polycephalum} hypernetworks}

We now analyze hypernetworks extracted from images of real data. We are interested in the evolution of the area covered by triangles in the sequences $ \{ H(\mu_t)\}_{t=0}^T$ extracted from real images of the slime mold \textit{P. polycephalum}. The behavior of this organism is the inspiration of the modeling ideas of the DMK equations described in \nameref{section:methods}. It has been shown that these slime molds follow a similar optimization strategy as that captured by the DMK dynamics while foraging for food in 2D surfaces \cite{nakagaki2000maze,tero2007mathematical,tero2010rules}.
  We extract hypernetworks from images using the idea described in  \nameref{section:methods} but instead of applying \cite{baptista2020network} to obtain the networks, we use the method proposed by \cite{baptista2020principlednet} which takes images as input. This pipeline uses the color intensities of the different image pixels to build a graph, by connecting adjacent meaningful nodes. We dedicate our attention to 4 image sequences from the Slime Mold Graph Repository \cite{dirnberger2017introducing}. The sequences are then describing the evolution of a \textit{P. polycephalum} placed in a rectangular Petri dish. Each image, and thus each hypernetwork, is a snapshot of the movement of this organism over periods of 120 seconds.

We study the covered area for every one of the 4 sequences, and plot the results for one of them (namely, image set \textit{motion12}; see Appendix) in \Cref{fig:image7}. We highlight 4 times along the property sequence to display the used images together with the corresponding hypernetworks. The lower leftmost plot shows a subsection of one of the studied snapshots. As can be seen there, this subhypernetwork topology exhibits a significant number of hyperedges of dimension 3, mainly around the thickest parts of the slime mold. On the other side, in the lower rightmost plot,  the evolution of $S$ is overall decreasing in time (similar results are obtained for other sequences,  as shown in the Appendix). This suggests that the thicker body parts tend to get thinner as the \textit{P. polycephalum} evolves into a consolidated state.  This pattern resembles what is shown above for the synthetic data, i.e. the covered area tends to decrease as time evolves similar to the behavior of the DMK-based hypernetwork sequence.  This suggests that the DMK model realistically mirrors a consolidation phase towards optimality of real slime molds \cite{dirnberger2017introducing}.

\begin{figure}[!ht]
    \centering
\begin{subfigure}[b]{1\textwidth}
[width=0.95\textwidth]{./fig7_hnx_surface_phys_insets_for_motion12.jpg}
\end{subfigure}
\caption{\textbf{\textit{P. polycephalum} hypergraphs.} On top: \textit{P. polycephalum} images and hypernetworks extracted from them. Bottom left: a zoomed-in part of the hypergraph shown inside the red rectangle on top. Bottom right: covered area as a function of time. The red shade highlights a tentative consolidation phase towards optimality.} \label{fig:image7}
\end{figure}

\section*{Conclusions}
We proposed a method to build higher-order structures from OT sequences. This method maps every member of the sequence into a hypergraph, outputting a temporal hypernetwork. We analyzed standard hypergraph properties on these temporal families and compared them to their continuous counterparts. We showed that convergence in terms of transportation cost tends to happen faster than that given by the covered area of the hypernetworks. This suggests that the dynamics used to solve the OT problems concentrates the displaced mass into main branches, and once this task is carried out, it slightly reduces the area covered by them. We studied this and other hypergraph properties, and compared them to those of their network versions. In some cases, hypernetworks reveal more information about the topology at convergence. This suggests that hypernetworks could be a better alternative representation to solutions of OT problems for some transportation schemes. The conclusions found in this work may further enhance our comprehension of OT solutions and the links between this field and that of hypergraphs.

\paragraph{Acknowledgements
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Diego Baptista.

\bibliographystyle{splncs03}