\section{Introduction}

Orthonormal transformations play a key role in most matrix decomposition techniques and spectral methods \cite{Belabbas369}. As such, manipulating them in an efficient manner is essential to many practical applications. While general matrix-vector multiplications with orthogonal matrices take $O(d^2)$ space and time, it is natural to ask whether faster approximate computations (say $O(d \log d)$) can be achieved while retaining enough accuracy.\\
Approximating an orthonormal matrix with just a few building blocks is hard in general. The standard decomposition technique meant to reduce complexity is a low-rank approximation. Unfortunately, for an orthonormal matrix, which is perfectly conditioned, this approach is meaningless.\\
In this work, we are inspired by the fact that several orthonormal/unitary transformations that exhibit low numerical complexity are known. The typical example is the discrete Fourier transform with its efficient implementation as the fast Fourier transform \cite{doi:10.1137/1.9781611970999} together with other Fourier-related algorithms: fast Walsh-Hadamard transforms \cite{1674569}, fast cosine transforms \cite{1163351}, and fast Hartley transforms \cite{1457236}. Other approaches include fast wavelet transforms \cite{doi:10.1002/cpa.3160440202}, banded orthonormal matrices \cite{SIMON2007120, Strang12413} and fast Slepian transforms \cite{KARNIK2019624}. Decomposition of orthogonal matrices into $O(d)$ Householder reflectors or $O(d^2)$ Givens rotations \cite{Golub1996}[Chapter 5.1] are known already. These basic building blocks have been further extended, for example we have fast Givens rotations \cite{Rath1982} and two generalizations of the Givens rotations \cite{BILOTI201356} and \cite{GGR}. In theoretical physics, unitary decompositions parametrize symmetry groups \cite{Tilma:2002ke}, and they are compactly parametrized using $\sigma$-matrices \cite{Spengler_2010} or symmetric positive definite matrices \cite{Barvinok}. To the best of our knowledge, none of these factorizations focus on reducing the computational complexity of using the orthogonal/unitary transformations but rather they model properties of physical systems.\\
Our idea is to approximately factor any orthonormal matrix into a product of a fixed number of sparse matrices such that their application (to a vector)
has linearithmic complexity. In this paper, we derive structured approximations to orthonormal matrices that can be found efficiently and applied remarkably fast. We pose the search for an efficient approximation as an optimization problem over a product of structured matrices, the extended orthogonal Givens transformations. These structures extend Givens rotations to also include reflectors with no computational drawback and suggest a decomposition of the main optimization problem into sub-problems that are easy to understand and solved via a greedy approach. The theoretical properties of the obtained solution are characterized in terms of approximation bounds while the empirical properties are studied extensively.\\
We illustrate our approach  considering dimensionality reduction with principal component analysis (PCA).
Here the goal is not to propose a new fast algorithm to compute the principal directions, plenty of efficient algorithms  for this task \cite{FastMCSVD}\cite{Golub1996}[Chapter~10] \cite{FindingStructureWithRandomness}. Rather, we aim at constructing fast dimensionality reduction operators.
While the calculation of the principal components is a one-off computation, a numerically efficient projection operation is critical since it is required multiple times in downstream applications. The problem of deriving fast projections has also been previously studied. Possible approaches include: fast wavelet transforms \cite{WaveletsForPCA}, sparse PCA \cite{MINIMAXOPTIMALSPCA, SparsePCA}, structured transformations such as circulant matrices \cite{CirculantDRoperator}, Kronecker products \cite{KroneckerPCA}, Givens rotations \cite{DCTandKLT, Treelets, LeCunFastApproximations} or structured random projections \cite{FastJL, ToeplitzJL}. Compared to these works, we propose a new way to factorize any orthogonal matrix, including PCA directions, into simple orthogonal structures that we call extended orthogonal Givens transformations and which naturally lead to optimization problems that have closed-form solutions and are therefore efficiently computed.\\
We note that our approach provides new perspectives on the structure of the orthogonal group and how to coarsely approximate it, which might have an impact on other open research questions.\\% where orthogonal matrices are involved.\\
The paper is organized as follows: Section 2 describes the basic building blocks and algorithm that we propose, Section 3 gives the theoretical guarantees for our contributions, Section 4 details the application of our method to PCA projections and Section 5 shows the numerical experiments.

\section{The proposed algorithm}
	Given a $d\times d$ orthonormal $\mathbf{U}$, the matrix-vector multiplication $\mathbf{Ux}$ takes $O(d^2)$ operations. We want to build $\mathbf{\bar{U}}$ such that $\mathbf{U} \approx \mathbf{\bar{U}}$ and $\mathbf{\bar{U}x}$ takes $O(d \log d)$ operations. Parametrizations of orthonormal matrices \cite{doi:10.1137/0908055} are known, but to be best of our knowledge, the problem of accurately approximating $\mathbf U$ as  product of only a few $O(d \log d)$ transformations is open. Given a $d \times p$ diagonal $\mathbf{\Sigma}_p$, in the spirit of previous work minimizing Frobenius norm approximations \cite{Kondor2014MMF, lemagoarou:hal-01416110}, we consider the problem
	\begin{equation}
	\underset{\mathbf{\bar{U}}, \ \mathbf{\bar{\Sigma}}_p}{\text{minimize}}\ \| \mathbf{U} \mathbf{\Sigma}_p - \mathbf{\bar{U}}\mathbf{\bar{\Sigma}}_p  \|_F^2 \text{ subject to } \mathbf{\bar{U}} \in \mathcal{F}_g,
	\label{eq:theoptimizationproblem_only_orthonormal}
	\end{equation}
	where $\mathbf{\bar{U}}$ is $d \times d$ and $\mathbf{\bar{\Sigma}}_p$ is a $d \times p$ diagonal matrix. Choosing $p = d$ while $\mathbf{\Sigma}_p$ and $\mathbf{\bar{\Sigma}}_p$ to be the identity, we simply approximate $\mathbf U$. We will also use $\mathbf{U}_p$ do denote the first $p \leq d$ columns of $\mathbf{U}$.
The above general formulation allows to also consider cases where  different directions might have different importance.
Then, $\mathcal{F}_g$ is a  set of orthogonal matrices -- defined next -- that can be applied fast and allow to efficiently but approximately solve
\eqref{eq:theoptimizationproblem_only_orthonormal}.

\subsection{The basic building blocks}

Classic matrix building blocks that are numerically efficiency include  circulant/Toeplitz matrices or Kronecker products. These choices are inefficient as they depend on $O(d)$ free parameters but their matrix-vector product cost is $O(d \log d)$ or even $O(d \sqrt{d})$, i.e., they do not scale linearly with the number of parameters they have. Consider the sparse orthogonal matrices
\begin{equation}
\mathbf{G}_{ij} = \begin{bmatrix} \mathbf{I}_{i-1} &  & &\\
& * & & *  \\
& & \mathbf{I}_{j-i-1} & \\
& * &  & *  \\
& & & & \mathbf{I}_{d-j} \\
\end{bmatrix}, \mathbf{\tilde{G}}_{i j} \in \left\{ \! \begin{bmatrix} c & -s \\ s & c \end{bmatrix}\!,\! \begin{bmatrix} c & s \\ s & -c	\end{bmatrix}   \! \right\},\text{ such that } c^2 + s^2 = 1,
\label{eq:theG}
\end{equation}
where the non-zero part (denoted by $*$ and $\mathbf{\tilde{G}}_{i j}$) on rows and columns $i$ and $j$. The transformation in \eqref{eq:theG}, with the first option in $\mathbf{\tilde{G}}_{i j}$, is a Givens (or Jacobi) rotation. With the second option, we have a very sparse Householder reflector. These transformations were first used by \citet{FastSparsifyingTransforms} to learn numerically efficient sparsifying dictionaries for sparse coding. The $\mathbf{G}_{ij}$s have the following advantages: i) they are orthogonal; ii) they are sparse and therefore fast to manipulate: matrix-vector multiplications $\mathbf{G}_{ij}\mathbf{x}$ take only $6$ operations; iii) there are two degrees of freedom to learn: $c$ (or $s$) and the binary choice; and iv) allowing both sub-matrices in $\mathbf{\tilde{G}}_{i j}$ enriches the structure and as we will see, leads to an easier (closed-form solutions) optimization problem.\\
We propose to consider matrices $\mathbf{\bar{U}} \in \mathcal{F}_g$ that are   products of $g$ transformations from \eqref{eq:theG}, that is
\begin{equation}
\mathbf{\bar{U}} =
 \prod_{k=1}^{g} \mathbf{G}_{i_k j_k} = \mathbf{G}_{i_1 j_1}  \dots \mathbf{G}_{i_g j_g}.
\label{eq:approxu}
\end{equation}
Matrix-vector multiplication with $\mathbf{\bar{U}}$ takes $6g$ operations -- when $g$ is $O(d \log d)$ this is significantly better than $O(d^2)$, while the coding complexity of each $\mathbf{G}_{i j}$ is approximately $2\log_2 d + C$: $2\log_2 d -1$ bits to encode the choice of the two indices, a constant factor $C$ for the pair $(c,s)$ and 1 bit for the choice between the rotation and reflector. The coding complexity of $\mathbf{\bar{U}}$ scales linearly with $g$.\\
	We note that Givens rotations have been used extensively to build numerically efficient transformations \cite{5560826, EffectiveGivens, Kondor2014MMF, lemagoarou:hal-01416110, Treelets}. However,  $2 \times 2$ reflector was not used before. This may be because in linear algebra (e.g. in QR factorization) and in optimization \cite{Shalit14} considering also the reflector has no additional benefit: the rotation alone introduces each zero in the QR factorization and the reflector does not have an exponential mapping on the orthogonal manifold, respectively. As we will show, considering both the rotation and the reflector has the advantage of providing a closed-form solution to our problem.

\subsection{The proposed greedy algorithm}

We propose to solve the optimization problem in \eqref{eq:theoptimizationproblem_only_orthonormal} with a greedy approach: we keep $\mathbf{\bar{\Sigma}}_p$ and all variables fixed except for a single $\mathbf{G}_{i_k j_k}$ from $\mathbf{\bar{U}}$ and minimize the objective function. When optimizing w.r.t.  $\mathbf{G}_{i_k j_k}$ it is convenient to write
\begin{equation}
\! \| \mathbf{U}\mathbf{\Sigma}_p  \! \! - \! \mathbf{\bar{U} \bar{\Sigma}}_p \|_F^2
\!  = \!  \|
\underbrace{ \prod_{t=1}^{k-1} \mathbf{G}_{i_t j_t}^T
\mathbf{U} \mathbf{\Sigma}_p \! \!  }_{{\mathbf L}^{(k)}}
- \mathbf{G}_{i_k j_k}
\underbrace{
 \prod_{t=k+1}^{g} \mathbf{G}_{i_t j_t}
 \mathbf{\bar{\Sigma}}_p}_{{\mathbf N}^{(k)}} \|_F^2 \!   = \! \| \mathbf{L}^{(k)} \! - \! \mathbf{G}_{i_k j_k} \mathbf{N}^{(k)} \|_F^2.
\label{eq:thestart}
\end{equation}
The next result characterizes the Givens transformation $\mathbf{G}_{i_k j_k}$ minimizing the above norm. We drop the dependence on $k$ for ease of notation.

\begin{theorem}
[Locally optimal $\mathbf{G}_{ij}$]
Let $\mathbf{L}$ and $\mathbf{N}$ be two $d \times p$ matrices. Further, let  $\mathbf{Z} = \mathbf{L} \mathbf{N}^T$ and $\mathbf{Z}_{\{i,j\}} = \begin{bmatrix} Z_{ii} & Z_{ij}  \\ Z_{ji} & Z_{jj} \end{bmatrix}$ then we have
\begin{equation}
\ C_{ij} = \| \mathbf{Z}_{ \{i,j\} } \|_* - \text{tr}(\mathbf{Z}_{ \{ i,j \} }) = \left\{ \begin{array}{lr} \sqrt{ (Z_{ii} + Z_{jj})^2 + (Z_{ij} - Z_{ji})^2} -Z_{ii} -Z_{jj} , \text{if } \det( \mathbf{Z}_{ \{ i,j \} } )  \geq 0 \\ \sqrt{ (Z_{ii} - Z_{jj})^2 + (Z_{ij} + Z_{ji})^2} -Z_{ii} -Z_{jj}, \text{if } \det( \mathbf{Z}_{ \{ i,j \} } )  < 0 \end{array}\right.
\label{eq:theCijcomputed}
\end{equation}
Let $\mathbf{Z}_{\{i^\star,j^\star\}} \! = \! \mathbf{V}_1 \mathbf{S  V}
_2^T$ be the SVD of $\mathbf{Z}_{\{i^\star,j^\star\}}$ , with
\begin{equation}
(i^\star, j^\star) = \underset{(i, j),\ j > i}{\arg \max}\ \ C_{ij}.
\label{eq:theCij}
\end{equation}
Then, the  extended orthogonal Givens transformation that minimizes $\| \mathbf{L} - \mathbf{G}_{i j} \mathbf{N} \|_F^2$ is given by
$\mathbf{\tilde{G}}_{i^\star j^\star}^\star = \mathbf{V}_1 \mathbf{V}_2^T.$
\end{theorem}

The above theorem  derives a locally optimal way to construct an approximation $\mathbf{\bar{U}}$. We iteratively apply the result to find, for each component $k$ in \eqref{eq:approxu}, the extended orthogonal Givens transformation that best minimizes the objective function \eqref{eq:theoptimizationproblem_only_orthonormal}. The full procedure is  in Algorithm 1 and can be viewed in two different ways: i) a coordinate minimization algorithm; or ii) a hierarchical decomposition where each stage is extremely sparse. The proposed algorithm is guaranteed to converge, in the sense of the objective function \eqref{eq:theoptimizationproblem_only_orthonormal}, to a stationary point. Indeed, no step in the algorithm can increase the objective function, since the sub-problems are minimized exactly: we choose the best indices and then perform the best $2 \times 2$ transformation. We note three remarks on the properties of Algorithm 1.
\begin{rem}[Complexity of Algorithm 1]
The computational complexity of the iterative part of Algorithm 1 is $O(d g)$ and the initialization is dominated by the computation of all the scores $C_{ij}$ which takes $O(d^2)$. Note that, the $C_{ij}$s are computed from scratch only once in the initialization phase. After that,  at each step $k$ we need to  recompute the $C_{ij}$ (redo the $2\times 2$ singular value decompositions) only for the indices $(i_k, j_k)$  currently used (the Givens transformations act on two coordinates at a time). All other scores are update by the same quantity: in \eqref{eq:theCij}, the $C_{ij}$ are the same except when $i$ or $j$ belong to the set $(i_k, j_k)$. This observation substantially reduces the running time.
\end{rem}
\begin{rem}[Complexity of applying $\mathbf{\bar{U} \mathbf{\bar{\Sigma}}}_p$]
When  $p <   d$ the computational complexity of $6g$ operations is an upper bound. Since we keep only  $p$ components, we need  be careful  not to perform operations whose result is thrown away by the mask $\mathbf{\bar{\Sigma}}_p$. Consider for example a transformation $\mathbf{G}_{1d}$ applied to a vector of size $d$  projected to a $p < d$ dimensional space. The three operations that take place on the $d^\text{th}$ component are unnecessary. Then, after  computing $\mathbf{\bar{U}}$, a pass is made through each of the $g$ transforms to decide which of two coordinates the computations are necessary for the final result. As we will show, this further improves the numerical efficiency of our method.
\end{rem}
\begin{rem}[On the choice of indices] Algorithm 1 greedily chooses at each step $k$ the indices  according to \eqref{eq:theCij}. Other factors might be considered: i) choosing indices based on previous choices so that only a select group of indices are used throughout the algorithm, or ii) make multiple choices at each step in order to speed up the algorithm.
\end{rem}
\begin{algorithm}[tb]
	\caption{Approximate orthonormal matrix factorization with extended Givens transformations}
	\label{alg:fpca}
	\begin{algorithmic}
		\State {\bfseries Input:} The $p$ orthogonal components $\mathbf{U}_p$ and their weights (singular values) $\mathbf{\Sigma}_p$, the size $g$ of the approximation \eqref{eq:approxu}, the update rule for $\mathbf{\bar{\Sigma}}_p$ in \{ `identity', `original', `update' \} and the stopping criterion $\epsilon$ (default taken to be $\epsilon = 10^{-2}$).

		\State {\bfseries Output:} The linear transformation $\mathbf{\bar{U}} \mathbf{\mathbf{\bar{\Sigma}}}_p$, the approximate solution to \eqref{eq:theoptimizationproblem_only_orthonormal}.
		\State {\bfseries Initialize:} $\mathbf{G}_{i_k j_k} \! = \! \mathbf{I}_{d \times d},k\! =\! 1,\dots,g$ and compute all scores $C_{ij}$ according to \eqref{eq:theCijcomputed} with $\mathbf{Z} = \mathbf{U}_p \mathbf{\bar{\Sigma}}_p^T$, where $\mathbf{\bar{\Sigma}}_p = \begin{bmatrix} \mathbf{I}_{p \times p}; & \mathbf{0}_{(d-p) \times p} \end{bmatrix}$ if the update rule is `identity' and $\mathbf{\bar{\Sigma}}_p = \mathbf{\Sigma}_p$ otherwise.
		\Repeat
		\State Set $\mathbf{L}^{(0)} = \mathbf{U}_p \mathbf{\Sigma}_p$ and set $\mathbf{N}^{(0)} = \mathbf{G}_{i_1 j_1} \dots \mathbf{G}_{i_g j_g} \mathbf{\bar{\Sigma}}_p$.

		\For{$k = 1$ {\bfseries to} $g$ }
		\State Update $\mathbf{N}^{(k)} = \mathbf{G}_{i_k j_k}^T \mathbf{N}^{(k-1)}$ and find best score according to \eqref{eq:theCij}.
		\State Compute the best $k^\text{th}$ transformation by Theorem 1.
		\State Update $\mathbf{L}^{(k)} = \mathbf{G}_{i_k j_k}^T \mathbf{L}^{(k-1)}$ and then update all scores $C_{ij}$ in \eqref{eq:theCijcomputed} but only for indices $i, j \in \{i_k, j_k\}$ with $\mathbf{Z}\! = \! \mathbf{L}^{(k)} (\mathbf{N}^{(k)})^T$ -- all other scores are unchanged.
		\EndFor

		\State Set $\mathbf{\bar{\Sigma}}_p = \begin{bmatrix}
		\text{diag}(\mathbf{L}^{(g)}); & \mathbf{0}_{(d-p) \times p}
		\end{bmatrix}$ if rule is `update', $i \leftarrow i+1$ and $\epsilon_i = \| \mathbf{L}^{(g)} - \mathbf{\bar{\Sigma}}_p \|_F^2 $.
		\Until $ |\epsilon_{i-1} - \epsilon_i| < \epsilon$, if $i > 1$.
	\end{algorithmic}
\end{algorithm}

\section{Analysis of the proposed algorithm}

We consider $p = d$, i.e., $\mathbf{\bar{\Sigma}}_p = \mathbf{I}_{d \times d}$ and therefore $\mathbf{Z} = \mathbf{U}$. We model the $\mathbf{U}$ as a random orthonormal matrix with Haar measure \cite{HowToRandomUnitary} updated so that the diagonal is positive. We perform this update because multiplication by a diagonal matrix with $\pm 1$ entries has no computational cost but it brings $\mathbf{U}$ closer to $\mathbf{I}_{d \times d}$.
The goal of this section is to establish upper bounds for the distance between $\mathbf{U}$ and $\mathbf{\bar{U}}$, as a function of $d$ and $g$. We first comment  on the inherent difficulty of the problem.

\begin{rem}[The approximation gap] Since  the orthogonal group has size $O(d^2)$, by the pigeonhole principle  a random orthogonal matrix as \eqref{eq:approxu} and only $g \ll d^2$ degrees of freedom cannot be exactly  approximated with less than $O(d^2)$ operations.
For our purposes, think $g$ either $O(d)$ or $O(d \log d)$.
Our goal is to show that the fast structures we propose can perform well in practice and have theoretical bounds that guarantee worse case or average accuracy.
\end{rem}
\noindent Next, we show two approximations bounds depending on the number of Givens transformations \eqref{eq:theG}.
\begin{theorem}[A special bound] Given a random $d \times d$ orthonormal $\mathbf{U}$, for large $d$, its approximation $\mathbf{\bar{U}}$ from \eqref{eq:theoptimizationproblem_only_orthonormal} with $g = d/2$ transformations  from \eqref{eq:theG} obeys
\begin{equation}
\mathbb{E}[ \| \mathbf{U} - \mathbf{\bar{U}} \|_F^2] \leq 2d - \sqrt{2\pi d}.
\end{equation}
\end{theorem}
\begin{theorem}[A general bound] Given a random $d \times d$ orthonormal $\mathbf{U}$, for large $d$, its approximation $\mathbf{\bar{U}}$ from \eqref{eq:theoptimizationproblem_only_orthonormal} with $g \leq d(d-1)/2$ transformations from \eqref{eq:theG} is bounded by
\begin{equation}
\mathbb{E}  \left[ \| \mathbf{U} - \mathbf{\bar{U}} \|_F^2 \right] \leq 2(d - \lfloor r \rfloor) - \frac{2\sqrt{2}}{\sqrt{\pi}} \sqrt{d - \lfloor r \rfloor},\text{ where } r = d - \frac{1 + \sqrt{(2d-1)^2 - 8g}}{2}.
\label{eq:theresultsofu3}
\end{equation}
\end{theorem}
Theorem 2 shows  that, on average, the performance might degrade with increasing $d$. As  stated in Remark 4, this is not surprising since the orthonormal group is much larger than the structure we are trying to approximate it with. The next result provides a bound for  other values of $g$. In Theorem 3, taking $g = c_1 d \log d$ for some positive constant $c_1$ we have that $r \approx c_1 \log d$. This means that whenever $p \ll d$ we will roughly need $O(d)$ Givens transformations from \eqref{eq:theG} to improve the $\lfloor r \rfloor$ term. Since the proof of the theorem uses only rotations (and furthermore, in a particular order of indices $(i_k,j_k)$) we expect our algorithm to perform much better than the bound indicates as it allows for a richer structure \eqref{eq:theG} and uses greedy steps that maximally improve the accuracy at each step.\\
The previous theorems consider  the Frobenius norm. In the Jacobi iterative process for diagonalizing a symmetric matrix with Givens rotations \cite{Golub1996}[Chapter~8.4] the progress of the procedure (convergence) is measured using the off-diagonal ``norm'' $\text{off}(\mathbf{U}) = \sqrt{\sum_{t}^d \sum_{q \neq t}^d U_{tq}^2}$.\\
\begin{theorem}[Convergence in the off-diagonal norm] Given a $d \times d$ orthonormal $\mathbf{U}$ and a single Givens transformation $\mathbf{G}_{ij}$, assuming $\text{det}(\mathbf{U}_{ \{i,j\} }) \geq 0$ we have
\begin{equation}
\text{off}(\mathbf{UG}_{ij}^T )^2 \leq \text{off}(\mathbf{U})^2 \! + \frac{1}{2}((U_{ii} - U_{jj})^2- (U_{ij} - U_{ji})^2).
\end{equation}
\end{theorem}
This result shows that, unlike with the Jacobi iterations, monotonic convergence in this quantity is not guaranteed and depends on the relative differences between the diagonal and the off-diagonal entries of $\mathbf{U}_{ \{i,j\} }$. Our method convergence monotonically to a stationary point when we measure the progress in the Frobenius norm.

\begin{rem}[The effect of a single $\mathbf{G}_{ij}$] Given a $d \times d$ orthonormal $\mathbf{U}$ and a Givens transformation $\mathbf{G}_{ij}$ from \eqref{eq:theG} we have that: i) $\mathbf{U}\mathbf{G}_{i j}^T$ is closer to the identity matrix in the sense that $\mathbf{\tilde{G}}_{ij}$ makes a positive contribution to the diagonal elements, i.e.,  $\text{tr}(\mathbf{U}\mathbf{G}_{i j}^T) = \text{tr}(\mathbf{U}) + C_{i j}$; and ii) $\mathbb{E}[C_{ij}] \approx 0.6956 d^{-1/2}$ if $\mathbf{U}$ is random with Haar measure and positive diagonal for large $d$.
\end{rem}
The above remark  suggests a metric  to study the convergence of the proposed method: each Givens transformation adds the score $C_{ij}$ to the diagonal entries of the current approximation (and therefore ensures that $\mathbf{U \bar{U}}^T $ converges to the identity -- the only diagonal orthonormal matrix). By choosing the maximum $C_{ij}$ we are taking the largest step in this direction.
\begin{rem}[Evolution of $C_{i_kj_k}$ with $k$] Given a fixed $0 < u < 1$, consider the toy construction $\mathbf{U}_{ \{i,j\} } = \begin{bmatrix} u & z_2 \\ z_1 & u \end{bmatrix}$, i.e., a $2 \times 2$ sub-matrix of a $d \times d$ orthonormal matrix where diagonal elements are equal and off-diagonals are two independent truncated standard normal random variables in the interval $[-\sqrt{1-u^2}, \sqrt{1-u^2}]$ (since rows and columns of $\mathbf{U}$ are $\ell_2$ normalized). Then, for large $d$, by direct calculation we have that the  expected score $\mathbb{E}[ C_{ij}(u) ]$, i.e., $C_{ij}$ as a function of $u$, obeys
\begin{equation}
\begin{aligned}
\mathbb{E}[C_{ij}(u)] \propto (1-u)^2,
\end{aligned}
\end{equation}
i.e., the expected $C_{ij}$ decreases on average quadratically with the increase in the diagonal elements.\\
The remark is intuitive: as $k$ increases $\mathbf{\bar{U}}$ is more accurate and $\mathbf{U\bar{U}}^T$ becomes diagonally dominant, i.e., $\mathbf{U\bar{U}}^T \to \mathbf{I}_d$ as $k \to O(d^2)$, and we do expect to reach lower scores $C_{i_k j_k}$, i.e., few Givens transformations from \eqref{eq:theG} provide a rough estimation while very good approximations require $k \approx d^2$.
\end{rem}

\subsection{Other ways to measure the approximation error}

Throughout this paper we use the Frobenius norm to measure and study the approximation error we propose. In this section we discuss this choice and explore a few alternatives.\\
In the linear algebra literature, a natural way to measure approximation error is through the operator norm. This is especially true in the randomized linear algebra field (where matrix concentration inequalities which bound the operator norm play a central role). Moreover, the review manuscript \citep{10.1561/2200000048} deals explicitly with some potential issues that might arise from using the Frobenius, instead of the operator, norm in matrix approximations: the discussion in Chapter 6 of \citep{10.1561/2200000048} entitled ``Warning: Frobenius--Norm Bounds''. The text highlights situations where the matrix to be approximated is low rank and corrupted by noise and/or scaling issues are present. While that discussion holds true, in our case we deal with a given perfectly conditioned matrix $\mathbf{U}$ and its approximation displays the same scaling ($\mathbf{\bar{U}}$ has normalized columns just as $\mathbf{U}$). Consider the following remark.

\noindent \textbf{Remark 7. [Simplifying the Frobenius norm objective function]} \textit{Consider for simplicity $p = d$ and $\mathbf{\bar{\Sigma}}_d = \mathbf{\Sigma}_d$ in \eqref{eq:theoptimizationproblem_only_orthonormal} and see that $\| (\mathbf{U} - \mathbf{\bar{U}})\mathbf{\Sigma}_d  \|_F^2 = 2\sum_{i=1}^d \sigma_i (1 - \mathbf{u}_i^T \mathbf{\bar{u}}_i) = 2\sum_{i=1}^d \sigma_i (1 - \cos (\theta_i))$,
	where $\theta_i$ is the angle between column vectors $\mathbf{u}_i$ and $\mathbf{\bar{u}}_i$ (the columns of $\mathbf{U}$ and $\mathbf{\bar{U}}$, respectively), and $\sigma_i > 0$ are the diagonal elements of $\mathbf{\Sigma}_d$.}\hfill$\square$

Therefore, the proposed optimization objective function minimizes the weighted sum of the cosines of the angles between the original columns and their approximations. As such, low-rank or scaling issues cannot arise. The only potential problem is that different columns might have very different approximation errors (i.e., there could exist $i$ such that $\cos(\theta_i) \approx 1$ while there could be some $j$ for which $\cos(\theta_j) \ll 1$). This issue can be mitigated by increasing $g$ in \eqref{eq:approxu} or choosing carefully the indices $(i_k, j_k)$ where the proposed transformations operate.

Assume that we consider the operator norm as the approximation error, i.e., we want to minimize $\|\mathbf{U} - \mathbf{\bar{U}}\|_2$ in \eqref{eq:theoptimizationproblem_only_orthonormal}. We have the following two results.

\noindent \textbf{Remark 8. [The spectrum of the error matrix]} \textit{Consider $p = d$ and $\mathbf{\bar{\Sigma}}_d = \mathbf{\Sigma}_d = \mathbf{I}_d$ in \eqref{eq:theoptimizationproblem_only_orthonormal} then for any orthonormal $\mathbf{U}$ and $\mathbf{\bar{U}}$ the error matrix $\mathbf{U} - \mathbf{\bar{U}}$ is normal and has all its eigenvalues on a circle of radius one centered at $(1,0)$ in the complex plane. As such, we have that $\| \mathbf{U} - \mathbf{\bar{U}} \|_2 \leq 2$.}\hfill$\square$

\noindent \textbf{Theorem 5. [A bound on the operator norm of the error matrix]} \textit{Consider $p = d$ and $\mathbf{\bar{\Sigma}}_d = \mathbf{\Sigma}_d = \mathbf{I}_d$ in \eqref{eq:theoptimizationproblem_only_orthonormal} and that $\mathbf{u}^T_i \mathbf{\bar{u}}_i \geq 0$ for all $i=1,\dots,d$, then the operator norm of the error matrix obeys $\| \mathbf{U} - \mathbf{\bar{U}} \|_2 \leq 1-\epsilon_\text{min} + \sqrt{(d-1) (1-\epsilon_\text{min}^2) }$ where $\epsilon_\text{min}  = \underset{i}{\min} \ \mathbf{u}^T_i \mathbf{\bar{u}}_i = \underset{i}{\min} \ \cos(\theta_i)$. The bound in Remark 8 is met when $\epsilon_{\min} \geq (d-2)/d$.}\hfill$\blacksquare$

Remark 8 describes the full spectrum of the error matrix. It is interesting to notice that the upper bound $\| \mathbf{U} - \mathbf{\bar{U}} \|_2 \leq 2$ coincides with the expectation result from \cite{CollinsMale2011}: as $d \to \infty$ if both $\mathbf{U}$ and $\mathbf{\bar{U}}$ are chosen uniformly at random with Haar measure then almost surely $\|\mathbf{U} + \mathbf{\bar{U}} \| \to 2$. Theorem 5 describes an upper bound on the operator norm of the error matrix. The bound is tight only for very high values of $\epsilon_{\min}$ and therefore is meant to give a qualitative measure of the approximation. As in Remark 7, the key quantity is $\cos(\theta_i)$ but now the bound depends on the worst approximation: while the Frobenius norm objective function minimizes the sum of the pairwise distances between the columns of $\mathbf{U}$ and $\mathbf{\bar{U}}$, when using the operator norm the approximation accuracy depends on the largest distance between the same pairwise columns. The assumption that $\mathbf{u}^T_i \mathbf{\bar{u}}_i \geq 0$ is not restrictive at all as we expect $\mathbf{U}^T \mathbf{\bar{U}}$ to be diagonally dominant with positive elements on the diagonal (see Remark 5 and the discussion after Remark 6). We note that in this context the Frobenius norm approach could be used to minimize a proxy for the operator norm. As in Remark 7 we have the choice of $\sigma_i$ we could update these values with each iteration of the proposed algorithm such that $\sigma_i^{\text{(new)}} \leftarrow \frac{\mathbf{u}_{i_{\max}}^T \mathbf{\bar{u}}_{i_{\max}} }{\mathbf{u}_i^T \mathbf{\bar{u}}_i}$ where $i_{\max} = \underset{i}{\arg \max}\ \mathbf{u}_i^T \mathbf{\bar{u}}_i$. This choice will encourage the algorithm to improve upon the worst pairwise column approximation (i.e., the highest $\mathbf{u}_i^T \mathbf{\bar{u}}_i$). This approach would be in the spirit of an iteratively reweighted least squares algorithm such as \cite{doi:10.1002/cpa.20303}.

Finally, the last measure of accuracy we consider is the angle between two subspaces as described by \cite{10.2307/2005662}. This value can be non-zero only when $p<d$, a case which is interesting for PCA projections and which we detail in the next section. Following \cite{doi:10.1137/S1064827500377332}, given $\mathbf{U}_p$ and $\mathbf{\bar{U}}_p$, whose columns span two $d$-dimensional subspaces of size $p$, we compute the angles between the two subspaces as
\begin{equation}
\beta_i = \arccos (\tau_i),\ i=1,\dots,p,
\end{equation}
where $0 \leq \tau_i \leq 1$ are the singular values of $\mathbf{U}_p^T \mathbf{\bar{U}}_p$ and where $0\leq \beta_1 \leq \dots \leq \beta_p \leq \pi/2$. We take the principal angle to be the largest angle above, i.e., $\beta = \beta_p = \arccos(\tau_p)$.

Finally, we would like to note that all the approximation errors we have previously discussed measure (in different ways) how well $\mathbf{U}_p^T \mathbf{\bar{U}}_p$ approaches the identity matrix.

\section{Application: fast PCA projections}

Consider a  training set of $d$-dimensional points $\{\mathbf{x}_i \}_{i=1}^N$ and  the $d \times N$ matrix $\mathbf{X} = \begin{bmatrix} \mathbf{x}_1 & \dots & \mathbf{x}_N \end{bmatrix}$. Given $1 \leq p < d$, PCA provides the optimal $p$-dimensional projection that minimally distorts, on average, the data points. The projection is given by the  eigenvectors of the $p$ largest eigenvalues of $\mathbf{X} \mathbf{X}^T$ or, equivalently, the left singular vectors of the $p$ largest singular values of $\mathbf{X}$, that is
\begin{equation}
\mathbf{XX}^T \approx \mathbf{U}_p (\mathbf{\Sigma}_p \mathbf{\Sigma}_p^T ) \mathbf{U}_p^T \text{ and } \mathbf{X} \approx \mathbf{U}_p \mathbf{\Sigma}_p \mathbf{V}_p^T.
\label{eq:eigandsvd}
\end{equation}
Given the above decompositions we can approximate $\mathbf{X} $ by $\mathbf{\bar{X}} = \mathbf{\bar{U}} \mathbf{\bar{\Sigma}}_p \mathbf{V}_p^T$, i.e., we keep $\mathbf{V}_p$ but we modify the principal components and their singular values, such that we minimize the error given by
\begin{equation}
\| \mathbf{U}_p \mathbf{\Sigma}_p \mathbf{V}_p^T -  \mathbf{\bar{U}} \mathbf{\bar{\Sigma}}_p \mathbf{V}_p^T \|_F^2 = \| \mathbf{U}_p \mathbf{\Sigma}_p -  \mathbf{\bar{U}} \mathbf{\bar{\Sigma}}_p \|_F^2,
\label{eq:Visnotneeded}
\end{equation}
where $\mathbf{U}_p$ is $d \times p$, the diagonal matrix $\mathbf{\Sigma}_p$ is $p \times p$, $\mathbf{\bar{U}}$ is of size $d \times d$, $\mathbf{\bar{\Sigma}}_p$ is $d \times p$ and is zero except for its main $p \times p$ diagonal. In this paper, we work with $\mathbf{X}$, as opposed to $\mathbf{XX}^T$, to keep the relationship with $\mathbf{U}_p$ linear, rather than quadratic. In the context of applying our approach to PCA, we use our decomposition on the principal components $\mathbf{U}_p$ which we assume are already calculated together with the associated singular values $\mathbf{\Sigma}_p$ which we may use as weights in \eqref{eq:theoptimizationproblem_only_orthonormal}.\\

Note that in \eqref{eq:Visnotneeded}, $\mathbf{V}_p$, which has size $N$, is not necessary and that the two-step procedure is equivalent to computing the projections $\mathbf{\bar{U}}$ directly from $\mathbf{X}$. Also note that based on \eqref{eq:eigandsvd}, we could factor $\mathbf{X} \approx \mathbf{\bar{U}} \mathbf{\Sigma} \mathbf{\bar{V}}^T$ where $\mathbf{\bar{U}}$ and $\mathbf{\bar{V}}^T$ are approximations in $\mathcal{F}_{O(d \log d)}$ and $\mathcal{F}_{O(N \log N)}$, respectively. The difficulty here is the dependency of $\mathbf{\bar{V}}$ on $N \gg d$ which would require a large running time.\\% can be very large and therefore finding $\mathbf{\bar{V}}$ would require a large running time. It is for this reason that we choose to find $\mathbf{U}$ or $\mathbf{U}_p$ first by some well-established algorithms and then perform our approximation $\mathbf{\bar{U}}$ or $\mathbf{\bar{U}}_p$ depending only on dimensions $d$ and $p$.
With $\mathbf{\bar{U}}$ fixed, for $\mathbf{\bar{\Sigma}}_p$ we have several strategies: i) set it to the identity, i.e., flatten the spectrum; ii) keep it to the original singular values $\mathbf{\Sigma}_p$; or iii) continuously update it to minimize the Frobenius norm, i.e., get the new ``singular values'' that are optimal with the approximation $\mathbf{\bar{U}}$. The first approach favors the accurate reconstruction of all components while the other approach favors mostly the few leading components only (depending on the decay rate of the corresponding singular values).\\%The good choice depends on the application at hand.
\subsection{Comparison with the symmetric diagonalization by Givens rotations approach} Because of the locally optimal way the Givens transformations are chosen (see Theorem 1), our proposed factorization algorithm is computationally slower than the Jacobi diagonalization process which chooses the Givens rotations on indices $(i,j)$ corresponding to the largest off-diagonal entry of the covariance matrix. Furthermore, the Jacobi decomposition uses each rotation to zero the largest absolute value off-diagonal entry and because of this sub-optimal choice needs $O(d^2 \log d)$ Givens rotations \cite{ManySweeps} to complete de diagonalization (more than the $\frac{d(d-1)}{2}$ needed to fully represent the orthonormal group).\\
 In all other aspects, our approach provides advantages over the Jacobi approach: i) we define a clear objective function that we locally optimize exactly; ii) it is known that the Jacobi process converges slowly when the number of rotations is low \cite{JacobiIsLInearEarlyOn}, which is exactly the practically relevant scenario we have, i.e., $g \ll d^2$; iii) with the same computational complexity, i.e., $g$ terms in the factorization, our proposed approach is always more accurate since we include as a special case the Givens rotations.
	\subsection{Comparison with  structured matrix factorization} Our approach requires the explicit availability of the orthonormal principal directions $\mathbf{U}_p$. Previous methods that factor using only Givens rotations are not applied directly on an orthogonal matrix. These methods rely on receiving as input a symmetric object (e.g., $\mathbf{XX}^T$) and then using a variant of Jacobi iterations for matrix diagonalization \cite{lemagoarou:hal-01416110} or multiresolution factorizations \cite{Kondor2014MMF} to find the good rotations that approximate the orthonormal eigenspace. Applying Givens transformations directly to $\mathbf{X}$ on the left, i.e., $\mathbf{G}_{ij} \mathbf{X}$, cannot lead to the computation of the PCA projections $\mathbf{U}$ but only to the polar decomposition. On the other hand, when applying Givens rotations on both sides of the covariance matrix, i.e., $\mathbf{G}_{ij} \mathbf{XX}^T \mathbf{G}_{ij}^T$, then the right eigenspace $\mathbf{V}$ cancels out in the product \eqref{eq:eigandsvd} and we are able to directly recover $\mathbf{U}$ (but we need $\mathbf{XX}^T$ explicitly). Finally, note that the diagonalization process approximates the full eigenspace $\mathbf{U}$ and cannot separate from the start the $p$ principal components $\mathbf{U}_p$ because they are solving the following problem
	\begin{equation}
	\underset{\mathbf{\bar{U}},\ \mathbf{\bar{\Lambda}}}{\text{minimize}}\ \|  \mathbf{XX}^T - \mathbf{\bar{U}}\mathbf{\bar{\Lambda}} \mathbf{\bar{U}} \|_F^2 \text{ subject to } \mathbf{\bar{U}} \in \mathcal{F}_g.
	\end{equation}
	This formulation is useful to approximate the whole symmetric  matrix $\mathbf{XX}^T$ (or $\mathbf{U}$), but not necessarily the $p$ principal eigenspace $\mathbf{U}_p$. To get these we would need to complete the diagonalization process, find the $p$ largest entries on the diagonal of $\mathbf{\bar{\Lambda}}$ and then work backward  to identify  the rotations that contributed  diagonalizing those largest elements. This  procedure would be prohibitively  expensive. Previous work, e.g. \cite{Kondor2014MMF, Treelets}, deals with approximating $\mathbf{XX}^T$ rather than computing PCA. In the same line of work, the one-sided Jacobi algorithm for SVD \cite{doi:10.1137/05063920X} can also be applied.

\section{Experimental results}

Given the rather pessimistic guarantees, we tackle problems: how well does Algorithm 1 recover random orthogonal matrices and principal components such that we benefit from the computational gains but do not significantly impact the approximation/classification accuracy. Source code available
\footnote{https://github.com/cristian-rusu-research/fast-orthonormal-approximation}

\subsection{Synthetic experiments}

For fixed $d$ we generate random orthonormal matrices from the Haar measure \cite{HowToRandomUnitary}.
Figure \ref{fig:random} shows the representation error $(2d)^{-1} \| \mathbf{U} - \mathbf{\bar{U}} \|_F^2$ for the proposed method. The plot shows that allowing for the Givens transformations $\mathbf{\tilde{G}}_{i j}$ in \eqref{eq:theG} brings a 17\% relative benefit as compared to using only the Givens rotations while, for the same $g$, the computational complexity is the same. The circulant approximation performs worst because it has the lowest number of degrees of freedom, only $d$ (computationally, it is comparable with the Givens and proposed approaches for $g = 100$). Lastly, we can observe that the bound is very pessimistic, especially for these values of $g$. In Figure \ref{fig:random_evo_stages} (left) we show for fixed number of transformations $g$ the progress that the proposed algorithm makes with each iteration. It is interesting to observe that the initialization steps (first $g$ steps) significantly decrease the approximation error while the other step make only moderate improvements. This indicates that convergence is slow and might take a large number of iterations with little progress made by the latter steps. Also in Figure \ref{fig:random_evo_stages} (right) we show the number of stages in each transformation. A stage is a set of extended orthogonal Givens transformations that can be applied in parallel, i.e., they do not share any indices $(i_k, j_k)$ among them. This is important from an implementation perspective as parallel processing can be exploited to speedup the transformations.

\subsection{MNIST digits and fashion}
\begin{figure}[!tbp]
	\centering
	\begin{minipage}[t]{0.495\textwidth}
		\includegraphics[trim = 18 5 30 22, clip, width=0.49\textwidth]{random_orthonormal_50-eps-converted-to.pdf}
		\includegraphics[trim = 18 5 28 22, clip, width=0.49\textwidth]{random_orthonormal-eps-converted-to.pdf}
		\caption{Average approximation errors and standard deviations over 100 realizations of random orthonormal matrices of size $d = 50$ (left) and $d = 100$ (right). For reference we show the bound developed in Theorem 3, the approximation accuracy of the circulant \cite{CirculantDRoperator} (Toeplitz performed just marginally better) and that of the using the factorization \eqref{eq:approxu} but allowing only Givens rotations. As expected, performance degrades with large $d$.}
		\label{fig:random}
	\end{minipage}
	\hfill
	\begin{minipage}[t]{0.495\textwidth}
		\includegraphics[trim = 18 5 28 20, clip, width=0.49\textwidth]{random_orthogonal_with_iterations-eps-converted-to.pdf}
		\includegraphics[trim = 18 5 30 20, clip, width=0.49\textwidth]{random_orthogonal_number_of_stages-eps-converted-to.pdf}
		\caption{Left: for $d=100$ and $g \in \{   332, 664, 1328 \}$ we show the evolution (mean and standard deviation) of the objective function with the number of iterations. In each case, the first $g$ iterations are the initialization process. Right: for $d \in \{50, 100\}$ we show the number of stages in each transformation we  learrn with the proposed  algorithm as a function of $g$. For both plots results are averaged over 100 realizations.}
		\label{fig:random_evo_stages}
	\end{minipage}
\end{figure}
\begin{figure}[!tbp]
	\centering
	\begin{minipage}[t]{0.495\textwidth}
		\includegraphics[trim = 18 5 26 20, clip, width=0.49\textwidth]{mnist_digits_new-eps-converted-to.pdf}
		\includegraphics[trim = 18 5 23 20, clip, width=0.49\textwidth]{mnist_fashion_new-eps-converted-to.pdf}
		\caption{Classification accuracy obtained by the k-NN algorithm for the MNIST digits (left) and fashion (right) datasets as a function of the complexity of the proposed projections. Dimensionality reduction was done with $p = 15$ principal components. The bold text represents the speedup (FLOPS) compared to the cost of projecting with the unstructured, optimal, PCA components which takes $2pd$ operations. For the sparser JL the variable $s$ is the number of non-zeros in each column.}
		\label{fig:mnistdetails}
	\end{minipage}
	\hfill
	\begin{minipage}[t]{0.495\textwidth}
		\includegraphics[trim = 18 5 28 20, clip, width=0.49\textwidth]{usps_with_iterations-eps-converted-to.pdf}
		\includegraphics[trim = 18 5 30 20, clip, width=0.49\textwidth]{digits_number_of_stages-eps-converted-to.pdf}
		\caption{For the USPS dataset ($d  =  256$) we have, left:  and $g \in \{  60, 120, 240 \}$ we show the evolution (mean and standard deviation) of the objective function with the number of iterations -- again, in  each case, the first $g$ iterrations are the initialization steps; right: the number of stages in each transformation created by the proposed method as a function of the number of extended orthogonal Givens transformations $g$.}
		\label{fig:usps_evo_stages}
	\end{minipage}
\end{figure}

We now turn to a classification problem. We use the MNIST digits
 and fashion

  datasets. The points have size $d = 400$ (we trimmed the bordering whitespace) and we have $N = 6\times 10^4$ training and $N_\text{test} = 10^4$ test points. In all cases, we use the k-nearest neighbors (k-NN) algorithm with $k = 10$, and we are looking to correctly classify the test points. Before k-NN we apply PCA and our proposed method. Results are shown in Figure \ref{fig:mnistdetails}. We deploy two variants of the proposed method: approximate the principal components as if they had equal importance and approximate the principal components while simultaneously also updating estimates of the singular values.\\
For comparison, we also show the sparse JL \cite{SparserJL}. In this case, the target dimension is $p \in \{15, 30\}$ while the random transformation of size $p \times d$ only has three non-zero entries per column. More non-zeros did not have any significant effect on the classification accuracy while increasing (doubling, in this case) the target dimension $p$ increases the classification accuracy by 10\%. The results reported in the plots are averages for 100 realizations and the standard deviation is below 1\%. Of course, the significant advantage of the JL approach over PCA is that no training is needed. The disadvantage is that if we choose greedily the target projection dimension, i.e., low $p$, the accuracy degrades significantly for JL. Results are identical for PCA when $p$ is $15$ of $30$.\\
Since we are dealing with image data, we also project using the discrete cosine transform (DCT). For the digits dataset, the performance is poor but, surprisingly, for the fashion dataset, this approach is competitive given the large speedup it reports (we used the sparse fast Fourier transform \cite{Indyk:2014:SSF:2634074.2634110} as we want only the largest $p$ components). We have also performed the projection by fast wavelet transforms with `haar' and several of the Daubechies `dbx' filters but the results were always similar to that of the DCT. For clarity of exposition, we did not add these results to the figures.\\
Our proposed methods report a clear trade-off between the classification accuracy and the numerical complexity of the projections. If we insist on an accuracy level close (within 1--2\%) of the full PCA then the speedup is only about x3. Reasonable accuracy is obtained for a speedup of x4--x5 after which the results degrade quickly. For $p=15$ better performance seems impossible via randomization. In this figure, the speedup is measured in terms of the number of operations (FLOPS).\\
Finally, in Figure \ref{fig:gt_vs_spca} (left) we compare our proposed methods against the sparse PCA on MNIST digits. sPCA performs exceptionally well in terms of the classification accuracy given the computational budget (a similar result is replicated for MNIST fashion). On other datasets where the principal components capture some global features (not local like in our example) we expect this performance to degrade. The training time of sPCA exceeds by 60\% the running time of PCA plus that of our method. We used the implementation of \citet{MINIMAXOPTIMALSPCA}.
\begin{table}[t]
	\caption{Average classification accuracies for k-NN when using PCA projections and our approximations without spectrum update, for various datasets. We also show the speedup (FLOPS and actual running time) and the number of features selected in the calculations as a proportion out of the total $d$ (see also Remark 2).  Results are averaged over 100 random realizations (train/test splits).}
	\label{tb:sample-table}
	\vskip 0.15in
	\begin{center}
		\begin{small}
			\begin{sc}
				\begin{tabular}{lcccccr}
					\toprule
					DATASET & FULL PCA & \multicolumn{4}{c}{PROPOSED ALGORITHM}  \\
					& accuracy & accuracy  & speedup(FLOPS) & speedup(TIME)  & selection \\
					\midrule
					PENDIGITS    & 95 $\pm$ 0.7& 91 $\pm$ 2.1 & $\times$1.6 & $\times$1.1 & 1 \\
					ISOLET & 92 $\pm$ 0.4 & 90 $\pm$ 1.0 & $\times$12 & $\times$10.1 & 1 \\
					USPS    & 95 $\pm$ 1.2& 94 $\pm$ 0.8 & $\times$7.7 &  $\times$4.7 & 0.64 \\
					UCI    & 90 $\pm$ 1.9& 87 $\pm$ 1.5 & $\times$2.5 & $\times$1.6  & 0.72 \\
					20NEWS     & 80 $\pm$ 3.1 & 77 $\pm$ 2.1 & $\times$3.1  & $\times$2.5 & 0.3 \\
					EMNIST digits & 97 $\pm$ 2.5& 95 $\pm$ 1.8 & $\times$13 & $\times$11.3 & 0.37 \\
					MNIST 8m     & 96 $\pm$ 2.0& 94 $\pm$ 0.9 & $\times$15 & $\times$13.7 & 0.28  \\
					\bottomrule
				\end{tabular}
			\end{sc}
		\end{small}
	\end{center}
	\vskip -0.1in
\end{table}
Because of ideas in Remark 1, we perform only the finally useful calculations and further reduce the computational cost on average by one third (these are accounted for already in the numbers in the plots).

\subsection{Experiments on other datasets}

The 20-newsgroups
 dataset consists of $18827$ articles from 20 newsgroups (approximately 1000 per class). The data set was tokenized using the rainbow package (www.cs.cmu.edu/~mccallum/bow/rainbow).

 Each article is represented by a word-count vector for the $d = 2\times10^4$ common words in the vocabulary. For this dataset we have $N_\text{test} = 5648$, $p = 200$, and as shown in Figure \ref{fig:gt_vs_spca} (right) in this case we outperform sparse PCA.\\
We also apply our algorithm to several other popular dataset from the literature: PENDIGITS (www.ics.uci.edu/$\sim$mlearn/MLRepository.html)
 with 10 classes and $d = 16$, $N = 7494$, $N_\text{test} = 3498$, $p = 4$; ISOLET (archive.ics.uci.edu/ml/datasets/isolet)

  with 26 classes and $d = 617$, $N = 6238$, $N_\text{test} = 1559$, $p = 150$; USPS (github.com/darshanbagul/USPS\_Digit\_Classification)

   with 10 classes and $d = 256$, $N = 7291$, $N_\text{test} = 2007$, $p = 12$; UCI (ftp.ics.uci.edu/pub/machine-learning-databases/optdigits)

    with 10 classes and $d = 64$, $N = 3823$, $N_\text{test} = 1797$, $p = 6$; EMNIST digits (nist.gov/itl/iad/image-group/emnist-dataset)

     with 10 classes and $d = 784$, $N = 24\times10^4$, $N_\text{test} = 4\times10^4$, $p = 15$; MNIST 8m (leon.bottou.org/papers/loosli-canu-bottou-2006)

      with 10 classes and $d = 784$, $N = 6.4 \times 10^6$, $N_\text{test} = 1.7 \times 10^6$, $p = 15$. All the results are shown in Table \ref{tb:sample-table}. Here we provide two measures for the speedup: the FLOPS (number of arithmetic operations, additions and multiplications) and the actual running time (in seconds). For the time speedup, we first computed $\mathbf{\bar{U}}$ and then implemented the matrix-vector multiplication and compared it to the generic matrix-vector multiplication with $\mathbf{U}$, both compiled in C using the gcc compiler and --O3 flag. Once computed by the proposed algorithm, the transformation could also be hardcoded and a further speedup improvement could be achieved. The running time speedups are slightly below the FLOPS speedups due to overhead in modern CPUs (fetching data, register loading etc.). Aside the computational benefits of the proposed transformations we note that while complex instruction set computing machines are closing the speedup gap in terms of running time the price is higher energy consumption and more complex execution pipelines/circuitry.\\
\begin{figure}[!tbp]
	\centering
	\includegraphics[trim = 110 0 132 12, clip, width=\textwidth]{fig_all_other_norms_new-eps-converted-to}
	\caption{We show for three datasets used in Section 5.3, as a function of the number of Givens transformations $g$ in $\mathbf{\bar{U}}_p$ that goes like $g = \alpha p \log_2 d$, the Frobenius (left most) and operator norms errors (second from the left), angle between subspaces distance measured in radians (third from the left) and correlations (right most shows the average correlations but also minimum and maximum values). The results are averaged over 10 realizations (random training/testing data samples) but the variance is always below 0.05.}
	\label{fig:all_other_norms}
\end{figure}
In Figure \ref{fig:usps_evo_stages}, for the  USPS dataset, we provide insights into the behavior of the proposed algorithm with each iteration and the number of stages in each transformation that we construct. Results are similar to the ones shown for the random orthogonal approximation. We observe again that the initialization step is very efficient in reducing the approximation error (especially when compared to the iterative process that follows) and that the number of stages that have to be applied sequentially can further improve the running time when implementing the proposed transformations.\\
Finally, one of the most famous applications of PCA is in the field of computer vision for the problem of human face recognition. The eigenfaces \cite{Sirovich:87} approach was used successfully for face recognition and classification tasks. Here, we want to reproduce the famous eigenfaces by using the proposed methods. The original eigenfaces and their approximations (sparse eigenfaces \cite{SparseFaces}), with different $g$ and therefore different levels of detail, are shown in Figure \ref{fig:eigenfaces}.

\begin{figure}[!tbp]
	\centering
	\begin{minipage}[t]{0.495\textwidth}
		\includegraphics[trim = 18 5 28 20, clip, width=0.49\textwidth]{gt_vs_spca_new-eps-converted-to.pdf}
		\includegraphics[trim = 18 5 30 20, clip, width=0.49\textwidth]{gt_vs_spca_20news_new-eps-converted-to.pdf}
		\caption{Classification accuracy versus number of operations on the MNIST digits (left) and 20NEWS (right) datasets for our proposed methods and the sparse PCA method \cite{MINIMAXOPTIMALSPCA, SparsePCA}.}
		\label{fig:gt_vs_spca}
	\end{minipage}
	\hfill
	\begin{minipage}[t]{0.495\textwidth}
		\centering
		\includegraphics[trim = 18 15 30 20, clip, width=0.75\textwidth]{eigenfaces_new-eps-converted-to.pdf}
		\caption{A few eigenfaces obtained by the optimal PCA (top) and by our proposed method with $g = 3059$ and $g = 1020$ (middle and bottom, respectively). The projection speedup (FLOPS) is x4.1 and x13.9, respectively.}
		\label{fig:eigenfaces}
	\end{minipage}
\end{figure}

\subsection{Results on other approximation errors}

In Section 3.1 we have explored several other ways to measure the approximation accuracy of the proposed algorithm. We note that in the proposed method we keep the Frobenius norm objective function but we also measure the operator norm, the angle between subspaces distance and the average/minimum/maximum correlations. All results for three datasets are shown in Figure \ref{fig:all_other_norms}, the choices of parameters are the same as in Section 5.3. The first observation is that the approximation errors increase with larger $d$ (as already clear from Remark 4 and Theorem 3), i.e, best results are obtained for UCI ($d = 64$)  and the worst for MNIST ($d = 400$). Second, note that improving the Frobenius norm error we also improve all the other error measures. Thirdly, note the similar behavior between the operator norm and subspace distance errors. It is interesting to observe that the two plots seem to suggest experimentally that $\arccos(\sigma_{\min}( \mathbf{U}_p^T \mathbf{\bar{U}}_p )) \approx 1-\sigma_{\max}( \mathbf{I}_p - \mathbf{U}_p^T \mathbf{\bar{U}}_p )$ which hold over a large array of number of Givens transformations $g$. Lastly, the two right-most plots highlight the connection between the operator norm optimization and the maximization of the lowest coherence between columns and their approximations (as per Theorem 5).\\
Depending on the application at hand we can choose how to measure the approximation error.

\subsection{Details of the implementation}

\begin{table}[t]
	\caption{Speed-up achieved with the proposed approximations $\mathbf{\bar{U}}_p$ against a vanilla C implementation of dense matrix-vector multiplication and BLAS Level 2 functions, in both cases with no parallelism.}
	\label{}
	\vskip 0.15in
	\begin{center}
		\begin{small}
			\begin{sc}
				\begin{tabular}{lcccccr}
					\toprule
					DATASET: & ISOLET & USPS  & EMNIST digits & MNIST 8m  \\
					\midrule
					C language (TIME) & $\times$10.1 & $\times$4.7 & $\times$11.3 & $\times$13.7 & \\
					BLAS (TIME) & $\times$4.8 & $\times$2.5 & $\times$5.7 & $\times$6.8 & \\
					\bottomrule
				\end{tabular}
			\end{sc}
		\end{small}
	\end{center}
	\vskip -0.1in
\end{table}
The source code attached to this paper is written for the Matlab environment. Besides the implementation of Algorithm 1, we also provide the code to efficiently perform the matrix-vector multiplication with the proposed $\mathbf{\bar{U}}$ and $\mathbf{\bar{U}}_p$, respectively (also taking into account Remark 3). Still, due to the characteristics of Matlab, our implementation is not faster than the dense matrix-vector multiplication ``*''. To provide an appropriate comparison, we implement the matrix-vector multiplication with our structures in the compiled lower-level programming language C. We provide comparisons of this implementation against two scenarios: a vanilla matrix-vector multiplication (using the appropriate ordering of the loops to exploit locality) and the Basic Linear Algebra Subprograms (BLAS) Level 2 routines for matrix-vector multiplication (SGEMV). We use the single thread variant of BLAS as for the dimensions $d$ we consider the overhead of the parallel implementation is significant.\\
Applying the proposed transformations on batch features, i.e., matrix-matrix multiplications, would require careful application of the proposed transformations \eqref{eq:theG} to fully use the computing architecture and maximize performance.

\section{Conclusions}
This paper proposes a new matrix factorization algorithm for orthogonal matrices based on a class of structured matrices called extended orthogonal Givens transformations. We show that there is a trade-off between the computational complexity and accuracy of the approximations created by our approach. We apply our method to the approximation of a fixed number of principal components and show that, with a minor decrease in performance, we can reach significant computational benefits.\\
Future research directions include strengthening the theoretical guarantees since they  are way above what we observe experimentally. Further, it would be of interest to improve the  complexity of the proposed algorithm either by a parallel implementation or using randomization (e.g. computing a random subset of the $O(d^2)$ scores). As an immediate application, it would be interesting to apply our decomposition to the recently proposed unitary recurrent neural networks \cite{URNN}.

\section*{Acknowledgment}
This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216, and the Italian Institute of Technology. Part of this work has been carried out at the Machine Learning Genoa (MaLGa) center, Universita di Genova (IT). Lorenzo Rosasco acknowledges the financial support of the European Research Council (grant SLING 819789), the AFOSR projects FA9550-17-1-0390 and BAA-AFRL-AFOSR-2016-0007 (European Office of Aerospace Research and Development). Cristian Rusu acknowledges support by the Romanian Ministry of Education and Research, CNCS-UEFISCDI, project number PN-III-P1-1.1-TE-2019-1843, within PNCDI III.

{\small
\bibliographystyle{plainnat}