$\newcommand{\Sig}{\Sigma}$
$\newcommand{\dist}{\operatorname{dist}}$
$\newcommand{\distSO}[1]{\dist(#1,\SO)}$
$\newcommand{\distO}[1]{\text{dist}(#1,\On)}$
$\newcommand{\tildistSO}[1]{\operatorname{dist}\_{\til d}(#1,\SO)}$
$\newcommand{\SOn}{\operatorname{SO}\_n}$
$\newcommand{\On}{\operatorname{O}\_n}$
$\newcommand{\GLp}{\operatorname{GL}\_n^+}$
$\newcommand{\GLtwo}{\operatorname{GL}\_2^+}$
$\newcommand{\GLm}{\operatorname{GL}\_n^-}$
$\newcommand{\sig}{\sigma}$
$\newcommand{\diag}{\operatorname{diag}}$


Let $A$ be a real $n \times n$ matrix, with **negative determinant.**
Suppose that the singular values of $A$ are pairwise distinct. Then, it can be proved that there exist a **unique** special orthogonal matrix $Q(A)$ which is closest to $A$ (w.r.t the Frobenius distance).



> 
> I want to find out if there exist a "formula" for $Q(A)$, say in terms of positive roots, inverses, matrix multiplication etc. Is there any hope for such a thing?
> 
> 
> 


By a formula, I do not really mean a "closed-form" formula. A closely related example of what I am looking for is the orthogonal polar factor of an invertible matrix:


If $A=OP$, where $O \in \On$, and $P$ is symmetric positive-definite, then $P=\sqrt{A^TA}$ (here $\sqrt{}$ is the unique symmetric positive-definite square root) and $O=O(A)=AP^{-1}=A(\sqrt{A^TA})^{-1}$. I consider this is an acceptable formula. 


**(Comment: The orthogonal factor $O(A)$ is the closest orthogonal matrix to $A$).**


**Edit 2:**


After some more thinking, I think the idea of a "reasonable formula" might be a bit hopeless: If we had any such "reasonable" formula, we could probably extend it continuously to all of $\GLm$. However, such a continuous extension does not exist:


Set $A\_n=\begin{pmatrix} -1 & 0 \\\ 0 & 1+\frac{1}{n} \end{pmatrix},B\_n=\begin{pmatrix} -(1+\frac{1}{n}) & 0 \\\ 0 & 1 \end{pmatrix}$.


Then $Q(A\_n)=\text{Id},Q(B\_n)=-\text{Id}$, while $A\_n ,B\_n$ both converge to $\begin{pmatrix} -1 & 0 \\\ 0 & 1 \end{pmatrix}$.


This phenomena implies that perhaps the best we can do is to find "partial expressions", as in [Dap's answer](https://mathoverflow.net/a/326656/46290).


Here is what I know:
Let $A=U\Sig V^T$ be the singular values decomposition of $A$; we can assume that $\Sig = \diag\left( \sig\_1,\dots\sig\_n \right)$ where $\sigma\_1$ is the **smallest** singular value of $A$, and that $U \in \SOn,V \in \On,\det V=-1$.


Set $\Sig':=\diag\left( -\sig\_1,\dots\sig\_n \right)$, and rewrite
$$
 A= U\Sig V^T = U (\Sig \diag\left( -1,1,1\dots ,1 \right)) (\diag\left( -1,1,1\dots ,1 \right) V ^T )
 =U \Sig' \tilde V^T,
$$
where $\tilde V \in \SOn$ is defined by requiring $\diag\left( -1,1,1\dots ,1 \right) V ^T=\tilde V^T$.



> 
> Then, it turns out that $Q(A)=U\tilde V^T$. 
> 
> 
> 


Specifically, we have 
$$
\dist(A,\SOn)= \dist(U \Sig' \tilde V^T ,\SOn)= \dist( \Sig' ,\SOn)=d(\Sig' ,\text{Id})\\=(\sig\_1+1)^2 + \sum\_{i=2}^n \left( \sig\_i-1 \right)^2,
$$
and one can prove that $\text{Id}$ is the unique closest matrix in $\SOn$ to $\Sig'$. (It is important here that $\sigma\_1$ is the **smallest** singular value of $A$).




---


*Comment:*


I prefer a formula which does not involve directly the **singular vectors** of $A$, since I want to understand how smoothly does $Q(A)$ varies with $A$. (The formula for the orthogonal factor mentioned above immediately implies that it is a smooth function of the matrix, once one knows that the positive square root is smooth). Finally, note that while $Q(A)=U\tilde V^T$, the orthogonal factor satisfies $O(A)=UV^T$).


**Edit:** I found out that the minimizer $Q(A)$ indeed changes smoothly; This follows from the fact that **locally**, [we can choose the matrices $U,V$ smoothly](https://math.stackexchange.com/a/3163368/104576). However, I think that an elegant formula would still be a nice thing to have. (Even though we do not need one to establish smoothness). 


A more abstract discussion about smoothness of minimizers can be found [here](https://mathoverflow.net/questions/242002/smoothness-of-the-closest-point-on-a-submanifold). 



Is there a "formula" for the point in $\text{SO}(n)$ which is closest to a given matrix?
Not quite an answer, but we can compute $Q$ using a single suitably chosen real parameter $\lambda.$
For any $\lambda>0,$ for all matrices $A$ with singular values $\sigma\_1\leq\sigma\_2\leq\cdots\leq\sigma\_n$ satisfying $0<\sigma\_1<\lambda<\sigma\_2,$
$$Q(A)=O(A)O(A^TA-\lambda^2 I)$$


where $I$ is the identity matrix, and $O(M)=M(\sqrt{M^TM})^{-1}$ as defined in the question.
To prove this, let $A=U\Sigma V^T$ as in the question. Then $\Sigma^2-\lambda^2I=|\Sigma^2-\lambda^2I|\operatorname{diag}(-1,1,1,\dots,1)$ where $|\Sigma^2-\lambda^2I|$ is a certain positive define diagonal matrix. This gives


$$O(A)O(A^TA-\lambda^2 I)=O(U\Sigma V^T)O(V|\Sigma^2-\lambda^2 I| \tilde{V}^T)=UV^TV\tilde{V}^T=U\tilde{V}^T.$$


If you were interested in integral formulas (still depending on $\lambda$): $Q$ of the form $Q(A)=A \cdot f(A^TA)$ where $f$ acts on the spectrum of $A^TA$ by taking $z>0$ to $z^{-1/2}$ for $z>\lambda$ and to $-z^{-1/2}$ for $z<\lambda.$ The holomorphic functional calculus then gives an integral formula for $f,$ locally in $A.$


Here are some relationships between the expressive power of various parameters.
If you happened to know the smallest two singular values you could plug in $\lambda=\sqrt{\sigma\_1(A)\sigma\_2(A)}.$ This is the inverse square root of the singular norm of $\bigwedge^2 A^{-1},$ which can be defined via spectral radius using $M^TM,$ and spectral radius has a sort of formula, [Gelfand's formula](https://en.wikipedia.org/wiki/Spectral_radius#Gelfand's_formula). On the other hand, for symmetric $A$ with $\sigma\_1(A)<\sigma\_2(A),$ the matrix $\tfrac12(O(A)-Q(A))$ is projection onto the right singular vector corresponding to $\sigma\_1(A).$


