$\DeclareMathOperator\Spin{Spin}\DeclareMathOperator\Sp{Sp}\DeclareMathOperator\SL{SL}$The exceptional isomorphism $\Spin(5,1)\simeq \SL(2,\mathbb{H})$ is well-known, and I can find references that say the maximal compact of $\Spin(5,1)$ is $\Spin(5) \simeq \Sp(2)$. So I know the answer to the question, but not the how or why. In particular, is there a proof that $\Sp(2)$ is maximal compact in $\SL(2,\mathbb{H})$ not via the exceptional isomorphisms? Perhaps some sort of analogue of Gram-Schmidt or other explicit factorisation?



Maximal compact subgroup of $\mathrm{SL}(2,\mathbb{H})$
$\DeclareMathOperator\GL{GL}\DeclareMathOperator\SL{SL}\DeclareMathOperator\SO{SO}\DeclareMathOperator\Sp{Sp}$YCor's [comment](https://mathoverflow.net/questions/195060/maximal-compact-subgroup-of-mathrmsl2-mathbbh#comment485760_195060) contains the essential idea needed for the proof, but maybe a few more details would be helpful. (If YCor does provide something similar later, feel free to award YCor's answer the bounty.)


Consider the mapping $\sigma:\GL(2,\mathbb{H})\to M\_2(\mathbb{H})$ given by
$$
\sigma(A) = A^\* A
$$
where $A^\*$ is the conjugate transpose of $A$ in $M\_2(\mathbb{H})$, the $2$-by-$2$ matrices with entries in $\mathbb{H}$. Then $\sigma(A)=\sigma(A)^\*$ (using the fact that $\overline{pq}= \overline{q}\,\overline{p}$ for $p,q\in\mathbb{H}$). Consequently, the image of $\sigma$ lies in the $6$-dimensional real subspace $S\_2(\mathbb{H})$, consisting of the matrices $s\in M\_2(\mathbb{H})$ that satisfy $s = s^\*$. Due to the associativity of multiplication in $\mathbb{H}$ and the above-mentioned conjugation identity, we have
$$
(AB)^\*sAB = B^\*(A^\*sA)B,
$$
so it follows that the mapping $\rho(s,A)=A^\*sA$ defines a (right) representation of $\GL\_2(\mathbb{H})$ on $S\_2(\mathbb{H})$.


Now, define the quadratic form $Q:S\_2(\mathbb{H})\to\mathbb{R}$ by
$$
Q\left(\begin{pmatrix}a&x\\\overline{x}&b\end{pmatrix}\right) = ab-x\overline{x}.
$$
when $a,b\in\mathbb{R}$ and $x\in\mathbb{H}$. Note that $Q$ has signature type $(1,5)$ as a real quadratic form.


The crucial identity (which can be proved by hand just by writing it out) is that
$$
Q(A^\*sA) = Q(s)\,Q(A^\*A).
$$
It follows that $Q(A^\*A) = 1$ defines $\SL(2,\mathbb{H})$ as a codimension $1$ closed subgroup of $\GL(2,\mathbb{H})$. In particular, the representation $\rho$ sends $\SL(2,\mathbb{H})$ into $\SO(Q)\simeq\SO(1,5)$, and it is easy to show that the kernel of this homomorphism is $\{\pm I\_2\}\subset\SL(2,\mathbb{H})$.


By definition, the stabilizer of $I\_2$ under the right representation $\rho$ is the 10-dimensional Lie group usually denoted in differential geometry by $\Sp(2)$, and its orbit under this right action by $\SL(2,\mathbb{H})$ must thus have dimension $5$ and hence be the nappe of the hyperboloid $Q(s)=1$ consisting of those $s$ with positive trace, i.e., hyperbolic $5$-space.


The connectedness of $\Sp(2)$ follows from the well-known fibration $\Sp(1)\to \Sp(2)\to S^7$, so it follows that $\Sp(2)$ is a nontrivial double cover of the identity component of $\SO(Q)$. Since $\SL(2,\mathbb{H})$ is simple and not compact, the signature type of its Killing form cannot be $(0,15)$, and since its Lie algebra splits as a module over ${\mathfrak{sp}}(2)$ into two irreducible pieces of dimension $10$ and $5$ corresponding to ${\mathfrak{sp}}(2)$ (on which it is negative definite) and its orthogonal complement, the type must be $(5,10)$. Thus, a maximal compact in $\SL(2,\mathbb{H})$ must have dimension $10$, so $\Sp(2)$ must be a maximal compact.


Finally, if you want a ‘factorization’ (analogous to the QR decomposition in the real case), you can show that every $A\in\SL(2,\mathbb{H})$ can be factored uniquely in the form
$$
A = Q\begin{pmatrix} a & 0\\ 0& a^{-1}\end{pmatrix}\begin{pmatrix} 1 & x\\ 0& 1\end{pmatrix},
$$
with $Q\in\Sp(2)$, $a\in\mathbb{R}^+$, and $x\in\mathbb{H}$, which also shows that $\SL(2,\mathbb{H})$ is diffeomorphic to $\Sp(2)\times\mathbb{R}^5$. (This is just the KAN decomposition for $\SL(2,\mathbb{H})$.)


