Let $O$ be a matrix sampled from the Haar measure on $O(n)$. Let $X$ be the upperleft $k\times k$ submatrix of $O$.


In a physics research project I am interested in the distribution of $X$, say $\rho(X)$. What I was able to prove for $k=1$ and $k=2$ is that:


$k=1$: $\rho(X)\propto(1-X^2)^{(n-3)/2}$.


$k=2$: $\rho(X)\propto (1-\text{Tr}(XX^T)+\det(XX^T))^{(n-5)/2}$.


It did not escape my attention that these expressions are in fact closely related to the characteristic polynomial $p(x)$ of $XX^T$. In both cases, the distribution $\rho(X)$ is a function of $p(1)$.


Given the simplicity of the result, I believe that this result is known to the math community for a long time; however, I can not find any literature that contains the result. I would be grateful if someone can point me to some existing literature or enlighten me on the significance of the characteristic polynomial.



Distribution of Submatrix of Orthogonal Matrix
The probability distribution of $X$ was calculated in [Random-matrix theory of thermal conduction in superconducting quantum dots](https://arxiv.org/abs/1004.2438). In the context of that physics problem, the $k\times k$ upper-left submatrix $X$ of an $n\times n$ orthogonal matrix $O$ is the reflection matrix of a superconducting quantum dot with $k$ modes in one opening and $n-k$ modes in the other opening. Such a system is described by the circular real ensemble (CRE) of random-matrix theory.


The probability distribution of $X$ is conveniently described in the singular value decomposition $X=O\_1 {\rm diag}\,(x\_1,x\_2,\ldots x\_k)O\_2$, where $x\_i\geq 0$ are the singular values and $O\_1,O\_2$ are orthogonal matrices.


The matrices $O\_1,O\_2$ are independently drawn from $O(k)$ with the Haar measure. The singular values $x\_i$ of $X$ are related to the transmission probabilities $T\_i$ by $T\_i=1-x\_i^2$. For $N=\min(k,n-k)$ there is a set $\{T\_1, T\_2,\ldots T\_N\}$ of nonzero transmission probabilities that have the probability distribution
$$P(\{T\_1,T\_2,\ldots T\_N\})\propto \prod\_i T\_i^{|n/2-k|-1/2}(1-T\_i)^{-1/2}\prod\_{j<\ell}|T\_\ell-T\_j|.$$
See equation (5) in the cited paper, with $\beta=1$ and $\gamma=-1$.


Equivalently, in terms of the $x\_i$'s themselves, there is a set $\{x\_1, x\_2,\ldots x\_N\}$ of singular values that are strictly smaller than unity in absolute value, with probability distribution
$$P(\{x\_1,x\_2,\ldots x\_N\})\propto \prod\_i (1-x\_i^2)^{|n/2-k|-1/2}\prod\_{j<\ell}|x\_\ell^2-x\_j^2|.$$


The probability distribution of the matrix elements of the matrix $X$ can indeed be written (for $k<n/2$) as $P(\{X\_{ij}\})\propto {\rm Det}\,(1-XX^\top)^{n/2-k-1/2}$, as in the OP. Equivalently, in terms of the symmetric matrix $H=XX^\top$ one has $P(\{H\_{ij}\})\propto {\rm Det}\,H^{-1/2}\,{\rm Det}\,(1-H)^{n/2-k-1/2}$.


The OP also asks for the significance of the appearance of the characteristic polynomial ${\rm Det}(1-XX^\top)$. This is easiest to understand for $k>n/2$, when $XX^T$ has $k-n/2$ eigenvalues pinned at unity. These repel the $n/2$ other eigenvalues with a term $\prod\_{i=1}^{n/2}(1-x\_i^2)^{k-n/2}$.


