If $U$ is a $N\times N$ random unitary matrix uniformly distributed with respect to Haar measure, a $M\times M$ block $A$ from it has distribution given by
$$ \det(1-AA^\dagger)^{N-2M}.$$


If $O$ is a $N\times N$ random real orthogonal matrix uniformly distributed with respect to Haar measure, a $M\times M$ block $A$ from it has distribution given by
$$ \det(1-AA^T)^{(N-2M-1)/2}.$$


My question is what is the distribution of the $M\times M$ top left block from a $N\times N$ random unitary symmetric matrix from the Circular Orthogonal Ensemble.



Distribution of top left block from unitary symmetric matrices
The general formula in the circular ensembles follows from equation 2.10 of [arXiv:cond-mat/9612179](https://arxiv.org/abs/cond-mat/9612179):
$$P(H)\propto\text{det}\,(1-H)^{\tfrac{1}{2}\beta(N-2M+1-2/\beta)},\;\;N-2M\geq 0,\qquad\qquad(\ast)$$
where $H=AA^\dagger$ and $A$ is an $M\times M$ principal submatrix of an $N\times N$ unitary matrix $U$ (circular unitary ensemble, $\beta=2$) or unitary symmetric matrix (circular orthogonal ensemble, $\beta=1$) or unitary selfdual matrix (circular symplectic ensemble, $\beta=4$).


The case in the OP is $\beta=1$, in that case $H=AA^\dagger=AA^\ast$ (in the physics notation that $\ast$ is complex conjugation and $\dagger$ is Hermitian conjugate). The assumption that $N-2M\geq 0$ is needed, because if $M>N/2$ there will be an additional set of $M-N/2$ eigenvalues of $H$ that are pinned to unity.


The case that $U$ is real orthogonal (circular real ensemble, $H=AA^\dagger=AA^\top$) has an additional factor,
$$P(H)\propto\text{det}\,H^{-1/2}\,\text{det}\,(1-H)^{\tfrac{1}{2}(N-2M-1)}.\qquad\qquad(\ast\ast)$$ That case is considered in an [earlier MO posting](https://mathoverflow.net/a/342124/11260). Note that $(\ast\ast)$ agrees with the corresponding result in the OP (the second equation), which gives $P(A)$ instead of $P(H)$.




---


To make contact with the physics literature: $U$ represents a scattering matrix, $A$ is the reflection matrix, and the eigenvalues of $1-H=1-AA^\dagger$ are the transmission eigenvalues $T\_1,T\_2,\ldots T\_M$. Equation (2.10) in the cited paper gives the joint probability distribution of the $T\_i$'s, which can equivalently be expressed as the equation $(\ast)$ above.
