If $M$ is a $2n\times 2n$ real symmetric matrix, I would like to ask what could be its canonical form under unitary congruence. We view a unitary $n\times n$ matrix $U$ as a real $2n\times 2n$ matrix, and try to see if we can ``diagonalize'' $M$ by
$$U^T M U.$$


In addition, what is the canonical form if we only require that $U$ is symplectic?



What is the canonical form of real symmetric 2n\times 2n matrix under unitary congruence?
The difficulty is that the space $S\_{2n}(\mathbb{R})$ of $2n$-by-$2n$ symmetric matrices with real entries is not irreducible under the action of $\mathrm{U}(n)$. In fact, there is a $\mathrm{U}(n)$-irreducible decomposition
$$
S\_{2n}(\mathbb{R}) = \mathbb{R}\cdot I\_{2n}\oplus A\_n\oplus B\_n
$$
where $A\_n$ is the space of matrices of the form
$$
A = \begin{pmatrix}s&a\cr-a&s\end{pmatrix}
$$
where $s$ is traceless symmetric $n$-by-$n$ and $a$ is antisymmetric $n$-by-$n$ and where $B\_n$ is the space of matrices of the form
$$
B = \begin{pmatrix}p&q\cr q&-p\end{pmatrix}
$$
where $p$ and $q$ are $n$-by-$n$ symmetric matrices. (In this realization, $\mathrm{U}(n)$ is the set of matrices of the form
$$
U = \begin{pmatrix}x&y\cr -y&x\end{pmatrix}
$$
where $x$ and $y$ are $n$-by-$n$ and $U = x+iy$ is unitary, i.e., $UU^\ast=I\_n$.)


When you write $M = \lambda I\_{2n} + A + B$ with $A\in A\_n$ and $B\in B\_n$, then you can use $U$ to diagonalize $A$ or you can use it to diagonalize $B$, but, usually, you cannot diagonalize both when $n>1$.


As for only requiring that $U$ be symplectic, now you are asking for a classification of the adjoint orbits of $\mathrm{Sp}(n,\mathbb{R})$ acting on its Lie algebra. Of course, you can conjugate the semi-simple elements into a maximal torus (of some signature), but a full classification of, say, the nilpotent elements, is complicated.


