$\DeclareMathOperator\Sym{Sym}\DeclareMathOperator\O{O}\DeclareMathOperator\GL{GL}$Let $k$ be an algebraically closed field of characteristic 0 (it can even be $\mathbb{C}$ if you like), and let $n\in\mathbb{N}$. Let $\Sym(n)$ denote the space of $n\times n$ symmetric matrices with entries in $k$. Let $\O(n)\subseteq \GL(n)$ denote the orthogonal group, that is all $n\times n$ matrices $A$ with entries in $k$ such that $AA^t=A^tA=I\_n$.


Then $\O(n)$ acts on $\Sym(n)$ by conjugation. I would like nice representatives for the orbits of this action.


Let $B\in \Sym(n)$. If $B$ is diagonalizable, then it is a theorem that it can be diagonalized by an orthogonal matrix, so all the 'semisimple orbits' have a diagonal matrix as a representative.


So my question is about when $B$ is not diagonalizable — what is a 'nice' form we can put it in under orthogonal conjugation? (For an example of a non-diagonalizable symmetric matrix, see <https://math.stackexchange.com/questions/1658393/why-a-complex-symmetric-matrix-is-not-diagonalizible.>)


Seeking explanations or references. Thank you!



Action of symmetric matrices under $\mathrm{O}(n)$
The adjoint action of $SO(n,\mathbb C)$ on $\mathrm{Sym}\_0(n,\mathbb C)$ (we can replace $O(n,\mathbb C)$ by $SO(n,\mathbb C)$, and $\mathrm{Sym}(n,\mathbb C)$
by the subspace of traceless matrices, since the
action splits off the scalar multiples of the identity) is the isotropy representation of the complex symmetric space
$SL(n,\mathbb C)/SO(n,\mathbb C)$. Kostant and Rallis (1971) investigated such
representations. This was later generalized by Vinberg (1976) to the case of a $\theta$-group (the $\mathfrak g\_0$-action on $\mathfrak g\_1$, where we consider the decomposition of a complex Lie algebra $\mathfrak g=\mathfrak g\_0 
+\mathfrak g\_1+\cdots + \mathfrak g\_m$ into eigenspaces of an automorphism
$\theta$ of order $m$ ($1$-parameter group of automorphisms in case $m=\infty$).


Among other things, they found that for such a representation $(G\_0,V)$:


1. There is a **Jordan decomposition**: every $x\in V$ can be uniquely written
as $x\_s+ x\_n$, where $x\_s$ is a semisimple element (this means that $ad\_{x\_s}$
is a semisimple transformation) , $x\_n$ is a nilpotent element (this means that
$\mathrm{ad}\_{x\_n}$ is a nilpotent transformation),
and they commute.
2. A maximal Abelian subspace $c$ of $V$ consisting of semisimple elements
is called a **Cartan subspace**. Any two Cartan subspaces are $G\_0$-conjugate.
3. **Semisimple elements** are characterized by the fact that their $G^0$-orbits are closed, or they lie in a Cartan subspace, or they are $G^0$-conjugate
to an element in a given Cartan subspace. In your example, a Cartan subspace is given by the diagonal matrices, and the semisimple elements are exactly the
diagonalizable symmetric matrices.
4. **Nilpotent elements** are characterized by the fact that $0$
lies in the closure of their orbits. Nilpotent elements of course come in
$G^0$-orbits, and there are only finitely many nilpotent orbits.
In the case of a complex symmetric space $\mathfrak g=\mathfrak k + \mathfrak p$
(under $\theta$),
the Kostant-Sekiguchi correspondence establishes a bijection between the set of nilpotent $K$-orbits on $\mathfrak p$ and the set of nilpotent $G\_{\mathbb R}$-orbits on $\mathfrak g\_{\mathbb R}$, where $\mathfrak g\_{\mathbb R}$ is the noncompact real form of $\mathfrak g$ whose Cartan involution induces $\theta$ on $\mathfrak g$. In turn, the nilpotent $G\_{\mathbb R}$-orbits on $\mathfrak g\_{\mathbb R}$ are parametrized by partitions.


More explicitly, in our case
we have the symmetric space $\mathfrak{sl}(n,\mathbb C)=\mathfrak{so}(n,\mathbb C)+\mathrm{Sym}\_0(n,\mathbb C)$ and the nilpotent $SO(n,\mathbb C)$-orbits in
$\mathrm{Sym}(n,\mathbb C)$ are in bijection with the nilpotent $SL(n,\mathbb R)$-orbits in $\mathfrak{sl}(n,\mathbb R)$. The latter can be understood in terms of the Jordan canonical form.
As an explicit example of the Kostant-Sekiguchi correspondence, the nilpositive element $\begin{pmatrix}0&1\\0&0\end{pmatrix}$
of an $\mathfrak{sl}\_2$-triple in $\mathfrak{sl}(2,\mathbb R)$ corresponds under the Cayley transform to the nilpositive element $\frac12\begin{pmatrix}i&1\\1&-i\end{pmatrix}$ of a normal $\mathfrak{sl}\_2$-triple in $\mathfrak{sl}(2,\mathbb C)$; that is essentially the example given by @JohannesEbert. One can work the details of the correspondence more generally, see e.g. the book  

[https://www.amazon.com/Nilpotent-Orbits-Semisimple-Lie-Algebra/dp/0534188346](https://rads.stackoverflow.com/amzn/click/com/0534188346)


