Let $A\in \mathcal{M}\_{m\times m}(\mathbb R)$ , $det(A)=1$ , $A$ is positively definite. Which matrices $P$ satisfy the equation
$$P^TAP=A$$
In fact I am interested in sequences of traces $tr P^n$ of the iterations of such solutions.


In dimension $2$ one can show that
$$P^n=\left(
 \begin{array}{cc}
 \cos n\phi& -\beta \sin n\phi \\
 \alpha \sin n\phi & \cos n\phi\\
 \end{array}\right)$$
for some $\phi, \alpha, \beta$ satisfying$\alpha\beta=1$
hence $tr P^n= 2\cos n\phi$.



Matrix equation $P^TAP=A$
As $A=L^\top L$, for some $L\in M\_{m\times m}(\mathbb{R})$, $\det L=1$, you can rewrite $P^\top L^\top LP=L^\top L$ and then multiply both sides by $L^{-1}$, etc., obtaining $(L^{\top})^{-1}P^\top L^\top LPL^{-1}=(LPL^{-1})^\top LPL^{-1}=I$, i.e. each $LPL^{-1}$ must be orthogonal.


As traces are preserved under conjugation, you can assume $L=I$, and so your question is reduced to studying traces of orthogonal matrices. The trace is the sum of eigenvalues, which are all modulus 1 complex numbers for orthogonal matrices.


