$\newcommand{\al}{\alpha}$


Let $M\_n$ be the space of $n \times n$ real matrices.


**Question:**


For which $n$, is there an inner product on $M\_n$ which satisfies:


$$(\*) \, \, \langle Q^TXQ,Q^TYQ \rangle = \langle X, Y\rangle $$


For every $Q \in SO(n)$,


but does **not** satisfy $(\*)$ for every $Q \in O(n)$.


(i.e, is there an inner product which is $SO(n)$-**isotropic** but not $O(n)$-**isotropic**?)


**Results so far:**


$(1)$ Since for odd $n$, $-Id \in O(n)\setminus SO(n)$ commutes with every matrix in $M\_n$, every $SO(n)$-invariant inner product is also $O(n)$-invariant. Thus the question is interesting only for even $n$.


$(2)$ A natural approach for this problem, is to use Riesz representation, as follows:


Let $\langle , \rangle\_F$ denote the Frobenius inner product on $M\_n$, and let $\langle , \rangle$ be some arbitrary inner product. Fixing $X \in M\_n$, we get a linear functional: $Y \to \langle X, Y \rangle$. There is a (unique) matrix $\al(x)$ such that:


$$(\*\*) \, \langle X, \cdot \rangle = \langle \al(X), \cdot \rangle\_F$$


So, to every inner product $\langle , \rangle$ there is an associated linear operator $\al:M\_n \to M\_n$ satisfying $(\*\*)$.


We say that $\al$ is $SO(n)$ (or $O(n)$)-**isotropic** if


$$ \al(Q^TXQ)=Q^T\alpha(X)Q $$ For every $Q \in SO(n)$ (or $O(n)$)


It's easy to see (using the fact $\langle , \rangle\_F$ is left-and right $O(n)$ invariant), that:


$$\langle , \rangle \text{ is } SO(n)\text{-isotropic } \iff \al \text{ is } SO(n)\text{-isotropic }$$ ,


(and similarly to $O(n)$).


Hence, the question can be partially reduced to finding $SO(n)$-isotropic operators which are not $O(n)$-isotropic. 
If no such operator exists, then we finished. However, it could be the case that such an operator does exist, but does not give rise to an inner product\* via $(\*\*)$. 


This is the case of the operator $\al(X)=R\_{\theta}\cdot X$ , where $R\_{\theta}$ is a rotation matrix, and $n=2$. (For details see [here](https://math.stackexchange.com/a/1603096/104576)). The corrseponding bilinear form turns out to be positive but not symmetric. Taking its symmetrization gives an inner product, which turns out to be $O(n)$-isotropic.


I do not know if there are any such operators for even $n \neq 2$.


In fact, for $O(n)$-invariant operators, there is a representation theorem which says they must be of the form of:


$\alpha(X)=a \text{tr}(X)I + bX + cX^T$


A proof is given for example [here](http://link.springer.com/article/10.1007%2FBF00041995#page-1).




---


\*To induce an inner product, $\al$ must be self-adjoint w.r.t the Frobenius product, and be positive, in the sense that $\langle \al(X), X\rangle\_F > 0 \, , \, \forall X \neq 0$



Existence of $SO(n)$-isotropic inner products which are not $O(n)$-isotropic
Disclaimer: this answer is at least 50% due to @Holonomia. If you like it, why don't you upvote some of her posts?


The groups $O(n)$ and $SO(n)$ act by conjugation on $M\_n(\mathbb R)$.
There is an equivariant decomposition
$$M\_n(\mathbb R)\cong\mathbb R E\_n\oplus\mathrm{Sym}\_0^2(\mathbb R^n)
\oplus\Lambda^2(\mathbb R^n)\;.$$
The second summand consists of trace-free symmetric matrices, the third one of skew-symmetric matrices.
Each of these spaces is an irreducible $O(n)$-representation (or $0$ if $n$ is small).
For almost all $n$, each one is also irreducible as an $SO(n)$-representation (or $0$).


The only exception is $n=4$. Here, $\Lambda^2(\mathbb R^4)\cong\Lambda^{2,+}(\mathbb R^4)\oplus\Lambda^{2,-}(\mathbb R^4)$ as an $SO(4)$-representation. This is the decomposition into the $\pm 1$-eigenspaces (also called selfdual and antiselfdual forms) of the Hodge-$\*$-operator, which needs a metrics and an orientation in its definition, and hence is $SO(4)$-invariant. Typical elements are
$$e\_1\wedge e\_2+e\_3\wedge e\_4=\begin{pmatrix}&1\\-1\\&&&1\\&&-1\end{pmatrix}\in\Lambda^{2,+}(\mathbb R^4)$$
and
$$e\_1\wedge e\_2+e\_4\wedge e\_3=\begin{pmatrix}&1\\-1\\&&&-1\\&&1\end{pmatrix}\in\Lambda^{2,-}(\mathbb R^4)\;.$$
Replacing the standard basis by another oriented ONB gives more elements. The elements of $O^-(4)=O(4)\setminus SO(4)$ swap both spaces. There is an $SO(4)$- but not $O(4)$-invariant scalar product on $\Lambda^2(\mathbb R^4)$ such that $$\langle\alpha,\beta\rangle\,e\_1\wedge\cdots\wedge e\_4=\alpha\wedge\beta\;.$$ For more details, please find a nice introduction to the representation theory of compact Lie groups.


So the only $SO(n)$- but not $O(n)$-invariant scalar products on $M\_n(\mathbb R)$ exist in dimension $n=4$.


Here is an explanation:
Let $G$ be any compact Lie group, and let $V$ be any $G$-representation over $\Bbbk=\mathbb R$ or $\mathbb C$.
Then $V$ admits a decomposition $V\cong V\_1\otimes\Bbbk^{n\_1}\oplus\dots\oplus V\_k\otimes\Bbbk^{n\_k}$. Here $V\_1,\dots,V\_n$ are irreducible, and $\gamma\in G$ acts as $\gamma\otimes\mathrm{id}$ on each summand. This decomposition is unique up to reordering.
Fix a $G$-invariant scalar product $g\_i$ on each $V\_i$. Then each $G$-invariant scalar product on $V$ is of the form $g=g\_1\otimes h\_1\oplus\cdots\oplus g\_k\otimes h\_k$, where $h\_i$ is an arbitrary scalar product on $\Bbbk^{n\_i}$. This is an application of Schur's lemma that you can find in most introductory books on representation theory.


If $H\subset G$ is a compact subgroup, then each irreducible $G$-representation $V\_i$ above is an $H$-representation, so it has an analogous decomposition. From this it is clear that every $H$-invariant scalar product on $V$ is $G$ invariant if and only if each $V\_i$ in the decomposition above is also irreducible as an $H$-representation (so the decomposition of $V\_i$ over $H$ is just $V\_i\cong V\_i\otimes\Bbbk$).


