During my researches I've come across the following question.


Let $A$ and $B$ be a couple of square $k\times k$ skew symmetric matrices on $\mathbb R$. Let us consider the (real) pencil generated by $A$ and $B$ that is
$$P(A,B)\doteq \lambda A+\eta B,\; \lambda,\eta\in\mathbb R.$$


What I am studying is the stabilization of $P(A,B)$ under the action of the orthogonal group $O(k)$, that is I would like to know what can be deduced about $A$ and $B$ under the following request that is 


$$(\*)\quad M P(A,B) M^{T}\subset P(A,B),\quad \forall M\in O(k).$$


Another interesting question for me would be to characterize all the $M\in O(k)$ such that $(\*)$ holds.


What I would like to know is if there are already similar results in literature and what is known on the subject, references are warmly welcomed and I thank you all in advance for your kind help.


Best Wishes.



Stabilization of the pencil of skew symmetric matrices by the orthogonal group
Your first question



> 
> "what can be deduced about $A$ and $B$ under the request $(\*)$"
> 
> 
> 


can be reformulated as follows.


In fact, you are asking if there exists a $2$-dimensional vector subspace $V \subset \mathfrak{so}(k)$ stable under the adjoint action of $O(k)$


$$\mathrm{Ad} : O(K) \to \mathrm{Aut}(\mathfrak{so}(k))$$


defined by $\mathrm{Ad}(M)N := MNM^\*$ for all $M \in O(k)$ and $N \in \mathfrak{so}(k)$


Thus you are asking if the adjoint representation of $O(k)$ is reducible (and admits a reduction of dimension $2$).


Your second question, i.e.



> 
> "characterize all the $M \in O(k)$ such that $(\*)$ holds"
> 
> 
> 


is indeed harder, but it has an explicit solution for $k=3$. In this case $\mathfrak{so}(3) \simeq \mathbb{R}^3$ and the adjoint representation of $O(3)$ is just the tautological representation.


Under the above identification, the stabilizer of a two dimensional subspace in $\mathfrak{so}(3)$ under the adjoint action is just the stabilizer of a two dimensional subspace in $\mathbb{R}^3$ under the tautological representations, namely the subgroup $SO(2)$.


