Consider the compact group $ G=\operatorname{SO}\_3(\mathbb{R}) $. The closed subgroups of $ G $ (other than the trivial group 1 and the whole group $ G $) are exactly $ O\_2$, $\operatorname{SO}\_2 $ and the finite groups $ C\_n$, $D\_{2n}$, $T \cong A\_4$, $O \cong S\_4$, $I \cong A\_5 $ (cyclic groups with $ n $ elements, dihedral groups with $ 2n $ elements and the three symmetry groups of the platonic solids). The normalizers of these groups are as follows:


\begin{align\*}
G&=N\_G(G)=N\_G(1) \\
O\_2&=N\_G(O\_2)=N\_G(\operatorname{SO}\_2)=N\_G(C\_n) \\
I&= N\_G(I) \\
O&=N\_G(O)=N\_G(T)=N\_G(D\_4) \\
D\_{4n} &= N\_G(D\_{2n}) 
\end{align\*}
where in the last equation $ n \geq 3 $. We say a (closed) subgroup is maximal if it is maximal among all proper closed subgroups of $ G $.


Observe that in the example above the maximal subgroups exactly coincide with the self-normalizing subgroups. Namely,
$$
O\_2, I,O.
$$
That the maximal subgroups are all self-normalizing is not too surprising. The normalizer of a closed subgroup is always closed. Thus a maximal subgroup is always either normal or self-normalizing. Since $ G $ is simple, adjoint (i.e. center-free), and connected that means the maximal subgroups must be self-normalizing. However I am a bit surprised that the reverse holds. That is, that every self-normalizing subgroup of $ G $ is maximal. That inspires my question:


**For a closed subgroup of a compact Lie group does self-normalizing imply maximal?**


This is true for the compact Lie group $ \operatorname{SO}\_3(\mathbb{R}) $ and thus also true for $ \operatorname{SU}\_2 $. What about the generic case? I am especially interested in $ \operatorname{SU}\_3 $.



Self-normalizing implies maximal for subgroup of compact Lie group
$\DeclareMathOperator\SO{SO}\DeclareMathOperator\U{U}\DeclareMathOperator\O{O}$Denote by $\U'(n)$ the normalizer of $\U(n)$ in $\mathrm{GL}\_{2n}(\mathbf{R})$. It is not hard to see that $\U(n)$ has index 2 in $\U'(n)$, which is generated by $\U(n)$ and by the coordinate-wise complex conjugation. Moreover, $\U'(n)$ is maximal in $\O(2n)$ (if $n\ge 2$).


In $G=\SO(5)$, consider the subgroup $H=\SO(5)\cap (\U'(2)\times \O(1))$ (which contains $\U(2)$ with index 2). I claim that $H$ is self-normalized, but not maximal, in $G=\SO(5)$.


The subgroup $H$ is not maximal in $G$ because it is properly contained in $L=SO(5)\cap (\O(4)\times \O(1))$.


It is self-normalized. Indeed, its action on $\mathbf{R}^5$ has the irreducible decomposition $4\oplus 1$, which is preserved by the normalizer. Hence, the normalizer of $H$ in $G$ equals the normalizer of $H$ inside $\SO(5)\cap (\O(4)\times \O(1))=L$. Using that $\U'(2)$ is maximal in $\O(4)$ one can deduce easily that $H$ is maximal in $L$. Since $H$ is not normal in $L$, it is self-normalized in $L$. Hence $H$ is self-normalized in $G$.


