Let $G$ be a reductive algebraic group over an algebraically closed field (of characteristic zero if it matters) and $H \subset G$ a subgroup, also reductive. Is the identity component of the normalizer of $H$ in $G$ always reductive?



Is the normalizer of a reductive subgroup reductive?
I assume $H$ and $G$ are connected, and will explain a purely algebraic proof of the affirmative answer in characteristic 0 (bypassing Cartan involutions) and address the possibility of failure in positive characteristic by putting the question into a broader context that also addresses Jim Humphreys' question of why this is a natural question to wonder about (though I don't know the OP's motivation). 


For reasons that will become clearer below, my expectation is that the answer is negative in positive characteristic, due to the failure of nontrivial connected semisimple groups (unlike tori) to have completely reducible representation theory in such characteristics. (EDIT: McNinch has now given such examples over any algebraically closed field of characteristic $p > 0$: the adjoint semisimple subgroup $H = {\rm{PGL}}\_n$ inside ${\rm{SL}}({\mathfrak{gl}}\_n)$ embedded via "conjugation" for any $n$ divisible by $p$.) No doubt in char. 0 the argument via Cartan involutions by Aakumadula is simpler than what is below in char. 0 (though the proof of Mostow's theorem upon which Aakumadula's argument rests is not easy).


Let $N\_G(H)$ and $Z\_G(H)$ denote the respective scheme-theoretic normalizer and centralizer of $H$ in $G$, so we want to determine if $N\_G(H)^0\_{\rm{red}}$ is reductive. The first point is that (in any characteristic) $N\_G(H)\_{\rm{red}}^0$ is reductive if and only if $Z\_G(H)\_{\rm{red}}^0$ is reductive. The "only if" direction follows from the normality of $Z\_G(H)\_{\rm{red}}$ in $N\_G(H)\_{\rm{red}}$. For the converse, consider the action of $N\_G(H)$ on $H$ through conjugation in $G$. This defines a map from $N\_G(H)$ to the automorphism functor of $H$ (over $k$), and since $H$ is connected reductive this functor is represented by a smooth $k$-group scheme ${\rm{Aut}}\_{H/k}$ whose identity component is the adjoint quotient $H^{\rm{ad}}$ (by SGA3 technology which could be avoided for our purposes by using some auxiliary ad hoc constructions). In particular, the action of $N\_G(H)^0$ on $H$ is given by composing a $k$-homomorphism
$$f:N\_G(H)^0 \rightarrow {\rm{Aut}}\_{H/k}^0 = H^{\rm{ad}}$$
with the natural action of $H^{\rm{ad}}$ on $H$. But $f|\_H$ is the natural quotient map (due to how $H^{\rm{ad}}$ is identified with a subgroup of ${\rm{Aut}}\_{H/k}$) and $(\ker f)\_{\rm{red}}^0 = Z\_G(H)^0\_{\rm{red}}$, so the insensitivity of reductivity to passing to an isogenous quotient and the reductivity of $H^{\rm{ad}}$ imply that $N\_G(H)^0\_{\rm{red}}$ is reductive if (and only if) $Z\_G(H)^0\_{\rm{red}}$ is reductive. 


So now we are faced with the question of determining when $Z\_G(H)^0\_{\rm{red}}$ is reductive. Observe that if $H$ were not given as a $k$-subgroup of $G$ (equipped with its conjugation action on $G$) but rather were given to us as an abstract connected reductive group equipped with an abstract action on $G$ as a $k$-group then we could always form the associated connected reductive semidirect product $G' = G \rtimes H$ in which $H$ lies as a subgroup and for which
$$Z\_{G'}(H) = G^H \times H$$
(where $G^H$ is the scheme-theoretic centralizer in $G$ of the $H$-action). Thus, the reductivity of $(G^H)^0\_{\rm{red}}$ is equivalent to that of $Z\_{G'}(H)^0\_{\rm{red}}$. 


To summarize, the original *general* problem over $k$ of reductivity of the identity component of normalizers of connected reductive subgroups of connected reductive groups is equivalent to the *general* problem over $k$ of when the 
underlying reduced scheme of the centralizer $G^H$ for an abstract action of a connected reductive $k$-group $H$ on a connected reductive $k$-group $G$ has reductive identity component. (In particular, the answer is affirmative to one of these two general questions over $k$ if and only if the answer is affirmative to the other general question over $k$.)


We shall now record a purely algebraic result (valid over algebraically closed fields of any characteristic) that gives an affirmative answer to the above questions in characteristic 0 and illuminates the search for counterexamples in positive characteristic. Recall that an affine $k$-group scheme of finite type is called *linearly reductive* if its finite-dimensional linear representation theory over $k$ is completely reducible. In characteristic 0 this is equivalent to reductivity of the identity component, whereas in positive characteristic the only linearly reductive $k$-group schemes are the affine finite type $k$-groups whose identity component is multiplicative type and component group has order not divisible by char($k$) (by a theorem of Nagata). 


$\mathbf{Theorem}$: If $H$ is a linearly reductive $k$-group equipped with an action on a connected reductive $k$-group $G$ then $(G^H)^0\_{\rm{red}}$ is reductive.


This (hard) theorem is Proposition A.8.12 of the book "Pseudo-reductive groups". (When ${\rm{char}}(k) = p > 0$ this Theorem is useful for $H = \mu\_{p^n}$, not just for the rather classical case when $H$ is a torus.) The proof rests on a beautiful result due independently to Borel (using etale cohomology) and Richardson (using Haboush's theorem), and it also involves considerations with non-representable functors.


Observe that finding a counterexample to the original problem in positive characteristic amounts to showing that the above Theorem has optimal hypotheses; e.g., it fails for some connected semisimple $H$ (and a suitable $G$). This is a very natural question in positive characteristic, and so (as I hope Jim Humphreys will agree) adequately motivates the original problem. 


