I have encountered an algebraic group $G$ over $\mathbb C$ such that there is a Zariski open orbit for the adjoint action of $G$ on the nilpotent radical $\mathfrak n$ of its Lie algebra, but there is an ideal $\mathfrak n'\subset\mathfrak n$ such that the adjoint action of $G$ on $\mathfrak n'$ does not have open orbit.


In my example $G$ has dimension $35$, $\mathfrak n$ has dimension $28$, $\mathfrak n'$ has dimension $25$ and the largest dimension of a $G$-orbit on $\mathfrak n'$ has dimension $24$. Details are sort of unpleasant but I am ready to provide them if needed, just ask about it in a comment.


I would like to understand this phenomenon better, so my question is what is the "smallest" example of it, i. e. of a linear algebraic group $G\subset\operatorname{GL}(V)$ over $\mathbb C$ possessing an open orbit and a subrepresentation $V'\subset V$ without any open $G$-orbits.


Note that such representation cannot be completely reducible, so $G$ must be nonreductive.



Non-hereditarily locally transitive linear algebraic groups
(1) Let $G$ be the pointwise stabilizer of the first coordinate line $L\subseteq V$ in $\mathrm{GL}\_2$ (so $G$ is 2-dimensional, connected, non-reductive). Then $G$ has an open orbit on the plane $V$ (the complement of $L$) but acts trivially on $L$, hence with no open orbit.


This answer the main question, although this is not the action of an algebraic group on its nilpotent radical.




---


Edit: (2) here's now an example, of dimension 6, in which the action is the $G$-action on its nilpotent radical (which coincides with the unipotent radical, as is necessarily the case in any $G$ having an open orbit on its nilpotent radical).


Let $G$ be the semidirect product of $\mathrm{SL}\_2$ with the 3-dimensional Heisenberg group $U$. Then $G$ fixes pointwise the line $[U,U]$. However it has an open orbit on $U$, namely $U\smallsetminus [U,U]$. Indeed, the $G$-action on $U$ is conjugate to the $G$-action on its Lie algebra. On the Lie algebra, $\mathrm{SL}\_2$ acts on $\mathfrak{u}=\mathfrak{a}\_2\oplus\mathfrak{a}\_1$ ($2+1$ decomposition) as $g\cdot (x,z)=(gx,z)$. While $U/[U,U]$ acts on $\mathfrak{u}$ as $y\cdot (x,z)=(x,z+\langle x,y\rangle)$, where $\langle,\rangle$ is a symplectic form on the plane $\mathfrak{a}\_2$. Since for each $x\neq 0$, the map $x\mapsto \langle x,y\rangle$ is surjective, we deduce that for every $x\neq 0$, $(x,z)$ is in the same orbit as $(x,0)$. Since $\mathrm{SL}\_2$ acts transitively on $\mathfrak{a}\_2\smallsetminus\{0\}$, it follows that the $G$-action on $U\smallsetminus [U,U]$ is transitive.




---


(3) Moreover, (2) gives the unique smallest-dimensional example. Indeed, as already mentioned, the existence of an open orbit in the (connected) nilpotent radical forces the nilpotent radical to be equal to the unipotent radical. So, we write $G=U\rtimes L$, with $L$ reductive. By minimality, we can suppose that $L$ acts faithfully on $U$. Then $U$ is non-abelian, since otherwise by semisimplicity, existence of open orbits passes to subspaces. Moreover, $G$ has an open orbit on $U/[U,U]$, and this action factors through $G/U=L$. Hence $\dim(L)\ge\dim(U/[U,U])$.


So, $U$ being non-abelian, has dimension $\ge 3$. If $U$ has dimension $3$, $L$ embeds into a Levi factor in the automorphism group of $U$, that is $\mathrm{GL}\_2$. To be an example, the action should be trivial on the center, so $L\subset\mathrm{SL}\_2$. The inequality above says $\dim(L)\ge 2$. Since $\mathrm{SL}\_2$ has no 2-dimensional reductive subgroup, we get $L=\mathrm{SL}\_2$, which is precisely the example above.


If we are looking at at most 6-dimensional examples, given that $\dim(U/[U,U])\ge 2$ in all cases, the only remaining possibility a priori is $\dim(U,[U,U])=\dim(L)=2$, $U$ 4-dimensional (so at this stage we have already shown $6$ is the minimal dimension).


In dimension $6$, the only possibility for such $U$ is the 4-dimensional filiform unipotent group (which occurs, for instance, as semidirect product (1-dimensional) $\ltimes$ (3-dimensional) abelian unipotent groups, the action being unipotent generated by a Jordan matrix. But for this group, a Levi factor in the automorphism group is precisely a 2-dimensional torus $T$. But in this case the proper $T$-invariant ideals in $\mathfrak{u}$ are: the terms of the lower central series (of dimension 0, 1 and 2, with an open $T$-orbit), and two hyperplanes, which also have an open $G$-orbit. So this is not an example.


