Let $G$ be a smooth affine group scheme over a base $S$. $G$ acts on a scheme $X$ over $S$. Let $x$ be an $S$-point in $X$. Then we have an orbit map $G\to X$. I wonder when the image (set-theoretically) of this map is locally closed, and the induced scheme structure (the minimal one) on the orbit is smooth over $S$.



Smoothness of orbit of group scheme
$\textbf{Edit by afh:}$ *Unfortunately this answer is not correct. I apologize, there is a small bug in one of the last steps in the argument below (the surjective morphism of flat schemes at the end does not need to be a closed immersion/isomorphism). In fact the statement of the proposition below is not true even if the base is a DVR and $G$ is etale and quasifinite. The proof below only shows that the set theoretic image is locally closed when the base is a DVR.*


Let me add another answer addressing the first comment of the OP above, where he asks for possible hypothesis that ensure the orbit is locally closed. I hope I am not making a lot of mistakes. In summary, there is a positive answer to the question if one assumes that the scheme theoretic stabilizer $G\_x$ of the section $x$ is flat.


Here is the setup: let $S$ be a Noetherian scheme (probably can be removed) and let $X$ be a scheme of finite type over $S$. Let $G$ be a smooth affine algebraic group over $S$, and fix an action of $G$ on $X$. Let $x: S \to X$ be a section. If the scheme theoretic stabilizer $G\_x$ is flat over $S$, then we can form the quotient algebraic space $G /G\_x$ over $S$. By construction, the action morphism
$$ G \to X, \; \; g \mapsto g \cdot x$$
factors through the quotient $G \to G/G\_x$. Therefore we get a natural morphism $G/G\_x \to X$. We shall denote $O= G/G\_x$ and call it the orbit.


$\textbf{Proposition}$ In the setup above (under the assumption that $G\_x$ is $S$-flat), the orbit $O$ is represented by a smooth scheme over $S$ and the natural morphism $O \to X$ is a locally closed immersion.


Proof: First, since smoothness can be checked flat locally and $G$ is smooth, the fppf quotient morphism $G \to G/G\_x$ shows that $O$ is smooth over $S$. It is not difficult to show that the orbit morphism $O \to X$ is a monomorphism of algebraic spaces (cf. Section 2.1 in <https://arxiv.org/abs/0804.2242>). Now, since $X$ is a scheme, we can apply <https://stacks.math.columbia.edu/tag/03XX> to the monomorphism $O \hookrightarrow X$ to conclude that $O$ is a scheme. It remains to check that $O \hookrightarrow X$ is a locally closed immersion.


We will use EGA IV (15.7.6). This Corollary in EGA says that if the valuative criterion for local properness (to be explained below) is satisfied for $ \phi: O \hookrightarrow X$, then the morphism $\phi$ factors as a composition $h \circ g$, where $h$ is an open immersion and $g$ is proper. In this case this would mean that $g$ is a proper monomorphism, and hence a closed immersion. In this (quasicompact) situation, this would in turn imply that $\phi: O \to X$ is a locally closed immersion. We are left to prove the valuative criteria mentioned above.


This is what we have to show. Let $R$ be a DVR with field of fractions $K$. Suppose that we are given a morphism $Spec(R) \to X$ that factors set theoretically through the set theoretic image $\phi(O)$. The local valuative criterion stipulates that any section $Spec(K) \to O \times\_{X} Spec(K)$ must extend uniquely to a section $Spec(R) \to O \times\_{X} Spec(R)$. In order to check this, we are free to base change using the morphism $Spec(R) \to X \to S$ in order to assume that the base $S$ is the spectrum of a DVR.


So we assume that $S$ is $Spec(R)$ with generic point $\eta$ and special point $s$. Take the scheme theoretic image of $Z \subset X$ of the morphism $\phi: O \hookrightarrow X$. Since $S$ is a DVR and $O$ is $S$-flat, it follows that $Z$ is automatically flat over $X$ (this is the crucial reason why we passed to a DVR). It can be checked that $G$ still acts on $Z$ (as the scheme theoretic image of a quasicompact $G$-equivariant morphism), so we might as well replace $X$ with $Z$ and assume that $X$ is flat and $O$ is scheme theoretically dense in $X$. Now taking scheme closure commutes with flat base change, so the generic fiber $O\_{\eta} \hookrightarrow X\_{\eta}$ is scheme theoretically dense. The usual argument for orbits over fields (notice that the construction of $O$ commutes with arbitrary base-change!) shows that $O\_{\eta} \hookrightarrow X\_{\eta}$ is an open immersion. Since $O$ is smooth, this shows that $X\_{\eta}$ is geometrically reduced with the same dimension as $O\_{\eta}$, and the boundary $B\_{\eta} = X\_{\eta} \setminus O\_{\eta}$ has strictly smaller dimension. By flatness, the dimension of $X\_{s}$ is the same as the dimension of $X\_{\eta}$, and so we must have that $X\_{s}$ and $O\_{s}$ have the same dimension. Again, the usual argument for fields implies that $O\_{s} \hookrightarrow X\_s$ is locally closed, and since it is full dimension and smooth this means that the image of $O\_s \subset X\_s$ is open (but note that $X\_s$ could be nonreduced, so we don't know yet that $O\_s \to X\_s$ is an open immersion).


We equip the boundary with $B\_{\eta}$ with its reduced subscheme structure. Since $G\_{\eta}$ is geometrically reduced, it acts on $B\_{\eta}$. Take the scheme theoretic closure in $X$ of the boundary $B\_{\eta} \to X\_{\eta}$, let's call it $B$. $B$ is $G$-stable, and the fibers have smaller dimension than the fibers of $O$. This means that $B\_s$ must be disjoint from the open orbit $O\_s$ of bigger dimension. Hence the closed subset $B$ is disjoint from the image of $\phi: O \hookrightarrow X$, and we can replace $X$ with $X \setminus B$. Hence we can assume that the generic fiber $\phi\_{\eta} : O\_{\eta} \to X\_{\eta}$ is an isomorphism. Finally, by removing the closed complement $X\_s \setminus O\_s$ inside the generic fiber $X\_s$, we can assume that the morphism of $R$-flat schemes $\phi: O \to X$ becomes an infinitesimal closed immersion when restricted to the special fiber ($O\_s$ will be the reduced subscheme of $X\_s$ if $X\_s$ is not reduced). Since the restriction to the generic fiber is also a closed immersion (isomorphism actually), this implies that $O \hookrightarrow X$ is a closed immersion. Since $O$ is schematically dense inside $X$, this means that $O \hookrightarrow X$ must be an isomorphism. In summary, we have shown that $O \to X$ induces an open immersion into its scheme theoretic closure. This implies that $O \to X$ is a locally closed immersion, and so by the same proposition in EGA IV (15.7.6), it satisfies the valuative criterion for local properness. QED


$\textbf{Last Remark/ Warning:}$ The scheme theoretic image $Z$ of the locally closed immersion $O \hookrightarrow X$ does not need to commute with passing to fibers. In other words, the fibers of the orbit $O$ do not need to be dense in the fibers of the closure $Z$. Therefore one cannot in principle conclude properties of the fibers of $Z$ in terms of properties of $O$. An example that I like is to let $R$ be a DVR with uniformizer $\pi$ and consider the scheme $X = Spec(R[s,t]/(st - \pi))$. We can let $\mathbb{G}\_m$ act on $X$ with weight $-1$ on $t$ and weight $1$ on $s$. Consider the section $(t,s) = (1,\pi)$. The stabilizer of this section is trivial, and the orbit $O$ is the open immersion $O \hookrightarrow X$ with closed complement the vanishing locus of $t$. At the generic fiber we have an isomorphism $O\_{\eta} \cong X\_{\eta}$ (there is a single orbit at the generic fiber), but at the special fiber the complement $X\_s \setminus O\_s$ is the vanishing locus of $t$, which contains two orbits: another open orbit of $(t,s) = (0,1)$, and the closed orbit at the origin $(t,s) = (0,0)$.


