Let $k$ be an algebraically closed field of characteristic $p>0$.
All the examples of non-smooth algebraic group schemes over $k$ that
I have seen (apart from "artificial" examples; see below) have been given by presentations with at least one defining
relation of degree a positive power of $p$. Here are the examples
I have looked at:


* Frobenius kernels.
* Some automorphism schemes of algebras.
The paper "Non-reduced automorphism schemes" by Geiss and Voigt has
an interesting example in Section 2, which is given by a presentation
including some relations of degree 2. The authors state that this group scheme is not reduced if and only if $p=2$. They also mention that if $G$ is a finite $p$-group,
then the group scheme $\mathrm{Aut}(k[G])$ is not reduced.
* Results by Sopkina, "Classification of all connected subgroup
schemes of a reductive group containing a split maximal torus",
imply, if I have understood things correctly, that every non-smooth
subgroup scheme of a reductive group over $k$ (or at least of
$\mathrm{GL}\_{n}$) containing a maximal torus, has a presentation including a
$p$-power relation.
* Non-smooth centralisers in classical groups in not very good characteristic.


On the other hand, it is easy to produce "artificial" examples
of a non-smooth group scheme in, for example, char 3 with a presentation
involving only quadratic relations. Namely, take $k$ of char 3 and
$\alpha\_{3}=\mathrm{Spec}\, k[x]/(x^{3})$. Since $k[x]/(x^{3})$ is isomorphic
to $k[x,y]/(xy,x-y^{2})$ as $k$-algebras, we can transport the Hopf
algebra structure from the former to the latter. 



> 
> **Question.** Is there a non-smooth algebraic group scheme $G$ over $k$, and an
>  embedding $G\rightarrow\mathbb{A}^{n}$ of $G$ as a closed subscheme of affine
>  $n$-space, with $n$ *minimal*, such that every defining relation
>  of $G$ in this embedding is of degree strictly less than $p$? (Even
>  the case $p=3$ would be interesting.) 
> 
> 
> 


Using explicit equations and a minimal embedding into affine space may seem a bit unnatural, but results of Kollár and Jelonek (see [this](https://mathoverflow.net/questions/209079/effective-nullstellensatz-and-bounds-on-the-nilpotency-index-of-reduced-ideal-to) previous MO question) - which boil down to estimating degrees and Bézout's theorem - imply that if we take
$p>d^{n}$, where $d$ is the maximal degree of a relation, then $p$ does not divide the nilpotency index of any element in the coordinate algebra of $G$, so a proof of Cartier's theorem can be carried through in this case. Hence an example as in the above question must have $d<p<d^n$.



Is there a non-smooth algebraic group scheme in char $p$, all of whose defining relations have degree less than $p$?
No. Let $f$ be a relation of minimal degree $2 \leq d <p$. Apply the comultiplication. This must be zero in $R \otimes R$, where $R$ is the ring of functions. So if $x\_1, \dots, x\_n$ are the variables, then it is zero in $k[x\_1, \dots, x\_n] \otimes k[x\_1,\dots x\_n]$ modulo the various relations.


Write $f$ as a sum of monomials in the $x\_i$. When we apply the comultiplication to a monomial, the leading term does not depend on which monomial we pick. It's just the sum over the ways of splitting that monomial into a product of two monomials. So each pair of monomials in $k[x\_1, \dots, x\_n] \otimes k[x\_1,\dots x\_n]$ comes from a unique monomial of $f$, so we may ignore cancellation among different monomials. Moreover, because $d \geq 2$, we may split it into two monomials, each of degree $<d$. Because $d$ was the minimal degree of relations, we may ignore the relations. So we end up with something nonzero unless the number of ways of splitting is nonzero. But the number of ways of splitting is a product of binomial coefficients. All numbers involved in the binomial coefficients are less than $p$, so the binomial coefficients are prime to $p$ and hence nonzer.


