Let $R$ be a (noncommutative) ring. (For me, the words "ring" and "algebra" are isomorphic, and all rings are associative with unit, and usually noncommutative.) Then I think I know what "linear algebra in characteristic $R$" should be: it should be the study of the category $R\text{-bimod}$ of $(R,R)$-bimodules. For example, an $R$-algebra on the one hand is a ring $S$ with a ring map $R \to S$. But this is the same as a ring object in the $R\text{-bimod}$. When $R$ is a field, we recover the usual linear algebra over $R$; in particular, when $R = \mathbb Z/p$, we recover linear algebra in characteristic $p$.


Suppose that $G$ is an algebraic group (or perhaps I mean "group scheme", and maybe I should say "over $\mathbb Z$"); then my understanding is that for any *commutative* ring $R$ we have a notion of $G(R)$, which is the group $G$ with coefficients in $R$. (Probably there are some subtleties and modifications to what I just said.)



> 
> **My question:** What is the right notion of an algebraic group "in characteristic $R$"?
> 
> 
> 


It's certainly a bit funny. For example, it's reasonable to want $GL(1,R)$ to consist of all invertible elements in $R$. On the other hand, in $R\text{-bimod}$, the group $\text{Aut}(R,R)$ consists of invertible elements in the *center* $Z(R)$.


Incidentally, I'm much more interested in how the definitions must be modified to accommodate noncommutativity than in how they must be modified to accommodate non-invertibility. So I'm happy to set $R = \mathbb H$, the skew field of quaternions. Or $R = \mathbb K[[x,y]]$, where $\mathbb K$ is a field and $x,y$ are noncommuting formal variables.



What is an algebraic group over a noncommutative ring?
It seems that you want some notions on noncommutative group scheme,right?
In fact, A.Rosenberg has introduced **noncommutative group scheme** in his work with Kontsevich
["noncommutative grassmannian and related constructions"](http://www.mpim-bonn.mpg.de/preblob/3621) (2008). Actually, this work gave a systematically treatment to the noncommutative grassmannian type space introduced in their early paper [noncommutative smooth space](https://ui.adsabs.harvard.edu/abs/1998math.....12158K/abstract) and the work of Rosenberg himself on [noncommutative spaces and schemes](http://www.mpim-bonn.mpg.de/preblob/950).


More comments: It seems that you want to know the linear algebra over noncommutative ring. I think you need to look at the paper by Gelfand and Retakh on [Quasideterminants, I](https://doi.org/10.1007/s000290050019). And the main motivation for the "noncommutative grassmannian and related constructions" is to give a geometric explanation to the work of Gelfand and Retakh.


All of these work are based on functor of point of view.


