Let $\mathfrak{gl}\_n(\mathbb{R})$ be the Lie algebra of matrices with real entries and $GL\_n(\mathbb{R})$ its associated Lie group. Recall that a linear subgroup $G \subseteq GL\_n(\mathbb{R})$ acts by conjugation on $\mathfrak{gl}\_n(\mathbb{R})$, that is, for $g \in G$ its action on $A \in \mathfrak{gl}\_n(\mathfrak{R})$, is defined by


$$g(A) = g^{-1}Ag.$$


Definition: Let $G \subseteq GL\_n(\mathbb{R})$ be a subgroup. A polynomial $f \in\mathbb{R}[(X\_{ij})\_{1\leq i,j\leq n}]$ is called invariant on $\mathfrak{gl}\_n(\mathbb{R})$ with respect to conjugation by elements in $G$ iff


$$\forall g \in G, \forall A \in \mathfrak{gl}\_n(\mathbb{R}): f(g^{-1}Ag) = f(A)$$


We will denote by $G'$ the set of all invariant polynomials.


Question: What is known about $G'$?



Invariant polynomials with respect to group actions on matrices
For the action of the upper unipotent group, which we shall denote by $G$ here, one can continue in the following way: Let $g\_r = \pmatrix{1 & r \\ 0 & 1}\in G$. Consider a $2\times 2$ real matrix $M=\pmatrix{a & b \\ c & d }$. Then $g\_rMg\_r^{-1} = \pmatrix{a+rc & b-r(a-d) - r^2c \\ c & d-rc}$. Let us write $x=a+d$ and 
$y = a-d$. Then $x$ and $c$ are invariants under the action of $G$, and the action of $g\_r$ on the coordinate functions $y$ and $b$ is given by $g\_r\cdot y = y+2rc$ and $g\_r\cdot b = b -ry - r^2c$. A direct calculation shows that the polynomial $bc+4y^2$ is also $G$-invariant (it is a linear combination, in fact, of the trace and determinant polynomials). 
I claim that $\mathbb{R}[a,b,c,d]^G = \mathbb{R}[bc+4y^2,x,c]$.


To see why this is true, consider the localization $\mathbb{R}[a,b,c,d]\_c$.
Since the action of $G$ on $c$ is trivial, we have an action of $G$ on this ring. We can then write $\mathbb{R}[a,b,c,d]\_c = \mathbb{R}[x,y,c,b+4y^2/c]\_c$.
In this presentation the action of $G$ on 3 of the generators is trivial, and one can show by an easy induction that if $G$ acts trivially on $p(x,y,c,b+4y^2/c)$ then no nontrivial power of $y$ appear in $p$.
We get that $\mathbb{R}[a,b,c,d]\_c^G = \mathbb{R}[x,c,b+4y^2/c]\_c$. 
The ring of invariants in $\mathbb{R}[a,b,c,d]$ will just be the intersection of the ring of invariants in $\mathbb{R}[a,b,c,d]\_c$ with $\mathbb{R}[a,b,c,d]$, and one can easily show that this ring is exactly $\mathbb{R}[c,x,bc+4y^2] = \mathbb{R}[c,tr,det]$.


About the embedding of $GL(1,\mathbb{C})$ in $GL(2,\mathbb{R})$: 
We denote the group by $G$ again. We need to find the invariants in $\mathbb{R}[a,b,c,d]$. We will find the invariants in the extension of scalars: $\mathbb{C}[a,b,c,d]$. The advatage is that now the action of $x+iy\in \mathbb{C}^{\times}$ is given by one dimensional representations: on $a+d$ and $c-b$ the action is trivial, while on $(c+b) + (d-a)i$ and $(c+b) + (a-d)i$ the action is given by $(x-iy)^2/(x^2+y^2)$ and $(x+iy)^2/(x^2+y^2)$. 
Since these two weights are of infinite order, and they are the inverses of each other, we get easily that the ring of invariants is now spanned by $a+d, b+c$ and $((c+b) + (d-a)i)((c+b) - (d-a)i) = (c+b)^2 + (a-d)^2$. Another set of generators will then indeed be $a+d,c-b, a^2+b^2+c^2+d^2$.


