$\DeclareMathOperator\SO{SO}$Suppose we have a (continuous) linear action of $\SO(n,\mathbb R)$ on a vector space $\mathbb R^N$. Consider the ring of invariants $A\subset \mathbb R[x\_1,\ldots, x\_N]$, which is an $\mathbb R$-algebra. Is it true that the orbits of the $\SO(n,\mathbb R)$ action are in one-to-one correspondence with the $\mathbb R$-algebra homomorphisms $A\to\mathbb R$?


It is clear that for each orbit of the action we get such an evaluation homomorphism, by the value of an invariant polynomial on the orbit. But is the inverse true? Is there some nice book that considers that type of setting, over the real numbers?



Invariant theory over $\mathbb R$
The answer depends on what you mean by "one-to-one correspondence". Is it bijective or just injective? Robert Bryant's (standard) argument shows that $\mathbb R^N/\mathrm{SO}(n)\to \mathrm{AlgHom}\_{\mathbb R}(A,\mathbb R)$ is injective. In general, this map is very far from being surjective, though. In other words, not every Homomorphism $A\to\mathbb R$ is an evaluation map.
Already the standard action of $\mathrm{SO}(n)$ on $\mathbb R^n$ is a counterexample. In this case, $A=\mathbb R[q]$ where $q(v)=\|v\|^2$ is the norm square function. Thus a homomorphisms $f:A\to\mathbb R$ is given by the value $f(q)$ which can be an arbitrary real number. But the image of $\mathbb R^n/\mathrm{SO}(n)$ is obviously only $\mathbb R\_{\ge0}$.


So what is the "correct quotient"? The answer is "It depends". A topologist would say it is the orbit space. The embedding into $\mathrm{AlgHom}\_{\mathbb R}(A,\mathbb R)$ gives the orbit space the structure of a semialgebraic set on which you have notions of continuous or smooth function. A theorem of G. Schwarz states that all continuous/smooth invariants are pull-backs of continuous/smooth function on the orbit space.


An algebraist would prefer to call $\mathrm{AlgHom}\_{\mathbb R}(A,\mathbb R)$ to be the quotient since it classified all closed orbits defined over $\mathbb R$ regardless whether the contain a real point or not.


A very nice paper calculating the image of $\mathbb R^N/\mathrm{SO}(n)\to \mathrm{AlgHom}\_{\mathbb R}(A,\mathbb R)$ (in particular) in terms of inequalities is Procesi, Claudio; Schwarz, Gerald: Inequalities defining orbit spaces.
Invent. Math. 81 (1985), 539–554


