Suppose $G$ is a sufficiently nice (maybe locally compact and abelian) group which acts on the separable Hilbert space $\mathcal{H}$ by unitary transformations. Is there a generalization of the spectral theorem to this context? Specifically, what ought the spectrum of such a group action be, and, do generalizations of spectral measures exist in this context?


This is motivated by the following line of thinking. A unitary operator $U$ naturally induces a $\mathbb{Z}$-action on $\mathcal{H}$ via $ k \cdot \psi = U^k \psi $. Then, one version of the spectral theorem produces Borel measures $\mu\_\psi^U$ supported on $\mathbb{T}$ with the property that
$$
\langle \psi, U^k \psi \rangle 
= \langle \psi, k \cdot \psi \rangle 
= \int\_{\mathbb{T}} \! z^k \, d\mu\_\psi^U(z).
$$
for all $ \psi \in \mathcal{H}, k \in \mathbb{Z} $. There are a few observations that one can make. First, the spectrum of $U$ lives in the dual group $ \widehat{\mathbb{Z}} = \mathbb{T} $. Also, if $ E\_k:\mathbb{T} \to \mathbb{T} $ is the map $E\_k(z) = z^k$, then $ k \mapsto E\_k $ is the usual Pontryagin isomorphism $ \mathbb{Z} \to \widehat{\mathbb{T}} $.


Similarly, a bounded self-adjoint operator $A$ acting on $\mathcal{H}$ induces a (continuous) unitary action of $\mathbb{R}$ on $\mathcal{H}$ via $ t \cdot \psi = e^{-itA} \psi $. Again, the spectral theorem produces measures $ \mu = \mu\_\psi^A $ supported on $\mathbb{R}$ such that
$$
\langle \psi, e^{-itA} \psi \rangle 
= \langle \psi, t \cdot \psi \rangle 
= \int\_{\mathbb{R}} \! e^{-itx} \, d\mu\_\psi^A(x),
$$
and the spectrum lives in the dual group $ \widehat{\mathbb{R}} = \mathbb{R} $. This leads me to wonder the following: given a (sufficiently nice) action of a (sufficiently nice) group $G$ by unitary operators, do there exist measures $ \mu\_\psi^G $ supported on $\widehat{G}$ such that
$$
\langle \psi, g \cdot \psi \rangle
=
\int\_{\widehat{G}} \! E\_g(\chi) \, d\mu\_\psi^G(\chi)?
$$
As above, $E\_g(\chi) = \chi(g)$.


This seems like a fairly reasonable question to ask, but I haven't been able to find a discussion of things like this in any of my texts on spectral theory (or any of my colleagues' texts on unitary representations).


Remark: In case this is nonstandard notation, $ \widehat{G} $ denotes the multiplicative group of continuous group homomorphisms $ G \to \mathbb{T} $, and $ \mathbb{T} = \{ z \in \mathbb{C} : |z| = 1\} $.



A version of the spectral theorem for group actions
This is true for locally compact abelian $G$; you ought to be able to find it in any text on abstract harmonic analysis (a reference I have at hand is Theorem 4.44 in Folland's *A Course in Abstract Harmonic Analysis*).


Here's a bit of general perspective that may be helpful. Unitary representations of $G$ are the same as representations of the group C\*-algebra $C^\*(G)$ (essentially by definition of $C^\*(G)$), and the spectrum of $C^\*(G)$ can be identified with $\widehat{G}$. By the classification of commutative C\*-algebras, this identifies $C^\*(G)$ with $C\_0(\widehat{G})$, the algebra of functions on $\widehat{G}$ vanishing at infinity. For any locally compact Hausdorff space $X$, there is a spectral theorem for representations of $C\_0(X)$ whose proof is not much harder than the case of a single normal operator (which is just the case when $X$ is a compact subset of $\mathbb{C}$).


