As we have known, the Sato-Tate measure for GL(2) turned out to be the half circle measure


$\frac{1}{2\pi} \sqrt{4-x^2}dx$ on [-2,2],


which appears in various versions of equi-distribution problems in GL(2).


My question is what the corresponding measure for GL(3) should be.


Firstly we shall note that Hecke eigenvalues in GL(3) can be imaginary. And we have to replace [-2,2] with {$z\in \mathbb{C}:|z|<=3$} the ball of radius 3 in complex plane. I don't know anything further yet.



Sato-Tate measure for GL(3) Automorphic forms
(2017-11-26 edit by j.c.: earlier versions of this answer consisted of David Hansen's screenshot of the following, with the text "Here is a screenshot of a semi-answer which froze my computer when I hit 'post'":)


Things get more complicated.
$\newcommand{\SU}{\mathop{\rm SU}\nolimits}
\newcommand{\C}{\bf C}
\newcommand{\Irr}{\mathop{\rm Irr}\nolimits}
$


Let $\pi=\pi\_\infty\otimes\bigotimes'\_p\pi\_p$ be a unitarily normalized cuspidal automorphic representation of $GL\_n(\bf{A}\_{\bf Q})$; suppose for simplicity that $\pi$ has trivial central character. The local factor $\pi\_p$ (for $p$ outside a finite set $S$) is determined by a matrix $s(\pi\_p)\in GL\_n(\C)$, which is well-defined up to conjugacy. The generalized Ramanujan conjecture predicts that $s(\pi\_p)$ may be chosen as an element of the unitary group $\SU\_n$. Suppose this is true, and let $f:\SU\_n\rightarrow\C$ be a continuous function which is invariant under conjugation. Here is a generalized Sato-Tate conjecture:



> 
> If $\pi$ is not a functorial lift from a "smaller" group, then $\frac{1}{\pi(X)}\sum\_{p\leq X}f(s(\pi\_p))\rightarrow \int\_{\SU\_n}f(g)dg$ where $dg$ is the Haar probability measure on $\SU\_n$.
> 
> 
> 


If $n=2$ and $\pi$ is associated with an elliptic curve $E$, this simplifies drastically: the conjugacy class $s(\pi\_p)\in\SU\_2$ is determined uniquely by its trace, which is $\frac{a\_p(E)}{\sqrt{p}}$. Likewise, any $f$ determines a unique continuous $f':[-2,2]\rightarrow\C$, with $f=f'\circ\mathrm{tr}$. The Haar probability measure $dg$ pushes forward to the measure $\frac{1}{2\pi}\sqrt{4-x^2}$ on $[-2,2]$, and the conjecture now takes the more familiar form.


What's the meaning of the vague condition "$\pi$ is not a functorial lift from a smaller group"? Well, let's try to give a plausibility argument for the above conjecture. Let $\Irr\_n$ denote the set of irreducible algebraic representations of $\SU\_n$; for $\sigma\in\Irr\_n$, write $\chi\_\sigma$ for its character. In $L^2(\SU\_n)$ we have 


$$f(g)=\sum\_{\sigma\in\Irr\_n}\chi\_\sigma(g)\langle f,\chi\_\sigma\rangle.$$


Changing $g$ to $s(\pi\_p)$, summing over $p\leq X$ and switching the order of summation gives


$$\frac{1}{\pi(X)}\sum\_{p\leq X}f(s(\pi\_p)) \approx \int\_{\SU\_n}f(g)dg + \sum\_{\sigma\text{ nontrivial}}\frac{1}{\pi(X)}\sum\_{p\leq X}\chi\_\sigma(s(\pi\_p)).$$


The behavior of the sum $\sum\_{p\leq X}\chi\_\sigma(s(\pi\_p))$ is ontrolled by the Langlands L-function $L(s,\pi,\sigma)$, and in particular this sum is $o(X)$ if the L-function is nonvanishing and without poles in $\mathrm{Re}(s)\geq 1$. The truth of this latter property for all $\sigma$ is expected to be equivalent to $\pi$ not arising via functoriality from a smaller group.


