Let $p\geq 3$ be a prime number. The modular curve $X(p)$ can be considered as a connected smooth projective curve over complex numbers. There is a subgroup $\mathrm{PSL}\_2(\mathbb{F}\_p)$ inside its group of automorphisms (Serre has proved that for $p\geq 7$ it is the full group of automorphisms). 


For which $p$ does there exist a geometrically connected smooth projective curve $X$ over $\mathbb{Q}$ such that $X(p)$ is the base change of $X$ and such that all $\mathrm{PSL}\_2(\mathbb{F}\_p)$-automorphisms of $X(p)$ are base changed from automorphisms of $X$?



Automorphisms of the modular curve defined over $\mathbb{Q}$
A nontrivial unipotent element of $PSL\_2(\mathbb F\_p)$ fixes $(p-1)/2$ cuspidal points of this modular curve. Its eigenvalue on the tangent space of those points is a $p$th root of unity.


If the curve and the action are defined over $\mathbb Q$, then this set of roots of unity must be Galois-invariant. However, there is no set of $(p-1)/2$ Galois-invariant roots of unity, as the Galois action on them is transitive.


In fact, I think one can check that the roots of unity obtained are a Galois orbit under the absolute Galois group of the quadratic extension of $\mathbb Q$ with conductor $p$, suggesting that the curve and the action may be defined over that field, as is true at least for $p=3$ and $p=5$.


