Let us work over $\mathbb{C}$ for the moment.


Assume we are given a real quadratic field $K$ with ring of integers $\mathcal{O}\_K$.


$\mathbf{Question:}$ Is there a smooth projective curve $C$ of genus $g=2$ such that $End(Jac(C))$ is a non-maximal order in $K$, that is $End(Jac(C))=\mathcal{O}\_{K,f}=\mathbb{Z}+f\mathcal{O}\_K$ for some $f>1$, especially $\neq \mathcal{O}\_K$? If yes can we find such a curve for every $K$ and every $f>1$?



Are there curves of genus 2 with real multiplication by a non-maximal order?
Yes, for every quadratic ring of discriminant $f^2 D$ (where $D$ is the discriminant of $K$ in your notation), there's a Humbert surface's worth of such Jacobians. See David Gruenewald's thesis 
 (available at <http://echidna.maths.usyd.edu.au/~davidg/thesis.pdf>) and the references there for calculations of Humbert surfaces. The double cover, which is the Hilbert modular surface, can also be computed in various ways. See <https://arxiv.org/abs/1209.3527> for a method using elliptic K3 surfaces (for fundamental discriminants) and <https://arxiv.org/abs/1412.2849> for non-fundamental discriminants if K is replaced by the disc 1 quadratic algebra $\mathbf{Q} \oplus \mathbf{Q}$.


