Modular forms could be defined for arbitrary subgroups of the modular group, and I have read that this is done in some papers, but every definition of a modular form I have seen has been with respect to congruence subgroups. 



Why are modular forms (usually) defined only for congruence subgroups?
At a certain level, it's mostly a matter of (i) terminology and (ii) reading the right books. Technically the word "modular" in modular forms refers to the "modular group $SL\_2(\mathbb{Z})$".


In Miyake's book *Modular Forms*, he defines an **automorphic form** with respect to an arbitrary Fuchsian group $\Gamma$ (i.e., a discrete subgroup of $SL\_2(\mathbb{R})$). Then he goes on to say (p. 114) that "Automorphic functions and forms for modular groups are called *modular functions* and *modular forms* respectively." Despite the title, plenty of the book deals with the general case, or with the special case of Fuchsian groups associated to quaternion algebras, which do not yield modular forms according to his definition.


In Shimura's book *Introduction to the Arithmetic Theory of Automorphic Functions* he defines (pp. 28-29) automorphic functions and forms with respect to an arbitrary Fuchsian group of the first kind (i.e., finite hyperbolic covolume). The phrase "modular forms" is sometimes used in his book, but doesn't appear to get a formal definition.


These are, to my mind, the two most standard and authoritative references on "modular forms", and they both entertain the concept of a modular form with respect to a rather general Fuchsian group, whatever they want to call it.


On the other hand, there are reasons for restricting to Fuchsian groups which are **arithmetic** (which is a technical term here) and **of congruence type**. A theorem of Margulis shows that arithmeticity is equivalent to having a sufficiently rich theory of Hecke operators, which is highly important in number-theoretic applications. Similarly, being arithmetic of congruence type puts you in the realm of Shimura varieties, and gives you a rich theory of models of the Riemann surfaces defined over various abelian number fields. On the other hand, it is a famous consequence of Belyi's theorem that every algebraic curve over $\mathbb{Q}$ can be uniformized by a finite-index subgroup of $SL\_2(\mathbb{Z})$ (generally of non-congruence type). So if one is interested in the "special" arithmetic properties of modular curves, it makes sense to restrict to congruence type.


Indeed, continuing work of John Voight and myself indicates that the congruence type condition is even more arithmetically significant than the arithmeticity [sic!]. We define congruence subgroups of non-arithmetic Fuchsian triangle groups and derive some of the arithmetic applications (using techniques from group theory and the arithmetic theory of branched coverings) that are parallel to those satisfied by the usual modular curves. See


[http://alpha.math.uga.edu/~pete/triangle-091309.pdf](http://alpha.math.uga.edu/%7Epete/triangle-091309.pdf)


Note that this work is not yet finished, to my consternation. (Mea culpa. **Mea** culpa.)


