I asked this over stackexchange with no answer so here we are,


I know by constructing some particular cases that I can find unitary matrices $X$, $Y$ and $Z$ such that


$X^m = Y^n = Z^p = XYZ = 1$


with 


$$
\frac{1}{m} + \frac{1}{n}+\frac{1}{p} < 1
$$


indicating an infinite von Dyck group unless the fact that the matrices are unitary implies some additional, non-trivial relations between $X$, $Y$ and $Z$. Is it possible for infinite von Dyck or triangle groups to be subgroups of $SU(n)$?


Also, can somebody point to me some references on representation theory of infinite discrete groups?



Can the infinite von Dyck groups be subgroups of $SU(n)$?
The question was somewhat sloppy so it was unclear if it was about *some* or *all* von Dyck groups admitting embeddings in $U(n)$. If the question was *some* then Victor's answer clearly suffices. 


If the question was about *all* von Dyck groups, then Victor's answer almost gets there but not quite, since only finitely many of these groups are arithmetic (first observed by Takeuchi who actually listed them all, but also follows from far more powerful finiteness theorem of Borel and Prasad: Only finitely many arithmetic groups with covolume $\le const$).
For non-arithmetic groups one has to worry about all Galois conjugations yielding noncompact forms of $SO(3)$. For non-arithmetic groups, one has to use a bit more elaborate version of Victor's answer. First, observe that every von Dyck group $\Lambda$ contains a closed surface subgroup $\Gamma$ of finite index. I will consider only the case when the genus is $\ge 2$ since virtually abelian case is much easier. Then, being a closed surface group, $\Gamma$ is isomorphic to a cocompact arithmetic subgroup $\Gamma'$ of $O(2,1)$. Now, $\Gamma'$ admits an embedding $\rho$ to some $U(n)$ as explained by Victor. To embed (unitarily) the original group $\Lambda$, use the representation induced from $\rho: \Gamma\to U(n)$ to $\Lambda$. 


A more interesting question is if every finitely generated subgroup of $SL(n, {\mathbb R})$ embeds in some orthogonal group. It seems that $SL(m, {\mathbb Z})$ is a counter-example for that. 


Update: a. Indeed, $SL(n, {\mathbb Z})$ does not embed in any $U(m)$. The same applies to all finitely generated groups which contain distorted cyclic subgroups, see Yves' comments. The simplest example will be Baumslag-Solitary groups $BS(1,p)=\langle a, b| aba^{-1}=b^p\rangle$, $p>1$. 


b. There is no way to construct sytematically *all* irreducible finite dimensional unitary representations of von Dyck groups since (starting in certain dimension) there will be continuous families of such representations. (The situation is very much unlike theory of Lie groups.) One way to see this is by observing that the spaces of irreducible unitary representations (modulo conjugation) of surface groups have positive dimension. Using induction (of representations) one can then prove the same for von Dyck groups. 


Here are few links where one can read about surface group representations (including unitary ones): 


<http://www.math.u-psud.fr/~labourie/preprints/pdf/surfaces.pdf>


<http://arxiv.org/pdf/math.GT/0509114.pdf>


<http://arxiv.org/abs/0710.5263>


The old paper by Andre Weil "Remarks on Cohomology of Groups" 
<http://www.jstor.org/stable/1970495> is particularly relevant for the discussion of von Dyck groups $\Gamma$, since Weil computes dimension of the Zariski tangent space of $Hom(\Gamma, G)$ in the end of the paper, where $G$ is an arbitrary Lie group. (He does much more, of course.) 


c. One can show that every hyperbolic von Dyck group $D(p,q,r)$, except for $D(2,6,6)$ and $D(2,4,6)$, admits a homomorphism to $PU(2)$ whose image is dense. In particular, for every von Dyck group $\Gamma$ (with the above exceptions), one gets an irreducible representation to $U(n), n\ge 3$. By working more carefully, one can probably prove the same for the two exceptions. 


d. Situation with irreducible representations to $SU(2)$ is more complicated: There will be more exceptions. 


