The cohomology of Shimura varieties and Drinfeld shtukas is conjectured to realize the representations sought for in the Langlands programme/conjectures, the cohomology of Deligne-Lusztig varieties realizes representations of the classical groups over finite fields: How did people find those varieties?



how to find the varieties whose cohomology realizes certain representations?
Regarding Shimura varieties:


One has to first consider the case of modular curves, which has served throughout as an impetus and inspiration for the general theory.


The study of modular curves (in various guises) goes back to the 19th century, with the work
of Jacobi and others on *modular equations* (which from a modern viewpoint are explicit equations for the modular curves $X\_0(N)$). The fact that these curves are defined over $\mathbb Q$ (or even $\mathbb Z$) also goes back (in some form) to the 19th century, in so far as it was noticed that modular equations have rational or integral coefficients. There is also the (strongly related) fact that interesting modular functions/forms have rational or integral $q$-expansion coefficients. Finally, there are the facts related to Kronecker's Jugendtraum, that modular functions/forms with integral Fourier coefficients, when evaluated at quadratic imaginary points in the upper half-plane, give algebraic numbers lying in abelian extensions of quadratic imaginary fields. These all go back to the 19th century in various forms, although complete theories/interpretations/explanations weren't known until well into the 20th century.


The idea that the cohomology of modular curves would be Galois theoretically interesting is more recent. I think that it goes back to Eichler, with Igusa, Ihara, Shimura, Serre, and then Deligne all playing important roles. It seems to be non-trivial to trace the history, in part because the intuitive idea seems to predate the formal introduction of etale cohomology (which is necessary to make the idea completely precise and general). Thus Ihara's work considers zeta-functions of modular curves (or of the Kuga--Satake varieties over them) rather than cohomology. (The zeta-function is a way of incarnating the information carried in cohomology without talking directly about cohomology). Shimura worked just with weight two modular forms (related to cohomology with constant coefficients), and instead of talking directly about etale cohomology worked with the Jacobians of the modular curves. (He explained how the Hecke operators break up the
Jacobian into a product of abelian varieties attached to Hecke eigenforms.) [Added: In fact,
I should add that Shimura also had an argument, via congruences, which reduced the
study of cohomology attached to higher weight forms to the case of weight two forms; this was elaborated on by Ohta. These kinds of arguments were then rediscovered and further developed by Hida, and have since been used by lots of people to relate modular forms of
different weights to one another.]


The basic idea, which must have been understood in some form by all these people, is
that a given Hecke eigenform $f$ contributes two dimensions to cohomology, represented by the two differential forms $f d\tau$ and $\overline{f}d\tau$. Thus Hecke eigenspaces in 
cohomology of modular curves are two-dimensional. Since the Hecke operators are defined over $\mathbb Q$, these eigenspaces are preserved by the Galois action on etale cohomology, and so we get two-dimensional Galois reps. attached to modular forms.


As far as I understand, Shimura's introduction of general Shimura varieties grew out of 
thinking about the theory of modular curves, and in particular, the way in which that
theory interacted with the theory of complex multiplication elliptic curves. In particular,
he and Taniyama developed the general theory of CM abelian varieties, and it was natural to try to embed that more general theory into a theory of moduli spaces generalizing the modular curves. A particular challenge was to try to give a sense to the idea that the
resulting varieties (i.e. Shimura varieties in modern terminology) had canonical models over number fields. This could no longer be done by studying rationality of $q$-expansions (since they could be compact, say, and hence have no cusps around which to form Fourier
expansions). Shimura introduced the Shimura reciprocity law, i.e. the description of the
Galois action on the special points (the points corresponding to CM abelian varieties) as the basic tool for characterizing and studying rationality questions for Shimura varieties.


In particular, Shimura varieties were introduced prior to the development of the Langlands programme, and for reasons other than the construction of Galois representations. However,
once one had these varieties, naturally defined over number fields, and having their origins in the theory of algebraic groups and automorphic forms, it was natural to try to calculate their zeta-functions, or more generally, to calculate the Galois action on their cohomology, and Langlands turned to this problem in the early 1970s. (Incidentally, my understanding is that it was he who introduced the terminology *Shimura varieties*.) The first question he tried to answer was: how many dimensions does a given Hecke eigenspace contribute to the cohomology.
He realized that the answer to this --- at least typically --- was given by Harish-Chandra's theory of (what are now called) discrete series $L$-packets, as is explained in his 
[letters to Lang](http://publications.ias.edu/rpl/section/26); the relationship of the resulting Galois representations to the Langlands program is not obvious --- in particular, it is not obvious how the dual group intervenes --- and this (namely, the intervention of the dual group) is the main topic of the letters to Lang. These letters to Lang are just the beginning of the story, of course. (For example, the typical situation does not always occur; there is the phenomenon of endoscopy. And then there is the problem of actually proving that the Galois action on cohomology gives what one expects it to!)


Regarding Drinfeld and Deligne--Lusztig varieties: 


I've studied these cases in much less detail, but I think 
that Drinfeld was inspired by the case of Shimura varieties, and (as Jim Humphreys has noted) Deligne--Lusztig drew insipration from Drinfeld. 


What can one conclude:


These theoretically intricate objects grew out of a long and involved history, with multiple motivations driving their creation and the investigations of their properties.


If you want to find a unifying (not necessarily historical) theme,
one could also note that Deligne--Lusztig varieties are built out of flag varieties in a certain sense, in fact as locally closed regions of flag varieties, and that Shimura varieties are also built out of (in the sense that they are
quotients of) symmetric spaces, which are again open regions in (partial) flag varieties.
This suggests a well-known conclusion, namely that the geometry of reductive groups and the various spaces associated to them seems to be very rich.


