Game theory with rational and irrational players Are there game theoretic models where one player is rational and the other is irrational (i.e., plays with behavioral limitations)?

The motivation for this question is that behavioral economics has documented how people often make choices that don't maximize expected utility, but those irrational people might be interacting with near-rational computers or organizations. I've never seen a game theory model that has both an expected utility-maximizing player and a behavioral player.
Yes, a whole book has been written on Behavioral Game Theory. 

More specifically, standard solution concept such as Nash equilibrium requires that players best respond to a correct belief about other players' moves. The following are examples that relax one of these cognitive restrictions:


Quantal response equilibrium allows for the possibility that players do not always choose a best response. Instead, they "quantal respond" (or best respond with mistakes) to a correct belief about others' quantal response. 
Level-k or Cognitive Hierarchy Theory relaxes the correctness of beliefs assumption, and allows players to best respond to a set of mutually inconsistent beliefs. 
Psychological games allow players to derive utility directly from beliefs (and belief-dependent emotions). Relevant solution concept include sequential reciprocity equilibrium and Rabin's fairness equilibrium. 


For a complete review of models of behavioral game theory, see Chapter 13 of Dhami (2016).

For a specific example of a game with mixed players, see this paper on robust protection of fisheries. A more comprehensive treatment of mixed games is available in Rani Spiegler's Bounded Rationality and Industrial Organization per Michael Greinecker's comment on the original question.