What are the arguments against the rational expectations hypothesis? I believe the rational expectations hypothesis says that agents in a model bear expectations that are the same as mathematical expectations.

Under what circumstances does this hypothesis become questionable? What are the usual arguments against this hypothesis?
The Rational Expectations Hypothesis (REH) is an hypothesis about aggregate expectations. I believe it is illuminating to post here a lengthy quote (part 2) from Muth (1961) paper where REH originated (bold letters are our emphasis):


  2. THE "RATIONAL EXPECTATIONS" HYPOTHESIS
   Two major conclusions from studies of expectations data are the following:
  1. Averages of expectations in an industry are more accurate than naive models and as accurate as elaborate equation systems, although
  there are considerable cross-sectional differences of opinion.
  2. Reported expectations generally underestimate the extent of changes that actually take place.    
  
  In order to explain these phenomena, I
  should like to suggest that expectations, since they are informed
  predictions of future events, are essentially the same as the
  predictions of the relevant economic theory (We show in Section 5 that
  the hypothesis is consistent with these two phenomena). At the risk of
  confusing this purely descriptive hypothesis with a pronouncement as
  to what firms ought to do, we call such expectations "rational." It is
  sometimes argued that the assumption of rationality in economics leads
  to theories inconsistent with, or inadequate to explain, observed
  phenomena, especially changes over time (e.g., Simon 1959). Our
  hypothesis is based on exactly the opposite point of view: that
  dynamic economic models do not assume enough rationality.  
  
  The hypothesis can be rephrased a little more precisely as follows: that
  expectations of firms (or, more generally, the subjective probability
  distribution of outcomes) tend to be distributed, for the same
  information set, about the prediction of the theory (or the
  "objective" probability distributions of outcomes).  
  
  The hypothesis asserts three things: (1) Information is scarce, and the economic
  system generally does not waste it. (2) The way expectations are
  formed depends specifically on the structure of the relevant system
  describing the economy. (3) A "public prediction," in the sense of
  Grunberg and Modigliani (1954), will have no substantial effect on the
  operation of the economic system (unless it is based on inside
  information). This is not quite the same thing as stating that the
  marginal revenue product of economics is zero, because expectations of
  a single firm may still be subject to greater error than the theory.  
  
  It does not assert that the scratch work of entrepreneurs resembles
  the system of equations in any way; nor does it state that predictions
  of entrepreneurs are perfect or that their expectations are all the
  same. ...


I believe that it should be clear from the above that:
1) REH  is not an assertion about each separate individual, but about the properties of the "prevailing" expectation produced by the black-box combination of individual expectations.  In other words the REH is assumed, without really making any assumptions about individual rationality.  

2) It has as much to do with  the "internal consistency" of the economic model itself, because by construction and without any economic assumptions, $E(X\mid I) = X+ e,\; E(e\mid I) =0 $.  

The fact that the predominant economic model framework has been that of the "representative" (identical) consumer, nevertheless blurred the distinction between the aggregate expectation, and individual expectations on aggregate variables. This provided shallow "micro-foundations" to the REH, (shallow because it is not really micro-founded, that which essentially assumes away the need to aggregate), but also, it moved the debate into the arena of individual expectations formation and whether individuals use information efficiently or not, which raised valid objections as those mentioned in the answer by @EnergyNumbers.  

But really, at the individual level, the hypothesis that individuals use the mathematical expected value comes essentially from Expected Utility theory, that predates the Rational Expectations, and has a debate on its own (also here in Economics.SE)

Another set of "arguments against" the REH (which gave very interesting literature), was collected early on in the book "Individual forecasting and aggregate outcomes - Rational Expectations examined" 1983 R. Frydman and E. Phelps (ed). Of which I mention two:  

1) Being an equilibrium concept, REH requires co-ordination of expectations formation (which is really not that realistic) or properties of Nash-equilibrium: this last insight gave us "Eductive Expectations" and some really thoughtful works by Roger Guesnerie.  

2) The second one, which became rather more widely spread than Eductive Expectations, is "Adaptive Learning" (see "Learning and Expectations in Macroeconomics" By Evans and Honkapohja, 2001).
Adaptive Learning pointed out that REH assumes that economic agents know the structure of their environment perfectly. So in Adaptive Learning models we have the first systematic approach to model uncertainty : as economists, so economic agents do not know the environment perfectly, and they have to estimate it and learn it gradually (hence "adaptive learning"). In this strand of literature, "learning" is done through econometric methods, mainly least-squares (which is a very intuitive least-distance mathematical approximation method). Roughly speaking, here agents' expectations are not the expected values, but the estimated expected values. This creates much more interesting and realistic dynamics, that some times may converge (someday) to an REH equilibrium (which makes Adaptive Learning a "selection mechanism" for the sometimes multiple REH equilibira), or to some other point, not predicted by REH.  

Research into the issue of aggregate expectations formation and modeling is currently exploding, see for example another Frydman &amp; Phelps (ed.) book, "Rethinking Expectations" (2012), in parallel with the emerging "Post-Walrasian" direction in Macroeconomics (see D. Colander (ed). Post-Walrasian Macroeconomics 2006).