Why are there fewer high schools than elementary schools? In nearly every public school district in the U.S., one can observe the following pattern: There are fewer middle schools than elementary schools. And there are fewer high schools than middle schools. (And the count of students per school goes inversely as the count of schools.)

But why is this the case? Is there any explanation for this based on economic principles?

For example, why is the reverse not the case? Why are there not fewer elementary schools than high schools, let's say?
Two relevant economic principles are:
Economies of scale: These are a more significant issue for middle and especially high schools which have a greater need for specialist teachers, as others have pointed out, and also for specialist facilities such as science labs with supporting technicians.  Such teachers and facilities would tend to be under-utilised in small schools, resulting in higher costs of provision per child. Elementary schools, on the other hand, rely more on generalist teachers so can operate efficiently on a smaller scale.
Based on a survey of literature on scale economies in education, Andrews, Duncombe &amp; Yinger (2002) found “some evidence” that the optimal size of elementary schools was 300-500 pupils while that of high schools was 600-900 students.  The reason for identifying an optimal size (rather than concluding that larger is always better) was that large schools can also have disadvantages, such as more bureaucracy leading to lower student and staff motivation and less parental involvement.  Note that these optimal ranges were based on studies focusing on the US: different ranges might be optimal elsewhere given different school curricula and different staff salary levels.
Travel costs: Older children are more likely to make their own way to school, while younger children are more likely to be accompanied by a parent. Hence the costs of travel to school, for a given journey, and including not only fares and/or fuel costs (if any) but also and especially the opportunity cost of a parent's time, tend to be greater for younger children.
More formally, family behaviour can usefully be considered within the framework of a household production model in which households derive utility $U$ from commodities $Z_i$ which they “produce” using inputs consisting of market goods $x_i$ and/or time periods $T_i$.  Becker (1965) writes this as (p 495):
$$U = U(Z_1,…,Z_m) ≡ U(f_1,…,f_m) ≡ U(x_1,…,x_m; T_1,…,T_m)$$
The $f_i$ are the “production” functions, although the term production is used here in a somewhat specialised sense: Becker gives the example of the commodity sleep being “produced” via input of a home, a bed and time. Households aim to maximise utility subject to constraints on both income and time (albeit there may be scope to trade off income against time by adjusting hours spent in paid work).  Within this framework a child’s education can be regarded as a commodity produced using time spent in school, doing homework and travelling to and from school, and perhaps market goods (school fees if applicable, fares or fuel costs if required for travel).  A longer journey to school will imply a greater input of time, with a consequent lesser availability of time for other commodities and therefore less overall utility.  Crucially, this effect will tend to be larger for younger children who are more likely to be accompanied by a parent, whose time should also be counted as an input.  Moreover in some cases a parent’s time on a twice-daily school trip will reduce their hours of availability for paid work, again resulting in lower overall family utility.  Hence families, seeking to maximise their utility, will have a strong incentive to try to ensure (via voting or campaigning) that public schools for younger children are small and numerous so that travel times to school are generally short, and less incentive to do so for schools for older children.