What is the proper term for "contraction economics?" By "contraction economics" I mean theories and work based on the assumption that the population of a country will decline over time.

An example might be a paper looking at the housing market for Japan over the next 50 years (if it assumed that population decline continued over that time).
It doesn't look like there's an established term. Combine "demographic decline", "population decline" or even "fertility decline" (although this doesn't imply population decline) with whatever you are interested in (markets, economic growth etc.)

These papers (if macro) usually get JEL-tagged "J11 Demographic Trends, Macroeconomic Effects, and Forecasts" in combination with whatever is getting modeled, e.g. "Scotland the Grey: A Linked Demographic–Computable General Equilibrium (CGE) Analysis of the Impact of Population Ageing and Decline" is tagged with J11 and a bunch of R1x codes, the latter for regional economics/models.