Anyone out hunting for an affordable home today knows that the pickings are slim – and they are about to get slimmer.

Housing inventory hit a record low about two years ago, but a lull in home sales over the past year helped build back much-needed supply, especially in the mid-priced range. Then a sharp drop in rates this summer brought demand back and depleted that supply dramatically.

National housing inventory fell 2.5% annually in September, a sharper decline than August's 1.8% decrease, according to realtor.com

Supply has always been leanest on the low end, as investors have been very active in that price range since the foreclosure crisis.

Roughly 5 million mostly entry-level homes have been turned into single-family rentals, and strong demand for those rentals means investors are unlikely to put the homes up for sale anytime soon.

In addition, an unseasonably strong surge in demand at the end of summer and into this fall now has the supply of homes priced below $200,000 down 10% compared with a year ago.

The demand is being fueled by lower mortgage rates. The average rate on the 30-year fixed surged over 5% last November and stayed above 4.5% through March, according to Mortgage News Daily. That made for a lackluster spring housing market, traditionally the busiest time for buying.

Rates then began falling in May and particularly sharply in July and August. By the start of September the average rate was around 3.5%, and sales of both new and existing homes were surging back. Clearly there was substantial pent-up demand from the spring.

Demand also surged in the move-up market, causing supplies there to fall as well. The supply of homes priced between $200,000 and $750,000, which make up 60% of the market, flatlined in September, after 18 months of strong inventory growth. Supply is now expected to decline in the months ahead.