Why did the Fed raise interest rates between 2004 and 2006? Between June 2004 and August 2006, the Fed raised interest rates again and again, because they were "...growing more uncomfortable about inflation".

Looking at the graph below, inflation did increase, but not in any totally insane way (note that it clearly didn't respond to the rate rises!):



Given that high interest rates are bad for mortgage-holders (particularly those with high loan-to-value ratios), you'd think quindrupling rates within 18 months would be bound to lead to a wave of defaults.

Were there accusations at the time that the Fed was blindly following its inflation targeting rule without thinking about the consequences in the rest of the economy?

Or was it about something other than inflation?
Good question! 

It seems that one of the reasons was precisely that inflation did not respond. They kept raising interest rates,  hoping it would respond at some point. In fact, nominal interest rates have to increase more than expected inflation for the real interest rates to go up, so it made sense.

The other reason is also related to the question. It seems that they were aware that a relatively long period of low interest rates was potentially fueling "asset price bubbles",  such as in housing, but also in asset backed securities, corporate bonds and equities. These also did not seem to respond quickly to the rate changes. 

Of course, apparently, they did not realize that it was dangerous moment to increase rates quickly. It would have potentially been a good idea to raise the rates very very slowly, so that leveraged agents had a chance to de-lever as the prices of their assets slowly came down... Alternatively they could have slowly tightened the regulation on lending instead of raising the rates at all. This last point seems to be the key to avoiding a new crisis: having a good too to increase and decrease economy wide leverage distinct from interest rates.