How does the federal funds rate compare to the federal discount rate? The federal funds rate is the interest rate charged to banks when they borrow from each other overnight in the federal funds market to satisfy their reserve requirements. This rate is influenced by the Fed through open market operations. The federal discount rate is the interest rate charged to banks when they borrow from the Fed to satisfy their reserve requirements. This rate depends on how much money banks are in need of and how much the Fed is willing to provide. 

It seems to me that both of these rates should be almost equal most of the time. If the discount rate is lower than the federal funds rate, then the depository institutions would borrow from the Fed, instead of from each other. Thus, this additional demand for the central bank's reserves drives up the discount rate such that it equals the federal funds rate. The opposite would occur if the federal funds rate is lower than the discount rate. Is this correct?
You are right the two series closely follow each other for the reasons you mention. During quite some time the discount rate actually used to be a ceiling for a funds rate. This is precisely, because if the federal funds rate was below
the discount rate, most banks adjusted their reserve positions in the federal funds market and when the federal funds rate rose to the level of the discount rate it was cheaper to borrow at discount rate so banks simply do that. 

However, they are not always as close as you would expect, especially the discount rate does not anymore acts as a ceiling. There are several reasons for that in literature, primarily because borrowing at both of those rates has a different strings attached to it. 

For example, Meulendyke (1998) argues that:


  Since the mid-1960s, the basic discount rate frequently has
  been below the prevailing Federal funds rate. The Federal Reserve
  has relied on administrative procedures to limit access to the window by restricting the frequency and amounts of borrowing.
   ... The Federal Reserve’s administrative
  restrictions used to be the primary factor that discouraged borrowing, but in the last decade banks themselves have been responsible
  for much of the limitation. Heavy borrowing in the 1980s by a few
  banks with financial difficulties caused others to avoid the window
  for fear depositors might conclude that they were also in trouble.
  Reluctance to borrow contributes to a seemingly contradictory
  result—namely, that increases in the amount of reserves in the banking system, when provided through the discount window, make
  reserve availability more restrictive on the margin because such
  increases put banks under pressure to find other sources of reserves
  to repay the loans.


and she further argues that:


  Until the mid-1960s, the Federal funds rate did
  not trade above the discount rate. During “tight money periods,” when the
  Trading Desk was fostering significant net borrowed reserve positions, funds
  generally traded at the discount rate, and the funds rate was not considered a
  useful indicator of money market conditions. When free reserves were high,
  funds often traded below the discount rate and showed some day-to-day
  variation. At such times, the funds rate received greater attention as an
  indicator of reserve availability.
  There was considerable surprise when the funds rate first rose above
  the discount rate, briefly in October 1964 and more persistently in 1965. As
  large banks became more active managers of the liability side of their balance sheets, they borrowed funds in the market in a sustained way. Banks
  had introduced large negotiable certificates of deposit (CDs) in 1961. But CD
  borrowings were (until 1991) subject to reserve requirements and (until
  1970) to interest rate ceilings under Regulation Q. Borrowings from other
  banks through the Federal funds market were free of reserve requirements
  and interest rate ceilings. Furthermore, they were not subject to the restrictions on prolonged use that were applied to the Federal Reserve’s discount
  window. The changes in liability management techniques meant that
  individual banks could expand credit even when they did not have free
  reserves if they were willing to bid aggressively for wholesale funding from
  other banks. 


These and possibly also some other reasons will lead the two to diverge in some situations as borrowing at those rate might be subject to different conditions and have different implications. You can see on the graph below which plots of the federal funds rate in red and the discount rate in blue (based on FRED data) that they can sometimes differ quite substantially but indeed tend to follow each other closely.