In a Markov Decision Process (MDP), the discounted total reward is defined as $\sum\_{t=0}^\infty \gamma^tr\_t$ where $r\_t$ is the reward perceived at time $t$ and $\gamma$ is a real number $\in ]0, 1[$. The average total reward is defined as $\lim\_{t\rightarrow \infty}\frac{\sum\_{i=0}^tr\_i}{t}$.


My question is: does a policy $\pi$ that maximizes the discounted total reward also maximize the average total reward and vice versa? Or there is a policy $\pi$ that maximizes the first and $\pi'$ that maximizes the second where $\pi \neq \pi'$?



Discounted total reward vs. Average total reward
Consider first the extreme case where future value is
steeply discounted, meaning that $\gamma$ is very small,
close to $0$. In this case, the discounted total reward
approaches identity with $r\_0$, and the maximizing policy
in that case will approach the policy of maximizing $r\_0$.
This makes sense, since if you don't value future rewards,
then you should try to maximize the present reward. The
average total reward, in contrast, can depend sensitively on future
rewards $r\_n$, and so we shouldn't expect the two policies to coincide in general.


This way of thinking guides one toward a counterexample where the
policies differ: in a Markov decision process where there
is comparatively large but deferred reward---that is, a
choice between a small present reward followed by a large
future payoff and a medium-sized present reward followed by
a small future payoff---the two policies will come to
different maximizing decisions.


