$N$-player discounted stochastic games with finite state and action spaces possess a Nash equilibrium in stationary strategies. This has been proved by Fink (1964) and a closely related result by Takahashi (1964). Various generalizations have appeared since then.


I have read Fink's proof and understood it (at least I think so). To prove equilibrium, he must compare his proposed strategy to all other possible strategies. However it seems to me that he only considers the set of stationary strategies.


I can see one reason that looking only at stationary strategies is sufficient. Namely, if every other player uses a stationary strategy (this is the candidate Nash equilibrium), then the $n$-th player must solve a Markov Decision problem and it has been proved (e.g., look at Puterman's or Derman's books) that for the MDP nothing is lost by using a stationary strategy.


However, I have looked and have not been able to find some work in which the above or some other argument is used to justify considering only stationary strategies for the $N$-player case.


So my **question** is this: *do you know of any work that addresses the above concerns*? Any help will be greatly appreciated.



Existence of stationary Nash equilibrium of discounted stochastic game
The point that stationary best responses to stationary strategies are best responses without any further restrictions was already made in the very first paper on stochastic games in the context of zero-sum games in



> 
> Shapley, Lloyd S. "[Stochastic games.](https://doi.org/10.1073/pnas.39.10.1095)" *Proceedings of the
> National Academy of Sciences* 39.10 (1953): 1095-1100.
> 
> 
> 


For a more explicit statement of the argument, which is exactly the one you gave, you can look at



> 
> Sobel, Matthew J. "[Noncooperative stochastic games.](https://doi.org/10.1214/aoms/1177693059)" *The Annals
> of Mathematical Statistics* 42.6 (1971): 1930-1935.
> 
> 
> 


