code: def value_iteration(mdp, epsilon=0.001):
      "Solving an MDP by value iteration. [Fig. 17.4]"
      U1 = dict([(s, 0) for s in mdp.states])
      R, T, gamma = mdp.R, mdp.T, mdp.gamma
      while True:
          U = U1.copy()
          delta = 0
          for s in mdp.states:
              U1[s] = R(s) + gamma * max([sum([p * U[s1] for (p, s1) in T(s, a)])
                                          for a in mdp.actions(s)])
              delta = max(delta, abs(U1[s] - U[s]))
          if delta < epsilon * (1 - gamma) / gamma:
               return f value_iteration(mdp, epsilon=0.001):
      "Solving an MDP by value iteration. [Fig. 17.4]"
      U1 = dict([(s, 0) for s in mdp.states])
      R, T, gamma = mdp.R, mdp.T, mdp.gamma
      while True:
          U = U1.copy()
          delta = 0
          for s in mdp.states:
              U1[s] = R(s) + gamma * max([sum([p * U[s1] for (p, s1) in T(s, a)])
                                          for a in mdp.actions(s)])
              delta = max(delta, abs(U1[s] - U[s]))
          if delta < epsilon * (1 - gamma) / gamma:
               return U
docs: Solving an MDP by value iteration. [Fig. 17.4]