Update of value function in continuous time - HJB When solving (numerically, by value function iteration) a dynamic programming problem in discrete time, such as

$$V_1(a) = \max_{c} \ u(c) + \dfrac{1}{1+\rho}V_0(a')$$

we maximize with respect to the control variable and get a first order condition that we then plug back into the functional equation shown above. The result of this step, $V(a)_1$, will then be used on the RHS of a second iteration

$$V_2(a) = \max_{c} \ u(c) + \dfrac{1}{1+\rho}V_1(a')$$

and we repeat this process until $V(a)_n-V(a)_{n+1}&lt;\epsilon$.

My question is how does the update of the value function work in continuous time?  I have been working on a paper that uses continuous time dynamic programming, so the Bellman equation looks as follows

$$\rho V_n(a) = \max_{c} \ u(c) + \dfrac{\partial V_n(a)}{\partial a}da_t \quad (*)$$

where the transition equation is represented by $da_t$. From what I have seen, the update of the value function is done by calculating $\Delta$:

$$ \Delta = \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

where $u(c(a^*))$ and $da_t(a^*)$ represent the control and transition equation as functions of the optimal policy. That is, we maximize the RHS as in the previous example (the discrete time case), but then we subtract $\rho V(a)$ from both sides. Then updating the value function is done as follows:

$$V_{n+1}(a) = V_n(a) + \Delta$$

How can this be so? I would have thought that I would just use the maximised RHS of (*) and plug back in a new iteration. How come the other method is the correct one?
You iterate towards a fixed point, so you want to reach a situation where plugging in your current iterated value produces itself. Now using your notation, we are told that we should calculate

$$V_{n+1}(a) = V_n(a) + \Delta$$

where

$$\Delta = \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

Insert the second into the first to see what the iteration rule is:

$$V_{n+1}(a) = V_n(a) + \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

When you reach a point where

$$V_{n+1}(a) = V_n(a) $$

(or $\epsilon$-so)

It will mean

$$\rho V_n(a)= \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) $$

which is what you have to satisfy.

Some stars etc may have to be adjusted in the above, for a fully consistent notation.