How to verify Value Function in nonzero sum two player Differential Game? There are two agents $i=1,2$. The state $k$ is governed by $\tau_i\in[0,1]$ where 
\begin{align}
\dot{k} = f(k,\tau_1,\tau_2).
\end{align}

Define the value function of player $i$ by
\begin{align}
v_i(k) := \sup_{\tau_i}\int^\infty_0{e^{-\rho t}F(k,\tau_i,\sigma_{-i})dt}.
\end{align}
where $\sigma_{-i}(k)$ is the policy function or a markovian stratgey profile of player $-i$. And $\rho&gt;0$ is the time preference rate.

The Hamilton-Jacobi-Bellman equation reads
\begin{align}
\rho v_i(k) = \sup_{\tau_i}\{F(k,\tau_i,\sigma_{-i}) + v'(k)f(k,\tau_i,\sigma_{-i})\}.
\end{align}

Now assume I approximated a function $\hat{v}_i$ by value function iteration. How do I know that it's the true value function?
The first panel shows the final value function after iterating 80k (!). Should probably try an implicit method. The second panel shows the evolution from frist guess to final result $v^0\to v^j$, $j=80k$. Third panel is the distance between the value functions and fourth shows the error. 

Note that the value function is convex-concave. This is due to the fact that for the limit case the law of motion tends to infinty $\lim_{k\to 0} \dot{k} = \infty$. 

By the maximum principle optimality requires that the Hamiltonian 
\begin{align}
H_1(k,\tau_1,\tau_2^*,\lambda_1) := F(k,\tau_1,\tau_2^*) + \lambda_1 f(k,\tau_1,\tau_2^*)
\end{align}
is concave in $(k,\tau_1,\tau_2)$ (sufficient condition). Now we have
\begin{align}
\rho v(k) = \max_{\tau_1} H_1(k,\tau_1,\tau_2^*,\lambda_1).
\end{align} 


I was wondering if the value function must be (strictly) concave to ensure optimality?


Any reference is appreciated.