\section{Introduction}

In the process of obtaining a sufficiently general version of the Fundamental Theorem of Asset Pricing (FTAP), semimartingales proved crucial in modelling discounted asset-price processes. The powerful tool of stochastic integration with respect to general predictable integrands, that semimartingales are exactly tailored for, finally lead to the culmination of the theory in \cite{MR1304434}. The FTAP connects the economical notion of \textsl{No Free Lunch with Vanishing Risk} (NFLVR) with the mathematical concept of existence of an \textsl{Equivalent Martingale Measure} (EMM), i.e., an auxiliary probability, equivalent to the original (in the sense that they have the same impossibility events), that makes the discounted asset-price processes have some kind of martingale property. For the above approach to work one has to utilize stochastic integration using \emph{general} predictable integrands, which translates to allowing for \emph{continuous-time} trading in the market. Even though continuous-time trading is of vast theoretical importance, in practice it is only an ideal approximation; the only feasible way of trading is via \emph{simple}, i.e., combinations of \emph{buy-and-hold}, strategies.

Recently, it has been argued that existence of an EMM is \emph{not} necessary for viability of the market; to this effect, see \cite{MR1774056}, \cite{MR1929597}, \cite{MR2210925}. Even in cases where classical arbitrage opportunities are present in the market, credit constraints will not allow for arbitrages to be scaled to any desired degree. It is rather the existence of a \textsl{strictly positive supermartingale deflator}, a concept weaker than existence of an EMM, that allows for a consistent theory to be developed.

\smallskip

Our purpose in this work is to provide answers to the following questions:

\begin{enumerate}
  \item Why are semimartingales important in modeling discounted asset-price processes?
  \item Is there an analogous result to the FTAP that involves weaker (both economic and mathematical) conditions, and  only assumes the possibility of simple trading?
\end{enumerate}

A partial, but precise, answer to question (1) is already present in \cite{MR1304434}. Roughly speaking, market viability already imposes the semimartingale property on discounted asset-price processes. In this paper, we elaborate on the previous idea, undertaking a different approach, which ultimately leads to an improved result. We also note that in \cite{MR2139030}, \cite{MR2157199} and \cite{LarZit05}, the semimartingale property of discounted asset-price processes is obtained via the finite value of a utility maximization problem; this approach will also be revisited.

All the conditions that have appeared previously in the literature are only \emph{sufficient} to ensure that discounted asset-price processes are semimartingales. Here, we shall also discuss a necessary and sufficient condition in terms of a natural market-viability notion that parallels the FTAP, under minimal initial structural assumptions on the discounted asset-price processes themselves. The weakened version of the FTAP that we shall come up with as an answer to question (2) above is a ``simple, no-short-sales trading'' version of Theorem 4.12 from \cite{MR2335830}.

\smallskip

The structure of the paper is as follows. In Section \ref{sec: UPBR, supdefl, semimarts, FTAP}, we introduce the market model, simple trading under no-short-sales constraints. Then, we discuss the market viability condition of \textsl{absence of arbitrages of the first kind} (a weakening of condition NFLVR), as well as the concept of \textsl{strictly positive supermartingale deflators}. After this, our main result, Theorem \ref{thm: FTAP_jump}, is formulated and proved, which establishes both the importance of semimartingales in financial modelling, as well as the weak version of the FTAP. Section \ref{sec: beyond main result} deals with remarks on, and ramifications of, Theorem \ref{thm: FTAP_jump}.
We note that, though hidden in the background, the proofs of our results depend heavily on the notion of the \emph{num\'eraire \ portfolio} (also called \emph{growth-optimal}, \emph{log-optimal} or \emph{benchmark portfolio}), as it appears in a series of works: \cite{KELLY}, \cite{LONG}, \cite{MR1849424}, \cite{MR1970286}, \cite{MR1929597}, \cite{MR2194899}, \cite{MR2335830}, \cite{MR2284490}, to mention a few.

\section{The Semimartingale Property of Discounted Asset-Price Process and a Version of the Fundamental Theorem of Asset Pricing} \label{sec: UPBR, supdefl, semimarts, FTAP}

\subsection{The financial market model and trading via simple, no-short-sales strategies}

The random movement of $d \in \mathbb N$ risky assets in the market is modelled via c\`adl\`ag,
nonnegative stochastic processes $S^i$, where $i \in \set{ 1, \ldots, d}$.
As is usual in the field of Mathematical Finance, we assume that all wealth processes are discounted by another special asset which is considered a ``baseline''. The above process $S = (S^i)_{i = 1, \ldots, d}$ is defined on a filtered probability space $(\Omega, \, \mathcal{F}, \, \pare{\mathcal{F}_t}_{t \in \Real_+}, \, \prob)$, where $\pare{\mathcal{F}_t}_{t \in \Real_+}$ is a filtration satisfying $\mathcal{F}_t \subseteq \mathcal{F}$ for all $t \in \mathbb R_+$, as well as the usual assumptions of right-continuity and saturation by all $\mathbb{P}$-null sets of $\mathcal{F}$.

Observe that there is no \emph{a priori} assumption on $S$ being a semimartingale. This property will come as a consequence of a natural market viability assumption.

\smallskip

In the market described above, economic agents can trade in order to reallocate their wealth. Consider a \textsl{simple} predictable process $\theta := \sum_{j=1}^n \vartheta_{j} \mathbb{I}_{\dbraoc{\tau_{j-1}, \tau_j}}$. Here, $\tau_0 = 0$, and for all $j \in \set{ 1, \ldots, n}$ (where $n$ ranges in $\mathbb N$), $\tau_j$ is a \emph{finite} stopping time and $\vartheta_{j} = (\vartheta^i_j)_{i=1, \ldots, d}$ is $\mathcal{F}_{\tau_{j-1}}$-measurable. Each $\tau_{j-1}$, $j \in \set{ 1, \ldots, n}$, is an instance when some given economic agent may trade in the market; then, $\vartheta^i_j$ is the number of units from the $i$th risky asset that the agent will hold in the trading interval $]\tau_{j-1}, \tau_{j}]$. This form of trading is called \textsl{simple}, as it comprises of a finite number of \textsl{buy-and-hold} strategies, in contrast to \textsl{continuous} trading where one is able to change the position in the assets in a continuous fashion. This last form of trading is only of theoretical value, since it cannot be implemented in reality, even if one ignores market frictions.
Starting from initial capital $x \in \mathbb R_+$ and following the strategy described by the simple predictable process $\theta := \sum_{j=1}^n \vartheta_j \mathbb{I}_{\dbraoc{\tau_{j-1}, \tau_j}}$, the agent's discounted wealth process is given by
\begin{equation} \label{eq: Xhat}
X^{x, \theta} \ = x + \int_0^\cdot \inner{\theta_t}{\, \mathrm d S_t} \ := \ x + \sum_{j = 1}^n \inner{\vartheta_j}{ S_{\tau_j \wedge \cdot} - S_{\tau_{j-1}  \wedge \cdot}}.
\end{equation}
(We use ``$\inner{\cdot}{\cdot}$'' throughout to denote the  usual Euclidean inner product on $\mathbb R^d$.)

The wealth process $X^{x, \theta}$ of \eqref{eq: Xhat} is c\`adl\`ag \ and adapted, but could in principle become negative. In real markets, some economic agents, for instance pension funds, face several institution-based constraints when trading. The most important constraint is prevention of having negative positions in the assets; we plainly call this  \emph{no-short-sales} constraints. In order to ensure that no short sales are allowed in the risky assets, which also include the baseline asset used for discounting, we define $\mathcal{X}_{\mathsf{s}} (x)$ to be the set of all wealth processes $X^{x, \theta}$ given by \eqref{eq: Xhat}, where $\theta = \sum_{j=1}^n \vartheta_j \mathbb{I}_{\dbraoc{\tau_{j-1}, \tau_j}}$ is simple and predictable and such that $\vartheta^i_j \geq 0$ and $\inner{ \vartheta_j}{S_{\tau_{j-1}}} \leq X^{x, \theta}_{\tau_{j-1}}$ hold for all  $i \in \set{1, \ldots, d}$ and $j \in \set{ 1, \ldots, n}$. (The subscript ``$\mathsf{s}$'' in $\mathcal{X}_{\mathsf{s}}(x)$ is a mnemonic for ``simple''; the same is true for all subsequent definitions where this subscript appears.) Note that the previous no-short-sales restrictions, coupled with the nonnegativity of $S^i$, $i \in \set{1, \ldots, d}$, imply the stronger $\theta^i \geq 0$ for all $i = 1, \ldots, d$ and $\inner{ \theta}{ S_-} \leq X^{x, \theta}_-$. (The subscript ``${}_-$'' is used to denote the left-continuous version of a c\`adl\`ag \ process.) It is clear that $\mathcal{X}_{\mathsf{s}}(x)$ is a convex set for all $x \in \mathbb R_+$. Observe also that $\mathcal{X}_{\mathsf{s}} (x) = x \mathcal{X}_{\mathsf{s}} (1)$ for all $x \in \mathbb R_+ \setminus \set{0}$. Finally, define $\mathcal{X}_{\mathsf{s}} := \bigcup_{x \in \mathbb R_+} \mathcal{X}_{\mathsf{s}}(x)$.

\subsection{Market viability} \label{subsec: UPBR}

We now aim at defining the essential ``no-free-lunch'' concept to be used in our discussion. For $T \in \mathbb R_+$, an $\mathcal{F}_T$-measurable random variable $\xi$ will be called an \textsl{arbitrage of the first kind on $[0, T]$} if $\mathbb{P}[\xi \geq 0] = 1$, $\mathbb{P}[\xi > 0] > 0$, and \emph{for all $x > 0$ there exists $X \in \mathcal{X}_{\mathsf{s}}(x)$, which may depend on $x$, such that $\mathbb{P}[X_T \geq \xi] = 1$}. If, in a market where only simple, no-short-sales trading is allowed, there are \emph{no} arbitrages of the first kind on \emph{any} interval $[0, T]$, $T \in \mathbb R_+$ we shall say that condition NA$1_{\mathsf{s}}$ \ holds. It is straightforward to check that condition NA$1_{\mathsf{s}}$ \ is weaker than condition NFLVR (appropriately stated for simple, no-short-sales trading). The next result describes an equivalent reformulation of condition NA$1_{\mathsf{s}}$ \ in terms of boundedness in probability of the set of outcomes of wealth processes, which is essentially condition ``\textsl{No Unbounded Profit with Bounded Risk}'' of \cite{MR2335830} for all finite time-horizons in our setting of simple, no-short-sales trading.

\begin{prop} \label{prop: NAone iff NUPBR}
Condition \emph{NA$1_{\mathsf{s}}$} holds if and only if, for all $T \in \mathbb R_+$, the set $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is bounded in probability, i.e., $\downarrow \lim_{\ell \to \infty}  \sup_{X \in \mathcal{X}_{\mathsf{s}} (1)} \, \mathbb{P} [X_T > \ell] = 0$ holds for all $T \in \mathbb R_+$.
\end{prop}

\begin{proof}
Using the fact that $\mathcal{X}_{\mathsf{s}}(x) = x \mathcal{X}_{\mathsf{s}}(1)$ for all $x > 0$, it is straightforward to check that if an arbitrage of the first kind exists on $[0, T]$ for some $T \in \mathbb R_+$ then $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is not bounded in probability. Conversely, assume the existence of $T \in \mathbb R_+$ such that $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is not bounded in probability. As $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is further convex, Lemma 2.3 of \cite{MR1768009} implies the existence of $\Omega_u \in \mathcal{F}_T$ with $\mathbb{P}[\Omega_u] > 0$ such that, for all $n \in \mathbb N$, there exists $\widetilde{X}^n \in \mathcal{X}_{\mathsf{s}}(1)$ with $\mathbb{P}[\{\widetilde{X}^n_T \leq  n \} \cap \Omega_u] \leq \mathbb{P}[\Omega_u] / 2^{n+1}$. For all $n \in \mathbb N$, let $A^n = \mathbb{I}_{\{\widetilde{X}^n_T > n\}} \cap \Omega_u \in \mathcal{F}_T$. Then, set $A := \bigcap_{n \in \mathbb N} A^n \in \mathcal{F}_T$ and $\xi :=  \mathbb{I}_A$. It is clear that $\xi$ is $\mathcal{F}_T$-measurable and that $\mathbb{P}[\xi \geq 0] = 1$. Furthermore, since $A \subseteq \Omega_u$ and
\[
\mathbb{P} \bra{\Omega_u \setminus A} = \mathbb{P} \bra{\bigcup_{n \in \mathbb N} \pare{\Omega_u \setminus A^n}} \leq \sum_{n \in \mathbb N} \mathbb{P} \bra{ \Omega_u \setminus A^n} = \sum_{n \in \mathbb N} \mathbb{P} \bra{ \{ \widetilde{X}^n_T \leq  n \} \cap \Omega_u} \leq \sum_{n \in \mathbb N} \frac{\mathbb{P}[\Omega_u]}{2^{n+1}} = \frac{\mathbb{P}[\Omega_u]}{2},
\]
we obtain $\mathbb{P}[A] > 0$, i.e., $\mathbb{P}[\xi > 0] > 0$. For all $n \in \mathbb N$ set $X^n := (1 / n) \widetilde{X}^n$, and observe that $X^n \in \mathcal{X}_{\mathsf{s}}(1 / n)$ and $\xi = \mathbb{I}_{A} \leq \mathbb{I}_{A^n} \leq X^n_T$ hold for all $n \in \mathbb N$. It follows that $\xi$ is and arbitrage of the first kind on $[0, T]$, which finishes the proof.
\end{proof}

\begin{rem} \label{rem: NAone for stop times}
The constant wealth process $X \equiv 1$ belongs to $\mathcal{X}_{\mathsf{s}}(1)$. Then, Proposition \ref{prop: NAone iff NUPBR} implies that  condition NA$1_{\mathsf{s}}$ \ is also equivalent to the requirement that the set $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is bounded in probability for all finite stopping times $T$.
\end{rem}

\subsection{Strictly positive supermartingale deflators} \label{subsec: supermart_defl}

Define the set $\mathcal{Y}_{\mathsf{s}}$ of \textsl{strictly positive supermartingale deflators for simple, no-short-sales trading}  to consist of all c\`adl\`ag \ processes $Y$ such that $\mathbb{P}[Y_0 = 1, \text{ and } Y_t > 0 \ \, \forall t \in \mathbb R_+] = 1$, and $Y X$ is a supermartingale for all $X \in \mathcal{X}_{\mathsf{s}}$. Note that existence of a strictly positive supermartingale deflator is a condition closely related, but strictly weaker, to existence of equivalent (super)martingale probability measures.

\subsection{The main result} \label{subsec: main result}

Condition NA$1_{\mathsf{s}}$, existence of strictly positive supermartingale deflators and the semimartingale property of $S$ are immensely tied to each other, as will be revealed below.

\smallskip

Define the (first) \textsl{bankruptcy time} of $X \in \mathcal{X}_{\mathsf{s}}$ to be $\zeta^X := \inf \{ t \in \mathbb R_+ \ | \  X_{t-} = 0 \text{ or } X_t = 0\}$. We shall say that $X \in \mathcal{X}_{\mathsf{s}}$ \textsl{cannot revive from bankruptcy} if $X_t = 0$ holds for all $t \geq \zeta^X$ on $\{ \zeta^X < \infty\}$. As $S^i \in \mathcal{X}_{\mathsf{s}}$ for $i \in \set{ 1, \ldots, d}$, the previous definitions apply in particular to each $S^i$, $i \in \set{ 1, \ldots, d}$.

Before stating our main Theorem \ref{thm: FTAP_jump}, recall that $S^i$, $i \in \set{1, \ldots, d}$, is an \textsl{exponential semimartingale} if there exists a semimartingale $R^i$ with $R^i_0 = 0$, such that $S^i = S^i_0 \mathcal E(R^i)$  where ``$\mathcal E$'' denotes the \textsl{stochastic exponential} operator.

\begin{thm} \label{thm: FTAP_jump}
Let $S = (S^i)_{i=1, \ldots, d}$ be an adapted, c\`adl\`ag \ stochastic process such that $S^i$ is nonnegative for all $i \in \set{1, \ldots, d}$. Consider the following four statements:
\begin{enumerate}
  \item[$(i)$] Condition \emph{NA1}$_\mathsf{s}$ holds in the market.
  \item[$(ii)$] $\mathcal{Y}_{\mathsf{s}} \neq \emptyset$.
  \item[$(iii)$] $S$ is a semimartingale, and $S^i$ cannot revive from bankruptcy for all $i \in \set{1, \ldots, d}$.
  \item[$(iv)$] For all $i \in \set{1, \ldots, d}$, $S^i$ is an exponential semimartingale.
\end{enumerate}
Then, we have the following:
\begin{enumerate}
  \item[(1)] It holds that $(i) \Leftrightarrow (ii) \Rightarrow (iii)$, as well as $(iv) \Rightarrow (i)$.
  \item[(2)] Assume further that $S^i_{\zeta^{S^i} -} > 0$ holds on $\{ \zeta^{S^i} < \infty\}$ for all $i \in \set{1, \ldots, d}$. Then, we have the equivalences $(i) \Leftrightarrow (ii) \Leftrightarrow (iii) \Leftrightarrow (iv)$.
\end{enumerate}

\end{thm}

\subsection{Proof of Theorem \ref{thm: FTAP_jump}, statement $(1)$}

\begin{proof}[$(i) \Rightarrow (ii)$]

Define the set of dyadic rational numbers $\mathbb{D} := \{ m / 2^k \ | \  k \in \mathbb N, \, m \in \mathbb N \}$, which is dense in $\mathbb R_+$. Further, for $k \in \mathbb N$, define the set of trading times $\mathbb{T}^k := \{ m / 2^k \ | \  m \in \mathbb N, \, 0 \leq m \leq k 2^k \}$. Then, $\mathbb{T}^k \subset \mathbb{T}^{k'}$ for $k < k'$ and $\bigcup_{k \in \mathbb N} \mathbb{T}^k = \mathbb{D}$. In what follows, $\mathcal{X}_{\mathsf{s}}^k (1)$ denotes the subset of $\mathcal{X}_{\mathsf{s}}(1)$ consisting of wealth processes where trading only may happen at times in $\mathbb{T}^k$. We now state and prove an intermediate result that will help to establish implication $(i) \Rightarrow (ii)$ of Theorem \ref{thm: FTAP_jump}.

\begin{lem} \label{lem: num exists for discrete-time}
Under condition \emph{NA$1_{\mathsf{s}}$}, and for each $k \in \mathbb N$, there exists a wealth process $\widetilde{X}^k \in \mathcal{X}_{\mathsf{s}}^k(1)$ with $\mathbb{P}[\widetilde{X}^k_t > 0] = 1$ for all $t \in \mathbb{T}^k$ such that, by defining $\widetilde{Y}^k := 1 / \widetilde{X}^k$, $\mathbb{E}[\widetilde{Y}^k_t X_t  \ | \  \mathcal{F}_s] \leq \widetilde{Y}^k_s X_s$ holds for all $X \in \mathcal{X}_{\mathsf{s}}^k(1)$, where $\mathbb{T}^k \ni s \leq t \in \mathbb{T}^k$.
\end{lem}

\begin{proof}
The existence of such ``num\'eraire \ portfolio'' $\widetilde{X}^k$ essentially follows from Theorem 4.12 of \cite{MR2335830}. However, we shall explain in detail below how one can \emph{obtain} the validity of Lemma \ref{lem: num exists for discrete-time} following the idea used to prove Theorem 4.12 of \cite{MR2335830} in this simpler setting, rather than using the latter heavy result. Throughout the proof we keep $k \in \mathbb N$ fixed, and we set $\mathbb{T}^k_{++} := \mathbb{T}^k \setminus \set{0}$.

First of all, it is straightforward to check that condition  NA$1_{\mathsf{s}}$ \ implies that each $X \in \mathcal{X}_{\mathsf{s}}$, and in particular also each $S^i$, $i \in \set{1, \ldots, d}$, cannot revive from bankruptcy. This implies that we can consider an alternative ``multiplicative'' characterization of wealth processes in $\mathcal{X}_{\mathsf{s}}(1)$, as we now describe. Consider a process $\pi = (\pi_t)_{t \in \mathbb{T}^k_{++}}$ such that,  for all $t \in \mathbb{T}^k_{++}$, $\pi_t \equiv (\pi^i_t)_{i \in \set{1, \ldots, d}}$ is $\mathcal{F}_{t - 1/2^k}$-measurable and takes values in the $d$-dimensional simplex $\triangle^d := \big\{ z = (z^i)_{i =1, \ldots, d} \in \mathbb R^d \ | \  z^i \geq 0 \text{ for } i=1, \ldots, d, \text{ and } \sum_{i=1}^d z^i \leq 1 \big\}$. Define $X^{(\pi)}_0 := 1$ and, for all $t \in \mathbb{T}^k_{++}$, $X^{(\pi)}_t := \prod_{\mathbb{T}^k_{++} \ni u \leq t} \pare{1 + \inner{\pi_u}{\Delta R^{k}_u}}$, where, for $u \in \mathbb{T}^k_{++}$, $\Delta R^k_u = (\Delta R^{k,i}_u)_{i \in \set{1, \ldots, d}}$ is such that $\Delta R^{k, i}_u = \big( S^i_u / S^i_{u - 1/2^k} - 1 \big) \mathbb{I}_{\{ S^i_{u - 1/2^k} > 0 \}}$ for $i \in \set{1, \ldots, d}$. Then, define a simple predictable $d$-dimensional process $\theta$ as follows: for $i \in \set{1, \ldots, d}$ and $u \in ]t - 1/2^k, t]$, where $t \in \mathbb{T}^k_{++}$, set $\theta^i_u = \big( \pi^i_t X^{(\pi)}_{t - 1/2^k} / S^i_{t - 1/2^k} \big) \mathbb{I}_{\{S^i_{t - 1/2^k}  > 0\}}$; otherwise, set $\theta = 0$. It is then straightforward to check that $X^{1, \theta}$, in the notation of \eqref{eq: Xhat}, is an element of $\mathcal{X}_{\mathsf{s}}^k(1)$, as well as that $X^{1, \theta}_t = X^{(\pi)}_t$ holds for all $t \in \mathbb{T}^k$. We have then established that $\pi$ generates a wealth process in $\mathcal{X}_{\mathsf{s}}^k(1)$. We claim that every wealth process of $\mathcal{X}_{\mathsf{s}}^k(1)$ can be generated this way. Indeed, starting with any predictable $d$-dimensional process $\theta$ such that $X^{1, \theta}$, in the notation of \eqref{eq: Xhat}, is an element of $\mathcal{X}_{\mathsf{s}}^k(1)$, we define $\pi^i_t = \big( \theta^i_t S^i_{t - 1/2^k} /X^{1, \theta}_{t - 1/2^k} \big) \mathbb{I}_{\{X^{1, \theta}_{t - 1/2^k}  > 0\}}$ for $i \in \set{1, \ldots, d}$ and $t \in \mathbb{T}^k_{++}$. Then, $\pi = (\pi_t)_{t \in \mathbb{T}^k_{++}}$ is $\triangle^d$-valued, $\pi_t \equiv (\pi^i_t)_{i \in \set{1, \ldots, d}}$ is $\mathcal{F}_{t - 1/2^k}$-measurable for $t \in \mathbb{T}^k_{++}$, and $\pi$ generates $X^{1, \theta}$ in the way described previously --- in particular, $X^{1, \theta}_t = X^{(\pi)}_t$ holds for all $t \in \mathbb{T}^k$. (In establishing the claims above it is important that all wealth processes of $\mathcal{X}_{\mathsf{s}}$ cannot revive from bankruptcy.)

Continuing, since $\triangle^d$ is a \emph{compact} subset of $\mathbb R^d$, for all $t \in \mathbb{T}^k$ there exists a $\mathcal{F}_{t - 1/2^k}$-measurable $\rho_t = (\rho^i_t)_{i \in \set{1, \ldots, d}}$ such that, for all $\mathcal{F}_{t - 1/2^k}$-measurable and $\triangle^d$-valued $\pi_t = (\pi^i_t)_{i \in \set{1, \ldots, d}}$, we have
\[
\mathbb{E} \bra{ \frac{ 1 + \inner{\pi_t}{\Delta R^{k}_t}}{1 + \inner{\rho_t}{\Delta R^{k}_t}} \ \Bigg| \ \mathcal{F}_{t - 1/2^k}} \leq 1
\]
(It is exactly the existence of such $\rho_t$ can be seen as a stripped-down version of Theorem 4.12 in \cite{MR2335830}; in effect, $\rho_t$ is the optimal proportions of wealth connected with the log-utility maximization problem, modulo technicalities arising when the value of the log-utility maximization problem has infinite value.) Setting $\widetilde{X}^k$ to be the wealth process in $\mathcal{X}_{\mathsf{s}}^k(1)$ generated by $\rho$ as described in the previous paragraph, the result of Lemma \ref{lem: num exists for discrete-time} is immediate.
\end{proof}

We proceed with the proof of implication $(i) \Rightarrow (ii)$ of Theorem \ref{thm: FTAP_jump}, using the notation from the statement of Lemma \ref{lem: num exists for discrete-time}. For all $k \in \mathbb N$, $\widetilde{Y}^k$ satisfies $\widetilde{Y}^k_0 = 1$ and is a positive supermartingale when sampled from times in $\mathbb{T}^k$, since $1 \in \mathcal{X}_{\mathsf{s}}^k$. Therefore, for any $t \in \mathbb{D}$, the \emph{convex hull} of the set $\{ \widetilde{Y}^k_t  \ | \  k \in \mathbb N\}$ is bounded in probability. We also claim that, under condition NA$1_{\mathsf{s}}$, for any $t \in \mathbb R_+$, the convex hull of the set $\{ \widetilde{Y}^k_t  \ | \  k \in \mathbb N \}$ is bounded away from zero in probability. Indeed, for any collection $(\alpha^k)_{k \in \mathbb N}$ such that $\alpha^k \geq 0$ for all $k \in \mathbb N$, having all but a finite number of $\alpha^k$'s non-zero and satisfying $\sum_{k =1 }^\infty \alpha^k = 1$, we have
\[
\frac{1}{\sum_{k = 1}^\infty \alpha^k \widetilde{Y}^k } \ \leq \ \sum_{k = 1}^\infty \alpha^k \frac{1}{\widetilde{Y}^k} \ = \ \sum_{k = 1}^\infty \alpha^k \widetilde{X}^k \, \in \, \mathcal{X}_{\mathsf{s}} (1).
\]
Since, by Proposition \ref{prop: NAone iff NUPBR}, $\set{X_t \ | \  X \in \mathcal{X}_{\mathsf{s}} (1)}$ is bounded in probability for all $t \in \mathbb R_+$, the previous fact proves that the convex hull of the set $\{ \widetilde{Y}^k_t \ | \  k \in \mathbb N \}$ is bounded away from zero in probability.

Now, using Lemma A1.1 of \cite{MR1304434}, one can proceed as in the proof of Lemma 5.2(a) in \cite{MR1469917} to infer the existence of a sequence $(\widehat{Y}^k)_{k \in \mathbb N}$ and some process $(\widehat{Y}_t)_{t \in \mathbb{D}}$ such that, for all $k \in \mathbb N$, $\widehat{Y}^k$ is a convex combination of $\widetilde{Y}^k , \widetilde{Y}^{k+1}, \ldots$,  and $\mathbb{P}[ \lim_{k \to \infty} \widehat{Y}_t^k = Y_t, \, \forall t \in \mathbb{D} ] = 1$. The discussion of the preceding paragraph ensures that $\mathbb{P}[0 < \widehat{Y}_t < \infty, \, \forall \, t \in \mathbb{D}] = 1$.

Let $\mathbb{D} \ni s \leq t \in \mathbb{D}$. Then, $s \in \mathbb{T}^k$ and $t \in \mathbb{T}^k$ for all large enough $k \in \mathbb N$. According to the conditional version of Fatou's Lemma, for all $X \in \bigcup_{k=1}^\infty \mathcal{X}_{\mathsf{s}}^k$ we have that
\begin{equation} \label{eq: Yhat is supermart defl}
\mathbb{E} [\widehat{Y}_t X_t \ | \  \mathcal{F}_s] \leq \liminf_{k \to \infty} \mathbb{E} [ \widehat{Y}^k_t X_t \ | \  \mathcal{F}_s] \leq \liminf_{k \to \infty} \widehat{Y}^k_s X_s = \widehat{Y}_s X_s.
\end{equation}
It follows that $(\widehat{Y}_t X_t)_{t \in \mathbb{D}}$ is a supermartingale  for all $X \in \bigcup_{k=1}^\infty \mathcal{X}_{\mathsf{s}}^k$. (Observe here that we sample the process $\widehat{Y} X$ only at times contained in $\mathbb{D}$.) In particular, $(\widehat{Y}_t)_{t \in \mathbb{D}}$ is a supermartingale.

For any $t \in \mathbb R_+$ define $Y_t := \lim_{s \downarrow \downarrow t, s \in \mathbb{D}} \widehat{Y}_s$ --- the limit is taken in the $\mathbb{P}$-a.s. sense, and exists in view of the supermartingale property of $(\widehat{Y}_t)_{t \in \mathbb{D}}$. It is straightforward that $Y$ is a c\`adl\`ag \ process; it is also adapted because $(\mathcal{F}_t)_{t \in \mathbb R_+}$ is right-continuous. Now, for $t \in \mathbb R_+$, let $T \in \mathbb{D}$ be such that $T > t$; a combination of the right-continuity of both $Y$ and the filtration $(\mathcal{F}_t)_{t \in \mathbb R_+}$, the supermartingale property of  $(\widehat{Y}_t)_{t \in \mathbb{D}}$, and L\'evy's martingale convergence Theorem, give $\mathbb{E}[\widehat{Y}_T \ | \  \mathcal{F}_t] \leq Y_t$. Since $\mathbb{P}[\widehat{Y}_T > 0] = 1$, we obtain $\mathbb{P}[Y_t > 0] = 1$. Right-continuity of the filtration $(\mathcal{F}_t)_{t \in \mathbb R_+}$, coupled with \eqref{eq: Yhat is supermart defl}, imply that $\mathbb{E} [Y_t X_t \ | \  \mathcal{F}_s] \leq  Y_s X_s$ for all $\mathbb R_+ \ni s \leq t \in \mathbb R_+$ and $X \in \bigcup_{k=1}^\infty \mathcal{X}_{\mathsf{s}}^k$. In particular, $Y$ is a c\`adl\`ag \ nonnegative supermartingale; since $\mathbb{P}[Y_t > 0] = 1$ holds for all $t \in \mathbb R_+$, we conclude that $\mathbb{P}[Y_t > 0, \, \forall t \in \mathbb R_+] = 1$.

Of course, $1 \in \mathcal{X}_{\mathsf{s}}^k$ and $S^i \in \mathcal{X}_{\mathsf{s}}^k$ hold for all $k \in \mathbb N$ and $i \in \set{1, \ldots, d}$. It follows that $Y$ is a supermartingale, as well as that $Y S^i$ is a supermartingale for all $i \in \set{1, \ldots, d}$. In particular, $Y$ and $Y S = (Y S^i)_{i \in \set{1, \ldots, d}}$ are semimartingales. Consider any $X^{x, \theta}$ in the notation of \eqref{eq: Xhat}. Using the integration-by-parts formula, we obtain
\[
 Y X^{x, \theta} = x + \int_0^\cdot \pare{ X^{x, \theta}_{t-} - \inner{\theta_t}{S_{t-}}} \, \mathrm d Y_t + \int_0^\cdot \inner{\theta_t}{\, \mathrm d (Y_t S_t)}.
\]
If $X^{x, \theta} \in \mathcal{X}_{\mathsf{s}}(x)$, we have $X^{x, \theta}_{-} - \inner{\theta}{S_-} \geq 0$, as well as $\theta^i \geq 0$ for $i \in \set{1, \ldots, d}$. Then, the supermartingale property of $Y$ and $Y S^i$, $i \in \set{1, \ldots, d}$, gives that $Y X^{x, \theta}$ is a supermartingale. Therefore, $Y \in \mathcal{Y}_{\mathsf{s}}$, i.e., $\mathcal{Y}_{\mathsf{s}} \neq \emptyset$.
\end{proof}

\begin{proof}[$(ii) \Rightarrow (i)$]
Let $Y \in \mathcal{Y}_{\mathsf{s}}$, and fix $T \in \mathbb R_+$. Then, $\sup_{X \in \mathcal{X}_{\mathsf{s}} (1)} \mathbb{E}[Y_T X_T] \leq 1$. In particular, the set $\set{Y_T X_T \ | \  X \in \mathcal{X}_{\mathsf{s}} (1)}$ is bounded in probability. Since $\mathbb{P}[Y_T > 0] = 1$, the set $\set{X_T \ | \  X \in \mathcal{X}_{\mathsf{s}} (1)}$ is bounded in probability as well. An invocation of Proposition \ref{prop: NAone iff NUPBR} finishes the argument.
\end{proof}

\begin{proof}[$(ii) \Rightarrow (iii)$]
Let $Y \in \mathcal{Y}_{\mathsf{s}}$. Since $S^i \in \mathcal{X}_{\mathsf{s}}$, $Y S^i$ is a supermartingale, thus a semimartingale, for all $i \in \set{1, \ldots, d}$. Also, the fact that $Y > 0$ and It\^o's formula give that $1 / Y$ is a semimartingale. Therefore, $S^i = (1 / Y) (Y S^i)$ is a semimartingale for all $i \in \set{1, \ldots, d}$. Furthermore, since $Y S^i$ is a nonnegative supermartingale, we have $Y_t S^i_t = 0$ for all $t \geq \zeta^{S^i}$ on $\{\zeta^{S^i} < \infty \}$, for $i \in \set{1, \ldots, d}$. Now, using $Y > 0$ again, we obtain that $S^i_t = 0$ holds for all $t \geq \zeta^{S^i}$ on $\{ \zeta^{S^i} < \infty \}$. In other words, each $S^i$, $i \in \set{1, \ldots, d}$, cannot revive after bankruptcy.
\end{proof}

\begin{proof}[$(iv) \Rightarrow (i)$]
Since $S$ is a semimartingale, we can consider continuous-time trading. For $x \in \mathbb R_+$, let $\mathcal{X} (x)$ be the set of all  wealth processes $X^{x, \theta} := x + \int_0^\cdot \inner{\theta_t}{\, \mathrm d S_t}$, where $\theta$ is $d$-dimensional, predictable and $S$-integrable, ``$\int_0^\cdot \inner{\theta_t}{\, \mathrm d S_t}$'' denotes a vector stochastic integral, $X^{x, \theta} \geq 0$ and $0 \leq \inner{\theta}{S_-} \leq X_-^{x, \theta}$. (Observe that the qualifying subscript ``$\mathsf{s}$'' denoting simple trading has been dropped in the definition of $\mathcal{X}_{\mathsf{s}}(x)$, since we are considering continuous-time trading.) Of course, $\mathcal{X}_{\mathsf{s}}(x) \subseteq \mathcal{X}(x)$. We shall show in the next paragraph that $\set{X_T \ | \  X \in \mathcal{X}(1)}$ is bounded in probability for all $T \in \mathbb R_+$, therefore establishing condition NA$1_{\mathsf{s}}$, in view of Proposition \ref{prop: NAone iff NUPBR}.

For all $i \in \set{1, \ldots, d}$, write $S^i = S^i_0 \mathcal E(R^i)$, where $R^i$ is a semimartingale with $R^i_0 = 0$. Let $R := (R^i)_{i=1, \ldots, d}$. It is straightforward to see that $\mathcal{X} (1)$ coincides with the class of all processes of the form $\mathcal E \pare{ \int_0^\cdot \inner{\pi_t}{\, \mathrm d R_t} }$, where $\pi$ is predictable and take values in the $d$-dimensional simplex $\triangle^d := \big\{ z = (z^i)_{i =1, \ldots, d} \in \mathbb R^d \ | \  z^i \geq 0 \text{ for } i=1, \ldots, d, \text{ and } \sum_{i=1}^d z^i \leq 1 \big\}$. Since, for all $T \in \mathbb R_+$,
\[
\log \pare{\mathcal E \pare{ \int_0^T \inner{\pi_t}{\, \mathrm d R_t} }} \leq \int_0^T \inner{\pi_t}{\, \mathrm d R_t}
\]
holds for all $\triangle^d$-valued and predictable $\pi$, it suffices to show the boundedness in probability of the class of all $\int_0^T \inner{\pi_t}{\, \mathrm d R_t}$, where $\pi$ ranges in all $\triangle^d$-valued and predictable processes. Write $R = B + M$, where $B$ is a process of finite variation and $M$ is a local martingale with $|\Delta M^i| \leq 1$, $i \in \set{1, \ldots, d}$. Then, $\int_0^T |\inner{\pi_t}{\, \mathrm d B_t}| \leq \sum_{i =1}^d \int_0^T |\, \mathrm d B^i_t| < \infty$. This establishes the boundedness in probability of the class of all $\int_0^T \inner{\pi_t}{\, \mathrm d B_t}$, where $\pi$ ranges in all $\triangle^d$-valued and predictable processes. We have to show that the same holds for the class of all $\int_0^T \inner{\pi_t}{\, \mathrm d M_t}$, where $\pi$ is $\triangle^d$-valued and predictable. For $k \in \mathbb N$, let $\tau^k := \inf \{ t \in \mathbb R_+ \ | \  \sum_{i =1}^d [M^i, M^i]_t \geq k \} \wedge T$, Note that $[ M^i, M^i]_{\tau^k} = [ M^i, M^i]_{\tau^k-} + |\Delta M^i_{\tau^k}|^2 \leq  k + 1$ holds for all $i \in \set{1, \ldots, d}$. Therefore, using the notation $\normtwo{\eta} := \sqrt{\mathbb{E}[|\eta|^2]}$ for a random variable $\eta$, we obtain
\[
 \normtwo{\int_0^{\tau^k} \inner{\pi_t}{\, \mathrm d M_t}} \leq \sum_{i=1}^d \normtwo{ \int_0^{\tau^k} \pi^i_t \, \mathrm d M^i_t  } \leq \sum_{i=1}^d \normtwo{  \sqrt{[ M^i, M^i]_{\tau^k}}} \leq d \sqrt{k + 1}
\]
Fix $\epsilon > 0$. Let $k = k (\epsilon)$ be such that $\mathbb{P}[\tau^k < T] < \epsilon / 2$, and also let $\ell := d \sqrt{2 (k + 1) / \epsilon} $. Then,
\[
 \mathbb{P} \bra{ \int_0^T \inner{\pi_t}{\, \mathrm d M_t} > \ell} \leq \mathbb{P} \bra{\tau^k < T} + \mathbb{P} \bra{ \int_0^{\tau^k} \inner{\pi_t}{\, \mathrm d M_t} > \ell} \leq \frac{\epsilon}{2} + \abs{\frac{\normtwo{ \int_0^{\tau^k} \inner{\pi_t}{\, \mathrm d M_t}}}{\ell}}^2 \leq \epsilon.
\]
The last estimate is uniform over all $\triangle^d$-valued and predictable $\pi$. We have, therefore, established the boundedness in probability of the class of all $\int_0^T \inner{\pi_t}{\, \mathrm d M_t}$, where $\pi$ ranges in all $\triangle^d$-valued and predictable processes. This completes the proof.
\end{proof}

\subsection{Proof of Theorem \ref{thm: FTAP_jump}, statement $(2)$.}

In view of statement (1) of Theorem \ref{thm: FTAP_jump}, we only need to show the validity of $(iii) \Leftrightarrow (iv)$ under the extra assumption of statement (2). This equivalence is really Proposition 2.2 in \cite{MR2322919}, but we present the few details for completeness.

For the implication $(iii) \Rightarrow (iv)$, simply define $R^i := \int_0^\cdot (1 / S^i_{t-}) \, \mathrm d S^i_t$ for $i \in \set{1, \ldots, d}$,  The latter process is a well-defined semimartingale because, for each $i \in \set{1, \ldots, d}$, $S^i$ is a semimartingale, $S^i_-$ is locally bounded away from zero on the stochastic interval $\dbra{0, \zeta^{S^i}}$, and $S = 0$ on $\dbraco{\zeta^{S^i}, \infty}$.

Now, for $(iv) \Rightarrow (iii)$, it is clear that $S$ is a semimartingale. Furthermore, for all $i \in \set{1, \ldots, d}$, $S^i$ cannot revive from bankruptcy; this follows because stochastic exponentials stay at zero once they hit zero.
\qed

\section{On and Beyond the Main Result} \label{sec: beyond main result}

\subsection{Comparison with the result of Delbaen and Schachermayer} \label{subsubsec: compar with DS}

Theorem 7.2  of the seminal paper \cite{MR1304434} establishes the semimartingale property of $S$ under condition NFLVR for simple admissible strategies, coupled with a local boundedness assumption on $S$ (always together with the c\`adl\`ag \ property and adaptedness). The assumptions of Theorem \ref{thm: FTAP_jump} are different than the ones in \cite{MR1304434}. Condition NA$1_{\mathsf{s}}$ \ (valid for simple, no-short-sales trading) is weaker than NFLVR for simple admissible strategies. Furthermore, local boundedness from above is not required in our context, but we do require that each $S^i$, $i \in \set{1, \ldots, d}$, is nonnegative. In fact, as we shall argue in \S \ref{subsubsec: local bdd from below} below, nonnegativity of each $S^i$, $i \in \set{1, \ldots, d}$, can be weakened by local boundedness from below, indeed making Theorem \ref{thm: FTAP_jump} a generalization of Theorem 7.2 in \cite{MR1304434}. Note that if the components of $S$ are unbounded both above and below, not even condition NFLVR is enough to ensure the semimartingale property of $S$; see Example 7.5 in \cite{MR1304434}.

\smallskip
Interestingly, and in contrast to \cite{MR1304434}, the proof of Theorem \ref{thm: FTAP_jump} provided here does \emph{not} use the deep Bichteler-Dellacherie theorem on the characterization of semimartingales as ``good integrators'' (see \cite{MR1906715}, \cite{MR2273672}, where one \emph{starts} by defining semimartingales as good integrators and obtains the classical definition as a by-product). Actually, and in view of Proposition \ref{prop: NAone iff NUPBR}, statement (2) of Theorem \ref{thm: FTAP_jump} can be seen as a ``multiplicative'' counterpart of the Bichteler-Dellacherie theorem. Its proof exploits two simple facts: (a) positive supermartingales are semimartingales, which follows directly from the Doob-Meyer decomposition theorem; and (b) reciprocals of strictly positive supermartingales are semimartingales, which is a consequence of It\^o's formula. Crucial in the proof is also the concept of the num\'eraire \ portfolio.

\subsection{The semimartingale property of $S$ when each $S^i$, $i \in \set{1, \ldots, d}$, is locally bounded from below} \label{subsubsec: local bdd from below}

As mentioned previously, implication $(i) \Rightarrow (iii)$ actually holds even when each $S^i$, $i \in \set{1, \ldots, d}$,  is locally bounded from below, which we shall establish now. We still, of course, assume that each $S^i$, $i \in \set{1, \ldots, d}$, is adapted and c\`adl\`ag. Since ``no-short-sales'' strategies have ambiguous meaning when asset prices can become negative, we need to make some changes in the class of admissible wealth processes. For $x \in \mathbb R_+$, let $\mathcal{X}_{\mathsf{s}}'(x)$ denote the class of all wealth processes $X^{x, \theta}$ using simple trading as in \eqref{eq: Xhat} that satisfy $X^{x, \theta} \geq 0$. Further, set $\mathcal{X}_{\mathsf{s}}' = \bigcup_{x \in \mathbb R_+} \mathcal{X}_{\mathsf{s}}'(x)$. Define condition NA$1'_{\mathsf{s}}$ \ for the class $\mathcal{X}_{\mathsf{s}}'$ in the obvious manner, replacing ``$\mathcal{X}_{\mathsf{s}}$'' with ``$\mathcal{X}_{\mathsf{s}}'$'' throughout in \S \ref{subsec: UPBR}. Assume then that condition NA$1'_{\mathsf{s}}$ \ holds. To show that $S$ is a semimartingale, it is enough to show that $(S_{\tau^k \wedge t})_{t \in \mathbb R_+}$ is a semimartingale for each $k \in \mathbb N$, where $(\tau^k)_{k \in \mathbb N}$ is a localizing sequence such that $S^i \geq -k$ on $\dbra{0, \tau^k}$ for all $i \in \set{1, \ldots, d}$ and $k \in \mathbb N$. In other words, we might as well assume that $S^i \geq - k$ for all $i \in \set{1, \ldots, d}$. Define $\tilde{S}^i := k + S^i$; then, $\tilde{S}^i$ is nonnegative for all $i \in \set{1, \ldots, d}$. Let $\tilde{S} = (\tilde{S}^i)_{i \in \set{1, \ldots, d}}$. If $\tilde{\mathcal{X}_{\mathsf{s}}}$ is (in self-explanatory notation) the collection of all wealth processes resulting from simple, no-short-sales strategies investing in $\tilde{S}$, it is straightforward that $\tilde{\mathcal{X}_{\mathsf{s}}} \subseteq \mathcal{X}_{\mathsf{s}}'$. Therefore, NA$1_{\mathsf{s}}$ \ holds for simple, no-short-sales strategies investing in $\tilde{S}$; using implication $(i) \Rightarrow (iii)$ in statement (1) of Theorem \ref{thm: FTAP_jump}, we obtain the semimartingale property of $\tilde{S}$. The latter is of course equivalent to $S$ being a semimartingale.

\smallskip

One might wonder why we do not simply ask from the outset that each $S^i$, $i \in \set{1, \ldots, d}$, is locally bounded from below, since it certainly contains the case where each $S^i$, $i \in \set{1, \ldots, d}$, is nonnegative. The reason is that by restricting trading to using only no-short-sales strategies (which we can do when each $S^i$, $i \in \set{1, \ldots, d}$, is nonnegative) enables us to be as general as possible in extracting the semimartingale property of $S$ from the NA$1_{\mathsf{s}}$ \ condition. Consider, for example, the discounted asset-price process given by $S = a \mathbb{I}_{\dbraco{0,1}} + b \mathbb{I}_{\dbraco{1,\infty}}$, where $a > 0$ and $b \in \mathbb R_+$ with $a \neq b$. This is a \emph{really} elementary example of a nonnegative semimartingale. Now, if we allow for any form of simple trading, as long as it keeps the wealth processes nonnegative, it is clear that condition NA$1'_{\mathsf{s}}$ \ will fail (since it is known that at time $t = 1$ there will be a jump of size $(b - a) \in \mathbb R \setminus \set{0}$ in the discounted asset-price process). On the other hand, if we only allow for no-short-sales strategies, NA$1_{\mathsf{s}}$ \ will hold --- this is easy to see directly using Proposition \ref{prop: NAone iff NUPBR}, since $X_T \leq |b - a| / a$ for all $T \geq 1$ and $X \in \mathcal{X}_{\mathsf{s}}(1)$. Therefore, we can conclude that $S$ is a semimartingale using implication $(i) \Rightarrow (iii)$ in statement (1) of Theorem \ref{thm: FTAP_jump}. (Of course, one might argue that there is no need to invoke Theorem \ref{thm: FTAP_jump} for the simple example here. The point is that allowing for all nonnegative wealth processes results in a rather weak sufficient criterion for the semimartingale property of $S$.)

\subsection{The semimartingale property of $S$ via bounded indirect utility}
There has been previous work in the literature obtaining the semimartingale property of $S$ using the finiteness of the value function of a utility maximization problem via use of only simple strategies --- see, for instance, \cite{MR2139030}, \cite{MR2157199}, \cite{LarZit05}. In all cases, there has been an assumption of local boundedness (or even continuity) on $S$. We shall offer a result in the same spirit, dropping the local boundedness requirement. We shall assume \emph{either} that discounted asset-price processes are nonnegative and \emph{only} no-short-sales simple strategies are considered (which allows for a sharp result), \emph{or} that discounted asset-price processes are locally bounded from below. In the latter case, Proposition \ref{prop: semimart via finite expec util} that follows is a direct generalization of the corresponding result in \cite{MR2139030}, where the authors consider locally bounded (both above and below) discounted asset-price processes. In the statement of Proposition \ref{prop: semimart via finite expec util} below, we use the notation $\mathcal{X}_{\mathsf{s}}'(x)$ introduced previously in \S \ref{subsubsec: local bdd from below}.

\begin{prop} \label{prop: semimart via finite expec util}
Let $S = (S^i)_{i =1, \ldots, d}$ be such that $S^i$ is adapted and c\`adl\`ag \ process for $i \in \set{1, \ldots, d}$. Also, let $U: \mathbb R_+ \mapsto \mathbb R \cup \{ - \infty \}$ be a nondecreasing function with $U > - \infty$ on $]0 , \infty]$ and $U(\infty) = \infty$. Fix some $x > 0$. Finally, let $T$ be a finite stopping time. Assume that either:
\begin{itemize}
	\item each $S^i$, $i \in \set{1, \ldots, d}$, is nonnegative and $\sup_{X \in \mathcal{X}_{\mathsf{s}}(x)} \mathbb{E}[U(X_T)] < \infty$, or
	\item each $S^i$, $i \in \set{1, \ldots, d}$, is locally bounded from below and $\sup_{X \in \mathcal{X}_{\mathsf{s}}'(x)} \mathbb{E}[U(X_T)] < \infty$.
\end{itemize}
Then, the process $(S_{T \wedge t})_{t \in \mathbb R_+}$ is a semimartingale.
\end{prop}

\begin{proof}
Start by assuming that each $S^i$, $i \in \set{1, \ldots, d}$ is nonnegative and that $\sup_{X \in \mathcal{X}_{\mathsf{s}}(x)} \mathbb{E}[U(X_T)] < \infty$. Since we only care about the semimartingale property of $(S_{T \wedge t})_{t \in \mathbb R_+}$, assume without loss of generality that $S_t = S_{T \wedge t}$ for all $t \in \mathbb R_+$. Suppose that condition NA$1_{\mathsf{s}}$ \ fails. According to Proposition \ref{prop: NAone iff NUPBR} and Remark \ref{rem: NAone for stop times}, there exists a sequence $(\widetilde{X}^n)_{n \in \mathbb N}$ of elements in $\mathcal{X}_{\mathsf{s}}(x)$ and $p > 0$ such that $\mathbb{P} [\widetilde{X}_T^n > 2 n] \geq p$ for all $n \in \mathbb N$. For all $n \in \mathbb N$, let $X^n := (x + \widetilde{X}^n)/2 \in \mathcal{X}_{\mathsf{s}}(x)$. Then, $\sup_{X \in \mathcal{X}_{\mathsf{s}}(x)} \mathbb{E}[U(X_T)] \geq \liminf_{n \to \infty} \mathbb{E}[U(X_T^n)] \geq (1 - p) U(x/2) + p \liminf_{n \to \infty} U(n) = \infty$. This is a contradiction to $\sup_{X \in \mathcal{X}_{\mathsf{s}}(x)} \mathbb{E}[U(X_T)] < \infty$. We conclude that $(S_{T \wedge t})_{t \in \mathbb R_+}$ is a semimartingale using implication $(i) \Rightarrow (iii)$ in statement (1) of Theorem \ref{thm: FTAP_jump}.

Under the assumption that each $S^i$, $i \in \set{1, \ldots, d}$ is locally bounded from below and that $\sup_{X \in \mathcal{X}_{\mathsf{s}}'(x)} \mathbb{E}[U(X_T)] < \infty$, the proof is exactly the same as the one in the preceding paragraph, provided that one replaces ``$\mathcal{X}_{\mathsf{s}}$'' with ``$\mathcal{X}_{\mathsf{s}}'$'' throughout, and uses the fact that condition NA$1'_{\mathsf{s}}$ \ for the class $\mathcal{X}_{\mathsf{s}}'$ implies the semimartingale property for $S$, as was discussed in \S \ref{subsubsec: local bdd from below}.
\end{proof}

\subsection{On the implication $(iii) \Rightarrow (i)$ in Theorem \ref{thm: FTAP_jump}}
If we do not require the additional assumption on $S$ in statement (2) of Theorem \ref{thm: FTAP_jump}, implication $(iii) \Rightarrow (i)$ might fail. We present below a counterexample where this happens.

On $(\Omega, \mathcal{F}, \mathbb{P})$, let $W$ be a standard, one-dimensional Brownian motion (with respect to its own natural filtration --- we have not defined $(\mathcal{F}_t)_{t \in \mathbb R_+}$ yet). Define the process $\xi$ via $\xi_t := \exp(- t / 4 + W_t)$ for $t \in \mathbb R_+$. Since $\lim_{t \to \infty} W_t / t = 0$, $\mathbb{P}$-a.s., it is straightforward to check that $\xi_\infty := \lim_{t \to \infty} \xi_t = 0$, and actually that $\int_0^\infty \xi_t \, \mathrm d t < \infty$, both holding $\mathbb{P}$-a.s. Write $\xi = A + M$ for the Doob-Meyer decomposition of the continuous submartingale $\xi$ under its natural filtration, where $A = (1 / 4) \int_0^\cdot \xi_t \, \mathrm d t$ and $M = \int_0^\cdot \xi_t \, \mathrm d W_t$. Due to $\int_0^\infty \xi_t \, \mathrm d t < \infty$, we have $A_\infty < \infty$ and $[M, M]_\infty = \int_0^\infty |\xi_t|^2 \, \mathrm d t < \infty$, where $[M, M]$ is the quadratic variation process of $M$. In the terminology of \cite{MR2126973}, $\xi$ is a semimartingale up to infinity. If we define $S$ via $S_t = \xi_{t / (1 - t)}$ for $t \in [0, 1[$ and $S_t = 0$ for $t \in [1, \infty[$, then $S$ is a nonnegative semimartingale. Define $(\mathcal{F}_t)_{t \in \mathbb R_+}$ to be the augmentation of the natural filtration of $S$. Observe that $\zeta^S = 1$ and $S_{\zeta^S -} = 0$; the condition of statement (2) of Theorem \ref{thm: FTAP_jump} is not satisfied. In order to establish that NA$1_{\mathsf{s}}$ \ fails, and in view of Proposition \ref{prop: NAone iff NUPBR}, it is sufficient to show that $\set{X_1 \ | \  X \in \mathcal{X}_{\mathsf{s}}(1)}$ is not bounded in probability. Using continuous-time trading, define a wealth process $\widehat{X}$ for $t \in [0, 1[$, via $\widehat{X}_0 = 1$ and the dynamics $\, \mathrm d \widehat{X}_t / \widehat{X}_t = (1 / 4) (\, \mathrm d S_t / S_t)$ for $t \in [0,1[$. Then, $\widehat{X}_t = \exp \pare{ (1/16) (t / (1-t)) + (1/4) W_{t / (1-t)}}$ for $t \in [0,1[$,
which implies that $\mathbb{P}[\lim_{t \uparrow \uparrow 1} \widehat{X}_t = \infty] = 1$, where ``$t \uparrow \uparrow 1$'' means that $t$ \emph{strictly} increases to $1$. Here, the percentage of investment is $1/4 \in [0,1]$, i.e, $\widehat{X}$ is the result of a no-short-sales strategy. One can then find an approximating sequence $(X^k)_{k \in \mathbb N}$ such that $X^k \in \mathcal{X}_{\mathsf{s}}(1)$ for all $k \in \mathbb N$, as well as $\mathbb{P}[| X^k_{1}  - \widehat{X}_{1 - 1 / k} | < 1] > 1 - 1 / k$. (Approximation results of this sort are discussed in greater generality in \cite{MR1971602}.) Then, $(X^k_1)_{k \in \mathbb N}$ is not bounded in probability; therefore,  NA$1_{\mathsf{s}}$ \ fails. Of course, in this example we also have $(iii) \Rightarrow (iv)$ of Theorem \ref{thm: FTAP_jump} failing.


\section{Introduction}

In this paper we describe basic concepts of dynamic programming in terms of categories of optics. The class of models we consider are discrete-time Markov decision processes, aka. discrete-time controlled Markov chains. There are classical methods of computing optimal control policies, underlying much of both classical control theory and modern reinforcement learning, known collectively as \emph{dynamic programming}. These are based on two operations that can be interleaved in many different ways: \emph{value improvement} and \emph{policy improvement}. The central idea of this paper is the slogan \emph{value improvement is optic precomposition}, or said differently, \emph{value improvement is a representable functor on optics}.

Given a control problem with state space $X$, a \emph{value function} $V : X \to \R$ represents an estimate of the long-run payoff of following a policy starting from any state, and can be equivalently represented as a costate $V : \binom{X}{\R} \to I$ in a category of optics. Every control policy $\pi$ also induces an optic $\lambda (\pi) : \binom{X}{\R} \to \binom{X}{\R}$. The general idea is that the forwards pass of the optic is a morphism $X \to X$ describing the dynamics of the Markov chain given the policy, and the backwards pass is a morphism $X \otimes \R \to \R$ which given the current state and the \emph{continuation payoff}, describing the total payoff from all future stages, returns the total payoff for the current stage given the policy, plus all future stages.

Given a policy $\pi$ and a value function $V : \binom{X}{\R} \to I$, the costate $\binom{X}{\R} \overset{\lambda (\pi)}\longrightarrow \binom{X}{\R} \overset{V}\longrightarrow I$ is a closer approximation of the value of $\pi$. This is called \emph{value improvement}. Iterating this operation
\[ \ldots \binom{X}{\R} \overset{\lambda (\pi)}\longrightarrow \binom{X}{\R} \overset{\lambda (\pi)}\longrightarrow \binom{X}{\R} \overset{V}\longrightarrow I \]
converges efficiently to the true value function of the policy $\pi$.

Replacing $\pi$ with a new policy that is optimal for its value function is called \emph{policy improvement}. Repeating these steps is known as \emph{policy iteration}, and converges to the optimal policy and value function.

Alternatively, instead of repeating value improvement until convergence before each step of policy improvement, we can also alternate them, giving the composition of optics
\[ \ldots \binom{X}{\R} \overset{\lambda (\pi_2)}\longrightarrow \binom{X}{\R} \overset{\lambda (\pi_1)}\longrightarrow \binom{X}{\R} \overset{V}\longrightarrow I \]
where each policy $\pi_i$ is optimal for the value function to the right of it. This is known as \emph{value iteration}, and also converges to the optimal policy and value function.
For an account of convergence properties of these algorithms, classic textbooks are \cite[Sec.6]{Puterman}, \cite[Ch.1]{bertsekas_dpoc_vol2}.

In this paper we illustrate this idea, using mixed optics to account for the categorical structure of transitions in a Markov chain and the convex structure of expected payoffs, which typically form the kleisli and Eilenberg-Moore categories of a probability monad. This paper is partially intended as an introduction to dynamic programming for category theorists, focussing on illustrative examples rather than on heavy theory.

\subsection{Related work}

The precursor of this paper was early work on value iteration using open games \cite{hedges_etal_compositional_game_theory}. The idea originally arose around 2016 during discussions of the first author with Viktor Winschel and Philipp Zahn. An early version was planned as a section of \cite{hedges_morphisms_open_games} but cut partly for page limit reasons, and partly because the idea was quite uninteresting until it was understood how to model stochastic transitions in open games \cite{bolt_hedges_zahn_bayesian_open_games} via optics \cite{Riley}. In this paper we have chosen to present the idea without any explicit use of open games, both in order to clarify the essential idea and also to bring it closer to the more recent framework of categorical cybernetics \cite{towards_foundations_categorical_cybernetics}, which largely subsumes open games \cite{capucci_etal_translating_extensive_form}. (Although, actually using this framework properly is left for future work.)

A proof-of-concept implementation of value iteration with open games was done in 2019 by the first author and Wolfram Barfuss\footnote{Source currently available at \url{https://github.com/jules-hedges/open-games-hs/blob/og-v0.1/src/OpenGames/Examples/EcologicalPublicGood/EcologicalPublicGood.hs}}, implementing a model from \cite{barfuss_learning_dynamics} - a model of the social dilemma of emissions cuts and climate collapse as a stochastic game, or jointly controlled MDP - and verifying it against Barfuss' Matlab implementation. A far more advanced implementation of reinforcement learning using open games was developed recently by Philipp Zahn, currently closed-source, and was used for the paper \cite{eschenbaum_etal_robust_algorithmic_collusion}.

The most closely related work to ours is \cite{DJM}, which formulates MDPs in terms of F-lenses \cite{Spivak} of the functor $\operatorname{BiKl}(C\times -,\Delta(\RR\times -))^{\op}$, where $C\times-$ is the reader comonad and $\Delta(\RR\times-)$ is a probability monad over actions with their expected value.
A MDP there is a lens from states and potential state changes and rewards to the agents observation and input $\binom{X}{\Delta(X\times\R)}\to\binom{O}{I}$.
Our approach differs in two ways.
We firstly assume that the readout function is the identity, as we are not dealing with partial observability \cite{POMDP}.
Secondly, we specify a concrete structure of the backwards update map $f^*: X\times I\to \Delta(X\times R)$, which allows us to rearrange the interface of this lens from policies to value functions.
Doing so opens up the possibility of composing these lenses sequentially, which is the heart of the dynamic programming approach explored in this paper.

Another approach is to model MDPs as coalgebras from states to rewards and potential transitions, as done by Feys et al. \cite{LTVMPDs}.
They observe that the Bellman optimality condition for value iteration is a certain coalgebra-to-algebra morphism. We believe that this approach is orthogonal to ours, and both could potentially be done simultaneously. We discuss this in the further work section.

A series of papers by Botta et al (for example \cite{botta13}) formulates dynamic programming in dependent type theory, accounting in a serious way for how different actions can be available in different states, a complication that we ignore in this paper. It may be possible to unify these approaches using dependent optics \cite{braithwaite_etal_fibre_optics,vertechi_dependent_optics}.

Finally, \cite{baez_erbele_categories_control} builds a category of signal flow diagrams, a widely used tool in control theory. Besides the common application to control theory there is little connection to this paper. In particular, time is implicit in their string diagrams, meaning their models have continuous time, whereas our approach is inherently discrete time. Said another way, composition in their category is `space-like' whereas ours is `time-like' - their morphisms are (open) systems whereas ours are processes.

\section{Dynamic programming}

\subsection{Markov Decision Processes}

A \emph{Markov decision process} (MDP) consists of a state space $X$, an action space $A$, a state transition function $f: X\times A\to X$, and a utility or reward function $U: X\times A\to \RR$. The state transition function is often taken to be stochastic, that is, to be given by probabilities $f (x' \mid x, a)$. In the stochastic case the utility function can be taken without loss of generality to be an expected utility function. We imagine actions to be chosen by an agent, who is trying to \emph{control} the Markov chain with the objective of optimising the long-run reward.

A \emph{policy} for an MDP is a function $\pi : X \to A$, which can also be taken to be either deterministic or stochastic. The type of policies encodes the Markov property: the choice of action depends only on the current state, and may not depend on any memory of past states.

Given an initial state $x_0 \in X$, a policy $\pi$ determines (possibly stochastically) a sequence of states
\[ x_0,\quad x_1=f(x_0,\pi(x_0)),\quad x_2=f(x_1,\pi(x_1)),\quad \dots \]
The total payoff is given by an infinite geometric sum of individual payoffs for each transition:
\begin{equation} \label{eq:policy_value}
    V_\pi(x_0) = \sum_{k=0}^\infty \beta^kU(x_k,\pi(x_k))
\end{equation}
where $0 < \beta < 1$ is a fixed \emph{discount factor} which balances the relevance of present and future payoffs. (There are other methods of obtaining a single objective from an infinite sequence of transitions, such as averaging, but we focus on discounting in this paper.)
A key idea behind dynamic programming is that this geometric sum can be equivalently written as a telescoping sum:
\[ V_\pi (x_0) = U (x_0, \pi (x_0)) + \beta (U (x_1, \pi (x_1)) + \beta (U (x_2, \pi (x_2) + \cdots))) \]

The \emph{control problem} is to choose a policy $\pi$ in order to maximise (the expected value of) $V_\pi (x_0)$.

\subsection{Deterministic dynamic programming}

In dynamic programming, the agent's objective of maximizing the overall utility can be divided into two orthogonal goals:
to determine the value of a given policy $\pi$ (which we call the \emph{value improvement} step), and to determine the optimal policy $\pi^*$ (the \emph{policy improvement} step).
Bellman's equation is used as an update rule for both:
\begin{align}
    \text{\textbf{Value improvement:}} &&&V'(x) = U(x,\pi(x)) + \beta V(f(x,\pi(x))) \label{eq:value_improvement} \\
    \text{\textbf{Policy improvement:}} &&&\pi'(x) = \arg\max_{a \in A} U(x,a) + \beta V(f(x,a)) \label{eq:policy_improvement}
\end{align}
A Bellman optimality condition on the other hand determines the fixpoint of this update rule, and is met when $V'=V$ and $\pi'=\pi$ respectively.

The update rule \eqref{eq:value_improvement} is the discounted sum \eqref{eq:policy_value} where the stream of states is co-recursively fixed by the policy $\pi$ and transition function $f$.
The co-recursive structure refers to the calculation of the utility of a state $x$, where one needs the utility of the \emph{next} state, while in a recursive structure, $x$ needs the \emph{previous} state, starting from an initial state as a base case.

Two classical algorithms use these two steps differently:
Policy iteration iterates value improvement until the current policy value is optimal before performing a policy improvement step, and value iteration interleaves both steps one after another.

In policy iteration, a initial value function is chosen (usually $V(x)=0$), and a randomly chosen policy $\pi$ is evaluated by \eqref{eq:value_improvement} repeatedly until the value reaches a fixpoint, which is assured by the contraction mapping \cite{denardo_contraction_in_dp}.
Once $V$ reaches (or in practice gets close to) a fixpoint $V'=V$ or another convergence condition, the policy improvement step \eqref{eq:policy_improvement} chooses a greedy policy as an improvement to $\pi$.

A $q$-function or \emph{state-action value function} $q_\pi : X \times A \to \R$ describes the value of being in state $x$ and then taking action $a$, assuming that subsequent actions are taken by the policy $\pi$
\begin{equation}\label{eq:q}
    q_\pi(x,a) = U(x,a) + \beta V(f(x,a))
\end{equation}
The \emph{policy improvement theorem} \cite{bellman_1957} states that if a pair of deterministic policies $\pi,\pi': X\to A$ satisfies for all $x\in X$
\[ q_\pi(x,\pi'(x)) \geq V_\pi(x) \]
then $V_{\pi'}(x)\geq V_\pi(x)$ for all $x\in X$.

The optimal policy $\pi^*$, if it exists, is the policy which if followed from any state, generates the maximum value.
This is a Bellman optimality condition which fuses the two steps \eqref{eq:value_improvement}, \eqref{eq:policy_improvement}:
\begin{equation} \label{eq:Bellman}
    V_{\pi^*}(x) = \max_{a \in A} U(x,a) + \beta V_{\pi^*}(f(x,a))
\end{equation}

\emph{Value iteration} is a special policy iteration algorithm insofar it stops the update rule for value improvement to one step, by truncating the sum \eqref{eq:policy_value} to the first summand.
Moreover, it introduces the value improvement step implicitly in the policy improvement, which assigns a value to states
\[ V'(x) = \max_{a \in A} U(x,a) + \beta V(f(x,a)) \]
while the policy in each iteration is still recoverable as
\[ \pi'(x) = \arg\max_{a \in A} U(x,a) + \beta V(f(x,a)) \]

\subsection{Stochastic dynamic programming}

Stochasticity can be introduced in different places in a MDP:
\begin{enumerate
    \item in the policy $\pi: X\to \Delta A$, where the probability of the policy $\pi$ taking action $a$ in a state $x$ is now notated $\pi(a \mid x)$.
    \item in the transition function $f: X\times A\to \Delta X$ and potentially the reward function $U: X\times A\to \Delta\R$ independently.
    \item usually the reward is included inside the transition function $f: X\times A\to \Delta(X\times\R)$, allowing correlated next states and rewards.
    This is relevant when the reward is morally from the next state, rather than the current state and action.
    If the reward were truly from the current state and action, the transition function can be decomposed into a function $f: X\times A\to \Delta X \times \Delta \R$.
\end{enumerate}
In this section we assume for simplicity that $\Delta$ is the finite support distribution monad, although the equations in the following can be formulated for arbitrary distributions by replacing the sum with an appropriate integral.

The policy value update rule \eqref{eq:value_improvement} becomes stochastic, and adopts a slightly different form depending on which part of the MDP is stochastic. For the cases 1. and 2.:
\begin{align}
    V'(x) &= \sum_a\pi(a \mid x)(U(x,a) + \beta V(f(x,a))) \\
    V'(x) &= \sum_r U(r \mid x,a)r + \sum_{x'}f(x' \mid x,a)\beta V(x') \\
    \intertext{In the most general case, that is 1. together with 3.:}
    V'(x) &= \sum_{a\in A}\pi(a \mid x)\sum_{x',r}f(x',r \mid x,a)(r + \beta V(x')) \label{eq:prediction}
\end{align}
(Note that the sum over $r$ is over the support of $f (- \mid x, a)$, which we assume here to be finite, although in general it can be replaced with an integral.)

The policy improvement theorem holds in the stochastic setting \cite[Sec.4.2]{RL_Intro} by defining
\[ q_\pi(s,\pi'(s))=\sum_a \sigma(a \mid s)q_\pi(s,a) \]

\subsection{Gridworld example}

A classic example in reinforcement learning is the Gridworld environment, where an agent moves in the four cardinal directions in a rectangular grid.
States of this finite MDP correspond to the positions that the agent can be in.

Assume that all transitions and policies are deterministic, and that the transition function prevents the agent from moving outside the boundary.
Suppose that the environment rewards 0 value for all states except the top left corner, where the reward is 1 (see figure \ref{fig:Gridworld}).

\begin{figure}[ht!]
    \centering
    \begin{tikzpicture}
        \tikzstyle{arrow} = [red,-stealth,shorten <=-3pt,shorten >=-3pt]
        \tikzstyle{matrixsty} = [matrix of nodes,nodes={inner sep=0pt,text width=.5cm,align=center,minimum height=.5cm}]
        \matrix(m0)[matrixsty] at (1,1){
        1 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m1)[matrixsty] at (4,1){
        1 & 0 & 0 & 0 \\
        $\beta$ & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m2)[matrixsty] at (7,1){
        1 & 0 & 0 & 0 \\
        $\beta$ & 0 & 0 & 0 \\
        $\beta^2$ & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m3)[matrixsty] at (10,1){
        1 & 0 & 0 & 0 \\
        $\beta$ & 0 & 0 & 0 \\
        $\beta^2$ & 0 & 0 & 0 \\
        $\beta^3$ & 0 & 0 & 0 \\};

        \matrix(m4)[matrixsty] at (13,1){
        1 & 0 & 0 & 0 \\
        $\beta$ & 0 & 0 & 0 \\
        $\beta^2$ & 0 & 0 & 0 \\
        $\beta^3$ & 0 & 0 & 0 \\};

        \matrix(m5)[matrixsty] at (1,-2){
        1 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m6)[matrixsty] at (4,-2){
        1 & $\beta$ & 0 & 0 \\
        $\beta$ & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m7)[matrixsty] at (7,-2){
        1 & $\beta$ & $\beta^2$ & 0 \\
        $\beta$ & $\beta^2$ & 0 & 0 \\
        $\beta^2$ & 0 & 0 & 0 \\
        0 & 0 & 0 & 0 \\};

        \matrix(m8)[matrixsty] at (10,-2){
        1 & $\beta$ & $\beta^2$ & $\beta^3$ \\
        $\beta$ & $\beta^2$ & $\beta^3$ & 0 \\
        $\beta^2$ & $\beta^3$ & 0 & 0 \\
        $\beta^3$ & 0 & 0 & 0 \\};

        \matrix(m9)[matrixsty] at (13,-2){
        1 & $\beta$ & $\beta^2$ & $\beta^3$ \\
        $\beta$ & $\beta^2$ & $\beta^3$ & $\beta^4$ \\
        $\beta^2$ & $\beta^3$ & $\beta^4$ & 0 \\
        $\beta^3$ & $\beta^4$ & 0 & 0 \\};

        \foreach \gridx in {0,3,...,12} {
            \pgfmathtruncatemacro{\gridxx}{\gridx+2}
            \draw[step=.5cm,color=gray] (\gridx,0) grid (\gridxx,2);
            \draw[step=.5cm,color=gray] (\gridx,-3) grid (\gridxx,-1);
        }

        \foreach \gridid in {m0,m1,m2,m3} {
            \foreach \row in {1,...,4} {
                \foreach \col in {1,...,4} {
                    \draw[arrow] (\gridid-\row-\col) -- ++ (0,.3cm);
                }
            }
        }

        \foreach \col in {1,3,4} {
            \foreach \row in {1,...,4} {
                \draw[arrow] (m4-\row-\col) -- ++ (0,.3cm);
            }
        }
        \foreach \row in {1,...,4} {
            \draw[arrow] (m4-\row-2) -- ++ (-.3cm,0);
        }

        \foreach \gridid in {m5,m6,m7,m8,m9} {
            \foreach \row in {2,3,4} {
                \foreach \col in {1,...,4} {
                    \draw[arrow] (\gridid-\row-\col) -- ++ (0,.3cm);
                }
            }
        }

        \foreach \gridid/\col in {m5/1,m5/2,m5/3,m5/4,m6/1,m6/3,m6/4,m7/1,m7/4,m8/1,m9/1} {
            \draw[arrow] (\gridid-1-\col) -- ++ (0,.3cm);
        }

        \foreach \gridid/\col in {m6/2,m7/2,m7/3,m8/2,m8/3,m8/4,m9/2,m9/3,m9/4} {
            \draw[arrow] (\gridid-1-\col) -- ++ (-.3cm,0);
        }

        \draw[->] (m0.east) -- (m1.west) node[above,midway]{$V$};
        \draw[->] (m1.east) -- (m2.west) node[above,midway]{$V$};
        \draw[->] (m2.east) -- (m3.west) node[above,midway]{$V$};
        \draw[->] (m3.east) -- (m4.west) node[below,midway]{$\pi$};
        \draw[->] (m4.east) -- ++ (.5cm,0) node[above,midway]{$\cdots$};

        \draw[->] (m5.east) -- (m6.west) node[above,midway]{$\pi$} node[below,midway]{$V$};
        \draw[->] (m6.east) -- (m7.west) node[above,midway]{$\pi$} node[below,midway]{$V$};
        \draw[->] (m7.east) -- (m8.west) node[above,midway]{$\pi$} node[below,midway]{$V$};
        \draw[->] (m8.east) -- (m9.west) node[above,midway]{$\pi$} node[below,midway]{$V$};
        \draw[->] (m9.east) -- ++ (.5cm,0) node[above,midway]{$\cdots$};

    \end{tikzpicture}
    \caption{Difference between policy iteration (above) and value iteration (below). The numbers in the cells are state values and the red arrows are the directions dictated by the policy at each stage. The arrows between grids indicate what kind of update the algorithm does, either value improvement ($V$) or policy improvement ($\pi$). Notice how policy iteration performs value improvement three times before updating the policy, whereas value iteration improves the value and the policy at each stage.}
    \label{fig:Gridworld}
\end{figure}

Starting with a policy which moves upwards in all states and a value function which rewards 1 only in the top left corner, a policy iteration algorithm would improve the value of the current policy until converging to the optimal values in the leftmost column, before updating the policy, while a value iteration algorithm would update the value function and also update the policy.

Take the finite set of positions as the state space $X$, and $A=\{\leftarrow,\rightarrow,\uparrow,\downarrow\}$ as the action space.

This example can be made stochastic if we add stochastic policies like $\epsilon$-greedy, where the action that the agent takes is the one with maximum value with probability $1 - \epsilon$ and a random one with probability $\epsilon$.
Another way is for the transition function to be stochastic, for example with a wind current that shifts the next state to the right with some probability $\epsilon$.

\subsection{Inverted pendulum example}

A task that illustrates a continuous state space MDP is the control of a pendulum balanced over a cart, which can be described in continuous-time exactly by two non-linear differential equations \cite[Example 2E]{control_systems_design}:
\begin{align*}
    (M+m)\ddot{y}+mL\ddot{\theta}\cos\theta-mL\dot{\theta}^2\sin\theta &= a \\
    mL\ddot{y}\cos\theta+mL^2\ddot{\theta}-mLg\sin\theta &= 0
\end{align*}
where $M$ is the mass of the cart, $m$ the mass of the pendulum, $L$ the length of the pendulum, $\theta$ the angle of the pendulum with respect to the upwards position, $y$ the carts horizontal position, $g$ the gravitational constant and $a$ is our control function (usually denoted $u$).
We rewrite the state variables as $x=[y,\dot{y},\theta,\dot{\theta}]^\top$.

Sampling the trajectory of continuous-time dynamics $\frac{\d}{\d t}x(t)=f(x(t))$ by $x_k=x(k\Delta t)$, one can define the discrete-time propagator $F_{\Delta t}$ by
\[ F_{\Delta t}(x(t))= x(t) + \int_t^{t+\Delta t} f(x(\tau)) \d\tau \]
which allows to model the system with $x_{k+1}=F_{\Delta t} (x_k)$.

A more common approach is to observe that the system of equations $\dot{x}=A(x)+B(x)a$ with $A$ and $B$ being non-linear functions of the state space, can be \emph{linearized} near a (not necessarily stable) equilibrium state, like the pendulum being in the upwards position.
There we can assume certain approximations like $\cos \theta\approx 1$ and $\sin\theta\approx \theta$, as well as small velocities leading to negligible quadratic terms $\dot{\theta}^2\approx 0$ and $\dot{y}^2\approx 0$.
This linearization around a fixpoint allows for the expression $\dot{x}=A x(t)+B a(t)$, where the matrix $A$ and vector $B$ are constants given by
\begin{align*}
    A &= \begin{pmatrix}
        0 & 1 & 0 & 0 \\
        0 & 0 & -\frac{mg}{M} & 0 \\
        0 & 0 & 0 & 1 \\
        0 & 0 & \frac{(M+m)g}{ML} & 0
    \end{pmatrix} & B &= \begin{pmatrix}
        0 \\
        \frac{1}{M} \\
        0 \\
        -\frac{1}{ML}
    \end{pmatrix}
\end{align*}

If we assume that the observation of the pendulum angle and cart position is discretized in time, an a priori time-discretization of this model using Euler approximation follows $x_{k+1}=x_k+\Delta t (A x_k+ B a_k)$, with the same constants, where $k$ indexes time steps.
Therefore we can say that the time-discretized, linearized model of the inverted pendulum over a cart follows a deterministic MDP for which a controller $u$ can be learned.
We take the state space as $\R^4$ and the action space of the force exerted to the cart as $\R$.

The time-discretised formulation of this problem is more common in reinforcement learning settings than in `classical' control theory. In that case, a a common payoff function is to obtain one unit of reward for each time step that the pendulum is maintained within a threshold of angles. The not linearized, not time-discretized setting, which is more common in optimal control theory, allows the reward, which is usually termed negatively as a cost function $J$, to have a much more flexible expression, in terms of time spent towards the equilibrium, energy spent to control the device, etc.
\[ J(x,a) = \int_0^\infty C(x(t),a(t)) \d t \]

\subsection{Savings problem example}

The \emph{savings problem} is one of the most important models in economics, modelling the dilemma between saving and consumption of resources \cite[part IV]{ljungqvist_sargent_recursive_macroeconomic_theory} (see also \cite{stokey_lucas_recursive_methods_economic_dynamics}). It is also mathematically closely related to the problem of charging a battery, for example choosing when to draw electricity from a power grid to raise the water level in a reservoir \cite{mody_steffen_optimal_charging_electric_vehicles}.

At each discrete time step $k$, an agent receives an income $i_k$. They also have a bank balance $x_k$, which accumulates interest over time (this could also be, for example, an investment portfolio yielding returns). At each time step the agent makes a choice of \emph{consumption}, which means converting their income into utility (or, more literally, things from which they derive utility). If the consumption in some stage is less than their income then the difference is added to the bank balance, and if it is more than the difference is taken from the bank balance. The dilemma is that the agent receives utility only from consumption, but saving gives the possibility of higher consumption later due to interest. The optimal balance between consumption and saving depends on the discount factor, which models the agent's preference between consumption now and consumption in the future.

In the most basic version of the model, all values can be taken as deterministic, and the income $i_k$ can also be taken as constant. This basic model can be expanded in many ways, for example with forecasts and uncertainty about income and interest rates. A straightforward extension, which we will consider in this paper, is that income is normally distributed $i \sim \mathcal N (\mu, \sigma)$, independently in each time step.

We take the state space and action space both as $X = A = [0, \infty)$. Given the current bank balance $x$ and consumption decision $a$, the utility in the current stage is $U (x, a) = \min \{ a, x + i \}$. (That is, the agent's consumption is capped by their current bank balance.) The state transition is given by $f (x, a) = \max \{ (1 + \gamma) x - a + i, 0 \}$, where $\gamma$ is the interest rate.

\section{Optics}

In this section we recall material on categories of mixed optics, mostly taken from \cite{roman_etal_profunctor_optics}.

\subsection{Categories of optics}

Given a monoidal category $\M$ and a category $\C$, an action of $\M$ on $\C$ is given by a functor $\bullet : \M \times \C \to \C$ with coherence isomorphisms $I \bullet X \cong X$ and $(M \otimes N) \bullet X \cong M \bullet (N \bullet X)$. $\C$ is called an $\M$-actegory.

Given a pair of $\M$-actegories $\C,\D$, we can form the category of optics $\Optic_{\C,\D}$. Its objects are pairs $\binom{X}{X'}$ where $X$ is an object of $\C$ and $X'$ is an object of $\D$. Hom-sets are defined by the coend
\[ \Optic_{\C,\D} \left( \binom{X}{X'}, \binom{Y}{Y'} \right) = \int^{M : \M} \C (X, M \bullet Y) \times \D (M \bullet Y', X') \]
in the category $\Set$. Such a morphism is called an optic, and consists of an equivalence class of triples $(M, f, f')$ where $M$ is an object of $\M$, $f : X \to M \bullet Y$ in $\C$ and $g : M \bullet Y' \to X'$ in $\D$. We call $M$ the residual, $f$ the forwards pass and $f'$ the backwards pass, so we think of the residual as mediating communication from the forward pass to the backward pass. Composition of optics works by taking the monoidal product in $\M$ of the residuals.

A common example is a monoidal category $\M = \C = \D$ acting on itself by the monoidal product, so
\[ \Optic_\C \left( \binom{X}{X'}, \binom{Y}{Y} \right) = \int^{M : \C} \C (X, M \otimes Y) \times \C (M \otimes Y', X') \]
This is the original definition of optics from \cite{Riley}. If $\C$ is additionally cartesian monoidal then we can eliminate the coend to produce \emph{concrete lenses}:
\begin{align*}
	\int^{M : \C} \C (X, M \times Y) \times \C (M \times Y', X') &\cong \int^{M : \C} \C (X, M) \times \C (X, Y) \times \C (M \times Y', X') \\
	&\cong \C (X, Y) \times \C (X \times Y', X')
\end{align*}
On the other hand, if $\C$ is monoidal closed then we can eliminate the coend in a different way to produce \emph{linear lenses}:
\begin{align*}
	\int^{M : \C} \C (X, M \otimes Y) \times \C (M \otimes Y', X') &\cong \int^{M : \C} \C (X, M \otimes Y) \times \C (M, [Y', X']) \\
	&\cong \C (X, [Y', X'] \otimes Y)
\end{align*}
Both of these proofs use the \emph{ninja Yoneda lemma} for coends \cite{loregian-coend-cofriend}.

\begin{example}
	Let $\Set$ act on itself by cartesian product. Optics $\binom{X}{X'} \to \binom{Y}{Y'}$ in $\Optic_\Set$ can be written equivalently as pairs of functions $X \to Y$ and $X \times Y' \to X'$, or as a single function $X \to Y \times (Y' \to X')$.
\end{example}

\begin{example}
	Let $\Euc$ be the category of Euclidean spaces and smooth functions, which is cartesian but not cartesian closed. Optics $\binom{X}{X'} \to \binom{Y}{Y'}$ in $\Optic_\Euc$ can be written as pairs of smooth functions $X \to Y$ and $X \times Y' \to X'$.
\end{example}

\begin{example}
	Let $\Mark$ be the category of sets and finite support Markov kernels, which is the kleisli category of the finite support probability monad $\Delta : \Set \to \Set$. It is a prototypical example of a Markov category \cite{Synthetic_approach}, and it is neither cartesian monoidal nor monoidal closed. Optics $\binom{X}{X'} \to \binom{Y}{Y'}$ in $\Optic_\Mark$ can only be written as optics, it is not possible to eliminate the coend. This is the setting used for Bayesian open games \cite{bolt_hedges_zahn_bayesian_open_games}.
\end{example}

\begin{example}
	Let $\Conv$ be the category of convex sets, which is the Eilenberg-Moore category of the finite support probability monad \cite{fritz09}. A convex set can be thought of a set with an abstract expectation operator $\mathbb E : \Delta X \to X$. Thus the functor $\Delta : \Mark \to \Conv$ given by $X \mapsto \Delta (X)$ on objects is fully faithful. $\Conv$ has finite products which are given by tupling in the usual way. $\Conv$ also has a closed structure: the set of convex functions $X \to Y$ themselves form a convex set $[X, Y]$ pointwise. However $\Conv$ is not cartesian closed: instead there is a different monoidal product making it monoidal closed \cite[section 2.2]{sturtz_categorical_probability} (see also \cite{kock_closed_categories_commutative_monads}). This monoidal product ``classifies biconvex maps'' in the same sense that the tensor product of vector spaces classifies bilinear maps. The embedding $\Delta : \Mark \to \Conv$ is strong monoidal for this monoidal product, not for the cartesian product of convex sets.

	We can define an action of $\Mark$ on $\Conv$, by $M \bullet X = \Delta (M) \otimes X$ \cite[section 5.5]{capucci_gavranovic_actegories}. Together with the self-action of $\Mark$, we get a category $\Optic_{\Mark,\Conv}$ given by
	\begin{align*}
		\Optic_{\Mark, \Conv} \left( \binom{X}{X'}, \binom{Y}{Y'} \right) &= \int^{M : \Mark} \Mark (X, M \otimes Y) \times \Conv (\Delta (M) \otimes Y', X') \\
		&\cong \int^{M : \Mark} \Mark (X, M \otimes Y) \times \Conv (\Delta (M), [Y', X'])
	\end{align*}
	(This coend cannot be eliminated because the embedding $\Delta : \Mark \to \Conv$ does not have a right adjoint.)

	This category of optics will be very useful for Markov decision processes, where the forwards direction is a Markov kernel and the backwards direction is a function involving expectations.
\end{example}

\subsection{Monoidal structure of optics}

A category of optics $\Optic_{\C,\D}$ is itself (symmetric) monoidal, when $\C$ and $\D$ are (symmetric) monoidal in a way that is compatible with the actions of $\M$. The details of this have been recently worked out in \cite{capucci_gavranovic_actegories}. The monoidal product on objects of $\Optic_{\C,\D}$ is given by pairwise monoidal product. All of the above examples are symmetric monoidal.

A monoidal category of optics comes equipped with a string diagram syntax \cite{hedges_coherence_lenses_open_games}. This has directed arrows representing the forwards and backwards passes, and right-to-left bending wires but not left-to-right bending wires. The residual of the denoted optic can be read off from a diagram, as the monoidal product of the wire labels of all right-to-left bending wires.
For example, a typical optic $(M, f, f') \in \Optic_\C \left( \binom{X}{X'}, \binom{Y}{Y'} \right)$ is denoted by the diagram
\begin{center}
	\tikzfig{generic_optic}
\end{center}

These diagrams have only been properly formalised for a monoidal category acting on itself, so for mixed optics we need to be very careful and are technically being informal.

Costates in monoidal categories of optics, that is optics $\binom{X}{X'} \to I$ (where $I = \binom{I}{I}$ is the monoidal unit of $\Optic_{\C,\D}$), are a central theme of this paper. When we have a monoidal category acting on itself, costates in $\Optic_\C$ are given by
\[ \Optic_\C \left( \binom{X}{X'}, I \right) = \int^{M : \C} \C (X, M \otimes I) \times \C (M \otimes I, X') \cong \C (X, X') \]
Thus \emph{costates in optics are functions}. A different way of phrasing this is by defining a functor $K : \Optic_\C^\op \to \Set$ given on objects by $K \binom{X}{X'} = \C (X, X')$, and then showing that $K$ is representable \cite{hedges_morphisms_open_games}. We will generally treat this isomorphism as implicit, sometimes referring to costates as though they are functions.

In the case of a cartesian monoidal category $\C$, given a concrete lens $f : X \to Y$, $f' : X \times Y' \to X'$ and a function $k : Y \to Y'$, the action of $K$ gives us the function $X \to X'$ given by $x \mapsto f' (x, k (f (x)))$.

When we have $\M = \C$ acting on both itself and on $\D$ (which includes all of the examples above) then similarly
\[ \Optic_{\C,D} \left( \binom{X}{X'}, I \right) = \int^{M : \C} \C (X, M \otimes I) \times \D (M \bullet I, X') \cong \D (X \bullet I, X') \]

\section{Dynamic programming with optics}

Given an MDP with state space $X$ and action space $A$, we can convert it to an optic $\binom{X \otimes A}{\R} \to \binom{X}{\R}$. The category of optics in which this lives can be `customised' to some extent, and depends on the class of MDPs that we are considering and how much typing information we choose to include. The definition of this optic is given by the following string diagram:
\begin{center}
	\tikzfig{mdp_optic}
\end{center}
To be clear, this diagram is not completely formal because we are making some assumptions about the category of optics we work in. In general, we require the forwards category $\C$ to be a Markov category (giving us copy morphisms $\Delta_X$ and $\Delta_A$), and the backwards category $\D$ must have a suitable object $\R$ together with morphisms $\times \beta : \R \to \R$ and $+ : \R \otimes \R \to \R$. Specific examples of interpretations of this diagram will be explored below. When the forwards category acts on the backwards category, then the forwards pass is a morphism $g : X \otimes A \to X \otimes X \otimes A$ in $\C$ where
\[ g = \Delta_{X \otimes A}\comp (f \otimes \id_{X \otimes A}) \]
and the backwards pass is a morphism $g' : X \bullet A \bullet \R \to \R$ in $\D$ encoding the function $g' (x, a, r) = \mathbb E U (x, a) + \beta r$.
The resulting optic is given by $\lambda = (X \otimes A, g, g') : \binom{X \otimes A}{\R} \to \binom{X}{\R}$ in $\Optic_{\C, \D}$.

Given a policy $\pi : X \to A$, we lift it to an optic $\overline\pi : \binom{X}{\R} \to \binom{X \otimes A}{\R}$, by
\begin{center}
	\tikzfig{pi_bar}
\end{center}
Here we are also assuming that the forwards category has a copy morphisms $\Delta_X$ (for example, because it is a Markov category), and the backwards category has a suitable object of real numbers. The interpretation of this diagram is the optic $(I, \Delta_X\comp (\id_X \otimes \pi), \id_\R)$.

\subsection{Discrete-space deterministic decision processes}\label{sec:4.1}

Consider a deterministic decision process with a discrete set of states $X$, discrete and finite set of actions $A$, transition function $f : X \times A \to X$, payoff function $U : X \times A \to \R$ and discount factor $\beta \in (0, 1)$. We convert this into an optic $\lambda = (X \times A, g, g') : \binom{X \times A}{\R} \to \binom{X}{\R}$ in $\Optic_\Set$, whose forwards pass is $g (x, a) = (x, a, f (x, a))$ and whose backwards pass is $g' (x, a, r) = U (x, a) + \beta r$.

Consider a dynamical system with the state space $A^X \times \Optic_\Set \left( \binom{X}{\R}, I \right)$. Elements of this are pairs $(\pi, V)$ of a policy $\pi : X \to A$ and a value function $V : X \to \R$. We can define two update steps:
\begin{align*}
	\text{\textbf{Value improvement:}}& &&(\pi, V) \mapsto (\pi, \overline\pi\comp \lambda\comp V) \\
	\text{\textbf{Policy improvement:}}& &&(\pi, V) \mapsto (x \mapsto \arg\max_{a \in A} (\lambda\comp V) (x, a), V)
\end{align*}
(We assume that $\arg\max$ is canonically defined, for example because $A$ is equipped with an enumeration so that we can always choose the first maximiser.)

Unpacking and applying the isomorphism between costates in lenses and functions, a step of value improvement replaces $V$ with
\[ V' (x) = U (x, \pi (x)) + \beta V (f (x, \pi (x))) \]
and a step of policy improvement replaces $\pi$ with
\[ \pi' (x) = \arg\max_{a \in A} U (x, a) + \beta V (f (x, a)) \]
Iterating the value improvement step converges to a value function which is the optimal value function for the current (not necessarily optimal) policy $\pi$.
A fixpoint of alternating steps of value improvement and policy improvement is a pair $(\pi^*, V^*)$ satisfying
\begin{align*}
	V^* (x) &= \max_{a \in A} (\lambda\comp V^*) (x, a) = \max_{a \in A} U (x, a) + \beta V^* (f (x, a)) \\
	\pi^* (x) &= \arg\max_{a \in A} (\lambda\comp V^*) (x, a) = \arg\max_{a \in A} U (x, a) + \beta V^* (f (x, a))
\end{align*}

\begin{example}
	[Gridworld example]
    A policy of an agent in our version of Gridworld (Figure \ref{fig:Gridworld}) is a function from the $4\times 4$ set of states $X = \{ 1, 2, 3, 4 \}^2$ that we index by $(i,j)$ to the four-element set of actions $A = \{ \leftarrow, \rightarrow, \uparrow, \downarrow \}$, i.e. an element of $A^X$.
    Initializing the value function $V$ with the environments immediate reward whose only non-zero value is $V(0,0)=1$ (top-left corner) and the policy with a upwards facing constant action $\pi(i,j)=\ \uparrow$ for all $(i,j)\in X$, a value improvement step would leave the policy unchanged while updating $V$ to $\overline{\pi}\comp\lambda\comp V$, which differs with $V$ only at $(0,1)\mapsto \beta$.

    If we instead perform a policy improvement step, the value function remains unchanged while the new policy differs with $\pi$ at $(1,0)\mapsto \arg\max_{a\in A}(\lambda\comp v)(1,0,a)=\ \leftarrow$.
\end{example}

\subsection{Continuous-space deterministic decision processes}

\begin{example}[Inverted pendulum]
    A state of our time-discretized inverted pendulum on a cart consists of $[y,\dot{y},\theta,\dot{\theta}]^\top$ in the state space $X=\R^4$.

    The linearized transition function that sends $x_k$ to $x_{k+1}=Ax_k+Ba_k$ is a smooth map $X\to Y$.

    The discretized cost $J(x,a)=\sum_{k=0}^\infty \beta^k C(x(k),a(k))$ defines the backwards smooth function $X\times A\times\R\to \R$ which adds the cost at the $k$th time step $C(x(k),a(k))$ to the discounted sum:
    \[ (x, a, r) \mapsto C (x, a) + \beta r \]
    These two maps form an optic $\binom{X\times A}{\R}\to\binom{X}{\R}$ in $\Optic_\Euc$. Note that the cost function $C$ is itself typically not affine, but rather convex (intuitively, since the `good states' that should minimise the cost fall in the middle of the state space).
\end{example}

This formalisation of the continuous state space misses however a practical problem.
Let $S$ be a continuous state space.
In numerical implementations, policy improvement over $S$ needs to map an action to every point in the space.
Two common approaches are to discretize the state space into a possibly non-uniform grid, or to restrict the space of values to a family of parametrized functions \cite[Sec.4.]{rust_numerical_dp_in_econ}.
The discretization approach treats the continuous state space as a distribution over a simplicial complex $X$ obtained e.g. by triangulation, $\Euc(1,S)\approx \Mark(I,X)$, where a continuous state gets mapped to a distribution over the barycentric coordinates of the simplex.
This effectively transforms the initial continuous-space deterministic decision process into a discrete-space MDP, modelling numerical approximation errors as stochastic uncertainty.

\subsection{Discrete-space Markov decision processes}

Consider a Markov decision process with a discrete set of states $X$, discrete and finite set of actions $A$, a transition Markov kernel $f : X \times A \to \Delta (X)$, expected payoff function $U : X \times A \to \R$ and discount factor $\beta \in (0, 1)$. We can write the transition function as conditional probabilities $f (x' \mid x, a)$.

We can convert this data into an optic $\lambda : \binom{X \otimes A}{\R} \to \binom{X}{\R}$ in the category $\Optic_{\Mark, \Conv}$ given by $\Mark$ acting on both itself and $\Conv$. This optic is given concretely by $(X \otimes A, g, g')$ where $g : X \otimes A \to X \otimes A \otimes X$ in $\Mark$ is given by $\Delta_{X\otimes A}\comp(f\otimes \id_{X\otimes A})$, and $g' : \Delta (X \otimes A) \to [\R, \R]$ in $\Conv$ is defined by $g' (\alpha) (r) = \mathbb E U (\alpha) + \beta r$, where $\alpha \in \Delta (X \times A)$ is a joint distribution on states and actions. Alternatively, we can note that the domain of $g'$ is free on the set $X \times A$ (although it cannot be considered free on an object of $\Mark$), and define it as the linear extension of $g' (x, a) (r) = U (x, a) + \beta r$.

With this setup, value improvement $(\pi, V) \mapsto (\pi, \overline\pi\comp \lambda\comp V)$ yields the value function
\[ V' (x) = \E_{a \sim \pi (x)} [U (x, a) + \beta V (f (x, a))] \]
Alternating steps of value and policy improvement converge to the optimal policy $\pi^*$ and value function $V^*$, which maximises the expected value of the policy:
\[ V^*_{\pi^*} (x_0) = \E \sum_{k = 0}^\infty \beta^k U (x_k, \pi^* (x_k)) \]

\begin{example}[Gridworld, continued]
    In a proper MDP, transition functions can be stochastic, and update steps have to take expectations over values: value improvement maps $(\pi, V) \mapsto (\pi, \overline\pi\comp \lambda\comp V)$ and policy improvement maps $(\pi, V) \mapsto (x \mapsto \arg\max_{a \in A} \E (\lambda\comp V) (x, a), V)$.
    This model also accepts stochastic policy improvement steps like $\epsilon$-greedy, which is an ad hoc heuristic technique of balancing exploration and exploitation in reinforcement learning \cite[Sec.2]{kaelbling_rl_survey}, a problem which is known in control theory as the identification-control conflict.
\end{example}

\subsection{Continuous-space Markov decision processes}

For continuous-space MDPs we need a category of continuous Markov kernels. There are several possibilities for this arising as the kleisli category of a monad, such as the Giry monad on measurable spaces \cite{giry82}, the Radon monad on compact Hausdorff spaces \cite{swirszcz_monadic_functors_convexity} and the Kantorovich monad on complete metric spaces \cite{fritz_perrone_probability_monad_colimit}. However, control theorists typically work with more specific parametrised families of distributions for computational reasons, the most common being normal distributions. We will work with the category $\Gauss$ of Euclidean spaces and affine functions with Gaussian noise \cite[section 6]{Synthetic_approach}. (This is an example of a Markov category that does not arise as the kleisli category of a monad, because its multiplication map would not be affine.) This works because the pushforward measure of a Gaussian distribution along an affine function is still Gaussian, which fails for more general functions.

\begin{example}
	We will formulate the savings problem with normally-distributed income. For purposes of example, it is convenient to ignore the restriction that the bank balance cannot become negative, since the binary $\max$ operator is nonlinear and hence does not preserve Gaussians under pushforward measure. We therefore work in the category $\Gauss$, and take the state and action spaces to both be $X = A = \R$. (This technically causes the optimisation problem to become unbounded and degenerate, since there is no reason for the agent to not consume an arbitrarily high amount in every stage, but we ignore this problem.)

	$\Gauss$ is a Markov category that is not cartesian (the monoidal product is the cartesian product of Euclidean spaces, which adds the dimensions), so it acts on itself by the monoidal product and we take the category $\Optic_\Gauss$. The transition function $f : \R \otimes \R \to \R$ is given by $f (x, a) = (1 + \gamma) x - a + \mathcal N (\mu, \sigma)$, and the payoff function $U : \R \otimes \R \to \R$ is given by $U (x, a) = a$.

	Since we do not have a corresponding Eilenberg-Moore category, the expectation over payoffs is not taken `automatically': a value improvement step $(\pi, V) \mapsto \overline\pi\comp \lambda\comp V$ will produce a function $V' : \R \to \R$ in $\Gauss$ with nonzero variance, even if $V$ always has zero variance.

	Finding a more appropriate categorical setting for this example (also allowing for the nonlinear $\min$ in payoffs to bound the optimisation problem) constitutes future work.
\end{example}

\section{Q-learning}

Consider a deterministic decision process corresponding to the optic $\lambda : \binom{X \times A}{\R} \to \binom{X}{\R}$.
The dynamical system with state space $A^X\times \Optic_\Set\left(\binom{X\times A}{\R},I\right)$ has elements $(\pi,q)$ consisting of a \emph{state-action} value function $q: X\times A\to \R$  as in \eqref{eq:q} rather than a state-value function $V : X \to \R$.

We can define similar update steps
\begin{align*}
	\text{\textbf{Value improvement:}}& &&(\pi, q) \mapsto (\pi, \lambda\comp\overline\pi\comp q) \\
	\text{\textbf{Policy improvement:}}& &&(\pi, q) \mapsto \left( x \mapsto \arg\max_{a \in A} q (x, a), q \right)
\end{align*}
These can also be fused into a single step:
\begin{align*}
	\text{\textbf{State-action value iteration:}} &&&(\pi, q) \mapsto \left( x \mapsto \arg\max_{a \in A} q (x, a), \lambda\comp \overline\pi\comp q \right)
\end{align*}
Observe that composition of the $\lambda$ optic with $\overline\pi$ is flipped compared to the case seen in Section \ref{sec:4.1}, as we want an element of $\Optic_\Set\left(\binom{X\times A}{\R},\binom{X\times A}{\R}\right)$ to compose with $q$.

The advantage of learning state-action value functions $X \otimes A \to \R$ rather than state-value functions $X \to \R$ is that is gives a way to approximate $\arg\max_{a \in A} (\lambda\comp \overline\pi\comp q) (x, a)$ without making any use of $\lambda$, namely by instead using $\arg\max_{a \in A} q (x, a)$. This leads to an effective method known as Q-learning for computing optimal control policies even when the MDP is unknown, with only a single state transition and payoff being sampled at each time-step. This is the essential difference between classical control theory and \emph{reinforcement learning}. The above method, despite learning a $q$-function, is \emph{not} Q-learning because it makes use of $\lambda$ during value improvement.

Q-learning \cite{q_learning} is a sampling algorithm that approximates the state-action value iteration, usually by a lookup table $Q$ referred as Q-matrix.
It treats the optic as a black box, having therefore no access to the transition or rewards functions used in \eqref{eq:q}, and instead updates $q$ by interacting with the environment dynamics:
\[ q'(x',a) = (1-\alpha)q(x,a)+\alpha(r+\beta \max_{a'}q(x',a')) \]
where $\alpha\in(0,1)$ is a weighting parameter.
Note that both the new state $x'$ and the reward $r$ are obtained by interacting with the system, rather than looked ahead by $x'=f(x,a)$ and $r=U(x,a)$.
It falls in the family of temporal difference algorithms.

\section{Further work}

At the end of the previous section, it can be seen that Q-learning is no longer essentially using the structure of the category of optics, instead treating the Q-function as a mere function. We believe this can be overcome using the framework of categorical cybernetics \cite{towards_foundations_categorical_cybernetics}, leading to a fully optic-based approach to reinforcement learning. By combining with other instantiations of the same framework, it is hoped to encompass the zoo of modern variants of reinforcement learning that have achieved spectacular success in many applications in the last few years. For example, deep Q-learning represents the Q-function not as a matrix but as a deep neural network, trained by gradient descent, allowing much higher dimensionality to be handled in practice. Deep learning is currently one of the main applications of categorical cybernetics \cite{cruttwell_etal_categorical_foundations_gradient_based_learning}.

The proof that dynamic programming algorithms converge to the optimal policy and value function typically proceed by noting that the set of all value functions form a complete ordered metric space and that value improvement is a monotone contraction mapping. The metric structure is used to prove that iteration converges to a unique fixpoint by the contraction mapping theorem, and then the order structure is used to prove that this fixpoint is indeed optimal. Since value improvement is optic composition, these facts would be a special case of the category of optics being enriched in the category of ordered metric spaces and monotone contraction mappings. We do not currently know whether such an enrichment is possible. Unlike costates, general optics have nontrivial forwards passes, so there are two possible approaches: either ignore the forwards passes and defining a metric only in terms of the backwards passes, or defining a metric also using the forwards passes, for example using the Kantorovich metric between distributions. This would also be a reasonable place to unify our approach with the coalgebraic approach with metric coinduction \cite{LTVMPDs}.

Finally, continuous time MDPs pose a serious challenge to any approach for which categorical composition is sequencing in time, since composition of two morphisms in a category appears to be inherently discrete-time. (Open games are similarly unable to handle dynamic games with continuous time, such as pursuit games.) A plausible approach to this is to associate an endomorphism in a category to every real interval, by treating that interval of time as a single discrete time-step, and then requiring that all morphisms compose together correctly, similar to a sheaf condition. It is hoped that the Bellman-Jacobi-Hamilton equation, a PDE that is the continuous time analogue of the discrete-time Bellman equation, will simiarly arise as a fixpoint in this way. Exploring this systematically is important future work.

\bibliographystyle{eptcs}