There is a supermartingale convergence theorem which is often cited in texts which use Stochastic Approximation Theory and Reinforcement Learning, in particular the famous book "Neuro-dynamic Programming" the theorem is:


"Let $Y\_t, X\_t, Z\_t, t = 1,2,3,....$ be three sequences of random variables and let $\mathcal{F\_t}$ be sets of random variables such that $\mathcal{F\_t} \subset \mathcal{F\_{t+1}}$ for all t, suppose that:


(a) The random variables $Y\_t, X\_t, Z\_t$ are nonnegative and are functions of the random variables in $\mathcal{F}\_t$


(b) For each $t$ we have $E[Y\_{t+1}|\mathcal{F\_t}] \leq Y\_t - X\_t +Z\_t$


(c) $\sum\_{t=0}^\infty Z\_t \lt \infty$ 


Then:


$\sum\_{t=0}^{\infty}X\_t \lt \infty $ and there exists a nonnegative random variable $Y$ such that $Y\_t \rightarrow Y$ with probability 1."


The problem is, I can't find any proofs of this anywhere. Most texts on probability theory prove the standard Martingale Convergence Theorem but I feel this is a big step from that.


Can anyone direct me to a source which proves the above or write out a proof from the standard convergence theorem? 



Proof of extended supermartingale convergence theorem
Here's one approach. 


First notice that 
$$
R\_t: = Y\_{t} + \sum\_{i=1}^{t-1} X\_i - \sum\_{i=1}^{t-1} Z\_i
$$
is a supermartingale, since
$$
R\_{t+1} - R\_t = Y\_{t+1} - Y\_t + X\_t - Z\_t, 
$$
giving
$$
E(R\_{t+1} - R\_t | \mathcal{F}\_t) = E (Y\_{t+1} | \mathcal{F}\_t) - Y\_t + X\_t - Z\_t 
$$
which is less than or equal to $0$ with probability $1$, by (b). 


Now, we don't have a fixed lower bound for the supermartingale $R$, so we can't apply the convergence theorem directly. 


However, for any $a>0$, consider the stopping time
$$
\tau\_a = \inf\{t: \sum\_{i=1}^t Z\_i>a\},
$$
with $\tau\_a=\infty$ if $\sum\_{i=1}^t Z\_i\leq a$ for all $t$.


We can define
$$
R^{(a)}(t):=R(t\wedge \tau\_a)=
\begin{cases}
R\_t&\text{ if }t<\tau\_a\\
R\_{\tau\_a}&\text{ if }t\geq \tau\_a
\end{cases}.
$$


$R^{(a)}$ is also a supermartingale for any $a$, and $R^{(a)}(t)$ is bounded below by $-a$. 


So by the martingale convergence theorem, for any given $a$, $R^{(a)}(t)$ converges to some finite limit with probability $1$. By countable additivity, 
we get that with probability $1$, $R^{(a)}(t)$ converges to a finite limit for all $a\in\mathbb{Z}$. 


But if $\sum\_{i=0}^\infty Z\_i<\infty$, 
which from (c) we assume happens with probability $1$, then for all large enough $a\in\mathbb{Z}$,
we have $\tau\_a=\infty$, and so $R^{(a)}(t)=R(t)$ 
for all $t$. Since we know $R^{(a)}(t)$ converges, we also get that $R(t)$ converges.


Finally, since $R(t)$ converges and $\sum\_{i=0}^{t-1} Z\_i$ converges, we also have that $Y\_t+\sum\_{i=1}^{t-1} X\_i$ converges. Since $\sum\_{i=1}^{t-1} X\_i$ is non-decreasing in $t$, and $Y\_t$ is non-negative for all $t$, the only way this can happen is if $Y\_t$ and $\sum\_{i=1}^{t-1} X\_i$ both converge, as required. 


