I'm trying to get my head around online (recursive) maximum-likelihood parameter estimation in the language of stochastic processes and in the context of stochastic filtering, i.e. where we have a partially observed Ito diffusion process
\begin{align}
dX\_t&=f(t,\theta\_0,X\_t)dt+g(t,\theta\_0,X\_t)dW\_t, \\
dY\_t&=h(t,\theta\_0,X\_t)dt+dV\_t.
\end{align}
Here, everything is scalar, $X\_t$ is the (unobserved) signal, $Y\_t$ is the observation, $W\_t$ and $V\_t$ are independent standard Wiener processes, and $\theta\_0$ is a parameter. Suppose we don't know $\theta\_0$ and we want to estimate it recursively as observations $Y\_t$ come in.


Moura & Mitter (1986) give a formula for the log likelihood function for parameter estimation based on only observations of $Y\_t$, namely
\begin{equation}
L\_t(\theta)=\frac{1}{r}\int\_0^t\left ( \hat h\_s dY\_s-\frac{1}{2} \hat h^2\_s ds \right), \quad \hat h\_t=E \left[ h(t,\theta,X\_t)|\mathcal{F}^Y\_t \right].
\end{equation}
First of all, it is not entirely clear why the above function would be the correct thing to look at, and the paper mentioned above does not really help me with a lot of detail. What I can guess so far is that they use a sequence of changes of measures, ending up with one (let's call it $\mathcal{Q}\_{\theta}$) under which the innovations process $n\_t$, defined as
\begin{equation}
dn\_t=dY\_t-\hat h\_t dt,
\end{equation}
is a Wiener process. By Girsanov's theorem, the above log likelihood function is the log of the Radon-Nikodym (RN) derivative $d\mathcal{Q}\_{\theta}/d\tilde{\mathcal{P}}\_{\theta}|\_{\mathcal{F}\_t}$ of that measure with respect to another measure $\tilde{\mathcal{P}}\_{\theta}$ under which $Y\_t$ is a Wiener process (is this correct?). However, what I would want to maximize looks something like
\begin{equation}
L'\_t(\theta)=\log \frac{d\mathcal{P}^Y\_{\theta}}{d\mathcal{P}^Y\_0},
\end{equation}
i.e. the RN derivative of the measure of the process $Y\_t$, with $X\_t$ 'integrated' or 'marginalized' out, with respect to a parameter-independent reference measure $\mathcal{P}^Y\_0$. So my first question is: why should $L\_t$ be the log likelihood function instead of $L'\_t$, or alternatively, why is it equivalent to maximize $L'\_t$?




---


For my second question, let's pretend that we understand why $L\_t$ is the log likelihood function, and let's maximize it. For simplicity, I assume that everything is linear,
\begin{align}
dX\_t&=-X\_tdt+dW\_t, \\
dY\_t&=\theta X\_tdt+dV\_t,
\end{align}
such that 
\begin{equation}
L\_t(\theta)=\frac{1}{r}\int\_0^t\left ( \theta\mu\_s dY\_s-\frac{1}{2} \theta^2(\mu\_s)^2 ds \right), 
\end{equation}
where $\mu\_t$ is the mean of the Kalman-Bucy filter
\begin{align}
d\mu\_t&=-\mu\_tdt+\theta \sigma^2\_t(dY\_t-\theta\mu\_tdt), \\
\frac{d\sigma^2\_t}{dt}&=1-2\sigma^2\_t-\theta^2\sigma^4\_t,
\end{align}
which is integrated using the parameter $\theta$, acquiring an implicit parameter dependence. To maximize $L\_t$ for fixed $t$, we could use a steepest ascent algorithm, i.e.
\begin{equation}
\begin{split}
\frac{d}{d\tau}\hat \theta\_{\text{ML,offline}}(\tau)&=\eta \partial\_{\theta}L\_t(\theta)\Big|\_{\theta=\hat \theta\_{\text{ML,offline}}(\tau)}\\
&=\frac{\eta}{r}\int\_0^t\left(dY\_s-\theta\mu\_sds\right)\left(\mu\_s+\theta\nu\_s\right)\Big|\_{\theta=\hat \theta\_{\text{ML,offline}}(\tau)}.
\end{split}
\end{equation}
Here, $\tau$ is the parameter of the offline gradient ascent, i.e. while the process $(X,Y)$ is stopped (observations from 0 to $t$ have been collected and stored), the gradient ascent is initialized at $\tau=0$ with an initial guess $\theta\_{\text{ML,offline}}(0)$, and then updated according to the differential equation above until it converges. We also introduced a learning rate $\eta>0$ and the derivative of the mean of the Kalman-Bucy filter, $\nu\_t=\partial\_{\theta}\mu\_t$, for which we find
\begin{align}
d\nu\_t&=-\nu\_tdt+(\sigma^2\_t+\theta\rho^2\_t)(dY\_t-\theta\mu\_tdt)-\theta\sigma^2\_t(\mu\_t+\theta\nu\_t)dt, \quad &\nu\_0=0,\\
\frac{d\rho^2\_t}{dt}&=-2\rho^2\_t-2\theta^2\sigma^2\_t\rho^2\_t-2\theta\sigma^4\_t\quad &\rho^2\_0=0.
\end{align}
Finally, we could turn the above gradient-based update equation into a recursive update rule by getting rid of the integral:
\begin{equation}
d\hat\theta\_t=\frac{\eta}{r}\left(dY\_t-\hat\theta\_t\mu\_tdt\right)\left(\mu\_t+\hat\theta\_t\nu\_t\right),
\end{equation}
a rule which works well enough in practice (simulations). The problem is reconciling this with another derivation one could come up with. Let's discretize the Ito SDEs above:
\begin{align}
X\_i&=X\_{i-1}-\delta t X\_{i-1}+\sqrt{\delta t}\xi\_i, \\
Y\_i&=Y\_{i-1}+\theta\delta t X\_{i-1}+\sqrt{\delta t}\eta\_i,
\end{align}
where $\xi\_i,\eta\_i$ are sampled i.i.d from $\mathcal{N}(0,1)$. This is the Euler-Maruyama discretization. We can write down a log likelihood function in discrete time:
\begin{align}
L^D\_n(\theta)&=\log P(\mathbf{Y}|\theta), \\
P(\mathbf{Y}|\theta)&=\int d\mathbf{X}\, P(\mathbf{X},\mathbf{Y}|\theta), \\
P(\mathbf{X},\mathbf{Y}|\theta)&=P(X\_1)P(Y\_1)\prod\_{i=2}^n P(X\_i|X\_{i-1}) P(Y\_i|X\_{i-1},Y\_{i-1},\theta).
\end{align}
Taking the derivative, we obtain
\begin{equation}
\begin{split}
\partial\_{\theta}L^D\_n(\theta)&=\frac{1}{P(\mathbf{Y}|\theta)}\int d\mathbf{X}\,\partial\_{\theta} P(\mathbf{X},\mathbf{Y}|\theta) \\
&=\int d\mathbf{X}\, P(\mathbf{X}|\mathbf{Y},\theta)\partial\_{\theta}\log P(\mathbf{X},\mathbf{Y}|\theta).
\end{split}
\end{equation}
Now, using the structure of the joint distribution in the integrand, and the explicit form of the Gaussian distribution, one ends up with
\begin{equation}
\partial\_{\theta}\log P(\mathbf{X},\mathbf{Y}|\theta) = \sum\_{i=2}^n \left[X\_{i-1}(Y\_i-Y\_{i-1})-\theta\delta t X^2\_{i-1} \right],
\end{equation}
which, after taking the continuum limit, expectations over $P(\mathbf{X}|\mathbf{Y},\theta)$ and getting rid of the integral, seems to lead to an update rule
\begin{equation}
d\hat\theta\_t\propto \left[\mu\_t dY\_t-\theta(\mu^2\_t+\sigma^2\_t)dt \right].
\end{equation}
So the second question (sorry for being so long-winded) is: Why are the two rules different? Note that they are fairly similar, with the only differences being that the first rule has expectations under the square, and the second one expectation of the square, and the second one does not contain filter derivatives. Is there a flaw in one of these derivations? Or is it to be expected that we get different rules because the problem is not well-posed?



Recursive parameter estimation for partially observed Ito SDEs
The formulation of a likelihood function for a continuous-time stochastic process requires the choice of a reference measure. In the context of parameter estimation, the reference measure has to be parameter-independent. In the case of partially observed diffusion processes with additive observation noise, the innovation process is a Brownian motion adapted to the observation filtration, and consequently there is a reference measure under which $Y\_t$ is a Brownian motion. 


The big miracle is that the Radon-Nikodym derivative gives us an expression of the likelihood function in terms of the filter, which makes it perfect for online parameter estimation. 


The full story, including numerical examples, is written up in the following article:
[Online Maximum Likelihood Estimation of the Parameters of Partially Observed Diffusion Processes](https://arxiv.org/abs/1611.00170)


