Suppose we have a parameter $\theta \in R^{n}$ that defines some noisy observation $z=\mu(\theta)+\eta, z\in R^{m}$ where the noise follows a Gaussian distribution whose covariance is a function of the parameter, i.e. $\eta\sim N(0,\Sigma(\theta))$. Then the observation random variable is $z \sim N(\mu(\theta),\Sigma(\theta))$. Assume observations of $z$ are iid and $\Sigma=diag(\sigma\_1^{2}(\theta), \sigma\_2^{2}(\theta)...\sigma\_m^{2}(\theta))$, where $\sigma\_i^{2}\ne\sigma\_j^{2}$.


Then, the log-likelihood function would be:


\begin{align}
L\_z\left(z;\theta\right)&=log\left(\prod^m\_{j=1}\frac{1}{\sqrt{2 \pi \sigma\_j^2\left(\theta\right)}} e^{-\frac{\left(z\_j-\mu\_j(\theta)\right)^2}{2 \sigma\_j^2\left(\theta\right)}}\right)\\
&=\sum\_{j=1}^{m} -\frac{1}{2}log(2\pi)-\frac{1}{2} log(\sigma\_{j}^{2}(\theta))-\frac{(z\_j-\mu\_j(\theta))^{2}}{2\sigma\_j^{2}(\theta)}\\
\end{align}


The first derivative of the log-likelihood function w.r.t the parameters would be:


\begin{aligned}
\frac{\partial L}{\partial \theta}= \sum\_{j=1}^{m}-\frac{1}{2}\left(\sigma\_j^2(\theta)\right)^{-1} \frac{\partial \sigma\_j^2}{\partial \theta}-(z\_j-\mu\_j(\theta)) \frac{\partial \mu\_j}{\partial \theta}\left(\sigma\_j^2(\theta)\right)^{-1}-\frac{1}{2}\left(\sigma\_j^2(\theta)\right)^{-2} (z\_j-\mu\_j(\theta))^2 \frac{\partial \sigma\_j^2(\theta)}{\partial \theta}
\end{aligned}


What I'm getting confused about is: how would the log-likelihood function derivative be zero at the parameter value, when the first term of the derivative does not depend on $(z\_j-\mu\_j(\theta))$?



Derivative of log-likelihood function for Gaussian distribution with parameterized variance
$\newcommand\th\theta\newcommand\si\sigma\newcommand\p\partial\newcommand\ol\overline$There is no reason to get confused here. Indeed, that "the first term of the derivative does not depend on $(z\_j-\mu\_j(\theta))$" does not at all prevent the derivative from taking the zero value.


If e.g. $\mu(\th)=\th$ and $\si\_i(\th)=\th$ for all real $\th>0$ and all $i=1,\dots,m$, then
$$\frac{\p L}{\p\th}=-\frac n{\th^3}\,(-\ol{z^2}+\th\ol z+\th^2),$$
where $\ol z:=\frac1m\,\sum\_{i=1}^m z\_i$ and $\ol{z^2}:=\frac1m\,\sum\_{i=1}^m z\_i^2$, so that $\frac{\p L}{\p\th}=0$ at a real $\th>0$ if and only if
$$\th=\hat\th:=\hat\th\_m:=\frac{-\ol z+\sqrt{\ol z^2+4\ol{z^2}}}2.$$


Moreover, here $\hat\th$ is the maximum likelihood estimator of $\th$. Using (say) the law of large numbers, one can easily check that $\hat\th\_m$ is consistent: $\hat\th\_m\to\th$ in probability as $m\to\infty$ (assuming that $\th$ is the true value of the parameter).


