# Maximum Likelihood and MAP continued...

## MLE of Mean of Gaussian

*  Let $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_N$ be samples from a multi-variance Normal distribution with known covariance matrix and an unknown mean.  Given this data, obtain the ML estimate of the mean vector. 
	\begin{equation}
	p(\mathbf{x}_k| {{\mu}}) = \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_k - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_k - {\mu})\right)
	\end{equation}
* We can define our likelihood given the $N$ data points.  We are assuming these data points are drawn independently but from an identical distribution (i.i.d.):
	\begin{equation}
	\prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)
	\end{equation}
*  We can apply our "trick" to simplify
	\begin{eqnarray}
	\mathscr{L} &=& \ln \prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \ln \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \left( \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}} + \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right) \right) \\
	&=&  - N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right) 
	\end{eqnarray}
* Now, lets maximize:
	\begin{eqnarray}
	\frac{\partial \mathscr{L}}{\partial \mu} &=& \frac{\partial}{\partial \mu} \left[- N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right)\right] = 0 \\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}(\mathbf{x}_n - {\mu}) = 0\\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}\mathbf{x}_n  = \sum_{n=1}^N  \Sigma^{-1} {\mu}\\
	&\rightarrow& \Sigma^{-1} \sum_{n=1}^N \mathbf{x}_n  = \Sigma^{-1} {\mu} N\\
	&\rightarrow& \sum_{n=1}^N \mathbf{x}_n  = {\mu} N\\
	&\rightarrow& \frac{\sum_{n=1}^N \mathbf{x}_n}{N} = {\mu}\\
	\end{eqnarray}
* So, the ML estimate of $\mu$ is the sample mean!


## MAP of Mean of Gaussian

* To get a MAP estimate of the mean of a Gaussian, we apply a prior distribution and maximize the posterior. 
* Lets use a Gaussian prior on the mean (because it has a *conjugate prior* relationship)

\begin{eqnarray}
p(\mu|X, \mu_0, \sigma_0^2, \sigma^2) &\propto& \mathscr{N}(X|\mu, \sigma^2)\mathscr{N}(\mu|\mu_0, \sigma_0^2)\\
&=& \prod_{n=1}^N \left(\frac{1}{\sqrt{2\pi \sigma^2}} \exp\left\{-\frac{1}{2\sigma^2}\left(x_n-\mu\right)^2 \right\}\right)\frac{1}{\sqrt{2\pi \sigma_0^2}} \exp\left\{-\frac{1}{2\sigma_0^2}\left(\mu-\mu_0\right)^2 \right\}\nonumber\\
\mathscr{L} &=& -\frac{N}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{n=1}^N(x_n-\mu)^2 - \frac{1}{2}\ln(2\pi \sigma_0^2) - \frac{1}{2\sigma_0^2}(\mu - \mu_0)^2\\
\frac{\partial \mathscr{L}}{\partial \mu} &=& \frac{1}{\sigma^2}\sum_{n=1}^N x_n - \frac{N}{\sigma^2}\mu - \frac{1}{\sigma_0^2}\mu + \frac{1}{\sigma_0^2}\mu_0 = 0\\
\mu\left(\frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2} \right) &=& \frac{1}{\sigma^2}\sum_{n=1}^N x_n + \frac{1}{\sigma_0^2}\mu_0 \\
\mu_{MAP} &=& \frac{\sigma_0^2}{N\sigma_0^2 + \sigma^2}\sum_{n=1}^N x_n + \frac{\mu_0\sigma^2}{N\sigma_0^2 + \sigma^2}
\end{eqnarray}

* *Does this result make sense?*


```python

```

b9ae2dc50aff00022d6006f921eb970f05614f38
ipynb
Jupyter Notebook
Lecture04_MLandMAPcont/Lecture 04 .ipynb
chenshenlv/LectureNotes
6febeec7b93f915ba4608fe85ee13364e03db73b
MIT
2020-03-17T19:11:19.000Z
2021-02-16T06:04:02.000Z
Lecture04_MLandMAPcont/Lecture 04 .ipynb
chenshenlv/LectureNotes
6febeec7b93f915ba4608fe85ee13364e03db73b
MIT
2018-10-02T16:11:44.000Z
2018-10-23T17:05:00.000Z
Lecture04_MLandMAPcont/Lecture 04 .ipynb
chenshenlv/LectureNotes
6febeec7b93f915ba4608fe85ee13364e03db73b
MIT
2020-03-18T00:29:54.000Z
2020-07-23T05:00:04.000Z
Qwen/Qwen-72B
1. YES
2. YES