### **Claim**

In the limit, maximizing the likelihood of a parametric distribution whose family contains the true unknown distribution is equivalent to minimizing the forward Kullback-Leibler divergence between them.

---

Let $P(X)$ be an unknown distribution producing observations $x$ with a pdf $p(x)$. We wish to approximate the pdf $p(x)$ of with a parametric family $q_{\theta}(x)$. We assume that the true distribution is within the family described by $q_{\theta}(x)$. 


The parameters $\theta_{ML}$ of the approximating distribution $q_{\theta_{ML}}$ are defined with the maximum likelihood estimator

$$\theta_{ML} = \max_{\theta} \frac{1}{m}\sum_{i=1}^m \log q_{\theta}(x_i)$$

where $x_i \sim P$ and $i \in \{1, \dots, m\}$. 


The approximating distribution $q_{\theta_{KL}}$ is defined by minimizing the KL divergence between $p(x)$ and $q_{\theta}(x)$.

$$\theta_{KL} = \min_{\theta} KL(P||Q_{\theta}) = \min_{\theta} \int_x p(x) \log \frac{p(x)}{q_{\theta}(x)} dx$$

where here we use the forward KL divergence.


If we take the limit as $m \rightarrow \infty$ then $\theta_{ML}$ found by maximizing the likelihood is the same as $\theta_{KL}$ found by minimizing the forward KL divergence.

\begin{align}
\lim_{m \rightarrow \infty} \theta_{ML} &= \lim_{m \rightarrow \infty} \max_{\theta} \frac{1}{m}\sum_{i=1}^m \log q_{\theta}(x_i) \\
&= \max_{\theta} \int_x p(x) \log q_{\theta}(x) dx \\
&= \min_{\theta} \bigg [ -\int_x p(x) \log q_{\theta}(x) dx \bigg] \\
&= \min_{\theta} \bigg [ \int_x p(x) \log p(x) dx -\int_x p(x) \log q_{\theta}(x) dx \bigg] \\
&= \min_{\theta} \bigg [ \int_x p(x) \log \frac{p(x)}{q_{\theta}(x)} dx \bigg] \\
&= \min_{\theta} KL(P||Q_{\theta}) \\
&= \theta_{KL}
\end{align}


```python

```

94a60cfebad933a71a107e7b11412e60fd218dad
ipynb
Jupyter Notebook
Maximizing Liklihood and Minimizing KL Divergence.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
2019-12-04T11:04:45.000Z
2019-12-04T11:04:45.000Z
Maximizing Liklihood and Minimizing KL Divergence.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
Maximizing Liklihood and Minimizing KL Divergence.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
Qwen/Qwen-72B
1. YES
2. YES