### Analytic Solutions to Variational Bayes

The general setting for variational Bayesian techniques begins from Bayesian inference, where we wish to infer a posterior.

\begin{align}
p(z|x) = \frac{p(x|z)p(z)}{p(x)}
\end{align}

Often times the marginal $p(x)$ is too complicated to derive analytically or calcuate computationally. In this case, we call it 'intractable'. When this happens we may wish to approximate the posterior $p(z|x)$. A common technique involves determining a surrogate distribution $q(z|x)$ that is as close as possible to the true posterior. We can measure the closeness between two distributions with the well known Kullback Leibler (KL) divergence. In this case we use the reverse KL divergence.

\begin{align}
D_{\text{KL}}(q(z|x)||p(z|x)) = \int_z q(z|x) \log{\frac{q(z|x)}{p(z|x)}} \text{d}z
\end{align}

However, posing the problem this way has not helped us yet as the KL retains the unknown posterior within it. In other words, we need the thing we are trying to approximate, to approximate it. There is a fix. Let us rewrite the KL term as follows...

\begin{align}
D_{\text{KL}}(q(z|x)||p(z|x)) &= \int_z q(z|x) \log{\frac{q(z|x)}{p(z|x)}} \text{d}z \\
&= \int_z q(z|x) \log{\frac{q(z|x)p(x)}{p(z|x)p(x)}} \text{d}z \\
&= \int_z q(z|x) \log{\frac{q(z|x)}{p(x,z)}} \text{d}z + \int_z q(z|x) \log{p(x)} \text{d}z \\
&= \int_z q(z|x) \log{\frac{q(z|x)}{p(x,z)}} \text{d}z + \log{p(x)} \\
\end{align}

We wish to make the right hand side as small as possible. Recall that the data generating distribution $p(x,z)$ is considered fixed and $x$ is considered given, or observed, therefore $\log{p(x)}$ is a constant. Minimizing the right hand side then amounts to minimizing the integral where we've replaced the posterior with the joint which is more often available. However, before we move on let us make a quick observation from information theory. As Claude Shannon demonstrated, KL divergence is greater than or equal to zero, therefore we know that 

\begin{align}
\int_z q(z|x) \log{\frac{q(z|x)}{p(x,z)}} \text{d}z + \log{p(x)} &\ge 0 \\
\log{p(x)} &\ge -\int_z q(z|x) \log{\frac{q(z|x)}{p(x,z)}} \text{d}z \\
\log{p(x)} &\ge \mathcal{L}[q]
\end{align}
We see that the functional $\mathcal{L}[q]$ is a lower bound for the marginal likelihood, or model evidence. For this reason $\mathcal{L}$ is often referred to as the *evidence lower bound* or *ELBO* for short. We will adopt this nomenclature.

Our initial objective was to minimize the reverse KL divergence between the true posterior and our surrogate distribution $q(z|x)$. We have now reformulated that problem to maximizing the ELBO. As posed thus far, the maximum for $\mathcal{L}$ is when $\mathcal{L} = \log{p(x)}$, in which case the KL divergence would be zero and the optimal variational distribution will equal the true posterior $q(z|x) = p(z|x)$. This is rarely possible, so assumptions are made regarding $q(z|x)$ that constrain it to a family of distributions which simplifies the analysis. Under such assumptions, the distribution $q$ is no longer free to take any form and we can not guarantee that our optimal $q^*(z|x)$ equals the posterior. It is in this sense that our technique becomes an approximation. Albeit, there are times when the true posterior lives within the family of our variational distribution $q(z|x)$ and an exact solution is possible. The larger the family, the more likely this is to happen. Though, as the family increases in size, the complexity of the problem often does as well. Therefore, one might consider variational inference as the art of maximizing the expressive power of your model while minimizing the complexity of the resulting analysis. In any case, we often have to settle for a reasonable approximation to the posterior, in the KL sense. 

### Mean Field Model

One simplifying assumption that is made to constrain the variational family is that the distribution $q(z|x)$ factorizes over the latent variables $q(z|x) = q(z_1|x)q(z_2|x)\cdots q(z_n|x)$. The assumption is often referred to as the *mean field* approximation, inspired by physics. 

**(expand on this in the future with your seminar notes)**

### Parametric Families

Another technique, sometimes used in conjuction with the former, is to assume a parametric family for our variational distribution $q_{\phi}(z|x)$, parameterized by $\phi$. The ELBO simplifies to a function of the variational parameters $\mathcal{L}(\phi)$. This can sometimes be solved analytically but is most often approached computationally. In this form standard stochastic optimization techniques can be applied.

\begin{align}
\mathcal{L}(\phi;x) &= -\int_z q_{\phi}(z|x)  \log{\frac{q_{\phi}(z|x)}{p(x,z)}} \text{d}z \\
&\approx \frac{1}{L} \sum_i^L \log{p(x,z_i)} - \log{q_{\phi}(z_i|x)}
\end{align}

where $z_i \sim q_{\phi}(z|x)$. From here, one takes the pathwise derivative of the ELBO with respect to the variational parameters $\phi$ and optimizes with different samples $x \sim p(x)$.

### Variational Autoencoders

For variational autoencoders we rewrite the ELBO as follows

\begin{align}
\mathcal{L}[q] &= -\int_z q(z|x) \log{\frac{q(z|x)}{p(x,z)}} \text{d}z \\
&= \int_z q(z|x) \log{\frac{p(x|z)p(z)}{q(z|x)}} \text{d}z \\
&= -D_{\text{KL}}(q(z|x)||p(z)) + \mathbb{E}_{q(z|x)} \big[ \log{p(x|z)} \big]\\
\end{align}

By specifying different assumptions about $q(z|x)$, $p(x|z)$ and $p(z)$ we will arrive at different algorithms. For example if we choose parametric forms amenable to analysis we can derive closed form solutions for the lower bound. These simplifications remove the need for Monte Carlo estimation. Let us examine the most common example. We begin by parameterizing the encoder $q_{\phi}(z|x)$ as a diagonal covariance Gaussian $\mathcal{N}(z|x; \mu,\sigma^2 \cdot I)$ where $\sigma^2 \in \mathbb{R}^N$ and assume $p(z)$ is a centered isotropic Gaussian $p(z) = \mathcal{N}(z; 0,I)$ (i.e. parameterless). 

\begin{align}
q_{\phi}(z|x) &= (2\pi )^{-\frac{N}{2}} \big( \prod_i \sigma_i^2 \big )^{-\frac{1}{2}} \text{exp}\big\{-\frac{1}{2}(z-\mu)^T(\sigma^2 \cdot I)^{-1}(z-\mu)\big\} \\
\end{align}
\begin{align}
p(z) &= (2\pi )^{-\frac{N}{2}} \text{exp}\big\{-\frac{z^T z}{2}\} \\
\log{p(z)} &= -\frac{N}{2}\log{2\pi} -\frac{z^T z}{2} \\
\end{align}

With these definitions we can evaluate the negative KL term as

\begin{align}
-D_{\text{KL}}(q_{\phi}(z|x)||p(z)) &= \int_z q_{\phi}(z|x) \log{\frac{p(z)}{q_{\phi}(z|x)}} \text{d}z \\
&= \int_z q_{\phi}(z|x) \log{p(z)} \text{d}z - \int_z q_{\phi}(z|x) \log{q_{\phi}(z|x)} \text{d}z \\
&= \int_z \bigg((2\pi )^{-\frac{N}{2}} \big( \prod_i \sigma_i^2 \big )^{-\frac{1}{2}} \text{exp}\big\{-\frac{1}{2}(z-\mu)^T(\sigma^2 \cdot I)^{-1}(z-\mu)\big\}\bigg)\bigg(-\frac{N}{2}\log{2\pi} -\frac{z^T z}{2}\bigg)\text{d}z + H\big[q_{\phi}(z|x)\big]
\end{align}

where we have identified the term on the right as the entropy $H$ which has a known closed form solution for the multivariate Gaussian case.

\begin{align}
H\big[\mathcal{N}(x; \mu,\Sigma)\big] &= \frac{1}{2} \log{\text{det}(2 \pi e \Sigma)} \\
\end{align}

Therefore in our case

\begin{align}
H\big[\mathcal{N}(z|x; \mu,\sigma^2 \cdot I)\big] &= \frac{1}{2} \log\big(\text{det}(2 \pi e \sigma^2 \cdot I)\big) \\
&= \frac{1}{2} \log\big((2 \pi e)^N \prod_i \sigma_i^2\big) \\
&= \frac{N}{2} \log(2 \pi e) + \frac{1}{2} \sum_i \log(\sigma_i^2) \\
\end{align}

Now to address the more daunting integral

\begin{align}
& \int_z q_{\phi}(z|x) \log{p(z)} \text{d}z \\
=& \int_z \bigg((2\pi )^{-\frac{N}{2}} \big( \prod_i \sigma_i^2 \big )^{-\frac{1}{2}} \text{exp}\big\{-\frac{1}{2}(z-\mu)^T(\sigma^2 \cdot I)^{-1}(z-\mu)\big\}\bigg)\bigg(-\frac{N}{2}\log{2\pi} -\frac{z^T z}{2}\bigg)\text{d}z
\end{align}

If we expand the inner term and factor out the constants we see that

\begin{align}
& \int_z \bigg((2\pi )^{-\frac{N}{2}} \big( \prod_i \sigma_i^2 \big )^{-\frac{1}{2}} \text{exp}\big\{-\frac{1}{2}(z-\mu)^T(\sigma^2 \cdot I)^{-1}(z-\mu)\big\}\bigg)\bigg(-\frac{N}{2}\log{2\pi} -\frac{z^T z}{2}\bigg)\text{d}z \\
=&  -\frac{N}{2}\log{2\pi} \int_z q_{\phi}(z|x) \text{d}z  -\int_z \frac{z^T z}{2}\bigg((2\pi )^{-\frac{N}{2}} \big( \prod_i \sigma_i^2 \big )^{-\frac{1}{2}} \text{exp}\big\{-\frac{1}{2}(z-\mu)^T(\sigma^2 \cdot I)^{-1}(z-\mu)\big\}\bigg) \text{d}z \\
\end{align}

---

assume we have only $M$ observations from the marginal data generating distribution $p(x)$. In this case we must not only parameterize the variational distribution $q_{\phi}(z|x)$ but also the true likelihood $p_{\theta}(x|z)$. Furthermore, we 


\begin{align}
\nabla_{\phi} \mathcal{L}(\phi;x) &= -\nabla_{\phi} \int_z q_{\phi}(z|x)  \log{\frac{q_{\phi}(z|x)}{p(x,z)}} \text{d}z \\
&=  -\nabla_{\phi} \int_z q_{\phi}(z|x) \log{q_{\phi}(z|x)} \text{d}z + \int_z \nabla_{\phi} q_{\phi}(z|x) \log{p(x,z)} \text{d}z \\
&=  -\nabla_{\phi} \text{H}[q_{\phi}(z|x)] + \int_z \nabla_{\phi} q_{\phi}(z|x) \log{p(x,z)} \text{d}z
\end{align}

where we chose to represent the first integral as the entropy $\text{H}$ because many parametrics models have known closed form solutions for the entropy and this will simplify things to come.

f669fa01634ed076c3ba6f405047f4700627a3c0
ipynb
Jupyter Notebook
notes/variational_inference.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
2019-12-04T11:04:45.000Z
2019-12-04T11:04:45.000Z
notes/variational_inference.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
notes/variational_inference.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
Qwen/Qwen-72B
1. YES
2. YES