Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

Variational Inference with Gaussian Score Matching

**Chirag Modi**

Center for Computational Astrophysics,

Center for Computational Mathematics,

Flatiron Institute, New York

cmodi@flatironinstitute.org

**Charles C. Margossian**

Center for Computational Mathematics,

Flatiron Institute, New York

cmargossian@flatironinstitute.org

**Yuling Yao**

Center for Computational Mathematics,

Flatiron Institute, New York

yyao@flatironinstitute.org

**Robert M. Gower**

Center for Computational Mathematics,

Flatiron Institute, New York

rgower@flatironinstitute.org

**David M. Blei**

Department of Computer Science, Statistics,

Columbia University, New York

david.blei@columbia.edu

**Lawrence K. Saul**

Center for Computational Mathematics,

Flatiron Institute, New York

lsaul@flatironinstitute.org

###### Abstract

Variational inference (VI) is a method to approximate the computationally intractable posterior distributions that arise in Bayesian statistics. Typically, VI fits a simple parametric distribution to be close to the target posterior, optimizing an appropriate objective such as the evidence lower bound (ELBO). In this work, we present a new approach to VI. Our method is based on the principle of score matching--namely, that if two distributions are equal then their score functions (i.e., gradients of the log density) are equal at every point on their support. With this principle, we develop score-matching VI, an iterative algorithm that seeks to match the scores between the variational approximation and the exact posterior. At each iteration, score-matching VI solves an inner optimization, one that minimally adjusts the current variational estimate to match the scores at a newly sampled value of the latent variables. We show that when the variational family is a Gaussian, this inner optimization enjoys a closed-form solution, which we call Gaussian score matching VI (GSM-VI). GSM-VI is a "black box" variational algorithm in that it only requires a differentiable joint distribution, and as such it can be applied to a wide class of models. We compare GSM-VI to black box variational inference (BBVI), which has similar requirements but instead optimizes the ELBO. We first study how GSM-VI behaves as a function of the problem dimensionality, the condition number of the target covariance matrix (when the target is Gaussian), and the degree of mismatch between the approximating and exact posterior distribution. We then study GSM-VI on a collection of real-world Bayesian inference problems from the posteriorDB database of datasets and models. We find that GSM-VI is faster than BBVI  and equally or more accurate. Specifically, over a wide range of target posteriors, GSM-VI requires 10-100x fewer gradient evaluations than BBVI to obtain a comparable quality of approximation.1Introduction

This paper is about variational inference for approximate Bayesian computation. Consider a statistical model \(p(\mathbf{\theta},\mathbf{x})\) of parameters \(\mathbf{\theta}\in\mathbb{R}^{d}\) and observations \(\mathbf{x}\). Bayesian inference aims to infer the posterior distribution \(p(\mathbf{\theta}\,|\,\mathbf{x})\), which is often intractable to compute. Variational inference is an optimization-based approach to approximate the posterior [5; 18].

The idea behind VI is to approximate the posterior with a member of a _variational family_ of distributions \(q_{\mathbf{w}}(\mathbf{\theta})\), parameterized by _variational parameters_\(\mathbf{w}\)[5; 18]. Specifically, VI methods establish a measure of closeness between \(q_{\mathbf{w}}(\mathbf{\theta})\) and the posterior, and then minimize it with an optimization algorithm. Researchers have explored many aspects of VI, including different objectives [8; 9; 20; 26; 27; 29; 34] and optimization strategies [1; 15; 28].

In its modern form, VI typically minimizes \(\text{KL}\left(q_{\mathbf{w}}(\theta)||p(\theta\,|\,x)\right)\) with stochastic optimization, and further satisfies the so-called "black-box" criteria [1; 28; 33]. Black-box VI (BBVI) only requires the practitioner to specify the log joint \(\log p(\mathbf{\theta},\mathbf{x})\) and (often) its gradient \(\nabla_{\mathbf{\theta}}\log p(\theta,\mathbf{x})\), which for many models can be obtained by automatic differentiation [14; 25]. BBVI has been widely implemented, and it is available in many probabilistic programming systems [4; 21; 31].

In this paper, we propose a new approach to VI. We begin with the principle of _score matching_[16], that when two densities are equal then their gradients are equal as well, and we use this principle to derive a new way to fit a variational distribution to be close to the exact posterior. The result is _score-matching VI_. Rather than explicitly minimize a divergence, score-matching VI iteratively projects the variational distribution onto the exact score-matching constraint. This strategy enables a new black-box VI algorithm.

Score-matching VI relies on the same ingredients as reparameterization BBVI [21]--a differentiable variational family and a differentiable log joint--and so it can be as easily incorporated into probabilistic programming systems as well. Further, when the variational family is a Gaussian, score-matching VI is particularly efficient: each iteration is computable in closed form. We call the resulting algorithm Gaussian score matching VI (GSM-VI).

Unlike BBVI, GSM-VI does not rely on stochastic gradient descent (SGD) for its core optimization. Though SGD has the appeal of simplicity, it requires careful tuning of learning rates. GSM-VI was inspired by a different tradition of constraint-based algorithms for online learning [3; 7; 12; 13; 23]. These algorithms have been extensively developed and analyzed for problems in classification, and under the right conditions, they have been observed to outperform SGD. This paper shows how to extend this constraint-based framework--and the powerful machinery behind it--from the problem of classification to the workhorse of Gaussian VI. The key insight is that score-matching (unlike ELBO maximization) lends itself naturally to a constraint-based formulation.

We empirically compared GSM-VI to reparameterization BBVI on several classes of models, and with both synthetic and real-world data. In general, we found that GSM requires 10-100x fewer gradient evaluations to converge to an equally good approximation. When the exact posterior is Gaussian, we found that GSM-VI scales significantly better with respect to dimensionality and is insensitive to the condition number of the target covariance. When the exact posterior is non-Gaussian, we found that GSM-VI converges more quickly without sacrificing the quality of the final approximation.

This paper makes the following contributions:

* We introduce _score-matching variational inference_, a new black-box approach to fitting \(q_{\mathbf{w}}(\mathbf{\theta})\) to be close to \(p(\mathbf{\theta}\,|\,\mathbf{x})\). Score-matching VI requires no tunable optimization hyperparameters, to which BBVI can be sensitive.
* When the variational family is Gaussian, we develop _Gaussian score-matching variational inference_ (GSM-VI). It establishes efficient closed-form iterates for score-matching VI.
* We empirically compare GSM-VI to reparameterization BBVI. Across many models and datasets, we found that GSM-VI enjoys faster convergence to an equally good approximation.

We develop score-matching VI in Section 2 and study its performance in Section 3.

**Related work.** This paper introduces a new method for black-box variational inference that relies only on having access to the gradients of the variational distribution and the log joint. GSM-VI has similar goals to automatic-differentiation variational inference (ADVI) [21] and Pathfinder [35], which also fit multivariate Gaussian variational families, but do so by maximizing the ELBO using stochastic optimization. As in GSM-VI, the algorithm of ref. [32] also seeks to match the scores of the variational and the target posterior, but it does so by minimizing the L2 loss between them.

A novel aspect of GSM-VI is how it fits the variational parameters. Rather than minimize a loss function, it aims to solve a set of nonlinear equations. Similar ideas have been pursued in the context of fitting a model to data using empirical risk minimization (ERM). For example, passive agressive (PA) methods [7] and the stochastic polyak stepsize (SPS) are also derived via projections onto sampled nonlinear equations [3; 13; 23]. A probabilistic extension of PA methods is known as confidence-weighted (CW) learning [12]. In CW learning, the learner maintains a multivariate Gaussian distribution over the weight vector of a linear classifier. Similarly, the second step of GSM-VI also minimizes a KL divergence between multivariate Gaussians. Unlike CW learning, however, the projection in GSM-VI enforces a score-matching constraint as opposed to a margin-based constraint for linear classification.

## 2 Score Matching Variational Inference

Suppose for the moment that the variational family \(q_{\mathbf{w}}(\mathbf{\theta})\) is rich enough to perfectly capture the posterior \(p(\mathbf{\theta}\,|\,\mathbf{x})\). In other words, there exists a \(\mathbf{w}^{*}\) such that

\[\log\,q_{\mathbf{w}^{*}}(\mathbf{\theta})=\log\,p(\mathbf{\theta}|\mathbf{x}),\qquad\forall\bm {\theta}\,\in\Theta. \tag{1}\]

If we could solve Eq. 1 for \(\mathbf{w}^{*}\), the resulting variational distribution would be a perfect fit. The challenge is that the posterior on the right side is intractable to compute.

To help, we appeal to the idea of score matching [16]. We define the score of a distribution to be the gradient of its log with respect to the variable2, e.g., \(\nabla_{\mathbf{\theta}}\log q_{\mathbf{w}}(\mathbf{\theta})\). The principle of score matching is that if two distributions are equal at every point in their support then their score functions are also equal.

Footnote 2: We make this clear because, in some literature, the score is defined as the gradient with respect to the parameter.

To use score matching for VI, we first write the log posterior as the log joint minus the normalizing constant, i.e., the marginal distribution of \(\mathbf{x}\),

\[\log\,p(\mathbf{\theta}\,|\,\mathbf{x})=\log\,p(\mathbf{\theta},\mathbf{x})-\log\,p(\mathbf{x}). \tag{2}\]

With this expression, the principle of score matching leads to the following Lemma.

**Lemma 2.1**.: Let \(\Theta\subset\mathbb{R}^{d}\) be a set that contains the support of \(p(\mathbf{\theta},\mathbf{x})\) and \(q_{\mathbf{w}^{*}}(\mathbf{\theta})\) for some \(\mathbf{w}^{*}\). That is \(p(\mathbf{\theta},\mathbf{x})=q_{\mathbf{w}^{*}}(\mathbf{\theta})=0\) for every \(\theta\not\in\Theta\). Furthermore, suppose that the support of \(p(\mathbf{\theta},\mathbf{x})\) and \(q_{\mathbf{w}^{*}}(\mathbf{\theta})\), and \(\Theta\) is path-connected. The parameter \(\mathbf{w}^{*}\) satisfies

\[\nabla_{\mathbf{\theta}}\log\,q_{\mathbf{w}^{*}}(\mathbf{\theta})=\nabla_{\mathbf{\theta}}\log \,p(\mathbf{\theta},\mathbf{x}),\qquad\forall\mathbf{\theta}\in\Theta, \tag{3}\]

if and only if \(\mathbf{w}^{*}\) also satisfies Eq. 1.

What is notable about Eq. 3 is that the right side is the gradient of the log joint. Unlike the posterior, the gradient of the log joint is tractable to compute for a large class of probabilistic models. (The proof of the lemma is in the appendix.)

This lemma motivates a new algorithm, _score-matching VI._ The idea is to iteratively refine the variational parameters \(\mathbf{w}\) in order to satisfy (as much as possible) the system of equations in Eq. 3. At each iteration \(t\), the algorithm first samples a new \(\mathbf{\theta}_{t}\) from the current variational approximation and then minimally adjusts \(\mathbf{w}\) to satisfy Eq. 3 for that value of \(\mathbf{\theta}_{t}\), see (4). We want this adjustment to be minimal because, eventually \(q_{\mathbf{w}_{t}}\) will learn a good fit--one that increasingly matches the scores over the support of the target posterior--and we want to preserve this good fit as much as possible.

**Score matching variational inference**

At iteration \(t\):

Sample \(\mathbf{\theta}_{t}\sim q_{\mathbf{w}_{t}}(\mathbf{\theta})\).

Update the variational parameters:

\[\begin{split}\mathbf{w}_{t+1}=\arg\min_{\mathbf{w}}\text{KL}\left(q_{\mathbf{w} _{t}}(\mathbf{\theta})\,||\,q_{\mathbf{w}}(\mathbf{\theta})\right)\\ \text{such that}&\nabla_{\mathbf{\theta}}\,\log q_{\mathbf{w} }(\mathbf{\theta}_{t})=\nabla_{\mathbf{\theta}}\,\log p(\mathbf{\theta}_{t},\mathbf{x}).\end{split} \tag{4}\]

This algorithm for score matching VI was inspired by earlier online algorithms for learning a classifier from a stream of labeled examples. One particularly elegant algorithm in this setting is known as passive-aggressive (PA) learning [7]; this algorithm incrementally updates its model by the minimal amount to classify each example correctly by a large margin. This approach was subsequently extended to a probabilistic setting, known as confidence-weighted (CW) learning [12] in which one minimally updates a _distribution_ over classifiers. Our algorithm is similar in that it minimally updates an approximating distribution for VI, but it is different in that it enforces constraints for score matching instead of large margin classification.

At a high level, what makes this approach to VI likely to succeed or fail? Certainly it is necessary that there are more variational parameters than elements of the latent variable \(\mathbf{\theta}\); when this is not the case, it may be impossible to satisfy a _single_ score matching constraint in Eq. 4. That said, it is standard (as in, e.g., a factorized or mean-field variational family) to have at least as many variational parameters as there are elements of the latent variable. It is also apparent that the algorithm may never converge if the target posterior is not contained in the variational family, or it may converge to a degenerate solution if the variational approximation collapses to a point mass, thus terminating the updates altogether. While we cannot dismiss these possibilities out of hand, we did not observe erratic or pathological outcomes in any of the empirical studies of Section 3.

For more intuition, Figure 0(a) illustrates the effect of the update in Eq. 4 when both the target and approximating distribution are 1d Gaussian. The target posterior \(p(\mathbf{\theta}\,|\,\mathbf{x})\) is shaded blue. The plot shows the initial variational distribution \(q_{\mathbf{w}_{0}}\) (light grey curve) and its update to \(q_{\mathbf{w}_{1}}\) (medium grey) so that the gradient of the updated distribution matches the gradient of the target at the sampled \(\mathbf{\theta}_{0}\) (dotted red tangent line). It also shows the update from \(\mathbf{w}_{1}\) to \(\mathbf{w}_{2}\), now matching the gradient at \(\mathbf{\theta}_{1}\). With these two updates, \(q_{\mathbf{w}_{2}}\) (dark grey) is very close to the target \(p(\mathbf{\theta}\,|\,\mathbf{x})\). With this picture in mind, we now develop the details of this algorithm for a more widely applicable setting.

**Gaussian Score-Matching VI.** Suppose the variational distribution belongs to a multivariate Gaussian family \(q_{\mathbf{w}}(\mathbf{\theta}):=\mathcal{N}(\mathbf{\mu},\mathbf{\Sigma})\); this is a common choice especially in systems for automated approximate inference [1, 21]. One of our main contributions is to show in this case that Eq. 4 has a closed-form solution. The solution \(\mathbf{w}_{t+1}=(\mathbf{\mu}_{t+1},\mathbf{\Sigma}_{t+1})\) has the following form:

\[\mathbf{\mu}_{t+1} =\ \mathbf{\mu}_{t}+\mathbf{A}_{t}\,\left(\nabla_{\mathbf{\theta}}\,\log p(\mathbf{ \theta}_{t},\mathbf{x})-\nabla_{\mathbf{\theta}}\,\log q_{\mathbf{w}_{t}}(\mathbf{\theta}_{t})\right) \tag{5}\] \[\mathbf{\Sigma}_{t+1} =\ \mathbf{\Sigma}_{t}+(\mathbf{\mu}_{t}-\mathbf{\theta}_{t})(\mathbf{\mu}_{t}- \mathbf{\theta}_{t})^{\top}-(\mathbf{\mu}_{t+1}-\mathbf{\theta}_{t})(\mathbf{\mu}_{t+1}-\mathbf{ \theta}_{t})^{\top} \tag{6}\]

where \(\mathbf{A}_{t}\in\mathbb{R}^{d\times d}\) is a matrix defined in the theorem below. Note that these updates, which exactly solve the optimization in Eq. 4, only require the score of the log joint \(\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta},\mathbf{x})\) and the score of the variational distribution \(\nabla_{\mathbf{\theta}}\log q_{\mathbf{w}}(\mathbf{\theta})\).

Before deriving Eqs. 5 and 6 in full, we highlight the intuitive form of this closed-form solution. Consider the approximation at the \(t\)th iteration \(q_{\mathbf{w}_{t}}\) and the current sample \(\mathbf{\theta}_{t}\). First suppose the scores already match at this sample, that is \(\nabla_{\mathbf{\theta}}\,\log p(\mathbf{\theta}_{t},\mathbf{x})=\nabla_{\mathbf{\theta}}\, \log q_{\mathbf{w}_{t}}(\mathbf{\theta}_{t})\). Then the mean does not change \(\mathbf{\mu}_{t+1}=\mathbf{\mu}_{t}\) and, similarly, the two rank-one terms in the covariance update in Eq. 6 cancel out so \(\mathbf{\Sigma}_{t+1}=\mathbf{\Sigma}_{t}\). This shows that when \(q_{\mathbf{w}_{t}}(\mathbf{\theta})=p(\mathbf{\theta},\mathbf{x})\) for all \(\mathbf{\theta}\), the method stops. On the other hand, if the scores do not match, then the mean is updated proportionally to the difference between the scores, and the covariance is updated by a rank-two correction. Figure 0(b) illustrates the vector field of updates for a one dimensional target \(p(\mathbf{\theta},\mathbf{x})=\mathcal{N}(0,1)\). The vector field points to the solution (green star), which is also the only fixed point of the updates.

We now formalize this result and give the complete expression for the matrix \(\mathbf{A}_{t}\) that appears in the update for the mean of the variational approximation in Eq. 5.

**Theorem 2.2**.: **(GSM-VI updates)** Let \(p(\mathbf{\theta},\mathbf{x})\) be given for some \(\mathbf{\theta}\in\mathbb{R}^{d}\), and let \(q_{\mathbf{w}^{\epsilon}}(\mathbf{\theta})\) and \(q_{\mathbf{w}}(\mathbf{\theta})\) be multivariate normal distributions with means \(\mathbf{\mu}_{t}\) and \(\mathbf{\mu}\) and covariance matrices \(\mathbf{\Sigma}_{t}\) and \(\mathbf{\Sigma}\), respectively. As shorthand, let \(\mathbf{g}_{t}:=\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta}_{t},\mathbf{x})\) and let

\[\mathbf{\mu}_{t+1},\mathbf{\Sigma}_{t+1}\ =\ \operatorname*{argmin}_{\mathbf{\mu},\mathbf{ \Sigma}\succeq 0}\left[\operatorname{KL}(q_{t},q)\right]\quad\text{such that}\quad\nabla_{\mathbf{\theta}}\log q(\mathbf{\theta}_{t})=\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta}_{t},\mathbf{x}). \tag{7}\]

The solution to eq. (7) is given by Eqs. 5 and 6 where

\[\mathbf{A}_{t} := \frac{1}{1+\rho}\left[\mathbf{I}-\frac{(\mathbf{\mu}_{t}-\mathbf{\theta}_ {t})\mathbf{g}_{t}^{\top}}{1+\rho+(\mathbf{\mu}_{t}-\mathbf{\theta}_{t})^{\top}\mathbf{g}_{t} }\right]\mathbf{\Sigma}_{t}\,, \tag{8}\]

and \(\rho\) is the positive root of the quadratic equation

\[\rho(1+\rho)\ =\ \mathbf{g}_{t}^{\top}\mathbf{\Sigma}_{t}\mathbf{g}_{t}+\left[(\mathbf{\mu}_{t}-\mathbf{ \theta}_{t})^{\top}\mathbf{g}_{t}\right]^{2}. \tag{9}\]

With the definition of \(\mathbf{A}_{t}\) in Eq. 8 we can see that the computational complexity of updating \(\mathbf{\mu}\) and \(\mathbf{\Sigma}\) via Eqs. 5 and 6 is \(\mathcal{O}(d^{2})\), where \(\mathbf{\theta}\in\mathbb{R}^{d}\) and we assume the cost of computing the gradients is \(\mathcal{O}(d)\). Note this is the best possible iteration complexity we can hope for, since we store and maintain the full covariance matrix of \(d^{2}\) elements. (The proof is in the appendix.)

Algorithm 1 presents the full GSM-VI algorithm. Here we also use mini-batching, where we average over \(B\in\mathbb{N}\) independently sampled updates of Eqs. 5 and 6 before updating the mean and covariance.

## 3 Empirical Studies

We evaluate how well GSM-VI can approximate a variety of target posterior distributions. Recall that GSM-VI uses a multivariate Gaussian distribution as its variational family. We investigate two separate cases--one when the target posterior is in this family, and the other when it is not.

**Algorithmic details and comparisons.** We compare GSM-VI with a reparameterization variant of BBVI as the baseline, similar to [21]. BBVI uses the same multivariate Gaussian variational family, which we fit by maximizing the ELBO. (Maximizing the ELBO is equivalent to minimizing KL).

```
Input :Initial mean estimate \(\mathbf{\mu}_{0}\), initial covariance estimate \(\mathbf{\Sigma}_{0}\), target distribution \(p(\mathbf{\theta}|\mathbf{x})\), number of iterations \(N\in\mathbb{N}\), batch size \(B\in\mathbb{N}\). Output :Multivariate normal variational distribution \(q_{\mathbf{w}}(\mathbf{\theta}):=\mathcal{N}(\mathbf{\mu},\mathbf{\Sigma})\) for\(i=0,\ldots,N-1\)\(\triangleright\) iteration loop do for\(j=0,\ldots,B-1\)\(\triangleright\) batch loop do  Sample \(\mathbf{\theta}^{(j)}\sim\mathcal{N}(\mathbf{\mu}_{i},\mathbf{\Sigma}_{i})\) \(\mathbf{g}\leftarrow\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta}^{(j)}|\mathbf{x})\) \(\mathbf{\varepsilon}\leftarrow\mathbf{\Sigma}_{i}\mathbf{g}-\mathbf{\mu}_{i}+\mathbf{\theta}\)  Solve \(\rho(1+\rho)\ =\ \mathbf{g}^{\top}\mathbf{\Sigma}_{i}\mathbf{g}+\left[(\mathbf{\mu}_{i}-\mathbf{ \theta})^{\top}\mathbf{g}\right]^{2}\) for \(\rho>0\) \(\mathbf{\delta}\mathbf{\mu}^{(j)}\leftarrow\frac{1}{1+\rho}\left[\mathbf{I}-\frac{( \mathbf{\mu}_{i}-\mathbf{\theta})\mathbf{g}^{\top}}{1+\rho+(\mathbf{\mu}_{i}-\mathbf{\theta})^{ \top}\mathbf{g}}\right]\mathbf{\varepsilon}\) \(\mathbf{\mu}^{(j)}_{i}\leftarrow\mathbf{\mu}_{i}+\mathbf{\delta}\mathbf{\mu}^{(j)}\) \(\mathbf{\delta}\mathbf{\Sigma}^{(j)}\leftarrow(\mathbf{\mu}_{i}-\mathbf{\theta})(\mathbf{\mu}_{i}- \mathbf{\theta})^{\top}-(\mathbf{\mu}^{(j)}_{i}-\mathbf{\theta})(\mathbf{\mu}^{(j)}_{i}-\mathbf{ \theta})^{\top}\)  end for  Update \(\mathbf{\mu}_{i+1}\leftarrow\mathbf{\mu}_{i}+\sum_{j}\mathbf{\delta}\mathbf{\mu}^{(j)}/B\)  Update \(\mathbf{\Sigma}_{i+1}-\mathbf{\Sigma}_{i}+\sum_{j}\mathbf{\delta}\mathbf{\Sigma}^{(j)}/B\)  end for \(q_{\mathbf{w}}(\mathbf{\theta})\leftarrow\mathcal{N}(\mathbf{\mu}_{N},\mathbf{\Sigma}_{N})\)
```

**Algorithm 1**Gaussian Score Matching VI

We use the ADAM optimizer [19] with default settings but vary the learning rate between \(10^{-1}\) and \(10^{-3}\). We report results only for the best performing setting. The full BBVI algorithm is shown in Algorithm 2.

The only free parameter in GSM-VI is the batch size \(B\). We find that for Gaussian targets, GSM performs equally well for all \(B\geq 1\). There are marginal gains for larger batches for high dimensional targets, but \(B=2\) is a good conservative default. For non-Gaussian targets, we find that all batch sizes converge to the same solution, but we also note that larger batch sizes help to dampen the oscillations in KL divergence around the optimal solution. Notwithstanding these effects, it is not generally necessary to fine-tune this hyperparameter. In all studies of this section we report results for \(B=2\) and show that it is a good default baseline. We use the same batch size for BBVI.

Both GSM-VI and BBVI require an initial distribution for the variational approximation. Unless specified otherwise, we initialize the variational approximation as a Gaussian distribution with zero mean and identity covariance matrix.

**Evaluation metric.** GSM-VI does not explicitly minimize any loss function. Hence to compare its performance against BBVI, we estimate empirical divergences between the variational and the target distribution and show their evolution with the number of gradient evaluations. We measure performance in terms of gradient evaluations as the actual running time can be sensitive to implementation details; moreover, we note that in real-world problems, the computation is often bottlenecked by evaluations of the target log density and its gradient. For completeness, we also measure running times of GSM-VI and BBVI for a full rank Gaussian target with 2048 dimensions using sample size of 1 in Jax after JIT-compilation. The GSM update takes 230 ms/update while BBVI takes 227 ms/update.

In the experiments with synthetic models in Sections 3.1, and 3.2 we have access to the true distribution; hence for these experiments, we estimate the forward KL divergence (FKL) empirically by \(\sum_{\mathbf{\theta}_{i}\sim p(\mathbf{\theta})}\log p(\mathbf{\theta}_{i})-\log q(\mathbf{ \theta}_{i})\). To reduce stochasticity, we always use the same pre-generated set of 1000 samples from the target distribution. In the experiments with real-world models in Section 3.3, we do not have access to the samples from the target distribution; for these experiments we monitor the negative ELBO. In all experiments, we show the results for 10 independent runs.

### GSM-VI for Gaussian approximation

We begin by studying GSM-VI where the target distribution is also a multivariate Gaussian.

**Scaling with dimensions.** How does GSM-VI scale with respect to the dimensionality, \(D\), of the latent variable? Figure 2 shows the convergence of FKL for GSM-VI and BBVI for different dimensionalities. Empirically, for GSM-VI, we find that the number of iterations required for convergence increases almost linearly with dimensionality. The scaling for BBVI is worse: even for small problems (\(D<64\)) it requires 100 times more iterations while also converging to a sub-optimal solution (as measured by the FKL metric). The figure also shows the evolution of the _reverse_ KL divergence, and it is noteworthy that GSM-VI also outperforms BBVI in this metric.

For completeness, we also experimented on these problems3 with methods for variational inference based on natural gradient descent (NGD-VI) [2, 22]. Natural gradient descent is a stochastic gradient method that uses the Fisher matrix as a preconditioner. We find NGD-VI is slower by factor of

Figure 2: Scaling with dimensionality: evolution of forward (a) and reverse KL (b) with the number of gradient evaluations of the target distribution, which here is a Gaussian distribution with a dense \(D\times D\) covariance matrix. Each panel shows the results for a different dimensionality \(D\) (indicated in the title). Translucent lines show the scatter of 10 different runs, and the solid line shows the average. GSM-VI and BBVI use batch sizes of \(B=2\), while the batch sizes for natural gradient descent are shown in the legend.

5-10x than GSM-VI. While NGD-VI outperforms BBVI, its performance is sensitive to the choice of batch size, with small batch sizes prone to divergence. Furthermore, we note that that per-iteration-complexity of NGD-VI is cubic in the dimensionality (\(D^{3}\)) as each iteration requires inverting a Hessian approximation; this is in contrast to the quadratic (\(D^{2}\)) per-iteration-complexities of GSM-VI and BBVI. For these reasons, we do not investigate NGD-VI further in this paper.

**Impact of condition number.** What is the impact of the shape of the target distribution? We again consider a Gaussian target distribution, but vary the condition number of its covariance matrix by fixing its smallest eigenvalue to \(0.1\) and scaling the largest eigenvalue to \(0.1\times c\). Figure 3 shows the results for a 10 dimensional Gaussian where we vary the condition number \(c\) from 1 to 1000. Convergence of GSM-VI seems to be largely insensitive to the condition number of the covariance matrix. BBVI on the other hand struggles with poorly conditioned problems, and it does not converge for \(c>100\) even with 100 times more iterations than GSM.

### GSM-VI for non-Gaussian target distributions

GSM-VI was designed to solve the exact score-matching equations in Eq. 3, but these equations only have a solution when the family of variational distributions contains the target distribution (see Lemma 2.1). Here we investigate the sensitivity of GSM-VI to this assumption by fitting non-Gaussian target distributions with varying degrees on non-Gaussianity. Specifically, we suppose that

Figure 4: Impact of non-Gaussianity: evolution of FKL with the number of gradient evaluations for Sinh-arcsinh distributions with 10-dimensional dense Gaussian as the base distribution. Gaussian distribution has \(s=0,\,t=1\). In the left panel, we vary skewness \(s\) while fixing \(t=1\), and in the right panel we vary the tail-weight \(t\) with skewness fixed to \(s=0\). Solid lines are the results for GSM, dashed for BBVI.

Figure 3: Impact of condition number: evolution of FKL with the number of gradient evaluations of the target distribution. The target is a 10-dimensional Gaussian albeit with a dense covariance matrix of different condition numbers \(c\) specified in the title of different panels. Translucent lines show the scatter of 10 different runs and the solid line shows the average.

the target has a multivariate Sinh-arcsinh normal distribution [17]

\[\mathbf{z}\sim\mathcal{N}(\mathbf{\mu},\mathbf{\Sigma});\quad\mathbf{x}=\sinh\left(\frac{1}{t} \left[\sinh^{-1}(\mathbf{z})+s\right]\right) \tag{10}\]

where the scalar parameters \(s\) and \(t\) control, respectively, the skewness and the heaviness of the tails. Note that Eq. 10 reduces as a special case to a Gaussian distribution for the choices \(s=0\) and \(t=1\) of these parameters.

Figure 4 shows the result for fitting the variational Gaussian to a 10-dimensional Sinh-arcsinh normal distribution for different values of \(s\) and \(t\). As the target departs further from Gaussianity, the quality of variational fit worsens for both GSM-VI (solid lines) and BBVI (dashed lines), but they converge to a fit of similar quality in terms of average FKL. GSM converges to this solution at least 10 times faster than BBVI. For highly non-Gaussian targets (\(s\geq 1\) or \(|t-1|\geq 0.8\)), we have found that GSM-VI does not converge to a fixed point, and it can experience oscillations that are larger in amplitude than BBVI; see for instance \(s=1.8\) and \(t=0.1\) on the left and right of Figure 4, respectively.

### GSM-VI on real-world data

We evaluate GSM-VI for approximate on real-world data with 8 models from the posterioridb database [24]. The database provides the Stan code, data and reference posterior samples, and we use bridgestan to access the gradients of these models [6, 30]. We study the following statistical models, each with a different complexity and dimensionality (D): diamonds (generalized linear models, D=26), hudson-lynx-bare (differential equation dynamics, D=8), bball-drive (hidden Markov models, D=8) and arK (time-series, D=7), eight-schools-centered and non-centered (hierarchical meta-analysis, D=10), gp-pois-regr (Gaussian processes, D=13), low-dim-gauss-mix (Gaussian mixture, D=5).

For each model (except hudson-lynx-bare), we initialize the variational parameter \(\mathbf{\mu}_{0}\) at the mode of the distribution, and we set \(\mathbf{\Sigma}_{0}=0.1\,\mathbf{I}_{d}\) where \(\mathbf{I}_{d}\) is the identity matrix of dimension \(d\)

Figure 5: Models from posterioridb: Convergence of the ELBO for four models with multivariate normal posteriors. We show results for 10 runs.

Figure 6: Models from posterioridb: Convergence of ELBO for four models with non-Gaussian posteriors. We show results for 10 runs.

For Hudson-lynx-hare, we initialize the variational distribution as standard normal. We also experimented with other initializations. We find that they do not qualitatively change the conclusions, but can have larger variance between different runs.

We show the evolution of the ELBO for 10 runs of these models. Four of the models have posteriors that can be fit with multivariate normal distribution: diamonds, hudson-lynx-hare, bball-drive, and arK. Figure 5 shows the result for these models. The other models have non-Gaussian posteriors: eight-schools-centered, eight-schools-non-centered, gp-pois-regr,, and low-dim-gauss-mix. Figure 6 shows the results.

Overall, GSM-VI outperforms BBVI by a factor of 10-100x. When the target posterior is Gaussian, GSM-VI leads to more stable solutions. When the target is non-Gaussian, it converges to the same quality of variational approximation as BBVI. Finally, we note that while GSM-VI yields noisy estimates of the ELBO, it does yield stable estimates of the 1-D marginals and their moments.

## 4 Conclusion and Future Work

In this paper we have proposed Gaussian score matching VI (GSM-VI), a new approach for VI when the variational family is multivariate Gaussian. GSM-VI does not explicitly minimize a divergence between the variational and target distribution; instead, it repeatedly enforces score-matching constraints with closed-form updates for the mean and covariance matrix of the variational distribution. This is in contrast to stochastic gradient based methods, such as in BBVI, that rely on first-order Taylor approximations of their objective function.

GSM-VI is implemented in an open-source Python code at [https://github.com/modichirag/GSM-VI](https://github.com/modichirag/GSM-VI).

Unlike approaches that are rooted in stochastic gradient descent, GSM-VI does not require the tuning of step-size hyper-parameters. Rather, it is able to adaptively make large jumps in the initial iterations (see Fig. 1a) and make smaller adjustments as the approximation converges to the target. GSM-VI has only one free parameter, the batch size, and we found a batch-size of 2 to perform competitively across all experiments. Another choice is how to initialize the variational distribution. For the experiments in this paper, we initialized the covariance matrix as the identity matrix, but additional gains could potentially be made with more informed choices derived from a Laplace approximation or L-BFGS Hessian approximation [35].

We evaluated the performance of GSM-VI on synthetic targets and real-world models from posterioriD. In general, we found that it requires 10-100x fewer gradient evaluations than BBVI for the target distribution to converge. When the target distribution is itself multivariate Gaussian, we observed that GSM-VI scales nearly _linearly_ with dimensionality, which is significantly better than BBVI, and that GSM-VI is fairly insensitive to the condition number of the target covariance matrix. We also found that GSM-VI converges more quickly than BBVI--and also to a solution with a larger ELBO, which is surprising given that BBVI explicitly maximizes the ELBO.

An important avenue for future work is to provide a proof that GSM-VI converges. We note that good convergence results have been obtained for analogous methods that project onto interpolation equations for empirical risk minimization. For instance the Stochastic Polyak Step achieves the min-max optimal rates of convergence for SGD [23]. Note that convergence of VI is a generally challenging problem, with no known rates of convergence even for BBVI [10; 11]. This and others directions are left to future work.

## References

* Agrawal et al. [2020] A. Agrawal, D. R. Sheldon, and J. Domke. Advances in black-box vi: normalizing flows, importance weighting, and optimization. _Neural Information Processing Systems_, 2020.
* Arenz et al. [2023] O. Arenz, P. Dahlinger, Z. Ye, M. Volpp, and G. Neumann. A unified perspective on natural gradient variational inference with gaussian mixture models. _Transactions on Machine Learning Research_, 2023. ISSN 2835-8856. URL [https://openreview.net/forum?id=tLbjsX4tjs](https://openreview.net/forum?id=tLbjsX4tjs).

* Berrada et al. [2020] L. Berrada, A. Zisserman, and M. P. Kumar. Training neural networks for and by interpolation. In _Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, 2020.
* Bingham et al. [2018] E. Bingham, J. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. Szerlip, P. Horsfall, and N. Goodman. Pyro: Deep universal probabilistic programming. _arXiv:1810.09538_, 2018.
* Blei et al. [2017] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. _Journal of the American statistical Association_, 112(518):859-877, 2017.
* Carpenter et al. [2017] B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan: A probabilistic programming language. _Journal of Statistical Software_, 76(1):1-32, 2017.
* Crammer et al. [2006] K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive-aggressive algorithms. _Journal of Machine Learning Research_, 7:551-585, 2006.
* Dhaka et al. [2021] A. K. Dhaka, A. Catalina, M. Welandawe, M. Riis Andersen, J. Huggins, and A. Vehtari. Challenges and Opportunities in High-dimensional Variational Inference. art. arXiv:2103.01085, March 2021.
* Dieng et al. [2017] A. Dieng, D. Tran, R. Ranganath, J. Paisley, and D. Blei. Variational inference via \(\chi\) upper bound minimization. In _Neural Information Processing Systems_, 2017.
* Domke [2019] J. Domke. Provable gradient variance guarantees for black-box variational inference. In _Neural Information Processing Systems_, volume 32, 2019.
* Domke [2020] J. Domke. Provable smoothness guarantees for black-box variational inference. In _International Conference on Machine Learning_, volume 119, pages 2587-2596. PMLR, 2020.
* Dredze et al. [2008] M. Dredze, K. Crammer, and F. Pereira. Confidence-weighted linear classification. In _International Conference on Machine Learning,_, volume 307, pages 264-271, 2008.
* Gower et al. [2021] R. M. Gower, A. Defazio, and M. Rabbat. Stochastic polyak stepsize with a moving target. _arXiv:2106.11851_, 2021.
* Gunes Baydin et al. [2015] A. Gunes Baydin, B. A. Pearlmutter, A. Andreyevich Radul, and J. M. Siskind. Automatic differentiation in machine learning: a survey. _arXiv e-prints_, art. arXiv:1502.05767, February 2015. doi: 10.48550/arXiv.1502.05767.
* Hoffman et al. [2013] M. Hoffman, D. Blei, C. Wang, and J. Paisley. Stochastic variational inference. _Journal of Machine Learning Research_, 14:1303-1347, 2013.
* Hyvarinen [2005] A. Hyvarinen. Estimation of non-normalized statistical models by score matching. _Journal of Machine Learning Research_, 6:695-709, 2005.
* Jones and Pewsey [2019] C. Jones and A. Pewsey. The Sinh-Arcsinh Normal Distribution. _Significance_, 16(2):6-7, 04 2019. ISSN 1740-9705.
* Jordan et al. [1999] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. _Machine learning_, 37:183-233, 1999.
* Kingma and Ba [2015] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In _ICLR_, 2015.
* Knoblauch et al. [2022] J. Knoblauch, J. Jewson, and T. Damoulas. An optimization-centric view on Bayes' rule: Reviewing and generalizing variational inference. _Journal of Machine Learning Research_, 23:1-109, 2022.
* Kucukelbir et al. [2016] A. Kucukelbir, D. Tran, R. Ranganath, A. Gelman, and D. M. Blei. Automatic Differentiation Variational Inference. art. arXiv:1603.00788, March 2016.
* Lin et al. [2019] W. Lin, M. Emtiyaz Khan, and M. Schmidt. Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations. _arXiv e-prints_, art. arXiv:1906.02914, June 2019. doi: 10.48550/arXiv.1906.02914.
* Loizou et al. [2021] N. Loizou, S. Vaswani, I. Laradji, and S. Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. _International Conference on Artificial Intelligence and Statistics_, 2021.
* Magnusson et al. [2022] M. Magnusson, P. Burkner, and A. Vehtari. posterioridb: a set of posteriors for Bayesian inference and probabilistic programming, November 2022. URL [https://github.com/stan-dev/posteriordb](https://github.com/stan-dev/posteriordb).

* [25] Charles C. Margossian. A review of automatic differentiation and its efficient implementation. _Wiley interdisciplinary reviews: data mining and knowledge discovery_, 9, 3 2019. doi: 10.1002/WIDM.1305.
* [26] T. Minka. Expectation propagation for approximate Bayesian inference. In _Uncertainty in Artificial Intelligence (UAI)_, pages 362-369, 2001.
* [27] C. Naesseth, F. Lindsten, and D. Blei. Markovian score climbing: Variational inference with \(\text{KL}(\text{p}\,\|\,\text{q})\). In _Neural Information Processing Systems_, 2020.
* [28] R. Ranganath, S. Gerrish, and D. M. Blei. Black Box Variational Inference. art. arXiv:1401.0118, December 2013.
* [29] R. Ranganath, J. Altosaar, D. Tran, and D. Blei. Operator variational inference. In _Neural Information Processing Systems_. 2016.
* [30] E. Roualdes, B. Ward, S. Axen, and B. Carpenter. BridgeStan: Efficient in-memory access to Stan programs through Python, Julia, and R, March 2023. URL [https://github.com/roualdes/bridgestan](https://github.com/roualdes/bridgestan).
* [31] J. Salvatier, Wiecki T., and C. Fonnesbeck. Probabilistic programming in Python using PyMC3. _PeerJ Computer Science_, 2016.
* [32] U. Seljak and B. Yu. Posterior inference unchained with EL_2O. art. arXiv:1901.04454, January 2019.
* [33] M. W., M. R. Andersen, A. Vehtari, and J. H. Huggins. Robust, automated, and accurate black-box variational inference. _arXiv:2203.15945_, 2022.
* [34] Y. Yang, R. Martin, and H. Bondell. Variational approximations using Fisher divergence. _arXiv e-prints_, art. arXiv:1905.05284, May 2019.
* [35] L. Zhang, B. Carpenter, A. Gelman, and A. Vehtari. Pathfinder: Parallel quasi-newton variational inference. _Journal of Machine Learning Research_, 23(306):1-49, 2022.

## Appendix A Proof of Lemma 2.1

**Lemma 2.1**.: Let \(\Theta\subset\mathbb{R}^{d}\) be a set that contains the support of \(p(\mathbf{\theta},\mathbf{x})\) and \(q_{\mathbf{w}^{*}}(\mathbf{\theta})\) for some \(\mathbf{w}^{*}\). That is \(p(\mathbf{\theta},\mathbf{x})=q_{\mathbf{w}^{*}}(\mathbf{\theta})=0\) for every \(\theta\not\in\Theta\). Furthermore, suppose that the support of \(p(\mathbf{\theta},\mathbf{x})\) and \(q_{\mathbf{w}^{*}}(\mathbf{\theta})\), and \(\Theta\) is path-connected. The parameter \(\mathbf{w}^{*}\) satisfies

\[\nabla_{\mathbf{\theta}}\log\,q_{\mathbf{w}^{*}}(\mathbf{\theta})=\nabla_{\mathbf{\theta}}\log \,p(\mathbf{\theta},\mathbf{x}),\qquad\forall\mathbf{\theta}\in\Theta, \tag{3}\]

if and only if \(\mathbf{w}^{*}\) also satisfies Eq. 1.

Proof.: (1) \(\implies\)(3): Differentiating both sides of (1) in \(\mathbf{\theta}\) gives

\[\nabla_{\theta}\log q_{\mathbf{w}^{*}}(\mathbf{\theta}) =\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta}|\mathbf{x})=\nabla_{\theta} \left(\log p(\mathbf{\theta},\mathbf{x})-\log p(\mathbf{x})\right)\] \[=\nabla_{\theta}\log p(\mathbf{\theta},\mathbf{x}),\qquad\forall\mathbf{ \theta}\in\Theta.\]

(3) \(\implies\)(1): Let \(\mathbf{\theta}_{0}\) be any arbitrary point in \(\Theta\). Because \(\Theta\) is path-connected, every \(\mathbf{\theta}\in\Theta\) is connected to \(\mathbf{\theta}_{0}\) via a differentiable path \(\mathbf{r}(t)\) where \(\mathbf{r}(0)=\mathbf{\theta}_{0}\) and \(\mathbf{r}(1)=\mathbf{\theta}\). Integrating both sides of (3) along this path \(\mathbf{r}(t)\), using again that \(\nabla_{\theta}\log p(\mathbf{\theta}|\mathbf{x})=\nabla_{\theta}\log p(\mathbf{\theta}, \mathbf{x})\), and using the Fundamental Theorem of Calculus gives

\[\log q_{\mathbf{w}^{*}}(\mathbf{\theta})-\log q_{\mathbf{w}^{*}}(\mathbf{\theta}_ {0}) =\int_{0}^{1}\left\langle\nabla_{\theta}\log q_{\mathbf{w}^{*}}( \mathbf{r}(t)),\mathbf{r}^{\prime}(t)\right\rangle dt\] \[=\int_{0}^{1}\left\langle\nabla_{\theta}\log p((\mathbf{r}(t))| \mathbf{x}),\mathbf{r}^{\prime}(t)\right\rangle dt\] \[=\log p(\mathbf{\theta}|\mathbf{x})-\log p(\mathbf{\theta}_{0}|\mathbf{x}),\quad \forall\mathbf{\theta}\in\Theta.\]

Rearranging and defining \(C:=\log q_{\mathbf{w}^{*}}(\mathbf{\theta}_{0})-\log p(\mathbf{\theta}_{0}|\mathbf{x})\) gives

\[\log q_{\mathbf{w}^{*}}(\mathbf{\theta})=\log p(\mathbf{\theta}|\mathbf{x})+C,\qquad\forall \mathbf{\theta}\in\Theta,\]By exponentiating both sides and integrating in \(\mathbf{\theta}\) over \(\Theta\) we have that

\[1=\int_{\mathbf{\theta}\in\Theta}q_{\mathbf{w}^{*}}(\mathbf{\theta})d\mathbf{\theta}=e^{C}\int_{ \mathbf{\theta}\in\Theta}p(\mathbf{\theta}|\mathbf{x})d\mathbf{\theta}=e^{C},\]

where we used that \(\Theta\) contains the support of both \(p(\mathbf{\theta}|\mathbf{x})\) and \(q_{\mathbf{w}^{*}}(\mathbf{\theta})\). Consequently \(C=0\), which gives our result.

## Appendix B Proof of Theorem 2.2

Here we give the proof for Theorem 2.2. We also re-introduce the theorem with a simplified notation, where we use \((\mathbf{\mu}_{0},\mathbf{\Sigma}_{0})\) to denote the mean and covariance at the previous time step of the method, thus dropping the iteration counter \(t\).

**Theorem B.1**.: **(GSM updates)** Let \(p(\mathbf{\theta},\mathbf{x})\) be given for some \(\mathbf{\theta}\in\mathbb{R}^{d}\), and let \(q_{0}(\mathbf{\theta})\) and \(q(\mathbf{\theta})\) be the multivariate normal distributions, respectively, with means \(\mathbf{\mu}_{0}\) and \(\mathbf{\mu}\) and covariance matrices \(\mathbf{\Sigma}_{0}\) and \(\mathbf{\Sigma}\). We seek the distribution

\[\operatorname*{arg\,min}_{\mathbf{\mu},\mathbf{\Sigma}=\mathbf{\Sigma}^{\top}}\left[ \operatorname{KL}(q_{0},q)\right]\quad\text{such that}\quad\nabla_{\mathbf{\theta }}\log q(\mathbf{\theta})=\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta},\mathbf{x}). \tag{11}\]

As shorthand, let \(\mathbf{g}\;:=\;\nabla_{\mathbf{\theta}}\log p(\mathbf{\theta},\mathbf{x}),\) and let \(\rho\) be the positive root of the quadratic equation

\[\rho(1\!+\!\rho)\;=\;\mathbf{g}^{\top}\mathbf{\Sigma}_{0}\mathbf{g}+\left[(\mathbf{\mu}_{0}\!- \!\mathbf{\theta})^{\top}\!\mathbf{g}\right]^{2}. \tag{12}\]

Then the solution is given by the following closed-form updates:

\[\mathbf{\mu} = \mathbf{\mu}_{0}\;+\frac{1}{1+\rho}\left[\mathbf{I}-\frac{(\mathbf{\mu}_{ 0}-\mathbf{\theta})\mathbf{g}^{\top}}{1+\rho+(\mathbf{\mu}_{0}-\mathbf{\theta})^{\top}\mathbf{g}} \right]\mathbf{\Sigma}_{0}\left(\mathbf{g}-\nabla_{\mathbf{\theta}}\log q_{0}(\mathbf{\theta}) \right), \tag{13}\] \[\mathbf{\Sigma} = \mathbf{\Sigma}_{0}+(\mathbf{\mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{\theta})^{\top}. \tag{14}\]

Furthermore, if \(\mathbf{\Sigma}_{0}\) is symmetric positive definite then so is \(\mathbf{\Sigma}\).

Proof.: The constraint in this optimization is given by

\[\mathbf{g} = \nabla_{\mathbf{\theta}}\log q(\mathbf{\theta}) \tag{15}\] \[= \nabla_{\mathbf{\theta}}\left[-\tfrac{1}{2}(\mathbf{\theta}-\mathbf{\mu})\mathbf{ \Sigma}^{-1}(\mathbf{\theta}-\mathbf{\mu})-\tfrac{1}{2}\log\left((2\pi)^{d}|\mathbf{ \Sigma}|\right)\right]\] (16) \[= -\mathbf{\Sigma}^{-1}(\mathbf{\theta}-\mathbf{\mu}). \tag{17}\]

The KL divergence is given by

\[\operatorname{KL}(q_{0},q)\;=\;\frac{1}{2}\left\{\mathbf{tr}[\mathbf{\Sigma}^{-1} \mathbf{\Sigma}_{0}]+\log\frac{|\mathbf{\Sigma}|}{|\mathbf{\Sigma}_{0}|}+(\mathbf{\mu}-\mathbf{\mu }_{0})^{\top}\mathbf{\Sigma}^{-1}(\mathbf{\mu}-\mathbf{\mu}_{0})-d\right\}. \tag{18}\]

Dropping irrelevant terms from the optimization, we obtain the Lagrangian

\[\mathcal{L}(\mathbf{\mu},\mathbf{\Sigma},\mathbf{\lambda})\;=\;\frac{1}{2}\left\{\mathbf{ tr}[\mathbf{\Sigma}^{-1}\mathbf{\Sigma}_{0}]-\log|\mathbf{\Sigma}^{-1}|+(\mathbf{\mu}-\mathbf{\mu}_{0 })^{\top}\mathbf{\Sigma}^{-1}(\mathbf{\mu}-\mathbf{\mu}_{0})\right\}+\mathbf{\lambda}^{\top} \big{(}\mathbf{g}-\mathbf{\Sigma}^{-1}(\mathbf{\mu}-\mathbf{\theta})\big{)}. \tag{19}\]

It is easier to optimize the matrix \(\mathbf{\Sigma}^{-1}\) instead of \(\mathbf{\Sigma}\). We can enforce the symmetry of \(\mathbf{\Sigma}^{-1}\) by writing

\[\mathbf{\Sigma}^{-1}\;=\;\tfrac{1}{2}\left(\mathbf{\Phi}+\mathbf{\Phi}^{\top}\right) \tag{20}\]

and performing an unconstrained optimization over \(\mathbf{\Phi}\). With respect to the latter, the gradients of the Lagrangian are given by

\[\frac{\partial\mathcal{L}}{\partial\Phi_{ij}}\;=\;\sum_{kl}\left(\frac{ \partial\mathcal{L}}{\partial\Sigma_{kl}^{-1}}\right)\left(\frac{\partial \Sigma_{kl}^{-1}}{\partial\Phi_{ij}}\right)\;=\;\sum_{kl}\left(\frac{\partial \mathcal{L}}{\partial\Sigma_{kl}^{-1}}\right)\left(\frac{1}{2}\delta_{ki} \delta_{lj}+\frac{1}{2}\delta_{kj}\delta_{li}\right)\;=\;\frac{1}{2}\left( \frac{\partial\mathcal{L}}{\partial\Sigma_{ij}^{-1}}+\frac{\partial\mathcal{L }}{\partial\Sigma_{ji}^{-1}}\right). \tag{21}\]Next we examine where the gradients of the Lagrangian vanish:

\[\mathbf{0}=\frac{\partial\mathcal{L}}{\partial\mathbf{\mu}}\;\;\Longrightarrow\;\; \mathbf{0}\;=\;\mathbf{\Sigma}^{-1}(\mathbf{\mu}-\mathbf{\mu}_{0})\,-\,\mathbf{\Sigma}^{-1} \mathbf{\lambda}\;\;\;\Longrightarrow\;\;\;\boxed{\mathbf{\lambda}=\mathbf{\mu}-\mathbf{\mu}_{0 }} \tag{22}\]

\[\mathbf{0}=\frac{\partial\mathcal{L}}{\partial\mathbf{\lambda}}\;\;\Longrightarrow\; \;\mathbf{0}\;=\;\mathbf{g}-\mathbf{\Sigma}^{-1}(\mathbf{\mu}-\mathbf{\theta})\;\;\;\Longrightarrow \;\;\;\boxed{\mathbf{\mu}-\mathbf{\theta}=\mathbf{\Sigma}\mathbf{g}} \tag{23}\]

\[\mathbf{0}=\frac{\partial\mathcal{L}}{\partial\mathbf{\Phi}}\;\;\Longrightarrow\; \;\mathbf{0}\;=\;\mathbf{\Sigma}_{0}-\mathbf{\Sigma}+(\mathbf{\mu}-\mathbf{\mu}_{0})(\mathbf{\mu} -\mathbf{\mu}_{0})^{\top}-\left[\mathbf{\lambda}(\mathbf{\mu}-\mathbf{\theta})^{\top}+(\mathbf{\mu }-\mathbf{\theta})\mathbf{\lambda}^{\top}\right], \tag{24}\]

\[\Longrightarrow\;\;\boxed{\mathbf{\Sigma}\;=\;\mathbf{\Sigma}_{0}\,+\,(\mathbf{\mu}-\mathbf{ \mu}_{0})(\mathbf{\mu}-\mathbf{\mu}_{0})^{\top}\,-\,\mathbf{\lambda}(\mathbf{\mu}-\mathbf{\theta} )^{\top}\,-\,(\mathbf{\mu}-\mathbf{\theta})\mathbf{\lambda}^{\top}} \tag{25}\]

We claim that these equations (though nonlinear) can be solved in closed form. The first step is to eliminate \(\mathbf{\lambda}\) from eq. (25) using eq. (22). In this way we find

\[\mathbf{\Sigma} = \mathbf{\Sigma}_{0}+(\mathbf{\mu}-\mathbf{\mu}_{0})(\mathbf{\mu}-\mathbf{\mu}_{0})^{ \top}-(\mathbf{\mu}-\mathbf{\mu}_{0})(\mathbf{\mu}-\mathbf{\theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta })(\mathbf{\mu}-\mathbf{\mu}_{0})^{\top} \tag{26}\] \[= \mathbf{\Sigma}_{0}-\mathbf{\mu}\mathbf{\mu}^{\top}+\mathbf{\mu}\mathbf{\theta}^{\top }+\mathbf{\theta}\mathbf{\mu}^{\top}+\mathbf{\mu}_{0}\mathbf{\mu}_{0}^{\top}-\mathbf{\mu}_{0}\mathbf{ \theta}^{\top}-\mathbf{\theta}\mathbf{\mu}_{0}^{\top}\] (27) \[= \mathbf{\Sigma}_{0}+(\mathbf{\mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{\theta})^{\top}. \tag{28}\]

It is worth highlighting the form of this equation:

\[\mathbf{\Sigma}\;=\;\mathbf{\Sigma}_{0}+(\mathbf{\mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{\theta})^{\top}\]

This is a simple rank-two update for \(\mathbf{\Sigma}\). Note that \(\mathbf{\Sigma}=\mathbf{\Sigma}_{0}\) if \(\mathbf{\mu}=\mathbf{\mu}_{0}\); also, the solution for \(\mathbf{\Sigma}\) is determined by the solution for \(\mathbf{\mu}\).

Ultimately we must solve for \(\mathbf{\mu}\), but first it is useful to solve for the intermediate quantity \(\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}>0\). From eq. (28), we obtain

\[\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}\;=\;\mathbf{g}^{\top}\mathbf{\Sigma}_{0}\mathbf{g}+\left[( \mathbf{\mu}_{0}-\mathbf{\theta})^{\top}\mathbf{g}\right]^{2}-\left[(\mathbf{\mu}-\mathbf{\theta })^{\top}\mathbf{g}\right]^{2}, \tag{29}\]

and from eq. (23), we obtain

\[\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}\;=\;\mathbf{g}^{\top}\mathbf{\Sigma}_{0}\mathbf{g}+\left[( \mathbf{\mu}_{0}-\mathbf{\theta})^{\top}\mathbf{g}\right]^{2}-\left(\mathbf{g}^{\top}\mathbf{ \Sigma}\mathbf{g}\right)^{2}. \tag{30}\]

As shorthand, let \(\rho=\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}\). Then from eq. (30) we see that \(\rho\) satisfies the quadratic equation

\[\rho(1\!+\!\rho)\;=\;\mathbf{g}^{\top}\mathbf{\Sigma}_{0}\mathbf{g}+\left[(\mathbf{\mu}_{0}- \mathbf{\theta})^{\top}\mathbf{g}\right]^{2}.\]

Note that there are no unknowns on the right side of this equation. The correct solution is given by the positive root since \(\rho=\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}>0\). Also note that \(\rho=(\mathbf{\mu}-\mathbf{\theta})^{\top}\mathbf{g}\) from eq. (23).

It is useful to define one final intermediate quantity before solving for \(\mathbf{\mu}\). Let

\[\mathbf{\varepsilon}_{0}\;=\;\mathbf{\Sigma}_{0}\mathbf{g}-\mathbf{\mu}_{0}+\mathbf{\theta}.\]

Note that \(\mathbf{\varepsilon}_{0}\) simply measures the degree to which the parameters of \(q_{0}(\mathbf{\theta})\) violate the desired constraint \(\nabla_{\mathbf{w}}\log q(\mathbf{\theta})=\nabla_{\mathbf{w}}\log p(\mathbf{\theta},\mathbf{y})\). Put another way, if \(\mathbf{\varepsilon}_{0}=\mathbf{0}\), then we have the trivial solution \(\mathbf{\mu}=\mathbf{\mu}_{0}\) and \(\mathbf{\Sigma}=\mathbf{\Sigma}_{0}\).

Now we have everything to express the solution for \(\mathbf{\mu}\) in a highly intuitive form; in particular, it will be immediately evident that \(\mathbf{\mu}\rightarrow\mathbf{\mu}_{0}\) as \(\mathbf{\varepsilon}_{0}\rightarrow\mathbf{0}\). Starting from eqs. (23) and (28), we find

\[\mathbf{\mu}-\mathbf{\mu}_{0} = \mathbf{\theta}-\mathbf{\mu}_{0}+\mathbf{\Sigma}\mathbf{g}, \tag{31}\] \[= \mathbf{\theta}-\mathbf{\mu}_{0}+\left[\mathbf{\Sigma}_{0}+(\mathbf{\mu}_{0}-\bm {\theta})(\mathbf{\mu}_{0}-\mathbf{\theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{ \theta})^{\top}\right]\mathbf{g},\] (32) \[= \mathbf{\varepsilon}_{0}\,+\,(\mathbf{\mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}- \mathbf{\theta})^{\top}\mathbf{g}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{\theta})^{\top} \mathbf{g},\] (33) \[= \mathbf{\varepsilon}_{0}\,+\,(\mathbf{\mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}- \mathbf{\theta})^{\top}\mathbf{g}\,-\,(\mathbf{\mu}-\mathbf{\mu}_{0}+\mathbf{\mu}_{0}-\mathbf{\theta})( \mathbf{\mu}-\mathbf{\theta})^{\top}\mathbf{g},\] (34) \[= \mathbf{\varepsilon}_{0}\,-\,\rho(\mathbf{\mu}-\mathbf{\mu}_{0})\,+\,(\mathbf{\mu}_ {0}-\mathbf{\theta})[(\mathbf{\mu}_{0}-\mathbf{\theta})-(\mathbf{\mu}-\mathbf{\theta})^{\top}\mathbf{g},\] (35) \[= \mathbf{\varepsilon}_{0}\,-\,\rho(\mathbf{\mu}-\mathbf{\mu}_{0})\,+\,(\mathbf{\mu} _{0}-\mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{\mu})^{\top}\mathbf{g},\] (36) \[= \mathbf{\varepsilon}_{0}\,-\,(\rho\mathbf{\Gamma}+(\mathbf{\mu}_{0}-\mathbf{\theta} )\mathbf{g}^{\top})(\mathbf{\mu}_{0}-\mathbf{\mu}). \tag{37}\]Note what has happened here: eq. (32) is a system of _nonlinear_ equations for \(\mu\), but in eq. (37), all the nonlinearity has been expressed in terms of \(\rho\). Since \(\rho\) can be determined via eq. (30), we arrive effectively at a system of _linear_ equations for \(\mu\). Collecting terms, we obtain

\[\left[(1+\rho)\mathbf{I}+(\mathbf{\mu}_{0}-\mathbf{\theta}) \mathbf{g}^{\top}\right](\mathbf{\mu}-\mathbf{\mu} _{0})\;=\;\mathbf{\varepsilon}_{0}. \tag{38}\]

We thus arrive at the closed-form update

\[\mathbf{\mu}\;=\;\mathbf{\mu}_{0}\,+\,\left[(1+\rho)\mathbf{ I}+(\mathbf{\mu}_{0}-\mathbf{\theta})\mathbf{g}^{\top} \right]^{-1}\mathbf{\varepsilon}_{0} \tag{39}\]

It is evident from this update that \(\mathbf{\mu}\rightarrow\mathbf{\mu}_{0}\) as \(\mathbf{\varepsilon}_{0}\rightarrow\mathbf{0}\). The matrix inverse in this update can also be computed efficiently from the Woodbury matrix identity.

In sum, the joint update for \(\mu\) and \(\Sigma\) can be efficiently computed as follows:

1. Set \(\mathbf{g}=\nabla_{\mathbf{w}}\log p(\mathbf{\theta },\mathbf{y})\) and \(\mathbf{\varepsilon}_{0}\;=\;\mathbf{\Sigma}_{0}\mathbf{g}-\mathbf{\mu}_{0}+\mathbf{\theta}\).
2. Solve \(\rho(1+\rho)\;=\;\mathbf{g}^{\top}\mathbf{\Sigma}_{0}\mathbf{g}+\left[(\mathbf{\mu}_{0}-\mathbf{\theta})^{\top }\mathbf{g}\right]^{2}\) for \(\rho>0\).
3. Compute \(\mathbf{\mu}\;=\;\mathbf{\mu}_{0}\,+\,\left[(1+\rho)\mathbf{ I}+(\mathbf{\mu}_{0}-\mathbf{\theta})\mathbf{g}^{\top} \right]^{-1}\mathbf{\varepsilon}_{0}\).
4. Compute \(\mathbf{\Sigma}\;=\;\mathbf{\Sigma}_{0}+(\mathbf{ \mu}_{0}-\mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}-(\mathbf{\mu}-\mathbf{\theta})(\mathbf{\mu}-\mathbf{\theta})^{\top}\).

Note that it straightforward to solve for the value of \(\rho\) in step 2 of this procedure. In particular, from the quadratic formula, we obtain the positive root

\[\boxed{\rho\;=\;\frac{\sqrt{1+4(\mathbf{g}^{\top}\mathbf{ \Sigma}_{0}\mathbf{g}+\left[(\mathbf{\mu}_{0}-\mathbf{\theta})\mathbf{\overline{g}}\right]^{2})-1}}{2}.}\]

We can solve the linear equation for \(\mu\) in step 3 of this procedure most efficiently by using the Sherman-Morrison formula

\[\left[a\mathbf{I}+\mathbf{u}\mathbf{g}^{\top}\right]^{-1}= \frac{1}{a}\left(\mathbf{I}-\frac{\mathbf{u}\mathbf{g}^{\top }}{a+\mathbf{u}^{\top}\mathbf{g}}\right),\quad\mbox{for every }\mathbf{u},\mathbf{g},a.\]

Applying this formula in step 3, we find that

\[\mathbf{\mu}\;=\;\mathbf{\mu}_{0}\,+\frac{1}{1+\rho}\left[ \mathbf{I}-\frac{(\mathbf{\mu}_{0}-\mathbf{\theta})\mathbf{g}^{\top}}{1+\rho+(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}\mathbf{g}}\right]\mathbf{\varepsilon}_{0}. \tag{40}\]

Next we relate this update to the difference in the scores of the target distribution and the current estimate of the variational approximation. Since \(\nabla_{\mathbf{\theta}}\log q_{0}(\mathbf{\theta})=-\mathbf{\Sigma}_{0}^{-1}(\mathbf{\theta}-\mathbf{\mu}_{0})\), we have that

\[\mathbf{\varepsilon}_{0}=\mathbf{\Sigma}_{0}\left(\mathbf{g}-\mathbf{\Sigma}_{0}^{-1}(\mathbf{\mu}_{0}+ \mathbf{\theta})\right)=\mathbf{\Sigma}_{0}\left(\mathbf{g}-\nabla_{\mathbf{\theta}}\log q_{0}(\mathbf{ \theta})\right).\]

It follows, by substituting the above expression into Eq. 40, that

\[\boxed{\mathbf{\mu}\;=\;\mathbf{\mu}_{0}\,+\frac{1}{1+\rho} \left[\mathbf{I}-\frac{(\mathbf{\mu}_{0}-\mathbf{\theta}) \mathbf{g}^{\top}}{1+\rho+(\mathbf{\mu}_{0}-\mathbf{ \theta})^{\top}\mathbf{g}}\right]\mathbf{\Sigma}_{0}\left( \mathbf{\nabla}_{\mathbf{\theta}}\log p(\mathbf{x}, \mathbf{\theta})-\nabla_{\mathbf{\theta}}\log q_{0}(\mathbf{\theta})\right),}\]

which gives the update in the statement of the theorem.

Finally we prove that if \(\mathbf{\Sigma}_{0}\) is positive definite, then so is \(\Sigma\). To do so, we show explicitly that all of the eigenvalues of \(\Sigma\) are positive. We begin by rewriting our results for \(\Sigma\) in eq. (28) and \(\rho\) in eq. (30) in a more convenient form. As shorthand, let

\[\mathbf{M}_{0}\;=\;\mathbf{\Sigma}_{0}+(\mathbf{\mu}_{0}- \mathbf{\theta})(\mathbf{\mu}_{0}-\mathbf{\theta})^{ \top}, \tag{41}\]

so that \(\mathbf{M}_{0}\) captures the first two terms on the right side of eq. (28). Note that \(\mathbf{M}_{0}\) is positive-definite, a fact that we will exploit repeatedly in what follows. In addition, recall that \(\mathbf{\mu}-\mathbf{\theta}=\mathbf{\Sigma}\mathbf{g}\) from eq. (23). Thus with this notation we can rewrite eqs. (28) and (30) as

\[\mathbf{\Sigma} = \mathbf{M}_{0}-(\mathbf{\Sigma}\mathbf{g})( \mathbf{\Sigma}\mathbf{g})^{\top}, \tag{42}\] \[\rho(1+\rho) = \mathbf{g}^{\top}\mathbf{M}_{0}\mathbf{g}. \tag{43}\]Now let \(\mathbf{e}\) be any normalized eigenvector of \(\mathbf{\Sigma}\); we want to show that its corresponding eigenvalue \(\lambda_{e}\) is positive. From eq. (42), it follows that

\[\lambda_{e} = \mathbf{e}^{\top}\mathbf{\Sigma}\mathbf{e} \tag{44}\] \[= \mathbf{e}^{\top}\left[\mathbf{M}_{0}-(\mathbf{\Sigma}\mathbf{g})(\mathbf{\Sigma} \mathbf{g})^{\top}\right]\mathbf{e}\] (45) \[= \mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{e}-\lambda_{e}^{2}(\mathbf{e}^{\top} \mathbf{g})^{2}. \tag{46}\]

Note that if \(\mathbf{e}^{\top}\mathbf{g}=0\), then it follows trivially that \(\lambda_{e}=\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{e}>0\). So we only need to consider the non-trivial case \(\mathbf{e}^{\top}\mathbf{g}\neq 0\). To proceed, we note that

\[(\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{g})^{2}\ =\ (\mathbf{e}^{\top}\mathbf{M}_{0}^{\frac{1} {2}}\mathbf{M}_{0}^{\frac{1}{2}}\mathbf{g})^{2}\ \leq\ (\mathbf{e}^{\top}\mathbf{M}_{0} \mathbf{e})(\mathbf{g}^{\top}\mathbf{M}_{0}\mathbf{g}), \tag{47}\]

where we have used the Cauchy-Schwartz inequality to bound \((\mathbf{e}^{\top}\mathbf{M}_{o}\mathbf{g})\) in terms of \((\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{e})\), the latter of which appears in eq. (46). Substituting this inequality into eq. (46), we find that

\[\lambda_{e}\ \geq\ \frac{(\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{g})^{2}}{\mathbf{g}^{ \top}\mathbf{M}_{0}\mathbf{g}}\ -\ \lambda_{e}^{2}(\mathbf{e}^{\top}\mathbf{g})^{2}. \tag{48}\]

To prove that \(\lambda_{e}>0\) we need one more intermediate result. Focusing on the rightmost term in this equality, we note that

\[\lambda_{e}(\mathbf{e}^{\top}\mathbf{g})\ =\ \mathbf{e}^{\top}\Sigma\mathbf{g}\ =\ \mathbf{e}^{\top} \left[\mathbf{M}_{0}-(\mathbf{\Sigma}\mathbf{g})(\mathbf{\Sigma}\mathbf{g})^{\top}\right]\mathbf{g} \ =\ \mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{g}-\lambda_{e}(\mathbf{e}^{\top}\mathbf{g})(\mathbf{g}^{ \top}\mathbf{\Sigma}\mathbf{g}), \tag{49}\]

and rearranging the terms in this equation, we find

\[\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{g}\ =\ \lambda_{e}(\mathbf{e}^{\top}\mathbf{g})(1+\mathbf{g}^{ \top}\mathbf{\Sigma}\mathbf{g}). \tag{50}\]

This intermediate result is useful because it relates the two terms on the right side of eq. (48). In particular, using eq. (50) to eliminate the term \(\mathbf{e}^{\top}\mathbf{M}_{0}\mathbf{g}\) in eq. (48), we find:

\[\lambda_{e} \geq \frac{\left[\lambda_{e}(\mathbf{e}^{\top}\mathbf{g})(1+\mathbf{g}^{\top}\mathbf{ \Sigma}\mathbf{g})\right]^{2}}{\mathbf{g}^{\top}\mathbf{M}_{0}\mathbf{g}}\ -\ \lambda_{e}^{2}(\mathbf{e}^{ \top}\mathbf{g})^{2}\] \[= \lambda_{e}^{2}(\mathbf{e}^{\top}\mathbf{g})^{2}\left[\frac{(1+\mathbf{g}^{ \top}\mathbf{\Sigma}\mathbf{g})^{2}}{\mathbf{g}^{\top}\mathbf{M}_{0}\mathbf{g}}\ -\ 1\right]\] \[= \lambda_{e}^{2}(\mathbf{e}^{\top}\mathbf{g})^{2}\left[\frac{(1+\rho)^{2} }{\rho(1+\rho)}\ -\ 1\right]\] \[= \frac{\lambda_{e}^{2}(\mathbf{e}^{\top}\mathbf{g})^{2}}{\rho},\] \[> 0,\]

where the final inequality follows because the individual terms \(\lambda_{e}^{2}\), \((\mathbf{e}^{\top}\mathbf{g})^{2}\), and \(\rho\) are all strictly positive; note that \(\lambda_{e}\) cannot be equal to zero because this contradicts the equality in eq. (46). This completes the proof. Perhaps it is useful that this derivation also gives upper bounds on \(\lambda_{e}\), namely

\[\frac{1}{\lambda_{e}}\ \geq\ \frac{(\mathbf{e}^{\top}\mathbf{g})^{2}}{\rho}\quad \Longrightarrow\quad\lambda_{e}\ \leq\ \frac{\rho}{(\mathbf{e}^{\top}\mathbf{g})^{2}}\ =\ \frac{\mathbf{g}^{\top}\mathbf{\Sigma}\mathbf{g}}{(\mathbf{e}^{\top}\mathbf{g})^{2}}. \tag{51}\]

Response:
## Review

### Summary

The paper presents a novel method for black-box variational inference (VI) based on score matching, specifically targeting Gaussian variational families. The approach derives a framework where the iterative updates of the variational distribution are guided by the alignment of gradients between the target and the variational distribution. This results in a method termed Gaussian Score Matching VI (GSM-VI), which is shown to outperform traditional BBVI in several experimental setups, particularly in terms of convergence speed and efficiency. However, the paper lacks a thorough theoretical analysis and detailed exploration of the methodology's performance beyond Gaussian distributions, with potential limitations regarding convergence in high-dimensional or non-Gaussian settings.

### Strengths

- The score matching approach is novel and effectively addresses variational inference.
- The paper is well-structured and clearly communicates complex ideas.
- GSM-VI shows significant performance improvements over standard BBVI in terms of convergence rates.
- The experimental results are robust and demonstrate the method's practical applicability.
- The availability of code promotes reproducibility.

### Weaknesses

- The paper is primarily empirical and lacks theoretical guarantees for the proposed method.
- There is insufficient explanation for the strong empirical results, particularly regarding scaling with dimension and batch size.
- The evaluation is limited to Gaussian distributions without exploring other potential variational families.
- Important experimental details, including the dimensionality of benchmarks and hyperparameter settings, are not well-documented.
- The paper does not adequately discuss the oscillatory behavior observed in experiments or its implications.

### Questions

- Why does GSM-VI demonstrate such effective scaling properties with dimensionality?
- What are the measured running times for the experiments, considering the O(d2) time complexity of the update step?
- Could the O(d2) update step negatively affect performance in very high-dimensional cases?
- What were the batch sizes used for BBVI in the benchmarks?
- How were the examples from the posteriordb selected and what is their dimensionality?
- Could the authors elaborate on the scenarios where the optimization could result in local minima?
- Can the paper clarify the implications of only reporting the Forward KL when comparing with other VI methods?

### Soundness

**Score:** 3

**Description:** Good - The methodology is promising and demonstrates empirical effectiveness, but lacks theoretical backing and comprehensive analysis of limitations, impacting the overall soundness.

### Presentation

**Score:** 3

**Description:** Good - The paper is generally well-written and clear, but could benefit from additional explanations and organization of results to enhance understanding.

### Contribution

**Score:** 3

**Description:** Good - The approach presents a novel insight into variational inference, but the limited exploration of non-Gaussian settings somewhat diminishes the overall contribution.

### Rating

**Score:** 7

**Description:** Accept: The paper demonstrates solid technical merit with a significant impact potential, but requires revisions to address theoretical gaps and clarify experimental details.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents an innovative framework for variational inference that shows promising empirical results. While it has some limitations, particularly regarding the theoretical analysis and exploration of non-Gaussian families, the overall contribution is significant. The clarity of presentation and the reproducibility of results further support the decision to accept the paper with minor revisions.