\section{Introduction}\label{sec:intro}

We argue that a very general family of covariance matrix distributions, known
as the \emph{Inverse G-Wishart} family, plays a fundamental role in modularization
of variational inference algorithms via variational message passing when a factor
graph fragment (Wand, 2017) approach is used. A factor graph fragment, or
\emph{fragment} for short, is a sub-graph of the relevant factor graph
consisting of a factor and all of its neighboring nodes. Even though use of the
Inverse G-Wishart distribution is not necessary, its adoption allows for fundamental
factor graph fragment natural parameter updates to be expressed elegantly and succinctly.
An essential aspect of this strategy is that the Inverse G-Wishart distribution
is the \emph{only} distribution used for covariance matrix and variance parameters.
The family includes as special cases the Inverse Chi-Squared, Inverse Gamma and
Inverse Wishart distributions. Therefore, just a single distribution is required
which leads to savings in notation and code. Whilst similar comments concerning modularity
apply to Monte Carlo-based approaches to approximate Bayesian inference, here
we focus on variational inference.

 Two of the most common contemporary approaches to fast approximate Bayesian inference
are mean field variational Bayes (e.g.\ Attias, 1999) and expectation propagation
(e.g.\ Minka, 2001). Minka (2005) explains how each approach can be expressed
as message passing on relevant \emph{factor graphs} with \emph{variational message passing}
(Winn \&\  Bishop, 2005) being the name used for the message passing version
of mean field variational Bayes. Wand (2017) introduced the concept of \emph{factor graph
fragments}, or \emph{fragments} for short, for compartmentalization of variational
message passing into atom-like components. Chen \&\  Wand (2020) demonstrate
the use of fragments for expectation propagation. Explanations of factor graph-based
variational message passing that match the current exposition are given in
Sections 2.4--2.5 of Wand (2017).

  Sections 4.1.2--4.1.3 of Wand (2017) introduce two variational message passing fragments
known as the \emph{Inverse Wishart prior} fragment and the \emph{iterated Inverse G-Wishart}
fragment. The first of these simply corresponds to imposing an Inverse Wishart prior
on a covariance matrix. In the scalar case this reduces to imposing an Inverse Chi-Squared
or, equivalently, an Inverse Gamma prior on a variance parameter. The iterated Inverse G-Wishart
fragment facilitates the imposition of arbitrarily non-informative priors on
standard deviation parameters such as members of the Half-$t$ family (Gelman, 2006
; Polson \&\  Scott, 2012). An extension to the covariance matrix case,
for which there is the option to impose marginal Uniform distribution priors
over the interval $(-1,1)$ on correlation parameters,
is elucidated in Huang \&\  Wand (2013). Mulder \&\  Pericchi (2018)
provide a different type of extension that is labelled the \emph{Matrix-$F$}
distribution. These two fragments arise in many classes of Bayesian models,
such as both Gaussian and generalized response linear mixed models
(e.g. McCulloch \textit{et al.}, 2008), Bayesian factor models
(e.g.  Conti \textit{et al.}, 2014), vector autoregressive models
(e.g. Assaf \textit{et al.}, 2019), and generalized additive mixed models
and group-specific curve models (e.g. Harezlak \textit{et al.}, 2018).

Despite the fundamentalness of Inverse G-Wishart-based fragments for variational
message passing, the main reference to date, Wand (2017), is brief in its exposition
and contains some errors that affect certain cases. In this article we provide
a detailed exposition of the Inverse G-Wishart distribution in the context of
variational message passing and list the Inverse Wishart prior and iterated Inverse G-Wishart
fragment updates in full ready-to-code forms. \textsf{R} functions
(\textsf{R} Core Team, 2020) that implement these algorithms are provided as
part of the supplementary material of this article.
We also explain the errors in Wand (2017).

Section \ref{sec:GWandIGW} contains relevant definitions and results concerning
the G-Wishart and Inverse G-Wishart distributions. Connections with the
Huang-Wand and Matrix-$F$ families of marginally noninformative prior distributions for
covariance matrices are summarized in Section \ref{sec:HWandMatrixFconns} and
in Section \ref{sec:VMPbackground} we point to background material
on variational message passing. In Sections \ref{sec:IGWprior}
and \ref{sec:iterIGWfrag} we provide detailed accounts
of the two variational message passing fragments pertaining
to variance and covariance matrix parameters, expanding on what is presented
in Sections 4.1.2 and 4.1.3 of Wand (2017), and making some corrections to what
is presented there. In Section \ref{sec:priorSpec} we provide explicit instructions on
how the two fragments are used to specify different types of prior distributions
on standard deviation and covariance matrix parameters in variational message
passing-based approximate Bayesian inference. Section \ref{sec:illustrative}
contains a data analytic example that illustrates the use of the covariance matrix
fragment update algorithms. Some closing discussion is given in Section \ref{sec:closing}.
A web-supplement contains relevant details.

\section{The G-Wishart and Inverse G-Wishart Distributions}\label{sec:GWandIGW}

A random matrix $\boldsymbol{X}$ has an Inverse G-Wishart distribution if and only if
$\boldsymbol{X}^{-1}$ has a G-Wishart distribution. In this section we first review
the G-Wishart distribution, which has an established literature.
Then we discuss the Inverse G-Wishart distribution and list properties
that are relevant to its employment in variational message passing.

Let $G$ be an undirected graph with $d$ nodes labeled $1,\ldots,d$ and
set $E$ consisting of pairs of nodes that are connected by an edge.
We say that the symmetric $d\times d$ matrix $\boldsymbol{M}$
\emph{respects} $G$ if
$$\boldsymbol{M}_{ij}=0\quad\mbox{for all}\quad \{i,j\}\notin E.$$
Figure \ref{fig:MandG} shows the zero/non-zero entries of
four $5\times5$ symmetric matrices. For each matrix, the $5$-node graph
that the matrix respects is shown underneath.

\begin{figure}[h]
\null\vskip4mm
$\Mone\quad\Mtwo\quad\Mthree\quad\Mfour$
\centering
     \subfigure{\includegraphics[width=0.22\textwidth]{MandGa.pdf}}
\quad\subfigure{\includegraphics[width=0.22\textwidth]{MandGb.pdf}}
\quad\subfigure{\includegraphics[width=0.22\textwidth]{MandGc.pdf}}
\quad\subfigure{\includegraphics[width=0.22\textwidth]{MandGd.pdf}}
\caption{\textit{The zero/non-zero entries of four $5\times5$ symmetric matrices
with non-zero entries denoted by ${\Large\mbox{$\times$}}$. Underneath each matrix is
the $5$-node undirected graph that the matrix respects.
The nodes are numbered according to the rows and
columns of the matrices. A graph edge is present between nodes $i$ and $j$ whenever
the $(i,j)$ entry of the matrix is non-zero. The graph respected by the
full matrix is denoted by} $G_{\mbox{\tiny{\rm full}}}$. \textit{The graph respected by the
diagonal matrix is denoted by} $G_{\mbox{\tiny{\rm diag}}}$.}
\label{fig:MandG}
\end{figure}

The first graph in Figure \ref{fig:MandG} is totally connected and
corresponds to the matrix being full. Hence we denote this graph
by $G_{\mbox{\tiny{\rm full}}}$. At the other end of the spectrum is the last graph of
Figure  \ref{fig:MandG}, which is totally disconnected. Since this
corresponds to the matrix being diagonal we denote this graph by
$G_{\mbox{\tiny{\rm diag}}}$.

An important concept in G-Wishart and Inverse G-Wishart distribution theory
is graph decomposability. An undirected graph $G$ is \emph{decomposable} if
and only if all cycles of four or more nodes have an edge that is
not part of the cycle but connects two nodes
of the cycle. In Figure \ref{fig:MandG} the first, third and fourth graphs are
decomposable. However, the second graph is not decomposable since it
contains a four-node cycle that is devoid of edges that connect pairs
of nodes within this cycle. Alternative labels
for decomposable graphs are \emph{chordal} graphs and \emph{triangulated} graphs.

In Sections \ref{sec:GWishartDistr} and \ref{sec:IGWdistn}
we define the G-Wishart and Inverse G-Wishart distributions and
treat important special cases. This exposition depends on particular
notation, which we define here. For a generic proposition ${\mathcal P}$ we
define $I({\mathcal P})$ to equal $1$ if ${\mathcal P}$ is true and zero otherwise.
If the random variables $x_j$, $1\le j\le d$, are independent
such that $x_j$ has distribution ${\mathcal D}_j$ we write $x_j\stackrel{{\tiny \mbox{ind.}}}{\sim}{\mathcal D}_j$,
$1\le j\le d$. For a $d\times 1$ vector $\boldsymbol{v}$ let $\mbox{diag}(\boldsymbol{v})$ be
the $d\times d$ diagonal matrix with diagonal comprising the entries of $\boldsymbol{v}$
in order. For a $d\times d$ matrix $\boldsymbol{M}$ let $\mbox{diagonal}(\boldsymbol{M})$ denote the
$d\times 1$ vector comprising the diagonal entries of $\boldsymbol{M}$ in order.
The $\mbox{\rm vec}$ and $\mbox{\rm vech}$ matrix operators are well-established
(e.g. Gentle, 2007). If $\boldsymbol{a}$ is a $d^2\times1$ vector then $\mbox{\rm vec}^{-1}(\boldsymbol{a})$ is the
$d\times d$ matrix such that $\mbox{\rm vec}\big(\mbox{\rm vec}^{-1}(\boldsymbol{a})\big)=\boldsymbol{a}$.
The matrix $\boldsymbol{D}_d$, known as the \emph{duplication matrix of order $d$},
is the $d^2\times\{\frac{1}{2}d(d+1)\}$ matrix containing only zeros and ones
such that $\boldsymbol{D}_d\mbox{\rm vech}(\boldsymbol{A})=\mbox{\rm vec}(\boldsymbol{A})$ for any symmetric $d\times d$ matrix $\boldsymbol{A}$
(Magnus \&\  Neudecker, 1999). For example,
$$\boldsymbol{D}_2=\left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{array}
\right].
$$
The Moore-Penrose inverse of $\boldsymbol{D}_d$
is $\boldsymbol{D}_d^+\equiv(\boldsymbol{D}_d^T\boldsymbol{D}_d)^{-1}\boldsymbol{D}_d^T$ and is such that
$\boldsymbol{D}_d^+\mbox{\rm vec}(\boldsymbol{A})=\mbox{\rm vech}(\boldsymbol{A})$ for a symmetric matrix $\boldsymbol{A}$.

\subsection{The G-Wishart Distribution}\label{sec:GWishartDistr}

The \textit{G-Wishart distribution} (Atay-Kayis \&\  Massam, 2005)
is defined as follows:
\begin{definition}
Let $\boldsymbol{X}$ be a $d\times d$ symmetric and
	positive definite random matrix and $G$ be a $d$-node undirected graph
	such that $\boldsymbol{X}$ respects $G$. For $\delta>0$ and a symmetric
        positive definite $d\times d$ matrix $\boldsymbol{\Lambda}$ we say that
        $\boldsymbol{X}$ has a G-Wishart distribution with graph $G$, shape parameter
        $\delta$ and rate matrix $\boldsymbol{\Lambda}$, and write
$$\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G,\delta,\boldsymbol{\Lambda}),$$
if and only if the non-zero values of the density function of $\boldsymbol{X}$ satisfy
\begin{equation}
p(\boldsymbol{X})\propto|\boldsymbol{X}|^{(\delta-2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X})\}.
\label{eq:GWishartDensKern}
\end{equation}
\label{def:GWdefn}
\end{definition}

Obtaining an expression for the normalizing factor of a general G-Wishart density function is a
challenging problem and recently was resolved by Uhler \textit{et al.} (2018). In the special case
where $G$ is a decomposable graph a relatively simple expression for the normalizing
factor exists and is given, for example, by equation (1.4) of Uhler \textit{et al.} (2018).
The non-decomposable case is much more difficult and treated in Section 3 of
Uhler \textit{et al.} (2018), but the normalizing factor does not have a succinct expression
for general $G$. Similar comments apply to expressions for the mean of a G-Wishart random matrix.
As discussed in Section 3 of Atay-Kayis \&\  Massam (2005), the G-Wishart distribution has
connections with other distributional constructs such as the hyper Wishart law defined by
Dawid \&\  Lauritzen (1993).

Let $G_{\mbox{\tiny{\rm full}}}$ be the totally connected $d$-node undirected graph and $G_{\mbox{\tiny{\rm diag}}}$ be
the totally disconnected $d$-node undirected graph. The special cases of $G=G_{\mbox{\tiny{\rm full}}}$
and $G=G_{\mbox{\tiny{\rm diag}}}$ are such that the normalizing factor and mean do have simple closed
form expressions. Since these cases arise in fundamental variational message passing
algorithms we now turn our attention to them.

\subsubsection{The $G=G_{\mbox{\tiny{\rm full}}}$ Special Case}

In the case where $G$ is a fully connected graph we have:

\begin{result}
If the $d\times d$ random matrix $\boldsymbol{X}$ is such that
$\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm full}}},\delta,\boldsymbol{\Lambda})$ then
\begin{equation}
\begin{array}{rcl}
	p(\boldsymbol{X})&=&\frac{|\boldsymbol{\Lambda}|^{(\delta+d-1)/2}}{2^{d(\delta+d-1)/2}\pi^{d(d-1)/4}
		\prod_{j=1}^d\Gamma(\frac{\delta+d-j}{2})}\,
	|\boldsymbol{X}|^{(\delta-2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X})\}\\[1ex]
	&&\quad\times
	I(\boldsymbol{X}\ \mbox{a symmetric and positive definite $d\times d$ matrix}).
\end{array}
\label{eq:pXfullCase}
\end{equation}
The mean of $\boldsymbol{X}$ is
$$E(\boldsymbol{X})=(\delta+d-1)\,\boldsymbol{\Lambda}^{-1}.$$
\label{res:GfullResult}
\end{result}

Result \ref{res:GfullResult} is not novel at all since the $G=G_{\mbox{\tiny{\rm full}}}$
case corresponds to $\boldsymbol{X}$ having a Wishart distribution.
In other words, (\ref{eq:pXfullCase}) is simply the density function of
a Wishart random matrix. However, it is worth pointing out the
the shape parameter used here is different from that commonly used
for the Wishart distribution. For example, in Table A.1 of
Gelman \textit{et al.} (2014) the shape parameter is denoted by $\nu$ and
is related to the shape parameter of (\ref{eq:pXfullCase}) according to
$$\nu=\delta+d-1$$
and therefore are the same only in the special case of $\boldsymbol{X}$ being scalar.
Also, note that Definition \ref{def:GWdefn} and  Result \ref{res:GfullResult}
use the rate matrix parameterisation, whereas Table A.1 of
Gelman \textit{et al.} (2014) uses the scale matrix parameterisation
for the Wishart distribution. The scale matrix is $\boldsymbol{\Lambda}^{-1}$.

\subsubsection{The $G=G_{\mbox{\tiny{\rm diag}}}$ Special Case}

Before treating the $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$ situation,
we define the notation
\begin{equation}
x\sim\mbox{Gamma}(\alpha,\beta)
\label{eq:FriedCigar}
\end{equation}
to mean that the scalar random variable $x$ has a Gamma distribution with
shape parameter $\alpha$ and rate parameter $\beta$. The density function corresponding
to (\ref{eq:FriedCigar}) is
$$p(x)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}\,x^{\alpha-1}\exp(-\beta\,x)I(x>0).$$

The $\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$ distribution is tied intimately
to the Gamma distribution, as Result \ref{res:GdiagResult} shows.

\begin{result}
Suppose that the $d\times d$ random matrix $\boldsymbol{X}$ is such that
$\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$.
Then the non-zero entries of $\boldsymbol{X}$ satisfy
$$X_{jj}\stackrel{{\tiny \mbox{ind.}}}{\sim}\mbox{Gamma}\big({\textstyle{\frac{1}{2}}}\delta,{\textstyle{\frac{1}{2}}}\Lambda_{jj}\big),\quad 1\le j\le d,$$
where $\Lambda_{jj}$ is the $j$th diagonal entry of $\boldsymbol{\Lambda}$.
The density function of $\boldsymbol{X}$ is
{\setlength\arraycolsep{3pt}
\begin{eqnarray*}
p(\boldsymbol{X})&=&\frac{|\boldsymbol{\Lambda}|^{\delta/2}}{2^{d\delta/2}\Gamma(\delta/2)^d}
\,|\boldsymbol{X}|^{(\delta-2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X})\}\prod_{j=1}^d I(X_{jj}>0)\\[1ex]
&=&\frac{\prod_{j=1}^d\Lambda_{jj}^{\delta/2}}{2^{d\delta/2}\Gamma(\delta/2)^d}\,
\prod_{j=1}^d\,X_{jj}^{(\delta-2)/2}\,\exp\left(-{\textstyle{\frac{1}{2}}}\sum_{j=1}^d\Lambda_{jj}\,X_{jj}\right)
\prod_{j=1}^d I(X_{jj}>0).
\end{eqnarray*}
}
The mean of $\boldsymbol{X}$ is
$$E(\boldsymbol{X})=\delta\,\boldsymbol{\Lambda}^{-1}=\delta\,\mbox{\rm diag}(1/\Lambda_{11},\ldots,1/\Lambda_{dd}).$$
\label{res:GdiagResult}
\end{result}

\noindent
We now make some remarks concerning Result \ref{res:GdiagResult}.
\begin{enumerate}
\item When $G=G_{\mbox{\tiny{\rm diag}}}$ the off-diagonal entries of $\boldsymbol{\Lambda}$ have no effect on
the distribution of $\boldsymbol{X}$. In other words, the declaration
$\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$ is equivalent to the declaration
$\boldsymbol{X}\sim\mbox{\rm G-Wishart}\big(G_{\mbox{\tiny{\rm diag}}},\delta,\mbox{diag}\{\mbox{diagonal}(\boldsymbol{\Lambda})\}\big)$.
\item The declaration $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$ is equivalent
to the diagonal entries of $\boldsymbol{X}$ being independent Gamma random variables with shape parameter
${\textstyle{\frac{1}{2}}}\delta$ and rate parameters equalling the diagonal entries of ${\textstyle{\frac{1}{2}}}\boldsymbol{\Lambda}$.
\item Even though statements concerning the distributions of independent random variables
may seem simpler than a statement of the form $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\delta,\boldsymbol{\Lambda})$,
the major thrust of this article is the elegance provided by key variational message passing
fragment updates being expressed in terms of a single family of distributions.
\end{enumerate}

\subsubsection{Exponential Family Form and Natural Parameterisation}

Suppose that $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G,\delta,\boldsymbol{\Lambda})$. Then for $\boldsymbol{X}$ such that $p(\boldsymbol{X})>0$
we have
\begin{equation}
p(\boldsymbol{X})\propto \exp\left\{
	\left[
	\begin{array}{c}
		\log|\boldsymbol{X}|\\
		\mbox{\rm vech}(\boldsymbol{X})
	\end{array}
	\right]^T
	\left[
	\begin{array}{c}
		{\textstyle{\frac{1}{2}}}(\delta-2)\\
		-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda})
	\end{array}
	\right]
	\right\}\\[1ex]=\exp\{\boldsymbol{T}(\boldsymbol{X})^T\boldsymbol{\eta}\}
\label{eq:vechFirst}
\end{equation}
where
$$
\boldsymbol{T}(\boldsymbol{X})\equiv\left[
\begin{array}{c}
\log|\boldsymbol{X}|\\
\mbox{\rm vech}(\boldsymbol{X})
\end{array}
\right]
\quad\mbox{and}\quad
\boldsymbol{\eta}
\equiv
\left[
\begin{array}{c}
\eta_1\\
\boldsymbol{\eta}_2
\end{array}
\right]
=
\left[
\begin{array}{c}
{\textstyle{\frac{1}{2}}}(\delta-2)\\
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda})
\end{array}
\right]
$$
are, respectively, sufficient statistic and natural parameter vectors.  The inverse of the natural parameter mapping is
$$
\left\{
\setlength\arraycolsep{1pt}{
\begin{array}{rcl}
\delta&=&2(\eta_1+1),\\[1ex]
\boldsymbol{\Lambda}&=&-2\,\mbox{\rm vec}^{-1}(\boldsymbol{D}_d^{+T}\boldsymbol{\eta}_2)
\end{array}.
}
\right.
$$
Note that, throughout this article, we use
$\mbox{\rm vech}(\boldsymbol{X})$ rather than $\mbox{\rm vec}(\boldsymbol{X})$ since the
former is more compact and avoids duplications.
Section \ref{sec:vecANDvech} in the web-supplement
has further discussion on this matter.

\subsection{The Inverse G-Wishart Distribution}\label{sec:IGWdistn}

Suppose that $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G,\delta,\boldsymbol{\Lambda})$,
where $\boldsymbol{X}$ is $d\times d$, and $\boldsymbol{Y}=\boldsymbol{X}^{-1}$.
Let the density functions of $\boldsymbol{X}$ and $\boldsymbol{Y}$ be denoted by $p_{\boldsymbol{X}}$
and $p_{\boldsymbol{Y}}$  respectively.
Then the density function of $\boldsymbol{Y}$ is
\begin{equation}
p_{\boldsymbol{Y}}(\boldsymbol{Y})=p_{\boldsymbol{X}}(\boldsymbol{Y}^{-1})\,|J(\boldsymbol{Y})|
\label{eq:IGWdensFunction}
\end{equation}
where
$$J(\boldsymbol{Y})\equiv\mbox{the determinant of}\ \frac{\partial\mbox{\rm vec}(\boldsymbol{Y}^{-1})}{\partial\mbox{\rm vec}(\boldsymbol{Y})^T}$$
is the Jacobian of the transformation.

An important observation is that the form of $J(\boldsymbol{Y})$
is dependent on the graph $G$. In the case of $G$ being a
decomposable graph an expression for $J(\boldsymbol{Y})$ is given by
(2.4) of Letac \&\  Massam (2007), with credit given to Roverato (2000).
Therefore, if $G$ is decomposable, the density
function of an Inverse G-Wishart random matrix can be obtained by substitution
of (2.4) of Letac \&\  Massam (2007) into (\ref{eq:IGWdensFunction}).
However, depending on the complexity of $G$, simplification
of the density function expression may be challenging.

With variational message passing in mind, we now turn to the $G=G_{\mbox{\tiny{\rm full}}}$ and $G=G_{\mbox{\tiny{\rm diag}}}$ special
cases. The $G=G_{\mbox{\tiny{\rm diag}}}$ case is simple since it involves products of univariate density functions
and we have
\begin{equation}
\mbox{if $G=G_{\mbox{\tiny{\rm diag}}}$ then $|J(\boldsymbol{Y})|=|\boldsymbol{Y}|^{-2}$ for any $d\in{\mathbb N}$}.
\label{eq:JacForDiag}
\end{equation}
The $G=G_{\mbox{\tiny{\rm full}}}$ case is more challenging and is the focus of
Theorem 2.1.8 of Muirhead (1982):
\begin{equation}
\mbox{if $G=G_{\mbox{\tiny{\rm full}}}$ then $|J(\boldsymbol{Y})|=|\boldsymbol{Y}|^{-(d+1)}$}.
\label{eq:JacForFull}
\end{equation}
This result is also stated as Lemma 2.1 in Letac \&\  Massam (2007).

Combining (\ref{eq:IGWdensFunction}), (\ref{eq:JacForDiag}) and (\ref{eq:JacForFull})
we have:

\begin{result}
Suppose that $\boldsymbol{Y}=\boldsymbol{X}^{-1}$ where $\boldsymbol{X}\sim\mbox{\rm G-Wishart}(G,\delta,\boldsymbol{\Lambda})$
and $\boldsymbol{X}$ is $d\times d$.
\begin{itemize}
        %
	\item[(a)] If $G=G_{\mbox{\tiny{\rm full}}}$ then $p(\boldsymbol{Y})\propto|\boldsymbol{Y}|^{-(\delta+2d)/2}
                   \exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{Y}^{-1})\}$.
        %
	\item[(b)] If $G=G_{\mbox{\tiny{\rm diag}}}$ then $p(\boldsymbol{Y})\propto|\boldsymbol{Y}|^{-(\delta+2)/2}
                   \exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{Y}^{-1})\}$.
	%
\end{itemize}
\label{res:GWtoIGW}
\end{result}

Whilst Result \ref{res:GWtoIGW} only covers $G=G_{\mbox{\tiny{\rm full}}}$ or $G=G_{\mbox{\tiny{\rm diag}}}$ it shows that,
in these special cases, the density function of an Inverse G-Wishart random matrix $\boldsymbol{Y}$
is proportional to a power of $|\boldsymbol{Y}|$ multiplied by an an exponentiated trace of a matrix
multiplied by $\boldsymbol{Y}^{-1}$. This form does not necessarily arise for $G\notin\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$.
Since the motivating variational message passing fragment update algorithms only involve the
$G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ cases we focus on them for the remainder of this section.

\subsubsection{The Inverse G-Wishart Distribution When $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$}

For succinct statement of variational message passing fragment update algorithms
involving variance and covariance matrix parameters it is advantageous to have
a single Inverse G-Wishart distribution notation for the $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$
cases.

\begin{definition}
Let $\boldsymbol{X}$ be a $d\times d$ symmetric and
	positive definite random matrix and $G$ be a $d$-node undirected graph
	such that $\boldsymbol{X}^{-1}$ respects $G$. Let $\xi>0$ and $\boldsymbol{\Lambda}$ be a symmetric
        positive definite $d\times d$ matrix $\boldsymbol{\Lambda}$.
\begin{itemize}
	%
	\item[(a)] If $G=G_{\mbox{\tiny{\rm full}}}$ and $\xi$ is restricted such that $\xi>2d-2$ then we
                   say that $\boldsymbol{X}$ has an Inverse G-Wishart distribution with graph $G$,
                   shape parameter $\xi$ and scale matrix $\boldsymbol{\Lambda}$, and write
                   %
                   $$\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G,\xi,\boldsymbol{\Lambda}),$$
                   %
                   if and only if the non-zero values of the density function of $\boldsymbol{X}$ satisfy
$$p(\boldsymbol{X})\propto|\boldsymbol{X}|^{-(\xi+2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X}^{-1})\}.$$
	\item[(b)]If $G=G_{\mbox{\tiny{\rm diag}}}$ then say that $\boldsymbol{X}$ has an Inverse G-Wishart distribution with graph $G$,
                   shape parameter $\xi$ and scale matrix $\boldsymbol{\Lambda}$, and write
                   %
                   $$\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G,\xi,\boldsymbol{\Lambda}),$$
                   %
                   if and only if the non-zero values of the density function of $\boldsymbol{X}$ satisfy
$$p(\boldsymbol{X})\propto|\boldsymbol{X}|^{-(\xi+2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X}^{-1})\}.$$

	%
	\item[(c)] If $G\notin\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ then $\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G,\xi,\boldsymbol{\Lambda})$ is not defined.
	%
\end{itemize}
\label{def:IGWdefn}
\end{definition}

The shape parameter $\xi$ used in Definition \ref{def:IGWdefn} is a reasonable compromise between
various competing parameterisation choices for the Inverse G-Wishart distribution
for $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ and for use in variational message passing algorithms. It has
the following attractions:
\begin{itemize}
\item The exponent of the determinant in the density function expression is $-(\xi+2)/2$ regardless
of whether $G=G_{\mbox{\tiny{\rm full}}}$ or $G=G_{\mbox{\tiny{\rm diag}}}$, which is consistent with the G-Wishart distributional
notation used in Definition \ref{def:GWdefn}.
\item In the $d=1$ case $\xi$ matches the shape parameter in the most common parameterisation
of the Inverse Chi-Squared distribution such as that used in Table A.1 of Gelman \textit{et al.} (2014).
\end{itemize}

In case where $\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G_{\mbox{\tiny{\rm full}}},\xi,\boldsymbol{\Lambda})$ we have
the following:

\begin{result}
If the $d\times d$ random matrix $\boldsymbol{X}$ is such that
$\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G_{\mbox{\tiny{\rm full}}},\xi,\boldsymbol{\Lambda})$ then
$$
\begin{array}{rcl}
	p(\boldsymbol{X})&=&\displaystyle{\frac{|\boldsymbol{\Lambda}|^{(\xi-d+1)/2}}
                  {2^{d(\xi-d+1)/2}\pi^{d(d-1)/4}
		\prod_{j=1}^d\Gamma(\frac{\xi-d-j}{2}+1)}}\,
	|\boldsymbol{X}|^{-(\xi+2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X}^{-1})\}\\[1ex]
	&&\quad\times
	I(\boldsymbol{X}\ \mbox{a symmetric and positive definite $d\times d$ matrix}).
\end{array}
$$
The mean of $\boldsymbol{X}^{-1}$ is
$$E(\boldsymbol{X}^{-1})=(\xi-d+1)\,\boldsymbol{\Lambda}^{-1}.$$
\label{res:IGfullWres}
\end{result}

Result \ref{res:IGfullWres} follows directly from the fact that
$\boldsymbol{X}\sim\mbox{\rm Inverse G-Wishart}(G_{\mbox{\tiny{\rm full}}},\xi,\boldsymbol{\Lambda})$ if and only if
$\boldsymbol{X}$ has an Inverse Wishart distribution and established
results for the density function and mean of this distribution
given in, for example, Table A.1 of Gelman \textit{et al.} (2014).

We now deal with the $G=G_{\mbox{\tiny{\rm diag}}}$ case.

\begin{definition}
Let $x$ be a random variable.
For $\delta>0$ and $\lambda>0$
we say that the random variable $x$ has an Inverse Chi-Squared
distribution with shape parameter $\delta$ and rate parameter
$\lambda$, and write
$$x\sim\mbox{{\rm Inverse}-$\chi^2$}(\delta,\lambda),$$
if and only if $1/x\sim\chi^2(\delta,\lambda)$. If $x\sim\mbox{{\rm Inverse}-$\chi^2$}(\delta,\lambda)$
then the density function of $x$ is
$$p(x)=\frac{(\lambda/2)^{\delta/2}}{\Gamma(\delta/2)}
x^{-(\delta+2)/2}\,\exp\{-(\lambda/2)\big/x\}I(x>0).
$$
\end{definition}

\noindent
We are now ready to state:
\begin{result}
Suppose that the $d\times d$ random matrix $\boldsymbol{X}$ is such that
$\boldsymbol{X}\sim\mbox{\rm Inverse-G-Wishart}(G_{\mbox{\tiny{\rm diag}}},\xi,\boldsymbol{\Lambda})$.
Then the non-zero entries of $\boldsymbol{X}$ satisfy
$$X_{jj}\stackrel{{\tiny \mbox{ind.}}}{\sim}\mbox{{\rm Inverse}-$\chi^2$}(\xi,\Lambda_{jj}),\quad 1\le j\le d,$$
where $\Lambda_{jj}$ is the $j$th diagonal entry of $\boldsymbol{\Lambda}$.
The density function of $\boldsymbol{X}$ is
{\setlength\arraycolsep{3pt}
\begin{eqnarray*}
p(\boldsymbol{X})&=&\frac{|\boldsymbol{\Lambda}|^{\xi/2}}{2^{d\xi/2}\Gamma(\xi/2)^d}
\,|\boldsymbol{X}|^{-(\xi+2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X}^{-1})\}\prod_{j=1}^d I(X_{jj}>0)\\[1ex]
&=&\frac{\prod_{j=1}^d\Lambda_{jj}^{\xi/2}}{2^{d\xi/2}\Gamma(\xi/2)^d}
\prod_{j=1}^d\,X_{jj}^{-(\xi+2)/2}\,\exp\left\{-{\textstyle{\frac{1}{2}}}\sum_{j=1}^d(\Lambda_{jj}/X_{jj})\right\}
\prod_{j=1}^d I(X_{jj}>0).
\end{eqnarray*}
}
The mean of $\boldsymbol{X}^{-1}$ is
$$E(\boldsymbol{X}^{-1})=\xi\boldsymbol{\Lambda}^{-1}=\xi\,\mbox{\rm diag}(1/\Lambda_{11},\ldots,1/\Lambda_{dd}).$$
\end{result}

\subsubsection{Natural Parameter Forms and Sufficient Statistic Expectations}

Suppose that $\boldsymbol{X}\sim\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{\Lambda})$ where $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$.
Then for $\boldsymbol{X}$ such that $p(\boldsymbol{X})>0$,
$$
	p(\boldsymbol{X})\propto\exp\left\{
	\left[
	\begin{array}{c}
		\log|\boldsymbol{X}|\\
		\mbox{\rm vech}(\boldsymbol{X}^{-1})
	\end{array}
	\right]^T
	\left[
	\begin{array}{c}
		-(\xi+2)/2\\
		-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda})
	\end{array}
	\right]
	\right\}=\exp\{\boldsymbol{T}(\boldsymbol{X})^T\boldsymbol{\eta}\}
$$
where
\begin{equation}
\boldsymbol{T}(\boldsymbol{X})\equiv\left[
\begin{array}{c}
\log|\boldsymbol{X}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{X}^{-1})
\end{array}
\right]
\quad\mbox{and}\quad
\boldsymbol{\eta}
\equiv
\left[
\begin{array}{c}
\eta_1\\
\boldsymbol{\eta}_2
\end{array}
\right]
=
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}(\xi+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda})
\end{array}
\right]
\label{eq:IsawLara}
\end{equation}
are, respectively, sufficient statistic and natural parameter vectors.
The inverse of the natural parameter mapping is
\begin{equation}
\left\{
\begin{array}{rcl}
\xi&=&-2\eta_1-2,\\[1ex]
\boldsymbol{\Lambda}&=&-2\,\mbox{\rm vec}^{-1}(\boldsymbol{D}_d^{+T}\boldsymbol{\eta}_2).
\end{array}
\right.
\label{eq:IGWinvMap}
\end{equation}
As explained in Section \ref{sec:vecANDvech} of the web-supplement,
alternatives to (\ref{eq:IsawLara}) are those that use $\mbox{\rm vec}(\boldsymbol{X})$
instead of $\mbox{\rm vech}(\boldsymbol{X})$. Throughout this article we use the more
compact ``$\mbox{\rm vech}$'' form.

The following result is fundamental to succinct formulation of
updates of covariance and variance parameter fragment updates
for variational message passing:

\begin{result}
If $\boldsymbol{X}$ is a $d\times d$ random matrix that has an Inverse G-Wishart distribution
with graph $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ and natural parameter vector $\boldsymbol{\eta}$. Then
$$E(\boldsymbol{X}^{-1})=\left\{
\begin{array}{ll}
\{\eta_1+{\textstyle{\frac{1}{2}}}(d+1)\}\{\mbox{\rm vec}^{-1}(\boldsymbol{D}_d^{+T}\boldsymbol{\eta}_2)\}^{-1} & \mbox{if}\ \ G=G_{\mbox{\tiny{\rm full}}}\\[1.5ex]
(\eta_1+1)\{\mbox{\rm vec}^{-1}(\boldsymbol{D}_d^{+T}\boldsymbol{\eta}_2)\}^{-1}& \mbox{if}\ \ G=G_{\mbox{\tiny{\rm diag}}}.
\end{array}
\right.
$$
\label{eq:recipMoment}
\end{result}

\subsubsection{Relationships with the Hyper Inverse Wishart Distributions}

Throughout this article we follow the G-Wishart nomenclature
as used by, for example, Atay-Kayis \&\  Massam (2005),
Letac \&\  Massam (2007) and Uhler \textit{et al.} (2018)
in our naming of the Inverse G-Wishart family.
Some earlier articles, such as Roverato (2000), use the
term \emph{Hyper Inverse Wishart} for the same
family of distributions. The naming used here is in keeping
with the more recent literature concerning Wishart distributions
with graphical restrictions.

\section{Connections with Some Recent Covariance Matrix Distributions}\label{sec:HWandMatrixFconns}

Recently Huang \&\  Wand (2013) and Mulder \&\  Pericchi (2018) developed
covariance matrix distributional families that have attractions in terms
of the types of marginal prior distributions that can be imposed on
interpretable parameters within the covariance matrix. Mulder \&\  Pericchi (2018)
referred to their proposal as the \emph{Matrix-F} family of distributions.

\subsection{The Huang-Wand Family of Distributions}

A major motivation for working with the Inverse G-Wishart distribution is the fact that
the family of marginally non-informative priors proposed in Huang \&\  Wand (2013)
can be expressed succinctly in terms of the $\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{\Lambda})$
family  where $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$. This means that variational message fragments that
cater for Huang-Wand prior specification, as well as Inverse-Wishart prior specification,
only require natural parameter vector manipulations within a single distributional family.

If $\boldsymbol{\Sigma}$ is a $d\times d$ symmetric positive definite matrix then, for
$\nu_{\mbox{\tiny HW}}>0$ and $s_1,\ldots,s_d>0$, the specification
\begin{equation}
\begin{array}{c}
\boldsymbol{\Sigma}|\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}\Big(G_{\mbox{\tiny{\rm full}}},\nu_{\mbox{\tiny HW}}+2d-2,\boldsymbol{A}^{-1}\Big),\\[2ex]
\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}\Big(G_{\mbox{\tiny{\rm diag}}},1,\big\{\nu_{\mbox{\tiny HW}}\,\mbox{diag}(s_1^2,\ldots,s_d^2)\big\}^{-1}\Big)
\end{array}
\label{eq:HuangWandGeneralPrior}
\end{equation}
places a distribution of the type given in Huang \&\  Wand (2013)
on $\boldsymbol{\Sigma}$ with shape parameter $\nu_{\mbox{\tiny HW}}$ and scale parameters
$s_1,\ldots,s_d$.

The specification (\ref{eq:HuangWandGeneralPrior}) matches (2) of Huang \&\  Wand (2013) but
with some differences in notation. Firstly, $d$ is used for matrix dimension here rather
than $p$ in Huang \&\  Wand (2013). Also, the $s_j$, $1\le j\le d$,
scale parameters are denoted by $A_j$
in  Huang \&\  Wand (2013). The $a_j$ auxiliary variables in (2) of Huang \&\  Wand (2013)
are related to the matrix $\boldsymbol{A}$ via the expression $\mbox{diag}(a_1,\ldots,a_d)=2\nu_{\mbox{\tiny HW}}\boldsymbol{A}$.

As discussed in Huang \&\  Wand (2013), special cases of (\ref{eq:HuangWandGeneralPrior})
correspond to marginally noninformative prior specification of the covariance matrix $\boldsymbol{\Sigma}$
in the sense that the standard deviation parameters $\sigma_j\equiv(\boldsymbol{\Sigma})^{1/2}_{jj}$,
$1\le j\le d$,
can have Half-$t$ priors with arbitrarily large scale parameters,
controlled by the $s_j$ values.
This is in keeping with the advice given in Gelman (2006). Moreover, correlation parameters
$\rho_{jj'}\equiv(\boldsymbol{\Sigma})^{1/2}_{jj'}/(\sigma_j\sigma_{j'})$, for each $j\ne j'$ pair, have
a Uniform distribution over the interval $(-1,1)$ when $\nu_{\mbox{\tiny HW}}=2$.
We refer to this special case as the \emph{Huang-Wand} marginally non-informative prior
distribution with scale parameters $s_1,\ldots,s_d$ and write
\begin{equation}
\boldsymbol{\Sigma}\sim\mbox{Huang-Wand}(s_1,\ldots,s_d)
\label{eq:HWprior}
\end{equation}
as a shorthand for (\ref{eq:HuangWandGeneralPrior}) with $\nu_{\mbox{\tiny HW}}=2$.

\subsection{The Matrix-$F$ Family of Distributions}

For  $\nu_{\mbox{\tiny MP}}>d-1$, $\delta_{\mbox{\tiny MP}}>0$ and $\boldsymbol{B}_{\mbox{\tiny MP}}$ a $d\times d$ symmetric positive
definite matrix Mulder \&\  Pericchi (2018) defined
a $d\times d$ random matrix $\boldsymbol{\Sigma}$ to have a Matrix-$F$ distribution,
written
\begin{equation}
\boldsymbol{\Sigma}\sim F(\nu_{\mbox{\tiny MP}},\delta_{\mbox{\tiny MP}},\boldsymbol{B}_{\mbox{\tiny MP}}),
\label{eq:CockOfTheNorth}
\end{equation}
if its density function has the form
$$p(\boldsymbol{\Sigma})\propto \frac{|\boldsymbol{\Sigma}|^{(\nu_{\mbox{\tiny MP}}-d-1)/2}\,
I(\boldsymbol{\Sigma}\ \mbox{symmetric and positive definite})}
{|\boldsymbol{I}_d+\boldsymbol{\Sigma}\boldsymbol{B}_{\mbox{\tiny MP}}^{-1}|^{(\nu_{\mbox{\tiny MP}}+\delta_{\mbox{\tiny MP}}+d-1)/2}}.
$$
However, standard manipulations of results given in Mulder \&\  Pericchi (2018)
show that specification (\ref{eq:CockOfTheNorth}) is equivalent to
\begin{equation}
\begin{array}{c}
\boldsymbol{\Sigma}|\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}\Big(G_{\mbox{\tiny{\rm full}}},\delta_{\mbox{\tiny MP}}+2d-2,\boldsymbol{A}^{-1}\Big),\\[2ex]
\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}\Big(G_{\mbox{\tiny{\rm full}}},\nu_{\mbox{\tiny MP}}+d-1,\boldsymbol{B}_{\mbox{\tiny MP}}^{-1}\Big)
\end{array}
\label{eq:FMatrixPrior}
\end{equation}
in the notation used in the current paper. An important difference
between (\ref{eq:HuangWandGeneralPrior}) and
(\ref{eq:FMatrixPrior}) is that the former involves $\boldsymbol{A}$ having
an Inverse G-Wishart distribution with the restriction $G=G_{\mbox{\tiny{\rm diag}}}$, whilst the
latter has $G=G_{\mbox{\tiny{\rm full}}}$. Section 2.4 of Mulder \&\  Pericchi (2018)
compares the two specifications in terms of the types of prior
distributions that can be imposed on standard deviation and
correlation parameters.

\section{Variational Message Passing Background}\label{sec:VMPbackground}

The overarching goal of this article is to identify and specify
algebraic primitives for flexible imposition of covariance matrix priors
within a variational message passing framework. In Wand (2017) these
algebraic primitives are organised into fragments. This formalism is
also used in Nolan \&\  Wand (2017), Maestrini \&\  Wand (2018) and
McLean \&\  Wand (2019).

  Despite it being a central theme of this article, we will not provide a
detailed description of variational message passing here. Instead
we refer the reader to Sections 2--4 of Wand (2017) for the relevant
variational message passing background material.

Since the notational conventions for messages used in this section's references
are used in the remainder of this article we summarize them here.
If $f$ denotes a generic factor and $\theta$ denotes a generic stochastic
variable that is a neighbour of $f$ in the factor graph then the message
passed from $f$ to $\theta$ and the message passed from $\theta$ to $f$
are both functions of $\theta$ and are denoted by, respectively,
$$\biggerm_{\mbox{\scriptsize$f\rightarrow\theta$}}(\theta)\quad\mbox{and}\quad\biggerm_{\mbox{\scriptsize$\theta\rightarrow f$}}(\theta).$$
Typically, the messages are proportional to an exponential family density function
with sufficient statistic $\boldsymbol{T}(\theta)$, and we have
$$\biggerm_{\mbox{\scriptsize$f\rightarrow\theta$}}(\theta)\propto\exp\left\{\boldsymbol{T}(\theta)^T\biggerbdeta_{\mbox{\scriptsize$f\rightarrow\theta$}}\right\}
\quad\mbox{and}\quad
\biggerm_{\mbox{\scriptsize$\theta\rightarrow f$}}(\theta)\propto\exp\left\{\boldsymbol{T}(\theta)^T\biggerbdeta_{\mbox{\scriptsize$\theta\rightarrow f$}}\right\}
$$
where $\biggerbdeta_{\mbox{\scriptsize$f\rightarrow\theta$}}$ and $\biggerbdeta_{\mbox{\scriptsize$\theta\rightarrow f$}}$ are the message natural
parameter vectors. Such vectors play a central role in variational message passing
iterative algorithms. We also adopt the notation
$$\biggerbdeta_{\mbox{\scriptsize$f\leftrightarrow\theta$}}\equiv\biggerbdeta_{\mbox{\scriptsize$f\rightarrow\theta$}}+\biggerbdeta_{\mbox{\scriptsize$\theta\rightarrow f$}}.$$

\section{The Inverse G-Wishart Prior Fragment}\label{sec:IGWprior}

The Inverse G-Wishart prior fragment corresponds to the following prior
imposition on a $d\times d$ covariance matrix $\boldsymbol{\Theta}$:
$$\boldsymbol{\Theta}\sim\mbox{Inverse-G-Wishart}(G_{\scriptscriptstyle\boldsymbol{\Theta}},\xi_{\scriptscriptstyle\boldsymbol{\Theta}},\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}})$$
for a $d$-node undirected graph $G_{\scriptscriptstyle\boldsymbol{\Theta}}$, scalar shape parameter $\xi_{\scriptscriptstyle\boldsymbol{\Theta}}$
and scale matrix $\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}$. The fragment's factor is
{\setlength\arraycolsep{3pt}
\begin{eqnarray*}
p(\boldsymbol{\Theta})&\propto&|\boldsymbol{\Theta}|^{-(\xi_{\scriptscriptstyle\boldsymbol{\Theta}}+2)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}\boldsymbol{\Theta}^{-1})\}\\[2ex]
&&\quad\times I(\boldsymbol{\Theta}\ \mbox{is symmetric and positive definite and}\ \boldsymbol{\Theta}^{-1}\ \mbox{respects}\ G_{\scriptscriptstyle\boldsymbol{\Theta}}).
\end{eqnarray*}
}

\begin{figure}[h]
\centering
{\includegraphics[width=0.25\textwidth]{InvGWishPriorFrag.pdf}}
\caption{\it Diagram of the Inverse G-Wishart prior fragment.}
\label{fig:InvGWishPriorFrag}
\end{figure}

Figure \ref{fig:InvGWishPriorFrag} is a diagram of the fragment, which shows that its only
factor to stochastic node message is
$$\biggerm_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}(\bTheta)\propto p(\boldsymbol{\Theta})$$
which leads to
$$\biggerm_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}(\bTheta)=\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{\Theta}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{\Theta}^{-1})
\end{array}
\right]^T
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi_{\scriptscriptstyle\boldsymbol{\Theta}}+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\,\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}})
\end{array}
\right]
\right\}.
$$
Therefore, the natural parameter update is
$$\biggerbdeta_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}\longleftarrow\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi_{\scriptscriptstyle\boldsymbol{\Theta}}+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\,\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}})
\end{array}
\right].
$$
Apart from passing the natural parameter vector out of the fragment, we
should also pass the graph out of the fragment. This entails the update:
$$G_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}\longleftarrow G_{\scriptscriptstyle\boldsymbol{\Theta}}.$$

Algorithm \ref{alg:InvGWishPriorFrag} provides the inputs,
updates and outputs for the Inverse G-Wishart prior fragment.

\begin{algorithm}[!th]
\begin{center}
\begin{minipage}[t]{165mm}
\textbf{Hyperparameter Inputs:} $G_{\scriptscriptstyle\boldsymbol{\Theta}},\xi_{\scriptscriptstyle\boldsymbol{\Theta}},\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}$.
\vskip3mm\noindent\noindent
\textbf{Updates:}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}\longleftarrow\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi_{\scriptscriptstyle\boldsymbol{\Theta}}+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\,\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}})
\end{array}
\right]$\ \ \ ;\ \ \ $G_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}\longleftarrow G_{\scriptscriptstyle\boldsymbol{\Theta}}$
\end{itemize}
\textbf{Outputs:} $G_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}$.
\vskip3mm\noindent
\end{minipage}
\end{center}
\caption{\it The inputs, updates and outputs for the Inverse G-Wishart prior fragment.}
\label{alg:InvGWishPriorFrag}
\end{algorithm}

\section{The Iterated Inverse G-Wishart Fragment}\label{sec:iterIGWfrag}

The iterated Inverse G-Wishart fragment corresponds to the
following specification involving a $d\times d$ covariance
matrix $\boldsymbol{\Sigma}$:
$$\boldsymbol{\Sigma}|\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{A}^{-1})$$
where $G$ is a $d$-node undirected graph such that $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ and
$\xi$ is a particular deterministic value of the Inverse G-Wishart shape parameter
according to Definition \ref{def:IGWdefn}. Figure \ref{fig:iterInvGWishFrag}
is a diagram of this fragment, showing that it has a
factor $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ connected to two stochastic nodes
$\boldsymbol{\Sigma}$ and $\boldsymbol{A}$.

\begin{figure}[h]
\centering
{\includegraphics[width=0.4\textwidth]{iterInvGWishFrag.pdf}}
\caption{\it Diagram of the iterated Inverse G-Wishart fragment.}
\label{fig:iterInvGWishFrag}
\end{figure}

The factor of the iterated Inverse G-Wishart fragment is,
as a function of both $\boldsymbol{\Sigma}$ and $\boldsymbol{A}$,
{\setlength\arraycolsep{3pt}
\begin{eqnarray*}
p(\boldsymbol{\Sigma}|\boldsymbol{A})\propto\left\{\begin{array}{l}
|\boldsymbol{A}|^{-(\xi-d+1)/2}|\boldsymbol{\Sigma}|^{-(\xi+2)/2}\,
\exp\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{A}^{-1}\boldsymbol{\Sigma}^{-1})\}\quad\mbox{if $G=G_{\mbox{\tiny{\rm full}}}$,}\\[2ex]
|\boldsymbol{A}|^{-\xi/2}|\boldsymbol{\Sigma}|^{-(\xi+2)/2}\,\exp\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{A}^{-1}\boldsymbol{\Sigma}^{-1})\}
\quad\mbox{if $G=G_{\mbox{\tiny{\rm diag}}}$.}
\end{array}
\right.
\end{eqnarray*}
}
As shown in Section \ref{sec:drvFirstMsg} of the web-supplement both
of the factor to stochastic node messages of this fragment,
$$\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}(\bSigma)\quad\mbox{and}\quad\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}(\bA),$$
are proportional to Inverse G-Wishart density functions with graph $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$.
We assume the following conjugacy constraints:

\begin{center}
\begin{minipage}[t]{130mm}
All messages passed to $\boldsymbol{\Sigma}$ and $\boldsymbol{A}$ from outside
the fragment are proportional to Inverse G-Wishart density functions
with graph $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$. The Inverse G-Wishart messages
passed between $\boldsymbol{\Sigma}$ and $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ have the same graph.
The Inverse G-Wishart messages
passed between $\boldsymbol{A}$ and $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ have the same graph.
\end{minipage}
\end{center}

\noindent
Under these constraints, and in view of e.g. (7) of Wand (2017), the message
passed from $\boldsymbol{\Sigma}$ to $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ has the form
$$
\biggerm_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}(\bSigma)=
\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{\Sigma}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]^T\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}\right\}
$$
and the message passed from $\boldsymbol{A}$ to $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ has the form
$$
\biggerm_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}(\bA)=
\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{A}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{A}^{-1})
\end{array}
\right]^T\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}\right\}.
$$
Algorithm \ref{alg:IterInvGWishFrag} gives the full set of updates
of the message natural parameter vectors and graphs
for the iterated Inverse-G-Wishart fragment. The derivation
of Algorithm \ref{alg:IterInvGWishFrag} is given
in Section \ref{sec:mainAlgDrv} of the web-supplement.

\begin{algorithm}[!th]
\begin{center}
\begin{minipage}[t]{165mm}
\textbf{Graph Input:} $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$.\\[1ex]
\textbf{Shape Parameter Input:} $\xi>0$.\\[1ex]
\textbf{Message Graph Input:}\ $G_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$.\\[1ex]
\textbf{Natural Parameter Inputs:} $\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}$,\ $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}$,\
$\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$,\ $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}$.\\[1ex]
\textbf{Updates:}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] $\GSUBpSigmaATOSigma\longleftarrow G$\ \ \ ;\ \ \
        $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\longleftarrowG_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$
\item[] $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}+\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}$
\item[] $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}+\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$
\item[] If $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm full}}}$ then $\omega_1\longleftarrow(d+1)/2$
\item[] If $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm diag}}}$ then $\omega_1\longleftarrow 1$
\item[] $E_{q}(\boldsymbol{A}^{-1})\longleftarrow\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_1+\omega_1\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_2\Big)\right\}^{-1}$
\item[] If $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm diag}}}$ then
$E_{q}(\boldsymbol{A}^{-1})\longleftarrow\mbox{diag}\left\{\mbox{diagonal}\Big(E_{q}(\boldsymbol{A}^{-1})\Big)\right\}$
\item[] $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}\longleftarrow
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}\Big(E_{q}(\boldsymbol{A}^{-1})\Big)
\end{array}
\right]$
\item[] If $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm full}}}$ then $\omega_2\longleftarrow(d+1)/2$
\item[] If $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm diag}}}$ then $\omega_2\longleftarrow 1$
\item[] $E_{q}(\boldsymbol{\Sigma}^{-1})\longleftarrow\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_1+\omega_2\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_2\Big)\right\}^{-1}$
\item[]If $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm diag}}}$ then
$E_{q}(\boldsymbol{\Sigma}^{-1})\longleftarrow\mbox{diag}\left\{\mbox{diagonal}\Big(E_{q}(\boldsymbol{\Sigma}^{-1})\Big)\right\}$
\item[] $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\longleftarrow
\left[
\begin{array}{c}
-(\xi+2-2\omega_2)/2 \\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}\Big(E_{q}(\boldsymbol{\Sigma}^{-1})\Big)
\end{array}
\right]$
\end{itemize}
\textbf{Outputs:} $\GSUBpSigmaATOSigma,G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}},\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}},\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}$.
\vskip3mm\noindent
\end{minipage}
\end{center}
\caption{\it The inputs, updates and outputs for the iterated Inverse G-Wishart fragment.}
\label{alg:IterInvGWishFrag}
\end{algorithm}
\null\vfill\eject

\subsection{Corrections to Section 4.1.3 of Wand (2017)}

The iterated Inverse G-Wishart fragment was introduced in Section 4.1.3 of Wand (2017)
and  it is one of the five fundamental fragments of semiparametric regression given
in Table 1. However, there are some errors due to the author of Wand (2017) failing
to recognise particular subtleties regarding the Inverse G-Wishart distribution,
as discussed in Section \ref{sec:IGWdistn}. We now point out misleading or
erroneous aspects in Section 4.1.3 of Wand (2017).

Firstly, in Wand (2017) $\boldsymbol{\Theta}_1$ plays the role of $\boldsymbol{\Sigma}$ and $\boldsymbol{\Theta}_2$ plays
the role of $\boldsymbol{A}$. The dimension of $\boldsymbol{\Theta}_1$ and $\boldsymbol{\Theta}_2$ is denoted by $d^{\Theta}$.
The first displayed equation of Section 4.1.3 is
\begin{equation}
\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2\sim\mbox{Inverse-G-Wishart}(G,\kappa,\boldsymbol{\Theta}_2^{-1})
\label{eq:XmasEve}
\end{equation}
for $\kappa>d^{\Theta}-1$ but it is only in the $G=G_{\mbox{\tiny{\rm full}}}$ case that such
a statement is reasonable for general $d^{\Theta}\in{\mathbb N}$. When $G=G_{\mbox{\tiny{\rm full}}}$
then $\kappa=\xi-d^{\Theta}+1$ according the notation used in the current article.
Therefore, (\ref{eq:XmasEve}) involves a different parameterisation to that
used throughout this article. Therefore, our first correction is to
replace the first displayed equation of Section 4.1.3 of Wand (2017) by:
$$\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2\sim\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{\Theta}_2^{-1})$$
where $\xi>0$ if $G=G_{\mbox{\tiny{\rm diag}}}$ and $\xi>2d^{\Theta}-2$ if $G=G_{\mbox{\tiny{\rm full}}}$.

The following sentence in Section 4.1.3 of Wand (2017):
``The fragment factor is of the form
$$p(\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2)\propto|\boldsymbol{\Theta}_2|^{-\kappa/2}
|\boldsymbol{\Theta}_1|^{-(\kappa+d^{\Theta}+1)/2}
\exp\left\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{\Theta}_1^{-1}\boldsymbol{\Theta}_2^{-1})\right\}\mbox{\null''}
$$
should instead be
``The fragment factor is of the form
$$p(\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2)\propto
\left\{
\begin{array}{l}
|\boldsymbol{\Theta}_2|^{-(\xi-d^{\Theta}+1)/2}
|\boldsymbol{\Theta}_1|^{-(\xi+2)/2}
\exp\left\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{\Theta}_1^{-1}\boldsymbol{\Theta}_2^{-1})\right\}
\quad\mbox{if $G=G_{\mbox{\tiny{\rm full}}}$,}\\[1ex]
|\boldsymbol{\Theta}_2|^{-\xi/2}
|\boldsymbol{\Theta}_1|^{-(\xi+2)/2}
\exp\left\{-{\textstyle{\frac{1}{2}}}\mbox{tr}(\boldsymbol{\Theta}_1^{-1}\boldsymbol{\Theta}_2^{-1})\right\}
\quad\mbox{if $G=G_{\mbox{\tiny{\rm diag}}}$.''}\\[1ex]
\end{array}
\right.
$$

In equation (31) of Wand (2017), the first entry
of the vector on the right-hand side of the $\longleftarrow$ should
be
$$-(\xi+2)/2\quad\mbox{rather than}\quad -(\kappa+d^{\Theta}+1)/2.$$
To match the correct parameterisation of the Inverse G-Wishart distribution,
as used in the current article, equation (32) of Wand (2017)
should be
$$\mbox{\null``}E(\boldsymbol{X}^{-1})\quad\mbox{where}\quad
\boldsymbol{X}\sim\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{\Lambda})\mbox{\null''}.$$

The equation in Section 4.1.3 of Wand (2017):
$$
\mbox{\null``}
\etaSUBpThetaOneThetaTwoTOThetaTwo\longleftarrow
\left[
\begin{array}{c}
-\kappa/2\\[1ex]
-{\textstyle{\frac{1}{2}}}\mbox{\rm vec}\Big(
E_{\mbox{\footnotesize$p(\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2)\to\boldsymbol{\Theta}_2$}}
(\boldsymbol{\Theta}_1^{-1})\Big)
\end{array}
\right]\mbox{\null''}
$$
should be replaced by
$$
\mbox{\null``}
\etaSUBpThetaOneThetaTwoTOThetaTwo\longleftarrow
\left[
\begin{array}{c}
-(\xi+2-2\omega_2)/2\\[1ex]
-{\textstyle{\frac{1}{2}}}\mbox{\rm vec}\Big(
E_{\mbox{\footnotesize$p(\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2)\to\boldsymbol{\Theta}_2$}}
(\boldsymbol{\Theta}_1^{-1})\Big)
\end{array}
\right]
$$
where $\omega_2$ depends on the graph of the Inverse G-Wishart distribution corresponding to
$E_{\mbox{\footnotesize$p(\boldsymbol{\Theta}_1|\boldsymbol{\Theta}_2)\to\boldsymbol{\Theta}_2$}}$.
If the graph is $G_{\mbox{\tiny{\rm full}}}$ then $\omega_2=(d^{\Theta}+1)/2$ and if the graph is $G_{\mbox{\tiny{\rm diag}}}$
then $\omega_2=1$.''

Lastly the iterated Inverse G-Wishart fragment natural parameter updates
given by equations (36) and (37) of Wand (2017) are affected by the oversights
described in the preceding paragraphs. They should be replaced by the updates
given in Algorithm \ref{alg:IterInvGWishFrag} with $\boldsymbol{\Theta}_1=\boldsymbol{\Sigma}$
and $\boldsymbol{\Theta}_2=\boldsymbol{A}$.

\section{Use of the Fragments for Covariance Matrix Prior Specification}\label{sec:priorSpec}

The underlying rationale for the Inverse G-Wishart prior and iterated Inverse G-Wishart fragments
is their ability to facilitate the specification of a wide range of covariance matrix priors
within the variational message passing framework. In the $d=1$ special case, covariance
matrix parameters reduce to variance parameters and their square roots are standard deviation
parameters. In this section we spell out how the fragments, and their natural parameter updates
in Algorithms \ref{alg:InvGWishPriorFrag} and \ref{alg:IterInvGWishFrag}, can be used
for prior specification in important special cases.

\subsection{Imposing an Inverse Chi-Squared Prior on a Variance Parameter}

Let $\sigma^2$ be a variance parameter and consider the prior imposition
$$\sigma^2\sim\mbox{Inverse-$\chi^2$}(\delta_{\sigma^2},\lambda_{\sigma^2})$$
for hyperparameters $\delta_{\sigma^2},\lambda_{\sigma^2}>0$,
within a variational message passing scheme. Then Algorithm
\ref{alg:InvGWishPriorFrag} should be called with inputs set to:
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm full}}},\quad \xi_{\scriptscriptstyle\boldsymbol{\Theta}}=\delta_{\sigma^2},\quad
\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=\lambda_{\sigma^2}.$$

\subsection{Imposing an Inverse Gamma Prior on a Variance Parameter}

Let $\sigma^2$ be a variance parameter and consider the prior imposition
\begin{equation}
\sigma^2\sim\mbox{Inverse-Gamma}(\alpha_{\sigma^2},\beta_{\sigma^2})
\label{eq:IGpriorSpec}
\end{equation}
for hyperparameters $\alpha_{\sigma^2},\beta_{\sigma^2}>0$.
The density function corresponding to (\ref{eq:IGpriorSpec}) is
$$p(\sigma^2;\alpha_{\sigma^2},\beta_{\sigma^2})\propto (\sigma^2)^{-\alpha_{\sigma^2}-1}
\exp\{-\beta_{\sigma^2}/(\sigma^2)\}I(\sigma^2>0).$$
Note that the Inverse Chi-Squared and Inverse Gamma distributions
are simple reparameterisations of each other since
$$x\sim\mbox{Inverse-$\chi^2$}(\delta,\lambda)
\quad\mbox{if and only if}\quad
x\sim\mbox{Inverse-Gamma}\big({\textstyle{\frac{1}{2}}}\delta,{\textstyle{\frac{1}{2}}}\lambda\big).
$$
To achieve (\ref{eq:IGpriorSpec}) Algorithm \ref{alg:InvGWishPriorFrag} should be called with inputs set to:
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm full}}},\quad \xi_{\scriptscriptstyle\boldsymbol{\Theta}}=2\alpha_{\sigma^2},\quad
\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=2\beta_{\sigma^2}.$$

\subsection{Imposing an Inverse Wishart Prior on a Covariance Matrix Parameter}

A random matrix $\boldsymbol{X}$ is defined to have an Inverse Wishart distribution
with shape parameter $\kappa$ and scale matrix $\boldsymbol{\Sigma}$,
written $\boldsymbol{X}\sim\mbox{Inverse-Wishart}(\kappa,\boldsymbol{\Lambda})$,
if and only if the density function of $\boldsymbol{X}$ is
\begin{equation}
\begin{array}{rcl}
	p(\boldsymbol{X})&=&\displaystyle{\frac{|\boldsymbol{\Lambda}|^{\kappa/2}}{2^{\kappa d/2}\pi^{d(d-1)/4}
		\prod_{j=1}^d\Gamma(\frac{\kappa+1-j}{2})}}\,
	|\boldsymbol{X}|^{-(\kappa+d+1)/2}\exp\{-{\textstyle{\frac{1}{2}}}\mbox{\rm tr}(\boldsymbol{\Lambda}\boldsymbol{X}^{-1})\}\\[2ex]
	&&\quad\times
	I(\boldsymbol{X}\ \mbox{a symmetric and positive definite $d\times d$ matrix}).
\end{array}
\label{eq:GelmanTable}
\end{equation}
Note that this is the common parameterisation of the Inverse Wishart distribution
(e.g. Table A.1 of Gelman \textit{et al.}, 2014).
Crucially, (\ref{eq:GelmanTable}) uses  a \emph{different} shape parametrization
from that used for the Inverse G-Wishart distribution in Definition \ref{def:IGWdefn}
when $G=G_{\mbox{\tiny{\rm full}}}$ with the relationship between the two shape
parameters given by $\kappa=\xi-d+1$. Even though the more general Inverse G-Wishart
family is important for the internal workings of variational message passing,
the ordinary Inverse Wishart distribution, with the parameterisation
as given in (\ref{eq:GelmanTable}), is more common when
imposing a prior on a covariance matrix.

Let $\boldsymbol{\Sigma}$ be a $d\times d$ matrix and consider the prior imposition
\begin{equation}
\boldsymbol{\Sigma}\sim\mbox{Inverse-Wishart}(\kappa_{\scriptscriptstyle\boldsymbol{\Sigma}},\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Sigma}})
\label{eq:classicalIW}
\end{equation}
for hyperparameters $\kappa_{\scriptscriptstyle\boldsymbol{\Sigma}},\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Sigma}}>0$,
within a variational message passing scheme.
Then Algorithm
\ref{alg:InvGWishPriorFrag} should be called with inputs set to:
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm full}}},\quad \xi_{\scriptscriptstyle\boldsymbol{\Theta}}=\kappa_{\scriptscriptstyle\boldsymbol{\Sigma}}+d-1,\quad
\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Sigma}}.$$

\subsection{Imposing a Half-$t$ Prior on a Standard Deviation Parameter}

Consider the prior imposition
\begin{equation}
\sigma\sim\mbox{Half-$t$}(s_{\sigma},\nu_{\sigma})
\label{eq:sigmaHt}
\end{equation}
for a scale parameter $s_{\sigma}>0$ and a degrees of freedom parameter
$\nu_{\sigma}>0$.
The density function corresponding to (\ref{eq:sigmaHt})
is such that $p(\sigma)\propto \{1+(\sigma/s_{\sigma})^2/\nu_{\sigma}\}^{-(\nu_{\sigma}+1)/2}I(\sigma>0)$.
This is equivalent to
\begin{equation}
\sigma^2|a\sim\mbox{Inverse-$\chi^2$}(\nu_{\sigma},1/a)\quad\mbox{and}\quad
a\sim\mbox{Inverse-$\chi^2$}(1,1/s_{\sigma}^2).
\label{eq:HtPriorAux}
\end{equation}
Since $d=1$, the graphs $G_{\mbox{\tiny{\rm full}}}$ and $G_{\mbox{\tiny{\rm diag}}}$ are the same --
a single node graph. Treating $\sigma^2$ and $a$ as $1\times1$ matrices
we can re-write (\ref{eq:HtPriorAux}) as
$$\sigma^2|a\sim\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm full}}},\nu_{\sigma},a^{-1})\quad\mbox{and}\quad
a\sim\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm diag}}},1,(\nu_{\sigma}\,s_{\sigma}^2)^{-1})
$$
(e.g. Armagan \textit{et al.}, 2011). The specification
$$a\sim\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm diag}}},1,(\nusigmas_{\sigma}^2)^{-1})$$
involves calling Algorithm \ref{alg:InvGWishPriorFrag}
with
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm diag}}},\quad\xi_{\scriptscriptstyle\boldsymbol{\Theta}}=1\quad\mbox{and}\quad
\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=(\nu_{\sigma}\,s_{\sigma}^2)^{-1}.$$
The output is the single node graph $G_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}$
and the $2\times1$ natural parameter vector
$$\biggerbdeta_{\mbox{\scriptsize$\pDens(\bTheta)\rightarrow\bTheta$}}=\mbox{\Large $\bdeta$}_{p(a)\rightarrow a}.$$

The specification
$$\sigma^2|a\sim\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm full}}},\nu_{\sigma},a^{-1})$$
implies that Algorithm \ref{alg:IterInvGWishFrag} is called
with graph input $G=G_{\mbox{\tiny{\rm full}}}$, shape parameter input
$\xi=\nu_{\sigma}$ and message parameter inputs
$$\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}=\mbox{\Large $\bdeta$}_{p(\sigma^2|a)\to\sigma^2},
\quad\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}=\mbox{\Large $\bdeta$}_{\sigma^2\to p(\sigma^2|a)},$$
and
$$G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm diag}}},\quad\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=\mbox{\Large $\bdeta$}_{p(\sigma^2|a)\to a}
\quad\mbox{and}\quad\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}=\mbox{\Large $\bdeta$}_{a\to p(\sigma^2|a)}.$$
Note that in this $d=1$ special case $G_{\mbox{\tiny{\rm full}}}$ and $G_{\mbox{\tiny{\rm diag}}}$ are both the single node graph.

\subsubsection{The Half-Cauchy Special Case}

The special case of
\begin{equation}
\sigma\sim\mbox{Half-Cauchy}(s_{\sigma}).
\label{eq:sigmaHC}
\end{equation}
corresponds to $\nu_{\sigma}=1$.
The density function corresponding to (\ref{eq:sigmaHC})
is such that $p(\sigma)\propto \{1+(\sigma/s_{\sigma})^2\}^{-1}I(\sigma>0)$.
Therefore, one should set $\xi=1$ in the call to Algorithm \ref{alg:IterInvGWishFrag}.

\subsection{Imposing a Huang-Wand Prior on a Covariance Matrix}

To impose the Huang-Wand prior
$$\boldsymbol{\Sigma}\sim\mbox{Huang-Wand}(s_{\mbox{\tiny{$\bSigma,1$}}},\ldots,s_{\mbox{\tiny{$\bSigma,d$}}})$$
in a variational message passing framework we should have the inputs to
Algorithm \ref{alg:InvGWishPriorFrag} being as follows:
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm diag}}},\quad\xi_{\scriptscriptstyle\boldsymbol{\Theta}}=1
\quad\mbox{and}\quad\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=\big\{2\,\mbox{diag}(s_{\mbox{\tiny{$\bSigma,1$}}}^2,\ldots,s_{\mbox{\tiny{$\bSigma,d$}}}^2)\big\}^{-1}.$$
The graph parameter input to Algorithm \ref{alg:IterInvGWishFrag} should be
$G=G_{\mbox{\tiny{\rm full}}}$ and the shape parameter input should be $\xi=2d$.

\subsection{Imposing a Matrix-$F$ Prior on a Covariance Matrix}

To impose the Matrix-$F$ prior
$$\boldsymbol{\Sigma}\sim F(\nu_{\mbox{\tiny MP}},\delta_{\mbox{\tiny MP}},\boldsymbol{B}_{\mbox{\tiny MP}})$$
in a variational message passing framework the inputs to
Algorithm \ref{alg:InvGWishPriorFrag} should be as follows:
$$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm full}}},\quad\xi_{\scriptscriptstyle\boldsymbol{\Theta}}=\nu_{\mbox{\tiny MP}}+d-1
\quad\mbox{and}\quad\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=\boldsymbol{B}_{\mbox{\tiny MP}}^{-1}.$$
The graph parameter input to Algorithm \ref{alg:IterInvGWishFrag} should be
$G=G_{\mbox{\tiny{\rm full}}}$ and the shape parameter input should be $\xi=\delta_{\mbox{\tiny MP}}+2d-2$.

\subsection{Tabular Summary of Fragment-Based Prior Specification}

Table \ref{tab:fragPrior} summarizes the results of this section and
is a crucial reference for placing priors of covariance matrix,
variance and standard deviation parameters in variational message passing
schemes that make use of Algorithms \ref{alg:InvGWishPriorFrag} and
\ref{alg:IterInvGWishFrag}.

\begin{table}[ht]
\begin{center}
{\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccccccc}
\hline
\multicolumn{1}{c}{\null}&
\multicolumn{3}{c}{Algorithm 1}&
\multicolumn{1}{c}{\null}&
\multicolumn{3}{c}{Algorithm 2}\\
\cmidrule(r){2-4}\cmidrule(l){6-8}
\multicolumn{1}{c}{prior specification}&
\multicolumn{1}{c}{$G_{\scriptscriptstyle\boldsymbol{\Theta}}$} &
\multicolumn{1}{c}{$\xi_{\scriptscriptstyle\boldsymbol{\Theta}}$}&
\multicolumn{1}{c}{$\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}$}&
\multicolumn{1}{c}{\null}&
\multicolumn{1}{c}{$\xi$}&
\multicolumn{1}{c}{$G$}&
\multicolumn{1}{c}{${\small G}_{\mbox{\tiny$\bA\rightarrow \pDens(\bSigma|\bA)$}}$ }
\\
\hline\\[-1.5ex]
\begin{small}$\sigma^2\sim\mbox{Inverse-$\chi^2$}(\delta_{\sigma^2},\lambda_{\sigma^2})$\end{small}
& $G_{\mbox{\tiny{\rm full}}}$ & $\delta_{\sigma^2}$   & $\lambda_{\sigma^2}$&\null&   N.A.& N.A. & N.A. \\[1ex]
\begin{small}$\sigma^2\sim\mbox{Inv.-Gamma}(\alpha_{\sigma^2},\beta_{\sigma^2})$\end{small}
& $G_{\mbox{\tiny{\rm full}}}$ & $2\alpha_{\sigma^2}$& $2\beta_{\sigma^2}$ &\null &  N.A.& N.A. & N.A. \\[1ex]
\begin{small}$\boldsymbol{\Sigma}\sim\mbox{Inv.-Wishart}(\kappa_{\scriptscriptstyle\boldsymbol{\Sigma}},\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Sigma}})$\end{small}
& $G_{\mbox{\tiny{\rm full}}}$ & $\kappa_{\scriptscriptstyle\boldsymbol{\Sigma}}+d-1$   & $\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Sigma}}$ &\null &   N.A.& N.A. & N.A. \\[1ex]
\begin{small}$\sigma\sim\mbox{Half-$t$}(s_{\sigma},\nu_{\sigma})$\end{small}
& $G_{\mbox{\tiny{\rm diag}}}$ & $1$   & $(\nusigmas_{\sigma}^2)^{-1}$ &\null & $\nu_{\sigma}$    & $G_{\mbox{\tiny{\rm full}}}$  &  $G_{\mbox{\tiny{\rm diag}}}$\\[1ex]
\begin{small}$\sigma\sim\mbox{Half-Cauchy}(s_{\sigma})$\end{small}
& $G_{\mbox{\tiny{\rm diag}}}$ & $1$   & $(s_{\sigma}^2)^{-1}$ &\null &  $1$    & $G_{\mbox{\tiny{\rm full}}}$  &  $G_{\mbox{\tiny{\rm diag}}}$\\[1ex]
\begin{small}$\boldsymbol{\Sigma}\sim\mbox{Huang-Wand}$ \end{small}
& $G_{\mbox{\tiny{\rm diag}}}$ & $1$   & $\big\{2\,\mbox{diag}(s_{\mbox{\tiny{$\bSigma,1$}}}^2,$ &\null &  $2d$    & $G_{\mbox{\tiny{\rm full}}}$  &  $G_{\mbox{\tiny{\rm diag}}}$\\[0ex]
$\qquad\ (s_{\mbox{\tiny{$\bSigma,1$}}},\ldots,s_{\mbox{\tiny{$\bSigma,d$}}})$               &   &       &      $\null\ \ \ldots,s_{\mbox{\tiny{$\bSigma,d$}}}^2)\big\}^{-1}$
&\null &       &       &     \\[1ex]
\begin{small}$\boldsymbol{\Sigma}\sim\mbox{Matrix-$F$}$ \end{small}
&$G_{\mbox{\tiny{\rm full}}}$ & $\nu_{\mbox{\tiny MP}}+$  & $\boldsymbol{B}_{\mbox{\tiny MP}}^{-1}$ &\null &  $\delta_{\mbox{\tiny MP}}+$ & $G_{\mbox{\tiny{\rm full}}}$  &  $G_{\mbox{\tiny{\rm full}}}$\\[0ex]
$\qquad\ (\nu_{\mbox{\tiny MP}},\delta_{\mbox{\tiny MP}},\boldsymbol{B}_{\mbox{\tiny MP}})$               &   & $\ d-1$      &
&\null & $\ 2d-2$       &       &     \\[1ex]
\hline
\end{tabular}
}
\end{center}
\caption{\it Specifications of inputs of Algorithms \ref{alg:InvGWishPriorFrag}
and \ref{alg:IterInvGWishFrag} for several variance, standard deviation and
covariance matrix prior impositions. The abbreviation N.A. stands for not
applicable since Algorithm \ref{alg:IterInvGWishFrag} is not needed for the
first three prior impositions.}
\label{tab:fragPrior}
\end{table}

\section{Illustrative Example}\label{sec:illustrative}

We illustrate the use of Algorithms \ref{alg:InvGWishPriorFrag}
and \ref{alg:IterInvGWishFrag} for the case of Bayesian linear
mixed models with $t$ distribution responses. Such $t$-based models
impose a form of robustness in situations where the responses are susceptible
to having outlying values (e.g. Lange \textit{et al.}, 1989).
The notation $y\sim t(\mu,\sigma,\nu)$ indicates that the random
variable $y$ has a $t$ distribution with location parameter $\mu$,
scale parameter $\sigma>0$ and degrees of freedom parameter $\nu>0$.
The corresponding density function of $y$ is
$$p(y)=\displaystyle{\frac{\Gamma\left(\frac{\nu+1}{2}\right)}
{\sigma\sqrt{\pi\nu}\Gamma(\nu/2)[1+\{(y-\mu)/\sigma\}^2/\nu]^{\frac{\nu+1}{2}}}}.
$$
Now suppose that the response data consists of repeated measures
within each of $m$ groups. Let
$$y_{ij}\equiv\mbox{the $j$th response for the $i$th group},\quad 1\le j\le n_i,\ 1\le i\le m,$$
and then let $\boldsymbol{y}_i$, $1\le i\le m$, be the $n_i\times 1$ vectors containing $y_{ij}$ data
for the $i$th group. For each $1\le i\le m$, let $\boldsymbol{X}_i$ be $n_i\times p$ design matrices
corresponding to the fixed effects and $\boldsymbol{Z}_i$ be $n_i\times q$ design matrices corresponding
to the random effects. Next put
\begin{equation}
\boldsymbol{y}\equiv\left[
\begin{array}{c}
\boldsymbol{y}_1\\
\vdots\\
\boldsymbol{y}_m
\end{array}
\right],
\quad
\boldsymbol{X}\equiv\left[
\begin{array}{c}
\boldsymbol{X}_1\\
\vdots\\
\boldsymbol{X}_m
\end{array}
\right]
\quad\mbox{and}\quad
\boldsymbol{Z}\equiv\blockdiag{1\le i\le m}(\boldsymbol{Z}_i)
\label{eq:yXZdefn}
\end{equation}
and define $N=n_1+\ldots+n_m$ to be the number of rows in each of $\boldsymbol{y}$, $\boldsymbol{X}$ and $\boldsymbol{Z}$.
Let $y_{\ell}$ be the $\ell$th entry of $\boldsymbol{y}$, $1\le\ell\le N$.
The family of Bayesian $t$ response linear mixed models that we consider is
\begin{equation}
\begin{array}{c}
y_{\ell}|\boldsymbol{\beta},\boldsymbol{u},\sigma\stackrel{{\tiny \mbox{ind.}}}{\sim} t\big((\boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{Z}\boldsymbol{u})_{\ell},\sigma,\nu\big),
\quad 1\le \ell\le N,\quad \boldsymbol{u}|\boldsymbol{\Sigma}\sim N(\boldsymbol{0},\boldsymbol{I}_m\otimes\boldsymbol{\Sigma}),\\[2ex]
\boldsymbol{\beta}\sim N(0,\sigma_{\boldsymbol{\beta}}^2\boldsymbol{I}),\quad
\sigma\sim\mbox{Half-Cauchy}(s_{\sigma}),\quad {\textstyle{\frac{1}{2}}}\nu\sim\mbox{Moon-Rock}(0,\lambda_{\nu}),\\[2ex]
\boldsymbol{\Sigma}\sim\mbox{Huang-Wand}\big(s_{\mbox{\tiny{$\bSigma,1$}}},\ldots,s_{\mbox{\tiny{$\bSigma,q$}}}\big)
\end{array}
\label{eq:tRespModel}
\end{equation}
for hyperparameters
$\sigma_{\boldsymbol{\beta}},s_{\sigma},\lambda_{\nu},s_{\mbox{\tiny{$\bSigma,1$}}},\ldots,s_{\mbox{\tiny{$\bSigma,q$}}}>0$.

As explained in McLean \&\  Wand (2019), the Moon Rock family of distributions is
conjugate for the parameter ${\textstyle{\frac{1}{2}}}\nu$, with the notation $x\sim\mbox{Moon-Rock}(\alpha,\beta)$
indicating that the corresponding density function satisfies
$p(x)\propto\{x^x/\Gamma(x)\}^{\alpha}\exp(-\beta x)I(x>0)$.
In the variational message passing treatment of the degrees of freedom parameter
it is simpler to work with
$$\upsilon\equiv {\textstyle{\frac{1}{2}}}\nu\quad\mbox{so that}\quad\upsilon\sim\mbox{Moon-Rock}(0,\lambda_{\nu}).$$
After the approximate posterior density function of $\upsilon$ is
obtained via variational message passing, it is trivial to then
obtain the same for $\nu$. Hence, we work with $\upsilon$,
rather than $\nu$, in the upcoming description of variational
message passing-based fitting and inference for (\ref{eq:tRespModel}).

Next note that
$$\boldsymbol{y}_{\ell}|\boldsymbol{\beta},\boldsymbol{u},\sigma\stackrel{{\tiny \mbox{ind.}}}{\sim} t\Big((\boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{Z}\boldsymbol{u})_{\ell},\sigma,2\upsilon\Big),
\quad 1\le \ell\le N$$
is equivalent to
\begin{equation}
\boldsymbol{y}_{\ell}|\boldsymbol{\beta},\boldsymbol{u},\sigma^2,b_{\ell}\sim
N\big((\boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{Z}\boldsymbol{u})_{\ell},b_{\ell}\sigma^2\big),
\quad b_{\ell}|\upsilon\stackrel{{\tiny \mbox{ind.}}}{\sim}\mbox{Inverse-$\chi^2$}\left(2\upsilon,2\upsilon\right),
\label{eq:auxVarOne}
\end{equation}
$\sigma\sim\mbox{Half-Cauchy}(s_{\sigma})$ is equivalent to
\begin{equation}
\sigma^2|a\sim\mbox{Inverse-$\chi^2$}(1,1/a)\quad\mbox{and}\quad
a\sim\mbox{Inverse-$\chi^2$}(1,1/s_{\sigma}^2)
\label{eq:auxVarTwo}
\end{equation}
and $\boldsymbol{\Sigma}\sim\mbox{Huang-Wand}\big(s_{\mbox{\tiny{$\bSigma,1$}}},\ldots,s_{\mbox{\tiny{$\bSigma,q$}}}\big)$
is equivalent to
\begin{equation}
\begin{array}{c}
\boldsymbol{\Sigma}|\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm full}}},2q,\boldsymbol{A}^{-1}),\\[1ex]
\boldsymbol{A}\sim\mbox{Inverse-G-Wishart}
\Big(G_{\mbox{\tiny{\rm diag}}},1,\big\{2\,\mbox{diag}(s_{\mbox{\tiny{$\bSigma,1$}}}^2,\ldots,s_{\mbox{\tiny{$\bSigma,q$}}}^2)\big\}^{-1}\Big).
\end{array}
\label{eq:auxVarThree}
\end{equation}
Substitution of (\ref{eq:auxVarOne}), (\ref{eq:auxVarTwo}) and (\ref{eq:auxVarThree})
into (\ref{eq:tRespModel}) leads to the hierarchical Bayesian model depicted
as a directed acyclic graph in Figure \ref{fig:tMixModDAG} with $\boldsymbol{b}\equiv(b_1,\ldots,b_N)$.
The unshaded circles in Figure \ref{fig:tMixModDAG} correspond to model parameters
and auxiliary variables and will be referred to as \emph{hidden nodes}.

\begin{figure}[h]
\centering
{\includegraphics[width=0.8\textwidth]{tMixModDAG.pdf}}
\caption{\it Directed acyclic graph corresponding to the
$t$ response linear mixed model (\ref{eq:tRespModel}) with auxiliary
variable representations (\ref{eq:auxVarOne})--(\ref{eq:auxVarThree}).
The shaded circle corresponds to the observed data. The unshaded circles
correspond to model parameters and auxiliary variables. The small
solid circles correspond to hyperparameters.
}
\label{fig:tMixModDAG}
\end{figure}

Consider the following mean field approximation of the joint
posterior of the hidden nodes in Figure \ref{fig:tMixModDAG}
\begin{equation}
p(\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\upsilon,\boldsymbol{\Sigma},a,\boldsymbol{A},\boldsymbol{b}|\boldsymbol{y})
\approx q(\boldsymbol{\beta},\boldsymbol{u},a,\boldsymbol{A},\boldsymbol{b})q(\sigma^2,\boldsymbol{\Sigma},\upsilon)
\label{eq:lessStringent}
\end{equation}
where $q$ denotes the approximate posterior density functions
of the relevant parameters. Application of induced factor results
(e.g. Bishop, 2006; Section 10.2.5) leads to the additional factorizations
$$q(\boldsymbol{\beta},\boldsymbol{u},a,\boldsymbol{A},\boldsymbol{b})=q(\boldsymbol{\beta},\boldsymbol{u})q(a)q(\boldsymbol{A})\prod_{\ell=1}^Nq(b_{\ell})
\quad\mbox{and}\quad
q(\sigma^2,\boldsymbol{\Sigma},\upsilon)=q(\sigma^2)q(\boldsymbol{\Sigma})q(\upsilon).
$$
and so the restriction given in (\ref{eq:lessStringent}) is equivalent to
\begin{equation}
p(\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\upsilon,\boldsymbol{\Sigma},a,\boldsymbol{A},\boldsymbol{b}|\boldsymbol{y})
\approx q(\boldsymbol{\beta},\boldsymbol{u})q(\sigma^2)q(\upsilon)q(\boldsymbol{\Sigma})q(a)q(\boldsymbol{A})
\prod_{\ell=1}^Nq(b_{\ell}).
\label{eq:finalMF}
\end{equation}
Figure \ref{fig:tMixModFacGraph} is a factor graph representation of
the joint density function of all random variables and vectors,
or stochastic nodes, in
Figure \ref{fig:tMixModDAG} hierarchical model, with unshaded circles
for each stochastic node according to the $q$-density factorization
given in (\ref{eq:finalMF}) and filled-in rectangles corresponding
to factors on the right-hand side of
\begin{equation}
\begin{array}{l}
p(\boldsymbol{y},\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\upsilon,\boldsymbol{\Sigma},a,\boldsymbol{A},\boldsymbol{b})\\[1ex]
\qquad\qquad
=p(\boldsymbol{A})p(a)p(\boldsymbol{\Sigma}|\boldsymbol{A})p(\sigma^2|a)
p(\boldsymbol{\beta},\boldsymbol{u}|\boldsymbol{\Sigma})p(\boldsymbol{y}|\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})
p(\boldsymbol{b}|\upsilon)p(\upsilon).
\end{array}
\label{eq:fullFactor}
\end{equation}
Edges join each factor to a stochastic node that appears in the factor.
To aid upcoming discussion, the fragments are numbered $1$ to $8$
according to appearance from left to right. Recall that a fragment
is a sub-graph consisting of a factor and all of its neighboring nodes.
Figure \ref{fig:tMixModFacGraph} uses shading to show the distinction
between adjacent fragments.

\begin{figure}[h]
\centering
{\includegraphics[width=1\textwidth]{tMixModFacGraph.pdf}}
\caption{\it Factor graph corresponding to the
$t$ response linear mixed model (\ref{eq:tRespModel}) with auxiliary
variable representations (\ref{eq:auxVarOne})--(\ref{eq:auxVarThree}).
The circular nodes correspond to stochastic nodes in the
$q$-density factorization in (\ref{eq:finalMF}). The rectangular
nodes correspond to the factors on the right-hand side of (\ref{eq:fullFactor}).
The fragments are numbered  $1$ to $8$ according to appearance from left to right.
Shading is used to show the distinction between adjacent fragments.
}
\label{fig:tMixModFacGraph}
\end{figure}

Note that (e.g. Minka, 2005; Wand, 2017) the variational message passing iteration
loop has the following generic steps:
$$
\begin{array}{lll}
&\mbox{1. Choose a factor.}\\
&\mbox{2. Update the parameter vectors of the messages passed from}\\
&\mbox{\ \ \ \ the factor's neighboring stochastic nodes to the factor.}\\
&\mbox{3. Update the parameter vectors of the messages passed}\\
&\mbox{\ \ \ \ from the factor to its neighboring stochastic nodes.}
\end{array}
$$
Step 2. is very simple and has generic form given by, for example, (7) of Wand (2017).
In the Figure \ref{fig:tMixModFacGraph} factor graph an example of Step 2. is:
\begin{equation}
{\setlength\arraycolsep{3pt}
\begin{array}{rcl}
&&\mbox{the message passed from $\boldsymbol{\Sigma}$ to $p(\boldsymbol{\beta},\boldsymbol{u}|\boldsymbol{\Sigma})$}\\[1ex]
&&\qquad\qquad=\mbox{the message passed from $p(\boldsymbol{\Sigma}|\boldsymbol{A})$ to $\boldsymbol{\Sigma}$
in the previous iteration.}
\end{array}
}
\label{eq:StoFexamp}
\end{equation}
In terms of natural parameter vector updates, (\ref{eq:StoFexamp}) corresponds to:
$$\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}.$$
Most of the other stochastic node to factor updates
in Figure \ref{fig:tMixModFacGraph} have an analogous form.
The exception are the messages passed within fragments 6 and 7,
which require use of the slightly more complicated form as
given by, for example, equation (7) of Wand (2017).

It remains to discuss Step 3., corresponding to the factor to stochastic
node updates:
\begin{itemize}
\item Fragments 1 and 2 are Inverse G-Wishart prior fragments and the
factor to stochastic node parameter vector updates are performed according
to Algorithm \ref{alg:InvGWishPriorFrag}. In view of Table \ref{tab:fragPrior},
the graph and shape hyperparameter inputs are
$G_{\scriptscriptstyle\boldsymbol{\Theta}}=G_{\mbox{\tiny{\rm diag}}}$ and $\xi_{\scriptscriptstyle\boldsymbol{\Theta}}=1$. For fragment 1 the
rate hyperparameter is $\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}
=\{2\mbox{diag}\big(s_{\mbox{\tiny{$\bSigma,1$}}}^2,\ldots,s_{\mbox{\tiny{$\bSigma,q$}}}^2\big)\}^{-1}$.
For fragment 2 the rate hyperparameter
is $\boldsymbol{\Lambda}_{\scriptscriptstyle\boldsymbol{\Theta}}=(s^2_{\sigma})^{-1}$.
\item Fragments 3 and 4 are iterated Inverse G-Wishart prior fragments and the
factor to stochastic node parameter vector updates are performed according
to Algorithm \ref{alg:IterInvGWishFrag}. As shown in Table \ref{tab:fragPrior},
the graph inputs should be
$$G_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}=G_{\mbox{\tiny{\rm diag}}},\ G_{\mbox{\scriptsize$a\rightarrow p(\sigma^2|a)$}}=G_{\mbox{\tiny{\rm diag}}},\ G_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}=G_{\mbox{\tiny{\rm full}}},
\ \mbox{and}\ G_{\mbox{\scriptsize$\sigma^2\rightarrow \pDens(\sigma^2|a)$}}=G_{\mbox{\tiny{\rm full}}}.
$$
The first two of these are imposed by the messages passed from fragments 1 and 2.
For fragment 3, the shape parameter input is $\xi=2q$.
For fragment 4, the shape parameter input is $\xi=1$.
\item Fragment 5 is the Gaussian penalization fragment described
in Section 4.1.4 of Wand (2017) with, in the notation given there, $L=1$,
$\boldsymbol{\mu}_{\boldsymbol{\theta}_0}=\boldsymbol{0}$ and $\boldsymbol{\Sigma}_{\boldsymbol{\theta}_0}=\sigma_{\boldsymbol{\beta}}^2\boldsymbol{I}$.
\item Fragments 6 and 7 correspond to the $t$ likelihood fragment.
Its natural parameter updates are provided by Algorithm 2 of
McLean \&\  Wand (2019).
\item Fragment 8 corresponds to the imposition of a Moon Rock
prior distribution on a shape parameter. This is a very simple
fragment for which the only inputs are the Moon Rock prior
specification hyperparameters and the output is the
natural parameter vector of the Moon Rock prior density function.
Since this fragment is not listed as an algorithm in this article
or elsewhere, we provide further details in the paragraph
after the next one.
\end{itemize}

For Fragments 5, 6 and 7 simple conversions between two different
versions of natural parameter vectors need to be made.
Section \ref{sec:vecANDvech} of the web-supplement
explains these conversions.

The most general Moon Rock prior specification for a generic parameter $\theta$ is
$$\theta\sim\mbox{Moon-Rock}(\alpha_{\theta},\beta_{\theta}).$$
This corresponds to the prior density function having exponential family form
$$p(\theta)\propto\exp\left\{
\left[
\begin{array}{c}
\theta\log(\theta)-\log\Gamma(\theta)\\[1ex]
\theta
\end{array}
\right]^T
\left[
\begin{array}{c}
\alpha_{\theta}\\[1ex]
-\beta_{\theta}
\end{array}
\right]
\right\}
$$
The inputs of the Moon Rock prior fragment are $\alpha_{\theta}\ge 0$ and $\beta_{\theta}>0$
and the output is the natural parameter vector
$$\biggerbdeta_{\mbox{\scriptsize$\pDens(\theta)\rightarrow\theta$}}\longleftarrow\left[
\begin{array}{c}
\alpha_{\theta}\\[1ex]
-\beta_{\theta}
\end{array}
\right].
$$
Since, for the $t$ response mixed model illustrative example,
we have the prior imposition
$\upsilon\sim\mbox{Moon-Rock}(0,\lambda_{\nu})$
we simply call the Moon Rock prior fragment with
$(\alpha_{\theta},\beta_{\theta})$ set to $(0,\lambda_{\nu})$.

To demonstrate variational message passing for fitting
and inference for model (\ref{eq:tRespModel}), we simulated
data according to the dimension values $p=q=2$ and the
true parameter values
\begin{equation}
\bbeta_{\mbox{\tiny true}}=
\left[
\begin{array}{c}
-0.58 \\[1ex]
1.89
\end{array}
\right],
\quad
\sigma^2_{\mbox{\tiny true}}=0.2,
\quad
\bSigma_{\mbox{\tiny true}}
=
\left[
\begin{array}{cc}
2.58 & 0.22 \\[1ex]
0.22 & 1.73
\end{array}
\right]
\quad\mbox{and}\quad
\nu_{\mbox{\tiny true}}=1.5.
\label{eq:trueValues}
\end{equation}
The sample sizes were $m=20$, with $n_i=15$ observations per group,
and the predictor data were generated from the Uniform distribution
on the unit interval. The hyperparameter values were set at
$$\sigma_{\boldsymbol{\beta}}=s_{\sigma}=s_{\mbox{\tiny{$\bSigma,1$}}}=s_{\mbox{\tiny{$\bSigma,2$}}}=10^5\quad\mbox{and}\quad
\lambda_{\nu}=0.01.$$
We ran the variational message passing algorithm as described above
until the relative change the variational parameters was below $10^{-10}$.
as well as Markov chain Monte Carlo via
the \textsf{R} language (\textsf{R} Core Team, 2020) package
\texttt{rstan} (Stan Development Team, 2019).
For Markov chain Monte Carlo fitting, a warmup of size 1000
was used, followed by chains of size 5000
retained for inference.

\begin{figure}[h]
\centering
{\includegraphics[width=0.95\textwidth]{tMixModPosterDens.pdf}}
\caption{\it Approximate posterior density functions for the parameters
in model (\ref{eq:tRespModel}) based on both variational message
passing (VMP) and Markov chain Monte Carlo (MCMC) algorithms applied to
data simulated according to the true values (\ref{eq:trueValues})
and sample sizes, predictor values and hyperparameter values
as described in the text. The vertical lines indicate true parameter values.
}
\label{fig:tMixModPosterDens}
\end{figure}

Figure \ref{fig:tMixModPosterDens} compares the approximate
posterior density functions based on both variational
message passing (VMP) and Markov chain Monte Carlo (MCMC).
The middle row performs the comparison for the
random intercept and slope parameters, $u_{i0}$ and $u_{i1}$,
for $i=1,2$. The parameters in the third row of Figure \ref{fig:tMixModPosterDens}
are for the standard deviation and correlation parameters in the $\boldsymbol{\Sigma}$
matrix, according to the notation $(\boldsymbol{\Sigma})_{11}=\sigma_1$, $(\boldsymbol{\Sigma})_{22}=\sigma_2$
and $(\boldsymbol{\Sigma})_{12}=\sigma_1\sigma_2\rho$. For most of the stochastic nodes,
the accuracy of variational message passing is seen to be very good.
For $\sigma$ and $\nu$, some under-approximation of the spread and locational shift
is apparent. A likely root cause is the imposition of the product restriction
$q(\sigma,\nu)=q(\sigma)q(\nu)$ even though these two parameters
have a significant amount of posterior dependence.

We have prepared a bundle of \textsf{R} language
code that carries out variational message passing for this illustrative example,
including use of Algorithms \ref{alg:InvGWishPriorFrag}
and \ref{alg:IterInvGWishFrag} for the imposition
of Half Cauchy and Huang-Wand priors. This code is part of
the web-supplement for this article.

Lastly, we point out that this illustrative example does not involve
matrix algebraic streamlining for random effects models. This relatively
new area for variational message passing research, which streamlines
calculations involving sparse matrix forms that arise in linear mixed models,
is described in Nolan, Menictas \&\  Wand (2020).

\section{Closing Remarks}\label{sec:closing}

Algorithms \ref{alg:InvGWishPriorFrag} and, especially, Algorithm \ref{alg:IterInvGWishFrag}
and their underpinnings are quite involved and dependent upon a careful study of particular
special cases of the inverses of G-Wishart random matrices. The amount of detail
provided by this article is tedious, but necessary, to ensure that the
fragment updates based on a single distributional
structure, the Inverse G-Wishart distribution with $G\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$,
are correct. The good news is that these algorithms only need to be derived once.
Their implementations, within a suite of computer
programmes for carrying out variational message passing for models containing
variance and covariance matrix parameters, can be isolated into subroutines
which, once working as intended, do not have
to be revisited ever again. Given the quintessence of variance and covariance
parameters in throughout statistics and machine learning,
Algorithms \ref{alg:InvGWishPriorFrag} and Algorithm \ref{alg:IterInvGWishFrag}
are important and fundamental contributions to variational message passing.

\section*{Acknowledgements}

We are grateful to two referees for their comments and suggestions.
This research was supported by Australian Research Council Discovery
Project DP140100441.

\section*{References}

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Armagan, A., Dunson, D.B. and Clyde, M. (2011). Generalized beta mixtures of Gaussians.
In \textit{Advances in Neural Information Processing Systems 24},
J.Shawe-Taylor, R.S. Zamel, P. Bartlett, F. Pereira and K.Q. Weinberger (eds.),
pp. 523--531.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Assaf, A.G., Li, G., Song, H. \&\  Tsionas, M.G. (2019).
Modeling and forecasting regional tourism demand using the Bayesian global vector
autoregressive (BGVAR) model.
\textit{Journal of Travel Research}, {\bf 58}, 383--397.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Attias, H. (1999). Inferring parameters and structure of latent variable
models by variational Bayes. In Laskey, K.B. and Prade, H. (editors)
\textit{Proceedings of the Fifteenth Conference
on Uncertainty in Artificial Intelligence}, pp. 21--30.
San Francisco: Morgan Kauffmann.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Atay-Kayis, A. \&\  Massam, H. (2005).
A Monte Carlo method for computing marginal likelihood
in nondecomposable Gaussian graphical models.
\textit{Biometrika}, {\bf 92}, 317--335.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Bishop, C.M. (2006). {\it Pattern Recognition and Machine Learning.}
New York: Springer.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Chen, W.Y and Wand, M.P. (2020).
Factor graph fragmentization of expectation propagation.
\textit{Journal of the Korean Statistical Society}, in press.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Conti, G., Fr\"{u}hwirth-Schnatter, S., Heckman, J.J. \&\  Piatek, R. (2014).
Bayesian exploratory factor analysis.
\textit{Journal of Econometrics}, {\bf 183}, 31--57.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Dawid, A.P. \&\  Lauritzen, S.L. (1993). Hyper Markov laws in the statistical analysis of
decomposable graphical models. \textit{The Annals of Statistics}, \textbf{21}, 1272--1317.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models.
\textit{Bayesian Analysis}, {\bf 1}, 515--533.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. \&\  Rubin, D.B. (2014).
\textit{Bayesian Data Analysis, Third Edition}, Boca Raton, Florida: CRC Press.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Gentle, J.E. (2007). \textit{Matrix Algebra}.
New York: Springer.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Harezlak, J., Ruppert, D. \&\  Wand, M.P. (2018).
{\it Semiparametric Regression with R}.
New York: Springer.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Huang, A. \&\  Wand, M.P. (2013).
Simple marginally noninformative prior distributions
for covariance matrices. \textit{Bayesian Analysis},
{\bf 8}, 439--452.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Lange, K.L., Little, R.J.A. and Taylor, J.M.G. (1989).
Robust statistical modeling using the $t$-distribution.
{\it Journal of the American Statistical Association},
{\bf 84}, 881-896.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Letac, G. \&\  Massam, H. (2007).
Wishart distributions for decomposable graphs.
\textit{The Annals of Statistics}, {\bf 35}, 1278--1323.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Maestrini, L., \&\  Wand, M.P. (2018). Variational message passing for skew t regression.
\emph{Stat}, {\bf7}, e196.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Magnus, J.R. \&\  Neudecker, H. (1999). \emph{Matrix Differential Calculus with Applications in
Statistics and Econometrics, Revised Edition}. Chichester U.K.: Wiley

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
McCulloch, C.E., Searle, S.R. \&\  Neuhaus, J.M. (2008). \textit{Generalized, Linear,
and Mixed Models, Second Edition}. New York: John Wiley \& Sons.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
McLean, M.W. \&\  Wand, M.P. (2019).
Variational message passing for elaborate response regression models.
\textit{Bayesian Analysis}, {\bf 14}, 371--398.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Minka, T.P. (2001).
Expectation propagation for approximate Bayesian inference.
In J.S. Breese \&\  D. Koller (eds),
\textit{Proceedings of the Seventeenth Conference on Uncertainty in
Artificial Intelligence}, pp. 362--369.
Burlington, Massachusetts: Morgan Kaufmann.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Minka, T. (2005).
Divergence measures and message passing.
\textit{Microsoft Research Technical Report Series},
{\bf MSR-TR-2005-173}, 1--17.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Muirhead, R.J. (1982). \textit{Aspects of Multivariate Statistical Theory.}
New York: John Wiley \& Sons.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Mulder, J. \&\  Pericchi, L.R. (2018). The Matrix-$F$ prior for estimating and
testing covariance matrices. \textit{Bayesian Analysis}, {\bf 13}, 1193--1214.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Nolan, T.H., Menictas, M. and Wand, M.P. (2020).
Streamlined computing for variational inference with higher level
random effects. Unpublished manuscript
available at \textit{https://arxiv.org/abs/1903.06616}.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Nolan, T.H. and Wand, M.P. (2017).
Accurate logistic variational message passing:
algebraic and numerical details. \textit{Stat}, {\bf 6},
102--112.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Polson, N. G. \&\  Scott, J. G. (2012). On the half-Cauchy prior for a global scale parameter.
\textit{Bayesian Analysis}, {\bf 7}, 887--902.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
\textsf{R} Core Team (2020). \textsf{R}: A language and environment
for statistical computing. \textsf{R} Foundation for Statistical Computing.
Vienna, Austria. \texttt{https://www.R-project.org/}

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Roverato, A. (2000). Cholesky decomposition of a hyper inverse Wishart matrix.
\textit{Biometrika}, {\bf 87}, 99-112.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Stan Development Team (2019). \textsf{RStan}: the
\textsf{R} interface to \textsf{Stan}.
\textsf{R} package version 2.19.2.
\texttt{http://mc-stan.org/.}

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Uhler, C., Lenkoski, A. and Richards, D. (2018).
Exact formulas for the normalizing constants of Wishart
distributions for graphical models. \textit{The Annals of Statistics}, {\bf 46}, 90--118.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Wand, M.P. (2017).
Fast approximate inference for arbitrarily large semiparametric
regression models via message passing (with discussion).
\textit{Journal of the American Statistical Association},
{\bf 112}, 137--168.

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Winn, J. \&\  Bishop, C.M. (2005).
Variational message passing. \textit{Journal of Machine Learning Research},
{\bf 6}, 661--694.

\vfill\eject
\renewcommand{\theequation}{S.\arabic{equation}}
\renewcommand{\thesection}{S.\arabic{section}}
\renewcommand{\thetable}{S.\arabic{table}}
\setcounter{equation}{0}
\setcounter{table}{0}
\setcounter{section}{0}
\setcounter{page}{1}
\setcounter{footnote}{0}

\begin{center}

{\Large Web-supplement for:}
\vskip3mm

\centerline{\Large\bf The Inverse G-Wishart Distribution and Variational Message Passing}
\vskip7mm
\ifthenelse{\boolean{UnBlinded}}{\centerline{\normalsize\sc By L. Maestrini and M.P. Wand}
\vskip5mm
\centerline{\textit{University of Technology Sydney}}
\vskip6mm}{\null}
\end{center}

\section{Natural Parameter Versions and Mappings}\label{sec:vecANDvech}

Throughout this article we use the ``$\mbox{\rm vech}$'' versions of the
natural parameter forms of the Multivariate Normal
and Inverse G-Wishart distributions. However, Wand (2017) and
McLean \&\  Wand (2019) used ``$\mbox{\rm vec}$'' versions of these
distributions. The ``$\mbox{\rm vech}$'' version has the attraction
of being more compact since entries of symmetric matrices
are not duplicated. However, adoption of the ``$\mbox{\rm vech}$'' version
entails use of duplication matrices. For implementation in the \textsf{R}
language (\textsf{R} Core Team, 2020) we note that the
function \texttt{duplication.matrix()} in the package
\textsf{matrixcalc} (Novomestky, 2012) returns the duplication matrix
of a given order.

First we explain the two versions for the Multivariate Normal distribution.
Suppose that the $d\times1$ random vector $\boldsymbol{v}$ has a $N(\boldsymbol{u},\boldsymbol{\Sigma})$
distribution. Then the density function of $\boldsymbol{v}$ is
$$p(\boldsymbol{v})\propto
\exp\left\{\left[
\begin{array}{c}
\boldsymbol{v}\\
\mbox{\rm vec}(\boldsymbol{v}\bv^T)
\end{array}
\right]^T\bdeta_v^{\mbox{\tiny vec}}
\right\}
=\exp\left\{\left[
\begin{array}{c}
\boldsymbol{v}\\
\mbox{\rm vech}(\boldsymbol{v}\bv^T)
\end{array}
\right]^T\bdeta_v^{\mbox{\tiny vech}}
\right\}
$$
where
$$\bdeta_v^{\mbox{\tiny vec}}\equiv
\left[
\begin{array}{c}
\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}\\[1ex]
-{\textstyle{\frac{1}{2}}}\mbox{\rm vec}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]
\quad
\mbox{and}
\quad
\bdeta_v^{\mbox{\tiny vech}}\equiv
\left[
\begin{array}{c}
\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right].
$$
The two natural parameter vectors can be mapped between each other using
\begin{equation}
\bdeta_v^{\mbox{\tiny vech}}=\mbox{blockdiag}(\boldsymbol{I}_d,\boldsymbol{D}_d^T)\bdeta_v^{\mbox{\tiny vec}}
\quad\mbox{and}\quad
\bdeta_v^{\mbox{\tiny vec}}=\mbox{blockdiag}(\boldsymbol{I}_d,\boldsymbol{D}_d^{+T})\bdeta_v^{\mbox{\tiny vech}}.
\label{eq:vecvechmapsMVN}
\end{equation}

Now we explain the interplay between the ``vec'' and ``vech'' forms of the
Inverse G-Wishart distribution. Let the $d\times d$ matrix $\boldsymbol{V}$
have an $\mbox{Inverse-G-Wishart}(G,\xi,\boldsymbol{\Lambda})$ distribution.
Then the density function of $\boldsymbol{V}$ is
$$p(\boldsymbol{V})\propto
\exp\left\{\left[
\begin{array}{c}
\log|\boldsymbol{V}|\\[1ex]
\mbox{\rm vec}(\boldsymbol{V}^{-1})
\end{array}
\right]^T\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vec}}
\right\}
=\exp\left\{\left[
\begin{array}{c}
\log|\boldsymbol{V}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{V}^{-1})
\end{array}
\right]^T\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vech}}
\right\}
$$
where
$$\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vec}}\equiv
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}(\xi+1)\\[1ex]
-{\textstyle{\frac{1}{2}}}\mbox{\rm vec}(\boldsymbol{\Lambda})
\end{array}
\right]
\quad\mbox{and}\quad
\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vech}}\equiv
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}(\xi+1)\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Lambda})
\end{array}
\right].
$$
Mappings between the two natural parameter vectors are as follows:
\begin{equation}
\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vech}}=\mbox{blockdiag}(1,\boldsymbol{D}_d^T)\,\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vec}}
\quad\mbox{and}\quad
\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vec}}=\mbox{blockdiag}(1,\boldsymbol{D}_d^{+T})\,\bdeta_{\mbox{\tiny$\bV$}}^{\mbox{\tiny vech}}.
\label{eq:vecvechmapsIGW}
\end{equation}

\section{Justification of Algorithm \ref{alg:IterInvGWishFrag}}\label{sec:mainAlgDrv}

We now provide justification for Algorithm \ref{alg:IterInvGWishFrag}, which is
concerned with the graph and natural parameter updates for the iterated
Inverse G-Wishart fragment.

\subsection{The Updates for $\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}(\bSigma)$}\label{sec:drvFirstMsg}
As a function of $\boldsymbol{\Sigma}$,
$$\log\,p(\boldsymbol{\Sigma}|\boldsymbol{A})=
\left[
\begin{array}{c}
\log|\boldsymbol{\Sigma}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]^T
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{A}^{-1})
\end{array}
\right]+\mbox{const}
$$
where `const' denotes terms that do not depend on $\boldsymbol{\Sigma}$.
Hence
$$\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}(\bSigma)=\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{\Sigma}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]^T\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}\right\}
$$
where
\begin{equation}
\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}
=
\left[
\begin{array}{c}
-{\textstyle{\frac{1}{2}}}\,(\xi+2)\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}\Big(E_{q}(\boldsymbol{A}^{-1})\Big)
\end{array}
\right]
\label{eq:roosterCrow}
\end{equation}
and $E_{q}$ denotes expectation with respect to the normalization of
$$\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}(\bA)\,\biggerm_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}(\bA).$$
Let $q(\boldsymbol{A})$ denote this normalized density function. Then
$q(\boldsymbol{A})$ is an Inverse-G-Wishart distribution with graph
$G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$ and natural parameter
vector $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}$. From Result \ref{eq:recipMoment},
$$E_{q}(\boldsymbol{A}^{-1})
=\left\{
\begin{array}{l}
\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_1+{\textstyle{\frac{1}{2}}}(d+1)\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_2\Big)\right\}^{-1}\\[2ex]
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\mbox{if $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm full}}}$},\\[2ex]
\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_1+1\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}}\big)_2\Big)\right\}^{-1}\\[2ex]
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
\mbox{if $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm diag}}}$}.
\end{array}
\right.
$$
Noting that the first factor of $E_{q}(\boldsymbol{A}^{-1})$ is
$(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bA$}})_1+\omega_1$,
where
$$\omega_1=\omega_1(d,G)=\left\{
\begin{array}{ll}
(d+1)/2    &  \mbox{if $G=G_{\mbox{\tiny{\rm full}}}$}\\[1ex]
1          &  \mbox{if $G=G_{\mbox{\tiny{\rm diag}}}$},
\end{array}
\right.
$$
the first update of $E_{q}(\boldsymbol{A}^{-1})$ in Algorithm \ref{alg:IterInvGWishFrag} is justified.
Lastly, we need to possibly adjust for the fact that $\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}(\bSigma)$
is proportional to an Inverse G-Wishart density function with $G=G_{\mbox{\tiny{\rm diag}}}$.
This is achieved by the conditional step:

\centerline{If $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm diag}}}$ then
$E_{q}(\boldsymbol{A}^{-1})\longleftarrow\mbox{diag}\left\{\mbox{diagonal}\Big(E_{q}(\boldsymbol{A}^{-1})\Big)\right\}.$}

\subsection{The Updates for $\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}(\bA)$}

As a function of $\boldsymbol{A}$,
$$\log\,p(\boldsymbol{\Sigma}|\boldsymbol{A})=
\left[
\begin{array}{c}
\log|\boldsymbol{A}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{A}^{-1})
\end{array}
\right]^T
\left[
\begin{array}{c}
-(\xi+2-2\omega_2)/2\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]+\mbox{const}
$$
where
\begin{equation}
\omega_2=\omega_2(d,G)=\left\{
\begin{array}{ll}
(d+1)/2    &  \mbox{if $G=G_{\mbox{\tiny{\rm full}}}$},\\[1ex]
1          &  \mbox{if $G=G_{\mbox{\tiny{\rm diag}}}$}
\end{array}
\right.
\label{eq:fryingPan}
\end{equation}
and `const' denotes terms that do not depend on $\boldsymbol{A}$.
Hence
$$\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}(\bA)=\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{A}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{A}^{-1})
\end{array}
\right]^T\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\right\}
$$
where
\begin{equation}
\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}
=
\left[
\begin{array}{c}
-(\xi+2-2\omega_2)/2\\[1ex]
-{\textstyle{\frac{1}{2}}}\boldsymbol{D}_d^T\mbox{\rm vec}\Big(E_{q}(\boldsymbol{\Sigma}^{-1})\Big)
\end{array}
\right]
\label{eq:boobyTrap}
\end{equation}
and $E_{q}$ denotes expectation with respect to the normalization of
$$\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}(\bSigma)\,\biggerm_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}(\bSigma).$$
Let $q(\boldsymbol{\Sigma})$ denote this normalized density function.
Then $q(\boldsymbol{\Sigma})$ is an Inverse-G-Wishart distribution with
graph $\GSUBpSigmaATOSigma\in\{G_{\mbox{\tiny{\rm full}}},G_{\mbox{\tiny{\rm diag}}}\}$
and natural parameter vector $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}$.
From Result \ref{eq:recipMoment},
$$E_{q}(\boldsymbol{\Sigma}^{-1})
=\left\{
\begin{array}{l}
\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_1+{\textstyle{\frac{1}{2}}}(d+1)\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_2\Big)\right\}^{-1}\\[2ex]
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\mbox{if $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm full}}}$},\\[2ex]
\Big\{\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_1+1\Big\}
\left\{\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_d^{+T}\big(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}}\big)_2\Big)\right\}^{-1}\\[2ex]
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
\mbox{if $\GSUBpSigmaATOSigma=G_{\mbox{\tiny{\rm diag}}}$}.
\end{array}
\right.
$$
Noting that the first factor of $E_{q}(\boldsymbol{\Sigma}^{-1})$ is
$(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\leftrightarrow\bSigma$}})_1+\omega_2$,
where $\omega_2$ is given by (\ref{eq:fryingPan}),
the first update of $E_{q}(\boldsymbol{\Sigma}^{-1})$ in Algorithm \ref{alg:IterInvGWishFrag} is justified.
Finally, there is the possible need to adjust for the fact that $\biggerm_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}(\bA)$
is proportional to an Inverse G-Wishart density function with $G=G_{\mbox{\tiny{\rm diag}}}$.
This is achieved by the conditional step:

\centerline{If $G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}=G_{\mbox{\tiny{\rm diag}}}$ then
$E_{q}(\boldsymbol{\Sigma}^{-1})\longleftarrow\mbox{diag}\left\{\mbox{diagonal}\Big(E_{q}(\boldsymbol{\Sigma}^{-1})\Big)\right\}.$}

\section{Illustrative Example Variational Message Passing Details}

The variational message passing approach to fitting and approximate
inference for statistical models is still quite a new concept.
In this section we provide details on the approach for the
illustrative example involving  the $t$ response linear mixed model
described in Section \ref{sec:illustrative}.

\subsection{Data and Hyperparameter Inputs}

Let $\boldsymbol{y}$ be the vector of responses as defined in (\ref{eq:yXZdefn}).
Also, let
$$\boldsymbol{C}=[\boldsymbol{X}\ \boldsymbol{Z}]$$
be the full design matrix, where the matrices $\boldsymbol{X}$ and $\boldsymbol{Z}$
are as defined in (\ref{eq:yXZdefn}).
The data inputs are $\boldsymbol{y}$ and $\boldsymbol{C}$.

The hyperparameter inputs are
$$\sigma_{\boldsymbol{\beta}},s_{\sigma^2},\lambda_{\nu},s_{\mbox{\tiny{$\bSigma,1$}}},\ldotss_{\mbox{\tiny{$\bSigma,q$}}}>0.$$

\subsection{Factor to Stochastic Node Parameter Initialisations}

\noindent
Initialize $G_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow\boldsymbol{A}$}}$ and  $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow \boldsymbol{A}$}}$ via a call to Algorithm \ref{alg:InvGWishPriorFrag} with
hyperparameter inputs:
$$G_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=G_{\mbox{\tiny{\rm diag}}},\quad\xi_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=1\quad\mbox{and}\quad
\boldsymbol{\Lambda}_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=\{2\mbox{diag}(s_{\mbox{\tiny{$\bSigma,1$}}}^2,\ldotss_{\mbox{\tiny{$\bSigma,q$}}}^2)\}^{-1}.
$$
\vskip5mm
\noindent
Initialize $G_{\mbox{\footnotesize$p(a)\rightarrow a$}}$ and $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(a)\rightarrow a$}}$ via a call to Algorithm \ref{alg:InvGWishPriorFrag} with
hyperparameter inputs:
$$G_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=G_{\mbox{\tiny{\rm diag}}},\quad\xi_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=1\quad\mbox{and}\quad
\boldsymbol{\Lambda}_{\mbox{\tiny{$\boldsymbol{\Theta}$}}}=(s_{\sigma}^2)^{-1}.
$$

\noindent
Note that the initialisations of $G_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow\boldsymbol{A}$}}$, $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow \boldsymbol{A}$}}$, $G_{\mbox{\footnotesize$p(a)\rightarrow a$}}$ and $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(a)\rightarrow a$}}$
are part of the prior impositions for $\boldsymbol{\Sigma}$ and $\sigma^2$. These four
factor to stochastic node parameters remain constant throughout the variational
message passing iterations.

\vskip5mm
\noindent
Initialize
$$
\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\upsilon)\rightarrow\upsilon$}}\longleftarrow
\left[
\begin{array}{c}
0\\[1ex]
-\lambda_{\nu}
\end{array}
\right].
$$

\noindent
This initialization of $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\upsilon)\rightarrow\upsilon$}}$ corresponds to the prior imposition
for $\upsilon$. This factor to stochastic node natural parameter remains constant
throughout the variational message passing iterations.

\vskip5mm
The remaining factor to stochastic node natural parameters in the
Figure \ref{fig:tMixModFacGraph} factor graph are updated in the
variational message passing iterations, but require initial values. In theory,
they can be set to any legal value according to the relevant exponential family.
The following initialisations, which are used in the code that
produced Figure \ref{fig:tMixModPosterDens}, are simple legal natural
parameter vectors:

$$G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\longleftarrowG_{\mbox{\tiny{\rm diag}}},\ \
\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}\longleftarrow\left[
\begin{array}{c}
-\frac{1}{2}\\[1ex]
-\frac{1}{2}\boldsymbol{D}_q^T\mbox{\rm vec}(\boldsymbol{I}_q)
\end{array}
\right],
$$

$$\GSUBpSigmaATOSigma\longleftarrowG_{\mbox{\tiny{\rm full}}},\ \
\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}\longleftarrow\left[
\begin{array}{c}
-\frac{1}{2}\\[1ex]
-\frac{1}{2}\boldsymbol{D}_q^T\mbox{\rm vec}(\boldsymbol{I}_q)
\end{array}
\right],
$$

$$G_{\mbox{\footnotesize$p(\sigma^2|\,a)\rightarrow a$}}\longleftarrowG_{\mbox{\tiny{\rm diag}}},\ \
\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow a$}}\longleftarrow\left[
\begin{array}{c}
-2\\[1ex]
-1
\end{array}
\right],
$$

$$G_{\mbox{\footnotesize$p(\sigma^2|\,a)\rightarrow\sigma^2$}}\longleftarrowG_{\mbox{\tiny{\rm full}}},\
\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}\longleftarrow\left[
\begin{array}{c}
-2\\[1ex]
-1
\end{array}
\right],
$$

$$\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}}\longleftarrow\left[
\begin{array}{c}
-\frac{1}{2}\\[1ex]
-\frac{1}{2}\mbox{\rm vec}(\boldsymbol{I}_q)
\end{array}
\right],\quad
\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}}\longleftarrow\left[
\begin{array}{c}
\boldsymbol{0}_{p+m q}\\[1ex]
-\frac{1}{2}\mbox{\rm vec}(\boldsymbol{I}_{p+m q})
\end{array}
\right],
$$

$$\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}}\longleftarrow\left[
\begin{array}{c}
\boldsymbol{0}_{p+m q}\\[1ex]
-\frac{1}{2}\mbox{\rm vec}(\boldsymbol{I}_{p+mq})
\end{array}
\right],\quad
\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}}
\longleftarrow
\left[
\begin{array}{c}
-2\\[1ex]
-1
\end{array}
\right]
$$
and
$$\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{b}|\,\upsilon)\rightarrow\upsilon$}}\longleftarrow\left[
\begin{array}{c}
1\\[1ex]
-1.1
\end{array}
\right].
$$
The messages involving the $b_{\ell}$, $1\le\ell\le N$, nodes do not need to
be included here since there messages are subsumed
in the calculations used for the natural parameter
updates for the model parameters in Algorithm 2 of
McLean \&\  Wand (2019).

\subsection{Variational Message Passing Iterations}

With all factor to stochastic node initialisations accomplished, now
we describe the iterative updates inside the variational message
passing cycle loop. Each iteration involves:
\begin{itemize}
\item updating the stochastic node to factor message parameters.
\item updating the factor to stochastic node message parameters.
\end{itemize}

\subsubsection{Stochastic Node to Factor Message Parameter Updates}

The stochastic node to factor message updates are quite simple
and follow from, e.g., equation (7) of Wand (2017).
For the Figure \ref{fig:tMixModFacGraph} factor graph the updates are:

$$G_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}\longleftarrowG_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow\boldsymbol{A}$}},\quad
\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}\longleftarrow\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{A})\rightarrow \boldsymbol{A}$}},
$$
$$
G_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}\longleftarrowG_{\mbox{\tiny{\rm full}}},\quad
\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}},
$$
$$
\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bbeta,\bu|\bSigma)$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}},\quad
\biggerbdeta_{\mbox{\scriptsize$(\bbeta,\bu)\rightarrow \pDens(\bbeta,\bu|\bSigma)$}}\longleftarrow\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}},
$$
$$\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$(\boldsymbol{\beta},\boldsymbol{u})\rightarrowp(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}},\quad
G_{\mbox{\scriptsize$a\rightarrow p(\sigma^2|a)$}}\longleftarrowG_{\mbox{\footnotesize$p(a)\rightarrow a$}},
$$
$$
\biggerbdeta_{\mbox{\scriptsize$a\rightarrow \pDens(\sigma^2|a)$}}\longleftarrow\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(a)\rightarrow a$}},\quad
G_{\mbox{\scriptsize$\sigma^2\rightarrow \pDens(\sigma^2|a)$}}\longleftarrowG_{\mbox{\tiny{\rm full}}},
$$
$$\biggerbdeta_{\mbox{\scriptsize$\sigma^2\rightarrow \pDens(\sigma^2|a)$}}\longleftarrow\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}},\quad
\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$\sigma^2\rightarrowp(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})$}}\longleftarrow\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}$$
and
$$\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$\upsilon\rightarrowp(\boldsymbol{b}|\,\upsilon)$}}\longleftarrow\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\upsilon)\rightarrow\upsilon$}}.$$

\vskip5mm
Some additional remarks concerning stochastic node to factor updates
are:
\begin{itemize}
\item The stochastic node to factor messages corresponding to the extremities of the
Figure \ref{fig:tMixModFacGraph} factor graph, such as the message from $\boldsymbol{A}$ to $p(\boldsymbol{A})$, are not
required in the variational message passing iterations. Therefore, updates for
these messages can be omitted.
\item Some of the stochastic node to factor message parameter updates, such as that for
$\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$, remain constant throughout the iterations. However, for
simplicity of exposition, we list all of the updates together.
\end{itemize}

\subsubsection{Factor to Stochastic Node Message Parameter Updates}

The updates for the parameters of factor to stochastic node messages
are a good deal more complicated than the reverse messages.
For the illustrative example, these updates are encapsulated in
three algorithms across three different articles. Algorithm \ref{alg:IterInvGWishFrag}
plays an important role for the variance and covariance matrix parameter
parts of the factor graph.

\begin{itemize}
\setlength\itemsep{0pt}
\item[] \textrm{Use Algorithm \ref{alg:IterInvGWishFrag} with:}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] \textbf{\small Shape Parameter Input}: $1$.
\item[] \textbf{\small Graph Inputs}:  $G_{\mbox{\scriptsize$\sigma^2\rightarrow \pDens(\sigma^2|a)$}}$, $G_{\mbox{\scriptsize$a\rightarrow p(\sigma^2|a)$}}$.
\item[] \textbf{\small Natural Parameter Inputs:} $\biggerbdeta_{\mbox{\scriptsize$\sigma^2\rightarrow \pDens(\sigma^2|a)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}$
 $\biggerbdeta_{\mbox{\scriptsize$a\rightarrow \pDens(\sigma^2|a)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow a$}}$
 \item[] \textbf{\small Outputs:}
        $G_{\mbox{\footnotesize$p(\sigma^2|\,a)\rightarrow\sigma^2$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}$,
	$G_{\mbox{\footnotesize$p(\sigma^2|\,a)\rightarrow a$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow a$}}$
\end{itemize}
				%

\item[] \textrm{Use Algorithm \ref{alg:IterInvGWishFrag} with:}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] \textbf{\small Shape Parameter Input}: $2q$
\item[] \textbf{\small Graph Inputs}: $G_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}$, $G_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$
\item[] \textbf{\small Natural Parameter Inputs:} $\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bSigma|\bA)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}$,
$\biggerbdeta_{\mbox{\scriptsize$\bA\rightarrow \pDens(\bSigma|\bA)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}$
\item[] \textbf{\small Outputs:} $\GSUBpSigmaATOSigma$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}$,
$G_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bA$}}$
\end{itemize}
\item[] \textrm{Use the Gaussian Penalisation Fragment of Wand (2017, Section 4.1.4):}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] \textbf{\small Hyperparameter Input:} $\sigma^2_{\boldsymbol{\beta}}$
\item[] \textbf{\small Natural Parameter Inputs:} $\biggerbdeta_{\mbox{\scriptsize$(\bbeta,\bu)\rightarrow \pDens(\bbeta,\bu|\bSigma)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}}$,\\
\null$\qquad\qquad\qquad\qquad\qquad\quad\,\biggerbdeta_{\mbox{\scriptsize$\bSigma\rightarrow \pDens(\bbeta,\bu|\bSigma)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}}$
\item[] \textbf{\small Outputs:} $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}}$, $\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}}$
\end{itemize}
\item[] \textrm{Use the $t$ Likelihood Fragment of McLean \&\  Wand (2019, Algorithm 2):}
\begin{itemize}
\setlength\itemsep{0pt}
\item[] \textbf{\small Data Inputs:} $\boldsymbol{y}$, $\boldsymbol{C}$
\item[] \textbf{\small Natural Parameter Inputs:} $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$(\boldsymbol{\beta},\boldsymbol{u})\rightarrowp(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})$}}$, $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}}$,\\
\null$\qquad\qquad\qquad\qquad\qquad\quad\,\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$\sigma^2\rightarrowp(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})$}}$, $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}}$,\\
\null$\qquad\qquad\qquad\qquad\qquad\quad\,\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$\upsilon\rightarrowp(\boldsymbol{b}|\,\upsilon)$}}$, $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{b}|\,\upsilon)\rightarrow\upsilon$}}$
\item[] \textbf{\small Outputs:} $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}}$, $\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}}$,
$\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{b}|\,\upsilon)\rightarrow\upsilon$}}$
\end{itemize}
\end{itemize}

Regarding, the last two fragment updates it should be noted
that Wand (2017) and McLean \&\  Wand (2019) work with
the ``$\mbox{\rm vec}$'' versions of Multivariate Normal and Inverse G-Wishart
natural parameter vectors. To match the ``$\mbox{\rm vech}$'' natural parameter
forms used in Algorithms \ref{alg:InvGWishPriorFrag} and
\ref{alg:IterInvGWishFrag} of the current article conversions
given by (\ref{eq:vecvechmapsMVN}) and (\ref{eq:vecvechmapsIGW})
are required.

\subsection{Determination of Posterior Density Function Approximations}

After convergence of the variational message passing iterations,
the optimal $q^*$-densities for each stochastic node are obtained by
multiplying each of the messages that pass messages to that node.
See, for example, (10) of Wand (2017). We now give
details for the model parameters $\boldsymbol{\Sigma}$, $\sigma^2$, $(\boldsymbol{\beta},\boldsymbol{u})$ and
$\upsilon$.

\subsubsection{Determination of $q^*(\boldsymbol{\Sigma})$}

From (10) of Wand (2017):
$$q^*(\boldsymbol{\Sigma})\propto\exp\left\{
\left[
\begin{array}{c}
\log|\boldsymbol{\Sigma}|\\[1ex]
\mbox{\rm vech}(\boldsymbol{\Sigma}^{-1})
\end{array}
\right]^T
\left(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}+\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}}\right)
\right\}.
$$
It is apparent that $q^*(\boldsymbol{\Sigma})$ is an Inverse Wishart
density function with natural parameter vector
$$\mbox{\Large $\bdeta$}_{q(\boldsymbol{\Sigma})}\equiv \biggerbdeta_{\mbox{\scriptsize$\pDens(\bSigma|\bA)\rightarrow\bSigma$}}+\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow\bSigma$}}.$$

\subsubsection{Determination of $q^*(\sigma^2)$}

Using (10) of Wand (2017):
$$q^*(\sigma^2)\propto\exp\left\{
\left[
\begin{array}{c}
\log(\sigma^2)\\[1ex]
1/\sigma^2
\end{array}
\right]^T
\left(\biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}}\right)
\right\}.
$$
We see that $q^*(\sigma^2)$ is an Inverse Chi-Squared
density function with natural parameter vector
$$\mbox{\Large $\bdeta$}_{q(\sigma^2)}\equiv \biggerbdeta_{\mbox{\scriptsize$\pDens(\sigma^2|a)\rightarrow\sigma^2$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow\sigma^2$}}.$$

\subsubsection{Determination of $q^*(\boldsymbol{\beta},\boldsymbol{u})$}

Another application of (10) of Wand (2017) leads to:
$$q^*(\boldsymbol{\beta},\boldsymbol{u})\propto\exp\left\{
\left[
\begin{array}{c}
\boldsymbol{\beta}\\[0ex]
\boldsymbol{u}\\[1ex]
\mbox{\rm vech}\left(\left[
\begin{array}{c}
\boldsymbol{\beta}\\[0ex]
\boldsymbol{u}
\end{array}
\right]
\left[
\begin{array}{c}
\boldsymbol{\beta}\\[0ex]
\boldsymbol{u}
\end{array}
\right]^T
\right)
\end{array}
\right]^T
\left(\biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}}\right)
\right\}.
$$
We then have $q^*(\boldsymbol{\beta},\boldsymbol{u})$ having a Multivariate Normal
density function with natural parameter vector
$$\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\equiv \biggerbdeta_{\mbox{\scriptsize$\pDens(\bbeta,\bu|\bSigma)\rightarrow(\bbeta,\bu)$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{y}|\,\boldsymbol{\beta},\boldsymbol{u},\sigma^2,\boldsymbol{b})\rightarrow(\boldsymbol{\beta},\boldsymbol{u})$}}.$$

\subsubsection{Determination of $q^*(\upsilon)$}

One last application of (10) of Wand (2017) gives:
$$q^*(\upsilon)\propto\exp\left\{
\left[
\begin{array}{c}
\upsilon\log(\upsilon)-\log\{\Gamma(\upsilon)\}\\[1ex]
\upsilon
\end{array}
\right]^T
\left(\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{b}|\,\upsilon)\rightarrow\upsilon$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\upsilon)\rightarrow\upsilon$}}\right)
\right\}.
$$
Therefore, $q^*(\upsilon)$ is a Moon Rock density function with natural parameter
vector
$$\mbox{\Large $\bdeta$}_{q(\upsilon)}\equiv\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\boldsymbol{b}|\,\upsilon)\rightarrow\upsilon$}}+\mbox{\Large $\bdeta$}_{\mbox{\footnotesize$p(\upsilon)\rightarrow\upsilon$}}.$$

\subsection{Conversion from Natural Parameters to Common Parameters}

A final set of steps involves conversion of the $q^*$-densities to common parameter
forms.

\subsubsection{Conversion of $q^*(\boldsymbol{\Sigma})$ to Common Parameter Form}

The common parameter form of $q^*(\boldsymbol{\Sigma})$ is the
$\mbox{Inverse-G-Wishart}(G_{\mbox{\tiny{\rm full}}},\xi_{q(\boldsymbol{\Sigma})},\boldsymbol{\Lambda}_{q(\boldsymbol{\Sigma})})$
density function where
$$\xi_{q(\boldsymbol{\Sigma})}=-2\big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\Sigma})}\big)_1-2
\quad\mbox{and}\quad
\boldsymbol{\Lambda}_{q(\boldsymbol{\Sigma})}=-2\mbox{\rm vec}^{-1}\Big(\boldsymbol{D}_q^{+T}\big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\Sigma})}\big)_2\Big).
$$
Alternatively, $q^*(\boldsymbol{\Sigma})$ is the
$\mbox{Inverse-Wishart}(\kappa_{q(\boldsymbol{\Sigma})},\boldsymbol{\Lambda}_{q(\boldsymbol{\Sigma})})$ density
function, as defined by (\ref{eq:GelmanTable}), where
$$\kappa_{q(\boldsymbol{\Sigma})}=\xi_{q(\boldsymbol{\Sigma})}-q+1.$$

\subsubsection{Conversion of $q^*(\sigma^2)$ to Common Parameter Form}

The common parameter form of  $q^*(\sigma^2)$ is the
$\mbox{{\rm Inverse}-$\chi^2$}(\delta_{q(\sigma^2)},\lambda_{q(\sigma^2)})$
density function where
$$\delta_{q(\sigma^2)}=-2\big(\mbox{\Large $\bdeta$}_{q(\sigma^2)}\big)_1-2
\quad\mbox{and}\quad
\lambda_{q(\sigma^2)}=-2\big(\mbox{\Large $\bdeta$}_{q(\sigma^2)}\big)_2.
$$

\subsubsection{Conversion of $q^*(\boldsymbol{\beta},\boldsymbol{u})$ to Common Parameter Form}

The common parameter form of $q^*(\boldsymbol{\beta},\boldsymbol{u})$ is the
$N\big(\boldsymbol{\mu}_{q(\boldsymbol{\beta},\boldsymbol{u})},\boldsymbol{\Sigma}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)$ density function
where
$$\boldsymbol{\mu}_{q(\boldsymbol{\beta},\boldsymbol{u})}=-{\textstyle{\frac{1}{2}}}\left\{\mbox{\rm vec}^{-1}
\Big(\boldsymbol{D}_{p+mq}^{+T} \big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)_2\Big)
\right\}^{-1}\big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)_1
$$
and
$$
\boldsymbol{\Sigma}_{q(\boldsymbol{\beta},\boldsymbol{u})}=-{\textstyle{\frac{1}{2}}}\left\{\mbox{\rm vec}^{-1}
\Big(\boldsymbol{D}_{p+mq}^{+T} \big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)_2\Big)
\right\}^{-1}.
$$
Here $\big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)_1$ denotes the first $p+mq$ entries of
$\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}$ and $\big(\mbox{\Large $\bdeta$}_{q(\boldsymbol{\beta},\boldsymbol{u})}\big)_2$
denotes the remaining entries of the same vector.

\subsubsection{Conversion of $q^*(\upsilon)$ to Common Parameter Form and Conversion to $q^*(\nu)$ }

Recall that $q^*(\upsilon)$ is a Moon Rock density function. The Moon Rock distribution
is not as established as the other distributions appearing in this subsection. Nevertheless,
the web-supplement of McLean \&\  Wand (2019) defines a random variable $x$ to have
a Moon Rock distribution with parameters $\alpha>0$ and $\beta>\alpha$, written
$x\sim\mbox{Moon-Rock}(\alpha,\beta)$, if the density function of $x$ is
$$p(x)=\left[\int_0^{\infty}\{t^t/\Gamma(t)\}^{\alpha}\exp(-\beta\,t)\,dt\right]^{-1}
\{x^x/\Gamma(x)\}^{\alpha}\exp(-\beta\,x),\quad x>0.
$$
Therefore, $q^*(\upsilon)$ has a $\mbox{Moon-Rock}(\alpha_{q(\upsilon)},\beta_{q(\upsilon)})$
density function where
$$\alpha_{q(\upsilon)}=\big(\mbox{\Large $\bdeta$}_{q(\upsilon)}\big)_1
\quad\mbox{and}\quad\beta_{q(\upsilon)}=\,-\big(\mbox{\Large $\bdeta$}_{q(\upsilon)}\big)_2.
$$
Explicitly,
$$q^*(\upsilon)=\left[\int_0^{\infty}\{t^t/\Gamma(t)\}^{\alpha_{q(\upsilon)}}
\exp(-\beta_{q(\upsilon)}\,t)\,dt\right]^{-1}
\{\upsilon^{\upsilon}/\Gamma(\upsilon)\}^{\alpha_{q(\upsilon)}}
\exp(-\beta_{q(\upsilon)}\,\upsilon),\quad \upsilon>0.
$$
Lastly, we note that since $\nu=2\upsilon$ the $q^*$-density function of
$\nu$ is
{\setlength\arraycolsep{3pt}
\begin{eqnarray*}
&&q^*(\nu)={\textstyle{\frac{1}{2}}}
\left[\int_0^{\infty}\{t^t/\Gamma(t)\}^{\alpha_{q(\upsilon)}}
\exp(-\beta_{q(\upsilon)}\,t)\,dt\right]^{-1}\\[1ex]
&&\qquad\qquad\qquad\qquad\times
\{(\nu/2)^{\nu/2}/\Gamma(\nu/2)\}^{\alpha_{q(\upsilon)}}
\exp\big(-{\textstyle{\frac{1}{2}}} \beta_{q(\upsilon)}\nu\big),
\quad\nu>0.
\end{eqnarray*}
}

\section*{Reference}

\vskip12pt\par\noindent\hangindent=1 true cm\hangafter=1
Novomestky, F. (2012).
\textsf{matrixcalc}: Collection of functions for matrix calculations.
\textsf{R} package. \texttt{https://CRAN.R-project.org/package=matrixcalc}

\end{document}