Consider the following mixture model for a univariate density function
$$
(1) \quad f(x)=\int\_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))
$$
where $D$ is a compact subset of $\mathbb{R}\times \mathbb{R}^+$, $(m, \sigma^2)$ denotes the pair of mean and variance, $g(\cdot; m, \sigma^2)$ is the univariate Normal density function
with mean $m$ and variance $\sigma^2$, $\mu$ is a probability measure over $D$.


I'm interested in understanding which classes of density functions can (or cannot) be "approximated" as in (1).


That is, can we characterise the class, $\mathcal{F}$, of densities, $f(\cdot)$, for which there exists $D,\mu$ such that the "distance" between $f(\cdot)$ and $\int\_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))$ is small?


I found in various papers/books sentences along the lines of "There is an obvious sense in which the mixture of normals approach, given enough components, can approximate any density" (see [here](http://assets.press.princeton.edu/chapters/s10259.pdf) for instance).
I also found some papers proposing formalisation of this sentence within the finite mixture case. See also [here](https://mathoverflow.net/questions/27589/mixtures-of-gaussian-distributions-dense-in-distributions) for a related question. However, I could not find anything formally dealing with the infinite mixture case as (1).


Would you have some references to suggest? If I set $D$ very large, wouldn't (1) encompass a quite general class of densities?



About a mixture
$\newcommand\R{\mathbb R}\newcommand\si{\sigma}\newcommand\ep{\varepsilon}\newcommand\de{\delta}$Any probability distribution on $\R$ can be approximated by a discrete probability distribution on $\R$. Any discrete probability distribution on $\R$ is a mixture of Dirac probability distributions on $\R$. The Dirac probability distribution supported at a point $a\in\R$ can be approximated by the normal distribution with a small variance centered at $a$.


Thus indeed, if the set $D$ is large enough, then any probability distribution on $\R$ can be approximated by a mixture of normal distributions $N(m,\sigma^2)$ with $(m,\si^2)\in D$.




---


Another way: any probability distribution $\nu$ on $\R$ can be approximated by the convolution $\nu\*N(0,\si^2)$ of $\nu$ with the centered normal distribution $N(0,\si^2)$ with a small variance $\si^2$, and such a convolution is a mixture of normal distributions:
\begin{equation\*}
 \nu\*N(0,\si^2)=\int\_\R N(y,\si^2)\nu(dy). \tag{0}
\end{equation\*}
If now the set $D$ is large enough to, say, contain a set of the form $C\_\de\times(0,\ep)$, where $\ep\in(0,\infty)$, $\de$ is a small positive number, and the set $C\_\de\subseteq\R$ is such that $\nu(C\_\de)>1-\de$, then for small $\si^2\in(0,\ep)$
\begin{equation\*}
 \nu\approx\nu\*N(0,\si^2)\approx\int\_{C\_\de} N(y,\si^2)\nu(dy)
=\int\_D N(y,s^2)\mu(dy\times d(s^2)), \tag{1}
\end{equation\*}
where $\mu(dy\times d(s^2)):=\nu(dy)1(y\in C\_\de,s^2=\si^2)$ --
so that $\nu$ will be approximated by the mixture $\int\_D N(y,s^2)\mu(dy\times d(s^2))$ of normal distributions $N(m,s^2)$ with $(m,s^2)\in D$.


**Details:** The approximate equalities in (1) can be understood as follows. Suppose that $\si\_n^2\downarrow0$ and $\de\_n\downarrow0$ (as $n\to\infty$). Then, by [Slutsky's theorem](https://en.wikipedia.org/wiki/Slutsky%27s_theorem), $\nu\*N(0,\si\_n^2)\to\nu$ weakly; that is, for any bounded continuous function $g\colon\R\to\R$ we have
\begin{equation\*}
 (\nu\*N(0,\si\_n^2))(g)\to\nu(g), 
\end{equation\*}
where $\nu(g):=\int\_\R g\,d\nu$; this is how the first approximate equality in (1) can be understood.


Next, by (0), for any bounded continuous function $g\colon\R\to\R$ with $M:=\sup\_{x\in\R}|g(x)|<\infty$ we have
\begin{equation\*}
\begin{aligned}
 &\Big|(\nu\*N(0,\si\_n^2))(g)-\int\_{C\_{\de\_n}} N(y,\si\_n^2)(g)\nu(dy)\Big| \\ 
 =&\Big|\int\_{\R} N(y,\si\_n^2)(g)\nu(dy)-\int\_{C\_{\de\_n}} N(y,\si\_n^2)(g)\nu(dy)\Big| \\ 
 \le&\int\_{\R\setminus C\_{\de\_n}} N(y,\si^2)(|g|)\nu(dy) \\ 
 \le& M\nu(\R\setminus C\_{\de\_n})\le M\de\_n\to0; 
\end{aligned}
\end{equation\*}
this is how the second approximate equality in (1) can be understood.


So, for any bounded continuous function $g\colon\R\to\R$,
\begin{equation\*}
 \Big(\int\_{C\_{\de\_n}} N(y,\si\_n^2)\nu(dy)\Big)(g)
=\int\_{C\_{\de\_n}} N(y,\si\_n^2)(g)\nu(dy)\to\nu(g). 
\end{equation\*}
That is, we have the weak convergence of the normal mixtures $\int\_{C\_{\de\_n}} N(y,\si\_n^2)\nu(dy)$ to $\nu$. (The equality in the latter display is an instance of the Fubini theorem.)


