arXiv:1410.6843v2 [] 22 Apr 2016

Posteriors, conjugacy, and exponential families
for completely random measures
Tamara Broderick

Ashia C. Wilson

Michael I. Jordan

April 25, 2016
Abstract
We demonstrate how to calculate posteriors for general Bayesian nonparametric priors and likelihoods based on completely random measures
(CRMs). We further show how to represent Bayesian nonparametric priors as a sequence of finite draws using a size-biasing approach—and how
to represent full Bayesian nonparametric models via finite marginals. Motivated by conjugate priors based on exponential family representations
of likelihoods, we introduce a notion of exponential families for CRMs,
which we call exponential CRMs. This construction allows us to specify
automatic Bayesian nonparametric conjugate priors for exponential CRM
likelihoods. We demonstrate that our exponential CRMs allow particularly straightforward recipes for size-biased and marginal representations
of Bayesian nonparametric models. Along the way, we prove that the
gamma process is a conjugate prior for the Poisson likelihood process and
the beta prime process is a conjugate prior for a process we call the odds
Bernoulli process. We deliver a size-biased representation of the gamma
process and a marginal representation of the gamma process coupled with
a Poisson likelihood process.

1

Introduction

An important milestone in Bayesian analysis was the development of a general
strategy for obtaining conjugate priors based on exponential family representations of likelihoods [DeGroot, 1970]. While slavish adherence to exponentialfamily conjugacy can be criticized, conjugacy continues to occupy an important
place in Bayesian analysis, for its computational tractability in high-dimensional
problems and for its role in inspiring investigations into broader classes of priors (e.g., via mixtures, limits, or augmentations). The exponential family is,
however, a parametric class of models, and it is of interest to consider whether
similar general notions of conjugacy can be developed for Bayesian nonparametric models. Indeed, the nonparametric literature is replete with nomenclature
that suggests the exponential family, including familiar names such as “Dirichlet,” “beta,” “gamma,” and “Poisson.” These names refer to aspects of the

1

random measures underlying Bayesian nonparametrics, either the Lévy measure used in constructing certain classes of random measures or properties of
marginals obtained from random measures. In some cases, conjugacy results
have been established that parallel results from classical exponential families; in
particular, the Dirichlet process is known to be conjugate to a multinomial process likelihood [Ferguson, 1973], the beta process is conjugate to a Bernoulli
process [Kim, 1999, Thibaux and Jordan, 2007] and to a negative binomial
process [Broderick et al., 2015]. Moreover, various useful representations for
marginal distributions, including stick-breaking and size-biased representations,
have been obtained by making use of properties that derive from exponential
families. It is striking, however, that these results have been obtained separately,
and with significant effort; a general formalism that encompasses these individual results has not yet emerged. In this paper, we provide the single, holistic
framework so strongly suggested by the nomenclature. Within this single framework, we show that it is straightforward to calculate posteriors and establish
conjugacy. Our framework includes the specification of a Bayesian nonparametric analog of the finite exponential family, which allows us to provide automatic
and constructive nonparametric conjugate priors given a likelihood specification
as well as general recipes for marginal and size-biased representations.
A broad class of Bayesian nonparametric priors—including those built on the
Dirichlet process [Ferguson, 1973], the beta process [Hjort, 1990], the gamma
process [Ferguson, 1973, Lo, 1982, Titsias, 2008], and the negative binomial
process [Zhou et al., 2012, Broderick et al., 2015]—can be viewed as models
for the allocation of data points to traits. These processes give us pairs of
traits together with rates or frequencies with which the traits occur in some
population. Corresponding likelihoods assign each data point in the population
to some finite subset of traits conditioned on the trait frequencies. What makes
these models nonparametric is that the number of traits in the prior is countably
infinite. Then the (typically random) number of traits to which any individual
data point is allocated is unbounded, but also there are always new traits to
which as-yet-unseen data points may be allocated. That is, such a model allows
the number of traits in any data set to grow with the size of that data set.
A principal challenge of working with such models arises in posterior inference. There is a countable infinity of trait frequencies in the prior which we
must integrate over to calculate the posterior of trait frequencies given allocations of data points to traits. Bayesian nonparametric models sidestep the full
infinite-dimensional integration in three principal ways: conjugacy, size-biased
representations, and marginalization.
In its most general form, conjugacy simply asserts that the prior is in the
same family of distributions as the posterior. When the prior and likelihood are
in finite-dimensional conjugate exponential families, conjugacy can turn posterior calculation into, effectively, vector addition. As a simple example, consider a
model with beta-distributed prior, θ ∼ Beta(θ|α, β), for some fixed hyperparameters α and β. For the likelihood, let each observation xn with n ∈ {1, . . . , N } be
iid

iid Bernoulli-distributed conditional on parameter θ: xn ∼ Bern(x|θ). Then the

2

posterior is simply another beta distribution, Beta(θ|αpost , βpost ), with paramePN
PN
ters updated via addition: αpost := α + n=1 xn and βpost := β + N − n=1 xn .
While conjugacy is certainly useful and popular in the case of finite parameter cardinality, there is arguably a stronger computational imperative for its
use in the infinite-parameter case. Indeed, the core prior-likelihood pairs of
Bayesian nonparametrics are generally proven [Hjort, 1990, Kim, 1999, Lo, 1982,
Thibaux and Jordan, 2007, Broderick et al., 2015], or assumed to be [Titsias,
2008, Thibaux, 2008], conjugate. When such proofs exist, though, thus far
they have been specialized to specific pairs of processes. In what follows, we
demonstrate a general way to calculate posteriors for a class of distributions that
includes all of these classical Bayesian nonparametric models. We also define
a notion of exponential family representation for the infinite-dimensional case
and show that, given a Bayesian nonparametric exponential family likelihood,
we can readily construct a Bayesian nonparametric conjugate prior.
Size-biased sampling provides a finite-dimensional distribution for each of
the individual prior trait frequencies [Thibaux and Jordan, 2007, Paisley et al.,
2010]. Such a representation has played an important role in Bayesian nonparametrics in recent years, allowing for either exact inference via slice sampling [Damien et al., 1999, Neal, 2003]—as demonstrated by Teh et al. [2007],
Broderick et al. [2015]—or approximate inference via truncation [Doshi et al.,
2009, Paisley et al., 2011]. This representation is particularly useful for building
hierarchical models [Thibaux and Jordan, 2007]. We show that our framework
yields such representations in general, and we show that our construction is
especially straightforward to use in the exponential family framework that we
develop.
Marginal processes avoid directly representing the infinite-dimensional prior
and posterior altogether by integrating out the trait frequencies. Since the
trait allocations are finite for each data point, the marginal processes are finite for any finite set of data points. Again, thus far, such processes have
been shown to exist separately in special cases; for example, the Indian buffet
process [Griffiths and Ghahramani, 2006] is the marginal process for the beta
process prior paired with a Bernoulli process likelihood [Thibaux and Jordan,
2007]. We show that the integration that generates the marginal process from
the full Bayesian model can be generally applied in Bayesian nonparametrics
and takes a particularly straightforward form when using conjugate exponential
family priors and likelihoods. We further demonstrate that, in this case, a basic, constructive recipe exists for the general marginal process in terms of only
finite-dimensional distributions.
Our results are built on the general class of stochastic processes known as
completely random measures (CRMs) [Kingman, 1967]. We review CRMs in
Section 2.1 and we discuss what assumptions are needed to form a full Bayesian
nonparametric model from CRMs in Section 2.3. Given a general Bayesian
nonparametric prior and likelihood (Section 2.2), we demonstrate in Section 3
how to calculate the posterior. Although the development up to this point is
more general, we next introduce a concept of exponential families for CRMs

3

(Section 4.1) and call such models exponential CRMs. We show that we can
generate automatic conjugate priors given exponential CRM likelihoods in Section 4.2. Finally, we show how we can generate recipes for size-biased representations (Section 5) and marginal processes (Section 6), which are particularly
straightforward in the exponential CRM case (Corollary 5.2 in Section 5 and
Corollary 6.2 in Section 6). We illustrate our results on a number of examples
and derive new conjugacy results, size-biased representations, and marginal processes along the way.
We note that some similar results have been obtained by Orbanz [2010] and
James [2014]. In the present work, we focus on creating representations that
allow tractable inference.

2

Bayesian models based on completely random
measures

As we have discussed, we view Bayesian nonparametric models as being composed of two parts: (1) a collection of pairs of traits together with their frequencies or rates and (2) for each data point, an allocation to different traits. Both
parts can be expressed as random measures. Recall that a random measure is
a random element whose values are measures.
We represent each trait by a point ψ in some space Ψ of traits. Further,
let θk be the frequency, or rate, of the trait represented by ψk , where k indexes
the countably many traits. In particular, θk ∈ R+ . Then (θk , ψk ) is a tuple
consisting of the frequency of the kth trait together with its trait descriptor.
We can represent the full collection of pairs of traits with their frequencies by
the discrete measure on Ψ that places weight θk at location ψk :
Θ=

K
X

θk δ ψ k ,

(1)

k=1

where the cardinality K may be finite or infinity.
Next, we form data point Xn for the nth individual. The data point Xn is
viewed as a discrete measure. Each atom of Xn represents a pair consisting of
(1) a trait to which the nth individual is allocated and (2) a degree to which
the nth individual is allocated to this particular trait. That is,
Xn =

Kn
X

xn,k δψn,k ,

(2)

k=1

where again ψn,k ∈ Ψ represents a trait and now xn,k ∈ R+ represents the degree
to which the nth data point belongs to trait ψn,k . Kn is the total number of
traits to which the nth data point belongs.
Here and in what follows, we treat X1:N = {Xn : n ∈ [N ]} as our observed
data points for [N ] := {1, 2, 3, . . . , N }. In practice X1:N is often incorporated

4

into a more complex Bayesian hierarchical model. For instance, in topic modeling, ψk represents a topic; that is, ψk is a distribution over words in a vocabulary [Blei et al., 2003, Teh et al., 2006]. θk might represent the frequency with
which the topic ψk occurs in a corpus of documents. xn,k might be a positive
integer and represent the number of words in topic ψn,k that occur in the nth
P Kn
document. So the nth document has a total length of k=1
xn,k words. In this
case, the actual observation consists of the words in each document, and the
topics are latent. Not only are the results concerning posteriors, conjugacy, and
exponential family representations that we develop below useful for inference in
such models, but in fact our results are especially useful in such models—where
the traits and any ordering on the traits are not known in advance.
Next, we want to specify a full Bayesian model for our data points X1:N .
To do so, we must first define a prior distribution for the random measure Θ
as well as a likelihood for each random measure Xn conditioned on Θ. We let
ΣΨ be a σ-algebra of subsets of Ψ, where we assume all singletons are in ΣΨ .
Then we consider random measures Θ and Xn whose values are measures on Ψ.
Note that for any random measure Θ and any measurable set A ∈ ΣΨ , Θ(A) is
a random variable.

2.1

Completely random measures

We can see from Eqs. (1) and (2) that we desire a distribution on random
measures that yields discrete measures almost surely. A particularly simple
form of random measure called a completely random measure can be used to
generate a.s. discrete random measures [Kingman, 1967].
A completely random measure Θ is defined as a random measure that satisfies one additional property; for any disjoint, measurable sets A1 , A2 , . . . , AK ∈
ΣΨ , we require that Θ(A1 ), Θ(A2 ), . . . , Θ(AK ) be independent random variables. Kingman [1967] showed that a completely random measure can always
be decomposed into a sum of three independent parts:
Θ = Θdet + Θf ix + Θord .

(3)

Here, Θdet is the deterministic component, Θf ix is the fixed-location component,
and Θord is the ordinary component. In particular, Θdet is any deterministic
measure. We define the remaining two parts next.
The fixed-location component is called the “fixed component” by Kingman
[1967], but we expand the name slightly here to emphasize that Θf ix is defined
to be constructed from a set of random weights at fixed (i.e., deterministic)
locations. That is,
Kf ix
X
θf ix,k δψf ix,k ,
(4)
Θf ix =
k=1

where the number of fixed-location atoms, Kf ix , may be either finite or infinity;
ψf ix,k is deterministic, and θf ix,k is a non-negative, real-valued random variable
(since Φ is a measure). Without loss of generality, we assume that the locations
5

ψf ix,k are all distinct. Then, by the independence assumption of CRMs, we must
have that θf ix,k are independent random variables across k. Although the fixedlocation atoms are often ignored in the Bayesian nonparametrics literature, we
will see that the fixed-location component has a key role to play in establishing
Bayesian nonparametric conjugacy and in the CRM representations we present.
The third and final component is the ordinary component. Let #(A) denote
the cardinality of some countable set A. Let µ be any σ-finite, deterministic
measure on R+ × Ψ, where R+ is equipped with the Borel σ-algebra and ΣR+ ×Ψ
is the resulting product σ-algebra given ΣΨ . Recall that a Poisson point process
with rate measure µ on R+ × Ψ is a random countable subset Π of R+ × Ψ such
that two properties hold [Kingman, 1992]:
1. For any A ∈ ΣR+ ×Ψ , #(Π ∩ A) ∼ Poisson(µ(A)).
2. For any disjoint A1 , A2 , . . . , AK ∈ ΣR+ ×Ψ , #(Π∩A1 ), #(Π∩A2 ), · · · , #(Π∩
AK ) are independent random variables.
To generate an ordinary component, start with a Poisson point process on R+ ×
Ψ, characterized by its rate measure µ(dθ×dψ). This process yields Π, a random
ord
and countable set of points: Π = {(θord,k , ψord,k )}K
k=1 , where Kord may be finite
or infinity. Form the ordinary component measure by letting θord,k be the weight
of the atom located at ψord,k :
Θord =

K
ord
X

θord,k δψord,k .

(5)

k=1

Recall that we stated at the start of Section 2.1 that CRMs may be used to
produce a.s. discrete random measures. To check this assertion, note that Θf ix
is a.s. discrete by construction (Eq. (4)) and Θord is a.s. discrete by construction
(Eq. (5)). Θdet is the one component that may not be a.s. atomic. Thus the
prevailing norm in using models based on CRMs is to set Θdet ≡ 0; in what
follows, we adopt this norm. If the reader is concerned about missing any atoms
in Θdet , note that it is straightforward to adapt the treatment of Θf ix to include
the case where the atom weights are deterministic. When we set Θdet ≡ 0, we
are left with Θ = Θf ix + Θord by Eq. (3). So Θ is also discrete, as desired.

2.2

Prior and likelihood

The prior that we place on Θ will be a fully general CRM (minus any deterministic component) with one additional assumption on the rate measure of the
ordinary component. Before incorporating the additional assumption, we say
that Θ has a fixed-location component with Kf ix atoms, where the kth atom
indep

has arbitrary distribution Ff ix,k : θf ix,k ∼ Ff ix,k (dθ). Kf ix may be finite
or infinity, and Θ has an ordinary component characterized by rate measure
µ(dθ × dψ). The additional assumption we make is that the distribution on

6

the weights in the ordinary component is assumed to be decoupled from the
distribution on the locations. That is, the rate measure decomposes as
µ(dθ × dψ) = ν(dθ) · G(dψ),

(6)

where ν is any σ-finite, deterministic measure on R+ and G is any proper
distribution on Ψ. While the distribution over locations has been discussed
extensively elsewhere [Neal, 2000, Wang and Blei, 2013], it is the weights that
affect the allocation of data points to traits.
Given the factorization of µ in Eq. (6), the ordinary component of Θ can
ord
be generated by letting {θf ix,k }K
k=1 be the points of a Poisson point process
1
ord
generated on R+ with rate ν. We then draw the locations {ψf ix,k }K
k=1 iid
iid

according to G(dψ): ψf ix,k ∼ G(dψ). Finally, for each k, θf ix,k δψf ix,k is an
atom in Θord. This factorization will allow us to focus our attention on the
trait frequencies, and not the trait locations, in what follows. Moreover, going
forward, we will assume G is diffuse (i.e., G has no atoms) so that the ordinary
component atoms are all at a.s. distinct locations, which are further a.s. distinct
from the fixed locations.
Since we have seen that Θ is an a.s. discrete random measure, we can write
it as
K
X
Θ=
θk δ ψ k ,
(7)
k=1

where K := Kf ix + Kord may be finite or infinity, and every ψk is a.s. unique.
That is, we will sometimes find it helpful notationally to use Eq. (7) instead of
separating the fixed and ordinary components. At this point, we have specified
the prior for Θ in our general model.
Next, we specify the likelihood; i.e., we specify how to generate the data
points Xn given Θ. We will assume each Xn is generated iid given Θ across the
data indices n. We will let Xn be a CRM with only a fixed-location component
given Θ. In particular, the atoms of Xn will be located at the atom locations
of Θ, which are fixed when we condition on Θ:
Xn :=

K
X

xn,k δψk .

k=1

Here, xn,k is drawn according to some distribution H that may take θk , the
weight of Θ at location ψk , as a parameter; i.e.,
indep

xn,k ∼ H(dx|θk ) independently across n and k.

(8)

Note that while every atom of Xn is located at an atom of Θ, it is not
necessarily the case that every atom of Θ has a corresponding atom in Xn . In
particular, if xn,k is zero for any k, there is no atom in Xn at ψk .
1 Recall that K
ord may be finite or infinity depending on ν and is random when taking
finite values.

7

We highlight that the model above stands in contrast to Bayesian nonparametric partition models, for which there is a large literature. In partition models
(or clustering models), Θ is a random probability measure [Ferguson, 1974]; in
this case, the probability constraint precludes Θ from being a completely random measure, but it is often chosen to be a normalized completely random
measure [James et al., 2009, Lijoi and Prünster, 2010]. The choice of Dirichlet process (a normalized gamma process) for Θ is particularly popular due
to a number of useful properties that coincide in this single choice [Doksum,
1974, Escobar, 1994, Escobar and West, 1995, 1998, Ferguson, 1973, Lo, 1984,
MacEachern, 1994, Perman et al., 1992, Pitman, 1996a,b, Sethuraman, 1994,
West and Escobar, 1994]. In partition models, Xn is a draw from the probability distribution described by Θ. If we think of such Xn as a random measure,
it is a.s. a single unit mass at a point ψ with strictly positive probability in Θ.
One potential connection between these two types of models is provided
by combinatorial clustering [Broderick et al., 2015]. In partition models, we
might suppose that we have a number of data sets, all of which we would like
to partition. For instance, in a document modeling scenario, each document
might be a data set; in particular each data point is a word in the document.
And we might wish to partition the words in each document. An alternative
perspective is to suppose that there is a single data set, where each data point
is a document. Then the document exhibits traits with multiplicities, where the
multiplicities might be the number of words from each trait; typically a trait
in this application would be a topic. In this case, there are a number of other
names besides feature or trait model that may be applied to the overarching
model—such as admixture model or mixed membership model [Airoldi et al.,
2014].

2.3

Bayesian nonparametrics

So far we have described a prior and likelihood that may be used to form a
Bayesian model. We have already stated above that forming a Bayesian nonparametric model imposes some restrictions on the prior and likelihood. We
formalize these restrictions in Assumptions A0, A1, and A2 below.
Recall that the premise of Bayesian nonparametrics is that the number of
traits represented in a collection of data can grow with the number of data
points. More explicitly, we achieve the desideratum that the number of traits is
unbounded, and may always grow as new data points are collected, by modeling
a countable infinity of traits. This assumption requires that the prior have
a countable infinity of atoms. These must either be fixed-location atoms or
ordinary component atoms. Fixed-location atoms represent known traits in
some sense since we must know the fixed locations of the atoms in advance.
Conversely, ordinary component atoms represent unknown traits, as yet to be
discovered, since both their locations and associated rates are unknown a priori.
Since we cannot know (or represent) a countable infinity of traits a priori, we
cannot start with a countable infinity of fixed-location atoms.

8

A0. The number of fixed-location atoms in Θ is finite.
Since we require a countable infinity of traits in total and they cannot come
from the fixed-location atoms by Assumption A0, the ordinary component must
contain a countable infinity of atoms. This assumption will be true if and only
if the rate measure on the trait frequencies has infinite mass.
A1. ν(R+ ) = ∞.
Finally, an implicit part of the starting premise is that each data point be
allocated to only a finite number of traits; we do not expect to glean an infinite
amount of information from finitely represented data. Thus, we require that
the number of atoms in every Xn be finite. By Assumption A0, the number
of atoms in Xn that correspond to fixed-location atoms in Θ is finite. But by
Assumption A1, the number of atoms in Θ from the ordinary component is
infinite. So there must be some restriction on the distribution of values of X at
the atoms of Θ (that is, some restriction on H in Eq. (8)) such that only finitely
many of these values are nonzero.
In particular, note that if H(dx|θ) does not contain an atom at zero for any
θ, then a.s. every one of the countable infinity of atoms of X will be nonzero.
Conversely, it follows that, for our desiderata to hold, we must have that H(dx|θ)
exhibits an atom at zero. One consequence of this observation is that H(dx|θ)
cannot be purely continuous for all θ. Though this line of reasoning does not
necessarily preclude a mixed continuous and discrete H, we henceforth assume
that H(dx|θ) is discrete, with support Z∗ = {0, 1, 2, . . .}, for all θ.
In what follows, we write h(x|θ) for the probability mass function of x given
θ. So our requirement that each data point be allocated to only a finite number of traits translates into a requirement that the number of atoms of Xn
with values in Z+ = {1, 2, . . .} be finite. Note that, by construction, the pairs
ord
{(θord,k , xord,k )}K
k=1 form a marked Poisson point process with rate measure
µmark (dθ × dx) := ν(dθ)h(x|θ). And the pairs with xord,k equal to any particular value x ∈ Z+ further form a thinned Poisson point process with rate
measure νx (dθ) := ν(dθ)h(x|θ). In particular, the number of atoms of X with
weight x is Poisson-distributed with mean νx (R+ ). So the number of atoms of
X is finite if and only if the following assumption holds.2
P∞
A2.
x=1 νx (R+ ) < ∞ for νx := ν(dθ)h(x|θ).

Thus Assumptions A0, A1, and A2 capture our Bayesian nonparametric
desiderata. We illustrate the development so far with an example.

Example 2.1. The beta process [Hjort, 1990] provides an example distribution
for Θ. In its most general form, sometimes called the three-parameter beta
2 When we have the more general case of a mixed continuous and discrete H, Assumption A2
becomes
R
R
A2b. x>0 θ∈R ν(dθ)H(dx|θ) < ∞.
+

9

process [Teh and Görür, 2009, Broderick et al., 2012], the beta process has an
ordinary component whose weight rate measure has a beta distribution kernel,
ν(dθ) = γθ−α−1 (1 − θ)c+α−1 dθ,

(9)

with support on (0, 1]. Here, the three fixed hyperparameters are γ, the mass
parameter ; c, the concentration parameter ; and α, the discount parameter.3
Moreover, each of its Kf ix fixed-location atoms, θk δψk , has a beta-distributed
weight [Broderick et al., 2015]:
θf ix,k ∼ Beta(θ|ρf ix,k , σf ix,k ),

(10)

where ρf ix,k , σf ix,k > 0 are fixed hyperparameters of the model.
By Assumption A0, Kf ix is finite. By Assumption A1, ν(R+ ) = ∞. To
achieve this infinite-mass restriction, the beta kernel in Eq. (9) must be improper; i.e., either −α ≤ 0 or c + α ≤ 0. Also, note that we must have γ > 0
since ν is a measure (and the case γ = 0 would be trivial).
Often the beta process is used as a prior paired with a Bernoulli process
likelihood [Thibaux
and Jordan, 2007]. The Bernoulli process specifies that,
P∞
given Θ = k=1 θk δψk , we draw
indep

xn,k ∼ Bern(x|θk ),

which is well-defined since every atom weight θk of Θ is in (0, 1] by the beta
process construction. Thus,
Xn =

∞
X

xn,k δψk .

k=1

The marginal distribution of the X1:N in this case is often called an Indian buffet process [Griffiths and Ghahramani, 2006, Thibaux and Jordan, 2007]. The
locations of atoms in Xn are thought of as the dishes sampled by the nth customer.
We take a moment to highlight the fact that continuous distributions for
H(dx|θ) are precluded based on the Bayesian nonparametric desiderata by considering an alternative likelihood. Consider instead if H(dx|θ) were continuous
here. Then X1 would have atoms at every atom of Θ. In the Indian buffet process analogy, any customer would sample an infinite number of dishes,
which contradicts our assumption that our data are finite. Indeed, any customer would sample all of the dishes at once. It is quite often the case in
practical applications, though, that the Xn are merely latent variables, with
the observed variables chosen according to a (potentially continuous) distribution given Xn [Griffiths and Ghahramani, 2006, Thibaux and Jordan, 2007];
3 In [Teh and Görür, 2009, Broderick et al., 2012], the ordinary component features the
beta distribution kernel in Eq. (9) multiplied not only by γ but also by a more complex, positive, real-valued expression in c and α. Since all of γ, c, and α are fixed hyperparameters, and
γ is an arbitrary positive real value, any other constant factors containing the hyperparameters
can be absorbed into γ, as in the main text here.

10

consider, e.g., mixture and admixture models. These cases are not precluded
by our development.
Finally, then, we may apply Assumption A2, which specifies that the number
of atoms in each observation Xn is finite; in this case, the assumption means
Z
∞ Z
X
ν(dθ) · h(1|θ)
ν(dθ) · h(x|θ) =
x=1

θ∈(0,1]

θ∈R+

since θ is supported on (0, 1] and x is supported on {0, 1}
Z
Z
θ1−α−1 (1 − θ)c+α−1 dθ < ∞.
γθ−α−1 (1 − θ)c+α−1 dθ · θ = γ
=
θ∈(0,1]

θ∈(0,1]

The integral here is finite if and only if 1 − α and c + α are the parameters of a
proper beta distribution: i.e., if and only if α < 1 and c > −α. Together with
the restrictions above, these restrictions imply the following allowable parameter
ranges for the beta process fixed hyperparameters:
γ > 0,

α ∈ [0, 1),

c > −α,

ρf ix,k , σf ix,k > 0 for all k ∈ [Kf ix ].

(11)

These correspond to the hyperparameter ranges previously found in [Teh and Görür,
2009, Broderick et al., 2012].


3

Posteriors

In Section 2, we defined a full Bayesian model consisting of a CRM prior for Θ
and a CRM likelihood for an observation X conditional on Θ. Now we would
like to calculate the posterior distribution of Θ|X.
Theorem 3.1 (Bayesian nonparametric posteriors). Let Θ be a completely random measure that satisfies Assumptions A0 and A1; that is, Θ is a CRM with
Kf ix fixed atoms such that Kf ix < ∞ and such that the kth atom can be written
θf ix,k δψf ix,k with
indep

θf ix,k ∼ Ff ix,k (dθ)
for proper distribution Ff ix,k and deterministic ψf ix,k . Let the ordinary component of Θ have rate measure
µ(dθ × dψ) = ν(dθ) · G(dψ),
P∞
where G is a proper distribution and ν(R+ ) = ∞. Write ΘP=
k=1 θk δψk ,
∞
and let X be generated conditional on Θ according to X = k=1 xk δψk with
indep

xk ∼ h(x|θk ) for proper, discrete probability mass function h. And suppose
X and Θ jointly satisfy Assumption A2 so that
∞ Z
X
ν(dθ)h(x|θ) < ∞.
x=1

θ∈R+

Then let Θpost be a random measure with the distribution of Θ|X. Θpost is
a completely random measure with three parts.
11

1. For each k ∈ [Kf ix ], Θpost has a fixed-location atom at ψf ix,k with weight
θpost,f ix,k distributed according to the finite-dimensional posterior Fpost,f ix,k (dθ)
that comes from prior Ff ix,k , likelihood h, and observation X({ψf ix,k }).
2. Let {xnew,k δψnew,k : k ∈ [Knew ]} be the atoms of X that are not at fixed
locations in the prior of Θ. Knew is finite by Assumption A2. Then Θpost
has a fixed-location atom at xnew,k with random weight θpost,new,k , whose
distribution Fpost,new,k (dθ) is proportional to
ν(dθ)h(xnew,k |θ).
3. The ordinary component of Θpost has rate measure
νpost (dθ) := ν(dθ)h(0|θ).
Proof. To prove the theorem, we consider in turn each of the two parts of
the prior: the fixed-location component and the ordinary component. First,
consider any fixed-location atom, θf ix,k δψf ix,k , in the prior. All of the other
fixed-location atoms in the prior, as well as the prior ordinary component, are
independent of the random weight θf ix,k . So it follows that all of X except
xf ix,k := X({ψf ix,k }) is independent of θf ix,k . Thus the posterior has a fixed
atom located at ψf ix,k whose weight, which we denote θpost,f ix,k , has distribution
Fpost,f ix,k (dθ) ∝ Ff ix,k (dθ)h(xf ix,k |θ),
which follows from the usual finite Bayes Theorem.
Next, consider the ordinary component in the prior. Let
Ψf ix = {ψf ix,1 , . . . , ψf ix,Kf ix }
be the set of fixed-location atoms in the prior. Recall that Ψf ix is deterministic,
and since G is continuous, all of the fixed-location atoms and ordinary component atoms of Θ are at a.s. distinct locations. So the measure Xf ix defined
by
Xf ix (A) := X(A ∩ Ψf ix )
can be derived purely from X, without knowledge of Θ. It follows that the
measure Xord defined by
Xord (A) := X(A ∩ (Ψ\Ψf ix ))
can be derived purely from X without knowledge of Θ. Xord is the same as
the observed data measure X but with atoms only at atoms of the ordinary
component of Θ and not at the fixed-location atoms of Θ.
Now for any value x ∈ Z+ , let
{ψnew,x,1 , . . . , ψnew,x,Knew,x }
be all of the locations of atoms of size x in Xord. By Assumption A2, the number
of such atoms, Knew,x , is finite. Further let θnew,x,k := Θ({ψnew,x,k }). Then
12

K

new,x
are generated from a thinned Poisson point process
the values {θnew,x,k }k=1
with rate measure
νx (dθ) := ν(dθ)h(x|θ).
(12)

And since νx (R+ ) < ∞ by assumption, each θnew,x,k has distribution equal to
the normalized rate measure in Eq. (12). Note that θnew,x,k δψnew,x,k is a fixedlocation atom in the posterior now that its location is known from the observed
Xord .
By contrast, if a likelihood draw at an ordinary component atom in the prior
returns a zero, that atom is not observed in Xord . Such atom weights in Θpost
thus form a marked Poisson point process with rate measure
ν(dθ)h(0|θ),
as was to be shown.
In Theorem 3.1, we consider generating Θ and then a single data point
X conditional on Θ. Now suppose we generate Θ and then N data points,
X1 , . . . , XN , iid conditional on Θ. In this case, Theorem 3.1 may be iterated to
find the posterior Θ|X1:N . In particular, Theorem 3.1 gives the ordinary component and fixed atoms of the random measure Θ1 := Θ|X1 . Then, using Θ1 as
the prior measure and X2 as the data point, another application of Theorem 3.1
gives Θ2 := Θ|X1:2 . We continue recursively using Θ|X1:n for n between 1 and
N − 1 as the prior measure until we find Θ|X1:N . The result is made explicit in
the following corollary.
Corollary 3.2 (Bayesian nonparametric posteriors given multiple data points).
Let Θ be a completely random measure that satisfies Assumptions A0 and A1;
that is, Θ is a CRM with Kf ix fixed atoms such that Kf ix < ∞ and such that
the kth atom can be written θf ix,k δψf ix,k with
indep

θf ix,k ∼ Ff ix,k (dθ)
for proper distribution Ff ix,k and deterministic ψf ix,k . Let the ordinary component of Θ have rate measure
µ(dθ × dψ) = ν(dθ) · G(dψ),
P∞
where G is a proper distribution and ν(R+ ) = ∞. Write Θ = P
k=1 θk δψk , and
∞
let X1 , . . . , Xn be generated conditional on Θ according to X = k=1 xn,k δψn,k
indep

with xn,k ∼ h(x|θk ) for proper, discrete probability mass function h. And
suppose X1 and Θ jointly satisfy Assumption A2 so that
∞ Z
X
ν(dθ)h(x|θ) < ∞.
x=1

θ∈R+

It is enough to make the assumption for X1 since the Xn are iid conditional on
Θ.
Then let Θpost be a random measure with the distribution of Θ|X1:N . Θpost
is a completely random measure with three parts.
13

1. For each k ∈ [Kf ix ], Θpost has a fixed-location atom at ψf ix,k with weight
θpost,f ix,k distributed according to the finite-dimensional posterior Fpost,f ix,k (dθ)
that comes from prior Ff ix,k , likelihood h, and observation X({ψf ix,k }).
2. Let {ψnew,k : k ∈ [Knew ]} be the union of atom locations across X1 , X2 , . . . , XN
minus the fixed locations in the prior of Θ. Knew is finite. Let xnew,n,k
be the weight of the atom in Xn located at ψnew,k . Note that at least one
of xnew,n,k across n must be non-zero, but in general xnew,n,k may equal
zero. Then Θpost has a fixed-location atom at xnew,k with random weight
θpost,new,k , whose distribution Fpost,new,k (dθ) is proportional to
ν(dθ)

N
Y

h(xnew,n,k |θ).

n=1

3. The ordinary component of Θpost has rate measure
n

νpost,n (dθ) := ν(dθ) [h(0|θ)] .
Proof. Corollary 3.2 follows from recursive application of Theorem 3.1. In order
to recursively apply Theorem 3.1, we need to verify that Assumptions A0, A1,
and A2 hold for the posterior Θ|X1:(n+1) when they hold for the prior Θ|X1:n .
Note that the number of fixed atoms in the posterior is the number of fixed atoms
in the prior plus the number of new atoms in the posterior. By Theorem 3.1,
these counts are both finite as long as Θ|X1:n satisfies Assumptions A0 and A2,
which both hold for n = 0 by assumption and n > 0 by the recursive assumption.
So Assumption A0 holds for Θ|X1:(n+1) .
Next we notice that since Assumption A1 implies that there is an infinite
number of ordinary component atoms in Θ|X1:n and only finitely many become
fixed atoms in the posterior by Assumption A2, it must be that Θ|X1:(n+1)
has infinitely many ordinary component atoms. So Assumption A1 holds for
Θ|X1:(n+1) .
Finally, we note that
∞ Z
X

x=1

=

νpost,n (dθ)h(x|θ)

θ∈R+

∞ Z
X

x=1

n

ν(dθ) [h(0|θ)] h(x|θ) ≤

θ∈R+

∞ Z
X

x=1

ν(dθ)h(x|θ) < ∞,

θ∈R+

where the penultimate inequality follows since h(0|θ) ∈ [0, 1] and where the
inequality follows by Assumption A2 on the original Θ (conditioned on no data).
So Assumption A2 holds for Θ|X1:(n+1) .
We now illustrate the results of the theorem with an example.
Example 3.3. Suppose we again start with a beta process prior for Θ as
in Example 2.1. This time we consider a negative binomial process likelihood
14

[Zhou et al., 2012, Broderick
et al., 2015]. The negative
binomial process specP∞
P∞
ifies that, given Θ = k=1 θk δψk , we draw X = k=1 xk δψk with
indep

xk ∼ NegBin(x|r, θk ),

for some fixed hyperparameter r > 0. So
Xn =

∞
X

xn,k δψk .

k=1

In this case, Assumption A2 translates into the following restriction.
∞ Z
X

x=1

=

Z

ν(dθ) · h(x|θ)

θ∈R+

ν(dθ) · [1 − h(0|θ)] =

Z

γθ−α−1 (1 − θ)c+α−1 dθ · [1 − (1 − θ)r ] < ∞,

θ∈(0,1]

θ∈R+

where the penultimate equality follows since the support of ν(dθ) is (0, 1].
By a Taylor expansion, we have 1 − (1 − θ)r = rθ + o(θ) as θ → 0, so we
require
Z
θ1−α−1 (1 − θ)c+α−1 dθ < ∞,

θ∈(0,1]

which is satisfied if and only if 1 − α and c + α are the parameters of a proper
beta distribution. Thus, we have the same parameter restrictions as in Eq. (11).
Now we calculate the posterior given the beta process prior on Θ and the
negative binomial process likelihood for X conditional on Θ. In particular,
the posterior has the distribution of Θpost , a CRM with three parts given by
Theorem 3.1.
First, at each fixed atom ψf ix,k of the prior with weight θf ix,k given by
Eq. (10), there is a fixed atom in the posterior with weight θpost,f ix,k . Let
xpost,f ix,k := X({ψf ix,k }). Then θpost,f ix,k has distribution
Fpost,f ix,k (dθ) ∝ Ff ix (dθ) · h(xpost,f ix,k |θ)
= Beta(θ|ρf ix,k , σf ix,k ) dθ · NegBin(xpost,f ix,k |r, θ)
∝ θρf ix,k −1 (1 − θ)σf ix,k −1 dθ · θxpost,f ix,k (1 − θ)r

(13)

∝ Beta (θ |ρf ix,k + xpost,f ix,k , σf ix,k + r ) dθ.
Second, for any atom xnew,k δψnew,k in X that is not at a fixed location in the
prior, Θpost has a fixed atom at ψnew,k whose weight θpost,new,k has distribution
Fpost,new,k (dθ) ∝ ν(dθ) · h(xnew,k |θ)
= ν(dθ) · NegBin(xnew,k |r, θ)
∝ θ−α−1 (1 − θ)c+α−1 dθ · θxnew,k (1 − θ)r
∝ Beta (θ |−α + xnew,k , c + α + r ) dθ,
15

(14)

which is a proper distribution since we have the following restrictions on its
parameters. For one, by assumption, xnew,k ≥ 1. And further, by Eq. (11), we
have α ∈ [0, 1) as well as c + α > 0 and r > 0.
Third, the ordinary component of Θpost has rate measure
ν(dθ)h(0|θ) = γθ−α−1 (1 − θ)c+α−1 dθ · (1 − θ)r = γθ−α−1 (1 − θ)c+r+α−1 dθ.
Not only have we found the posterior distribution Θpost above, but now
we can note that the posterior is in the same form as the prior with updated
ordinary component hyperparameters:
γpost = γ,

αpost = α,

cpost = c + r.

The posterior also has old and new beta-distributed fixed atoms with beta distribution hyperparameters given in Eq. (13) and Eq. (14), respectively. Thus,
we have proven that the beta process is, in fact, conjugate to the negative binomial process. An alternative proof was first given by Broderick et al. [2015].

As in Example 3.3, we can use Theorem 3.1 not only to calculate posteriors but also, once those posteriors are calculated, to check for conjugacy. This
approach unifies existing disparate approaches to Bayesian nonparametric conjugacy. However, it still requires the practitioner to guess the right conjugate
prior for a given likelihood. In the next section, we define a notion of exponential families for CRMs, and we show how to automatically construct a conjugate
prior for any exponential family likelihood.

4

Exponential families

Exponential families are what typically make conjugacy so powerful in the finite
case. For one, when a finite likelihood belongs to an exponential family, then existing results give an automatic conjugate, exponential family prior for that likelihood. In this section, we review finite exponential families, define exponential
CRMs, and show that analogous automatic conjugacy results can be obtained
for exponential CRMs. Our development of exponential CRMs will also allow
particularly straightforward results for size-biased representations (Corollary 5.2
in Section 5) and marginal processes (Corollary 6.2 in Section 6).
In the finite-dimensional case, suppose we have some (random) parameter θ
and some (random) observation x whose distribution is conditioned on θ. We
say the distribution Hexp,like of x conditional on θ is in an exponential family if
Hexp,like (dx|θ) = hexp,like (x|θ) dx = κ(x) exp {hη(θ), φ(x)i − A(θ)} µ(dx),
(15)
where η(θ) is the natural parameter, φ(x) is the sufficient statistic, κ(x) is the
base density, and A(θ) is the log partition function. We denote the density

16

of Hexp,like here, which exists by definition, by hexp,like . The measure µ—
with respect to which the density hexp,like exists—is typically Lebesgue measure
when Hexp,like is diffuse or counting measure when Hexp,like is atomic. A(θ) is
determined by the condition that Hexp,like (dx|θ) have unit total mass on its
support.
It is a classic result [Diaconis and Ylvisaker, 1979] that the following distribution for θ ∈ RD constitutes a conjugate prior:
Fexp,prior (dθ) = fexp,prior (θ) dθ = exp {hξ, η(θ)i + λ [−A(θ)] − B(ξ, λ)} dθ.
(16)
Fexp,prior is another exponential family distribution, now with natural parameter (ξ ′ , λ)′ , sufficient statistic (η(θ)′ , −A(θ))′ , and log partition function B(ξ, λ).
Note that the logarithms of the densities in both Eq. (15) and Eq. (16) are linear
in η(θ) and −A(θ). So, by Bayes Theorem, the posterior Fexp,post also has these
quantities as sufficient statistics in θ, and we can see Fexp,post must have the
following form.
Fexp,post (dθ|x) = fexp,post (θ|x) dθ
= exp {hξ + φ(x), η(θ)i + (λ + 1) [−A(θ)] − B(ξ + φ(x), λ + 1)} dθ.

(17)

Thus we see that Fexp,post belongs to the same exponential family as Fexp,prior
in Eq. (16), and hence Fexp,prior is a conjugate prior for Hexp,like in Eq. (15).

4.1

Exponential families for completely random measures

In the finite-dimensional case, we saw that for any exponential family likelihood,
as in Eq. (15), we can always construct a conjugate exponential family prior,
given by Eq. (16).
In order to prove a similar result for CRMs, we start by defining a notion of
exponential families for CRMs.
Definition 4.1. We say that a CRM Θ is an exponential CRM if it has the
following two parts. First, let Θ have Kf ix fixed-location atoms, where Kf ix
may be finite or infinite. The kth fixed-location atom is located at any ψf ix,k ,
unique from the other fixed locations, and has random weight θf ix,k , whose
distribution has density ff ix,k :
ff ix,k (θ) = κ(θ) exp {hη(ζk ), φ(θ)i − A(ζk )} ,
for some base density κ, natural parameter function η, sufficient statistic φ,
and log partition function A shared across atoms. Here, ζk is an atom-specific
parameter.
Second, let Θ have an ordinary component with rate measure µ(dθ × dψ) =
ν(dθ) · G(dψ) for some proper distribution G and weight rate measure ν of the
form
ν(dθ) = γ exp {hη(ζ), φ(θ)i} .
In particular, η and φ are shared with the fixed-location atoms, and fixed hyperparameters γ and ζ are unique to the ordinary component.
17

4.2

Automatic conjugacy for completely random measures

With Definition 4.1 in hand, we can specify an automatic Bayesian nonparametric conjugate prior for an exponential CRM likelihood.
P∞
Theorem 4.2 (Automatic conjugacy). Let Θ =
k=1 θk δψk , in accordance
with Assumption A1. Let X be generated conditional on Θ according to an
exponential CRM with fixed-location atoms at {ψk }∞
k=1 and no ordinary component. In particular, the distribution of the weight xk at ψk of X has the
following density conditional on the weight θk at ψk of Θ:
h(x|θk ) = κ(x) exp {hη(θk ), φ(x)i − A(θk )} .
Then a conjugate prior for Θ is the following exponential CRM distribution.
First, let Θ have Kprior,f ix fixed-location atoms, in accordance with Assumption A0. The kth such atom has random weight θf ix,k with proper density
fprior,f ix,k (θ) = exp {hξf ix,k , η(θ)i + λf ix,k [−A(θ)] − B(ξf ix,k , λf ix,k )} ,
where (η ′ , −A)′ here is the sufficient statistic and B is the log partition function.
ξf ix,k and λf ix,k are fixed hyperparameters for this atom weight.
Second, let Θ have ordinary component characterized by any proper distribution G and weight rate measure
ν(dθ) = γ exp {hξ, η(θ)i + λ [−A(θ)]} ,
where γ, ξ, and λ are fixed hyperparameters of the weight rate measure chosen
to satisfy Assumptions A1 and A2.
Proof. To prove the conjugacy of the prior for Θ with the likelihood for X, we
calculate the posterior distribution of Θ|X using Theorem 3.1. Let Θpost be
a CRM with the distribution of Θ|X. Then, by Theorem 3.1, Θpost has the
following three parts.
First, at any fixed location ψf ix,k in the prior, let xf ix,k be the value of X
at that location. Then Θpost has a fixed-location atom at ψf ix,k , and its weight
θpost,f ix,k has distribution
Fpost,f ix,k (dθ) ∝ fprior,f ix,k (θ) dθ · h(xf ix,k |θ)
∝ exp {hξf ix,k , η(θ)i + λf ix,k [−A(θ)]} dθ · exp {hη(θ), φ(xf ix,k )i − A(θ)} dθ
= exp {hξf ix,k + φ(xf ix,k ), η(θ)i + (λf ix,k + 1) [−A(θ)]} dθ.
It follows, from putting in the normalizing constant, that the distribution of
θpost,f ix,k has density
fpost,f ix,k (θ) = exp {hξf ix,k + φ(xf ix,k ), η(θ)i + (λf ix,k + 1) [−A(θ)]
− B(ξf ix,k + φ(xf ix,k ), λf ix,k + 1)} .
Second, for any atom xnew,k δψnew,k in X that is not at a fixed location in the
prior, Θpost has a fixed atom at ψnew,k whose weight θpost,new,k has distribution
Fpost,new,k (θ) ∝ ν(dθ) · h(xnew,k |θ)
18

∝ exp {hξ, η(θ)i + λ [−A(θ)]} · exp {hη(θ), φ(xnew,k )i − A(θ)} dθ
= exp {hξ + φ(xnew,k ), η(θ)i + (λ + 1) [−A(θ)]} dθ
and hence density
fpost,new,k (θ) = exp {hξ + φ(xnew,k ), η(θ)i + (λ + 1) [−A(θ)] − B(ξ + φ(xnew,k ), λ + 1)} .
Third, the ordinary component of Θpost has weight rate measure
ν(dθ) · h(0|θ)
= γ exp {hξ, η(θ)i + λ [−A(θ)]} · κ(0) exp {hη(θ), φ(0)i − A(θ)}
= γκ(0) · exp {hξ + φ(0), η(θ)i + (λ + 1) [−A(θ)]} .
Thus, the posterior rate measure is in the same exponential CRM form as
the prior rate measure with updated hyperparameters:
γpost = γκ(0),

ξpost = ξ + φ(0),

λpost = λ + 1.

Since we see that the posterior fixed-location atoms are likewise in the same
exponential CRM form as the prior, we have shown that conjugacy holds, as
desired.
We next use Theorem 4.2 to give proofs of conjugacy in cases where conjugacy has not previously been established in the Bayesian nonparametrics literature.
4
Example 4.3. Let X be generated
P∞according to a Poisson likelihood
P∞process
conditional on Θ. That is, X = k=1 xk δψk conditional on Θ = k=1 θk δψk
has an exponential CRM distribution with only a fixed-location component.
The weight xk at location ψk has support on Z∗ and has a Poisson density with
parameter θk ∈ R+ :

h(x|θk ) =

1 x −θk
1
θk e
=
exp {x log(θk ) − θk } .
x!
x!

(18)

The final line is rewritten to emphasize the exponential family form of this
density, with
κ(x) =

1
,
x!

φ(x) = x,

η(θ) = log(θ),

A(θ) = θ.

By Theorem 4.2, this Poisson likelihood process has a Bayesian nonparametric
conjugate prior for Θ with two parts.
First, Θ has a set of Kprior,f ix fixed-location atoms, where Kprior,f ix < ∞
by Assumption A0. The kth such atom has random weight θf ix,k with density
fprior,f ix,k (θ) = exp {hξf ix,k , η(θ)i + λf ix,k [−A(θ)] − B(ξf ix,k , λf ix,k )}
4 We use the term “Poisson likelihood process” to distinguish this specific Bayesian nonparametric likelihood from the Poisson point process.

19

= θξf ix,k e−λf ix,k θ exp {−B(ξf ix,k , λf ix,k )}
= Gamma(θ |ξf ix,k + 1, λf ix,k ),

(19)

where Gamma(θ|a, b) denotes the gamma density with shape parameter a > 0
and rate parameter b > 0. So we must have fixed hyperparameters ξf ix,k > −1
and λf ix,k > 0. Further,
ξ

ix,k
exp {−B(ξf ix,k , λf ix,k )} = λffix,k

+1

/Γ(ξf ix,k + 1)

to ensure normalization.
Second, Θ has an ordinary component characterized by any proper distribution G and weight rate measure
ν(dθ) = γ exp {hξ, η(θ)i + λ [−A(θ)]} dθ = γθξ e−λθ dθ.

(20)

Note that Theorem 4.2 guarantees that the weight rate measure will have the
same distributional kernel in θ as the fixed-location atoms.
Finally, we need to choose the allowable hyperparameter ranges for γ, ξ, and
λ. First, γ > 0 to ensure ν is a measure. By Assumption A1, we must have
ν(R+ ) = ∞, so ν must represent an improper gamma distribution. As such, we
require either ξ + 1 ≤ 0 or λ ≤ 0. By Assumption A2, we must have
Z
Z
∞ Z
X


γθξ e−λθ dθ · 1 − e−θ < ∞.
ν(dθ) · [1 − h(0|θ)] =
ν(dθ) · h(x|θ) =

x=1

θ∈R+

θ∈R+

θ∈R+

To ensure the integral over [1, ∞) is finite, we must have λ > 0. To ensure the
integral over (0, 1) is finite, we note that 1 − e−θ = θ + o(θ) as θ → 0. So we
require
Z
γθξ+1 e−λθ dθ < ∞,

θ∈(0,1)

which is satisfied if and only if ξ + 2 > 0.
Finally, then the hyperparameter restrictions can be summarized as:
γ > 0,

ξ ∈ (−2, −1],

λ > 0;

ξf ix,k > −1 and λf ix,k > 0

for all k ∈ [Kprior,f ix ].

The ordinary component of the conjugate prior for Θ discovered in this
example is typically called a gamma process. Here, we have for the first time
specified the distribution of the fixed-location atoms of the gamma process and,
also for the first time, proved that the gamma process is conjugate to the Poisson
likelihood process. We highlight this result as a corollary to Theorem 4.2.
Corollary 4.4. Let the Poisson likelihood process be a CRM with fixed-location
atom weight distributions as in Eq. (18). Let the gamma process be a CRM with
fixed-location atom weight distributions as in Eq. (19) and ordinary component
weight measure as in Eq. (20). Then the gamma process is a conjugate Bayesian
nonparametric prior for the Poisson likelihood process.

20

Example 4.5. Next, let X be generated according to a new process we call
an odds Bernoulli process. We have previously seen a typical Bernoulli process
likelihood in Example 2.1. In the odds Bernoulli process, we say that X, conditional on Θ, has an exponential CRM distribution. In this case, the weight of
the kth atom, xk , conditional on θk has support on {0, 1} and has a Bernoulli
density with odds parameter θk ∈ R+ :
h(x|θk ) = θkx (1 + θk )−1
= exp {x log(θk ) − log(1 + θk )} .

(21)

That is, if ρ is the probability of a successful Bernoulli draw, then θ = ρ/(1 − ρ)
represents the odds ratio of the probability of success over the probability of
failure.
The final line of Eq. (21) is written to emphasize the exponential family form
of this density, with
κ(x) = 1,

φ(x) = x,

η(θ) = log(θ),

A(θ) = log(1 + θ).

By Theorem 4.2, the likelihood for X has a Bayesian nonparametric conjugate
prior for Θ. This conjugate prior has two parts.
First, Θ has a set of Kprior,f ix fixed-location atoms. The kth such atom has
random weight θf ix,k with density
fprior,f ix,k (θ) = exp {hξf ix,k , η(θ)i + λf ix,k [−A(θ)] − B(ξf ix,k , λf ix,k )}
= θξf ix,k (1 + θ)−λf ix,k exp {−B(ξf ix,k , λf ix,k )}
= BetaPrime (θ |ξf ix,k + 1, λf ix,k − ξf ix,k − 1 ) ,

(22)

where BetaPrime(θ|a, b) denotes the beta prime density with shape parameters
a > 0 and b > 0. Further,
exp {−B(ξf ix,k , λf ix,k )} =

Γ(λf ix,k )
Γ(ξf ix,k + 1)Γ(λf ix,k − ξf ix,k − 1)

to ensure normalization.
Second, Θ has an ordinary component characterized by any proper distribution G and weight rate measure
ν(dθ) = γ exp {hξ, η(θ)i + λ [−A(θ)]} dθ = γθξ (1 + θ)−λ dθ.

(23)

We need to choose the allowable hyperparameter ranges for γ, ξ, and λ.
First, γ > 0 to ensure ν is a measure. By Assumption A1, we must have
ν(R+ ) = ∞, so ν must represent an improper beta prime distribution. As such,
we require either ξ + 1 ≤ 0 or λ − ξ − 1 ≤ 0. By Assumption A2, we must have
∞ Z
X

x=1

θ∈R+

ν(dθ) · h(x|θ) =

Z

ν(dθ) · h(1|θ)

θ∈R+

since the support of x is {0, 1}
21

=

Z

γθξ (1 + θ)−λ dθ · θ1 (1 + θ)−1 = γ

θ∈R+

Z

θξ+1 (1 + θ)−λ−1 dθ < ∞.
θ∈R+

Since the integrand is the kernel of a beta prime distribution, we simply require
that this distribution be proper; i.e., ξ + 2 > 0 and λ − ξ − 1 > 0.
The hyperparameter restrictions can be summarized as:
γ > 0, ξ ∈ (−2, −1], λ > ξ + 1; ξf ix,k > −1 and λf ix,k > ξf ix,k + 1 for all k ∈ [Kprior,f ix ].
We call the distribution for Θ described in this example the beta prime process. Its ordinary component has previously been defined by Broderick et al.
[2015]. But this result represents the first time the beta prime process is described in full, including parameter restrictions and fixed-location atoms, as well
as the first proof of its conjugacy with the odds Bernoulli process. We highlight
the latter result as a corollary to Theorem 4.2 below.
Corollary 4.6. Let the odds Bernoulli process be a CRM with fixed-location
atom weight distributions as in Eq. (21). Let the beta process be a CRM with
fixed-location atom weight distributions as in Eq. (22) and ordinary component
weight measure as in Eq. (23). Then the beta process is a conjugate Bayesian
nonparametric prior for the odds Bernoulli process.


5

Size-biased representations

We have shown in Section 4.2 that our exponential CRM (Definition 4.1) is
useful in that we can find an automatic Bayesian nonparametric conjugate prior
given an exponential CRM likelihood. We will see in this section and the next
that exponential CRMs allow us to build representations that allow tractable
inference despite the infinite-dimensional nature of the models we are using.
The best-known size-biased representation of a random measure in Bayesian
nonparametrics is the stick-breaking representation of the Dirichlet process ΘDP
[Sethuraman, 1994]:
ΘDP =

∞
X

θDP,k δψk ;

k=1

For k ∈ Z∗ , θDP,k = βk

k−1
Y

(24)
(1 − βj ),

iid

βk ∼ Beta(1, c),

iid

ψk ∼ G,

j=1

where c is a fixed hyperparameter satisfying c > 0.
The name “stick-breaking” originates from thinking of the unit interval as
a stick of length one. At each round k, only some of the stick remains; βk
describes the proportion of the remaining stick that is broken off in round k,
and θDP,k describes the total amount of remaining stick that is broken off in

22

round k. By construction, not only is each θDP,k ∈ (0, 1) but in fact the θDP,k
add to one (the total stick length) and thus describe a distribution.
Eq. (24) is called a size-biased representation for the following reason. Since
the weights {θDP,k }∞
k=1 describe a distribution, we can make draws from this
distribution; each such draw is sometimes thought of as a multinomial draw with
a single trial. In that vein, typically we imagine that our data points Xmult,n are
described as iid draws conditioned on ΘDP , where Xmult,n is a random measure
with just a single atom:
Xmult,n = δψmult,n ;

ψmult,n = ψk with probability θDP,k .

(25)

Then the limiting proportion of data points Xmult,n with an atom at ψmult,1
(the first atom location chosen) is θDP,1 . The limiting proportion of data points
with an atom at the next unique atom location chosen will have size θDP,2 , and
so on [Broderick et al., 2013].
The representation in Eq. (24) is so useful because there is a familiar, finitedimensional distribution for each of the atom weights θDP,k of the random measure ΘDP . This representation allows approximate inference via truncation
[Ishwaran and James, 2001] or exact inference via slice sampling [Walker, 2007,
Kalli et al., 2011].
Since the weights {θDP,k }∞
k=1 are constrained to sum to one, the Dirichlet
process is not a CRM.5 Indeed, there has been much work on size-biased representations for more general normalized random measures, which include the
Dirichlet process as just one example [Perman et al., 1992, Pitman, 1996a,b,
2003].
By contrast, we here wish to explore size-biasing for non-normalized CRMs.
In the normalized CRM case, we considered which atom of a random discrete
probability measure was drawn first and what is the distribution of that atom’s
size. In the non-normalized CRM case considered in the present work, when
drawing X conditional on Θ, there may be multiple atoms (or one atom or no
atoms) of Θ that correspond to non-zero atoms in X. The number will always
be finite though by Assumption A2. In this non-normalized CRM case, we
wish to consider the sizes of all such atoms in Θ. Size-biased representations
have been developed in the past for particular CRM examples, notably the beta
process [Paisley et al., 2010, Broderick et al., 2012]. And even though there is
typically no interpretation of these representations in terms of a single stick
representing a unit probability mass, they are sometimes referred to as stickbreaking representations as a nod to the popularity of Dirichlet process stickbreaking.
In the beta process case, such size-biased representations have already been
shown to allow approximate inference via truncation [Doshi et al., 2009, Paisley et al.,
2011] or exact inference via slice sampling [Teh et al., 2007, Broderick et al.,
2015]. Here we provide general recipes for the creation of these representations and illustrate our recipes by discovering previously unknown size-biased
5 In fact, the Dirichlet process is a normalized gamma process (cf. Example 4.3) [Ferguson,
1973].

23

representations.
We have seen that a general CRM Θ takes the form of an a.s. discrete random
measure:
∞
X
θk δ ψ k .
(26)
k=1

The fixed-location atoms are straightforward to simulate; there are finitely many
by Assumption A0, their locations are fixed, and their weights are assumed to
come from finite-dimensional distributions. The infinite-dimensionality of the
Bayesian nonparametric CRM comes from the ordinary component (cf. Section 2.3 and Assumption A1). So far the only description we have of the ordinary component is its generation from the countable infinity of points in a
Poisson point process. The next result constructively demonstrates that we
can represent the distributions of the CRM weights {θk }∞
k=1 in Eq. (26) as a
sequence of finite-dimensional distributions, much as in the familiar Dirichlet
process case.

Theorem 5.1 (Size-biased representations). Let Θ be a completely random
measure that satisfies Assumptions A0 and A1; that is, Θ is a CRM with Kf ix
fixed atoms such that Kf ix < ∞ and such that the kth atom can be written
θf ix,k δψf ix,k . The ordinary component of Θ has rate measure
µ(dθ × dψ) = ν(dθ) · G(dψ),
P∞
where G is a proper distribution and ν(R+ ) = ∞. Write P
Θ =
k=1 θk δψk ,
∞
and let Xn be generated iid given Θ according to Xn =
k=1 xn,k δψk with
indep

xn,k ∼ h(x|θk ) for proper, discrete probability mass function h. And suppose
Xn and Θ jointly satisfy Assumption A2 so that
∞ Z
X

x=1

ν(dθ)h(x|θ) < ∞.

θ∈R+

Then we can write
Θ=

m,x
∞ X
∞ ρX
X

θm,x,j δψm,x,j

m=1 x=1 j=1
iid

ψm,x,k ∼ G iid across m, x, j
 Z

indep
ρm,x ∼ Poisson ρ
ν(dθ)h(0|θ)m−1 h(x|θ) across m, x

(27)

θ

indep

θm,x,j ∼ Fsize,m,x (dθ) ∝ ν(dθ)h(0|θ)m−1 h(x|θ)
iid across j and independently across m, x.
Proof. By construction, Θ is an a.s. discrete random measure with a countable
infinity of atoms. Without loss of generality, suppose that for every (non-zero)
value of an atom weight θ, there is a non-zero probability of generating an atom
24

with non-zero weight x in the likelihood. Now suppose we generate X1 , X2 , . . ..
Then, for every atom θδψ of Θ, there exists some finite n with an atom at ψ.
Therefore, we can enumerate all of the atoms of Θ by enumerating
• Each atom θδψ such that there is an atom in X1 at ψ.
• Each atom θδψ such that there is an atom in X2 at ψ but there is not an
atom in X1 at ψ.
..
.
• Each atom θδψ such that there is an atom in Xm at ψ but there is not an
atom in any of X1 , X2 , . . . , Xm−1 at ψ.
..
.
Moreover, on the mth round of this enumeration, we can further break down
the enumeration by the value of the observation Xm at the atom location:
• Each atom θδψ such that there is an atom in Xm of weight 1 at ψ but
there is not an atom in any of X1 , X2 , . . . , Xm−1 at ψ.
• Each atom θδψ such that there is an atom in Xm of weight 2 at ψ but
there is not an atom in any of X1 , X2 , . . . , Xm−1 at ψ.
..
.
• Each atom θδψ such that there is an atom in Xm of weight x at ψ but
there is not an atom in any of X1 , X2 , . . . , Xm−1 at ψ.
..
.
Recall that the values θk that form the weights of Θ are generated according
to a Poisson point process with rate measure ν(dθ). So, on the first round, the
values of θk such that x1,k = x also holds are generated according to a thinned
Poisson point process with rate measure
ν(dθ)h(x|θ).
In particular, since the rate measure has finite total mass by Assumption A2,
we can define
Z
M1,x := ν(dθ)h(x|θ),
θ

which will be finite. Then the number of atoms θk for which x1,k = x is
ρ1,x ∼ Poisson(ρ|M1,x ).
And each such θk has weight with distribution
Fsize,1,x (dθ) ∝ ν(dθ)h(x|θ).

25

Finally, note from Theorem 3.1 that the posterior Θ|X1 has weight rate measure
ν1 (dθ) := ν(dθ)h(0|θ).
Now take any m > 1. Suppose, inductively, that the ordinary component of
the posterior Θ|X1 , . . . , Xm−1 has weight rate measure
νm−1 (dθ) := ν(dθ)h(0|θ)m−1 .
The atoms in this ordinary component have been selected precisely because they
have not appeared in any of X1 , . . . , Xm−1 . As for m = 1, we have that the
atoms θk in this ordinary component with corresponding weight in Xm equal to
x are formed by a thinned Poisson point process, with rate measure
νm−1 (dθ)h(x|θ) = ν(dθ)h(0|θ)m−1 h(x|θ).
Since the rate measure has finite total mass by Assumption A2, we can define
Z
Mm,x := ν(dθ)h(0|θ)m−1 h(x|θ),
θ

which will be finite. Then the number of atoms θk for which x1,k = x is
ρm,x ∼ Poisson(ρ|Mm,x ).
And each such θk has weight
Fsize,m,x ∝ ν(dθ)h(0|θ)m−1 h(x|θ).
Finally, note from Theorem 3.1 that the posterior Θ|X1:m , which can be thought
of as generated by prior Θ|X1:(m−1) and likelihood Xm |Θ, has weight rate measure
ν(dθ)h(0|θ)m−1 h(0|θ) = νm (dθ),
confirming the inductive hypothesis.
Recall that every atom of Θ is found in exactly one of these rounds and that
x ∈ Z+ . Also recall that the atom locations may be generated independently
and identically across atoms, and independently from all the weights, according
to proper distribution G (Section 2.2). To summarize, we have then
Θ=

m,x
∞ X
∞ ρX
X

θm,x,j δψm,x,j ,

m=1 x=1 j=1

where
iid

ψm,x,k ∼ G iid across m, x, j
Z
Mm,x = ν(dθ)h(0|θ)m−1 h(x|θ) across m, x
θ

26

indep

ρm,x ∼ Poisson(ρ|Mm,x ) across m, x
Fsize,m,x (dθ) ∝ ν(dθ)h(0|θ)m−1 h(x|θ) across m, x
indep

θm,x,j ∼ Fsize,m,x (dθ) iid across j and independently across m, x,
as was to be shown.
The following corollary gives a more detailed recipe for the calculations in
Theorem 5.1 when the prior is in a conjugate exponential CRM to the likelihood.
Corollary 5.2 (Exponential CRM size-biased representations). Let Θ be an
exponential CRM with no fixed-location atoms (thereby trivially satisfying Assumption A0) such that Assumption A1 holds.
Let X be generated conditional on Θ according to an exponential CRM with
fixed-location atoms at {ψk }∞
k=1 and no ordinary component. Let the distribution
of the weight xn,k at ψk have probability mass function
h(x|θk ) = κ(x) exp {hη(θk ), φ(x)i − A(θk )} .
Suppose that Θ and X jointly satisfy Assumption A2. And let Θ be conjugate
to X as in Theorem 4.2. Then we can write
Θ=

m,x
∞ X
∞ ρX
X

θm,x,j δψm,x,j

m=1 x=1 j=1
iid

ψm,x,j ∼ G

iid across m, x, j

Mm,x = γ · κ(0)m−1 κ(x) · exp {B(ξ + (m − 1)φ(0) + φ(x), λ + m)}
indep

ρm,x ∼ Poisson (ρ|Mm,x )
independently across m, x

(28)

indep

θm,x,j ∼ fsize,m,x (θ) dθ
= exp {hξ + (m − 1)φ(0) + φ(x), η(θ)i + (λ + m)[−A(θ)]
− B(ξ + (m − 1)φ(0) + φ(x), λ + m)}
iid across j and independently across m, x.
Proof. The corollary follows from Theorem 5.1 by plugging in the particular
forms for ν(dθ) and h(x|θ).
In particular,
Z
ν(dθ)h(0|θ)m−1 h(x|θ)
Mm,x =
θ∈R+
Z
γ exp {hξ, η(θ)i + λ [−A(θ)]}
=
θ∈R+

m−1

· [κ(0) exp {hη(θ), φ(0)i − A(θ)}]
· κ(x) exp {hη(θ), φ(x)i − A(θ)} dθ
27

= γκ(0)m−1 κ(x) exp {B (ξ + (m − 1)φ(0) + φ(x), λ + m)} ,

Corollary 5.2 can be used to find the known size-biased representation of
the beta process [Thibaux and Jordan, 2007]; we demonstrate this derivation in
detail in Example B.1 in Appendix B. Here we use Corollary 5.2 to discover a
new size-biased representation of the gamma process.
Example 5.3. Let Θ be a gamma process, and let Xn be iid Poisson likelihood
processes conditioned on Θ for each n as in Example 4.3. That is, we have
ν(dθ) = γθξ e−λθ dθ.
And
h(x|θk ) =

1 x −θk
θ e
x! k

with
γ > 0,

ξ ∈ (−2, −1],

ξf ix,k > −1 and λf ix,k > 0

λ > 0;

for all k ∈ [Kprior,f ix ]

by Example 4.3.
We can pick out the following components of h:
κ(x) =

1
,
x!

φ(x) = x,

η(θ) = log(θ),

A(θ) = θ.

Thus, by Corollary 5.2, we have
fsize,m,x (θ) ∝ θξ+x e−(λ+m)θ ∝ Gamma (θ |ξ + x + 1, λ + m ) .
We summarize the representation that follows from Corollary 5.2 in the following
result.
Corollary 5.4. Let the gamma process be a CRM Θ with fixed-location atom
weight distributions as in Eq. (19) and ordinary component weight measure as
in Eq. (20). Then we may write
Θ=

m,x
∞ X
∞ ρX
X

θm,x,j δψm,x,j

m=1 x=1 j=1
iid

ψm,x,j ∼ G
Mm,x = γ ·

iid across m, x, j
1
· Γ(ξ + x + 1) · (λ + m)−(ξ+x+1) across m, x
x!

indep

ρm,x ∼ Poisson (ρ|Mm,x) across m, x
indep

θm,x,j ∼ Gamma (θ |ξ + x + 1, λ + m )
iid across j and independently across m, x.

28

6

Marginal processes

In Section 5, although we conceptually made use of the observations {X1 , X2 , . . .},
we focused on a representation of the prior Θ: cf. Eqs. (27) and (28). In this
section, we provide a representation of the marginal of X1:N , with Θ integrated
out.
The canonical example of a marginal process again comes from the Dirichlet
process (DP). In this case, the full model consists of the DP-distributed prior
on ΘDP (as in Eq. (24)) together with the likelihood for Xmult,n conditional on
ΘDP (iid across n) described by Eq. (25). Then the marginal distribution of
Xmult,1:N is described by the Chinese restaurant process. This marginal takes
the following form.
For each n = 1, 2, . . . , N ,
K

n−1
be the union of atom locations in Xmult,1 , . . . , Xmult,n−1 .
1. Let {ψk }k=1
Then Xmult,n |Xmult,1 , . . . , Xmult,n−1 has a single atom at ψ, where

PKn−1
ψk
with probability ∝
k=1 Xmult,m ({ψk })
ψ=
ψnew with probability ∝ c

ψnew ∼ G

In the case of CRMs, the canonical example of a marginal process is the Indian buffet process [Griffiths and Ghahramani, 2006]. Both the Chinese restaurant process and Indian buffet process have proven popular for inference since
the underlying infinite-dimensional prior is integrated out in these processes
and only the finite-dimensional marginal remains. By Assumption A2, we know
that the marginal will generally be finite-dimensional for our CRM Bayesian
models. And thus we have the following general marginal representations for
such models.
Theorem 6.1 (Marginal representations). Let Θ be a completely random measure that satisfies Assumptions A0 and A1; that is, Θ is a CRM with Kf ix
fixed atoms such that Kf ix < ∞ and such that the kth atom can be written
θf ix,k δψf ix,k . The ordinary component of Θ has rate measure
µ(dθ × dψ) = ν(dθ) · G(dψ),
P∞
where G is a proper distribution and ν(R+ ) = ∞. Write P
Θ =
k=1 θk δψk ,
∞
and let Xn be generated iid given Θ according to Xn =
x
δψk with
n,k
k=1
indep

xn,k ∼ h(x|θk ) for proper, discrete probability mass function h. And suppose
Xn and Θ jointly satisfy Assumption A2 so that
∞ Z
X
ν(dθ)h(x|θ) < ∞.
x=1

θ∈R+

Then the marginal distribution of X1:N is the same as that provided by the
following construction.
For each n = 1, 2, . . . , N ,
29

K

n−1
be the union of atom locations in X1 , . . . , Xn−1 . Let xm,k :=
1. Let {ψk }k=1
Xm ({ψk }). Let xn,k denote the weight of Xn |X1 , . . . , Xn−1 at ψk . Then
xn,k has distribution described by the following probability mass function:
R
Qn−1

m=1 h(xm,k |θ)
θ∈R+ ν(dθ)h(x|θ)
.
hcond xn,k = x x1:(n−1),k =
R
Qn−1
m=1 h(xm,k |θ)
θ∈R+ ν(dθ)

2. For each x = 1, 2, . . .

ρ

n,x
,
• Xn has ρn,x new atoms. That is, Xn has atoms at locations {ψn,x,j }j=1
where
ρn,x
Kn−1
= ∅ a.s.
∩ {ψk }k=1
{ψn,x,j }j=1

Moreover,
 Z

indep
ρn,x ∼ Poisson ρ
ν(dθ)h(0|θ)n−1 h(x|θ) across n, x
θ

iid

ψn,x,j ∼ G(dψ) across n, x, j.
Proof. We saw in the proof of Theorem 5.1 that the marginal for X1 can be
expressed as follows. For each x ∈ Z+ , there are ρ1,x atoms of X1 with weight
x, where
Z

indep
ν(dθ)h(x|θ) across x.
ρ1,x ∼ Poisson
These atoms have locations

θ
ρ1,x
{ψ1,x,j }j=1 ,

where

iid

ψ1,x,j ∼ G(dψ) across x, j.
P∞
1
For the upcoming induction, let K1 := x=1 ρ1,x . And let {ψk }K
k=1 be the (a.s.
ρ1,x
disjoint by assumption) union of the sets {ψ1,x,j }j=1 across x. Note that K1 is
finite by Assumption A2.
We will also find it useful in the upcoming induction to let Θpost,1 have the
distribution of Θ|X1 . Let θpost,1,x,j = Θpost,1 ({ψ1,x,j }). By Theorem 3.1 or the
proof of Theorem 5.1, we have that
indep

θpost,1,x,j ∼ Fpost,1,x,j (dθ) ∝ ν(dθ)h(x|θ)
independently across x and iid across j.
K

n−1
is the union
Now take any n > 1. Inductively, we assume {ψn−1,k }k=1
of all the atom locations of X1 , . . . , Xn−1 . Further assume Kn−1 is finite. Let
Θpost,n−1 have the distribution of Θ|X1 , . . . , Xn−1 . Let θn−1,k be the weight of
Θpost,n−1 at ψn−1,k . And, for any m ∈ [n − 1], let xm,k be the weight of Xm at
ψn−1,k . We inductively assume that

indep

θn−1,k ∼ Fn−1,k (dθ) ∝ ν(dθ)

n−1
Y

m=1

independently across k.
30

h(xm,k |θ)

(29)

Now let ψn,k equal ψn−1,k for k ∈ [Kn−1 ]. Let xn,k denote the weight of Xn
at ψn,k for k ∈ [Kn−1 ]. Conditional on the atom weight of Θ at ψn,k , the atom
weights of X1 , . . . , Xn−1 , Xn are independent. Since the atom weights of Θ are
independent as well, we have that xn,k |X1 , . . . , Xn−1 has the same distribution
as xn,k |x1,k , . . . , xn−1,k . We can write the probability mass function of this
distribution as follows.
hcond (xn,k = x |x1,k , . . . , xn−1,k )
Z
Fn−1,k (dθ)h(x|θ)
=
θ∈R+

=

R

θ∈R+

R

h

ν(dθ)

θ∈R+

Qn−1

m=1

ν(dθ)

i
h(xm,k |θ) · h(x|θ)

Qn−1

m=1

h(xm,k |θ)

,

where the last line follows from Eq. (29).
We next show the inductive hypothesis in Eq. (29) holds for n and k ∈
[Kn−1 ]. Let xn,k denote the weight of Xn at ψn,k for k ∈ [Kn−1 ]. Let Fn,k (dθ)
denote the distribution of xn,k and note that
Fn,k (dθ) ∝ Fn−1,k (dθ) · h(xn,k |θ)
n
Y
= ν(dθ)
h(xm,k |θ),
m=1

which agrees with Eq. (29) for n when we assume the result for n − 1.
The previous development covers atoms that are present in at least one of
X1 , . . . , Xn−1 . Next we consider new atoms in Xn ; that is, we consider atoms in
Xn for which there are no atoms at the same location in any of X1 , . . . , Xn−1 .
We saw in the proof of Theorem 5.1 that, for each x ∈ Z+ , there are ρn,x
new atoms of Xn with weight x such that
 Z

indep
ρn,x ∼ Poisson ρ
ν(dθ)h(0|θ)n−1 h(x|θ) across x.
θ

ρ

n,x
with
These new atoms have locations {ψn,x,j }j=1

iid

ψn,x,j ∼ G(dψ) across x, j.
By Assumption A2,

P∞

x=1

ρn,x < ∞. So
Kn := Kn−1 +

∞
X

ρn,x

x=1

remains finite. Let ψn,k for k ∈ {Kn−1 + 1, . . . , Kn } index these new locations.
Let θn,k be the weight of Θpost,n at ψn,k for k ∈ {Kn−1 + 1, . . . , Kn }. And let
xn,k be the value of X at ψn,k .
31

We check that the inductive hypothesis holds. By repeated application of
Theorem 3.1, the ordinary component of Θ|X1 , . . . , Xn−1 has rate measure
ν(dθ)h(0|θ)n−1 .
So, again by Theorem 3.1, we have that
indep

θn,k ∼ Fn.k (dθ) ∝ ν(dθ)h(0|θ)n−1 h(xn,k |θ).
Since Xm has value 0 at ψn,k for m ∈ {1, . . . , n − 1} by construction, we have
that the inductive hypothesis holds.
As in the case of size-biased representations (Section 5 and Corollary 5.2),
we can find a more detailed recipe when the prior is in a conjugate exponential
CRM to the likelihood.
Corollary 6.2 (Exponential CRM marginal representations). Let Θ be an exponential CRM with no fixed-location atoms (thereby trivially satisfying Assumption A0) such that Assumption A1 holds.
Let X be generated conditional on Θ according to an exponential CRM with
fixed-location atoms at {ψk }∞
k=1 and no ordinary component. Let the distribution
of the weight xn,k at ψk have probability mass function
h(x|θk ) = κ(x) exp {hη(θk ), φ(x)i − A(θk )} .
Suppose that Θ and X jointly satisfy Assumption A2. And let Θ be conjugate
to X as in Theorem 4.2. Then the marginal distribution of X1:N is the same as
that provided by the following construction.
For each n = 1, 2, . . . , N ,
K

n−1
be the union of atom locations in X1 , . . . , Xn−1 . Let xm,k :=
1. Let {ψk }k=1
Xm ({ψk }). Let xn,k denote the weight of Xn |X1 , . . . , Xn−1 at ψk . Then
xn,k has distribution described by the following probability mass function:

hcond xn,k = x x1:(n−1),k
(
)
n−1
n−1
X
X
= κ(x) exp −B(ξ +
xm , λ + n − 1) + B(ξ +
xm + x, λ + n) .

m=1

m=1

2. For each x = 1, 2, . . .
ρ

n,x
,
• Xn has ρn,x new atoms. That is, Xn has atoms at locations {ψn,x,j }j=1
where
ρn,x
Kn−1
= ∅ a.s.
∩ {ψk }k=1
{ψn,x,j }j=1

Moreover,
Mn,x := γ · κ(0)n−1 κ(x) · exp {B(ξ + (n − 1)φ(0) + φ(x), λ + n)}
across n, x
indep

ρn,x ∼ Poisson (ρ |Mn,x ) across n, x
iid

ψn,x,j ∼ G(dψ) across n, x, j.
32

Proof. The corollary follows from Theorem 6.1 by plugging in the forms for
ν(dθ) and h(x|θ).
In particular,
Z

ν(dθ)

θ∈R+

=

Z

n
Y

h(xm,k |θ)

m=1

γ exp {hξ, η(θ)i + λ [−A(θ)]} ·
θ∈R+

=γ

"

n
Y

"

n
Y

κ(xm,k ) exp {hη(θ), φ(xm,k )i − A(θ)}

m=1

#

κ(xm,k ) B

m=1

ξ+

n
X

#

!

φ(xm,k ), λ + n .

m=1

So

hcond xn,k = x x1:(n−1),k
R
Qn−1
ν(dθ)h(x|θ) m=1
h(xm,k |θ)
θ∈R+
=
R
Qn−1
m=1 h(xm,k |θ)
θ∈R+ ν(dθ)
(
)
n−1
n−1
X
X
= κ(x) exp −B(ξ +
xm , λ + n − 1) + B(ξ +
xm + x, λ + n) .
m=1

m=1

In Example C.1 in Appendix C we show that Corollary 6.2 can be used to
recover the Indian buffet process marginal from a beta process prior together
with a Bernoulli process likelihood. In the following example, we discover a new
marginal for the Poisson likelihood process with gamma process prior.
Example 6.3. Let Θ be a gamma process, and let Xn be iid Poisson likelihood
processes conditioned on Θ for each n as in Example 4.3. That is, we have
ν(dθ) = γθξ e−λθ dθ

and

h(x|θk ) =

1 x −θk
θ e
x! k

with
γ > 0,

ξ ∈ (−2, −1],

λ > 0;

ξf ix,k > −1 and λf ix,k > 0

for all k ∈ [Kprior,f ix ]

by Example 4.3.
We can pick out the following components of h:
κ(x) =
And we calculate
Z
exp {B(ξ, λ)} =

1
,
x!

φ(x) = x,

η(θ) = log(θ),

exp {hξ, η(θ)i + λ[−A(θ)]} dθ =

A(θ) = θ.

Z

θ∈R+

θ∈R+

33

θξ e−λθ = Γ(ξ + 1)λ−(ξ+1) .

So, for k ∈ Z∗ , we have
(

P(xn = x) = κ(x) exp −B(ξ +

n−1
X

xm , λ + n − 1) + B(ξ +

m=1
Pn−1

n−1
X

)

xm + x, λ + n)

m=1

Pn−1
1 (λ + n − 1)ξ+ m=1 xm +1 Γ(ξ + m=1 xm + x + 1)
·
·
Pn−1
Pn−1
x!
Γ(ξ + m=1 xm + 1)
(λ + n)ξ+ m=1 xm +x+1
Pn−1
ξ+Pnm=1 xm +1 
x

Γ(ξ + m=1 xm + x + 1)
1
λ+n−1
=
·
Pn−1
λ+n
λ+n
Γ(x + 1)Γ(ξ + m=1 xm + 1)
!
n−1
X
= NegBin x ξ +
xm + 1, (λ + n)−1 .
=

m=1

And
Mn,x := γ · κ(0)n−1 κ(x) · exp {B(ξ + (n − 1)φ(0) + φ(x), λ + n)}
1
=γ·
· Γ(ξ + x + 1)(λ + n)−(ξ+x+1) .
x!
We summarize the marginal distribution representation of X1:N that follows
from Corollary 6.2 in the following result.
Corollary 6.4. Let Θ be a gamma process with fixed-location atom weight distributions as in Eq. (19) and ordinary component weight measure as in Eq. (20).
Let Xn be drawn, iid across n, conditional on Θ according to a Poisson likelihood process with fixed-location atom weight distributions as in Eq. (18). Then
X1:N has the same distribution as the following construction.
For each n = 1, 2, . . . , N ,
K

n−1
be the union of atom locations in X1 , . . . , Xn−1 . Let xm,k :=
1. Let {ψk }k=1
Xm ({ψk }). Let xn,k denote the weight of Xn |X1 , . . . , Xn−1 at ψk . Then
xn,k has distribution described by the following probability mass function:
!
n−1
X

xm,k + 1, (λ + n)−1 .
hcond xn,k = x x1:(n−1),k = NegBin x ξ +

m=1

2. For each x = 1, 2, . . .
ρ

n,x
,
• Xn has ρn,x new atoms. That is, Xn has atoms at locations {ψn,x,j }j=1
where
ρn,x
Kn−1
= ∅ a.s.
∩ {ψk }k=1
{ψn,x,j }j=1

Moreover,
1 Γ(ξ + x + 1)
·
x! (λ + n)ξ+x+1
across n, x

Mn,x := γ ·

34

indep

ρn,x ∼ Poisson (ρ |Mn,x ) independently across n, x
iid

ψn,x,j ∼ G(dψ) independently across n, x and iid across j.


7

Discussion

In the preceding sections, we have shown how to calculate posteriors for general
CRM-based priors and likelihoods for Bayesian nonparametric models. We have
also shown how to represent Bayesian nonparametric priors as a sequence of finite draws, and full Bayesian nonparametric models via finite marginals. We
have introduced a notion of exponential families for CRMs, which we call exponential CRMs, that has allowed us to specify automatic Bayesian nonparametric
conjugate priors for exponential CRM likelihoods. And we have demonstrated
that our exponential CRMs allow particularly straightforward recipes for sizebiased and marginal representations of Bayesian nonparametric models. Along
the way, we have proved that the gamma process is a conjugate prior for the
Poisson likelihood process and the beta prime process is a conjugate prior for
the odds Bernoulli process. We have discovered a size-biased representation of
the gamma process and a marginal representation of the gamma process coupled
with a Poisson likelihood process.
All of this work has relied heavily on the description of Bayesian nonparametric models in terms of completely random measures. As such, we have worked
very particularly with pairings of real values—the CRM atom weights, which we
have interpreted as trait frequencies or rates—together with trait descriptors—
the CRM atom locations. However, all of our proofs broke into essentially two
parts: the fixed-location atom part and the ordinary component part. The fixedlocation atom development essentially translated into the usual finite version of
Bayes Theorem and could easily be extended to full Bayesian models where the
prior describes a random element that need not be real-valued. Moreover, the
ordinary component development relied entirely on its generation as a Poisson
point process over a product space. It seems reasonable to expect that our development might carry through when the first element in this tuple need not be
real-valued. And thus we believe our results are suggestive of broader results
over more general spaces.

Acknowledgements
Support for this project was provided by ONR under the Multidisciplinary University Research Initiative (MURI) program (N00014-11-1-0688). T. Broderick
was supported by a Berkeley Fellowship. A. C. Wilson was supported by an
NSF Graduate Research Fellowship.

35

A

Further automatic conjugate priors

We use Theorem 4.2 to calculate automatic conjugate priors for further exponential CRMs.
Example A.1. Let X be generated according to a Bernoulli process as in
Example 2.1. That is, X has an exponential CRM distribution with Klike,f ix
fixed-location atoms, where Klike,f ix < ∞ in accordance with Assumption A0:
Klike,f ix

X=

X

xlike,k δψlike,k .

k=1

The weight of the kth atom, xlike,k , has support on {0, 1} and has a Bernoulli
density with parameter θk ∈ (0, 1]:
h(x|θk ) = θkx (1 − θk )1−x
= exp {x log(θk /(1 − θk )) + log(1 − θk )} .
The final line is rewritten to emphasize the exponential family form of this
density, with
κ(x) = 1
φ(x) = x
η(θ) = log



θ
1−θ



A(θ) = − log(1 − θ).
Then, by Theorem 4.2, X has a Bayesian nonparametric conjugate prior for
Klike,f ix

Θ :=

X

θk δ ψ k .

k=1

This conjugate prior has two parts.
First, Θ has a set of Kprior,f ix fixed-location atoms at some subset of the
Klike,f ix fixed locations of X. The kth such atom has random weight θf ix,k
with density
fprior,f ix,k (θ) = exp {hξf ix,k , η(θ)i + λf ix,k [−A(θ)] − B(ξf ix,k , λf ix,k )}
= θξf ix,k (1 − θ)λf ix,k −ξf ix,k exp {−B(ξf ix,k , λf ix,k )}
= Beta (θ |ξf ix,k + 1, λf ix,k − ξf ix,k + 1 ) ,
where Beta(θ|a, b) denotes the beta density with shape parameters a > 0 and
b > 0. So we must have fixed hyperparameters ξf ix,k > −1 and λf ix,k >
ξf ix,k − 1. Further,
exp {−B(ξf ix,k , λf ix,k )} =

Γ(λf ix,k + 2)
Γ(ξf ix,k + 1)Γ(λf ix,k − ξf ix,k + 1)
36

to ensure normalization.
Second, Θ has an ordinary component characterized by any proper distribution G and weight rate measure
ν(dθ) = γ exp {hξ, η(θ)i + λ [−A(θ)]} dθ
= γθξ (1 − θ)λ−ξ dθ.
Finally, we need to choose the allowable hyperparameter ranges for γ, ξ, and
λ. γ > 0 ensures ν is a measure. By Assumption A1, we must have ν(R+ ) = ∞,
so ν must represent an improper beta distribution. As such, we require either
ξ + 1 ≤ 0 or λ − ξ ≤ 0. By Assumption A2, we must have
∞ Z
X
ν(dθ) · h(x|θ)
θ∈R+

x=1

=

Z

ν(dθ)h(1|θ)

θ∈(0,1]

since the support of x is {0, 1} and the support of θ is (0, 1]
Z
=γ
θξ (1 − θ)λ−ξ dθ · θ
θ∈(0,1]

<∞

Since the integrand is the kernel of a beta distribution, the integral is finite if
and only if ξ + 2 > 0 and λ − ξ + 1 > 0.
Finally, then the hyperparameter restrictions can be summarized as:
γ>0
ξ ∈ (−2, −1]
λ>ξ−1
ξf ix,k > −1 and λf ix,k > ξf ix,k − 1 for all k ∈ [Kprior,f ix ]
By setting α = ξ+1, c = λ+2, ρf ix,k = ξf ix,k +1, and σf ix,k = λf ix,k −ξf ix,k +1,
we recover the hyperparameters of Eq. (11) in Example 2.1. Here, by contrast to
Example 2.1, we found the conjugate prior and its hyperparameter settings given
just the Bernoulli process likelihood. Henceforth, we use the parameterization
of the beta process above.


B

Further size-biased representations

Example B.1. Let Θ be a beta process, and let Xn be iid Bernoulli processes
conditioned on Θ for each n as in Example A.1. That is, we have
ν(dθ) = γθξ (1 − θ)λ−ξ dθ.
And
h(x|θk ) = θkx (1 − θk )1−x
37

with
γ>0
ξ ∈ (−2, −1]
λ>ξ−1
ξf ix,k > −1 and λf ix,k > ξf ix,k − 1 for all k ∈ [Kprior,f ix ]
by Example A.1.
We can pick out the following components of h:
κ(x) = 1
φ(x) = x
η(θ) = log



θ
1−θ



A(θ) = − log(1 − θ).
Thus, by Corollary 5.2,
Θ=

m,x
∞ X
∞ ρX
X

θm,x,j δψm,x,j

m=1 x=1 j=1
iid

ψm,x,j ∼ G iid across m, x, j
indep

θm,x,j ∼ fsize,m,x (θ) dθ
∝ θξ+x (1 − θ)λ+m−ξ−x dθ
∝ Beta (θ |ξ + x, λ − ξ + m − x ) dθ
iid across j and independently across m, x
Mm,x := γ ·

Γ(ξ + x + 1)Γ(λ − ξ + m − x + 1)
Γ(λ + m + 2)

indep

ρm,x ∼ Poisson (Mm,x )
across m, x
Broderick et al. [2012] and Paisley et al. [2012] have previously noted that
this size-biased representation of the beta process arises from the Poisson point
process.


C

Further marginals

Example C.1. Let Θ be a beta process, and let Xn be iid Bernoulli processes
conditioned on Θ for each n as in Examples A.1 and B.1.
We calculate the main components of Corollary 6.2 for this pair of processes.
In particular, we have
(
)
n−1
n−1
X
X
P(xn = 1) = κ(k) exp −B(ξ +
xm , λ + n − 1) + B(ξ +
xm + 1, λ + n)
m=1

m=1

38

Γ(λ + n − 1 + 2)
Pn−1
x
+
1)Γ(λ + n − 1 − ξ − m=1 xm + 1)
m=1 m
Pn−1
Pn−1
Γ(ξ + m=1 xm + 1 + 1)Γ(λ + n − ξ − m=1 xm − 1 + 1)
·
Γ(λ + n + 2)
Pn−1
ξ + m=1 xm + 1
=
λ+n+1

=

Γ(ξ +

Pn−1

And
Mn,1 := γ · κ(0)n−1 κ(1) · exp {B(ξ + (n − 1)φ(0) + φ(1), λ + n)}
=γ·

Γ(ξ + 1 + 1)Γ(λ + n − ξ − 1 + 1)
Γ(λ + n + 2)

Thus, the marginal distribution of X1:N is the same as that provided by the
following construction.
For each n = 1, 2, . . . , N ,
1. At any location ψ for which there is some atom in X1 , . . . , Xn−1 , let xm be
the weight of Xm at ψ for m ∈ [n−1]. Then we have that Xn |X1 , . . . , Xn−1
has weight xn at ψ, where
!
Pn−1
ξ + m=1 xm + 1
P(dxn ) = Bern xn
λ+n+1
2. Xn has ρn,1 atoms at locations {ψn,1,j } with j ∈ [ρn,1 ] where there have
not yet been atoms in any of X1 , . . . , Xn−1 . Moreover,
Γ(ξ + 1 + 1)Γ(λ + n − ξ − 1 + 1)
Γ(λ + n + 2)
across n

Mn,1 := γ ·

indep

ρn,1 ∼ Poisson (Mn,1 ) across n, x
iid

ψn,1,j ∼ G(dψ) across n, j
Here, we have recovered the three-parameter extension of the Indian buffet
process [Teh and Görür, 2009, Broderick et al., 2013].


39

References
E. M. Airoldi, D. Blei, E. A. Erosheva, and S. E. Fienberg. Handbook of Mixed
Membership Models and Their Applications. CRC Press, 2014.
D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of
Machine Learning Research, 3:993–1022, 2003.
T. Broderick, M. I. Jordan, and J. Pitman. Beta processes, stick-breaking, and
power laws. Bayesian Analysis, 7(2):439–476, 2012.
T. Broderick, M. I. Jordan, and J. Pitman. Cluster and feature modeling from
combinatorial stochastic processes. Statistical Science, 2013.
T. Broderick, L. Mackey, J. Paisley, and M. I. Jordan. Combinatorial clustering
and the beta negative binomial process. IEEE TPAMI, 2015.
P. Damien, J. Wakefield, and S. Walker. Gibbs sampling for Bayesian nonconjugate and hierarchical models by using auxiliary variables. Journal of the
Royal Statistical Society: Series B, 61(2):331–344, 1999.
M. H. DeGroot. Optimal Statistical Decisions. John Wiley & Sons, Inc, 1970.
P. Diaconis and D. Ylvisaker. Conjugate priors for exponential families. The
Annals of Statistics, 7(2):269–281, 1979.
K. Doksum. Tailfree and neutral random probabilities and their posterior distributions. The Annals of Probability, pages 183–201, 1974.
F. Doshi, K. T. Miller, J. Van Gael, and Y. W. Teh. Variational inference for
the Indian buffet process. In AISTATS, 2009.
M. D. Escobar. Estimating normal means with a Dirichlet process prior. Journal
of the American Statistical Association, 89(425):268–277, 1994.
M. D. Escobar and M. West. Bayesian density estimation and inference using
mixtures. Journal of the American Statistical Association, 90(430):577–588,
1995.
M. D. Escobar and M. West. Computing nonparametric hierarchical models. In
Practical nonparametric and semiparametric Bayesian statistics, pages 1–22.
Springer, 1998.
T. S. Ferguson. A Bayesian analysis of some nonparametric problems. The
Annals of Statistics, pages 209–230, 1973.
T. S. Ferguson. Prior distributions on spaces of probability measures. The
Annals of Statistics, pages 615–629, 1974.
T. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian
buffet process. In NIPS, 2006.
40

N. L. Hjort. Nonparametric Bayes estimators based on beta processes in models
for life history data. The Annals of Statistics, pages 1259–1294, 1990.
H. Ishwaran and L. F. James. Gibbs sampling methods for stick-breaking priors.
Journal of the American Statistical Association, 96(453), 2001.
L. F. James. Poisson latent feature calculus for generalized Indian buffet processes. arXiv preprint arXiv:1411.2936, 2014.
L. F. James, A. Lijoi, and I. Prünster. Posterior analysis for normalized random
measures with independent increments. Scandinavian Journal of Statistics,
36(1):76–97, 2009.
M. Kalli, J. E. Griffin, and S. G. Walker. Slice sampling mixture models. Statistics and Computing, 21(1):93–105, 2011.
Y. Kim. Nonparametric Bayesian estimators for counting processes. Annals of
Statistics, pages 562–588, 1999.
J. F. C. Kingman. Completely random measures. Pacific Journal of Mathematics, 21(1):59–78, 1967.
J. F. C. Kingman. Poisson Processes, volume 3. Oxford University Press, 1992.
A. Lijoi and I. Prünster. Models beyond the Dirichlet process. In N. L. Hjort,
C. Holmes, P. Müller, and S. G. Walker, editors, Bayesian Nonparametrics.
Cambridge Series in Statistical and Probabilistic Mathematics, 2010.
A. Y. Lo. Bayesian nonparametric statistical inference for Poisson point processes. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 59
(1):55–66, 1982.
A. Y. Lo. On a class of Bayesian nonparametric estimates: I. Density estimates.
Annals of Statistics, 12(1):351–357, 1984.
S. N. MacEachern. Estimating normal means with a conjugate style Dirichlet
process prior. Communications in Statistics-Simulation and Computation, 23
(3):727–741, 1994.
R. M. Neal. Markov chain sampling methods for Dirichlet process mixture
models. Journal of Computational and Graphical Statistics, 9(2):249–265,
2000.
R. M. Neal. Slice sampling. Annals of Statistics, pages 705–741, 2003.
P. Orbanz. Conjugate projective limits. arXiv preprint arXiv:1012.0363, 2010.
J. W. Paisley, A. K. Zaas, C. W. Woods, G. S. Ginsburg, and L. Carin. A
stick-breaking construction of the beta process. In ICML, pages 847–854,
2010.

41

J. W. Paisley, L. Carin, and D. M. Blei. Variational inference for stick-breaking
beta process priors. In ICML, pages 889–896, 2011.
J. W. Paisley, D. M. Blei, and M. I. Jordan. Stick-breaking beta processes and
the Poisson process. In AISTATS, pages 850–858, 2012.
M. Perman, J. Pitman, and M. Yor. Size-biased sampling of poisson point
processes and excursions. Probability Theory and Related Fields, 92(1):21–39,
1992.
J. Pitman. Random discrete distributions invariant under size-biased permutation. Advances in Applied Probability, pages 525–539, 1996a.
J. Pitman. Some developments of the Blackwell-MacQueen urn scheme. Lecture
Notes-Monograph Series, pages 245–267, 1996b.
J. Pitman. Poisson-Kingman partitions. Lecture Notes-Monograph Series, pages
1–34, 2003.
J. Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica,
4:639–650, 1994.
Y. W. Teh and D. Görür. Indian buffet processes with power-law behavior. In
NIPS, pages 1838–1846, 2009.
Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet
processes. Journal of the American Statistical Association, 101(476), 2006.
Y. W. Teh, D. Görür, and Z. Ghahramani. Stick-breaking construction for the
Indian buffet process. In AISTATS, pages 556–563, 2007.
R. Thibaux and M. I. Jordan. Hierarchical beta processes and the Indian buffet
process. In AISTATS, pages 564–571, 2007.
R. J. Thibaux. Nonparametric Bayesian Models for Machine Learning. PhD
thesis, UC Berkeley, 2008.
M. K. Titsias. The infinite gamma-Poisson feature model. In NIPS, pages
1513–1520, 2008.
S. G. Walker. Sampling the Dirichlet mixture model with slices. Communications in Statistics—Simulation and Computation, 36(1):45–54, 2007.
C. Wang and D. M. Blei. Variational inference in nonconjugate models. The
Journal of Machine Learning Research, 14(1):1005–1031, 2013.
M. West and M. D. Escobar. Hierarchical priors and mixture models, with application in regression and density estimation. In P. R. Freeman and A. F. M.
Smith, editors, Aspects of Uncertainty: A Tribute to D. V. Lindley. Institute
of Statistics and Decision Sciences, Duke University, 1994.
M. Zhou, L. Hannah, D. Dunson, and L. Carin. Beta-negative binomial process
and Poisson factor analysis. AISTATS, 2012.
42

