On the frequentist validity of Bayesian limits
B. J. K. Kleijn∗
Korteweg-de Vries Institute for Mathematics, University of Amsterdam

arXiv:1611.08444v3 [] 27 Nov 2017

November 2017
Abstract
To the frequentist who computes posteriors, not all priors are useful asymptotically: in
this paper Schwartz’s 1965 Kullback-Leibler condition is generalised to enable frequentist
interpretation of convergence of posterior distributions with the complex models and often
dependent datasets in present-day statistical applications. We prove four simple and fully
general frequentist theorems, for posterior consistency; for posterior rates of convergence;
for consistency of the Bayes factor in hypothesis testing or model selection; and a theorem
to obtain confidence sets from credible sets. The latter has a significant methodological
consequence in frequentist uncertainty quantification: use of a suitable prior allows one to
convert credible sets of a calculated, simulated or approximated posterior into asymptotically consistent confidence sets, in full generality. This extends the main inferential implication of the Bernstein-von Mises theorem to non-parametric models without smoothness
conditions. Proofs require the existence of a Bayesian type of test sequence and priors
giving rise to local prior predictive distributions that satisfy a weakened form of Le Cam’s
contiguity with respect to the data distribution. Results are applied in a wide range 0f
examples and counterexamples.

1

Introduction

In this paper (following [5]:“Statisticians should readily use both Bayesian and frequentist
ideas.”) we examine for which priors Bayesian asymptotic conclusions extend to conclusions
valid in the frequentist sense: how Doob’s prior-almost-sure consistency is strengthened to
reach Schwartz’s frequentist conclusion that the posterior is consistent, or how a test that is
consistent prior-almost-surely becomes a test that is consistent in all points of the model, or
how a Bayesian credible set can serve as a frequentist confidence set asymptotically.
The central property to enable frequentist interpretation of posterior asymptotics is defined as
remote contiguity in section 3. It expresses a weakened form of Le Cam’s contiguity, relating
the true distribution of the data to localized prior predictive distributions. Where Schwartz’s
Kullback-Leibler neighbourhoods represent a choice for the localization appropriate when the
∗

email: B.Kleijn@uva.nl, web: https://staff.fnwi.uva.nl/b.j.k.kleijn

1

sample is i.i.d., remote contiguity generalises the notion to include non-i.i.d. samples, priors
that change with the sample size, weak consistency with the Dirichlet process prior, etcetera.
Although firstly aimed at enhancing insight into asymptotic relations by simplification and
generalisation, this paper also has a significant methodological consequence: theorem 4.12
demonstrates that if the prior is such that remote contiguity applies, credible sets can be
converted to asymptotically consistent confidence sets in full generality. So the asymptotic
validity of credible sets as confidence sets in smooth parametric models [53] extends much
further: in practice, the frequentist can simulate the posterior in any model, construct his
preferred type of credible sets and ‘enlarge’ them to obtain asymptotic confidence sets, provided
his prior induces remote contiguity. This extends the main inferential implication of the
Bernstein-von Mises theorem to non-parametric models.
In the remainder of this section we discuss posterior consistency. In section 2 we concentrate
on an inequality that relates testing to posterior concentration and indicates the relation with
Le Cam’s inequality. Section 3 introduces remote contiguity and the analogue of Le Cam’s
First Lemma. In section 4, frequentist theorems on the asymptotic behaviour of posterior
distributions are proved, on posterior consistency, on posterior rates of convergence, on consistent testing and model selection with Bayes factors and on the conversion of credible sets
to confidence sets. Section 5 formulates the conclusions.
Definitions, notation, conventions roughly follow those of [51] and are collected in appendix A
with some other preliminaries. All applications, illustrations, examples and counterexamples
have been collected in appendix B. Proofs are found in appendix C.

1.1

Posterior consistency and inconsistency

For a statistical procedure to be consistent, it must infer the truth with arbitrarily large
accuracy and probability, if we gather enough data. For example, when using sequential data
X n ∼ Pθ0 ,n to estimate the value θ0 , a consistent estimator sequence θn converges to θ0 in
Pθ0 ,n -probability. For a posterior Π(·|X n ) to be consistent, it must concentrate mass arbitrarily
close to one in any neighbourhood of θ0 as n → ∞ (see definition 4.1).
Consider a model P for i.i.d. data with single-observation distribution P0 . Give P a Polish
topology with Borel prior Π so that the posterior is well-defined (see definition A.3). The first
general consistency theorem for posteriors is due to Doob.
Theorem 1.1 (Doob (1949))
For all n ≥ 1, let (X1 , X2 , . . . , Xn ) ∈ X n be i .i .d . − P0 , where P0 lies in a model P. Suppose
X and P are Polish spaces. Assume that P 7→ P (A) is Borel measurable for every Borel set
A ⊂ X . Then for any Borel prior Π on P the posterior is consistent, for Π-almost-all P .
In parametric applications Doob’s Π-null-set of potential inconsistency can be considered small
2

(for example, when the prior dominates Lebesgue measure). But in non-parametric context
these null-sets can become very large (or not, see [54]): the first examples of unexpected
posterior inconsistency are due to Schwartz [61], but it was Freedman [29] who made the point
famous with a simple non-parametric counterexample (discussed in detail as example B.1). In
[30] it was even shown that inconsistency is generic in a topological sense: the set of pairs
(P0 , Π) for which the posterior is consistent is meagre: posteriors that only wander around,
placing and re-placing mass aimlessly, are the rule rather than the exception. (For a discussion,
see example B.2.)
These and subsequent examples of posterior inconsistency established a widespread conviction
that Bayesian methods were wholly unfit for frequentist purposes, at least in non-parametric
context. The only justifiable conclusion from Freedman’s meagreness, however, is that a
condition is missing: Doob’s assertion may be all that a Bayesian requires, a frequentist
demands strictly more, thus restricting the class of possible choices for his prior. Strangely,
a condition representing this restriction had already been found when Freedman’s meagreness
result was published.
Theorem 1.2 (Schwartz (1965))
For all n ≥ 1, let (X1 , X2 , . . . , Xn ) ∈ X n be i .i .d . − P0 , where P0 lies in a model P. Let U
denote an open neighbourhood of P0 in P. If,
(i) there exist measurable φn : X n → [0, 1], such that,
P0n φn = o(1),

sup Qn (1 − φn ) = o(1),

(1)

Q∈U c

(ii) and Π is a Kullback-Leibler prior, i.e. for all δ > 0,


dP
< δ > 0,
Π P ∈ P : −P0 log
dP0

(2)

P -a.s.

then Π(U |X n ) −−0−−→ 1.
Over the decades, examples of problematic posterior behaviour in non-parametric setting continued to captivate [20, 21, 17, 22, 23, 31, 32], while Schwartz’s theorem received initially limited but steadily growing amounts of attention: subsequent frequentist theorems (e.g. by Barron [3], Barron-Schervish-Wasserman [4], Ghosal-Ghosh-van der Vaart [34], Shen-Wasserman
[63], Walker [70] and Walker-Lijoi-Prünster [72], Kleijn-Zhao [46] and many others) have extended the applicability of theorem 1.2 but not its essence, condition (2) for the prior. The
following example illustrates that Schwartz’s condition cannot be the whole truth, though.
Example 1.3 Consider X1 , X2 , . . . that are i.i.d.-P0 with Lebesgue density p0 : R → R supported on an interval of known width (say, 1) but unknown location. Parametrize in terms
of a continuous density η on [0, 1] with η(x) > 0 for all x ∈ [0, 1] and a location θ ∈ R:
pθ,η (x) = η(x − θ) 1[θ,θ+1] (x). A moment’s thought makes clear that if θ 6= θ0 ,
pθ0 ,η0
−Pθ,η log
= ∞,
pθ,η
3

for all η, η 0 . Therefore Kullback-Leibler neighbourhoods do not have any extent in the θdirection and no prior is a Kullback-Leibler prior in this model. Nonetheless the posterior is
consistent (see examples B.14 and B.15).



Similar counterexamples exist [46] for the type of prior that is proposed in the analyses of
posterior rates of convergence in (Hellinger) metric setting [34, 63]. Although methods in
[46] avoid this type of problem, the essential nature of condition (2) in i.i.d. setting becomes
apparent there as well.
This raises the central question of this paper: is Schwartz’s Kullback-Leibler condition perhaps a manifestation of a more general notion? The argument leads to other questions for
which insightful answers have been elusive: why is Doob’s theorem completely different from
Schwartz’s? The accepted explanation views the lack of congruence as an indistinct symptom of differing philosophies, but is this justified? Why does weak consistency in the full
non-parametric model (e.g. with the Dirichlet process prior [28], or more modern variations
[19]) reside in a corner of its own (with tailfreeness [30] as sufficient property of the prior),
apparently unrelated to posterior consistency in either Doob’s or Schwartz’s views? Indeed,
what would Schwartz’s theorem look like without the assumption that the sample is i.i.d. (e.g.
with data that form a Markov chain or realize some other stochastic process) or with growing
parameter spaces and changing priors? And to extend the scope further, what can be said
about hypothesis testing, classification, model selection, etcetera? Given that the Bernsteinvon Mises theorem cannot be expected to hold in any generality outside parametric setting
[17, 32], what relationship exists between credible sets and confidence sets? This paper aims
to shed more light on these questions in a general sense, by providing a prior condition that
enables strengthening Bayesian asymptotic conclusions to frequentist ones, illustrated with a
variety of examples and counterexamples.

2

Posterior concentration and asymptotic tests

In this section, we consider a lemma that relates concentration of posterior mass in certain
model subsets to the existence of test sequences that distinguish between those subsets. More
precisely, it is shown that the expected posterior mass outside a model subset V with respect
to the local prior predictive distribution over a model subset B, is upper bounded (roughly)
by the testing power of any statistical test for the hypotheses B versus V : if a test sequence
exists, the posterior will concentrate its mass appropriately.

4

2.1

Bayesian test sequences

Since the work of Schwartz [62], test sequences and posterior convergence have been linked
intimately. Here we follow Schwartz and consider asymptotic testing; however, we define test
sequences immediately in Bayesian context by involving priors from the outset.
Definition 2.1 Given priors (Πn ), measurable model subsets (Bn ), (Vn ) ⊂ G and an ↓ 0, a
sequence of Bn -measurable maps φn : Xn → [0, 1] is called a Bayesian test sequence for Bn
versus Vn (under Πn ) of power an , if,
Z
Z
Pθ,n φn dΠn (θ) +

Pθ,n (1 − φn ) dΠn (θ) = o(an ).

(3)

Vn

Bn

We say that (φn ) is a Bayesian test sequence for Bn versus Vn (under Πn ) if (3) holds for
some an ↓ 0.
Note that if we have sequences (Cn ) and (Wn ) such that Cn ⊂ Bn and Wn ⊂ Vn for all n ≥ 1,
then a Bayesian test sequence for (Bn ) versus (Vn ) of power an is a Bayesian test sequence for
(Cn ) versus (Wn ) of power (at least) an .
Lemma 2.2 For any B, V ∈ G with Π(B) > 0 and any measurable φ : X → [0, 1],
Z
Z
Z
1
Pθ Π(V |X) dΠ(θ|B) ≤ Pθ φ dΠ(θ|B) +
Pθ (1 − φ) dΠ(θ).
Π(B) V

(4)

So the mere existence of a test sequence is enough to guarantee posterior concentration, a fact
expressed in n-dependent form through the following proposition.
Proposition 2.3 Assume that for given priors Πn , sequences (Bn ), (Vn ) ⊂ G and an , bn ↓ 0
such that an = o(bn ) with Πn (Bn ) ≥ bn > 0, there exists a Bayesian test sequence for Bn
versus Vn of power an . Then,
PnΠn |Bn Π(Vn |X n ) = o(an b−1
n ),

(5)

for all n ≥ 1.
To see how this leads to posterior consistency, consider the following: if the model subsets
Vn = V are all equal to the complement of a neighbourhood U of P0 , and the Bn are chosen
Π |Bn

such that the expectations of the random variables X n 7→ Π(V |X n ) under Pn n

‘dominate’

their expectations under P0,n in a suitable way, sufficiency of prior mass bn given testing power
an ↓ 0, is enough to assert that P0,n Π(V |X n ) → 0, so an arbitrarily large fraction of posterior
mass is found in U with high probability for n large enough.

2.2

Existence of Bayesian test sequences

Lemma 2.2 and proposition 2.3 require the existence of test sequences of the Bayesian type.
That question is unfamiliar, frequentists are used to test sequences for pointwise or uniform
5

testing. For example, an application of Hoeffding’s inequality demonstrates that, weak neighbourhoods are uniformly testable (see proposition A.6). Another well-known example concerns
testability of convex model subsets. Mostly the uniform test sequences in Schwartz’s theorem
are constructed using convex building blocks B and V separated in Hellinger distance (see
proposition B.7 and subsequent remarks).
Requiring the existence of a Bayesian test sequence c.f. (3) is quite different. We shall illustrate
this point in various ways below. First of all the existence of a Bayesian test sequence is linked
directly to behaviour of the posterior itself.
Theorem 2.4 Let (Θ, G , Π) be given. For any B, V ∈ G with Π(B) > 0, Π(V ) > 0, the
following are equivalent,
(i) there are Bn -measurable φn : Xn → [0, 1] such that for Π-almost-all θ ∈ B, θ0 ∈ V ,
Pθ,n φn → 0,

Pθ0 ,n (1 − φn ) → 0,

(ii) there are Bn -measurable φn : Xn → [0, 1] such that,
Z
Z
Pθ,n φn dΠ(θ) +
Pθ,n (1 − φn ) dΠ(θ) → 0,
B

V

(iii) for Π-almost-all θ ∈ B, θ0 ∈ V ,
Pθ,n

Π(V |X n ) −−−→ 0,

Pθ0 ,n

Π(B|X n ) −−−→ 0.

The interpretation of this theorem is gratifying to supporters of the likelihood principle and
pure Bayesians: distinctions between model subsets are Bayesian testable, if and only if, they
are picked up by the posterior asymptotically, if and only if, there exists a pointwise test for
B versus V that is Π-almost-surely consistent.
For a second, more frequentist way to illustrate how basic the existence of a Bayesian test
sequences is, consider a parameter space (Θ, d) which is a metric space with fixed Borel prior
Π and d-consistent estimators θ̂n : Xn → Θ for θ. Then for every θ0 ∈ Θ and  > 0, there
exists a pointwise test sequence (and hence, by dominated convergence, also a Bayesian test
sequence) for B = {θ ∈ Θ : d(θ, θ0 ) < 12 } versus V = {θ ∈ Θ : d(θ, θ0 ) > }. This approach is
followed in example B.19 on random walks, see the definition of the test following inequality
(B.36).
A third perspective on the existence of Bayesian tests arises from Doob’s argument. From
our present perspective, we note that theorem 2.4 implies an alternative proof of Doob’s
consistency theorem through the following existence result on Bayesian test sequences. (Note:
here and elsewhere in i.i.d. setting, the parameter space Θ is P, θ is the single-observation
distribution P and θ 7→ Pθ,n is P 7→ P n .)
Proposition 2.5 Consider a model P of single-observation distributions P for i.i.d. data
(X1 , X2 , . . . , Xn ) ∼ P n , (n ≥ 1). Assume that P is a Polish space with Borel prior Π. For
any Borel set V there is a Bayesian test sequence for V versus P \ V under Π.
6

Doob’s theorem is recovered when we let V be the complement of any open neighbourhood
U of P0 . Comparing with conditions for the existence of uniform tests, Bayesian tests are
quite abundant: whereas uniform testing relies on the minimax theorem (forcing convexity,
compactness and continuity requirements into the picture), Bayesian tests exist quite generally
(at least, for Polish parameters with i.i.d. data).
The fourth perspective on the existence of Bayesian tests concerns a direct way to construct
a Bayesian test sequence of optimal power, based on the fact that we are really only testing
barycentres against each other: let priors (Πn ) and G -measurable model subsets Bn , Vn be
given. For given tests (φn ) and power sequence an , write (3) as follows:
Πn (Bn ) PnΠn |Bn φn (X n ) + Πn (Vn ) PnΠn |Vn φn (X n ) = o(an ),
Π |Bn

and note that what is required here, is a (weighted) test of (Pn n
likelihood-ratio test (denote the density for
pBn ,n , and similar for

Π |B
Pn n n

Π |Vn

) versus (Pn n

with respect to µn =

Π |B
Pn n n

). The

Π |V
+ Pn n n

by

Π |V
Pn n n ),

φn (X n ) = 1{Πn (Vn ) pVn ,n (X n )>Πn (Bn ) pBn ,n (X n )} ,
Π |Bn

is optimal and has power kΠn (Bn ) Pn n

Π |Bn

∧ Πn (Vn ) Pn n

k. This proves the following useful

proposition that re-expresses power in terms of the relevant Hellinger transform (see, e.g.
section 16.4 in [51], particularly, Remark 1).
Proposition 2.6 Let priors (Πn ) and measurable model subsets Bn , Vn be given. There exists
a test sequence φn : Xn → [0, 1] such that,
Z
Z
Pθ,n φn dΠn (θ) +
Pθ,n (1 − φn ) dΠn (θ)
Bn
Vn
Z 
α 
1−α
Πn (Bn ) pBn ,n (x)
Πn (Vn ) pVn ,n (x)
≤
dµn (x),

(6)

for every n ≥ 1 and any 0 ≤ α ≤ 1.
Proposition 2.6 generalises proposition 2.5 and makes Bayesian tests available with a (close-to)sharp bound on the power under fully general conditions. For the connection with minimax
tests, we note the following. If {Pθ,n : θ ∈ Bn } and {Pθ,n : θ ∈ Vn } are convex sets (and the
Πn are Radon measures, e.g. in Polish parameter spaces), then,

H PnΠn |Bn , PnΠn |Vn ≥ inf{H(Pθ,n , Pθ0 ,n ) : θ ∈ Bn , θ0 ∈ Vn }.
Combination with (6) for α = 1/2, implies that the minimax upper bound in i.i.d. cases, c.f.
proposition B.7 remains valid:
Z
Z
n
P φn dΠn (P ) +
Bn

Qn (1 − φn ) dΠn (Q) ≤

p
2
Πn (Bn ) Πn (Vn ) e−nn ,

(7)

Vn

where n = inf{H(P, Q) : P ∈ Bn , Q ∈ Vn }. Given an ↓ 0, any Bayesian test φn that satisfies
(3) for all probability measures Πn on Θ, is a (weighted) minimax test for Bn versus Vn of
power an .
7

Note that the above enhances the role that the prior plays in the frequentist discussion of the
asymptotic behaviour of the posterior: the prior is not only important in requirements like
(2), but can also be of influence in the testing condition: where testing power is relatively
weak, prior mass should be scarce to compensate and where testing power is strong, prior
mass should be plentiful. To make use of this, one typically imposes upper bounds on prior
mass in certain hard-to-test subsets of the model (as opposed to lower bounds like (2)). See
example B.19 on random-walk data. In the Hellinger-geometric view, the prior determines
Π |Bn

whether the local prior predictive distributions Pn n

Π |Vn

and Pn n

lie close together or not in

Hellinger distance, and thus to the r.h.s. of (6) for α = 1/2. This phenomenon plays a role in
example B.17 on the estimation of a sparse vector of normal means, where it explains why the
slab-component of a spike-and-slab prior must have a tail that is heavy enough.

2.3

Le Cam’s inequality

Referring to the argument following proposition 2.3, one way of guaranteeing that the exΠ|Bn

pectations of X n 7→ Π(V |X n ) under Pn

approximate those under P0,n , is to choose

Bn = {θ ∈ Θ : kPθ,n − Pθ0 ,n k ≤ δn }, for some sequence δn → 0, because in that case,
Π|Bn

|P0,n ψ − Pn

Π|Bn

ψ| ≤ kP0,n − Pn

k ≤ δn , for any random variable ψ : Xn → [0, 1]. Without

fixing the definition of the sets Bn , one may use this step to specify inequality (4) further:
P0,n Π(Vn |X) ≤ P0,n − PnΠ|Bn
Z
Z
Πn (Vn )
Pθ,n (1 − φn ) dΠn (θ|Vn ),
+ Pθ,n φn dΠn (θ|Bn ) +
Πn (Bn )

(8)

for Bn and Vn such that Πn (Bn ) > 0 and Πn (Vn ) > 0. Le Cam’s inequality (8) is used, for
example, in the proof of the Bernstein-von Mises theorem, see lemma 2 in section 8.4 of [53]. A
less successful application pertains to non-parametric posterior rates of convergence for i.i.d.
data, in an unpublished paper [50]. Rates of convergence obtained in this way are suboptimal:
Le Cam qualifies the first term on the right-hand side of (8) as a “considerable nuisance”
and concludes that “it is unclear at the time of this writing what general features, besides the
metric structure, could be used to refine the results”, (see [51], end of section 16.6). In [74],
Le Cam relates the posterior question to dimensionality restrictions [49, 63, 34] and reiterates,
“And for Bayes risk, I know that just the metric structure does not catch everything, but I
don’t know what else to look at, except calculations.”

3

Remote contiguity

Le Cam’s notion of contiguity describes an asymptotic version of absolute continuity, applicable
to sequences of probability measures in a limiting sense [48]. In this section we weaken the

8

property of contiguity in a way that is suitable to promote Π-almost-everywhere Bayesian
limits to frequentist limits that hold everywhere.

3.1

Definition and criteria for remote contiguity

The notion of ‘domination’ left undefined in the argument following proposition 2.3 is made
rigorous here.
Definition 3.1 Given measurable spaces (Xn , Bn ), n ≥ 1 with two sequences (Pn ) and (Qn )
of probability measures and a sequence ρn ↓ 0, we say that Qn is ρn -remotely contiguous with
respect to Pn , notation Qn C ρ−1
n Pn , if,
Pn φn (X n ) = o(ρn )

Qn φn (X n ) = o(1),

⇒

(9)

for every sequence of Bn -measurable φn : Xn → [0, 1].
Note that for a sequence (Qn ) that is an -remotely contiguous with respect to (Pn ), there
exists no test sequence that distinguishes between Pn and Qn with power an . Note also that
given two sequences (Pn ) and (Qn ), contiguity Pn C Qn is equivalent to remote contiguity
Pn C a−1
n Qn for all an ↓ 0. Given sequences an , bn ↓ 0 with an = O(bn ), bn -remote contiguity
implies an -remote contiguity of (Pn ) with respect to (Qn ).
Example 3.2 Let P be a model for the distribution of a single observation in i.i.d. samples
X n = (X1 , . . . , Xn ). Let P0 , P and  > 0 be such that −P0 log(dP/dP0 ) < 2 . The law of large
numbers implies that for large enough n,
n 2
dP n n
(X ) ≥ e− 2  ,
n
dP0

(10)

with P0n -probability one. Consequently, for large enough n and for any Bn -measurable sequence ψn : Xn → [0, 1],
1

2

P n ψn ≥ e− 2 n P0n ψn .

(11)

Therefore, if P n φn = o(exp (− 21 n2 )) then P0n φn = o(1). Conclude that for every  > 0, the
Kullback-Leibler neighbourhood {P : −P0 log(dP/dP0 ) < 2 } consists of model distributions
for which the sequence (P0n ) of product distributions are exp (− 21 n2 )-remotely contiguous
with respect to (P n ).



Criteria for remote contiguity are given in the lemma below; note that, here, we give sufficient conditions, rather than necessary and sufficient, as in Le Cam’s First Lemma. (For the
definition of (dPn /dQn )−1 , see appendix A, notation and conventions.)
Lemma 3.3 Given (Pn ), (Qn ), an ↓ 0, Qn C a−1
n Pn , if any of the following hold:
P

Qn

n
(i) for any Bn -measurable φn : Xn → [0, 1], a−1
−→
0 implies φn −−→ 0,
n φn −

(ii) given  > 0, there is a δ > 0 such that Qn (dPn /dQn < δ an ) < , for large enough n,
9

−1
(iii) there is a b > 0 such that lim inf n b a−1
n Pn (dQn /dPn > b an ) = 1,

(iv) for any  > 0, there is a constant c > 0 such that kQn − Qn ∧ c a−1
n Pn k < , for large
enough n,
(v) under Qn every subsequence of (an (dPn /dQn )−1 ) has a weakly convergent subsequence.
Proof The proof of this lemma can be found in appendix C. It actually proves that ((i) or
(iv)) implies remote contiguity; that ((ii) or (iii)) implies (iv) and that (v) is equivalent to
(ii).



Contiguity and its remote variation are compared in the context of (parametric and nonparametric) regression in examples B.11 and B.12. We may specify the definition of remote
contiguity slightly further.
Definition 3.4 Given measurable spaces (Xn , Bn ), (n ≥ 1) with two sequences (Pn ) and
(Qn ) of probability measures and sequences ρn , σn > 0, ρn , σn → 0, we say that Qn is ρn -to-σn
remotely contiguous with respect to Pn , notation σn−1 Qn C ρ−1
n Pn , if,
Pn φn (X n ) = o(ρn )

⇒

Qn φn (X n ) = o(σn ),

for every sequence of Bn -measurable φn : Xn → [0, 1].
Like definition 3.1, definition 3.4 allows for reformulation similar to lemma 3.3, e.g. if for some
sequences ρn , σn like in definition 3.4,
Qn − Qn ∧ σn ρ−1
n Pn = o(σn ),
then σn−1 Qn C ρ−1
n Pn . We leave the formulation of other sufficient conditions to the reader.
2
n
−1 n
Note that inequality (11) in example 3.2 implies that b−1
n P0 C an P , for any an ≤ exp(−nα )
Qn -a.s.

with α2 > 21 2 and bn = exp(−n(α2 − 12 2 )). It is noted that this implies that φn (X n ) −−−−−→ 0
for any φn : Xn → [0, 1] such that Pn φn (X n ) = o(ρn ) (more generally, this holds whenever
P
n σn < ∞, as a consequence of the first Borel-Cantelli lemma).

3.2

Remote contiguity for Bayesian limits

The relevant applications in the context of Bayesian limit theorems concern remote contiguity
of the sequence of true distributions Pθ0 ,n with respect to local prior predictive distributions
Π |Bn

Pn n

, where the sets Bn ⊂ Θ are such that,
Πn |Bn
Pθ0 ,n C a−1
,
n Pn

(12)

for some rate an ↓ 0.
In the case of i.i.d. data, Barron [3] introduces strong and weak notions of merging of Pθ0 ,n
with (non-local) prior predictive distributions PnΠ . The weak version imposes condition (ii) of
10

lemma 3.3 for all exponential rates simultaneously. Strong merging (or matching [2]) coincides
with Schwartz’s almost-sure limit, while weak matching is viewed as a limit in probability.
By contrast, if we have a specific rate an in mind, the relevant mode of convergence is Prohorov’s weak convergence: according to lemma 3.3-(v), (12) holds if inverse likelihood ratios
Zn have a weak limit Z when re-scaled by an ,
Zn = (dPnΠn |Bn /dPθ0 ,n )−1 (X n ),

Pθ

,n -w.

0
an Zn −−−
−−→ Z.

To better understand the counterexamples of section B, notice the high sensitivity of this
criterion to the existence of subsets of the sample spaces assigned probability zero under
some model distributions, while the true probability is non-zero. More generally, remote
contiguity is sensitive to subsets En assigned fast decreasing probabilities under local prior
Π |Bn

predictive distributions Pn n

(En ), while the probabilities Pθ0 ,n (En ) remain high, which is

what definition 3.1 expresses. The rate an ↓ 0 helps to control the likelihood ratio (compare
to the unscaled limits of likelihood ratios that play a central role in the theory of convergence
of experiments [51]), conceivably enough to force uniform tightness in many non-parametric
situations.
But condition (12) can also be written out, for example to the requirement that for some
constant δ > 0,
Pθ0 ,n


 Z dP
θ,n
(X n ) dΠn (θ|Bn ) < δ an → 0,
dPθ0 ,n

with the help of lemma 3.3-(ii).
Example 3.5 Consider again the model of example 1.3. In example B.14, it is shown that if
the prior Π for θ ∈ R has a continuous and strictly positive Lebesgue density and we choose
Bn = [θ0 , θ0 + 1/n], then for every δ > 0 and all an ↓ 0,

Z

dPθ,n
n
n
Pθ0
(X ) dΠ(θ|Bn ) < δ an ≤ Pθn0 n(X(1) − θ0 ) < 2δ an ,
dPθ0 ,n
for large enough n ≥ 1, and the r.h.s. goes to zero for any an because the random variables
n(X(1) − θ0 ) have a non-degenerate, positive weak limit under Pθn0 as n → ∞. Conclude that
with these choices for Π and Bn , (12) holds, for any an .



The following proposition should be viewed in light of [52], which considers properties like
contiguity, convergence of experiments and local asymptotic normality in situations of statistical information loss. In this case, we are interested in (remote) contiguity of the probability
measures that arise as marginals for the data X n when information concerning the (Bayesian
random) parameter θ is unavailable.
Proposition 3.6 Let θ0 ∈ Θ and a prior Π : G → [0, 1] be given. Let B be a measurable
subset of Θ such that Π(B) > 0. Assume that for some an ↓ 0, the family,
 

dPθ,n −1 n
an
(X ) : θ ∈ B, n ≥ 1 ,
dPθ0 ,n
11

Π|B

is uniformly tight under Pθ0 ,n . Then Pθ0 ,n C a−1
n Pn

.

Other sufficient conditions from lemma 3.3 may replace the uniform tightness condition. When
the prior Π and subset B are n-dependent, application of lemma 3.3 requires more. (See, for
instance, example B.12 and lemma B.13, where local asymptotic normality is used to prove
(12).)
To re-establish contact with the notion of merging, note the following. If remote contiguity
of the type (12) can be achieved for a sequence of subsets (Bn ), then it also holds for any
sequence of sets (e.g. all equal to Θ, in Barron’s case) that contain the Bn but at a rate that
differs proportionally to the fraction of prior masses.
Lemma 3.7 For all n ≥ 1, let Bn ⊂ Θ be such that Πn (Bn ) > 0 and Cn such that Bn ⊂ Cn
with cn = Πn (Bn )/Πn (Cn ) ↓ 0, then,
Πn |Cn
PnΠn |Bn C c−1
.
n Pn
Π |Bn

n
Also, if for some sequence (Pn ), Pn C a−1
n Pn

Π |Cn

n
−1
then Pn C a−1
n cn Pn

.

So when considering possible choices for the sequence (Bn ), smaller choices lead to slower
rates an , rendering (9) applicable to more sequences of test functions. This advantage is to be
balanced against later requirements that Πn (Bn ) may not decrease too fast.

4

Posterior concentration

In this section new frequentist theorems are formulated involving the convergence of posterior
distributions. First we give a basic proof for posterior consistency assuming existence of
suitable test sequences and remote contiguity of true distributions (Pθ0 ,n ) with respect to local
prior predictive distributions. Then it is not difficult to extend the proof to the case of posterior
rates of convergence in metric topologies. With the same methodology it is possible to address
questions in Bayesian hypothesis testing and model selection: if a Bayesian test to distinguish
between two hypotheses exists and remote contiguity applies, frequentist consistency of the
Bayes Factor can be guaranteed. We conclude with a theorem that uses remote contiguity to
describe a general relation that exists between credible sets and confidence sets, provided the
prior induces remotely-contiguous local prior predictive distributions.

4.1

Consistent posteriors

First, we consider posterior consistency generalising Schwartz’s theorem to sequentially observed (non-i.i.d.) data, non-dominated models and priors or parameter spaces that may
depend on the sample size. For an early but very complete overview of literature and developments in posterior consistency, see [33].
12

Definition 4.1 The posteriors Π( · |X n ) are consistent at θ ∈ Θ if for every neighbourhood U
of θ,
Pθ,n

Π(U |X n ) −−−→ 1.

(13)

The posteriors are said to be consistent if this holds for all θ ∈ Θ. We say that the posterior
is almost-surely consistent if convergence occurs almost-surely with respect to some coupling
for the sequence (Pθ0 ,n ).
Equivalently, posterior consistency can be characterized in terms of posterior expectations of
bounded and continuous functions (see proposition B.5).
Theorem 4.2 Assume that for all n ≥ 1, the data X n ∼ Pθ0 ,n for some θ0 ∈ Θ. Fix a prior
Π : G → [0, 1] and assume that for given B, V ∈ G with Π(B) > 0 and an ↓ 0,
(i) there exist Bayesian tests φn for B versus V ,
Z
Z
Pθ0 ,n (1 − φn ) dΠ(θ0 ) = o(an ),
Pθ,n φn dΠ(θ) +
Π|B

(ii) the sequence Pθ0 ,n satisfies Pθ0 ,n C a−1
n Pn
Pθ

(14)

V

B

.

,n

Then Π(V |X n ) −−−0−→ 0.
These conditions are to be interpreted as follows: theorem 2.4 lends condition (i) a distinctly
Bayesian interpretation: it requires a Bayesian test to set V apart from B with testing power
an . Lemma 2.2 translates this into the (still Bayesian) statement that the posteriors for V
Π|B

go to zero in Pn

-expectation. Condition (ii) is there to promote this Bayesian point to a

frequentist one through (9). To present this from another perspective: condition (ii) ensures
Π|B

that the Pn

cannot be tested versus Pθ0 ,n at power an , so the posterior for V go to zero in

Pθ0 ,n -expectation as well (otherwise a sequence φn (X n ) ∝ Π(V |X n ) would constitute such a
test).
To illustrate theorem 4.2 and its conditions Freedman’s counterexamples are considered in
detail in example B.4.
A proof of a theorem very close to Schwartz’s theorem is now possible. Consider condition
(i) of theorem 1.2: a well-known argument based on Hoeffding’s inequality guarantees the
existence of a uniform test sequence of exponential power whenever a uniform test sequence
test sequence exists, so Schwartz equivalently assumes that there exists a D > 0 such that,
P0n φn + sup Qn (1 − φn ) = o(e−nD ).
Q∈P\U

We vary slightly and assume the existence of a Bayesian test sequence of exponential power.
In the following theorem, let P denote a Hausdorff space of single-observation distributions
on (X , B) with Borel prior Π.

13

Corollary 4.3 For all n ≥ 1, let (X1 , X2 , . . . , Xn ) ∼ P0n for some P0 ∈ P. Let U denote an
open neighbourhood of P0 and define K() = {P ∈ P : −P0 log(dP/dP0 ) < 2 }. If,
(i) there exist  > 0, D > 0 and a sequence of measurable ψn : X n → [0, 1], such that,
Z
Z
n
Qn (1 − ψn ) dΠ(Q) = o(e−nD ),
P ψn dΠ(P ) +
K()

P\U

(ii) and Π(K()) > 0 for all  > 0,
P -a.s.

then Π(U |X n ) −−0−−→ 1.
An instance of the application of corollary 4.3 is given in example B.10. Example B.23 demonstrates posterior consistency in total variation for i.i.d. data from a finite sample space, for
priors of full support. Extending this, example B.24 concerns consistency of posteriors for
priors that have Freedman’s tailfreeness property [30], like the Dirichlet process prior. Also
interesting in this respect is the Neyman-Scott paradox, a classic example of inconsistency for
the ML estimator, discussed in Bayesian context in [5]: whether the posterior is (in)consistent
depends on the prior. The Jeffreys prior follows the ML estimate while the reference prior
avoids the Neyman-Scott inconsistency. Another question in a sequence model arises when we
analyse FDR-like posterior consistency for a sequence vector that is assumed to be sparse (see
example B.17).

4.2

Rates of posterior concentration

A significant extension to the theory on posterior convergence is formed by results concerning
posterior convergence in metric spaces at a rate. Minimax rates of convergence for (estimators
based on) posterior distributions were considered more or less simultaneously in Ghosal-Ghoshvan der Vaart [34] and Shen-Wasserman [63]. Both propose an extension of Schwartz’s theorem
to posterior rates of convergence [34, 63] and apply Barron’s sieve idea with a well-known
entropy argument [7, 8] to a shrinking sequence of Hellinger neighbourhoods and employs a
more specific, rate-related version of the Kullback-Leibler condition (2) for the prior. Both
appear to be inspired by contemporary results regarding Hellinger rates of convergence for sieve
MLE’s, as well as on Barron-Schervish-Wasserman [4], which concerns posterior consistency
based on controlled bracketing entropy for a sieve, up to subsets of negligible prior mass,
following ideas that were first laid down in [3]. It is remarked already in [4] that their main
theorem is easily re-formulated as a rate-of-convergence theorem, with reference to [63]. More
recently, Walker, Lijoi and Prünster [72] have added to these considerations with a theorem
for Hellinger rates of posterior concentration in models that are separable for the Hellinger
metric, with a central condition that calls for summability of square-roots of prior masses of
covers of the model by Hellinger balls, based on analogous consistency results in Walker [70].
More recent is [46], which shows that alternatives for the priors of [34, 63] exist.

14

Theorem 4.4 Assume that for all n ≥ 1, the data X n ∼ Pθ0 ,n for some θ0 ∈ Θ. Fix priors
Πn : G → [0, 1] and assume that for given Bn , Vn ∈ G with Πn (Bn ) > 0 and an , bn ↓ 0 such
that an = o(bn ),
(i) there are Bayesian tests φn : Xn → [0, 1] such that,
Z
Z
Pθ,n (1 − φn ) dΠn (θ) = o(an ),
Pθ,n φn dΠn (θ) +
Bn

(15)

Vn

(ii) The prior mass of Bn is lower-bounded, Πn (Bn ) ≥ bn ,
Π |Bn

n
(iii) The sequence Pθ0 ,n satisfies Pθ0 ,n C bn a−1
n Pn

Pθ

.

,n

Then Π(Vn |X n ) −−−0−→ 0.



Example 4.5 To apply theorem 4.4, consider again the situation of a uniform distribution
with an unknown location, as in examples 1.3 and 3.5. Taking Vn equal to {θ : θ − θ0 > n }
{θ : θ0 − θ > n } respectively, with n = Mn /n for some Mn → ∞, suitable test sequences
are constructed in example B.15, and in combination with example 3.5, lead to the conclusion
that with a prior Π for θ that has a continuous and strictly positive Lebesgue density, the
posterior is consistent at (any n slower than) rate 1/n.
Example 4.6 Let us briefly review the conditions of [4, 34, 63] in light of theorem 4.4: let
n ↓ 0 denote the Hellinger rate of convergence we have in mind, let M > 1 be some constant
and define,
Vn = {P ∈ P : H(P, P0 ) ≥ M n },
Bn = {P ∈ P : −P0 log dP/dP0 < 2n , P0 log2 dP/dP0 < 2n }.
Theorems for posterior convergence at a rate propose a sieve of submodels satisfying entropy
conditions like those of [7, 8, 51] and a negligibility condition for prior mass outside the sieve
[3], based on the minimax Hellinger rate of convergence n ↓ 0. Together, they guarantee the
existence of Bayesian tests for Hellinger balls of radius n versus complements of Hellinger
balls of radius M n of power exp(−DM 2 n2n ) for some D > 0 (see example B.8). Note that
Bn is contained in the Hellinger ball of radius n around P0 , so (15) holds. New in [34, 63] is
the condition for the priors Πn ,
2

Πn (Bn ) ≥ e−Cnn ,

(16)

for some C > 0. With the help of lemmas B.16 and 3.3-(ii), we conclude that,
2

P0n C ecnn PnΠ|Bn ,

(17)
P

0
for any c > 1. If we choose M such that DM 2 −C > 1, theorem 4.4 proves that Π(Vn |X n ) −−→
0,

i.e. the posterior is Hellinger consistent at rate n .
Certain (simple, parametric) models do not allow the definition of priors that satisfy (16), and
alternative less restrictive choices for the sets Bn are possible under mild conditions on the
model [46].
15

4.3

Consistent hypothesis testing with Bayes factors

The Neyman-Pearson paradigm notwithstanding, hypothesis testing and classification concern
the same fundamental statistical question, to find a procedure to choose one subset from a
given partition of the parameter space as the most likely to contain the parameter value of
the distribution that has generated the data observed. Asymptotically one wonders whether
choices following such a procedure focus on the correct subset with probability growing to one.
From a somewhat shifted perspective, we argue as follows: no statistician can be certain of
the validity of specifics in his model choice and therefore always runs the risk of biasing his
analysis from the outset. Non-parametric approaches alleviate his concern but imply greater
uncertainty within the model, leaving the statistician with the desire to select the correct
(sub)model on the basis of the data before embarking upon the statistical analysis proper (for
a recent overview, see [69]). The issue also makes an appearance in asymptotic context, where
over-parametrized models leave room for inconsistency of estimators, requiring regularization
[9, 10, 12].
Model selection describes all statistical methods that attempt to determine from the data which
model to use. (Take for example sparse variable selection, where one projects out the majority
of covariates prior to actual estimation, and the model-selection question is which projection
is optimal.) Methods for model selection range from simple rules-of-thumb, to cross-validation
and penalization of the likelihood function. Here we propose to conduct the frequentist analysis
with the help of a posterior: when faced with a (dichotomous) model choice, we let the so-called
Bayes factor formulate our preference. For an analysis of hypothesis testing that compares
Bayesian and frequentist views, see [5]. An objective Bayesian perspective on model selection
is provided in [73].
Definition 4.7 For all n ≥ 1, let the model be parametrized by maps θ 7→ Pθ,n on a parameter
space (Θ, G ) with priors Πn : G → [0, 1]. Consider disjoint, measurable B, V ⊂ Θ. For given
n ≥ 1, we say that the Bayes factor for testing B versus V ,
Fn =

Π(B|X n ) Πn (V )
,
Π(V |X n ) Πn (B)
Pθ,n

Pθ,n

is consistent for testing B versus V , if for all θ ∈ V , Fn −−−→ 0 and for all θ ∈ B, Fn−1 −−−→ 0.
Let us first consider this from a purely Bayesian perspective: for fixed prior Π and i .i .d . data,
theorem 2.4 says that the posterior gives rise to consistent Bayes factors for B versus V in a
Bayesian (that is, Π-almost-sure) way, iff a Bayesian test sequence for B versus V exists. If
the parameter space Θ is Polish and the maps θ 7→ Pθ (A) are Borel measurable for all A ∈ B,
proposition 2.5 says that any Borel set V is Bayesian testable versus Θ \ V , so in Polish models
for i.i.d. data, model selection with Bayes factors is Π-almost-surely consistent for all Borel
measurable V ⊂ Θ.
16

The frequentist requires strictly more, however, so we employ remote contiguity again to bridge
the gap with the Bayesian formulation.
Theorem 4.8 For all n ≥ 1, let the model be parametrized by maps θ 7→ Pθ,n on a parameter
space with (Θ, G ) with priors Πn : G → [0, 1]. Consider disjoint, measurable B, V ⊂ Θ with
Πn (B), Πn (V ) > 0 such that,
(i) There exist Bayesian tests for B versus V of power an ↓ 0,
Z
Z
n
Qn (1 − φn ) dΠn (Q) = o(an ),
P φn dΠn (P ) +
V

B
Π |B

Π |V

n
n
(ii) For every θ ∈ B, Pθ,n C a−1
, and for every θ ∈ V , Pθ,n C a−1
.
n Pn
n Pn

Then the Bayes factor for B versus V is consistent.
Note that the second condition of theorem 4.8 can be replaced by a local condition: if, for every
Π |Bn

n
θ ∈ B, there exists a sequence Bn (θ) ⊂ B such that Πn (Bn (θ)) ≥ bn and Pθ,n C a−1
n bn Pn

then Pθ,n C

Πn |B
a−1
n Pn

,

(as a consequence of lemma 3.7 with Cn = B).

In example B.19, we use theorem 4.8 to prove the consistency of the Bayes factor for a goodnessof-fit test for the equilibrium distribution of an stationary ergodic Markov chain, based on largelength random-walk data, with prior and posterior defined on the space of Markov transition
matrices.

4.4

Confidence sets from credible sets

The Bernstein-von Mises theorem [53] asserts that the posterior for a smooth, finite-dimensional
parameter converges in total variation to a normal distribution centred on an efficient estimate with the inverse Fisher information as its covariance, if the prior has full support. The
methodological implication is that Bayesian credible sets derived from such a posterior can
be reinterpreted as asymptotically efficient confidence sets. This parametric fact begs for
the exploration of possible non-parametric extensions but Freedman discourages us [32] with
counterexamples (see also [17]) and concludes that: “The sad lesson for inference is this. If frequentist coverage probabilities are wanted in an infinite-dimensional problem, then frequentist
coverage probabilities must be computed.”
In recent years, much effort has gone into calculations that address the question whether nonparametric credible sets can play the role of confidence sets nonetheless. The focus lies on
well-controlled examples in which both model and prior are Gaussian so that the posterior
is conjugate and analyse posterior expectation and variance to determine whether credible
metric balls have asymptotic frequentist coverage (for examples, see Szabó, van der Vaart and
van Zanten [68] and references therein). Below, we change the question slightly and do not seek
to justify the use of credible sets as confidence sets; from the present perspective it appears
17

more natural to ask in which particular fashion a credible set is to be transformed in order to
guarantee the transform is a confidence set, at least in the large-sample limit.
In previous subsections, we have applied remote contiguity after the concentration inequality
to control the Pθ0 ,n -expectation of the posterior probability for the alternative V through its
Π|Bn

Pn

-expectation. In the discussion of the coverage of credible sets that follows, remote conti-

guity is applied to control the Pθ0 ,n -probability that θ0 falls outside the prospective confidence
Π|Bn

set through its Pn

-probability. The theorem below then follows from an application of

Bayes’s rule (A.22). Credible levels provide the sequence an .
Definition 4.9 Let (Θ, G ) with prior Π, denote the sequence of posteriors by Π(·|·) : G ×Xn →
[0, 1]. Let D denote a collection of measurable subsets of Θ. A sequence of credible sets (Dn ) of
credible levels 1−an (where 0 ≤ an ≤ 1, an ↓ 0) is a sequence of set-valued maps Dn : Xn → D
such that Π(Θ \ Dn (x)|x) = o(an ) for PnΠn -almost-all x ∈ Xn .
Definition 4.10 For 0 ≤ a ≤ 1, a set-valued map x 7→ C(x) defined on X such that, for all
θ ∈ Θ, Pθ (θ 6∈ C(X)) ≤ a, is called a confidence set of level 1 − a. If the levels 1 − an of a
sequence of confidence sets Cn (X n ) go to 1 as n → ∞, the Cn (X n ) are said to be asymptotically
consistent.
Definition 4.11 Let D be a (credible) set in Θ and let B = {B(θ) : θ ∈ Θ} denote a collection
of model subsets such that θ ∈ B(θ) for all θ ∈ Θ. A model subset C 0 is said to be (a confidence
set) associated with D under B, if for all θ ∈ Θ \ C 0 , B(θ) ∩ D = ∅. The intersection C of
all C 0 like above equals {θ ∈ Θ : B(θ) ∩ D 6= ∅} and is called the minimal (confidence) set
associated with D under B (see Fig 1).
Example B.25 makes this construction explicit in uniform spaces and specializes to metric
context.
Theorem 4.12 Let θ0 ∈ Θ and 0 ≤ an ≤ 1, bn > 0 such that an = o(bn ) be given. Choose
priors Πn and let Dn denote level-(1 − an ) credible sets. Furthermore, for all θ ∈ Θ, let
Bn = {Bn (θ) ∈ G : θ ∈ Θ} denote a sequence such that,
(i) Πn (Bn (θ0 )) ≥ bn ,
Π |Bn (θ0 )

n
(ii) Pθ0 ,n C bn a−1
n Pn

.

Then any confidence sets Cn associated with the credible sets Dn under Bn are asymptotically
consistent, i.e. for all θ0 ∈ Θ,

Pθ0 ,n θ0 ∈ Cn (X n ) → 1.

(18)

This refutes Freedman’s lesson, showing that the asymptotic identification of credible sets and
confidence sets in smooth parametric models (the main inferential implication of the Bernsteinvon Mises theorem) generalises to the above form of asymptotic congruence in non-parametric

18

B(θ)

C

θ
D

Figure 1: The relation between a credible set D and
its associated (minimal) confidence set C under B in
Venn diagrams: the extra points θ in the associated
confidence set C not included in the credible set D
are characterized by non-empty intersection B(θ) ∩
D 6= ∅.
models. The fact that this statement holds in full generality implies very practical ways
to obtain confidence sets from posteriors, calculated, simulated or approximated. A second
remark concerns the confidence levels of associated confidence sets. In order for the assertion
of theorem 4.12 to be specific regarding the confidence level (rather than just resulting in
asymptotic coverage), we re-write the last condition of theorem 4.12 as follows,
Π |Bn (θ0 )

n
−1
(ii’) c−1
n Pθ0 ,n C bn an Pn

,

so that the last step in the proof of theorem 4.12 is more specific; particularly, assertion (18)
becomes,

Pθ0 ,n θ ∈ Dn (X n ) = o(cn ),
i.e. the confidence level of the sets Dn (X n ) is 1−Kcn asymptotically (for some constant K > 0
and large enough n).
The following corollary that specializes to the i.i.d. situation is immediate (see example B.26).
Let P denote a model of single-observation distributions, endowed with the Hellinger or totalvariational topology.
Corollary 4.13 For n ≥ 1 assume that (X1 , X2 , . . . , Xn ) ∈ X n ∼ P0n for some P0 ∈ P.
Let Πn denote Borel priors on P, with constant C > 0 and rate sequence n ↓ 0 such that
(16) is satisfied. Denote by Dn credible sets of level 1 − exp(−C 0 n2n ), for some C 0 > C.
Then the confidence sets Cn associated with Dn under radius-n Hellinger-enlargement are
asymptotically consistent.
19

Note that in the above corollary,
diamH (Cn (X n )) = diamH (Dn (X n )) + 2n ,
P0n -almost surely. If, in addition to the conditions in the above corollary, tests satisfying
(15) with an = exp(−C 0 n2n ) exist, the posterior is consistent at rate n and sets Dn (X n )
have diameters decreasing as n , c.f. theorem 4.4. In the case n is the minimax rate of
convergence for the problem, the confidence sets Cn (X n ) attain rate-optimality [55]. Rateadaptivity [42, 13, 68] is not possible like this because a definite, non-data-dependent choice
for the Bn is required.

5

Conclusions

We list and discuss the main conclusions of this paper below.
Frequentist validity of Bayesian limits
There exists a systematic way of taking Bayesian limits into frequentist ones, if priors
satisfy an extra condition relating true data distributions to localized prior predictive
distributions. This extra condition generalises Schwartz’s Kullback-Leibler condition and
amounts to a weakened form of contiguity, termed remote contiguity.
For example regarding consistency with i.i.d. data, Doob shows that a Bayesian form of posterior consistency holds without any real conditions on the model. To the frequentist, ‘holes’ of
potential inconsistency remain, in null-sets of the prior. Remote contiguity ‘fills the holes’ and
elevates the Bayesian form of consistency to the frequentist one. Similarly, prior-almost-surely
consistent tests are promoted to frequentist consistent tests and Bayesian credible sets are
converted to frequentist confidence sets.
The nature of Bayesian test sequences
The existence of a Bayesian test sequence is equivalent to consistent posterior convergence
in the Bayesian, prior-almost-sure sense. In theorems above, a Bayesian test sequence
thus represents the Bayesian limit for which we seek frequentist validity through remote
contiguity. Bayesian test sequences are more abundant than the more familiar uniform
test sequences. Aside from prior mass requirements arising from remote contiguity, the
prior should assign little weight where testing power is weak and much where testing
power is strong, ideally.
Example B.19 illustrates the influence of the prior when constructing a test sequence. Aside
from the familiar lower bounds for prior mass that arise from remote contiguity, existence of
Bayesian tests also poses upper bounds for prior mass.
Systematic analysis of complex models and datasets
Although many examples have been studied on a case-by-case basis in the literature,
20

the systematic analysis of limiting properties of posteriors in cases where the data is
dependent, or where the model, the parameter space and/or the prior are sample-size
dependent, requires generalisation of Schwartz’s theorem and its variations, which the
formalism presented here provides.
To elaborate, given the growing interest in the analysis of dependent datasets gathered from
networks (e.g. by webcrawlers that random walk linked webpages), or from time-series/stochastic
processes (e.g. financial data of the high-frequency type), or in the form of high-dimensional
or even functional data (biological, financial, medical and meteorological fields provide many
examples), the development of new Bayesian methods involving such aspects benefits from a
simple, insightful, systematic perspective to guide the search for suitable priors in concrete
examples.
To illustrate the last point, let us consider consistent community detection in stochastic block
models [64, 6]. Bayesian methods have been developed for consistent selection of the number
of communities [41], for community detection with a controlled error-rate with a growing
number of communities [16] and for consistent community detection using empirical priors
[67]. A moment’s thought on the discrete nature of the community assignment vector suggests
a sequence of uniform priors, for which remote contiguity (of Bn = {P0,n }) is guaranteed
(at any rate) and prior mass lower bounded by bn = Kn !Kn−n (where Kn is the number of
communities at ‘sample size’ n). It would be interesting to see under which conditions a
Bayesian test sequence of power an = o(bn ) can be devised that tests the true assignment
vector versus all alternatives (in the sparse regime [18, 1, 58]). Rather than apply a Chernoff
bound like in [16], one would probably have to start from the probabilistic [58] or informationtheoretic [1] analyses of respective algorithmic solutions in the (very closely related) planted bisection model. If a suitably powerful test can be shown to exist, theorem 4.4 proves frequentist
consistency of the posterior.
Methodology for uncertainty quantification
Use of a prior that induces remote contiguity allows one to convert credible sets of a calculated, simulated or approximated posterior into asymptotically consistent confidence
sets, in full generality. This extends the main inferential implication of the Bernsteinvon Mises theorem to non-parametric models without smoothness conditions.
The latter conclusion forms the most important and practically useful aspect of this paper.

A

Definitions and conventions

Because we take the perspective of a frequentist using Bayesian methods, we are obliged to
demonstrate that Bayesian definitions continue to make sense under the assumptions that the
data X is distributed according to a true, underlying P0 .
21

Remark A.1 We assume given for every n ≥ 1, a measurable (sample) space (Xn , Bn ) and
random sample X n ∈ Xn , with a model Pn of probability distributions Pn : Bn → [0, 1].
It is also assumed that there exists an n-independent parameter space Θ with a Hausdorff,
completely regular topology T and associated Borel σ-algebra G , and, for every n ≥ 1, a
bijective model parametrization Θ → Pn : θ 7→ Pθ,n such that for every n ≥ 1 and every
A ∈ Bn , the map Θ → [0, 1] : θ 7→ Pθ,n (A) is measurable. Any prior Π on Θ is assumed to be
a Borel probability measure Π : G → [0, 1] and can vary with the sample-size n. (Note: in i.i.d.
setting, the parameter space Θ is P1 , θ is the single-observation distribution P and θ 7→ Pθ,n
is P 7→ P n .) As frequentists, we assume that there exists a ‘true, underlying distribution for
the data; in this case, that means that for every n ≥ 1, there exists a distribution P0,n from
which the n-th sample X n is drawn.



Often one assumes that the model is well-specified : that there exists a θ0 ∈ Θ such that
P0,n = Pθ0 ,n for all n ≥ 1. We think of Θ as a topological space because we want to discuss
estimation as a procedure of sequential, stochastic approximation of and convergence to such
a ‘true parameter value θ0 . In theorem 2.4 and definition 4.1 we assume, in addition, that
the observations X n are coupled, i.e. there exists a probability space (Ω, F , P0 ) and random
variables X n : Ω → Xn such that P0 ((X n )−1 (A)) = P0,n (X n ∈ A) for all n ≥ 1 and A ∈ Bn .
Definition A.2 Given n, m ≥ 1 and a prior probability measure Πn : G → [0, 1], define the
n-th prior predictive distribution on Xm as follows:
Z
Πn
Pθ,m (A) dΠn (θ),
Pm (A) =

(A.19)

Θ

for all A ∈ Bm . If the prior is replaced by the posterior, the above defines the n-th posterior
predictive distribution on Xm ,
Πn |X n
Pm
(A)

Z
=

Pθ,m (A) dΠ(θ|X n ),

(A.20)

Θ

for all A ∈ Bm . For any Bn ∈ G with Πn (Bn ) > 0, define also the n-th local prior predictive
distribution on Xm ,
Πn |Bn
Pm
(A) =

1
Πn (Bn )

Z
Pθ,m (A) dΠn (θ),

(A.21)

Bn

as the predictive distribution on Xm that results from the prior Πn when conditioned on Bn .
If m is not mentioned explicitly, it is assumed equal to n.
The prior predictive distribution PnΠn is the marginal distribution for X n in the Bayesian
perspective that considers parameter and sample jointly (θ, X n ) ∈ Θ × Xn as the random
quantity of interest.
Definition A.3 Given n ≥ 1, a (version of) the posterior is any map Π( · |X n = · ) : G ×
Xn → [0, 1] such that,
(i) for B ∈ G , the map Xn → [0, 1] : xn 7→ Π(B|X n = xn ) is Bn -measurable,
22

(ii) for all A ∈ Bn and V ∈ G ,
Z
Z
n
Πn
Pθ,n (A) dΠn (θ).
Π(V |X ) dPn =

(A.22)

V

A

Bayes’s Rule is expressed through equality (A.22) and is sometimes referred to as a ‘disintegration’ (of the joint distribution of (θ, X n )). If the posterior is a Markov kernel, it is a
PnΠn -almost-surely well-defined probability measure on (Θ, G ). But it does not follow from the
definition above that a version of the posterior actually exists as a regular conditional probability measure. Under mild extra conditions, regularity of the posterior can be guaranteed: for
example, if sample space and parameter space are Polish, the posterior is regular; if the model
Pn is dominated (denote the density of Pθ,n by pθ,n ), the fraction of integrated likelihoods,
Z
Z
n
n
Π(V |X ) =
pθ,n (X ) dΠn (θ)
pθ,n (X n ) dΠn (θ),
(A.23)
V

Θ

for V ∈ G , n ≥ 1 defines a regular version of the posterior distribution. (Note also that there
is no room in definition (A.22) for X n -dependence of the prior, so ‘empirical Bayes’ methods
must be based on data Y n independent of X n , i.e. sample-splitting.)
Remark A.4 As a consequence of the frequentist assumption that X n ∼ P0,n for all n ≥ 1, the
PnΠn -almost-sure definition (A.22) of the posterior Π(V |X n ) does not make sense automatically
[29, 46]: null-sets of PnΠn on which the definition of Π( · |X n ) is ill-determined, may not be
null-sets of P0,n . To prevent this, we impose the domination condition,
P0,n  PnΠn ,

(A.24)

for every n ≥ 1.



To understand the reason for (A.24) in a perhaps more familiar way, consider a dominated
model and assume that for certain n, (A.24) is not satisfied. Then, using (A.19), we find,
Z

P0,n
pθ,n (X n ) dΠn (θ) = 0 > 0,
so the denominator in (A.23) evaluates to zero with non-zero P0,n -probability.
To get an idea of sufficient conditions for (A.24), it is noted in [46] that in the case of i.i.d.
data where P0,n = P0n for some marginal distribution P0 , P0n  PnΠ for all n ≥ 1, if P0 lies in
the Hellinger- or Kullback-Leibler-support of the prior Π. For the generalisation to the present
setting we are more precise and weaken the topology appropriately.
Definition A.5 For all n ≥ 1, let Fn denote the class of all bounded, Bn -measurable f :
Xn → R. The topology Tn is the initial topology on Pn for the functions {P 7→ P f : f ∈ Fn }.

Finite intersections of sets Uf, = {(P, Q) ∈ Pn2 : |(P − Q)f | < } (f ∈ Fn , 0 ≤ f ≤ 1 and
 > 0), form a fundamental system of entourages for a uniformity Un on Pn . A fundamental
23

system of neighbourhoods for the associated topology Tn on P is formed by finite intersections
of sets of the form,
WP,f, = {Q ∈ Pn : |(P − Q)f | < },
with P ∈ Pn , f ∈ Fn , 0 ≤ f ≤ 1 and  > 0.
If we model single-observation distributions P ∈ P for an i.i.d. sample, the topology Tn on
Pn = P n induces a topology on P (which we also denote by Tn ) for each n ≥ 1. The
union T∞ = ∪n Tn is an inverse-limit topology that allows formulation of conditions for the
existence of consistent estimates that are not only sufficient, but also necessary [47], offering
a precise perspective on what is estimable and what is not in i.i.d. context. The associated
strong topology is that generated by total variation (or, equivalently, the Hellinger metric).
For more on these topologies, the reader is referred to Strasser (1985) [65] and to Le Cam
(1986) [51]. We note explicitly the following fact, which is a direct consequence of Hoeffding’s
inequality.
Proposition A.6 (Uniform Tn -tests)
Consider a model P of single-observation distributions P for i.i.d. samples (X1 , X2 , . . . , Xn ) ∼
P n , (n ≥ 1). Let m ≥ 1,  > 0, P0 ∈ P and a measurable f : X m → [0, 1] be given. Define


B = P ∈ P : |(P m − P0m )f | <  , and V = P ∈ P : |(P m − P0m )f | ≥ 2 . There exist a
uniform test sequence (φn ) such that,
sup P n φn ≤ e−nD ,

sup Qn (1 − φn ) ≤ e−nD ,

P ∈B

Q∈V

for some D > 0.
Proof The proof is an application of Hoeffding’s inequality for the sum

Pn

i=1 f (Xi )

left to the reader.

and is


The topologies Tn also play a role for condition (A.24).
Proposition A.7 Let (Πn ) be Borel priors on the Hausdorff uniform spaces (Pn , Tn ). For
any n ≥ 1, if P0,n lies in the Tn -support of Πn , then P0,n  PnΠn .
Proof Let n ≥ 1 be given. For any A ∈ Bn and any U 0 ⊂ Θ such that Πn (U 0 ) > 0,
Z
P0,n (A) ≤ Pθ,n (A) dΠn (θ|U 0 ) + sup |Pθ,n (A) − P0,n (A)|.
θ∈U 0

Let A ∈ Bn be a null-set of PnΠn ; since Πn (U 0 ) > 0,

R

Pθ,n (A) dΠn (θ|U 0 ) = 0. For some  > 0,

take U 0 equal to the Tn -basis element {θ ∈ Θ : |Pθ,n (A) − Pθ0 ,n (A)| < } to conclude that
Pθ0 ,n (A) <  for all  > 0.



In many situations, priors are Borel for the Hellinger topology, so it is useful to observe that
the Hellinger support of Πn in Pn is always contained in the Tn -support.

24

Notation and conventions
l.h.s. and r.h.s. refer to left- and right-hand sides respectively. For given probability measures
P, Q on a measurable space (Ω, F ), we define the Radon-Nikodym derivative dP/dQ : Ω →
[0, ∞), P -almost-surely, referring only to the Q-dominated component of P , following [51].
We also define (dP/dQ)−1 : Ω → (0, ∞] : ω 7→ 1/(dP/dQ(ω)), Q-almost-surely. Given a
σ-finite measure µ that dominates both P and Q (e.g. µ = P + Q), denote dP/dµ = q and
dQ/dµ = p. Then the measurable map p/q 1{q > 0} : Ω → [0, ∞) is a µ-almost-everywhere
version of dP/dQ, and q/p 1{q > 0} : Ω → [0, ∞] of (dP/dQ)−1 . Define total-variational and
R
Hellinger distances by kP − Qk = supA |P (A) − Q(A)| and H(P, Q)2 = 1/2 (p1/2 − q 1/2 )2 dµ,
respectively. Given random variables Zn ∼ Pn , weak convergence to a random variable Z is
P -w.

P

n
denoted by Zn −−n−−→ Z, convergence in probability by Zn −−→
Z and almost-sure convergence

P ∞ -a.s.

(with coupling P ∞ ) by Zn −−−−−→ Z. The integral of a real-valued, integrable random variable
X with respect to a probability measure P is denoted P X, while integrals over the model with
respect to priors and posteriors are always written out in Leibniz’s notation. For any subset
B of a topological space, B̄ denotes the closure, B̊ the interior and ∂B the boundary. Given
 > 0 and a metric space (Θ, d), the covering number N (, Θ, d) ∈ N ∪ {∞} is the minimal
cardinal of a cover of Θ by d-balls of radius . Given real-valued random variables X1 , . . . , Xn ,
the first order statistic is X(1) = min1≤i≤n Xi . The Hellinger diameter of a model subset C is
denoted diamH (C) and the Euclidean norm of a vector θ ∈ Rn is denoted kθk2,n .

B

Applications and examples

In this section of the appendix examples and applications are collected.

B.1

Inconsistent posteriors

Calculations that demonstrate instances of posterior inconsistency are many (for (a far-fromexhaustive list of) examples, see [20, 21, 17, 22, 23, 31, 32]). In this subsection, we discuss
early examples of posterior inconsistency that illustrate the potential for problems clearly and
without distracting technicalities.
Example B.1 (Freedman (1963) [29])
Consider a sample X1 , X2 , . . . of random positive integers. Denote the space of all probability
distributions on N by Λ and assume that the sample is i.i.d.-P0 , for some P0 ∈ Λ. For any
P ∈ Λ, write p(i) = P ({X = i}) for all i ≥ 1. The total-variational and weak topologies on
Λ are equivalent (defined, P → Q if p(i) → q(i) for all i ≥ 1). Let Q ∈ Λ \ {P0 } be given.
To arrive at a prior with P0 in its support, leading to a posterior that concentrates on Q, we
consider sequences (Pm ) and (Qn ) such that Qm → Q and Pm → P0 as m → ∞. The prior Π

25

places masses αm > 0 at Pm and βm > 0 at Qm (m ≥ 1), so that P0 lies in the support of Π.
A careful construction of the distributions Qm that involves P0 , guarantees that the posterior
satisfies,

Π({Qm }|X n ) P0 -a.s.
−−−−→ 0,
Π({Qm+1 }|X n )

that is, posterior mass is shifted further out into the tail as n grows to infinity, forcing all
posterior mass that resides in {Qm : m ≥ 1} into arbitrarily small neighbourhoods of Q. In
a second step, the distributions Pm and prior weights αm are chosen such that the likelihood
at Pm grows large for high values of m and small for lower values as n increases, so that the
posterior mass in {Pm : m ≥ 1} also accumulates in the tail. However, the prior weights αm
may be chosen to decrease very fast with m, in such a way that,
Π({Pm : m ≥ 1}|X n ) P0 -a.s.
−−−−→ 0,
Π({Qm : m ≥ 1}|X n )
thus forcing all posterior mass into {Qm : m ≥ 1} as n grows. Combination of the previous
two displays leads to the conclusion that for every neighbourhood UQ of Q,
P -a.s.

Π(UQ |X n ) −−0−−→ 1,
so the posterior is inconsistent. Other choices of the weights αm that place more prior mass
in the tail do lead to consistent posterior distributions.



Some objected to Freedman’s counterexample, because knowledge of P0 is required to construct
the prior that causes inconsistency. So it was possible to argue that Freedman’s counterexample
amounted to nothing more than a demonstration that unfortunate circumstances could be
created, probably not a fact of great concern in any generic sense. To strengthen Freedman’s
point one would need to construct a prior of full support without explicit knowledge of P0 .
Example B.2 (Freedman (1965) [30])
In the setting of example B.1, denote the space of all distributions on Λ by π(Λ). Note that
since Λ is Polish, so is π(Λ) and so is the product Λ × π(Λ).
Theorem B.3 (Freedman (1965) [30])
Let X1 , X2 , . . . form an sample of i.i.d.-P0 random integers, let Λ denote the space of all
distributions on N and let π(Λ) denote the space of all Borel probability measures on Λ, both
in Prohorov’s weak topology. The set of pairs (P0 , Π) ∈ Λ × π(Λ) such that for all open U ⊂ Λ,
lim sup P0n Π(U |X n ) = 1,
n→∞

is residual.
And so, the set of pairs (P0 , Π) ∈ Λ × π(Λ) for which the limiting behaviour of the posterior
is acceptable to the frequentist, is meagre in Λ × π(Λ). The proof relies on the following
construction: for k ≥ 1, define Λk to be the subset of all probability distributions P on N such
26

that P (X = k) = 0. Also define Λ0 as the union of all Λk , (k ≥ 1). Pick Q ∈ Λ \ Λ0 . We
assume that P0 ∈ Λ \ Λ0 and P0 6= Q. Place a prior Π0 on Λ0 and choose Π = 12 Π0 + 12 δQ .
Because Λ0 is dense in Λ, priors of this type have full support in Λ. But P0 has full support in
N so for every k ∈ N, P0∞ (∃m≥1 : Xm = k) = 1: note that if we observe Xm = k, the likelihood
equals zero on Λk so that Π(Λk |X n ) = 0 for all n ≥ m, P0∞ -almost-surely. Freedman shows
this eliminates all of Λ0 asymptotically, if Π0 is chosen in a suitable way, forcing all posterior
mass onto the point {Q}. (See also, Le Cam (1986) [51], section 17.7).



The question remains how Freedman’s inconsistent posteriors relate to the work presented
here. Since test sequences of exponential power exist to separate complements of weak neighbourhoods, c.f. proposition A.6, Freedman’s inconsistencies must violate the requirement of
remote contiguity in theorem 4.2.
Example B.4 As noted already, Λ is a Polish space; in particular Λ is metric and second
countable, so the subspace Λ \ Λ0 contains a countable dense subset D. For Q ∈ D, let V be
the set of all prior probability measures on Λ with finite support, of which one point is Q and
the remaining points lie in Λ0 . The proof of the theorem in [30] that asserts that the set of
consistent pairs (P0 , Π) is of the first category in Λ × π(Λ) departs from the observation that
if P0 lies in Λ \ Λ0 and we use a prior from V , then,
P -a.s.

Π({Q}|X n ) −−0−−→ 1,
(in fact, as is shown below, with P0∞ -probability one there exists an N ≥ 1 such that
Π({Q}|X n ) = 1 for all n ≥ N ). The proof continues to assert that V lies dense in π(Λ),
and, through sequences of continuous extensions involving D, that posterior inconsistency for
elements of V implies posterior inconsistency for all Π in π(Λ) with the possible exception of
a set of the first category.
From the present perspective it is interesting to view the inconsistency of elements of V in
light of the conditions of theorem 4.2. Define, for some bounded f : N → R and  > 0, two
subsets of Λ,
B = {P : |P f − P0 f | < 21 },

V = {P : |P f − P0 f | ≥ }.

Proposition A.6 asserts the existence of a uniform test sequence for B versus V of exponential
power. With regard to remote contiguity, for an element Π of V with support of order M + 1,
write,
Π = βδQ +

M
X

αm δPm ,

m=1

where β +

P

m αm

= 1 and Pm ∈ Λ0 (1 ≤ m ≤ M ). Without loss of generality, assume that 

and f are such that Q does not lie in B. Consider,
Π|B

1
dPn
(X n ) =
n
dP0
Π(B)

Z
B

M
n
dP n n
1 X
dPm
(X
)
dΠ(P
)
≤
α
(X n ).
m
dP0n
Π(B)
dP0n
m=1

27

For every 1 ≤ m ≤ M , there exists a k(m) such that Pm (X = k(m)) = 0, and the probability
of the event En that none of the X1 , . . . , Xn equal k(m) is (1 − P0 (X = k(m)))n . Note that
n /dP n (X n ) > 0.
En is also the event that dPm
0

Hence for every 1 ≤ m ≤ M and all X in an event of P0∞ -probability one, there exists an
n /dP n (X n ) = 0 for all n ≥ N . Consequently, for all X in an event
Nm ≥ 1 such that dPm
m
0
Π|B

of P0∞ -probability one, there exists an N ≥ 1 such that dPn

/dP0n (X n ) = 0 for all n ≥ N .

Therefore, condition (ii) of lemma 3.3 is not satisfied for any sequence an ↓ 0. A direct proof
that (9) does not hold for any an is also possible: given the prior Π ∈ V , define,
M
Y

φn (X n ) =

1{∃1≤i≤n :Xi =k(m)} .

m=1

Then the expectation of φn with respect to the local prior predictive distribution equals zero,
Π|B

so Pn

φn = o(an ) for any an ↓ 0. However, P0n φn (X n ) → 1, so the prior Π does not give rise
Π|B

to a sequence of prior predictive distributions (Pn

) with respect to which (P0n ) is remotely

contiguous, for any an ↓ 0.

B.2



Consistency, Bayesian tests and the Hellinger metric

Let us first consider characterization of posterior consistency in terms of the family of realvalued functions on the parameter space that are bounded and continuous.
Proposition B.5 Assume that Θ is a Hausdorff, completely regular space. The posterior is
consistent at θ0 ∈ Θ, if and only if,
Z
Pθ ,n
f (θ) dΠ(θ|X n ) −−−0−→ f (θ0 ),

(B.25)

for every bounded, continuous f : Θ → R.
Proof Assume (13). Let f : Θ → R be bounded and continuous (with M > 0 such that |f | ≤
M ). Let η > 0 be given and let U ⊂ Θ be a neighbourhood of θ0 such that |f (θ) − f (θ0 )| < η
for all θ ∈ U . Integrate f with respect to the (Pθ0 ,n -almost-surely well-defined) posterior and
to δθ0 :
Z

f (θ) dΠ(θ|X n ) − f (θ0 )
Z
Z
n
≤
|f (θ) − f (θ0 )| dΠ(θ|X ) +
|f (θ) − f (θ0 )| dΠ(θ|X n )
Θ\U

U
n

≤ 2M Π(Θ \ U |X )

+ sup |f (θ) − f (θ0 )| Π(U |X n ) ≤ η + oPθ0 ,n (1),
θ∈U

as n → ∞, so that (B.25) holds. Conversely, assume (B.25). Let U be an open neighbourhood
of θ0 . Because Θ is completely regular, there exists a continuous f : Θ → [0, 1] such that f = 1
at {θ0 } and f = 0 on Θ \ U . Then,
Z
Z
Pθ ,n
Π(U |X n ) ≥ f (θ) dΠ(θ|X n ) −−−0−→ f (θ) dδθ0 (P ) = 1.
28

Consequently, (13) holds.



Proposition B.5 is used to prove consistency of frequentist point-estimators derived from the
posterior.
Example B.6 Consider a model P of single-observation distributions P on (X , B) for i.i.d.
data (X1 , X2 , . . . , Xn ) ∼ P n , (n ≥ 1). Assume that the true distribution of the data is P0 ∈ P
and that the model topology is Prohorov’s weak topology or stronger. Then for any bounded,
continuous g : X → R, the map,
f : P → R : P 7→ (P − P0 )g(X) ,
is continuous. Assuming that the posterior is weakly consistent at P0 ,
Z
Pθ0
Πn |X n
0,
(P − P0 )g dΠ(P |X n ) −−→
P1
g − P0 g ≤

(B.26)

so posterior predictive distributions are consistent point estimators in Prohorov’s weak topology. Replacing the maps g by bounded, measurable maps X → R and assuming posterior
consistency in T1 , one proves consistency of posterior predictive distributions in T1 in exactly the same way. Taking the supremum over measurable g : X → [0, 1] in (B.26) and
assuming that the posterior is consistent in the total variational topology, posterior predictive
distributions are consistent in total variation as frequentist point estimators.



The vast majority of non-parametric applications of Bayesian methods in the literature is
based on the intimate relation that exists between testing and the Hellinger metric (see [51],
section 16.4). Proofs concerning posterior consistency or posterior convergence at a rate rely
on the existence of tests for small parameter subsets Bn surrounding a point θ0 ∈ Θ, versus the
complements Vn of neighbourhoods of the point θ0 . The building block in such constructions
is the following application of the minimax theorem.
Proposition B.7 (Minimax Hellinger tests)
Consider a model P of single-observation distributions P for i.i.d. data. Let B, V ⊂ P be
convex with H(B, V ) > 0. There exists a test sequence (φn ) such that,
sup P n φn ≤ e−nH
P ∈B

2 (B,V

)

, sup Qn (1 − φn ) ≤ e−nH

2 (B,V

)

.

Q∈V

Proof This is an application of the minimax theorem. See Le Cam (1986) [51], section 16.4
for details.



Questions concerning consistency require the existence of tests in which at least one of the two
hypotheses is a non-convex set, typically the complement of a neighbourhood. Imposing the
model P to be of bounded entropy with respect to the Hellinger metric allows construction
of such tests, based on the uniform tests of proposition B.7. Below, we apply well-known
constructions for the uniform tests in Schwartz’s theorem from the frequentist literature [49,
29

7, 8, 34] to the construction of Bayesian tests. Due to relations that exist between metrics for
model parameters and the Hellinger metric in many examples and applications, the material
covered here is widely applicable in (non-parametric) models for i.i.d. data.
Example B.8 Consider a model P of distributions P for i.i.d. data X n ∼ P n , (n ≥ 1) and,
in addition, suppose that P is totally bounded with respect to the Hellinger distance. Let
P0 ∈ P and  > 0 be given, denote V () = {P ∈ P : H(P0 , P ) ≥ 4}, BH () = {P ∈ P :
H(P0 , P ) < }. There exists an N () ≥ 1 and a cover of V () by H-balls V1 , . . . , VN () of
radius  and for any point Q in any Vi and any P ∈ BH (), H(Q, P ) > 2. According to
proposition 2.6 with α = 1/2 and (7), for each 1 ≤ i ≤ N () there exists a Bayesian test
sequence (φi,n ) for BH () versus Vi of power (upper bounded by) exp(−2n2 ). Then, for any
subset B 0 ⊂ BH (),
N ()
0

PnΠ|B Π(V |X n ) ≤

X

0

PnΠ|B Π(Vi |X n )

i=1
N ()Z

1 X
P n φn dΠ(P ) +
Π(B 0 )
0
B
i=1
s
N ()
X Π(Vi )
≤
exp(−2n2 ),
Π(B 0 )

≤

Z


P n (1 − φn ) dΠ(P )

(B.27)

Vi

i=1

2

which is smaller than or equal to e−n for large enough n. If  = n with n ↓ 0 and n2n → ∞,
and the model’s Hellinger entropy is upper-bounded by log N (n , P, H) ≤ Kn2n for some
K > 0, the construction extends to tests that separate Vn = {P ∈ P : H(P0 , P ) ≥ 4n } from
Bn = {P ∈ P : H(P0 , P ) < n } asymptotically, with power exp(−nL2n ) for some L > 0. (See
also the so-called Le Cam dimension of a model [49] and Birgé’s rate-oriented work [7, 8].) It
is worth pointing out at this stage that posterior inconsistency due to the phenomenon of ‘data
tracking’ [4, 71], whereby weak posterior consistency holds but Hellinger consistency fails, can
only be due to failure of the testing condition in the Hellinger case.
Note that the argument also extends to models that are Hellinger separable: in that case (B.27)
remains valid, but with N () = ∞. The mass fractions Π(Vi )/Π(B 0 ) become important (we
point to strong connections with Walker’s theorem [70, 72]). Here we see the balance between
prior mass and testing power for Bayesian tests, as intended by the remark that closes the
subsection on the existence of Bayesian test sequences in section 2.



To balance entropy and prior mass differently in Hellinger separable models, Barron (1988)
[3] and Barron et al. (1999) [4] formulate an alternative condition that is based on the Radon
property that any prior on a Polish space has.
Example B.9 Consider a model P of distributions P for i.i.d. data X n ∼ P n , (n ≥ 1), with
priors (Πn ). Assume that the model P is Polish in the Hellinger topology. Let P0 ∈ P and
 > 0 be given; for a fixed M > 1, define V = {P ∈ P : H(P0 , P ) ≥ M }, BH = {P ∈
30

P : H(P0 , P ) < }. For any sequence δm ↓ 0, there exist compacta Km ⊂ P such that
Π(Km ) ≥ 1 − δm for all m ≥ 1. For each m ≥ 1, Km is Hellinger totally bounded so there
exists a Bayesian test sequence φm,n for BH () ∩ Km versus V () ∩ Km . Since,
Z
Z
P n φn dΠ(P ) +
Qn (1 − φn ) dΠ(Q)
BH
V
Z
Z
n
Qn (1 − φm,n ) dΠ(Q) + δm ,
P φm,n dΠ(P ) +
≤
V ∩Km

BH ∩Km

and all three terms go to zero, a diagonalization argument confirms the existence of a Bayesian
test for BH versus V . To control the power of this test and to generalise to the case where
 = n is n-dependent, more is required: as we increase m with n, the prior mass δm(n) outside
of Kn = Km(n) must decrease fast enough, while the order of the cover must be bounded: if
Πn (Kn ) ≥ 1−exp(−L1 n2n ) and the Hellinger entropy of Kn satisfies log N (n , Kn , H) ≤ L2 n2n
for some L1 , L2 > 0, there exist M > 1, L > 0, and a sequence of tests (φn ) such that,
Z
Z
2
P n φn dΠ(P ) +
Qn (1 − φn ) dΠ(Q) ≤ e−Lnn ,
BH (n )

V (n )

for large enough n. (For related constructions, see Barron (1988) [3], Barron et al. (1999) [4]
and Ghosal, Ghosh and van der Vaart (2000) [34].)
To apply corollary 4.3 consider the following steps.
Example B.10 As an example of the tests required under condition (i) of corollary 4.3, consider P in the Hellinger topology, assuming totally-boundedness. Let U be the Hellinger-ball
of radius 4 around Pθ0 of example B.8 and let V be its complement. The Hellinger ball BH ()
in equation (B.27) contains the set K(). Alternatively we may consider the model in any of
the weak topologies Tn : let  > 0 be given and let U denote a weak neighbourhood of the
form {P ∈ P : |(P n − P0n )f | ≥ 2}, for some bounded measurable f : Xn → [0, 1], as in
proposition A.6. The set B of proposition A.6 contains a set K(δ), for some δ > 0. Both these
applications were noted by Schwartz in [62].

B.3

Some examples of remotely contiguous sequences

The following two examples illustrate the difference between contiguity and remote contiguity
in the context of parametric and non-parametric regression.
Example B.11 Let F denote a class of functions R → R. We consider samples X n =
((X1 , Y1 ), . . . , (Xn , Yn )), (n ≥ 1) of points in R2 , assumed to be related through Yi = f0 (Xi )+ei
for some unknown f0 ∈ F , where the errors are i.i.d. standard normal e1 , . . . , en ∼ N (0, 1)n
and independent of the i.i.d. covariates X1 , . . . , Xn ∼ P n , for some (ancillary) distribution P on
R
R. It is assumed that F ⊂ L2 (P ) and we use the L2 -norm kf k2P,2 = f 2 dP to define a metric d
on F , d(f, g) = kf −gkP,2 . Given a parameter f ∈ F , denote the sample distributions as Pf,n .
31

We distinguish two cases: (a) the case of linear regression, where F = {fθ : R → R : θ ∈ Θ},
where θ = (a, b) ∈ Θ = R2 and fθ (x) = ax + b; and (b) the case of non-parametric regression,
where we do not restrict F beforehand.
Let Π be a Borel prior Π on F and place remote contiguity in context by assuming, for the
moment, that for some ρ > 0, there exist 0 < r < ρ and τ > 0, as well as Bayesian tests φn
for B = {f ∈ F : kf − f0 kP,2 < r} versus V = {f ∈ F : kf − f0 kP,2 ≥ ρ} under Π of power
an = exp(− 21 nτ 2 ). If this is the case, we may assume that r < 12 τ without loss of generality.
Suppose also that Π has a support in L2 (P ) that contains all of F .
Let us concentrate on case (b) first: a bit of manipulation casts the an -rescaled likelihood ratio
for f ∈ F in the following form,
a−1
n

1 Pn
2
2
dPf,n
(X n ) = e− 2 i=1 (ei (f −f0 )(Xi )+(f −f0 ) (Xi )−τ ) ,
dPf0 ,n

(B.28)

under X n ∼ Pf0 ,n . The exponent is controlled by the law of large numbers,
n

 Pf∞0 -a.s.
1X
−−→ kf − f0 k2P,2 − τ 2 .
ei (f − f0 )(Xi ) + (f − f0 )2 (Xi ) − τ 2 −−−
n
i=1

Hence, for every  > 0 there exists an N (f, ) ≥ 1 such that the exponent in (B.28) satisfies
the upper bound,
n
X


ei (f − f0 )(Xi ) + (f − f0 )2 (Xi ) − τ 2 ≤ n(kf − f0 k2P,2 − τ 2 + 2 ),

i=1

for all n ≥ N (f, ). Since Π(B) > 0, we may condition Π on B, choose  = 21 τ and use Fatou’s
inequality to find that,
1 2 dPnΠ|B
2
lim inf e nτ
(X n )
n→∞
dPf0 ,n

1

2

≥ lim inf e 4 nτ = ∞,
n→∞

Pf∞
-almost-surely. Consequently, for any choice of δ,
0

Pf0 ,n
1

Π|B

1 2
dPn
(X n ) < δ e− 2 nτ
dPf0 ,n

2

Π|B

and we conclude that Pf0 ,n C e 2 nτ Pn


→ 0,

. Based on theorem 4.2, we conclude that,

 Pf ,n
Π kf − f0 kP,2 < ρ X n −−−0−→ 1,
i.e. posterior consistency for the regression function in L2 (P )-norm obtains.



Example B.12 As for case (a), one has the choice of using a prior like above, but also to
proceed differently: expression (B.28) can be written in terms of a local parameter h ∈ Rk
which, for given θ0 and n ≥ 1, is related to θ by θ = θ0 + n−1/2 h. For h ∈ R2 , we write
Ph,n = Pθ0 +n−1/2 h,n , P0,n = Pθ0 ,n and rewrite the likelihood ratio (B.28) as follows,
Pn
dPh,n n
√1
h·` (X ,Y )− 1 h·I ·h+Rn
(X ) = e n i=1 θ0 i i 2 θ0
,
dP0,n

32

(B.29)

where `θ0 : R2 → R2 : (x, y) 7→ (y − a0 x − b0 )(x, 1) is the score function for θ, Iθ0 = Pθ0 ,1 `θ0 `Tθ0
Pθ

,n

is the Fisher information matrix and Rn −−−0−→ 0. Assume that Iθ0 is non-singular and note
the central limit,

n

Pθ0 ,n -w.
1 X
√
`θ0 (Xi , Yi ) −−−
−−→ N2 (0, Iθ0 ),
n
i=1

which expresses local asymptotic normality of the model [48] and implies that for any fixed
h ∈ R2 , Ph,n C P0,n .
Lemma B.13 Assume that the model satisfies LAN condition (B.29) with non-singular Iθ0 and
that the prior Π for θ has a Lebesgue-density π : Rd → R that is continuous and strictly positive
in all of Θ. For given H > 0, define the subsets Bn = {θ ∈ Θ : θ = θ0 + n−1/2 h, khk ≤ H}.
Then,
Π|Bn
P0,n C c−1
,
n Pn

(B.30)

for any cn ↓ 0.
Proof According to lemma 3 in section 8.4 of Le Cam and Yang (1990) [53], Pθ0 ,n is conΠ|Bn

tiguous with respect to Pn

. That implies the assertion.

√
Note that for some K > 0, Π(Bn ) ≥ bn := K(H/ n)d . Assume again the existence of
Bayesian tests for V = {θ ∈ Θ : kθ − θ0 k > ρ} (for some ρ > 0) versus Bn (or some B
such that Bn ⊂ B), of power an = exp(− 12 nτ 2 ) (for some τ > 0). Then an b−1
n = o(1),
Pθ

,n

and, assuming (B.30), theorem 4.4 implies that Π(kθ − θ0 k > ρ|X n ) −−−0−→ 0, so consistency is
straightforwardly demonstrated.
The case becomes somewhat more complicated if we are interested in optimality of parametric
rates: following the above, a logarithmic correction arises from the lower bound Π(Bn ) ≥
√
K(H/ n)d when combined in the application of theorem 4.4. To alleviate this, we adapt the
construction somewhat: define Vn = {θ ∈ Θ : kθ − θ0 k ≤ Mn n−1/2 } for some Mn → ∞ and
Bn like above. Under the condition that there exists a uniform test sequence for any fixed
V = {θ ∈ Θ : kθ − θ0 k > ρ} versus Bn (see, for example, [45]), uniform test sequences for Vn
0

2

versus Bn of power e−K Mn exist, for some k 0 > 0. Alternatively, assume that the Hellinger
distance and the norm on Θ are related through inequalities of the form,
K1 kθ − θ0 k ≤ H(Pθ , Pθ0 ) ≤ K2 kθ − θ0 k,
for some constants K1 , K2 > 0. Then cover Vn with rings,


(Mn + k − 1)
(Mn + k)
√
√
Vn,k = θ ∈ Vn :
≤ kθ − θ0 k ≤
,
n
n
for k ≥ 1 and cover each ring with balls Vn,k,l of radius n−1/2 , where 1 ≤ l ≤ Ln,k and Ln,k the
minimal number of radius-n−1/2 balls needed to cover Vn,k , related to the Le Cam dimension

33

[49]. With the Bn defined like above, and the inequality,
Z
Pθ,n Π(Vn,k,l |X n ) dΠn (θ|Bn )
≤ sup Pθ,n φn,k,l +
θ∈Bn

Πn (Vn,k,l )
sup Pθ,n (1 − φn,k,l ),
Πn (Bn ) θ∈Vn,k,l

where the φn,k,l are the uniform minimax tests for Bn versus Vn,k,l of lemma B.7, of power
exp(−K 0 (Mn + k − 1)2 ) for some K 0 > 0. We define φn,k = max{φn,k,l : 1 ≤ l ≤ Ln,k } for Vn,k
versus Bn and note,
Z

Πn (Vn,k )  −K(Mn +k−1)2
e
,
Pθ,n Π(Vn,k |X n ) dΠn (θ|Bn ) ≤ Ln,k +
Πn (Bn )
where the numbers Ln,k are upper bounded by a multiple of (Mn +k)d and the fraction of prior
masses Πn (Vn,k )/Πn (Bn ) can be controlled without logarithmic corrections when summing over
k next.



But remote contiguity also applies in more irregular situations: example 1.3 does not admit
KL priors, but satisfies the requirement of remote contiguity. (Choose η equal to the uniform
density for simplicity.)
Example B.14 Consider X1 , X2 , . . . that form an i.i.d. sample from the uniform distribution
on [θ, θ + 1], for unknown θ ∈ R. The model is parametrized in terms of distributions Pθ
with Lebesgue densities of the form pθ (x) = 1[θ,θ+1] (x), for θ ∈ Θ = R. Pick a prior Π on Θ
with a continuous and strictly positive Lebesgue density π : R → R and, for some rate δn ↓ 0,
choose Bn = (θ0 , θ0 + δn ). Note that for any α > 0, there exists an N ≥ 1 such that for all
n ≥ N , (1 − α)π(θ0 )δn ≤ Π(Bn ) ≤ (1 + α)π(θ0 )δn . Note that for any θ ∈ Bn and X n ∼ Pθn0 ,
dPθn /dPθn0 (X n ) = 1{X(1) > θ}, and correspondingly,
Π|B

dPn n n
(X ) = Πn (Bn )−1
dPθn0
≥

Z

θ0 +δn

1{X(1) > θ} dΠ(θ)
θ0

1 − α δn ∧ (X(1) − θ0 )
,
1+α
δn

for large enough n. As a consequence, for every δ > 0 and all an ↓ 0,
Pθn0



Π|B

dPn n n
(X ) < δ an
dPθn0




≤ Pθn0 δn−1 (X(1) − θ0 ) < (1 + α)δ an ,

for large enough n ≥ 1. Since n(X(1) − θ0 ) has an exponential weak limit under Pθn0 , we choose
Π |Bn

n
δn = n−1 , so that the r.h.s. in the above display goes to zero. So Pθ0 ,n C a−1
n Pn

an ↓ 0.

, for any


To show consistency and derive the posterior rate of convergence in example 1.3, we use
theorem 4.4.

34

Example B.15 Continuing with example B.14, we define Vn = {θ : θ − θ0 > n }. It is noted
that, for every 0 < c < 1, the likelihood ratio test,
φn (X n ) = 1{dPθ0 +n ,n /dPθ0 ,n (X n ) > c} = 1{X(1) > θ0 + n },
satisfies Pθn (1 − φn )(X n ) = 0 for all θ ∈ Vn , and if we choose δn = 1/2 and n = Mn /n for
some Mn → ∞, Pθn φn ≤ e−Mn +1 for all θ ∈ Bn , so that,
Z
Z
Pθn φn ( dΠ(θ) +
Pθn (1 − φn ) dΠ(θ) ≤ Π(Bn ) e−Mn +1 ,
Bn

Vn
Π|Bn

Using lemma 2.2, we see that Pn

Π(Vn |X n ) ≤ e−Mn +1 . Based on the conclusion of exam-

ple B.14 above, remote contiguity implies that Pθn0 Π(Vn |X n ) → 0. Treating the case θ < θ0 −n
similarly, we conclude that the posterior is consistent at (any n slower than) rate 1/n.
To conclude, we demonstrate the relevance of priors satisfying the lower bound (16). Let us
repeat lemma 8.1 in [34], to demonstrate that the sequence (P0n ) is remotely contiguous with
respect to the local prior predictive distributions based on the Bn of example 4.6.
Lemma B.16 For all n ≥ 1, assume that (X1 , X2 , . . . , Xn ) ∈ X n ∼ P0n for some P0 ∈ P
and let n ↓ 0 be given. Let Bn be as in example 4.6. Then, for any priors Πn such that
Πn (Bn ) > 0,
Z
Pθ0 ,n

dPθn
2
(X n ) dΠn (θ|Bn ) < e−cnn
dPθn0


→ 0,

for any constant c > 1.

B.4

The sparse normal means problem

For an example of consistency in the false-detection-rate (FDR) sense, we turn to the most
prototypical instance of sparsity, the so-called sparse normal means problem: in recent years
various types of priors have been proposed for the Bayesian recovery of a nearly-black vector
in the Gaussian sequence model. Most intuitive in this context is the class of spike-and-slab
priors [57], which first select a sparse subset of non-zero components and then draws those
from a product distribution. But other proposals have also been made, e.g. the horseshoe prior
[14], a scale-mixture of normals. Below, we consider FDR-type consistency with spike-and-slab
priors.
Example B.17 Estimation of a nearly-black vector of locations in the Gaussian sequence
model is based on n-point samples X n = (X1 , . . . , Xn ) assumed distributed according to,
Xi = θi + εi ,

(B.31)

(for all 1 ≤ i ≤ n), where ε1 , . . . , εn form an i.i.d. sample of standard-normally distributed
errors. The parameter θ is a sequence (θi : i ≥ 1) in R, with n-dimensional projection
35

θn = (θ1 , . . . , θn ), for every n ≥ 1. The corresponding distributions for X n are denoted Pθ,n
for all n ≥ 1.
Denoting by pn the number of non-zero components of the vector θn = (θ1 , . . . , θn ) ∈ Rn ,
sparsity is imposed through the assumption that θ is nearly black, that is, pn → ∞, but
pn = o(n) as n → ∞. For any integer 0 ≤ p ≤ n, denote the space of n-dimensional vectors
θn with exactly p non-zero components by `0,n (p). For later reference, we introduce, for every
subset S of In := {1, . . . , n}, the space RnS := {θn ∈ Rn : θi = 1{i ∈ S}θi , 1 ≤ i ≤ n}.
Popular sub-problems concern selection of the non-zero components [10] and (subsequent)
minimax-optimal estimation of the non-zero components [25] (especially with the LASSO in
related regression problems, see, for example, [75]). Many authors have followed Bayesian
approaches; for empirical priors, see [44], and for hierarchical priors, see [15] (and references
therein).
As n grows, the minimax-rate at which the L2 -error for estimation of θn grows, is bounded in
the following, sparsity-induced way [24],
inf

sup

θ̂n

θn ∈`0,n (pn )

Pθ,n θ̂n − θn

2
2,n

≤ 2pn log

n
(1 + o(1)),
pn

as n → ∞, where θ̂n runs over all estimators for θn .
A natural proposal for a prior Π for θ [57] (or rather, priors Πn for all θn (n ≥ 1)), is to draw
a sparse θn hierarchically [15]: given n ≥ 1, first draw p ∼ πn (for some distribution πn on
{0, 1, . . . , n}), then draw a subset S of order p from {1, . . . , n} uniformly at random, and draw
θn by setting θi = 0 if i 6∈ S and (θi : i ∈ S) ∼ Gp , for some distribution G on (all of) R.
The components of θn can therefore be thought of as having been drawn from a mixture of a
distribution degenerate at zero (the spike) and a full-support distribution G (the slab).
To show that methods presented in this paper also apply in complicated problems like this, we
give a proof of posterior convergence in the FDR sense. We appeal freely to useful results that
appeared elsewhere, in particular in [15]: we adopt some of Castillo and van der Vaart’s more
technical steps to reconstitute the FDR-consistency proof based on Bayesian testing and remote
contiguity: to compare, the testing condition and prior-mass lower bound of theorem 4.4 are
dealt with simultaneously, while the remote contiguity statement is treated separately. (We
stress that only the way of organising the proof, not the result is new. In fact, we prove only
part of what [15] achieves.)
Assume that the data follows (B.31) and denote by θ0 the true vector of normal means. For
each n ≥ 1, let pn (respectively p) denote number of non-zero components of θ0n (respectively
θn ). We do not assume that the true degree of sparsity pn is fully known, but for simplicity
and brevity we assume that there is a known sequence of upper bounds qn , such that for some
constant A > 1, pn ≤ qn ≤ A pn , for all n ≥ 1. (Indeed, theorem 2.1 in [15] very cleverly shows

36

that if G has a second moment and the prior density for the sparsity level has a tail that is
slim enough, then the posterior concentrates on sets of the form, {θn ∈ Rn : p ≤ Apn } under
P0 , for some A > 1.)
Set rn2 = pn log(n/pn ) and define two subsets of Rn ,

Vn = θn : p ≤ Apn , kθn − θ0n k2,n > M rn ,

Bn = θn : kθn − θ0n k2,n ≤ d rn , ,
assuming for future reference that Π(Bn ) > 0. As for Vn , we split further: define, for all j ≥ 1,

Vn,j = θn ∈ Vn : jM rn < kθn − θ0n k2,n ≤ (j + 1)M rn .
Next, we subdivide Vn,j into intersections with the spaces RnS for S ⊂ In : we write Vn,j =
∪{Vn,S,j : S ⊂ In } with Vn,S,j = Vn,j ∩ RnS . For every n ≥ 1, j ≥ 1 and S ⊂ In , we cover Vn,S,j
by Nn,S,j L2 -balls Vn,j,S,i of radius 12 jM rn and centre points θj,S,i . Comparing the problem of
covering Vn,j with that of covering Vn,1 , one realizes that Nn,S,j ≤ Nn,S := Nn,S,1 .
Fix n ≥ 1. Due to lemma 2.2, for any test sequences φn,j,S,i ,
PnΠ|Bn Π(Vn |X n )

≤

n,S,j
X X NX

PnΠ|Bn Π(Vn,j,S,i |X n )

j≥1 S⊂In i=1
Nn,S,j

≤

1 X X X
Π(Bn )
j≥1 S⊂In i=1
Z
Z
Pθ,n φn,j,S,i dΠ(θ) +
Bn

≤

Apn 
X
p=0

Pθ,n (1 − φn,j,S,i ) dΠ(θ)



Vn,j,S,i

n X
an (j)
,
Nn,S
p
bn
j≥1

where bn := Π(Bn ), an (j) := maxS⊂In ,1≤i≤Nn,S,j an (j, S, i) and,
an (j, S, i)
=
bn

Z

Pθ,n φn,j,S,i dΠ(θ|Bn )
Z
Π(Vn,j,S,i )
+
Pθ,n (1 − φn,j,S,i ) dΠ(θ|Vn,j,S,i )
Π(Bn )
Π(Vn,j,S,i )
≤ sup Pθ,n φn,j,S,i +
sup Pθ,n (1 − φn,j,S,i ).
n
Π(Bn ) θn ∈Vn,j,S,i
θ ∈Bn

A standard argument (see lemma 5.1 in [15]) shows that there exists a test φn,j,S,i such that,
P0,n φn,j,S,i +

Π(Vn,j,S,i )
sup Pθ,n (1 − φn,j,S,i )
Π(Bn ) θn ∈Vn,j,S,i
s
Π(Vn,j,S,i ) − 1 j 2 M 2 pn log(n/pn )
≤2
e 128
Π(Bn )

37

Note that for every measurable 0 ≤ φ ≤ 1, Cauchy’s inequality implies that, for all θ, θ0 ∈ Rn
Pθ,n φ ≤ Pθ0 ,n φ2

1/2

Pθ0 ,n (dPθ,n /dPθ0 ,n )2

1/2

≤ Pθ0 ,n φ

1/2

1

0 2

e 2 kθ−θ k2,n

(B.32)

We use this to generalise the first term in the above display to the test uniform over Bn at the
expense of an extra factor, that is,
sup Pθ,n φn,j,S,i +

θn ∈Bn

Π(Vn,j,S,i )
sup Pθ,n (1 − φn,j,S,i )
Π(Bn ) θn ∈Vn,j,S,i
s
Π(Vn,j,S,i ) − 1 j 2 M 2 rn2 + 1 d2 rn2
2
e 256
≤2
Π(Bn )

In what appears to be one of the essential (and technically very demanding) points of [15], the
proofs of the lemma 5.4 (only after the first line) and of proposition 5.1 show that there exists
a constant K > 0 such that,
s

Π(Vn,j,S,i )
2
≤ eKrn ,
Π(Bn )

if G has a Lebesgue density g : R → R such that there exists a constant c > 0 such that
| log g(θ) − log g(θ0 )| ≤ c(1 + |θ − θ0 |) for all θ, θ0 ∈ R. This allows for demonstration that (see
the final argument in the proof of proposition 5.1 in [15]) if we choose M > 0 large enough,
there exists a constant K 0 > 0 such that for large enough n,
0 2

PnΠ|Bn Π(Vn |X n ) ≤ e−K rn .
Remote contiguity follows from (B.32): fix some n ≥ 1 and note that for any θn ∈ Bn ,
(P0,n φ)2 ≤ ed

2 r2
n

Pθ,n φ.

Integrating with respect to Π(·|Bn ) on both sides shows that,
P0,n φ ≤ e
2 r2
n

so that P0,n C ed

Π|Bn

Pn

d2
2 rn (PnΠ|Bn φ)1/2 ,

. So if we choose d2 < K 0 , remote contiguity guarantees that

P0,n Π(Vn |X n ) → 0.

B.5

Goodness-of-fit Bayes factors for random walks

Consider the asymptotic consistency of goodness-of-fit tests for the transition kernel of a
Markov chain with posterior odds or Bayes factors. Bayesian analyses of Markov chains on
a finite state space are found in [66] and references therein. Consistency results c.f. [70] for
random walk data are found in [36]. Large-deviation results for posterior distributions are
derived in [59, 27]. The examples below are based on ergodicity for remote contiguity and
Hoeffding’s inequality for uniformly ergodic Markov chains [56, 37] to construct suitable tests.
We first prove the analogue of Schwartz’s construction in the case of an ergodic random walk.
38

Let (S, S ) denote a measurable state space for a discrete-time, stationary Markov process P
describing a random walk X n = {Xi ∈ S : 0 ≤ i ≤ n} of length n ≥ 1 (conditional on a
starting position X0 ). The chain has a Markov transition kernel P (·|·) : S × S → [0, 1] that
describes Xi |Xi−1 for all i ≥ 1.
Led by Pearson’s approach to goodness-of-fit testing, we choose a finite partition α = {A1 , . . . , AN }
of S and ‘bin the data’ in the sense that we switch to a new process Z n taking values in the
finite state space Sα = {ej : 1 ≤ j ≤ N } (where ej denotes the j-th standard basis vector
in Rn ), defined by Z n = {Zi ∈ Sα : 0 ≤ i ≤ n}, with Zi = (1{Xi ∈ A1 }, . . . , 1{Xi ∈ AN }).
The process Z n forms a stationary Markov chain on Sα with distribution Pα,n . The model is
parametrized in terms of the convex set Θ of N × N Markov transition matrices pα on the
finite state space Sα ,
pα (k|l) = Pα,n (Zi = ek |Zi−1 = el ) = P (Xi ∈ Ak |Xi−1 ∈ Al ),

(B.33)

for all 0 ≤ i ≤ n and 1 ≤ k, l ≤ N . We assume that Pα,n is ergodic with equilibrium
distribution that we denote by πα , and πα (k) := πα (Z = k). We are interested in Bayes
factors for goodness-of-fit type questions, given a parameter space consisting of transition
matrices.
Example B.18 Assume that the true transition kernel P0 gives rise to a matrix p0 ∈ Θ that
generates an ergodic Markov chain Z n . Denote the true distribution of Z n by P0,n and the
equilibrium distribution by π0 (with π0 (k) := π0 (Z = k)). For given  > 0, define,
N
o
n
X
pα (l|k)
< 2 .
B = pα ∈ Θ :
−p0 (l|k)π0 (k) log
p0 (l|k)
0

k,l=1

Assume that Π(B 0 ) > 0. According to the ergodic theorem, for every pα ∈ B 0 ,
n
N
1X
p(l|k)
pα (Zi |Zi−1 ) P0,n -a.s. X
log
−−−−−→
p0 (l|k)π0 (k) log
,
n
p0 (Zi |Zi−1 )
p0 (l|k)
i=1

k,l=1

(compare with the rate-function in the large-deviation results in [59, 27]) so that, for large
enough n,

n

Y pα (Zi |Zi−1 )
n 2
dPα,n n
(Z ) =
≥ e− 2  ,
dP0,n
p0 (Zi |Zi−1 )
i=1

P0,n -almost-surely. Just like in Schwartz’s proof [62], in proposition B.22 and in example B.11,
the assumption Π(B 0 ) > 0 and Fatou’s lemma imply remote contiguity because,
Z dP
n 2
α,n
P0,n
(Z n ) dΠ(pα |B 0 ) < e− 2  → 0.
dP0,n
Π|B 0

So lemma 3.3 says that P0,n C exp( n2 2 )Pn

.

However, exponential remote contiguity will turn out not to be enough for goodness-of-fit
tests below, unless we impose stringent model conditions. Instead, we shall resort to local
asymptotic normality for a sharper result.
39

Example B.19 We formulate goodness-of-fit hypotheses in terms of the joint distribution for
two consecutive steps in the random walk. Like Pearson, we fix some such distribution P0 and
consider hypotheses based on differences of ‘bin probabilities’ pα (k, l) = pα (k|l)πα (l),
H0 : max pα (k, l) − p0 (k, l) < ,
1≤k,l≤N

(B.34)

H1 : max pα (k, l) − p0 (k, l) ≥ ,
1≤k,l≤N

for some fixed  > 0. The sets B and V are defined as the sets of transition matrices pα ∈ Θ
that satisfy hypotheses H0 and H1 respectively. We assume that the prior is chosen such that
Π(B) > 0 and Π(V ) > 0.
Endowed with some matrix norm, Θ is compact and a Borel prior on Θ can be defined in
various ways. For example, we may assign the vector (pα (·|1), . . . , pα (·|N )) a product of
Dirichlet distributions. Conjugacy applies and the posterior for pα is again a product of
Dirichlet distributions [66]. For an alternative family of priors, consider the set E of N N
N × N -matrices E that have standard basis vectors ek in RN as columns. Each E ∈ E is a
deterministic Markov transition matrix on Sα and E is the extremal set of the polyhedral set
Θ. According to Choquet’s theorem, every transition matrix pα can then be written in the
form,
pα =

X

λE E,

(B.35)

E∈E

for a (non-unique) combination of λE := {λE : E ∈ E } such that λE ≥ 0,

P

E

λE = 1. If

λE > 0 for all E ∈ E , the resulting Markov chain is ergodic and we denote the corresponding
distributions for Z n by Pα,n . Any Borel prior Π0 (e.g. a Dirichlet distribution) on the simplex
SN N in RN

N

is a prior for λE and induces a Borel prior Π on Θ. Note that all non-ergodic

transition matrices lie in the boundary ∂Θ, so if we choose Π0 such that Π(Θ̊) = 1, ergodicity
may be assumed in all prior-almost-sure arguments. This is true for any Π0 that is absolutely
continuous with respect to the (N N − 1-dimensional) Lebesgue measure on SN N (for example
when we choose Π0 equal to a Dirichlet distribution). Note that if the associated density is
continuous and strictly positive, Π(B) > 0 and Π(V ) > 0.
We intend to use theorem 4.8 with B and V defined by H0 and H1 , so we first demonstrate that
a Bayesian test sequence for B versus V exists, based on a version of Hoeffding’s inequality
valid for random walks [37]. First, define, for given 0 < λn ≤ N −N such that λn ↓ 0,

N
Sn0 := λE ∈ S N : λE ≥ λn /N N −1 , for all E ∈ E ,
and denote the image of Sn0 under (B.35) by Sn . Note that if Π(∂Θ) = 0, then πS,n :=
Π(Θ \ Sn ) → 0.
Now fix n ≥ 1 for the moment. Recalling the nature of the matrices E, we see that for every
1 ≤ k, l ≤ N , pα (k|l) as in equation (B.35) is greater than or equal to λn . Consequently,
the corresponding Markov chain satisfies condition (A.1) of Glynn and Ormoneit [37] (closely
40

related to the notion of uniform ergodicity [56]): starting in any point X0 under a transition
from Sn , the probability that X1 lies in A ⊂ Sα is greater than or equal to λn φ(A), where φ is
the uniform probability measure on Sα . This mixing condition enables a version of Hoeffding’s
inequality (see theorem 2 in [37]): for any λE ∈ Sn0 and 1 ≤ k, l ≤ N , the transition matrix of
P
equation (B.35) is such that, with p̂n (k, l) = n−1 i 1{Zi = k, Zi−1 = l},
 λ2 (nδ − 2λ−1 )2 

n
.
Pα,n p̂n (k, l) − pα (k, l) ≥ δ ≤ exp − n
2n

(B.36)

Now define for a given sequence δn > 0 with δn ↓ 0 and all n ≥ 1, 1 ≤ k, l ≤ N ,
Bn = {pα ∈ Θ : max pα (k, l) − p0 (k, l) <  − δn },
k,l

Vk,l = {pα ∈ Θ : pα (k, l) − p0 (k, l) ≥ },
V+,k,l,n = {pα ∈ Θ : pα (k, l) − p0 (k, l) ≥  + δn },
V−,k,l,n = {pα ∈ Θ : pα (k, l) − p0 (k, l) ≤ − − δn }.
N

Note that if Π0 is absolutely continuous with respect to the Lebesgue measure on S N , then
πB,n := Π(B \ Bn ) → 0 and πn,k,l := Π(Vk,l \ (V+,k,l,n ∪ V−,k,l,n )) → 0.
If we define the test φ+,k,l,n (Z n ) = 1{p̂n (k, l) − p0 (k, l) ≥ }, then for any pα ∈ Bn ∩ Sn ,
Pα,n φ+,k,l,n (Z n ) ≤ Pα,n p̂n (k, l) − pα (k, l) ≥ δn
 λ2 (nδ − 2λ−1 )2 
n
n
≤ exp − n
.
2n



If on the other hand, pα lies in the intersection of V+,n,k,l with Sn , we find,
Pα,n (1 − φ+,n,k,l (Z n )) = Pα,n p̂n (k, l) − pα (k, l) < −δn
 λ2 (nδ − 2λ−1 )2 
n
n
.
≤ exp − n
2n



Choosing the sequences δn and λn such that nδn2 λ2n → ∞, we also have λ−1
n = o(nδn ), so the
exponent on the right is smaller than or equal to − 81 nλ2n δn2 .

41

So if we define φn (Z n ) = maxk,l {φ−,k,l,n (Z n ), φ+,k,l,n (Z n )},
Z
Z
Pα,n φn dΠ(pα ) +
Qα,n (1 − φn ) dΠ(qα )
B
V
Z
Z
≤
Pα,n φn dΠ(pα ) +
Qα,n (1 − φn ) dΠ(qα ) + Π(Θ \ Sn )
B∩Sn
N
X

V ∩Sn

Z
≤

Pα,n (φ−,k,l,n + φ+,k,l,n ) dΠ(pα )

B k,l=1

+

N Z
X

Qα,n (1 − φ−,k,l,n ) dΠ(qα )

V−,k,l

k,l=1

Z


Qα,n (1 − φ+,k,l,n ) dΠ(qα )

+
V+,k,l

+

N
X


Π Vn,k,l \ (V+,n,k,l ∪ V+,n,k,l ) +Π(Θ \ Sn ) + Π(B \ Bn )

k,l=1

≤

1 2 2
2N 2 e− 8 nλn δn

+ πB,n + πS,n +

N
X

πn,k,l .

k,l=1

So if we choose a prior Π0 on S N

N

that is absolutely continuous with respect to Lebesgue

measure, then (φn ) defines a Bayesian test sequence for B versus V .
Because we have not imposed control over the rates at which the terms on the r.h.s. go to zero,
remote contiguity at exponential rates is not good enough. Even if we would restrict supports
of a sequence of priors such that πB,N = πS,n = πn,k,l = 0, the first term on the r.h.s. is subexponential. To obtain a rate sharp enough, we note that the chain Z n is positive recurrent,
which guarantees that the dependence pα → dPα,n /dP0,n is locally asymptotically normal
[43, 38]. According to lemma B.13, this implies that local prior predictive distributions based
on n−1/2 -neighbourhoods of p0 in Θ are cn -remotely contiguous to P0,n for any rate cn , if the
prior has full support. If we require that the prior density π 0 with respect to Lebesgue measure
on S N
that

N

is continuous and strictly positive, then we see that there exists a constant π > 0 such

π 0 (λ)

N

≥ π for all λ ∈ S N , so that for every n−1/2 -neighbourhood Bn of p0 , there exists a

K > 0 such that Π(Bn ) ≥ bn := K n−N

N /2

. Although local asymptotic normality guarantees

remote contiguity at arbitrary rate, we still have to make sure that cn → 0 in lemma B.13, i.e.
that an = o(bn ). Then the remark directly after theorem 4.8 shows that condition (ii) of said
theorem is satisfied.
The above leads to the following conclusion concerning goodness-of-fit testing c.f. (B.34).
Proposition B.20 Let X n be a stationary, discrete time Markov chain on a measurable state
space (S, S ). Choose a finite, measurable partition α of S such that the Markov chain Z n is
ergodic. Choose a prior Π0 on S N

N

absolutely continuous with respect to Lebesgue measure

with a continuous density that is everywhere strictly positive. Assume that,
(i) nλ2n δn2 / log(n) → ∞,
42

(ii) Π(B \ Bn ), Π(Θ \ Sn ) = o(n−(N

N /2)

),

(iii) maxk,l Π(Vk,l \ (V+,k,l,n ∪ V−,k,l,n )) = o(n−(N

N /2)

).

Then for any choice of  > 0, the Bayes factors Fn are consistent for H0 versus H1 .
To guarantee ergodicity of Z n one may use an empirical device, i.e. we may use an independent, finite-length realization of the random walk X n to find a partition α such that for all
1 ≤ k, l ≤ N , we observe some m-step transition from l to k. An interesting generalisation
concerns a hypothesized Markov transition kernel P0 for the process X n and partitions αn
(with projections p0,αn as in (B.33)), chosen such that αn+1 refines αn for all n ≥ 1. Bayes
factors then test a sequence of pairs of hypotheses (B.34) centred on the p0,αn . The arguments
leading to proposition B.20 do not require modification and the rate of growth Nn comes into
the conditions of proposition B.20.



Example B.19 demonstrates the enhancement of the role of the prior as intended by the remark
that closes the subsection on the existence of Bayesian test sequences in section 2: where testing
power is relatively weak, prior mass should be scarce to compensate and where testing power
is strong, prior mass should be plentiful. A random walk for which mixing does not occur
quickly enough does not give rise to (B.36) and alternatives for which separation decreases too
fast lose testing power, so the difference sets of proposition B.20 are the hard-to-test parts of
the parameter space and conditions (ii)–(iii) formulate how scarce prior mass in these parts
has to be.

B.6

Finite sample spaces and the tailfree case

Example B.21 Consider the situation where we observe an i.i.d. sample of random variables
X1 , X2 , . . . taking values in a space XN of finite order N . Writing XN as the set of integers
{1, . . . , N }, we note that the space M of all probability measures P on (XN , 2XN ) with the
total-variational metric (P, Q) 7→ kP − Qk is in isometric correspondence with the simplex,

SN = p = (p(1), . . . , p(N )) : min p(k) ≥ 0, Σi p(i) = 1 ,
k

with the metric (p, q) 7→ kp − qk = Σk |p(k) − q(k)| it inherits from RN with the L1 -norm,
when k 7→ p(k) is the density of P ∈ M with respect to the counting measure. We also define
M 0 = {P ∈ M : P ({k}) > 0, 1 ≤ k ≤ N } ⊂ M (and RN = {p ∈ SN : p(k) > 0, 1 ≤ k ≤ N } ⊂
SN ).
Proposition B.22 If the data is an i.i.d. sample of XN -valued random variables, then for
any n ≥ 1, any Borel prior Π : G → [0, 1] of full support on M , any P0 ∈ M and any ball B
around P0 , there exists an 0 > 0 such that,
1

2

P0n C e 2 n PnΠ|B ,
43

(B.37)

for all 0 <  < 0 .
Proof By the inequality kP − Qk ≤ −P log(dQ/dP ), the ball B around P0 contains all sets
of the form K() = {P ∈ M 0 : −P0 log(dP/dP0 ) < }, for some 0 > 0 and all 0 <  < 0 . Fix
such an . Because the mapping P 7→ −P0 log(dP/dP0 ) is continuous on M 0 , there exists an
open neighbourhood U of P0 in M such that U ∩ M 0 ⊂ K(). Since both M 0 and U are open
and Π has full support, Π(K()) ≥ Π(U ∩ M 0 ) > 0. With the help of example 3.2, we see that
for every P ∈ K(),
1

e 2 n

2

dP n n
(X ) ≥ 1,
dP0n

for large enough n, P0 -almost-surely. Fatou’s lemma again confirms condition (ii) of lemma 3.3
is satisfied. Conclude that assertion (B.37) holds.



Example B.23 We continue with the situation where we observe an i.i.d. sample of random
variables X1 , X2 , . . . taking values in a space XN of finite order N . For given δ > 0, consider
the hypotheses,
B = {P ∈ M : kP − P0 k < δ}, V = {Q ∈ M : kQ − P0 k > 2δ}.
Noting that M is compact (or with the help of the simplex representation SN ) one sees that
entropy numbers of M are bounded, so the construction of example B.8 shows that uniform
tests of exponential power e−nD (for some D > 0) exist for B versus V . Application of
proposition B.22 shows that the choice for an 0 <  < 0 small enough, guarantees that
Π(V |X n ) goes to zero in P0n -probability. Conclude that the posterior resulting from a prior Π
of full support on M is consistent in total variation.
Example B.24 With general reference to Ferguson (1973) [28], one way to construct nonparametric priors concerns a refining sequence of finite, Borel measurable partitions of a Polish
sample space, say X = R: to define a ‘random distribution’ P on X , we specify for each such
partition α = {A1 , . . . , AN }, a Borel prior Πα on SN , identifying (p1 , . . . , pN ) with the ‘random
variables’ (P (A1 ), . . . , P (AN )). Kolmogorov existence of the stochastic process describing all
P (A) in a coupled way subjects these Πα to consistency requirements expressing that if A1 , A2
partition A, then P (A1 ) + P (A2 ) must have the same distribution as P (A). If the partitions
refine appropriately, the resulting process describes a probability measure Π on the space of
Borel probability measures on X , i.e. a ‘random distribution’ on X . Well-known examples of
priors that can be constructed in this way are the Dirichlet process prior (for which a so-called
base-measure µ supplies appropriate parameters for Dirichlet distributions Πα , see [28]) and
Polya Tree prior (for detailed explanations, see, for example, [35]).
A special class of priors constructed in this way are the so-called tailfree priors. The process
prior associated with a family of Πα like above is said to be tailfree, if for all α, β such
that β = {B1 , . . . , BM } refines α = {A1 , . . . , AN }, the following holds: for all 1 ≤ k ≤ N ,
(P (Bl1 |Ak ), . . . , P (BlL (k) |Ak )) (where the sets Bl1 , . . . , BlL (k) ∈ β partition Ak ) is independent
44

of (P (A1 ), . . . , P (AN )). Although somewhat technical, explicit control of the choice for the
Πα render the property quite feasible in examples.
Fix a finite, measurable partition α = {A1 , . . . , AN }. For every n ≥ 1, denote by σα,n the
σ-algebra σ(αn ) ⊂ B n , generated by products of the form Ai1 × · · · × Ain ⊂ X n , with
1 ≤ i1 , . . . , in ≤ N . Identify XN with the collection {e1 , . . . , eN } ⊂ RN and define the
projection ϕα : X 7→ XN by,

ϕα (x) = 1{x ∈ A1 }, . . . , 1{x ∈ AN } .
We view XN (respectively XNn ) as a probability space, with σ-algebra σN equal to the power
set (respectively σN,n , the power set of XNn ) and probability measures denoted Pα : σN → [0, 1]
that we identify with elements of SN . Denoting the space of all Borel probability measures on
X by M 1 (X ), we also define ϕ∗α : M 1 (X ) → SN ,

ϕ∗α (P ) = P (A1 ), . . . , P (AN ) ,
which maps P to its restriction to σα,1 , a probability measure on XN . Under the projection
φα , any Borel-measurable random variable X taking values in X distributed P ∈ M 1 (X ) is
mapped to a random variable Zα = ϕα (X) that takes values in XN (distributed Pα = ϕ∗α (P )).
We also define Zαn = (ϕα (X1 ), . . . , ϕα (Xn )), for all n ≥ 1.
Let Πα denote a Borel prior on SN . The posterior on SN is then a Borel measure denoted
Πα (·|Zαn ), which satisfies, for all A ∈ σN,n and any Borel set V in SN ,
Z
Z
Πα (V |Zαn ) dPnΠα =
Pαn (A) dΠα (Pα ),
A

V

by definition of the posterior. In the model for the original i.i.d. sample X n , Bayes’s rule takes
the form, for all A0 ∈ Bn and all Borel sets V 0 in M 1 (X ),
Z
Z
Π(V 0 |X n ) dPnΠ =
P n (A0 ) dΠ(P ),
A0

V0

defining the posterior for P . Now specify that V 0 is the pre-image ϕ−1
∗α (V ) of a Borel measurable
V in SN : as a consequence of tailfreeness, the data-dependence of the posterior for such a V 0 ,
X n 7→ Π(V 0 |X n ), is measurable with respect to σα,n (see Freedman (1965) [30] or Ghosh
(2003) [35]). So there exists a function gn : XNn → [0, 1] such that,
Π(V 0 |X n = xn ) = gn (ϕα (x1 ), . . . , ϕα (xn )),
for PnΠ -almost-all xn ∈ X n . Then, for given A0 ∈ σα,n (with corresponding A ∈ σN,n ),
Z
Z
0
n
Π
Π(V |X ) dPn = P n (1A0 (X n ) Π(V 0 |X n )) dΠ(P )
0
A
Z
Z

= Pαn 1A (Zαn ) gn (Zαn ) dΠα (Pα ) =
gn (Zαn ) dPnΠα ,
A

45

while also,
Z

n

0

Z

P (A ) dΠ(P ) =
This shows that Zαn 7→

V0
gn (Zαn )

Pαn (A) dΠα (Pα ).

V

is a version of the posterior Πα ( · |Zαn ). In other words, we can

write Π(V 0 |X n ) = Πα (V |φα (X n )) = Πα (V |Zαn ), PnΠ -almost-surely.
Denote the true distribution of a single observation from X n by P0 . For any V 0 of the form
ϕ−1
∗α (V ) for some α and a neighbourhood V of P0,α = ϕ∗α (P0 ) in SN , the question whether
Π(V 0 |X n ) converges to one in P0 -probability reduces to the question whether Π(V |Zαn ) converges to one in P0,α -probability. Remote contiguity then only has to hold as in example B.21.
Another way of saying this is to note directly that, because X n 7→ Π(V 0 |X n ) is σα,n -measurable,
remote contiguity (as in definition 3.1) is to be imposed only for φn : X n → [0, 1] that are measurable with respect to σα,n (rather than B n ) for every n ≥ 1. That conclusion again reduces
the remote contiguity requirement necessary for the consistency of the posterior for the parameter (P (A1 ), . . . P (AN )) to that of a finite sample space, as in example B.21. Full support of the
prior Πα then guarantees remote contiguity for exponential rates as required in condition (ii)
of theorem 4.2. In the case of the Dirichlet process prior, full support of the base measure
µ implies full support for all Πα , if we restrict attention to partitions α = (A1 , . . . , AN ) such
that µ(Ai ) > 0 for all 1 ≤ i ≤ N . (Particularly, we require P0  µ for consistent estimation.)
Uniform tests of exponential power for weak neighbourhoods complete the proof that tailfree
priors lead to weakly consistent posterior distributions: (norm) consistency of Πα ( · |Zαn ) for
all α guarantees (weak T1 -)consistency of Π( · |X n ), in this proof based on remote contiguity
and theorem 4.2.

B.7

Credible/confidence sets in metric spaces

When enlarging credible sets to confidence sets using a collection of subsets B as in definition 4.11, measurability of confidence sets is guaranteed if B(θ) is open in Θ for all θ ∈ Θ.
Example B.25 Let G be the Borel σ-algebra for a uniform topology on Θ, like the weak and
metric topologies of appendix A. Let W denote a symmetric entourage and, for every θ ∈ Θ,
define B(θ) = {θ0 ∈ Θ : (θ, θ0 ) ∈ W }, a neighbourhood of θ. Let D denote any credible set.
A confidence set associated with D under B is any set C 0 such that the complement of D
contains the W -enlargement of the complement of C 0 . Equivalently (by the symmetry of W ),
the W -enlargement of D does not meet the complement of C 0 . Then the minimal confidence
set C associated with D is the W -enlargement of D. If the B(θ) are all open neighbourhoods
(e.g. whenever W is a symmetric entourage from a fundamental system for the uniformity on
Θ), the minimal confidence set associated with D is open. The most common examples include
the Hellinger or total-variational metric uniformities, but weak topologies (like Prohorov’s or
Tn -topologies) and polar topologies are uniform too.
46



Example B.26 To illustrate example B.25 with a customary situation, consider a parameter
space Θ with parametrization θ 7→ Pθn , to define a model for i.i.d. data X n = (X1 , . . . , Xn ) ∼
Pθn0 , for some θ0 ∈ Θ. Let D be the class of all pre-images of Hellinger balls, i.e. sets D(θ, ) ⊂ Θ
of the form,
D(θ, ) =



θ0 ∈ Θ : H(Pθ , Pθ0 ) <  ,

for any θ ∈ Θ and  > 0. After choice of a Kullback-Leibler prior Π for θ and calculation
of the posteriors, choose Dn equal to the pre-image D(θ̂n , ˆn ) of a (e.g. the one with the
smallest radius, if that exists) Hellinger ball with credible level 1 − o(an ), an = exp(−nα2 )
for some α > 0. Assume, now, that for some 0 <  < α, the W of example B.25 is the
Hellinger entourage W = {(θ, θ0 ) : H(Pθ , Pθ0 ) < }. Since Kullback-Leibler neighbourhoods
are contained in Hellinger balls, the sets D(θ̂n , ˆn + ) (associated with Dn under the entourage
W ), is a sequence of asymptotic confidence sets, provided the prior satisfies (2). If we make
 vary with n, neighbourhoods of the form Bn in example 4.6 are contained in Hellinger balls
of radius n , and in that case,
Cn (X n ) = D(θ̂n , ˆn + n ),
is a sequence of asymptotic confidence sets, provided that the prior satisfies (16).

C



Proofs

In this section of the appendix, proofs from the main text are collected.
Proof (theorem 1.1)
The argument (see, e.g., Doob (1949) [26] or Ghosh and Ramamoorthi (2003) [35]) relies on
martingale convergence and a demonstration of the existence of a measurable f : X ∞ → P
such that f (X1 , X2 , . . .) = P , P ∞ -almost-surely for all P ∈ P (see also propositions 1 and 2
of section 17.7 in [51]).



Proof (proposition 2.2)
Due to Bayes’s Rule (A.22) and monotone convergence,
Z
Pθ (1 − φ) Π(V |X) dΠ(θ)
B
Z
Z
≤ (1 − φ) Π(V |X) dP Π =
Pθ (1 − φ) dΠ(θ).
V

Inequality (4) follows from the fact that Π(V |X) ≤ 1.



Proof (theorem 2.4)
Condition (i) implies (ii) by dominated convergence. Assume (ii) and note that by lemma 2.2,
Z
Pθ,n Π(V |X n ) dΠ(θ|B) → 0.
47

Assuming that the observations X n are coupled and can be thought of as projections of
a random variable X ∈ X ∞ with distribution Pθ , martingale convergence in L1 (X ∞ × Θ)
R
(relative to the probability measure Π∗ defined by Π∗ (A×B) = B Pθ (A) dΠ(θ) for measurable
A ⊂ X ∞ and B ⊂ Θ), shows there is a measurable g : X ∞ → [0, 1] such that,
Z
Pθ Π(V |X n ) − g(X) dΠ(θ|B) → 0.
So

R

Pθ g(X) dΠ(θ|B) = 0, implying that g = 0, Pθ -almost-surely for Π-almost-all θ ∈ B.

Using martingale convergence again (now in L∞ (X ∞ × Θ)), conclude Π(V |X n ) → 0, Pθ almost-surely for Π-almost-all θ ∈ B, from which (iii) follows. Choose φ(X n ) = Π(V |X n ) to
conclude that (i) follows from (iii).



Proof (proposition 2.5)
Apply [51], section 17.1, proposition 1 with the indicator for V . See also [11].



Proof (lemma 3.3)
Assume (i). Let φn : Xn → [0, 1] be given and assume that Pn φn = o(an ). By Markov’s
inequality, for every  > 0, Pn (a−1
n φn > ) = o(1). By assumption, it now follows that
Qn

φn −−→ 0. Because 0 ≤ φn ≤ 1 the latter conclusion is equivalent to Qn φn = o(1).
Assume (iv). Let  > 0 and φn : Xn → [0, 1] be given. There exist c > 0 and N ≥ 1 such that
for all n ≥ N ,


Qn φn < c a−1
n Pn φn + .
2
If we assume that Pn φn = o(an ) then there is a N 0 ≥ N such that c a−1
n Pn φn < /2 for all
n ≥ N 0 . Consequently, for every  > 0, there exists an N 0 ≥ 1 such that Qn φn <  for all
n ≥ N 0.
To show that (ii) ⇒ (iv), let µn = Pn + Qn and denote µn -densities for Pn , Qn by pn , qn :
Xn → R. Then, for any n ≥ 1, c > 0,
Z
Z
Z

−1
Qn −Qn ∧ c an Pn k = sup
qn dµn −
qn dµn ∧
c a−1
p
dµ
n
n
n
A∈Bn
A
A
A
Z
≤ sup
(qn − qn ∧ c a−1
n pn ) dµn
A∈Bn A
Z
−1
= 1{qn > c a−1
n pn } (qn − c an pn ) dµn .

(C.38)

Note that the right-hand side of (C.38) is bounded above by Qn (dPn /dQn < c−1 an ).
To show that (iii) ⇒ (iv), it is noted that, for all c > 0 and n ≥ 1,
Z
−1
−1
0 ≤ c a−1
n Pn (qn > c an pn ) ≤ Qn (qn > c an pn ) ≤ 1,
−1
so (C.38) goes to zero if lim inf n→∞ c a−1
n Pn (dQn /dPn > c an ) = 1.

To prove that (v) ⇔ (ii), note that Prohorov’s theorem says that weak convergence of a
subsequence within any subsequence of an (dPn /dQn )−1 under Qn (see appendix A, notation
48

and conventions) is equivalent to the asymptotic tightness of (an (dPn /dQn )−1 : n ≥ 1) under
Qn , i.e. for every  > 0 there exists an M > 0 such that Qn (an (dPn /dQn )−1 > M ) <  for all
n ≥ 1. This is equivalent to (ii).



Proof (proposition 3.6)
For every  > 0, there exists a constant δ > 0 such that,

 
dPθ,n −1 n
1
(X ) >
< ,
Pθ0 ,n an
dPθ0 ,n
δ
for all θ ∈ B, n ≥ 1. For this choice of δ, condition (ii) of lemma 3.3 is satisfied for all θ ∈ B
simultaneously, and c.f. the proof of said lemma, for given  > 0, there exists a c > 0 such
that,
kPθ0 ,n − Pθ0 ,n ∧ c a−1
n Pθ,n k < ,

(C.39)

for all θ ∈ B, n ≥ 1. Now note that for any A ∈ Bn ,
Π|B
0 ≤Pθ0 ,n (A) − Pθ0 ,n (A) ∧ c a−1
(A)
n Pn
Z

≤
Pθ0 ,n (A) − Pθ0 ,n (A) ∧ c a−1
n Pθ,n (A) dΠ(θ|B).

Taking the supremum with respect to A, we find the following inequality in terms of total
variational norms,
Pθ0 ,n − Pθ0 ,n ∧

Π|B
c a−1
n Pn

Z
≤

Pθ0 ,n − Pθ0 ,n ∧ c a−1
n Pθ,n dΠ(θ|B).

Since the total-variational norm is bounded and Π(·|B) is a probability measure, Fatou’s lemma
says that,
Π|B
lim sup Pθ0 ,n − Pθ0 ,n ∧ c a−1
n Pn
n→∞
Z
≤ lim sup Pθ0 ,n − Pθ0 ,n ∧ c a−1
n Pθ,n dΠ(θ|B),
n→∞

and the r.h.s. equals zero c.f. (C.39). According to condition (iv) of lemma 3.3 this implies
the assertion.



Proof (lemma 3.7)
Fix n ≥ 1. Because Bn ⊂ Cn , for every A ∈ Bn , we have,
Z
Z
Pθ,n (A) dΠ(θ) ≤
Pθ,n (A) dΠ(θ),
Bn
Π |Bn

so Pn n
have

Cn
Π |Cn

(A) ≤ Πn (Cn )/Πn (Bn ) Pn n

Π |C
Pn n n φn (X n )

(A). So if for some sequence φn : Xn → [0, 1], we
Π |Bn

= o(Πn (Bn )/Πn (Cn )), then the Pn n

proving the first claim. If

Π |C
Pn n n φn (X n )

-expectations of φn (X n ) are o(1),
Π |Bn

= o(an Πn (Bn )/Πn (Cn )), then Pn n

o(an ) and, hence, Pn φn (X n ) = o(1).

φn (X n ) =


Proof (theorem 4.2)
Π|B

Choose Bn = B, Vn = V and use proposition 2.3 to see that Pn
49

Π(V |X n ) is upper bounded

by Π(B)−1 times the l.h.s. of (14) and, hence, is of order o(an ). Condition (ii) then implies
Pθ

,n

that Pθ0 ,n Π(V |X n ) = o(1), which is equivalent to Π(V |X n ) −−−0−→ 0 since 0 ≤ Π(V |X n ) ≤ 1,
Pθ0 ,n -almost-surely, for all n ≥ 1.



Proof (corollary 4.3)
A prior Π satisfying condition (ii) guarantees that P0n  PnΠ for all n ≥ 1, c.f. the remark
preceding proposition A.7. Choose  such that 2 < D. Recall that for every P ∈ B(), the
exponential lower bound (10) for likelihood ratios of dP n /dP0n exists. Hence the limes inferior
of exp( 21 n2 )(dP n /dP0n )(X n ) is greater than or equal to one with P0∞ -probability one. Then,
with the use of Fatou’s lemma and the assumption that Π(B()) > 0,
Z
dPθn
enD
(X n ) dΠ(θ) ≥ 1,
lim inf
n→∞ Π(B) B dP n
θ0
with Pθ∞
-probability one, showing that sufficient condition (ii) of lemma 3.3 holds. Conclude
0
that,
P0n C enD PnΠ|B ,
Pθ

,n

and use theorem 4.2 to see that Π(U |X n ) −−−0−→ 1.



Proof (theorem 4.4)
Π |Bn

Proposition 2.3 says that Pn n

Π(Vn |X n ) is of order o(b−1
n an ). Condition (iii) then implies
Pθ

,n

that Pθ0 ,n Π(Vn |X n ) = o(1), which is equivalent to Π(Vn |X n ) −−−0−→ 0 since 0 ≤ Π(Vn |X n ) ≤ 1,
Pθ0 ,n -almost-surely for all n ≥ 1.



Proof (theorem 4.12)
Fix n ≥ 1 and let Dn denote a credible set of level 1 − o(an ), defined for all x ∈ Fn ⊂ Xn
such that PnΠn (Fn ) = 1. For any x ∈ Fn , let Cn (x) denote a confidence set associated with
Dn (x) under B. Due to definition 4.11, θ0 ∈ Θ \ Cn (x) implies that Bn (θ0 ) ∩ Dn (x) = ∅.
Hence the posterior mass of B(θ0 ) satisfies Π(Bn (θ0 )|x) = o(an ). Consequently, the function
x 7→ 1{θ0 ∈ Θ \ Cn (x)} Π(B(θ0 )|x) is o(an ) for all x ∈ Fn . Integrating with respect to the n-th
prior predictive distribution and dividing by the prior mass of Bn (θ0 ), one obtains,
Z
1
an
1{θ0 ∈ Θ \ Cn } Π(Bn (θ0 )|X n ) dPnΠn ≤
.
Πn (Bn (θ0 ))
bn
Applying Bayes’s rule in the form (A.22), we see that,
Z


an
Πn |Bn (θ0 )
n
Pn
.
θ0 ∈ Θ \ Cn (X ) = Pθ,n θ0 ∈ Θ \ Cn (X n ) dΠn (θ|Bn ) ≤
bn
By the definition of remote contiguity, this implies asymptotic coverage c.f. (18).



Proof (corollary 4.13)
Define an = exp(−C 0 n2n ), bn = exp(−Cn2n ), so that the Dn are credible sets of level 1−o(an ),
2
the sets Bn of example 4.6 satisfy condition (i) of theorem 4.12 and bn a−1
n = exp(cnn ) for

some c > 0. By (17), we see that condition (ii) of theorem 4.12 is satisfied. The assertion now
follows.


50

References
[1] E. Abbe, A. Bandeira and G. Hall, Exact recovery in the stochastic block model,
arXiv:1405.3267 [cs.SI]
[2] A. Barron, Discussion on Diaconis and Freedman: the consistency of Bayes estimates.
Ann. Statist. 14 (1986), 26–30.
[3] A. Barron, The exponential convergence of posterior probabilities with implications for
Bayes estimators of density functions, Technical Report 7 (1988), Dept. Statistics, Univ.
Illinois.
[4] A. Barron, M. Schervish and L. Wasserman, The consistency of distributions in
nonparametric problems, Ann. Statist. 27 (1999), 536-561.
[5] M. Bayarri and J. Berger, The Interplay of Bayesian and Frequentist Analysis, Statist.
Sci. 19 (2004), 58–80.
[6] P. Bickel and P. Chen, A nonparametric view of network models and Newman-Girvan
and other modularities, Proc. Natl. Acad. Sci. USA 106 (2009), 21068-21073.
[7] L. Birgé, Approximation dans les espaces métriques et théorie de l’estimation, Zeitschrift
für Wahrscheinlichkeitstheorie und Verwandte Gebiete 65 (1983), 181–238.
[8] L. Birgé, Sur un théorème de minimax et son application aux tests, Probability and
Mathematical Statistics 3 (1984), 259–282.
[9] L. Birgé, and P. Massart, From model selection to adative estimation, Festschrift
for Lucien Le Cam (eds. D. Pollard, E. Torgesen, G. Yang), Springer, New York (1997),
55–87.
[10] L. Birgé, and P. Massart, Gaussian model selection, J. Eur. Math. Soc. 3 (2001),
203–268.
[11] L. Breiman, L. Le Cam and L. Schwartz, Consistent Estimates and Zero-One Sets,
Ann. Math. Statist. 35 (1964), 157–161.
[12] P. Bühlman and S. van de Geer, Statistics for High-Dimensional Data, Springer
Verlag, New York (2011).
[13] T. Cai, M. Low and Y. Xia, Adaptive confidence interval for regression functions under
shape constraints, Ann. Statist. 41 (2013), 722–750.
[14] C. Carvalho, N. Polson, and J. Scott, The horseshoe estimator for sparse signals,
Biometrika 97 (2010), 465–480.
[15] I. Castillo and A. van der Vaart, Needles and straw in a haystack: posterior concentration for possibly sparse sequences, Ann. Statist. 40 (2012), 2069–2101.
51

[16] D. Choi, P. Wolfe and E. Airoldi, Stochastic block models with a growing number
of classes, Biometrika 99 (2012), 273–284.
[17] D. Cox, An analysis of Bayesian inference for non-parametric regression, Ann. Statist.
21 (1993), 903–924.
[18] A. Decelle, F. Krzakala, C. Moore and L. Zdeborová, Inference and Phase
Transitions in the Detection of Modules in Sparse Networks, Phys. Rev. Lett. 107 (2011),
065701
[19] P. De Blasi, A. Lijoi, and I. Prünster, An asymptotic analysis of a class of discrete
nonparametric priors, Statist. Sinica 23 (2013), 1299–1322.
[20] P. Diaconis and D. Freedman, On the Consistency of Bayes Estimates, Ann. Statist.
14 (1986), 1–26.
[21] P. Diaconis and D. Freedman, On Inconsistent Bayes Estimates of Location, Ann.
Statist. 14 (1986), 68–87.
[22] P. Diaconis and D. Freedman, Nonparametric Binary Regression: A Bayesian Approach, Ann. Statist. 21 (1993), 2108–2137.
[23] P. Diaconis and D. Freedman, Consistency of Bayes estimates for nonparameteric
regression: normal theory, Bernoulli 4 (1998), 411–444.
[24] D. Donoho, I. Johnstone, J. Hoch and A. Stern, Maximum entropy and the nearly
black object, J. Roy. Statist. Soc. B54 (1992), 41–81.
[25] D. Donoho, and I. Johnstone, Minimax risk over `p -balls for `q -error, Probab. Theory
Related Fields, 99 (1994), 277-303.
[26] J. Doob, Application of the theory of martingales, Colloque international Centre nat.
Rech. Sci., Paris (1949), 22–28.
[27] P. Eichelsbacher and A. Ganesh, Bayesian inference for Markov chains, J. Appl.
Probab. 39 (2002), 91–99.
[28] T. Ferguson, A Bayesian Analysis of Some Nonparametric Problems, Ann. Statist. 1
(1973), 209–230.
[29] D. Freedman, On the asymptotic behavior of Bayes estimates in the discrete case I,
Ann. Math. Statist. 34 (1963), 1386–1403.
[30] D. Freedman, On the asymptotic behavior of Bayes estimates in the discrete case II,
Ann. Math. Statist. 36 (1965), 454–456.
[31] D. Freedman, and P. Diaconis, On Inconsistent Bayes Estimates in the Discrete Case,
Ann. Statist. 11 (1983), 1109–1118.

52

[32] D. Freedman, On the Bernstein-von Mises theorem with infinite dimensional parameters, Ann. Statist. 27 (1999), 1119–1140.
[33] S. Ghosal, J. Ghosh, and R. Ramamoorthi, Consistency issues in Bayesian nonparametrics, In Asymptotics, Nonparametrics and Time Series: A Tribute to Madan Lal
Puri (Subir Ghosh, ed.) Dekker, New York (1999), 639-667.
[34] S. Ghosal, J. Ghosh and A. van der Vaart, Convergence rates of posterior distributions, Ann. Statist. 28 (2000), 500–531.
[35] J. Ghosh and R. Ramamoorthi, Bayesian nonparametrics, Springer Verlag, New York
(2003).
[36] S. Ghosal and Y.-Q. Tang Bayesian Consistency for Markov Processes, Sankhya 68
(2006), 227–239.
[37] P. Glynn and D. Ormoneit, Hoeffding’s inequality for uniformly ergodic Markov chains,
Statist. Probab. Lett. 56 (2002), 143–146.
[38] E. Gobet, LAN property for ergodic diffusions with discrete observations, Ann. I. H.
Poincaré PR 38 (200), 711–737.
[39] P. Greenwood and A. Shiryaev, Contiguity and the statistical invariance principle,
Gordon and Breach, New York (1985).
[40] J. Hajék and Z. S̆idák, Theory of rank tests, Academic Press, New York (1967).
[41] K. Hayashi, T. Konishi and T. Kawamoto, A tractable fully Bayesian method for the
stochastic block model, arxiv:1602.02256 [cs.LG]
[42] N. Hengartner, P. Stark, Finite-sample confidence envelopes for shape-restricted densities, Ann. Statist. 23 (1995), 525–550.
[43] R. Höpfner, J. Jacod and L. Ladelli, Local asymptotic normality and mixed normality for Markov statistical models, Probab. Th. Rel. Fields (1990) 86: 105.
[44] I. Johnstone and B. Silverman, Needles and straw in haystacks: empirical Bayes
setimates of possibly sparse sequences, Ann. Statist. 32 (2004), 1594–1649.
[45] B. Kleijn and A. van der Vaart, The Bernstein-Von-Mises theorem under misspecification, Electron. J. Statist. 6 (2012), 354–381.
[46] B. Kleijn and Y.-Y. Zhao, Criteria for posterior consistency, arxiv:1308.1263
[MATH.ST].
[47] L. Le Cam and L. Schwartz, A necessary and sufficient condition for the existence of
consistent estimates, Ann. Math. Statist. 31 (1960), 140–150.

53

[48] L. Le Cam, Locally asymptotically normal families of distributions, University of California Publications in Statistics 3 (1960), 37-98.
[49] L. Le Cam, Convergence of estimates under dimensionality restrictions, Ann. Statist. 1
(1973), 38–55.
[50] L. Le Cam, An inequality concerning Bayes estimates, University of California, Berkeley
(197X), unpublished.
[51] L. Le Cam, Asymptotic methods in statistical decision theory, Springer, New York
(1986).
[52] L. Le Cam and G. Yang, On the preservation of local asymptotic normality under
information loss, Ann. Statist. 16 (1988), 483–520.
[53] L. Le Cam and G. Yang, Asymptotics in Statistics: some basic concepts, Springer, New
York (1990).
[54] A. Lijoi, I. Prünster and S. Walker, Extending Doob’s consistency theorem to nonparametric densities, Bernoulli 10 (2004), 651–663.
[55] M. Low, On nonparametric confidence intervals, Ann. Statist. 25 (1997), 2547–2554.
[56] S. Meyn and R. Tweedie, Markov Chains and Stochastic Stability, Cambridge University Press, New York (2009).
[57] T. Mitchell, and J. Beauchamp, Bayesian Variable Selection in Linear Regression, J.
Amer. Statist. Assoc. 83 (1988), 1023–1032.
[58] E. Mossel, J. Neeman and A. Sly, Consistency thresholds for the planted bisection
model, Electron. J. Probab. 21 (2016), 1–24.
[59] F. Papangelou, Large-deviations and the Bayesian estimation of higher-order Markov
chains, J. Appl. Probab. 33 (1996), 18–27.
[60] G. Roussas, Contiguity of probability measures: some applications in statistics, Cambridge Tracts in Mathematics and Mathematical Physics 63 (1972), Cambridge University
Press, London-New York.
[61] L. Schwartz, Consistency of Bayes procedures, PhD. thesis, Dept. of Statistics, University of California, Berkeley (1961).
[62] L. Schwartz, On Bayes procedures, Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 4 (1965), 10–26.
[63] X. Shen and L. Wasserman, Rates of convergence of posterior distributions, Ann.
Statist. 29 (2001), 687–714.

54

[64] K. Nowicki and T. Snijders, Estimation and Prediction for Stochastic Blockstructures,
J. Amer. Statist. Assoc. 96 (2001), 1077–1087.
[65] H. Strasser, Mathematical theory of statistics, de Gruyter, Berlin, 1985.
[66] C.Strelioff, J. Crutchfield and A. Hubler, Inferring Markov chains: Bayesian estimation, model comparison, entropy rate, and out-of-class modeling, arXiv:math/0703715
[] (2007).
[67] S. Suwan, D. Lee, R. Tang, D. Sussman, M. Tang and C. Priebe Empirical Bayes
estimation for the stochastic blockmodel, Electron. J. Statist. 10 (2016), 761–782.
[68] B. Szabó, A. van der Vaart and J. van Zanten, Frequentist coverage of adaptive
nonparametric Bayesian credible sets, Ann. Statist. 43 (2015), 1391–1428.
[69] J. Taylor and R. Tibshirani, Statistical learning and selective inference, Proc. Natl.
Acad. Sc. 112 (2016), 7629–7634.
[70] S. Walker, New approaches to Bayesian consistency, Ann. Statist. 32 (2004), 2028–2043.
[71] S. Walker, A. Lijoi and I. Prünster, Data tracking and the understanding of Bayesian
consistency, Biometrika 92 (2005), 765–778.
[72] S. Walker, A. Lijoi and I. Prünster, On rates of convergence for posterior distributions in infinite-dimensional models, Ann. Statist. 35 (2007), 738–746.
[73] L. Wasserman, Bayesian Model Selection and Model Averaging, J. Math. Psychology
44 (2000), 92-107.
[74] G. Yang, A conversation with Lucien Le Cam, Statist. Sc. 14 (1999), 223–241.
[75] C.-H. Zhang and J. Huang, The sparsity and bias of the LASSO selection in highdimensional linear regression, Ann. Statist. 36 (2008), 1567–1594.

55

