A note on conditional versus joint unconditional weak
convergence in bootstrap consistency results
Axel Bücher ∗ and Ivan Kojadinovic †

arXiv:1706.01031v4 [] 1 Mar 2018

March 5, 2018

Abstract
The consistency of a bootstrap or resampling scheme is classically validated by weak
convergence of conditional laws. However, when working with stochastic processes in the
space of bounded functions and their weak convergence in the Hoffmann-Jørgensen sense,
an obstacle occurs: due to possible non-measurability, neither laws nor conditional laws are
well-defined. Starting from an equivalent formulation of weak convergence based on the
bounded Lipschitz metric, a classical circumvent is to formulate bootstrap consistency in
terms of the latter distance between what might be called a conditional law of the (nonmeasurable) bootstrap process and the law of the limiting process. The main contribution
of this note is to provide an equivalent formulation of bootstrap consistency in the space of
bounded functions which is more intuitive and easy to work with. Essentially, the equivalent
formulation consists of (unconditional) weak convergence of the original process jointly with
two bootstrap replicates. As a by-product, we provide two equivalent formulations of bootstrap consistency for statistics taking values in separable metric spaces: the first in terms of
(unconditional) weak convergence of the statistic jointly with its bootstrap replicates, the
second in terms of convergence in probability of the empirical distribution function of the
bootstrap replicates. Finally, the asymptotic validity of bootstrap-based confidence intervals and tests is briefly revisited, with particular emphasis on the, in practice unavoidable,
Monte Carlo approximation of conditional quantiles.
Keywords: Bootstrap; conditional weak convergence; confidence intervals; resampling; stochastic processes; weak convergence.
MSC 2010: 62E20; 62G09

1

Introduction

It is not uncommon in statistical problems that the limiting distribution of a statistic of interest
be intractable. To carry out inference on the underlying quantity, one possibility consists of
using a bootstrap or resampling scheme. Ideally, prior to its use, its consistency or asymptotic
validity should be mathematically demonstrated. For a real or vector-valued statistic Sn (or,
more generally, a statistic taking values in a separable metric space D), the latter classically
consists of establishing weak convergence of certain conditional laws (assuming that these are
well-defined; see, e.g., Faden, 1985 or Section 6 in Kallenberg, 2002). Specifically, a resampling
∗
Ruhr-Universität Bochum, Fakultät für Mathematik, Universitätsstr. 150, 44780 Bochum, Germany. E-mail:
axel.buecher@rub.de
†
CNRS / Université de Pau et des Pays de l’Adour, Laboratoire de mathématiques et applications – IPRA,
UMR 5142, B.P. 1155, 64013 Pau Cedex, France. E-mail: ivan.kojadinovic@univ-pau.fr

1

scheme can be considered asymptotically consistent if an appropriate distance between the
conditional distribution of a bootstrap replicate of Sn given the available observations and the
distribution of Sn is shown to converge to zero in probability; see, for instance, Bickel and
Freedman (1981), van der Vaart (1998, Chapter 23), Horowitz (2001) and the references therein,
or Assertions (c) and (e) in Lemma 2.2 below. A first contribution of this note is to show
that, under minimal conditions, the aforementioned convergence of conditional laws is actually
equivalent to the (unconditional) weak convergence of Sn jointly with two bootstrap replicates
to independent copies of the same limit. As pointed out by a referee, the proof of this result
relies on a key idea dating back to Hoeffding (1952), which has been used to derive quite similar
statements since then; see, for instance, Lemma 4.1 in Dümbgen and Del Conte-Zerial (2013)
and the additional references given in Section 2. Furthermore, we provide an interesting third
equivalent formulation of the consistency of a bootstrap for Sn . It roughly states that the
distance between the empirical distribution of the bootstrap replicates and the unobservable
distribution of Sn converges in probability to zero as the number of replicates and the sample
size increase (see also Beran et al., 1987, Section 4, for a similar result). The latter is particularly
meaningful given that most applications of resampling involve at some point approximating the
unobservable distribution of Sn by the empirical distribution of a finite number of bootstrap
replicates.
In many situations, the D-valued statistic of interest Sn is a “sufficiently smooth” functional
of a certain stochastic process Gn belonging to the space `∞ (T ) of bounded functions defined
on some arbitrary set T (think of the general empirical process, for instance, as defined in
Chapter 2 of van der Vaart and Wellner, 2000). Note that `∞ (T ), when equipped with the
supremum distance, is in general neither separable nor complete, and that Gn is usually allowed
to be non-measurable as well. As a consequence, neither laws nor conditional laws are welldefined in general, which complicates the theoretical analysis of bootstraps for Gn . Following
Giné and Zinn (1990), the consistency of a resampling scheme is then commonly defined by
the requirement that the bounded Lipschitz distance between the candidate limiting law and a
suitable adaptation of what might be called a conditional law of the bootstrap replicate (even
though the latter does not exist in the classical sense due to non-measurability) converges to
zero in outer probability. For instance, for the general empirical process based on independent
and identically distributed observations, such an investigation is carried out in Præstgaard
and Wellner (1993) (see also van der Vaart and Wellner, 2000, Section 3.6) for the so-called
empirical bootstrap and various other exchangeable bootstraps. The appeal of working at the
stochastic process level then arises from the fact that such bootstrap consistency results can be
transferred to the D-valued statistic level (often, D = Rd ) by means of appropriate extensions
of the continuous mapping theorem and the functional delta method.
It may however be argued that the aforementioned generalization of the classical conditional
formulation of bootstrap consistency is unintuitive and complicated to use given the subtlety
of the underlying mathematical concepts (in particular, relying on “conditional laws” of nonmeasurable maps). The latter seems all the more true for instance for empirical processes based
on estimated or serially dependent observations (see, e.g., Rémillard and Scaillet, 2009; Segers,
2012; Bücher and Kojadinovic, 2016a). The main contribution of this note is to show that the
D-valued results from Section 2 continue to hold for stochastic processes with bounded sample
paths: the “conditional” formulation is actually equivalent to the (unconditional) weak convergence of the initial stochastic process jointly with two bootstrap replicates. From a practical
perspective, using the latter unconditional formulation may have two important advantages.
First and most importantly, it may be easier to prove in certain situations than the conditional formulation. For this reason, it was for instance used, as explained above, for empirical
processes based on estimated or serially dependent observations; see also Section 3 below for
2

additional references. Second, the unconditional formulation may be transferable to the statistic
level for a slightly larger class of functionals of the stochastic process under consideration. The
latter follows for instance from the fact that continuous mapping theorems for the bootstrap,
that is, adapted to the conditional formulation, require more than just continuity of the map
that transforms the stochastic process into the statistic of interest (see, e.g., Kosorok, 2008,
Section 10.1.4). Furthermore, there does not seem hitherto to exist an extended continuous
mapping theorem (see, e.g., van der Vaart and Wellner, 2000, Theorem 1.11.1) for the bootstrap. Once the unconditional formulation is transferred to a separable metric space D (with D
being typically Rd ), the classical conditional statement immediately follows by the equivalence
at the D-valued statistic level mentioned above. Finally, let us mention that the equivalence at
the stochastic process level is well-known for the special case of multiplier central limit theorems
(CLTs) for the general empirical process based on i.i.d. observations using results of van der
Vaart and Wellner (2000, Section 2.9) (note that multiplier CLTs are sometimes also referred
to as multiplier or weighted bootstraps; see, e.g., Kosorok, 2008, Cheng and Huang (2010) and
the references therein). As such, our proven equivalence at the stochastic process level can be
seen as an extension of the latter work.
As an illustration of our results, we revisit the fact that bootstrap consistency implies that
bootstrap-based confidence intervals are asymptotically valid in terms of coverage and that
bootstrap-based tests hold their level asymptotically; see, for instance, van der Vaart (1998,
Lemma 23.3) for a related result and Horowitz (2001, Sections 3.3 and 3.4) for more specialized
and deeper results. In particular, we provide results which explicitly take into account that
(unobservable) conditional quantiles must be approximated by Monte Carlo in practice.
Finally, we would like to stress that the asymptotic results in this note are all of first order.
Higher order correctness of a resampling scheme (usually considered for real-valued statistics)
may still be important in small samples. The reader is referred to Hall (1992) for more details.
This note is organized as follows. The equivalence between the aforementioned formulations
of asymptotic validity of bootstraps of statistics taking values in separable metric spaces is
proved in Section 2. Section 3 states conditions under which the results of Section 2 extend
to stochastic processes with bounded sample paths. In Section 4, it is formally verified that,
as expected, bootstrap consistency implies asymptotic validity of bootstrap-based confidence
intervals and tests. A summary of results and concluding remarks are given in the last section.
a.s.

In the rest of the document, the arrow ‘ ’ denotes weak convergence, while the arrows ‘−→’
P
and ‘→’ denote almost sure convergence and convergence in probability, respectively.

2

Equivalent statements of bootstrap consistency in separable
metric spaces

The generic setup considered in this section is as follows. The available data will be denoted
by Xn . Apart from measurability, no assumptions are made on Xn , but it is instructive to think
of Xn as an n-tuple of multivariate observations which may possibly be serially dependent.
Let (D, d) denote a separable metric space. We are interested in approximating the law of some
D-valued statistic computed from Xn , denoted by Sn = Sn (Xn ). D-valued bootstrap replicates
of Sn , on which inference could be based, will be denoted by Sn(1) = Sn(1) (Xn , Wn(1) ), Sn(2) =
Sn(2) (Xn , Wn(2) ), . . . , where Wn(1) , Wn(2) , . . . , typically R-valued, are identically distributed and
represent additional sources of randomness such that Sn(1) , Sn(2) , . . . are independent conditionally
on Xn .
The previous setup is general enough to encompass most if not all types of resampling
3

procedures. For instance, when D = Rd , the classical empirical (multinomial) bootstrap of
Efron (1979) based on resampling with replacement from some original i.i.d. data set Xn =
(i)
(i)
(X1 , . . . , Xn ) can be obtained by letting the Wn(i) = (Wn1
, . . . , Wnn
) be i.i.d. multinomially
distributed with parameter (n, 1/n, . . . , 1/n). Indeed, for fixed i ∈ N, the sample Xn∗ =
(i)
(X1∗ , . . . , Xn∗ ) constructed by including the jth original observation Xj exactly Wnj
times,
j ∈ {1, . . . , n}, may be identified with a sample being drawn with replacement from the original
observations. Many other resampling schemes are included as well: block bootstraps for time
series such as the one of Künsch (1989), (possibly dependent) multiplier (or weighted, wild )
bootstraps (see, e.g., Shao, 2010) or the parametric bootstrap (see, e.g., Stute et al., 1993; Genest and Rémillard, 2008). For all but the last mentioned resampling scheme, Wn(1) , Wn(2) , . . . ,
could be interpreted as i.i.d. vectors of bootstrap weights, independent of Xn . Several examples
of such weights when Xn corresponds to n i.i.d. observations are given for instance in van der
Vaart and Wellner (2000, Section 3.6.2).
The previous setup is formally summarized in the following assumption. Recall the notions
of conditional independence and regular conditional distribution; see, e.g., Kallenberg (2002),
Section 6.
Condition 2.1 (D-valued resampling mechanism). Let (D, d) denote a separable metric space
equipped with the Borel sigma field D, and let (Ω, A, P) denote a probability space. For n ∈ N, let
Xn : Ω → Xn be a random variable in some measurable space Xn . Furthermore, let Wn(i) : Ω →
Wn , i ∈ N, denote identically distributed random variables in some measurable space Wn and
let Sn(i) = Sn(i) (Xn , Wn(i) ), i ∈ N, be D-valued statistics (to be considered as bootstrap replicates
of some D-valued statistic Sn = Sn (Xn )) that are independent conditionally on Xn . Finally,
(1)
assume that P(Sn(1) ∈ · | Xn ) has a regular version, denoted by PSn |Xn : Xn × D → R and called
the (regular) conditional distribution of Sn(1) given Xn .
The last assumption in the previous condition concerning the existence of the conditional
distribution of Sn(1) given Xn is automatically satisfied if there exists a possibly different metric e
on D which is equivalent to d such that (D, e) is complete. In that case, D is a Borel space,
see Theorem A1.2 in Kallenberg (2002), and the assertion follows from Theorem 6.3 in that
reference. The existence of the aforementioned conditional distribution can also be guaranteed
if the underlying probability space has a product structure, that is, if Ω = Ω0 × Ω1 × · · · with
probability measure P = P0 ⊗P1 ⊗· · · , where Pi denotes the probability measure on Ωi , such that,
for any ω ∈ Ω, Xn (ω) only depends on the first coordinate of ω and Wn(i) (ω) only depends on
the (i + 1)-coordinate of ω, implying in particular that Xn , Wn(1) , Wn(2) , . . . are independent. In
that case, it can readily be checked by Fubini’s theorem that (xn , A) 7→ P1 (Sn(1) (xn , Wn(1) ) ∈ A)
defines a regular version of the conditional distribution of Sn(1) given Xn .
In a related way, for arbitrary real-valued functions h such that E|h(Sn(1) )| < ∞, conditional
expectations E{h(Sn(1) ) | Xn } are always to be understood as integration of h(Sn(1) ) with respect
(1)
to PSn |Xn (Kallenberg, 2002, Theorem 6.4).
Lemma 2.2 below is one of the main result of this note and essentially shows that the
unconditional weak convergence of a statistic jointly with two of its bootstrap replicates is
equivalent to the convergence in probability of the conditional law of a bootstrap replicate. The
latter (with convergence in probability possibly replaced by almost sure convergence) is the
classical mathematical definition of the asymptotic validity of a resampling scheme. A further
equivalent formulation, of interest for applications, is also provided. Parts of these equivalences
can also be found in Dümbgen and Del Conte-Zerial (2013), Lemma 4.1, relying on ideas put
forward in Hoeffding (1952) and also exploited in Romano (1989) and Chung and Romano
(2013).
4

Recall that the bounded Lipschitz metric dBL between probability measures P, Q on a separable metric space (D, d) equipped with the Borel sigma field D is defined by
R
R
dBL (P, Q) = sup
f dP − f dQ ,
f ∈BL1 (D)

where BL1 (D) denotes the set of functions h : D → [−1, 1] such that |h(x) − h(y)| ≤ d(x, y) for
all x, y ∈ D. Moreover, recall the Kolmogorov distance dK between probability measures P, Q
on Rd , defined by
dK (P, Q) = sup P {(−∞, x]} − Q{(−∞, x]} .
x∈Rd

Finally, denote the empirical distribution of the sample Sn(1) , . . . , Sn(M ) by
n
P̂S
M =

M
1 X
δS (i) .
n
M
i=1

Lemma 2.2 (Equivalence of unconditional and conditional formulations). Suppose that Condition 2.1 is met. Assume further that Sn = Sn (Xn ) converges weakly to some random variable
S in D. Then, the following four assertions are equivalent:
(a)
(b)
(c)
(d)

(1)

P(Sn ,Sn

(2)

,Sn )

(1)
(M )
(Sn ,Sn ,...,Sn )

PS ⊗ PS ⊗ PS

as n → ∞,

(PS )⊗(M +1) as n → ∞ and for any M ≥ 2,

 (1)
P
dBL PSn |Xn , PSn → 0 as n → ∞,


P
Sn
n
dBL P̂S
→ 0 as n, M → ∞.
M ,P
P

If, additionally, D = Rd and the (cumulative) distribution function (d.f.) of S is continuous,
then the preceding four assertions are also equivalent to
 (1)

P
(e)
dK PSn |Xn , PSn → 0 as n → ∞,


P
Sn
n
,
P
→ 0 as n, M → ∞.
(f )
dK P̂S
M
Before providing a proof of this lemma, let us give an interpretation of the assertions. The
intuition behind Assertions (a) and (b) is that a resampling scheme should be considered consistent if the bootstrap replicates Sn(1) , Sn(2) , . . . behave approximately as independent copies of Sn ,
the more so that n is large. Assertions (c) and (e) translate mathematically the idea that a
resampling scheme should be considered valid if the distribution of a bootstrap replicate given
the data is close to the distribution of the original statistic Sn , the more so that n is large.
Assertions (d) and (f ) can be regarded as empirical analogues of Assertions (c) and (e), respectively: the unobservable conditional law of a bootstrap replicate is replaced by the empirical
law of a sample of M bootstrap replicates, providing an approximation of the law of Sn that
improves as n, M increase.
Assertions (c) and (e) are known to hold for many statistics and resampling schemes, possibly
as a consequence of general consistency results such as the one of Beran and Ducharme (1991)
(see also Horowitz, 2001, Section 2.1). Assertions (a) and (b) are substantially less frequently
encountered in the literature and appear mostly as a consequence of similar assertions at a
stochastic process level; see Lemma 3.1 in Section 3 and the references therein.
Let us finally turn to the proof of Lemma 2.2. The latter is in fact a corollary of the
following, slightly more general lemma which does not rely on the additional assumption that
Sn converges weakly.
5

Lemma 2.3. Suppose that Condition 2.1 is met and let Q be a fixed probability measure on
(D, D). Then, the following four assertions are equivalent:
(a)
(b)
(c)
(d)

(1)

P(Sn

(2)

,Sn )

(1)
(M )
(Sn ,...,Sn )

Q⊗Q

as n → ∞,

Q⊗M as n → ∞ and for any M ≥ 2,

(1)
P
dBL PSn |Xn , Q → 0 as n → ∞,


P
n
dBL P̂S
,
Q
→ 0 as n, M → ∞.
M

P



If, additionally, D = Rd and the d.f. of Q is continuous, then the preceding four assertions are
also equivalent to
(e)
(f )

 (1)

P
dK PSn |Xn , Q → 0 as n → ∞,


P
n
→ 0 as n, M → ∞.
dK P̂S
,
Q
M

The proof of this lemma will in turn be based on the following two possibly well-known
lemmas about metrizing weak convergence in separable metric spaces. Note that the results are
stated in terms of nets which generalize sequences (see, e.g., van der Vaart and Wellner, 2000,
Section 1.1) in order to account for the net convergences in Assertions (d) and (f ) of the two
preceding lemmas.
For sequences, the forthcoming assertions regarding the Kolmogorov distance can for instance be found in van der Vaart (1998), see Lemma 2.11 and Problem 23.1, while the assertions
regarding the bounded Lipschitz metric can be found in Dudley (2002), Theorem 11.3.3, for
the non-random version (see Lemma 2.4 below) and in Dümbgen and Del Conte-Zerial (2013),
Section 2, for the random one (see Lemma 2.5 below). Detailed proofs are provided in the
supplementary material for the sake of completeness.
Lemma 2.4. Suppose that (D, d) is a separable metric space and let Pα be a net of probability
measures on (D, D), where D denotes the Borel sigma field. Then Pα
P if and only if
dBL (Pα , P ) → 0. If D = Rd and if the d.f. of P is continuous, we also have equivalence to
dK (Pα , P ) → 0.
A random probability measure P̂ on a separable metric space (D, d) is a mapping from
some
probability space (Ω, A, P) into the set of Borel probability measures on (D, D) such that
R
f dP̂ considered as a function from Ω to R is measurable for any bounded and continuous
function f on D (see, e.g., Dümbgen and Del Conte-Zerial, 2013, Section 2). Note that, under
(1)
Condition 2.1, ω 7→ PSn |Xn (Xn (ω), ·) is a sequence of such random probability measures.
Lemma 2.5. Suppose that (D, d) is a separable metric space and let (P̂α )α denote a net of
random probability measures on (D, D) defined on a probability space (Ω, A, P). Then,
Z
Z
P
f dP̂α → f dP
(2.1)
for any f bounded and Lipschitz continuous if and only if dBL (P̂α , P ) → 0 in probability. Further, dBL (P̂α , P ), considered as a map from Ω to R, is measurable.
If D = Rd and if the d.f. of P is continuous, then (2.1) is also equivalent to dK (P̂α , P ) → 0
in probability, and dK (P̂α , P ) is measurable as well.
6

We can now prove Lemma 2.3.
Proof of Lemma 2.3. We begin by showing the equivalence between (a), (b), (c) and (d). Note
that, even though the equivalence between (a) and (c) is almost identical to Lemma 4.1 of
Dümbgen and Del Conte-Zerial (2013), we provide a self-contained proof to ease readability.
(b) ⇒ (a): trivial.

(a) ⇒ (c): by Lemma 2.4, we only need to show that
Z
Z
(1)
Sn |Xn P
→ f dQ as n → ∞,
f dP

for all bounded and Lipschitz continuous f . As in the proof of Lemma 4.1 in Dümbgen and Del
Conte-Zerial (2013), we can even prove L2 -convergence. Let S denote a random variable with
distribution Q. Then, by the law of iterated expectation,
2 o

n R
2 
R
(1)
(1)
= E E{f (Sn ) | Xn } − E{f (S)}
E
f dPSn |Xn − f dQ

2 

2
= E E{f (Sn(1) ) | Xn }
− 2E{f (Sn(1) )}E{f (S)} + E{f (S)} .
(1)

(2)

Since Sn and Sn are identically distributed and conditionally independent given Xn , the first
term on the right-hand side can be written as



E E{f (Sn(1) )f (Sn(2) ) | Xn } = E f (Sn(1) )f (Sn(2) ) .
The function (x, y) 7→ f (x)f (y) being bounded and continuous, the convergence in (a) implies
that, as n → ∞,
2 o
n R



2
R
(1)
→ E f (S (1) )f (S (2) ) − 2E{f (S (1) )}E{f (S)} + E{f (S)} = 0,
E
f dPSn |Xn − f dQ
where S (1) and S (2) are independent copies of S.
(c) ⇒ (b): by Lemma 2.4 and Corollary 1.4.5 in van der Vaart and Wellner (2000), it suffices
to show that, as n → ∞,
E{f1 (Sn(1) ) · · · fM (Sn(M ) )} →

M
Y

E{fj (S)}

j=1
(1)

(M )

for any f1 , . . . , fM bounded and Lipschitz continuous. By independence of Sn , . . . , Sn
ditionally on Xn , we can write the left-hand side as




E E{f1 (Sn(1) ) · · · fM (Sn(M ) ) | Xn } = E E{f1 (Sn(1) ) | Xn } · · · E{fM (Sn(M ) ) | Xn } ,

con-

and the assertion follows from (c), Lemma 2.5 and dominated convergence for convergence in
probability.
(c) ⇔ (d): fix f bounded and Lipschitz continuous and ε > 0, and denote by K a bound
on f . Then, for any n ∈ N,
n R
o
h n R
oi
R
(1)
(1)
Sn
Sn |Xn ≥ ε = E P
n
P
f dP̂S
−
f
dP
f
d
P̂
−
Ef
(S
)
≥
ε
X
n
n
M
M
h
i

PM
1
K
(i)
≤ 2 2 E Var
≤ 2
(2.2)
i=1 f (Sn ) Xn
ε M
ε M
7

by Chebychev’s inequality. As a consequence,
Z
Z
(1)
P
Sn
f dP̂M − f dPSn |Xn → 0
as n, M → ∞ since the upper bound on the right-hand side of (2.2) is independent of n. The
equivalence (c) ⇔ (d) is then a consequence of Lemma 2.5.

Finally, if D = Rd and if the d.f. of Q is continuous, the equivalences (c) ⇔ (e) and (d) ⇔ (f )
are immediate consequences of Lemma 2.5.

Lemma 2.2 arises finally as a simple corollary of Lemma 2.3.

Proof of Lemma 2.2. Denote the assertions (a)–(f ) in Lemma 2.3 by (a0 )–(f 0 ), respectively.
Let Q = PS and note that, by Lemma 2.4, Sn
S implies that dBL (PSn , Q) → 0. Then, the
triangle inequality and Lemma 2.3 immediately imply the equivalences (c) ⇔ (c0 ) ⇔ (d0 ) ⇔ (d).
Since (b) ⇒ (a) ⇒ (a0 ) ⇔ (c0 ), to show the equivalence between (a)–(d), it remains to be
shown that (c0 ) implies (b). By Corollary 1.4.5 in van der Vaart and Wellner (2000), it suffices
to show that
M
Y
E{f0 (Sn )f1 (Sn(1) ) · · · fM (Sn(M ) )} →
E{fj (S)}
j=0

for any f0 , . . . , fM bounded and Lipschitz continuous. By independence of Sn(1) , . . . , Sn(M ) conditionally on Xn , we obtain that




E E{f0 (Sn )f1 (Sn(1) ) · · · fM (Sn(M ) ) | Xn } = E f0 (Sn )E{f1 (Sn(1) ) | Xn } · · · E{fM (Sn(M ) ) | Xn } ,
and the assertion follows from (c0 ), Lemma 2.5 and dominated convergence for convergence in
probability.
Finally, if D = Rd and if the d.f. of Q = PS is continuous, (c0 ) ⇔ (e0 ) by Lemma 2.3 and the
equivalences (e) ⇔ (e0 ) ⇔ (f 0 ) ⇔ (f ) follow from the fact that dK (PSn , Q) → 0 (a consequence
of Lemma 2.4), the triangular inequality and Lemma 2.3.


3

Extension to stochastic processes with bounded sample paths

As in the previous section, let Xn be some data formally seen as a random variable in some
measurable space Xn . Furthermore, let T denote an arbitrary non-empty set and let `∞ (T )
denote the set of real-valued bounded functions on T equipped with the supremum distance.
Since, as already mentioned in the introduction, the latter metric space is in general neither
separable nor complete, one cannot typically set D = `∞ (T ) and apply the results of the previous
section.
To remedy this shortcoming, we are hereafter specifically interested in the situation in which
the D-valued statistic Sn of the previous section is a stochastic process Gn = Gn (Xn ) on T
constructed from Xn . It is assumed that every sample path t 7→ Gn (t, Xn (ω)) is a bounded
function so that Gn may formally be regarded as a map from the underlying probability space
Ω into `∞ (T ) without however imposing any measurability conditions. We additionally suppose
that, as n → ∞, Gn converges weakly in `∞ (T ) to some tight, Borel measurable stochastic
process G in the sense of Hoffmann-Jørgensen (see, e.g., van der Vaart and Wellner, 2000, Section 1.3) (which in fact implies that Gn is asymptotically measurable). Extending the setting
(1)
(1)
(2)
(2)
(2)
of Section 2, we further assume that G(1)
n = Gn (Xn , Wn ), Gn = Gn (Xn , Wn ), . . . are bootstrap replicates of Gn , that is, stochastic processes on T depending on additional identically
8

distributed random variables Wn(1) , Wn(2) , . . . in some measurable space Wn that can, in many
cases, be interpreted as bootstrap weights and should in general be seen as the additional sources
of randomness introduced by the resampling scheme. As for Gn , it is assumed that the sample
(2)
∞
∞
paths of G(1)
n , Gn , . . . also belong to ` (T ) and, when seen as maps into ` (T ), no measurability assumptions are made on these bootstrap replicates either. When Xn represents i.i.d.
observations and Gn is the general empirical process constructed from Xn , several examples
of possible bootstrap replicates of Gn can for instance be found in van der Vaart and Wellner
(2000, Section 3.6). As in Section 3.6 of the latter reference, we assume throughout this section
that the underlying probability space is independent of n and has a product structure, that is,
Ω = Ω0 × Ω1 × · · · with probability measure P = P0 ⊗ P1 ⊗ · · · , where Pi denotes the probability measure on Ωi , such that, for any ω ∈ Ω, Xn (ω) only depends on the first coordinate
of ω and Wn(i) (ω) only depends on the (i + 1)-coordinate of ω, implying in particular that
Xn , Wn(1) , Wn(2) , . . . are independent.
Some additional notation is needed before our main result can be stated. For any map
Z : Ω → R, let Z ∗ be any minimal measurable majorant of Z with respect to P, that is, Z ∗ : Ω →
[−∞, ∞] is measurable, Z ∗ ≥ Z and Z ∗ ≤ U almost surely for any measurable function U : Ω →
[−∞, ∞] with U ≥ Z almost surely. A maximal measurable minorant of Z with respect to P is
denoted by Z∗ and defined by Z∗ = −(−Z)∗ (see van der Vaart and Wellner, 2000, Section 1.2).
Furthermore, for any i ∈ {0, 1, . . . }, we define the map Z i∗ : Ω → [−∞, ∞] such that, for any
(ω0 , . . . , ωi−1 , ωi+1 , . . . ) ∈ Ω0 ×. . . Ωi−1 ×Ωi+1 ×· · · , the map ωi 7→ Z i∗ (ω0 , . . . , ωi−1 , ωi , ωi+1 , . . . )
is a minimal measurable majorant of ωi 7→ Z(ω0 , . . . , ωi−1 , ωi , ωi+1 , . . . ) with respect to Pi .
Finally, for a real-valued function Y on Xn × Wn such that w 7→ Y (x, w) is measurable for all
x ∈ Xn , we further use the notation
Z
(i)
Y (Xn , w) dPWn (w),
E(Y | Xn ) =
Wn

provided the integral exists. Note that if Y is jointly Borel measurable, the right-hand side of
the last displays defines a version of the conditional expectation of Y given Xn , whence the
notation.
Lemma 3.1. With the previous notation and under the above assumptions, the following three
assertions are equivalent:
(a) As n → ∞,

(2)
(Gn , G(1)
n , Gn )

(G, G(1) , G(2) )

where G, G(1) , G(2) are i.i.d.

in {`∞ (T )}3 ,

(3.1)

(b) For any M ≥ 2, as n → ∞,
(M )
(Gn , G(1)
n , . . . , Gn )

(G, G(1) , . . . , G(M ) )

in {`∞ (T )}M +1 ,

(3.2)

where G, G(1) , . . . , G(M ) are i.i.d.
(c) As n → ∞,

sup
h∈BL1 (`∞ (T ))

P∗

1∗
E{h(G(1)
| Xn } − E{h(G)} → 0,
n )

(3.3)

P∗

and G(1)
n is asymptotically measurable, where → denotes convergence in outer probability.
Let us make a few comments on this result:
• Assertion (c) is the extension put forward by Giné and Zinn (1990) of the conditional
formulation of bootstrap consistency in a separable metric space D to the non-necessarily
9

separable space `∞ (T ). Section 3.6 in van der Vaart and Wellner (2000) and Chapter 10
in Kosorok (2008) in particular provide proofs of Assertion (c) for various bootstraps of
the general empirical process constructed from i.i.d. observations along with continuous
mapping theorems for the bootstrap and a functional delta method for the bootstrap that
can be used to transfer (3.3) to the statistic level in certain situations.
• In van der Vaart and Wellner (2000), van der Vaart (1998) and Kosorok (2008), the
expression on the left-hand side of (3.3) appears without the minimal measurable majorant
with respect to the “weights”. This is a consequence of the fact that, for all the resampling
schemes considered in these monographs, the function w 7→ G(1)
n (x, w) is continuous for
all x ∈ Xn , implying that w 7→ h{G(1)
(x,
w)}
is
measurable
for
all x ∈ Xn and all h ∈
n
∞
BL1 (` (T )). However, the minimal measurable majorant becomes for instance necessary
if one wishes to apply Lemma 3.1 to certain stochastic processes appearing when using
the parametric bootstrap (e.g., for goodness-of-fit testing, see, Stute et al., 1993; Genest
and Rémillard, 2008). To see this, suppose that Xn is an i.i.d. sample of size n from some
d.f. G on the real line, with G from some parametric family {Gθ }. A natural stochastic
process, from which one may for instance construct classical goodness-of-fit statistics, is
√
then Gn (t) = n{Gn (t) − G(t)}, t ∈ R, where Gn is the empirical d.f. of Xn . Bootstrap
samples are generated by sampling from Gθn , where θn = θn (Xn ) is an estimator of θ.
Note in passing that the latter way of proceeding is compatible with the product-structure
condition on the underlying probability space since bootstrap samples can equivalently
be regarded as obtained by applying G−1
θn component-wise to independent random vectors
Wn(1) , Wn(2) , . . . independent of Xn and whose components are i.i.d. standard uniform.
√
Now, corresponding parametric bootstrap replicates of Gn are given by G(i)
n(G(i)
n =
n −
(i)
(i)
−1
−1
Gn ), where G(i)
n is the empirical d.f. of the sample (Gθn (Wn1 ), . . . , Gθn (Wnn )). The need
for the minimal measurable majorant with respect to the “weights” in (3.3) is then a
consequence of the fact that the function from Rn to R defined by

n
i
1 Xh
(i)
−1
(i)
(i)
(i)
w 7→ h{Gn (x, w )} = h √
1{Gθn (x) (wj ) ≤ ·} − 1(xj ≤ ·)
n
j=1

is not measurable for all h ∈ BL1 (`∞ (R)) and all x ∈ Xn , as can for instance be verified
by adapting arguments from Billingsley (1999, Section 15).
• Bootstrap asymptotic validity in the form of Assertions (a) or (b) is less frequently encountered in the literature, although, as discussed in the introduction, it may be argued
that this unconditional formulation is more intuitive and easy to work with. It is proved
for example in Genest and Rémillard (2008) (for M = 1), Rémillard and Scaillet (2009),
Segers (2012), Genest and Nešlehová (2014), Berghaus and Bücher (2017) and Bücher
and Kojadinovic (2016a,b), among many others, for various stochastic processes arising
in statistical tests on copulas or for assessing stationarity.
• As mentioned in the introduction, note that Assertions (b) and (c) are known to be
equivalent for the special case of the multiplier CLT for the general empirical process
based on i.i.d. observations and, in this case, it is even sufficient to consider M = 1 in (b):
Corollary 2.9.3 in van der Vaart and Wellner (2000) corresponds to Assertion (b), while
Theorem 2.9.6 corresponds to Assertion (c). The equivalence between the two follows by
combining Theorem 2.9.6 with Theorem 2.9.2.
Before proving Lemma 3.1, we provide a useful corollary which is an immediate consequence
of Lemma 3.1 and Lemma 2.2. It may be regarded as an analogue of Theorem 1.5.4 in van der
Vaart and Wellner (2000) in a conditional setting and, roughly speaking, states that conditional
weak convergence of a sequence of stochastic processes is equivalent to the conditional weak
10

convergence of finite-dimensional distributions and (unconditional) asymptotic tightness.
Corollary 3.2. Suppose that the assumptions of Lemma 3.1 are met. Then, any of the equivalent assertions in that lemma is equivalent to the fact that the finite dimensional distributions
of G(1)
n conditionally weakly converge to those of G in probability, that is, for any k ∈ N and
s1 , . . . , s k ∈ T ,

 (1)
(1)
P
(3.4)
dBL P(Gn (s1 ),...,Gn (sk ))|Xn , P(G(s1 ),...,G(sk )) → 0
as n → ∞, and that G(1)
n is (unconditionally) asymptotically tight.
Proof of Lemma 3.1. We closely follow the proof of Theorem 2.9.6 of van der Vaart and Wellner
(2000) and rely on Lemma 2.2 when necessary.
(b) ⇒ (a): trivial.

(a) ⇒ (c): Asymptotic measurability of G(1)
n is an immediate consequence of the weak con(1)
(1)
∞
vergence of Gn to G in ` (T ) (van der Vaart and Wellner, 2000, Lemma 1.3.8). Next, by
Theorems 1.5.4 and 1.5.7 in van der Vaart and Wellner (2000), the latter convergence implies
that there exists a semimetric ρ on T such that (T, ρ) is totally bounded and such that, for any
ε > 0,
n
o
(1)
sup |G(1)
(s)
−
G
(t)|
>
ε
= 0.
(3.5)
lim lim sup P∗
n
n
δ↓0

n→∞

ρ(s,t)<δ

Fix ` ∈ N. For any s ∈ T , let B(s, 1/`) = {t ∈ T : ρ(s, t) < 1/`} denote the ball of radius 1/`
centered at s. Since (T, ρ) is totally bounded, there exists k = k(`) ∈ N and si = si (`) ∈ T ,
i ∈ {1, . . . , k}, such that T is included in the union of all balls B(si , 1/`), i ∈ {1, . . . , k}. The
latter allows us to define a mapping Π` : T → T defined, for any s ∈ T , by Π` (s) = si∗ where
si∗ is the center of a ball containing s. Now, to prove (3.3), we consider the decomposition
sup
h∈BL1

(`∞ (T ))

1∗
E{h(G(1)
| Xn } − E{h(G)} ≤ In (`) + Jn (`) + K(`),
n )

` ∈ N,

where
(1)

(1)

In (`) = suph∈BL1 (`∞ (T )) E{h(Gn )1∗ | Xn } − E{h(Gn ◦ Π` )1∗ | Xn } ,
(1)

Jn (`) = suph∈BL1 (`∞ (T )) E{h(Gn ◦ Π` )1∗ | Xn } − E{h(G ◦ Π` )} ,
K(`) = suph∈BL1 (`∞ (T )) E{h(G ◦ Π` )} − E{h(G)} .
Some thought reveals that (3.3) is proved if, for any ε > 0,
lim lim sup P∗ {In (`) > ε} = 0,

`→∞ n→∞

(3.6)

and similarly for Jn (`) and K(`).
Term In (`): By Markov’s inequality for outer probabilities (Lemma 6.10 in Kosorok, 2008),
it suffices to show (3.6) with P∗ {In (`) > ε} replaced by E∗ {In (`)}. For any ` ∈ N, we have, by
Lemma 1.2.2 (iii) in van der Vaart and Wellner (2000),
n
o
(1) 1∗
In (`) ≤
sup
E |h(G(1)
◦
Π
)
−
h(G
)|
|
X
n
`
n
n
h∈BL1 (`∞ (T ))

hn
o∗
i
n
o
(1)
≤ E sup |G(1)
| Xn ≤ E Ln (`)∗ | Xn ,
n ◦ Π` (s) − Gn (s)| ∧ 1
s∈T

11

(1)
where ∧ denotes the minimum operator and Ln (`) = supρ(s,t)<1/` |G(1)
n (s)−Gn (t)|∧1. It follows
that E∗ {In (`)} ≤ E{Ln (`)∗ }. Note that, by Lemma 1.2.2 (viii) in van der Vaart and Wellner
(2000), we may choose Ln (`)∗ in such a way that ` 7→ Ln (`)∗ is nonincreasing almost surely.
Then ` 7→ P{Ln (`)∗ > ε} is nonincreasing as well, and from (3.5) and Problem 2.1.5 in van
der Vaart and Wellner (2000, see also Section 2.1.2), we have that Ln (`n )∗ → 0 in probability
as n → ∞ for any sequence `n → ∞, which, by dominated convergence for convergence in
probability, implies that E{Ln (`n )∗ } → 0. Hence, lim`→∞ lim supn→∞ E[Ln (`)∗ ] = 0 by invoking
Problem 2.1.5 in van der Vaart and Wellner (2000) again.

Term Jn (`): Fix ` ∈ N and recall that the centers of the balls defining Π` were denoted
by s1 , . . . , sk . Since the weak convergence stated in (3.1) implies weak convergence of the
respective finite dimensional distributions, we may invoke the equivalence between (a) and (c)
in Lemma 2.2 to conclude (with the help of the triangular inequality and Lemma 2.4) that (3.4)
holds, that is, that
Bn (s1 , . . . , sk ) =

sup
h∈BL1

(Rk )

P

(1)
E[h{G(1)
n (s1 ), . . . , Gn (sk )} | Xn ] − E[h{G(s1 ), . . . , G(sk )}] → 0.

(3.7)
Next, let h ∈ BL1 (`∞ (T )) be arbitrary. Define f : Rk → `∞ (T ) such that, for any x ∈ Rk and
s ∈ T , f (x)(s) = xi if Π` (s) = si . Furthermore, let g : Rk → R be defined as g(x) = h(f (x))
implying that h(G◦Π` ) = g(G(s1 ), . . . , G(sk )). Some thought reveals that g ∈ BL1 (Rk ), whence
Jn (`) ≤ Bn (s1 , . . . , sk ) → 0 in probability as n → ∞ for all ` ∈ N, implying the analogue of (3.6)
for Jn (`).
Term K(`): For any ` ∈ N, we have
n
o
n
K(`) ≤ E sup |G ◦ Π` (s) − G(s)| ∧ 1 ≤ E
s∈T

sup
ρ(s,t)<1/`

o
|G(s) − G(t)| ∧ 1 .

By tightness of G, Addendum 1.5.8 in van der Vaart and Wellner (2000) and dominated convergence, the expectation on the right converges to zero as ` → ∞, implying the analogue of (3.6)
for K(`).
(c) ⇒ (b): To prove (3.2), we need to show the weak convergence of the finite-dimensional
distributions and marginal asymptotic tightness. We start with the former. Let M, k ∈ N and
s1 , . . . , sk ∈ T . It suffices to show that, as n → ∞,
(1)
(M )
(M )
Gn (s1 ), . . . , Gn (sk ), G(1)
n (s1 ), . . . , Gn (sk ), . . . , Gn (s1 ), . . . , Gn (sk )



G(s1 ), . . . , G(sk ), G(1) (s1 ), . . . , G(1) (sk ), . . . , G(M ) (s1 ), . . . , G(M ) (sk )



(3.8)

in R(M +1)k . Now, for any g ∈ BL1 (Rk ), the function h : `∞ (T ) → R defined by h(f ) =
g(f (s1 ), . . . , f (sk )) is an element of BL1 (`∞ (T )). From (3.3), we then obtain that (3.7) holds,
or, equivalently, that (3.4) holds. We may hence invoke the equivalence between (b) and (c) in
Lemma 2.2 to obtain (3.8).
(M )
It remains to show marginal tightness. Since Gn
G in `∞ (T ) and G(1)
are
n , . . . , Gn
(1)
(1)
∞
identically distributed, it is sufficient to show that Gn
G in ` (T ). Then, as in the proof
of Theorem 2.9.6 of van der Vaart and Wellner (2000), for any h ∈ BL1 (`∞ (T )),

h
i
h
i
(1)
(1) ∗
∗
(1) 1∗
|E∗ {h(G(1)
)}
−
E{h(G
)}|
≤
E
E{h(G
)
|
X
}
−
E
E{h(G
)
|
X
}
n
n
n
n
n
h
i
1∗
(1)
+ E∗ E{h(G(1)
)
|
X
}
−
E{h(G
)}
.
n
n

12

By dominated convergence for convergence in outer probability and (3.3), the second term
1∗ ≥ {h(G(1) ) }1∗ = h(G(1) ) almost surely, the first term is
converges to zero. Since h(G(1)
n )
n ∗
n ∗
bounded above by
h
i
h
i
∗
(1)
(1) ∗
(1)
E E{h(G(1)
n ) | Xn } − E E{h(Gn )∗ | Xn } = E{h(Gn ) } − E{h(Gn )∗ }.
The latter expression converges to zero since G(1)
n is assumed asymptotically measurable. The
assertion follows from the Portmanteau Theorem (see, e.g., van der Vaart and Wellner, 2000,
Theorem 1.3.4 (i) and (vii)).


4

Validity of bootstrap-based confidence intervals and tests

Whether the consistency of a resampling scheme is shown at the stochastic process level and
then transferred to D = Rd or is directly proved at the statistic level, one naturally expects corresponding bootstrap-based confidence intervals and tests to be asymptotically valid. Specifically,
the latter amounts to verifying that confidence intervals have the correct asymptotic coverage
and that tests maintain their level asymptotically. To formally establish these expected consequences, in this section, we restrict ourselves to the classical situation of a real-valued statistic
whose weak limit has a continuous distribution function.
Condition 4.1 (R-valued resampling mechanism). Assume that Condition 2.1 holds with D = R
and that, additionally, Sn converges weakly to a random variable S with continuous d.f. F .
A result in the desired direction is for example Lemma 23.3 in van der Vaart (1998) and
more specialized and deeper results are for instance collected in Horowitz (2001, Sections 3.3 and
3.4). Most results of that type do not however take into account the necessary approximation
of the unobservable conditional d.f. of a bootstrap replicate by the empirical d.f. of a sample of
bootstrap replicates. The following simple lemma does so and thus allows one to easily verify
the asymptotic validity of bootstrap-based confidence intervals and tests constructed from a
consistent resampling scheme in the sense of Lemma 2.2.
As we continue, for n, M ∈ N and x ∈ R, we use the following notation:
FnM (x) =

1
M

(i)
i=1 1(Sn

PM

≤ x),

(1)

Fn (x) = P(Sn ≤ x | Xn )

and F (x) = P(S ≤ x).

Lemma 4.2. Suppose that Condition 4.1 is met and that one of the equivalent assertions in
Lemma 2.2 holds. Then, for any α ∈ (0, 1),
lim P{Sn ≥ (Fn )−1 (1 − α)} = α

n→∞

and

lim

n,M →∞

P{Sn ≥ (FnM )−1 (1 − α)} = α,

where G−1 denotes the generalized inverse of d.f. G, that is G−1 (y) = inf{x ∈ R : G(x) ≥ y},
y ∈ (0, 1]. The statements with ‘≥’ replaced by ‘>’ in the previous display hold as well.
The assertion of this lemma involving conditional quantiles (or a version thereof) is usually
provided in textbooks on the bootstrap to validate its use for the construction of confidence
intervals and tests (see, e.g., Lemma 23.3 in van der Vaart, 1998). For completeness, we shall
prove it at the end of this section. The assertion involving empirical quantiles is the one to
be used in practice as conditional quantiles are not available and must thus be approximated
by Monte Carlo. Note in particular that the above formulation is general enough to allow
M = M (n) with M (n) → ∞ as n → ∞.
13

Let us now briefly verify that the asymptotic validity of bootstrap-based confidence intervals
and tests is an immediate consequence of the preceding lemma. Start with the former and
√
assume that Sn = n(θn − θ), where θn is an estimator of some parameter θ ∈ R. Then, a
natural confidence interval for θ is given by
h
i
In,M,α = θn − n−1/2 (FnM )−1 (1 − α/2), θn − n−1/2 (FnM )−1 (α/2) ,
α ∈ (0, 1/2).
Note in passing that the above confidence interval is related to the so-called basic bootstrap confidence interval (see, e.g., Davison and Hinkley, 1997, Chapter 5). A consequence of Lemma 4.2
is then that, if one of the equivalent assertions in Lemma 2.2 hold, In,M,α is of asymptotic level
1 − α in the sense that, as n, M → ∞,
P(θ ∈ In,M,α ) = P{Sn ≥ (FnM )−1 (α/2)} − P{Sn > (FnM )−1 (1 − α/2)} → 1 − α.
Let us now discuss the case of bootstrap-based tests. Assume that Sn is a test statistic for
some null hypothesis H0 such that large values of Sn provide evidence against H0 . It is then
natural to reject H0 at level α ∈ (0, 1) when Sn > (FnM )(1 − α). Should one of the equivalent
assertions in Lemma 2.2 holds under H0 , Lemma 4.2 immediately implies that this test holds it
level asymptotically in the sense that, under H0 , P{Sn ≥ (FnM )−1 (1 − α)} → α as n, M → ∞.
If the bootstrap replicates are stochastically bounded under the alternative, then the test will
also be consistent provided Sn converges to infinity in probability under the alternative.
Under the same setting, another statistic of interest is
pM
n

M
1 X
=
1(Sn(i) > Sn ) = 1 − FnM (Sn ),
M
i=1

which may be interpreted as an approximate p-value for the test based on Sn . The theoretical
analogue of the latter is
pn = P(Sn(1) > Sn | Xn ) = 1 − Fn (Sn ).
Intuitively, the resampling scheme being valid should imply that, under the null hypothesis, the
statistics pM
n and pn are approximately standard uniform. The following result formalizes this.
Corollary 4.3. Suppose that Condition 4.1 is met and that one of the equivalent assertions in
Lemma 2.2 holds. Then, as n → ∞,
pn

Uniform(0, 1)

and

n
pM
n

Uniform(0, 1),

for any sequence Mn → ∞ as n → ∞.
The proofs of Lemma 4.2 and Corollary 4.3 are given hereafter.
Proof of Lemma 4.2. Consider the assertion involving conditional quantiles. Notice first that
the weak convergence of Sn to S, the continuity of F and Lemma 2.4 imply that
dK (PSn , PS ) = sup |P(Sn ≤ x) − F (x)| → 0
x∈R

as n → ∞.

(4.1)

Next, combine Assertion (e) in Lemma 2.2 with (4.1) to obtain that every subsequence of
(1)
dK (PSn |Xn , PS ) has a further subsequence along which this expression converges almost surely
to zero as n → ∞. Let α ∈ (0, 1) such that F −1 is continuous at 1 − α. As a consequence of
14

a.s.

Lemma 21.2 in van der Vaart (1998), we obtain that Fn−1 (1 − α) −→ F −1 (1 − α) along that
subsequence. Hence, the random vector (Sn , Fn−1 (1 − α)) converges weakly to (S, F −1 (1 − α)),
again along that subsequence. Since P{S = F −1 (1 − α)} = 0 by continuity, the Portmanteau
Theorem implies that
P{Sn ≥ Fn−1 (1 − α)} → P{S ≥ F −1 (1 − α)} = α

(4.2)

along that subsequence. The latter equation holds for all expect at most countably many α ∈
(0, 1). Because the left (resp. right) side of (4.2) is an increasing (resp. increasing continuous)
function of α, (4.2) must hold for all α ∈ (0, 1). The first assertion follows since the subsequence
we started with was arbitrary. Finally, note that one may replace ‘≥’ by ‘>’ in the last display.
Consider the assertion involving empirical quantiles. Let q1−α denote the (1 − α)-quantile
of S. Since limn→∞ P(Sn ≥ q1−α ) = P(S ≥ q1−α ) = α as a consequence of the Portmanteau
Theorem and the continuity of F , it suffices to show that
lim

n,M →∞

|P{Sn ≥ (FnM )−1 (1 − α)} − P(Sn ≥ q1−α )|
=

lim

n,M →∞

|P{FnM (Sn ) ≥ 1 − α} − P{F (Sn ) ≥ 1 − α}| = 0,

where the equality follows from the fact that FnM and F are right-continuous. Using the fact
that, for any a, b, x ∈ R and ε > 0, |1(x ≤ a) − 1(x ≤ b)| ≤ 1(|x − a| ≤ ε) + 1(|a − b| > ε), we
can estimate
|P{FnM (Sn ) ≥ 1 − α} − P{F (Sn ) ≥ 1 − α}|

≤ P{|F (Sn ) − 1 + α| ≤ ε} + P{|F (Sn ) − FnM (Sn )| > ε}.

By the continuous mapping theorem and the Portmanteau Theorem, the first term on the
right converges to P{|F (S) − 1 + α| ≤ ε} as n → ∞, which can be made arbitrary small by
decreasing ε. Combining Assertion (f ) from Lemma 2.2 with (4.1) immediately implies that
the second term converges to zero as n, M → ∞, hence the first claim.

The claim with ‘≥’ replaced by ‘>’ follows from the fact that, by continuity of F and
the Portmanteau Theorem, for any x ∈ R, P(Sn < x) → P(S < x) as n → ∞. The latter
convergence can be made uniform by arguments as in Lemma 2.11 in van der Vaart (1998),
which, combined with (4.1) implies that supx∈R P(Sn = x) converges to zero in probability as
n → ∞.


Proof of Corollary 4.3. The weak convergence Sn
S as n → ∞ together with the continuous
mapping theorem implies that 1 − F (Sn )
1 − F (S) ∼ Uniform(0, 1) as n → ∞. Combining
Assertion (e) in Lemma 2.2 with (4.1), we additionally immediately obtain that Fn (Sn )−F (Sn )
converges to zero in probability as n → ∞, which implies that pn has the same weak limit as
1−F (Sn ) as n → ∞. Similarly, Assertion (f ) in Lemma 2.2 combined with (4.1) readily implies
n
that pM

n has the same limit distribution as 1 − F (Sn ) as n → ∞.

5

Concluding remarks

As a picture often speaks better than words, we summarized in the diagram of Figure 1 the
way the results obtained in this note could typically be used to prove the validity of bootstrapbased statistical inference procedures. From the point of view of applications of resampling
schemes starting at the stochastic process level, the diagram highlights two paths to proving
15

Stochastic processes in ℓ∞ (T )
Lemma 3.1:

⇐⇒

Assertion (a)

Assertion (c)

CMT,
extended CMT,
FDT
Rd -valued statistics

Conditional path

Unconditional path

under "product-structure" condition

CMT,
FDT
for the
bootstrap

Lemma 2.2 with D = Rd :
Assertion (a)

⇐⇒

under minimal conditions

Assertion (c)

⇓

Lemma 4.2 and Corollary 4.3:
Validity of bootstrap-based confidence intervals and tests

Figure 1:
Summary of typical uses of the obtained results; CMT stands for “continuous
mapping theorem” and FDT for “functional delta method”.

the asymptotic validity of bootstrap-based confidence intervals and tests: an unconditional
path starting at Assertion (a) of Lemma 3.1 and a conditional path starting at Assertion (c) of
Lemma 3.1.
We conclude by summarizing the main consequences and features of the results obtained in
this note, some of which explicitly appear in the diagram of Figure 1:
• At the stochastic process level, it may be argued that one needs to deal with less subtle
mathematical concepts to prove unconditional bootstrap consistency than to show its conditional version. Roughly speaking, the unconditional approach avoids the need to work
with the seemingly awkward notion of “conditional law” of a non-measurable function.
• Focusing for instance on existing continuous mapping theorems for the bootstrap (Kosorok,
2008, Section 10.1.4), it appears that, for transferring Assertion (c) of Lemma 3.1 into
Assertion (c) of Lemma 2.2, more assumptions than just continuity of the underlying functional are necessary, thereby suggesting that the unconditional formulation of bootstrap
consistency might be slightly more useful. Additionally, Assertion (a) of Lemma 3.1 can
be combined with the extended continuous mapping theorem (van der Vaart and Wellner,
2000, Theorem 1.11.1), while a version of the latter result for the bootstrap does not
hitherto seem to exist.
• The equivalence between the unconditional and the conditional formulation of bootstrap
consistency at the stochastic process level only holds if the additional randomness in the
bootstrap replicates is independent of the data (in fact, this assumption is only needed to
make Assertion (c) well-defined). Interestingly enough, such a condition does not seem to
be a restriction in practice as it seems satisfied by most if not all resampling schemes.
16

• Although, as already discussed, one cannot in general rely on Lemma 2.2 to deal with
stochastic processes with bounded sample paths, this lemma remains general enough to
deal with stochastic processes living in the Skorohod space (see, e.g., Billingsley, 1999)
since the latter can be metrized in such a way that it is separable and complete.

Acknowledgments
The authors are very grateful to an anonymous referee for making them aware of “Hoeffding’s
trick” (Hoeffding, 1952) and for several other very relevant suggestions which contributed to
significantly increasing the scope of this note. The authors would also like to thank JeanDavid Fermanian for fruitful discussions. This research has been supported by the Collaborative
Research Center “Statistical modeling of nonlinear dynamic processes” (SFB 823) of the German
Research Foundation, which is gratefully acknowledged.

References
Beran, R. and G. Ducharme (1991). Asymptotic theory for bootstrap methods in statistics. Centre
de recherches mathématiques, Université de Montréal, Canada: Les publication CRM.
Beran, R. J., L. Le Cam, and P. W. Millar (1987). Convergence of stochastic empirical measures.
J. Multivariate Anal. 23 (1), 159–168.
Berghaus, B. and A. Bücher (2017). Goodness-of-fit tests for multivariate copula-based time
series models. Econometric Theory 33 (2), 292–330.
Bickel, P. J. and D. A. Freedman (1981). Some asymptotic theory for the bootstrap. Ann.
Statist. 9 (6), 1196–1217.
Billingsley, P. (1999). Convergence of probability Measures. New York: Wiley. Second edition.
Bücher, A. and I. Kojadinovic (2016a). A dependent multiplier bootstrap for the sequential
empirical copula process under strong mixing. Bernoulli 22 (2), 927–968.
Bücher, A. and I. Kojadinovic (2016b). Dependent multiplier bootstraps for non-degenerate
u-statistics under mixing conditions with applications. Journal of Statistical Planning and
Inference 170, 83–105.
Cheng, G. and J. Z. Huang (2010). Bootstrap consistency for general semiparametric M estimation. Ann. Statist. 38 (5), 2884–2915.
Chung, E. and J. P. Romano (2013). Exact and asymptotically robust permutation tests. Ann.
Statist. 41 (2), 484–507.
Davison, A. C. and D. V. Hinkley (1997). Bootstrap Methods and Their Application. Cambridge
University Press.
Dudley, R. M. (2002). Real analysis and probability. Cambridge University Press, Cambridge.
Revised reprint of the 1989 original.
Dümbgen, L. and P. Del Conte-Zerial (2013). On low-dimensional projections of highdimensional distributions. In From probability to statistics and back: high-dimensional models
and processes, Volume 9 of Inst. Math. Stat. (IMS) Collect., pp. 91–104. Inst. Math. Statist.,
Beachwood, OH.
17

Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Statist. 7 (1), 1–26.
Faden, A. M. (1985). The existence of regular conditional probabilities: Necessary and sufficient
conditions. Ann. Probab. 13 (1), 288–298.
Genest, C. and J. G. Nešlehová (2014). On tests of radial symmetry for bivariate copulas.
Statistical Papers 55, 1107–1119.
Genest, C. and B. Rémillard (2008). Validity of the parametric bootstrap for goodness-offit testing in semiparametric models. Annales de l’Institut Henri Poincaré: Probabilités et
Statistiques 44, 1096–1127.
Giné, E. and J. Zinn (1990). Bootstrapping general empirical measures. Ann. Probab. 18 (2),
851–869.
Hall, P. (1992). The bootstrap and Edgeworth expansion. Springer Series in Statistics. SpringerVerlag, New York.
Hoeffding, W. (1952). The large-sample power of tests based on permutations of observations.
Ann. Math. Statistics 23, 169–192.
Horowitz, J. (2001). The bootstrap. In J. J. Heckman and E. E. Leamer (Eds.), Handbook of
Econometrics, Volume 5, pp. 3159–3228. Amsterdam: North-Holland.
Kallenberg, O. (2002). Foundations of modern probability (Second ed.). Probability and its
Applications (New York). Springer-Verlag, New York.
Kosorok, M. (2008). Introduction to empirical processes and semiparametric inference. New
York: Springer.
Künsch, H. (1989). The jacknife and the bootstrap for general stationary observations. The
Annals of Statistics 17 (3), 1217–1241.
Præstgaard, J. and J. A. Wellner (1993). Exchangeably weighted bootstraps of the general
empirical process. Ann. Probab. 21 (4), 2053–2086.
Rémillard, B. and O. Scaillet (2009). Testing for equality between two copulas. Journal of
Multivariate Analysis 100 (3), 377–386.
Romano, J. P. (1989). Bootstrap and randomization tests of some nonparametric hypotheses.
Ann. Statist. 17 (1), 141–159.
Segers, J. (2012). Asymptotics of empirical copula processes under nonrestrictive smoothness
assumptions. Bernoulli 18, 764–782.
Shao, X. (2010). The dependent wild bootstrap. Journal of the American Statistical Association 105 (489), 218–235.
Stute, W., W. Gonzáles Manteiga, and M. Presedo Quindimil (1993). Bootstrap based goodnessof-fit tests. Metrika 40, 243–256.
van der Vaart, A. (1998). Asymptotic statistics. Cambridge University Press.
van der Vaart, A. and J. Wellner (2000). Weak convergence and empirical processes. New York:
Springer. Second edition.
18

Supplementary material for
“A note on conditional versus joint unconditional weak
convergence in bootstrap consistency results”
Axel Bücher1 and Ivan Kojadinovic2
March 5, 2018

Abstract
This supplementary material contains the proofs of Lemmas 2.4 and 2.5.

A

Proof of Lemma 2.4

Lemma 2.4. Suppose that (D, d) is a separable metric space and let Pα be a net of probability
measures on (D, D), where D denotes the Borel sigma field. Then Pα
P if and only if
dBL (Pα , P ) → 0. If D = Rd and if the d.f. of P is continuous, we also have equivalence to
dK (Pα , P ) → 0.
Proof. The assertion for D = Rd and if the d.f. of P is continuous is, up to a slight generalization
to nets, Lemma 2.11 in van der Vaart (1998). The general case is, again up to a generalization
to nets, Theorem 11.3.3 in Dudley (2002). Since we will need arguments from the latter proof
in the proof of Lemma 2.5 below, we reproduce them for completeness.
We start by proving the claim under the additional assumption that (D, d) is complete. By
Ulam’s Theorem (see, e.g., Theorem 1.3 in Billingsley, 1999), the measure P is tight. Hence,
for any given ε > 0, we can find a compact set K in (D, d) such that P (K) > 1 − ε. Let
K ε = {x ∈ D : d(x, y) < ε for some y ∈ K} denote the ε-enlargement of K. Furthermore,
consider the function g(x) = max{1 − d(x, K)/ε, 0}, x ∈ D, and note that g is Lipschitz
continuous and satisfies 1K ≤ g ≤ 1K ε . As a consequence of the Portmanteau Theorem (see,
e.g., Theorem 1.3.4 in van der Vaart and Wellner, 2000), we obtain that
Z
Z
ε
lim inf Pα (K ) ≥ lim inf gdPα ≥ gdP ≥ P (K) > 1 − ε.
Next, consider the space FK of functions h = f |K : K → [−1, 1] with f ∈ BL1 (D). It can
be verified that FK is equicontinuous and bounded, whence, by the Arzelá–Ascoli theorem, FK
is relatively compact in the Banach space (C(K), k · k∞ ). Relative compactness in a complete
metric space is equivalent to total boundedness, so we can find functions f1 , . . . , fm ∈ FK such
that, for any f ∈ FK , there exists fj with supx∈K |f (x) − fj (x)| < ε. As a consequence, by
Lipschitz continuity of f and fj on D, we have
sup |f (x) − fj (x)| < 3ε.

x∈K ε
1

Ruhr-Universität Bochum, Fakultät für Mathematik, Universitätsstr. 150, 44780 Bochum, Germany. E-mail:
axel.buecher@rub.de
2
CNRS / Université de Pau et des Pays de l’Adour, Laboratoire de mathématiques et applications – IPRA,
UMR 5142, B.P. 1155, 64013 Pau Cedex, France. E-mail: ivan.kojadinovic@univ-pau.fr

Finally, assembling bounds obtained so far, we obtain that, for any f ∈ BL1 (D),
R
R
R
f d(Pα − P ) = (f − fj )d(Pα − P ) + fj d(Pα − P )
R
R
R
≤ f − fj dPα + f − fj dP +
fj d(Pα − P )
R
R
R
≤ K ε f − fj dPα + 2Pα {(K ε )c } + K f − fj dP + 2P (K c ) +
fj d(Pα − P )
R
m
≤ 3ε + 2ε + ε + 2ε + max
fj d(Pα − P )
j=1

for sufficiently large α. The Portmanteau Theorem implies that that maximum on the righthand side converges to 0. The assertion follows since ε > 0 was arbitrary.
Finally, the case where D is not complete can be treated analogously by passing to the
completion of D; see, e.g., Dudley (2002).


B

Proof of Lemma 2.5

Lemma 2.5. Suppose that (D, d) is a separable metric space and let (P̂α )α denote a net of
random probability measures on (D, D) defined on a probability space (Ω, A, P). Then,
Z
Z
P
f dP̂α → f dP
(B.1)
for any f bounded and Lipschitz continuous if and only if dBL (P̂α , P ) → 0 in probability. Further, dBL (P̂α , P ), considered as a map from Ω to R, is measurable.
If D = Rd and if the d.f. of P is continuous, then (B.1) is also equivalent to dK (P̂α , P ) → 0
in probability, and dK (P̂α , P ) is measurable as well.
Proof. We only consider the case where D is complete. For the general case, one can pass to the
completion of D as mentioned in the proof of Lemma 2.4. Sufficiency follows from linearity of
integrals, since any bounded Lipschitz function can be scaled to a function in BL1 (D). Necessity
follows by carefully
R following
R the proof of Lemma 2.4, which only made use of the fact that
Pα
P implies f dPα → f dP for all f bounded and Lipschitz continuous. More precisely,
let δ > 0 and η > 0. We need to show that P(dBL (P̂α , P ) > δ) < η for all sufficiently large α.
Let ε = δ/11. Choose K = K(ε) and, subsequently, the functions g, f1 , . . . , fm as in the proof
of Lemma 2.4. By assumption, we have P(Aα ) < η for all sufficiently large α, where
n
o
 R
R
R
Aα = max
gd(P̂α − P ) , f1 d(P̂α − P ) , . . . , fm d(P̂α − P ) > ε .
On the event Acα , we can verify that P̂Rα (K ε ) > 1 − 2ε and subsequently follow the proof of
Lemma 2.4 to establish that supf ∈BL1
f d(P̂α − P ) ≤ 11ε = δ for sufficiently large α, which
implies the assertion.
Let us next prove measurability of dBL (P̂α , P ). Choose a countable dense subset S of D.
Let H denote the set of all real-valued functions on D of the form
h(x) = q max{1 − pd(x, s), 0},

x ∈ D,

where s ∈ S, p, q ∈ Q ∩ [0, ∞) with q ≤ 1 and pq ≤ 1. Note that that H is a countable subset
of BL1 (D), and that any nonnegative function f ∈ BL1 (D) can be written as
f (x) = sup{h(x) : h ≤ f, h ∈ H}.
20

(B.2)

Indeed, either a picture helps, or the following formal argument: for fixed x ∈ D with c =
f (x) > 0, choose a sequence (sm )m in S converging to x and let δm = d(x, sm ). Without loss
of generality, we may assume that δm < c for all m. Choose cm ∈ Q such that 0 < cm < c − δm
+
and such that cm → c. Consider the function hm (z) = cm {1 − c−1
m d(z, sm )} . Then hm ∈ H
and hm (z) ≤ f (z) for all z ∈ D. Indeed, this bound is trivial for z with d(z, sm ) ≥ cm , and
otherwise, we have
c − f (z) = f (x) − f (z) ≤ d(x, z) ≤ d(x, sm ) + d(sm , z) = δm + d(sm , z),
which implies that
f (z) ≥ c − δm − d(sm , z) > cm − d(sm , z) = hm (z).
The assertion in (B.2) then follows from the fact that hm (x) = cm − δm → c = f (x) as m → ∞.

Next, let H∨ denote the set of functions which are maxima of a finite number of functions in
H. Note that H∨ ⊂ BL1 (D). Further, let H∨,− denote the the set of functions h = h1 − h2 with
h1 , h2 ∈ H∨ such that h ∈ BL1 (D). Note that H∨,− is still countable and a subset of BL1 (D).

Now, consider an arbitrary function f ∈ BL1 (D), and write f = f + −f − with f + = max(f, 0)
and f − = max(−f, 0). The supremum in (B.2), with f replaced by f + , is over countably many
+
+
+
m
functions. Denote them by h+
1 , h2 , . . . For m ∈ N, let gm = maxj=1 hj ∈ H∨ , such that
+ . Similarly, we can write f − as a limit of functions g − ∈ H . Note that g ±
f + = limm→∞ gm
∨
m
m
+ − g − is an element of
can only be positive on the support of f ± . As a consequence, gm = gm
m
+ (x)−g + (y)| ≤
H∨,− . Indeed, if x and y are both in the support of f + , then |gm (x)−gm (y)| = |gm
m
−
d(x, y). The case where both are in the support of f or where at least one of the points is
in neither support is similar. Finally, consider the case where x is in the support of f + and
y is in the support of f − , and without loss of generality assume that gm (x) > gm (y). Then,
+ (x) + g − (y) ≤ f + (x) + f − (y) = f (x) − f (y) ≤ d(x, y).
|gm (x) − gm (y)| = gm (x) − gm (y) = gm
m
This implies, by monotone convergence,
R
R
R
gm d(P̂α − dP ) ≤ supg∈H∨,−
gd(P̂α − dP ) .
f d(P̂α − dP ) = limm→∞

As a consequence, since f was arbitrary and since H∨,− ⊂ BL1 (D), we obtain that
R
dBL (P̂α , P ) = supg∈H∨,−
gd(P̂α − dP ) ,
and the assertion follows since the supremum on the right-hand side is a countable supremum
over measurable random variables.
Finally, consider the assertions regarding the Kolmogorov distance. Note that P̂α ((−∞, x]),
considered as map from Ω to R with x ∈ Rd fixed, is measurable. Hence, by right-continuity of
d.f.s, the Kolmogorov distance is measurable as well. To conclude, it is sufficient to show that
the convergence in (B.1) is equivalent to
P

P̂α ((−∞, x]) → P ((−∞, x])
for all x ∈ Rd ; see, e.g, Problem 23.1 in van der Vaart (1998). This equivalence in turn follows
by standard approximation arguments as in the proof of the classical Portmanteau Theorem;
see, e.g., van der Vaart (1998), Lemma 2.2.


21

