arXiv:1407.4546v7 [] 28 Sep 2016

The Annals of Statistics
2016, Vol. 44, No. 5, 1931–1956
DOI: 10.1214/15-AOS1375
c Institute of Mathematical Statistics, 2016

CRAMÉR-TYPE MODERATE DEVIATIONS FOR STUDENTIZED
TWO-SAMPLE U -STATISTICS WITH APPLICATIONS
By Jinyuan Chang1,∗,† , Qi-Man Shao2,‡ and Wen-Xin Zhou3,§,†
Southwestern University of Finance and Economics,∗ University of
Melbourne,† Chinese University of Hong Kong‡ and Princeton University§
Two-sample U -statistics are widely used in a broad range of applications, including those in the fields of biostatistics and econometrics. In this paper, we establish sharp Cramér-type moderate deviation theorems for Studentized two-sample U -statistics in a general framework, including the two-sample t-statistic and Studentized
Mann–Whitney test statistic as prototypical examples. In particular,
a refined moderate deviation theorem with second-order accuracy is
established for the two-sample t-statistic. These results extend the
applicability of the existing statistical methodologies from the onesample t-statistic to more general nonlinear statistics. Applications to
Tribute: Peter was a brilliant and prolific researcher, who has made enormously influential contributions to mathematical statistics and probability theory. Peter had extraordinary knowledge of analytic techniques that he often applied with ingenious simplicity
to tackle complex statistical problems. His work and service have had a profound impact
on statistics and the statistical community. Peter was a generous mentor and friend with
a warm heart and keen to help the young generation. Jinyuan Chang and Wen-Xin Zhou
are extremely grateful for the opportunity to learn from and work with Peter in the last
two years at the University of Melbourne. Even in his final year, he had afforded time to
guide us. We will always treasure the time we spent with him. Qi-Man Shao is so grateful
for all the helps and supports that Peter had provided during the various stages of his
career. Peter will be dearly missed and forever remembered as our mentor and friend.
Received June 2015.
1
Supported in part by the Fundamental Research Funds for the Central Universities
(Grant No. JBK160159, JBK150501), NSFC (Grant No. 11501462), the Center of Statistical Research at SWUFE and the Australian Research Council.
2
Supported by Hong Kong Research Grants Council GRF 603710 and 403513.
3
Supported by NIH Grant R01-GM100474-4 and a grant from the Australian Research
Council.
AMS 2000 subject classifications. Primary 60F10, 62E17; secondary 62E20, 62F40,
62H15.
Key words and phrases. Bootstrap, false discovery rate, Mann–Whitney U test, multiple hypothesis testing, self-normalized moderate deviation, Studentized statistics, twosample t-statistic, two-sample U -statistics.

This is an electronic reprint of the original article published by the
Institute of Mathematical Statistics in The Annals of Statistics,
2016, Vol. 44, No. 5, 1931–1956. This reprint differs from the original in
pagination and typographic detail.
1

2

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU
two-sample large-scale multiple testing problems with false discovery
rate control and the regularized bootstrap method are also discussed.

1. Introduction. The U -statistic is one of the most commonly used nonlinear and nonparametric statistics, and its asymptotic theory has been well
studied since the seminal paper of Hoeffding (1948). U -statistics extend the
scope of parametric estimation to more complex nonparametric problems
and provide a general theoretical framework for statistical inference. We refer to Koroljuk and Borovskich (1994) for a systematic presentation of the
theory of U -statistics, and to Kowalski and Tu (2007) for more recently
discovered methods and contemporary applications of U -statistics.
Applications of U -statistics can also be found in high dimensional statistical inference and estimation, including the simultaneous testing of many
different hypotheses, feature selection and ranking, the estimation of high
dimensional graphical models and sparse, high dimensional signal detection. In the context of high dimensional hypothesis testing, for example,
several new methods based on U -statistics have been proposed and studied
in Chen and Qin (2010), Chen, Zhang and Zhong (2010) and Zhong and
Chen (2011). Moreover, Li et al. (2012) and Li, Zhong and Zhu (2012) employed U -statistics to construct independence feature screening procedures
for analyzing ultrahigh dimensional data.
Due to heteroscedasticity, the measurements across disparate subjects
may differ significantly in scale for each feature. To standardize for scale,
unknown nuisance parameters are always involved and a natural approach
is to use Studentized, or self-normalized statistics. The noteworthy advantage of Studentization is that compared to standardized statistics, Studentized ratios take heteroscedasticity into account and are more robust against
heavy-tailed data. The theoretical and numerical studies in Delaigle, Hall
and Jin (2011) and Chang, Tang and Wu (2013, 2016) evidence the importance of using Studentized statistics in high dimensional data analysis. As
noted in Delaigle, Hall and Jin (2011), a careful study of the moderate
deviations in the Studentized ratios is indispensable to understanding the
common statistical procedures used in analyzing high dimensional data.
Further, it is now known that the theory of Cramér-type moderate deviations for Studentized statistics quantifies the accuracy of the estimated
p-values, which is crucial in the study of large-scale multiple tests for controlling the false discovery rate [Fan, Hall and Yao (2007), Liu and Shao
(2010)]. In particular, Cramér-type moderate deviation results can be used
to investigate the robustness and accuracy properties of p-values and critical
values in multiple testing procedures. However, thus far, most applications
have been confined to t-statistics [Fan, Hall and Yao (2007), Wang and Hall
(2009), Delaigle, Hall and Jin (2011), Cao and Kosorok (2011)]. It is conjectured in Fan, Hall and Yao (2007) that analogues of the theoretical properties

STUDENTIZED TWO-SAMPLE U -STATISTICS

3

of these statistical methodologies remain valid for other resampling methods based on Studentized statistics. Motivated by the above applications,
we are attempting to develop a unified theory on moderate deviations for
more general Studentized nonlinear statistics, in particular, for two-sample
U -statistics.
The asymptotic properties of the standardized U -statistics are extensively
studied in the literature, whereas significant developments are achieved in
the past decade for one-sample Studentized U -statistics. We refer to Wang,
Jing and Zhao (2000) and the references therein for Berry–Esseen-type
bounds and Edgeworth expansions. The results for moderate deviations can
be found in Vandemaele and Veraverbeke (1985), Lai, Shao and Wang (2011)
and Shao and Zhou (2016). The results in Shao and Zhou (2016) paved the
way for further applications of statistical methodologies using Studentized
U -statistics in high dimensional data analysis.
Two-sample U -statistics are also commonly used to compare the different (treatment) effects of two groups, such as an experimental group and
a control group, in scientifically controlled experiments. However, due to
the structural complexities, the theoretical properties of the two-sample U statistics have not been well studied. In this paper, we establish a Cramértype moderate deviation theorem in a general framework for Studentized
two-sample U -statistics, especially the two-sample t-statistic and the Studentized Mann–Whitney test. In particular, a refined moderate deviation
theorem with second-order accuracy is established for the two-sample tstatistic.
The paper is organized as follows. In Section 2, we present the main
results on Cramér-type moderate deviations for Studentized two-sample U statistics as well as a refined result for the two-sample t-statistic. In Section 3, we investigate statistical applications of our theoretical results to the
problem of simultaneously testing many different hypotheses, based particularly on the two-sample t-statistics and Studentized Mann–Whitney tests.
Section 4 shows numerical studies. A discussion is given in Section 5. All the
proofs are relegated to the supplementary material [Chang, Shao and Zhou
(2016)].
2. Moderate deviations for Studentized U -statistics. We use the following notation throughout this paper. For two sequences of real numbers an
and bn , we write an ≍ bn if there exist two positive constants c1 , c2 such that
c1 ≤ an /bn ≤ c2 for all n ≥ 1, we write an = O(bn ) if there is a constant C
such that |an | ≤ C|bn | holds for all sufficiently large n, and we write an ∼ bn
and an = o(bn ), respectively, if limn→∞ an /bn = 1 and limn→∞ an /bn = 0.
Moreover, for two real numbers a and b, we write for ease of presentation
that a ∨ b = max(a, b) and a ∧ b = min(a, b).

4

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

2.1. A review of Studentized one-sample U -statistics. We start with a
brief review of Cramér-type moderate deviation for Studentized one-sample
U -statistics. For an integer s ≥ 2 and for n > 2s, let X1 , . . . , Xn be independent and identically distributed (i.i.d.) random variables taking values in a
metric space (X, G), and let h : Xd 7→ R be a symmetric Borel measurable
function. Hoeffding’s U -statistic with a kernel h of degree s is defined as
Un =

X

1
n
s

h(Xi1 , . . . , Xis ),

1≤i1 <···<is ≤n

which is an unbiased estimate of θ = E{h(X1 , . . . , Xs )}. In particular, we
focus on the case where X is the Euclidean space Rr for some integer r ≥ 1.
When r ≥ 2, write Xi = (Xi1 , . . . , Xir )T for i = 1, . . . , n.
Let
h1 (x) = E{h(X1 , . . . , Xs )|X1 = x}

for any x = (x1 , . . . , xr )T ∈ Rr

and
σ 2 = var{h1 (X1 )},

vh2 = var{h(X1 , X2 , . . . , Xs )}.

Assume that 0 < σ 2 < ∞, then the standardized nondegenerate U -statistic
is given by
Zn =

n1/2
(Un − θ).
sσ

Because σ is usually unknown, we are interested in the following Studentized U -statistic:
1/2
bn = n (Un − θ),
U
sb
σ

(2.1)

where σ
b2 denotes the leave-one-out jackknife estimator of σ 2 given by
n

(n − 1) X
(qi − Un )2
σ
b =
(n − s)2
2

with

i=1

qi =

1

n−1
s−1

X

h(Xi , Xℓ1 , . . . , Xℓs−1 ).

1≤ℓ1 <···<ℓs−1 ≤n
ℓj 6=i for each j=1,...,s−1

Shao and Zhou (2016) established a general Cramér-type moderate deviation theorem for Studentized nonlinear statistics, in particular for Studentized U -statistics.

5

STUDENTIZED TWO-SAMPLE U -STATISTICS

Theorem 2.1. Assume that vp := [E{|h1 (X1 ) − θ|p }]1/p < ∞ for some
2 < p ≤ 3. Suppose that there are constants c0 ≥ 1 and κ ≥ 0 such that for
all x1 , . . . , xs ∈ R,
#
"
s
X
2
2
2
(2.2)
{h1 (xi ) − θ} .
{h(x1 , . . . , xs ) − θ} ≤ c0 κσ +
i=1

Then there exist constants C, c > 0 depending only on d such that
bn ≥ x)
P(U
= 1 + O(1){(vp /σ)p (1 + x)p n1−p/2 + (as1/2 + vh /σ)(1 + x)3 n−1/2 }
1 − Φ(x)

holds uniformly for 0 ≤ x ≤ c min{(σ/vp )n1/2−1/p , (n/as )1/6 }, where |O(1)| ≤
C and as = max(c0 κ, c0 + s). In particular, we have
bn ≥ x)
P(U
→1
1 − Φ(x)

holds uniformly in x ∈ [0, o(n1/2−1/p )).

Condition (2.2) is satisfied for a large class of U -statistics. Below are some
examples.
Statistic
t-statistic
Sample variance
Gini’s mean difference
One-sample Wilcoxon’s statistic
Kendall’s τ

Kernel function
h(x1 , x2 ) = 0.5(x1 + x2 )
h(x1 , x2 ) = 0.5(x1 − x2 )2
h(x1 , x2 ) = |x1 − x2 |
h(x1 , x2 ) = I{x1 + x2 ≤ 0}
h(x1 , x2 ) = 2I{(x22 − x21 )(x12 − x11 ) > 0}

c0
2
10
8
1
1

κ
0
(θ/σ)2
(θ/σ)2
σ −2
σ −2

2.2. Studentized two-sample U -statistics. Let X = {X1 , . . . , Xn1 } and Y =
{Y1 , . . . , Yn2 } be two independent random samples, where X is drawn from
a probability distribution P and Y is drawn from another probability distribution Q. With s1 and s2 being two positive integers, let
h(x1 , . . . , xs1 ; y1 , . . . , ys2 )
be a kernel function of order (s1 , s2 ) which is real and symmetric both in its
first s1 variates and in its last s2 variates. It is known that a nonsymmetric kernel can always be replaced with a symmetrized version by averaging
across all possible rearrangements of the indices.
Set θ := E{h(X1 , . . . , Xs1 ; Y1 , . . . , Ys2 )}, and let
Un̄ =

1

X

X

h(Xi1 , . . . , Xis1 ; Yj1 , . . . , Yjs2 ),
n1  n2 
s1 s2 1≤i1 <···<is ≤n1 1≤j1 <···<js ≤n2
2
1

6

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

be the two-sample U -statistic, where n̄ = (n1 , n2 ). To lighten the notation,
we write Xi1 ,...,iℓ = (Xi1 , . . . , Xiℓ ), Yj1 ,...,jk = (Yj1 , . . . , Yjk ) such that
h(Xi1 ,...,iℓ ; Yj1 ,...,jk ) = h(Xi1 , . . . , Xiℓ ; Yj1 , . . . , Yjk ),
and define
h1 (x) = E{h(X1,...,s1 ; Y1,...,s2 )|X1 = x},

(2.3)

h2 (y) = E{h(X1,...,s1 ; Y1,...,s2 )|Y1 = y}.
Also let vh2 = var{h(X1,...,s1 ; Y1,...,s2 )}, σ12 = var{h1 (Xi )}, σ22 = var{h2 (Yj )}
and
σ 2 = σ12 + σ22 ,

(2.4)

2 2 −1
σn̄2 = s21 σ12 n−1
1 + s 2 σ2 n 2 .

For the standardized two-sample U -statistic of the form σn̄−1 (Un̄ − θ), a
uniform Berry–Esseen bound of order O{(n1 ∧ n2 )−1/2 } was obtained by
Helmers and Janssen (1982) and Borovskich (1983). Using a concentration
inequality approach, Chen and Shao (2007) proved a refined uniform bound
and also established an optimal nonuniform Berry–Esseen bound. For large
deviation asymptotics of two-sample U -statistics, we refer to Nikitin and
Ponikarov (2006) and the references therein.
Here, we are interested in the following Studentized two-sample U -statistic:
(2.5)
where

bn̄ = σ
U
bn̄−1 (Un̄ − θ)
n

n

i=1

i=1

1
1
1 X
1 X
σ
b12 =
qi
qi −
n1 − 1
n1

and

2 2 −1
with σ
bn̄2 = s21 σ
b12 n−1
b2 n2 ,
1 + s2 σ

!2

,

n

n

j=1

j=1

2
2
1 X
1 X
pj
pj −
σ
b22 =
n2 − 1
n2

1

X

X

1

X

X

!2

qi =

h(Xi,i2 ,...,is1 ; Yj1 ,...,js2 ),
n1 −1 n2 
s1 −1 s2 1≤i2 <···<is1 ≤n1 1≤j1 <···<js2 ≤n2
iℓ 6=i,ℓ=2,...,s1

pj =

h(Xi1 ,...,is1 ; Yj,j2,...,js2 ).
n1  n2 −1
s1 s2 −1 1≤i1 <···<is1 ≤n1 1≤j2 <···<js2 ≤n2
jk 6=j,k=2,...,s2

Note that σ
b12 and σ
b22 are leave-one-out jackknife estimators of σ12 and σ22 ,
respectively.

7

STUDENTIZED TWO-SAMPLE U -STATISTICS

bn̄ . For p > 2, let
2.2.1. Moderate deviations for U

(2.6) v1,p = [E{|h1 (X1 ) − θ|p }]1/p

and v2,p = [E{|h2 (Y1 ) − θ|p }]1/p .

Moreover, put
s = s1 ∨ s2 ,

n̄ = (n1 , n2 ),

n = n1 ∧ n2

and
λn̄ = vh



n1 + n2
σ12 n2 + σ22 n1

1/2

with vh2 = var{h(X1,...,s1 ; Y1,...,s2 )}.

bn̄ given
The following result gives a Cramér-type moderate deviation for U
in (2.5) under mild assumptions. A self-contained proof can be found in the
supplementary material [Chang, Shao and Zhou (2016)].
Assume that there are constants c0 ≥ 1 and κ ≥ 0 such

Theorem 2.2.
that
2

(2.7) {h(x; y) − θ} ≤ c0

"

s2
s1
X
X
2
2
{h2 (yj ) − θ}2
{h1 (xi ) − θ} +
κσ +
j=1

i=1

#

for all x = (x1 , . . . , xs1 ) and y = (y1 , . . . , ys2 ), where σ 2 is given in (2.4).
Assume that v1,p and v2,p are finite for some 2 < p ≤ 3. Then there exist
constants C, c > 0 independent of n1 and n2 such that
bn̄ ≥ x)
P(U
1 − Φ(x)
(2.8)

(

= 1 + O(1)

holds uniformly for

p
2
X
vℓ,p
(1 + x)p
p/2−1

ℓ=1

σℓp nℓ

p/2−1

0 ≤ x ≤ c min[(σ1 /v1,p )n1

1/2
+ (ad

3

+ λn̄ )(1 + x)

p/2−1

, (σ2 /v2,p )n2



n1 + n2
n1 n2

1/2 )

, a−1/6
{n1 n2 /(n1 + n2 )}1/6 ],
s

where |O(1)| ≤ C and as = max(c0 κ, c0 + s). In particular, as n → ∞,
(2.9)

bn̄ ≥ x)
P(U
→1
1 − Φ(x)

holds uniformly in x ∈ [0, o(n1/2−1/p )).

Theorem 2.2 exhibits the dependence between the range of uniform convergence of the relative error in the central limit theorem and the optimal moment conditions. In particular, if p = 3, the region becomes 0 ≤ x ≤

8

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

O(n1/6 ). See Theorem 2.3 in Jing, Shao and Wang (2003) for similar results
on self-normalized sums. Under higher order moment conditions, it is not
clear if our technique can be adapted to provide a better approximation for
bn̄ ≥ x) for x lying between n1/6 and n1/2 in order.
the tail probability P(U
It is also worth noticing that many commonly used kernels in nonparametric statistics turn out to be linear combinations of the indicator functions
and, therefore, satisfy condition (2.7) immediately.
2.2.2. Two-sample t-statistic. As a prototypical example of two-sample
U -statistics, the two-sample t-statistic is of significant interest due to its
wide applicability. The advantage of using t-tests, either one-sample or twosample, is their high degree of robustness against heavy-tailed data in which
the sampling distribution has only a finite third or fourth moment. The robustness of the t-statistic is useful in high dimensional data analysis under
the sparsity assumption on the signal of interest. When dealing with two
experimental groups, which are typically independent, in scientifically controlled experiments, the two-sample t-statistic is one of the most commonly
used statistics for hypothesis testing and constructing confidence intervals
for the difference between the means of the two groups.
Let X = {X1 , . . . , Xn1 } be a random sample from a one-dimensional population with mean µ1 and variance σ12 , and let Y = {Y1 , . . . , Yn2 } be a random
sample from another one-dimensional population with mean µ2 and variance
σ22 independent of X . The two-sample t-statistic is defined as
Tbn̄ = q

X̄ − Ȳ

,
2 n−1
σ
b12 n−1
+
σ
b
1
2 2
Pn1
−1 Pn2
where n̄ = (n1 , n2 ), X̄ = n−1
1
j=1 Yj and
i=1 Xi , Ȳ = n2
n

σ
b12

1
1 X
(Xi − X̄)2 ,
=
n1 − 1

i=1

n

σ
b22

2
1 X
=
(Yj − Ȳ )2 .
n2 − 1

j=1

The following result is a direct consequence of Theorem 2.2.
Theorem 2.3. Assume that µ1 = µ2 , and E(|X1 |p ) < ∞, E(|Y1 |p ) < ∞
for some 2 < p ≤ 3. Then there exist absolute constants C, c > 0 such that
2

X
P(Tbn̄ ≥ x)
1−p/2
= 1 + O(1)(1 + x)p
(vℓ,p /σℓ )p nℓ
1 − Φ(x)
ℓ=1

1/2−1/p

holds uniformly for 0 ≤ x ≤ c minℓ=1,2 {(σℓ /vℓ,p )nℓ
}, where |O(1)| ≤ C
p
1/p
p
1/p
and v1,p = {E(|X1 − µ1 | )} , v2,p = {E(|Y1 − µ2 | )} .

9

STUDENTIZED TWO-SAMPLE U -STATISTICS

Motivated by a series of recent studies on the effectiveness and accuracy
of multiple-hypothesis testing using t-tests, we investigate whether a higher
order expansion of the relative error, as in Theorem 1.2 of Wang (2005) for
self-normalized sums, holds for the two-sample t-statistic, so that one can
use bootstrap calibration to correct skewness [Fan, Hall and Yao (2007),
Delaigle, Hall and Jin (2011)] or study power properties against sparse alternatives [Wang and Hall (2009)]. The following theorem gives a refined
Cramér-type moderate deviation result for Tbn̄ , whose proof is placed in the
supplementary material [Chang, Shao and Zhou (2016)].
Theorem 2.4. Assume that µ1 = µ2 . Let γ1 = E{(X1 − µ1 )3 } and γ2 =
E{(Y1 − µ2 )3 } be the third central moment of X1 and Y1 , respectively. Moreover, assume that E(|X1 |p ) < ∞, E(|Y1 |p ) < ∞ for some 3 < p ≤ 4. Then


−2
P(Tbn̄ ≥ x)
γ1 n−2
3
1 − γ2 n2
x
= exp −
2 −1 3/2
1 − Φ(x)
3(σ12 n−1
1 + σ2 n 2 )
(2.10)
"
#
p
2  3
X
vℓ,3 (1 + x) vℓ,p (1 + x)p
+ p p/2−1
× 1 + O(1)
1/2
σℓ3 nℓ
σℓ n ℓ
ℓ=1
holds uniformly for
(2.11)

1/2

1/2−1/p

0 ≤ x ≤ c min min{(σℓ /vℓ,3 )3 nℓ , (σℓ /vℓ,p )nℓ
ℓ=1,2

},

where |O(1)| ≤ C and for every q ≥ 1, v1,q = {E(|X1 − µ1 |q )}1/q , v2,q =
{E(|Y1 − µ2 |q )}1/q .
A refined Cramér-type moderate deviation theorem for the one-sample tstatistic was established in Wang (2011), which to our knowledge, is the best
result for the t-statistic known up to date, or equivalently, self-normalized
sums.
2.2.3. More examples of two-sample U -statistics. Beyond the two-sample
t-statistic, we enumerate three more well-known two-sample U -statistics
and refer to Nikitin and Ponikarov (2006) for more examples. Let X =
{X1 , . . . , Xn1 } and Y = {Y1 , . . . , Yn2 } be two independent random samples
from population distributions P and Q, respectively.
Example 2.1 (The Mann–Whitney test statistic).
order (s1 , s2 ) = (1, 1), defined as
h(x; y) = I{x ≤ y} − 1/2

The kernel h is of

with θ = P(X1 ≤ Y1 ) − 1/2,

10

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

and in view of (2.3),
h1 (x) = 1/2 − G(x),

h2 (y) = F (y) − 1/2.

In particular, if F ≡ G, we have σ12 = σ22 = 1/12.
Example 2.2 (The Lehmann statistic).
(2, 2), defined as

The kernel h is of order (s1 , s2 ) =

h(x1 , x2 ; y1 , y2 ) = I{|x1 − x2 | ≤ |y1 − y2 |} − 1/2
with θ = P(|X1 − X2 | ≤ |Y1 − Y2 |) − 1/2. Then under H0 : θ = 0, E{h(X1 , X2 ;
Y1 , Y2 )} = 0, and
h1 (x) = G(x){1 − G(x)} − 1/6,

h2 (y) = F (y){F (y) − 1} + 1/6.

In particular, if F ≡ G, then σ12 = σ22 = 1/180.
Example 2.3 (The Kochar statistic). The Kochar statistic was constructed by Kochar (1979) to test if the two hazard failure rates are different.
Denote by F the class of all absolutely continuous cumulative distribution
functions (CDF) F (·) satisfying F (0) = 0. For two arbitrary CDF’s F, G ∈ F ,
and let f = F ′ , g = G′ be their densities. Thus, the hazard failure rates are
defined by
rF (t) =

f (t)
,
1 − F (t)

rG (t) =

g(t)
,
1 − G(t)

as long as both 1 − F (t) and 1 − G(t) are positive. Kochar (1979) considered the problem of testing the null hypothesis H0 : rF (t) = rG (t) against
the alternative H1 : rF (t) ≤ rG (t), t ≥ 0 with strict inequality over a set of
nonzero measures. Observe that H1 holds if and only if δ(s, t) = F̄ (s)Ḡ(t) −
F̄ (t)Ḡ(s) ≥ 0 for s ≥ t ≥ 0 with strict inequality over a set of nonzero measures, where F̄ (·) := 1 − F (·) for any F ∈ F .
Recall that X1 , . . . , Xn1 and Y1 , . . . , Yn2 are two independent samples
drawn respectively from F and G. Following Nikitin and Ponikarov (2006),
we see that
η(F ; G) = E{δ(X ∨ Y, X ∧ Y )}
= P(Y1 ≤ Y2 ≤ X1 ≤ X2 ) + P(X1 ≤ Y2 ≤ Y2 ≤ X2 )
− P(X1 ≤ X2 ≤ Y1 ≤ Y2 ) − P(Y1 ≤ X1 ≤ X2 ≤ Y2 ).
Under H0 , η(F ; G) = 0 while under H1 , η(F ; G) > 0. The U -statistic with
the kernel of order (s1 , s2 ) = (2, 2) is given by
h(x1 , x2 ; y1 , y2 ) = I{yyxx or xyyx} − I{xxyy or yxxy}.

STUDENTIZED TWO-SAMPLE U -STATISTICS

11

Here, the term “yyxx” refers to y1 ≤ y2 ≤ x1 ≤ x2 and similar treatments
apply to xyyx, xxyy and yxxy. Under H0 : rF (t) = rG (t), we have
h1 (x) = −4G3 (x)/3 + 4G2 (x) − 2G(x),
h2 (y) = 4F 3 (y)/3 − 4F 2 (y) + 2F (y).
In particular, if F ≡ G, then σ12 = σ22 = 8/105.
3. Multiple testing via Studentized two-sample tests. Multiple-hypothesis
testing occurs in a wide range of applications including DNA microarray
experiments, functional magnetic resonance imaging analysis (fMRI) and
astronomical surveys. We refer to Dudoit and van der Laan (2008) for a systematic study of the existing multiple testing procedures. In this section, we
consider multiple-hypothesis testing based on Studentized two-sample tests
and show how the theoretical results in the previous section can be applied
to these problems.
3.1. Two-sample t-test. A typical application of multiple-hypothesis testing in high dimensions is the analysis of gene expression microarray data.
To see whether each gene in isolation behaves differently in a control group
versus an experimental group, we can apply the two-sample t-test. Assume
that the statistical model is given by

Xi,k = µ1k + εi,k ,
i = 1, . . . , n1 ,
(3.1)
Yj,k = µ2k + ωj,k ,
j = 1, . . . , n2 ,
for k = 1, . . . , m, where index k denotes the kth gene, i and j indicate the
ith and jth array, and the constants µ1k and µ2k , respectively, represent the
mean effects for the kth gene from the first and the second groups. For each k,
ε1,k , . . . , εn1 ,k (resp., ω1,k , . . . , ωn2 ,k ) are independent random variables with
2 > 0 (resp., σ 2 > 0). For the kth marginal test,
mean zero and variance σ1k
2k
2 and σ 2 are unequal, the two-sample twhen the population variances σ1k
2k
statistic is most commonly used to carry out hypothesis testing for the null
H0k : µ1k = µ2k against the alternative H1k : µ1k 6= µ2k .
Since the seminal work of Benjamini and Hochberg (1995), the Benjamini
and Hochberg (B–H) procedure has become a popular technique in microarray data analysis for gene selection, which along with many other procedures
depend on p-values that often need to be estimated. To control certain simultaneous errors, it has been shown that using approximated p-values is
asymptotically equivalent to using the true p-values for controlling the kfamilywise error rate (k-FWER) and false discovery rate (FDR). See, for
example, Kosorok and Ma (2007), Fan, Hall and Yao (2007) and Liu and
Shao (2010) for one-sample tests. Cao and Kosorok (2011) proposed an alternative method to control k-FWER and FDR in both large-scale one-

12

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

and two-sample t-tests. A common thread among the aforementioned literature is that theoretically for the methods to work in controlling FDR at a
given level, the number of features m and the sample size n should satisfy
log m = o(n1/3 ).
Recently, Liu and Shao (2014) proposed a regularized bootstrap correction
method for multiple one-sample t-tests so that the constraint on m may
be relaxed to log m = o(n1/2 ) under less stringent moment conditions as
assumed in Fan, Hall and Yao (2007) and Delaigle, Hall and Jin (2011). Using
Theorem 2.4, we show that the constraint on m in large scale two-sample
t-tests can be relaxed to log m = o(n1/2 ) as well. This provides theoretical
justification of the effectiveness of the bootstrap method which is frequently
used for skewness correction.
To illustrate the main idea, here we restrict our attention to the special
case in which the observations are independent. Indeed, when test statistics are correlated, false discovery control becomes very challenging under
arbitrary dependence. Various dependence structures have been considered
in the literature. See, for example, Benjamini and Yekutieli (2001), Storey,
Taylor and Siegmund (2004), Ferreira and Zwinderman (2006), Leek and
Storey (2008), Friguet, Kloareg and Causeur (2009) and Fan, Han and Gu
(2012), among others. For completeness, we generalize the results to the
dependent case in Section 3.1.3.
3.1.1. Normal calibration and phase transition. Consider the large-scale
significance testing problem:
H0k : µ1k = µ2k

versus

H1k : µ1k 6= µ2k ,

1 ≤ k ≤ m.

Let V and R denote, respectively, the number of false rejections and the
number of total rejections. The well-known false discovery proportion (FDP)
is defined as the ratio FDP = V / max(1, R), and FDR is the expected FDP,
that is, E{V / max(1, R)}. Benjamini and Hochberg (1995) proposed a
distribution-free method for choosing a p-value threshold that controls the
FDR at a prespecified level where 0 < α < 1. For k = 1, . . . , m, let pk be
the marginal p-value of the kth test, and let p(1) ≤ · · · ≤ p(m) be the order
statistics of p1 , . . . , pm . For a predetermined control level α ∈ (0, 1), the B–H
procedure rejects hypotheses for which pk ≤ p(k̂) , where


αk
k̂ = max 0 ≤ k ≤ m : p(k) ≤
(3.2)
m
with p(0) = 0.
In microarray analysis, two-sample t-tests are often used to identify differentially expressed genes between two groups. Let
Tk = q

X̄k − Ȳk
2 n−1 + σ
2 n−1
σ
b1k
b2k
1
2

,

k = 1, . . . , m,

STUDENTIZED TWO-SAMPLE U -STATISTICS

where X̄k = n−1
1

Pn 1

i=1 Xi,k ,

Ȳk = n−1
2

n

2
=
σ
b1k

1
1 X
(Xi,k − X̄k )2 ,
n1 − 1

i=1

Pn 2

j=1 Yj,k
2
=
σ
b2k

13

and
n

2
1 X
(Yj,k − Ȳk )2 .
n2 − 1

j=1

1
2
Here and below, {Xi,1 , . . . , Xi,m }ni=1
and {Yj,1 , . . . , Yj,m}nj=1
are independent
random samples from {X1 , . . . , Xm } and {Y1 , . . . , Ym }, respectively, generated according to model (3.1), which are usually non-Gaussian in practice.
Moreover, assume that the sample sizes of the two samples are of the same
order, that is, n1 ≍ n2 .
Before stating the main results, we first introduce a number of notation.
Set H0 = {1 ≤ k ≤ m : µ1k = µ2k }, let m0 = #H0 denote the number of true
null hypotheses and m1 = m − m0 . Both m = m(n1 , n2 ) and m0 = m0 (n1 , n2 )
are allowed to grow as n = n1 ∧ n2 increases. We assume that
m0
= π0 ∈ (0, 1].
lim
n→∞ m
In line with the notation used in Section 2, set

2
= var(Xk ),
σ1k

2
= var(Yk ),
σ2k

γ1k = E{(Xk − µ1k )3 },

γ2k = E{(Yk − µ2k )3 }

2 = σ 2 n−1 + σ 2 n−1 . Throughout this subsection, we focus on the
and σn̄,k
2k 2
1k 1
normal calibration and let pbk = 2 − 2Φ(|Tk |), where Φ(·) is the standard
normal distribution function. Indeed, the exact null distribution of Tk and
thus the true p-values are unknown without the normality assumption.

Theorem 3.1. Assume that {X1 , . . . , Xm , Y1 , . . . , Ym } are independent
nondegenerate random variables; n1 ≍ n2 , m = m(n1 , n2 ) → ∞ and log m =
o(n1/2 ) as n = n1 ∧ n2 → ∞. For independent random samples {Xi,1 , . . . ,
1
2
Xi,m }ni=1
and {Yj,1 , . . . , Yj,m}nj=1
, suppose that
max max{E(ξk4 ), E(ηk4 )} ≤ C < ∞

(3.3) min min(σ1k , σ2k ) ≥ c > 0,
1≤k≤m

1≤k≤m

−1
−1
for some constants C and c, where ξk = σ1k
(Xk − µ1k ) and ηk = σ2k
(Yk −
µ2k ). Moreover, assume that

(3.4)

#{1 ≤ k ≤ m : |µ1k − µ2k | ≥ 4(log m)1/2 σn̄,k } → ∞

as n → ∞, and let
(3.5)

c0 = lim inf



n,m→∞


n1/2 X −3
−2
−2
σn̄,k |γ1k n1 − γ2k n2 | .
m0
k∈H0

14

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

(i) Suppose that log m = o(n1/3 ). Then as n → ∞, FDPΦ →P απ0 and
FDRΦ → απ0 .
(ii) Suppose that c0 > 0, log m ≥ c1 n1/3 for some c1 > 0 and that log m1 =
1/3
o(n ). Then there exists some constant β ∈ (α, 1] such that
lim P(FDPΦ ≥ β) = 1

n→∞

and

lim inf FDRΦ ≥ β.
n→∞

(log m)/n1/3

(iii) Suppose that c0 > 0,
→ ∞ and log m1 = o(n1/3 ). Then
as n → ∞, FDPΦ →P 1 and FDRΦ → 1.
Here, FDRΦ and FDPΦ denote, respectively, the FDR and the FDP of the
B–H procedure with pk replaced by pbk in (3.2).

Together, conclusions (i) and (ii) of Theorem 3.1 indicate that the number
of simultaneous tests can be as large as exp{o(n1/3 )} before the normal calibration becomes inaccurate. In particular, when n1 = n2 = n, the skewness
parameter c0 given in (3.5) reduces to


1 X |γ1k − γ2k |
.
c0 = lim inf
2 + σ 2 )3/2
m→∞
m0
(σ1k
2k
k∈H
0

As noted in Liu and Shao (2014), the limiting behavior of the FDRΦ varies
in different regimes and exhibits interesting phase transition phenomena as
the dimension m grows as a function of (n1 , n2 ). The average of skewness c0
plays a crucial role. It is also worth noting that conclusions (ii) and (iii) hold
under the scenario π0 = 1, that is, m1 = o(m). This corresponds to the sparse
settings in applications such as gene detections. Under finite 4th moments of
Xk and Yk , the robustness of two-sample t-tests and the accuracy of normal
calibration in the FDR/FDP control have been investigated in Cao and
Kosorok (2011) when m1 /m → π1 ∈ (0, 1). This corresponds to the relatively
dense setting, and the sparse case that we considered above is not covered.
3.1.2. Bootstrap calibration and regularized bootstrap correction. In this
subsection, we first use the conventional bootstrap calibration to improve
the accuracy of FDR control based on the fact that the bootstrap approximation removes the skewness term that determines first-order inaccuracies
of the standard normal approximation. However, the validity of bootstrap
approximation requires the underlying distribution to be very light tailed,
which does not seem realistic in real data applications. As pointed in the
literature of gene study, many gene data are commonly recognized to have
heavy tails which violates the assumption on underlying distribution used to
make conventional bootstrap approximation work. Recently, Liu and Shao
(2014) proposed a regularized bootstrap method that is shown to be more
robust against the heavy tailedness of the underlying distribution and the
dimension m is allowed to be as large as exp{o(n1/2 )}.

15

STUDENTIZED TWO-SAMPLE U -STATISTICS

†
†
†
†
Let Xk,b
= {X1,k,b
, . . . , Xn† 1 ,k,b }, Yk,b
= {Y1,k,b
, . . . , Yn†2 ,k,b }, b = 1, . . . , B,
denote bootstrap samples drawn independently and uniformly, with replacement, from Xk = {X1,k , . . . , Xn1 ,k } and Yk = {Y1,k , . . . , Yn2 ,k }, respectively.
†
†
Let Tk,b
be the two-sample t-statistic constructed from {X1,k,b
− X̄k , . . . ,
†
Xn† 1 ,k,b − X̄k } and {Y1,k,b
− Ȳk , . . . , Yn†2 ,k,b − Ȳk }. Following Liu and Shao
(2014), we use the following empirical distribution:
m

†
Fm,B
(t) =

B

1 XX
†
I{|Tk,b
| ≥ t}
mB
k=1 b=1

to approximate the null distribution, and thus the estimated p-values are
†
given by pbk,B = Fm,B
(|Tk |). Respectively, FDPB and FDRB denote the FDP
and the FDR of the B–H procedure with pk replaced by pbk,B in (3.2).
The following result shows that the bootstrap calibration is accurate provided log m increases at a strictly slower rate than (n1 ∧ n2 )1/2 , and the
underlying distribution has sub-Gaussian tails.
Theorem 3.2.

Assume the conditions in Theorem 3.1 hold and that
2

2

max max{E(et0 ξk ), E(et0 ηk )} ≤ C < ∞

1≤k≤m

for some constants t0 , C > 0.
(i) Suppose that log m = o(n1/3 ). Then as n → ∞, FDPB →P απ0 and
FDRB → απ0 .
(ii) Suppose that log m = o(n1/2 ) and m1 ≤ mρ for some ρ ∈ (0, 1). Then
as n → ∞, FDPB →P α and FDRB → α.
The sub-Gaussian condition in Theorem 3.2 is quite stringent in practice,
whereas it can hardly be weakened in general when the bootstrap method
is applied. In the context of family-wise error rate control, Fan, Hall and
Yao (2007) proved that the bootstrap calibration is accurate if the observed
data are bounded and log m = o(n1/2 ). The regularized bootstrap method,
however, adopts the very similar idea of the trimmed estimators and is a twostep procedure that combines the truncation technique and the bootstrap
method.
First, define the trimmed samples
bi,k = Xi,k I{|Xi,k | ≤ λ1k },
X
Ybj,k = Yi,k I{|Yj,k | ≤ λ2k }

for i = 1, . . . , n1 , j = 1, . . . , n2 , where λ1k and λ2k are regularized parameters
†
b† , . . . , X
b†
b†
b†
b†
to be specified. Let Xbk,b
= {X
1,k,b
n1 ,k,b } and Yk,b = {Y1,k,b , . . . , Yn2 ,k,b },
b = 1, . . . , B, be the corresponding bootstrap samples drawn by sampling
randomly, with replacement, from
b1,k , . . . , X
bn ,k } and Ybk = {Yb1,k , . . . , Ybn ,k },
Xbk = {X
1

2

16

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

†
respectively. Next, let Tbk,b
be the two-sample t-test statistic constructed from
P
†
n
−1
−1 Pn1 b
−1
1
b†
b
b†
b
{X
i=1 Xi,k } and {Y1,k,b − n2 ×
i=1 Xi,k , . . . , Xn1 ,k,b − n1
1,k,b − n1
Pn 2 b
−1 Pn2 b
b†
j=1 Yj,k }. As in the previous procedure, define
j=1 Yj,k , . . . , Yn2 ,k,b − n2
the estimated p-values by
m

†
pbk,RB = Fbm,RB
(|Tk |)

B

1 XX
†
†
with Fbm,RB
(t) =
I{|Tbk,b
| ≥ t}.
mB
k=1 b=1

Let FDPRB and FDRRB denote the FDP and the FDR, respectively, of the
B–H procedure with pk replaced by pbk,RB in (3.2).
Theorem 3.3.

(3.6)

Assume the conditions in Theorem 3.1 hold and that

max max{E(|Xk |6 ), E(|Yk |6 )} ≤ C < ∞.

1≤k≤m

The regularized parameters (λ1k , λ2k ) are such that


1/6
1/6
n1
n2
λ1k ≍
(3.7)
and λ2k ≍
.
log m
log m
(i) Suppose that log m = o(n1/3 ). Then as n → ∞, FDPRB →P απ0 and
FDRRB → απ0 .
(ii) Suppose that log m = o(n1/2 ) and m1 ≤ mρ for some ρ ∈ (0, 1). Then
as n → ∞, FDPRB →P α and FDRRB → α.
In view of Theorem 3.3, the regularized bootstrap approximation is valid
under mild moment conditions that are significantly weaker than those required for the bootstrap method to work theoretically. The numerical performance will be investigated in Section 4. To highlight the main idea, a
self-contained proof of Theorem 3.1 is given in the supplementary material
[Chang, Shao and Zhou (2016)]. The proofs of Theorems 3.2 and 3.3 are
based on straightforward extensions of Theorems 2.2 and 3.1 in Liu and
Shao (2014), and thus are omitted.
3.1.3. FDR control under dependence. In this section, we generalize the
results in previous sections to the dependence case. Write ̺ = n1 /n2 . For
2 + ̺σ 2 and define
every k, ℓ = 1, . . . , m, let σk2 = σ1k
2k
(3.8)

rkℓ = (σk σℓ )−1 {cov(Xk , Xℓ ) + ̺ cov(Yk , Yℓ )},

which characterizes the dependence between (Xk , Yk ) and (Xℓ , Yℓ ). Partic2 = σ 2 , we see that r = 1 {corr(X , X ) +
ularly, when n1 = n2 and σ1k
kℓ
k
ℓ
2k
2
corr(Yk , Yℓ )}. In this subsection, we impose the following conditions on the
dependence structure of X = (X1 , . . . , Xm )T and Y = (Y1 , . . . , Ym )T .

STUDENTIZED TWO-SAMPLE U -STATISTICS

17

(D1) There exist constants 0 < r < 1, 0 < ρ < (1 − r)/(1 + r) and b1 > 0
such that
max

1≤k6=ℓ≤m

|rkℓ | ≤ r

and

max sk (m) ≤ b1 mρ ,

1≤k≤m

where for k = 1, . . . , m,
sk (m) = {1 ≤ ℓ ≤ m : corr(Xk , Xℓ ) ≥ (log m)−2−γ
or corr(Yk , Yℓ ) ≥ (log m)−2−γ }
for some γ > 0.
(D2) There exist constants 0 < r < 1, 0 < ρ < (1 − r)/(1 + r) and b1 > 0
such that max1≤k6=ℓ≤m |rkℓ | ≤ r and for each Xk , the number of variables Xℓ
that are dependent of Xk is less than b1 mρ .
The assumption max1≤k6=ℓ≤m |rkℓ | ≤ r for some 0 < r < 1 imposes a constraint on the magnitudes of the correlations, which is natural in the sense
that the correlation matrix R = (rkℓ )1≤k,ℓ≤m is singular if max1≤k6=ℓ≤m |rkℓ | =
1. Under condition (D1), each (Xk , Yk ) is allowed to be “moderately” correlated with at most as many as O(mρ ) other vectors. Condition (D2) enforces
a local dependence structure on the data, saying that each vector is dependent with at most as many as O(mρ ) other random vectors and independent
of the remaining ones. The following theorem extends the results in previous sections to the dependence case. Its proof is placed in the supplementary
material [Chang, Shao and Zhou (2016)].
Theorem 3.4. Assume that either condition (D1) holds with log m =
O(n1/8 ) or condition (D2) holds with log m = o(n1/3 ).
(i) Suppose that (3.3) and (3.4) are satisfied. Then as n → ∞, FDPΦ →P
απ0 and FDRΦ → απ0 .
(ii) Suppose that (3.3), (3.6) and (3.7) are satisfied. Then as n → ∞,
FDPRB →P απ0 and FDRRB → απ0 .
In particular, assume that condition (D2) holds with log m = o(n1/2 ) and
m1 ≤ mc for some 0 < c < 1. Then as n → ∞, FDPRB →P απ0 and FDRRB →
απ0 .
3.2. Studentized Mann–Whitney test. Let X = {X1 , . . . , Xn1 } and Y =
{Y1 , . . . , Yn2 } be two independent random samples from distributions F and
G, respectively. Let θ = P(X ≤ Y ) − 1/2. Consider the null hypothesis H0 :
θ = 0 against the one-sided alternative H1 : θ > 0. This problem arises in
many applications including testing whether the physiological performance
of an active drug is better than that under the control treatment, and testing
the effects of a policy, such as unemployment insurance or a vocational
training program, on the level of unemployment.

18

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

The Mann–Whitney (M–W) test [Mann and Whitney (1947)], also known
as the two-sample Wilcoxon test [Wilcoxon (1945)], is prevalently used for
testing equality of means or medians, and serves as a nonparametric alternative to the two-sample t-test. The corresponding test statistic is given
by
n

(3.9)

n

2
1 X
1 X
Un̄ =
I{Xi ≤ Yj },
n1 n2

n̄ = (n1 , n2 ).

i=1 j=1

The M–W test is widely used in a wide range of fields including statistics, economics and biomedicine, due to its good efficiency and robustness
against parametric assumptions. Over one-third of the articles published
in Experimental Economics use the Mann–Whitney test and Okeh (2009)
reported that thirty percent of the articles in five biomedical journals published in 2004 used the Mann–Whitney test. For example, using the M–W U
test, Charness and Gneezy (2009) developed an experiment to test the conjecture that financial incentives help to foster good habits. They recorded
seven biometric measures (weight, body fat percentage, waist size, etc.) of
each participant before and after the experiment to assess the improvements
across treatments. Although the M–W test was originally introduced as a
rank statistic to test if the distributions of two related samples are identical, it has been prevalently used for testing equality of medians or means,
sometimes as an alternative to the two-sample t-test.
It was argued and formally examined recently in Chung and Romano
(2016) that the M–W test has generally been misused across disciplines. In
fact, the M–W test is only valid if the underlying distributions of the two
groups are identical. Nevertheless, when the purpose is to test the equality of
distributions, it is recommended to use a statistic, such as the Kolmogorov–
Smirnov or the Cramér–von Mises statistic, that captures the discrepancies
of the entire distributions rather than an individual parameter. More specifically, because the M–W test only recognizes deviation from θ = 0, it does
not have much power in detecting overall distributional discrepancies. Alternatively, the M–W test is frequently used to test the equality of medians.
However, Chung and Romano (2013) presented evidence that this is another
improper application of the M–W test and suggested to use the Studentized
median test.
Even when the M–W test is appropriately applied for testing H0 : θ = 0,
the asymptotic variance depends on the underlying distributions, unless
the two population distributions are identical. As Hall and Wilson (1991)
pointed out, the application of resampling to pivotal statistics has better
asymptotic properties in the sense that the rate of convergence of the actual
significance level to the nominal significance level is more rapid when the

19

STUDENTIZED TWO-SAMPLE U -STATISTICS

pivotal statistics are resampled. Therefore, it is natural to use the Studentized Mann–Whitney test, which is asymptotic pivotal.
Let
bn̄ = σ
U
bn̄−1 (Un̄ − 1/2)

(3.10)

bn̄2 = σ
b12 n−1
denote the Studentized test statistic for Un̄ as in (3.9), where σ
1 +
−1
2
σ
b2 n2 ,
!2
!2
n1
n2
n1
n2
1 X
1 X
1 X
1 X
2
2
qi −
pj −
qi ,
pj
σ
b2 =
σ
b1 =
n1 − 1
n1
n2 − 1
n2
i=1

j=1

i=1

j=1

−1 Pn1

−1 Pn2

with qi = n2
i=1 I{Xi ≤ Yj }.
j=1 I{Yj < Xi } and pj = n1
When dealing with samples from a large number of geographical regions (suburbs, states, health service areas, etc.), one may need to make
many statistical inferences simultaneously. Suppose we observe a family
of paired groups, that is, for k = 1, . . . , m, Xk = {X1,k , . . . , Xn1 ,k }, Yk =
{Y1,k , . . . , Yn2 ,k }, where the index k denotes the kth site. Assume that Xk is
drawn from Fk , and independently, Yk is drawn from Gk .
For each k = 1, . . . , m, we test the null hypothesis H0k : θk = P(X1,k ≤
Y1,k )−1/2 = 0 against the one-sided alternative H1k : θk > 0. If H0k is rejected,
we conclude that the treatment effect (of a drug or a policy) is acting within
the kth area. Define the test statistic
−1
bn̄,k = σ
U
bn̄,k
(Un̄,k − 1/2),

bn̄,k is constructed from the kth paired samples according to (3.10).
where U
Let
bn̄,k ≤ t|H k )
Fn̄,k (t) = P(U
0

and Φ(t) = P(Z ≤ t),

where Z is the standard normal random variable. Then the true p-values
bn̄,k ), and pbk = 1 − Φ(U
bn̄,k ) denote the estimated p-values
are pk = 1 − Fn̄,k (U
based on normal calibration.
To identify areas where the treatment effect is acting, we can use the
B–H method to control the FDR at α level by rejecting the null hypotheses
indexed by S = {1 ≤ k ≤ m : pbk ≤ pb(k̂) }, where k̂ = max{1 ≤ k ≤ m : pb(k) ≤
αk/m}, and {b
p(k) } denote the ordered values of {b
pk }. As before, let FDRΦ
be the FDR of the B–H method based on normal calibration.
Alternative to normal calibration, we can also consider bootstrap calibra†
†
†
†
tion. Recall that Xk,b
= {X1,k,b
, . . . , Xn† 1 ,k,b } and Yk,b
= {Y1,k,b
, . . . , Yn†2 ,k,b },
b = 1, . . . , B, are two bootstrap samples drawn independently and uniformly,
with replacement, from Xk = {X1,k , . . . , Xn1 ,k } and Yk = {Y1,k , . . . , Yn2 ,k }, reb†
spectively. For each k = 1, . . . , m, let U
n̄,k,b be the bootstrapped test statistic

20

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

†
†
constructed from Xk,b
and Yk,b
, that is,
#
"
n2
n1 X
X
1
†
−1
b
I{Xi,k ≤ Yj,k } ,
U
bn̄,k,b Un̄,k,b −
n̄,k,b = σ
n1 n2
i=1 j=1

where Un̄,k,b and σ
bn̄,k,b are the analogues of Un̄ given in (3.9) and σ
bn̄ specified
†
†
below (3.10) via replacing Xi and Yj by Xi,k,b
and Yj,k,b
, respectively. Using
the empirical distribution function
m

B

XX
b † | ≤ t},
b† (t) = 1
I{|U
G
n̄,k,b
m,B
mB
k=1 b=1

b† (U
b † ). For a predewe estimate the unknown p-values by pbk,B = 1 − G
m,B
n̄,k,b
termined α ∈ (0, 1), the null hypotheses indexed by SB = {1 ≤ k ≤ m : pbk,B ≤
pb(k̂B ),B } are rejected, where k̂B = max{0 ≤ k ≤ m : pbk,B ≤ αk/m}. Denote by
FDRB the FDR of the B–H method based on bootstrap calibration.
Applying the general moderate deviation result (2.9) to Studentized Mann–
bn̄,k leads to the following result. The proof is based on
Whitney statistics U
a straightforward adaptation of the arguments we used in the proof of Theorem 3.1, and hence is omitted.
Theorem 3.5. Assume that {X1 , . . . , Xm , Y1 , . . . , Ym } are independent
random variables with continuous distribution functions Xk ∼ Fk and Yj ∼
Gk . The triplet (n1 , n2 , m) is such that n1 ≍ n2 , m = m(n1 , n2 ) → ∞, log m =
o(n1/3 ) and m−1 #{k = 1, . . . , m : θk = 1/2} → π0 ∈ (0, 1] as n = n1 ∧ n2 →
1
2
∞. For independent samples {Xi,1 , . . . , Xi,m }ni=1
and {Yj,1 , . . . , Yj,m}nj=1
, suppose that min1≤k≤m min(σ1k , σ2k ) ≥ c > 0 for some constant c > 0 and as
n → ∞,
#{1 ≤ k ≤ m : |θk − 1/2| ≥ 4(log m)1/2 σn̄,k } → ∞,
2 = var{G (X )}, σ 2 = var{F (Y )} and σ 2 = σ 2 n−1 + σ 2 n−1 .
where σ1k
k k
k
k
2k 2
1k 1
n̄,k
2k
Then as n → ∞, FDPΦ , FDPB →P απ0 and FDRΦ , FDRB → απ0 .

Attractive properties of the bootstrap for multiple-hypothesis testing were
first noted by Hall (1990) in the case of the mean rather than its Studentized
counterpart. Now it has been rigorously proved that bootstrap methods
are particularly effective in relieving skewness in the extreme tails which
leads to second-order accuracy [Fan, Hall and Yao (2007), Delaigle, Hall
and Jin (2011)]. It is interesting and challenging to investigate whether these
advantages of the bootstrap can be inherited by multiple U -testing in either
the standardized or the Studentized case.

21

STUDENTIZED TWO-SAMPLE U -STATISTICS

4. Numerical study. In this section, we present numerical investigations
for various calibration methods described in Section 3 when they are applied to two-sample large-scale multiple testing problems. We refer to the
simulation for two-sample t-test and Studentized Mann–Whitney test as
Sim1 and Sim2 , respectively. Assume that we observe two groups of m1
2
dimensional gene expression data {Xi }ni=1
and {Yj }nj=1
, where X1 , . . . , Xn1
and Y1 , . . . , Yn2 are independent random samples drawn from the distributions of X and Y, respectively.
For Sim1 , let X and Y be such that
(4.1)

X = µ1 + {ε1 − E(ε1 )} and

Y = µ2 + {ε2 − E(ε2 )},

where ε1 = (ε1,1 , . . . , ε1,m )T and ε2 = (ε2,1 , . . . , ε2,m )T are two sets of i.i.d.
random variables. The i.i.d. components of noise vectors ε1 and ε2 follow two
types of distributions: (i) the exponential distribution Exp(λ) with density
function λ−1 e−x/λ ; (ii) Student t-distribution t(k) with k degrees of freedom.
The exponential distribution has nonzero skewness, while the t-distribution
is symmetric and heavy-tailed. For each type of error distribution, both cases
of homogeneity and heteroscedasticity were considered. Detailed settings for
the error distributions are specified in Table 1.
For Sim2 , we assume that X and Y satisfy
(4.2)

X = µ1 + ε1

and

Y = µ2 + ε2 ,

where ε1 = (ε1,1 , . . . , ε1,m )T and ε2 = (ε2,1 , . . . , ε2,m )T are two sets of i.i.d.
random variables. We consider several distributions for the error terms ε1,k
and ε2,k : standard normal distribution N (0, 1), t-distribution t(k), uniform
distribution U (a, b) and Beta distribution Beta(a, b). Table 2 reports four
settings of (ε1,k , ε2,k ) used in our simulation. In either setting, we know
P(ε1,k ≤ ε2,k ) = 1/2 holds. Hence, the power against the null hypothesis
H0k : P(Xk ≤ Yk ) = 1/2 will generate from the magnitude of the difference
between the kth components of µ1 and µ2 .
In both Sim1 and Sim2 , we set µ1 = 0, and assume that the first m1 =
1/2 and
2 −1
⌊1.6m1/2 ⌋ components of µ2 are equal to c{(σ12 n−1
1 + σ2 n2 ) log m}
2
2
the rest are zero. Here, σ1 and σ2 denote the variance of ε1,k and ε2,k , and
Table 1
Distribution settings in Sim1

Exponential distributions
Student t-distributions

Homogeneous case

Heteroscedastic case

ε1,k ∼ Exp(2)
ε2,k ∼ Exp(2)

ε1,k ∼ Exp(2)
ε2,k ∼ Exp(1)

ε1,k ∼ t(4)
ε2,k ∼ t(4)

ε1,k ∼ t(4)
ε2,k ∼ t(3)

22

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU
Table 2
Distribution settings in Sim2
Identical distributions

Nonidentical distributions

Case 1

ε1,k ∼ N (0, 1)
ε2,k ∼ N (0, 1)

ε1,k ∼ N (0, 1)
ε2,k ∼ t(3)

Case 2

ε1,k ∼ U (0, 1)
ε2,k ∼ U (0, 1)

ε1,k ∼ U (0, 1)
ε2,k ∼ Beta(10, 10)

c is a parameter employed to characterize the location discrepancy between
the distributions of X and Y. The sample size (n1 , n2 ) was set to be (50, 30)
and (100, 60), and the discrepancy parameter c took values in {1, 1.5}. The
significance level α in the B–H procedure was specified as 0.05, 0.1, 0.2 and
0.3, and the dimension m was set to be 1000 and 2000. In Sim1 , we compared
three different methods to calculate the p-values in the B–H procedure: normal calibration given in Section 3.1.1, bootstrap calibration and regularized
bootstrap calibration proposed in Section 3.1.2. For regularized bootstrap
calibration, we used a cross-validation approach as in Section 3 of Liu and
Shao (2014) to choose regularized parameters λ1k and λ2k . In Sim2 , we
compared the performance of normal calibration and bootstrap calibration
proposed in Section 3.2. For each compared method, we evaluated its performance via two indices: the empirical FDR and the proportion among the
true alternative hypotheses was rejected. We call the latter correct rejection
proportion. If the empirical FDR is low, the proposed procedure has good
FDR control; if the correct rejection proportion is high, the proposed procedure has fairly good performance in identifying the true signals. For ease
of exposition, we only report the simulation results for (n1 , n2 ) = (50, 30)
and m = 1000 in Figures 1 and 2. The results for (n1 , n2 ) = (100, 60) and
m = 2000 are similar, which can be found in the supplementary material
[Chang, Shao and Zhou (2016)]. Each curve corresponds to the performance
of a certain method and the line types are specified in the caption below.
The horizontal ordinates of the four points on each curve depict the empirical FDR of the specified method when the pre-specified level α in the B–H
procedure was taken to be 0.05, 0.1, 0.2 and 0.3, respectively, and the vertical
ordinates indicate the corresponding empirical correct rejection proportion.
We say that a method has good FDR control if the horizontal ordinates
of the four points on its performance curve are less than the prescribed α
levels.
In general, as shown in Figures 1 and 2, the B–H procedure based on
(regularized) bootstrap calibration has better FDR control than that based
on normal calibration. In Sim1 where the errors are symmetric (e.g., ε1,k and
ε2,k follow the Student t-distributions), the panels in the first row of Figure 1 show that the B–H procedures using all the three calibration methods

STUDENTIZED TWO-SAMPLE U -STATISTICS

23

Fig. 1. Performance comparison of B–H procedures based on three calibration methods in Sim1 with (n1 , n2 ) = (50, 30) and m = 1000. The first and second rows show the
results when the components of noise vectors ε1 and ε2 follow t-distributions and exponential distributions, respectively; left and right panels show the results for homogeneous
and heteroscedastic cases, respectively; horizontal and vertical axes depict empirical false
discovery rate and empirical correct rejection proportion, respectively; and the prescribed
levels α = 0.05, 0.1, 0.2 and 0.3 are indicated by unbroken horizontal black lines. In each
panel, dashed lines and unbroken lines represent the results for the discrepancy parameter
c = 1 and 1.5, respectively, and different colors express different methods employed to calculate p-values in the B–H procedure, where blue line, green line and red line correspond
to the procedures based on normal, conventional and regularized bootstrap calibrations,
respectively.

are able to control or approximately control the FDR at given levels, while
the procedures based on bootstrap and regularized bootstrap calibrations
outperform that based on normal calibration in controlling the FDR. When
the errors are asymmetric in Sim1 , the performances of the three B–H procedures are different from those in the symmetric cases. From the second row
of Figure 1, we see that the B–H procedure based on normal calibration is
distorted in controlling the FDR while the procedure based on (regularized)
bootstrap calibration is still able to control the FDR at given levels. This

24

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

Fig. 2. Performance comparison of B–H procedures based on two different calibration
methods in Sim2 with (n1 , n2 ) = (50, 30) and m = 1000. The first and second rows show
the results when the components of noise vectors ε1 and ε2 follow the distributions specified in cases 1 and 2 of Table 2, respectively; left and right panels show the results for
the cases of identical distributions and nonidentical distributions, respectively; horizontal
and vertical axes depict empirical false discovery rate and empirical correct rejection proportion, respectively; and the prescribed levels α = 0.05, 0.1, 0.2 and 0.3 are indicated by
unbroken horizontal black lines. In each panel, dashed lines and unbroken lines represent
the results for the discrepancy parameter c = 1 and 1.5, respectively, and different colors
express different methods employed to calculate p-values in the B–H procedure, where blue
line and red line correspond to the procedures based on normal and bootstrap calibrations,
respectively.

phenomenon is further evidenced by Figure 2 for Sim2 . Comparing the B–H
procedures based on conventional and regularized bootstrap calibrations, we
find that the former approach is uniformly more conservative than the latter
in controlling the FDR. In other words, the B–H procedure based on regularized bootstrap can identify more true alternative hypotheses than that
using conventional bootstrap calibration. This phenomenon is also revealed
in the heteroscedastic case. As the discrepancy parameter c gets larger so
that the signal is stronger, the correct rejection proportion of the B–H pro-

STUDENTIZED TWO-SAMPLE U -STATISTICS

25

cedures based on all the three calibrations increase and the empirical FDR
is closer to the prescribed level.
5. Discussion. In this paper, we established Cramér-type moderate deviations for two-sample Studentized U -statistics of arbitrary order in a general framework where the kernel is not necessarily bounded. Two-sample
U -statistics, typified by the two-sample Mann–Whitney test statistic, have
been widely used in a broad range of scientific research. Many of these applications rely on a misunderstanding of what is being tested and the implicit
underlying assumptions, that were not explicitly considered until relatively
recently by Chung and Romano (2016). More importantly, they provided
evidence for the advantage of using the Studentized statistics both theoretically and empirically.
Unlike the conventional (one- and two-sample) U -statistics, the asymptotic behavior of their Studentized counterparts has barely been studied
in the literature, particularly in the two-sample case. Recently, Shao and
Zhou (2016) proved a Cramér-type moderate deviation theorem for general
Studentized nonlinear statistics, which leads to a sharp moderate deviation
result for Studentized one-sample U -statistics. However, extension from onesample to two-sample in the Studentized case is totally nonstraightforward,
and requires a more delicate analysis on the Studentizing quantities. Further,
for the two-sample t-statistic, we proved moderate deviation with secondorder accuracy under a finite 4th moment condition (see Theorem 2.4),
which is of independent interest. In contrast to the one-sample case, the
two-sample t-statistic cannot be reduced to a self-normalized sum of independent random variables, and thus the existing results on self-normalized
ratios [Jing, Shao and Wang (2003), Wang (2005, 2011)] cannot be directly
applied. Instead, we modify Theorem 2.1 in Shao and Zhou (2016) to obtain
a more precise expansion that can be used to derive a refined result for the
two-sample t-statistic.
Finally, we show that the obtained moderate deviation theorems provide
theoretical guarantees for the validity, including robustness and accuracy, of
normal, conventional bootstrap and regularized bootstrap calibration methods in multiple testing with FDR/FDP control. The dependence case is also
covered. These results represent a useful complement to those obtained by
Fan, Hall and Yao (2007), Delaigle, Hall and Jin (2011) and Liu and Shao
(2014) in the one-sample case.
Acknowledgements. The authors would like to thank Peter Hall and Aurore Delaigle for helpful discussions and encouragement. The authors sincerely thank the Editor, Associate Editor and three referees for their very
constructive suggestions and comments that led to substantial improvement
of the paper.

26

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

SUPPLEMENTARY MATERIAL
Supplement to “Cramér-type moderate deviations for Studentized twosample U -statistics with applications” (DOI: 10.1214/15-AOS1375SUPP;
.pdf). This supplemental material contains proofs for all the theoretical results in the main text, including Theorems 2.2, 2.4, 3.1 and 3.4, and additional numerical results.
REFERENCES
Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical
and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 57
289–300. MR1325392
Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in
multiple testing under dependency. Ann. Statist. 29 1165–1188. MR1869245
Borovskich, Y. V. (1983). Asymptotics of U -statistics and Von Mises’ functionals. Soviet
Math. Dokl. 27 303–308.
Cao, H. and Kosorok, M. R. (2011). Simultaneous critical values for t-tests in very high
dimensions. Bernoulli 17 347–394. MR2797995
Chang, J., Shao, Q. and Zhou, W.-X. (2016). Supplement to “Cramér-type
moderate deviations for Studentized two-sample U -statistics with applications.”
DOI:10.1214/15-AOS1375SUPP.
Chang, J., Tang, C. Y. and Wu, Y. (2013). Marginal empirical likelihood and sure
independence feature screening. Ann. Statist. 41 2123–2148. MR3127860
Chang, J., Tang, C. Y. and Wu, Y. (2016). Local independence feature screening
for nonparametric and semiparametric models by marginal empirical likelihood. Ann.
Statist. 44 515–539. MR3476608
Charness, G. and Gneezy, U. (2009). Incentives to exercise. Econometrica 77 909–931.
Chen, S. X. and Qin, Y.-L. (2010). A two-sample test for high-dimensional data with
applications to gene-set testing. Ann. Statist. 38 808–835. MR2604697
Chen, L. H. Y. and Shao, Q.-M. (2007). Normal approximation for nonlinear statistics
using a concentration inequality approach. Bernoulli 13 581–599. MR2331265
Chen, S. X., Zhang, L.-X. and Zhong, P.-S. (2010). Tests for high-dimensional covariance matrices. J. Amer. Statist. Assoc. 105 810–819. MR2724863
Chung, E. and Romano, J. P. (2013). Exact and asymptotically robust permutation
tests. Ann. Statist. 41 484–507. MR3099111
Chung, E. and Romano, J. (2016). Asymptotically valid and exact permutation tests
based on two-sample U -statistics. J. Statist. Plann. Inference. 168 97–105. MR3412224
Delaigle, A., Hall, P. and Jin, J. (2011). Robustness and accuracy of methods for high
dimensional data analysis based on Student’s t-statistic. J. R. Stat. Soc. Ser. B. Stat.
Methodol. 73 283–301. MR2815777
Dudoit, S. and van der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics. Springer, New York. MR2373771
Fan, J., Hall, P. and Yao, Q. (2007). To how many simultaneous hypothesis tests can
normal, Student’s t or bootstrap calibration be applied? J. Amer. Statist. Assoc. 102
1282–1288. MR2372536
Fan, J., Han, X. and Gu, W. (2012). Estimating false discovery proportion under arbitrary covariance dependence. J. Amer. Statist. Assoc. 107 1019–1035. MR3010887
Ferreira, J. A. and Zwinderman, A. H. (2006). On the Benjamini–Hochberg method.
Ann. Statist. 34 1827–1849. MR2283719

STUDENTIZED TWO-SAMPLE U -STATISTICS

27

Friguet, C., Kloareg, M. and Causeur, D. (2009). A factor model approach to multiple testing under dependence. J. Amer. Statist. Assoc. 104 1406–1415. MR2750571
Hall, P. (1990). On the relative performance of bootstrap and Edgeworth approximations
of a distribution function. J. Multivariate Anal. 35 108–129. MR1084945
Hall, P. and Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing.
Biometrics 47 757–762. MR1132543
Helmers, R. and Janssen, P. (1982). On the Berry–Esseen theorem for multivariate
U -statistics. In Math. Cent. Rep. SW 90 1–22. Mathematisch Centrum, Amsterdam.
Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. Ann.
Math. Statistics 19 293–325. MR0026294
Jing, B.-Y., Shao, Q.-M. and Wang, Q. (2003). Self-normalized Cramér-type large
deviations for independent random variables. Ann. Probab. 31 2167–2215. MR2016616
Kochar, S. C. (1979). Distribution-free comparison of two probability distributions with
reference to their hazard rates. Biometrika 66 437–441. MR0556731
Koroljuk, V. S. and Borovskich, Y. V. (1994). Theory of U -Statistics. Mathematics
and Its Applications 273. Kluwer Academic, Dordrecht. MR1472486
Kosorok, M. R. and Ma, S. (2007). Marginal asymptotics for the “large p, small
n” paradigm: With applications to microarray data. Ann. Statist. 35 1456–1486.
MR2351093
Kowalski, J. and Tu, X. M. (2007). Modern Applied U -Statistics. Wiley, Hoboken, NJ.
MR2368050
Lai, T. L., Shao, Q.-M. and Wang, Q. (2011). Cramér type moderate deviations for
Studentized U -statistics. ESAIM Probab. Stat. 15 168–179. MR2870510
Leek, J. T. and Storey, J. D. (2008). A general framework for multiple testing dependence. Proc. Natl. Acad. Sci. USA 105 18718–18723.
Li, R., Zhong, W. and Zhu, L. (2012). Feature screening via distance correlation learning.
J. Amer. Statist. Assoc. 107 1129–1139. MR3010900
Li, G., Peng, H., Zhang, J. and Zhu, L. (2012). Robust rank correlation based screening.
Ann. Statist. 40 1846–1877. MR3015046
Liu, W. and Shao, Q.-M. (2010). Cramér-type moderate deviation for the maximum of
the periodogram with application to simultaneous tests in gene expression time series.
Ann. Statist. 38 1913–1935. MR2662363
Liu, W. and Shao, Q.-M. (2014). Phase transition and regularized bootstrap in largescale t-tests with false discovery rate control. Ann. Statist. 42 2003–2025. MR3262475
Mann, H. B. and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Statistics 18 50–60. MR0022058
Nikitin, Y. and Ponikarov, E. (2006). On large deviations of nondegenerate two-sample
U - and V -statistics with applications to Bahadur efficiency. Math. Methods Statist. 15
103–122. MR2225432
Okeh, U. M. (2009). Statistical analysis of the application of Wilcoxon and Mann–
Whitney U test in medical research studies. Biotechnol. Molec. Biol. Rev. 4 128–131.
Shao, Q.-M. and Zhou, W.-X. (2016). Cramér type moderate deviation theorems for
self-normalized processes. Bernoulli 22 2029–2079.
Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative
point estimation and simultaneous conservative consistency of false discovery rates: A
unified approach. J. R. Stat. Soc. Ser. B. Stat. Methodol. 66 187–205. MR2035766
Vandemaele, M. and Veraverbeke, N. (1985). Cramér type large deviations for Studentized U -statistics. Metrika 32 165–179. MR0824452
Wang, Q. (2005). Limit theorems for self-normalized large deviation. Electron. J. Probab.
10 1260–1285 (electronic). MR2176384

28

J. CHANG, Q.-M. SHAO AND W.-X. ZHOU

Wang, Q. (2011). Refined self-normalized large deviations for independent random variables. J. Theoret. Probab. 24 307–329. MR2795041
Wang, Q. and Hall, P. (2009). Relative errors in central limit theorems for Student’s t
statistic, with applications. Statist. Sinica 19 343–354. MR2487894
Wang, Q., Jing, B.-Y. and Zhao, L. (2000). The Berry–Esseen bound for Studentized
statistics. Ann. Probab. 28 511–535. MR1756015
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics 1 80–83.
Zhong, P.-S. and Chen, S. X. (2011). Tests for high-dimensional regression coefficients
with factorial designs. J. Amer. Statist. Assoc. 106 260–274. MR2816719
J. Chang
School of Statistics
Southwestern University of Finance
and Economics
Chengdu, Sichuan 611130
China
and
School of Mathematics and Statistics
University of Melbourne
Parkville, Victoria 3010
Australia
E-mail: jingyuan.chang@unimelb.edu.au

Q.-M. Shao
Department of Statistics
Chinese University of Hong Kong
Shatin, NT
Hong Kong
E-mail: qmshao@cuhk.edu.hk

W.-X. Zhou
Department of Operations Research
and Financial Engineering
Princeton University
Princeton, New Jersey 08544
USA
and
School of Mathematics and Statistics
University of Melbourne
Parkville, Victoria 3010
Australia
E-mail: wenxinz@princeton.edu

