Chernoff Index for Cox Test of Separate Parametric Families
Xiaoou Li, Jingchen Liu, and Zhiliang Ying
Columbia University

arXiv:1606.08248v1 [] 27 Jun 2016

June 28, 2016

Abstract
The asymptotic efficiency of a generalized likelihood ratio test proposed by Cox is studied
under the large deviations framework for error probabilities developed by Chernoff. In particular,
two separate parametric families of hypotheses are considered [Cox, 1961, 1962]. The significance
level is set such that the maximal type I and type II error probabilities for the generalized
likelihood ratio test decay exponentially fast with the same rate. We derive the analytic form
of such a rate that is also known as the Chernoff index [Chernoff, 1952], a relative efficiency
measure when there is no preference between the null and the alternative hypotheses. We further
extend the analysis to approximate error probabilities when the two families are not completely
separated. Discussions are provided concerning the implications of the present result on model
selection.

1

Introduction

Cox [1961, 1962] introduced the problem of testing two separate parametric families. Let X1 , . . . , Xn
be independent and identically distributed real-valued observations from a population with density
f with respect to some baseline measure µ. Let {gθ , θ ∈ Θ} and {hγ , γ ∈ Γ} denote two separate
parametric families of density functions with respect to the same measure µ. Consider testing
H0 : f ∈ {gθ , θ ∈ Θ} against H1 : f ∈ {hγ , γ ∈ Γ}. To avoid singularity, we assume that all the
distributions in the families gθ and hγ are mutually absolutely continuous so that the likelihood ratio
stays away from zero and infinity. Furthermore, we assume that the model is correctly specified,
that is, f belongs to either the g-family or the h-family.
Recently revisiting this problem, Cox [2013] mentioned several applications such as the one-hit
and two-hit models of binary dose-response and testing of interactions in a balanced 2k factorial
experiment. Furthermore, this problem has been studied in econometrics [Vuong, 1989, White,
1982a,b, Pesaran, 1974, Pesaran and Deaton, 1978, Davidson and MacKinnon, 1981]. For more
applications of testing separate families of hypotheses, see Berrington de González and Cox [2007]
and Braganca Pereira [2005], and the references therein. Furthermore, there is a discussion of model
misspecification, that is, f belongs to neither the g-family nor the h-family, which is beyond the

1

current discussion. For semiparametric models, Fine [2002] proposed a similar test for non-nested
hypotheses under the Cox proportional hazards model assumption.
In the discussion of Cox [1962], the test statistic l = lg (θ̂) − lh (γ̂) − Egθ̂ {lg (θ̂) − lh (γ̂)} is
considered. The functions lg (θ) and lh (γ) are the log-likelihood functions under the g-family and the
h-family and θ̂ and γ̂ are the corresponding maximum likelihood estimators. Rigorous distributional
discussions of statistic l can be found in Huber [1967] and White [1982a,b]. In this paper, we consider
the generalized likelihood ratio statistic
Q
maxγ∈Γ ni=1 hγ (Xi )
Q
LRn =
= elh (γ̂)−lg (θ̂)
maxθ∈Θ ni=1 gθ (Xi )

(1)

that is slightly different from Cox’s approach. We are interested in the Chernoff efficiency, whose
definition is provided in Section 2.1, of the generalized likelihood ratio test.
In the hypothesis testing literature, there are several measures of asymptotic relative efficiency
for simple null hypothesis against simple alternative hypothesis. Let n1 and n2 be the necessary
sample sizes for each of two testing procedures to perform equivalently in the sense that they admit
the same type I and type II error probabilities. Then, the limit of ratio n1 /n2 in the regime that
both sample sizes tend to infinity represents the asymptotic relative efficiency between these two
procedures.
Relative efficiency depends on the asymptotic manner of the two types of error probabilities
with large samples. Under different asymptotic regimes, several asymptotic efficiency measures
are proposed and they are summarized in Chapter 10 of Serfling [1980]. Under the regime of
Pitman efficiency, several asymptotically equivalent tests to Cox test exist. Furthermore, Pesaran
[1984] and Rukhin [1993] applied Bahadur’s criterion of asymptotic comparison [Bahadur, 1960,
1967] to tests for separate families and compared different tests for lognormal against exponential
distribution and for non-nested linear regressions. There are other efficiency measures that are
frequently considered, such as Kallenberg efficiency [Kallenberg, 1983].
In the context of testing a simple null hypothesis against a fixed simple alternative hypothesis,
Chernoff [1952] introduces a measure of asymptotic efficiency for tests based on sum of independent
and identically distributed observations, a special case of which is the likelihood ratio test. This
efficiency is introduced by showing no preference between the null hypothesis and the alternative
hypothesis. The rejection region is setup such that the two types of error probabilities decay at the
same exponential rate ρ. The rate ρ is later known as the Chernoff index. A brief summary of the
Chernoff index is provided in Section 2.1.
The basic strategy of Chernoff [1952] is applying large deviations techniques to the log-likelihood
2

ratio statistic and computes/approximates the probabilities of the two types of errors. Under the
situation when either the null hypothesis or the alternative hypothesis is composite, one naturally
considers the generalized likelihood ratio test. To the authors’ best knowledge, the asymptotic
behavior of the generalized likelihood ratio test under the Chernoff’s regime remains an open
problem. This is mostly because large deviations results are not directly applicable as the test
statistic is the ratio of the supremums of two random functions. This paper fills in this void and
provides a definitive conclusion of the asymptotic efficiency of the generalized likelihood ratio test
under Chernoff’s asymptotic regime. We define the Chernoff index via the asymptotic decay rate
of the maximal type I and type II error probabilities that is also the minimax risk corresponding
to the zero-one loss function.
We compute the generalized Chernoff index of the generalized likelihood ratio test for two
separate parametric families that keep a certain distance away from each other. That is, the
Kullback-Leibler distance between gθ and hγ are bounded away from zero for all θ ∈ Θ and γ ∈ Γ.
We use ρθγ to denote the Chernoff index of the likelihood ratio test for the simple null H0 : f = gθ
against simple alternative H1 :

f = hγ . Under mild moment conditions, we show that the

exponential decay rate of the maximal error probabilities is simply the minimum of the one-to-one
Chernoff index ρθγ over the parameter space, that is, ρ = minθ,γ ρθγ . This result suggests that the
generalized likelihood ratio test is asymptotically the minimax strategy in the sense that with the
same sample size it achieves the optimal exponential decay rate of the maximal type I and type
II error probabilities when they decay equally fast. The present result can also be generalized to
asymptotic analysis of Bayesian model selection among two or more families of distributions. A
key technical component is to deal with the excursion probabilities of the likelihood functions, for
which random field and non-exponential change of measure techniques are applied. This paper
also in part corresponds to the conjecture in Cox [2013] “formal discussion of possible optimality
properties of the test statistics would, I think, require large deviation theory” though we consider
a slightly different statistic.
We further extend the analysis to the cases when the two families may not be completely
separate, that is, one may find two sequences of distributions in each family and the two sequences
converge to each other. For this case, the Chernoff index is zero. We provide asymptotic decay
rate of the type I error probability under a given distribution gθ0 in H0 . To have the problem wellposed, the minimum Kullback-Leibler divergence between gθ0 and all distributions in H1 has to be
bounded away from zero. The result is applicable to both separated and non-separated families and

3

thus it provides a means to approximate the error probabilities of the generalized likelihood ratio test
for general parametric families. This result has important theoretical implications in hypothesis
testing, model selection, and other areas where maximum likelihood is employed. We provide a
discussion concerning variable selection for regression models.
The rest of this paper is organized as follows. We present our main results for separate families of
hypotheses in Section 2. Further extension to more than two families and Bayesian model selection
is discussed in Section 3. Results for possibly non-separate families are presented in Section 4.
Numerical examples are provided in Section 5. Lastly a concluding remark is give in Section 6.

2

Main results

2.1

Simple null against simple alternative – a review of Chernoff index

In this section we state the main results and their implications. To start with, we provide a brief
review of Chernoff index for simple null versus simple alternative; then, we proceed to the case of
simple null versus composite alternative; furthermore, we present the generalized Chernoff index
for the composite null versus composite alternative.
Under the context of simple null hypothesis versus simple alternative hypothesis, we have
the null hypothesis H0 : f = g and the alternative hypothesis H1 : f = h. We write the loglikelihood ratio of each observation as li = log h(Xi ) − log g(Xi ). Then, the likelihood ratio is
P
LRn = exp( ni=1 li ). We use l to denote the generic random variable equal in distribution to
li . We define the moment generating function of l under distribution g as Mg (z) = Eg (ezl ) =
R
{h(x)/g(x)}z g(x)µ(dx), which must be finite for z ∈ [0, 1] by the Hölder inequality. Furthermore,
define the rate function
mg (t) = max[zt − log{Mg (z)}].
z

The following large deviations result is established by Chernoff (1952).
Proposition 1 If −∞ < t < Eg (l), then log Pg (LRn < ent ) ∼ −n × mg (t); if Eg (l) < t < ∞, then
log Pg (LRn > ent ) ∼ −n × mg (t).
We write an ∼ bn if an /bn → 1 as n → ∞. The above proposition provides an asymptotic decay
rate of the type I error probability: for any t > Eg (l)
Pg (LRn > ent ) = e−{1+o(1)}n×mg (t)

as n → ∞.

Similarly, we switch the roles of g and h and define Mh (z) and mh (t) by flipping the sign of the
log-likelihood ratio l = log g(X) − log h(X) and computing the expectations under h. One further
4

defines ρ(t) = min{mg (t), mh (−t)} that is the slower rate among the type I and type II error
probabilities. A measure of efficiency is given by
ρ=

max

ρ(t)

(2)

Eg (l)<t<Eh (l)

that is known as the Chernoff index between g and h.
In the decision framework, we consider the zero-one loss function

if f = g and (X1 , ..., Xn ) ∈ C
 1
1
if f = h and (X1 , ..., Xn ) ∈
/C
L(C, f, X1 , ..., Xn ) =

0
otherwise
where C ⊂ Rn and f is a density function. Then, the risk function is

Pg (C)
if f = g
R(C, f ) = Ef {L(C, f, X1 , ..., Xn )} =
.
c
Ph (C )
if f = h

(3)

(4)

The Chernoff index is the asymptotic exponential decay rate of the minimax risk minC maxf R(C, f )
within the family of tests. In the following section, we will generalize the Chernoff efficiency
following the minimaxity definition.
Using the fact that Mg (z) = Mh (1 − z), one can show that the optimization in (2) is solved at
t = 0 and
ρ = ρ(0).

(5)

Both mg (t) and mh (−t) are monotone functions of t and (5) suggests that ρ = mg (0) = mh (0). To
achieve the Chernoff index, we reject the null hypothesis if the likelihood ratio statistic is greater

0.010
0.000

0.005

-log(Mg(z))

0.015

0.020

than 1 and the type I and type II error probabilities have identical exponential decay rate ρ.

0.0

0.2

0.4

0.6

0.8

1.0

z

Figure 1: Plot of − log{Mg (z)} (y-coordinate) against z (x-coordinate) for the example of lognormal
distribution versus exponential distribution

5

To have a more concrete idea of the above calculations, Figure 1 shows one particular − log{Mg (z)}
as a function of z where g(x) is a lognormal distribution and h(x) is an exponential distribution.
There are several useful facts. First, − log{Mg (z)} is a concave function of z and − log{Mg (0)} =
− log{Mg (1)} = 0. The maximization maxz [zt − log{Mg (z)}] is solved at d log{Mg (z)}/dz = t.
Furthermore, the Chernoff index is achieved at t = 0. We insert t = 0 into the maximization and
the Chernoff index is ρ = maxz [− log{Mg (z)}].

2.2

Generalized Chernoff index for testing composite hypothesis

In this subsection, we develop the corresponding results for testing composite hypotheses. Some
technical conditions are required as follows.
A1 Complete separation: minθ∈Θ,γ∈Γ Egθ {log gθ (X) − log hγ (X)} > 0.
A2 The parameter spaces Θ and Γ are compact subsets of Rdg and Rdh with continuously differentiable boundary ∂Θ and ∂Γ, respectively.
A3 Define lθγ = log hγ (X) − log gθ (X), S1 = supθ,γ |∇θ lθγ |, and S2 = supθ,γ |∇γ lθγ |. There exists
some η, x0 > 0, that are independent with θ and γ, such that for x > x0
sup max{Pgθ (Si > x), Phγ (Si > x)} ≤ e−(log x)

1+η

,

(i = 1, 2).

(6)

θ∈Θ,γ∈Γ

Remark 2 Condition A3 requires certain tail conditions of Si . It excludes some singularity cases.
This condition is satisfied by most parametric families. For instance, if gθ (x) = g0 (x)eθx−ϕg (θ) and
hγ = h0 (x)eγx−ϕh (γ) are exponential families, then
|∇θ lθγ | = |x − ϕ0g (θ)| ≤ |x| + O(1).
Thus (6) is satisfied if |x| has a finite moment generating function.
If gθ = g(x − θ) is the scale family, then
|∇θ lθγ | =

g 0 (x − θ)
g(x − θ)

that usually has finite moment generating function for light-tailed distributions (Gaussian, exponential, etc) and is usually bounded for heavy-tailed distributions (e.g. t-distribution). Similarly, one
may verify (6) for scale families. Thus, A3 is a weak condition and is applicable to most parametric
families practically in use.

6

We start the discussion for a simple null hypothesis against a composite alternative hypothesis
H0 : f = g

and H1 : f ∈ {hγ : γ ∈ Γ}.

In this case, the likelihood ratio takes the following form
Q
maxγ∈Γ ni=1 hγ (Xi )
Qn
LRn =
.
i=1 g(Xi )

(7)

(8)

For each distribution hγ in the alternative family, we define ργ to be the Chernoff index of the
likelihood ratio test for H0 : f = g against H1 : f = hγ , whose form is given as in (2). The first
result is given as follows.
Lemma 3 Consider the hypothesis testing problem given as in (7) and the generalized likelihood
ratio test with rejection region Cλ = {(x1 , ..., xn ) : LRn > λ} where LRn is given by (8). If
conditions A1-3 are satisfied and we choose λ = 1, then the asymptotic decay rate of the type I and
maximal type II error probabilities are identical, more precisely,
log Pg (C1 ) ∼ sup log Phγ (C1c ) ∼ −n × min ργ .
γ

γ∈Γ

For composite null versus composite alternative
H0 : f ∈ {gθ : θ ∈ Θ} against H1 : f ∈ {hγ : γ ∈ Γ}

(9)

similar results can be obtained. The generalized likelihood ratio statistic is given by (1). For each
single pair (gθ , hγ ), we let ρθγ denote the corresponding Chernoff index of the likelihood ratio test
for H0 : f = gθ and H1 : f = hγ . The following theorem states the main result.
Theorem 4 Consider a composite null hypothesis against a composite alternative hypothesis given
as in (9) and the generalized likelihood ratio test with rejection region Cλ = {(x1 , ..., xn ) : LRn > λ}
where LRn is given by (1). If conditions A1-3 are satisfied and we choose λ = 1, then the asymptotic
decay rate of the maximal type I and type II error probabilities are identical, more precisely,
sup log Pgθ (C1 ) ∼ sup log Phγ (C1c ) ∼ −n × min ρθγ .
θ∈Θ

θ∈Θ,γ∈Γ

γ∈Γ

(10)

We call
ρ = min ρθγ
θ,γ

the generalized Chernoff index between the two families {gθ } and {hγ } that is the exponential
decay rate of the maximal type I and type II error probabilities for the generalized likelihood ratio
7

test. We would like to make a few remarks. Suppose that ρθγ is minimized at θ∗ and γ∗ . The
maximal type I and type II error probabilities of C1 have identical exponential decay rate as that
of the error probabilities of the likelihood ratio test for the simple null H0 : f = gθ∗ versus simple
alternative H1 : f = hγ∗ problem. Then, according to the Neyman-Pearson lemma, we have the
following statement. Among all the tests for (9) that admit maximal type I error probabilities that
decays exponentially at least at rate ρ, their maximal type II error probabilities decay at most at
rate ρ. This asymptotic efficiency can only be obtained at the particular threshold λ = 1, at which
the maximal type I and the type II error probabilities decay exponentially equally fast. Consider
the loss function as in (3) and the risk function is

Pf (C)
if f ∈ {gθ : θ ∈ Θ}
R(C, f ) =
.
c
Pf (C )
if f ∈ {hγ : γ ∈ Γ}

(11)

According to the above discussion, the maximum risk of the rejection region C1 = {LRn > 1}
achieves the same asymptotic decay rate as that of the minimax risk that is
minn

C⊂R

log{R(C, f )}
→ −ρ.
n
f ∈{gθ }∪{hγ }
max

Upon considering the exponential decay rate of the two types of error probabilities, one can
simply reduce the problem to testing H0 : f = gθ∗ against H1 : f = hγ∗ . Each of these two
distributions can be viewed as the least favorable distribution if its own family is chosen to be
the null family. The results in Lemma 3 and Theorem 4 along with their proofs suggest that the
maximal type I and type II error probabilities are achieved at f = gθ∗ and f = hγ∗ . In addition,
under the distribution gθ∗ and conditional on the event C1 , in which H0 is rejected, the maximum
likelihood estimator γ̂ converges to γ∗ ; vice versa, under the distribution f = hγ∗ , if H0 is not
rejected, the maximum likelihood estimator θ̂ converges to θ∗ .

2.3

Relaxation of the technical conditions

The results of Lemma 3 and Theorem 4 require three technical conditions. Condition A1 ensures
that the two families are separated and it is crucial for the exponential decay of the error probabilities. Condition A2, though important for the proof, can be relaxed for most parametric families.
They can be replaced by certain localization conditions for the maximum likelihood estimator. We
present one as follows.
A4 There exist parameter-dependent compact sets Aθ , Ãγ ⊂ Γ and Bγ , B̃θ ⊂ Θ such that for all

8

θ and γ
1
log Pgθ (θ̂ ∈ B̃θc or γ̂ ∈ Acθ ) < −ρ,
n→∞ n
1
lim inf log Phγ (θ̂ ∈ Bγc or γ̂ ∈ Ãcγ ) < −ρ
n→∞ n

lim inf

(12)

where θ̂ and γ̂ are the maximum likelihood estimators under the two families. Condition A3
is satisfied if the maximization in the definition of Si is taken on the set Aθ and B̃θ when the
tail is computed under gθ and is taken on the set Ãγ and Bγ when the tail is computed under
hγ .
Remark 5 Assumption A4 can be verified by means of large deviations of the maximum likelihood
estimator; see Arcones [2006]. Under regularity conditions, the probability that the maximum likelihood estimator deviates from the true parameter by a constant decreases exponentially. One can
choose the constant large enough so that it decays at a faster rate than ρ and thus Assumption 4 is
satisfied.
Consider the first probability in (12) under gθ . We typically choose B̃θ to be a reasonably large
compact set containing θ and thus Pgθ (θ̂ ∈ B̃θc ) decays exponentially fast at a higher rate than ρ.
For the choice of Aθ , we first define
γθ = arg max Egθ {log hγ (X)}
γ∈Γ

that is the limit of γ̂ under gθ . Then, we choose Aθ be a sufficiently large compact set containing
γθ so that the decay rate of Pgθ (γ̂ ∈ Acθ ) is higher than ρ. Similarly, we can choose Bγ and
Ãγ . Furthermore, the maximum score function for a single observation over a compact set usually
has a sufficiently light tail to satisfy condition A4, for instance, Pgθ (supθ∈B̃θ ,γ∈Aθ |∇θ lθγ | > x) ≤
1+η

e−(log x)

.

Corollary 6 Consider a composite null hypothesis against composite alternative hypothesis given
as in (9). Suppose that conditions A1 and A4 are satisfied. Then, the asymptotic decay rates of
the maximal type I and type II error probabilities are identical, more precisely,
sup log Pgθ (C1 ) ∼ sup log Phγ (C1c ) ∼ −n × min ρθγ .
θ∈Θ

θ,γ

γ∈Γ

9

3

Extensions

3.1

On the asymptotic behavior of Bayes factor

The result in Theorem 4 can be further extended to the study of Bayesian model selection. Consider
the two families in (9) each of which is endowed with a prior distribution on its own parameter
space, denoted by φ(θ) and ϕ(γ). We use M to denote the family membership: M = 0 for the
g-family and M = 1 for the h-family. Then, the Bayes factor is
R
Qn
p(X1 , ..., Xn |M = 1)
i=1 hγ (Xi )dγ
γ∈Γ ϕ(γ)
Qn
BF =
.
= R
p(X1 , ..., Xn |M = 0)
i=1 gθ (Xi )dθ
θ∈Θ φ(θ)

(13)

With a similar derivation as that of Bayesian information criterion [Schwarz, 1978], the marginalized
likelihood p(X1 , ..., Xn |M = i) is the maximized likelihood multiplied by a polynomial prefactor
depending on the dimension of the parameter space. Therefore, we can approximate the Bayesian
factor by the generalized likelihood ratio statistic as follows
κ−1 n−β ≤

BF
≤ κnβ
LRn

(14)

for some κ and β sufficiently large. Therefore, log BF = log LRn + O(log n). Since the expectation
of log LRn is of order n, the O(log n) term does not affect the exponential rate. Therefore, we have
the following result.
Theorem 7 Consider two families of distributions given as in (9) satisfying conditions A1-3. The
prior densities ϕ and φ are positive and Lipschitz continuous. We select M = 1 if BF > 1 and
M = 0 otherwise where BF is given by (13). Then, the asymptotic decay rate of selecting the
wrong model are identical under each of the two families. More precisely,
Z
log
Pgθ (BF > 1)φ(θ)dθ ∼ sup log Pgθ (BF > 1)
θ∈Θ
θ∈Θ
Z
∼ log
Phγ (BF ≤ 1)ϕ(γ)dγ ∼ sup log Phγ (BF ≤ 1) ∼ −n × min ρθγ .
γ∈Γ

γ∈Γ

θ,γ

The proof of the above theorem is an application of Theorem 4 and (14) and thus we omit it.
The above result does not rely on the validity of the prior distributions. Therefore, model selection
based on Bayes factor is asymptotically efficient even if the prior distribution is misspecified. That
is, the Bayes factor is calculated based on the probability measures with density functions ϕ and φ
that are different from the true prior probability measures under which θ and γ are generated.

10

3.2

Extensions to more than two families

Suppose that there are K non-overlapping families {gk,θk : θk ∈ Θk } for k = 1, ..., K, among which
we would like to select the true family to which the distribution f belongs. Let
Lk (θk ) =

n
Y

gk,θk (Xi )

i=1

be the likelihood of family k. A natural decision is to select the family that has the highest
likelihood, that is,
k̂ = arg max sup Lk (θk ).
k=1,...,K θk

According to the results in Theorem 4, we obtain that
sup log Pgk,θk (k̂ 6= k) ∼ −nρ
k,θk

where ρ is the smallest generalized Chernoff indices, defined as in Theorem 4, among all the (K −
1)K/2 pairs of families. To obtain the above limit, one simply considers each family k as the null
hypothesis and the union of the rest K − 1 altogether as the alternative hypothesis.
With the same argument as in Section 3.1, we consider Bayesian model selection among the
K families each of which is endowed with a prior φk (θk ). Consider the marginalized maximum
likelihood estimator
Z
k̂B = arg max

Lk (θk )φk (θk )dθk

k

that admits the same misclassification rate
Z
sup log Pgk,θk (k̂B 6= k) ∼ sup log
k,θk

4
4.1

Pgk,θk (k̂B 6= k)φk (θk )dθk ∼ −nρ.

k

Results for possibly non-separated families
The asymptotic approximation of error probabilities

In this section we extend the results to the cases when the g-family and the h-family are not
necessarily separated, that is,
min Egθ {log gθ (X) − log hγ (X)} = 0.

θ∈Θ,γ∈Γ

(15)

In the case of (15), the Chernoff index is trivially zero. We instead derive the asymptotic decay
rate of the following error probabilities. For some θ0 ∈ Θ such that
min Egθ0 {log gθ0 (X) − log hγ (X)} > 0,
γ

11

we consider the type I error probability
Pgθ0 (LRn > enb )

as n → ∞

(16)

where LRn is the generalized likelihood ratio statistic as in (1). For b, we require that
sup Egθ0 {log hγ (X) − log gθ0 (X)} < b

(17)

γ∈Γ

ensuring that Pgθ0 (LRn > enb ) eventually converges to zero.
The statement of the theorem requires the following construction. For each θ and γ, we first
define the moment generating function of log hγ (X) − log gθ (X) − b
h
i
Mgθ0 (θ, γ, λ) = Egθ0 exp{λ(log hγ (X) − log gθ (X) − b)}

(18)

and consider the optimization problem
Mg†θ , inf sup inf Mgθ0 (θ, γ, λ).
0

θ∈Θ γ∈Γ λ∈R

(19)

Under Assumption A2, there exists at least one solution to the above optimization we assume one
of the solutions is
(θ† , γ † , λ† ) = arg inf sup inf Mgθ0 (θ, γ, λ).
θ∈Θ γ∈Γ λ∈R

Furthermore, we define a measure Q† that is absolutely continuous with respect to Pgθ0
n
o
dQ†
= exp λ† (log hγ † (X) − log gθ† (X) − b) /Mg†θ .
0
dPgθ0

(20)

Definition 8 (Solid tangent cone) For a set A ⊂ Rd and x ∈ A, the solid tangent cone Tx A is
defined as
Tx A = {y ∈ Rd : ∃ ym and λm such that ym → y, λm → 0 as m → ∞, and x + λm ym ∈ A}.
If A has continuously differentiable boundary and x ∈ ∂A, then Tx A consists of all the vectors
in Rd that have negative inner products with the normal vector to ∂A at x pointing outside of A;
if x is in the interior of A, then Tx A = Rd . We consider the following technical conditions for the
main theorem in this section.
A5 The moment generating function Mgθ0 is twice differentiable at (θ† , γ † , λ† ).

12

A6 Under Q† , the the solution to the Euler condition is unique, that is, the equation with respect
to θ and γ
†

E Q {y > ∇θ log gθ (X)} ≤ 0 for all y ∈ Tθ Θ

(21)

†

E Q {y > ∇γ hγ (X)} ≤ 0 for all y ∈ Tγ Γ
has a unique solution (θ̄, γ̄). In addition,
†

†

E Q {sup |∇2θ log gθ (X)|} < ∞ and E Q {sup |∇2γ log hγ (X)|} < ∞.
γ∈Γ

θ∈Θ

We also assume that under measure Q† as n → ∞,
√

n(θ̂ − θ̄) = OQ† (1) and

√

n(γ̂ − γ̄) = OQ† (1),

where θ̂ and γ̂ are the maximum likelihood estimators
θ̂ = arg sup
θ

n
X

log gθ (Xi ) and γ̂ = arg sup
γ

i=1

n
X

log hγ (Xi ),

i=1

and a random sequence an = OQ† (1) means it is tight under measure Q† .
A7 We assume that gθ0 does not belong to the closure of the family of distributions {hγ : γ ∈ Γ},
that is, inf γ∈Γ D(gθ0 khγ ) > 0.
Assumption A6 requires n−1/2 convergence of θ̂ and γ̂ under Q† . It also requires the local
maximum of the function E Q† log gθ (X) and E Q† log hγ (X) to be unique. We elaborate the Euler
condition for θ ∈ int(Θ) and θ ∈ ∂Θ separately. If θ ∈ int(Θ), then Tθ Θ = Rdg . The Euler condition
†

is equivalent to E Q ∇θ log gθ (X) = 0, which is the usual first order condition for a local maximum.
†

If θ ∈ ∂Θ, then the Euler condition requires that the directional derivative of E Q {log gθ (X)} along
a vector pointing towards inside Θ is non-positive. Assumption A7 guarantees that the probability
limn→∞ Pgθ0 (LRn > enb ) = 0 for some b.
Theorem 9 Under Assumptions A2-A3 and A5-A7, for each b satisfying (17), we have
log Pgθ0 (LRn > enb ) ∼ −n × ρ†gθ ,
0

where ρ†gθ0 = − log Mg†θ0 and Mg†θ0 is defined in (19).
This theorem provides a means to approximate the type I and type II error probabilities for general
parametric families. The above results are applicable to the both cases that the two families are
13

separated or not separated. According to standard large deviations calculation for random walk,
we have that for each θ ∈ Θ and γ ∈ Γ,
log Pgθ0

n
X


log hγ (Xi ) − log gθ (Xi ) − nb > 0 ∼ inf log Mgθ0 (θ, γ, λ).
λ

i=1

Theorem 9 together with the above display implies that
log Pgθ0 (LRn > 1) ∼ inf sup log Pgθ0
θ

γ

∼ log Pgθ0

n
X


log hγ (Xi ) − log gθ (Xi ) > nb

i=1
n
X


log hγ † (Xi ) − log gθ† (Xi ) > nb

i=1

The exponential decay rate of the error probabilities under gθ0 is the same as the exponential decay
rate of the probability that hγ † is preferred to gθ† .
One application of Theorem 9 is to compute the power function asymptotically. Consider the
fixed type I error α and the critical region of the generalized likelihood ratio test is determined by
the quantile of a χ2 distribution, that is {LRn > eλα } where 2λα is the (1 − α)th quantile of the χ2
distribution. This correspond to choosing b = o(1). For a given alternative distribution hγ , one can
compute the type II error probability asymptotically by means of Theorem 9 switching the role of
the null and the alternative families. Thus, the power function can be computed asymptotically.

4.2

Application to model selection in generlized linear models

We discuss the application of Theorem 9 on model selection for generalized linear models [McCullagh and Nelder, 1989]. Let Yi be the response of the ith observation and X (i) = (Xi1 , ..., Xip )T
and Z (i) = (Zi1 , ..., Ziq )T be two sets of predictors, i = 1, ..., n. Consider a generalized linear model
with canonical link function and the true conditional distribution of Yi is
n
o
0 T (i)
0 T (i)
gi (yi , β ) = exp (β ) X yi − b((β ) X ) + c(yi ) ,
0

i = 1, 2, ..., n,

(22)

where f (y) = ec(y) is the base-line density, b(·) is the logarithm of the moment generating function,
β 0 = (β10 , ..., βp0 )T is the vector of true regression coefficients, and X is the set of true predictors.
Let the null hypothesis be
n
o
H0 : gi (yi , β) = exp β T X (i) yi − b(β T X (i) ) + c(yi ) ,

i = 1, 2, ..., n;

(23)

i = 1, 2, ..., n.

(24)

the alternative hypothesis is
n
o
H1 : hi (yi , γ) = exp γ T Z (i) yi − b(γ T Z (i) ) + c(yi ) ,
14

We further assume that H1 does not contain (22). Conditional on the covariates X and Z, we
consider the asymptotic decay rate of the type I error probability
Pβ 0 (LRn ≥ 1),
where LRn =

Q
supγ n
h (Y ,γ)
Qi=1 i i
supβ n
i=1 gi (Yi ,β)

is the generalized likelihood ratio.

We present the construction of the rate function as follows. For each β ∈ Rp , γ ∈ Rq and λ ∈ R,
define
ρen (β, γ, λ) =

n
i

o
1 X n h T (i)
λ b(γ Z )−b(β T X (i) ) +b((β 0 )T X (i) )−b (β 0 )T X (i) +λ(γ T Z (i) −β T X (i) ) .
n
i=1
(25)

Taking derivative with respect to λ, we have
n



o
∂
1 X n T (i)
ρen (β, γ, λ) =
b(γ Z )−b(β T X (i) )−b0 (β 0 )T X (i) +λ(γ T Z (i) −β T X (i) ) (γ T Z (i) −β T X (i) ) .
∂λ
n
i=1
(26)
According to fact that b(·) is a convex function, we have
lim sup
λ→+∞

∂
ρen (β, γ, λ) < 0,
∂λ

if β T X (i) 6= γ T Z (i) for some i. Define the set Bn ⊂ Rp such that
Bn = {β : inf
γ

∂
ρen (β, γ, 0) ≥ 0}.
∂λ

Then for each β ∈ Bn and γ ∈ Rq , there is a λ ≥ 0 such that

∂
en (β, γ, 0)
∂λ ρ

= 0. Thanks to the

convexity of b, β 0 ∈ Bn and thus Bn is never empty. The second derivative is
n

∂2
1 X 00  0 T (i)
T (i)
T (i)
ρ
e
(β,
γ,
λ)
=
−
(β
)
X
+
λ(γ
Z
−
β
X
)
(γ T Z (i) − β T X (i) )2 < 0,
b
n
(∂λ)2
n
i=1

if β T X (i) 6= γ T Z (i) for some i. Therefore, there is a unique solution to the maximization supλ ρen (β, γ, λ).
We further consider the optimization
ρe†n = sup inf sup ρen (β, γ, λ).
β∈Bn γ

(27)

λ

We consider the following technical conditions.
A8 For each n, the solution to (27) exists, denoted by (βn† , γn† , λ†n ). There exists a constant κ1
such that
kβn† k ≤ κ1 , kγn† k ≤ κ1 and λ†n ≤ κ1 for all n.
Here, k · k is the Euclidean norm.
15

A9 There exists a constant δ1 > 0 such that inf γ supλ ρen (β 0 , γ, λ) > δ1 for all n.
A10 There exists a constant κ2 such that kX (i) k ≤ κ2 and kZ (i) k ≤ κ2 for all i. Additionally,
P
there exits δ2 > 0 such that for all n the smallest eigenvalue of n1 ni=1 X (i) X (i)T is bounded
below by δ2 .
A11 For any compact set K ⊂ R, inf u∈K b00 (u) > 0. In addition, b(·) is four-time continuously
differentiable.
Assumption A8 requires that the solution of the optimization (27) does not tend to infinity as n increases, which is a mild condition. In particular, if the Kullback-Leibler divergence D(gi (·, β 0 )|gi (·, β))
tend to infinity uniformly for all i as kβk goes to infinity, then Bn is a bounded subset of Rp and
kβn† k is also bounded. Similar checkable sufficient conditions can be obtained for γn† and λ†n .
Theorem 10 Under Assumptions A8-A11, conditional on the covariates X (i) and Z (i) , i = 1, ..., n,
we have
log Pβ 0 (LRn ≥ 1) ∼ −n × ρe†n ,
where ρe†n is defined in (27).
For generalized linear models, the moment generating function of likelihood ratio is
n
n X
o
[log hi (Yi , γ) − log gi (Yi , β)] = e−neρn (β,γ,λ) .
Eβ 0 λ
i=1

Therefore, ρe†n is a natural generalization of ρ†gθ0 for the nonidentical distribution case.
Theorem 10 provides the asymptotic rate of selecting the wrong model by maximizing the
likelihood. The asymptotic rate as a function of the true regression coefficients β 0 quantifies the
strength of the signals. The larger the rate is, the easier it is to select the correct variables. The
rate also depends on covariates. If Z is highly correlated with X, then the rate is small. Overall,
the rate serves as an efficiency measure of selecting the true model from families that mis-specifies
the model.

5

Numerical examples

In this section, we present numerical examples to illustrate the asymptotic behavior of the maximal
type I and type II error probabilities and the sample size tends to infinity. The first one is an
example of continuous distributions and the second one is an example of discrete distributions.

16

The third one is an example of linear regression models where the null hypotheses and alternative
are not separated. In these examples, we compute the error probabilities using importance sampling
corresponding to the change of measure in the proof with sufficiently large number of Monte Carlo
replications to ensure that our estimates are sufficiently accurate.
Example 11 Consider the lognormal distribution and exponential distribution. For x > 0, let
gθ (x) =

(log x)2
1
− 2θ
e
x(2πθ)1/2

Θ = (0, +∞),

hγ (x) =

1 − γx
e
γ

Γ = (0, +∞)

be the density functions of the lognormal distribution and the exponential distribution.
For each θ and γ, we compute ρθγ numerically. Figure 2 shows the contour plot of ρθ,γ . The
minimum of ρθγ is 0.020 and is obtained at (θ∗ , γ ∗ ) = (1.28, 1.72). From the theoretical analysis,
the maximal type I and type II error probabilities for the test decay at rate e−nρθ∗ γ ∗ .
Figure 3 is the plot of the maximal type I and type II error probabilities as a function of the
sample size for the composite versus composite test
H0 : f ∈ {gθ ; θ ∈ Θ}

against

H1 : f ∈ {hγ ; γ ∈ Γ}

against

H1 : f = hγ∗ .

and simple versus simple test
H0 : f = gθ∗

We also fit a straight line to the logarithm of error probabilities against the sample sizes using least
squares and the slope is −0.022. This confirms the theoretical findings. The error probabilities
shown in Figure 3 range from 7 × 10−5 to 0.12 and the range for sample size is from 50 to 370.

Example 12 We now proceed to the Poisson distribution versus the geometric distribution. Let
gθ (x) =

e−θ θx
x!

Θ = [1, +∞),

hγ (x) =

γx
(1 + γ)x+1

Γ = [0.5, +∞),

for x ∈ Z+ . The parameter γ is the failure to success odds. The minimum Chernoff index without
constraint is attained at θ = γ = 0 and ρ00 = 0. Thus we truncate the parameter spaces away from
zero to separate the two families.
The Chernoff index ρθ,γ can be computed numerically and is minimized at (θ∗ , γ ∗ ) = (1, 0.93),
with ρθ∗ ,γ ∗ = 0.023. Figure 4 shows the contour plot of ρθ,γ . Same as in the previous example, we
compute the maximal type I and type II error probabilities of the composite versus composite test
and simple versus simple test. Figure 5 shows the maximal type I and type II error probabilities as
17

10
8
6
4
0

2

gamma

0

2

4

6

8

10

theta

Figure 2: Contour plot for ργ,θ in Example 11. The triangle point indicates the minimum.

a function of the sample size. The error probabilities appeared in Figure 5 range from 1.0 × 10−4 to
0.10 with the sample sizes range from 40 to 400. We also fit a straight line to the logarithm of error
probabilities against the sample sizes and the slope is −0.025. This numerical analysis confirms our
theorems.
Example 13 We consider two regression models,
H0 : Y = β1 X1 + β2 X2 + ε1 against H1 : Y = β1 X1 + ζ1 Z1 + ε2 ,
where (X1 , X2 , Z1 ) jointly follows the multivariate Gaussian distribution with mean (0, 0, 0)T and
the covariance matrix Σ. The random noises ε1 and ε2 follow the normal distributions N (0, σ12 )
and N (0, σ22 ) respectively and are independent of (X1 , X2 , Z1 ). We assume the true model to be
Y = β10 X1 + β20 X2 + ε,
with the following parameters



1 0.1 0.1
β10 = 1, β20 = 2, ε ∼ N (0, 1), and Σ = 0.1 1 0.1 .
0.1 0.1 1
Let (Xi1 , Xi2 , Zi1 , Yi )T be i.i.d. copies of (X1 , X2 , Z1 , Y ) generated under the true model, for i =
1, ..., n. Let θ = (β1 , β2 ) and γ = (β1 , ζ1 ) be the regression coefficients for the null and the alternative hypotheses respectively. The maximum likelihood estimators for θ and γ are the least square
18

●

●

●
●

●

●

5e−02

●
●

●
●
●

●

●

●
●
●

type I error probability of simple versus simple test
type II error probability of simple versus simple test
type I error probability of composite versus composite test
type II error probability of composite versus composite test

●

●
●
●
●

●
●
●
●

5e−03

error probability

●
●
●

●

●
●
●
●

●
●
●
●

●
●
●
●

●
●
●

●

●
●
●

●

5e−04

●
●
●
●

●
●
●

●

●
●
●
●

●
●
●

1e−04

●

●
●
●

●

●
●
●

50

100

150

200

250

300

350

n

Figure 3: Decay rate of type I and type II error probabilities (y-coordinate) as a function of sample
size (x-coordinate) in Example 11.

estimators
θ̂ = (X̃ > X̃)−1 X̃ > Ỹ and γ̂ = (Z̃ > Z̃)−1 Z̃ > Ỹ ,
where






 
X11 X12
X11 Z11
Y1
 X21 X22 
 X21 Z21 
 Y2 
 , Z̃ = 

 
X̃ = 
···
 · · · · · ·  , and Ỹ = · · ·
··· 
Xn1 Xn2
Xn1 Zn1
Yn

are the design matrices for linear models under H0 and H1 . We consider the error probability that
the maximized log-likelihood of H0 is smaller than that of H1 , equivalently, the residual sum of
squares under H0 is larger than that under H1


Pβ 0 ,Σ kỸ − X̃ θ̂k2 > kỸ − Z̃ γ̂k2 .
From the theoretical analysis, the above probability decays at rate e
definition of
obtain

ρ†gθ0

ρ†gθ0

−nρ†gθ

0

as n → ∞, where the

is given in Theorem 9. We solve the optimization problem (19) numerically and

= 0.45. Figure 6a and Figure 6b are scatter plots of the error probability in the above

display as a function of the sample size with different ranges for error probabilities. In Figure 6a,
the range of the error probability is from 10−4 to 0.25 and the range of sample size is from 3 to
18. In Figure 6b, the range of error probabilities is from 1.2 × 10−8 to 4.0 × 10−6 with the sample


size from 24 to 36. We fit straight lines for log Pβ 0 ,Σ kỸ − X̃ θ̂k2 > kỸ − Z̃ γ̂k2 against n using

19

least square. The fitted slope in Figure 6a is −0.52 and the fitted slope in Figure 6b is −0.47. This

3
1

2

gamma

4

5

confirms our theoretical results.

1

2

3

4

5

6

theta

Figure 4: Contour plot for ργ,θ in Example 12. The triangle point indicates the minimum.

6

Concluding remarks

The generalized likelihood ratio test of separate parametric families that was put forth by Cox
in his two seminal papers has received a great deal of attention in the statistics and econometrics
literature. The present investigation takes the viewpoint of an early work by Chernoff (1952) where
testing a simple null versus a simple alternative is considered. By imposing that the two types of
error probabilities decay at the same rate, we extend the Chernoff index to the case of the Cox test.
Our results are under the basic assumption that the data come from one of the parametric
families under consideration. It is often the case that none is the true model. It would be of
interest to formulate error probabilities for this case and to see if similar exponential decay results
continue to hold.
An initial motivation that led to the Cox formulation of the problem comes from the survival
analysis where different models are used to fit failure time data. The econometrics literature also
contains much subsequent development. Semiparametric models that contain infinite dimensional
nuisance parameters are widely used in both econometrics and survival analysis. It would be of
interest to develop parallel results for testing separate semiparametric models.
20

1e−01

●
●
●
●

●

●
●
●
●

●

●

●
●
●
●

1e−02

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

1e−03

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

●
●
●
●
●
●
●

1e−04

error probability

type I error probability of simple versus simple test
type II error probability of simple versus simple test
type I error probability of composite versus composite test
type II error probability of composite versus composite test

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

1e−05

●
●
●
●
●
●
●
●

50

100

150

200

250

300

350

400

n

Figure 5: Maximal type I and type II error probabilities (y-coordinate) as a function of sample
size (x-coordinate) in Example 12.

A

Proof of Lemma 3

Throughout the proof, we adopt the following notation an ∼
= bn if log an ∼ log bn . We define the
log-likelihood ratio as
lγ (x) = log hγ (x) − log g(x).
The generalized log-likelihood ratio statistic is defined as
l = sup

n
X

γ

lγi

i=1

where lγi = lγ (Xi ). The generalized likelihood ratio test admits the rejection region
Cλ = {el > λ}.
We consider the case that λ = 1 and show that for this particular choice of λ the maximal type I
and type II error probabilities decay exponentially fast with the same rate. We let γ∗ = arg inf ργ
and thus ρ = ργ∗ .
Based on Chernoff’s calculation of large deviations for the log-likelihood ratio statistic, we
proceed to the calculation of the type I error probability
n


X
Pg (l > 0) = Pg sup
lγi > 0 .
γ

21

i=1

●

●

2e−06

5e−02

●
●

●

●

●
●

●
●
●

●
●

2e−07

error probability

●

5e−03

error probability

●

5e−07

●

●

●

●

5e−08

●

5e−04

●
●

●

●
●

●

1e−08

1e−04

●
●

5

10

●

24

15

26

28

30

n

n

(a)

(b)

32

34

36

Figure 6: Error probability(y-coordinate) in Example 13 as a function of sample size(x-coordinate).
We now provide an approximation of the right-hand side, which requires a lower bound and an
upper bound. We start with the lower bound by noticing that
n
n


X

X
Pg sup
lγi > 0 ≥ sup Pg
lγi > 0
γ

γ

i=1

(28)

i=1

that is a simple lower bound. According to Proposition 1, the right-hand side is bounded from
below by
≥ e−{1+o(1)}nρ
where ρ = min ργ . For the upper bound and with some β > 0, we split the probability


Pg sup
γ

n
X
i=1

lγi

>0





≤ Pg sup
γ

n
X

lγi

> 0, sup
γ

i=1



+ Pg sup
γ

n
X

n
X

∇lγi < en

1−β



i=1
1−β

∇lγi ≥ en



.

(29)

i=1

The first term on the right-hand side is bounded by Lemma 14.
Lemma 14 Consider a random function ηn (θ) living on a d-dimensional compact domain θ ∈ D,
where n is an asymptotic parameter that will be send to infinity. Suppose that ηn (θ) is almost surely
differentiable with respect to θ and for each θ, there exists a rate ρ(θ) such that
P {ηn (θ) > ζn } ∼
= e−nρ(θ)

for all ζn /n → 0 as n → ∞
22

where the above convergence is uniform in θ. Then, we have the following approximation
1
1−β
lim inf − log P {sup ηn (θ) > 0, sup |∇ηn (θ)| < en } ≥ min ρ(θ)
n→∞
θ
n
θ∈D
θ∈D
for all β > 0.
With the aid of Proposition 1, we have that the random function

Pn

i
i=1 lγ

satisfies the assumption in

Lemma 14 with ρ(γ) = ρθγ . Then the first term in (29) is bounded from the above by e−{1+o(1)}nρ .
For the second term in (29), according to condition A3, we choose β sufficiently small such that


Pg sup
γ

n
X

1−β

∇lγi ≥ en



≤ n × Pg (sup |∇lγi | > n−1 en

1−β

) = o(e−nρ ).

γ

i=1

Thus, we obtain an upper bound


Pg sup
γ

n
X


lγi > 0 ≤ e−n{ρ+o(1)} .

i=1

Then, the type I error probability is approximated by
e

−nρ

∼
= sup Pg
γ

n
X

lγi





> 0 ≤ Pg sup
γ

i=1

n
X


lγi > 0 ≤ e−n{ρ+o(1)} .

(30)

i=1

We now consider the type II error probability α2 = supγ Phγ (l < 0). For each γ, note that
n
n

X


X
lγi 1 < 0 ≤ Phγ
lγi < 0 .
Phγ (l < 0) = Phγ sup
γ1

i=1

i=1

Note that the right-hand side is the type II error probability of the likelihood ratio test. According
to Chernoff’s calculation, we have that
Phγ (l < 0) ≤ Phγ

n
X


lγi < 0 ∼
= e−nργ

i=1

for all γ. We take maximum with respect to γ on both sides and obtain that
sup Phγ (l < 0) ≤ sup Pg
γ

γ

n
X


lγi > 0 ∼
= e−n minγ ργ .

(31)

i=1

Thus, the maximal type II error probability has an asymptotic upper bound that decays at the
rate of the Chernoff index.
In what follows, we show that this asymptotic upper bound is asymptotically achieved. We
choose λn possibly depending on g such that
n
n


X

X
lγi ∗ > nλn .
Pg sup
lγi > 0 = Pg
γ

i=1

i=1

23

Note that g is fixed and the probabilities on both sides of the above identity decay at the rate e−nρ .
Together with the continuity of the large deviations rate function, it must be true that λn → 0−. We
apply Neyman-Pearson lemma to the simple null H0 : f = g versus simple alternative H1 : f = hγ∗ .
P
P
Note that { ni=1 lγi ∗ > nλn } is a uniformly most powerful test and {supγ ni=1 lγi > 0} is a test
with the same type I error probability. Then, we have that
n
n


X

X
Phγ∗ sup
lγi < 0 ≥ Phγ∗
lγi ∗ < nλn .
γ

i=1

(32)

i=1

That is, the type II error probability of the generalized likelihood ratio test must be greater than
that of the likelihood ratio test under the simple alternative hγ∗ . Note that λn → 0−. Thanks to
the the continuity of the large deviations rate function, we have that
Phγ∗

n
X

lγi ∗

< nλn



∼
= Phγ∗

n
X

i=1


lγi ∗ < 0 ∼
= e−nρ .

(33)

i=1

Put together (31), (32), and (33), we have that
sup Phγ (l < 0) ∼
= e−nρ .
γ

Thus, we conclude the proof.

B

Proof of Theorem 4

The one-to-one log-likelihood ratio is
lθγ (x) = log hγ (x) − log gθ (x).
The generalized log-likelihood ratio statistic is
l = sup
γ

n
X
i=1

log hγ (Xi ) − sup
θ

n
X

log gθ (Xi ) = inf sup
θ

i=1

γ

n
X

i
lθγ

i=1

and the rejection region is
Cλ = {el > λ}.
We define that γ(θ) = arg inf γ ρθγ , and θ(γ) = arg inf θ ρθγ , and (θ∗ , γ∗ ) = arg inf θ,γ ρθγ . Note that
the null and the alternative are now symmetric, thus we only need to consider one of the two types
of error probabilities. We consider the type II error probability. We now define
kθ = sup

n
X

γ

24

i=1

i
lθγ
.

For each given θ and γ, we have a simple upper bound
n
X

i
Phγ (kθ < 0) ≤ Phγ
lθγ
<0 ∼
= e−nρθγ .

(34)

i=1

We now proceed to the type II error probability if hγ is the true distribution, that is
1−β

Phγ (inf kθ < 0) ≤ Phγ (inf kθ < 0; sup |∇kθ | < en
θ

θ

) + Phγ (sup |∇kθ | ≥ en

θ

1−β

).

θ

The first term on the right-hand-side is bounded by Lemma 14 combined with (34)
1−β

Phγ (inf kθ < 0; sup |∇kθ | < en
θ

) ≤ e−n{inf θ ρθγ +o(1)} .

θ

For the second term, we have that
Phγ {sup |∇(sup
θ

γ

n
X

i
lθγ
)| ≥ en

1−β

} ≤ Phγ (sup sup
γ

θ

i=1

n
X

1−β

i
|∇lθγ
| ≥ en

)

i=1
1−β

i
| ≥ n−1 en
≤ nPhγ (sup sup |∇lθγ
θ

) = o(e−nρ ).

γ

Thus, we have that
Phγ (inf kθ < 0) = Phγ (l < 0) ≤ e−n{inf θ ρθγ +o(1)} ,
θ

which provides an upper bound for the type II error probability
sup Phγ (l < 0) ≤ e−n{inf θ,γ ρθγ +o(1)} .
γ

We now provide a lower bound. For a given θ and γ(θ) = arg inf γ ρθγ , applying proof of Lemma
3 for the type II error probability by considering H0 : f = gθ and H1 : f ∈ {hγ : γ ∈ Γ}, we have
that
Phγ(θ) (kθ < 0) ∼
= e−nρθγ(θ) .
and thus
Phγ(θ) (inf kθ < 0) ≥ Phγ(θ) (kθ < 0) ∼
= e−nρθγ(θ) .
θ

We set θ = θ∗ in the above asymptotic identity and conclude the proof.

C

Proof of Lemma 14

We consider a change of measure on the continuous sample path space Qζ that admits the following
Radon-Nikodym derivative
dQζ
mes(Aζ )
=R
,
dP
P
{η
n (θ) > ζ}dθ
D

(35)

where Aζ = {θ ∈ D : ηn (θ) > ζ} and mes(·) is the Lebesgue measure. Throughout the proof,
we choose ζ = −1. To better understand the measure Qζ , we provide another description of the
sample path generation of ηn from Qζ , that requires the following three steps
25

1. Sample a random index τ ∈ D following the density function
h(τ ) = R

P {ηn (τ ) > ζ}
;
D P {ηn (θ) > ζ}dθ

2. Sample ηn (τ ) given that ηn (τ ) > ζ;
3. Sample {ηn (θ) : θ 6= τ } from the original conditional distribution given the realized value of
ηn (τ ), that is, P {·|ηn (τ )}.
To verify that the measure induced by the above sampling procedure is the same as that given by
(35), see Adler et al. [2012] that provides a discrete analogue of the above change of measure.
With these constructions, the interesting probability is given by
P {sup ηn (θ) > 0, sup |∇ηn (θ)| < en
θ∈D

1−β

}

θ∈D

o
n dP
1−β
; sup ηn (θ) > 0, sup |∇ηn (θ)| < en
dQζ θ∈D
θ∈D
n
o
1
1−β
= E Qζ
; sup ηn (θ) > 0, sup |∇ηn (θ)| < en
mes(Aζ ) θ∈D
θ∈D
Z
×
P (ηn (θ) > ζ)dθ
= E Qζ

D

Via the condition of this lemma, we have that
Z
P (ηn (θ) > ζ)dθ ∼
= e−n minθ ρ(θ) .
D

Thus, it is sufficient to show that
E Qζ

n

o
1
1−β
; sup ηn (θ) > 0, sup |∇ηn (θ)| < en
mes(Aζ ) θ∈D
θ∈D

cannot be too large. On the set {supθ∈D ηn (θ) > 0, supθ∈D |∇ηn (θ)| < n1−β }, the volume mes(Aζ )
is in fact lower bounded. Let θ∗ be the maximizer of ηn (θ) and thus ηn (θ∗ ) > 0. On the other
1−β

hand, the gradient of ηn is upper bounded by en
1−β

e−n

. Therefore, there exists a small region of radius

in which ηn will be above ζ = −1. Thus, mes(Aζ ) is lower bounded by ε0 e−dn

1−β

the bound
1−β

edn
P (sup ηn (θ) > 0, sup |∇ηn (θ)| < n ) ≤
ε0
θ∈D
θ∈D
β

concludes the proof.

26

Z
D

P (ηn (θ) > ζ)dθ ∼
= e−n minθ ρ(θ)

. Thus,

D

Proof of Corollary 6

The proof is very similar to that of Theorem 4 and therefore we omit some repetitive steps. We
first consider the type I error probability,
sup Pgθ (LRn > 1).
θ∈Θ

For each θ ∈ Θ, we establish an upper bound for


Pgθ (LRn > 1) = Pgθ sup log hγ (Xi ) − sup log gθ (Xi ) > 0 .
γ∈Γ

(36)

θ∈Θ

The event
{sup log hγ (Xi ) − sup log gθ (Xi ) > 0}
γ∈Γ

θ∈Θ

implies
{sup log hγ (Xi ) − log gθ (Xi ) > 0}.
γ∈Γ

Thus, we have


Pgθ (LRn > 1) ≤ Pgθ sup log hγ (Xi ) − log gθ (Xi ) > 0 .
γ∈Γ

We split the probability


Pgθ sup log hγ (Xi ) − log gθ (Xi ) > 0
γ∈Γ




≤ Pgθ sup log hγ (Xi ) − log gθ (Xi ) > 0, γ̂ ∈ Aθ + Pgθ γ̂ ∈ Acθ


γ∈Γ

≤ Pgθ






sup log hγ (Xi ) − log gθ (Xi ) > 0 + Pgθ γ̂ ∈ Acθ .

(37)

γ∈Aθ

According to Assumption A4, the second term is o(e−nρ ). For the first term, notice that Aθ is a
compact subset of Rdg . The conditions for Lemma 3 are satisfied. According to Lemma 3, the first
term in (37) is bounded above by
e−(1+o(1))n×minγ∈Aθ ρθγ ≤ e−(1+o(1))n×minγ∈Γ ρθγ ≤ e−(1+o(1))n×minθ,γ ρθγ .
Combining the upper bounds for the first and second terms in (37), we have
Pgθ (LRn > 1) ≤ e−(1+o(1))n×minθ,γ ρθγ .
The above derivation is uniform in θ. We obtain an upper bound for the type I error
sup Pgθ (LRn > 1) ≤ e−(1+o(1))n×minθ,γ ρθγ .
θ

27

Similarly, we obtain an upper bound for the type II error probability
sup Phγ (LRn ≤ 1) ≤ e−(1+o(1))n×minθ,γ ρθγ .
γ

Now we proceed to a lower bound for the type I error probability. Upon having the upper bounds
for both type I and type II error probabilities, the lower bounds for type I and type II error
probabilities can be derived using the same argument as that in the proof of Theorem 4. We omit
the details.

E

Proof of Theorem 9

The proof of the theorem consists of establishing upper and lower bounds for the probability
n


X
Pgθ0 (LRn > enb ) = Pgθ0 sup inf
[log hγ (Xi ) − log gθ (Xi )] > nb .
γ∈Γ θ∈Θ i=1

Upper bound

The event
n

sup inf

n
X

γ∈Γ θ∈Θ i=1

implies
n

sup

n
X

γ∈Γ i=1

o
log hγ (Xi ) − log gθ (Xi ) > nb

o
log hγ (Xi ) − log gθ† (Xi ) > nb .

Therefore, we have an upper bound


Pgθ0 (LRn > 1) ≤ Pgθ0 sup
γ

n
X


log hγ (Xi ) − log gθ† (Xi ) > nb .

(38)

i=1

We split the probability
n


X
Pgθ0 sup
[log hγ (Xi ) − log gθ† (Xi )] > nb
γ

≤ Pgθ 0



(39)

i=1

n
n

X
X
1−β
sup
[log hγ (Xi ) − log gθ† (Xi )] > nb, sup
∇γ log hγ (Xi ) < en
γ

i=1
n
X



+Pgθ0 sup
γ

γ


1−β

∇γ log hγ (Xi ) ≥ en

i=1

.

i=1

We establish upper bounds of the first and second terms in (39) separately. For the first term,
Pn
let ηn (γ) =
i=1 [log hγ (Xi ) − log gθ† (Xi )] − nb. For each γ, the exponential decay rate of the
probability
log Pgθ0 (ηn (γ) ≥ 0) ≤ n log inf Mgθ0 (λ, γ, θ† ).
λ

28

(40)

is established through standard large deviation calculation. Thanks to Lemma 14 and (40), the
first term in (39) is bounded above by
sup inf {Mgθ0 (θ† , λ, γ)}(1+o(1))n = e

−(1+o(1))nρ†gθ

0

λ

γ

.

For the second term, according to the Assumption A3,
n


X
1−β
1−β
−nρ†
Pgθ0 sup
∇γ log hγ (Xi ) ≥ en
≤ n × Pgθ0 (sup |∇γ log hγ (Xi )| > n−1 en ) = o(e gθ0 ).
γ

γ

i=1

Combining the analyses for both the first and the second term, we arrive at an upper bound
Pgθ0 (LRn > enb ) ≤ e
Lower bound

−(1+o(1))nρ†gθ

0

.

Recall that
n
o
dQ†
= exp λ† (log hγ † (X) − log gθ† (X)) − nb /Mg†θ .
0
dPgθ0

Then, the probability can be written as
Pgθ0 (LRn > enb ) = E Q

†

n dPg

θ0

dQ†

;

n
X

o
[log hγ̂ (Xi ) − log gθ̂ (Xi )] > nb ,

i=1

where γ̂ and θ̂ are the maximum likelihood estimators for the h-family and the g-family respectively.
According to the definition of Q† , the above display is equal to
−nρ†gθ

e

0

EQ

†

n
n
o
P
X
−λ† [ n
i=1 log hγ † (Xi )−log gθ † (Xi )−nb]
;
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb ,
e

(41)

i=1

where ρ†gθ0 = − log Mg†θ0 . We now establish a lower bound for
I , EQ

†

n
n
o
P
X
−λ† [ n
i=1 log hγ † (Xi )−log gθ † (Xi )−nb]
;
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb .
e
i=1

Because e

−λ† [

Pn

I≥E

i=1

Q†

log hγ † (Xi )−log gθ† (Xi )−nb]

is positive, we have

n
n
o
P
X
−λ† [ n
i=1 log hγ † (Xi )−log gθ † (Xi )−nb]
e
;
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb, E1 ,
i=1

where
E1 =

n
n X

log hγ † (Xi ) − log gθ† (Xi ) − nb ≤

√ o
n .

i=1

On the set E1 , we have the following inequality of the integrand
−λ† [

e

Pn

i=1

log hγ † (Xi )−log gθ† (Xi )−nb]

29

≥ e−|λ

† | √n

.

(42)

We plug the above inequality back to (42) and obtain a lower bound for
I ≥ e−|λ

† |√ n

n
 X

Q† {
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb} ∩ E1 .

(43)

i=1

For the rest of the proof, we develop a lower bound for the probability
n
 X

Q† {
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb} ∩ E1 .
i=1

The maximum likelihood estimator γ̂ satisfies the inequality
n
X

{log hγ̂ (Xi ) − log hγ † (Xi )} ≥ 0.

(44)

i=1

Furthermore, with the aid of Rolle’s Theorem, there exists θ̃ such that
n
X
{log gθ̂ (Xi ) − log gθ† (Xi )}
i=1

= (θ̂ − θ† ) ·

n
X
i=1

n

X
1
∇2θ gθ̃ (Xi )(θ̂ − θ† ),
∇θ log gθ† (Xi ) + (θ̂ − θ† )>
2

(45)

i=1

where “∇2θ ” denotes the Hessian matrices with respect to θ and “·” denotes the inner product
between vectors. (44) and (45) together give
n
X

{log hγ̂ (Xi ) − log gθ̂ (Xi )} −

i=1

n
X

{log hγ † (Xi ) − gθ† (Xi )}

i=1
†

≥ −(θ̂ − θ ) ·

n
X
i=1

We define

n

X
1
∇θ log gθ† (Xi ) − (θ̂ − θ† )>
∇2θ gθ̃ (Xi )(θ̂ − θ† ).
2

(46)

i=1

√ o
n
n
X
n
† >
E2 = (θ̂ − θ )
∇θ log gθ† (Xi ) ≤
,
4
i=1

E3 =

E4 =

n1

n √n
2

2

† 2

|θ̂ − θ | sup

<

θ
n
X

n
X

|∇2θ log gθ (Xi )|

i=1

√ o
n
≤
,
4

[log hγ † (Xi ) − log gθ† (Xi )] − nb ≤

√ o
n .

i=1

Based on (46), we have that
n
X
(E2 ∩ E3 ∩ E4 ) ⊂ {
log hγ̂ (Xi ) − log gθ̂ (Xi ) > nb} ∩ E1 .
i=1

We insert this to (42), and obtain that
I ≥ e−|λ

† | √n

Q† (E2 ∩ E3 ∩ E4 ) ≥ e−|λ

† |√ n

30

n
o
Q† (E4 ) − Q† (E2c ) − Q† (E3c ) .

(47)

For the rest of the proof, we develop upper bounds for Q† (E2c ) and Q† (E3c ) and a lower bound for
Q† (E4 ). For Q† (E4 ), because λ† = arg inf λ Mgθ0 (θ† , γ † , λ), we have
∂
Mg (θ† , γ † , λ† ) = 0.
∂λ θ0
Consequently,
†

E Q (log hγ † (X) − log gθ† (X) − b) = (Mg†θ )−1
0

∂
Mg (θ† , γ † , λ† ) = 0.
∂λ θ0

According to the central limit theorem, there exists ε0 > 0 such that
lim inf Q† (E4 ) > ε0 .
n→∞

Thus a lower bound for Q† (E4 ) has been derived. Before we proceed to upper bounds for Q† (E2c )
and Q† (E3c ), we establish the following lemma, whose proof is provided in Appendix G.
Lemma 15 Under the settings of Theorem 9, we have
γ † = γ̄ and θ† = θ̄.
We now proceed to an upper bound of Q† (E2c ). We split the sum
(θ̂ − θ† )>

n
X

∇θ log gθ† (Xi )

(48)

i=1
† >

= (θ̂ − θ )

n
X

†

†

[∇θ log gθ† (Xi ) − E Q ∇θ gθ† (Xi )] + n(θ̂ − θ† )> E Q ∇θ gθ† (X)

i=1
†

Note that θ̂ ∈ Tθ† Θ, according to Assumption A6 and Lemma 15, we have that (θ̂−θ† )> E Q ∇θ gθ† (X) ≤
0. Therefore, (48) implies
(θ̂ − θ† )>

n
X

∇θ log gθ† (Xi ) ≤ (θ̂ − θ† )>

i=1

n
X

†

[∇θ log gθ† (Xi ) − E Q ∇θ gθ† (Xi )].

i=1

Using Chebyshev’s inequality and the fact E(|∇θ log gθ† (X)|2 ) < ∞, we have
3

n− 4

n
X

†

[∇θ log gθ† (Xi ) − E Q ∇θ gθ† (Xi )] → 0 in probability Q† .

i=1

According to Slutsky’s theorem and
√

† > − 43

n(θ̂ − θ ) n

√

n(θ̂ − θ† ) = OQ† (1), we have

n
X

∇θ log gθ† (Xi ) → 0 in probability Q† .

i=1

31

(49)

Consequently,
†



† >

lim Q (θ̂ − θ )

n→∞

n
X

[∇θ log gθ† (Xi ) − E

i=1

Q†

√ 
n
= 0.
∇θ gθ† (Xi )] >
4

According to (49) and the above display, we have
†



† >

lim Q (θ̂ − θ )

n
X

n→∞

i=1

√ 
n
= 0.
∇θ log gθ† (Xi ) >
4

Thus, Q† (E2c ) → 0 as n → ∞. We provide an upper bound of Q† (E3c ) using a similar method.
With the aid of Chebyshev’s inequality, we have
5

n− 4

n
X

sup |∇2θ log gθ (Xi )|→0 in probability Q† .

i=1

θ

According to Slutsky’s theorem and

√

n(θ̂ − θ† ) = OQ† (1), we have
5

n|θ̂ − θ† |2 × n− 4

n
X

d

sup |∇2θ log gθ (Xi )| → 0.
θ

i=1

Consequently,
†



† 2

lim Q |θ̂−θ | sup

n→∞

θ

Therefore,

Q† (E3 )

n
X

|∇2θ log gθ (Xi )|

i=1

√ 
√ 
n

X
n
n
† 2
2
†
>
sup |∇θ log gθ (Xi )| >
≤ lim Q |θ̂−θ |
= 0.
n→∞
4
4
θ
i=1

→ 0 as n → ∞. We combine the results for
I≥

Q† (E2c ), Q† (E3c ),

Q† (E4 ), and (47),

ε0 −|λ† |√n
e
for n sufficiently large.
2

Combining the above display with (41), we arrive at the lower bound
Pgθ0 (LRn > enb ) ≥ e

−n(1+o(1))ρ†gθ

0

.

We complete the proof by combining the lower bound and upper bound for the probability Pgθ0 (LRn >
1).

F

Proof of Theorem 10

The proof is similar to that of Theorem 9. Throughout the proof, we will use κ as a generic
notation to denote large and not-so-important constants whose value may vary from place to place.
Similarly, we use ε as a generic notation for small positive constants. The proof of the theorem
consists of establishing upper and lower bounds for the probability
n


X
Pβ 0 (LRn ≥ 1) = Pβ 0 sup inf
[log hi (Yi , γ) − log gi (Yi , β)] ≥ 0 .
γ

β

i=1

32

Upper bound

Similar to (38), we have
n


X
Pβ 0 (LRn ≥ 1) ≤ Pβ 0 sup
[log hi (Yi , γ) − log gi (Yi , βn† )] ≥ 0
γ

i=1

According to the definition of hi (Yi , γ) and gi (Yi , β), we have
n
n
n
X
X
X
[log hi (Yi , γ) − log gi (Yi , β)] =
[γ T Z (i) Yi − b(γ T Z (i) )] −
[βn†T X (i) Yi − b(βn†T X (i) )].
i=1

i=1

i=1

Consequently, we have
n
n

 1X
1 X (i)
(i)
Z Yi ,
X Yi ) ∈ An ,
Pβ 0 (LRn ≥ 1) ≤ Pβ 0 (
n
n
i=1

(50)

i=1

where
n
n
o
n
1X
1X
T (i)
†T
p
q
T
b(γ Z )] ≥ [βn s1 −
b(βn†T X (i) )] .
An = (s1 , s2 ) : s1 ∈ R , s2 ∈ R and sup[γ s2 −
n
n
γ
i=1

i=1

We consider the change of measure
n
n
n X
o
X
dQ†
T
= exp λ†n
(γn†T Z (i) Yi −βn†T X (i) Yi )− [b((β 0 )T X (i) +λ†n {γn†T Z (i) −βn†T X (i) })−b((β 0 ) X (i) )] .
dP
i=1
i=1
(51)

According to (50), we have
Pβ 0 (LRn ≥ 1) ≤ E Q

†

n
n
h dP 1 X
i
1 X (i)
(i)
;
(
Z
Y
,
X
Y
)
∈
A
.
i
i
n
n
dQ† n
i=1

i=1

The above display and (51) together gives
Pβ 0 (LRn ≥ 1) ≤ exp

n h 
nX


io
T
T
b (β 0 ) X (i) + λ†n {γn†T Z (i) − βn†T X (i) } − b (β 0 ) X (i)
i=1

n
n
h
i
† Pn
†T (i)
†T
1 X (i)
1 X (i)
†
(i)
× E Q e−λn i=1 (γn Z Yi −βn X Yi ) ; (
Z Yi ,
X Yi ) ∈ An . (52)
n
n
i=1

i=1

The next lemma shows a property of βn† and An .
Lemma 16 For all (s1 , s2 ) ∈ An ,
[γ

†T

n

n

i=1

i=1

1X
1X
s2 −
b(γ †T Z (i) )] ≥ [βn†T s1 −
b(βn†T X (i) )].
n
n

33

According to Lemma 16, the right-hand side of (52) is further bounded above by
Pβ 0 (LRn ≥ 1)
n h 
n h 
nX

i
 
io
X
T
T
≤ exp
b (β 0 ) X (i) +λ†n {γn†T Z (i) −βn†T X (i) } −b((β 0 ) X (i) ) −λ†n
b γn†T Z (i) −b βn†T X (i)
i=1

i=1
n
n
h 1X
i
1 X (i)
× Q† (
Z (i) Yi ,
X Yi ) ∈ An . (53)
n
n
i=1

h

Because Q† ( n1

Pn

i=1 Z

Pβ 0 (LRn ≥ 1) ≤ exp

(i) Y , 1
i n

n
nX

Pn

i=1 X

(i) Y )
i

i=1

i

∈ An ≤ 1, we arrive at

T

T

[b((β 0 ) X (i) +λ†n {γn†T Z (i) −βn†T X (i) })−b((β 0 ) X (i) )]−λ†n

i=1

n
X

o
[b(γn†T Z (i) )−b(βn†T X (i) )] .

i=1
†

According to the definition of ρe†n , the right-hand side of the above inequality equals e−neρn . Therefore, we arrive at the upper bound
†

Pβ 0 (LRn ≥ 1) ≤ e−neρn .
Lower bound

Notice that the event
{

n
X

log hi (Yi , γn† ) − sup
β

i=1

implies the event
{sup
γ

n
X

n
X

log hi (Yi , γ) − sup

n
X

β

i=1

log gi (Yi , β) ≥ 0}.

i=1

log gi (Yi , β) ≥ 0}.

i=1

Therefore, a lower bound for the probability Pβ 0 (LRn ≥ 1) is
Pβ 0

n
X
i=1

log hi (Yi , γn† )

− sup
β

n
X


log gi (Yi , β) ≥ 0 .

i=1

According to the definition of Q† in (51), the above probability equals
n
nX
o
T
T
exp
[b((β 0 ) X (i) + λ†n {γn†T Z (i) − βn†T X (i) }) − b((β 0 ) X (i) )]
i=1

h
i
† Pn
†T (i)
†T
†
(i)
× E Q e−λn i=1 (γn Z Yi −βn X Yi ) ; E , (54)
where the event
E=

n
nX

o
γn†T Z (i) Yi − β̂nT X (i) Yi − b(γn†T X (i) ) + b(β̂nT X (i) ) ≥ 0 ,

i=1

34

and β̂n is the maximum likelihood estimator
β̂n = arg sup
β

n
X

β T X (i) Yi − b(βX (i) ).

i=1

Notice that
†

e−neρn = exp

n
nX

o
T
T
[b((β 0 ) X (i) +λ†n {γn†T Z (i) −βn†T X (i) })−b((β 0 ) X (i) )]−λ†n [b(γn†T Z (i) )−b(βn†T X (i) )] .

i=1

Therefore,
Pβ 0 (LRn ≥ 1) ≥ e−neρn × J,

(55)

where we define the quantity
h
i
† Pn
†T (i)
†T
†T
†T
†
(i)
(i)
(i)
J = E Q e−λn [ i=1 γn Z Yi −βn X Yi −b(γn X )+b(βn X )] ; E .
We proceed to establishing a lower bound of J. We consider two events
E1 =
and

n √n
2

<

n
X

γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) ≤

√ o
n

i=1

√ o
n
nX
n
T (i)
†T (i)
T (i)
†T (i)
.
E2 =
[β̂n X Yi − βn X Yi − b(β̂n X ) + b(βn X )] ≤
2
i=1

Because E1 together with E2 implies E, we have E ⊃ E1 ∩ E2 . Consequently,
h
i
† Pn
†T (i)
†T
†T
†T
†
(i)
(i)
(i)
J ≥ E Q e−λn [ i=1 γn Z Yi −βn X Yi −b(γn X )+b(βn X )] ; E1 ∩ E2 .
Notice that on the set E1 ,

†T (i)
†T (i)
†T (i)
†T (i)
i=1 γn Z Yi − βn X Yi − b(γn X ) + b(βn X )

Pn

†√

J ≥ e−λn

n

†√

Q† (E1 ∩ E2 ) ≥ e−λn

n



≤

√

n. Therefore,


Q† (E1 ) − Q† (E2c ) .

(56)

We provide an upper bound for Q† (E1 ) and a lower bound for Q† (E2c ).
Lemma 17 Let
vn = V arQ

†

n
X


γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) ,

i=1

then vn = O(n) as n → ∞. Furthermore, we have
n
 −1 h X
i
L vn 2
γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) → N (0, 1).
i=1

Here, L(·) denotes the law of random variables and N (0, 1) is the distribution of standard normal.
35

According to Lemma 17, there exists a constant ε > 0 such that
Q† (E1 ) ≥ ε.

(57)

We proceed to a lower bound for Q† (E2 ). Define the function for µ ∈ Rp
u(µ, β) = (β −

βn† )T µ

n
X
−
[b(β T X (i) ) − b(βn† X (i) )].
i=1

We further define the function
v(µ) = sup u(µ, β).
β

Lemma 18 Let
µ† =

n


X
T
b0 λ†n (γn†T Z (i) − βn† X (i) ) + (β 0 ) X (i) X (i) ,
i=1

then v(µ) is twice continuous differentiable around µ† , with v(µ† ) = 0 and ∇v(µ† ) = 0. Moreover,
we have
∇2 v(µ) =

n
hX

i−1


,
b00 β(µ)T X (i) X (i) X (i)T

i=1

where β(µ) = arg supβ u(µ, β).
According to Lemma 18 and Taylor expansion of v(µ) around µ† , we have
√ o
√ o n
n
n
1
n
† 2
2
†
⊂
kµ − µ k k∇ v(µ )k2 ≥
,
v(µ) ≥
2
2
2

(58)

where k · k2 is denotes the spectral norm of matrices. According to Lemma 18 and Assumptions
A10 and A11, k∇2 v(µ† )k2 = O(n). Therefore, (58) implies
√ o n
o
n
3
n
v(µ) ≥
⊂ kµ − µ† k ≥ εn 4 .
2
√
P
Notice that the event E2c = {v( ni=1 X (i) Yi ) ≥ 2n }, we have
Q

†

(E2c )

n
 X

3
≤Q k
X (i) Yi − µ† k ≥ εn 4 .
†

i=1

With the aid of Chebyshev’s inequality, the above display implies
†

Q

(E2c )

≤ (ε

−2 − 32

n

)E

Q†

k

n
X

X (i) Yi − µ† k2

i=1

Because E

Q†

k

Pn

i=1 X

(i) Y

i

− µ† k2 = O(n), we have Q† (E2c ) tend to zero as n goes to infinity.

Combining this result with (56) and (57), we arrive at a lower bound for J
†√
ε
J ≥ e−λn n .
2

36

The above inequality together with (55) gives a lower bound
†
†√
ε
P (LRn ≥ 1) ≥ e−neρn −λn n .
2

(59)

√
According to Assumption A9, ρe†n ≥ inf γ supλ ρen (β 0 , γ, λ) ≥ δ1 , so λ†n n = o(1)ne
ρ†n . Therefore, (59)
†

implies Pβ 0 (LRn ≥ 1) ≥ e−neρn (1+o(1)) . We complete the proof by combining the lower and upper
bound for Pβ 0 (LRn ≥ 1)

G

Proof of Lemma 15

Proof of Lemma 15. According to condition A6, it is sufficient to show that for all y ∈ Tγ † Γ,
†

E Q y > ∇γ hγ † (X) ≤ 0,

(60)

and for all y ∈ Tθ† Θ,
†

E Q y > ∇θ gθ† (X) ≤ 0.

(61)

We first prove (60). We discuss two cases: γ † ∈ int(Γ) and γ † ∈ ∂Γ, where int(Γ) denotes the
interior of Γ.
Case 1: γ † ∈ int(Γ)

Because λ† = arg inf λ Mgθ0 (θ† , γ † , λ), we have

∂
† † †
∂λ Mgθ0 (θ , γ , λ )

= 0.

According to the definition of γ † , (γ † , λ† ) is a solution of the constrained optimization problem,
max Mgθ0 (θ† , γ, λ) such that
γ,λ

∂
Mg (θ† , γ, λ) = 0,
∂λ θ0

(62)

and thus it satisfies the Karush-Kuhn-Tucker conditions. That is, there exists a constant µ such
that


∂
† † †
† † †

 ∇γ Mgθ0 (θ , γ , λ ) = µ∇γ ∂λ Mgθ0 (θ , γ , λ )
2
∂
† † †
= µ ∂∂2 λ Mgθ0 (θ† , γ † , λ† ) .
∂λ Mgθ0 (θ , γ , λ )

 ∂ M (θ† , γ † , λ† ) = 0
∂λ gθ0

The second and third equations in the above display together imply that µ = 0. We plug µ = 0 to
the first equation and obtain that
∇γ Mgθ0 (θ† , γ † , λ† ) = 0.

(63)

According to the definition of Mgθ0 (θ, γ, λ), we have
∇γ Mgθ0 (θ, γ, λ) = λEgθ0 exp{λ(log hγ (X) − log gθ (X) − b)}∇γ log hγ (X)/Mg†θ .
0

We plug this in (63), and obtain
†

E Q ∇γ log hγ † (X) = 0.
Consequently, for all y ∈ Rdh , (60) holds.
37

(64)

Case 2: γ † ∈ ∂Γ

Because ∂Γ is continuously differentiable, with possibly relabeling the coordinate

of γ, there exists a continuously differentiable function v : Rdh −1 → R and r > 0 such that
B(γ † , r) ∩ Γ = {γ ∈ B(γ † , r) : γdh ≥ v(γ1 , ..., γdh −1 )},

(65)

where B(γ † , r) = {γ : |γ − γ † | ≤ r} is a closed ball centered around γ † . Similar to Case 1, we
consider the constrained optimization problem (62) with the additional constraint
γdh ≥ v(γ1 , ..., γdh −1 ).
The definition of γ † implies that (γ † , λ† ) is a local maximum to this optimization problem. Again,
it satisfies the Karush-Kuhn-Tucker conditions for optimization problem with inequality constraint.
That is, there exists constant µ1 and µ2 such that µ1 ≥ 0 and
 ∂
†
†
∂
∂
† † †
† † †


∂γi Mgθ0 (θ , γ , λ ) = µ1 ∂γi v(γ1 , ..., γdh −1 ) + µ2 ∇γ ∂λ Mgθ0 (θ , γ , λ ) for i = 1, ..., dh − 1

 ∂ M (θ† , γ † , λ† ) = −µ + µ ∇ ∂ M (θ† , γ † , λ† )
gθ 0
1
2 γ ∂λ gθ0
∂γd
.
2
∂
∂
†
†
†

= µ2 ∂ 2 λ Mgθ0 (θ† , γ † , λ† )

∂λ Mgθ0 (θ , γ , λ )

 ∂
† † †
= 0
∂λ Mgθ0 (θ , γ , λ )
Similar to the Case 1, the third and the fourth equalities together imply that µ2 = 0. We plug this
in the first and the second equalities and obtain that
∇γ Mgθ0 (θ† , γ † , λ† ) = µ1 (∇v(γ1† , ..., γd†h −1 )T , −1)T .

(66)

We now prove that γ † satisfies (60). Notice that ∂Γ is continuously differentiable, therefore the
tangent cone is
Tγ † Γ = {y ∈ Rdh : y · (∇v(γ1† , ..., γd†h −1 )T , −1)T ≤ 0}.
Consequently, for all y ∈ Tγ † Γ, (66) implies
∇γ Mgθ0 (θ† , γ † , λ† ) · y = µ1 y · (∇v(γ1† , ..., γd†h −1 )T , −1)T ≤ 0.

(67)

Notice that
∂
Mg (θ† , γ † , 0) = Egθ0 [log hγ † (X) − log gθ† (X) − b] < 0,
∂λ θ0
and
n
o
∂2
† †
λ[log hγ (X)−log gθ (X)−b]
2
M
(θ
,
γ
,
λ)
=
E
e
[log
h
> 0.
† (X) − log gθ † (X) − b]
g
g
γ
θ
θ
0
0
∂2λ
Thus λ† > 0. We prove (60) by plugging (64) in (67) and notice that λ† > 0. Now we proceed to
the proof of (61). Again, we consider two cases: γ † ∈ int(Γ) and γ † ∈ ∂Γ.
38

Case 1: γ † ∈ int(Γ).

According to the definition of (θ† , γ † , λ† ) and (63), (θ† , γ † , λ† ) is a local

minimum of the optimization problem
inf Mgθ0 (θ, γ, λ) such that

θ,γ,λ

∂
Mg (θ, γ, λ) = 0, and ∇γ Mgθ0 (θ, γ, λ) = 0.
∂λ θ0

We prove (61) using a similar proof as that for (60) and treating θ† and (γ † , λ† ) as γ † and λ†
respectively. The details are omitted.
Case 2: γ † ∈ ∂Γ.

We will first transform the Case 2 to Case 1. Recall the definition of v and r

in (65), for γ ∈ B(γ † , r) ∩ ∂Γ, we have
γd = v(γ1 , ..., γdh −1 ).
Let Φ : Rdh → Rdh −1 be a function such that Φ(γ) = (γ1 , ..., γdh −1 )T . Let ξ = Φ(γ), and ξ † = Φ(γ † ),
then for γ ∈ B(γ † , r) ∩ ∂Γ, γ = (ξ T , v(ξ))T . We abuse the notation a little and write
fg (θ, ξ, λ) = Mg (θ, γ, λ),
M
θ0
θ0
where γ = (ξ T , v(ξ))T . We further let Ξ = Φ(B(γ † , r) ∩ Γ). We compute the partial derivatives of
fg (θ, ξ, λ) at (θ† , ξ † , λ† ),
M
θ0
∂ f
∂
fg (θ† , ξ † , λ† ) = dγ (ξ † )T ∇γ Mg (θ† , γ † , λ† ),
Mgθ0 (θ† , ξ † , λ† ) =
Mgθ0 (θ† , γ † , λ† ) = 0, and ∇ξ M
θ0
θ0
∂λ
∂λ
dξ
(68)
where

dγ
dξ

is a dh × (dh − 1) Jacobian matrix


dγ
Idh −1
=
,
∇v(ξ † )T
dξ

and Idh −1 is the (dh − 1) × (dh − 1) identity matrix. We plug (66) and the above expression in (68),
and obtain
fg (θ† , ξ † , λ† ) = µ1 (∇v(ξ † ) − ∇v(ξ † ))T = 0.
∇ξ M
θ0
Therefore, (θ† , ξ † , λ† ) is a local minimum under the constrained optimization problem
fg (θ, ξ, λ) such that ∇ξ M
fg (θ, ξ, λ) = 0 and ∂ M
fg (θ, ξ, λ) = 0.
inf M
θ0
θ0
∂λ θ0

θ,ξ,λ

We complete the proof by replacing γ and Γ by ξ and Ξ respectively in the proof for Case 1.

39

H

Proof of Lemma 16

Define the function
w(s1 , s2 ) = sup[γ T s2 −
γ

n

n

i=1

i=1

1X
1X
b(γ T Z (i) )] − [βn†T s1 −
b(βn†T X (i) )].
n
n

Then An = {(s1 , s2 ) : w(s1 , s2 ) ≥ 0}. Let
n
n


1 X 0  † †T (i) †T (i)
1 X 0  † †T (i) †T (i)
†
†
0 T (i)
(i)
0 T (i)
s1 =
b λn (γn Z −βn X )+(β ) X
X and s2 =
b λn (γn Z −βn X )+(β ) X
Z (i) .
n
n
i=1

i=1

With similar proof as that for (60), we have that

γn†

satisfies first order conditions

n

∇γ ρen (βn† , γn† , λ†n ) = λ†n


i
1 X h 0 †T (i) (i) 0  † †T (i)
T
b (γn Z )Z −b λn (γn Z −βn†T X (i) )+(β 0 ) X (i) Z (i) = 0q .
n
i=1

(69)
(69) is also the first order condition for the optimization problem
sup[γ T s†2 −
γ

n

n

i=1

i=1

1X
1X
b(γ T Z (i) )] − [βn†T s†1 −
b(βn†T X (i) )].
n
n

Notice that this optimization is concave in γ. Therefore, γn† is a solution of the above optimization
problem, and
n

w(s†1 , s†2 ) = sup[γ T s†2 −
γ

n

1X
1X
b(γ T Z (i) )] − [βn†T s†1 −
b(βn†T X (i) )]
n
n
i=1

i=1

n
n
1X
1X
= γn†T s†2 −
b(γn†T Z (i) ) − [βn†T s†1 −
b(βn†T X (i) )]. (70)
n
n
i=1

Also notice that

λ†n

=

arg supλ ρen (βn† , γn† , λ).

i=1

Therefore, it satisfies the first order condition

n

0=



∂
1X
T
ρen (βn† , γn† , λ†n ) =
b(γn† Z (i) )−b(βn† X (i) )−b0 λ†n (γn†T Z (i) −βn†T X (i) )+(β 0 ) X (i) [γn†T Z (i) −βn†T X (i) ].
∂λ
n
i=1

(71)
(70) and (71) together gives
w(s†1 , s†2 ) = 0.
Therefore, (s†1 , s†2 ) is a boundary point of An . Furthermore, we have
∇s1 w(s†1 , s†2 ) = 0p and ∇s2 w(s†1 , s†2 ) = βn† .
Consequently, the normal vector of An at (s†1 , s†2 ) is −(∇s1 w(s†1 , s†2 ), ∇s2 w(s†1 , s†2 )) = −(0p , γn† ).
Because An is a convex set, for all (s1 , s2 ) ∈ An , we have
−(s1 − s†1 , s2 − s†2 ) · (0p , γn† ) ≤ 0.
We complete the proof by combining the above display and that w(s†1 , s†2 ) = 0.
40

I

Proof of Lemma 17

Because Yi s are independent, we have
vn =

n
n


X
X
†
T
(γn†T Z (i) −βn†T X (i) )2 V arQ (Yi ) =
(γn†T Z (i) −βn†T X (i) )2 b00 λ†n (γn†T Z (i) −βn†T X (i) )+(β 0 ) X (i) .
i=1

i=1

(72)
According to Assumption A10 and A11, we have vn = O(n). We define a triangular array for
n, i ≥ 1
h
i
−1
Un,i = vn 2 γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) .
It is sufficient to show that Un,i satisfies conditions for the Lyapuvov central limit theorem [Billingsley, 1995, page 362] for triangular arrays. That is,
lim

n→∞

n
X

†

E Q |Un,i |3 = 0.

(73)

i=1

According to Assumption A11, b(·) is four times continuously differentiable. This guarantees that
n
X

h
i3
†
E Q γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) = O(n).

i=1

Now we show that vn−1 = O(n−1 ). According to Assumptions A10 and A11 and (25), we have
|e
ρ†n | ≤ κ

n

n

i=1

i=1

1
1 X
1 X †T (i)
2
|γn Z − βn†T X (i) | ≤ κ
(γn†T Z (i) − βn†T X (i) )2 .
n
n

On the other hand, Assumption A9 implies that
ρe†n ≥ inf sup ρen (β 0 , γ, λ) ≥ δ1 .
γ

Therefore,

λ

n

1 X †T (i)
(γn Z − βn†T X (i) )2 ≥ δ12 κ−2 .
n
i=1

According to Assumption A11 and (72), vn ≥ ε

†T (i)
†T (i) 2
i=1 (γn Z − βn X ) .

Pn

Together with the above

display, we have vn−1 = O(n−1 ). Therefore,
n
X
i=1

E

Q†

3

|Un,i | = lim

n→∞

−3
vn 2

n
X

h
i3
1
†
E Q γn†T Z (i) Yi − βn†T X (i) Yi − b(γn†T X (i) ) + b(βn†T X (i) ) = O(n− 2 ),

i=1

and (73) is proved.

41

J

Proof of Lemma 18

Let β(µ) = arg supβ u(µ, β), then β(µ) satisfies
∂
u(µ, β(µ)) = 0.
∂β

(74)

We first show that β(µ† ) = βn† . Similar to (61), we have
n

∇β ρen (βn† , γn† , λ†n ) = λ†n



i
1 Xh
T
− b0 (βn†T X (i) )X (i) + b0 λ†n (γn†T Z (i) − βn†T X (i) ) + (β 0 ) X (i) X (i) = 0.
n
i=1

Therefore, we have

n

X
∂
u(µ† , βn† ) = µ† −
b0 (βn†T X (i) )X (i) = 0.
∂β
i=1

Notice that supβ u(µ† , β) is a strictly concave optimization problem. Therefore, βn† is its unique
solution β(µ). Now we compute ∇v(µ).
∇v(µ) = ∇u(µ, β(µ)) =

d
∂
∂
u(µ, β(µ)) +
β(µ) u(µ, β(µ)).
∂µ
dµ
∂β

The above display together with (74) gives
∇v(µ) =

∂
u(µ, β(µ)) = β(µ) − βn† .
∂µ

(75)

Because β(µ† ) = βn† , we have that v(µ) is continuously differentiable and v(µ† ) = 0 and ∇v(µ† ) = 0.
We proceed to the second derivatives of v(µ). Applying implicit function theorem to (74), we have
∇β(µ) = −

2
∂2
∂2
−1 ∂
u(µ,
β)
u(µ,
β(µ))
=
−
u(µ, β)−1 .
(∂β)2
∂µ∂β
(∂β)2

According to (75) and the above equation, we complete the proof.

References
R.J. Adler, J.H. Blanchet, and J. Liu. Efficient Monte Carlo for large excursions of Gaussian
random fields. Ann. Appl. Probab., 22(3):1167–1214, 2012.
Miguel A Arcones. Large deviations for m-estimators. Ann. Inst. Statist. Math., 58(1):21–52, 2006.
Raghu Raj Bahadur. Stochastic comparison of tests. Ann. Math. Statist., 31(2):276–295, 1960.
Raghu Raj Bahadur. Rates of convergence of estimates and test statistics. Ann. Math. Statist., 38
(2):303–324, 1967.
42

Amy Berrington de González and David R Cox. Interpretation of interaction: A review. Ann.
Appl. Stat., 1(2):371–385, 2007.
P Billingsley. Probability and measure. Wiley Series in Probability and Mathematical Statistics,
1995.
Basilio Braganca Pereira. Separate families of hypotheses. Encyclopedia of Biostatistics, 2005.
Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum
of observations. Ann. Math. Statist., 23(4):493–507, 1952.
D. R. Cox. Tests of separate families of hypotheses. In Proc. 4th Berkeley Sympos. Math. Statist.
and Prob., Vol. I, pages 105–123. Univ. California Press, Berkeley, Calif., 1961.
D. R. Cox. Further results on tests of separate families of hypotheses. J. R. Stat. Soc. Ser. B., 24:
406–424, 1962. ISSN 0035-9246.
D. R. Cox. A return to an old paper: ‘Tests of separate families of hypotheses’. J. R. Stat.
Soc. Ser. B., 75(2):207–215, 2013. ISSN 1369-7412. doi: 10.1111/rssb.12003. URL http:
//dx.doi.org/10.1111/rssb.12003.
Russell Davidson and James G MacKinnon. Several tests for model specification in the presence
of alternative hypotheses. Econometrica, 49(3):781–793, 1981.
Jason P. Fine. Comparing nonnested Cox models. Biometrika, 89(3):635–647, 2002.
Peter J. Huber. The behavior of maximum likelihood estimates under nonstandard conditions. In
Proc. 5th Berkeley Sympos. Math. Statist. and Prob., Vol. I: Statistics, pages 221–233. Univ.
California Press, Berkeley, Calif., 1967.
Wilbert C. M. Kallenberg. Intermediate efficiency, theory and examples. Ann. Statist., 11(1):
170–182, 1983.
P. McCullagh and J. A. Nelder. Generalized Linear Models. Chapman & Hall, London, 1989.
ISBN 0-412-31760-5. doi: 10.1007/978-1-4899-3242-6. URL http://dx.doi.org/10.1007/
978-1-4899-3242-6.
M Hashem Pesaran. On the general problem of model selection. Rev. Econom. Stud., 41(2):153–171,
1974.

43

M Hashem Pesaran and Angus S Deaton. Testing non-nested nonlinear regression models. Econometrica, 46(3):677–94, 1978.
Mohammad Hashem Pesaran. Asymptotic power comparisons of tests of separate parametric families by bahadur’s approach. Biometrika, 71(2):245–252, 1984.
Andrew L Rukhin. Bahadur efficiency of tests of separate hypotheses and adaptive test statistics.
J. Amer. Statist. Assoc., 88(421):161–165, 1993.
Gideon Schwarz. Estimating the dimension of a model. Ann. Statist., 6(2):461–464, 1978.
Robert J. Serfling. Approximation Theorems of Mathematical Statistics. Wiley, New York, 1980.
ISBN 0-471-02403-1.
Quang H Vuong. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2):307–33, 1989.
Halbert White. Maximum likelihood estimation of misspecified models. Econometrica, 50(1):1–25,
1982a. ISSN 0012-9682. doi: 10.2307/1912526. URL http://dx.doi.org/10.2307/1912526.
Halbert White. Regularity conditions for Cox’s test of nonnested hypothesis. J. Econometrics,
19(2-3):301–318, 1982b. ISSN 0304-4076. doi: 10.1016/0304-4076(82)90007-0. URL http:
//dx.doi.org/10.1016/0304-4076(82)90007-0.

44

