On Asymptotics Related to Classical Inference in Stochastic
Differential Equations with Random Effects
Trisha Maitra and Sourabh Bhattacharya∗

arXiv:1407.3968v3 [] 11 May 2016

Abstract
Delattre et al. (2013) considered n independent stochastic differential equations (SDE’s), where
in each case the drift term is associated with a random effect, the distribution of which depends upon
unknown parameters. Assuming the independent and identical (iid) situation the authors provide
independent proofs of weak consistency and asymptotic normality of the maximum likelihood estimators (M LE’s) of the hyper-parameters of their random effects parameters.
In this article, as an alternative route to proving consistency and asymptotic normality in the
SDE set-up involving random effects, we verify the regularity conditions required by existing relevant theorems. In particular, this approach allowed us to prove strong consistency under weaker
assumption. But much more importantly, we further consider the independent, but non-identical
set-up associated with the random effects based SDE framework, and prove asymptotic results associated with the M LE’s.
Keywords: Asymptotic normality; Burkholder-Davis-Gundy inequality; Itô isometry; Maximum
likelihood estimator; Random effects; Stochastic differential equations.

1 Introduction
Delattre et al. (2013) study mixed-effects stochastic differential equations (SDE’s) of the following
form:
dXi (t) = b(Xi (t), φi )dt + σ(Xi (t))dWi (t), with Xi (0) = xi , i = 1, . . . , n.
(1.1)
Here, for i = 1, . . . , n, the stochastic process Xi (t) is assumed to be continuously observed on the
time interval [0, Ti ] with Ti > 0 known, and {xi ; i = 1, . . . , n} are the known initial values of the
i-th process. The processes {Wi (·); i = 1, . . . , n} are independent standard Brownian motions, and
{φi ; i = 1, . . . , n} are independently and identically distributed (iid) random variables with common
distribution g(ϕ, θ)dν(ϕ) (for all θ, g(ϕ, θ) is a density with respect to a dominating measure on Rd ,
where R is the real line and d is the dimension), which are independent of the Brownian motions. Here
θ ∈ Ω ⊂ Rp (p ≥ 2d) is an unknown parameter to be estimated. The functions b : R × Rd 7→ R and
σ : R 7→ R are the drift function and the diffusion coefficient, respectively, both assumed to be known.
Delattre et al. (2013) impose regularity conditions that ensure existence of solutions of (1.1). We adopt
their assumptions, which are as follows.
(H1)

(i) The function (x, ϕ) 7→ b(x, ϕ) is C 1 (differentiable with continuous first derivative) on
R × Rd , and such that there exists K > 0 so that
b2 (x, ϕ) ≤ K(1 + x2 + |ϕ|2 ),
for all (x, ϕ) ∈ R × Rd .

(ii) The function σ(·) is C 1 on R and
σ 2 (x) ≤ K(1 + x2 ),
for all x ∈ R.
(H2) Let Xiϕ be associated with the SDE of the form (1.1) with drift function b(x, ϕ). Also letting
i

Qxϕ ,Ti denote the joint distribution of {Xiϕ (t); t ∈ [0, Ti ]}, it is assumed that for i = 1, . . . , n,

∗
Trisha Maitra is a PhD student and Sourabh Bhattacharya is an Associate Professor in Interdisciplinary Statistical Research
Unit, Indian Statistical Institute, 203, B. T. Road, Kolkata 700108. Corresponding e-mail: sourabh@isical.ac.in.

1

and for all ϕ, ϕ′ , the following holds:

Z Ti 2
b (Xiϕ (t), ϕ′ )
xi ,Ti
Qϕ
dt < ∞ = 1.
σ 2 (Xiϕ (t))
0
(H3) For f =

∂b
∂ϕj ,

j = 1, . . . , d, there exist c > 0 and some γ ≥ 0 such that
|f (x, ϕ)|
≤ c (1 + |x|γ ) .
2
ϕ∈Rd σ (x)
sup

Statistically, the i-th process Xi (·) can be thought of as modelling the i-th individual and the corresponding random variable φi denotes the random effect of individual i. For statistical inference, we
follow Delattre et al. (2013) who consider the special case where b(x, φi ) = φi b(x). We assume
(H1′ )

(i) b(·) and σ(x) are C 1 on R satisfying b2 (x) ≤ K(1 + x2 ) and σ 2 (x) ≤ K(1 + x2 ) for all
x ∈ R, for some K > 0.

(ii) Almost surely for each i ≥ 1,

Z

Ti
0

b2 (Xi (s))
ds < ∞.
σ 2 (Xi (s))

Under this assumption, (H3) is no longer required; see Delattre et al. (2013). Moreover, Proposition 1
of Delattre et al. (2013) holds; in particular, if for k ≥ 1, E|φi |2k < ∞, then for all T > 0,
sup E [Xi (t)]2k < ∞.

(1.2)

t∈[0,T ]

As in Delattre et al. (2013) we assume that φi are normally distributed. Hence, (1.2) is satisfied in our
case. Delattre et al. (2013) show that the likelihood, depending upon θ, admits a relatively simple form
composed of the following sufficient statistics:
Ui =

Z

0

Ti

b(Xi (s))
dXi (s),
σ 2 (Xi (s))

Vi =

The exact likelihood is given by
L(θ) =

n
Y

Z

Ti
0

b2 (Xi (s))
ds,
σ 2 (Xi (s))

i = 1, . . . , n.

λi (Xi , θ),

(1.3)

(1.4)

i=1

where


ϕ2
g(ϕ, θ) exp ϕUi −
λi (Xi , θ) =
Vi dν(ϕ).
(1.5)
2
R

Assuming that g(ϕ, θ)dν(ϕ) ≡ N µ, ω 2 , Delattre et al. (2013) obtain the following form of λi (Xi , θ):
"
 2
 #

1
Ui
Vi
Ui 2
λi (Xi , θ) =
exp −
exp
,
(1.6)
µ−
2
1/2
2 (1 + ω Vi )
Vi
2Vi
(1 + ω 2 Vi )


Z

where θ = (µ, ω 2 ) ∈ Ω ⊂ R × R+ . As in Delattre et al. (2013), here we assume that
(H2′ ) Ω is compact.
Delattre et al. (2013) consider xi = x and Ti = T for i = 1, . . . , n, so that the set-up boils down
to the iid situation, and investigate asymptotic properties of the M LE of θ, providing proofs of consistency and asymptotic normality independently, without invoking the general results already existing
2

in the literature. In this article, as an alternative, we prove asymptotic properties of the M LE in this
SDE set-up by verifying the regularity conditions of relevant theorems already existing in the literature. Our approach allowed us to prove strong consistency of M LE, rather than weak consistency
proved by Delattre et al. (2013). Also, importantly, our approach does not require assumption (H4) of
Delattre et al. (2013) which required (U1 , V1 ) to have density with respect to the Lebsegue measure on
R × R+ , which must be jointly continuous and positive on an open ball of R × R+ ).
Far more importantly, we consider the independent but non-identical case (we refer to the latter as
non-iid), and prove consistency and asymptotic normality of the M LE in this set-up. In what follows,
in Section 2 we investigate asymptotic properties of M LE in the iid context. In Section 3 we investigate
classical asymptotics in the non-iid set-up. We summarize our work and provide concluding remarks in
Section 4.
a.s.
P
L
Notationally, “ → ”, “→” and “→” denote convergence “almost surely”, “in probability” and “in
distribution”, respectively.

2 Consistency and asymptotic normality of MLE in the iid set-up
2.1 Strong consistency of MLE
Consistency of the M LE under the iid set-up can be verified by validating the regularity conditions of
the following theorem (Theorems 7.49 and 7.54 of Schervish (1995)); for our purpose we present the
version for compact Ω.
Theorem 1 (Schervish (1995)) Let {Xn }∞
n=1
 be conditionally iid given θ with density f1 (x|θ) with
respect to a measure ν on a space X 1 , B 1 . Fix θ0 ∈ Ω, and define, for each M ⊆ Ω and x ∈ X 1 ,
Z(M, x) = inf log
ψ∈M

f1 (x|θ0 )
.
f1 (x|ψ)

Assume that for each θ 6= θ0 , there is an open set Nθ such that θ ∈ Nθ and that Eθ0 Z(Nθ , Xi ) > −∞.
Also assume that f1 (x|·) is continuous at θ for every θ, a.s. [Pθ0 ]. Then, if θ̂n is the M LE of θ
corresponding to n observations, it holds that lim θ̂n = θ0 , a.s. [Pθ0 ].
n→∞

2.1.1

Verification of strong consistency of M LE in our SDE set-up

To verify the conditions of Theorem 1 in our case, we note that for any x, f1 (x|θ) = λ1 (x, θ) = λ(x, θ)
given by (1.6), which is clearly continuous in θ. Also, it follows from the proof of Proposition 7 of
Delattre et al. (2013) that for every θ 6= θ0 ,


1 + ω 2 V1
(ω02 − ω 2 )U12
1
1
f1 (x|θ0 )
= log
+
log
f1 (x|θ)
2
2 (1 + ω 2 V1 )(1 + ω02 V1 )
1 + ω02 V1


µ0 U1
µ20 V1
µU1
µ2 V1
−
−
−
+
2(1 + ω 2 V1 ) 1 + ω 2 V1
2(1 + ω02 V1 ) 1 + ω02 V1



 

2 
U1
1 2
1
ω2
|ω 2 − ω02 |
ω02
2
− |ω0 − ω |
log 1 + 2 +
≥−
1+ 2
2
ω2
2
ω
ω0
1 + ω02 V1


2
2
2
U1
µ 0 V1
|ω − ω |
µ0 U1
− |µ|
−
.
(2.1)
−
1+ 0 2
2
2
ω
1 + ω 0 V1
2(1 + ω0 V1 )
1 + ω02 V1
2



U1
, Eθ0
Taking Nθ = µ, µ × ω2 , ω 2 , and noting that Eθ0 1+ω
2V
0 1

U1
1+ω02 V1

and Eθ0

finite due to Lemma 1 of Delattre et al. (2013), it follows that Eθ0 Z(Nθ , Xi ) > −∞.
[Pθ0 ]. We summarize the result in the form of the following theorem:

3

µ20 V1
are
2(1+ω02 V1 )
a.s.
Hence, θ̂n → θ0





Theorem 2 Assume the iid setup and conditions (H1′ ) and (H2′ ). Then the M LE is strongly consistent
a.s.
in the sense that θ̂n → θ0 [Pθ0 ].

2.2 Asymptotic normality of MLE
To verify asymptotic normality of M LE we invoke the following theorem provided in Schervish (1995)
(Theorem 7.63):
Theorem 3 (Schervish (1995)) Let Ω be a subset of Rd , and let {Xn }∞
n=1 be conditionally iid given
P

θ each with density f1 (·|θ). Let θ̂n be an M LE. Assume that θ̂n → θ under Pθ for all θ. Assume
that f1 (x|θ) has continuous second partial derivatives with respect to θ and that differentiation can be
passed under the integral sign. Assume that there exists Hr (x, θ) such that, for each θ0 ∈ int(Ω) and
each k, j,
sup
kθ−θ0 k≤r

∂2
∂2
log fX1 |Θ (x|θ0 ) −
log fX1 |Θ (x|θ) ≤ Hr (x, θ0 ),
∂θk ∂θj
∂θk ∂θj

(2.2)

with
lim Eθ0 Hr (X, θ0 ) = 0.

r→0

Assume that the Fisher information matrix I(θ) is finite and non-singular. Then, under Pθ0 ,


√ 
L
n θ̂n − θ0 → N 0, I −1 (θ0 ) .
2.2.1

(2.3)

(2.4)

Verification of the above regularity conditions for asymptotic normality in our SDE set-up
P

In Section 2.1.1 we proved almost sure consistency of the M LE θ̂n in the SDE set-up. Hence, θ̂n → θ
under Pθ for all θ. In the proof of Proposition 5, Delattre et al. (2013) show that differentiation can be
Vi
i −µVi
passed under the integral sign. Letting γi (θ) = U
and Ii = 1+ω
2 V , note that (see the proof of
1+ω 2 Vi
i
Proposition 6 of Delattre et al. (2013))
∂2
∂2
2
log
f
(x|θ)
=
−I
(ω
),
log f1 (x|θ) = −γ1 (θ)I1 (ω 2 );
1
1
∂µ2
∂µ∂ω 2

∂2
1
log f1 (x|θ) = − 2γ12 (θ)I1 (ω 2 ) − I12 (ω 2 ) .
2
2
∂ω ∂ω
2

(2.5)
(2.6)

2

It follows from (2.5) and (2.6) that in our case ∂θ∂k ∂θj log f1 (x|θ) is differentiable in θ = (µ, ω 2 ), and the
derivative has finite expectation; see the proof of Proposition 8 of Delattre et al. (2013)). Hence, (2.2)
and (2.3) clearly hold. Following Delattre et al. (2013) we assume:
(H3′ ) The true value θ0 ∈ int (Ω).
That the information matrix I(θ) is finite and is the covariance matrix of the vector γ1 (θ), 21 γ12 (θ) − I1 ω 2
(hence, nonnegative-definite), are shown in Delattre et al. (2013). We additionally assume, as Delattre et al.
(2013):
(H4′ ) The information matrix I(θ0 ) is invertible.
Hence, asymptotic normality of the M LE, of the form (2.4), holds in our case. Formally,
Theorem 4 Assume the iid setup and conditions (H1′ ) – (H4′ ). Then the M LE is asymptotically normally distributed as (2.4).

4



3 Consistency and asymptotic normality of MLE in the non-iid set-up
We now consider the case where the processes Xi (·); i = 1, . . . , n, are independently, but not identically distributed. This happens when we no longer enforce the restrictions Ti = T and xi = x for
i = 1, . . . , n. However, we do assume that the sequences {T1 , T2 , . . .} and {x1 , x2 , . . . , } are sequences
entirely contained in compact sets T and X, respectively. Due to compactness, there exist convergent
subsequences with limits in T and X. Abusing notation, we continue to denote the convergent subsequences as {T1 , T2 , . . .} and {x1 , x2 , . . .}. Let the limts be T ∞ ∈ T and x∞ ∈ X.
Now, since the distributions of the processes Xi (·) are uniquely defined on the space of real, continuous functions C ([0, Ti ] 7→ R) = {f : [0, Ti ] 7→ R such that f is continuous}, given any t ∈ [0, Ti ], f (t)
is clearly a continuous function of the initial value f (0) = x. To emphasize dependence on x, we denote
the function as f (t, x). In fact, for any ǫ > 0, there exists δǫ > 0 such that whenever |x1 − x2 | < δǫ ,
|f (t, x1 ) − f (t, x2 )| < ǫ for all t ∈ [0, Ti ].
Henceforth, we denote the process associated with the initial value x and time point t as X(t, x),
and by φ(x) the random effect parameter associated with the initial value x such that φ(xi ) = φi . We
assume that
(H5′ ) φ(x) is a real-valued, continuous function of x, and that for k ≥ 1,
sup E [φ(x)]2k < ∞.

(3.1)

x∈X

For x ∈ X and T ∈ T, let
U (x, T ) =
V (x, T ) =

Z

Z

T
0
T
0

b(X(s, x))
dX(s, x);
σ 2 (X(s, x))

(3.2)

b2 (X(s, x))
ds.
σ 2 (X(s, x))

(3.3)

Clearly, U (xi , Ti ) = Ui and V (xi , Ti ) = Vi , where Ui and Vi are given by (1.3). In this non-iid set-up
we assume that
(H6′ )

b2 (x)
< K(1 + xτ ), for some τ ≥ 1.
σ 2 (x)

(3.4)

This assumption ensures that moments of all orders of V (x, T ) are finite. Then the moments of uniformly integrable continuous functions of U (x, T ), V (x, T ) and θ are continuous in x, T and θ. The
result is formalized as Theorem 5, the proof of which is presented in the Appendix.
Theorem 5 Assume (H5′ ) and (H6′ ). Let h(u, v, θ) be any continuous function of u, v and θ, such that
∞
∞
for any sequences {xm }∞
m=1 , {Tm }m=1 and {θm }m=1 , converging to x̃, T̃ and θ̃, respectively, for any
x̃ ∈ X, T̃ ∈ T and θ̃ ∈ Ω, the sequence {h (U (xm , Tm ), V (xm , Tm ), θm )}∞
m=1 is uniformly integrable.
Then, as m → ∞,
h 
i
E [h (U (xm , Tm ), V (xm , Tm ), θm )] → E h U (x̃, T̃ ), V (x̃, T̃ ), θ̃ .
(3.5)
Corollary 6 As in Delattre et al. (2013), consider the function


u
,
h(u, v) = exp ψ
1 + ξv

(3.6)

∞
where ψ ∈ R and ξ ∈ R+ . Then, for any sequences {xm }∞
m=1 and {Tm }m=1 converging to x̃ and T̃ ,

5

for any x̃ ∈ X and T̃ ∈ T, and for k ≥ 1,

as m → ∞.

h 
ik
E [h (U (xm , Tm ), V (xm , Tm ))]k → E h U (x̃, T̃ ), V (x̃, T̃ )
,

(3.7)

The proof of the above corollary only entails proving uniform integrability of {h (U (xm , Tm ), V (xm , Tm ))}∞
m=1 ,
which simply follows from the proof of Lemma 1 of Delattre et al. (2013).
Note that in our case, the Kullback-Leibler distance and Fisher’s information are expectations of
functions of the form h(u, v, θ), continuous in u, v and θ. Assumption (H6′ ), the upper bounds provided
in Delattre et al. (2013), Corollary 6, and compactness of Ω, can be used to easily verify uniform integrability of the relevant sequences. It follows that in our situation the Kullback-Leibler distance, which
we now denote by Kx,T (θ0 , θ) (or Kx,T (θ, θ0 )) to emphasize dependence on x, T and θ are continuous
in θ, x and T . Similarly, the elements of the Fisher’s information matrix Ix,T (θ) are continuous in θ, x
and T . For x = xk and T = Tk , we denote the Kullback-Leibler distance and the Fisher’s information
as Kk (θ0 , θ) (Kk (θ, θ0 )) and Ik (θ), respectively.
Continuity of Kx,T (θ0 , θ) (or Kx,T (θ, θ0 )) and Ix,T (θ0 ) with respect to x and T ensures that as
xk → x∞ and Tk → T ∞ , Kxk ,Tk (θ0 , θ) → Kx∞ ,T ∞ (θ0 , θ) = K(θ0 , θ), say. Similarly, Kxk ,Tk (θ, θ0 ) →
K(θ, θ0 ) and Ixk ,Tk (θ) → Ix∞ ,T ∞ (θ) = I(θ), say. Since X ∞ and T ∞ are contained in the respective
compact sets, the limits K(θ0 , θ), K(θ, θ0 ) and I(θ) are well-defined Kullback-Leibler divergences and
Fisher’s information, respectively. From the above limits, it follow that for any θ ∈ Ω,
Pn
k=1 Kk (θ0 , θ)
= K(θ0 , θ);
(3.8)
lim
n→∞
Pn n
k=1 Kk (θ, θ0 )
= K(θ, θ0 );
(3.9)
lim
n→∞
Pnn
k=1 Ik (θ)
= I(θ).
(3.10)
lim
n→∞
n
We investigate consistency and asymptotic normality of M LE in our case using the results of Hoadley
(1971). The limit results (3.8), (3.9) and (3.10) will play important roles in our proceedings.

3.1 Consistency and asymptotic normality of MLE in the non-iid set-up
Following Hoadley (1971) we define the following:
Ri (θ) = log
=0

fi (Xi |θ)
fi (Xi |θ0 )

if fi (Xi |θ0 ) > 0
otherwise.

Ri (θ, ρ) = sup {Ri (ξ) : kξ − θk ≤ ρ}
Vi (r) = sup {Ri (θ) : kθk > r} .

(3.11)

(3.12)
(3.13)

Following Hoadley (1971) we denote by ri (θ), ri (θ, ρ) and vi (r) to
Pbe expectations of Ri (θ), Ri (θ, ρ)
and Vi (r) under θ0 ; for any sequence {ai ; i = 1, 2, . . .} we denote ni=1 ai /n by ān .
P

θ0 :

Hoadley (1971) proved that if the following regularity conditions are satisfied, then the MLE θ̂n →

(1) Ω is a closed subset of Rd .
(2) fi (Xi |θ) is an upper semicontinuous function of θ, uniformly in i, a.s. [Pθ0 ].
6

(3) There exist ρ∗ = ρ∗ (θ) > 0, r > 0 and 0 < K ∗ < ∞ for which
(i) Eθ0 [Ri (θ, ρ)]2 ≤ K ∗ ,

(ii) Eθ0 [Vi (r)]2 ≤ K ∗ .
(4)

(i) lim r̄n (θ) < 0,
n→∞

0 ≤ ρ ≤ ρ∗ ;

θ 6= θ0 ;

(ii) lim v̄n (r) < 0.
n→∞

(5) Ri (θ, ρ) and Vi (r) are measurable functions of Xi .
Actually, conditions (3) and (4) can be weakened but these are more easily applicable (see Hoadley
(1971) for details).
3.1.1

Verification of the regularity conditions

Since Ω is compact in our case, the first regularity condition clearly holds.
For the second regularity condition, note that given Xi , fi (Xi |θ) is continuous, in fact, uniformly
continuous in θ in our case, since Ω is compact. Hence, for any given ǫ > 0, there exists δi (ǫ) > 0,
independent of θ, such that kθ1 − θ2 k < δi (ǫ) implies |f (Xi |θ1 ) − f (Xi |θ2 )| < ǫ. Now consider a
strictly positive function δx,T (ǫ), continuous in x ∈ X and T ∈ T, such that δxi ,Ti (ǫ) = δi (ǫ). Let
δ(ǫ) =
inf δx,T (ǫ). Since X and T are compact, it follows that δ(ǫ) > 0. Now it holds that
x∈X,T ∈T

kθ1 − θ2 k < δ(ǫ) implies |f (Xi |θ1 ) − f (Xi |θ2 )| < ǫ, for all i. Hence, the second regularity condition
is satisfied.
Let us now focus attention on condition (3)(i). It follows from (2.1) that



 

2 
Ui
1 2
ω2
|ω 2 − ω02 |
ω02
1
2
+ |ω0 − ω |
log 1 + 2 +
1+ 2
Ri (θ) ≤
2
ω2
2
ω
ω0
1 + ω02 Vi


 
2
2
2
µ 0 Ui
|ω − ω |
Ui
µ 0 Vi
−
1+ 0 2
.
(3.14)
+ |µ|
+
2
2
ω
1 + ω 0 Vi
2(1 + ω0 Vi ) 1 + ω02 Vi
Let us denote {ξ ∈ R × R+ : kξ − θk ≤ ρ} by S(ρ, θ). Here 0 < ρ < ρ∗ (θ), and ρ∗ (θ) is so small that
S(ρ, θ) ⊂ Ω for all ρ ∈ (0, ρ∗ (θ)). It then follows from (3.14) that


 
|ω 2 − ω02 |
1
ω2
sup Ri (ξ) ≤
sup
log 1 + 2 +
ω2
ω0
ξ∈S(ρ,θ)
(µ,ω 2 )∈S(ρ,θ) 2
2




1 2
Ui
ω02
2
×
sup
+
ω
−
ω
1
+
0
ω2
1 + ω02 Vi
(µ,ω 2 )∈S(ρ,θ) 2
 

|ω02 − ω 2 |
Ui
×
sup
|µ| 1 +
+
ω2
1 + ω02 Vi
(µ,ω 2 )∈S(ρ,θ)
+

µ 0 Ui
µ20 Vi
.
+
2
2(1 + ω0 Vi )
1 + ω02 Vi

(3.15)

The supremums in (3.15) are finite due to compactness of S(ρ, θ). Since under Pθ0 , Ui /(1 + ω02 Vi )
Vi
admits moments of all orders and 0 < Ii (ω02 ) = 1+ω
< ω12 (see Delattre et al. (2013)), it follows
2
0 Vi
0
from (3.15) that
Eθ0 [Ri (θ, ρ)]2 ≤ Ki (θ),
(3.16)

where Ki (θ) = K(xi , Ti , θ), with K(x, T, θ) being a continuous function of (x, T, θ), continuity being
a consequence of Theorem 5. Since because of compactness of X, T and Ω,
Ki (θ) ≤

sup
x∈X,T ∈T,θ∈Ω

7

K(x, T, θ) < ∞,

regularity condition (3)(i) follows.
To verify condition (3)(ii), first note that we can choose r > 0 such that kθ0 k < r and {θ ∈ Ω :
kθk > r} =
6 ∅. It then follows that
sup
Ri (θ) ≤ sup Ri (θ) for every i ≥ 1. The right hand side
θ∈Ω

{θ∈Ω:kθk>r}

is bounded by the same expression as the right hand side of (3.15), with only S(θ, ρ) replaced with Ω.
The rest of the verification follows in the same way as verification of (3)(i).
To verify condition (4)(i) note that by (3.8)
Pn
i=1 Ki (θ0 , θ)
= −K(θ0 , θ) < 0 for θ 6= θ0 .
(3.17)
lim r̄n = − lim
n→∞
n→∞
n
In other words, (4)(i) is satisfied.
To verify (4)(ii) we first show that lim v̄n exists for suitably chosen r > 0. Then we prove
n→∞

that the limit is negative. To see that the limit exists, we first write Rx,T (θ) = −Kx,T (θ0 , θ). Clearly,
Ri (θ) = Rxi ,Ti (θ). Using the arguments provided in the course of verification of (3)(ii), and the moment
existence result of Delattre et al. (2013), yield
sup Eθ0
i≥1

"

sup

#2

Ri (θ)

{θ∈Ω:kθk>r}

≤ sup Eθ0
i≥1



2
sup Ri (θ) ≤
θ∈Ω

sup
x∈X,T ∈T

K1 (x, T ),

(3.18)

where K1 (x, T ) is a continuous function of x and T . That K1 (x, T ) is continuous in x and T follows
from Theorem 5; the required
 uniform integrability follows due to finiteness of the moments of the
random variable U (x, T )/ 1 + ω 2 V (x, T ) , for every x ∈ X and T ∈ T, and compactness of X
and T. Now, because of compactness(of X and T it also )
follows that the right hand side of (3.18) is
∞

finite, proving uniform integrability of
"
#
vx,T = Eθ0

sup

{θ∈Ω:kθk>r}

sup

Ri (θ)

{θ∈Ω:kθk>r}

. Hence, it follows from Theorem 5 that

i=1

Rx,T (θ) is continuous in x and T . Since xi → x∞ and Ti → T ∞ ,
v̄i = v̄xi ,Ti → v̄x∞ ,T ∞ .

Since v̄x,T is well-defined for every x ∈ X, T ∈ T, and since x∞ ∈ X, T ∞ ∈ T, v̄x∞ ,T ∞ is also
well-defined. It follows that
lim v̄n = v̄x∞ ,T ∞
n→∞

exists.
To show that the limit lim v̄n is negative, let us first re-write Vi (r) as
n→∞


fi (Xi |θ0 )
fi (Xi |θ)
{θ∈Ω:kθk>r}


fi (Xi |θ0 )
≤−
inf
log
fi (Xi |θ)
{θ∈Ω:kθk≥r}
fi (Xi |θ0 )
,
= − log
fi (Xi |θi∗ (Xi ))

Vi (r) = −



inf

log

(3.19)

for some θi∗ (Xi ), depending upon Xi , contained in Ωr = Ω ∩ {θ : kθk ≥ r}. Recall that we chose r > 0
such that kθ0 k < r and Ω ∩ {θ : kθk > r} =
6 ∅, so that θi∗ (Xi ) 6= θ0 as kθi∗ (Xi )k ≥ r > kθ0 k for all Xi .
∗
It is important to observe that θi (Xi ) can not be a one-to-one function of Xi ≡ (Ui , Vi ). To see this, first
observe that for any given constant c, the equation log fi (Xi |θ0 ) − log fi (Xi |θ) = c, equivalently, the
equation log fi (Ui , Vi |θ0 ) − log fi (Ui , Vi |θ) = c, admits infiniteh number of solutions
in (Ui , Vi ), for any
i
fi (Xi |θ0 )
fi (Xi |θ0 )
2
∗
given θ = (µ, ω ). Hence, for θi (Xi ) = ϕ such that inf
log fi (Xi |θ) = log fi (Xi |ϕ) = c, there
{θ:kθk≥r}

8

exist infinitely many values of (Ui , Vi ) with the same infimum c for the same value ϕ, thereby proving
that θi∗ (Xi ) is a many-to-one function of Xi . A consequence of thishis non-degeneracy
i of the conditional

fi (Xi |θ0 )
= Ki (θ0 , θi∗ (Xi ))
distribution of Xi , given θi∗ (Xi ), which ensures that EXi |θi∗ (Xi ),θ0 log fi (X
∗
i |θi (Xi ))
is well-defined and strictly positive, since θi∗ (Xi ) 6= θ0 .
Given the above arguments, now note that,




fi (Xi |θ0 )
fi (Xi |θ0 )
Eθ0 log
= Eθi∗ (Xi )|θ0 EXi |θi∗ (Xi )=ϕi ,θ0 log
fi (Xi |θi∗ (Xi ))
fi (Xi |θi∗ (Xi ) = ϕi )

= Eθi∗ (Xi )|θ0 [Ki (θ0 , ϕi )]


≥ Eθi∗ (Xi )|θ0 inf Ki (θ0 , ϕi )
ϕi ∈Ωr

= Eθi∗ (Xi )|θ0 [Ki (θ0 , ϕ∗i )]
= Ki (θ0 , ϕ∗i ),

(3.20)

where ϕ∗i ∈ Ωr is where the infimum of Ki (θ0 , ϕi ) is achieved. Since ϕ∗i is independent of Xi , the last
step (3.20) follows. Hence,
Eθ0 Vi (r) ≤ −Ki (θ0 , ϕ∗i ) ≤ −

inf

x∈X,T ∈T,θ∈Ωr

Kx,T (θ0 , ϕ) = −Kx∗ ,T ∗ (θ0 , ϕ∗ ),

(3.21)

for some x∗ ∈ X, T ∗ ∈ T and ϕ∗ ∈ Ωr . Since Kx∗ ,T ∗ (θ0 , ϕ∗ ) is a well-defined Kullback-Leibler distance, it is strictly positive since ϕ∗ 6= θ0 . Hence, it follows from (3.21) and the fact that Kx∗ ,T ∗ (θ0 , ϕ∗ ) >
0, that
Pn
i=1 Eθ0 Vi (r)
lim v̄n = lim
n→∞
n→∞
Pn n
∗
i=1 Ki (θ0 , ϕi )
≤ − lim
n→∞
n
≤ −Kx∗ ,T ∗ (θ0 , ϕ∗ )
< 0.

Thus, condition (4)(ii) holds.
Regularity condition (5) holds because for any θ ∈ Ω, Ri (θ) is an almost surely continuous function
of Xi rendering it measurable for all θ ∈ Ω, and due to the fact that supremums of measurable functions
are measurable.
In other words, in the non-iid set-up in the non-iid SDE framework, the following theorem holds:
Theorem 7 Assume the non-iid SDE setup and conditions (H1′ ) (i) and (H2′ ) – (H6′ ). Then it holds
P
that θ̂n → θ0 .

3.2 Asymptotic normality of MLE in the non-iid set-up
′ (x, θ) =
Let ζi (x, θ) = log fi (x|θ); also, let ζi′ (x, θ) be the d × 1 vector with j-th component ζi,j
2

∂
′′ (x, θ) =
and let ζi′′ (x, θ) be the d × d matrix with (j, k)-th element ζi,jk
∂θj ∂θk ζi (x, θ).
For proving asymptotic normality in the non-iid framework, Hoadley (1971) assumed the following
regularity conditions:
∂
∂θj ζi (x, θ),

(1) Ω is an open subset of Rd .
P

(2) θ̂n → θ0 .
(3) ζi′ (Xi , θ) and ζi′′ (Xi , θ) exist a.s. [Pθ0 ].
9

(4) ζi′′ (Xi , θ) is a continuous function of θ, uniformly in i, a.s. [Pθ0 ], and is a measurable function of
Xi .
(5) Eθ [ζi′ (Xi , θ)] = 0 for i = 1, 2, . . ..


(6) Ii (θ) = Eθ ζi′ (Xi , θ)ζi′ (Xi , θ)T = −Eθ [ζi′′ (Xi , θ)], where for any vector y, y T denotes the
transpose of y.
(7) Īn (θ) → Ī(θ) as n → ∞ and Ī(θ) is positive definite.
′ (X , θ )
(8) Eθ0 ζi,j
i 0

3

≤ K2 , for some 0 < K2 < ∞.

(9) There exist ǫ > 0 and random variables Bi,jk (Xi ) such that
o
n
′′ (X , ξ) : kξ − θ k ≤ ǫ ≤ B
(i) sup ζi,jk
0
i
i,jk (Xi ).
(ii) Eθ0 |Bi,jk (Xi )|1+δ ≤ K2 , for some δ > 0.

Condition (8) can be weakened but is relatively easy to handle. Under the above regularity conditions,
Hoadley (1971) prove that


√ 
L
n θ̂n − θ0 → N 0, Ī −1 (θ0 ) .
(3.22)
3.2.1

Validation of asymptotic normality of M LE in the non-iid SDE set-up

Note that condition (1) requires the parameter space Ω to be an open subset. However, the proof of
asymptotic normality presented in Hoadley (1971) continues to hold for compact Ω, since for any open
cover of Ω we can extract a finite subcover, consisting of open sets.
Conditions (2), (3), (5), (6) are clearly valid in our case. Condition (4) can be verified in exactly
the same way as condition (2) of Section 3.1 is verified; measurability of ζi′′ (Xi , θ) follows due its
continuity with respect to Xi . Condition (7) simply followsfrom (3.10) and condition (8) holds due to
finiteness of the moments of the random variable U (x, T )/ 1 + ω 2 V (x, T ) , for every x ∈ X, T ∈ T,
and compactness of X and T.
′′ (X , θ) for j, k = 1, 2 are given by ∂ 2 log f (X |θ) =
For conditions (9)(i) and (9)(ii) note that ζi,jk
i
i
∂µ2

∂2
∂2
1
2
2
2
2
2
−Ii (ω ), ∂µ∂ω2 log f (Xi |θ) = −γi (θ)Ii (ω ), and ∂ω2 ∂ω2 log f (Xi |θ) = − 2 2γi (θ)Ii (ω ) − Ii (ω 2 ) .
Also since Delattre et al. (2013) establish


Ui
|µ̄|
ω02
(3.23)
sup |γi (θ)| ≤
2+ 2 + 2,
2
ω
ω
1 + ω 0 Vi
θ∈Ω
it follows from (3.23), the fact that 0 < Ii (ω 2 ) < 1/ω 2 , finiteness of moments of all orders of the
previously mentioned derivatives for every x ∈ X, T ∈ T, and compactness of X and T, that conditions
(9)(i) and (9)(ii) hold.
In other words, in our non-iid SDE case we have the following theorem on asymptotic normality.
Theorem 8 Assume the non-iid SDE setup and conditions (H1′ ) (i) and (H2′ ) – (H6′ ). Then (3.22)
holds.

4 Summary and conclusion
In SDE based random effects model framework, Delattre et al. (2013) considered the linearity assumption in the drift function given by b(x, φi ) = φi b(x), where φi are supposed to be Gaussian random
variables with mean µ and variance ω 2 , and obtained a closed form expression of the likelihood of the
above parameters. Assuming the iid set-up, they proved convergence in probability and asymptotic
10

normality of the maximum likelihood estimator of the parameters. In this paper, we proved strong consistency, rather than weak consistency, and asymptotic normality of the maximum likelihood estimator
under weaker assumptions in the iid set-up. Moreover, we extended the model of Delattre et al. (2013)
to the independent, but non-identical set-up, proving weak consistency and asymptotic normality.
In Maitra and Bhattacharya (2015), we extended our classical asymptotic theory to the Bayesian
framework, for both iid and non-iid situations. Specifically, we proved posterior consistency and asymptotic posterior normality, for both iid and non-iid set-ups. There we have also illustrated our theoretical
development with several examples and simulation studies. It is to be noted that those examples, illustrating consistency and inconsistency of the associated Bayes estimators, remains valid in the classical
paradigm with the Bayes estimators replaced by the maximum likelihood estimators.

Acknowledgments
Sincere gratitude goes to the reviewer whose suggestions have led to much improved presentation of our
article. The first author gratefully acknowledges her CSIR Fellowship, Govt. of India.

Appendix
Proof of Theorem 5. We can decompose (3.2) as
U (x, T ) =

Z

T

b(X(s, x))
φ(x)b(X(s, x))ds
σ 2 (X(s, x))

Z

T

0

+

0

= φ(x)
+

Z

0

Z

T

b(X(s, x))
(dX(s, x) − φ(x)b(X(s, x))ds)
σ 2 (X(s, x))
T

b2 (X(s, x))
ds
σ 2 (X(s, x))

(4.1)

b(X(s, x))
dW (s)
σ(X(s, x))

(4.2)

0

= φ(x)U (1) (x, T ) + U (2) (x, T ),

(say),

(4.3)

where W (s) is the standard Weiner process defined on [0, T ].
Given the process X(·, ·), continuity of (4.1) with respect to x and T can be seen as follows. Let

11

T1 , T2 ∈ T; without loss of generality, let T2 > T1 . Also, let x1 , x2 ∈ X. Then,
U (1) (x1 , T1 ) − U (1) (x2 , T2 )
Z T2 2
Z T1 2
b (X(s, x2 ))
b (X(s, x1 ))
ds −
ds
=
2
σ (X(s, x1 ))
σ 2 (X(s, x2 ))
0
0

Z T1  2
b (X(s, x1 ))
b2 (X(s, x2 ))
=
−
ds
σ 2 (X(s, x1 )) σ 2 (X(s, x2 ))
0
Z T2 2
b (X(s, x2 ))
−
ds
2
T1 σ (X(s, x2 ))
Z T1 2
b2 (X(s, x2 ))
b (X(s, x1 ))
ds
−
≤
σ 2 (X(s, x1 )) σ 2 (X(s, x2 ))
0
Z T2 2
b (X(s, x2 ))
+
ds
σ 2 (X(s, x2 ))
T1
≤ T1 sup

s∈[0,T1 ]

b2 (X(s, x2 ))
b2 (X(s, x1 ))
−
σ 2 (X(s, x1 )) σ 2 (X(s, x2 ))

+ |T2 − T1 |

sup

s∈[T1 ,T2 ],x∈X

≤ Tmax

b2 (X(s, x))
σ 2 (X(s, x))

b2 (X(s∗ , x1 ))
b2 (X(s∗ , x2 ))
+ C2 |T2 − T1 |,
−
σ 2 (X(s∗ , x1 )) σ 2 (X(s∗ , x2 ))

(4.4)
(4.5)
(4.6)

where Tmax = sup T; s∗ ∈ [0, T1 ] is such that the supremum in (4.4) is attained. That there exists such
s∗ is clear due to continuity of the functions in s and compactness of the interval. In (4.6), C2 is the
2
upper bound for the function σb 2(X(s,x))
(X(s,x)) .
Since X(·, x) is continuous in x, due to continuity of b(·) and σ(·), for any ǫ > 0, one can choose
δ1 (ǫ) > 0 such that |x1 − x2 | < δ1 (ǫ) implies
ǫ
b2 (X(s∗ , x2 ))
b2 (X(s∗ , x1 ))
<
−
,
σ 2 (X(s∗ , x1 )) σ 2 (X(s∗ , x2 ))
2Tmax
so that the first term in (4.6) is less than ǫ/2. Choosing δ2 (ǫ) = 2Cǫ 2 yields that if |T2 − T1 | < δ2 (ǫ),
then the second term in (4.6) is less than ǫ/2. This shows continuity of U (1) (x, T ) with respect to x and
∞
T for given X(·, ·). It follows that, for sequences {xm }∞
m=1 , {Tm }m=1 such that xm → x̃ and Tm → T̃
as m → ∞,
L
U (1) (xm , Tm ) → U (1) (x̃, T̃ ).
(4.7)
It is also clear that

L

φ(xm ) → φ(x̃).

(4.8)

h
i2k
sup E φ(xm )U (1) (xm , Tm )
< ∞,

(4.9)

Now note that due to assumptions (H5′ ), (H6′ ) (observing that U (1) (x, T ) = V (x, T ) for all x ∈ X and
T ∈ T), and compactness of X and T, we have, for any k ≥ 1,
m≥1

for all m ≥ 1, ensuring requisite uniform integrability. Hence, it follows that
h
i2
E φ(xm )U (1) (xm , Tm ) − φ(x̃)U (1) (x̃, T̃ ) → 0.

(4.10)

Let us now deal with U (2) (x, T ) given by (4.2). Letting for any set A, δA (s) = 1 if s ∈ A and 0
12

otherwise, be the indicator function, we define
2
Z Tmax 
b(X(s, x̃))
b(X(s, xm ))
δ
(s) −
δ
(s) ds
Q(xm , Tm ) =
σ(X(s, xm )) [0,Tm ]
σ(X(s, x̃)) [0,T̃ ]
0
Z Tmax 2
Z Tmax 2
b (X(s, xm ))
b (X(s, x̃))
=
δ[0,Tm ] (s)ds +
δ
(s)ds
2
σ (X(s, xm ))
σ 2 (X(s, x̃)) [0,T̃ ]
0
0
Z Tmax
b(X(s, xm )) b(X(s, x̃))
δ
(s)ds
−2
σ(X(s, xm )) σ(X(s, x̃)) [0,min{Tm ,T̃ }]
0
Z Tm 2
Z T̃ 2
b (X(s, xm ))
b (X(s, x̃))
=
ds +
ds
2
2
σ (X(s, xm ))
0
0 σ (X(s, x̃))
Z min{Tm ,T̃ }
b(X(s, xm )) b(X(s, x̃))
−2
ds.
σ(X(s,
xm )) σ(X(s, x̃))
0

(4.11)

It follows in the same way as in the proof of continuity of U (1) (·, ·) that the first and the third
integrals in (4.11) associated with the function Q(·, ·), are continuous at (x̃, T̃ ). As a result, for given
L

X(·, ·), Q(xm , Tm ) → 0 as m → ∞. It follows that Q(xm , Tm ) → 0.
Now note that
" Z
2 Z Tmax 2
2 #
Tmax 2
b (X(s, x̃))
b (X(s, xm ))
Q(xm , Tm ) ≤ 2
,
ds +
ds
σ 2 (X(s, xm ))
σ 2 (X(s, x̃))
0
0
so that, for any k ≥ 2,
E [Q(xm , Tm )]k ≤ 22k E

"Z

Tmax
0

b2 (X(s, xm ))
ds
σ 2 (X(s, xm ))

2k

+

Z

Tmax
0

b2 (X(s, x̃))
ds
σ 2 (X(s, x̃))

2k #

.

(4.12)

Since, by assumption (H6′ ) moments of all orders of V (x, T ) are finite, for any x ∈ X and T ∈ T, and
since X and T are compact, it follows that
sup E [Q(xm , Tm )]k < ∞,

m≥1

guaranteeing uniform integrabiility. Hence,
E [Q(xm , Tm )] → 0,

as m → ∞.

(4.13)

By Itô isometry (see, for example, Øksendal (2003)) it then follows that
E

Z

Tmax

0

b(X(s, xm ))
δ
(s)dW (s) −
σ(X(s, xm )) [0,Tm ]

Z

Tmax

0

That is,
E

"Z

Tm

0

b(X(s, xm ))
dW (s) −
σ(X(s, xm ))

It follows that

Z

T̃

0

2
b(X(s, x̃))
δ
(s)dW (s) → 0.
σ(X(s, x̃)) [0,T̃ ]

(4.14)

#2
b(X(s, x̃))
dW (s) → 0.
σ(X(s, x̃))

(4.15)

L

U (2) (xm , Tm ) → U (2) (x̃, T̃ ).

(4.16)

Using the Burkholder-Davis-Gundy inequality (see, for example, Delattre et al. (2013)) we obtain
h

E U

(2)

Z
i2k
(xm , Tm )
≤ Ck E

0

13

Tm

b2 (X(s, xm ))
ds
σ 2 (X(s, xm ))

k

.

(4.17)


2k
<
Again, due to assumption (H6′ ) and compactness of X and T it follows that sup E U (2) (xm , Tm )
m≥1

∞, so that uniform integrability is assured. It follows that

h
i2
E U (2) (xm , Tm ) − U (2) (x̃, T̃ ) → 0.

(4.18)

From (4.10) and (4.18) it follows, using the Cauchy-Schwartz inequality, that
h
i2
E U (xm , Tm ) − U (x̃, T̃ )
h
i2
h
i2
≤ E φ(xm )U (1) (xm , Tm ) − φ(x̃)U (1) (x̃, T̃ ) + E U (2) (xm , Tm ) − U (2) (x̃, T̃ )
r h
i
h
+2

E φ(xm )U (1) (xm , Tm ) − φ(x̃)U (1) (x̃, T̃ )

→ 0.

2

i2
E U (2) (xm , Tm ) − U (2) (x̃, T̃ )

(4.19)

Since V (x, T ) = U (1) (x, T ), due to (4.7) and assumption (H6′ ) (the latter ensuring uniform integrability), it easily follows that
h
i2
E V (xm , Tm ) − V (x̃, T̃ ) → 0.

(4.20)

Let G(x, T ) = (U (x, T ), V (x, T )). Then it follows from (4.19) and (4.20), that
L

G(xm , Tm ) → G(x̃, T̃ ).

(4.21)



L
(U (xm , Tm ), V (xm , Tm )) → U (x̃, T̃ ), V (x̃, T̃ ) .

(4.22)

That is,
Consider also a sequence {θm }∞
m=1 in Ω, converging to θ̃ ∈ Ω. Then, for any function h(u, v, θ),
which is continuous in u, v and θ, and such that the sequence {h(U (xm , Tm ), V (xm , Tm ), θm )}∞
m=1 is
uniformly integrable, we must have
h
i
E [h(U (xm , Tm ), V (xm , Tm ), θm )] → E h(U (x̃, T̃ ), V (x̃, T̃ ), θ̃) ,
(4.23)
ensuring continuity of E [h(U (x, T ), V (x, T ), θ)] with respect to x, T and θ.

References
Delattre, M., Genon-Catalot, V., and Samson, A. (2013). Maximum Likelihood Estimation for Stochastic Differential Equations with Random Effects. Scandinavian Journal of Statistics, 40, 322–343.
Hoadley, B. (1971). Asymptotic Properties of Maximum Likelihood Estimators for the Independent not
Identically Distributed Case. The Annals of Mathematical Statistics, 42, 1977–1991.
Maitra, T. and Bhattacharya, S. (2015). On Bayesian Asymptotics in Stochastic Differential Equations with Random Effects. Statistics and Probability Letters, 103, 148–159. Also available at
“http://arxiv.org/abs/1407.3971”.
Øksendal, B. (2003). Stochastic Differential Equations. Springer-Verlag, New York.
Schervish, M. J. (1995). Theory of Statistics. Springer-Verlag, New York.

14

