1

Cluster-Seeking James-Stein Estimators

arXiv:1602.00542v4 [] 16 Mar 2018

K. Pavan Srinath and Ramji Venkataramanan

Abstract—This paper considers the problem of estimating a
high-dimensional vector of parameters θ ∈ Rn from a noisy
observation. The noise vector is i.i.d. Gaussian with known
variance. For a squared-error loss function, the James-Stein (JS)
estimator is known to dominate the simple maximum-likelihood
(ML) estimator when the dimension n exceeds two. The JSestimator shrinks the observed vector towards the origin, and
the risk reduction over the ML-estimator is greatest for θ that lie
close to the origin. JS-estimators can be generalized to shrink the
data towards any target subspace. Such estimators also dominate
the ML-estimator, but the risk reduction is significant only when
θ lies close to the subspace. This leads to the question: in
the absence of prior information about θ, how do we design
estimators that give significant risk reduction over the MLestimator for a wide range of θ?
In this paper, we propose shrinkage estimators that attempt
to infer the structure of θ from the observed data in order
to construct a good attracting subspace. In particular, the
components of the observed vector are separated into clusters,
and the elements in each cluster shrunk towards a common
attractor. The number of clusters and the attractor for each
cluster are determined from the observed vector. We provide
concentration results for the squared-error loss and convergence
results for the risk of the proposed estimators. The results
show that the estimators give significant risk reduction over the
ML-estimator for a wide range of θ, particularly for large n.
Simulation results are provided to support the theoretical claims.
Index Terms—High-dimensional estimation, Large deviations
bounds, Loss function estimates, Risk estimates, Shrinkage estimators

Onsider the problem of estimating a vector of parameters
θ ∈ Rn from a noisy observation y of the form
y = θ + w.

The noise vector w ∈ Rn is distributed as N (0, σ 2 I), i.e., its
components are i.i.d. Gaussian random variables with mean
zero and variance σ 2 . We emphasize that θ is deterministic,
so the joint probability density function of y = [y1 , . . . , yn ]T
for a given θ is
pθ (y) =

1
n

(2πσ 2 ) 2

e

ky−θk2
− 2σ2

where the expectation is computed using the density in (I).
The normalized risk is R(θ, θ̂)/n.
Applying the maximum-likelihood (ML) criterion to (1)
yields the ML-estimator θ̂M L = y. The ML-estimator is an
unbiased estimator, and its risk is R(θ, θ̂M L ) = nσ 2 . The goal
of this paper is to design estimators that give significant risk
reduction over θ̂M L for a wide range of θ, without any prior
assumptions about its structure.
In 1961 James and Stein published a surprising result [1],
proposing an estimator that uniformly achieves lower risk than
θ̂M L for any θ ∈ Rn , for n ≥ 3. Their estimator θ̂JS is given
by


(n − 2)σ 2
y,
(2)
θ̂JS = 1 −
kyk2
and its risk is [2, Chapter 5, Thm. 5.1]




1
R θ, θ̂JS = nσ 2 − (n − 2)2 σ 4 E
.
kyk2

(3)

Hence for n ≥ 3,
R(θ, θ̂JS ) < R(θ, θ̂M L ) = nσ 2 , ∀θ ∈ Rn .

(4)

An estimator θ̂1 is said to dominate another estimator θ̂2 if

I. I NTRODUCTION

C

where k·k denotes the Euclidean norm. The risk of the
estimator for a given θ is the expected value of the loss
function:
h
i
R(θ, θ̂) := E kθ̂(y) − θk2 ,

.

(1)

The performance of an estimator θ̂ is measured using the
squared-error loss function given by
L(θ, θ̂(y)) := kθ̂(y) − θk2 ,
This work was supported in part by a Marie Curie Career Integration Grant
(Grant Agreement No. 631489) and an Early Career Grant from the Isaac
Newton Trust. This paper was presented in part at the 2016 IEEE International
Symposium on Information Theory.
K. P. Srinath and R. Venkataramanan are with Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, UK (e-mail: {pk423,
rv285}@cam.ac.uk).

R(θ, θ̂1 ) ≤ R(θ, θ̂2 ), ∀θ ∈ Rn ,
with the inequality being strict for at least one θ. Thus (4) implies that the James-Stein estimator (JS-estimator) dominates
the ML-estimator. Unlike the ML-estimator, the JS-estimator
is non-linear and biased. However, the risk reduction over the
ML-estimator can be significant, making it an attractive option
in many situations — see, for example, [3].
By evaluating the expression in (3), it can be shown that
the risk of the JS-estimator depends on θ only via kθk [1].
Further, the risk decreases as kθk decreases. (For intuition
about this, note in (3) that for large n, kyk2 ≈ nσ 2 + kθk2 .)
The dependence of the risk on kθk is illustrated in Fig. 1,
where the average loss of the JS-estimator is plotted versus
kθk, for two different choices of θ.
The JS-estimator in (2) shrinks each element of y towards
the origin. Extending this idea, JS-like estimators can be
defined by shrinking y towards any vector, or more generally,
towards a target subspace V ⊂ Rn . Let PV (y) denote the projection of y onto V, so that ky−PV (y)k2 = minv∈V ky−vk2 .
Then the JS-estimator that shrinks y towards the subspace V

2

1.6
Regular JS−Estimator
JS−Estimator Positive Part
Lindley’s Estimator
Lindley’s Estimator Positive Part
ML−Estimator

1.4

R̃(θ, θ̂)/n

1.2
1
0.8
0.6
0.4
0.2
0
0

1

2

3

4

5

6

7

∥θ∥
(a)
1.6

Regular JS−Estimator
JS−Estimator Positive Part
Lindley’s Estimator
Lindley’s Estimator Positive Part
ML−Estimator

1.4

R̃(θ, θ̂)/n

1.2
1
0.8
0.6
0.4
0.2
0
0

1

2

3

4

5

6

7

∥θ∥
(b)

Fig. 1: Comparison of the average normalized loss of the regular
JS-estimator, Lindley’s estimator, and their positive-part versions for
2
n = 10 as a function of
√ =
√ kθk. The loss of the ML-estimator is σ
1. In (a) θi = kθk/ 10, i = √1, · · · , 5, and θi = −kθk/ 10,
i = 6, · · · , 10. In (b), θi = kθk/ 10, ∀i.

is
"
θ̂ = PV (y) + 1 −

(n − d − 2)σ 2
2

ky − PV (y)k

#
(y − PV (y)) ,

(5)

where d is the dimension of V.1 A classic example of such an
estimator is Lindley’s estimator [4], which shrinks y towards
the one-dimensional subspace defined by the all-ones vector
1. It is given by


(n − 3)σ 2
θ̂L = ȳ1 + 1 −
(y − ȳ1) ,
(6)
ky − ȳ1k2
Pn
where ȳ := n1 i=1 yi is the empirical mean of y.
It can be shown that the different variants of the JS-estimator
such as (2),(5),(6) all dominate the ML-estimator.2 Further,
all JS-estimators share the following key property [6]–[8]:
the smaller the Euclidean distance between θ and the
attracting vector, the smaller the risk.
Throughout this paper, the term “attracting vector” refers
to the vector that y is shrunk towards. For θ̂JS in (2), the
attracting vector is 0, and the risk reduction over θ̂M L is
larger when kθk is close to zero. Similarly, if the components
of θ are clustered around some value c, a JS-estimator with
1 The dimension n has to be greater than d + 2 for the estimator to achieve
lower risk than θ̂M L .
2 The risks of JS-estimators of the form (5) can usually be computed using
Stein’s lemma [5], which states that E[Xg(X)] = E[g 0 (X)], where X is a
standard normal random variable, and g a weakly differentiable function.

attracting vector c1 would give significant risk reduction over
θ̂M L . One motivation for Lindley’s estimator in (6) comes
from a guess that the components of θ are close to its empirical
mean θ̄ — since we do not know θ̄, we approximate it by ȳ
and use the attracting vector ȳ1.
Fig. 1 shows how the performance of θ̂JS and θ̂L depends
on the structure of θ. In the left panel of the figure, the empirical mean θ̄ is always 0, so the risks of both estimators increase
monotonically with kθk. In the right panel, all the components
of θ are all equal to θ̄. In this case, the
from the
p distance
Pn
attracting vector for θ̂L is kθ − ȳ1k = ( i=1 wi )2 /n, so
the risk does not vary with kθk; in contrast the risk of θ̂JS
increases with kθk as its attracting vector is 0.
The risk reduction obtained by using a JS-like shrinkage estimator over θ̂M L crucially depends on the choice of attracting
vector. To achieve significant risk reduction for a wide range
of θ, in this paper, we infer the structure of θ from the data
y and choose attracting vectors tailored to this structure. The
idea is to partition y into clusters, and shrink the components
in each cluster towards a common element (attractor). Both
the number of clusters and the attractor for each cluster are to
be determined based on the data y.
As a motivating example, consider
a θ in which half the
√
n
and
the
other half are equal
components
are
equal
to
kθk/
√
to −kθk/ n. Fig. 1(a) shows that the risk reduction of both
θ̂JS and θ̂L diminish as kθk gets larger. This is because the
empirical mean ȳ is close to zero, hence θ̂JS and θ̂L both
shrink y towards 0. An ideal√
JS-estimator would shrink the √
yi ’s
corresponding to θi = kθk/ n towards the attractor
kθk/
n,
√
and the remaining observations towards −kθk/ n. Such an
estimator would give handsome gains over θ̂M L for all θ with
the above structure. On the other hand, if θ is such that all
its components are equal (to θ̄), Lindley’s estimator θ̂L is an
excellent choice, with significantly smaller risk than θ̂M L for
all values of kθk (Fig. 1(b)).
We would like an intelligent estimator that can correctly
distinguish between different θ structures (such as the two
above) and choose an appropriate attracting vector, based
only on y. We propose such estimators in Sections III and
IV. For reasonably large n, these estimators choose a good
attracting subspace tailored to the structure of θ, and use
an approximation of the best attracting vector within the
subspace.
The main contributions of our paper are as follows.
• We construct a two-cluster JS-estimator, and provide concentration results for the squared-error loss, and asymptotic convergence results for its risk. Though this estimator does not dominate the ML-estimator, it is shown to
provide significant risk reduction over Lindley’s estimator
and the regular JS-estimator when the components of θ
can be approximately separated into two clusters.
• We present a hybrid JS-estimator that, for any θ and for
large n, has risk close to the minimum of that of Lindley’s
estimator and the proposed two-cluster JS-estimator. Thus
the hybrid estimator asymptotically dominates both the
ML-estimator and Lindley’s estimator, and gives significant risk reduction over the ML-estimator for a wide
range of θ.

3

•

•

We generalize the above idea to define general multiplecluster hybrid JS-estimators, and provide concentration
and convergence results for the squared-error loss and
risk, respectively.
We provide simulation results that support the theoretical
results on the loss function. The simulations indicate that
the hybrid estimator gives significant risk reduction over
the ML-estimator for a wide range of θ even for modest
values of n, e.g. n = 50. The empirical risk of the hybrid
estimator converges rapidly to the theoretical value with
growing n.

A. Related work
George [7], [8] proposed a “multiple shrinkage estimator”,
which is a convex combination of multiple subspace-based JSestimators of the form (5). The coefficients defining the convex
combination give larger weight to the estimators whose target
subspaces are closer to y. Leung and Barron [9], [10] also
studied similar ways of combining estimators and their risk
properties. Our proposed estimators also seek to emulate the
best among a class of subspace-based estimators, but there
are some key differences. In [7], [8], the target subspaces are
fixed a priori, possibly based on prior knowledge about where
θ might lie. In the absence of such prior knowledge, it may not
be possible to choose good target subspaces. This motivates the
estimators proposed in this paper, which use a target subspace
constructed from the data y. The nature of clustering in θ is
inferred from y, and used to define a suitable subspace.
Another difference from earlier work is in how the attracting
vector is determined given a target subspace V. Rather than
choosing the attracting vector as the projection of y onto
V, we use an approximation of the projection of θ onto V.
This approximation is computed from y, and concentration
inequalities are provided to guarantee the goodness of the
approximation.
The risk of a JS-like estimator is typically computed using
Stein’s lemma [5]. However, the data-dependent subspaces
we use result in estimators that are hard to analyze using
this technique. We therefore use concentration inequalities to
bound the loss function of the proposed estimators. Consequently, our theoretical bounds get sharper as the dimension
n increases, but may not be accurate for small n. However,
even for relatively small n, simulations indicate that the risk
reduction over the ML-estimator is significant for a wide range
of θ.
Noting that the shrinkage factor multiplying y in (2) could
be negative, Stein proposed the following positive-part JSestimator [1]:


(n − 2)σ 2
y,
(7)
θ̂JS+ = 1 −
kyk2
+
where X+ denotes max(0, X). We can similarly define
positive-part versions of JS-like estimators such as (5) and
(6). The positive-part Lindley’s estimator is given by


(n − 3)σ 2
θ̂L+ = ȳ1 + 1 −
(y − ȳ1) .
(8)
ky − ȳ1k2 +

Baranchik [11] proved that θ̂JS+ dominates θ̂JS , and his result
also proves that θ̂L+ dominates θ̂L . Estimators that dominate
θ̂JS+ are discussed in [12], [13]. Fig. 1 shows that the positivepart versions can give noticeably lower loss than the regular
JS and Lindley estimators. However, for large n, the shrinkage
factor is positive with high probability, hence the positive-part
estimator is nearly always identical to the regular JS-estimator.
2
kθk2
2
Indeed, for large n, kyk
n ≈ n +σ , and the shrinkage factor
is
 


(n − 2)σ 2
(n − 2)σ 2
≈
1
−
> 0.
1−
kyk2
kθk2 + nσ 2
We analyze the positive-part version of the proposed hybrid
estimator using concentration inequalities. Though we cannot
guarantee that the hybrid estimator dominates the positivepart JS or Lindley estimators for any finite n, we show that
for large n, the loss of the hybrid estimator is equal to the
minimum of that of the positive-part Lindley’s estimator and
the cluster-based estimator with high probability (Theorems 3
and 4).
The rest of the paper is organized as follows. In Section
II, a two-cluster JS-estimator is proposed and its performance
analyzed. Section III presents a hybrid JS-estimator along
with its performance analysis. General multiple-attractor JSestimators are discussed in Section IV, and simulation results
to corroborate the theoretical analysis are provided in Section
V. The proofs of the main results are given in Section VI.
Concluding remarks and possible directions for future research
constitute Section VII.
B. Notation
Bold lowercase letters are used to denote vectors, and plain
lowercase letters for their entries. For example, the entries of
y ∈ Rn are yi , i = 1, · · · , n. All vectors have length n and
are column vectors, unless otherwise mentioned. For vectors
y, z ∈ Rn , hy, zi denotes their Euclidean inner product. The
all-zero vector and the all-one vector of length n are denoted
by 0 and 1, respectively. The complement of a set A is
denoted by Ac . For a finite set A with real-valued elements,
min(A) denotes the minimum of the elements in A. We
use 1{E} to denote the indicator function of an event E. A
central chi-squared distributed random variable with n degrees
of freedom
Xn2 . The Q-function is given by
R ∞ is 1denoted by
2
x
Q(x) = x √2π exp(− 2 )dx, and Qc (x) := 1 − Q(x). For a
random variable X, X+ denotes max(0, X). For real-valued
functions f (x) and g(x), the notation f (x) = o(g(x)) means
that limx→0 [f (x)/g(x)] = 0, and f (x) = O(g(x)) means that
limx→∞ [f (x)/g(x)] = c for some positive constant c.
P
For a sequence of random variables {Xn }∞
n=1 , Xn −→ X,
a.s.

L1

Xn −→ X, and Xn −→ X respectively denote convergence
in probability, almost sure convergence, and convergence in
L1 norm to the random variable X.
We use the following shorthand for concentration inequalities. Let {Xn (θ), θ ∈ Rn }∞
n=1 be a sequence of random
.
variables. The notation Xn (θ) = X, where X is either a
random variable or a constant, means that for any  > 0,
nk min(2 ,1)

P (|Xn (θ) − X| ≥ ) ≤ Ke

− max(kθk2 /n,1)

,

(9)

4

where K and k are positive constants that do not depend on
n or θ. The exact values of K and k are not specified.
The shrinkage estimators we propose have the general form
"
#
nσ 2
θ̂ = ν + 1 −
(y − ν) .
2
ky − νk +
For 1 ≤ i ≤ n, the ith component of the attracting vector ν
is the attractor for yi (the point towards which it is shrunk).
II. A TWO - CLUSTER JAMES -S TEIN ESTIMATOR
Recall the example
√
√in Section I where θ has half its components equal to kθk/ n, and the other half equal to kθk/ n.
Ideally, we would like to√shrink the yi ’s corresponding to the
first group
√ towards kθk/ n, and the remaining points towards
−kθk/ n. However, without an oracle, we cannot accurately
guess which point each yi should be shrunk towards. We would
like to obtain an estimator that identifies separable clusters in
y, constructs a suitable attractor for each cluster, and shrinks
the yi in each cluster towards its attractor.
We start by dividing the observed data into two clusters
based on a separating point sy , which is obtained from y. A
natural choice for the sy would be the empirical mean θ̄; since
this is unknown we use sy = ȳ. Define the clusters
C1 := {yi , 1 ≤ i ≤ n | yi > ȳ},
C2 := {yi , 1 ≤ i ≤ n | yi ≤ ȳ}.
The points in C1 and C2 will be shrunk towards attractors
a1 (y) and a2 (y), respectively, where a1 , a2 : Rn → R are
defined in (21) later in this section. For brevity, we henceforth
do not indicate the dependence of the attractors on y. Thus
the attracting vector is




1{y1 ≤ȳ}
1{y1 >ȳ}
 1{y2 ≤ȳ} 
 1{y2 >ȳ} 




(10)
ν2 := a1 
,
 + a2 
..
..




.
.
1{yn ≤ȳ}
1{yn >ȳ}
with a1 and a2 defined in (21). The proposed estimator is
"
#
nσ 2
θ̂JS2 = ν2 + 1 −
(y − ν2 )
2
ky − ν2 k +


2
σ
  (y − ν2 ) ,
= ν2 + 1 − 
(11)
2
g ky − ν2 k /n
where the function g is defined as
g(x) := max(σ 2 , x),

Analogously, the vector in the two-dimensional subspace
defined by (10) that is closest to θ is the projection of θ onto
this subspace. Computing this projection, the desired values
for a1 , a2 are found to be
Pn
Pn
θi 1{yi ≤ȳ}
i=1 θi 1{yi >ȳ}
des
des
P
, a2 = Pi=1
. (13)
a1 =
n
n
1
i=1 {yi >ȳ}
i=1 1{yi ≤ȳ}
As the θi ’s are not available, we define the attractors a1 , a2
des
as approximations of ades
1 , a2 , obtained using the following
concentration results.
Lemma 1. We have
n
n
n
1X
σ X − (θ̄−θ2i )2
. 1X
yi 1{yi >ȳ} =
θi 1{yi >ȳ} + √
e 2σ ,
n i=1
n i=1
n 2π i=1
(14)
n
n
n
1X
σ X − (θ̄−θ2i )2
. 1X
yi 1{yi ≤ȳ} =
θi 1{yi ≤ȳ} − √
e 2σ ,
n i=1
n i=1
n 2π i=1
(15)


n
n
θ̄ − θi
1X
. X
θi Q
θi 1{yi >ȳ} =
,
(16)
n i=1
σ
i=1


n
n
θ̄ − θi
1X
. 1X
θi 1{yi ≤ȳ} =
θi Qc
,
(17)
n i=1
n i=1
σ
!


n
n
X
2
θ̄ − θi
1 X
≥  ≤ Ke−nk ,
1{yi >ȳ} −
Q
P
n i=1
σ
i=1
(18)
!


n
n
X
X
2
1
θ̄ − θi
P
1{yi ≤ȳ} −
Qc
≥  ≤ Ke−nk .
n i=1
σ
i=1
(19)




i
i
:= 1 − Q θ̄−θ
. Recall from Section I-B
where Qc θ̄−θ
σ
σ
.
that the symbol = is shorthand for a concentration inequality
of the form (9).
The proof is given in Appendix D.
des
Using Lemma 1, we can obtain estimates for ades
1 , a2
in (13) provided we have an estimate for the term
Pn − (θ̄−θ2i )2
σ
√
2σ
. This is achieved via the following
i=1 e
n 2π
concentration result.
Lemma 2. Fix δ > 0. Then for any  > 0, we have
n
σ2 X
1{|yi −ȳ|≤δ} −
2nδ i=0
!

P
x ∈ R.

(12)

The attracting vector ν2 in (10) lies in a twodimensional subspace defined by the orthogonal vectors
[1{y1 >ȳ} , · · · , 1{yn >ȳ} ]T and [1{y1 ≤ȳ} , · · · , 1{yn ≤ȳ ]T . To derive the values of a1 and a2 in (10), it is useful to compare ν2
to the attracting vector of Lindley’s estimator in (6). Recall
that Lindley’s attracting vector lies in the one-dimensional
subspace spanned by 1. The vector lying in this subspace
that is closest in Euclidean distance to θ is its projection θ̄1.
Since θ̄ is unknown, we use the approximation ȳ to define the
attracting vector ȳ1.

≥

n
σ X − (θ̄−θ2i )2
√
e 2σ
+ κn δ
n 2π i=0

2

≤ 10e−nk ,

where k is a positive constant and |κn | ≤

!

(20)
√1 .
2πe

The proof is given in Appendix E.
Note 1. Henceforth in this paper, κn is used to denote a
generic bounded constant (whose exact value is not needed)
that is a coefficient of δ in expressions of the form f (δ) =
a + κn δ + o(δ) where a is some constant. As an example
1
to illustrate its usage, let f (δ) = a+bδ
, where a > 0 and

5

|bδ| < a. Then, we have f (δ) =
o(δ)) = a1 + κn δ + o(δ).

1
a(1+bδ/a)

=

1
a (1

+ ab δ +

Using Lemmas 1 and 2, the two attractors are defined to be
Pn
2 Pn
yi 1{yi >ȳ} − σ2δ i=0 1{|yi −ȳ|≤δ}
Pn
a1 = i=1
,
i=1 1{yi >ȳ}
(21)
Pn
Pn
σ2
i=0 1{|yi −ȳ|≤δ}
i=1 yi 1{yi ≤ȳ} + 2δ
Pn
.
a2 =
i=1 1{yi ≤ȳ}
With δ > 0 chosen to be a small positive number, this
completes the specification of the attracting vector in (10),
and hence the two-cluster JS-estimator in (11).
Note that ν2 , defined by (10), (21), is an approximation of the projection of θ onto the two-dimensional subspace V spanned by the vectors [1y1 >ȳ , · · · , 1yn >ȳ ]T and
[1y1 ≤ȳ , · · · , 1yn ≤ȳ ]T . We remark that ν2 , which approximates
the vector in V that is closest to θ, is distinct from the
projection of y onto V. While the analysis is easier (there
would be no terms involving δ) if ν2 were chosen to be
a projection of y (instead of θ) onto V, our numerical
simulations suggest that this choice yields significantly higher
risk. The intuition behind choosing the projection of θ onto V
is that if all the yi in a group are to be attracted to a common
point (without any prior information), a natural choice would
be the mean of the θi within
group, as in (13). This mean is
Pthe
n
determined
by
the
term
E(
i=1 θi 1{yi ≥ȳ} ), which is different
Pn
from E( i=1 yi 1{yi ≥ȳ} ) because
!
!
n
n
X
X
E
(yi − θi )1{yi ≥ȳ} = E
wi 1{yi ≥ȳ} 6= 0.
i=1

TheP term
involving
n
E( i=1 wi 1{yi ≥ȳ} ).

i=1

δ

in

(21)

approximates

Note 2. The attracting vector ν2 is dependent not just on y but
also on δ, through the two attractors a1 and a2 . In Lemma
2, for the deviation probability in (20) to fall exponentially
in n, δ needs to be held constant and independent of n.
From a practical design point
Pn of view, what is needed is
σ2
nδ 2  1. Indeed , for 2nδ
i=0 1{|yi −ȳ|≤δ} to be a reliable
(θ̄−θi )2
Pn
approximation of the term n√σ2π i=1 e− 2σ2 , it is shown
in Appendix E, specifically in (100) that we √
need nδ 2  1.
Numerical experiments suggest a value of 5/ n for δ to be
large enough for a good approximation.
We now present the first main result of the paper.
Theorem 1. The loss function of the two-cluster JS-estimator
in (11) satisfies the following:
(1) For any  > 0, and for any fixed δ > 0 that is independent
of n,



1
βn σ 2
P
kθ − θ̂JS2 k2 − min βn ,
+ κn δ
n
αn + σ 2
!

nk min(2 ,1)
−
+ o(δ) ≥  ≤ Ke max(kθk2 /n,1) ,
(22)
where αn , βn are given by (25) and (24) below, and K is

a positive constant that is independent of n and δ, while
k = Θ(δ 2 ) is another positive constant that is independent
of n (for a fixed δ).
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have




1
βn σ 2
+ κn δ + o(δ)
lim
R(θ, θ̂JS2 ) − min βn ,
n→∞ n
αn + σ 2
= 0.
(23)
The constants βn , αn are given by




n
n
kθk2
c21 X
θ̄ − θi
c22 X c θ̄ − θi
Q
−
Q
,
βn :=
−
n
n i=1
σ
n i=1
σ
(24)
!
 X

n
(θ̄−θ )2
2σ
− 2σ2i
√
e
(c1 − c2 ) , (25)
αn := βn −
n 2π
i=1
where
Pn

i=1 θi Q

c1 := P
n

i=1 Q





θ̄−θi
σ

θ̄−θi
σ



Pn

c
i=1 θi Q

 , c2 := P
n

c
i=1 Q





θ̄−θi
σ

θ̄−θi
σ



 . (26)

The proof of the theorem is given in Section VI-B.
Remark 1. In Theorem 1, βn represents the concentrating
value for the distance between θ and the attracting vector ν2 .
(It is shown in Sec. VI-B that kθ−ν2 k2 /n concentrates around
βn +κn δ.) Therefore, the closer θ is to the attracting subspace,
the lower the normalized asymptotic risk R(θ, θ̂JS2 )/n. The
term αn + σ 2 represents the concentrating value for the
distance between y and ν2 . (It is shown in Sec. VI-B that
ky − ν2 k2 /n concentrates around αn + σ 2 + κn δ.)
Remark 2. Comparing βn in (24) and αn in (25), we note
that βn ≥ αn because


Pn
−n i=1 (θi − θ̄)Q θiσ−θ̄

 P
 ≥ 0.

c1 − c2 = P
n
n
θi −θ̄
c θi −θ̄
Q
Q
i=1
i=1
σ
σ
(27)
To see (27), observe that in the sum in the numerator, the Q(·)
function assigns larger weight to the terms with (θi − θ̄) < 0
than to the terms with (θi − θ̄) > 0.
Furthermore, αn ≈ βn for large n if either |θi − θ̄| ≈ 0, ∀i,
or |θi − θ̄| → ∞, ∀i. In the first case, if θi = θ̄, for
i = 1, · · · , n, we get βn = αn = kθ − θ̄1k2 /n = 0.
In the second case, suppose that n1 of the θi values equal
p1 and the remaining (n − n1 ) values equal −p2 for some
p1 , p2 > 0. Then, as p1 , p2 → ∞, it can be verified that
βn → [kθk2 − n1 p21 − (n − n1 )p22 ]/n = 0. Therefore, the
asymptotic normalized risk R(θ, θ̂JS2 )/n converges to 0 in
both cases.
The proof of Theorem 1 further leads to the following
corollaries.
Corollary 1. The loss function of the positive-part JSestimator in (7) satisfies the following:

6

1.2

P

kθ − θ̂JS+ k2
γn σ 2
≥
−
n
γn + σ 2

!
2

≤ Ke−nk min(

,1)

,

2
where γn := kθk /n, and K and k are positive constants.
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have

lim

n→∞

1
γn σ 2
= 0.
R(θ, θ̂JS+ ) −
n
γn + σ 2

min {βn , βn /(αn + 1)}

(1) For any  > 0,

1

ρ = 0.1
0.8
0.6

Corollary 2. The loss function of the positive-part Lindley’s
estimator in (8) satisfies the following:

ρ = 0.5

ρ = 0.25

0.4

ρ=1
0.2
0
−0.2
0

Note that the positive-part Lindley’s estimator in (8) is
essentially a single-cluster estimator which shrinks all the
points towards ȳ. Henceforth, we denote it by θ̂JS1 .

ρ = 0.15

5

10

15

20

25

τ

30

35

40

45

50

Fig. 2: The asymptotic risk term min{βn , βn /(αn + σ 2 )} for the
two-cluster estimator is plotted vs τ for n = 1000, σ = 1, and
different values of ρ. Here, the components of θ take only two
values, τ and −ρτ . The number of components taking the value
τ is bnρ/(1 + ρ)c.

(1) For any  > 0,
P

kθ − θ̂JS1 k2
ρn σ 2
≥
−
n
ρn + σ 2

!
≤ Ke−nk min(

2

,1)

,

where K and k are positive constants, and
2

θ − θ̄1
.
(28)
n
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have

1 
ρn σ 2
lim
= 0.
(29)
R θ, θ̂JS1 −
n→∞ n
ρn + σ 2
ρn :=

Remark 3. Statement (2) of Corollary 1, which is known
in the literature [14], implies that θ̂JS+ is asymptotically
minimax over Euclidean balls. Indeed, if Θn denotes the set of
θ such that γn = kθk2 /n ≤ c2 , then Pinsker’s theorem [15,
Ch. 5] implies that the minimax risk over Θn is asymptotically
2 2
c
(as n → ∞) equal to cσ2 +σ
2.
Statement (1) of Corollary 1 and both the statements of
Corollary 2 are new, to the best of our knowledge. Comparing
Corollaries 1 and 2, we observe that ρn ≤ γn since kθ−θ̄1k ≤
kθk for all θ ∈ Rn with strict inequality whenever θ̄ 6= 0.
Therefore the positive-part Lindley’s estimator asymptotically
dominates the positive part JS-estimator.
It is well known that both θ̂JS+ and θ̂JS1 dominate the MLestimator [11]. From Corollary 1, it is clear that asymptotically,
the normalized risk of θ̂JS+ is small when γn is small, i.e.,
when θ is close to the origin. Similarly, from Corollary 2, the
asymptotic normalized risk of θ̂JS1 is small when ρn is small,
which occurs when the components of θ are all very close to
the mean θ̄. It is then natural to ask if the two-cluster estimator
θ̂JS2 dominates θ̂M L , and when its asymptotic normalized risk
is close to 0. To answer these questions, we use the following
example, shown in Fig. 2. Consider θ whose components take
one of two values, τ or −ρτ , such that θ̄ is as close to zero as
possible. Hence the number of components taking the value
τ is bnρ/(1 + ρ)c. Choosing σ = 1, n = 1000, the key
asymptotic risk term min{βn , βn /(αn + σ 2 )} in Theorem 1
is plotted as a function of τ in Fig. 2 for various values of ρ.

Two important observations can be made from the plots.
Firstly, min{βn , βn /(αn + σ 2 )} exceeds σ 2 = 1 for certain
values of ρ and τ . Hence, θ̂JS2 does not dominate θ̂M L .
Secondly, for any ρ, the normalized risk of θ̂JS2 goes to
zero for large enough τ . Note that when τ is large, both
γn = kθk2 /n and ρn = kθ − θ̄1k2 /n are large and hence,
the normalized risks of both θ̂JS+ and θ̂JS1 are close to 1.
So, although θ̂JS2 does not dominate θ̂M L , θ̂JS+ or θ̂JS2 ,
there is a range of θ for which R(θ, θ̂JS2 ) is much lower than
both R(θ, θ̂JS+ ) and R(θ, θ̂JS1 ). This serves as motivation for
designing a hybrid estimator that attempts to pick the better
of θ̂JS1 and θ̂JS2 for the θ in context. This is described in
the next section.
In the example of Fig. 2, it is worth examining why the
two-cluster estimator performs poorly for a certain range of
τ , while giving significantly risk reduction for large enough
τ . First consider an ideal case, where it is known which
components of theta are equal to τ and which ones are equal
to −ρτ (although the values of ρ, τ may not be known). In
this case, we could use a James-Stein estimator θ̂JSV of the
form (5) with the target subspace V being the two-dimensional
subspace with basis vectors




1{θ1 =τ }
1{θ1 =−ρτ }
 1{θ2 =τ } 
 1{θ2 =−ρτ } 




u1 := 
 , u2 := 
.
..
..




.
.
1{θn =τ }

1{θn =−ρτ }

Since V is a fixed subspace that does not depend on the data,
it can be shown that θ̂JSV dominates the ML-estimator [7],
[8]. In the actual problem, we do not have access to the ideal
basis vectors u1 , u2 , so we cannot use θ̂JSV . The two-cluster
estimator θ̂JS2 attempts to approximate θ̂JSV by choosing the
target subspace from the data. As shown in (10), this is done
using the basis vectors:




1{y1 ≥ȳ}
1{y1 <ȳ}
 1{y2 ≥ȳ} 
 1{y2 <ȳ} 




:=
b 1 := 
b
u
,
u


.
..
..
2




.
.
1{yn ≥ȳ}

1{yn <ȳ}

7

Since ȳ is a good approximation for θ̄ = 0, when the
separation between θi and θ̄ is large enough, the noise term wi
is unlikely to pull yi = θi + wi into the wrong region; hence,
b1, u
b 2 will be close to the ideal
the estimated basis vectors u
ones u1 , u2 . Indeed, Fig. 2 indicates that when the minimum
separation between θi and θ̄ (here, equal to ρτ ) is at least
b1, u
b 2 approximate the ideal basis vectors very
4.5σ, then u
well, and the normalized risk is close to 0. On the other hand,
the approximation to the ideal basis vectors turns out to be
poor when the components of θ are neither too close to nor
too far from θ̄, as evident from Remark 1.

where a1 , a2 are defined in (21). We now use (34) and (35)
to estimate the concentrating value in (22), noting that


βn σ 2
βn σ 2
=
min βn ,
,
αn + σ 2
g(αn + σ 2 )
where g(x) = max(x, σ 2 ). This yields the following estimate
of L(θ, θ̂JS2 )/n:
1
L̂(θ, θ̂JS2 ) =
n 
2
σ 2 n1 ky − ν2 k − σ 2 +

σ2
nδ (a1

− a2 )

Pn

i=0

1{|yi −ȳ|≤δ}



g(ky − ν2 k2 /n)
III. H YBRID JAMES -S TEIN ESTIMATOR WITH UP TO TWO
CLUSTERS

Depending on the underlying θ, either the positive-part
Lindley estimator θ̂JS1 or the two-cluster estimator θ̂JS2 could
have a smaller loss (cf. Theorem 1 and Corollary 2). So we
would like an estimator that selects the better among θ̂JS1 and
θ̂JS2 for the θ in context. To this end, we estimate the loss
of θ̂JS1 and θ̂JS2 based on y. Based on these loss estimates,
denoted by L̂(θ, θ̂JS1 ) and L̂(θ, θ̂JS2 ) respectively, we define
a hybrid estimator as
θ̂JSH = γy θ̂JS1 + (1 − γy )θ̂JS2 ,

(30)

where θ̂JS1 and θ̂JS2 are respectively given by (8) and (11),
and γy is given by

1
1
1
if
n L̂(θ, θ̂JS1 ) ≤ n L̂(θ, θ̂JS2 ),
γy =
0 otherwise.
(31)
The loss function estimates L̂(θ, θ̂JS1 ) and L̂(θ, θ̂JS2 ) are
obtained as follows. Based on Corollary 2, the loss function
of θ̂JS1 can be estimated via an estimate of ρn σ 2 /(ρn + σ 2 ),
where ρn is given by (28). It is straightforward to check, along
the lines of the proof of Theorem 1, that
!
2

ky − ȳ1k
.
g
= g ρn + σ 2 = ρn + σ 2 .
(32)
n
Therefore, an estimate of the normalized loss L(θ, θ̂JS1 )/n is


2
1
σ
 .
L̂(θ, θ̂JS1 ) = σ 2 1 − 
(33)
2
n
g ky − ȳ1k /n
The loss function of the two-cluster estimator θ̂JS2 can be
estimated using Theorem 1, by estimating βn and αn defined
in (25) and (24), respectively. From Lemma 13 in Section
VI-B, we have
1
2 .
ky − ν2 k = αn + σ 2 + κn δ + o(δ).
(34)
n
Further, using the concentration inequalities in Lemmas 1 and
2 in Section II, we can deduce that
!
n
1
σ2 X
2
2
ky − ν2 k − σ +
1{|yi −ȳ|≤δ} (a1 − a2 )
n
nδ i=0
.
= βn + κn δ + o(δ),
(35)

.

(36)
The loss function estimates in (33) and (36) complete the
specification of the hybrid estimator in (30) and (31). The
following theorem characterizes the loss function of the hybrid
estimator, by showing that the loss estimates in (33) and (36)
concentrate around the values specified in Corollary 2 and
Theorem 1, respectively.
Theorem 2. The loss function of the hybrid JS-estimator in
(30) satisfies the following:
(1) For any  > 0,
P

kθ − θ̂JSH k2
− min
n
!

kθ − θ̂JS1 k2 kθ − θ̂JS2 k2
,
n
n

!

nk min(2 ,1)

≥

− max(kθk2 /n,1)

≤ Ke

,

where K and k are positive constants.
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have

1 
lim sup
R θ, θ̂JSH
n→∞ n
h 


i 
− min R θ, θ̂JS1 , R θ, θ̂JS2
≤ 0.
The proof of the theorem in given in Section VI-C. The
theorem implies that the hybrid estimator chooses the better of
the θ̂JS1 and θ̂JS2 with high probability, with the probability
of choosing the worse estimator decreasing exponentially in
n. It also implies that asymptotically, θ̂JSH dominates both
θ̂JS1 and θ̂JS2 , and hence, θ̂M L as well.
Remark 4. Instead of picking one among the two (or several)
candidate estimators, one could consider a hybrid estimator
which is a weighted combination of the candidate estimators.
Indeed, George [8] and Leung and Barron [9], [10] have
proposed combining the estimators using exponential mixture
weights based on Stein’s unbiased risk estimates (SURE) [5].
Due to the presence of indicator functions in the definition of
the attracting vector, it is challenging to obtain a SURE for
θ̂JS2 . We therefore use loss estimates to choose the better
estimator. Furthermore, instead of choosing one estimator
based on the loss estimate, if we were to follow the approach
in [9] and employ a combination of the estimators using
exponential mixture weights based on the un-normalized loss
estimates, then the weight assigned to the estimator with the

8

smallest loss estimate is exponentially larger (in n) than the
other. Therefore, when the dimension is high, this is effectively
equivalent to picking the estimator with the smallest loss
estimate.

for 1 ≤ j ≤ L. With δ > 0 chosen to be a small positive
number as before, this completes the specification of the
attracting vector in (38), and hence the L-cluster JS-estimator
in (39).

IV. G ENERAL MULTIPLE - CLUSTER JAMES -S TEIN

Theorem 3. The loss function of the L-cluster JS-estimator
in (39) satisfies the following:
(1) For any  > 0,



1
βn,L σ 2
P
kθ − θ̂JSL k2 − min βn,L ,
n
αn,L + σ 2
!

ESTIMATOR

In this section, we generalize the two-cluster estimator of
Section II to an L-cluster estimator defined by an arbitrary
partition of the real line. The partition is defined by L − 1
functions sj : Rn → R, such that
.
sj (y) := sj (y1 , · · · , yn ) = µj , ∀j = 1, · · · , (L − 1), (37)
with constants µ1 > µ2 > · · · > µL−1 . In words, the partition
can be defined via any L − 1 functions of y, each of which
concentrates around a deterministic value as n increases. In the
two-cluster estimator, we only have one function s1 (y) = ȳ,
which concentrates around θ̄. The points in (37) partition the
real line as
R = (−∞, sL−1 (y)]∪(sL−1 (y), sL−2 (y)]∪· · ·∪(s1 (y), ∞) .
The clusters are defined as Cj = {yi , 1 ≤ i ≤ n | yi ∈
(sj (y), sj−1 (y)]}, for 1 ≤ j ≤ L, with s0 (y) = ∞ and
sL (y) = −∞. In Section IV-B, we discuss one choice of
partitioning points to define the L clusters, but here we first
construct and analyse an estimator based on a general partition
satisfying (37).
The points in Cj are all shrunk towards the same point aj ,
defined in (41) later in this section. The attracting vector is


1{y1 ∈Cj }
L
X


..
(38)
νL :=
aj 
,
.
j=1

1{yn ∈Cj }

and the proposed L-cluster JS-estimator is


σ2
(y − νL ) ,
θ̂JSL = νL + 1 −
g (ky − ν2 k2 /n)

(39)

where g(x) = max(σ 2 , x).
The attracting vector νL lies in an L-dimensional
subspace defined by the L orthogonal vectors
[1{y1 ∈C1 } , · · · , 1{yn ∈C1 } ]T , . . . , [1{y1 ∈CL } , · · · , 1{yn ≤CL } ]T .
The desired values for a1 , . . . , aL in (38) are such that
the attracting vector νL is the projection of θ onto the
L-dimensional subspace. Computing this projection, we find
the desired values to be the means of the θi ’s in each cluster:
Pn
Pn
θi 1{yi ∈CL }
i=1 θi 1{yi ∈C1 }
des
des
P
, . . . , aL = Pi=1
.
a1 =
n
n
1
i=1 {yi ∈C1 }
i=1 1{yi ∈CL }
(40)
As the θi ’s are unavailable, we set a1 , . . . , aL to be approxides
mations of ades
1 , . . . , aL , obtained using concentration results
similar to Lemmas 1 and 2 in Section II. The attractors are
given by
Pn
yi 1{yi ∈Cj }
aj = Pi=1
n
i=1 1{yi ∈Cj }
 (41)
Pn 
σ2
i=0 1{|yi −sj (y)|≤δ} − 1{|yi −sj−1 (y)|≤δ}
2δ
Pn
−
i=1 1{yi ∈Cj }

nk min(2 ,1)

− κn δ + o(δ) ≥ 

− max(kθk2 /n,1)

≤ Ke

,

where K and k are positive constants, and
kθk2
n



n  
L
X
c2j X
µj − θi
µj−1 − θi
−
Q
−Q
n i=1
σ
σ
j=0

L
n 
(µj−1 −θi )2
2σ X X − (µj −θ2i )2
2σ
cj
e
− e− 2σ2
− √
,
n 2π j=1 i=1
(42)
kθk2
βn,L :=
n



L
n  
X
c2j X
µj − θi
µj−1 − θi
−
Q
−Q
,
n i=1
σ
σ
j=0

αn,L :=

(43)
with


i
h 
µ
−θ
µ −θ
Q j σ i − Q j−1σ i
h 


i ,
cj := P
µj −θi
µj−1 −θi
n
−
Q
i=1 Q
σ
σ
Pn

i=1 θi

(44)

for 1 ≤ j ≤ L.
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have



σ 2 βn,L
1 
R θ, θ̂JSL − min βn,L ,
lim
n→∞ n
αn,L + σ 2
− κn δ + o(δ) = 0.
The proof is similar to that of Theorem 1, and we provide its
sketch in Section VI-D.
To get intuition on how the asymptotic normalized risk
R(θ, θ̂JSL )/n depends on L, consider the four-cluster estimator with L = 4. For the same setup as in Fig. 2, i.e.,
the components of θ take one of two values: τ or −ρτ , Fig.
3a plots the asymptotic risk term min{βn , βn,4 /(αn,4 + 1)}
versus τ for the four-cluster estimator. Comparing Fig. 3a
and Fig. 2, we observe that the four-cluster estimator’s risk
min{βn , βn,4 /(αn,4 + 1)} behaves similarly to the two-cluster
estimator’s risk min{βn , βn,2 /(αn,2 + 1)} with the notable
difference being the magnitude. For ρ = 0.5, 1, the peak value
of βn,4 /(αn,4 + 1) is smaller than that of βn,2 /(αn,2 + 1).
However, for the smaller values of ρ, the reverse is true. This
means that θ̂JS4 can be better than θ̂JS2 , even in certain

9

define a hybrid estimator as

min {βn,4 , βn,4 /(αn,4 + 1)}

1.4
1.2

θ̂JSH,L =

1
0.8

ρ = 0.5

ρ = 0.1

ρ=1



ρ = 0.25

γ` =

0
−0.2
0

5

10

15

20

25

30

τ

35

40

45

50

(a)

min {βn,4 , βn,4 /(αn,4 + 1)}

ρ = 0.25
0.8

ρ = 0.5
ρ = 0.1

0.4

ρ = 0.15
0.2

0
0

5

10

15

20

τ

25

1
0

if n1 L̂(θ, θ̂JS` ) = min1≤k≤L
otherwise

1
n L̂(θ, θ̂JSk ),

with L̂(θ, θ̂JS` ) denoting the loss function estimate of θ̂JS` .
For ` ≥ 2, we estimate the loss of θ̂JS` using Theorem
3, by estimating βn,` and αn,` which are defined in (42) and
(43), respectively. From (78) in Section VI-D, we obtain
1
2 .
ky − ν` k = αn,` + σ 2 + κn δ + o(δ).
(46)
n
Using concentration inequalities similar to those in Lemmas
1 and 2 in Section II, we deduce that

`
n
2
X
X

1
σ
2

ky − ν` k − σ 2 +
aj
1{|yi −sj (y)|≤δ}
n
nδ j=1 i=0
!
 .
− 1{|yi −sj−1 (y)|≤δ}
= βn,` + κn δ + o(δ),
(47)

1

0.6

(45)

where

ρ = 0.15
0.2

γ` θ̂JS`

`=1

0.6
0.4

L
X

30

35

40

(b)

Fig. 3: The asymptotic risk term min{βn , βn,4 /(αn,4 + 1)} for the
four-cluster estimator is plotted vs τ , for n = 1000, σ = 1, and
different values of ρ. In (a), the components of θ take only two
values τ and −ρτ , with b1000ρ/(1 + ρ)c components taking the
value τ . In (b), they take values from the set {τ, ρτ, −ρτ, −τ } with
equal probability.

scenarios where the θi take only two values. In the twovalue example, θ̂JS4 is typically better when two of the four
attractors of θ̂JS4 are closer to the θi values, while the two
attracting points of θ̂JS2 are closer to θ̄ than the respective θi
values.
Next consider an example where θi take values from
{τ, ρτ, −ρτ, −τ } with equal probability. This is the scenario favorable to θ̂JS4 . Figure 3b shows the plot of
min{βn , βn,4 /(αn,4 + 1)} as a function of τ for different
values of ρ. Once again, it is clear that when the separation
between the points is large enough, the asymptotic normalized
risk approaches 0.

A. L-hybrid James-Stein estimator
Suppose that we have estimators θ̂JS1 , . . . , θ̂JSL , where
θ̂JS` is an `-cluster JS-estimator constructed as described
above, for ` = 1, . . . , L. (Recall that ` = 1 corresponds to
Lindley’s positive-part estimator in (8).) Depending on θ, any
one of these L estimators could achieve the smallest loss. We
would like to design a hybrid estimator that picks the best
of these L estimators for the θ in context. As in Section III,
we construct loss estimates for each of the L estimators, and

where a1 , . . . , a` are as defined in (41). We now use (46) and
(47) to estimate the concentrating value in Theorem 3, and
thus obtain the following estimate of L(θ, θ̂JS` )/n:
σ2
1
L̂(θ, θ̂JS` ) =
n
g(ky − νl k2 /n)

1
ky − ν` k2 − σ 2
n


`
n

σ2 X X 
+
aj
1{|yi −sj (y)|≤δ} − 1{|yi −sj−1 (y)|≤δ}  .
nδ j=1 i=0
(48)
The loss function estimator in (48) for 2 ≤ ` ≤ L, together
with the loss function estimator in (33) for ` = 1, completes
the specification of the L-hybrid estimator in (45). Using steps
similar to those in Theorem 2, we can show that



1 
σ 2 βn,`
.
L̂ θ, θ̂JS` = min βn,` ,
+ κn δ + o(δ), (49)
n
αn,` + σ 2
for 2 ≤ ` ≤ L.
Theorem 4. The loss function of the L-hybrid JS-estimator in
(45) satisfies the following:
(1) For any  > 0,
!
!
kθ − θ̂JSH,L k2
kθ − θ̂JS` k2
P
− min
≥
1≤`≤L
n
n
nk min(2 ,1)

− max(kθk2 /n,1)

≤ Ke

,

where K and k are positive constants.
(2) For a sequence of θ with increasing dimension n, if
lim supn→∞ kθk2 /n < ∞, we have
 



1
lim sup
R θ, θ̂JSH,L − min R θ, θ̂JS`
≤ 0.
1≤`≤L
n→∞ n
The proof of the theorem is omitted as it is along the same

10

lines as the proof of Theorem 3. Thus with high probability,
the L-hybrid estimator chooses the best of the θ̂JS1 , . . . , θ̂JSL ,
with the probability of choosing a worse estimator decreasing
exponentially in n.
1

R̃(θ, θ̂)/n

B. Obtaining the clusters
In this subsection, we present a simple method to obtain
the (L − 1) partitioning points sj (y), 1 ≤ j ≤ (L − 1), for
an L-cluster JS-estimator when L = 2a for an integer a > 1.
We do this recursively, assuming that we already have a 2a−1 cluster estimator with its associated partitioning points s0j (y),
j = 1, · · · , 2a−1 − 1. This means that for the 2a−1 -cluster
estimator, the real line is partitioned as


R = −∞, s02a−1 −1 (y) ∪ s02a−1 −1 (y), s02a−1 −2 (y) ∪ · · ·

n = 10

0.8
0.6
Regular JS−estimator
Lindley’s estimator
Two−cluster JS−estimator
Hybrid JS−estimator
ML−estimator

0.4
0.2
0
0

1

2

3

4

5

τ

6

7

8

9

10

(a)

∪ (s01 (y), ∞) .

n = 50

i

1

R̃(θ, θ̂)/n

Recall that Section II considered the case of a = 1, with the
single partitioning point being ȳ.
The new partitioning points sk (y), k = 1, · · · , (2a − 1), are
obtained as follows. For j = 1, · · · , (2a−1 − 1), define
Pn
i=1 yi 1{s0j (y)<yi ≤s0j−1 (y)}
0
s2j (y) = sj (y), s2j−1 (y) = Pn
,
i=1 1{s0j (y)<yi ≤s0j−1 (y)}
Pn
n
o
i=1 yi 1 −∞<yi ≤s0
(y)
2a−1 −1
s2a −1 (y) = Pn
n
o
i=1 1 −∞<y ≤s0
(y)

0.8
0.6
0.4

Regular JS−estimator
Lindley’s estimator
Two−cluster JS−estimator
Hybrid JS−estimator
ML−estimator

0.2
0
0

1

2

3

4

where s00 (y) = ∞. Hence, the partition for the L-cluster
estimator is

8

9

10

1

R̃(θ, θ̂)/n

0.8
0.6
0.4

Regular JS−estimator
Lindley’s estimator
Two−cluster JS−estimator
Hybrid JS−estimator
ML−estimator

0.2
0
0

1

2

3

4

5

τ

6

7

8

9

10

(c)
n = 1000

1

R̃(θ, θ̂)/n

V. S IMULATION R ESULTS
In this section, we present simulation plots that compare
the average normalized loss of the proposed estimators with
that of the regular JS-estimator and Lindley’s estimator, for
various choices of θ. In each plot, the normalized loss, labelled
1
n R̃(θ, θ̂) on the Y -axis, is computed by averaging over 1000
realizations of w. We use w ∼ N (0, I), i.e., the noise variance
σ 2 = 1. Both the regular JS-estimator θ̂JS and Lindley’s
estimator θ̂JS1 used are the positive-part √
versions, respectively
given by (7) and (8). We choose δ = 5/ n for our proposed
estimators.
In Figs. 4–7, we consider three different structures for
θ, representing varying degrees of clustering. In the first
structure, the components {θi }ni=1 are arranged in two clusters.
In the second structure for θ, {θi }ni=1 are uniformly distributed
within an interval whose length is varied. In the third structure,
{θi }ni=1 are arranged in four clusters. In both clustered structures, the locations and the widths of the clusters as well as the
number of points within each cluster are varied; the locations
of the points within each cluster are chosen uniformly at
random. The captions of the figures explain the details of each
structure.

7

n = 100

R = (−∞, s2a −1 (y)] ∪ (s2a −1 (y), s2a −2 (y)] ∪ · · ·
We use such a partition to construct a 4-cluster estimator for
our simulations in the next section.

6

(b)

2a−1 −1

∪ (s1 (y), ∞) .

5

τ

0.8
0.6
0.4

Regular JS−estimator
Lindley’s estimator
Two−cluster JS−estimator
Hybrid JS−estimator
ML−estimator

0.2
0
0

1

2

3

4

5

τ

6

7

8

9

10

(d)

Fig. 4: Average normalized loss of various estimators for different
values of n. The {θi }n
i=1 are placed in two clusters, one centred at
τ and another at −τ . Each cluster has width 0.5τ and n/2 points.

11

1.5
Lindley’s estimator

R̃(θ, θ̂)/n

R̃(θ, θ̂)/n

1
0.8
0.6
Regular JS−estimator

1

ρn σ 2
ρn + σ 2

Two−cluster JS−estimator
Hybrid JS−estimator

0.5

Lindley’s estimator

0.4

βn σ 2
= min
αn + σ 2

Two−cluster JS−estimator
Hybrid JS−estimator

0.2

ML−estimator
0
0

1

2

3

4

5

τ

6

7

0 1
10

8

9

2

!

ρn σ 2
βn σ 2
, βn ,
ρn + σ 2
αn + σ 2

"

3

10

n

10

4

10

10

(a)

(a)
1.5
1.4

Lindley’s JS−estimator
Two−cluster JS−estimator

1.2

Hybrid JS−estimator

R̃(θ, θ̂)/n

! "
R̃ θ, θ̂ /n

1
0.8
0.6

Regular JS−estimator

1

ρn σ 2
ρn + σ 2
0.5

Lindley’s estimator
0.4

Two−cluster JS−estimator

!

ρn σ 2
βn σ 2
, βn ,
2
ρn + σ
αn + σ 2

"

Hybrid JS−estimator

0.2
0
0

βn σ 2
= min
αn + σ 2

ML−estimator
1

2

3

4

5

τ

6

0 1
10
7

8

9

2

3

10

n

10

4

10

10

(b)

(b)
0.5

R̃(θ, θ̂)/n

R̃(θ, θ̂)/n

0.8
Regular JS−estimator
Lindley’s estimator

0.6

Two−cluster JS−estimator
Hybrid JS−estimator

0.4

ML−estimator

1

2

3

4

5

τ

6

7

8

Hybrid JS−estimator

0.2

0.1

9

ρn σ 2
= min
ρn + σ 2

0 1
10

10

Two−cluster JS−estimator

0.3

0.2
0
0

Lindley’s estimator

βn σ 2
αn + σ 2

0.4

1

!

ρn σ 2
βn σ 2
, βn ,
2
ρn + σ
αn + σ 2

2

"

3

10

n

4

10

10

(c)

(c)
1.5

Lindley’s estimator

R̃(θ, θ̂)/n

R̃(θ, θ̂)/n

1
0.8
Regular JS−estimator

0.6

Lindley’s estimator
0.4

1

Two−cluster JS−estimator
Hybrid JS−estimator

0.5

ρn σ 2
= min
ρn + σ 2

Two−cluster JS−estimator
Hybrid JS−estimator

0.2

βn σ 2
αn + σ 2

!

ρn σ 2
βn σ 2
, βn ,
ρn + σ 2
αn + σ 2

"

ML−estimator
0
0

1

2

3

4

5

τ

6

7

8

9

10

0 1
10

arrangements of the samples {θi }n
i=1 , with n = 1000. In (a), there
are 2 clusters each of width 0.5τ , one around 0.25τ containing 300
points, and the other around −τ and containing 700 points. In (b),
θ consists of 200 components taking the value τ and the remaining
800 taking the value −0.25τ . In (c), there are two clusters of width
0.125τ , one around τ containing 300 points and another around −τ
containing 700 points. In (d), {θi }n
i=1 are arranged uniformly from
−τ to τ .

3

n

10

4

10

(d)

(d)

Fig. 5: Average normalized loss of various estimators for different

2

10

Fig. 6: Average normalized loss of various estimators versus
βn
,
αn +σ 2

βn
n
min( ρnρ+σ
2 , βn , α +σ 2 )
n
n
arrangements of {θi }i=1 . In (a),

ρn
,
ρn +σ 2

as a function of n for different
the {θi }n
i=1 are placed in two
clusters of width 1, one around 2 and the other around −2, each
containing an equal number of points. In (b), {θi }n
i=1 are placed in
two clusters of width 1.25, one around 5 and the other around −5,
each containing an equal number of points. In (c), {θi }n
i=1 are placed
in two clusters of width 0.25, one around 0.5 and the other around
−0.5, each containing an equal number of points. In (d), {θi }n
i=1 are
placed uniformly between −2 and 2.

12

VI. P ROOFS

1

R̃(θ, θ̂)/n

0.8

A. Mathematical preliminaries

0.6

Lindley’s estimator
Two−cluster JS−estimator
Four−cluster JS−estimator
Four−hybrid JS−estimator
ML−estimator

0.4

0.2

0
0

1

2

3

4

5

τ

Here we list some concentration results that are used in the
proofs of the theorems.
6

7

8

9

10

(a)

Lemma 3. Let {Xn (θ), θ ∈ Rn }∞
n=1 be a sequence of random
.
variables such that Xn (θ) = 0, i.e., for any  > 0,
nk min(2 ,1)

P(|Xn (θ)| ≥ ) ≤ Ke

1

R̃(θ, θ̂)/n

,
:=

where K and k are positive constants. If C
a.s.
lim supn→∞ kθk2 /n < ∞, then Xn −→ 0.

0.8

0.6

Proof. For any τ > 0, there exists a positive integer M such
that ∀n ≥ M , kθk2 /n < C + τ . Hence, we have, for any
 > 0, and for some τ > 1,

Lindley’s estimator

0.4

Two−cluster JS−estimator
Four−cluster JS−estimator

0.2

∞
X

Four−hybrid JS−estimator
ML−estimator

0
0

− max(kθk2 /n,1)

1

2

3

4

5

τ

6

7

8

9

10

P(|Xn (θ)| ≥ ) ≤

∞
X

nk min(2 ,1)

n=1

n=1

(b)

≤ C0 +

Fig. 7: Average normalized loss of various estimators for n = 1000
{θi }n
i=1 .

{θi }ni=1

In Fig. 4,
are arranged in two clusters, one centred
at −τ and the other at τ . The plots show the average
normalized loss as a function of τ for different values of n,
for four estimators: θ̂JS , θ̂JS1 , the two-attractor JS-estimator
θ̂JS2 given by (11), and the hybrid JS-estimator θ̂JSH given
by (30). We observe that as n increases, the average loss of
θ̂JSH gets closer to the minimum of that of θ̂JS1 and θ̂JS2 ;
Fig. 5 shows the the average normalized loss for different
arrangements of {θi }, with n fixed at 1000. The plots illustrate
a few cases where θ̂JS2 has significantly lower risk than θ̂JS1 ,
and also the strength of θ̂JSH when n is large.
Fig. 6 compares the average normalized losses of θ̂JS1 ,
θ̂JS2 , and θ̂JSH with their asymptotic risk values, obtained
in Corollary 2, Theorem 1, and Theorem 2, respectively. Each
subfigure considers a different arrangement of {θi }ni=1 , and
shows how the average losses converge to their respective
theoretical values with growing n.
Fig. 7 demonstrates the effect of choosing four attractors
when {θi }ni=1 form four clusters. The four-hybrid estimator
θ̂JSH,4 attempts to choose the best among θ̂JS1 , θ̂JS2 and
θ̂JS4 based on the data y. It is clear that depending on the
values of {θi }, θ̂JSH,4 reliably tracks the best of these. and
can have significantly lower loss than both θ̂JS1 and θ̂JS2 ,
especially for large values of n.

∞
X

Ke−

nk min(2 ,1)
C+τ

< ∞.

n=M

{θi }n
i=1

In (a),
and for different arrangements of the samples
are placed in four equal-sized clusters of width 0.5τ and in (b), the
clusters are of width 0.25τ . In both cases, the clusters are centred at
1.5τ , 0.9τ , −0.5τ and −1.25τ .

− max(kθk2 /n,1)

Ke

Therefore, we can use the Borel-Cantelli lemma to conclude
a.s.
that Xn −→ 0.

Lemma 4. For sequences of random variables {Xn }∞
n=1 ,
.
.
{Yn }∞
n=1 such that Xn = 0, Yn = 0, it follows that
.
Xn + Yn = 0.
nk

Proof. For  > 0, if P(|Xn | ≥ ) ≤ K1 e
nk

min(2 ,1)

1
− max(kθk
2 /n,1)

and

min(2 ,1)

2
− max(kθk
2 /n,1)

P(|Yn | ≥ ) ≤ K2 e
for positive constants K1 ,
K2 , k1 and k2 , then by the triangle inequality,




P(|Xn + Yn | ≥ ) ≤ P |Xn | ≥
+ P |Yn | ≥
2
2
nk min(2 ,1)

− max(kθk2 /n,1)

≤ Ke

where K = K1 + K2 and k = min

k1 k2
4 . 4



.

Lemma 5. Let X and Y be random variables such that for
any  > 0,
2
P (|X − aX | ≥ ) ≤ K1 e−nk1 min( ,1) ,
2
P (|Y − a | ≥ ) ≤ K e−nk2 min( ,1)

Y

2

where k1 , k2 are positive constants, and K1 , K2 are positive
integer constants. Then,
P (|XY − aX aY | ≥ ) ≤ Ke−nk min(

2

,1)

where K = 2(K1 + K2 ), and k is a positive constant
depending on k1 and k2 .

13

Proof. We have
P (|XY − aX aY | ≥ )
= P (|(X − aX )(Y − aY ) + XaY + Y aX − 2aX aY | ≥ )


≤ P |(X − aX )(Y − aY )| ≥
2


+ P |XaY + Y aX − 2aX aY | ≥
2


≤ P |(X − aX )(Y − aY )| ≥
2 



+ P |(X − aX )aY | ≥
+ P |(Y − aY )aX | ≥
r 4
r 4




≤ P |(X − aX )| ≥
+ P |(Y − aY )| ≥
2
2




+ P |(Y − aY )aX | ≥
+ P |(X − aX )aY | ≥
4
4
−nk100 min(2 ,1)
−nk10 min(,1)
−nk20 min(,1)
≤ K1 e
+ K2 e
+ K1 e
−nk200 min(2 ,1)
−nk min(2 ,1)
+K e
≤ Ke
2

k1
2 ,
min(k1 ,k2 )
16(aY )2 .

where k10 =
k=

k20 =

k2
2 ,

k100 =

k1
16(aY )2 ,

k2
16(aY )2 ,

k200 =

and

Y > 0. Therefore,
!


1
1
1
−
≤ − = P Y ≥ 1
P
Y
aY
a −
!Y
1
= P Y − aY ≥ 1
− aY
aY − 



aY
= P Y − aY ≥ aY
1 − aY
≤ K2 e


2 
(aY )2
−nk2 min
,1
1−a
Y

0

2

≤ K2 e−nk2 min(

≤ K2 e−nk2 min(

(51)

where k20 = k2 min (aY )4 , 1 . Using (50) and (51), we
obtain, for any  > 0,


2
1
1
≥  ≤ Ke−nk min( ,1)
P
−
Y
aY
where k = min(k10 , k20 ).
Lemma 7. Let {Xn }∞
n=1 be a sequence of random variables
and X be another random variable (or a constant) such that
2
for any  > 0, P (|Xn − X| ≥ ) ≤ Ke−nk min( ,1) for
positive constants K and k. Then, for the function g(x) :=
max(σ 2 , x), we have

Proof. Let An := {ω

2
P (Y − aY ≤ −) ≤ K1 e−nk1 min( ,1) ,
2
P (Y − a ≥ ) ≤ K e−nk2 min( ,1)

Y

where K = K1 + K2 , and k is a positive constant.
Proof. We have




1
1
1
1
P
−
≥ =P
≥
+
Y
aY
Y
a


 Y

1
1
=P Y ≤
= P Y − aY ≤
− aY
 + 1/aY
 + 1/aY



aY
= P Y − aY ≤ −aY
1 + aY

−nk1 min

(aY )2
1+aY

2 
,1

0

2

≤ K1 e−nk1 min(

2 
(aY )2
where k10 = k1 min
, 1 . Similarly
1+aY
≤ K1 e


P

Note that when  >

1
aY



(50)


1
1
=P
≤
− .
Y
aY


, P Y1 − a1Y ≤ − = 0 because

1
1
−
≤ −
Y
aY



,1)

,1)

.

P (|g(Xn ) − g(X)| ≥ )
= P (An ) P |g(Xn ) − g(X)| ≥ 

n

where k1 , k2 are positive constants, and K1 , K2 are positive
integer constants. Then, for any  > 0,


2
1
1
P
−
≥  ≤ Ke−nk min( ,1)
Y
aY

2

|Xn (ω) − X(ω)| ≥ }. We have
An




+ P (Acn ) P |g(Xn ) − g(X)| ≥  Acn
2
≤ Ke−nk min( ,1) + P |g(X ) − g(X)| ≥ 

2

(aY )4 ,1)

,1)

P (|g(Xn ) − g(X)| ≥ ) ≤ Ke−nk min(
Lemma 6. Let Y be a non-negative random variable such
that there exists aY > 0 such that for any  > 0,

2


Acn . (52)

Now, when Xn ≥ σ 2 and X ≥ σ 2 , it follows that
g(Xn ) − g(X) = Xn − X, and the second term of the
RHS of (52) equals 0, as it also does when Xn < σ 2 and
X < σ 2 . Let us consider the case where Xn ≥ σ 2 and
X < σ 2 . Then, g(Xn ) − g(X) = Xn − σ 2 < Xn − X < ,
as we condition on the fact that |Xn − X| < ; hence in
this case P |g(Xn ) − g(X)| ≥  Acn = 0. Finally, when
Xn < σ 2 and X ≥ σ 2 , we have g(X) − g(Xn ) =
X − σ 2 < X − Xn < ; hence
 in this case also we
have P |g(Xn ) − g(X)| ≥  Acn = 0. This proves the
lemma.
Lemma 8. (Hoeffding’s Inequality [16, Thm. 2.8]). Let
X1 , · · · , Xn be independent random variables such
Pnthat Xi ∈
[ai , bi ] almost surely, for all i ≤ n. Let Sn =
i −
i=12(X


2
− Pn 2n
2
|Sn |
E[Xi ]). Then for any  > 0, P n ≥  ≤ 2e i=1 (bi −ai ) .
Lemma 9. (Chi-squared concentration [16]). For i.i.d. Gaussian random variables w1 , . . . , wn ∼ N (0, σ 2 ), we have for
any  > 0,
!
n
2
1X 2
2
P
wi − σ ≥  ≤ 2e−nk min(, ) ,
n i=1

where k = min 4σ1 4 , 2σ1 2 .

14

Lemma 10. For i = 1, · · · , n, let wi ∼ (0, σ 2 ) be independent, and ai be real-valued and finite constants. We have for
any  > 0,
!
n
n
σ X − a2i2
1 X
P
wi 1{wi >ai } − √
e 2σ ≥ 
n i=1
2π i=1
2

≤ 2e−nk1 min(, ) ,
!
n
n
1 X
σ X − a2i2
P
wi 1{wi ≤ai } + √
e 2σ ≥ 
n i=1
2π i=1
≤ 2e−nk2 min(,

2

)

(53)

(54)

where k1 and k2 are positive constants.

The proof is given in Appendix A.

−nk2

P (|f (y) − a| ≥ ) ≤ 2e

for some constants a, k, such that k > 0. Then for any  > 0,
we have
!
n
n
X
2
1 X
P
1{yi >a} ≥  ≤ 4e−nk1  ,
1{yi >f (y)} −
n i=1
i=1
(55)
!
n
n
nk1 2
X
1 X
−
P
θi 1{yi >f (y)} −
θi 1{yi >a} ≥  ≤ 4e kθk2 /n ,
n i=1
i=1
(56)
!
n
n
X
X
1
wi 1{yi >a} ≥ 
wi 1{yi >f (y)} −
P
n i=1
i=1
2

,)

(57)

where k1 is a positive constant.

The proof is given in Appendix B.

Lemma 12. With the assumptions of Lemma 11, let h : Rn →
R be a function such that b > a and P (|h(y) − b| ≥ ) ≤
2
2e−nl for some l > 0. Then for any  > 0, we have
!
n
n
X
1 X
P
1{h(y)≥yi >f (y)} −
1{b≥yi >a} ≥ 
n i=1
i=1
≤ 8e

−nk2

We have,
1
kθ − θ̂JS2 k2
n


2
σ2
1
θ − ν2 − 1 −
(y
−
ν
)
=
2
n
g (ky − ν2 k2 /n)


1
σ2
=
y − ν2 − 1 −
(y − ν2 ) − w
n
g (ky − ν2 k2 /n)


2
1
σ2
=
(y
−
ν
)
−
w
2
n
g (ky − ν2 k2 /n)
=


Lemma 11. Let y ∼ N θ, σ 2 I , and let f : Rn → R be a
function such that for any  > 0,

≤ 4e−nk1 min(

B. Proof of Theorem 1

.

Proof. The result follows from Lemma 11 by noting that
1{h(y)≥yi >f (y)} = 1{yi >f (y)} − 1{yi >h(y)} , and 1{b≥yi >a} =
1{yi >a} − 1{yi >b} .

2

2

σ 4 ky − ν2 k2 /n
2

+

(g (ky − ν2 k2 /n))

2
σ2
−
n g (ky − ν2 k2 /n)

kwk
n

hy − ν2 , wi .

(58)

We also have
1
1
2
2
kθ − ν2 k = ky − ν2 − wk
n
n
1
1
2
2
2
= ky − ν2 k + kwk − hy − ν2 , wi
n
n
n
and so,
2
1
1
1
2
2
2
− hy − ν2 , wi = kθ − ν2 k − ky − ν2 k − kwk .
n
n
n
n
(59)
Using (59) in (58), we obtain
kθ − θ̂JS2 k2
n


2
kwk
σ2
σ 4 ky − ν2 k2 /n
+
+
=
2
n
g (ky − ν2 k2 /n)
(g (ky − ν2 k2 /n))


1
1
1
2
2
2
×
kθ − ν2 k − ky − ν2 k − kwk .
(60)
n
n
n
We now use the following results whose proofs are given in
Appendix F and Appendix G.
Lemma 13.
1
2 .
ky − ν2 k = αn + σ 2 + κn δ + o(δ).
n
where αn is given by (25).

(61)

Lemma 14.
1
2 .
kθ − ν2 k = βn + κn δ + o(δ)
n
where βn is given by (24).
Using Lemma 7 together with (61), we have
!
2
ky − ν2 k
.
g
= g(αn + σ 2 ) + κn δ + o(δ).
n

(62)

(63)

Using (61), (62) and (63) together with Lemmas 5, 6, and 9,

15

we obtain
4

2

2



2





kθ − θ̂JS2 k . σ αn + σ
σ
2
=
2 +σ +
n
g (αn + σ 2 )
(g (αn + σ 2 ))


× β 2 − αn + σ 2 − σ 2 + κn δ + o(δ) (64)
(
βn + κn δ + o(δ),
if αn < 0,
=
βn σ 2
αn +σ 2 + κn δ + o(δ) otherwise.
Therefore, for any  > 0,


kθ − θ̂JS2 k2
βn σ 2
− min βn ,
n
αn + σ 2

nk min(2 ,1)
− kθk2 /n
+ κn δ + o(δ) ≥  ≤ Ke
.

P

This proves (22) and hence, the first part of the theorem.
To prove the second part of the theorem, we use the
following definition and result.
Definition VI.1. (Uniform Integrability [17, p. 81]) A sequence {Xn }∞
n=1 is said to be uniformly integrable (UI) if




lim lim sup E |Xn |1{|Xn |≥K} = 0.
(65)
K→∞

Now, using (60) and (64), we can write




kθ − θ̂JS2 k2
βn σ 2
+ κn δ + o(δ)
− min βn ,
n
αn + σ 2
!
2
kwk
2
= Sn + Tn − Un − Vn +
−σ .
n
Note from Jensen’s inequality that |E[Xn ] − EX| ≤ E(|Xn −
X|). We therefore have




1
βn σ 2
+
κ
δ
+
o(δ)
R(θ, θ̂JS2 ) − min βn ,
n
n
αn + σ 2




kθ − θ̂JS2 k2
βn σ 2
≤E
+ κn δ + o(δ)
− min βn ,
n
αn + σ 2
= E Sn + Tn − Un − Vn +

≤ E |Sn | + E |Tn | − E |Un | − E |Vn | + E

L1

kwk2
n

−→ σ 2 , i.e.,


kwk2
2
lim E
−σ
= 0.
n→∞
n

We first show that

1
Fact 1. [18, Sec. 13.7] Let {Xn }∞
n=1 be a sequence in L ,
L1

equivalently E|Xn | < ∞, ∀n. Also, let X ∈ L . Then Xn −→
X, i.e., E(|Xn − X|) → 0, if and only if the following two
conditions are satisfied:

kwk2
− σ2 .
n
(68)

n→∞

1

kwk2
− σ2
n

(69)

Now, consider the individual terms of the RHS of (60).
Using Lemmas 5, 6 and 7, we obtain

σ 4 ky − ν2 k2 /n . σ 4 αn + σ 2
=
2
2 + κn δ + o(δ),
(g (ky − ν2 k2 /n))
(g (αn + σ 2 ))

This holds because


 Z ∞ 
kwk2
kwk2
2
2
E
−σ
=
− σ > x dx
P
n
n
0
Z
Z
1
∞
(i)
2
≤
2e−nkx dx +
2e−nkx dx
0
1
Z ∞
Z ∞
2
≤
2e−nkx dx +
2e−nkx dx
0
0
Z ∞
Z ∞
2
2
n→∞
−t2
e dt +
e−t dt −→ 0,
=√
nk 0
nk 0

and so, from Lemma 3,

where inequality (i) is due to Lemma 9.

P

1) Xn −→ X,
2) The sequence {Xn }∞
n=1 is UI.

σ 4 ky − ν2 k2 /n

Sn :=

2

(g (ky − ν2 k2 /n))
#
"

σ 4 αn + σ 2
a.s.
−
2 + κn δ + o(δ) −→ 0.
(g (αn + σ 2 ))

(66)

Similarly, we obtain
2

σ 2 kθ − ν2 k /n
g (ky − ν2 k2 /n)


βn σ 2
a.s.
−
+ κn δ + o(δ) −→ 0,
g(αn + σ 2 )

Tn :=

and since the sum of the terms in (66) that involve δ have
bounded absolute value for a chosen and fixed δ (see Note 1),
there exists M > 0 such that ∀n, |Sn | ≤ 2σ 2 + M . Hence,
from Definition VI.1, {Sn }∞
n=1 is UI. By a similar argument,
so is {Un }∞
.
Next,
considering
Vn , we have
n=1

2

σ 2 ky − ν2 k /n
g (ky − ν2 k2 /n)

 2
σ (αn + σ 2 )
a.s.
−
+ κn δ + o(δ) −→ 0,
g(αn + σ 2 )

Un :=

2

σ 2 kwk /n
g (ky − ν2 k2 /n)


σ4
a.s.
−
+ κn δ + o(δ) −→ 0.
g(αn + σ 2 )

Vn :=

Thus, from (68), to prove (23), it is sufficient to show that
E |Sn |, E |Tn |, E |Un | and E |Vn | all converge to 0 as n →
∞. From Fact 1 and (66), (67), this implies that we need
∞
∞
∞
to show that {Sn }∞
n=1 , {Tn }n=1 , {Un }n=1 , {Vn }n=1 are UI.
Considering Sn , we have

σ 4 αn + σ 2
σ 4 ky − ν2 k2 /n
2
2
∀n,
2 ≤σ ,
2 ≤σ ,
(g (ky − ν2 k2 /n))
(g (αn + σ 2 ))

(67)

2

2

σ 2 kwk /n
kwk
≤
,
g (ky − ν2 k2 /n)
n
2

σ4
≤ σ2 ,
g(αn + σ 2 )

∀n,

+ σ 2 + M , ∀n. Note from (69) and
and hence, |Vn | ≤ kwk
n
Fact 1 that {kwk2 /n}∞
n=1 is UI. To complete the proof, we
use the following result whose proof is provided in Appendix
C.

16

Lemma 15. Let {Yn }∞
n=1 be a UI sequence of positive-valued
random variables, and let {Xn }∞
n=1 be a sequence of random
variables such that |Xn | ≤ cYn + a, ∀n, where c and a are
positive constants. Then, {Xn }∞
n=1 is also UI.
Hence, {Vn }∞
n=1 is UI. Finally, considering Tn in (67), we see
that


2
2
2
2
2k
2k
2 ky−ν2 −wk
2σ 2 ky−ν
+ kwk
σ 2 kθ−ν
σ
n
n
n
n

= 
 ≤


ky−ν2 k2
ky−ν2 k2
ky−ν2 k2
g
g
g
n
n
n


2
kwk
2
≤2 σ +
,
n
βn σ 2
≤ βn < ∞.
g(αn + σ 2 )
Note that the last inequality is due to the assumption that
2
lim supn→∞ kθk2 /n < ∞. Therefore, |Tn | ≤ 2kwk /n +
2
2σ + M , ∀n, where M is some finite constant. Thus, by
Lemma 15, Tn is UI. Therefore, each of the terms of the RHS
of (68) goes to 0 as n → ∞, and this completes the proof of
the theorem.

for some positive constants k and K. Let P (γy = 0, ∆n > )
denote the probability that γy = 0 and ∆n >  for a chosen
 > 0. Therefore,
P (γy = 0, ∆n > )


1
1
=P
L̂(θ, θ̂JS1 ) > L̂(θ, θ̂JS2 ) , ∆n > 
n
n


1
1
≤P
L̂(θ, θ̂JS1 ) + ∆n > L̂(θ, θ̂JS2 ) + 
n
n
nk min(2 ,1)

≤ Ke

− max(kθk2 /n,1)

,

(74)

where the last inequality is obtained from (73). So for any
 > 0, we have
!
kθ − θ̂JSH k2
kθ − θ̂JS1 k2
P
−
≥  y ∈ En
n
n
nk min(2 ,1)

− max(kθk2 /n,1)

≤ P (γy = 0, ∆n > ) ≤ Ke

.

In a similar manner, we obtain for any  > 0,
kθ − θ̂JS2 k2
kθ − θ̂JSH k2
−
≥
n
n

P
C. Proof of Theorem 2

!
y ∈ Enc

nk min(2 ,1)

− max(kθk2 /n,1)

≤ P (γy = 1, −∆n > ) ≤ Ke

Let
n
o
En := y ∈ Rn : kθ − θ̂JS1 k2 < kθ − θ̂JS2 k2 ,

.

Therefore, we arrive at

kθ − θ̂JS2 k2
kθ − θ̂JS1 k2
−
.
∆n :=
n
n
Without loss of generality, for a given  > 0, we can assume
that |∆n | >  because if not, it is clear that
!
kθ − θ̂JSH k2
kθ − θ̂JS1 k2 kθ − θ̂JS2 k2
− min
,
≤ .
n
n
n

P

kθ − θ̂JSH k2
− min
i=1,2
n

kθ − θ̂JSi k2
n

!

!
≥

nk min(2 ,1)

− max(kθk2 /n,1)

≤ Ke

.

This proves the first part of the theorem.

From (32) and Lemma 6, we obtain the following concentration inequality for the loss estimate in (33):
1
. ρn σ 2
L̂(θ, θ̂JS1 ) =
.
n
ρn + σ 2
Using this together with Corollary 2, we obtain
1
. 1
L̂(θ, θ̂JS1 ) = kθ − θ̂JS1 k2 .
(70)
n
n
Following steps similar to those in the proof of Lemma 13,
we obtain the following for the loss estimate in (36):
1
βn σ 2
.
L̂(θ, θ̂JS2 ) =
+ κn δ + o(δ).
n
g(αn + σ 2 )

(71)

Combining this with Theorem 1, we have
1
. 1
L̂(θ, θ̂JS2 ) = kθ − θ̂JS2 k2 .
n
n

(72)

Then, from (70), (72), and Lemma 4, we have n1 L̂(θ, θ̂JS1 ) −
.
1
n L̂(θ, θ̂JS2 ) = −∆n . We therefore have, for any  > 0,


1
1
P
L̂(θ, θ̂JS1 ) − L̂(θ, θ̂JS2 ) − (−∆n ) ≥ 
n
n
nk min(2 ,1)
− max(kθk2 /n,1)

≤ Ke

(73)

For the second part, fix  > 0. First suppose that θ̂JS1 has
lower risk. For a given θ, let
AJS1 (θ) := {y ∈ Rn : γy = 1},
AJS2a (θ) := {y ∈ Rn : γy = 0, and ∆n ≤ } ,
AJS2b (θ) := Rn \(AJS1 (θ) ∪ AJS2a (θ))
= {y ∈ Rn : γy = 0, and ∆n > } .

17



2
by φ(y; θ), we have
exp − ky−θk
2σ 2
Z
R(θ, θ̂JSH ) =
φ(y; θ)kθ̂JS1 − θk2 dy
AJS1 (θ)
Z
+
φ(y; θ)kθ̂JS2 − θk2 dy
√ 1
2πσ 2

Denoting

D. Proof of Theorem 3
The proof is similar to that of Theorem 1, so we only
provide a sketch. Note that for ai , bi , real-valued and finite,
i = 1, · · · , n, with ai < bi ,
 
n
n  

bi
1X
ai 
1X 
−Q
E 1{ai <wi ≤bi } =
Q
,
n i=1
n i=1
σ
σ

n
n 
b2

i
1X 
σ X − a2i2
E wi 1{ai <wi ≤bi } = √
e 2σ − e− 2σ2 .
n i=1
n 2π i=1

AJS2a (θ) ∪ AJS2b (θ)
(a)

Z

φ(y; θ)kθ̂JS1 − θk2 dy

≤
AJS1 (θ)

Z

φ(y; θ) (kθ̂JS1 − θk2 + n) dy

+
AJS2a (θ)

Z

Since 1{ai <wi ≤bi } ∈ [0, 1], it follows that wi 1{ai <wi ≤bi } ∈
[mi , ni ] where mi = min(0, ai ), ni = max(0, bi ). So, from
Hoeffding’s inequality, we obtain
!
 
n
n  
X
1 X
bi
ai 
P
1{ai <wi ≤bi } −
−Q
≥
Q
n i=1
σ
σ
i=1

2

φ(y; θ)kθ̂JS2 − θk dy

+
AJS2b (θ)
(b)

1/2

≤ R(θ, θ̂JS1 ) + n + (P (γy = 0, ∆n > ))
!1/2
Z
4
×
φ(y; θ)kθ̂JS2 − θk dy

2

AJS2b (θ)
(c)

≤ R(θ, θ̂JS1 ) + n
nk min(2 ,1)

− max(kθk2 /n,1)

+ Ke



Ekθ̂JS2 − θk4

1/2

(75)

where step (a) uses the definition of AJS2a , in step (b) the last
term is obtained using the
p Cauchy-Schwarz
p inequality on the
product of the functions φ(y; θ), and φ(y; θ)kθ̂JS2 −θk2 .
Step (c) is from (74).
Similarly, when θ̂JS2 has lower risk, we get
R(θ, θ̂JSH ) ≤ R(θ, θ̂JS2 ) + n
nk min(2 ,1)

− max(kθk2 /n,1)

+ Ke



Ekθ̂JS1 − θk4

1/2

. (76)

Subsequently, the steps of Lemma 13 are used to obtain
1
ky − νyL k2
n



L
2 n  
µj − θi
µj−1 − θi
. kθk2 X cj X
Q
−
−Q
=
n
n i=1
σ
σ
j=0

 
X

L
n
X
−θi )2
(µ
(µ −θ )2
2
σ
− j2σ2i
− j−1
2
2σ
√
cj
e
−
−e
n
2π j=1 i=1
+ κn δ + o(δ).

Hence, from (75)-(76), we obtain



1
1
R(θ, θ̂JSH ) ≤
min R(θ, θ̂JSi ) + n
n
n i=1,2

1/2  
nk min(2 ,1)
− max(kθk2 /n,1)
4
max
Ekθ̂JSi − θk
+ Ke
.

(78)

Finally, employing the steps of Lemma 14, we get

i=1,2

Now,

noting
that
by
assumption,

1/2
/n is finite, we get
lim supn→∞ Ekθ̂JSi − θk4




1
R(θ, θ̂JSH ) − min R(θ, θ̂JSi ) −  ≤ 0.
lim sup
i=1,2
n→∞ n
Since this is true for every  > 0, we therefore have



1
lim sup
R(θ, θ̂JSH ) − min R(θ, θ̂JSi ) ≤ 0.
i=1,2
n→∞ n

≤ 2e−2n ,

n 
n
b2
i
1 X
σ X − a2i2
−
2
P
e 2σ − e 2σ
wi 1{ai <wi ≤bi } − √
n i=1
2π i=1

2
− Pn 2n
2
≥  ≤ 2e i=1 (ni −mi ) .

1
2
kθ − νyL k
n



L
2 n  
µj − θi
µj−1 − θi
. kθk2 X cj X
=
−
Q
−Q
n
n i=1
σ
σ
j=0
+ κn δ + o(δ).
The subsequent steps of the proof are along the lines of that
of Theorem 1.
VII. C ONCLUDING REMARKS

(77)

This completes the proof of the theorem.
2
Note3. Note that in the best case
 scenario, kθ − θ̂JSH k =
min kθ − θ̂JS1 k2 , kθ − θ̂JS2 k2 , which occurs when for
each realization of y, the hybrid estimator picks the better
of the two rival estimators θ̂JS1 and θ̂JS2 . In this case, the
inequality in (77) is strict, provided that there are realizations
of y with non-zero probability measure for which one estimator is strictly better than the other.

In this paper, we presented a class of shrinkage estimators
that take advantage of the large dimensionality to infer the
clustering structure of the parameter values from the data.
This structure is then used to construct an attracting vector for
the shrinkage estimator. A good cluster-based attracting vector
enables significant risk reduction over the ML-estimator even
when θ is composed of several inhomogeneous quantities.
We obtained concentration bounds for the squared-error loss
of the constructed estimators and convergence results for the
risk. The estimators have significantly smaller risks than the
regular JS-estimator for a wide range of θ ∈ Rn , even though

18

they do not dominate the regular (positive-part) JS-estimator
for finite n.
An important next step is to test the performance of the
proposed estimators on real data sets. It would be interesting
to adapt these estimators and analyze their risks when the
sample values are bounded by a known value, i.e., when |θi | ≤
τ , ∀i = 1, · · · , n, with τ known. Another open question is
how one should decide the maximum number of clusters to
be considered for the hybrid estimator.
An interesting direction for future research is to study
confidence sets centered on the estimators in this paper, and
compare them to confidence sets centered on the positive-part
JS-estimator, which were studied in [19], [20].
The James-Stein estimator for colored Gaussian noise, i.e.,
for w ∼ N (0, Σ) with Σ known, has been studied in [21],
and variants have been proposed in [22], [23]. It would be
interesting to extend the ideas in this paper to the case of
colored Gaussian noise, and to noise that has a general subGaussian distribution. Yet another research direction is to
construct multi-dimensional target subspaces from the data that
are more general than the cluster-based subspaces proposed
here. The goal is to obtain greater risk savings for a wider
range of θ ∈ Rn , at the cost of having a more complex
attractor.

b2

Clearly, f (−∞; b) = e 2 , and since b > 0, we have for
x ≤ 0,
i
x2 h
b2
b2
− √b2π e− 2
e 2 Q (x − b) + e 2 (1 − Q (x))
f (x; b) < e
 

x2
b2
− √b2π e− 2
e 2 [Q (x − b) + 1 − Q (x)]
= e

 

Z x
x2
b2
1 − u2
− √b2π e− 2
2
2
√ e
= e
e
1+
du
2π
x−b




 b2 
x2
c2
b
(i)
− √b e− 2
1 + √ e− 2
= e 2π
e2
2π


2 
2 
(j)
2 
−x
−c
(k) b2
b
b
b
2
√
− √2π e 2
e
e 2π
≤ e
e2
< e2
where (i) is from the first mean value theorem for integrals
for some c ∈ (x − b, x), (j) is because ex ≥ 1 + x for x ≥ 0,
2
2
and (k) is because for x ≤ 0, e−x > e−(x−b) for b > 0.
Therefore,
b2
sup f (x; b) = e 2 .
(80)
x∈(−∞,0]

Now, for x ≥ 0, consider
h(x) :=

f (−x; b) − f (x; b)
e

=e

A PPENDIX
A. Proof of Lemma 10


Note that E wi 1{wi >ai } =

a2

i
√σ e− 2σ2
2π

. So, with

n
σ X − a2i2
e 2σ ,
X :=
wi 1{wi >ai } − √
2π i=1
i=1
n
X

a2
i

we have EX = 0. Let mi := √σ2π e− 2σ2 , and consider the
moment generating function (MGF) of X. We have
Z ∞ 

n
2
wi
 λX  Y
e−λmi
λwi 1{w >a }
i
i
√
E e
=
e
e− 2σ2 dwi
2πσ 2 −∞
i=1

Z
Z ai
n
∞
Y
w2
w2
e−λmi
− 2σi2
λwi − 2σi2
√
=
e e
dwi +
e
dwi
2πσ 2 ai
−∞
i=1

Z ∞
n
 a 
Y
w2
1
i
−λmi
λwi − 2σi2
√
=
e
e e
dwi + 1 − Q
2
σ
2πσ
a
i
i=1
n
h λ2 σ 2  a

 a i
Y
i
i
=
e−λmi e 2 Q
− λσ + 1 − Q
.
σ
σ
i=1
(79)
Now, for any positive real number b, consider the function
i
x2 h
b2
− √b e− 2
f (x; b) = e 2π
e 2 Q (x − b) + 1 − Q (x) .
Qn
Note that the RHS of (79) can be written as i=1 f ( aσi ; λσ).
We will bound the MGF in (79) by bounding f (x; b).

b2
2

− √b2π e−

x2
2

[Q (−x − b) − Q (x − b)] + Q (x) − Q (−x) .

We have h(0) = 0 and


√ dh(x)
(x+b)2
(x−b)2
b2
x2
2π
= e 2 e− 2 + e− 2
− 2e− 2
dx

−x2 
x2
= e 2 e−bx + ebx − 2e− 2

−x2 
= e 2 e−bx + ebx − 2
= 2e

−x2
2

[cosh(bx) − 1] ≥ 0

because cosh(bx) ≥ 1. This establishes that h(x) is monotone
non-decreasing in [0, ∞) with h(0) = 0, and hence, for x ∈
[0, ∞),
h(x) ≥ 0 ⇒ f (−x; b) ≥ f (x; b).
(81)
Finally, from (80) and (81), it follows that
sup

b2

f (x; b) = e 2 .

(82)

x∈(−∞,∞)



nλ2 σ 2
Using (82) in (79), we obtain E eλX ≤ e 2 . Hence,
applying the Chernoff trick, we have for λ > 0:




2 2
 E eλX
− λ− nλ2 σ
λX
λ
P (X ≥ ) = P e
≥e
≤
≤
e
.
eλ
Choosing λ =


nσ 2

which minimizes e



2 2
− λ− nλ2 σ

, we get

19

2

P (X ≥ ) ≤ e− 2nσ2 and so,


X
P
≥
n
!
!
n
n
1 X
σ X − a2i2
=P
≥
wi 1{wi >ai } − √
e 2σ
n i=1
2π i=1
n2

≤ e− 2σ2 .

(83)

To obtain the lower tail inequality, we use the following result:
Fact 2. [24, Thm. 3.7]. For independent random variables
Xi satisfying Xi ≥ −M , for 1 ≤ i ≤ n, we have for any
 > 0,
!
2
n
n
− Pn
X
X
2(
E[X 2 ]+ M  )
i=1
i
3
P
X −
E[X ] ≤ − ≤ e
.
i

i

i=1

i=1

So, for Xi = w
 i 1{w
 i >ai } , we have Xi ≥ min{0, ai , i =
1, · · · , n}, and E Xi2 ≤ σ 2 , ∀i = 1, · · · , n. Clearly, we can
take M = − min{0, ai , i = 1, · · · , n} < ∞. Therefore, for
any  > 0,
!
2
n
n
− Pn
X
X
M
2
P
X −
E[X ] ≤ − ≤ e 2( i=1 E[Xi ]+ 3 )
i

i

i=1

i=1

≤e

2
−
2 nσ 2 + M 
3

(

Then, for any t > 0, we have
P(E)
≤ P(E, {a < f (y) ≤ a + t}) + P(E, {a − t ≤ f (y) ≤ a})
+ P(|f (y) − a| > t)
!
n
1 X
=P
θi 1{a<yi ≤f (y)} ≥ , {a < f (y) ≤ a + t}
n i=1
+P

!
n
1 X
θi 1{f (y)<yi ≤a} ≥ , {a − t < f (y) ≤ a}
n i=1

+ P(|f (y) − a| > t)
!
 X

n
1
≤P
|θi |1{a<yi ≤a+t} ≥ 
n i=1
!

 X
n
2
1
|θi |1{a−t<yi ≤a} ≥  + 2e−nkt .
+P
n i=1

Now,
Z
P(1{a<yi ≤a+t} = 1) =

)

(85)

a

a+t

√

1
2πσ 2

e−

(yi −θi )2
2σ 2

dyi ≤ √

and hence,
P

n
X

n
σ X − a2i2
wi 1{wi >ai } − √
e 2σ
2π
i=1
i=1

1
n
−

≤e

!

≤ −

n

2 σ2 + M 
3

).

≤ 2e

n2

(

2 σ2 + M 
3

1X
|θi | P(1{a<yi ≤a+t} = 1)
0 ≤ EY =
n i=1

(84)

Using the upper and lower tail inequalities obtained in (83)
and (84), respectively, we get
!
n
n
σ X − a2i2
1 X
e 2σ ≥ 
wi 1{wi >ai } − √
P
n i=1
2π i=1
−

2πσ 2
(86)

(yi −θi )2
Pn
where we have used e− 2σ2
≤ 1. Let Y := n1 i=1 Yi
where Yi := |θi |1{a<yi ≤a+t} . Then, from (86), we have

!

n2

(

t

) ≤ 2e−nk min(,2 )

where k is a positive constant (this is due to M being finite).
This proves (53). The concentration inequality in (54) can be
similarly proven, and will not be detailed here.

n
X
t
|θi |.
≤ √
n 2πσ 2 i=1

Since Yi ∈ [0, |θi |], from Hoeffding’s inequality, for any
2n21
1 > 0, we have P (Y − EY ≥ 1 ) ≤ exp{− kθk2 /n
}, which
implies


2n2
1
tkθk1
−
≤ e kθk2 /n ,
P Y ≥ 1 + √
2
n 2πσ
Pn
where kθk1 := i=1 |θi |. Now, set 1 = /2 and
p
 πσ 2 /2
t=
(87)
kθk1 /n
to obtain
2

B. Proof of Lemma 11
n

We first prove (56). Then (55) immediately follows by
setting θi = 1, ∀i.
Let us denote the event whose probability we want to bound
by E. In our case,
)
(
n
n
X
1 X
θi 1{yi >f (y)} −
θi 1{yi >a} ≥  .
E=
n i=1
i=1

⇒P

P (Y ≥ ) ≤ e
!

1X
|θi |1{a<yi ≤a+t} ≥ 
n i=1

≤e

n
− 2kθk
2 /n
2

n
− 2kθk
2 /n

.

(88)

A similar analysis yields
n

P

1X
|θi |1{a−t<yi ≤a} ≥ 
n i=1

!

2

≤e

n
− 2kθk
2 /n

.

(89)

Using (88) and (89) in (85) and recalling that t is given by

20

(87), we obtain

we obtain,

n
X

n

!

n
X

1
P
θi 1{yi >f (y)} −
θi 1{yi >a} ≥ 
n i=1
i=1
 n2 kπσ2

2
n2
−
− nk
− 2kθk
2
2 /n
2kθk2
/n
1
+e
≤2 e
≤ 4e kθk2 /n

where k is a positive constant. The last inequality holds
because kθk21 /n2 < kθk2 /n (by the Cauchy-Schwarz inequality), and lim supn→∞ kθk2 /n < ∞ (by assumption). This
proves (56).

P

1X
|wi |1{a<yi ≤a+t} ≥ 
n i=1

!
≤ 2e−nk min(

2

,)

(92)

where k is a positive constant. Using similar steps, it can be
shown that the third term on the RHS of (90) can also be
bounded as
!
n
2
1X
|wi |1{a−t<yi ≤a} ≥  ≤ 2e−nk min( ,) . (93)
P
n i=1
This completes the proof of (57).

Next, we prove (57). Using steps very similar to (85), we
have, for t > 0,  > 0,
!
n
n
X
1 X
wi 1{yi >f (y)} −
P
wi 1{yi >a} ≥ 
n i=1
i=1
!
n
1X
−nkt2
≤ 2e
+P
|wi |1{a<yi ≤a+t} ≥ 
n i=1
!
n
1X
|wi |1{a−t<yi ≤a} ≥  .
(90)
+P
n i=1
Pn
Now, let Y := n1 i=1 Yi where
Yi := |wi |1{a<yi ≤a+t} = |wi |1{a−θi <wi ≤a−θi +t} .
Noting that |wi | ≤ t + |a − θi | when wi ∈ [a − θi , a − θi + t],
we have
Z a−θi +t
|w| −w2 /2σ2
√
E[Yi ] =
e
dw
2
2πσ
a−θi

 (j)
2
2
t
|c|
(i)
e−c /2σ
≤ √
=t √
.
2
2πe
2πσ
Note that (i) is from the mean value theorem for integrals √
with
2
c ∈ (a − θi , a − θi + t), and (j) is because xe−x ≤ 1/ 2e
for x ≥ 0. Hence
n

t
1X
E[Yi ] ≤ √
.
0 ≤ E[Y ] =
n i=1
2πe

So,





lim sup E |Xn |1{|Xn |≥K}
K→∞
n→∞

i
h
≤ c lim lim sup E Yn 1{Yn ≥ K−a }
c
K→∞
n→∞


K −a
+ a lim lim sup P Yn ≥
= 0.
K→∞
c
n→∞
lim

D. Proof of Lemma 1
We first prove (16) and (17). Then, (18) and (19) immediately follow by setting θi = 1, for 1 ≤ i ≤ n.
From Lemma 11, for any  > 0,
!
n
n
2
X
1 X
− nk
P
θi 1{yi >ȳ} −
θi 1{yi >θ̄} ≤ 4e kθk2 /n . (94)
n
i=1

As each Yi takes values in an interval of length at most t, by
Hoeffding’s inequality we have for any 1 > 0
2

C. Proof of Lemma 15
Since {Yn }∞
VI.1, we have
n=1 is UI, from Definition

limK→∞ lim supn→∞ E Yn 1{Yn ≥K} = 0. Therefore,


E |Xn |1{|Xn |≥K}




≤ E c|Yn |1{|Xn |≥K} + E a1{|Xn |≥K}




≤ cE Yn 1{cYn +a≥K} + aE 1{cYn +a≥K}
i
h
i
h
= cE Yn 1{Yn ≥ K−a } + aE 1{Yn ≥ K−a }
c
c


h
i
K −a
= cE Yn 1{Yn ≥ K−a } + aP Yn ≥
.
c
c

2

P(Y ≥ 1 + E[Y ]) ≤ 2e−2n1 /t


2
2
t
⇒ P Y ≥ 1 + √
≤ 2e−2n1 /t .
(91)
2πe
√
t
Now, set √2πe
= 1 . Using this value of t in the RHS of
(91), we obtain
!
n
√
1X
P
|wi |1{a<yi ≤a+t} ≥ 1 + 1 ≤ 2e−nk1 1
n i=1
√
√
where
k1 = 1/(πe). Setting 1 + 1 = , we get 1 =
√
4+1−1
. Using the following inequality for x > 0:
2
 2
√
2
x /32, 0 ≤ x ≤ 3
1+x−1 ≥
3x/4,
x > 3,

i=1

Since θi 1{yi >θ̄} ∈ {0, θi } are independent for 1 ≤ i ≤ n,
from Hoeffding’s inequality, we have, for any  > 0,
!
n
n
h
i
1X
1X
P
θi 1{yi >θ̄} −
θi E 1{yi >θ̄} > 
n i=1
n i=1
2

≤ 2e

2n
− kθk
2 /n

.

(95)

Also for each i,
h
i


E 1{yi >θ̄} = P yi > θ̄ = P wi > θ̄ − θi


θ̄ − θi
=Q
.
σ
Therefore, from (94) and (95), we obtain


n
n
1X
θ̄ − θi
. 1X
θi 1{yi >ȳ} =
θi Q
.
n i=1
n i=1
σ

(96)

21

The concentration result in (17) immediately follows by writing 1{yi ≤ȳ} = 1 − 1{yi >ȳ} .
To prove (14), we write
n

n

n

1X
1X
1X
yi 1{yi >ȳ} =
θi 1{yi >ȳ} +
wi 1{yi >ȳ} .
n i=1
n i=1
n i=1
Hence, we have to show that
n
n
σ X − (θ̄−θ2i )2
1X
.
wi 1{yi >ȳ} = √
e 2σ .
n i=1
n 2π i=1

(97)

From the first mean value theorem for integrals, ∃εi ∈ (−δ, δ)
such that


Z θ̄+δ
(y −θ )2
(θ̄+εi −θi )2
1
1
− i2σ2i
−
2
2σ
√
√
e
dyi = 2δ
e
2πσ 2
2πσ 2
θ̄−δ
and so the RHS of (102) can be written as
n Z
(yi −θi )2
σ 2 X θ̄+δ
1
√
e− 2σ2 dyi
2nδ i=0 θ̄−δ
2πσ 2
n
2
i)
σ X − (θ̄+εi −θ
2σ 2
= √
.
e
n 2π i=0

From Lemma 11, for any  > 0, we have
n
n
X
1 X
wi 1{yi >ȳ} −
wi 1{yi >θ̄}
n i=1
i=1

P

!

Now, let xi := θ̄ − θi . Then, since |εi | ≤ δ, we have

2

≤ 4e

nk
− kθk
2 /n

.

(98)

Now,


h
i Z ∞
w2
1
− 2σi2
e
dwi
E wi 1{yi >θ̄} =
wi 1{yi >θ̄} √
2πσ 2
Z−∞
∞
2
wi
(θ̄−θi )2
w
σ
√ i e− 2σ2 dwi = √ e− 2σ2 .
=
2π
2πσ 2
θ̄−θi

1

n
X

n 2πσ 2

i=0

√

d
≤ δ max
dx

P

2

(99)

We obtain (97) by combining (98) and (99).
Similarly, (15) can be shown using Lemma 11 and Lemma
10 to establish that
n
n
σ X − (θ̄−θ2i )2
1X
.
e 2σ .
wi 1{wi ≤ȳ} = − √
n i=1
n 2π i=1

√

1

2σ 2

2

2πσ 2

e

x
− 2σ
2


=

σ2

δ
√
.
2πe

n Z
(yi −θi )2
σ 2 X θ̄+δ
1
√
e− 2σ2 dyi
2nδ i=0 θ̄−δ
2πσ 2
n
σ X − (θ̄−θ2i )2
= √
e 2σ
+ κn δ
n 2π i=0

!

≤ 2e−nk min(, ) .



(xi +εi )2

Therefore,

Using Lemma 10, we get, for any  > 0,
n
n
1 X
σ X − (θ̄−θ2i )2
e 2σ
≥
wi 1{wi >θ̄} − √
n i=1
2π i=1

x2
i

e− 2σ2 − e−

(103)

1
where |κn | ≤ √2πe
. Using (103) in (102), and then the
obtained result in (101) and (100), the proof of the lemma
is complete.

F. Proof of Lemma 13
We have

E. Proof of Lemma 2

n

1
1X
2
2
ky − ν2 k =
(yi − a1 ) 1{yi >ȳ}
n
n i=1

From Lemma 12, we have, for any  > 0,
P

n
n
X
1 σ2 X
1{|yi −ȳ|≤δ} −
1{|yi −θ̄|≤δ} ≥ 
n 2δ i=0
i=0

≤ 8e−

nk2 δ 2
σ4

.

(100)

Further, from Hoeffding’s inequality,
P

!
n
n
i
1 σ2 X
σ2 X h
1
−
E 1{|yi −θ̄|≤δ} ≥ 
n 2δ i=0 {|yi −θ̄|≤δ} 2δ i=0

≤ 2e−

8nδ 2 2
σ4

.

n

!

(101)

Also,
n
n
i σ2 X

σ2 X h
E 1{|yi −θ̄|≤δ} =
P yi − θ̄ ≤ δ
2δ i=0
2δ i=0
n Z
(yi −θi )2
σ 2 X θ̄+δ
1
√
=
e− 2σ2 dyi .
(102)
2δ i=0 θ̄−δ
2πσ 2

+

1X
2
(yi − a2 ) 1{yi ≤ȳ} .
n i=1

(104)

Now,
n

1X
2
(yi − a1 ) 1{yi >ȳ}
n i=1
" n
#
n
n
X
X
1 X 2
2
y 1{yi >ȳ} +
a1 1{yi >ȳ} − 2
a1 yi 1{yi >ȳ}
=
n i=1 i
i=1
i=1
" n
n
X
1 X 2
=
wi 1{wi >ȳ−θi } +
θi2 1{yi >ȳ}
n i=1
i=1
+2

n
X

θi wi 1{wi >ȳ−θi } +

i=1

−2

n
X
i=1

n
X
i=1


a1 yi 1{yi >ȳ}

a21 1{yi >ȳ}

22

and similarly,
1
n

n
X

2

(yi − a2 ) 1{yi ≤ȳ} =

i=1

+
+

n
X
i=1
n
X

θi2 1{yi ≤ȳ} + 2
a22 1{yi ≤ȳ} − 2

i=1

n
X
i=1
n
X

1
n

X
n

Employing the same steps as above, we get
!


n
n
1X
. c2 X c θ̄ − θi
2
a2
1{yi ≤ȳ} = 2
Q
+ κn δ + o(δ),
n i=1
n i=1
σ
(110)
!


n
n
2X
. 2c2 X c θ̄ − θi
yi 1{yi ≤ȳ} = 2
Q
a2
n i=1
n i=1
σ

wi2 1{wi ≤ȳ−θi }

i=1

θi wi 1{wi ≤ȳ−θi }

a2 yi 1{yi ≤ȳ} .

n
2c2 σ X − (θ̄−θ2i )2
√
−
+ κn δ + o(δ).
e 2σ
n 2π i=1

i=1

Therefore, from (104)
n

Therefore, using (106)-(111) in (105), we finally obtain

n

1 X 2 kθk2
2X
1
2
ky − ν2 k =
wi +
+
θi wi
n
n i=1
n
n i=1
+
+

1
n
n
X

n
X
i=1

a22 1{yi ≤ȳ}

i=1

Since

1
n

Pn

a21 1{yi >ȳ} − 2

n
X

a1 yi 1{yi >ȳ}

i=1

−2

n
X


a2 yi 1{yi ≤ȳ} .

(105)

i=1



2
∼ N 0, kθk
,
n2
!
n
2
1X
− n
θi wi ≥  ≤ e 2kθk2 /n .
n i=1

which completes the proof of the lemma.
(106)

(107)

where c1 , c2 are defined in (26). The concentration in (107)
follows from Lemmas 1 and 2, together with the results on
concentration of products and reciprocals in Lemmas 5 and
6, respectively. Further, using (107) and Lemma 5 again, we
.
obtain a21 = c21 + κn δ + o(δ) and
!


n
n
θ̄ − θi
1X
. c2 X
2
1{yi >ȳ} = 1
Q
+ κn δ + o(δ).
a1
n i=1
n i=1
σ
(108)
Similarly,
!
n
2X
a1
yi 1{yi >ȳ}
n i=1
!


n
n
θ̄ − θi
σ X − (θ̄−θ2i )2
. 2c1 X
=
θi Q
+√
e 2σ
n
σ
2π i=1
i=1
+ κn δ + o(δ)
!


n
n
X
θ̄ − θi
σ X − (θ̄−θ2i )2
. 2c1
=
c1
Q
+√
e 2σ
n
σ
2π i=1
i=1
+ κn δ + o(δ)


n
n
θ̄ − θi
2c1 σ X − (θ̄−θ2i )2
. 2c2 X
= 1
Q
+ √
e 2σ
n i=1
σ
n 2π i=1
+ κn δ + o(δ).

G. Proof of Lemma 14
The proof is along the same lines as that of Lemma 13. We
have
1
kθ − ν2 k2
n
" n
#
n
X
1 X
2
2
=
(θi − a1 ) 1{yi >ȳ} +
(θi − a2 ) 1{yi ≤ȳ}
n i=1
i=1

From Lemma 9, we have, for any  > 0,
!
n
2
1X 2
2
P
wi − σ ≥  ≤ 2e−nk min(, )
n i=1
where k is a positive constant. Next, we claim that
.
.
a1 = c1 + κn δ + o(δ), a2 = c2 + κn δ + o(δ),

1
2
ky − ν2 k
n




n
n
c2 X
θ̄ − θi
c2 X c θ̄ − θi
. kθk2
=
+ σ2 − 1
Q
− 2
Q
n
n i=1
σ
n i=1
σ
!
 X
 
n
(θ̄−θi )2
σ
2
√
e− 2σ2
(c1 − c2 )
−
n
2π
i=1
+ κn δ + o(δ),

i=1 θi wi

P

(111)

=

1
kθk2
+
n
n
+

n
X

n
X

a21 1{yi >ȳ} − 2

i=1

a1 θi 1{yi >ȳ}

i=1

a22 1{yi ≤ȳ} − 2

i=1
2

n
X

n
X

!
a2 θi 1{yi ≤ȳ}

i=1





n
n
c21 X
θ̄ − θi
2c1 X
θ̄ − θi
. kθk
=
+
Q
−
θi Q
n
n i=1
σ
n i=1
σ

!


n
n
X
X
1
θ̄ − θi
θ̄ − θi
2
c
c
+
c2
Q
− 2c2
θi Q
n
σ
σ
i=1
i=1
+ κn δ + o(δ)




n
n
c21 X
θ̄ − θi
c22 X c θ̄ − θi
. kθk2
−
Q
−
Q
=
n
n i=1
σ
n i=1
σ
+ κn δ + o(δ).
Acknowledgement
The authors thank R. Samworth for useful discussions on
James-Stein estimators, and A. Barron and an anonymous
referee for their comments which led to a much improved
manuscript.
R EFERENCES

(109)

[1] W. James and C. M. Stein, “Estimation with Quadratic Loss,” in Proc.
Fourth Berkeley Symp. Math. Stat. Probab., pp. 361–380, 1961.

23

[2] E. L. Lehmann and G. Casella, Theory of Point Estimation. Springer,
New York, NY, 1998.
[3] B. Efron and C. Morris, “Data Analysis Using Stein’s Estimator and Its
Generalizations,” J. Amer. Statist. Assoc., vol. 70, pp. 311–319, 1975.
[4] D. V. Lindley, “Discussion on Professor Stein’s Paper,” J. R. Stat. Soc.,
vol. 24, pp. 285–287, 1962.
[5] C. Stein, “Estimation of the mean of a multivariate normal distribution,”
Ann. Stat., vol. 9, pp. 1135–1151, 1981.
[6] B. Efron and C. Morris, “Stein’s estimation rule and its competitors—an
empirical Bayes approach,” J. Amer. Statist. Assoc., vol. 68, pp. 117–
130, 1973.
[7] E. George, “Minimax Multiple Shrinkage Estimation,” Ann. Stat.,
vol. 14, pp. 188–205, 1986.
[8] E. George, “Combining Minimax Shrinkage Estimators,” J. Amer. Statist.
Assoc., vol. 81, pp. 437–445, 1986.
[9] G. Leung and A. R. Barron, “Information theory and mixing leastsquares regressions,” IEEE Trans. Inf. Theory, vol. 52, no. 8, pp. 3396–
3410, 2006.
[10] G. Leung, Improving Regression through Model Mixing. PhD thesis,
Yale University, 2004.
[11] A. J. Baranchik, “Multiple Regression and Estimation of the Mean of a
Multivariate Normal Distribution,” Tech. Report, 51, Stanford University,
1964.
[12] P. Shao and W. E. Strawderman, “Improving on the James-Stein Positive
Part Estimator,” Ann. Stat., vol. 22, no. 3, pp. 1517–1538, 1994.
[13] Y. Maruyama and W. E. Strawderman, “Necessary conditions for
dominating the James-Stein estimator,” Ann. Inst. Stat. Math., vol. 57,
pp. 157–165, 2005.

[14] R. Beran, The unbearable transparency of Stein estimation. Nonparametrics and Robustness in Modern Statistical Inference and Time Series
Analysis: A Festschrift in honor of Professor Jana Jurečková, pp. 25–34.
Institute of Mathematical Statistics, 2010.
[15] I. M. Johnstone, Gaussian estimation: Sequence and wavelet models.
[Online]: http://statweb.stanford.edu/∼imj/GE09-08-15.pdf, 2015.
[16] S. Boucheron, G. Lugosi, and P. Massart, Concentration Inequalities: A
Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[17] L. Wasserman, All of Statistics: A Concise Course in Statistical Inference. Springer, New York, NY, 2nd ed., 2005.
[18] D. Williams, Probability with Martingales. Cambridge University Press,
1991.
[19] J. T. Hwang and G. Casella, “Minimax confidence sets for the mean
of a multivariate normal distribution,” The Annals of Statistics, vol. 10,
no. 3, pp. 868–881, 1982.
[20] R. Samworth, “Small confidence sets for the mean of a spherically
symmetric distribution,” Journal of the Royal Statistical Society: Series
B (Statistical Methodology), vol. 67, no. 3, pp. 343–361, 2005.
[21] M. E. Bock, “Minimax estimators of the mean of a multivariate normal
distribution,” Ann. Stat., vol. 3, no. 1, pp. 209–218, 1975.
[22] J. H. Manton, V. Krishnamurthy, and H. V. Poor, “James-Stein State
Filtering Algorithms,” IEEE Trans. Sig. Process., vol. 46, pp. 2431–
2447, Sep. 1998.
[23] Z. Ben-Haim and Y. C. Eldar, “Blind Minimax Estimation,” IEEE Trans.
Inf. Theory, vol. 53, pp. 3145–3157, Sep. 2007.
[24] F. Chung and L. Lu, “Concentration inequalities and martingale inequalities: a survey,” Internet Mathematics, vol. 3, no. 1, pp. 79–127, 2006.

