The Dispersion Bias

arXiv:1711.05360v4 [stat.ME] 15 Feb 2018

Lisa Goldberg∗ Alex Papanicolaou† Alex Shkolnik‡
November 4, 2017
This Version: February 16, 2018§

Abstract
Estimation error has plagued quantitative finance since Harry Markowitz
launched modern portfolio theory in 1952. Using random matrix theory,
we characterize a source of bias in the sample eigenvectors of financial
covariance matrices. Unchecked, the bias distorts weights of minimum
variance portfolios and leads to risk forecasts that are severely biased
downward. To address these issues, we develop an eigenvector bias correction. Our approach is distinct from the regularization and eigenvalue
shrinkage methods found in the literature. We provide theoretical guarantees on the improvement our correction provides as well as estimation
methods for computing the optimal correction from data.

∗

Departments of Economics and Statistics and Consortium for Data Analytics in Risk,
University of California, Berkeley, CA 94720 and Aperio Group, lrg@berkeley.edu.
†
Consortium for Data Analytics in Risk, University of California, Berkeley, CA 94720,
apapanicolaou@berkeley.edu.
‡
Consortium for Data Analytics in Risk, University of California, Berkeley, CA 94720,
ads2@berkeley.edu.
§
We thank the Center for Risk Management Research, the Consortium for Data Analytics
in Risk, and the Coleman Fung Chair for financial support. We thank Marco Avellaneda,
Bob Anderson, Kay Giesecke, Nick Gunther, Guy Miller, George Papanicolaou, Yu-Ting
Tai, participants at the the 3rd Annual CDAR Symposium in Berkeley, participants at the
Swissquote Conference 2017 on FinTech, and participants at the UC Santa Barbara Seminar
in Statistics and Applied Probability for discussion and comments. We are grateful to Stephen
Bianchi, whose incisive experiment showing that it is errors in eigenvectors, and not in
eigenvalues, that corrupt large minimum variance portfolios, pointed us in a good direction.

1

1

Introduction

Harry Markowitz transformed finance in 1952 by framing portfolio construction as a tradeoff between mean and variance of return. This application of
mean-variance optimization is the basis of theoretical breakthroughs as fundamental as the Capital Asset Pricing Model (CAPM) and Arbitrage Pricing Theory (APT), as well as practical innovations as impactful as Exchange Traded
Funds.1 Still, all financial applications of mean-variance optimization suffer
from estimation error in covariance matrices, and we highlight two difficulties.
First, a portfolio that is optimized using an estimated covariance matrix is
never the true Markowitz portfolio. Second, in current practice, the forecasted
risk of the optimized portfolio is typically too low, sometimes by a wide margin.
Thus, investors end up with the wrong portfolio, one that is riskier, perhaps a
lot riskier, than anticipated.
In this article, we address these difficulties by correcting a systematic bias
in the first eigenvector of a sample covariance matrix. Our setting is that of a
typical factor model,2 but our statistical setup differs from most recent literature. In the last two decades, theoretical and empirical emphasis has been on
the case when the number of assets N and number of observations T are both
large. In this regime, consistency of principal component analysis (PCA) estimates may be established (Bai & Ng 2008). Motivated by many applications,
we consider the setting of relatively few observations (in asymptotic theory: N
grows and T is fixed). Indeed, an investor often has a portfolio of thousands of
securities but only hundreds of observations.3 PCA is applied in this environment in the early, pioneering work by Connor & Korajczyk (1986) and Connor
& Korajczyk (1988), but also very recently (Wang & Fan 2017). In this high
dimension, low sample-size regime, PCA factor estimates necessarily carry a
finite-sample bias. This bias is further amplified by the optimization procedure
that is required to compute a Markowitz portfolio.
An elementary simulation experiment reveals that in a large minimum
variance portfolio, errors in portfolio weights are driven by the first principal
component, not its variance.4 The fact that the eigenvalues of the sample covariance matrix are not important requires some nontrivial analysis, which we
carry out. In particular, we show (in our asymptotic regime) that the bias
in the dominant sample eigenvalue does not effect the performance of the estimated minimum variance portfolio. Only the bias in the dominant sample
eigenvector needs to be addressed. We measure portfolio performance using
1

The seminal paper is Markowitz (1952). See Treynor (1962) and Sharpe (1964) for the
Capital Asset Pricing Model and Ross (1976) for the Arbitrage Pricing Theory.
2
More precisely, the eigenvalues of the covariance matrix corresponding to the factors
grow linearly in the dimension. This is not the traditional random matrix theory setting in
which all eigenvalues are bounded, nor that of “weak” factors, e.g., Onatski (2012).
3
While high frequency data are available in some markets, many securities are observed
only at a daily horizon or less frequently. Moreover, markets are non-stationary, so even
when there is a long history of data available, its relevance to some problems is questionable.
4
This experiment was first communicated to us by Stephen Bianchi.

2

two well-established metrics. Tracking error, the workhorse of financial practitioners, measures deviations in weights between the estimated (optimized) and
optimal portfolios. We use the variance forecast ratio, familiar to both academics and practitioners, to measure the accuracy of the risk forecast of the
portfolio, however right or wrong that portfolio may be.
To develop some intuition for the results to come, consider a simplistic
world where all security exposures to the dominant (market) factor are identical. With probability one, a PCA estimate of our idealized, dominant factor
will have higher dispersion (variation in its entries). Decreasing this dispersion,
obviously, mitigates the estimation error. We denote our idealized, dominant
factor by z. We prove that the same argument applies to any other dominant
factor along the direction of z with high probablity for N large. Thus moving our PCA estimate towards z, by some amount, is very likely to decrease
estimation error. In the limit (N ↑ ∞), the estimation error is reduced with
probability one. The larger the component of the true dominant factor along
z is, the more we can decrease the estimation error.
While a careful proof of our result relies on some recent theory on sample
eigenvector asymptotics, rule of thumb versions have been known to practitioners since the 1970s (see footnote 10 and the corresponding discussion). Indeed,
the dominant risk factor consistently found in the US and many other developed public equity markets has most (if not all) individual equities positively
exposed to it. In other words, empirically, the dominant risk factor has a significant component in z. Our characterization of the dispersion bias may then
be viewed as a formalization of standard operation procedure.
The remainder of the introduction discusses our contributions and the
related literature. Section 2 describes the problem and fundamental results
around the sample covariance matrix and PCA. In Section 3, we present our
main results on producing a bias corrected covariance estimate. Section 4 discusses the implementation of our correction for obtaining data-driven estimates.
Finally, in Section 5 we present numerical results illustrating the performance
of our method in improving the estimated portfolio and risk forecasts.

1.1

Our contributions

We contribute to the literature by providing a method that significantly improves the performance of PCA-estimated minimum-variance portfolios. Our
approach and perspective appear to be new. We summarize some of the main
points.
Several authors (see above) have noted that sample eigenvectors carry a
bias in the statistical and model setting we adopt. We contribute in this direction by, first, recognizing that it is the bias in the first sample eigenvector that
drives the performance of PCA-based, minimum-variance portfolios. Second,
we show that this bias may in fact be corrected to some degree (cf. discussion
below (3.7) in Wang & Fan (2017)). In our domain of application this degree
is material. We point out that eigenvalue bias, which has been the focus in
3

most literature, does not have a material impact on minimum-variance portfolio performance. This motivates lines of research into more general Markowitz
optimization problems. Finally, our correction can be framed geometrically
in terms of the spherical law of cosines. This perspective illuminates possible
extensions of our work. We discuss this further in our concluding remarks.
We also develop a bias correction and show that it outperforms standard
PCA. Minimum variance portfolios constructed with our corrected covariance
matrix are materially closer to optimal, and their risk forecasts are materially more accurate. In an idealized one-factor setting, we provide theoretical
guarantees for the size of the improvement. Our theory also identifies some
limitations. We demonstrate the efficacy of the method with an entirely datadriven correction. In an empirically calibrated simulation, its performance is
far closer to the theoretically optimal than to standard PCA.

1.2

Related literature

The impact of estimation error on optimized portfolios has been investigated
thoroughly in simulation and emprical settings. For example, see Jobson &
Korkie (1980), Britten-Jones (1999), Bianchi, Goldberg & Rosenberg (2017)
and the references therein. DeMiguel, Garlappi & Uppal (2007) compare a
variety of methods for mitigating estimation error, benchmarking againt the
equally weigted portfolio in out-of-sample tests. They conclude that unreasonably long estimation windows are required for current methods to consistently
outperform the benchmark. We review some methods most closely related to
our approach.5
Early work on estimation error and the Markowitz problem was focused on
Bayesian approaches. Vasicek (1973) and Frost & Savarino (1986) were perhaps
the first to impose informative priors on the model parameters.6 More realistic priors incorporating multi-factor modeling are analyzed in Pástor (2000)
(sample mean) and Gillen (2014) (sample covariance). Formulae for Bayes’ estimates of the return mean and covariance matrix based on normal and inverted
Wishart priors may be found in Lai & Xing (2008, Chapter 4, Section 4.4.1).
A related approach to the Bayesian framework is that of shrinkage or regularization of the sample covariance matrix.7 Shrinkage methods have been
proposed in contexts where little underlying structure is present (Bickel &
Levina 2008) as well as those in which a factor or other correlation struc5

The literature on this topic is extensive. We briefly mention a few important references
that do not overlap at all with out work. Michaud & Michaud (2008) recommends the use
of bootstap resampling. Lai, Xing & Chen (2011) reformulate the Markowitz problem as
one of stochastic optimization with unknown moments. Goldfarb & Iyengar (2003) develop
a robust optimization procedure for the Markowitz problem by embedding a factor structure
in the constraint set.
6
Preceeding work analyzed diffuse priors and was shown to be inneficient (Frost & Savarino
1986). The latter, instead, presumes all stocks are identical and have the same correlations.
Vasicek (1973) specified a normal prior on the cross-sectional market betas (dominant factor).
7
In the Bayesian setup, sample estimates are “shrunk” toward the prior (Lai & Xing 2008).

4

ture is presumed to exist (e.g. Ledoit & Wolf (2003), Ledoit & Wolf (2004),
Fan, Liao & Mincheva (2013) and Bun, Bouchaud & Potters (2016)). Perhaps
surprisingly, shrinkage methods turn out to be related to placing constraints
on the portfolio weights in the Markowitz optimization. Jagannathan & Ma
(2003) show that imposing a positivity constraint typically shrinks the large
entries of the sample covariance downward.8
As already mentioned, factor analysis and PCA in particular play a prominent role in the literature. It appears that while eigenvector bias is acknowledged, direct9 bias corrections are made only to the eigenvalues corresponding
to the principal components (e.g. Ledoit & Péché (2011) and Wang & Fan
(2017)). Some work on characterizing the behavior of sample eigenvectors may
be found in Paul (2007) and Shen, Shen, Zhu & Marron (2016). In the setting
of Markowitz portfolios, the impact of eigenvalue bias and optimal corrections
are investigated in in El Karoui et al. (2010) and El Karoui (2013).
Our approach also builds upon several profound contributions in the literature on portfolio composition. In an influential paper, Green & Hollifield
(1992) observe the importance of the structure of the dominant factor to the
composition of minimum variance portfolios. In particular, the “dispersion”
of the dominant factor exposures drives the extreme positions in the portfolio composition. This dispersion is further amplified by estimation error, as
pointed out in earlier work by Blume (1975) (see also Vasicek (1973)). These
early efforts have led to a number of heuristics10 to correct the sample bias of
dominant factor estimates.

2

Problem formulation

We address the impact of estimation error on Markowitz portfolio optimization.
To streamline the exposition, we restrict our focus to the minumum variance
b estimated from T obserportfilio. In particular, given a covariance matrix Σ
vations of returns to N securities, we consider the optimization problem,
b
min w> Σw

w∈RN
>

(1)

w 1N = 1.

We denote by wb the solution to (1), the estimated minimum variance portfolio.
Throughout, 1N is the N -vector of all ones. It is well-known that the portfolio
b are extremely sensitive to errors in the estimated model, and risk
weights, w,
8

This is generalized and analyzed further in DeMiguel, Garlappi, Nogales & Uppal (2009).
Several approaches to alter the sample eigenvectors indirectly (e.g. shrinking the sample
towards some structured covariance) do exist. However, the analysis of these approaches is
not focused on characterizing the bias inherent to the sample eigenvectors themselves.
10
For example, the Blume and Vasicek (beta) adjustments. See the discussion of Exhibit
3 and footnote 7 in Clarke, De Silva & Thorley (2011).
9

5

forecasts for the optimized portfolio tend to be too low.11 We aim to address
these issues in a high-dimension, low sample-size regime, i.e., T  N .
We are also interested in the equally weigthed portfolio where we = 1N /N ,
a very simple non-optimized portfolio frequenly employed as a benchmark. We
use this benchmark to test whether the corrections we make for improving the
minimum variance portfolio are not offset by degraded performance elsewhere.

2.1

Model specification & assumptions

We consider a one-factor, linear model for returns to N ∈ N securities. Here, a
generating process for the N -vector of excess returns R takes the form
R = φβ + 

(2)

where φ is the return to the factor, β = (β1 , . . . , βN ) is the vector of factor
exposures, and  = (1 , . . . , N ) is the vector of diversifiable specific returns.
While the returns (φ, ) ∈ R × RN are random, we treat each exposure βn ∈ R
as a constant to be estimated. Assuming φ and the {n } are mean zero and
pairwise uncorrelated, the N × N covariance matrix of R can be expressed as
Σ = σ 2 ββ > + ∆.

(3)

Here, σ 2 is the variance of φ and ∆ a diagonal matrix with nth entry ∆nn = δn2 ,
the variance of n . Estimation of Σ is central to numerous applications.
We consider a setting in which T observations {Rt } Tt=1 of the vector R are
generated by a latent time-series {φt , t } Tt=1 of (φ, ). It is standard to assume
the observations are i.i.d. and we do so throughout. Finite-sample error distorts
measurement of the parameters (σ 2, β, δ 2 ) leading to the estimate,
c
b = σ̂ 2 β̂ β̂ > + ∆,
Σ

(4)

which approximates (3) by using the estimated model (σ̂ 2, β̂, δ̂ 2 ).
Without loss of generality we assume the following condition throughout.
Condition 2.1. Both β and any estimate β̂ are normalized as kβk = kβ̂k = 1.
We require further statistical and regularity assumptions on our model for
our techical results. These conditions stem from our use of recent work on
spiked covariance models (Wang & Fan 2017). Some may be relaxed in various
ways. Our numerical results (Section 5) investigate a much more general setup.
2
Assumption 2.2. The factor variance σ 2 = σN
satisfies TNσ2 → c1 as N ↑ ∞
N
for fixed integer T ≥ 1 and c1 ∈ (0, ∞). Also, ∆ = δ 2 I for fixed δ ∈ (0, ∞).
11

Extreme sensitivity of portfolio weights to estimation error and the downward bias of risk
forecasts are also found in the optimized portfolios constructed by asset managers. Portfolio
specific corrections of the dispersion bias discussed in this article are useful in addressing
these practical problems. The focus on the global minimum variance portfolio in this article
highlights the essential logic of our analysis in the simplest possible setting.

6

Assumption 2.3. The returns {Rt } Tt=1 are i.i.d. with R1 ∼ N (0, Σ).
√
Assumption 2.4. For z = 1N / N we have supN γβ,z < 1 and supN γβ̂,z < 1.
Also, β and β̂ are oriented in such a way that β > z and β̂>z are nonnegative.
The requirement in Assumption 2.2 that the factor variance grow in dimension while the specific variance stays bounded is pervasive in the factor
modeling literature (Bai & Ng 2008). The extra requirement that the specific
risk is homogenous (i.e. ∆ is a scalar matrix) is restrictive but shortens the
technical proofs significanly. It is also commonplace to the spiked covariance
model literature.12 We discuss the adjustments required for heterogenous specific risk (i.e. ∆ diagonal) in Section 4.2. The distributional Assumption 2.3
facilitates several steps in the proofs. In particular, it allows for an elegant characterization of the systematic bias in PCA, i.e., the bias in the first eigenvector
of the sample covariance matrix of the returns {Rt } (see Section 2.2). Assumption 2.4 is not much of a restriction. First, all results can be easily extended
to the case β = z, which is simply a point of singularity and thus requires a
separate treatment. The orientation requirement is essentially without loss of
generality. We will see (in Section 3) that the vector z plays a special role in
our bias correction procedure. And if β>z < 0, we would simply consider −z.

2.2

PCA bias characterization

We consider PCA as the starting point for our analysis as its use for risk
factor identification is widespread in the literature. It is appropriate when σ 2
is much larger than k∆k (e.g., Assumption 2.2). Assembling a data matrix
R = (R1 , . . . , RT ), we will denote the (data) sample covariance matrix by
S = T −1 RR> . PCA identifies σ̂ 2 with the largest eigenvalue of S and β̂ with the
corresponding eigenvector (the first principal component). The diagonal matrix
c = diag(S − σ̂ 2 β̂ β̂ > ), which corresponds to
of specific risks ∆ is estimated as ∆
least-squares regression of R onto the estimated factors. Finally, the estimate
b of the covariance Σ is assembled as in (4).
Σ
Bias in the basic PCA estimator above arises from the use of the sample13
covariance matrix S. We focus on the high-dimension, low sample size regime
which is most appropriate for the practical applications we consider. Asymptotically, T is fixed and N ↑ ∞, a regime in which the PCA estimates of σ 2 and
β are not consistent. We summarize some recent results from the literature
below. We also state our characterization of PCA bias as it pertains to its
pricipal components. Our result is novel in that it suggests a remedy.
12

In particular, our need for this condition stem from our use of results from Shen et al.
(2016). The assumption may be relaxed by imposing regularity conditions on the entries
of ∆ at the expense of a more cumbersome exposition. It may also be removed entirely if
we consider the regime N, T ↑ ∞ as is more common in the literature. We do not pursue
this because many important pratical applications are restricted to only a small number of
observations.
13
If Σ replaces S, sample bias vanishes and the estimator is asymptotically exact as N ↑ ∞.

7

(b) θβ̂,z > θβ,z w.h.p. for large N
(their difference is shaded). PCA
estimates exhibit a systematic
(dispersion) bias relative to vector
z.

(a) β̂ lies near the cone defined by
the Ψ in (5) w.h.p. for large N .
There is no reference frame to detect the bias since β is unknown.

√
Figure 1: A unit sphere with example vectors β, β̂ and z = 1N / N . The
vector z provides a reference frame in which PCA bias may be identified. Note,
the angle θx,y is also the arc-length between points x and y on the unit sphere.
Let θβ̂,β denote the angle between β and its PCA estimate β̂. Shen et al.
(2016) showed, under Assumption 2.2 and mild conditions on the moments of
{Rt }, that there exist random variables Ψ > 1 and ξ ≥ 0 such that14
a.s.

cos θβ̂,β → Ψ−1
2

2 a.s.

2

σ̂ /σ → Ψ ξ

(5)
(6)

as N ↑ ∞ where ξ and Ψ depend on T (fixed). The pair (ξ, Ψ) characterizes
the error in the PCA estimated model asymptotically. As the sample size T increases, both ξ and Ψ approach one almost surely, i.e., PCA supplies consistent
estimates. Since Ψ > 1, the estimate σ̂ 2 tends to be biased upward (whenever
ξ fluctuates around one). Under Assumption 2.3, ξ = χ2T /T where χ2T has
the chi-square distribution with T degrees of freedom. Here, ξ is concentrated
around one with high probability (w.h.p.) for even moderate values of T .
Identifying a (systematic) bias in PCA estimates of β requires a more subtle
analysis. Observe from (5) that Ψ defines a cone near which the estimate β̂ will
lie with high probability for large N (see panel (a) of Figure 1). However, this
does not point to a systematic error that can be corrected.15 Indeed, it is not
apriori clear where on the cone the vector β̂ resides as (5) provides information
14

More precisely, Ψ2 = 1 + δ 2 c1 /ξ where c1 is identified in Assumption 2.2.
Recently, Wang & Fan (2017) provided CLT-type results for the asymptotic distribution
of sample eigenvectors (Theorem 3.2). They remark on the near “impossibility” of correcting
sample eigenvector bias. This follows from their choice of coordinate system (cf. Figure 1).
15

8

only about the angle θβ̂,β away from the unknown vector β. We provide a result
(see Theorem 2.5 below) that sheds
light on this problem.
√
Recall the vector z = 1N / N . We consider, not the angle θβ̂,β between β
and β̂, but the their respective angles θβ,z and θβ̂,z to this reference vector.
Theorem 2.5 (PCA bias). Suppose that Assumptions 2.2, 2.3 and 2.4 hold
a.s.
and let β̂ be a PCA estimate of β. Then, cos θβ,z ∼ Ψ cos θβ̂,z as N ↑ ∞ and,
in particular, we have that θβ̂,z exceeds θβ,z with high probability for large N .
The proof of this result is deferred to Appendix C. It applies the spiked
covariance model results of Shen et al. (2016) and Paul (2007) on sample eigenvectors using a decomposition in terms of the reference vector z.

2.3

Errors in optimized portfolios

Estimation error causes two types of difficulties in optimized portfolios. It distorts portfolio weights, and it biases the risk of optimized portfolios downward.
b constructed as
Both effects are present for the minimum variance portfolio w,
b
the solution to (1) using some estimate Σ of the returns covariance Σ. We now
define the metrics for assessing the magnitude of these two errors.
b
We denote by w∗ the optimal portfolio, i.e., the solution of (1) with Σ
replaced by Σ. Since the latter is positive definite, the optimal portfolio weights
w∗ may be given explicitly.16 We define,
b > Σ (w∗ − w)
b ,
Twb2 = (w∗ − w)

(8)

b Here, T 2 measures the distance between the
the (squared) tracking error of w.
w
b
b Specifically, it is the square of the
optimal and estimated portfolios, w∗ and w.
b
width of the distribution of return differences w∗ − w.
>
b
The variance of portfolio wb is given by wb Σwb and its true variance is
>
b
b We define,
w Σw.

Rwb =

bw
b
wb > Σ
,
wb > Σwb

(9)

the variance forecast ratio. Ratio (9) is less than one when the risk of the
portfolio wb is underforecast.17
Metrics (8) and (9) quantify the errors in portfolio weights and risk forecasts induced by estimation error.18 We analyze Twb and Rwb asymptotically.
16

Indeed, the Sherman-Morrison-Woodbury formula yields an explicit solution even for a
multi-factor model and with a guaranteed mean return (El Karoui 2008). In our setting,
−1

w∗ ∝ ∆

(1N βMV − β) ,

βMV

17

PN
1 + σ 2 k=1 βk2 /δk2
=
.
PN
σ 2 k=1 βk /δk2

(7)

With respect to the equally weighted portfolio, we , (the tracking error of which is zero)
we only consider the variance forecast ratio.
18
For a relationship to more standard error norms see Wang & Fan (2017).

9

√
Again recall z = 1N / N and let γx,y = x> y. Note, γx,y = cos θx,y whenever x
and y lie on the surface of a unit sphere as in Figure 1. Define,
E =

γβ,z − γβ,β̂ γβ̂,z
.
sin2 θβ̂,z

(10)

The variable E drives the asymptotics of our error metrics. Note that E = 0
when β̂ = β. Indeed, it is not difficult to show that for any estimates (σ̂, β̂, δ̂)
satisfying Assumption 2.4, if E is bounded away from zero (i.e. inf N E > 0),
Twb2 ∼ µ2 E 2
Rwb ∼

N −1 δ̂ 2
µ2 E 2 sin2 θβ̂,z

(N ↑ ∞).

(11)

where µ2 = σ 2 /N which is bounded in N . This result (with our all assumptions
above relaxed) is given by Proposition D.1 of Appendix D.
Corollary 2.6 (PCA performance). Suppose Assumptions 2.2, 2.3 and 2.4
b the tracking
hold. For the PCA-estimator of the minimum variance portfolio w,
2
error squared Twb is bounded away from zero and variance forecast ratio Rwb
tends to zero as N ↑ ∞. In particular, for the PCA estimator, the error E in
(10) satisfies
a.s.

E ∼ γβ,z

1 − Ψ−2
2
1 − Ψ−2 γβ,z

!

(12)

as N ↑ ∞ where Ψ > 1 is the random variable in (5).
a.s.

Proof. Expression (12) follows from (5), Theorem 2.5 (i.e. γβ,z ∼ Ψγβ̂,z ) and
2
the identity 1 − γβ̂,z
= sin2 θβ̂,z . The remaining claims follow from (11).

The result states that the variance forecast ratio for the minimum variance
portfolio wb that uses the PCA estimator of Section 2.2 is asymptotically 0.
The estimated risk will be increasingly optimistic as N → ∞. This is entirely
due to estimation error between the sample eigenvector and the population
eigenvector. As N grows, the forecast risk becomes negligible relative to the true
risk, rendering the PCA estimated minimum variance portfolio worse and worse.
The tracking error is also driven by the error of the sample eigenvector and for
increasing dimension, its proximity to the true minimum variance portfolio as
measured by tracking error is asymptotically bounded below (away from zero).

3

Bias correction

Our Theorem 2.5 characterizes the bias of the PCA estimator in terms of the
vector z. This is the unique (up to negation) dispersionless vector on the unit
sphere, i.e., its entries do not vary. Of course when z = β, then the its PCA
10

estimate β̂ will have higher dispersion with probability one. The argument
works along the projection of any β along z, given by γβ,z . Our PCA bias
characterization implies that γβ,z > γβ̂,z , or equivalently, θβ̂,z > θβ,z . with high
probability (for large N ). Figure 1b illustrates this systematic PCA bias and
clearly suggests a correction: a shrinkage of the PCA estimate β̂ towards z.

3.1

Intuition for the correction

Given an estimate β̂, we analyze a parametrized correction of the form
β̂ (ρ) =

β̂ + ρz
kβ̂ + ρzk

(13)

.

for ρ ∈ R. The curve β̂ (ρ) represents a geodesic between −z and z that passes
through β̂ as ρ passes the origin. We select the optimal “shrinkage” parameter
ρ∗ as the ρ that minimizes the error in our metrics Twb2 and Rwb asymptotically.
With SN −1 denoting the unit N -sphere, we define the space
n

o

Sβ = x ∈ SN −1 : γβ,z − γx,z γx,β = 0 .

(14)

Lemma 3.1. Let β 6= z. There is a ρ∗ such that β̂(ρ∗ ) ∈ Sβ and β̂ (ρ∗ ) 6= z.
Proof. By the spherical law of cosines, we obtain
γβ,z − γx,z γx,β = sin θx,z sin θx,β cos κ

(15)

where κ is the angle (in SN −1 ) between the geodesic from β̂ to z and the one
from β̂ to β (see Figure 2a). Write x = β̂ (ρ) and κ = κρ . Then, cos κρ = 0
when ρ = ρ∗ for which the geodesic between β̂ (ρ∗ ) and β is perperdicular to
that between β̂ and z.19 By construction, β̂(ρ∗ ) is not z and is in Sβ .

We observe that β̂ (ρ∗ ) of Lemma 3.1 minimizes the asymptotic error of
our metrics. In particular, replacing β̂ with β̂ (ρ∗ ) ensures that E is zero. Note
that β̂ (ρ∗ ) 6= z (unless β = z), i.e., we do not shrink β̂ all the way to z. While
z ∈ Sβ , when β 6= z, the asymptotic error E explodes as β̂ (ρ) → z.
In a setting where β̂ is the PCA estimator, observe that our selection of
∗
β̂ (ρ ) does more than just correct PCA bias. Theorem 2.5, under the proper
a.s.
assumptions, states that γβ,z ∼ Ψγβ̂,z for a random variable Ψ > 1. This suggests taking ρ such that γβ,z equals γβ̂(ρ),z . This choice is not optimal however
as it lies on the contour {x ∈ SN −1 : γx,z = γβ,z } (see Figure 2b). It is not in
Sβ unless β = z and its asymptotic error E is bounded away from zero.
19

If β̂ and β do not lie on the same side of the sphere we must amend (14) by replacing β̂
with −β̂. Note, γβ,z − γ·,z γ·,β has the same value for x and −x.

11

(b) Setting the angle κ = π/2 corresponds to setting the error driving
term E = 0. Thus, β̂ (ρ∗ ) ∈ Sβ .

(a) The spherical law of cosines states:
γβ,z − γβ̂,z γβ̂,β = sin θβ̂,z sin θβ̂,β cos κ
where κ is the angle of the corner opposite θβ,z , arc-length between β and
z.

Figure 2: Illustration of the spherical law of cosines and the geodesic β̂ (ρ)
between the points β̂ and z along which we shrink the PCA estimate β̂.

3.2

Statement of the main theorem

As noted above, we can improve tracking error and variance forecast ratio by
reducing the angle between the estimated eigenvector and the true underlying
eigenvector, β, equivalently, by replacing β̂ with an appropriate choice of β̂ρ .
We find an optimal value ρ∗N for a particular N . We present our method
and its impact on tracking error and variance forecast ratio for a minimum
variance portfolio below. We restrict to the case of homogeneous specific risk
where ∆ = δ 2 I for expositional purposes but consider the full- fledged case in
empirical results in Section 5.
In the following theorem, we also provide a correction to the sample eigenvalue. Our bias correction for the sample eigenvector introduces a bias in the
variance forecast ratio for the equally weighted portfolio. We shrink the sample eigenvalue, treated as the variance of the estimated factor, to debias the
variance forecast ratio for the equally weighted portfolio.
Theorem 3.2. Suppose Assumptions 2.2, 2.3, 2.4 hold and denote by δ̂ any
estimator of the specific risk δ. Define the oracle corrected estimate by β̂ρ∗ :=
β̂ρ∗N where the finite sample (fixed N and T ) optimal value ρ∗N solves the equation
0 = rβ̂ρ − γβ,β̂ρ (or equivalently, maximizes γβ,β̂ρ ).
i. The finite sample optimal oracle ρ∗N is given by,
ρ∗N =

γβ,z − γβ,β̂ γβ̂,z
.
γβ,β̂ − γβ,z γβ̂,z
12

(16)

ii. For the oracle value ρ∗N , the tracking error and forecast variance ratio for
b ∗ = σ̂ 2 β̂ ∗ β̂ T∗ + δ̂ 2 I satisfy,
the minimum variance portfolio for Σ
ρN
ρ ρ
a.s.
Twb2 ∼

2
γβ̂2 ∗ ,z − γβ,z
1 2
ρ
δ
2
N (1 − γβ̂2 ∗ ,z )(1 − γβ,z
)

(17)

ρ

a.s.

Rwb ∼ δ̂ /δ 2 .
2

(18)

That is, after the optimal correction, the forecast variance ratio for the
minimum variance portfolio no longer converges to 0 while the tracking
error to the true minimum variance portfolio does.
a.s.

iii. For T fixed, N → ∞, we have, ρ∗N ∼ ρ̄N where
ρ̄N =


γβ,z 
−1
Ψ
−
Ψ
.
2
1 − γβ,z

(19)

where Ψ2 = 1 + δ 2 c1 /ξ, ξ = χ2T /T , and χ2T is the chi-squared distribution
with T degrees of freedom. Also, ρ̄N > 0 almost surely if γβ,z > 0. And the
asymptotic improvement of the optimal angles θβ,β̂ρ∗ and θβ,β̂ρ̄ over the
N
original angle θβ,β̂ as N → ∞ is,
sin2 θβ,β̂ρ∗

N

sin2 θβ,β̂

2
sin2 θβ,β̂ρ̄
1 − γβ,z
N
∼
.
γβ,z 2 =
2
1−( Ψ )
sin θβ,β̂

a.s.

Remark 3.3. Geometrically, there are two views of β̂ρ∗ . One is that β̂ρ∗ is the
projection of β̂ onto Sβ . The other is that β̂ρ∗ is the projection of β onto the
geodesic defined by β̂ρ . In either case, our goal is to find the intersection of the
geodesic and the space Sβ .
Remark 3.4. While we consider a specific target, z, in principle the target does
not matter. It is possible that these kind of factor corrections can be applied
beyond the first factor, given enough information to create a reasonable prior.
The first takeaway from this result is that in the high dimensional limit,
it is always possible to improve on the PCA estimate by moving along the
geodesic between β̂ and z. As γβ,z approaches 0 or for a larger T , the optimal
correction approaches 0. Conversely, as γβ,z approaches 0 or for smaller T ,
the magnitude of the correction is larger. For γβ,z = 1, the proper choice is
naturally to choose z since β and z are aligned in that case.
The improvement in the angle as measured by the ratio of squared sines
2
is bounded in the interval (1 − γβ,z
, 1). As γβ,z approaches 0 or for larger T ,
the improvement diminishes and the ratio approaches 1. Conversely, for large
2
values of c1 , the improvement approaches 1 − γβ,z
, indicating that improvement
is naturally constrained by how close β is to z in the first place.
13

In the application to the minimum variance portfolio, the initial idea is to
correct the sample eigenvector so that we reduce the angle to the population
eigenvector. However, it is not immediately clear that this should have a dramatic effect. Even more surprising is that underestimation of risk has a large
component due to sample eigenvector bias and not any sample eigenvalue bias.
While an improved estimate β̂ρ has the potential to greatly improve forecast
risk, this represents only a single dimension on which to evaluate the performance of a portfolio. We could be sacrificing small tracking error to the true
long-short minimum variance portfolio in exchange for better forecasting. That
however is not the case here.
Since ξ and Ψ are unobservable non-degenerate random variables, determining their realized values, even with asymptotic estimates, is an impossible
task. Hence perfect corrections to kill off the driving term of underestimation of
risk are not possible. However, it is possible to make corrections that materially
improve risk forecasts.

3.3

Eigenvalue corrections

Our bias correction, based on formula (14), adjusts the dominant eigenvector
of the sample covariance matrix S. It does not involve standard corrections to
the eigenvalues of S, which are well known to be biased. This distinguishes our
results from the existing literature (see Section 1.2).
For the purpose of improving accuracy of minimum variance portfolios,
there is no need to adjust the dominant eigenvalue. As shown in formulas
(11) of Section 2.3, the main drivers of T 2 and R for large minimum variance
portfolios do not depend on the dominant eigenvalue of S when returns follow
a one-factor model. This is a particular feature of minimum variance, where
the dominant factor is effectively hedged.
Since our correction removes a systematic form of bias, it can be used to
improve accuracy of other portfolios. If these portfolios have substantial factor
exposure, however, a compatibility adjustment to the dominant eigenvalue may
be required. As our eigenvector adjustment, the compatibility adjustment to
the eigenvalue is distinct from the eigenvalue corrections in the literature.
Here, we provide some discussion of compatible eigenvalue corrections for
“simple” portfolios, i.e., those that do not depend on the realized matrix of
returns R. Note, that for these weights, the tracking error T is zero, so we
treat the risk forecast ratio R only.
Under assumptions on our one-factor model that hold for most cases of
interest, one can write, for a simple portfolio w that
Rw ∼

2
σ̂N
C (w, β, β̂) 2
2
σN

(20)

where C (w, β, β̂) = (w> β)/(w> β̂). Our correction addresses only the quantity C, but asymptotic formula (20) reveals that for simple portfolios, sample
14

eigenvalues play a material role. Another difference between simple portfolios
and minimum variance is that the estimate δ̂ 2 of δ 2 does not play any role; the
factor risk is all that matters. From the discusion in Section 2.2 we know that
2
2
for large N , but it is not apriori clear that an
tends to be larger than σN
σ̂N
2
eigenvalue correction should aim to lower σ̂N
. This would depend on the behavior of the coefficient C (w, β, β̂) given the estimate β̂ and the simple portfolio w
2
will adjust risk forecast raat hand. Moreover, a correction that decreases σ̂N
tios downward, potentially leading to unintended underforecasts. Thus, sample
eigenvalue corrections should be coupled with those for the sample eigenvectors
to balance their respective terms in (20).
We state a sharp result in this direction for the equally weighted portfolio,
a widely used simple portfolio.
Proposition 3.5. Suppose Assumptions 2.2, 2.3, 2.4 hold. Let we = 1N /N
and define the corrected eigenvalue via
σ̂ρ2∗N =



γβ̂,z 2 2
σ̂N
γβ̂ρ∗ ,z

where β̂ρ∗ is our corrected PCA estimate β̂ of Theorem 3.2. Then, the forecast
a.s.
variance ratio satisfies Rw ∼ χ2T /T as N → ∞ where χ2T has a chi-squared
distribution with T degrees of freedom.
2
downward since by design since γβ̂,z ≤ γβ̂ρ∗ ,z .
Note that we adjust σ̂N

4

Algorithm and extensions

For the precise statement of our algorithm see Appendix A. In what follows we
address data-driven corrections for the case ∆ = δ 2 I of Theorem 3.2 as well as
the extension to heterogeneous specific risk.

4.1

Data-driven estimator for homogenous specific risk

We introduce procedure for constructing an estimator ρ̂ for the asymptotic
oracle correction parameter ρ̄N given in (19). It is based on estimates the
specific variance δ 2 and c1 from Assumption 2.2. From Yata & Aoshima (2012),
we have the estimator given by,
δ̂ 2 =

Tr(S) − λ̂1
,
N −1− N
T

(21)

where S is the sample covariance matrix for the data matrix R and λ̂1 = σ̂ 2 is
the first eigenvalue of S. A natural estimate for the true eigenvalue λ1 (Σ) is
b S = max{λ̂ − δ̂ 2 N/T, 0}. For λ̂ sufficiently large, the estimate of c is given
λ
1
1
1
1

15

by,
ĉ1 =

N
T λˆ1 − N δ̂ 2

(22)

.

Given the estimates of δ 2 and c1 , we need a precise value for ξ, as well as Ψ,
in order to have a data-driven estimator. We approximate ξ by its expectation
E[ξ] = 1 to obtain a completely data driven correction parameter estimate ρ̂,
ρ̂ =

b
Ψγ
β̂,z



b
2
1 − (Ψγ
β̂,z )



b −Ψ
b −1 ,
Ψ

(23)

b 2 = 1 + δ̂ 2 ĉ .
where Ψ
1
We compute the factor variance as,

σ̂ρ̂2 = Φ2ρ̂ σ̂ 2 ,

Φρ̂ =

γβ̂,z
,
γβ̂ρ̂ ,z

where σ̂ 2 = λ̂1 is the first sample eigenvalue from S.

4.2

An extension to heterogenous specific risk

Our analysis thus far has rested on the simplifying assumption that security
return specific variances have a common value. Empirically, this is not the case,
and the numerical experiments discussed below in Section 5 allow for the more
complex and realistic case of heterogenous specific variances. To address the
issue, we modify both the oracle estimator and the data-driven estimator by
rescaling betas by specific variance.
Under heterogeneous specific variance, the oracle value ρ∗N is given by the
formula,
ρ∗N

=

b
b ∆
∆
∆
− γβ,
γb
γβ,z
β̂ β̂,z
b
b ∆
b
∆
∆
γβ,
− γβ,z
γβ̂,z
β̂

,

(24)

q

b
b
b ∆
∆
c −1 y/ γ ∆
where γx,y
= xT ∆
x,x γy,y is a weighted inner product. Furthermore, the
e given by,
risk adjusted returns R∆−1/2 have covariance Σ
e = σ 2 ∆−1/2 ββ > ∆−1/2 + I.
Σ

For the risk adjusted returns, Theorem 3.2 holds. The oracle formula coupled
c −1/2 as the data matrix
f = R∆
with the risk adjusted returns suggest we use R
c is the specific risk estimate from the standard PCA method. Why
where ∆
should we expect this to work? The purpose of the scaling R∆−1/2 is to make
f approximates that. Since we
the specific return distribution isotropic, and R
are only trying to obtain an estimator ρ̂ that is close to ρ∗N , this approximation
16

ends up fine. And for ellipses specified by ∆ with relatively low eccentricity, the
estimator in (23) actually works in practice since the distribution is relatively
close to isotropic. So for larger eccentricity, we require the following adjustment
just to get the data closer to an isotropic specific return distribution.
The updated formulas for the heterogenous specific risk correction estimators are given below, and we use them in our numerical experiments. For an
c the modified quantities are,
initial estimate of specific risk ∆,
c −1/2 β̂
∆
˜
,
β̂ = c
k∆−1/2 β̂k2
c −1/2 S∆
c −1/2 ,
e =∆
S
˜
e − λ̂
Tr(S)
˜
1
δ̂ 2 =
,
N −1− N
T

ρ̂ =

c −1/2 z
∆
z̃ = c
,
k∆−1/2 zk2
˜
c −1/2 β̂k2 ,
λ̂1 = λ̂1 k∆

N

(25)
(26)

ĉ1 =

˜
˜ ,
T λ̂1 − N δ̂ 2

(27)





(28)

b
Ψγ
˜
β̂,z̃
b
2
1 − (Ψγ
˜ )
β̂,z̃

b −Ψ
b −1 .
Ψ

c = diag(S − σ̂ 2 β̂ β̂ > ) as the
We use the PCA estimate of the specific risk ∆
initial estimator.
Once we have the estimated ρ̂, we return to the original data matrix R
and apply the correction as before to β̂, the first eigenvector of the sample
covariance matrix. The method for correcting the sample eigenvalue remains
the same and we opt to recompute the specific variances using the corrected
factor exposures and variance.

5

Numerical study

We use simulation to quantify the dispersion bias and its mitigation in minimum
variance and equally portfolios. We design our simulations around a return
generating process that is more realistic than the single-factor, homogenousspecific-risk model featured in Theorems 2.6 and 3.2.20 Returns to N ∈ N
securities are generated by a multi-factor model with heterogenous specific risk,
R = φB + ,

(29)

where φ is the vector of factor returns, B is the matrix of factor exposures, and
 = (1 , 2 , . . . , N ) is the vector of diversifiable specific returns.
Our examples are based on a model with K = 4 factors. For consistency
with Sections 2 and 3, we continue to adopt the mathematically convenient
convention of scaling exposure vectors to have L2 norm 1, and we denote the
20

The simplistic setting for our theoretical results showcases the main theoretical tools used
in their proofs without the distraction of regularity conditions required for generalization.
Global equity risk models used by investors typically include more than 100 factors.

17

vector of exposures to the first factor (the first column of the exposure matrix
B) by β. The recipe for constructing β with a target value γβ,z is to generate a
random vector, rescale the vector to length 1, and then modify the component
in the z direction to have the correct magnitude (while maintaining length 1).
A similar approach would be to construct a random vector β 0 with aveage equal
to 1 and variance equal to τ 2 , where τ 2 is the dispersion of the “market beta,”
and then rescale β 0 to length 1 to obtain β. The parameter τ 2 would control
the concentration of the market betas, just like γβ,z , and tends to be greater in
calmer regimes. The√connection between τ 2 and the dispersion parameter γβ,z
2
= 1/ 1 + τ 2 .
is given by γβ,z
The three remaining factors are fashioned from equity styles such as volatility, earnings yield and size.21 We draw the exposure of each security to each
factor from a mean 0, variance 0.75 normal distribution and again normalize
each vector of factor exposures to have L2 length 1.
We calibrate the risk of the market factor in accordance with Clarke et al.
(2011) and Goldberg, Leshem & Geddes (2014); both report the annualized
volatility of the US market to be roughly 16%, based on estimates that rely
on decades of data. We calibrate the risk of factors 2, 3 and 4 in accordance
with Morozov, Wang, Borda & Menchero (2012, Table 4.3), by setting their
annualized volatilities to be 8%, 4% and 4%. We assume that the returns to
the factors are pairwise uncorrelated.
We draw annualized specific volatilities {δn2 } from a uniform distribution
on [32%, 64%]. This range is somewhat broader than the estimated range in
Clarke et al. (2011).22
In each experiment, we simulate a year’s worth of daily returns, T = 250,
to N securities. From this data set, we construct a sample covariance matrix,
S, from which we extract three estimators of the factor covariance matrix Σ.
The first is the data-driven estimator, the implementation specifics of which are
discussed in Section 4.2 and precisely summarized of Appendix A. The second is
the oracle estimator which is the same as the data-driven estimator but with the
true value of γβ,z , the projection of the true factor onto the z-vector, supplied.23
Our third estimator is classical PCA, which is specified in detail in Section 2.2.
We use the three estimated covariance matrices to construct minimum variance
portfolios and to forecast their risk. We also use these covariance matrices to
forecast the risk of an equally weighted portfolio.
In the experiments described below, we vary the number of securities, N ,
and the concentration of the market factor, γβ,z . We run 50 simulations for
each pair, (N, γβ,z ). Results are shown in Figure 3 for a fixed concentration
and varying number of securities, and in Figures 4 and 5 for a fixed number
of securities and varying concentration. Panels (a) and (b) in each figure show
21

Seminal references on the importance of volatility (or beta), earnings yield and size in
explaining cross-sectional correlations is in Black, Jensen & Scholes (1972), Basu (1983) and
Banz (1981).
22
For example, the empirically observed range in Clarke et al. (2011) is [25%, 65%].
23
For the data-driven estimator, this quantity is estimated from the observed data.

18

annualized tracking error and volatility for a minimum variance portfolio. Panels (c) and (d) show variance forecast ratio for a minimum variance portfolio
and an equally weighted portfolio.

Figure 3: Performance statistics for 50 simulated data sets of T = 250 observations as N varies and γβ,z = 0.9. Panel (a): Annualized tracking error for
minimum variance portfolios. Panel (b): Annualized volatility for minimum
variance portfolios. Panel (c): Variance forecast ratio for minimum variance
portfolios. Panel (d): Variance forecast ratio for equally weighted portfolios.
In Figure 3, the concentration of the market factor is γβ,z = 0.9. Panels (a),
(b) and (c) show that for minimum variance portfolios optimized with dispersion bias-corrected PCA models, tracking error and volatility decline materially
as N grows from 500 to 3000, ranges of outcomes compress, and variance forecast ratios are near 1 for all N considered. These desirable effects are less
pronounced, or even absent, in a PCA model without dispersion bias mitigation. A comparison of panels (c) and (d) highlights the difference in accuracy
of risk forecasts between a minimum variance portfolio and an equally weighted
portfolio. Dispersion bias mitigation materially improves variance forecast ratio
for the former and has no discernible impact for the latter.

19

Figure 4: Performance statistics for 50 simulated data sets of T = 250 observations as γβ,z varies and N = 500. Panel (a): Annualized tracking error for
minimum variance portfolios. Panel (b): Annualized volatility for minimum
variance portfolios. Panel (c): Variance forecast ratio for minimum variance
portfolios. Panel (d): Variance forecast ratio for equally weighted portfolios.
In Figure 4, the number of securities is N = 500. Panels (a), (b) and
(c) show that for minimum variance portfolios optimized with PCA models,
tracking error and volatility increase materially as γβ,z grows from 0.5 to 0.9,
variance forecast ratio diminishes, and the ranges of outcomes expand. These
undesirable effects are diminished when the dispersion bias is mitigated.
Results for values γβ,z ∈ [0.9, 1.0] are shown separately in Figure 5. The
severe decline in performance for all risk metrics for γβ,z in this range is a
consequence of the sea change in the composition of a minimum variance portfolio that can occur when the dominant eigenvalue of the covariance matrix
is sufficiently concentrated. Here, the true minimum variance portfolio tends
to lose its short positions.24 Errors in estimation of the dominant factor lead
to long-short optimized portfolios approximating long-only optimal portfolios.
The market factor is hedged in the former but not in the latter, and this discrepancy propagates to the error metrics.
A comparison of panels (c) and (d) in Figures 4 and 5 highlights, again,
the difference in accuracy of risk forecasts between a minimum variance portfolio and an equally weighted portfolio. Dispersion bias mitigation materially
improves variance forecast ratio for the former and has no discernible impact
on the latter.
24

If specific variances are all equal, the minimum variance portfolio becomes equally
weighted when the dominant eigenvector is dispersionless.

20

Figure 5: Performance statistics for 50 simulated data sets of T = 250 observations as γβ,z varies and N = 500. Panel (a): Annualized tracking error for
minimum variance portfolios. Panel (b): Annualized volatility for minimum
variance portfolios. Panel (c): Variance forecast ratio for minimum variance
portfolios. Panel (d): Variance forecast ratio for equally weighted portfolios.
A casual inspection of the figures suggests that the data-driven estimator
performs nearly as well as the oracle. However, there can be substantial differences between the two estimators in some cases, as shown in tracking error and
variance forecast ratio of minimum variance Figure 5 when γβ,z ∈ [0.9, 1.0]. The
origin of the differences can be seen Lemma 3.1. In order for the tracking error
and variance forecast ratio to have good asymptotic properties, the estimator
β̂ must lie in the null space Sβ . This condition is guaranteed for the oracle
estimators but will not generally be satisfied for data-driven estimators.

6

Summary

In this article, we develop a correction for bias in PCA-based covariance matrix
estimators. The bias is excess dispersion in a dominant eigenvector, and the
form of the correction is suggested by formulas for estimation error metrics
applied to minimum variance portfolios.
We identify an oracle correction that optimally shrinks the sample eigenvector along a spherical geodesic toward the distinguished zero-dispersion vector, and we provide asymptotic guarantees that oracle shrinkage reduces both
types of error. These findings are especially relevant to equity return covariance matrices, which feature a dominant factor whose overwhelmingly positive
exposures tend to be overly dispersed by PCA estimators.
Our results fit into two streams of academic literature. The first is the
large-N -fixed-T branch of random matrix theory, which belongs to statistics.
The second is empirical finance, which features results about Bayesian adjust21

ments to estimated betas.
To enable practitioners to use our results, we develop a data-driven estimator of the oracle. Simulation experiments support the practical value of our
correction, but much work remains to be done. That includes the development
of estimates of the size and likelihood of the exposure bias in finite samples,
the identification and correction of biases in other risk factors, and empirical
studies. Explicit formulas for error metrics in combination with the geometric
perspective in this article provide a way to potentially improve construction
and risk forecasts of investable optimized portfolios.

22

A

Algorithm

Our corrected covariance matrix algorithm is given in Algorithm 1 where the
input is a data matrix of returns R.
Algorithm 1 1-Factor Bias Corrected PCA Covariance Estimator
Require: R = (R1√, . . . , RT )
Require: z = 1N / N
1: procedure Bias Corrected PCA Covariance(R)
2:
S ← T1 RRT
3:
Û ← [û1 , . . . , ûN ], Λ̂ ← Diag(λ̂1 , . . . , λ̂N ) . Eigendecomposition of S
. Orient such that β̂ T z > 0.
4:
β̂ ← sign(ûT1 z)û1
5:
σ̂ 2 ← λ̂1
c ← Diag(S − L),
b
b ← σ̂ 2 β̂ β̂ T
6:
∆
L
. Initial PCA estimate.
˜
c −1/2 β̂k2
c −1/2 S∆
c −1/2 ,
e ←∆
λ̂1 ← λ̂1 k∆
7:
S
−1/2
˜
b
b −1/2 z
∆
β̂
∆
β̂ ← k∆
z̃ ← k∆
b −1/2 β̂k2 ,
b −1/2 zk2
˜
e
˜
S)−λ̂1
b ← 1 + δ̂˜2 ĉ ,
δ̂ 2 ← Tr(
ĉ1 ← ˜ N ˜2
8:
Ψ
N ,
1
N −1−
T λ̂1 −N δ̂

T

9:

β̂+ρ̂z
β̂ρ̂ ← √1+2ρ̂γ

,
+ρ̂2

ρ̂ ←

b
Diag(S − L

b
L

β̂,z

10:

c
∆

ρ̂

←

ρ̂ ),

bγ ˜
Ψ

β̂,z̃

b γ ˜ )2
1−(Ψ



b −Ψ
b −1
Ψ



β̂,z̃

ρ̂

← σ̂ρ̂2 β̂ρ̂ β̂ρ̂T ,

σ̂ρ̂2 ←

γ2

β̂,z

γ2

σ̂ 2

β̂ρ̂ ,z

11:
12:

c
b +∆
return Σ̂ ← L
ρ̂
ρ̂
end procedure

. The 1-factor bias corrected PCA estimator.

23

B

Tables
N

500

1000

1500

2000

2500

3000

PCA
2.35% 2.18% 2.06% 2.01% 2.03% 2.00%
Oracle
1.64% 1.21% 0.97% 0.83% 0.77% 0.70%
Data-Driven 1.66% 1.21% 0.98% 0.84% 0.77% 0.71%
(a) Annualized Tracking Error: Minimum Variance

N

500

1000

1500

2000

2500

3000

PCA
5.10% 3.92% 3.34% 3.03% 2.91% 2.75%
Oracle
4.82% 3.47% 2.80% 2.42% 2.22% 2.01%
Data-Driven 4.82% 3.47% 2.81% 2.42% 2.23% 2.02%
(b) Annualized Volatility: Minimum Variance

N

500

1000

1500

2000

2500

3000

PCA
0.695 0.604 0.541 0.492 0.452 0.413
Oracle
0.963 0.962 0.960 0.961 0.963 0.961
Data-Driven 0.951 0.946 0.943 0.945 0.953 0.947
(c) Forecast Ratio: Minimum Variance

N

500

1000

1500

2000

2500

3000

PCA
0.983 0.991 1.014 1.011 1.001 1.006
Oracle
0.983 0.991 1.015 1.011 1.001 1.006
Data-Driven 0.983 0.991 1.015 1.011 1.001 1.006
(d) Forecast Ratio: Equal Weighted

Table 1: Median values derived from Figure 3. Performance statistics for 50
simulated data sets of T = 250 observations as γβ,z varies and N = 500. Panel
(a): Annualized tracking error for minimum variance portfolios. Panel (b):
Annualized volatility for minimum variance portfolios. Panel (c): Variance
forecast ratio for minimum variance portfolios. Panel (d): Variance forecast
ratio for equally weighted portfolios.

24

γβ,z

0.50

0.60

0.70

0.80

0.90

0.95

1.00

PCA
0.51% 0.59% 0.88% 1.26% 2.35% 3.72% 9.11%
Oracle
0.41% 0.42% 0.67% 0.86% 1.64% 2.91% 0.63%
Data-Driven 0.43% 0.45% 0.70% 0.89% 1.66% 2.94% 2.18%
(a) Annualized Tracking Error: Minimum Variance

0.50

γβ,z

0.60

0.70

0.80

0.90

0.95

1.00

PCA
2.42% 2.57% 2.99% 3.62% 5.10% 7.02% 20.95%
Oracle
2.40% 2.54% 2.94% 3.50% 4.82% 6.63% 18.88%
Data-Driven 2.41% 2.54% 2.95% 3.51% 4.82% 6.65% 18.99%
(b) Annualized Volatility: Minimum Variance

γβ,z

0.50

0.60

0.70

0.80

0.90

0.95

1.00

PCA
0.915 0.904 0.860 0.816 0.695 0.579 0.264
Oracle
0.958 0.965 0.949 0.959 0.963 0.967 1.001
Data-Driven 0.955 0.959 0.945 0.949 0.951 0.928 0.967
(c) Forecast Ratio: Minimum Variance

γβ,z

0.50

0.60

0.70

0.80

0.90

0.95

1.00

PCA
1.019 1.015 1.035 0.984 0.983 1.04 1.002
Oracle
1.020 1.015 1.035 0.984 0.983 1.04 1.002
Data-Driven 1.020 1.015 1.035 0.984 0.983 1.04 1.002
(d) Forecast Ratio: Equal Weighted

Table 2: Median values derived from Figures 4 and 5. Performance statistics
for 50 simulated data sets of T = 250 observations as γβ,z varies and N = 500.
Panel (a): Annualized tracking error for minimum variance portfolios. Panel
(b): Annualized volatility for minimum variance portfolios. Panel (c): Variance
forecast ratio for minimum variance portfolios. Panel (d): Variance forecast
ratio for equally weighted portfolios.

25

C

Proof of main results

We start off with some foundational asymptotic results from the literature. Let
X ∼ N (0, Λ) where Λ = diag(λ1 , λ2 , . . . , λ2 ) is a diagonal matrix satisfying
with λ1 satisying Assumption C1 in Shen et al. (2016). Let SX = T1 XXT be
the sample covariance for X with eigendecomposition SX = V̂Λ̂V̂T . Further
let v̂1 be the first sample eigenvector given by,




v̂11

v̂1 =  . . . 
,
v̂N 1
iT

h

and define ṽ = v̂21 . . . v̂N 1 . By Paul (2007, Theorem 6), ṽ ∼ Unif(B(N −
2)) where B(N − 2) is a unit N − 2 sphere.
Via a simple scaling by λ2 , by Shen et al. (2016, Theorem 6.3) we have
a.s.
T
e1 v̂1 = v̂11 → Ψ−1 where Ψ2 = 1 + λ2 c1 /ξ and ξ = χ2T /T .
We introduce the following lemma, which we will use in the proofs of our
results. We leave its proofs until the end.
Lemma C.1. If Xn ∼ Unif(B(n − 1)) and Yn ∈ B(n − 1) is a sequence
a.s.
independent of Xn , then XnT Yn → 0 as n → ∞.
Proof of Theorem 2.5. Let X = UR where U is the matrix of eigenvectors
(β, u2 , . . . uN ) of Σ so that Cov(X) = Λ as introduced in the beginning of this
section. Also as before let SX = T1 XXT = V̂Λ̂V̂T be the sample covariance of
X and its eigendecomposition. Then S = T1 RRT = UV̂Λ̂V̂T UT so that the
first sample eigenvector of S, β̂, is given by,
β̂ = v̂11 β +

N
X

v̂j1 uj .

j=2

Then we have γβ̂,β = v̂11 and,
γβ̂,z = v̂11 γβ,z + ṽ T ωN ,
h

i

where ωN = uT2 z · · · uTN z . As noted before, by Shen et al. (2016, Theorem
a.s.
6.3), v̂11 → Ψ−1 . We know that both kṽk2 and kωN k2 are bounded as,
kṽk2 ≤ 1,

1 = kU T zk22 = (β T z)2 +

X

2
(uTj z)2 = γβ,z
+ kωN k22 .

ṽ
Therefore, from Lemma C.1 for XN = kṽk
and YN =
2
converges almost surely to 0 so we conclude the result.

ωN
kωN k2

we have ṽ T ωN


Proof of Theorem 3.2.
i. It is easy to verify that ρ∗N solves γβ,z − γβ̂(ρ),z γβ̂(ρ),β and maximizes γβ,β̂ρ .
26

ii. For the oracle value ρ∗N , β̂ρ∗N ∈ Sβ so the result follows by Lemma 3.1.
iii. Convergence of ρ∗N to ρ̄N stems from Theorem2.5. It is also clear that ρ̄N
if γβ,z > 0 since Ψ − Ψ−1 > 0.
For the asymptotic improvement due to shrinkage, we rely on Björck &
Golub (1973, Theorem 1), which shows that principal angle can be derived
from the singular value decomposition of,
"

β

T

#

β̂−γβ̂,z z

z q

"

= γβ,z

1−γ 2

γβ,β̂ −γβ̂,z γβ,z

q

#

1−γ 2

.

β̂,z

β̂,z

By maximizing γβ,β̂ρ (or equivalently minimizing the angle between β and
βρ ), we are directly choosing ρ∗N such that β̂ρ∗N is the principal vector with
corresponding principal angle to β̂. Finding the vector in terms of correction quantity is easier through direct maximization of γβ,β̂ρ , and finding the
improvement is easier through the principal angle computation, despite the
results being equivalent.
From the above product, the squared cosine of the principal angle and its
asymptotic value is,
γβ,β̂ −γβ̂,z γβ,z

"
2
γβ,
β̂ρ∗

γβ,z

=

q

1−γ 2

β̂,z

N

2
= γβ,z
+

a.s.

2

(γβ,β̂ − γβ̂,z γβ,z )2
2
1 − γβ̂,z

2
∼ γβ,z
+

2
= γβ,
β̂ρ̄

# 2

2 2
Ψ−2 (1 − γβ,z
)
γβ,z 2
1−( Ψ )

N

a.s.

2
Since sin2 θβ,β̂ = 1 − γβ,
→ 1 − Ψ−2 , we see clearly that,
β̂

sin2 θβ,β̂ρ∗

N

sin2 θβ,β̂

2
sin2 θβ,β̂ρ̄
1 − γβ,z
N
∼
γβ,z 2 =
2
1−( Ψ )
sin θβ,β̂

a.s.


Proof of Lemma C.1. By orthogonal invariance of Xn and the independence of Yn ,
D
XnT Yn = XnT QTn Qn Yn = (Qn Xn )1 = Xn1
where Qn is an orthongal matrix such that Qn Yn = e1 , e1 is the first canonical
vector, and Xn1 is the first entry of Xn . We know from Muller (1959) for Xn1 ,
D
Xn1 = q

Z1
Z12 + χ2n−1

27

,

where Z1 ∼ N (0, 1) and is independent of χ2n−1 . We have,


4
h i
EXn1
1
n
P(|Xn1 | ≥ ) ≤
≤
E
Z14 E  2
4
4
2

n
χn

!2 


≤

C
,
n2

where the inverse chi-squared distribution has finite moment for n large enough
and C is some constant related to the moments of the standard normal distribution and the inverse chi-squared distribution. By the Borel-Cantelli lemma,
we conclude the result.

Proof of Proposition 3.5. For the equally weighted portfolio w =
using β̂ρ∗N ,
σ̂ρ2∗N N γβ̂2

ρ∗ ,z
N

Rw =

where

σ̂ρ2∗N

we get,

=

σ̂ρ2∗N γβ̂2

ρ∗ ,z
N

2
σ 2 γβ,z

,

!2

γβ̂,z
γβ̂

+ δ̂ 2 N1

2
σ 2 N γβ,z
+ δ 2 N1

a.s.

∼

1
1
N N

σ̂ 2 is the corrected factor variance. Using Theorem 2.5,

ρ∗ ,z
N

a.s.

Rw ∼

2
σ 2 ξΨ2 Ψ−2 γβ,z
a.s.
∼ ξ.
2
2
σ γβ,z



D

Asymptotic estimates

Throughout, we assume Assumption 2.2. The estimates (σ̂, β̂, δ̂), are general
(but with kβ̂k = 1) and not necessarily from PCA. In our setting, formula (7)
for the minimum variance portfolio simplifies to (cf. Clarke et al. (2011))
w∗ ∝ βMV z − β where βMV =

σ2 + δ2
.
σ 2 γβ,z

Analogously, wb and βbMV are defined in terms of the estimates (σ̂, β̂, δ̂). The
normalizing factor for the portolio weights that ensures 1>
N w∗ = 1 is given by
√
W = N (βMV − γβ,z ) .
(30)

28

We develop (N ↑ ∞)-asymtotics for the tracking error T and variance ratio R
in equations (8)–(9) for our simple one-factor model. It is easy to see that
b 2 + δ̂ 2 kwk
b 22
σ̂ 2 (β̂ > w)
,
b 2 + δ 2 kwk
b 22
σ 2 (β > w)
b 22 .
Twb2 = σ 2 (β > wb − β > w∗ )2 + δ 2 kw∗ − wk

Rwb =

(31)
(32)

Note that all quantities involved depend on N . We suppress this dependence
to ease the notation. Recall E in (10) defined as
E =

γβ,z − γβ,β̂ γβ̂,z
.
2
1 − γβ̂,z

Proposition D.1 (Asymptotics). Suppose Assumption 2.2 holds. Let µ2 =
σ 2 /N and µ̂2 = σ̂ 2 /N and assume that (µ̂, δ̂) are bounded in N .
(i) Suppose that supN γβ,z < 1 and supN γβ̂,z < 1. Then,
Rwb ∼

δ̂ 2
δ 2 + µ2 E 2 N sin2 θβ̂,z

Twb2 ∼ µ2 E 2 + δ 2 N −1

(N ↑ ∞).

2
2
(γβ̂,z
− γβ,z
)

(33)

2
2
(1 − γβ,z
) (1 − γβ̂,z
)

(ii) Suppose that supN γβ,z < 1 and γβ̂,z = 1 eventually in N . Then,
Rwb ∼

µ̂2
2
µ2 γβ,z

Twb2

2
µ2 γβ,z

∼

(N ↑ ∞).

(34)

(iii) Suppose that γβ,z = 1 eventually in N and supN γβ̂,z < 1. Then,




δ̂ 2 /µ2 
Rwb ∼ N −1  2
sin θβ̂,z


Twb2 = N −1 

2
δ 2 γβ̂,z



sin2 θβ̂,z



(N ↑ ∞).

(35)

(iv) Suppose that both γβ,z = 1 and γβ̂,z = 1 eventually in N . Then, Twb2 = 0
eventually in N and Rwb ∼ σ̂ 2 /σ 2 as N ↑ ∞.
Proof. All claims follow from the collection of Lemmas below.

29



It is not difficult to show the following identities in our setting.
>

2

2

σ (β w∗ ) = µ

!2

δ 2 γβ,z
2
δ 2 + σ 2 (1 − γβ,z
)

2


b 2 = µ2 
σ 2 (β > w)

γβ,z δ̂ 2 + σ̂ 2 (γβ,z − γβ,β̂ γβ̂,z )
2
)
δ̂ 2 + σ̂ 2 (1 − γβ̂,z

w∗> w∗ =

2
(σ 2 + δ 2 )2 − (σ 2 + 2δ 2 )σ 2 γβ,z
2
N (δ 2 + σ 2 (1 − γβ,z
))2

wb > w∗ =

δ 2 + σ 2 − σ 2 γβ,z E
2
(δ 2 + σ 2 (1 − γβ,z
) )N

E=

2


(36)

(δ̂ 2 + σ̂ 2 )γβ,z − σ̂ 2 γβ̂,z γβ̂,β
2
)
δ̂ 2 + σ̂ 2 (1 − γβ̂,z

These are suffiecient to prove the following Lemmas. As in Propositon D.1 we
set µ2 = σ 2 /N and µ̂2 = σ̂ 2 /N and assume that (µ̂, δ̂) are bounded in N .
Lemma D.2 (True portfolio variance due to factor). When supN γβ,z < 1 and
supN γβ̂,z < 1, the factor components of the true variance of w∗ and wb satisfy
δ4
σ (β w∗ ) ∼ 2 2
µN
2

>

2

γβ,z
2
1 − γβ,z

!2

(N ↑ ∞).

(37)

b 2 ∼ µ2 E 2
σ 2 (β > w)

Lemma D.3 (Porfolio weights). Suppose supN γβ,z < 1 and supN γβ̂,z < 1.
w∗> w∗

∼N

−1

wb w∗ ∼ N

−1

>

1
2
1 − γβ,z

!

1 − γβ,z E
2
1 − γβ,z

!

(N ↑ ∞).

(38)

References
Bai, J. & Ng, S. (2008), ‘Large dimensional factor analysis’, Foundations and
Trends R in Econometrics 3(2), 89–163.
Banz, R. W. (1981), ‘The relationship between return and market value of
common stock’, Journal of Financial Economics 9, 3–18.
Basu, S. (1983), ‘The relationship between earnings’ yield, market value and
return for nyse common stocks’, Journal of Financial Economics 12, 129–
156.

30

Bianchi, S. W., Goldberg, L. R. & Rosenberg, A. (2017), ‘The impact of estimation error on latent factor models of portfolio risk’, The Journal of
Portfolio Management 43(5), 145–156.
Bickel, P. J. & Levina, E. (2008), ‘Covariance regularization by thresholding’,
The Annals of Statistics pp. 2577–2604.
Björck, Ȧ. & Golub, G. H. (1973), ‘Numerical methods for computing angles
between linear subspaces’, Mathematics of computation 27(123), 579–594.
Black, F., Jensen, M. C. & Scholes, M. (1972), The capital asset pricing model:
some empirical tests, in M. C. Jensen, ed., ‘Studies in the Theory of Capital
Markets’, Praeger.
Blume, M. E. (1975), ‘Betas and their regression tendencies’, The Journal of
Finance 30(3), 785–795.
Britten-Jones, M. (1999), ‘The sampling error in estimates of mean-variance
efficient portfolio weights’, The Journal of Finance 54(2), 655–671.
Bun, J., Bouchaud, J. & Potters, M. (2016), ‘Cleaning correlation matrices’,
Risk Magazine .
Clarke, R., De Silva, R. & Thorley, S. (2011), ‘Minimum-variance portfolio
composition’, Journal of Portfolio Management 2(37), 31–45.
Connor, G. & Korajczyk, R. A. (1986), ‘Performance measurement with the arbitrage pricing theory: A new framework for analysis’, Journal of financial
economics 15, 373–394.
Connor, G. & Korajczyk, R. A. (1988), ‘Risk and return in equilibrium apt:
Application of a new test methodology’, Journal of financial economics
21, 255–289.
DeMiguel, V., Garlappi, L., Nogales, F. J. & Uppal, R. (2009), ‘A generalized
approach to portfolio optimization: Improving performance by constraining portfolio norms’, Management Science 55(5), 798–812.
DeMiguel, V., Garlappi, L. & Uppal, R. (2007), ‘Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy?’, The review of
Financial studies 22(5), 1915–1953.
El Karoui, N. (2013), ‘On the realized risk of high-dimensional markowitz portfolios’, SIAM Journal on Financial Mathematics 4(1), 737–783.
El Karoui, N. E. (2008), ‘Spectrum estimation for large dimensional covariance
matrices using random matrix theory’, The Annals of Statistics pp. 2757–
2790.

31

El Karoui, N. et al. (2010), ‘High-dimensionality effects in the markowitz problem and other quadratic programs with linear constraints: risk underestimation’, The Annals of Statistics 38(6), 3487–3566.
Fan, J., Liao, Y. & Mincheva, M. (2013), ‘Large covariance estimation by
thresholding principal orthogonal complements’, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 75(4), 603–680.
Frost, P. A. & Savarino, J. E. (1986), ‘An empirical bayes approach to efficient portfolio selection’, Journal of Financial and Quantitative Analysis
21(3), 293–305.
Gillen, B. J. (2014), ‘An empirical bayesian approach to stein-optimal covariance matrix estimation’, Journal of Empirical Finance 29, 402–420.
Goldberg, L., Leshem, R. & Geddes, P. (2014), ‘Restoring value to minimum
variance’, Journal of Investment Management 12(2), 32–39.
Goldfarb, D. & Iyengar, G. (2003), ‘Robust portfolio selection problems’, Mathematics of operations research 28(1), 1–38.
Green, R. C. & Hollifield, B. (1992), ‘When will mean-variance efficient portfolios be well diversified?’, The Journal of Finance 47(5), 1785–1809.
Jagannathan, R. & Ma, T. (2003), ‘Risk reduction in large portfolios: Why imposing the wrong constraints helps’, The Journal of Finance 58(4), 1651–
1683.
Jobson, J. D. & Korkie, B. (1980), ‘Estimation for markowitz efficient portfolios’, Journal of the American Statistical Association 75(371), 544–554.
Lai, T. L. & Xing, H. (2008), Statistical models and methods for financial markets, Springer.
Lai, T. L., Xing, H. & Chen, Z. (2011), ‘Mean-variance portfolio optimization when means and covariances are unknown’, The Annals of Applied
Statistics pp. 798–823.
Ledoit, O. & Péché, S. (2011), ‘Eigenvectors of some large sample covariance
matix ensembles’, Probability Theory and Related Fields 151(1–2), 233–
264.
Ledoit, O. & Wolf, M. (2003), ‘Improved estimation of the covariance matrix
of stock returns with an application to portfolio selection’, Journal of
empirical finance 10(5), 603–621.
Ledoit, O. & Wolf, M. (2004), ‘Honey, i shrunk the sample covariance matrix’,
The Journal of Portfolio Management 30, 110–119.

32

Markowitz, H. (1952), ‘Portfolio selection’, The Journal of Finance 7(1), 77–91.
Michaud, R. O. & Michaud, R. O. (2008), Efficient asset management: a practical guide to stock portfolio optimization and asset allocation, Oxford University Press.
Morozov, A., Wang, J., Borda, L. & Menchero, J. (2012), The barra global
equity model (gem3). MSCI Barra model documentation.
Muller, M. E. (1959), ‘A note on a method for generating points uniformly on
n-dimensional spheres’, Communications of the ACM 2(4), 19–20.
Onatski, A. (2012), ‘Asymptotics of the principal components estimator of large
factor models with weakly influential factors’, Journal of Econometrics
168(2), 244–258.
Pástor, L. (2000), ‘Portfolio selection and asset pricing models’, The Journal
of Finance 55(1), 179–223.
Paul, D. (2007), ‘Asymptotics of sample eigenstructure for a large dimensional
spiked covariance model’, Statistica Sinica pp. 1617–1642.
Ross, S. A. (1976), ‘The arbitrage theory of capital asset pricing’, Journal of
economic theory 13(3), 341–360.
Sharpe, W. F. (1964), ‘Capital asset prices: A theory of market equilibrium
under conditions of risk’, The Journal of Finance 19(3), 425–442.
Shen, D., Shen, H., Zhu, H. & Marron, S. (2016), ‘The statistics and mathematics of high dimensional low sample size asympotics’, Statistica Sinica
26(4), 1747–1770.
Treynor, J. L. (1962), Toward a theory of market value of risky assets. Presented
to the MIT Finance Faculty Seminar.
Vasicek, O. A. (1973), ‘A note on using cross-sectional information in bayesian
estimation of security betas’, The Journal of Finance 28(5), 1233–1239.
Wang, W. & Fan, J. (2017), ‘Asymptotics of empirical eigenstructure for high
dimensional spiked covariance’, The Annals of Statistics 45(3), 1342–1374.
Yata, K. & Aoshima, M. (2012), ‘Effective pca for high-dimension, low-samplesize data with noise reduction via geometric representations’, Journal of
multivariate analysis 105(1), 193–215.

33

