4), October, 621–651.Butler, C. and H. Davies (1998), “Assessing Market Views on Monetary Policy: The Use of ImpliedRisk-Neutral Probability Distributions”, paper presented to the BIS/CEPR Conference “AssetPrices and Monetary Policy”, January.Campa, J. M. and P. H. K. Chang (1995), “Testing the Expectations Hypothesis on the Term Structure of Volatilities in Foreign Exchange Options”, Journal of Finance, 50, 529–547.Campa, J. M. and P. H. K. Chang (1996), “Arbitrage-based Tests of Target-Zone Credibility: Evidence from ERM Cross-Rate Options”, American Economic Review, 86(4), September.Campa, J. M., P. H. K. Chang and R. L. Reider (1997), “ERM Bandwidths for EMU and After:Evidence from Foreign Exchange Options”, Economic Policy, 24, April, 55–89.Clews, R., N. Panigirtzoglou and J. Proudman (2000), “Recent Developments in Extracting Information from Option Markets”, Bank of England Quarterly Bulletin, Feb, 50–60.Deutsche Bundesbank (1995), “The Information Content of Derivatives for Monetary Policy”,Deutsche Bundesbank Monthly Report, Nov.Dunis, C. and P. Lequeux (2001), “The Information Content of Risk-Reversals”, Derivatives Uses,Trading and Regulation, 2, 98–117.Garman, M. M. and S. W. Kohlhagen (1983), “Foreign Currency Option Values”, Journal of International Money and Finance, 2, December, 231–237.Loretan, M. and W. B. English (2000), “Evaluating Correlation Breakdowns during Periods ofMarket Volatility”, BIS Conference Papers, Vol. 8, Bank for International Settlements, March.Luı́s, J. B. (2001), “Essays on Extracting Information from Financial Asset Prices”, PhD thesis,University of York.Malz, A. M. (1996), “Using Option Prices to Estimate Realignment Probabilities in the EuropeanMonetary System: the Case of Sterling–Mark”, Journal of International Money and Finance,15(5), 717–748.Malz, A. M. (1997), “Estimating the Probability Distribution of the Future Exchange Rate fromOption Prices”, Journal of Derivatives, Winter, 18–36.McCauley, R. and W. Melick (1996), “Risk Reversal Risk”, Risk, 9(11), November, 54–57.Melick, W. R. and C. P. Thomas (1997), “Recovering an Asset’s Implied PDF from Option Prices:An Application to Crude Oil during the Gulf Crisis”, Journal of Financial and QuantitativeAnalysis, 32(1), March, 91–115.Melick, W. R. and C. P. Thomas (1998), “Conﬁdence Intervals and Constant-Maturity Series forProbability Measures Extracted from Options Prices”, paper presented to the Bank of CanadaConference “Information in Financial Asset Prices”, May.
380
Applied Quantitative Methods for Trading and Investment
Merton, R. C. (1976), “Option Pricing when Underlying Stock Returns are Discontinuous”, Journalof Financial Economics, Jan/Mar(3), 125–144.Nelson, C. R. and A. F. Siegel (1987), “Parsimonious Modelling of Yield Curves”, Journal ofBusiness, 60(4), 473–489.Ritchey, R. R. (1990), “Call Option Valuation for Discrete Normal Mixtures”, The Journal of Financial Research, XIII(4), Winter.Söderlind, P. and L. E. O. Svensson (1997), “New Techniques to Extract Market Expectations fromFinancial Instruments”, Journal of Monetary Economics, 40, 383–429.Svensson, L. E. O. (1991), “The Simplest Test of Target Zone Credibility”, International MonetaryFund Staff Papers, 38, 655–665.Svensson, L. E. O. (1994), “Estimating and Interpreting Forward Interest Rates: Sweden 1992–4”,CEPR Discussion Paper Series No. 1051.Xu, X. and S. J. Taylor (1994), “The Term Structure of Volatility Implied by Foreign ExchangeOptions”, Journal of Financial and Quantitative Analysis, 29(1), March.
13
Filling Analysis for Missing Data: AnApplication to Weather Risk Management∗CHRISTIAN L. DUNIS AND VASSILIOS KARALIS
ABSTRACTThis chapter examines and analyses the use of alternative methods when confronted withmissing data, a common problem when not enough historical data or clean historical dataexist: this will typically be the case when trying to develop a decision tool either for anew asset in a given asset class (say a recently issued stock in a given company sector)or for a new asset class as such (for instance weather derivatives).Weather derivatives are used to illustrate this analysis because most weather derivativespricing methodologies rely heavily on “clean” weather temperature data. The methodology adopted is to discard certain observed values and treat them as missing data. Wethen examine and analyse the imputation accuracy of different interpolation techniquesand ﬁlling methods for missing historical records of temperature data: the ExpectationMaximisation (EM) algorithm, the Data Augmentation (DA) algorithm, the Kalman ﬁlter (KF), Neural Networks Regression (NNR) models and, ﬁnally, Principal ComponentAnalysis (PCA).These methodologies are benchmarked against simpler techniques like the fallbackmethodology and a naı̈ve approach whereby missing temperature observations are imputedwith the same day value of the previous year. Their performance is evaluated against theactual values of the “missing data” using standard measures of forecasting accuracy widelyused in the economic literature.Overall, it is found that, for the periods and the data series concerned, the results ofPCA outperformed the other methodologies in all cases of missing observations analysed.
13.1 INTRODUCTIONThis entire book has been devoted to answering this apparently simple question: withthe help of different methods, what can we learn about the future from the ﬁnancialdata available from the past? In other words, we have taken data availability for granted.∗
The opinions expressed herein are those of the authors, and not necessarily those of Girobank.
Applied Quantitative Methods for Trading and Investment. 2003 John Wiley & Sons, Ltd ISBN: 0-470-84885-5
Edited by C.L. Dunis, J. Laws and P. Naı̈m
382
Applied Quantitative Methods for Trading and Investment
But the absence of data, or of clean data, is a rather common problem facing users ofquantitative methods in ﬁnancial markets.Admittedly, in recent years, the increasing availability of powerful computers hasmade it ever easier to collect, store and process higher frequency databanks, at a timewhen the use of advanced technologies for trading, risk and asset management hasgrown exponentially among ﬁnancial institutions. Nevertheless, in a multivariate context, taking into account the fact that most ﬁnancial markets do not move independently from each other, the nonsynchronicity of such high frequency data arrival clearlyposes major practical problems. Similarly, even at daily frequencies, a risk managerwill possibly face a dilemma over which data to retain for the marking-to-market andValue-at-Risk (VaR) computation of a portfolio containing assets from different markets when some of these are not trading either because of the time difference or apublic holiday.Such problems will be exacerbated when not enough historical data or clean historicaldata exist: this will typically be the case when trying to develop a decision tool either fora new asset in a given asset class (say a recently issued stock in a given company sector)or for a new asset class as such (for instance weather derivatives).The motivation for this chapter is to consider and examine the problem of missing data.Weather data are used to illustrate this analysis because most weather derivatives pricing methodologies rely heavily on “clean” weather temperature data. The methodologyadopted is to discard certain observed values and treat them as “missing data”. We thenexamine and analyse the imputation accuracy of different interpolation techniques and ﬁlling methods for missing historical records of temperature data. These are the ExpectationMaximisation (EM) algorithm, which is a common technique for determining maximumlikelihood estimates, the Data Augmentation (DA) algorithm, an iterative simulation verysimilar to the EM algorithm and which may be regarded as a stochastic edition of theEM, the Kalman ﬁlter (KF), a smoothing algorithm which can be used to estimate missingobservations once a model has been represented in state space form, Principal ComponentAnalysis (PCA), an approach adapted to ﬁll in appropriate values for missing observations from data that are available from correlated variables and, ﬁnally, Neural NetworksRegression (NNR) models which have been used here to identify and possibly beneﬁtfrom nonlinear correlations between temperature series.These methodologies are benchmarked against simpler and widely used techniques likethe fallback methodology and a naı̈ve approach whereby missing temperature observations are imputed with the same day value of the previous year. This benchmarkingis carried out to gauge the potential added value of the different methods retainedin the imputation process. The performance of these techniques as predictors of the“pseudo missing” values is evaluated using standard measures of forecasting accuracywidely used in the economic literature, such as mean absolute error, root mean squarederror, etc.Overall, we conclude that, for the periods and the data series concerned, the results ofPCA outperformed the other methodologies in all cases of missing observations analysed.The rest of the chapter is organised as follows. Section 13.2 gives a brief overview ofthe importance of clean weather data for weather derivatives pricing and discusses theweather data used in this research. Section 13.3 documents the different ﬁlling methodsretained while Section 13.4 presents our empirical results. Finally, Section 13.5 concludesthis chapter.
Filling Analysis for Missing Data
383
13.2 WEATHER DATA AND WEATHER DERIVATIVES13.2.1 The importance of weather data for weather derivatives pricingThe history of the weather derivatives market dates back to 1996, when electricityderegulation in the USA caused the power market to begin changing from series of localmonopolies to competitive regional wholesale markets. Energy companies realising theimpact of weather on their operations took control of their weather risk and created a newmarket around it. While a number of pioneering trades were made as early as 1996, themarket did not really take off until after September 1999, when the Chicago MercantileExchange (CME) embarked on listing and trading standard futures and options contractson US temperature indexes. Since 2000, the weather risk market has grown signiﬁcantlyand become progressively more diversiﬁed across industries, even if this market originallyevolved from the energy sector. According to the Second Annual Weather Risk Industrysurvey commissioned by the Weather Risk Management Association (WRMA), the totalmarket size for weather derivatives increased to an impressive $11.5 billion at the end of2001 (see PricewaterhouseCoopers (2002)).Most weather derivatives contracts traded were temperature related (over 82%). Theseare “heating degree days” (HDDs) and “cooling degree days” (CDDs) contracts. A degreeday is the deviation of the average daily temperature (ADT) from a predeﬁned temperaturedeﬁned as K in the following equations. HDDs and CDDs are the most common degreeday measurements. The temperature below K represents the temperature below whichheating devices are expected to turn on (HDDs), and above which air conditioners areexpected to turn on (CDDs). Since HDDs and CDDs are measuring heating and coolingneeds compared to the base temperature, they are calculated according to the following equations: Tmax + TminDaily HDDs = max(0, (K − ADT)) = max 0, K −2 Tmax + Tmin−KDaily CDDs = max(0, (ADT − K)) = max 0,2where Tmax and Tmin are the maximum and minimum, respectively, recorded temperatureduring the day.1 The calculation of daily degree days for longer periods (week, season,year) is based on straightforward summation:HDDs =
 n Tmax + Tminmax 0, K −2i
 n Tmax + Tminmax 0,−KCDDs =2i
In order to demonstrate how a degree day index may be structured to moderate risk,suppose that an energy company through the past 10 years’ supply and temperature1In the USA, the standard baseline temperature (K) is 65◦ F (18.3◦ C), but the 65◦ F base temperature canﬂuctuate from one region to another.
384
Applied Quantitative Methods for Trading and Investment
analysis has determined the expected cumulative number of HDDs during the heatingseason. The company is expecting to sell a given amount of energy units according tothis number and sets its budgets to this level. Its analysis also suggests that, for each HDDbelow the 10-year average, demand will decrease by a set number of energy units, creatinga marginal loss of $55 000 versus budget. In such a case the company may purchase aHDD index derivative to achieve protection at a price of $55 000 for each HDD lowerthan a given strike (see Corbally and Dang (2001)).Traditionally, ﬁnancial derivatives such as options on equities, bonds, forex or commodities are priced using no-arbitrage models such as the Black–Scholes pricing model.In the case of weather derivatives, the underlying asset is a physical quantity (temperature, rain, wind or snow) rather than a traded asset. Given that the underlying weatherindexes are not traded, a no-arbitrage model cannot be directly applied to price weatherderivatives.Another approach is used in the insurance industry, known as Historical Burn Analysis(HBA). The central assumption of the method is that the historical record of weathercontracts payouts gives a precise illustration of the distribution of potential payouts. Asnoted by Henderson (2001), if weather risk is calculated as the payouts standard deviation,then the price of the contract will be given by the equation:Pricebid/offer (t) = D(t, T ) × (µ ± ασ )where D(t, T ) is the discount factor from contract maturity T to the pricing time t, µ isthe historical average payout, σ is the historical standard deviation of payouts and α is apositive number denoting the protection seller’s risk tolerance.HBA therefore crucially depends on historical temperature data. Collecting historicaldata may be somewhat difﬁcult and costly but, even when the data is available, there areseveral types of errors like missing data or unreasonable readings. Consequently, the datamust be “cleaned” – that is, the errors and omissions must be ﬁxed – in order to be usedfor pricing and risk management purposes.13.2.2 The weather databankFor this research, we use daily temperature data (cleaned data) for Philadelphia International (WMO: 724080) weather station (the index station) and for its “fallback station”according to the WRMA, the Allentown-Bethlehem (WMO: 725170).2 This data wasobtained from QuantWeather (www.quantweather.com). Furthermore, to perform PCAand NNR, we use temperature data for all the neighbouring stations for which cleaneddata are available. In order to create a series of consequent daily temperature observationsfor Tmax (maximum daily temperature), Tmin (minimum daily temperature) and the corresponding Tavg ((Tmax + Tmin )/2) for all the available stations, the databank spans from1 July 1960 to 31 December 1993 (i.e. 33 years and 12 237 observations). Moreover, toexamine the impact of seasonality, this databank was divided into two separate subsets.The ﬁrst one includes only the “Autumn” (September, October and November) temperature observations and the second one only November temperature observations, totalling2Weather stations are identiﬁed in several ways. The most common is a six-digit WMO ID assigned by theWorld Meteorological Organisation. WMO IDs are the primary means of identifying weather stations outsidethe USA, while in the USA the ﬁve-digit Weather Bureau Army Navy identities (WBAN IDs) are widely used.
Filling Analysis for Missing Data
385
3094 and 1020 observations respectively. Standard tests like the serial correlation LM test,ADF, Phillips–Perron, ARCH-LM and Jarque–Bera tests (not reported here in order toconserve space) showed that all the series are autocorrelated, stationary, heteroskedasticand not normally distributed.In order to assess the imputation accuracy of the methodologies, several “holes” werecreated in the datasets. Arbitrarily, it is assumed that all the missing observations occurredin November 1993. We also supposed that the missing data were missing at random. Thefallback methodology employs different procedures to handle missing data according tothe number of consecutive missing days, with 12 days or more an important breakpoint.3Therefore, ﬁve different gaps were created including 1, 7, 10 (<12), 20 and 30 (>12)consecutive missing days. Since HDDs and CDDs are calculated as the deviation of theaverage daily temperature from the standard baseline temperature and, again, in orderto conserve space only the imputed values of the average temperature dataset (Tavg ) arepresented and compared later in this chapter.
13.3 ALTERNATIVE FILLING METHODS FOR MISSING DATAIn order to ensure that more complex methodologies do indeed add value in the imputationprocess, it is essential to benchmark them against simpler and more widely used methodslike the naı̈ve approach and the fallback method. We ﬁrst present these benchmarks beforeturning to more sophisticated methods.13.3.1 The naı̈ve approachThe naı̈ve hypothesis retained is to impute the missing temperature observation with thesame day value of the previous year. This simple model is formed by:Yt = Y ∗where Yt is the missing temperature data at period t and Y ∗ the corresponding imputedvalue, i.e. the temperature that prevailed on the same day the previous year.13.3.2 The fallback methodologyThe second part of the WRMA conﬁrmation4 establishes the weather industry’s standardfor determining and adjusting weather data. It presents an estimation of temperaturebased on a weather index station, which is usually an agreed region designated in theconﬁrmation by an airport location, WBAN or WMO ID. In the event that weather datais not available for the index station, the WRMA conﬁrmation sets forth the interpolationprocedures to be used to determine the missing data under the provision entitled “fallbackmethodology” (see Raspé (2001), p. 233). The fallback method provides an adjustmentbased on temperature at a selected alternative weather index station referred to as the“fallback station”.3
See Section 13.3.2 below.Standard forms of documents and other conﬁrming evidence exchanged between the parties in a weathercontract.4
386
Applied Quantitative Methods for Trading and Investment
13.3.2.1 Fallback terminologyIt is important at this point to deﬁne some basic terminology of the fallback methodology:• Adjustment indicates the mean of the arithmetic difference of the daily maximum and/orminimum temperature (if the unavailable temperature is a daily maximum and/or minimum respectively) at the fallback station subtracted from the corresponding temperatureat the index station during the period in which temperature data is unavailable.• Missing data day denotes any day during the calculation period for which data for theindex station is unavailable.• Missing data calculation day means the same day and same month as the relevantmissing data day.• Leap year exception stipulates that, in the event that the missing data day is 29 February,the missing data calculation day shall be considered as 1 March.13.3.2.2 Data missing for less than 12 consecutive daysIf temperature data is unavailable for less than 12 consecutive days during the calculationperiod, the “adjustment” will be computed for each “missing data day” using a period ofthe 15 days immediately prior to, and the 15 days immediately following, the relevant“missing data day”. If there are unavailable temperature data at the fallback station and/orthe index station within the 15-day periods prior to and following a “missing data day”,then an adjustment period of the ﬁrst 15 days on which the relevant temperature valuesare available (out of a maximum of 25 days) on either side of each “missing data day”shall be used to calculate the “adjustment”. To illustrate this process consider the caseof one missing observation, say Y0 . Let Z be the temperature on the fallback station andY the temperature being measured. Then the gap between Y and Z is measured overthe period t−1 to t−15 and t1 to t15 and the average gap is calculated over this periodand designated as “avg”. Thus the interpolated value is Y0 = Z0 + avg. Similar methodswould apply to the computation of missing data for longer periods.13.3.2.3 Data missing for 12 or more consecutive daysIf there are missing temperature observations for more than 12 consecutive days, the“adjustment” for each “missing data day” will be computed using temperature valuesfrom a period of the 15 days prior to (including the relevant “missing data calculationday”) and the 15 days after the applicable “missing data calculation day” for the threeprior years. If there is no data for the index station and/or the fallback station in theimmediate three previous years, the adjustment period will be extended until three yearsof temperature values can be obtained. In the event that there are unavailable temperaturesat the fallback station and/or the index station within the periods of the 15 days priorto a “missing data calculation day” or the 15 following days, then an adjustment periodshall be the ﬁrst 15 days on which the relevant temperature data are available (out of amaximum of 25 days) on either side of each “missing data calculation day”.13.3.3 The expectation maximisation algorithmThe EM algorithm is a common technique for determining maximum-likelihood estimatesfor parametric models when data are not entirely available. Developed by Dempster et al.
Filling Analysis for Missing Data
387
(1977), the EM algorithm generated an innovation in the analysis of incomplete data,making it possible to calculate efﬁcient parameter estimates in a wide variety of statisticalproblems, under the assumption that the data are normally distributed.13.3.3.1 Maximum-likelihood estimationThe principle of maximum likelihood provides a means of choosing an asymptoticallyefﬁcient estimator for a parameter or a set of parameters. Consider a random sample ofn observations from a normal distribution with mean µ and variance σ 2 . The probabilitydensity function for each observation is (see e.g. Greene (2000), p. 125):−
f (xi ) = (2π)
11−2 × (σ 2 ) 2 × exp
1 (xi − µ)2−2σ2
Since the observations are independent, their joint density is:f (x1 , x2 , . . . , xn |µ, σ 2 ) =
n
nnf (xi ) = (2π)− 2 × (σ 2 )− 2 × exp
i=1

n1  (xi − µ)2− ×2 i=1σ2

This equation gives the probability of observing this speciﬁc sample. We are now interested in the values of µ and σ 2 that make this sample most probable, in other words, themaximum-likelihood estimates (MLE) θMLE of µ and σ 2 . Since the log function is monotonically increasing and easier to work with, we usually calculate θMLE by maximisingthe natural logarithm of the likelihood function:n nn1  (xi − µ)22ln L(µ, σ |x1 , x2 , . . . , xn ) = − × ln(2π) − × ln(σ ) −222 i=1σ22
This translates into ﬁnding solutions to the following ﬁrst-order conditions:n∂ ln L1= 2×(xi − µ) = 0∂µσi=1
n∂ ln Ln1=− 2 +×(xi − µ)2 = 0∂σ 22σ2σ 4i=1
Finally, the MLE estimates are:n1 µ̂ML = ×xi = x nn i=1
and
12σ̂ML= ×n
n
(xi − x n )2
i=1
13.3.3.2 The E-step and M-step of the algorithmWhen some observations Yobs of the sample are missing, the MLE estimates θMLE are notobtainable, because the likelihood function is not deﬁned at the missing data points. Inorder to overcome this problem we apply the EM algorithm which exploits the interdependence between the missing data Ymis and parameters θMLE . The Ymis enclose informationrelevant to estimating θMLE , and θMLE in turn helps us to compute probable values ofYmis . This relation suggests the following scheme for estimating θMLE in the presence of
388
Applied Quantitative Methods for Trading and Investment
Yobs alone: “ﬁll in” the missing data Ymis based on an initial estimate of θMLE , re-estimateθMLE based on Yobs and the ﬁlled-in Ymis and iterate until the estimates converge (seeSchafer (1997)). In any missing data case, the distribution of the complete dataset Y canbe factored as:P (Y |θ ) = P (Yobs |θ ) × P (Ymis |Yobs , θ )(13.1)Considering each term in the above equation as a function of θ , we have:l(θ |Y ) = l(θ |Yobs ) + log P (Ymis |Yobs , θ ) + c
(13.2)
where l(θ |Y ) = log P (Y |θ ) indicates the complete data log-likelihood, l(θ |Yobs ) =log L(θ |Yobs ) the observed data log-likelihood, and c a random constant. Averagingequation (13.2) over the distribution P (Ymis |Yobs , θ (t) ), where θ (t) is an initial estimateof the unknown parameter, we get:Q(θ |θ (t) ) = l(θ |Yobs ) + H (θ |θ (t) ) + c
(13.3)
whereQ(θ |θ (t) ) =
(l(θ |Y ) × P (Ymis |Yobs , θ (t) )) dYmis
andH (θ |θ (t) ) =
(log P (Ymis |Yobs , θ ) × P (Ymis |Yobs , θ (t) )) dYmis
A basic conclusion of Dempster et al. (1977) is that if we consider θ (t+1) as the value ofθ that maximises Q(θ |θ (t) ), then θ (t+1) is an improved estimation compared to θ (t) :l(θ (t+1) |Yobs ) ≥ l(θ (t) |Yobs )In summary, it is convenient to think of the EM as an iterative algorithm that operates intwo steps as follows:• E-step: The expectation step, in which the function Q(θ |θ (t) ) is calculated by averagingthe complete data log-likelihood l(θ |Y ) over P (Ymis |Yobs , θ (t) ).• M-step: The maximisation step, in which θ (t+1) is calculated by maximising Q(θ |θ (t) ).513.3.4 The data augmentation algorithmData augmentation is an iterative simulation algorithm, a special kind of Markov chainMonte Carlo (MCMC). As underlined by Schafer (1997), DA is very similar to the EMalgorithm, and may be regarded as a stochastic edition of EM. In many cases with missingvalues, the observed data distribution P (θ |Yobs ) is difﬁcult to deﬁne. However, if Yobs is“augmented” by a preliminary value of Ymis , the complete data distribution P (θ |Yobs , Ymis )can be handled.5
The EM algorithm was implemented with the SPSS software.
Filling Analysis for Missing Data
389
For an initial guess θ (t) of the parameter, we select a value of the missing data fromthe conditional distribution of Ymis :(t+1)Ymis∼ P (Ymis |Yobs , θ (t) )
(13.4)
(t+1), we select a new value of θ from its complete data distribution:Conditioning on Ymis(t+1))θ (t+1) ∼ P (θ |Yobs , Ymis
(13.5)
Alternating (13.4) and (13.5) from an initial value θ (0) , we obtain a stochastic sequence(t)): t = 1, 2, . . .} that converges to a stationary distribution P (θ ,Ymis |Yobs ), the{(θ (t) ,Ymisjoint distribution of the missing data and parameters given the observed data, and the(t)subsequences {θ (t) : t = 1, 2, . . .} and {Ymis: t = 1, 2, . . .} with P (θ |Yobs ) and P (Ymis |Yobs )their stationary distributions respectively. For a large value of t we can consider θ (t) as(t)as an approximatean approximate draw from P (θ |Yobs ); alternatively, we can regard Ymisdraw from P (Ymis |Yobs ).In summary, DA is an iterative algorithm that operates in two steps as follows:• I-step: The imputation step, in which the missing data are imputed by drawing themfrom their conditional distribution given the observed data and assumed initial valuesfor the parameters θ (t) .• P-step: The posterior step, in which the values for the parameters are simulated bydrawing them from their complete data distribution given the most recently imputed(t+1)for the missing data.values YmisThe convergence performance of the DA algorithm depends on the amount of missinginformation (how much information about the parameters is contained in the missingpart of the data relative to the observed part). High rates of missing information causesuccessive iterations to be highly correlated, and a large number of cycles will be neededfor the algorithm to converge. Low rates of missing information produce low correlationand rapid convergence.6The EM and DA algorithms provide optimal solutions under the assumption that thedata are normally distributed. However, weather temperature data deviate from normality.Nevertheless, in many cases the normal model is useful even when the actual data are nonnormal (see Schafer (1997), p. 147). The EM algorithm is used to calculate the missingvalues on the level of the series. Additionally, it is almost always better to run theEM algorithm before using the DA to impute missing data because running the EMﬁrst will provide good starting values for the DA and will help to predict its likelyconvergence behaviour.13.3.5 The Kalman ﬁlter modelsThe seminal works of Harvey (1989) and Hamilton (1994) have underlined the advantages of using state space modelling for representing dynamic systems where unobserved6The DA algorithm was implemented using Schafer’s stand alone NORM software, which can be downloadedat http://www.stat.psu.edu/∼jls/misoftwa.html.
390
Applied Quantitative Methods for Trading and Investment
variables (the so-called “state” variables) can be integrated within an “observable” model.The advantage of handling problems of missing observations with the state space formis that the missing values can be estimated by a smoothing algorithm like the Kalmanﬁlter. The ﬁltering is used to estimate the expected value of the state vector accordingto the