1.10
19
Restricted ARMA(10,10) model Excel spreadsheet (in-sample)
Yt∗ = 0.2492 − 0.3613X1,t − 0.2872X2,t + 0.2862X3,t + 0.2525X4,t− 0.3692X5,t − 0.3937X6,t + εtwhere X1,t , . . . , X6,t are the JP yc(−2), UK yc(−9), JAPDOWA(−1), ITMIB30(−19),JAPAYE$(−10), and OILBREN(−1) binary explanatory variables, respectively.9All of the coefﬁcients in the model are signiﬁcant at the 98% conﬁdence interval. Theoverall signiﬁcance of the model is tested using the LR test. The null hypothesis that allcoefﬁcients except the constant are not signiﬁcantly different from zero is rejected at the99% conﬁdence interval. The results of this are presented in Table 1.10.To justify the use of Japanese variables, which seems difﬁcult from an economic perspective, the joint overall signiﬁcance of this subset of variables is tested using the LR testfor redundant variables. The null hypothesis that these coefﬁcients, except the constant,are not jointly signiﬁcantly different from zero is rejected at the 99% conﬁdence interval.The results of this are presented in Table 1.11. In addition, a model that did not includethe Japanese variables, but was otherwise identical to logit1, was produced and the trading performance evaluated, which we shall name nojap (equation nojap of the logit.wf1EViews workﬁle). The Sharpe ratio, average gain/loss ratio and correct directional changeof the nojap model were 1.34, 1.01 and 54.38%, respectively. The corresponding valuesfor the logit1 model were 2.26, 1.01 and 58.13%. The evaluation can be reviewed inSheet 2 of the is logit1.xls and is nojap.xls Excel spreadsheets, and is also presented inFigures 1.12 and 1.13, respectively.9
Datastream mnemonics as mentioned in Table 1.1, yield curves and lags in brackets are used to save space.
20
Applied Quantitative Methods for Trading and Investment
Figure 1.11 Alternative restricted ARMA(10,10) model Excel spreadsheet (in-sample)
The logit1 model was retained for out-of-sample estimation. As, in practice, the estimation of the model is based upon the cumulative distribution of the logistic function for theerror term, the forecasts produced range between zero and one, requiring transformationinto a binary series. Again, the procedure is quite simple: a binary variable equal to oneis produced if the forecast is greater than 0.5 and zero otherwise.The performance of the strategy is evaluated in terms of forecast accuracy via thecorrect directional change measure and in terms of trading performance. Several otheradequate models were produced and their performance evaluated. None performed betterin-sample, therefore the logit1 model was retained.
1.5 NEURAL NETWORK MODELS: THEORYAND METHODOLOGYNeural networks are “data-driven self-adaptive methods in that there are few a prioriassumptions about the models under study” (Zhang et al., 1998: 35). As a result, they arewell suited to problems where economic theory is of little use. In addition, neural networksare universal approximators capable of approximating any continuous function (Horniket al., 1989).Many researchers are confronted with problems where important nonlinearitiesexist between the independent variables and the dependent variable. Often, insuch circumstances, traditional forecasting methods lack explanatory power. Recently,nonlinear models have attempted to cover this shortfall. In particular, NNR modelshave been applied with increasing success to ﬁnancial markets, which often containnonlinearities (Dunis and Jalilov, 2002).
Applications of Advanced Regression AnalysisTable 1.10
21
Logit1 EUR/USD returns estimation
Dependent Variable: BDR− USEURSPMethod: ML – Binary LogitSample(adjusted): 20 1459Included observations: 1440 after adjusting endpointsConvergence achieved after 3 iterationsCovariance matrix computed using second derivativesVariable
Coefﬁcient
Std. error
z-Statistic
Prob.
CBDR− JP− YC(−2)BDR− UK− YC(−9)BDR− JAPDOWA(−1)BDR− ITMIB31(−19)BDR− JAPAYE$(−10)BDR− OILBREN(−1)
0.249231−0.361289−0.2872200.2862140.252454−0.369227−0.393689
0.1405790.1089110.1083970.1086870.1080560.1083410.108476
1.772894−3.317273−2.6496962.6333692.336325−3.408025−3.629261
0.07620.00090.00810.00850.01950.00070.0003
Mean dependent var.S.E. of regressionSum squared resid.Log likelihoodRestr. log likelihoodLR statistic (6 df)Prob(LR statistic)
0.4576390.490514344.7857−967.3795−992.957751.156352.76E-09
Obs. with dep = 0Obs. with dep = 1
781659
S.D. dependent var.Akaike info. criterionSchwarz criterionHannan–Quinn criterionAvg. log likelihoodMcFadden R-squared
Total obs.
0.4983751.3533051.3789351.362872−0.6717910.025760
1440
Theoretically, the advantage of NNR models over traditional forecasting methods isbecause, as is often the case, the model best adapted to a particular problem cannot beidentiﬁed. It is then better to resort to a method that is a generalisation of many models,than to rely on an a priori model (Dunis and Huang, 2002).However, NNR models have been criticised and their widespread success has been hindered because of their “black-box” nature, excessive training times, danger of overﬁtting,and the large number of “parameters” required for training. As a result, deciding on theappropriate network involves much trial and error.For a full discussion on neural networks, please refer to Haykin (1999), Kaastra andBoyd (1996), Kingdon (1997), or Zhang et al. (1998). Notwithstanding, we provide belowa brief description of NNR models and procedures.1.5.1 Neural network modelsThe will to understand the functioning of the brain is the basis for the study of neuralnetworks. Mathematical modelling started in the 1940s with the work of McCulloch andPitts, whose research was based on the study of networks composed of a number of simpleinterconnected processing elements called neurons or nodes. If the description is correct,
22
Applied Quantitative Methods for Trading and InvestmentTable 1.11 Logit1 estimation redundant variables LR test
Redundant Variables: BDR− JP− YC(−2), BDR− JAPDOWA(−1), BDR− JAPAYE$(−10)F -statisticLog likelihood ratio
9.72202328.52168
ProbabilityProbability
0.0000020.000003
Test Equation:Dependent Variable: BDR− USEURSPMethod: ML – Binary LogitSample: 20 1459Included observations: 1440Convergence achieved after 3 iterationsCovariance matrix computed using second derivativesVariable
Coefﬁcient
Std. error
z-Statistic
Prob.
CBDR− UK− YC(−9)BDR− ITMIB31(−19)BDR− OILBREN(−1)
−0.013577−0.2472540.254096−0.345654
0.1052800.1069790.1067250.106781
−0.128959−2.3112452.380861−3.237047
0.89740.02080.01730.0012
Mean dependent var.S.E. of regressionSum squared resid.Log likelihoodRestr. log likelihoodLR statistic (3 df)Prob(LR statistic)
0.4576390.494963351.8032−981.6403−992.957722.634674.81E-05
S.D. dependent var.Akaike info. criterionSchwarz criterionHannan–Quinn criterionAvg. log likelihoodMcFadden R-squared
0.4983751.3689451.3835901.374412−0.6816950.011398
Obs. with dep = 0Obs. with dep = 1
781659
Total obs.
1440
they can be turned into models mimicking some of the brain’s functions, possibly withthe ability to learn from examples and then to generalise on unseen examples.A neural network is typically organised into several layers of elementary processingunits or nodes. The ﬁrst layer is the input layer, the number of nodes correspondingto the number of variables, and the last layer is the output layer, the number of nodescorresponding to the forecasting horizon for a forecasting problem.10 The input and outputlayer can be separated by one or more hidden layers, with each layer containing one ormore hidden nodes.11 The nodes in adjacent layers are fully connected. Each neuronreceives information from the preceding layer and transmits to the following layer only.12The neuron performs a weighted summation of its inputs; if the sum passes a thresholdthe neuron transmits, otherwise it remains inactive. In addition, a bias neuron may beconnected to each neuron in the hidden and output layers. The bias has a value of positive10Linear regression models may be viewed analogously to neural networks with no hidden layers (Kaastra andBoyd, 1996).11Networks with hidden layers are multilayer networks; a multilayer perceptron network is used for this chapter.12If the ﬂow of information through the network is from the input to the output, it is known as “feedforward”.
Applications of Advanced Regression Analysis
Figure 1.12 Logit1 estimation Excel spreadsheet (in-sample)
Figure 1.13 Nojap estimation Excel spreadsheet (in-sample)
23
24
Applied Quantitative Methods for Trading and Investmentxt[1]
xt[2]
Σ
∫
ht[1]Σ
xt[3]
Σ
∫
∫
ht[2]
xt[4]
~ytyt
xt[5]
where xt [i ] (i = 1, 2, ..., 5) are the NNR model inputs at time tht [ j ] ( j = 1, 2) are the hidden nodes outputs∼yt and yt are the actual value and NNR model output, respectively
Figure 1.14 A single output fully connected NNR model
one and is analogous to the intercept in traditional regression models. An example ofa fully connected NNR model with one hidden layer and two nodes is presented inFigure 1.14.The vector A = (x [1] , x [2] , . . . , x [n] ) represents the input to the NNR model where xt[i] isthe level of activity of the ith input. Associated with the input vector is a series of weightvectors Wj = (w1j , w2j , . . . , wnj ) so that wij represents the strength of the connectionbetween the input xt[i] and the processing unit bj . There may also be the input bias ϕjmodulated by the weight w0j associated with the inputs. The total input of the node bjis the dot product between vectors A and Wj , less the weighted bias. It is then passedthrough a nonlinear activation function to produce the output value of processing unit bj :bj = f
 n
x wij − w0j ϕj[i]
= f (Xj )
(1.8)
i=1
Typically, the activation function takes the form of the logistic function, which introducesa degree of nonlinearity to the model and prevents outputs from reaching very largevalues that can “paralyse” NNR models and inhibit training (Kaastra and Boyd, 1996;Zhang et al., 1998). Here we use the logistic function:f (Xj ) =
11 + e−Xj
(1.9)
The modelling process begins by assigning random values to the weights. The outputvalue of the processing unit is passed on to the output layer. If the output is optimal,the process is halted, if not, the weights are adjusted and the process continues until anoptimal solution is found. The output error, namely the difference between the actualvalue and the NNR model output, is the optimisation criterion. Commonly, the criterion
Applications of Advanced Regression Analysis
25
is the root-mean-squared error (RMSE). The RMSE is systematically minimised throughthe adjustment of the weights. Basically, training is the process of determining the optimalsolutions network weights, as they represent the knowledge learned by the network. Sinceinadequacies in the output are fed back through the network to adjust the network weights,the NNR model is trained by backpropagation13 (Shapiro, 2000).A common practice is to divide the time series into three sets called the training, test andvalidation (out-of-sample) sets, and to partition them as roughly 23 , 16 and 16 , respectively.The testing set is used to evaluate the generalisation ability of the network. The techniqueconsists of tracking the error on the training and test sets. Typically, the error on thetraining set continually decreases, however the test set error starts by decreasing andthen begins to increase. From this point the network has stopped learning the similaritiesbetween the training and test sets, and has started to learn meaningless differences, namelythe noise within the training data. For good generalisation ability, training should stopwhen the test set error reaches its lowest point. The stopping rule reduces the likelihoodof overﬁtting, i.e. that the network will become overtrained (Dunis and Huang, 2002;Mehta, 1995).An evaluation of the performance of the trained network is made on new examples notused in network selection, namely the validation set. Crucially, the validation set shouldnever be used to discriminate between networks, as any set that is used to choose thebest network is, by deﬁnition, a test set. In addition, good generalisation ability requiresthat the training and test sets are representative of the population, inappropriate selectionwill affect the network generalisation ability and forecast performance (Kaastra and Boyd,1996; Zhang et al., 1998).1.5.2 Issues in neural network modellingDespite the satisfactory features of NNR models, the process of building them should notbe taken lightly. There are many issues that can affect the network’s performance andshould be considered carefully.The issue of ﬁnding the most parsimonious model is always a problem for statisticalmethods and particularly important for NNR models because of the problem of overﬁtting.Parsimonious models not only have the recognition ability but also the more importantgeneralisation ability. Overﬁtting and generalisation are always going to be a problemfor real-world situations, and this is particularly true for ﬁnancial applications where timeseries may well be quasi-random, or at least contain noise.One of the most commonly used heuristics to ensure good generalisation is the application of some form of Occam’s Razor. The principle states, “unnecessary complex modelsshould not be preferred to simpler ones. However . . . more complex models always ﬁtthe data better” (Kingdon, 1997: 49). The two objectives are, of course, contradictory.The solution is to ﬁnd a model with the smallest possible complexity, and yet which canstill describe the data set (Haykin, 1999; Kingdon, 1997).A reasonable strategy in designing NNR models is to start with one layer containing afew hidden nodes, and increase the complexity while monitoring the generalisation ability.The issue of determining the optimal number of layers and hidden nodes is a crucial factor13Backpropagation networks are the most common multilayer network and are the most used type in ﬁnancialtime series forecasting (Kaastra and Boyd, 1996). We use them exclusively here.
26
Applied Quantitative Methods for Trading and Investment
for good network design, as the hidden nodes provide the ability to generalise. However,in most situations there is no way to determine the best number of hidden nodes withouttraining several networks. Several rules of thumb have been proposed to aid the process,however none work well for all applications. Notwithstanding, simplicity must be theaim (Mehta, 1995).Since NNR models are pattern matchers, the representation of data is critical for asuccessful network design. The raw data for the input and output variables are rarely fedinto the network, they are generally scaled between the upper and lower bounds of theactivation function. For the logistic function the range is [0,1], avoiding the function’ssaturation zones. Practically, as here, a normalisation [0.2,0.8] is often used with thelogistic function, as its limits are only reached for inﬁnite input values (Zhang et al.,1998).Crucial for backpropagation learning is the learning rate of the network as it determinesthe size of the weight changes. Smaller learning rates slow the learning process, whilelarger rates cause the error function to change wildly without continuously improving.To improve the process a momentum parameter is used which allows for larger learningrates. The parameter determines how past weight changes affect current weight changes,by making the next weight change in approximately the same direction as the previousone14 (Kaastra and Boyd, 1996; Zhang et al., 1998).1.5.3 Neural network modelling procedureConforming to standard heuristics, the training, test and validation sets were partitionedas approximately 23 , 16 and 16 , respectively. The training set runs from 17 October 1994to 8 April 1999 (1169 observations), the test set runs from 9 April 1999 to 18 May2000 (290 observations), and the validation set runs from 19 May 2000 to 3 July 2001(290 observations), reserved for out-of-sample forecasting and evaluation, identical to theout-of-sample period for the benchmark models.To start, traditional linear cross-correlation analysis helped establish the existence ofa relationship between EUR/USD returns and potential explanatory variables. AlthoughNNR models attempt to map nonlinearities, linear cross-correlation analysis can givesome indication of which variables to include in a model, or at least a starting point tothe analysis (Diekmann and Gutjahr, 1998; Dunis and Huang, 2002).The analysis was performed for all potential explanatory variables. Lagged termsthat were most signiﬁcant as determined via the cross-correlation analysis are presentedin Table 1.12.The lagged terms SPCOMP(−1) and US yc(−1) could not be used because of time-zonedifferences between London and the USA, as discussed at the beginning of Section 1.3.As an initial substitute SPCOMP(−2) and US yc(−2) were used. In addition, variouslagged terms of the EUR/USD returns were included as explanatory variables.Variable selection was achieved via a forward stepwise NNR procedure, namely potential explanatory variables were progressively added to the network. If adding a newvariable improved the level of explained variance (EV) over the previous “best” network,the pool of explanatory variables was updated.15 Since the aim of the model-building14The problem of convergence did not occur within this research; as a result, a learning rate of 0.1 andmomentum of zero were used exclusively.15EV is an approximation of the coefﬁcient of determination, R 2 , in traditional regression techniques.
Applications of Advanced Regression Analysis
27
Table 1.12 Most signiﬁcant lagof each potential explanatoryvariable (in returns)Variable
Best lag
DAXINDXDJES50IFRCAC40FTSE100GOLDBLNITMIBJAPAYE$OILBRENJAPDOWASPCOMPUSDOLLRBD ycEC ycFR ycIT ycJP ycUK ycUS ycNYFECRB
10101051991011511219292619120
procedure is to build a model with good generalisation ability, a model that has a higherEV level has a better ability. In addition, a good measure of this ability is to comparethe EV level of the test and validation sets: if the test set and validation set levels aresimilar, the model has been built to generalise well.The decision to use explained variance is because the EUR/USD returns series is astationary series and stationarity remains important if NNR models are assessed on thelevel of explained variance (Dunis and Huang, 2002). The EV levels for the training,test and validation sets of the selected NNR model, which we shall name nnr1 (nnr1.prvPrevia ﬁle), are presented in Table 1.13.An EV level equal to, or greater than, 80% was used as the NNR learning terminationcriterion. In addition, if the NNR model did not reach this level within 1500 learningsweeps, again the learning terminates. The criteria selected are reasonable for daily dataand were used exclusively here.If after several attempts there was failure to improve on the previous “best” model,variables in the model were alternated in an attempt to ﬁnd a better combination. ThisTable 1.13 nnr1 model EV for the training,test and validation setsTraining set
Test set
Validation set
3.4%
2.3%
2.2%
28
Applied Quantitative Methods for Trading and Investment
procedure recognises the likelihood that some variables may only be relevant predictorswhen in combination with certain other variables.Once a tentative model is selected, post-training weights analysis helps establish theimportance of the explanatory variables, as there are no standard statistical tests for NNRmodels. The idea is to ﬁnd a measure of the contribution a given weight has to theoverall output of the network, in essence allowing detection of insigniﬁcant variables.Such analysis includes an examination of a Hinton graph, which represents graphicallythe weight matrix within the network. The principle is to include in the network variablesthat are strongly signiﬁcant. In addition, a small bias weight is preferred (Diekmann andGutjahr, 1998; Kingdon, 1997; Previa, 2001). The input to a hidden layer Hinton graphof the nnr1 model produced by Previa is presented in Figure 1.15. The graph suggeststhat the explanatory variables of the selected model are strongly signiﬁcant, both positive(green) and negative (black), and that there is a small bias weight. In addition, the inputto hidden layer weight matrix of the nnr1 model produced by Previa is presented inTable 1.14.The nnr1 model contained the returns of the explanatory variables presented inTable 1.15, having one hidden layer containing ﬁve hidden nodes.Again, to justify the use of the Japanese variables a further model that did not includethese variables, but was otherwise identical to nnr1, was produced and the performanceevaluated, which we shall name nojap (nojap.prv Previa ﬁle). The EV levels of the training
Figure 1.15
Hinton graph of the nnr1 EUR/USD returns model
Applications of Advanced Regression Analysis
29
Table 1.14 Input to hidden layer weight matrix of the nnr1 EUR/USD returns modelGOLD JAPAYBLNE$(−19) (−10)C[1,0]C[1,1]C[1,2]C[1,3]C[1,4]
JAPDOWA(−15)
OILBREN(−1)
USDOLLR(−12)
FR yc(−2)
IT yc(−6)
JP yc JAPAY JAP(−9)E$DOWA(−1)(−1)
0.2316 −0.2120 −0.4336 −0.4579 −0.2621 −0.39110.4016 −0.1752 −0.3589 −0.5474 −0.3663 −0.46230.2490 −0.3037 −0.4462 −0.5139 −0.2506 −0.34910.3382 −0.3588 −0.4089 −0.5446 −0.2730 −0.45310.3338 −0.3283 −0.4086 −0.6108 −0.2362 −0.4828
0.24080.24380.29000.25550.3088
0.42950.27860.36340.46610.4192
0.40670.27570.27370.41530.4254
Bias
0.4403 −0.08240.4831 −0.02250.4132 −0.00880.5245 0.03730.4779 −0.0447
Table 1.15 nnr1 model explanatory variables (in returns)Variable
Lag
GOLDBLNJAPAYE$JAPDOWAOILBRENUSDOLLRFR ycIT ycJP ycJAPAYE$JAPDOWA
19101511226911
and test sets of the nojap model were 1.4 and 0.6 respectively, which are much lowerthan the nnr1 model.The nnr1 model was retained for out-of-sample estimation. The performance of thestrategy is evaluated in terms of traditional forecasting accuracy and in terms of tradingperformance.Several other adequate models were produced and their performance evaluated, including RNN models.16 In essence, the only difference from NNR models is the addition of aloop back from a hidden or the output layer to the input layer. The loop back is then usedas an input in the next period. There is no theoretical or empirical answer to whether thehidden layer or the output should be looped back. However, the looping back of eitherallows RNN models to keep the memory of the past,17 a useful property in forecastingapplications. This feature comes at a cost, as RNN models require more connections,raising the issue of complexity. Since simplicity is the aim, a less complex model thatcan still describe the data set is preferred.The statistical forecasting accuracy results of the nnr1 model and the RNN model,which we shall name rnn1 (rnn1.prv Previa ﬁle), were only marginally different, namelythe mean absolute percentage error (MAPE) differs by 0.09%. However, in terms of16
For a discussion on recurrent neural network models refer to Dunis and Huang (2002).The looping back of the output layer is an error feedback mechanism, implying the use of a nonlinearerror-correction model (Dunis and Huang, 2002).17
30
Applied Quantitative Methods for Trading and Investment
Figure 1.16 nnr1 model Excel spreadsheet (in-sample)
Figure 1.17 rnn1 model Excel spreadsheet (in-sample)
Applications of Advanced Regression Analysis
31
trading performance there is little to separate the nnr1 and rnn1 models. The evaluationcan be reviewed in Sheet 2 of the is nnr1.xls and is rnn1.xls Excel spreadsheets, and isalso presented in Figures 1.16 and 1.17, respectively.The decision to retain the nnr1 model over the rnn1 model is because the rnn1 model ismore complex and yet does not possess any decisive added value over the simpler model.
1.6 FORECASTING ACCURACY AND TRADING SIMULATIONTo compare the performance of the strategies, it is necessary to evaluate them on previously unseen data. This situation is likely to be the closest to a true forecasting ortrading situation. To achieve this, all models retained an identical out-of-sample periodallowing a direct comparison of their forecasting accuracy and trading performance.1.6.1 Out-of-sample forecasting accuracy measuresSeveral criteria are used to make comparisons between the forecasting ability of thebenchmark and NNR models, including mean absolute error (MAE), RMSE,18 MAPE,and Theil’s inequality coefﬁcient (Theil-U).19 For a full discussion on these measures, referto Hanke and Reitsch (1998) and Pindyck and Rubinfeld (1998). We also include correctdirectional change (CDC), which measures the capacity of a model to correctly predict thesubsequent actual change of a forecast variable, an important issue in a trading strategythat relies on the direction of a forecast rather than its level. The statistical performancemeasures used to analyse the forecasting techniques are presented in Table 1.16.1.6.2 Out-of-sample trading performance measuresStatistical performance measures are often inappropriate for ﬁnancial applications. Typically, modelling techniques are optimised using a mathematical criterion, but ultimatelythe results are analysed on a ﬁnancial criterion upon which it is not optimised. In otherwords, the forecast error may have been minimised during model estimation, but theevaluation of the true merit should be based on the performance of a trading strategy.Without actual trading, the best means of evaluating performance is via a simulated trading strategy. The procedure to create the buy and sell signals is quite simple: a EUR/USDbuy signal is produced if the forecast is positive, and a sell otherwise.20For many traders and analysts market direction is more important than the value ofthe forecast itself, as in ﬁnancial markets money can be made simply by knowing thedirection the series will move. In essence, “low forecast errors and trading proﬁts are notsynonymous since a single large trade forecasted incorrectly . . . could have accounted formost of the trading system’s proﬁts” (Kaastra and Boyd, 1996: 229).The trading performance measures used to analyse the forecasting techniques are presented in Tables 1.17 and 1.18. Most measures are self-explanatory and are commonlyused in the fund management industry. Some of the more important measures includethe Sharpe ratio, maximum drawdown and average gain/loss ratio. The Sharpe ratio is a18The MAE and RMSE statistics are scale-dependent measures but allow a comparison between the actual andforecast values, the lower the values the better the forecasting accuracy.19When it is more important to evaluate the forecast errors independently of the scale of the variables, theMAPE and