. However, our approach also has a technicalcomponent: it is somewhat irrational in a financial context since it depends heavily on the concepts of statistical inference. The fact that the input data is publiclyavailable lets us challenge the EMH in its semi-strong version. The procedure ofobtaining an exploratory dataset was divided into two phases according to [327]:C. Ullrich, Forecasting and Hedging in the Foreign Exchange Markets, Lecture Notesin Economics and Mathematical Systems 623, DOI: 10.1007/978-3-642-00495-7 10,c Springer-Verlag Berlin Heidelberg 2009
83
84
10 Description of Empirical Study and Results
1. Specifying and collecting a large amount of data2. Reducing the dimensionality of the dataset by selecting a subset of that data forefficient training (feature extraction)Since there is a trade-off between accuracy as represented by the entire dataset andthe computational overheads of retaining all parameters without application of feature extraction/selection techniques, the data selection procedure is also referred toas the curse of dimensionality which was first noted by [32]. The merit of featureextraction is to avoid multicollinearity, a problem that is common to all sorts ofregression models. If multicollinearity exists, explanatory variables have a high degree of correlation between themselves meaning that only a few important sourcesof information in the data are common to many variables. In this case, it may not bepossible to determine their individual effects.
10.1.1 Phase One: Input Data SelectionThe obvious place to start selecting data, along with the EUR/GBP, EUR/JPY, andEUR/USD is with the other leading traded exchange rates. In addition, we selectedrelated financial market data, including stock market price indices, 3-month interestrates, 10-year government bond yields and spreads, the price of Brent Crude oil, andthe prices of silver, gold, and platinum. Due to the bullish commodity markets, wealso decided to include daily prices of assorted metals being traded on the LondonMetal Exchange, as well as agricultural commodities. Fundamental variables hardlyplay a role in daily FX movements and were disregarded. The reason is that suchmacroeconomic data are collected on a monthly, quarterly, or annual basis only andthat it is thus believed to be more suitable for longer-run exchange rate predictions.All data were obtained from Bloomberg.All the series span a 7-year time period from 1 January 1997 to 31 December2004, totaling 2,349 trading days. The data is divided into two periods: the first period runs from 1 January 1997 to 31 August 2003 (1,738 observations), is used formodel estimation and is classified in-sample. The second period, from 1 September 2003 to 31 December 2004 (350 observations), is reserved for out-of-sampleforecasting and evaluation. Missing observations on bank holidays were filled bylinear interpolation, which is a quick and easy approach to smoothen out theseirregularities.
10.1.2 Phase Two: Dimensionality ReductionHaving collected an extensive list of candidate variables, the explanatory viabilityof each variable was evaluated. The aim was to remove those input variables thatdo not contribute significantly to model performance. For this purpose, we took atwo-step procedure.
10.1 Explanatory Dataset
85
First, pair-wise Granger Causality tests [170] with lagged values until k = 20were performed on stationary I(1) candidate variables. The Granger approach tothe question of whether an independent variable x causes a dependent variable y isto see how much of the current y can be explained by past values of y and then tosee whether adding lagged values of x can improve the explanation. The variable yis said to be Granger-caused by x if x helps in the prediction of y, or equivalently ifthe coefficients on the lagged x’s are statistically significant. The pair-wise Grangercausality test is carried out by running bivariate regressions of the formyt = α0 + α1 yt−1 + . . . + αl yt−l + β1 xt−1 + . . . + βl xt−l + εtxt = α0 + α1 xt−1 + . . . + αl xt−l + β1 yt−1 + . . . + βl yt−l + ut
(10.1)
for all possible pairs of (x, y) series in the group. In general it is better to use morerather than fewer lags in the test regressions, since the theory is couched in terms ofthe relevance of all past information. It is also important that the lag length pickedcorresponds to reasonable beliefs about the longest time over which one of the variables could help predict the other. The reported F-statistics are the Wald statisticsfor the joint hypothesis:β1 = β2 = . . . = βl = 0(10.2)for each equation. The null hypothesis is that x does not Granger-cause y in thefirst regression and that y does not Granger-cause x in the second regression. Themajor advantage of the Granger causality principle is that it is able to distinguishcausation from correlation. Hence the known problem of spurious correlations canbe avoided [173]. Still, it is important to note that the statement “x Granger causesy” does not imply that y is the effect or the result of x. Granger causality measuresprecedence and information content but does not by itself indicate causality in themore common use of the term. Tables 10.1–10.3 summarize the variables that werefound to be relevant for explaining the single exchange rates as can be seen by thevalues of the F-statistic which leads at least to a rejection of the null hypothesis atthe 10% significance level. Note that only the first regression from (10.1) has to beconsidered in our context.We find that EUR/GBP is Granger-caused by 11 variables, namely• EUR/USD, JPY/USD, and EUR/CHF exchange rates• IBEX, MIB30, CAC, and DJST stock market indices• The prices of platinum (PLAT) and nickel (LMNIDY)• Ten-year Australian (GACGB10) and Japanese (GJGB10) government bondyieldsFurther, we identify ten variables that significantly Granger-cause EUR/JPY,namely• EUR/CHF exchange rate• IBEX stock market index• The price of silver (SILV)• Australian 3-month interest rate (AU0003M)
86
10 Description of Empirical Study and Results
Table 10.1 Granger causality test results for EUR/GBP (k = 20)Null hypothesis
F-Statistic
EUR/USD does not Granger cause EUR/GBPGACGB10 does not Granger cause EUR/GBPGJGB10 does not Granger cause EUR/GBPIBEX does not Granger cause EUR/GBPJPY/USD does not Granger cause EUR/GBPLMNIDY does not Granger cause EUR/GBPMIB30 does not Granger cause EUR/GBPPLAT does not Granger cause EUR/GBPCAC does not Granger cause EUR/GBPCHF/USD does not Granger cause EUR/GBPDJST does not Granger cause EUR/GBP
1.4366∗1.4297∗1.4250∗1.7812∗∗1.6432∗∗1.4927∗1.9559∗∗∗1.4732∗1.7350∗∗1.6131∗∗1.8096∗∗
Table 10.2 Granger causality test results for EUR/JPY (k = 20)Null hypothesis
F-Statistic
GACGB10 does not Granger cause EUR/JPYGDBR10 does not Granger cause EUR/JPYGJGB10 does not Granger cause EUR/JPYGSWISS10 does not Granger cause EUR/JPYGT10 does not Granger cause EUR/JPYIBEX does not Granger cause EUR/JPYSILV does not Granger cause EUR/JPYAU0003M does not Granger cause EUR/JPYBPSW10-GUKG10 does not Granger cause EUR/JPYEUR/CHF does not Granger cause EUR/JPY
1.6036∗∗1.5556∗1.7328∗∗1.4303∗1.4502∗1.5328∗1.5305∗2.1236∗∗∗1.5779∗∗2.1408∗∗∗
Table 10.3 Granger causality test results for EUR/USD (k = 20)Null hypothesis
F-Statistic
LMSNDY does not Granger cause EUR/USDLMZSDY does not Granger cause EUR/USDKC1 does not Granger cause EUR/USDLMCADY does not Granger cause EUR/USDSPX does not Granger cause EUR/USDAUD/USD does not Granger cause EUR/USDCO1 does not Granger cause EUR/USD
1.4905∗1.5742∗1.6184∗∗1.5256∗2.1828∗∗∗1.4324∗1.8744∗∗
• Australian (GACGB10), German (GDBR10), Japanese (GJGB10), Swiss(GSWISS10), and US (GT10) government bond yields• UK bond spreads (BPSW10-GUKG10)For EUR/USD, Granger causality tests yield seven significant explanatoryvariables:• AUD/USD exchange rate• SPX stock market index
10.1 Explanatory Dataset
87
• The prices of copper (LMCADY), tin (KMSNDY), zinc (LMZSDY), coffee(KC1), and cocoa (CO1)Second, we carried out linear principal component analysis (PCA) on Grangercaused explanatory datasets in order to check for computational overheads. PCA isgenerally considered as a very efficient method for dealing with the problem of multicollinearity. It allows for reducing the dimensionality of the underlying dataset byexcluding highly intercorrelated explanatory variables. This results in a meaningfulinput for the learning machine.The procedure is based on an eigenvalue and eigenvector analysis of V = X X/T ,the k × k symmetric matrix of correlations between the variables in X. Each principal component is a linear combination of these columns where the weights arechosen in such a way that• The first principal component explains the greatest amount of the total variationin X, the second component explains the greatest amount of the remaining variation, and so on• The principal components are uncorrelated with each otherThe results of the PCAs for the respective exchange rates are given byTables 10.4–10.6. The column headed by “C1” and “V1” corresponds to the firstprincipal component, “C2” and “V2” denote the second principal component andso on. The row labeled “Eigenvalue” reports the eigenvalues of the sample secondmoment matrix in descending order from left to right. The “Variance Prop.” rowdisplays the variance proportion explained by each principal component. This valueis simply the ratio of each eigenvalue to the sum of all eigenvalues. The “Cumulative Prop.” row displays the cumulative sum of the Variance Prop. row from left toright and is the variance proportion explained by principal components up to that
Table 10.4 PCA test results for EUR/GBP
EigenvalueVar. Prop.Cum. Prop.
EUR/USDGACGB10GJGB10IBEXJPY/USDLMNIDYCACMIB30PLATCHF/USDDJST
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
C11
3.690.340.34
1.370.120.46
1.090.100.56
1.010.090.65
0.980.090.74
0.950.090.83
0.840.080.90
0.610.060.96
0.210.020.98
0.200.020.99
0.050.011.00
V1
V2
V3
V4
V5
V6
V7
V8
V9
V10
V11
0.03 −0.20 −0.04 0.68 −0.64 0.03 −0.29 −0.01 0.02 −0.01 −0.010.070.20 −0.70 0.02 −0.30 0.02 −0.61 0.07 0.01 −0.04 0.000.050.26 −0.61 −0.11 −0.43 −0.09 0.58 0.15 0.01 0.01 0.000.08 0.09 −0.04 −0.05 −0.02 −0.02 0.07 −0.61 −0.61 −0.110.47−0.080.66 0.14 −0.11 −0.25 −0.07 −0.18 −0.66 0.02 −.01 0.000.080.27 −0.02 0.48 0.33 0.69 0.30 −0.10 0.00 −0.02 0.000.06 0.06 −0.03 −0.02 −0.01 −0.05 0.02 −0.09 0.60 −0.610.490.47 −0.05 0.10 −0.05 −0.02 −0.03 −0.01 0.06 0.79 −0.37 −0.08−0.020.51 0.29 −0.08 −0.18 0.17 −0.21 0.71 0.01 0.04 0.00−0.200.50 0.29 −0.08 −0.18 0.17 −0.21 0.71 0.01 0.04 0.000.502 0.06 0.07 −0.02 −0.02 −0.01 −0.04 0.03 −0.08 0.35 0.78
88
10 Description of Empirical Study and Results
Table 10.5 PCA test results for EUR/USD
EigenvalueVar. Prop.Cum. Prop.
LMSNDYLMZSDYKC1LMCADYSPXAUDUSDCO1
C1
C2
C3
C4
C5
C6
C7
1.760.250.25
1.070.150.40
0.990.140.55
0.970.140.68
0.910.130.81
0.750.110.92
0.550.081.00
V1
V2
V3
V4
V5
V6
V7
0.490.560.030.600.210.180.05
−0.02−0.08−0.44−0.01−0.230.520.69
−0.150.210.050.80 −0.22−0.070.11 −0.01 −0.55 −0.60−0.670.590.010.010.04−0.020.100.02 −0.190.770.13 −0.040.60 −0.51 −0.520.16 −0.560.590.03 −0.06−0.37 −0.10 −0.61 −0.01 −0.01
Table 10.6 PCA test results for EUR/JPY
EigenvalueVar. Prop.Cum. Prop.
GACGB10GDBR10GJGB10GSWISS10GT10IBEXSILVAU0003MBPSW10GUKG10EURCHF
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
2.300.230.23
1.220.120.35
1.050.110.46
1.020.100.56
1.010.100.66
0.960.100.76
0.790.080.84
0.670.070.90
0.650.070.97
0.330.031.00
V1
V2
V3
V4
V5
V6
V7
V8
V9
V10
0.35 0.45 0.12 −0.26 −0.14 0.08 −0.03 −0.09 −0.75 −0.060.55 −0.08 −0.15 0.25 −0.00 0.04 −0.09 −0.06 0.17 −0.750.16 −0.01 −0.06 −0.49 −0.18 −0.81 −0.13 0.04 0.17 −0.020.50 −0.04 −0.17 0.13 0.05 0.03 −0.16 −0.58 0.16 0.550.45 −0.10 0.10 0.31 −0.12 −0.06 0.01 0.74 −0.05 0.350.25 −0.41 0.17 −0.31 0.24 0.03 0.76 −0.07 −0.05 0.000.00 −0.12 −0.30 −0.24 −0.82 0.35 0.17 0.01 0.14 0.030.16 0.60 0.34 −0.27 0.10 0.25 0.10 0.11 0.57 0.02−0.06 −0.02 0.70 0.39 −0.44 −0.23 0.13 −0.30 0.03 −0.050.08 −0.48
0.44 −0.38
0.03
0.30 −0.57
0.03 −0.01 −0.02
order. The second part of the output table displays the eigenvectors corresponding toeach eigenvalue. The first principal component is computed as a linear combinationof the series in the group with weights given by the first eigenvector. The secondprincipal component is the linear combination with weights given by the secondeigenvector and so on. Per cumulative R2 , which we required to be not lower than0.99, significant multicollinearity could not be detected for any dependent variable.Consequently, the datasets were not reduced any further and all variables were kept.
10.3 Sequential Minimization Optimization (SMO) Algorithm
89
10.2 SVM ModelWe use the C-support vector classification (C-SVC) algorithm as described in [83,405] and implemented in R packages “e1071” [70] and “kernlab” [220]:Problem 10.1. Given training vectors xi ∈ Rn , i = 1, . . . , l, in two classes, and avector y ∈ Rl such that y ∈ {+1, −1}, C-SVC solves the following problem:l1min wT w + C ∑ ζiw,b,ζ 2i=1w.r.t.yi wT φ (xi ) + b ≥ 1 − ζi
(10.3)
ζi ≥ 0, i = 1, . . . , lIts dual representation is1min α T Qα − eT αα 2w.r.t.
(10.4)
0 ≤ αi ≤ C, i = 1, . . . , lyT α = 0where e is the vector of all ones, C is the upper bound, Q is an l × l positive semidefinite matrix, Qi j ≡ yi y j K(xi , x j ), and K(xi , x j ) ≡ φ (xi )T φ (x j ) is the kernel, whichmaps training vectors xi into a higher dimensional, inner product feature space bythe function φ . The decision function isf (x) = sgn
l
∑ yi y j K(xi , x) + b
(10.5)
i=1
10.3 Sequential Minimization Optimization (SMO) AlgorithmTraining of a SVM requires the solution of a quadratic programing problem (QP),i.e., maximizing a convex quadratic form subject to linear constraints. Such convexquadratic programs have no local maxima and their solution can always be foundanalytically. Furthermore, the dual representation of the problem showed how training can be successfully affected even in very high dimensional feature spaces. Theproblem of minimizing differentiable functions of many variables has been widelystudied, especially in the convex case, and many standard approaches can be directly
90
10 Description of Empirical Study and Results
applied to SVM training.1 For large learning problems with many training examples,however, the constrained quadratic optimization approach to solving SVMs quicklybecomes intractable in terms of time and memory requirements. For instance, atraining set of 50,000 examples will yield a Q-matrix with 2.5 billion elements,which cannot fit easily into the memory of a standard computer. Consequently, traditional optimization algorithms, such as Newton or Quasi-Newton, cannot be directly applied. Several researchers have proposed decomposition methods to solvethis optimization problem [55, 207, 222, 274, 317, 326, 404].Vapnik et al. describe a “chunking” method [55] using the fact that the solutionof a QP problem is the same if we remove the rows and columns of the matrix Q thatcorrespond to zero Lagrange multipliers. Thus, a large QP problem can be decomposed into a series of smaller QP subproblems in which all of the nonzero Lagrangemultipliers are identified and all zero Lagrange multipliers are discarded. After allthe nonzero Lagrange multipliers in Q have been identified, the last step then solvesthe remaining QP problem. While chunking reduces the size of the matrix from N 2(where N is the number of training examples) to the number of nonzero Lagrangemultipliers, it still does not handle large scale training problems since the matrixstill may not fit in the memory. Osuna [317] proposed a new decomposition algorithm for solving the SVM QP problem by showing that a large QP problem canbe broken into a series of subproblems by maintaining a small working set. At eachiteration step one or more examples that violate the KKT conditions are added to thesmaller QP. Osuna’s decomposition algorithm uses a constant size matrix for everysubproblem and adds/subtracts one example at every step. Since the working set isusually small, this method does not have memory problems. However, a numericalQP solver is still required which raises numerical precision issues. Joachims [207]improves Osuna’s methods with a strategy to select good working sets.The algorithm that is opted for in the underlying context, however, is Platt’s sequential minimization optimization (SMO) algorithm [326]. SMO is based on theidea that the quadratic programing problems can be broken up into a series of smallest possible QP problems which can be solved analytically. SMO solves the smallestpossible optimization problem at every step, by• Creating working sets of size 2 (involving two Lagrange multipliers) with a setof heuristics• Jointly optimizing the two Lagrange multipliersThe main advantage is that the solution for the multipliers at each step is analyticand no QP solver is used. By avoiding the large matrix computation, SMO canhandle very large training sets in between linear and quadratic time with a linear1 The first iterative algorithm for separating points from two populations by means of a hyperplane
is the perceptron algorithm proposed by [345]. This algorithm starts with an initial weight vectorw0 = 0 and adapts it each time a training point is misclassified by the current weights. In its primalform the algorithm updates the weight vector and bias directly whereas in its dual form the finalhypothesis is a linear combination of the training points. This procedure is guaranteed to convergeprovided there exists a hyperplane (i.e., there is a hyperplane whose geometric margin is positive)that correctly classifies the training data. In this case the data are said to be linearly separable. Ifno such hyperplane exists the data are said to be nonseparable.
10.4 Kernel Selection
91
amount of memory in the training set size. For comparison, standard approachessuch as chunking and Osuna’s algorithm can be on the order of cubic time. SMOcan therefore be seen as a special case of the Osuna decomposition algorithm, whichat the same time improves the latter method. Comparative testing against other algorithms, done by Platt, has shown that SMO is often much faster and has betterscaling properties [326].
10.4 Kernel SelectionEver since the introduction of the SVM algorithm, the question of choosing thekernel has been considered as very important. This is largely due to the effect thatthe performance highly depends on data preprocessing and less on the linear classification algorithm to be used. How to efficiently find out which kernel is optimalfor a given learning task is still a rather unexplored problem and subject to intensecurrent research. One approach is to use cross validation to select the parametersof the kernels and SVMs [104, 291] with varying degrees of success. The notionof Kernel target alignment [86, 87] uses the objective function of kernels spannedby the eigenvectors of the kernel matrix of the combined training and test data. Thesemidefinite programing (SDP) approach [241] uses a more general class of kernels,namely a linear combination of positive semidefinite matrices. They minimize themargin of the resulting SVM using a SDP for kernel matrices with constant trace.Similar to this, [56] further restrict the class of kernels to the convex hull of the kernel matrices normalized by their trace. This restriction, along with minimization ofthe complexity class of the kernel, allows them to perform gradient descent to findthe optimum kernel. Using the idea of boosting, [84] optimize where are the weightsused in the boosting algorithm. The class of base kernels is obtained from the normalized solution of the generalized eigenvector problem. Another possibility is tolearn the kernel using Bayesian methods by defining a suitable prior, and learningthe hyperparameters by optimizing the marginal likelihood [417, 418]. As an example of this, when other information is available, an auxiliary matrix can be usedwith the EM algorithm for learning the kernel [397]. Furthermore, [314] recentlyproposed a statistical inference framework for making kernels that are adaptive andallow independent scales for each dimension. This can be achieved by defining areproducing Kernel Hilbert space (RKHS) on the space of kernels itself in order tolet hyperkernels parameterize the kernels for each dimension.We choose to take a pragmatic approach by comparing a range of kernels with regards to their effect on SVM performance. The argumentation is that even if a strongtheoretical method for selecting a kernel is developed, unless this can be validatedusing independent test sets on a large number of problems, methods such as bootstrapping and cross validation will remain the preferred method for kernel selection.In addition, with the inclusion of many mappings within one framework it is easier to make a comparison. Standard kernels, which satisfy Mercer’s conditions and
92
10 Description of Empirical Study and Results
whose performance is compared within the underlying classification task, includethe following:Linear : k(x, x ) = x, xPolynomial : k(x, x ) = (scale · x, x + offset)degreeLaplace : k(x, x ) = exp(−σ  x − x )Gaussian Radial Basis Function : k(x, x ) = exp(−σ  x − x 2 )Hyperbolic : k(x, x ) = tanh(scale · x, x + offset)Besselnν +1 (σ  x − x )Bessel : k(x, x ) =( x − x )−n(ν +1)
(10.6)
The estimated parameters are provided in Table 10.7. In addition, we verify theuse of the exotic p-Gaussian kerneld(xi , x) pK(xi , x j ) = exp −(10.7)σpTo our knowledge, the p-Gaussian has hardly been tested yet on real datasets, butin theory, has interesting properties. Compared to the widely used RBF kernels, pGaussians include a supplementary degree of freedom in order to better adapt to thedistribution of data in high-dimensional spaces [131]. The p-Gaussian is thereforedetermined by two parameters p, σ ∈ R which define the Euclidean distanced(xi , x) =
n
∑ xi − x
1/22
(10.8)
i=1
between data points. The two parameters depend on the specific input set for eachexchange rate return time series and are calculated as proposed in [131]:p=
ln
ln(0.05)ln(0.95) dF ln dN
;σ=
dFdN=1/p(− ln(0.05))(− ln(0.95))1/p
(10.9)
In the case of EUR/USD, for instance, we are considering 1,737 eight-dimensionalobjects from which we calculate the kernel matrix. We consider 1,737 eightdimensional objects because at each 1 ≤ t ≤ 1, 737, the similarity among the eightvariables, as expressed by their Euclidean distance, has to be determined. The elements of the kernel matrix are then given by the single Euclidean distances of onevariable to the remaining variables of the same system according to (10.8). For thisreason 1, 737 × 1, 737 Euclidean distances, based on eight coordinates, have to becalculated in order to compute the 5% (dN ) and 95% (dF ) percentiles in that distribution which are needed to estimate p and s. The estimated parameters for allexchange rates are provided in Table 10.8.
10.4 Kernel Selection
93
Table 10.7 SVM parameter estimates with standard kernel functionsLinear
SVsCostDegreeScaleGammaCoef 0Offsetσ
Polynomial
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
1,6631––0.09–––
1,6841––0.10–––
1,6501––0.14–––
1,6461––0.09–––
1,6111––0.10–––
1,6291––0.14–––
Laplace
SVsCostDegreeScaleGammaCoef 0Offsetσ
Gaussian
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
1