,6881–––––0.04
1,7091–––––0.05
1,6851–––––0.06
1,6361––0.09–––
1,6621––0.10–––
1,6301––0.14–––
Hyperbolic
SVsCostOrderDegreeScaleGammaCoef 0Offsetσ
Bessel
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
8781––1––1–
9121––1––1–
9301––1––1–
1,182111––––1
1,268111––––1
1,472111––––1
EUR/GBP
EUR/JPY
EUR/USD
157110.12070.03072.96780.0834
158410.17810.03382.44820.1138
157110.10390.02542.88540.0711
Table 10.8 p-Gaussianmodel estimatesSVsCostdFdNpσ
94
10 Description of Empirical Study and Results
10.5 Cross ValidationThe development of multivariate models generally requires checking their validity. Whereas the reliability of linear models is normally expressed by theoreticallyjustified metrics, such as R2 , it is more difficult to examine the reliability of nonlinear methods, such as supervised learning methods. Cross validation is a commontechnique for estimating the quality (accuracy) of a classifier induced by supervisedlearning algorithms. In addition, cross validation may be used for reasons of modelselection, i.e., choosing a classifier from a given set or combination of classifiers([420]). In k-fold cross validation, the dataset D is randomly split into k mutuallyexclusive subsets of approximately equal size. Each time t ∈ {1, . . ., k}, the learningalgorithm is trained on D\Dt and tested k times on Dt . The cross validation accuracyis the overall number of correct classifications, divided by the number of instancesin the dataset, i.e., the average accuracy rate of the k single runs. Formally, if D(i)denotes the test set that includes instance xi = (xi , yi ), then the cross validation accuracy is given byaccCV =
1δ Γ (D\D(i) , xi ), yi∑n (x ,y )∈D
(10.10)
i i
Complete cross validation is the average of all possibilities for choosing m/k instances out of m. However, this can be a very time consuming process. In our case,the underlying in-sample dataset consisting of 1,737 instances is divided into k = 5,k = 10, k = 15, and k = 20 subsets. Now k test runs are initiated where (k − 1)/kinstances are considered for training and 1/k are considered for testing. This process is being repeated for all subsets until every subset has been considered oncefor testing. Tables 10.9 and 10.10 display the computing times required for trainingabove specified SVMs per currency pair. Table 10.9 describes the runtime behavior of SVMs with standard kernels, whereas Table 10.10 separately summarizes thecomputational expenses obtained when training the SVMs with a p-Gaussian.Obviously, the more subsets we choose to validate a model, the more increasestraining time. Moreover, if we rank the SVM models from lowest to highest withregards to their required running-times, we end up at the following ordering: Linear< Hyperbolic < Gaussian RBF < Polynomial < Laplace < Bessel < p-Gaussian.This can be explained by the input lengths of the different kernel functions (see(10.6)). The most basic kernel function requiring the least arithmetic operationsis the linear kernel. The most sophisticated kernel, requiring by far the most thecomputing time, is the p-Gaussian (compare Tables 10.9 and 10.10). This can beattributed to a noninteger p in the exponent of the Euclidean distance matrix. Itshould also be mentioned that due to the random nature of dividing the data intosubsets, the single accuracy rates change each time. This leads to a change of thetotal accuracy rate. If the change is significant, which could not be confirmed in ouranalyses, it is often recommended to change or optimize the SVM parameter.
10.6 Benchmark Models
95
Table 10.9 Runtime behavior of standard kernelsLinear
No CV5-fold CV10-fold CV15-fold CV20-fold CV
Polynomial
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
00:0200:0500:0900:1200:16
00:0100:0400:0700:1100:15
00:0100:0300:0600:0900:12
00:0200:0500:1000:1500:20
00:0100:0400:1100:1500:21
00:0100:0500:1000:1500:21
Laplace
No CV5-fold CV10-fold CV15-fold CV20-fold CV
Gaussian
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
00:0500:1100:1700:2200:27
00:0500:1100:1700:2200:27
00:0500:1000:1500:2000:26
00:0200:0500:1000:1400:18
00:0200:0500:1000:1400:18
00:0100:0500:0900:1400:17
Hyperbolic
No CV5-fold CV10-fold CV15-fold CV20-fold CV
Bessel
EUR/GBP
EUR/JPY
EUR/USD
EUR/GBP
EUR/JPY
EUR/USD
00:0200:0400:0700:1100:16
00:0200:0400:0700:1000:14
00:0200:0400:0700:1000:13
00:0800:1700:2200:2700:32
00:0800:1700:2300:2900:35
00:1000:2000:2600:3200:38
Table 10.10 Runtime behavior of p-Gaussian
p-GaussianEUR/GBP EUR/JPY EUR/USDNo CV5-fold CV10-fold CV15-fold CV20-fold CV
02:0205:5510:5218:3228:15
02:0105:5510:5018:2728:06
01:5505:4210:3918:2527:52
10.6 Benchmark ModelsLetting yt represent the exchange rate at time t, we forecasted the variablesgn(Δ yt+h ) = sgn(yt+h − yt )
(10.11)
where h = 1 for a one-period forecast with daily data. Even small tremors were interpreted as real changes of directions.Kernel-dependent SVM forecasts were compared to the forecasts of two kindsof benchmark models: the naive model and the ARMA(p,q) model. The naive
96
10 Description of Empirical Study and Results
strategy assumes that the most recent period change is the best predictor of thefuture, i.e.,(10.12)sgn(Δ ŷt+1 ) = sgn(Δ yt )where Δ yt is the actual rate of return at period t and Δ ŷt+1 is the predicted rate ofreturn for the next period.The ARMA(p,q) model as described in Sect. 8.2.3 is the most general familyof models for representing stationary processes and therefore represents a suitablebenchmark. We use the model estimates as given by (8.12)–(8.14)• Δ yt = −0.0526Δ yt−1 − 0.0562Δ yt−3 + εt for the EUR/GBP series• Δ yt = 0.0287Δ yt−1 + εt for the EUR/JPY series• Δ yt = −0.5959Δ yt−1 + 0.5323εt−1 + εt for the EUR/USD seriesThe EUR/GBP and EUR/JPY models are AR(p) models. Dynamic forecasting withAR(p) models is done by generating the one-step ahead forecast and then using thisfor a two-step-ahead forecast and so on. That is, the optimal one-step-ahead prediction at time T is the conditional expectation of yT +1 given {yT , yT −1 , yT −2 , . . .}:ŷT +1 − ĉ = α̂1 (yT − ĉ) + α̂2 (yT −1 − ĉ) + . . . + α̂ p (yT −p+1 − ĉ)
(10.13)
and the two-step prediction is:ŷT +2 − ĉ = α̂1 (yT +1 − ĉ) + α̂2 (yT − ĉ) + . . . + α̂ p (yT −p+2 − ĉ)
(10.14)
and so on. Since in our case ĉ = 0, the s-step-ahead forecast for the EUR/GBP modelis given byŷT +s = α̂1 yT +s−1 + α̂3 yT +s−3(10.15)and the s-step-ahead forecast for the EUR/JPY model isŷT +s = α̂1 yT +s−1
(10.16)
For a general ARMA(p,q) model the s-step-ahead predictions areŷT +s − ĉ = α̂1 (ŷT +s−1 − ĉ)+ α̂2 (ŷT +s−2 − ĉ) + . . . + α̂ p (ŷT +s−p − ĉ)
(10.17)
+ β̂s εT + β̂s+1εT −1 + . . . + β̂qεT −q−sfor s ≤ q. For s > q only the AR part determines the forecasts. The EUR/USD modelis an ARMA(1,1) model with ĉ = 0 whose s-step-ahead predictions are thereforeŷt+s = α̂1 yT +s−1 + β̂s + εt
(10.18)
Note that the GARCH models, as estimated in Sect. 8.2.4 are not useful for servingas benchmark models within the realm of tackling the BCP. The purpose of thisfamily of models is to describe the development of the second statistical moment,whereas the task here is to predict directional movements, i.e., ups and downs of theexchange rate.
10.7 Evaluation Procedure
97
10.7 Evaluation ProcedureThe evaluation procedure is twofold. Out-of-sample forecasts are evaluated bothstatistically via confusion matrices and practically via trading simulations.
10.7.1 Statistical EvaluationGenerally, a predictive test is a single evaluation of the model performance based oncomparison of actual data with the values predicted by the model. We consider theBCP where, formally, each instance I is mapped to one element of the set {−1, +1}of positive and negative class labels by a classification model. Given a classificationmodel and an instance, there are four possible outcomes. If the instance is positiveand it is classified as positive, it is counted as a true positive. If it is classified asnegative, it is counted as a false negative. If the instance is negative and it is classifiedas negative, it is counted as a true negative. If it is classified as positive, it is countedas a false positive. Given a classification model and a set of instances (the test set),a two-by-two confusion matrix (also called a contingency table), as is illustrated inFig. 10.1, can be constructed representing the dispositions of the set of instances.This confusion matrix forms the basis for many statistical performance metrics (seefor instance [127], pp. 2–4). Since we are equally interested in predicting ups anddowns, the accuracy rateAccuracy =
TP+TNP+N
(10.19)
defined as the sum of true positives T P and true negatives T N divided by the sum ofthe total amount of positive observations P and negative observations N is the rightstatistical performance measure to apply.
Hypothesized Class
True Class
Fig. 10.1 Confusion matrix
+1
+1
−1
TruePositives(TP)
FalsePositives(FP)
TrueFalse−1 Negatives Negatives(TN)(FN)
98
10 Description of Empirical Study and Results
10.7.2 Operational EvaluationHowever, potential evidence for the predictability of exchange rate returns does notnecessarily imply market inefficiency. It is likely that the small average excess returns would not generate net gains when employed in a trading strategy once transaction costs have been taken into account. This idea has been referred to as the concept of speculative efficiency which implies the absence of unexploited speculativeprofits. Therefore, under this definition of market efficiency, these markets wouldnot be classified as inefficient. Practical or operational evaluation methods focus onthe context in which the prediction is used by imposing a metric on prediction results which permits testing above hypothesis. More generally, when predictions areused for trading or hedging purposes, the performance of a trading or hedging metric provides a measure of the model’s success. We formulate a trading simulationand calculate its profitability in simulated trading. First of all, return predictions ytwere translated into positions. Next, a decision framework was established that tellsus when the underlying asset is bought or sold depending on the level of the priceforecast.⎧⎨ 1 : ŷt < yt−1 − τ(10.20)It = −1 : ŷt > yt−1 + τ⎩0 : otherwiseIf the model predicts a negative return, the trader finds himself in a long position(It = 1). A long position is characterized by a “sell” signal concerning the base currency (EUR) as per today. The intuition is as follows: if the exchange rate is believedto go down tomorrow, then one should sell the base currency today, since 1 EUR isworth more in terms of foreign currency today than it will be tomorrow. Otherwise,if the model predicts a positive return, the trader finds himself in a short position(It = −1). A short position is characterized by a “buy” signal concerning the basecurrency (EUR) as per today. The intuition is as follows: if the exchange rate is believed to go up tomorrow, then one should buy the base currency today, since it ischeaper in terms of foreign currency than it will be tomorrow.The parameter t describes a subjective threshold, which can be added to the lastprice change leading to different signals. This may be useful if one intends to follow a longer run strategy where t guarantees that the base currency will not bebought/sold above/below a certain level. We decided to set t = 0.For measuring prediction performance on the operational level, the followingnine metrics were chosen: cumulated P/L, shape ratio as the quotient of annualizedP/L and annualized volatility, maximum daily profit, maximum daily loss, maximumdrawdown, value-at-risk with 95% confidence, average gain/loss ratio, and trader’sadvantage. A formal definition for each of these metrics is given in Table 10.11.Note that every metric is somewhat dependent on the gain or loss
πt = It−1 (yt − yt−1 )
(10.21)
on the position at time t but makes a unique statement. Accounting for transactioncosts (TC) is important in order to assess trading performance in a realistic way.
10.8 Numerical Results and Discussion
99
Table 10.11 Operational performance measuresMeasure
Definition
Cumulated profit and lossSharpe ratio
TPLCT = ∑t=1πtATSR = PL,withPLA = 252 × T1 ∑t=1πt andσA T√1TA2σT = 252 × T −1 × ∑t=1 (πt − π̄ )max(π1 , π2 , . . ., πT )min(π1 , π2 , . . ., πT )MD = min(PLCt − maxi=1,...,t (PLCi ))VaR = μ − Q(π , 0.05), μ = 0T(πt − It × TC), where It = 1 if πt−1 × πt < 0,NPLCT = ∑t=1else It = 0(Sum of all πt >0)/# upAGAL = (Sum of all πt <0)/# down×AL)TA = 0.5 × 1 + √(W T ×AG)+(LTwith W T :=22
Maximum daily profitMaximum daily lossMaximum drawdownValue-at-RiskNet cumulated profit and lossAverage gain/lossTrader’s advantage
(W T ×AG )+(LT ×AL )
number of winning trades, LT := number of losing trades,AG := average gain in up periods, and AL := average lossin down periods
Between market-makers an average cost of three pips (0.0003) per trade for a tradable amount of typically 10–20 million EUR is considered as a reasonable guessand thus incorporated in net cumulated profit.
10.8 Numerical Results and DiscussionIn order to compare forecasts for the same series across different models, accuracy rates for the out-of-sample period are depicted by bar charts as shown inFigs. 10.2–10.4.One immediate drawback may be the obtained accuracy, which is only 5% betterfor the best SVM models compared to the simple naive predictor. However, it shouldbe stressed that the tested datasets are very small due to the short history of theEuro. As argued by [246], there are important data mining applications where thedata is scarce and more research is needed towards methods that can deal with suchdatasets. This work backs this claim.Furthermore, Part II has shown that forecasting exchange rate directions is a verydifficult task. Foreign exchange markets are highly liquid and considered as very efficient, and it is difficult to gain superior information out of publicly available data.Consequently, if SVM accuracy rates outperform those of nave or random strategies, the SVM technique can be generally justified to predict exchange rate returndirections. This is the case for all three exchange rates. In particular, we find that hyperbolic SVMs deliver superior performance for out-of-sample prediction across allthree currency pairs. In the case of EUR/GBP, the Laplace SVM performs equally
100
10 Description of Empirical Study and Results
p-GaussianBesselLaplaceHyperbolicRBFPolynomialLinearAR(1,3)Naive40%
42%
44%
46%
48%
50%
52%
54%
56%
58%
60%
48%
50%
52%
54%
56%
58%
60%
Fig. 10.2 Statistical evaluation: EUR/GBP
p-GaussianBesselLaplaceHyperbolicRBFPolynomialLinearAR(1)Naive40%
42%
44%
46%
Fig. 10.3 Statistical evaluation: EUR/JPY
well as the hyperbolic SVM. Other models are out-performed by the hyperbolic kernel SVM more clearly in the cases of EUR/JPY and EUR/USD. This observationmakes hyperbolic kernels promising candidates to map all sorts of financial marketreturn data into high dimensional feature spaces.Tables 10.12–10.14 give the results of the trading simulation. The depicted values simply represent an aggregation of the EUR returns bought or sold with foreign
10.8 Numerical Results and Discussion
101
p-GaussianBesselLaplaceHyperbolicRBFPolynomialLinearARM A(1,1)Naive40%
42%
44%
46%
48%
50%
52%
54%
56%
RBF
Hyper. Laplace
58%
60%
Fig. 10.4 Statistical evaluation: EUR/USD
Table 10.12 Operational performance: EUR/GBPNaivePLCTSRmax Pmax LMDVaRNPLCTAG/ALTA
ARMA
Linear
Polyn.
Bessel
p-Gauss.
−0.0075 −0.0911 −0.0936 −0.0936 −0.0390 0.1036 0.0155 −0.0411 0.0596−0.0797 −0.9670 −0.9937 −0.9937 −0.4135 1.0994 0.1641 −0.4367 0.63230.0149 0.0149 0.0168 0.0168 0.0168 0.0149 0.0168 0.0139 0.0123−0.0168 −0.0168 −0.0149 −0.0149 −0.0139 −0.0168 −0.0139 −0.0168 −0.0168−0.0381 −0.0381 −0.0362 −0.0362 −0.0350 −0.0381 −0.0351 −0.0356 −0.0381−0.0070 −0.0077 −0.0075 −0.0075 −0.0073 −0.0070 −0.0069 −0.0074 −0.0069−0.0612 −0.1955 −0.1275 −0.1275 −0.0903 −0.0559 −0.0196 −0.0921 0.01431.0581 0.9279 0.8037 0.8037 0.9171 1.0398 0.8993 0.8823 1.01890.0000 0.4185 0.5300 0.5300 0.4872 0.4814 0.5899 0.3935 0.4351
Table 10.13 Operational performance: EUR/JPYNaivePLCTSRmax Pmax LMDVaRNPLCTAG/ALTA
ARMA
Linear
Polyn.
RBF
Hyper. Laplace
Bessel
p-Gauss.
0.0544 0.0525 −0.0948 −0.0948 −0.2191 −0.1387 −0.2867 −0.3114 −0.24980.3868 0.3729 −0.6743 −0.6743 −1.5568 −0.9862 −2.0360 −2.2112 −1.7460.0219 0.0219 0.0207 0.0207 0.0207 0.0217 0.0207 0.0207 0.0205−0.0205 −0.0217 −0.0219 −0.0219 −0.0219 −0.0219 −0.0219 −0.0219 −0.0219−0.0853 −0.0866 −0.0648 −0.0648 −0.0867 −0.0620 −0.0867 −0.0648 −0.0867−0.0100 −0.0097 −0.0109 −0.0109 −0.0111 −0.0108 −0.0113 −0.0115 −0.01130.0028 0.0522 −0.1527 −0.1527 −0.2761 −0.1984 −0.3446 −0.3618 0.30261.0411 1.0033 0.9000 0.9000 0.8828 0.8646 0.8332 0.8375 0.82180.0000 1.0000 0.4301 0.4301 0.4325 0.4365 0.4115 0.4035 0.4014
102
10 Description of Empirical Study and Results
Table 10.14 Operational performance: EUR/USDNaivePLCTSRmax Pmax LMDVaRNPLCTAG/ALTA
ARMA
Linear
Polyn.
RBF
Hyper. Laplace
Bessel
p-Gauss.
−0.1807 −0.1035 −0.1326 −0.1326 −0.0930.0480 −0.1005 −0.1617 0.1018−1.2345 −0.7052 −0.9043 −0.9043 −0.0630 0.3252 −0.6851 −1.1037 0.68900.0196 0.0196 0.0167 0.0167 0.0196 0.0196 0.0189 0.0187 0.0189−0.0189 −0.0187 −0.0196 −0.0196 −0.0187 −0.0189 −0.0196 −0.0196 −0.0196−0.0417 −0.0439 −0.0448 −0.0448 −0.0439 −0.0441 −0.0448 −0.0448 −0.0448−0.0125 −0.0110 −0.0126 −0.0126 −0.0118 −0.0108 −0.0118 −0.0117 −0.0112−0.2368 −0.2079 −0.1743 −0.1743 −0.0597 0.0000 0.1452 −0.2106 0.05110.9471 1.0579 0.8812 0.8812 1.0362 0.9627 0.9457 0.9457 1.10870.0000 0.4536 0.6253 0.6253 0.5683 0.5531 0.5838 0.4219 0.4992
currency independent of the notional amount. The following conclusions can bedrawn. The hypothesis of speculative efficiency may be rejected if NPLCT > 0 holdsfor a particular strategy, i.e., if the cumulated profit after transaction costs over theconsidered trading period is positive. Consider Table 10.12. It is interesting to observe that only SVMs equipped with hyperbolic, Laplace, and p-Gaussian kernelsare able to achieve a positive net cumulated profit.However, if we look at the results for EUR/JPY as given in line 7 of Table 10.13,one may have serious concerns on whether SVMs are able to make speculative profits. In fact, all trading strategies result in a negative net cumulated profit and the navestrategy is the only one generating a positive net cumulated profit. For EUR/USD,however, the p-Gaussian SVM is able to reject the hypothesis of speculative efficiency by generating positive net cumulated profit. To summarize, we can reject thespeculative efficiency hypothesis since for all three exchange rates speculative profits can be made. The SVM approach has only merit for EUR/GBP and EUR/USD.Thus, in the spirit of computational complexity, the superior models seem to be acceptable approximations to a hypothetical optimal forecasting technique that wouldhave total knowledge about new information.A different way of analyzing the results could be as follows: let the best forecasting technique be the one that does not only satisfy one criterion best, but thatis superior with regards to multiple trading metrics. Dominant strategies are represented by the maximum value(s) in each row. Operational evaluation results confirmstatistical ones in the case of EUR/GBP. Both the hyperbolic and the Laplace SVMgive best results along with the RBF SVM. For EUR/JPY and EUR/USD the results differ. The statistical superiority of hyperbolic SVMs cannot be confirmed onan operational level which is contradictory to the EUR/JPY and EUR/USD operational results at first glance. The reason for this phenomenon stems from the factthat operational evaluation techniques do not only measure the number of correctlypredicted exchange rate ups and downs, they also include the magnitude of returns.Consequently, if local extremes can be exploited, forecasting methods with less statistical performance may yield higher profits than methods with greater statisticalperformance. Thus, in the case of EUR/USD, the trader would have been betteroff applying a p-Gaussian SVM in order to maximize profit. Regarding EUR/JPY,
10.8 Numerical Results and Discussion
103
we find that no single model is able to outperform the nave strategy. The hyperbolic SVM, however, still dominates two performance measures. We also see thatp-Gaussian SVMs perform reasonably well in predicting EUR/GBP and EUR/USDreturn directions but not EUR/JPY. For the EUR/GBP and EUR/USD currency pairs,p-Gaussian data representations in high dimensional space lead to better generalization than standard Gaussians due to an additional degree of freedom p.To our knowledge, this is the first time, financial time series directions in general, and exchange rate directions in particular have been approached by SVMs.Further exploratory research therefore needs to be performed which could focus onSVM model improvements, for instance, examination of other sophisticated kernels,proper adjustment of kernel parameters