ariseout of the following two cases as is visualized in Fig. 9.2:
9.4 Structural Risk Minimization
69
• Case 1: The hypothesis space is smaller than the target space. As a consequence,the underlying function may lie outside the hypothesis space. A poor choice ofthe model space will result in a model mismatch which is measured by the approximation error.• Case 2: The technique for selecting a model from the hypothesis space is notoptimal. As a consequence, there is an error that is attributed to the learningprocedure. This type of error is referred to as estimation error.If we account for the importance of choosing an appropriate hypothesis space, theBCP can be summarized as follows. Given a set of decision functions H ∈ {±1}Xand a set of training examples according to (9.1) which are randomly generated according to a fixed unknown probability distribution P(x, y), the goal is to choosea function h∗ which best reflects the relationship between x and y for a set of testexamples, drawn from the same probability distribution. The relationship betweenx and y is modeled by a probability distribution P(x, y) which contains as a specialcase the possibility of a deterministic relationship y = f (x). The best function is theone which reproduces the relationship between x and y best on average. This function can be found by minimizing the expected risk defined as the average probabilityof misclassified test examplesR[ f ] =
12

| f (x) − y|dP(x, y)
(9.2)
However, since the true probability distribution which generates the relationshipbetween x and y is unknown, an inference procedure is required in order to at leastapproximate the function based on the observed training data.
9.4 Structural Risk MinimizationAn intuitive and widespread inference procedure is empirical risk minimization(ERM), which considers the average probability of misclassified training examplesin order to approximate (9.2)Remp [ f ] =
1 l∑ | f (xi ) − yi|l i=1
(9.3)
However, it has been argued that ERM is an incomplete inductive principle [295]since it is does not guarantee high generalization ability: an out-of-sample examplegenerated by the same probability distribution does not necessarily lie on the dottedline from Fig. 9.1. In other words, it is problematic to infer from low empirical riskto low expected risk. In order to do so the capacity/complexity of the hypothesisspace has to be examined which can be measured by the Vapnik–Chervonenkis(VC-) Dimension.The VC-dimension is an important property of a hypothesis space H and can bedefined for a variety of hypotheses spaces. If a given set of l training points can be
70
9 Support Vector Classification
labeled in all possible 2l ways, and for each labeling, a member of the hypothesisspace can be found which correctly assigns those labels, the set of points is said tobe shattered by that set of functions S p .Definition 9.2. The VC-dimension for the set of functions S p is defined as the maximum number of training points that can be shattered by S p .If the VC-dimension is p, then there exists at least one set of p points that can beshattered, but in general it will not be true that every set of p points can be shattered.For instance, suppose that R2 is the space in which the training data live and that forgiven separating lines, all points on the one side are assigned the class 1, and allpoints on the other side the class −1. According to Fig. 9.3, which is taken from[361], p. 10, there exist 23 = 8 possibilities to divide 3 training points into 2 classes.While it is possible to find three points that can be shattered by this set of functions,it is not possible to find four. Thus the VC-dimension of the set of oriented lines inR2 is p = 3. Hence p is referred to as the VC-dimension of a hypothesis space S p , ifpand only if there exists a set of points {xi }i=1such that these points can be separatedpin all 2 possible configurations and that no set {xi }qi=1 exists where q > p satisfiesthis property.The trade-off between ERM and hypothesis space capacity as measured by theVC-dimension is described by probabilistic bounds whose study is a major subjectof statistical learning theory. The meaning of these bounds can be explained as follows: if it is possible to explain the training data (i.e., to keep the empirical risk low)by a simple model (i.e., a hypothesis space whose VC-dimension is small comparedto the number of training examples), there is good reason for assuming that the truefunctional relationship has been found. Otherwise, if the training data can only beexplained by a hypothesis space of higher VC-dimension, this is not the case, sincethe machine may have used its capacity to memorize the single training examples(overfitting) instead of learning a more compact underlying regularity. Accordingly,it cannot be expected that out-of-sample examples can be reliably classified. Theprinciple of structural risk minimization (SRM) uses probabilistic bounds in order tominimize the expected risk by controlling both empirical risk and VC-dimension inorder to find a function that generalizes well on new unseen examples [85,404,405]
min Remp [ f ] +Sh
Fig. 9.3 Vapnik–Chervonenkis (VC-)dimension
p ln
2lp +1
l
− ln
δ4
(9.4)
9.5 Support Vector Machines
71
In this sense, SRM adapts the complexity of the learning machine to the problemunder consideration and is therefore superior to the traditional ERM principle [176]which has been used in many traditional neural network approaches.In some cases, the VC-dimension is equal to the number of parameters of thehypothesis space which in turn is related to the dimensionality of the observationsxi . Therefore, the number of free parameters is sometimes considered as a measureof complexity or capacity of a hypothesis space. However, this is not always truewith the negative consequence that the estimation of the capacity of a hypothesisspace cannot be reduced to counting of parameters, but in fact may be a difficultmathematical problem. On the other hand, the advantage is that one may still hope tobe capable of generalizing even if the data is high-dimensional. A class of learningalgorithms which have confirmed these hopes in various areas are support vectormachines (SVM), or more generally, kernel algorithms.
9.5 Support Vector MachinesThe difficulty in estimating the capacity of hypothesis spaces is that measures suchas the VC-Dimension are of combinatorial nature. This leads to a dilemma: goodestimates exist particularly for simple (i.e., linear) hypothesis spaces. However, inorder to analyze complex phenomena in the scientific domain, the goal is often toconstruct learning algorithms for nonlinear hypothesis spaces. SVMs are kernel algorithms which solve this dilemma in an elegant way by focusing on two majorprinciples: in a first step, SVMs map training examples in a high-dimensional feature space in order to construct a hyperplane which separates the two classes in asecond step. This will be explained in more detail in the following two subsections.
9.5.1 Learning in Feature SpaceDefinition 9.3. SVMs are kernel algorithms which apply a mapping φ from the original input space X into a high-dimensional feature space FX = ((x1 , y1 ), . . . , (xl , yl )) → F = φ (X) = (φ (x1 , y1 ), . . . , φ (xl , yl ))
(9.5)
where F is a Hilbert space, i.e., a complete vector space provided with an innerproduct.Figure 9.4 shows a conceptual example of such a feature mapping from a twodimensional input space to a two dimensional feature space ([85], p. 28). Thedata cannot be separated by a linear function in the input space. However, thetransformed feature vectors φ (xi , yi ) can be separated by a linear function in feature space. The decision boundary which is linear in F corresponds to a nonlineardecision boundary in X.
72
9 Support Vector ClassificationΦ
X
FΦ( )Φ( )Φ( )
Φ( )
Φ( )Φ( )
Φ( )
Φ( )
Φ( )
Φ( )Φ( )Φ( )Φ( )
Φ( )
Φ( )
Φ( )Φ( )
Φ( )
Fig. 9.4 Mapping from input space to feature space
A feature map can therefore simplify the classification task. But how can oneidentify the possibly best feature map for a given classification task? In the machinelearning community, the task of choosing the most suitable representation of thedata is known as feature selection. The objective of feature selection is to identifythe smallest set of features that still conveys the essential information contained inthe original attributes. This requires reducing the dimensionality of a given datasetwhich is beneficial since both the computational and generalization performance candegrade as the number of features grow. However, transforming the vectors in thetraining set D into a higher-dimensional space may incur computational problems.For very large training sets, the high dimensionality of F makes it very expensive,both in terms of memory and time, to represent the feature vectors φ (xi , yi ) corresponding to the training vectors (xi , yi ).
9.5.2 Kernel FunctionsFor a formal definition of a kernel function we refer to [85], p. 30.Definition 9.4. A kernel is a symmetric function K : X × X → R so that for all xi andx j in X, K φ (xi ), φ (x j ) where φ is a (nonlinear) mapping from the input space Xinto the Hilbert space F provided with the inner product K ., . .The inner product, therefore, does not need to be evaluated in the feature spacewhich provides a way of addressing the curse of dimensionality and has been
9.5 Support Vector Machines
73
referred to as the so called kernel trick [360]. However, the computation is stillcritically dependent upon the number of training patterns.A characterization of when a function K x, z is a kernel has been provided byMercer’s theorem [289] which is founded on reproducing Kernel Hilbert spaces(RKHS) (see for instance [11, 412]):Theorem 9.1. (Mercer’s Theorem). Let X be a compact subset of Rn . Suppose K isa continuous symmetric function such that the integral operator TK : L2 (X) → L2 (X)(TK f )(.) =is positive, that is
 

K(., X) f (x)dx
(9.6)
K(x, z) f (x) f (z)dxdz ≥ 0
(9.7)
X
for all f ∈ L2 (X). Then we can expand K x, z in a uniformly convergent series (onX × X) in terms of TK ’s eigenfunctions φ j ∈ L2 (X), normalized in such a way thatφ j L2 , and positive associated eigenvalues λ j ≥ 0,∞
K(x, z) = ∑ λ j φ j (x)φ j (z)
(9.8)
j=1
Hence, an inner product in feature space has an equivalent kernel in input space,K(x, z) = φ (x), φ (z) if K is a symmetric positive definite function that satisfiesMercer’s conditions as represented by (9.6) and (9.7).The main attraction of kernel functions is not only the fact that the applicationof kernel algorithms has proven to be very successful and includes world recordson famous benchmark data [92, 415]. Kernels are also very interesting from a theoretical perspective since they deal with three fundamental problems of empiricalinference:• Data representation: a kernel K(x, z) induces an imbedding of the data in thevector space• Similarity: a kernel K(x, z) can be perceived as a (nonlinear) measure of similarity which can be used for comparing data points• A-priori-knowledge: the solutions of kernel learning algorithms may be generallyexpressed as kernel developments in the training data. For this reason, a kernelK(x, z) parameterizes the hypothesis space which contains the solution, i.e., theknowledge which enters the learning process jointly with the training data.
9.5.3 Optimal Separating HyperplaneOnce the training examples are mapped into a high-dimensional feature space, thegoal is to construct a hyperplane in order to separate the two classes. There exist many possible linear classifiers that can separate the data and therefore solve
74
9 Support Vector Classification
Fig. 9.5 Optimal separatinghyperplane
the BCP. However, there is only one that maximizes the distance between itselfand the nearest data point of each class. This linear classifier is termed the optimalseparating hyperplane or maximum margin classifier. By explicitly maximizing themargin, the advantage of this sort of classifier is that it reduces the function capacity/complexity and therefore, minimizes bounds on the generalization error. It hasmaximal stability because it ensures the lowest probability of misclassification. Thiscan be explained by using geometric arguments paired with the ideas from Sect. 9.4.Consider the example in Fig. 9.5 which illustrates the binary classification toy problem of separating white from black dots.The data can be separated in different ways with margins indicated by the slimlines. The optimal separating hyperplane is represented by the fat line. Figure 9.5also shows how a linear discriminant that separates two classes with a small marginhas more capacity to fit the data than one with a large margin. The capacity of alinear discriminant is a function of the margin of separation. A hyperplane with asmall margin has more capacity since it can take many possible orientations and stillstrictly separate all the data. On the other hand, a hyperplane with a large marginhas limited flexibility to separate the data and has, therefore, a lower capacity thana small margin one. In general, it is thought that the capacity of a linear function isbeing determined by the number of variables. However, the size of the margin is notdirectly dependent on the dimensionality of the data. Thus, if the margin is fat, thenthe capacity of a function as measured by its VC-dimension can be low even if thenumber of variables is very high. As a consequence, problems caused by overfittingof high-dimensional data are greatly reduced and good performance, even for veryhigh-dimensional data, can be expected.Formally, the set of training examples D belonging to two different classes is saidto be optimally separated by the hyperplanew, x + b = 0
(9.9)
9.5 Support Vector Machines
75
Fig. 9.6 Separating hyperplane (w, b) for a two dimensional training set
x(w,b)
wxj
xi
d(w,b;xj)
d(w,b;xi)d(w,b;x)
b
d(w,b;x)
if it is separated without error and the distance between the closest vector to thehyperplane is maximal. The parameters (w, b) determine the weight vector and bias,and must be learned from the data by a given learning methodology. A geometricinterpretation is given in Fig. 9.6 where the hyperplane is the dark line with thepositive region (black dots) above and the negative region (white dots) below. Thevector w defines a direction perpendicular to the hyperplane, while varying the valueof b moves the hyperplane parallel to itself.The functional margin of a training example (xi , yi ) with respect to a hyperplane(w, b) is described by(9.10)dF (w, b; xi ) = yi ( w, xi + b)The distribution of the margins of the examples in a training set S is the functionalmargin distribution of (w, b). The minimum of the functional margin distribution iscalled the functional margin of (w, b) with respect to S and is given bydF = min dF (w, b; xi )i
(9.11)
Equation 9.9 can be simplified without loss of generality by considering a canonicalhyperplane [404], where the parameters w and b are constrained bymin | w, xi + b| = 1i
(9.12)
It states that the norm of the weight vector should be equal to the inverse of the distance of the nearest point in the data set to the hyperplane. A separating hyperplanein canonical form must satisfy the following constraintsd(w, b; xi ) = yi ( w, xi + b) ≥ 1
(9.13)
76
9 Support Vector Classification
According to above, the distance d(w, b; xi ) of a point xi from the canonical hyperplane (w, b) is termed geometric margin of xi from (w, b). The distribution of thegeometric margins of the examples in a training set S is the geometric margin distribution of (w, b). The minimum of the functional margin distribution is called thegeometric margin of (w, b) with respect to S and is given byd(w, b; x) = mini
1d(w, b; xi )w
(9.14)
and measures the Euclidean distances of the points from the decision boundary inthe input space. Thus, the geometric margin d will equal the functional margin ifthe weight vector is a unit vector. Figure 9.6 illustrates the geometric margin at twopoints xi and x j with respect to a hyperplane in two dimensions. The maximum geometric margin over all hyperplanes is referred to as the margin ξ of a training set S.A hyperplane realizing this maximum is known as a maximum margin hyperplane.The optimal hyperplane is given by maximizing the margin
ξ (w, b) = min d(w, b; xi ) + min d(w, b; xi )xi :yi =−1
xi :yi =−1
(| w, xi + b|)(| w, xi + b|)+ min= minxi :yi =−1xi :yi =+1ww1=min | w, xi + b| + min | w, xi + b|xi :yi =+1w xi :yi =−12=w
(9.15)
Hence, the hyperplane that optimally separates the data is the one that minimizes
Φ (w) =
1w22
(9.16)
It is independent of b because provided (9.13) is satisfied (i.e., it is a separatinghyperplane), changing b will move it in the normal direction to itself. Accordingly,the margin remains unchanged but the hyperplane is no longer optimal in that itwill be nearer to one class than the other. To consider how minimizing (9.16) isequivalent to implementing the SRM principle, suppose that the following boundholds,w < A(9.17)Then from (9.13) and (9.14),1(9.18)AAccordingly, the hyperplanes cannot be nearer than (1/A) to any of the data points.This reduces the amount of possible hyperplanes and hence, the capacity. TheVC-dimension p, of the set of canonical hyperplanes in n dimensional space isbounded by(9.19)p ≤ min R2 A2 , n + 1d(w, b; x) ≥
9.5 Support Vector Machines
77
where R is the radius of a hypersphere enclosing all the data points. Hence, minimizing (9.16) is equivalent to minimizing an upper bound on the VC-dimension.The solution to the optimization problem of (9.16) under the constraints of (9.13) isgiven by the saddle point of the Lagrange function [296].
Φ (w, b, α ) =
1w2 − ∑ αi (yi [ w, xi + b] − 1)2i=1
(9.20)
where α are the Lagrange multipliers. (9.20) has to be minimized with respect to wand b, and maximized with respect to α ≥ 0. Classical Lagrangian duality enablesthe primal problem, (9.20), to be transformed to its dual problem, which is easier tosolve. The dual problem is given bymax W (α ) = max min Φ (w, b, α )(9.21)α
α
w,b
The minimum with respect to w and b of the Lagrange function Φ is given byl∂Φ= 0 ⇒ ∑ αi yi = 0∂bi=1
(9.22)
l∂Φ= 0 ⇒ w = ∑ αi yi xi∂wi=1
Hence, from (9.20)–(9.22), the dual problem ismax W (α ) = max −α
α
l1 l lαi α j yi y j xi , x j + ∑ αk∑∑2 i=1 j=1k=1
(9.23)
The solution to the problem is given by
α ∗ = argmin −α
l1 l lααyyx,x−ijijij∑∑∑ αk2 i=1j=1k=1
w.r.t.
(9.24)
αi ≥ 0, i = 1, . . . , ll
∑ αjy j = 0
j=1
where the sign of the coefficient of xi is given by the classification of yi , the αi arepositive values proportional to the number of times misclassification of xi has causedthe weight to be updated. Points that have caused fewer mistakes will have smallerαi , whereas difficult points will have larger values. Thus, in the case of nonseparabledata, the coefficients of misclassified points must grow indefinitely.
78
9 Support Vector Classification
Solving (9.24) determines the Lagrange multipliers. The optimal separating hyperplane is then given byl
w∗ = ∑ αi yi xi
(9.25)
i=1
b∗ = −
1 ∗w , xr + xs2
where xr and xs are any support vector from each class satisfying
αr , αs > 0, yr = −1, ys = +1
(9.26)
f (x) = sgn( w∗ , x + b)
(9.27)
The hard classifier is then
Alternatively, a soft classifier may be used which linearly interpolates the margin⎧⎨ −1 : z < −1z : −1 ≤ z ≤ 1f (x) = h( w∗ , x + b), where h(z) =(9.28)⎩1 : z>1This may be more appropriate than the hard classifier of (9.27) in cases where thetraining data is linearly separable except for some exceptions. The soft classifierproduces a real valued output between −1 and 1 when the classifier is queried withinthe margin, where no training data resides. From the Karush–Kuhn–Tucker (KKT)conditionsαi (yi [ w, xi + b] − 1) = 0, i = 1, . . . , l(9.29)and hence only the points xi which satisfyyi [ w, xi + b] = 1
(9.30)
will have nonzero Lagrange multipliers. These points are termed support vectors(SV). If the data is linearly separable all the SV will lie on the margin and hence thenumber of SV can be very small. Consequently, the hyperplane is determined by asmall subset of the training set; the other points could be removed from the training set and recalculating the hyperplane would produce the same answer. Hence,SVM can be used to summarize the information contained in a data set by the SVproduced. If the data is linearly separable the following equality will holdl
w2 = ∑ αi = ∑ αi = ∑i=1
i∈SV
∑ αi α j yi y j xi , x j
(9.31)
i∈SV j∈SV
Hence, from (9.19) the VC dimension of the classifier is bounded byp ≤ min R2 ∑ , n + 1i∈SV
(9.32)
9.5 Support Vector Machines
79
and if the training data X is normalized to lie in the unit hypersphere, the VCdimension of the classifier is bounded byp ≤ 1 + min
∑ ,n
(9.33)
i∈SV
9.5.4 Generalized Optimal Separating HyperplaneSo far the discussion has been restricted to the case where the training data is(strictly) linearly separable. However, this must not be the case. Figure 9.7 showsthat there is no linear classifier that is able to separate the two classes without errorin the input space. Linear separation of input points does, therefore, not work well:a reasonably sized margin requires misclassifying four points.There are two approaches to generalizing the problem, which are dependent uponprior knowledge of the problem and an estimate of the noise on the data. In the casewhere it is expected or possibly even known that a hyperplane can correctly separatethe data, it is appropriate to focus on a different measure of the margin distribution,an additional cost function associated with misclassification, which generalizes thenotion of margin to account for a more global property of the training sample.Alternatively, a more complex function can be used to describe the boundary (seeSect. 9.5.2). To enable the optimal separating hyperplane method to be generalized,[83] introduced a nonnegative margin slack variable ζi of an example (xi , yi ) withrespect to the hyperplane (w, b) as
ζi = ζ ((xi , yi ), (w, b), d(w, b; x))= max(0, d(w, b; x) − yi ( w, xi + b))
ζi
ζj
Fig. 9.7 Generalized optimalseparating hyperplane
d(w,b;xj)
(9.34)
80
9 Support Vector Classification
given a fixed target margin d(w, b; x). Informally, this quantity measures how mucha point fails to have a margin of d(w, b; x) from the hyperplane. If ζi > d(w, b; x),then xi is misclassified by (w, b). The norm ζ 2 measures the amount by which thetraining set fails to have a margin d(w, b; x) and takes into account any misclassifications of the training data. Figure 9.7 shows the size of the margin slack variablesfor two misclassified points for a hyperplane with unit norm. All of the other pointsin the figure have their slack variable equal to zero since they have a positive marginof more than d(w, b; x).The slack variables are associated with a penalty functionFσ (ζ ) = ∑ ζiσ , σ > 0
(9.35)
i
where ζi are a measure of the classification errors. The optimization problem isto minimize the classification error as well as minimizing the bound on the VCdimension of the classifier. The constraints of (9.13) are modified for the nonseparable case toyi ( w, xi + b) ≥ 1 − ζi
(9.36)
where ζi ≥ 0. The generalized optimal separating hyperplane is determined by thevector w, that minimizes the function
Φ (w, ζ ) =
1w2 + C ∑ ζi2i
(9.37)
where C is a given value subject to the constraints of (9.36). The solution to theoptimization problem of (9.37) under the constraints of (9.36) is given by the saddlepoint of the Lagrange function [296]
Φ (w, b, α , ζ , β ) =
1w2 + C ∑ ζi2il
− ∑ αi (yi [wT xi + b] − 1 + ζi)
(9.38)
i=1l
− ∑ βi ζij=1
where α , β are the Lagrange multipliers. The Lagrangian has to be minimized withrespect to w, b, x, and maximized with respect to α , β . As before, classical Lagrangian duality enables the primal problem, (9.37), to be transformed to its dualproblem. The dual problem is given by,(9.39)max W (α , β ) = max W min Φ (w, b, α , ζ , β )α
α ,β
w,b,ζ
9.5 Support Vector Machines
81
The minimum with respect to w, b, ζ of the Lagrangian F is given byl∂Φ= 0 ⇒ ∑ αi yi = 0∂bi=1l∂Φ= 0 ⇒ w = ∑ αi yi xi∂wi=1
(9.40)
∂Φ= 0 ⇒ αi + βi = C∂ζHence, from (9.38)–(9.40), the dual problem becomesmax W (α ) = max −α
α
l1 l lαi α j yi y j xi , x j + ∑ αk∑∑2 i=1 j=1k=1
(9.41)
and the solution to the problem is given by,
α ∗ = arg minα
l1 l lαi α j yi y j xi , x j − ∑ αk∑∑2 i=1 j=1k=1
w.r.t.0 ≤ αi ≤ C, i = 1, . . . , l
(9.42)
l
∑ α jy j = 0
j=1
9.5.5 Generalization in High Dimensional Feature SpaceThere are some restrictions on the nonlinear mapping that can be employed (seeSect. 9.5.1). The optimization problem of (9.42) becomes,
α ∗ = arg minα
l1 l lαi α j yi y j K xi , x j − ∑ αk∑∑2 i=1 j=1k=1
w.r.t.
(9.43)
0 ≤ αi ≤ C, i = 1, . . . , ll
∑ α jy j = 0
j=1
(9.44)where K(x, x ) is the kernel function performing the nonlinear mapping into feature space, and the constraints are unchanged. Solving Problem 9.43 determines
82
9 Support Vector Classification
the Lagrange multipliers, and a hard classifier implementing the optimal separatinghyperplane in the feature space is given byf (x) = sgn
∑ αi yi K(xi , x) + b
(9.45)
i∈SV
wherel
w∗ , x = ∑ αi yi K(xi , x)
(9.46)
i=1
b∗ = −
1 l∑ αi yi [K(xi , xr ) + K(xi, xr )]2 i=1
The bias is computed using two support vectors, but can be computed using all theSV on the margin for stability [406]. If the kernel contains a bias term, the bias canbe accommodated within the kernel, and hence the classifier isf (x) = sgn
∑ αi K(xi , x)
(9.47)
i∈SV
Many employed kernels have a bias term and any finite Kernel can be made tohave one [158]. This simplifies the optimization problem by removing the equalityconstraint of (9.43). However, even though SVMs implement the SRM principle andhence can generalize well, a choice of the kernel function is necessary to producea classification boundary that is topologically appropriate. It is always possible tomap the input space into a dimension greater than the number of training points andproduce a classifier with no classification errors on the training set. However, thiswill generalize badly since the function learnt is very complex.
Chapter 10
Description of Empirical Study and Results
In the following two sections we will focus on the task of predicting a rise (labeled“+1”) or fall (labeled “−1”) of daily EUR/GBP, EUR/JPY, and EUR/USD exchangerate returns. To predict that the level of the EUR/USD, for instance, is close to thelevel today, is trivial. On the contrary, to determine if the market will rise or fallis much more interesting for a currency trader whose primary focus is to buy thebase currency if the exchange rate went down and to sell the base currency if theexchange rate went up.All of the following experiments were performed on a PC equipped with an Intel Pentium M Processor 750 and running at 1,866 MHz with 1,024 MB of systemmemory under the Windows XP operating system. The simulations were programedin the R environment [201], an open source and high-level programing language thatprovides powerful tools for statistical analysis. The R packages 1071 [70] and kernlab [220] were adopted for the SVM model fitting. These packages use the SMOalgorithm that is implemented by the LIBSVM tool [70].
10.1 Explanatory DatasetExchange rate forecasting methods generally fall into one of the following threecategories: fundamental forecasts, technical forecasts, or mixed forecasts. The forecasting approach that we adopted is what we call a statistical or purely data drivenapproach that borrows from both fundamental and technical analysis principles. Itis fundamental since it considers relationships between the exchange rate and otherexogenous financial market variables