TheCattle Trader,” the “Silver Day-Trading System,” or “Easy Profits through Stock Index Trad-ing.” It is most likely that each of these systems has been finely tuned with rules unique tothis one market. Once aware of the optimization process and its results, these offers mustbe viewed more critically; after all, with optimization techniques able to test combinationsof indicators and rules, you could just as easily create a system that seemed to make asmuch on a single market. To prove that you have actually found something that worksrequires an understanding of the rules and the way it was tested, or time to watch the sys-tem operate in the real market. If possible, a comparison of the real trading profile com-pared with the expected performance based on testing should be available.
Student t-Test \
Among tests that help determine whether the results are significant, the student ¢-test isone of the most useful. It tells you whether there is a systematic bias in the data by show-ing whether the mean of the data is significantly different from zero. This is a useful pieceof information when you try to decide whether the results of only a few trades represent agood system, or whether a series of losing trades implies that a system has no value, For asingle set of trades produced by computer testing or by actual trading, the ¢-test is:°
average trade resultsts ————_ vege Trace resus sy V number of trades -
1 standard deviation of trade results
The critical value of t, which can be found in Appendix A, depends on the number of tradesin the sampie. For the case of a series of trades produced from the same market and samesystem, the degrees of freedom = number of trades — 1. Using the table in Appendix A, ifthere were only 10 trades, the degrees of freedom would be 9, and the value of ¢ must begreater than 1.833 to reach the 95% confidence level.
Where you want to know if two systems are producing results that are ‘significantly dif-ferent from each other, the technique is more complicated. It requires comparing the
5 Ben Warwick, Event Trading (Irwin, 1996).POINT-AND-FIGURE TESTING 525
 
mean, variance, and number of trades of the two systems; most important is the approxi-mation for the degrees of freedom, which allows you to determine the confidence level ofthe results:
    
my 2where %, =the average of method 1 trades2 = the average of method 2 trades= variance of method 1 trades$2 = variance of method 2 tradesn, = number of trades in method 1nz = number of trades in method 2
The calculation of degrees of freedom uses Satterthwaite’s approximation:2a8mM 7n:cianny+o
(2)Ye(nz - m)
(m-
df=
 
POINT-AND-FIGURE TESTING
The point-and-figure system is representative of the group of swing systems, Buy and sellsignals occur when prices move above or below previous highs and lows, respectively. Thecurrent market direction changes when a price reversal exceeds a specified cumulativeprice change, designated in points. Because it is a charting method, the reversal is shownin terms of boxes; the reversal criterion traditionally is expressed as a “3-box reversal” (fora complete explanation, see Chapter 11).
The size of the box and the number of boxes that define a reversal are the two variablesthat make up the reversal criteria. If silver were the object of the test, the box size mightstart as low as 10 points (1/10th of one cent, the minimum move) and increase in incre-ments of 10 points; the number of boxes in a reversal would be the integer values 1,2,3,
. Figure 21-114 shows that the expected pattern of test results is symmetric, extendingacross the diagonal drawn from the top left to the bottom right corners. This example fol-lowed the convention of putting the fastest trend change in the upper left (smallest boxsize and a 1-box reversal) and the slowest in the lower right corners.
The symmetry is the result of very similar performance for equal reversal values, theproduct of the box size, and the number of boxes in a reversal. For example, a trendchange of 40 points for the Swiss franc could have occurred using combinations of 40points x 1 box, 20 points x 2 boxes, 10 points x 4 boxes, 4 points x 10 boxes, 2 points x 20boxes, and 1 point x 40 boxes (among others). The difference in profitability occursbecause the selection with the smallest box size will satisfy the penetration criteria sooner,giving slightly faster signals and more trades. The average profitability of all combinationsof reversal size will appear as in Figure 21-118, This representation can be used with the 2-dimensional map to find the most consistent set of parameters. .
When tested over long periods, most markets have increasingly wide price ranges, asclearly seen in the S&P Testing for fixed combinations of box size and number of reversal526 TESTINGeS
 
FIGURE 21-11 Point-and-figure mapping strategy. (a) Reversal number of boxes. (6) Reversal(box size x number of boxes).
 
 
Reversal number of boxes
(a)
Profit or score
Reversal {box size x number of boxes)
(8)
 
boxes for the past 10 years would find the single parameter set that performed best overthis period; however, the increasing volatility would result in much larger profits andlosses in recent years. This change of risk will force the optimized results to favor parame-ters that performed best in the last few years and might ignore the profits and losses of theearlier period, which would be relatively small. To put it in perspective, a 10% move in the”S&P in 1987 would only be about 25 points, equal to a total of $12,500, while it would be90 points in 1997, or $45,000. If prices moved 1% per day (3.00 points) in 1987, a reason-able reversal combination would be from .45 points for fast trading (.15-point box with 3-box reversal) to a full day's move of 3.00 (1.00-point*box with a 3-box reversal). In 1997,either combination is likely to produce a new trade every few hours, creating a very differ-ent performance profile than prior years.
The problem of volatility was discussed in Chapter 11. The alternative to a fixed boxover time was to-vary the box size as a percentage of price. Therefore, if the best reversalin 1987 was .45 for the S&P at a value of 300, it would be 1.35 in 1997 at 900. However,this may not have increased enough, and many market analysts treat volatility as an expo-nential function with respect to price. This can be approximated using a square root func-tion or log function, both readily available in a spreadsheet or testing program:COMPARING THE RESULTS OF TWO SYSTEMS 527
 
Box size = p x V priceBox size = p x In (price)
where p is the percentage that is varied as a parameter in the optimization. This lastmethod would be similar to the technique of using point-and-figure on a log chart.
COMPARING THE RESULTS OF TWO SYSTEMS
The easiest way to understand the use of testing is by example. Using daily IMM Swiss francdata from January 1986 through February 1996, we compare the results of two trend sys-tems, one based on a moving average and the other on exponential smoothing. These cal-culations can be found in Chapter 4 (“Trend Calculations”). In both cases, the systemgenerates a buy signal or a sell signal when the trendline turns up or down, respectively.Table 21-1 shows the results of varying the calculation interval from 10 to 250 days in stepsof 10 days. Parts a and 6 give the results using a commission of $25 per trade, and parts ¢and d use $100 per trade.
Which System Is Better?
If we scan the results of the net profit (NetPrft) in column two of parts a and 6, we can findthat the largest profit was $52,125 for the exponential smoothing at 120 days, the largestreturn on account (ROA) was 376% for exponential smoothing at 160 days, and the high-est profit factor was 3.47 for the moving average at 250 days. However, looking at individ-ual values can be confusing and misleading. Choosing the one best performer is onlysensible if you.can guarantee that those parameters will be the ones that perform best inthe next period—an impossible request.
At the bottom of each part of Table 21-1 are the average and standard deviation of eachset of tests. For parts a and 8, the average net profits for the moving average system wereslightly better than exponential smoothing; however, looking at all the averages and vari-ance of the two systems show that the results are nearly identical. From a practical point ofview there is really no difference between the performance of the two systems. One mightprofit from one particular situation in one system,-but over the long run they will beextremely close. This may be easier to see in Figure 21-12, which shows the net profits ofthe two methods on the same chart. It may seem that the moving average technique per-forms slightly better with shorter intervals (30 to 100 days) and the exponential at longerones (120 to 170 days), but these distinctions are very small. The overall picture is that atrending system produces profits based on the time interval used in the calculation, and isnot dependent upon the specific method of calculation. This observation allows us to viewthe robustness of trend following as a trading approach.
A comparison of optimization results usually selects the one with the highest profits,but that may not be the best choice. Comparisons should account for the following mini-mum criteria:
& A good sample of parameter combinations were tested. This will include thosevariables that cause frequent, as well as slow, trading. The distribution of fastand slow test results should be similaz, which may involve scaling the calculationperiods in different ways. One simple way of knowing if you have succeeded inthis distribution is by observing the average number of trades per test on eachsystem.2. The average results of all tests and the variance of results Should be compared. Sig-nificantly higher and more consistent overall results is a strong argument for a fun-damentally sound approach.TABLE 21-1 Comparison of Swiss Franc Trend TestsSwiss Franc 1986-1995, 10 years
 
{a) Moving Average with $25 Commission .Period NetPrft PFact ROA MaxDD #Trds %Prft AvgTrd
 
 
250 39063 3.47 354 ~11038 36 39 logs240 3900 1.07 12 ~31700 él 28 64230 22350 1.48 129 ~17263 AT 36 476220 25950 1.54 136 ~19088 57 25 455210 25900 1.46 87 ~29725 65 37 398200 20213 1.44 68 -29575 47 36 430190 7388 116 20 ~371 13 54 27 145180 28438 174 106 ~26825 41 34 694170 38150 2.26 283 ~13463 31 33 748160 34363 1.82 110 31238 69 29 498150 30475 1.62 180 20375 69 26 442140 37363 1.79 180 20738 67 31 558130 40988 5.88 200 ~20488 64 31 640120 39488 _ (165 133 ~29750 76 37 520(10 17725 1.24 62 ~28463 90 37 197400 44513 1.68 294 ~15138 109 36 40890 21088 » 1.25 73 ~28988 113 35 18780 40988 1.60 330 ~12425 tol 3) 40670 33050 1.44 '71 ~19313 105 37 31560 34300 " 139 150 ~22850 us 34 29850 22488 1.23 - 19 ~28413 Wl 33 20340 46563 149 317 ~14700 139 36 33530 A6\75 140 | 269 ~17163 177 - 36 26120 16850 1.10 47 ~35963 : 238 31 7I10 5763 1.03 23 ~25150 - 340 33 17Average 2894} 1.57 Ist -23478 +: 98 33 394Stdev 12243 0.48 lol 7333 67 4 236
 
{b) Exponential Smoothing with $25 CommissionPeriod NetPrft PFact ROA MaxDD #Trds %Prft AvgTrd
 
250 37363 2.81 339 -11038 42 “+ 38 890240 11013 1.26 38 -28763 51 22 216230 26150 1.67 170 -15388 45 40 581220 20963 L4l 99° 21163 él 31 344210 27400 157 89 30725 St 33 3537200 9300 LA7 23 -39988 49 29 190190 18600 1.42 55 -33863 55 35 338180 23088 1.56 a ~25238 4l 32 563170 45388 3.06 336 -13500 4l 44 4107160 45863 242 376 -12188 él 38 752150 39338 1.89 181 -21788 67 36 587140 38313 1.82, 174 22063 65 28 589130 36113 171 179 20163 73 29 495,_120 52125 1.95 272 —19138 80 45 652110 12250 Las 29 42050 100 28 123100 35800 1.48 254 14088 113 38 31799 30163 1.40 WI7— -25788 Who 38 27280 29875 1.37, 1s9 = 15000 103 27 29070 7728 1.08 24 -31588 125 30 6260 37538 1.44 24 30363 107 30 35159 26563 129 128 ~20788 123 31 21640 37838 1.37 193 —19400 151 32 24930 30813 (24 Hé —26600 185 33 16720 23938 115 81 -29638 234 32 10210 -8975 0.96 = —20 44325 362 33 25Average 27776 1.58 147 24585 100 33 399Stdev 13709 0.55 {04 9056 7) 5 267
eet anata enn Hs ee RN RAR NT
528TABLE 21-1 (Continued)
 
{c) Moving Average with $100 Commission
 
 
 
 
 
Period NetPrft PFact ROA MaxDD #lrds ‘%Prft AvgTrd250 36363 3.07 325 11188 36 36 1010240 675 0.99 2 —35525 él 28 ail230 18825 1.38 92 20488 47 34 401220 21675 (42 100 -21775 57 25 380210 21025 1.36 66 -31900 65 37 323200 16688 1.34 SI 32725 Al 32 355190 3563 1.07 9 —-39888 5h 25 70180 25363 1.63 89 —28475 41 29 619170 34325 2.04 227 ~15150 Si 29 673160 29188 1.64 85 —34463 69 25 423150 25300 1.48 110 23000 69 26 367140 32338 1.64 141 -22913 67 30 483130 36188 1.72 161 -22438 64 27 565120 33788 1.53 104 -32600 76 34 445110 10975 1.14 35 —31163 90 33 122100 36338 1.51 209 ~17350 109 33 33390 12613 1.14 40 -31763 113 32 1280 33413 1.45 260 -12875 (or 28 33170 25175, 131 112 ~22463 105 33 24060 25675 1.28 96 ~26675 His 33 22350 14163 1.14 48 -29763 Wl 32 12840 36138 1.36 224 16125 (39 33 26030 32900 1.27 176 -18738 177 “31 18620 ~1000 (.00 2 42563 238 28 “410 -19738 0.92 -57 -34575 340 31 58Average 21624 1.43 108 ~ ~26263 98 31 319Stdev 14127 0.42 89 8323 67 3 236(d) Exponential Smoothing with $100 Commission
Period NetPrft PFact ROA MaxDD #Trds %Prft Avgird250 34213 2.51 306 11188 42 36 81S240 7188 1.16 23 31538 SI 22 l41230 22775 1.56 142 15988 45 36 506220 16388 1,30 7I ~23088 él 28 269210 23575 1.46 73 - = -32450 31 29 462200 5625 1.10 13 42763 49 29 15190 14475 1.31 40 36413 55 33 263180 20013 1.46 74 -26888 4l 32 488170 42313 277 284 -14875 41 4l 1032160 41288 217 286 —14450 6l 31 677150 34313 1.72 (42 —24100 67 36 512140 33438 1.66 138 24313 65 26 514130 30638 156 129 —23675 73 27 420120 46125 1.79 219 21025 80] 39 577(10 4750 1.05 10 —45650 100 24 48100 27325 1.34 162 -16863 13 33 24290 21838 1.27 77 —28488 M1 34 19780 22150 1.25 125, -17700 103 27 21570 1650 0.98 -5 —36088 425 27 -1360 29513 1.32 88 32663 107 29 27650 17338 ts 72 -24088 123 30 I4140 26213 1.24 126 20825 ISI 30 17430 16938 (12 60 ~28100 jg5 29 9220 6388 1.04 18 35250 234 30 2710 36125 0.85 62 58600 362 30 -100Average 20282 LA5 105 ~27523 100 31 324Stdev 16746 (0.45 92. 10798 7 4. 267
 
LT
529530 TESTING
 
 
FIGURE 21-12 Comparison of Swiss franc trend-following techniques.
Comparison of Swiss franc tests1986-1995
60,00050,00040,000
30,000
Total PL
20,000
10,000
  
TTT
 
0
 
 
10,000. 0 50 100 150 200 250Calculation period
-®- Moving average, $25 commission  -O- Exponential smoothing, $25 commission
 
3. Results should be scored in some way to include profitability and risk. Other mea-
surements can be used, but stability is of great importance.~ 4, Results, or scores, should be averaged to avoid spikes or unusual distortions.
5. Execution costs, including liquidity factors, must be included in the manner appro-Priate to each system. A trend-following system with entries and exits in the directionof the trend should have greater slippage than a countertrend entry method. ~
6. Make-mistakes in the direction of being conservative. If two systems have compara-ble returns, select the one with the lower risk. If they have similar returns and risk,choose the one with fewer trades.
Choosing the speed of the trading system is often the most critical decision in the per-_formance profile. Parameters that cause a trend method to trade faster are most desirable,because the higher number of trades adds confidence to the validity of the test results. Youmight also find that there is more consistency in the performance of a faster trading pro-gram compared with a slower one that generates the same net profits. To compensate forits advantages, faster trading is more susceptible to execution problems, so that perfor-mance will deteriorate quickly if slippage increases. Figure 21-13 shows the change in per-formance of all test intervals from 10 to 250 days for the moving average method whentransaction costs are increased from $25 to $100 per trade. Note that the shortest intervalturns from a small profit to a large loss and profits for intervals through 40 days are halved.Some areas of particularly poor performance, such as at 70 and 110 days, are magnified bythe increased costs. In general, there is very little impact on the performance of slowerparameter selections. “
PRCOFITING FROM THE WORST RESULTS
Under the right conditions, the worst results of an optimization or test sequence may holdvaluable information, it presents the worst-case scenario that should be carefully reviewedfor system problems, in particular with regard to risk control. If the area of poor results isbroad and continuous, it may also show a way to improve entry and exit timing. Figure 21-14a and b shows the continuous results of 1- and 2-variable tests. Both identify faster trad-ing areas as having maximum loss and slower trends as having best performance.RETESTING PROCKCOURE 531ee
ir
FIGURE 21-13 Impact of increased transaction costs on performance.
 
Impact of commissionsIMM Swiss franc, 1986-1995
 
60,000
40,000
20,000
Total PL
-10,000
-20,000
 
 
40,000. 0 50 100 150 200 250
Moving average calculation period
— Moving average, $25 commission -O- Moving average, $100 commissioni
The worst results are only useful if there is a net profit from taking the opposite posi-tion. This is the case only if the net loss is greater than the transaction costs, including com-missions and slippage. But even if the worst score has a near-zero return before transactioncosts, itcan be used profitably. Unprofitable results of a fast trend mean that a long or shortPosition taken at that point lasts only a few days and does not indicate a sustained trend.Use the following rules:
1. Trade a system of two moving averages, a long-term, designated by the best scoreD2, and the short-term, selected as the worst score D1.
2. When prices cross the long-term moving average, a long position becomes eligible.A long position is entered when the short-term moving average signals a short sig-nal. Positions ‘may be entered on a 1-day delay.
3. Short positions may be closed out when prices cross above the long-term trendline;they must be closed out when a long position is entered.
4. Short positions and exits from long positions are the opposite of rules 2 and 3.
Because the short-term signals are not good indicators of trends, they can be used as acountertrend entries. That allows for better execution when trading larger positions or inless liquid markets. The choice of immediate exit or delayed exit is the trader's preference.Once a position is held, there is greater risk waiting to exit, yet most exits would beimproved by better timing. - °
RETESTING PROCEDURE
One optimization is never the end of the testing process. If you have successfully finishedthe development of a system, and have retained the most recent data for out-of-sample ver-ification, then you should retest the system including the out-of-sample data before begin-ning to trade the program. Any additional data adds robustness to the system. The impactof adding data will depend on the nature of the market during the new period and theamount of data originally used in the testing.
As in step-forward testing, some analysts prefer to test a fixed amount of data, drop-ping off the oldest. This does not mean that they test only 6 months, or 1 year of data and532 TESTING
 
FIGURE 21-14 Dual use of test map. (a) Single-variable test. (b) Two-variable test.(D2) Best score ,
   
 
Profit /loss
°
    
\ Transaction
costs
     
(D,) Worst score
{a}
Fast SlowFast Fast
     
Follow trend
sy
~ Slow
    
     
Use tofade entry
      
Transactioncosts
        
SlowFast
 
retest every month. You may decide that data before a particular date is no longer relevant.In some cases, there are structural changes that support that approach, such as the forma-tion of a unified European currency, which would put controls on the variance in exchangerates between the participating countries even before the inception date. You might alsoconsider a new market with low liquidity during its earlier years as different from moreactive, current market conditions. Retesting remains an important way to adjust the system.to characteristics of new data. You might decide that retesting is necessary when time rep-resenting 5% to 10% of the test period has elapsed. Using the 2-variable test as an example,a shift may be