variance, and number of trades of the two systems; most important is the approxi-
mation for the degrees of fr i imation for er -eedom, which allows you to determine the confidence level of
 
ny nz
where %, = the average of method I trades%, = the average of method 2 trades's, = variance of method 1 trades5; = variance of method 2 tradesn, = number of trades in method 1n = number of trades in method 2
‘The calculation of degrees of freedom uses Satterthwaite's approximation:24)nm 7any
Gai-m) * Ga-my)
 
POINT-AND-FIGURE TESTING
The poine-and.Agure system i representative of the group of swing systems. Buy and sellsignals occur when prices move above or below previous highs and lows, respectively. Theurrent market direction changes when a price reversal exceeds a specified cumulativePr ce c ange designated in points. Because it is a charting method, the reversal is shownrms of boxes; the reversal criterion traditionally is expressed as a “3-box reversal” (fora complete explanation, see Chapter 11). -nec eS ofthe box and the number of boxes that define a reversal are the two variablesreversal criteria. If silver were the object of the test, the box size mistart as low as 10 points (1/10th of one ce: ini “dacease in incre1 nt, the minimum move) and increase in incre-ments of 10 points; the number of boxes ina Pe eal would be tne integer values 1.2.3“Figure, 21-114 shows that the expected pattern of test results is symmetric, extendingacros s the diagonal drawn from the top left to the bottom right corners. This example fol-o ion of putting the fastest trend change in the upper left (smalksize and a L-box reversal) and the slowest in the lower right comers. (smallest Boxroa hs INCL is the result of very similar performance for equal reversal values, thex size, and the number of boxes in a reversal. Fo: tchange of 40 points for the Swiss franc co apneic obra2 1 wuld have occurred using combinations of 40points x 1 box, 20 points x 2 boxes, 10 points x 4 boxes, 4 points x 10 boxes, 2 points x 20boxes, and 1 point x 40 boxes (among others). The difference in profitability occurscause the selection with the smallest box size will satisfy the penetration criteria sooner,giving slightly faster signals and more trades. The average profitability of all combinationsof reversal size will appear a in Figure 21-115. This representation can be used with the 2-imensional map road the most consistent set of parameters.fon tested over long periods, most markets have increasingly Wi é: F , ingly wide price ranges,clearly seen in the S&P. Testing for fixed combinations of box size and number of reversal 
TESTING526
eeeFIGURE 21-11 Point-and-figure mapping strategy. (a) Reversal number of boxes. (b) Reversal(box size x number of boxes).
4
 
 
 
 
 
 
i=
 
 
Box size
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Reversal number of boxes(a)
 
Profit oF score
Reversal (box size x number of boxes)
(e)
1
boxes for the past 10 years would find the single parameter set chat performed best overii i i ii large! roi
is period; however, the increasing volatility would result in muc !
jossee in recent years. This change of risk will force the optimized results to favor parame
ters that performed best in the last few years and might ignore the profits and losses of the _
earlier period, which would be relatively small. To put itn perspective, a 10% move in thein 1987 would only be about 25 poinis, equal to a total of $12,500, :50 points in 1997, oF 845,000. I prices moved 1% per day (3.00 points, in 1987, reasonbinati ints for fast trading (.15-p. -able reversal combination would be from .45 points trading (15-point box with 5sal) to a full day’s move of 3.00 (1.00-point-box with a in. 1997,aoe ce ver tienvion is likely ce produce a new trade every few hours, creating a very differformance profile than prior years.see ine problem of volatility was discussed in Chapter 11, The alternative to a xed boxovertime was t0 vary the box size asa percentage of price. Therefore, If the best reversal5 for d a i Id be 1.35 in at 900.-in 1987 was .45 for the S&P at a value of 300, it wou 9this may not have increased enough, and many market analysts treat volatility as an expo"nential Function with respect to price. This can be approximated using a square r00)tion or log function, both readily available in a spreadsheet or testing program:
iiii41i4
 
COMPARING THE RESULTS OF TWO SYSTEMS 327
Box size = p x VpriceBox size = p x In (price)
where p is the percentage that is varied as a parameter in the optimization. This lastmethod would be similar to the technique of using point-and-figure on a log chart,
COMPARING THE RESULTS OF TWO SYSTEMS
The easiest way to understand the use of testing is by example. Using daily IMM Swiss francdata from January 1986 through February 1996, we compare the results of two trend sys.tems, one based on a moving average and the other on exponential smoothing, These calculations can be found in Chapter 4 (“Tread Calculations"). In both cases, the systemgenerates a buy signal or a sell signal when the trendline turns up or down, respectively.Table 21-1 shows the results of varying the calculation interval from 10 to 250 days in stepsof 10 days. Parts a and 6 give the results using a commission of $25 per trade, and parts ¢and @ use $100 per trade.
Which System Is Better?
If we scan the results of the net profit (NetPrft) in column two of parts @ and 6, we can findthat the largest profit was $52,125 for the exponential smoothing at 120 days, the largestreturn on account (ROA) was 376% for exponential smoothing at 160 days, and the high-est profit factor was 3.47 for the moving average at 250 days. However, looking at individ.ual values can be confusing and misleading. Choosing the one best performer is onlysensible if you: can guarantee that those parameters will be the ones that perform best inthe next period—an impossible request.
At the bottom of each part of Table 21-1 are the average and standard deviation of eachset of tests. For parts @ and 6, the average net profits for the moving average system wereslightly better than exponential smoothing; however, looking at all the averages and variance of the two systems show that the results are nearly identical. From a practical point ofview there is really no difference between the performance of the two systems. One mightProfit from one particular situation in one system, but over the long run they will beextremely close. This may be easier to see in Figure 21-12, which shows the net profits ofthe two methods on the same chart. It may seem that the moving average technique per-forms slightly better with shorter intervals (30 to 100 days) and the exponential at longerones (120 to 170 days), but these distinctions are very small. The overall picture is that atrending system produces profits based on the time interval used in the calculation, and isnot dependent upon the specific method of calculation. This observation allows us to viewthe robustness of trend following as a trading approach.
A comparison of optimization results usually selects the one with the highest profits,but that may not be the best choice. Comparisons shouid account for the following mini.mum criteri.
 
¥ A good sample of parameter combinations were tested. This will include thosevariables that cause frequent, as well as slow, trading. The distribution of fastand slow test results should be similar, which may involve scaling the calculationperiods in different ways. One simple way of knowing if you have succeeded inthis distribution is by observing the average number of trades per test on eachsystem.2. The average results of all tests and the variance of results should be compared. Sig-Rificantly higher and more consistent overall results is a strong argument for a fun.Gamentally sound approach.TABLE 21-1 Comparison of Swiss Franc Trend Tests TABLE 21-1 (Continued)
Swiss Franc 1986-1995, 10 years (c) Moving Average with $100 Commission
 
 
 
 
 
Seeger tempest pores caetteges
 
 
 
   
 
 
 
 
 
(a) Moving Average with $25 Commission : Period Newrfe Pact ROA Maxbb winds aadPeriod ‘NetPrft PFact ROA MaxDD #Trds ae ave § 250 36363 3 07 325 =11188 36 36 1010| 7 354 —11038 36 }08s x 99 -2 35525 61 28 =i| 330 73800 tor 12 ~31700 61 28 6 230 18825 138 2 20488 47 34 401230 22350 148 429 ~17263 47 36 476 220 21675 142 100 21775 87 25 38020 25950 1S4 136 -19088 57 25 455 210 21025 136 66 31900 65 37 323210 25900 146 a7 ~29725 65 37 398 | 200 16688 134 51 32725 47 32 355200 20213 144 68 ~29575 47 36 430 i 130 3563 1.07 9 39888 st 25 70190 7388 16 20 5h 2 145, | 80 25363 163 89 28475 4 29 619180 28438 174 106 4t 4 694 : 170 34325, 2.04 227 15150 3 29 673170 38150 2.26 283 st 33 748 160 29188 1.64 85 —34463 69 25 423160 34363 182 110 69 29 498 ' 150 25300 Lag 110 -23000 69 26 367150 30475 4.62 150 69 26 442 133 32338 eq 141 22913 67 30 483140 37363 479 180 67 3 588 | 139 36188 172 161 ~22438 64 27 565130 40988 138 200 64 31 640 20 33788 153 104 -32600 76 34 445120 39488 1.65 133 7 7 520 | Ho 10975 14 35 -31163 90 33 122110 17725 “424 62 90 37 197 i 100 36338 1st 209 ~17350 109 33 333400 44513 168 294 109 36 408 if 90 12613 14 40 31763 13 32 11290 21088 1.25 73 113 35 137 i 80 33413 145 260 ~12875 tor 28 33180 40988 1.60 330 tor 31 406 70 25175 131 12 -22463 105, 33 24070 33050 1.44 17L 405 37 345 | 60 25675 1.28 96 ~26675 us 33 22360 34300 139 150 15 34 298 50 14163 (4 48 -29763 mw 32 12850 22488 1.23 19 Vt 33 203 0 36138 136 224 ~16125 139 33 26040 46563 149 317 139 36 335 30 32900 127 176 18738 17 31 18630 46175 1.40 269 17 36 261 | 20 =1000 (00 -2 42563 238 28 430 16850 iio 47 » 238 31 71 10 -19738 0.92 $7 -34575, 340 31 5810 5763 1.03 B -25150 340 33 7 Average 21624 1.43 108 ~ ~26263 98 31 319Average 28941 157 ist “27a 38 33 334 Stdev 14127 042 89 8323 67 3 2367Stdev 12213 0.48 tot (4) Exponential Smoothing with $100 Commission(b} Exponential Smoothing with $25 Commission son Period NetPrft Pact ROA’ MaxDD. lids ‘%Prft AvgirdPeriod NetPrft Pact ROA MoxDD- #Trds ‘MP rf a 250 3 13 251 306 —11188 42 36 Bis81 339 =11038 42 38 23 -31538 sl 2 1410 ior 136 38 -28763 5] 22 216 230 22775 142 15988 45 36 506230 26150 167 170 -15388 45 40 581 220 16388 7 23088 6 28 269220 20963 99 21163 61 31 344 210 23575 730.  -32450 5t 29 462210 27400 89 -30725 51 33 537 200 5625 13 42763 49 29 15,200 9300 . 23 ~39988 49 29 190 190 14475 40 ~36413 55 33 263190 18600 2 55 33863 55 35 338 180 20013 74 —26888 4 32 488180 23088 : on —25238 4l 32 563 10 42313 284 —14875 4! 4 1032170 45388 5. 336 -13500 4l 44 t107 41288 286 -14450 él 3t 677160 45863 : 376 él 38 752 150 34313 142 ~24100 67 36 512180 39338 189 181 67 36 587 ! 33438 138 24313 65 26 514140 38313 1.82 174 65 28 589 30 30638 129 23675 73 27 420130 36113 We 179 73 29 495 120 46125 219 -21025 80 39 377420 52125 19s 272 80 45 652 110 4750 io 45650 100 24 48“V0 12250 Las 29 100 28 123 100 27328 162 16863, 113 33 242100 35800 148 254 113 38 317 90 21838 7 ~28488 mw 4 19799 30163 140 117— su 38 272 80 22150 125 -17700 103 27 21580 29875, 1.37, 189 103 27 290 70 1650 -s5 36088 125 27 =1370 7725 108 24 125 30 62 60 29513 88 ~32663 107 29 27660 37538 144 124 107 30 3st 50 17338 72 ~24088 123 30 14150 26563 129 128 123 3 216 40 26213 126 ~20825 151 30 1740 37838 137 193 1st 32 249 30 16938 60 ~28100 18s 29 9230 30813 (24 16 185 33 167 20 6388 18 ~35256 234 30 2720 23938 is al 234 32 192 10 36125 -62 ~58600 362 30 10010 8975 0.96. -20 362 3 ~ Average 20282 105 ~27523 100 31 324Average 27770 158 47 100 33 399 Stdev 16746 92 10798 7. 42 267Stdev 13709 1 (04 7 5 2 TT529
 
 RETESTING PROCEOURE
 
 
 
 
 
    
 
 
 
 
 
 
530 TESTINGeeu =eeFIGURE 21-12 Comparison of Swiss franc trend-following techniques. i FIGURE 21-13 Impact of i ‘i pact of increased transaction costs on performance.| & Comparison of Swiss franc tests i Impact of commissi1986-1995 198615IMM Swiss rang, 1986-199560,000 60,00050,000 |- 40,000 |-i [ ,000! 40,000 |— F@ 30,000 4 zB senor| 3B 20,000 | :| B 20,000|- © -10,000' 40,000 | [L 20,000 ~°1 1 1 |10,000 “ , ‘ ,00 4 abe a ho 0,000 5 + = L: ; 200Calculation period Moving average calculation period #0“ae Moving average, $25 commission -O~ Exponential smoothing, $25 commission + Moving average, $25 commi. ission _ ~O- Moving average, $100 commi. ssion
 
3, Results should be scored in some way to include profitability and risk. Other mea-surements can be used, but stability is of great importance.+. 4 Results, or scores, should be averaged to avoid spikes or unusual distortions.5, Execution costs, including liquidity factors, must be included in the manner appro-priate to each system. A trend-following system with entries and exits in the direction
The worst re ion rs sults are only usefal if there is a net profit from taking the opposite posi-aawions and clippage we the <t fess is greater than the transaction costs, including com-cao team be teed peaanet ™ e worst score has a near-zero return before transactioncosts itcan be used profi ly. Unprofitable results of a fast trend mean that a long or sh. at point lasts only a few days and does not indicate a sustained trend.
 
Peete vond should have greater slippage than a countertrend cry method. pese th i ;6. Make mistakes in the direction of being conservative. If two systems have compara- ¢ following rules:MaKe urns, select the one with the lower risk. If they have ‘similar returns and risk, 1. Trade a system of two movi7 7 in achoose the one with fewer trades. D2, and the short-term, see ee ee by the best score- oni = 2. i .Choosing the speed of the trading system is often the most critical decision in the per-. ‘When prices cross the long-term moving average, a long position becomes eligible.
; 3 A long position imn method to trade faster are most desirable, position is entered when the short-term movi 'nal, Positions may be entered ona L-day delay. average signals a short sig-
3. Short positions mayyy be closed out when prices cross ab:ove the long- line;they must be closed out when a long position is entered. ongeterm trendline:
4. Shi iti iort positions and exits from long positions are the opposite of rules 2 and 3.
formance profile. Parameters that cause a Urbecause the higher number of trades adds confidence to the validity of the test results. Youmight also find that there is more consistency in the performance of a faster trading pro-gram compared with a slower one that generates the same net profits. To compensate forits advantages, faster trading is more susceptible to execution problems, so that perfor-mance will deteriorate quickly if slippage increases. Figure 21-13 shows the change in per-formance of all test intervals from 10 to 250 days for the moving average method whentransaction costs are increased from $25 to $100 per trade. Note that the shortest intervalturns from a small profit to a large loss and profits for intervals through 40 days are halved.Some areas of particularly poor performance, such as at 70 and 110 days, are magnified byal, there is very little impact on the performance of slower
Because the short- i
con eset he short ‘com signals are not good indicators of trends, they can be used as a
ee Maud nd entries, That a lows for better execution when trading larger positions or in. The choice of immediate exit or delayed exit is the trader's preference.
Once a position is h i
eld, there is greater risk waiting to exi
, , is! i
improved by better timing. aiting to exit, yet most exits would be
 
the increased costs. In gene
- parameter selections. RETESTING PROCEDURE
One optimizat i i
One opt imization is never the end of the testing process. If you have successfully finished
ification, tren you sh oe , and have retained the most recent data for out-of-sample ver-
iat, then you wuld retest the system including the out-of-sample data before begin-
oF widing data eee eeend fey additional data adds robustness to the system. The impact
A in the nature of i ciamount of data originally used in the testing. the marker during the new period and thein st i
ping i Pn steprforward resting, some analysts prefer to test a fixed amount of data, drop-
This does not mean that they test only 6 months, or 1 year of data aad
PROFITING FROM THE WORST RESULTS
Under the right conditions, the worst results of an optimization or test sequence may hold
valuable information; it presents the worst-case scenario that should be carefully reviewed
. for system problems, in particular with regard to tisk control. If thé area of poor results isbroad and continuous, it may also show a way tO improve entry and exit timing. Figure 2}-Jag and b shows the continuous results of 1- and 2-variable tests. Both identify faster trading areas as having maximum loss and ‘slower trends as having best performance.
 
 
 id
 
 
TESTING532
FIGURE 21-14 Dual use of test map. (a) Single-variable test. (6) Two-variable test.(Dz) Best score
       
Profit oss
Transactioncosts
(D,) Worst score
(a)
Fast
Fotlow trend
 
 
(b) ~
 
decide that data before a particular date is no longer relevant.In some cases, there are structural changes that support that approach, such as the forma-tion of a unified European currency, which would put controls on the variance in exchangerates between the participating countries even before the inception date. You might alsoconsider a new market with low liquidity during its earlier years as different from moreactive, current market conditions. Retesting remains an important way to adjust the system.to characteristics of new data. You might decide that cetesting is necessary when time rep-resenting 5% to 10% of the test period has elapsed. Using the 2-variable test as an example,a shift may be expected in the area of best performance as shown in Figure 21-15. 1sBecause only a small amount of data was added, the size of the shift in Figure 21-should be small. Ifa large shift actually occurs, then two possibilities should be considered:
retest every month. You may
1. The data was unusually volatile and introduced patterns not previously seen in the”
data. In this case, the shift in parameters is justified. . . -2. ‘The data period for the original tests was short and did not include enough patternsto make the parameter choice robust. ‘With a small amount of test data, there is a
 
 
 
 
COMPREHENSIVE $1 UDIES 533
 
— sees
FIGURE 21-15 Consecutive tests.
 
Best performancetest no. 2
Best performancetest no. 1
2 Represents shift inbest performance areas
 
 
greater likelihood that a faster trading model will be selected. As the data shifts andthat faster model is no longer good, the parameters may jump to a very slow model.‘This erratic pattern of shifting from fast to slow is indicative of too little test data.
Optimizing after a Major Move
When your system fails to produce the right amount of profit from a major move, or whenthe stock market takes an uncomfortably large plunge, it is natural to question whether thesystem has the right parameters. Is the trend fast enough? Is the stop-loss too close? This isthe same process as retesting. If you have used enough data for your original tests, then itis very likely that the new tests won't change anything. If you make up new rules to fix thisone problem, you are overfitting and accomplish nothing. If you have doubts about thetest period, use more data rather than less. The new patterns will only give you a more real-istic idea of the market risk than you had before. A major move that contains a price shockis a special situation discussed later in this chapter, as well as Chapters 22 and 23.
COMPREHENSIVE STUDIES -
A comprehensive study points out the robustness of a trading strategy. Wherra system per-forms well in many markets using similar calculation intervals, it is fair to assume that themethod is sound. If a technique works in the Swiss franc but not the Deurschemark, or inEuroyen but not Eurodoliars, you must be able to understand the significant differences
© Readers with an interest in performing optimizations such as these should read the basic material in Chapter 4and the additional systems in Chapter 5. 
SMSC
 
  
 
534 TESTINGbefore accepting the method. Markets have individual characteristics that might justify dif-ferences in performance; however, a strategy that can generate profits in a broad range ofmarkets is robust. In addition to the studies that follow, Chapter 5 (“Trend Systems”)includes a section “Comparison of Major Trend Systems,” which will be helpful to anyoneinterested in viewing the results of different systems side by side.
Colby and Meyers, in The Encyclopedia of Technical Market Indicators (Dow Jones.Irwin, 1988), have given a very useful comparison of major market indicators simply bypresenting all of their test results in a common form. Any one of the methods may haveproved good or bad during the past 10 or 20 years, but taken as a whole, the consistencyofvarious techniques adds to the robustness of that approach. For example, if one volume-based indicator was profitable,