98 63 18/120Coppert 15x 25 17,337 15,087 8I 46 57 29/120Gold 1x25 27,990 8,900 93 43 46 78/120Swiss franc 7x55 79,600 13,912 54 20 37 109/120German mark 25 x 50 $2,387 10,750 44-19 43 103/120Japanese yen thx 20 121,362 14,112 13 53 47 110/120British pound 25x 45 61,612 —19,025 62 31 50 99/129S&P 500 29 x 23 194,750 83,650 199 126 63 8/120NYSE Indext 20 x 23 58,400 31,475 179 96 54 27/120T-Bills 19x 45 12,100 4,975 55 23 42 83/120T-Noces {1x 30 58,643 ~7,368 79 34 43 103/120T-Bonds W1 x 35- 45,305 14,543 67 28 42 88/120
re eR ee NS neta“Fast trend tested irom 3 to 25 in steps of 2; slow trend tested from |5 to 60 in steps of 5. 7Test data began in 1987 or 1988,COMPREHENSIVE S+ JDIES 541
 
 
TABLE 21-8 Results of Interday Closing Price Channel
 
 
 
 
Max. Trades
Best Net Run of
Day PIL Losses Total Profit % ProfCocoa 53 $147,913 $ ~6,248 110 60 55%Corn 49 39,533 ~5,048 18 50 42Sugar 40 296,027 ~20,758 133 57 43Cotton 62 206,575 ~6,870 89 48 54Silver 14 27.690 12,365 552 212 38Copper 27 151,671 -7,225 217 Ws 53Soybeans 5h 244,839 11,325 120 68 57Soybean meal 49 104,690 ~7,000 166 7 43Wheat 23 111,087 6,900 205 80 39Pork bellies $2 60,263 9,892 94 51 54Soybean oil 5! 8,646 ~3,622 109 48 44Plywood 49. 8,632 4,432 120 57 48Hogs 9 83,702 9,854 606 253 42
 
In the original tests shown in Table 21-8, the channel method has much lower profitsand proportionally greater risk than the crossovers; results tend to be more erratic as well.This method is clearly better than the single moving average, with more consistent profits,fewer trades, greater reliability, and even smaller drawdowns.
Testing the breakout strategy for 1986 to 1996 (Table 21-9), we see a different pat-tern than the update of the moving average method. The number of trades havedropped, the maximum drawdown has increased, while only one market showed all
 
TABLE 21-9 Results of an N-Day Breakout (1986~1996), $50 Transaction Costs*
 
Best Net Max trades
Market Day PIL Loss Total Prof % Prof RobustnessCorn 55 5,275 3,375 23 10 4B 9/20Cotton 35 54,605 10,700 34 16 47 19/20Soybeans 9S 13,225 6,150 12 7 58 4/20Silvert 85 ~2,685 13,250 13 6 46 0/20Copper! 40 9,400 6,887 23 9 39 40/20Gold 95 1,310 —14,430 14 5 36 - 1420Swiss frane 40 67,862 -16,600 31 16 52 19/20German mark 40 42,775 9,137 SN 16 $2 17/20Japanese yen 25 76,062 17,062 44 20 45 17/20British pound 35 62,012 ~13,625 36 15 42 18/20S&P 500 85 15,025 38,750 16 7 44 3/20NYSE Indext 55 950 21,200 7 7 4l 2/20T-Bills 30 10,875 3,825 4i (7 41 15/20T-Notes 2s 42,600 9,087 53 27 5t 20/20T-Bonds - 70 41,302 20,323 19 10 53 17/20ne“Breakout tested from 5 to 100 days in steps of 5. ; .
Test data began in 1987 or 1988,542 TESTING
 
losses. Of the 300 total tests, 57% were profitable, greater than either the moving aver-
age or crossover methods.\
Modified Three-Crossover Model
The use of one or more slow-moving averages may result in a buy or sell signal at a timewhen the prices are actually moving opposite to the position that is about to be entered,This may happen when:
1. The rules consider the crossover of the moving average, rather than a penetrationof the price.
2. The change ina simple moving average value based on the new price is less than thechange in the oldest price that was dropped.
For example, if a long signal occurs and the oldest price showed a decline of 50 pointswhile today’s price declined 40 points, the new moving average value will rise by the dif-ference, +10, divided by the number of days in the moving average. This may also occurusing exponential smoothing under the special conditions
By ~ Pe > Pry Bron
By using a third, faster-moving average, the slope can be used as a confirmation ofdirection to avoid entry into a trade that is going the wrong way. This filter can be added toany moving average or multiple moving average system with the following rule:
Do not enter a new long (or short) position unless the slope of the confirmingmoving average (the change in the moving average value from the prior day totoday) was up (or down), ,
The speed of this third, confirming moving average only makes sense if it is equal to orfaster than the faster of the trends used in the Crossover System.
Test Results
Fortunately, the results of both the Crossover System as well as the Modified Three-Crossover System were available for the same commodities and the same years. Becausethe crossovers used in the latter model are exactly those of the first system, tlie compari-son will show whether the confirming feature improved overall results. From the 22 mar-kets tested, plywood was removed because its poor results on both systems tended todistort the comparison. The important statistics are shown in Table 21-10,
The difference in using the Modified Three-Crossover System versus the simplerCrossover System are: —
Average change in profits ~ 15.3%Average change in equity drop -15,5%Average change in percentage of profitable trades 9%Average change in number of trades U7 726.1%
A decline in profits equal to a decline in equity drop (or risk) is the same as using theoriginal Crossover System with a 15.4% smaller investment, The percentage of profitabletrades increased negligibly, indicating that the confirmation filter did not eliminate morelosing trades as was expected, The last point, however, shows a larger decline in the totalnumber of trades, indicating that the profit per trade has increased. The new filterappears to catch the entries at a better point, and eliminate some trades with smallerprofits. This type of improvement means that there is more latitude for trading error,COMPREHENSIVE S) UDIES 543ee
ATABLE 21-10 Comparison of Systems 1970-1979
 
 
 
Crossovers Modified Three-Crossover ChangeNo, %  % Profitable No. % Profitable % %
PIL Trades Drop ‘Trades PIL Trades Drop Trades PiL = TradesDeutschemark $ 96,510 259 7.6 54 $ 97,698 226 34 $2 1.20 -127Japanese yen 94,575 131 3.4 51 92,287 th} 29 58 ~24 -153Canadian dollar 71,940 ° 158 7.0 52 69,690 116 5.2 60 ~3.1 +266British pound 80,262 113 8.2 46 69,808 Bi 8.3 53 -13.0 -28.3Swiss franc 120,674 = 121 48 48 100,89! WW 46 46 -16.4  -24.8Cocoa 408,262 353 24 46 282,453 291 76 43 -30.8 -17.6Coffee 83,586 332 39 39 $6,338 231 5.8 35 32,6 -30.4Sugar 348,833 4493.8 36 331,526 322 4.1 4! -5.0 -283Cotton 378,440 697 28 47 282,460 483 3.3 41 25.4 -30.7Silver 96,995 1098 13,9 35 89,165 990 15,7 37 “81 -9.8Copper 218,790 ° 571 27 43 217,689 408 29 43 ~5  -28.5Soybeans 386,137 483 4.8 47 308,233 317 57 48 20.2 -34.4Soybean meal 165,294 ° 429 5.3 43 135,643 316 5.9 “39 -17.9 -26.3Wheat 151,871 3059 2,7 43 90,093 207 69 44 40,7 -32!Pork bellies 78,207 . 312 27.5 44 76,363 246 14.2 43 24-212Soybean oil 125,848 488 5.2 39 89,834 346 9.3 40 28.6 -29.1Hogs 88,097 853 9.2 39 94,634 647 79 41 74° -24.2Cattle 162,280 1186 48 4! 155,135 875 3.3 40 44 -26.2GNMAs 76,29' 197 4.4 52 56.217 133 65 47 ~26.3 -325bills 28,210 388 27.0 38 20,436 = 284 3.4 37 ~27.6 -26.8Gold 131,325 275 16 53 98,030 158 | 22 57 -25.4 -42,5Average 7.3 45 6! 45 -15.3 -26.1
 
such as slippage. Although the overall profile of the Modified Three-Crossover is notmuch better than the Crossover System, it would be the preferred choice based on thisinformation.
4-9-18 Crossover Model Results
This model, using the same rules as the Modified Three-Crossover System, was well knownbefore Hochheimer's studies. It can be assumed that the selection of 4, 9, and 18 days wasa conscious effort to be slightly ahead of the 5, 10, and 20 days frequently used in movingaverage systems. Ir is likely that the system was not developed by extensive testing, sinceall markets are traded with these same speeds.
. As simple as it seems, there are a few very sound concepts in this approach:
1. Each moving average is twice the speed of the prior, enhancing their independencein recognizing different trends. -
2. They are slightly faster than the conventional 5-, 10-, and 20-day moving averages.
3. They are the same for all products, implying an attempt at robustness.
It would not be possible for either the profits or the equity drops to be better in thismodel than an optimized strategy, such as the Modified Three-Crossover Method; yet, outof 17 commodity markets simulated during the 1970 through 1976 period, only two lostmoney. That is a very impressive record. There should be a greater dégree of confidence inthis performance than in an optimized program.544 TESTING
 
Importance of Failed Tests ~
When the period from 1986 to 1996 is tested, as seen in Table 21-11, we see much lowerprofits, large losses in the index markets, and uniformly high risk. Net results are worsethan other methods tested. This certainly means that this technique is no longer good, Per-haps it means that a rigid set of trends cannot survive over the long term; the lengths of theunderlying trends have changed; they are no longer uniform because of changing marketparticipation, or there is more noise in the market, making it difficult to identify the wendin a timely fashion.
These results should make us rethink the conclusions drawn from the previous tests.We have looked at the best days and the robustness of the technique, Because we are nolonger confident that the best day of the past will be the best day of the future, we need todepend on the robustness of the system even more. If a large percentage of all tests areprofitable, then there is a much greater chance that any parameter selected will be prof-itable. Our goal is to find the most robust system.
To remove the dependence upon specific fixed parameters that might return unstableperformance over many years, it is necessary to generalize many of the features in a system,For example, instead of a fixed stop-loss at 10 points or $500, it will be necessary to allowthat stop to vary based on market volatility. Even the calculation period of the trend maychange according to volatility, market noise, volume, or some other factars. This approachis discussed further in Chapter 17 (‘Adaptive Techniques”).
Test Criteria
The method used in these tests is not bad, but real market factors must be included or theresults are deceiving. Although the researchers may have been pleased with the out-ofsample profits, they were only paper profits. In real trading, the chances greatly favor losses.
Testing should closely approximate trading. If there is doubt about the costs, makethem larger. A system that can profit under testing penalties should make money when actu-ally traded. Even when using prepackaged computer optimization software, the slippage
 
TABLE 21-11 Results of a 4-9-18 Crossover (1986-1996), $50 Transaction Costs =.
 
 
Net Max odes
" Market PIL Loss Tatal Prof % ProfCorn 425 —12,087 14 38 33Cotton 28,110 21,000 Wg 46 39Soybeans —19,087 33,987 139 sho 37Silver* 20,740 =21,640 94 28 30Copper* 6,862 10,062 - 86 28 33Gold 2,250 -12910  ~ 130 43 33
Swiss franc 3,325 43,350 124 48 39 —German mark 11,562 31,525, JH9 47 39Japanese yen 86.412 -17,075 104 4 39British pound 45,512 27,362 12 44 36S&P 500 -76,650 ~110,350 136 48 35NYSE Index -31,475 -51,075 98 36 37TBills ~7,025 14,400 125 40 32T-Notes 15,674 “11,406 116 42 - 36T-Bonds -14,668 44,013 130 38 29
eee renee eR AN NE*Test daca began in 1987 or 1988.COMPREHENSIVE STUDIES 545
 
can be added by making the commission costs high; both are per trade costs. If a mistake isto be made, do it by being too conservative; however, it is always best to be realistic.
Getting the Big Picture through Testing
As mentioned throughout this chapter, the key to robustness is to test more data, moremarkets, more of everything, and then to observe the results looking for a common patternof overall consistency, There is no better way to tell ifa market has trends than to test it fora variety of trending methods. A market that performs well using an interday breakout, butfails with a dual-trend crossover, is not a good candidate for trending profits in the future.A comprehensive look at 12 trading methods applied to 12 futures markets gives us achance to see generalized performance.’ The annualized percentage returns for a variety ofU.S. markets are shown for 12 well-defined technical methods in Table 21-12, These trad-ing systems are:CHL Channel breakout, using the closing price penetration of the N-day high andlowPAR Wilder's Parabolic System, in which the stop and reverse gets closer each dayDRM _ Directional Movement, a Wilder method that averages the ups and downs sep-arately .RNQ__ Range Quotient, basing a breakout on a ratio of current change to past changeDRP - Directional Parabolic, combining DRM and PAR systems‘MII: MI Price Channel, a breakout system using only the oldest and most recentprices -sO L-S-O Price Channel, using two parameters, including an interval of oldest. Prices, to decide a channel breakout .REF Reference Deviation, determining a volatility breakout based on a standard-deviation of past closing price changes
* Louis P Lukac, B, Wade Brorsen, and Scott H. Irwin, “How to test profitability of technical trading systems,”Futures (October 1987). .
 
TABLE 21-12 Percentage Returns by System and Market, 1978-1984*CHL PAR DRM RNQ DRP Mil LSO REF DMC DRI MAB ALX Average
 
 
Com 224 38 298 -62 30.1 433 107 46 379 -29 -+47 229 160Cocoa 10.0 -101.7 -121,9 -345.0 ~1}2,] -73.6 -256.9 -120.5 -72,9 ~281.8 -219.2 -352 1442Copper 154 2.9 46.1 -784 42-314 -B3.0 ~66.2 -39.8 -94.3 -1184-168 493Cate ~122 -284 -124 -725 -165 ~348 -449 -563 -11.4 -70.0 -589 -55 -353Pork bellies 30.8 220 ~6.8 -1344 48 20.6 -117.4 -145.3 -65 ~127.6 -1120-121 -53.8Lumber _ 38.7 43.6 -04 -19.2 -40.6 31.3 -463 -184 366 -I9) -47.] 246 -86Soybeans 77-191 25.8 -57.9 -10.0 13,3 -215 -456 26 -107 -225 131 ~104Silver 605 54.4 0.2 ~15.9 823-191 125 -724 342 -762 -180 559 8.2Sugar 103.4 462 61.2 -42.1 634 726 -O08 473 823 176 152 716 369
British pound = ~3.5 81 20.7 -36.! 303 19 78 3.5 25.9 -10.2 ~658~-398 -6.1German mark 665 35.4 682 188 780 63.3 19.2 -I7,8 463 246 66-500 299Tills 108.7 485 132.7 40.1 225.5 189.9 221.8 2398 127.8 39.2 -120 329 1096
Average 297° 24 12.6 -69.1 27.6 23.1 -26.2 -285 21.9 -510 -547 51 -89a*Calculation of annual percent returns assumes that 30% of the initial investment is used for initial margin, in which margin is estimated as 10%of contract value.546 TESTING
 
DMC Dual Moving Average Crossover, holding a trend position when both movingaverages have the same trend
1DRI Directional Indicator, a ratio of current price change to total past price change
MAB Moving Average with % Price Band, giving a signal when prices penetrate theband
ALX  Alexander’s Filter Rule, generating signals when prices reverse from a previousswing high or low by a percentage amount
To create the results in Table 21-12, each market was tested for 3 years and the bestparameters used to generate the returns for the next years, in a step-forward approach. Itis easy to see that some markets are not profitable for any strategy and that some strategiesare generally poor. Unless you were creating a strategy for a specific group of markets, youwould not want to trade a system that was not profitable in less than 50% of the markets,nor would you want to trade a market that failed in most of the trend strategies. For exam-ple, cattle lost in every system, and the RNQ, REF, and MAB systems were consistentlyunprofitable; therefore, none of those would be good candidates for trading. The mostconsistent systems were CHL, PAR, MII, and DMC, each posting 8 of 12 winning markets;yet, each has a noticeably different technique. T-bills present an interesting choice becausethe highest profit, 239.8%, occurred in one of the least reliable systems, REF; choosing DRPis likely to be a better choice.
Displaying the system results by year for all markets is another way to look at robust-ness, In Table 21-13, we see that RNQ and MAB net losses over all markets in all years,while CHL, MII, and DMC stand out as very consistent. The combined presentation ofresults across all markets and systems is a clear way of avoiding overfitting by looking at thebig picture. ‘
PRICE SHOCKS
Price shocks are large changes in price caused by unexpected, unpredictable events. Theimpact of price shocks on historic tests can change the results from profits to losses, andvary the risk from small to extremely large and, unfortunately, not enough thought is givento how these moves affect test results and future performance. A discussion of price shockscould easily fill an entire book, but the concepts that are important can be explained
 
TABLE 21-13 Percentage Returns by System and Year*
 
 
System 1978 1979 1980 1981 1982 1983 1984CHL 61 479 816° 21.8 28.3 19.9 28.0PAR 18,5 16.4 54.5 29.9 317 43.2 19.4DRM 29,3 217 92.2 31.6 22.7 78 -7.3RNQ -57.2 -69.0 9.6 -15210 00+ 312 135.5 ~100.!Dep 34.8 63,8- 88,7 31,0 — -13.4 20,0 38.5Mil 129 41.6 . 378 546 47 -307 3,1 7)LsSO 71 +40 394 ~55.4 ~55.1 -37.6 -59,5REF —$3.5 233 53.2 85.2 2.0 109.6 ~69.2DMC 17,6 26.8 85.4 59 22.9 19 -16DRI -60,7 154 46.3 ~108.) -W44 105.3 -34.0MAB ~38,0 443 77 -1144 WA -81.8 45.7ALX 28.3 = 33.6 82.6 34,4 -8.2 ~2.0 -144Average 5.8 12.3 57.8 -36.5 28.8 AND -23.1
a EN ES*Equal percentage amounts of margin are assumed to be invested in each market.ANATOMY OF AN OPTIMIZATION 547
 
briefly. When it comes to actual trading, the difference between your expected results andactual performance (not attributed to slippage) is entirely dependent on the number ofprice shocks.
By its very nature, a price shock must be unexpected. However, not all events causeshocks. An assassination, abduction, or political coup is likely to be a surprise, while a cropfreeze can be anticipated as weather turns unusually cold, Some economic reports, such asa ,5% increase in the Producer Price Index or jump in the balance of trade, will come as ashock, but a low carryover supply of soybeans or a tightening of the money supply aftersteady economic growth can be anticipated. When a change is anticipated, the marketadjusts to the correct level before the news is announced; when it is wrong, prices react inproportion to how poorly it was anticipated.
The problem comes when you test historic data, We know at the time of a price shockthat we could not have anticipated the event; at best we have an even chance of being onthe right side of the price move. But the computer doesn’t know that, and the best resultsfrom a large series of tests often include profiting from the largest price shocks—a situa-tion that would be impossible in actual trading. For example, if there were 10 price shocksin 10 years of data, and your historic tests profited from 8 of those shocks, then you haveoverestimated your profits, Even worse, you have underestimated your risk by believingthat a price shock produced a profit and not a loss.
How.can you correct this problem? First, you can look at the pattern of profits andlosses from a series of tests. If a 20-day and 30-day moving average each showed a maxi-mum drawdown of $10,000, but the 25-day average only lost $5,000 you should assumethat the 25-day results could_have had the same drawdown as the nearby tests. By somequirk of timing one test was able to avoid a loss while the normal test was not. It would berisky to assume such good fortune in trading.
If you have a record of major price shocks, such as a chart published by the exchangesshowing key events, you can verify how many of these events produced profits or losses inyour final system. If you have profited from more than 50% then you should reassess yourrisk based on losses rather than profits from