average, with more consistent profits,fewer trades, greater reliability, and even smaller drawdowns.
Testing the breakout strategy for 1986 to 1996 (Table 21-9), we see a different pat-tern than the update of the moving average method. The number of trades havedropped, the maximum drawdown has increased, while only one market showed all
—_— — ————TABLE 21-9 Results of an N-Day Breakout (1986-1996), $50 Transaction Costs*
 
Best Net Max Trades
Market Dey Pit Loss Totat Prof % Prof RobustnessCorn SS 5,275 3,375 23 10 43 9/20Cotton 35 54,605 -10,700 34 16 47 19/20Soybeans 95 13,225 6,150 12 7 58 4120Silvert 8S -2,685 13,250 13 6 46 0/20Coppert 40 ‘A400 6,887 23 9 39 10/20Gold 95 1310 -14,430 i4 5 36 - 120‘Swiss franc 40 67,862 -16,600 31 16 52 19/20German mark 40 42,775, 9,137 Bh 16 52 17720Japanese yen 25 76,062 -17,062 44 20 45 17/20British pound 35 62,012 13,625 36 1s 42 18/20S&P 500 85 15,025 35,750 16 7 44 3/20NYSE Index* 55 950 =21,200 7 7 4l 2/20T-Bills 30 10,875 3,825 4t 7 4! 15/20T-Notes 2s 42,600 -9,087 53 27 SI 2020TBonds - 70 41,302 20,323 19 10 53 17/720Teens Ee*Breakout tested from 5 to 100 days in steps of 5. -
"Test data began in 1987 or 1988. 
tena
 
 
542 TESTING
losses. Of the 300 total tests, 57% were profitable, greater than either the moving aver-age or crossover methods.
Modified Three-Crossover Model
The use of one or more slow-moving averages may result in a buy or sell signal at a timewhen the prices are actually moving opposite to the position that is about to be entered.This may happen when:
1. The rules consider the crossover of the moving average, rather than a penetrationof the price.
2. The change in a simple moving average value based on the new price is less than thechange in the oldest price that was dropped.
For example, if a long signal occurs and the oldest price showed a decline of 50 pointswhile today’s price declined 40 points, the new moving average value will rise by the difference, +10, divided by the number of days in the moving average. This may also occurusing exponential smoothing under the special conditions
Ey-1— Pp > Pry Ey
By using a third, faster-moving average, the slope can be used as a confirmation ofdirection to avoid entry into a trade that ig going the wrong way. This filter can be added toany moving average or multiple moving average system with the following rule:
Do not enter a new long (or short) position unless the slope of the confirmingmoving average (the change in the moving average value from the prior day totoday) was up (or down).
The speed of this third, confirming moving average only makes sense if it is equal to orfaster than the faster of the trends used in the Crossover System.
Test Results
Fortunately, the results of both the Crossover System as well as the Modified Three-Crossover System were available for the same commodities and the same years. Becausethe crossovers used in the latter model are exactly those of the first system, te compari-son will show whether the confirming feature improved overall results. From the 22 mar-kets tested, plywood was removed because its poor results on both systems tended todistort the comparison. The important statistics are shown in Table 21-10.
The difference in using the Modified Three-Crossover System versus the simplerCrossover System are: —
Average change in profits -15.3%Average change in equity drop -15.5%Average change in percentage of profitable trades ~ 9%Average change in number of trades = 726.1%
A decline in profits equal to a decline in equity drop (or risk) is the same as using theoriginal Crossover System with a 15.4% smaller investment. The percentage of profitabletrades increased negligibly, indicating that the confirmation filter did not eliminate morelosing trades as was expected, The last point, however, shows a larger decline in the totalnumber of trades, indicating that the profit per trade has increased. The new filterappears to catch the entries at a better point, and eliminate some trades with smallerProfits. This type of improvement means that there is more latitude for trading error,
 
COMPREHENSIVE S) UDIES 543
—_TABLE 21-10 Comparison of Systems 1970-1979
 
 
 
 
Crossovers Modified Three-Crossover ChangeNo, %  % Profitable No, % %Profitable % %
PiL Trades Drop ‘Trades PiL ‘Trades ‘Drop ‘Trades Pi TradesDeutschemark $ 96510 259 7.6 54 $97,698 2263.4 52 12-127Japanese yen 94575 1313.1 St 92,287 Jl 29 58 -24 -153Canadian dollar 71,940 1587.0 52 69,690 116 5.2 60 -21 -26.6British pound 80,262,113. 8.2. 46 69808 «81 83 53 13.0 -28.3‘Swiss franc 120,674 121 48 48 00.891 91 4.6 46 -164 -24.8Cocoa 408,262 353-24 46 282,453 2917.6 43 308 -17.6Coffee 83,586 3323.9 39 56,338 231 | 5.8 35 -32.6 -30.4Sugar 348,833 4493.8 36 331,526 322 4.1 41 5.0 -283Cotton 378,440 697 28 47 282,460 4833.3 4l 254-307Silver 96,995 1098 13.9 35 89,165 990 15.7 37 8.1‘Copper 218,790 ° 57k 27 43 217,689 408 29 43 -5  -28.5Soybeans 386,137 48348 a7 308,233 317) 5.7 48 20.2 -34.4Soybean meal 165,294 4295.3 43 135.643 316 5.9 39 -17.9 -263Wheat 151,871 305 27 43 90,093 207 6.9 44 40.7 -32.1Pork bellies 78,207 , 312 275 44 76,363 246 «14.2 43 24-22Soybean oil 125,848 488 5.2 39 89,834 346 9.3 40 -28.6 ~29.1Hogs 88,097 8539.2 39 94,634 647° 7.9 4l 74-242Cattle 162,280 1186 4.8 41 155135 875 3.3 40 44 -262GNMAs 76291 1974.4 52 $6217 133° 6.5 47 -263 325Tbills 28210 388 27.0 38 20,436 «2843.4 37 -276 -268Gold 131,325 275 1.6 53 98,030 158 | 2.2 57 254 425Average 73 45 6.1 45 -15.3 -26.
a
such as slippage. Although the overall profile of the Modified Three-Crossover is notmuch better than the Crossover System, it would be the preferred choice based on thisinformation.
4-9-18 Crossover Model Results
This model, using the same rules as the Modified Three-Crossover System, was well knownbefore Hochheimer’s studies. It can be assumed that the selection of 4, 9, and 18 days wasa conscious effort to be slightly ahead of the 5, 10, and 20 days frequently used in movingaverage systems. It is likely that the system was not developed by extensive testing, sinceall markets are traded with these same speeds.
‘As simple as it seems, there are a few very sound concepts in this approach:
1. Fach moving average is twice the speed of the prior, enhancing their independencein recognizing different trends. -
2. They are slightly faster than the conventional 5-, 10-, and 20-day moving averages
3. They are the same for all products, implying an attempt at robustness.
It would not be possible for either the profits or the equity drops to be better in thismodel than an optimized strategy, such as the Modified Three-Crossover Method; yet, outof 17 commodity markets simulated during the 1970 through 1976 period, only two lostmoney. That is a very impressive record. There should be a greater degree of confidence inthis performance than in an optimized program. 
 
S44 TESTING
 
Importance of Failed Tests ~
When the period from 1986 to 1996 is tested, as seen in Table 21-11, we see much lowerprofits, large losses in the index markets, and uniformly high risk. Net results are worsethan other methods tested. This certainly means that this technique is no longer good. Per-haps it means that a rigid set of trends cannot survive over the long term; the lengths of theunderlying trends have changed; they are no longer uniform because of changing marketparticipation; or there is more noise in the market, making it difficult to identify the trendina timely fashion.
‘These results should make us rethink the conclusions drawn from the previous tests,We have looked at the best days and the robustness of the technique. Because we are nolonger confident chat the best day of the past will be the best day of the future, we need todepend on the robustness of the system even more. If a large percentage of all tests areprofitable, then there is a much greater chance that any parameter selected will be prof-itable. Our goal is to find the most robust system.
To remove the dependence upon specific fixed parameters that might return unstableperformance over many years, it is necessary to generalize many of the features in a system.For example, instead of a fixed stop-loss at 10 points or $500, it will be necessary to allowthat stop to vary based on market volatility. Even the calculation period of the trend maychange according. to volatility, market noise, volume, or some other factars. This approachis discussed further in Chapter 17 (‘Adaptive Techniques”).
Test Criteria
The method used in these tests is not bad, but real market factors must be included or theresults are deceiving. Although the researchers may have been pleased with the outofsample profits, they were only paper profits. In real trading, the chances greatly favor losses.
Testing should closely approximate trading. If there is doubt about the costs, makethem larger. A system that can profit under testing penalties should make money when actu-ally traded. Even when using prepackaged computer optimization software, the slippage
 
     
 
 
TABLE 21-11 Results of a 4-9-18 Crossover (1986-1996), $50 Transaction Costs.
Net Mox —— es
Market PIL Loss Total Prof % ProfCorn A425 —12,087 14 38 33Cotton 28,110 +21,000 19 46 39Soybeans -19,087 33,987 139 sl. 37Silver* 20,740 21,640 94 28 30Copper* 6,862 10,062 — 86 28 33Gold 2,250 -12,910 ~ 130 43 33
Swiss franc 3,325 43,350 124 48 39—German mark 11,562 31,525, U9 47 39Japanese yen 86.412 17,075 104 4l 39British pound 45,512 27,362 122 44 36S&P 500 78,650 110,350 136 48 35NYSE Index* 31475 81075 98 36 WTBills, -7.025 14,400 125 40 32T-Notes 15,674 -11406 16 42 ~ 36T-Bonds 14,668 44,013 130 38 29
—Test daca began in 1587 or 1988.
 
 
COMPREHENSIVE STUDIES 545
can be added by making the commission costs high; both are per trade costs. If a mistake isto be made, do it by being too conservative; however, it is always best to be realistic
Getting the Big Picture through Testing
As mentioned throughout this chapter, the key to robustness is to test more data, moremarkets, more of everything, and then to observe the results looking for a common patternof overall consistency. There is no better way to tell if a market has trends than to test it fora variety of trending methods. A market that performs well using an interday breakout, butfails with a dual-trend crossover, is not a good candidate for trending profits in the future.A comprehensive look at 12 trading methods applied to 12 futures markets gives us achance to sce generalized performance.® The annualized percentage returns for a variety ofU.S. markets are shown for 12 well-defined technical methods in Table 21-12. These trad-ing systems are:CHL Channel breakout, using the closing price penetration of the N-day high andlowPAR Wilder's Parabolic System, in which the stop and reverse gets closer each dayDRM _ Directional Movement, a Wilder method that averages the ups and downs sep-arately .RNQ Range Quotient, basing a breakout on a ratio of current change to past changeDRP © Directional Parabolic, combining DRM and PAR systemsMIL . MII Price Channel, a breakout system using only the oldest and most recentprices“LSO  L-S-O Price Channel, using two parameters, including an interval of oldestprices, to decide a channel breakout -REF Reference Deviation, determining a volatility breakout based on a standard.deviation of past closing price changes
° Louis P Lukac, B, Wade Brorsen, and Scott H. Irwin, "How to test profitability of technical trading systems,”Futures (October 1987).
 
TABLE 21-12 Percentage Returns by System and Market, 1978-1984*
CHL PAR DRM RNQ DRP Mil LSO REF DMC DRI MAB ALX Average
 
Corn 224 38 298 62 301 433 10.7 46 379 -29 47 229 160Cocoa 10.0 -101.7 -121.9 -345.0 -112.1 -73.6 -256.9 -120.5 -72.9 -281.8 -219.2 ~35.2 -144.2Copper “154 29 46.1 -784 42-314 83.0 -66.2 -39.8 -94.3 -1184-168 493Cate “122 -284 -124 -725 -165 -348 44.9 -563 -114 -700 -58.9 -55 -353Pork bellies -308 220 68-1344 48 206 -I17.4 -145.3 -65 —127.6 -1120 -I21Lumber _ 38.7 436 ~O4 -192 —406 31.3 46.3 -184 366 -I9.1 47.1 24.6Soybeans 77-19. 258 57.9 ~10.0 133 -215 45.6 26 -10.7 -225 131Silver 605 544 02 -159 823-191 125 -724 342 -762 -180 55.9Sugar 103.4 462 61.2 421 634 726 -O8 -473 823 176 15.2 716
British pound -35 81 20.7 -361 303 19 -78 35 289 -10.2 -65.8 -39.8German mark 665 354 682 [88 780 63.3 192 -I78 463 246 66-500Tills 108.7 485 1327 40.1 225.5 189.9 221.8 239.8 1278 39.2 -120 329Average 29.7 24 126 69.1 27.6 23.1_ 26.2 -285 21.9 -51.0 -547 51 3.Calculation of annual percent returns assumes that 30% of the initial investment is used for intial margin, in which margin is estimated as 10%of contract value.
  
 
546 TESTING
 
 
DMC Dual Moving Average Crossover, holding a trend position when both movingaverages havé the same trend
‘
DRI Directional Indicator, a ratio of current price change to total past price change
MAB Moving Average with % Price Band, giving a signal when prices penetrate theband
ALX Alexander's Filter Rule, generating signals when prices reverse from a previousswing high or low by a percentage amount
To create the results in Table 21-12, each market was tested for 3 years and the bestparameters used to generate the rerurns for the next years, in a step-forward approach. Itis easy to see that some markets are not profitable for any strategy and that some strategiesare generally poor. Unless you were creating a strategy for a specific group of markets, youwould not want to trade a system that was not profitable in less than 50% of the markets,nor would you want to trade a market that failed in most of the trend strategies. For exam-ple, cattle lost in every system, and the RNQ, REF, and MAB systems were consistentlyunprofitable; therefore, none of those would be good candidates for trading. The mostconsistent systems were CHL, PAR, MII, and DMC, each posting 8 of 12 winning markets;yet, each has a noticeably different technique. T-bills present an interesting choice becausethe highest profit, 239.8%, occurred in one of the least reliable systems, REF; choosing DRPis likely to be a better choice.
Displaying the system results by year for all markets is another way to look at robust-ness. In Table 21-13, we see that RNQ and MAB net losses over all markets in all years,while CHL, MII, and DMC stand out as very consistent. The combined presentation ofresults across all markets and systems is a clear way of avoiding overfitting by looking at thebig picture. :
PRICE SHOCKS
Price shocks are large changes in price caused by unexpected, unpredictable events. Theimpact of price shocks on historic tests can change the results from profits to losses, andvary the risk from small to extremely large and, unfortunately, not enough thought is givento how these moves affect test results and future performance. A discussion of price shockscould easily fill an entire book, but the concepts that are important can be explained
 
TABLE 21-13 Percentage Returns by System and Year*
 
 
 
System 1978 1979 1980 1981 1982 1983CHL 6 479 B16 28 28.3 19.9PAR 18.5 16.4 545, 29.9 37 43.2DRM 29.3 27 92.2 31.6 22.7 78RNQ 57.2 $9.0 9.6 -152100 © 34.2 —135.5DRP 34.8 63.87 88.7 31.0 — 13.4 -20.0Mil 12.9 416 . B78 546 0-7 -30.7 3.1LSO 471 40 39d ~55.1 -55.1 37.6REF 13.5 23.3 53.2 85.2 20 — 109.6DMc 17.6 26.8 85.4 5.9 229 19DRI 60.7 -15.4 46.3 — 108.1 -li44 —108.3MAB 38.0 443 77 144 Hd 818ALX 28.3 = 38.6 82.6 34.4 8.2 -2.0Average 5.8 12.3 57.8 36.5 28.8 4g
**E qual percentage amounts of margin are assumed to be invested in each market
 
 
ANATOMY OF AN OPTIMIZATION 547
 
briefly. When it comes to actual trading, the difference between your expected results andactual performance (not attributed to slippage) is entirely dependent on the number ofprice shocks.
By its very nature, a price shock must be unexpected. However, not all events causeshocks. An assassination, abduction, or political coup is likely to be a surprise, while a cropfreeze can be anticipated as weather turns unusually cold. Some economic reports, such asa .5% increase in the Producer Price Index or jump in the balance of trade, will come as ashock, but a iow carryover supply of soybeans or a tightening of the money supply aftersteady economic growth can be anticipated. When a change is anticipated, the marketadjusts to the correct level before the news is announced; when it is wrong, prices react inproportion to how poorly it was anticipated.
The problem comes when you test historic data. We know at the time of a price shockthat we could not have anticipated the event; at best we have an even chance of being onthe right side of the price move. But the computer doesn’t know that, and the best resultsfrom a large series of tests often include profiting from the largest price shocks—a situa-tion that would be impossible in actual trading. For example, if there were 10 price shocksin 10 years of data, and your historic tests profited from 8 of those shocks, then you haveoverestimated your profits. Even worse, you have underestimated your risk by believingthat a price shock produced a profit and not a loss.
How. can you correct this problem? First, you can look at the pattern of profits andlosses from a series of tests. If a 20-day and 30-day moving average each showed a maxi-mum drawdown of $10,000, but the 25-day average only lost $5,000 you should assumethat the 25-day results could have had the same drawdown as the nearby tests. By somequirk of timing one test was able to avoid a loss while the normal test was not. It would berisky to assume such good fortune in trading.
If you have a record of major price shocks, such as a chart published by the exchangesshowing key events, you can verify how many of these events produced profits or losses inyour final system. If you have profited from more than 50% then you should reassess yourrisk based on losses rather than profits from some of those trades. Also check to see thatthere is an even distribution with regard to the size of the price shocks. You may haveselected parameters that took small losses on minor shocks and large profits on major ones.
If you do not treat price shocks as a serious risk, you may overleverage and undercap-italize your trading account. This is the most common cause of catastrophic loss. Any priceshock that causes a windfall profit could have, just as easily, produced a devastating loss.The example that follows, “Anatomy of an Optimization,” shows how expectations andreality are often quite different. Further examples can be found in Chapter 22 (“PracticalConsiderations”).
ANATOMY OF AN OPTIMIZATION’?
Using the Iraqi invasion of Kuwait on August 6, 1990, and the U.S. retaliation on January17, 1991, we optimized a simple moving average strategy for crude oil. The test periodbegan January 2, 1990, but 100 days were needed to initialize the calculations; therefore,the first trade was entered May 24, 1990 (see Tables 21-14, 21-15, and 21-16). In both Test1 and Test 2, the optimization selects moving averages that produced the highest profits.By accepting these choices, the trader would have held a short position before the invasionof Kuwait, and a long position before the U.S. retaliation. In reality, there would have beena large loss rather than an exceptional profit. When a price shock is an important part ofthe data being tested, the best system is the one that took the most profits out of that move,even at the cost of other trades. .
* perry Kaufman, “Price Shocks: Reevaluating Risk/Return Expectations,” Futures Industry Qunefuly 1995). 
 
548 TESTING
—TABLE 21.14 Test |: Optimizing Crude Oil, January 2, 1990,through August 3, 1990* '
 
 
 
 
 
through August 3, 19908Period Newer ROA MaxDD Tet5 2615 82.9 ~3155 WW10 6765 959.6 -705 51S 6975 989.4 =705 320 4975 705.7 =705 325 6975 989.4 108 330 4635 6575 705 335 3635 3134 ~1160 340 3215 276.0 ~1165 345 3255 187.6 ~1735 350 235 10.6 ~2415 355 115 2.4 ~3385 560 2625 156.7 ~1675 565 2785, 239.1 1165 s70 ~385 ~126 ~3055 775 1955 W127 ~1735 380 5040 100.0 ~5040 285 $040 100.0 ~5040 290 8040 100.0 5040 295 5040 100.0 ~5040 2100 5040 100.0 5040 2 _Trade Detail from TestingProfit! CumTrade Date BIS Price Loss PIL ~May 24,90 Buy 23.58. | May 25,90 Sell 23.55 55 55Jul 11,90 Buy 22.49 1035 980‘Aug3,90 Sell 28.54 5995 6975ee eSSFSsSs—* Testing stopped the day before the invasion of Kuwait. \
Test selects 15-day moving average. :Holding short on August 6, 1990, when Iraq invades Kuwaft.
In Test 2, although there were large profits from the first price shock, the trader wouldhave still held the wrong position when the second shock occurred. This method of test-ing hides the real trading risk, focuses on profits that