expected in the area of best performance as shown in Figure 21-15.Because only a small amount of data was added, the size of the shift in Figure 21-15should be small. Ifa large shift actually occurs, then two possibilities should be considered:
1. The data was unusually volatile and introduced patterns not previously seen in thedata. In this case, the shift in parameters is justified. +
2. The data period for the origina! tests was short and did not include enough patternsto make the parameter choice robust. With a smal! amount of test data, there is aCOMPREHENSIVE 81 UDIES 533
 
————.FIGURE 21-15 Consecutive tests.
    
Best performancetest no. 2
 
Best performancetest no. 1 —-_
  
 
2 Represents shift in
dest performance areas
 
 
greater likelihood that a faster trading model will be selected. As the data shifts andthat faster model is no longer good, the parameters may jump to a very slow model.This erratic pattern of shifting from fast to slow is indicative of too little test data.
Optimizing after a Major Move
When your system fails to produce the right amount of profit from a major move, or whenthe stock market takes an uncomfortably large plunge, it is natural to question whether thesystem has the right parameters. Is the trend fast enough? Is the stop-loss too close? This isthe same process as retesting. If you have used enough data for your original tests, then itis very likely that the new tests won’t change anything. If you make up new rules to fix thisone problem, you are overfitting and accomplish nothing. If you have doubts about thetest period, use more data rather than less. The new patterns will only give you a more real-istic idea of the market risk than you had before. A major move that contains a price shockis a special situation discussed later in this chapter, as well as Chapters 22 and 23.
COMPREHENSIVE STUDIES‘ -
A comprehensive study points out the robustness of a trading strategy. When-a system per-forms well in many markets using similar calculation intervals, it is fair to assume that themethod is sound. If a technique works in the Swiss franc but not the Deutschemark, or inEuroyen but not Eurodollars, you must be able to understand the significant differences© Readers with an interest in performing optimizations such as these should read the basic material'in Chapter 4and the additional systems in Chapter 5.534 TESTING
 
before accepting the method. Markets have individual characteristics that might justify dif.ferences in performance; however, a strategy that can generate profits in a broad range ofmarkets is robust. In addition to the studies that follow, Chapter 5 (“Trend Systems”)includes a section “Comparison of Major Trend Systems,” which will be helpful to anyoneinterested in viewing the results of different systems side by side.
Colby and Meyers, in The Encyclopedia of Technical Market Indicators (Dow Jones-Irwin, 1988), have given a very useful comparison of major market indicators simply bypresenting all of their test results in a common form, Any one of the methods may haveproved good or bad during the past 10 or 20 years, but taken as a whole, the consistencyof various techniques adds to the robustness of that approach. For example, if one volume-based indicator was profitable, while four others generated consistent losses, you wouldneed to understand why that one method worked before it could be used; the evidence ofthe four poor indicators argues that the use of volume in this way is not robust.
There are three previous, well-known comprehensive studies of moving average sys-tems by Maxwell, Davis and Thiel, and Hochheimer. In addition, optimizations of Wilder’sDirectional Movement and RS will be found in the sections that discuss those techniques.Although dated, these studies are very similar in structure to the way anyone would beginto test a trading idea; they emphasize different features of trading that were important tothe authors and give us useful insight. By virtue of hindsight, we can see that along withcompleteness in one area often comes a deficiency in another. Davis and Thiel analyze thegreatest variety of markets, covering virtually all of the U.S. crops as well as cattle, eggs, andthe soybean complex. They include about 5 years of data and use relatively fast movingaverages (up to 10 days); they introduce variations in lead-oriented plotting and in testingnonconsecutive days; the data used are only the closing prices. The results are clearly pre-sented in both detail and summary, and yield generally good returns.
The authors R.E. Davis and C.C. Thiel, Jr., present excellent credentials and experiencein systems testing. Their study of moving averages uses combinations of simple buy andsell signals, leading plots, and skips in the selection of sequential prices used (e.g., a skipof two uses every other price). They test a total of 100 combinations of the three factors:
A skip from 1 day (none) to 5 days (1 week)An average of 5, 6, 7, 8, or 10 days
A leap of 0 or 2 days s
Markets tested were soybeans, bellies, cattle, cocoa, copper, corn, eggs, soybean meal,oats, soybean oil, potatoes, rye, sugar, and wheat.
Maxwell's study is extremely comprehensive but limited by its application to only porkbellies. His idea was to apply combinations of features and test the results. The first featurewas the choice of trend and included the possibilities of a simple mean, or a moving aver-age of 3, 5, or 10 days, of either a conventional, average-modified, or weighted type. Thesecond feature was the delay factor, used to improve timing of both entry or exit. Some ofthe possibilities were: (1) act without delay; (2) act if the signal condition persists for oneadditional day; (3) enter if the signal condition persists for 2 additional days, but liquidatewithout delay; and others. Combinations of two types-of moving averages and a delay fac-tor were tested, with and without fixed or moving stops. With 10 types of averages, 6 delayfactors, and different stops, Maxwell has a lot of combinations to examine.
The study is then expanded to 3-factor systems with a list of 18 combinations of rulesto generate 324 systems, of which the results of 285 are recorded, with 49% profitable and51% losers. The largest loss was generated by a system with the rules:
1. Enter a new position when both the weighted 3-day and weighted 10-day movingaverages cross the price-mean, as long as the signal condition persists for 2 addi-tional days (buy if averages cross moving down, sell if up).COMPREHENSIVE > 1 UDIES 535
 
2. Liquidate positions when the 3-day average reverses its direction through the price-mean as long as it continues for 1 additional day.3. No fixed stops are used.
The best profits in the 3-factor system required both a 5-day average-modified and a10-day weighted average to move across the price-mean, provided the 5-day average laggedbehind the 10-day. The current position was liquidated when the shorter-term averagecrossed the longer term. No fixed stops were used.
Maxwell’s study represents a great amount of work and some simple and sound phi-losophy, but does not cover an adequate sampling of data to justify most of his conclu-sions. Conclusions were drawn based on a selected 50-day test period using May 72 porkbellies, leaving many questions regarding the success of this system or any other systemwhen applied to this short test interval. Maxwell does test four other selected 50-day peri-ods, but the reader cannot know how these periods were chosen. Considering the effort inoutlining a testing program and establishing rules, the extent of the system testing is dis-appointing.
Comparing Methods of Calculation
Hochheimer’ performs rwo interesting studies: a comparison of three types of moving aver-age calculations and a test of the use of channels and crossovers. In the first analysis,Hochheimer compares simple, exponentially smoothed, and linearly weighted moving aver-ages, tested on a good sampling of markets (without financials and currencies, which did notexist during the 1970 through (1976 test period). The simple moving average and exponen-tially smoothed calculations are covered in Chapter 4 (“Trend Calculations”), and the fin-early weighted refers to step-weighting, in which integer values are used and each successiveday is incremented by one, the most recent day having the highest weighting factor.The rules that apply regardless of which calculation was performed, were:
1. The trend calculation used the closing price only.
2. A buy signal occurred on an upward penetration of the moving average by the clos-ing price; a se// signal occurred under the opposite conditions.
3. The model is always in the market. -
4. A trade cannot be executed when the day’s high and low prices are the same. It isassumed to be a locked-limit day.
The test covered 7 years of data from 1970 through 1976 and moving average days (orequivalent smoothing constant converted as 2/(days + 1)), which appear to range from 2to 70. The results shown in Table 21-2 are interesting and logical, considering the rules.The longer trends (using more days) were consistently better than shorter ones, regardlessof-which technique was used. This would imply that the longer trends are more reliable.With a few exceptions, the best selection of days ranged from 40 to 70.
Amore careful look at the results proves interesting. During the 7 years of test data, soy-beans had 728 trades using a 55-day simple moving average. That is more than 100 tradeseach year, one every 2 or 3 days. Hogs had 1,093 trades during the same period, and othercommodities were similarly high. That seems to be an unusually large number of trades forthe time interval used, indicating that there must have been many false trends that resultedin small losses. This is confirmed by the very low percentage of profitable trades, only 21%for soybeans. Nothing is indicated about transaction costs, and 728 trades costing $50 pertrade in commissions and slippage would generate $36,400 in costs, about 15% of the Net7 Frank Hochheimer, “Moving Averages,” and “Channels and Crossovers,” in Tecbnical Analysis in Commodities,Perry J. Kaufman (ed.) (John Wiley & Sons, New York, NY, 1980).536 TESTING
 
nLTABLE 21-2 Results of a Simple Moving Average Model
 
 
Max. Trades
Best Best Net Run of
Day Range PIL Losses Total Profit % ProfCocoa 54 53-59 $87,957 $-14,155 600 157 26Corn 43 43-46 24,646 6,537 565 126 22Sugar 60 55-60 270,402 ~15,563 492 99 20Cotton 57 52-57 68,685 ~14,330 64) 121 19Silver 19 _ 42,920 ~15,285 1,393 429 31Copper 59 54-59 165,143 —7,687 432 158 37Soybeans 55 55-60 222,195 ~10,800 728 151 21Soy meal 68 65-70 22,506 20,900 704 148 21Wheat 4l 40-45 65,806 ~12,550 480 124 26Pork bellies \9 16-25 97,925. 9,498 774 281 36Soy oil 69 65-70 89,920 -8,920 586 122 21Plywood 68 65-70 1,622. 3,929 372 98 26Hogs 16 16-20 35,595 -7,190 4,093 318 29
 
P/L shown in the table. The total soybean profit of $222,195 was equal to $305 per tradeprofit over the test period, a very good net return and one that might have been unusual,because of the exceptional volatility of the agricultural markets during the early 1970s.
Comparing the Hochheimer results with the same system applied to the 10 years from1986 to 1996, we see a very different picture (see Table 21-3). Many of the markets that pro-duced large profits now show losses. The currencies stand out as uniformly successful;however, the time periods for calculations vary from 40 to 75 days. The appearance of largecalculation periods for the “Best Day” column indicates either a very successful long trend,as in the T-bonds, or an attempt to minimize the losses of markets without profitable trends.
In most cases, you will find that the shorter trend periods are unstable. Becauseconsistent short-term trends are rare over long test intervals, suich as 10 years, profitableresults for a 15-day moving average are usually caused by one or two very largt profits, ora sustained move in one direction. The choice of a short-term trend rarely produces prof-its in the future.
The ease of new computerized testing platforms allows a convenient look at robust-ness. For the.single moving average, 20 periods from 5 to 100 days were tested in steps of
—~5 days. The last column of Table 21-3 gives the number of profitable tests. Six of the 15 mar-
kets tested had no profitable results using any calculation period. Overall, 45% of all testswere profitable. The interest rates show very erratic results with a wide range of best days,even for the T-notes and T-bonds, which had a large number of profitable combinations.Only the currencies appear to be robust within a framework of moderate performance,
Channels and Crossovers “
This Hochheimer study consisted of two tests. The first is a crossover of two simple mov-ing averages. The short-term average (fast trend) ranged from 3 to 25 days and the long-term from 5 to 50 days. The objective was to eliminate the whipsaws that were evident inthe first study. The rules for this systein were:
1. Each moving average used closing prices only; the long-term number of days was al-ways greater than the short-term days.COMPREHENSIVE ST~ _iES 537
 
—_—TABLE 21-3 Results of a Simple Moving Average Model (1986-1996), $50 Transaction Costs
 
 
Best Net Max ee Robust
Market Day PIL Loss Total Prof % Prof nessCorn 80 -937 8,462 98 23 23 0/20Cotton 60 62,060 -76,640 99 38 4! 18/20Soybeans 95 ~1,825 sUIA75 124 33 27 0/20Silver* 40 485 12,720 V2 24 21 0/20Copper* 45 18,987 -9,000 89 34 38 9120Gold 45 3,240 16,500 146 44 30 0/20Swiss franc 45 76,025 =15,000 125 54 43 18/20German mark 75 46,462 ~13,462 9l 29 32 20/20Japanese yen 60 85,475 13,737 169 61 36 18/20British pound 40 61,825 -33,237 158 54 34 44/20S&P 500 35 4,650 -75,750 200 68 34 0/20NYSE index* 90 -7,100 -21,350 88 29 33 0/20TBills 30 8,350 6,325 156 49 31 8/20T-Notes 15 52,215 -9,235 267 102 38 16/20T-Bonds 100 51,084 ~14,036 74 33 45 15/20
 
‘Test data began in 1987 or 1988,
2. A buy signal occurred when the short-term average moved above the long-term av-erage; a sel signal occurred when the short-term average moved below the long-term average.
3. The model was always in the market.
4. If the high and low prices of the day were equal, a a locked. limit day was assumedand no execution occurred.
The use of a longer and shorter trend introduces the ability to identify a trend bias inthe market and trade only in that direction. Theoretically, this should apply to the interestrates and to the index markets, which have either sustained-trends or an upward bias.During the period of the original tests, it may be the seasonality of the agricultural mar-kets that dominate the trend. Using two moving averages, the faster one can be consid-ered a timing mechanism, while the slower one tracks the underlying market direction. InTable 21-4, the number of trades are less than the simple moving average model, and thepercentage of profitable trades was increased, the type of results that indicate a morerobust method.
Retesting
The testing performed to find the best set of parameters is never the last test. It is alwaysimportant to monitor performance and compare the out-of sample results with a new opti-mization to determine the success of the system rules and the testing process. A 1982 pub-lication of Hochheimer* provides a rare opportunity to see how the original tests fared andhow parameter selection would have changed based on both additional data and changesin price patterns. We can do this ourselves by testing 5- or 10-year periods and comparingthe results of the same portfolio. From these results we can visualize an overall change in® Frank L. Hochheimer and Richard J. Vaughn, Computerized Trading Techniques 1982 (Merrill Lynch Com.modities, New York, 1982).538 TESTING—_—_
eeeTABLE 21-4 Results of Twd Simple Moving Average Crossovers
 
 
Max. Trades
Best Net Run of.
Days PAL Losses Tatal Profit % Prof.Cocoa 725 $176,940 $-4,436 468 199 43Corn (147 69,275 -2,697 225 93 35Sugar 550 335,843 -13,851 311 109 35Cotten 16 25 304,485 4,755 485 233 48Silver 326 100,790 —8,610 661 262 40Copper 1733 212,939 -3,057 354 77 50Soybeans 1650 286,440 -15,665 311 148 48Soybean meal 18 50 117,155 -8,155 272 8 43Wheat 1147 113,118 -2,660 209 87 42Pork bellies 25 46 13,124 -21,538 226 100 42Soybean oil 1450 121,749 6,585 327 128 39Plywood 2442 18,505 —3,184 219 100 46Hogs 314 97,448 7805 793 321 40
 
which the market moves. We may find that noise has increased, causing a shift to longertrends, or that risk is much greater and the returns are proportionally smaller.
Crossovers Retested
In the 1982 update of the Crossover System optimization using data through 1979, it isinteresting to note that 11 out of 17 markets selected two moving average speeds eitheridentical (in six cases) or nearly identical to the previous test. Five of the remaining six testsshowed new selections that traded slower than the previous test. Only gold resulted in afaster selection, but the original test in 1976 could have had only one year of data.
Because the results of using the parameters selected in 1976 were not shown in thenew study, their performance could not be assessed. Eleven cases, however, in which thenew parameters were either identical or had very small changes, can be used as a conser-vative estimate. Their out-of-sample performance is shown in Table 21-5. If the parametersare not identical, the 1979 test must show results that are higher than the actual out-ofsample would have been.
The first reaction to the update of the 1976 crossover test should be very positive.Almost identical parameters for 11 out of 17 products originally tested were selected. Thismeans that the traders who used this system were using the optimum parameters duringthe 3-year period from 1977 through 1979. How did they do?
Table 21-6 is a comparison of the out-of-sample period, 1977 through 1979. In thecases in which the parameter selection was slightly different, it must be assumed that thetotal performance, 1970 through 1979, was better than the performance using the originalparameters. Therefore, the out-of-sample period should be at least as good as the resultsactually achieved. Total profits were $201,649, trading oné contract of each market.
Two additional factors should be considered. Although only 3 of the 11 showed losses,the fast nature of this system results in very !ow profits per trade in 4 of the remaining 8markets. This may not seem serious until the realities of trading are considered, where theexact execution price is rarely achieved. A trend-following system, with orders entered asintgaday stops, must always allow for slippage caused by lack of liquidity and directlyrelated to the size-of the order being executed (See the discussion of liquidity in Chapter18). In a fast market, this slippage can be large, often hundreds of dollars (in contractCOMPREHENSIVE S.. UDIES
539
eeaTABLE 21-5 Retesting of the Crossover System
 
Through 1976
Through 1979
 
 
Trades TradesBest Net Profitable! Profits! Best Net Profitable/Days PIL Total % Trade Days PIL TotalCorn 1147 $ 69,275 93/225 4l $ 308 12 48 $ 83,586 129/332Sugar 550 335,843 lo9/311 38 1080 650 348,833 160/449Cotton 1625 304,485 © 223/485 46 628 $625 378,440 329/697Silver 326 100,790 262/661 40 152 326° 96,995 385/1098Copper 17 33 212,939 1771354 50 602 17 32 218,790 245/571Wheat $147 433,118 87/209 42 637 12 48 151,871 130/305Pork bellies 25 46 (3,124 400/226 44 58 25 46 78,207 137/312Soy oil 1450 121,749 428/327 39 372 1450 125,848 190/488Plywood 24 42 18,505 00/219 46 84 20 46 2,667 132/351Catcle 713 147,540 337/792 43 186 713 162,280 490/1 186T-Bills 618 39,710 69/148 47 268 618 28,210 148/388
 
value) away from the intended price.:Even Market-on-Close orders are subject to slippage.A buyer using a Close-Only order should expect to get the high of the closing range; a seller
will get the low.
In Table 21-6, the furthest right column assumes a conservative slippage, one that
should be included in any test program. In some cases, such as copper and bellies, the slip-page is small and does not seem to impact on profits; even the large profits of cotton sur-vive well. But in the fastest trading commodities, even the smallest slippage will severelyreduce large profits, frequently turning them into losses. The final total shows that thosesmall costs, when applied to every product, were greater than the profits. Instead of mak-ing a killing in the market, these traders netted a loss.
 
TABLE 21-6 Qut-of-Sample Results for the Crossover System
 
 
Trades
Net Profitable! Profits! |-Way Slippage
Pit Tatal % Trade (5% Daily Range)Corn $ 14311 36/107 34 $ 133* Ag = 25 x 250 =~ $ -5,350Sugar 12,990 SIA3S- 37 94* 10 pts = 112 = 34500Cotton 73,995 106/212 50 349 10 pts = 50 =  -21,200Silver 3,795 123/437 28 -9 ad 50 = 43,700Copper 5.851 68/217 31 27* 20 pts = 50 = 21,700Wheat 18,753 43/96 45 195* he = 25 = -2,400Pork bellies 65,083 37/86 43 757 log = 42 = -7,224Soy oil 4,099 62/161 39 25 5¢ = 30 = —9,660Plywood ~15,838 32/132 24 -120* 20 pts = 15 = 3,960Cattle 14,740 153/394 39 37 10 pts = 40 = 31,520Bills 14,500 791240 33 ~48 2pt- = 50 =  ~-24,000Totals $201,649 -$205,214
 
“Best case (could have been no better. possibly worse).540 TESTING
 
Crossover System, 1986 to-1996
Bringing this technique forward to current markets, seen in Table 21-7, there is again a ten-dency to select longer calculation periods, especially for the short-term trend; therefore,there has been a shift from the optimum trend speeds of 5 to 10 years earlier. The numberof total trades has dropped significantly across all markets, and may be attributed to moremarket noise and less definitive trends. Because the use of two trends greatly increases thechance of overfitting the data, it should not be surprising that all of the markets show prof-its for some combination of fast and slow moving averages. The last column, indicating thenumber of profitable combinations, is needed to get a picture of the robustness.
Taking a closer look at Table 21-7, it appears at first that the S&P would be highly prof.itable; however, the profit of $194,750, drawdown of $63,650, and 63% profitable trades isdeceiving because only 8 of 120 combinations were profitable. As a group, the currenciesand interest rates performed well, and the total of all profitable combinations was 56%,showing that the technique of using two moving averages is likely to be more robust thanthe 45% given by the simple moving average. When making these system comparisons, youmust be cautious that these tests represent a comparable range of test speeds.
ChannelsThe next model relies on trends but are very different from the moving averages used inthe previous optimizations. Originally called a price channel, it is now more familiar as anN-day breakout, the penetration of the high and low of the past N days. The penetration,which causes a buy or sell signal, occurs if the closing price penetrates the high-low band.Table 21-8 shows the results of the interday test. Acommon variation of this methodis to buy when the intraday high penetrates the upper band; however, this can cause ambi-guities when both the high and low of the day exceed both bands. It is not possible toknow which one occurred first. Intuitively, the closing price is expected to be a more reli-able indicator of direction than the intraday high or low.
 
TABLE 21-7 Results of a Two-Moving Average Crossover Model (1986-1996),$50 Transaction Costs* .
 
 
Best Net Max ____Tades
Market Day PHL Loss Total Prof % Prof RobustnessCorn 17 x 60 3,775 75,375 44 \7 39 24/420Cotton 11x 60 66,725 5,390 40 18 45 105/120Soybeans 20 x 25 73,450 44,100 131 80 6t 28/)20Silvert 15x19 18,230 -40,180 155