99.5 percent chance that any choice will yield returns greaterthan -4 percent (the average minus 3 standard deviations).The minimum test criterion should be an 84 percent chance of success,given by the Best Choice Index.
Testing ProcessThe total solution is the test process. It begins with conceptualization. Itmust be followed by clear steps that lead to a well-defined result. Experience shows that if you do not control the process, the process willcontrol you. The test procedure can be separated into five parts:1. Deciding what to test.2. Deciding how to test it.3. Evaluating the results.4. Choosing the specific parameters to trade.5. Trading and monitoring performance.Each of these steps is critical to the success of the program. Setting upthis process for the first time will take a lot of careful work, but most of
182
Making a Trading Strategy Robust
it will only need to be done once. There will be many decisions to makewith regard to the data, testing software, and the method of evaluation.Because the proper development of a trading strategy is so important toits success, these issues will be discussed in detail in this chapter. Box10-1 provides a checklist that will serve as a reminder.
Part 1: DecidingWhat to TestBefore you begin testing, define the system and the test plan completely. You must tell the computer what to do, not allow the computer to tellyou. Do not drift from one idea to another as you reach obstacles. Try tofollow the original idea to completion and learn its advantages and disadvantages.Step 1. Is the Strategy Logical?Did you write the rules before you began testing? Where did you get yourideas? Successful trading programs are based on sound ideas such aseconomic relationships (e.g., arbitrage, seasonality, and the spreadingof strong and weak economies) or valid technical strategies (e.g., breakout of a support or resistance level, selling volatility with options, or useof divergence). Letting the computer uncover an obscure short-termpattern, no matter how reliable it seems, is not a sound tradingapproach. Price patterns can always be found, but they have doubtfulpredictability and often change without notice.When you develop your program, the strategy must make sense forthe market and fit your own objectives, as in the following examples:• For the stock market, you might want a long-term buying strategy withno short positions.• For the bond market, a long-term strategy that parallels slow-changingeconomic and government policy might be most conservative.• For foreign exchange, a short-term method that would buy or sell in thedirection of an intraday breakout with small profit-taking objectivesmight make more sense for tactical hedging and limited overnightpositions.Using Logical Ideas. A logical idea does not need to be based on fundamentals. Years of watching price movement on Chicago's InternationalMonetary Market (IMM) may give you the idea that dependable entry signals occur only during the three periods of high volume each day—at the
Box 10-1. CHECKLIST FOR ROBUST TESTINGPARTI: Deciding What to Test••••
1. Is the strategy logical?2. Can you program all the rules?3. Does the strategy make sense only under certain conditions?4. Take a guess as to the expected results.
PART 2: Deciding How to Test•5. Choose the testing tools and method.•6. Do you have enough of the "right" data?•7. Have you included realistic transaction costs?•8. Will you test a full range of parameters?•9. In what order will the parameters be tested?D 10. Are the parameters distributed properly?• 11. Have you defined the evaluation criteria?• 12. How will the output be presented?PART 3: Evaluating the Results•••
13. Are the calculations correct?14. Were there enough trades to be "significant?"15. Does the trading system produce profits for mostcombinations of parameters?Q 16. Did logic changes improve overall test performance?• 17. How did it perform on out-of-sample data?PART 4: Choosing the Specific Parameters to Trade•••••n
18. Did the last test include the most recent data?19. Did you choose from an area of broad success?20. Are profits distributed relatively evenly over the tested history?21. Are the profits per trade large enough to absorb errors?22. Did the historic results show any large losses due to price shocks?23. Have you risk-adjusted the returns to your acceptablerisk level?
PART 5: Trading and Monitoring Performance• 24. Are you following the same rules that were tested?• 25. Are you trading the same data that was tested?D 26. Are you monitoring the difference between the system andactual entries and exits?183
184
Making a Trading Strategy Robust
open, close, and just after traders return from lunch. The low volume periods between give less dependable indication of direction and require amore demanding price move to enter a trade. The important rule is to knowwhat you want to do, and then use the computer to verify your idea. You mustcontrol the process.
Starting with One Idea and Ending with Another. Be sure thatcomputer feedback does not cause you to stray from your original idea. Alogical strategy can evolve into meaningless patterns. There is a naturaltendency to explain why a system must be fundamentally sound, simplybecause you have already seen that the test results are good.Step 2. Can You ProgramAll the Rules?Can all the rules in the trading strategy be entered into the computer or aspreadsheet program? Have you assumed anything that was not programmed?
A strategy that cannot be tested cannot be evaluated. If you assume thatyou would not have been caught in a price shock because the programdoes not trade overnight, then you leave yourself open to unexpectedlosses, undercapitalization, and justifiable criticism.Writing clear trading rules is essential to testing. You must be certainthat you can account for entry and exit conditions, risk control, types oforders, time of day, and other situations that completely describe yourplan. Writing the rules will tell you the type of data needed for testing(whether it is only prices, the Producer Price Index, or API statistics) thefrequency and extent of the data (open, high, low, close, or 30-minuteprices with tick volume). As carefully as you try, you will always needto add details later.Intraday Breakout Example. Start with the most basic approach,omitting risk control, profit-taking, or qualified entries. If you believe thatan intraday breakout system is a sound idea, then first test only the breakout entry and the basic exit signals. You might want to close out the tradeat the end of each day; or, you might exit if prices reverse and breakout inthe opposite direction. It is important that you know whether the underlying idea works before adding profit-taking, risk control, and other morespecialized features.Decide which parts of the system can vary. You know that a breakoutearly in the day allows more time to reach bigger profits during the restof the trading session. Therefore, you will want to test the time of thebreakout. You will not want to accept an entry signal late in the day,because of the limited potential for profits before the close.
Testing for Robustness
185
If the program is fully computerized, you will want to look at the datano more often than every 5 minutes. Although you may be able to execute an order within 60 seconds of a breakout, it is not practical toassume good executions. Using 5-minute bars for testing; rather than 1minute, will also reduce the time needed to test the strategy.Before you start testing, you know that an intraday breakout systemdepends on the period over which the breakout is measured, the time ofday for entry, the size of the profit-taking objective, and some risk control.Trend System Example. All systems have common features: entryand exit rules, risk control, and possibly profit-taking situations. A trendsystem requires a trend speed. This can vary significantly with your application and objectives. Equities programs, with little leverage and highertransaction costs, require a range from 50 to 500 days. A futures trader,with margins of only 5%, will favor faster trends, from 5 to 30 days.It is a mistake to use a smoothing approach on intraday data. As thetime period between data observations gets shorter, the level of noiseincreases. Because of illiquid periods in all markets, prices can jump ineither direction without indicating a true trend change. This causes frequent false signals that cannot be eliminated by using a longer trendbased on the same intraday prices. The combination of intraday noiseand trend lag will be a difficult obstacle to overcome.Step 3. Does the Strategy MakeSense Only under CertainConditions?Decide, in advance, whether the strategy targets certain market movement, or a specific set of conditions. The idea may only make sense forlong or short time intervals. For example, a day-trading program using15-minute data would not use a 200-period moving average, while along-term investment program in stocks would not use a 3-day trend.By defining the range over which the trading model will operate, youreduce the chance of being diverted from your objective. Write out themost reasonable test range for each of the parameters that are considered important to the strategy. The more you can define your expectations, the better the results.Step 4. Take a Guess as to theExpected ResultsDecide the expected rate of return, the percentage of profitable trades, andthe size of the losses. The objective is to compare the test results with your
186
Making a Trading Strategy Robust
expectations. Whether the results are much better or worse than planned,when you have a basis for evaluation it will be easier to correct and moveforward with the development of the system. To say "something iswrong" with the test results, you must first decide what you expect.
Part 2: DecidingHow to TestStep 5. Choose the Testing Toolsand MethodWith more sophisticated strategy-testing software, it is no longer necessary to program the trading method in FORTRAN, BASIC, or C to test itssuccess. In a few minutes, using a strategy testing package such asTeleTrac, Omega's System Writer, or even a Lotus or Quattro spreadsheetprogram, you can have a good idea of the viability of the technique.An increasing number of programmable graphics terminals and newstrategy-testing software are available at very competitive prices. They allhave the advantage of calculating profits and losses accurately, the flexibility of rule changes and data selection, and the ability to plot both dataand profitability. In some cases, results can be read into spreadsheets forfurther evaluation. The time saved is well worth the price. For the moresophisticated analysts, supplementary software such as ManugisticsStatgraphics and Mathsoft Mathcad are impressive tools for evaluatingcomplex statistical relationships and expressing mathematical formulas.Long Test, Short Test, or "Step-Forward" Test? The pattern inTable 10-1 is not unusual. Tests using a small amount of data give resultsshowing that many combinations of parameters will work. The shorter thetest period, the more profitable the system will appear. Consider a bondmarket that has moved steadily up for 3 months. If there were only smallretracements, then any moving average from 10 days and longer wouldhave yielded the same results, which is the net move from the beginningto the end of the period (see Figure 10-1 (a)).When a short test interval has one or more price swings, the slowertrends give back profits, while some of the faster ones are very successful. The size of the swing and the amount of market noise determinewhich trend speeds are best (see Figure 10-1 (b)). In general, tests ofsmall amounts of data give:• Individual and average test results that are much higher• Risk that is sometimes lower• Profitable results for more models that trade faster• Erratic forecasting ability
Testing for Robustness
187
(a)
(b)
Pi jure XO-1. Moving average results for a short test period, (a) A shortperiod with a strongly trending market allows most trend speeds to performwell, (b) A short period with price swings may allow fast trends to capture profits, but net losses for the longer term trends.
It is much more difficult to find a trading method that is good overlonger test periods. The best tested performance (annualized rate ofreturn) of a system tested over many years will never be as high as therate of return of a similar system tested over a few months or a year.Using more data, you should expect:
188
Making a Trading Strategy Robust
• Much lower returns• Larger risk when positions are held longer• Difficulty in getting consistent profits from short-term trading• Questions as to the relevance of older data• Better forecasting abilityTherefore, test results using smaller amounts of data look better, but donot perform as expected; results based on longer tests look worse butperform closer to expectations. You should not be disappointed in theresults of a long test period when compared with shorter tests. It is onlythat the shorter tests are misleading.Select a Long, Representative Test Period. When more data aretested, there is a greater variety of unusual situations, longer profitableprice moves, sequences of losses, and price shocks. When longer periodsare tested, both risk and reward increase; however, risk increases fasterthan returns. Testing shorter periods can give an unrealistically small risk,cause undercapitalized trading and fatal results. A good rule is to be certain that the data contain two full cycles, that is, there should be two clearbull markets, two bear markets and two prolonged sideways intervals.Because results never look as good when the same strategy is testedover longer periods, you might argue that markets have changed andthe old data are no longer representative; that globalization and regional alliances have changed the price relationships and patterns in manysectors, or that government controls will prevent an economic collapse.By saying that the market will continue to exhibit only the price patternsseen recently is unrealistic. It will evolve to new patterns; however, wehave no way of knowing what they will be. The past contains the mostaccessible, practical, and realistic examples of changing situations. Box10-2 shows that performance drops but predictability increases with theuse of more data. Short test periods produce unreasonable expectationsof profits."Step-Forward Testing" versus One Long Test. The technique of"step-forward testing" seems to be a sensible approach to resolving someof the testing dilemmas. It works as follows:1. Select a short data interval, called a "test window" (e.g., 2 years ofdata).2. Test (optimize) a full set of parameters on the test window and selectthe "best."
Testing for Robustness
189
3. Run the model on a short period of out-of-sample data, immediatelyfollowing the test window (e.g., the next 3 months), using the "best"parameters (see Figure 10-2).4. Collect performance data on the "out-of-sample" period, including acomparison with the "in-sample" test-window returns.5. Move the test window forward and repeat steps 2 through 4 until done.6. The parameters that perform most consistently in the out-of-sampleperiod are considered best.Hidden Problems. Step-forward testing seems to duplicate the way wewould operate a trading program. But there are hidden problems:• Shorter test periods favor faster strategies that produce higher profitsand more trades. But the performance of fast-trading strategies variessharply from one test window to another, as discussed in the previous section and shown in Table 10-2. The "best" parameters are not agood choice for trading.• Short test periods do not represent long-term trading fairly. Each shorttest period can have only a few long-term trades, and they may startand end in the middle of a trade. Moving the test window forward doesnot correct the treatment of these trades.• Retesting the same system with modified rules means that the "outof-sample" data is no longer new. This is called "feedback." Onceused, the data is no longer "out-of-sample." You know what to expectand how to make the data patterns show more profits.The step-forward process will usually select an inconsistent, fasttrading method over a better long-term system simply because the testwindow forces this result. Instead, use all the data in one long test to getcontinuous performance over as many changing patterns as possible.Step 6: Do You Have Enoughof the "Right" Data?The more data you test, the more situations the program will experience. There must be at least two bull markets, two bear markets, andtwo sideways periods. Unless you can prove that the older data is misleading, or no longer valid, you should use as much data as possible.Put some data aside for out-of-sample validation after the final systemhas been selected. This is discussed further in Step 18.
Box 10-2. MORE DATA IMPROVE TEST RESULTSUsing more data produces more consistent and realistic results. Finalresults may show that risk is higher and profits are lower, but these figures are more likely to be achieved in trading. It is more difficult to findpersistent short-term patterns in a longer data series; therefore, selections favor slower trading. Long-term solutions, in turn, include realistic equity fluctuations because they cannot be fine-tuned to avoid specific losing periods. This performance profile shows higher risk andmakes it necessary to have higher capitalization.A simple test of the MATIF CAC-40 Index (Table 10-2) shows thepredictive ability of tests based on 1,2,3, and 5 years of data. The system tested was1. An exponential moving average from 5 to 50 days, in incrementsof 5 days.2. A trend change criterion ("filter") from 0 to 10 points, in increments of 1 point.3. A buy signal that occurred when the trend turned up by theamount of the filter; a sell signal that was generated when thetrend turned down by the amount of the filter value.The highest profits for each test determined the trend speed and entryfilter that were to be used to evaluate the next 1 year of data. The averages for each test case were compared.
Table 10-2. Tests of CAC-40 (MATIF) French Stock Index1-Year TestYear(s)Tested19821983198419851986198719881989199019911992Avg/Yr
2-Year Test
Best PerformanceSpeed Filter P/L55010101030351035201521
3062710670765
6311881287683578515378708266328364
1 Year Year(s)Ahead Tested(54)(70)(134)5564(600)(427)232(126)208
—(41)
Speed and filter vary.Test is sensitive to current patterns.
190
82-8383-8484-8585-8686-8787-8888-8989-9090-9191-92Avg/Yr
Best PerformanceSpeed FilterP/L
1 YearAhead
402510201525451020
215339186
21612438593812421160569734501
(79)(162)620362(447)(338)3(141)237
—
—
—
—
23
4
326
6
Average trend slows.In-sample profits decline.1 year ahead improves.
Summary of results• 1-year test. The best trend speed and filter varied considerablyfrom year to year. The average speed was lowest of all tests, showing that a fast trend often looks best for a short test period. Performance was inconsistent in 1-year-ahead tests, averaging a loss of41 points. Using 1 year of data to forecast 1 year ahead does notlook promising.• 2-year test. Overall profits per year declined and the average bestchoice slowed slightly, showing that more test data become harder to fit. However, results for 1 year ahead increased and becamenet positive.• 3-year test. The performance pattern continued to improve. The 1year-ahead tests for 1988 and 1989, which show large losses, werestudied to find out that those years posted new highs. Performancein the out-of-sample data seems to be better when the price movement is within the range seen in the tested period.• 5-year test. Improvement continued overall. Longer trends wereselected and tested performance declined. These are especiallygood results because 1985 and 1986, which showed large profits inthe 3-year test, were not part of this out-of-sample performance.
Table 10-2. Tests of CAC-40 (Matif) French Stock Index[Continued)3-Year TestYear(s)Tested82-8483-8584-8684-8786-8887-8988-9089-9190-92Avg/Yr
5-Year Test
Best PerformanceSpeed Filter P/L15202020255010302524
22339174105
1624721165141412259109008321072302
1 Year Year(s)Ahead Tested215119362(454)(388)362(71)7327
Best PerformanceSpeed FilterP/L
1 YearAhead
82-8683-8784-8885-8986-9087-9188-92Avg/Yr
2020204010253028
22207904
968151513271374164013941024264
355(39)(344)345(71)9
—43
Average trend is slower.Average trend is slowest.In-sample profits are lower.In-sample profits are lowest.1 year ahead is better.1 year ahead is best.New highs in 1988 generated losses because it is not part of sample data.
191
192
Making a Trading Strategy Robust
2-Year3-MonthTest Interval Out of Sample
Figure 10-2. Step-forward testing. A process that follows a veryappealing and strict testing approach, often jumps from fast to slowparameters because of the short test window. It may miss selectingslower, more conservative strategies.
Are Tou Testing the Same Data That Ton Will Trade? Do not testone set of data, then trade another. Do not use a "continuation" seriesbecause either the gaps have been removed, or they cause windfall profitsor losses that would not have happened in trading. A "perpetual" contracthas prices that never existed and usually dampens any severe price movecausing the risks to look smaller.Did Ton Verify the Accuracy of the Data? Data can be inaccurate evenwhen prepared by a reliable vendor. Look for prices at the beginning or endof a contract that are completely different. Sometimes the data will haveprices from another market that have not been erased, or an erroneous dateone or two years earlier. Check for blank or zero entries. If you chart the data,you will easily see errors. The ones that are too small to see can be ignored.Special Cases in Selecting Test Data. It is not always possible tohave enough data for testing. New markets or changing situations mayrender old data questionable. Or, you are looking to profit from a recentprice pattern, without expectation of using the system for very long. Thefollowing sections offer some alternatives.Selecting Similar Data Periods. A stock that has dropped to a verylow level can have a very different performance pattern from a period ofhigh prices and high volatility. Selecting similar historic periods, such asthose following a prolonged decline, or after a sell-off of 10%, may be theonly way to model your strategy.
Testing for Robustness
193
Using Cash Markets to Model Futures. Cash markets are often usedto test a system that will be traded as a new futures contract; however,a new contract can be illiquid. A good model will account for similar situations in other new markets, adapting to the change between the cashand futures. Because there are many examples of changing markets, thisshould be a successful exercise.Stock and Futures Markets under Special Situations. All markets gothrough severe changes: a corporate scandal or mismanagement, suddennew competition or government regulation; a price shock in coffee ororange juice due to a freeze. These special situations must be alsostudied separately, rather than absorbing them into the flow of everydayprice movement. Market reaction to special situations is often similarbecause of the human response, rather than the fundamentals of a company or commodity. Similar cases can be found in other markets. Whenthe special situation is a "price shock," a new set of rules can be used.This is discussed in Chapter 7.Structural Changes and Not Enough Data. The European MonetarySystem (EMS) imposed a structure on participating currencies that had little precedent. A previous period, under the Bretton Woods agreement,may not provide enough similarity or adequate data for modeling a trading strategy. In this case, a fundamental analysis is the only course. Resultsbased on small amounts of data are unreliable. A sound understanding ofthe fundamental interaction and the rules under which the new agreementoperates may allow some confirmation by testing. To date, the EMS hasproved to be unstable, therefore, a test of the 1 to 2 years of data wouldhave led to poor results.Creating More Data. For some markets, it is possible to create syntheticdata. By studying volatility at different price levels, sequences of runs, variation in periodicity between highs and lows, and seasonality, it is possibleto use random price generation to create data with the same statistical qualities as the one being evaluated. Synthetic data gives you the ability to testmore situations and develop a more robust solution, but it is not the same asreal data. It is best to use synthetic data first, before testing actual data.Type of Data to Test. The data used for testing strategies shouldalways be the same as the data to be traded. This is very straightforwardfor stocks, but becomes more difficult when you use foreign exchange orfutures prices. The forex market will require adjustment for interest overthe holding period, but the futures market presents the greatest problem.Although the nearest futures has the greatest liquidity, it may tradeactively for as little as one month, and rarely more than three months.This frequent expiration makes testing inconvenient. The following sections will show how to fix this problem.
194
Making a Trading Strategy Robust
Original Data Series. For stocks, foreign exchange, interest rates, orother cash market data, a long series of original, unadjusted data is available for testing. When trading the cash market you will also need the spotinterest rate to calculate a forward price. Treating the cash price as a validentry and exit point omits the need to roll the position forward daily. Eachrollover has an implied transaction cost that eats away at profits.Alternately, you can use spot prices for entry and exit, and calculate thenet interest rate credit and debit when liquidating the position. Additionaltransaction costs must be included in testing each time the position isrolled over.Futures Contracts. Original futures contract data can also be used fortesting, without modification. Use the following steps:1. Read the futures contract data.2. Start the strategy calculations at the beginning of the series ("winding up the strategy"), or as soon as there is enough data.3. Begin taking positions in the new contract on a specific date, or onthe day that the previous contract stopped trading. If the previouscontract had an open position when it was "rolled," then assume thesame position on the same day in the new contract. Exceptionsshould be made for markets that do not have a high correlationbetween delivery months, such as livestock, or at critical seasonalperiods such as the change for old to new crop for grains, orFebruary-March in heating oil.4. Exit any trade on a specific preset date before expiration. For interestrates, this is usually the last day of the month prior to delivery. Forcurrencies, it is about 3 days before expiration.This method is inconvenient because results are usually given by contract. For a 10-year test of interest rate futures, 40 separate sets of resultsmust be accumulated. In addition, it is difficult to assess maximumdrawdown unless you can treat the segments of data as a continuousequity stream.Continuous Data Series for Futures. Many data vendors provide aconstructed series that pieces together the nearest futures contract to create a new series (e.g., a 3-month price) calculated in a way that resemblesthe London Metal Exchange forward contracts. These choices are unacceptable for testing because they do not show the data that will be tradedin a way that can duplicate a realistic trading environment. The constructed 3-month series, with interpolated carrying charges or interest, is frequently a smoothed version of prices that occurred at that time, reducingboth the profit and risk.
Testing for Robustness
195
Gap-Adjusted Series and Index Series. A gap-adjusted data series isa good alternative for most technical applications in futures. It puts thenearest-to-delivery segments together into a single price series by closingthe gaps at the time one contract rolls into the next. By proceeding backward, the most recent futures or forward contact has today's prices, andthe older contracts are adjusted up or down according to the gaps.The gap-adjusted series works well for trend-following applicationsand strategies where the comparative price, rather than the actual price,is needed. It does not work for chart analysis, economic studies (supply/demand/price relationships), and similar uses. One problem withgap-adjusting, where older prices are changed, is that the very old datacan take on negative or unrealistic values. Because the prices are notreal, the rate of return and risk measurements must refer back to theactual prices, rather than base their values on the gap-adjusted series.With one additional step—indexing—the gap-adjusted series becomesmore useful. Indexing is simply starting with the value of 100 (or 1000,depending on convenience), then adding or subtracting successive values as a percentage change. For example,index = index[1] + (price - price[1])/price[1]
Today's new index value, index, is yesterday's value, index[1 ], plus the percentage change in yesterday's price. The notation [1 ] means the 1-day priorvalue. The index price represents a percentage change and allows simplecomparisons between returns of different markets. It eliminates the needto reference the original price data to calculate risk and returns.Building a Gap-Adjusted Series. If you are working with futurescontracts, a continuous series can be very useful. There are three steps tofollow, (1) creating a continuous series with duplicate entries on the day ofthe rollover, (2) gap-adjusting the series, and (3) indexing. Figure 10-3shows a flowchart of this process beginning with Step 2. Use the prices inTable 10-3 to follow the flowchart. For example, if the S&P 500 were beingcombined, Step 1 causes the June 93 contract to stop on the last day of May,and the September 93 contract to start on that day. Step 2 would gap-adjustthe prices, working backward, whenever it identified a duplicate date. InTable 10-3 the June contract values are adjusted up by 12.00, equal to theroll-forward gap on May 31. Step 3 would assign 100 to the first value, thencalculate the percentage changes for each successive entry. Note: A cleveranalyst can eliminate Step 2 if an index is the only output.Alternatives. The only remaining problem with gap-adjusting is thattransaction costs cannot be posted at the time of the roll-forward, becausethat date can no longer be identified. It may be more difficult, but prefer-
106
Making a Trading Strategy Robust
Step 2: Gap-Adjusting1. Start at the most recent data entry and scanbackward.Initialize BIAS, the accumulated size of the gap.Initialize price counter N.2. If the prior date is not equal to current date,add BIAS and keep data, otherwiseadd the new gap to BIAS and skip thisduplicate entry.3. If done with data, then go to Step 3, otherwiseincrement N to look at next older data.
Step 3: Indexing4. Start at oldest data, MInitialize index to 100.5. Increment M to look at next sequentialdata item.6. Calculate index value as the percentagechange.7. If not done, continue at (