5).
Figure 10-3. Flowchart for continuous, gap-adjusted, indexed series. Once youhave a continuous series built (Step 1) according to the form shown in Table 10-3,it becomes easy to scan backward and gap-adjust, the go forward to index.
able, to write program logic around the continuation file, which containsthe duplicate dates and data. When a duplicate date is encountered, theold trade is closed out and the new trade entered.Shock-Adjusted Series. A FORTRAN program for removing priceshocks, then restoring the continuity of the data by indexing, can be foundin Chapter 7. It is a similar program to the one in Figure 10-3 and givescoding details.
Testing for Robustness
197
Table 10-3. Sample S&P Prices Combined before GapAdjustingContract
Date
Price
Gap-Adj Value
Index
JUN93JUN93JUN93JUN93SEP93SEP93SEP93
930528930529930530930531930531930601930604
451.50449.25446.50448.00460.00462.50460.75
463.50461.25458.50no entry460.00462.50460.75
100.00099.50299.38899.225no entry100.00599.622
S&P prices have been combined into a single series, and still show the originalprices. A duplicate entry appears on May 31, which will be the date of the rollforward where the gap is adjusted. The "Gap-Adj" and "Index" columns showthe values after those steps have been completed.
Step 7. Have You IncludedRealistic Transaction Costs?Transaction costs include brokerage and slippage. But other factorsreduce performance.Do You Expect Any Missed Trades? "Unables" have a great impact onresults because they reduce only the profits and not the losses. If you overtrade the liquidity of the market, then unables become an important factor. Programs that trade intraday will face more problems than those thattrade on the close. Part of a successful program is achieving actual tradingresults similar to expectations. A full discussion of slippage and unablescan be found in Chapter 2.Step 8. Will You Test a FullRange of Parameters?Determine, in advance, the range of parameters that is sensible for this strategy. If you are trading stocks for an institutional portfolio, a moving average test range may be 50 to 400 days. Stop-losses must be equally large.However, do not prescan and remove very fast and slow ranges becausethey showed losses. That is the same as eliminating everything except theone set of parameters that was profitable. You cannot develop a robustmodel by looking at a narrow range that has been preselected to work.Step 9. In What Order Will theParameters Be Tested?Test the most important variables first, the ones that cause the largestchange in performance. That would be the number of days (the "period")
198
Making a Trading Strategy Robust
in a moving average, Relative Strength Index (RSI), or stochastic; the timeof day or number of days in a breakout system; or, the deviation from thenorm in a countertrend or arbitrage approach. These variables usuallyhave the greatest effect on profits. Tests of other rules should follow, inorder of most impact on profits or most frequently applied.Testing the variables that are most important will speed up the testprocess. Rather than testing all combinations of all variables in one procedure, selecting the test range for one variable at a time can reduce thenumber of tests and the total time of the testing process.In some cases, the most profitable combination of parameters occurswhen the primary variable is "suboptimized." For example, profittaking opportunities may be increased when the moving average is veryfast, therefore you want high-momentum situations for very fast profittaking objectives and a short holding time. If two features must worktogether, testing both the trending period and the profit-taking levelsimultaneously can work. It may also be that the profit-taking level is themost important variable, and the trending period is not as significant.Step lO. Are the ParametersDistributed Properly?Not only should the range of parameters be set in advance, but the distribution of those tests is important. Box 10-3 describes what needs to bedone. This is a crucial step in preparing to see the whole test picture,which is essential for a robust system.Because the final decision is based on the average of all tests, the distribution of parameters must not favor either the fast or slow strategies.They must be evenly distributed. When a moving average system is tested, it is generally thought that a test of 5,10,15, 20,... days is a reasonable choice. Equal increments, however, favor the very slowest trading.Figure 10-4 shows how equal days have very unequal percentagechanges from one test to the next. A change from a 5- to a 10-day moving average is a 100 percent change in the amount of data. A changefrom 10 to 15 is a 50 percent change, but a change from 95 to 100 is onlya 5.2 percent shift. An equal distribution of days will skew the resultstoward the slower tests.Visual Distribution. It is not necessary to use mathematics to decidethe distribution of parameters for testing. A very effective visual methodcan be best shown by the following example. If the fast end of the testshows 100 trades and the slow end has 10 trades, choose test periods sothat 11 tests give results showing trades of 100,90,80,..., 20,10. In reality,
Testing for Robustness
199
a perfect distribution is impossible, but the goal is clear. Try to find theparameters that cause the number of trades to be evenly distributed acrossthe full range of tests.Step 1 1 . Have Yon Defined theEvaluation Criteria?What do you measure to decide which system is better? To evaluateresults, it is necessary to produce a minimum number of statistics for eachtest. Selecting the test with the highest profits may not be as important asfinding the one with the best return/risk ratio. Decide in advance howyou will select the best strategy. Most often, you need a combination ofstatistics, including reward/risk ratio, profits per trade, and risk-adjusted returns.• Return/risk ratio is the compounded, annualized rate of return divided by one standard deviation of the annual equity changes. For practical purposes, monthly or daily values can be used for comparisontesting as long as all tests are the same. This standard, set by the securities industry, allows a fast, uniform comparison of tests over different time intervals.Compounded annualized rate of return,CROR = (Ending value - Starting value)A(1/years)Standard deviation,SD = @STD(Monthly changes in equity)Return/risk ratio,RR = CROR/SD
Calculations should use returns on cash to see the raw performancebefore deciding on the potential use of leverage. The importance anduse of these three statistics are discussed thoroughly in Chapter 4.• Profits per trade show how much room you have for unexpected problems and allow you* to see the impact of transaction costs. If two systems have the same percentage returns and similar equity swings, theone with the highest profits per trade is the better choice. If marketvolume drops when trading a new market, or when trading is at anilliquid time of day, the test showing the highest profits per trade canabsorb more transaction costs. A system with less than $50 profit pertrade is an unlikely candidate to succeed.
Making a Trading Strategy Robust
Box 10-3. DISTRIBUTION OF TRENDS FOR TESTINGThe selection of which trend speeds to test will give a correct or distorted view of the potential of the system. If moving averages from 5days to 100 days are tested, the total picture is skewed toward longertrends; that is, the results of trend periods from 55 to 100 days can bevery similar, while those of 5 to 50 days may each show very differentperformance.By viewing the percentage change in consecutive tests, it is evidentthat there should be fewer tests as the trend speed becomes longer. Table10-4 shows (1) Days, equal test periods, in days, for an exponential moving average; (2) %Change, the percentage change in the length of theperiod; (3) ExpSC, the equivalent exponential smoothing constant;(4) Equal, an equal distribution of smoothing constants, calculated assmoothing_constant = 2/(days + 1)
and (5) Days, the equivalent number of days corresponding to thesmoothing constants in column (4). The averages are at the bottom ofthe columns.Table 10-4 Distribution of Trends(1) Days
(2) % Change
(3) ExpSC
(4) Equal
(5) Days
5.00010.00015.00020.00025.00030.00035.00040.00045.00050.00055.00060.00065.00070.00075.00080.00085.00090.00095.000100.000
100.00050.00033.33325.00020.00016.66714.28612.50011.11110.0009.0918.3337.6927.1436.6676.2505.8825.5565.263
0.3330.1820.1250.0950.0770.0650.0560.0490.0430.0390.0360.0330.0300.0280.0260.0250.0230.0220.0210.020
0.3330.3170.3000.2840.2670.2510.2340.2180.2010.1850.1680.1520.1350.1190.1020.0860.0690.0530.0360.020
5.0005.3135.6596.0476.4816.9747.5358.1828.9349.82110.88212.17413.78115.83318.54822.30827.85736.87554.091100.002
0.177
19.115
Averages:52.500
17.739
0.066
Testing for Robustness
201
Column (2) shows that the percentage change is very large when theperiod is short. The average change falls near the 35-day test, althoughthe middle test is 50 to 55 days, indicating a large imbalance of longerperiod tests in which the change is very small. Figure 10-4(a) alsoshows the large changes in the faster trends, rapidly leveling off to verysmall changes for most of the remaining tests.
(a)
5 Through 100-Day Periods
(b)Figure 10-4. Trend distribution, (a) Equal test periods.Equal test period increments result in very different %changes, (b) Equal smoothing constants. Smoothing constants, which can be viewed as a percentage, show how thetest periods, in days, are closer together for faster trends.
202
Making a Trading Strategy Robust
A series of tests in which the trend speeds change by an equal percentage gives a much better sample of overall performance than equally spaced periods. An exponential moving average is an easier choicefor accomplishing this because an equal spacing of smoothing constants is the same as an equal percentage change. Column (4) has anequal distribution of smoothing constants, beginning and ending at thesame values as in column (3). Column (5) gives the number of daysapproximately equal to the smoothing constants in column (4), converted usingdays = (2/smoothing_constant) - 1
Figure 10-4(b) compares the pattern of the test periods in equal dayswith the pattern of equal percentages necessary to achieve an even distribution of performance.
• The number of trades will show whether there are enough trades tohave sound results. A rough idea of the accuracy is given bysample error = 1/@SQRT(number of trades)
• Maximum drawdown, on a day-to-day basis measures the peak-tovalley decline in equity, and gives the minimum capital needed fortrading. Although one test may have a smaller equity variation, measured by the standard deviation, the maximum drawdown canremain the same because both models were on the same side of asevere price shock. The model with the smaller standard deviation .shows a more acceptable equity variation during normal markets,but both require the same investment from peak to valley. It is oftenused for a worst-case scenario. Unfortunately, it is rarely the worstcase.• Risk-adjusted returns is the most important performance measurement. It compares standardized returns at the same risk level.• Percentage of profitable trades gives an indication of the consistency ofperformance. More frequent profits normally translate into less equity fluctuation. A very low percentage shows dependence on a fewlarge price moves. Each type of system, trend-following or countertrend, has a recognizable profile. Trend-following systems shouldhave from 35 percent to 45 percent profitable trades, while countertrend programs should exceed more than 60 percent successfultrades. Variations from these patterns should be examined closely.
Testing for Robustness
203
• Time to recovery, although similar to risk, gives a different interpretation. It measures the time between new equity highs. From a practical view, a larger equity drop but a very fast recovery may be preferable to a smaller decline with a slow recovery.Step 1 2 . How Will theOutput Be Presented?If you only saw the most profitable result from a set of 500 historic testsof various parameter combinations, you would have no idea whether thestrategy was robust. This chapter tries to stress that the combined performance of a wide range of parameters determines the level of confidence.Within this total picture, patterns of performance can be used formaking the final parameter selection. For example, positions heldlonger will normally have a higher profit per trade; other tests that limitrisk may show a better return/risk ratio.As the parameters that indicate trading frequency or risk controlmove from small to large values, performance should change in a continuous pattern. The presentation of test results can make the final parameter selection a much simpler task. Tests are commonly presented lineby line, giving the results of the first moving average speed and theincremented stop-loss, similar to the presentation in Table 10-1. Bychanging the form to a two- or three-dimensional chart, the resultsbecome much more useful.A Two-Dimensional Display. A bar or line chart is a two-dimensional display. It can show net profits or profits per trade versus trend speed.In Figure 10-5, line a shows that the profits per trade are erratic for a verysmall stop-loss and trend speeds under 20 days. Results become more consistent above 20 days. The center gray zone holds the best trends. Line bshows the profits per trade from the same trends speeds with a slightlylarger stop-loss. Results improve uniformly, but the original patternremains the same.The line chart in Figure 10-5 works for this example, but becomesunreadable when many lines are drawn for each stop-loss tested.Instead, a contour map (Figure 10-6(a)) shows the patterns clearly. InFigure 10-6(b), which holds the values plotted in the contour map, thetrend speed is the left scale and the stop-loss is along the bottom. Thefastest strategy, combining the shortest trend and smallest stop-loss,shows profits per trade of .07 percent in the upper left corner. The slowest strategy and the largest stop-loss give a much larger profit per tradeof .22 percent in the lower right corner. Clustered in the center are thepeak results.
204
Making a Trading Strategy Robust
15
20
25
30
35
40
Moving-AverageDays of test results. MovingFigure 10-5. A two-dimensionalpresentation
average results (line a) are more erratic using a small stop-loss. A largerstop-loss (line b) improves results keeping the same overall pattern.
The white areas in Figure 10-6 have the largest profits per trade, whilethe black areas have the smallest. It is easy to distinguish that the strategy improves as it moves away from the upper left corner, but falls offnear the center right side of the chart. If instead of these clear patterns,there were scattered peaks and valleys, the strategy would be erraticand risky. Other examples of contour maps can be found in the discussion of stops, Chapter 6.Smoothing the Table and Chart. In most cases, the contour map seemsto be a smooth display of results. However, an isolated peak or valley maymake it difficult to choose the best parameters using an automated selectionmethod. The results in Figure 10-6(b) can be smoothed by creating a newtable where each entry is the average of the nine boxes for which it is the center. Exceptions can be made for smoothing the entries on the sides using sixboxes and the corners with four boxes. Figure 10-7 shows that in this newgrid, the shaded box is the average of its surrounding group, includingitself. This 2-dimensional smoothing will help parameter selection. For larger tests or more smoothing, blocks of 5 X 5 or 7 X 7 can be used.
PART 3: Evaluatingthe ResultsUsing Averages and MapsThe average minus the standard deviation gives the Best Choice Index,which is simply the chance of picking a trading model that will producean average result. The contour map display can help locate broad areas
0.07
0.07
0.08
0.11
0.13
0.16
0.16
0.16
0.19
0.20
0.21
0.25
0.26
0.26
0.12
0.19
0.19
0.21
0.23
0.15
0.25
0.16
0.17
0.21
0.28
0.20
0.20
0.16
0.17
0.21
0.24
0.31
0.27
0.19
0.13
0.19
0.24
0.24
0.24
0.26
0.26
0.20
0.18
0.23
0.25
0.25
0.33
0.23
0.13
0.18
0.24
0.21
0.23
0.18
0.24
0.18
0.17
0.20
0.26
0.25
0.25
0.29
0.23
0.H
0.14
0.20
0.23
0.22
0.25
0.22
(b)
Figure 10-6. Contour map of test results, (a) This contourmap was produced by Mathsoft MATHCAD by importing thespreadsheet shown In chart b. The contour map of test resultsis similar to a topological map of a mountainous terrain. Areaswhere profitably jumps from high to low within a few testsresemble jagged, Irregular formations. Robust systems andareas of stable performance tend to have larger, more gradualcontour changes, (b) This chart shows the profits per trade of atrend system with a percentage stop-loss. The system buyswhen the trend turns up and sells when the trend turns down.A stop-loss is placed at the time of the original entry point andcauses an exit when the trade shows an absolute loss greaterthan the stop-loss. Once the trade exits, it does not reenter themarket until a new trend signal occurs.20S
206
Making a Trading Strategy Robust
Test grid
Figure 10-7. Smoothing the results of the raw tests. Asmoothed test table and contour map makes parameter selectioneasier. Each original entry can be replaced by (a) the square ofnine boxes, (b) the six boxes along the edge, and (c) the four corner boxes. The shaded box holds the new results.
of success and prevent the selection of a trading model that targets aprofit per trade too small for practical use. If the overall picture is good,the strategy is profitable, and results are smooth over most of the map,the chance of choosing a successful model is also good. The followingquestions will help qualify the results.Step 1 3 . Are the CalculationsCorrect?Before going further, step back and ask yourself whether you have checkedall the calculations. Did you manually verify a few lines in the spreadsheet?Did you calculate, in advance, the exact entry and exit prices for a numberof trades that used different rules? Do the answers look reasonable? Eventhe best analyst can make an error typing a formula. Do not waste time running hundreds of tests without verifying the results.Step 1 4 . Were There EnoughTrades to Be "Significant?"In Step 11, the sample error was given as sample error = 1/@SQRT(number of trades). Therefore, if there are only 16 trades, the error in the per-
Testing for Robustness
207
formance is ± 25 percent. It requires 400 trades to keep the error to 5 percent, considered the minimum acceptable size, but few systems producethat many trades. The only alternative is to be sure that the underlyingpremise is sound, and to produce as many sample trades as possible.Step 1 5 . Does the TradingSystem Produce Profits for MostCombinations of Parameters?What are the chances that any selection will be profitable? Are the patternscontinuous?A robust system must be broadly successful. When you look at thetest results, you should see mostly profits, and the Best Choice Indexmust be positive, giving an 84 percent chance of success. Use the average less 2 standard deviations to get the 97.5 percent level, and the average less 3 standard deviations to find the 99.5 percent level. The higherthe probability, the more robust. The contour map display should showcontinuous patterns, as in Figure 10-6(a). Jagged peaks and valleys maybe caused by specific rules that work in one test case but not others.Step 1 6 . Did Logic ChangesImprove Overall TestPerformance?When a new rule or calculation is added to the program, the results arerobust if they improve the Best Choice Index. This assures that thechange in logic was not pointed toward a specific event, but was a general improvement. A higher Best Choice Index occurs when the averageof all tests increases while the standard deviation does not increase, orthe average remains the same while the standard deviation decreases.A smaller standard deviation indicates improved consistency andmakes it easier to select successful parameters. These cases are shownin Figure 10-8.Step 17. How Did It Perform onOut-of-Sample Data?At least 10 percent of the test data should have been set aside. Even better, the 10 percent oldest and most recent data should not have beenused for testing. Once the trading strategy has been finalized, test thatdata separately and compare the average of all tests against the averageof the final tests of the longer set of historic data. Even in the best ofcases, you can expect profits to be lower and risk higher; however, thepattern should be similar to the tested profile.
208
Making a Trading Strategy Robust
(a)
(b)
Figure 10-8. Selecting a robust system using the performance curve and Best Choice Index, (a) When the performancecurve flattens and widens, the results get worse. The averagereturns remain the same, but the standard deviation gets largercausing the Best Choice Index to drop, (b) When the averageshifts to the right or left, the overall performance gets better orworse, as long as the standard deviation remains the same.
Testing for Robustness
209
Results of the out-of-sample test that are very different from the othertests must be reviewed carefully. Poor results indicate that the strategyis not working. The use of a chi-square test (see Chapter 11) will showwhether this failure is part of the long-term performance profile or indicates that something is wrong. You may have an error in the rules or calculations, but that should have been corrected long before this point.OrA the test period might have been too short, resulting in unstableresults.Feedback Dilemma. Once you have used the out-of-sample data toverify the system, you can no longer use that data again. Inspecting thetrades and adding rules may produce a valid improvement, but you havemade it work in the "unseen" data; therefore, you have no way to checkthe results. You might include the new data and omit some other piece;however, the reliability of the results has dropped.
Part 4: Choosing theSpecific Parametersto TradeThe final section of a trading model is a combination of profits, risk, andpersonal preference. A program that holds trades for weeks may produce the highest profits per trade but may not meet the investor's shortterm objectives. Even though individuals may choose differently, themost robust systems offer the best platform from which to select. Thissection asks questions that are important, regardless of your specificgoals.Generally, selecting from models that hold positions longer givesmore dependable results. It is also more difficult to assess the expectedreturns from faster trading models. Figure 10-9 shows the hypotheticalresults of a trend system, where the fastest trading model is posted atthe left. Performance is erratic although a smoothed line can give a better idea of expectations. In actual trading, the 6-day trend may capturethe next big profit, while the 4- and 8-day trends post losses.A comparison of fast and slow strategies shows that:• Faster trading is more sensitive to current market patterns.• Faster trading gives up a large percentage of profits and losses totransaction costs.• Faster trading may have the same large losses due to price shocks, butthese losses will be a much larger than typical profits and losses.
210
Making a Trading Strategy Robust
Regardless of the trading strategy, taking the long-term view is themore conservative, reliable approach. Although the long-term strategymay have larger absolute losses, it often has a better return/risk ratiothan faster programs. This does not mean that you cannot have a systemthat works well trading fast. The performance must be high when youdraw the smooth line through the irregular results. You must alsoexpect real returns to be erratic. Tests plotted in Figure 10-9 show thatresults can vary significantly from expectations, especially with fasttrading methods. You should expect real returns to vary even more thanthe tests show.Step 1 8 . Did the Last TestInclude the Most Recent Data?Having reserved some data for out-of-sample testing (see Step 17), theprogram should be retested using all data. This is particularly important if the out-of-sample data is the most recent. Once the model is operational, retesting should be performed whenever 5 to 10 percent newdata is available, or unique market patterns occur. The model may beadjusted by a small amount, but it will become ever so slightly morerobust.Step 1 9 . Did You Choose from anArea of Broad Success?Was it the slow selection?The contour map shows whether the performance of the strategy hasa smooth or irregular pattern with respect to parameter changes. Theareas of broad success show stability and are often associated withslower trading models.A choice of a faster strategy must be justified by a larger profit pertrade and reasonably high reliability to compensate for inherent uncertainty. The worst-performance case in the neighborhood of the selectionshould still be acceptable. Figure 10-9 shows that erratic results associated with short-term trading should be considered as smoothed whenselecting from this region.Step 2 0 . Are Profits DistributedEvenly over the Tested History?Study the trades and equity of the final model to see whether profits andlosses alternate in a reasonable pattern. A standard deviation of theequity changes, time to recovery, and other statistical measures give the
Testing for Robustness
Figure 10-9. The typical results of a trend-following strategy optimization. By selecting the peak profits, or return/risk ratio, results oftenfavor isolated returns of short-term trends. The chance of repeating thisperformance in actual trading is very slim. The smoothed line is the mostlikely return.
relative merits of one test against another, but only a visual study isgood enough before you begin trading. It may be helpful to look atquarterly results to see consistency.Step 2 1 . Are the Profitsper Trade Large Enoughto Absorb Errors?When two tests have similar risks and returns, the best choice is the onewith the largest profits per trade. Larger profits absorb unexpectedproblems (e.g., slippage in a fast market) that result in lost profits whenan order cannot be executed. Establish a minimum acceptable profit pertrade.Step 2 2 . Did the Historic ResultsShow Any Large Losses due toPrice Shocks?Price shocks are unpredictable events. Your program should have anequal number of losses as it has profits due to price shocks, although
212
Making a Trading Strategy Robust
some may be controlled by a stop-loss. Check the obvious past priceshocks against the system trades. If the system profited from all of them,or avoided the losses, the results are overfitted or just lucky. You cannotexpect the program to profit from unpredictable events in the future.The danger of trading a system which has not shown losses from priceshocks is that the risk is unreasonably small. This leads to greater leverage and large losses.Step 2 3 . Have Ton Risk-Adjustedthe Returns to Tour AcceptableRisk Level?The return/risk ratio turns absolute performance into relative returnsand allows the fair comparison of each model. Traders however mustestablish their own acceptable risk level. Decide, for example, that youare willing to take a 1 percent chance of losing more than 10 percentduring any month. Then the system you trade must show a risk (measured as 1 standard deviation of the monthly equity changes) of lessthan 3V3 percent. Three standard deviations will be 10 percent.Remember that equity changes based on monthly data are alreadysmoothed. You can expect larger mid-month equity fluctuations, sometimes as much as 50 percent greater.
P u t 5: Trading andMonitoring PerformanceNo amount of testing can substitute for trading. As soon as the firstposition is set, you may realize that the transaction costs used in testingwere too low, you cannot execute the full position in the cash marketafter the New York close, or that a breakout signal produced liquiditygaps. Monitoring the system signals against actual trading providesinformation that will continue to improve the testing process.Step 2 4 . Are Ton Following theSame Rules That Were Tested?Real trading results often vary from test results because the rules usedin testing are not followed. The size of the transaction costs or the liquidity of the market may also prevent you from executing the full position. Most often, it is the execution technique. By waiting until after thecomputer has given a trading signal, the trade price and the theoreticalcomputer signal are far apart. This is solved by anticipating the com-
212
Making a Trading Strategy Robust
some may be controlled by a stop-loss. Check the obvious past priceshocks against the system trades. If the system profited from all of them,or avoided the losses, the results are overfitted or just lucky. You cannotexpect the program to profit from unpredictable events in the future.The danger of trading a system which has not shown losses from priceshocks is that the risk is unreasonably small. This leads to greater leverage and large losses.Step 2 3 . Have Ton Risk-Adjustedthe Returns to Tour AcceptableRisk Level?The return/risk ratio turns absolute performance into relative returnsand allows the fair comparison of each model. Traders however mustest