ablish their own acceptable risk level. Decide, for example, that youare willing to take a 1 percent chance of losing more than 10 percentduring any month. Then the system you trade must show a risk (measured as 1 standard deviation of the monthly equity changes) of lessthan 3V3 percent. Three standard deviations will be 10 percent.Remember that equity changes based on monthly data are alreadysmoothed. You can expect larger mid-month equity fluctuations, sometimes as much as 50 percent greater.
Part 5: Trading andMonitoring PerformanceNo amount of testing can substitute for trading. As soon as the firstposition is set, you may realize that the transaction costs used in testingwere too low, you cannot execute the full position in the cash marketafter the New* York close, or that a breakout signal produced liquiditygaps. Monitoring the system signals against actual trading providesinformation that will continue to improve the testing process.Step 2 4 . Are Ton Following theSame Rules That Were Tested?Real trading results often vary from test results because the rules usedin testing are not followed. The size of the transaction costs or the liquidity of the market may also prevent you from executing the full position. Most often, it is the execution technique. By waiting until after thecomputer has given a trading signal, the trade price and the theoreticalcomputer signal are far apart. This is solved by anticipating the com-
212
Making a Trading Strategy Robust
some may be controlled by a stop-loss. Check the obvious past priceshocks against the system trades. If the system profited from all of them,or avoided the losses, the results are overfitted or just lucky. You cannotexpect the program to profit from unpredictable events in the future.The danger of trading a system which has not shown losses from priceshocks is that the risk is unreasonably small. This leads to greater leverage and large losses.Step 2 3 . Have Tou Risk-Adjustedthe Returns to Tour AcceptableRisk Level?The return/risk ratio turns absolute performance into relative returnsand allows the fair comparison of each model. Traders however mustestablish their own acceptable risk level. Decide, for example, that youare willing to take a 1 percent chance of losing more than 10 percentduring any month. Then the system you trade must show a risk (measured as 1 standard deviation of the monthly equity changes) of lessthan 3% percent. Three standard deviations will be 10 percent.Remember that equity changes based on monthly data are alreadysmoothed. You can expect larger mid-month equity fluctuations, sometimes as much as 50 percent greater.
Part 5: Trading andMonitoring PerformanceNo amount of testing can substitute for trading. As soon as the firstposition is set, you may realize that the transaction costs used in testingwere too low, you cannot execute the full position in the cash marketafter the New* York close, or that a breakout signal produced liquiditygaps. Monitoring the system signals against actual trading providesinformation that will continue to improve the testing process.Step 2 4 . Are You Following theSame Rules That Were Tested?Real trading results often vary from test results because the rules usedin testing are not followed. The size of the transaction costs or the liquidity of the market may also prevent you from executing the full position. Most often, it is the execution technique. By waiting until after thecomputer has given a trading signal, the trade price and the theoreticalcomputer signal are far apart. This is solved by anticipating the com-
Testing for Robustness
213
puter signal. To be a successful system trader, you must execute at thesame time the system is executing. Chapter 11 shows how to anticipatea computer signal.Step 2 5 . Are You Trading theSame Data That Was Tested?Although it is convenient to test a strategy using a continuation or "perpetual" contract, the results will not be the same when you trade cash orfutures contracts. Be sure that you are trading the same market that wastested, and that you tested the same market you are trading.Step 2 6 . Are Ton Monitoring theDifference between the Systemand Actual Entries and Exits?Understanding how to test a strategy comes from identifying why testing and actual trading results are different. Monitor the theoretical signals, real executions, and the percentage of trades that cannot be executed, then retest the strategy with these improved values. In time, youwill be able to show very realistic test results.
Other Important PracticalGuidelinesEven the most careful, responsible testing cannot show how the systemwill perform when it is traded. From the preceding guidelines, experience shows how the following points should be highlighted:• Slower systems, those using longer periods of evaluation, performcloser to expectations than faster models.• Avoid systems that do not show downside risk. Absence of risk is anindication of overfitting or a coincidental good fortune that is notlikely to be repeated in trading.• View test results as a smoothed line. In a robust system, expect peakresults to be lower, and poor performers better, both moving towardthe average when traded.• Avoid systems with low reliability. They may indicate dependenceon a few exceptional trades, rather than steady performance.• Avoid systems that have only a few trades. They may not yet show anaccurate picture of results.
214
Making a Trading Strategy Robust
More Data GiveMore Predictable ResultsIt is worth repeating the importance of using more data, rather than lessdata, for testing. More data contain more price patterns, sustainedmoves, and price shocks. Many people argue that old data lack relevance—markets have changed. In specific cases, and for some applications, that is true. It is safer to assume that there is more danger thanbenefit in using small amounts of data.A system tested over the past 3 years will not see the largest priceshocks of the recent 10 years. Yet you must expect that even largershocks will come. If you capitalize an investment according to recentrisk, you will not survive for long. The greatest failure in trading isundercapitalization, and this is the direct result of unrealistic expectations of risk. If recent data are best for maximizing profits, more data arebest for risk evaluation.It is possible to test a strategy twice, once for parameter selection andonce for risk. Because tests of more years of data show lower profits andhigher risk, they are not viewed as desirable. It is much more pleasingto choose from the high returns and low risk of shorter test intervals.But the reality is that the longer tests are more representative of realtrading results. Choosing to ignore these results does not producegreater profits.Start by Knowing the AnswerThe best use of computer testing is to verify a theory. If your idea is good,then testing various time intervals, entry and exit criteria, and risk management parameters should show reasonably consistent returns. It mayshow that your theory is good for short-term patterns, but not for thelonger view; however, it should verify your idea. A concept based on anunderstanding of the market—whether economic, statistical, or pricepatterns—is a valid, valuable basis for a system and the best way tobegin the development of a trading program. Feeding a test package amultitude of indicators, rules, and prices series, and letting it crunchaway until it combines them into a profitable result, has a very lowchance of being a successful trading system.Errors of Omission"Survivor bias" and the failure to apply a worst-case scenario are twoproblems classified as errors of omission. Omissions constitute an unseentrap for analysts. It is far easier to account for odd patterns and priceshocks than to consider situations that do not appear in the data.
Testing for Robustness
215
Survivor Bias. The selection of certain stocks, funds, arid investmentmanagers for testing unconsciously omits the worst cases—those wherethe company or manager went out of business. A classic case of survivorbias is in the review of investment managers. The one who generates thehighest profits may have the highest risk. If you review only those managers currently reporting, you do not find out that all managers with comparably high risk were previously forced out of business by losses. Thesecomparisons result in unrealistically low risk.Similarly, the selection of specific stock issues means that those firmshave not seen the patterns that precede failure. Even the largest firmsare no longer as secure as we once thought. Drexel Burnham, E.F.Hutton, Stotler, and the Pennsylvania Railroad (also the Penn-Central,with the most assets of any company in the United States) proved thatmismanagement and litigation can ruin even the biggest. IBM, the autogiants, and insurance companies no longer look inviolate. It is difficultto assess risk properly if you only study the winners.Worst-Case Scenarios. More difficult, yet just as important, is the ability to conceive "worst-case scenarios." What might cause a market to go tonew high prices, fall to new lows, or become twice as volatile as the worstperiod in history? If this happens, what steps do you take to stabilize risk?Or, do you remove those markets from your portfolio? Will the tradingstrategy perform properly if prices move to levels not seen in historicdata? Will previously uncorrelated markets move together?These scenarios are critical to risk control. Often, there are no immediate answers to these hypothetical cases, but only a general confidencethat the current strategy has the flexibility to adapt to market change.That is not always enough. A sharp drop in one market can force a needfor capital, causing investors to liquidate unrelated assets to finance thelosing ones. This results in a broad reversal in many investment areas.Data IntegrityThe assumption that a historic data series is correct can result in atremendous loss of time. All data should be scanned for gross errorsbefore being used. Data received electronically or on disk from a reliable vendor may still have problems. Testing and evaluating a systemtakes time. To find a data error after weeks of work means that all thetesting must be done again. A few fast steps can avoid that aggravationand cost.1. Look at a price chart of all the data to be used. Any serious data problem will be obvious.
216
Making a Trading Strategy Robust
2. If you have strategy-testing software, identify opening, high, low, orclosing prices that were greater than 3 percent from the previousprice. Look at those days one at a time. Many of them will be errors.3. When the final model has been selected, look at the profits and losses of each trade. Be critical of the largest profits and losses; verifythat the entry and exit prices are reasonable.Patching the ProblemsTrading strategies succeed by generalization. Most plans are profitablebecause they grind out larger profits than losses. The problem with ageneral or statistical solution is that it is blind to specific cases, but thetrader is well aware of the reasons for big price moves.Each major move and price shock can be explained. By carefullystudying the cause and patterns of larger trading losses, indicators andrules can be combined to control the losses, leaving a more profitableperformance profile. But the next big move is always different. They canbe explained in retrospect, but rarely fit a prescribed pattern. Explainingeach loss has intellectual satisfaction but falls short of reducing tradingrisk. Fixing each case based on its own features is still "overfitting."Do Not Oversolve the ProblemA young analyst, trying to do his best, produces an answer to four decimal places, when each of the inputs had only two places of accuracy.You cannot create more accuracy than you have. Technical models,based on either price patterns or statistics, do not depend on one pricemove or a single trade. They succeed over a large number of events.Fine-tuning a moving average can be counterproductive because itmoves away from the general solution. A specific trend speed thatavoided a large loss has no way of avoiding similar losses in the future.Oversolving or overtesting produces unrealistic expectations of systemperformance.Accuracy and Test Time. For most system tests, there is a direct relationship between accuracy and calculation time. The more time it takes,the better the result. Is it better to test exponential smoothing constants insteps of .1, .01, or .001? There can be 10,100, or 1000 tests in the range .1 to.9, based on the test increment. But 1000 tests is wasted accuracy, just astesting stop-losses in $5 total investment increments is naive.Is it important if the one that was not tested showed twice the profitof the two adjacent tests? If you are still trying to find peak profits rather
Testing for Robustness
217
than the best system or contour, then you are wasting your time. Trendsare intended to smooth data. Fine-tuning a trend seems inconsistentwith the concept of smoothing. If you select a 154-day moving averagebecause a large loss was averted, while a 153-day average was caught,you have a basic misunderstanding about the implied accuracy of a system.New powerful computers with increased speed have made it painlessto run large, meaningless tests. When computers were slower orresources limited, it was necessary to reason out the benefits of eachhour spent on the machine. The "broad-brush" approach may still bethe most efficient use of time and a way to prevent overfitting.
SummaryThe method of finding a trading strategy can increase or decrease yourchance for success. Using sound procedures and statistical methods issafe and conservative. This includes long data series that encompass asmany unique situations as possible. In addition, global statistics, whichaverage all the tests, are an excellent measure of a robust system andprevent the temptation to seek high-profit simulations. When usingaverages, it is clear which strategies and new techniques are best.
11Improving thePerformance ofExisting SystemsAfter the completion of tests, measuring and monitoring performancecontinues the process that results in a successful trading program. Acareful comparison of actual results with expectations shows how wellthe testing was performed. Because there can be a tremendous gapbetween testing and trading, it should not be surprising that results aredifferent. To be successful, however, they cannot be so divergent thatexpected profits are turned into real losses. It will be necessary to tradea system to know whether the testing assumptions were realistic. Afterthat, it is necessary to figure out how to improve the program whilemaintaining its integrity. The following sections discuss some of theseimprovements.
Measuring and MonitoringPredictabilityThe most important part of performance monitoring is to discoverwhether you have correctly measured the risk of loss, and to find out assoon as possible. There are many ways to proceed when there are trading profits, but only one choice when losses are larger than expected.Monitoring actual results gives the only accurate assessment of expectations. Before that, we can only estimate. Burdening tests with slippageand other costs that are too big will make good strategies look bad and219
220
Making a Trading Strategy Robust
increase the time and effort needed to find a good trading method. Onthe other hand, expecting larger costs is safer than underestimatingthem.What Do Ton Monitor?We always monitor trading to find the difference between expected andactual results. Expected performance comes from testing; actual resultscome from trading. Actual results are not accurate if they reflect thetrading of a small position when the intention is to trade a large one.Deltas. The difference between an expected and an actual value will becalled a delta (shown as the symbol A). There can be execution deltas tocompare fill prices, and total performance deltas, as follows:• Record the difference between the program's estimated trade priceand the actual average execution price. Separate the entry and exitvalues, because there is often more finessing of entries:long entry A = (system entry - actual entry)short entry A = (actual entry - system entry)long exit A = (actual exit - system exit)short exit A = (system exit - actual exit)
• Record the system profits and losses from those trades not filled at all(unables).• Calculate the total unit profit and loss for the system and actualtrades:total unit A = (total actual P/L - total system P/L)/number of contracts traded
Although the most important value is the unit difference between thesystem's expected profit or loss and the actual results, the breakdown ofthose values provides information to help improve executions. The unitdifference should be used to estimate future results, if nothing changes,and to use realistic transaction costs in testing. The other values are allevident. Chapter 2 discussed the impact of unables (those orders notfilled) on profitability and showed that unables reduce profits but notlosses, increasing the diffculty of trading a program successfully.
Improving the Performance of Existing Systems
221
LiquidityExecution problems can sometimes be caused by the type of order usedto implement the trading system, but they are all related to market liquidity. If your order is too big for the market at the moment it hits, thenthe execution is bad. It may be necessary to average in over a few hours,or shift the orders closer to the opening or closing of the trading session.Limit orders may need to be replaced by market orders spaced over alonger period. One thing is certain: If you do not execute each ordergiven by the trading program, you cannot expect to achieve its results.FeedbackMonitoring performance is the only way to find the real cost of trading.It is valuable information and can be used for testing other systems.Although different types of orders have their own peculiar costs, all ofthem have some slippage, and all of them have unables. It is mostimportant that we know what they are.Chi-Sqnare TestIn the spirit of simplicity, it does not require high-powered mathematics to know that your trading is not going as planned. A loss that is larger than any historic one, or a series of losses longer than any before, issure to get your attention. But not all situations are clear until theybecome a problem. The chi-square test is a simple way to compare historic (expected) and trading (actual) results to find out whether something is wrong.For example, your new system has had 20 trades and only 4 wereprofitable, but historic testing showed that 40 percent of the tradesshould be profitable. What are the chances that something is wrong?Use the chi-square test:chi-square = ©SUM ((actual - expected) A2/expected)
The test is the sum of the percentage difference in the actual versusexpected results. When the two numbers are very close, the chi-squarevalue is small. To find out whether a chi-square value is large enough toindicate a problem, Table 11-1 must be used.If you want to compare the frequency of profits, the expected profitfrequency and the expected loss frequency are both used because thechi-square test requires a minimum of two cases:
222
Making a Trading Strategy Robust
Table 11-1. Distribution of Chi-SquareProbability iof Occurring by ChanceCasesLessl
.70
.50
.30
.20
.10
.05
.02
.01
.001
12345678910
.15.711.422.203.003.834.675.536.397.27
.461.392.373.364.355.356.357.348.349.34
1.072.413.674.886.067.238.389.5210.6611.78
1.643.224.645.997.298.569.8011.0312.2413.44
2.714.616.257.789.2410.6512.0213.36. 14.6815.99
3.845.997.829.4911.0712.5914.0715.5116.9218.31
5.417.829.8411.6713.3915.0316.6218.1719.6821.16
6.649.2111.3413.2815.0916.8118.4820.0921.6723.21
10.8313.8216.2718.4720.5222.4624.3226.1327.8829.59
chi-square =
A(actual profit freq - expected profit: freq) 2expected profit freq
(actual loss freq - expected loss freq)A2+expected loss frequency
where actual is the real trading performance and expected is the testedresult. The expected profit frequency and expected loss frequency musttotal 100 percent:chi-square =
(20-40)*2(80-60^2__—+—= 16.674060
Referring to Table 11-1, we compare the results of 16.67 with the firstline because there are two cases. We see that 16.67 is greater than thevalue associated with .001 and is considered highly significant.Therefore, there is something wrong with the system if it shows a 20percent trading reliability. But we have not yet considered the numberof trades. Based on 20 trades, the sample error would be 1/@sqrt(20) =.22, or 22 percent. Then the value of 16.67 could fall to 13.00, still above10.83. Certain levels are considered important for the chi-square test:chi-square > .001 probability, then it is highly significant.chi-square > .01 probability, then it is significant.chi-square > .05 probability, then it is probably significant.
The chi-square test can show whether the actual pattern of price runs(the number of sequences of moves in the same direction) compares
Improving the Performance of Existing Systems
223
Table 11-2. Chi-Square Evaluation of Price Runs
with a random distribution. Table 11-2 gives the columns of a spreadsheet and the total of the column (E - A)A2/E is the value of chi-square.For eight cases, Table 11-1 places the chi-square value at a level indicating more than a 50 percent chance of the pattern of runs being random. As more cases are included, the statistic shows that the comparison gets closer.
AnticipationTheoretical profits can only be realized with anticipation. Chapter 2tried to point out that screen lag, slippage, and unables could easilychange a theoretically sound trading strategy into a losing venture. Onesolution offered was to target profits per trade that are large enough toabsorb the loss. Another way is to anticipate the trading signal. Be prepared to execute an order at the exact time the technical system gets itssignal, rather than waiting until you get a confirmation. Even better,execute the order just ahead of the computer.To show the importance of anticipating signals, consider a movingaverage system using the closing prices. The system buys when thetrendline turns up, and sells when the trendline turns down. Figure 11-1and Table 11-3 compare performance of a selection of moving averagespeeds for entries taken on the same day as the moving average calculation with executions on the close of the next day. Three very differentmarkets were tested, the Hong Kong Hang Seng Index, the Deutschemark, and IBM. The trend speeds covered a reasonably broad range of 5to 75 days.The results form a clear pattern. Faster trends lose from 50 to 400 percent of their profits when entries are delayed for one day. Slow trends
224
Making a Trading Strategy Robust
Pignre 1 1 - 1 . Comparison of 1-day lag performance. The three very different markets show similar results from a 1 -day entry lag of a trend system. Fasttrends have much worse performance while slow trends are not affected.
Table 11-3. Comparison of Same Day and1-Day Lag PerformanceHang Seng
Deutsche Mark
IBM
MovingAverageDays
SameDay
NextDay
PercentChange
SameDay
NextDay
PercentChange
SameDay
Next PercentDay Change
515253545556575
106.882.755.943.437.8. 18.610.216.3
55.244.122.312.27.918.619.333.1
(48.3)(46.6)(60.0)(71.8)(79.1)0.290.3102.9
10.023.035.921.338.3.32.121.616.3
(10.7)
(206.8)(105.3)(110.0(160.9)(91.8)(132.2)(55.7)(55.0)
38.78.79.715.018.524.924.038.4
4.9(25.1)8.69.219.914.522.832.7
(1.2)(3.6)(13.0)3.2(10.3)9.67.3
(87.4)(387.1)(11.5)(38.6)7.5(42.0)(5.1)(14.8)
are not affected. This shows that timing is critical for the 5-day movingaverage and that the first day holds the largest profits: Hang Seng profits dropped 48 percent, the Deutsche mark fell from a 10 percent profitto a 10 percent loss, and IBM lost 87 percent. Longer trends are not asdependent on a specific entry price, and while the first day may be profitable, it is not a large percentage of the total profit. The profits shown
Improving the Performance of Existing Systems
225
in the 75-day Hang Seng due to the delay should not be expected. It ismost likely that it is simply a distortion due to fewer trades.It seems reasonable that any short-term trading that depends onmomentum, or a burst of price movement, to generate a buy or sell signal, will be hurt by a delayed entry. By knowing the price, in advance,at which a trading system will get a new signal, it is possible to eliminate a number of problems. The most important is the ability to executeat the system price, not afterward. Small orders could be placed as stopsor resting orders.The need for better executions with faster trading implies that thebulk of the profits occur early in the trade. This supports the earlier discussion of profit-taking, which argues that holding a trade until thetrend reverses causes the return/risk ratio to drop. The amount of profit compared with the risk gets worse as you hold the trade longer.Windowing Large OrdersLarge orders can be executed by creating a window around the systemcalculation time, and entering orders throughout that window. Forexample, a forex trader has a momentum program based on hourlydata. Once the 11:00 calculation has passed, he knows that a new buysignal will occur if prices are above 156.50 at 12:00. Because his experience has shown that an order of 25 million $/sterling should only take11 minutes to fill at that time of day, he starts buying at 11:55 if the priceis safely above 156.50 at that time. If the executions take 10 minutes, theaverage price should be close to the price at 12:00, when the computercalculates its signal and posts an entry price.As the hour nears when the system signals are calculated, it usuallybecomes clear whether or not the order should be placed. Sometimes,prices are right at, or just below the (buy) signal price and you are notcertain whether the order should be placed. If you start buying andpush prices higher, then you force the system signal yourself. Yet waiting until after 12:00 might mean getting a much worse fill.The fact that anticipation greatly improves returns tips the balance infavor of executing marginal cases. You begin filling the order slowly,watching for the 12:00 price. If it gives a signal, you finish filling theorder; if not, you reverse the position as quickly as possible. Exiting a"false anticipation" is less costly over the long run than waiting untilafter the signal to begin executing the order.A false anticipation can occur at any time. Prices can seem safelyabove a buy signal level, then plummet in the 60 seconds before 12:00,even while you are buying. Once the 12:00 price is fixed, you knowwhether to continue or reverse the positions that have been set.
226
Making a Trading Strategy Robust
Calculating the Anticipated PriceFinding the signal price in advance is straightforward, but it requiressome algebra. You write the formula for the moving average, RelativeStrength Index (RSI), or other indicators based on the next period price(e.g., tomorrow for daily data), then solve for the next price. For example, a 5-day moving average for today is:@ moving_average(price,5)= (price + price[1] + price[2] + price[3] + price[4])/5
Using the function @sum(price,days), which sums the previous n daysof price, we could shorten this to:@moving_average(price,5) = @sum(price,5)/5
where price is the last price, price[1 ] is the prior price, and so forth. Usingreal numbers, we get:@moving_average(price,5) =
(154.50 + 153.20 + 153.60 + 152.70 + 152.50)/5= 153.30What price is needed for tomorrow's close so that the moving averageturns down by .10? The new value of the 5-day moving average @moving_average would need to be equal to 153.20. By moving the calculationforward one day, we can find today's value using simple algebra:@buy_signal = 153.20 = (next_price + @sum(price,4))/5then next_price = 153.20*5 - @sum(price,4)
which solves for next_price by multiplying both sides by 5 and subtracting the sum of the four known values from both sides. The values@sum(price,4) are the most recent four prices. Using the sample pricesgives:766.00 = next_price + 154.50 + 153.20 + 153.60 + 152.70152.00 = next_price
Therefore, the moving average turns down by .10 if the price of the sterling closes at 152.00 or lower. Box 11-1 gives a few common formulasand the calculations for anticipating the next price.
Improving the Performance of Existing Systems
227
Quote Equipment withProgrammed StudiesComplicated calculations are not necessary if you use one of the manypieces of quote equipment with preprogrammed studies. TeleTrac,TradeStation, MarketView, CQG, and many others already calculate moving averages, stochastics, and other indicators in a way that allows" youto change the number of periods in the calculation. The last price isautomatically used to find the next value, therefore the machine is constantly telling you whether you will get a signal at the next 15-minute,hour, or daily interval. Institutions that plan to customize the processwill find the formulas in Box 11-1 to be helpful.
Filtering System SignalsTrading risk increases with high prices and high volatility. Becausethere are so many unique strategies and time frames, the "high" levelassociated with risk that is "too high" is likely to be different for eachone. One approach to controlling risk is to use a protective stop; however, the market can jump through your risk level at exactly the timeyou need that safety the most. Financial stops based on personal riskpreferences have been discussed as ineffective protection. And, whilelogical stops (based on significant support and resistance levels, or outside factors such as economic indicators) may reduce day-to-day risk,they cannot protect the loss due to a price shock. Whether you intend tohold a trade for an hour, a day, or a year, a price shock will produce thesame loss if you are unlucky enough to be in the wrong position.Filtering Price LevelsOften, a pattern of trading performance is associated with entering amarket at a significantly high or low price. And, while the term "low" isreasonably clear, "high" is not obvious. Physical products can be considered low at levels equal to the lowest prices seen during the past 10years, or at the cost of production. Controlled markets