2-dimensional table), with the moving average days along theleft (rows) and the stop-loss values at the bottom (columns). Figure 21-4 shows that thelowest number of days and the smallest stop-loss value will give the profit/loss value in thetop left corner box.
By presenting the results as shown in Figure 21-4, there is a continuity of performancein all directions. The upper left corner represents the fastest system—the one with themost trades; the bottom right corner shows the results of the slowest strategy. The patternsof the upper right may be similar to those of the lower left if the system displays some sym-inetry. A faster trend speed with a large stop (top right) and a slow trend with a small stop(bottom left) are likely to have the same number of trades and similar profitability.
Depending on the data used for testing as well as the trading rules, three patterns aremost likely to appear in the results of a 2-dimensional display. Figure 21-5a shows the sim-plest case of a single area of successful performance gradually tapering off. This is analo-gous to the single-parameter test shown in Figure 21-3a. Selecting the parameters that512 TESTINGPee
 
FIGURE 21-4 Standard configuration for a 2-dimensional optimization.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2 +A4 Lfge L_I@|e +—sSel | | ||2 10 L_| |3 122Ss 143 t= x16 SF18 *++ —20LL .12 3 4 5 67 8 9 10
Percentage stops
 
gave the center of the best performance area is the most reasonable, because moderateshifts in price patterns are still likely to be profitable.
The appearance of two areas of profitability are usually the results of a highly volatilemarket (Figure 21-56). Prices that are moving quickly and sustain a_major trend can betraded using a fast or slow model. The fast model is often more profitable because it cap-tures more of the rising trend; it reacts faster and also profits from the decline. The slowertrend-following approach captures less of the price move but keeps clear of the sharp reac-tions that stop out the middle-speed trends. The selection of parameters to use in real trad-ing follows the same reasoning applied to Figures 21-38 and c. The third case, shown inFigure 21-5c, is one of erratic profits and losses. The absence of a consistent pattern in theperformance indicates that the trading strategy does not apply to the data.
Alternate Ways of Visualizing Results
With software such as MathCad, you can see the 2-parameter test as a contour or topolog-ical relief map. For some users this form can make it much easier to see the peaks and val-leys of performance and is highly recommended. Any spreadsheet program will allow asurface plot, which will give you a rough look at the continuity of a 2-parameter test, asshown in Figure 21-6, This 3-dimensional graph gives the net profits of a moving averagecrossover system applied to the Swiss franc. The slower trend is shown along the frontscale and the faster one along the right. There is a ridgé running from the front to the back,indicating that a slow trend of about 50 to 60 days dominates the results. Other than thefastest trends of 3 days, it is easy to see that there are many combinations of fast and slowMoving averages that are historically profitabie.
Visualizing with Scatter Diagrams - .To see the performance of more than two variables, you can plot the results of all testsagainst each parameter (see Figure 21-7) using a scatter diagram (also called an XY Pfot).VISUALIZING ANL INTERPRETING THE RESULTS 513
eeFIGURE 21-5 Patterns resulting from a 2-dimensional optimization. (a) Single profitablearea. (b) Two distinct profitable areas. (c) No obvious pattern.
Stop-loss (points of percent)
 
B:g:&:e:3 2oo:2g vt + Marginally
successful
UnsuccessfulStop-loss (points or percent)
zs§: Marginallys successful -£33
 
(by514 TESTING
 
 
FIGURE 21-5 (Continued) ~
 
Successful
— wens successful
Unsuccessful
  
This will give you a better understanding of how each value affects the total picture andallow you a way to visualize the robustness of each variable. Using a 2-variable example, arange of fast and slow moving averages were tested for the Swiss franc based on a crossoverstrategy. The results of the slow moving averages are plotted for all combinations of the fastmoving average in Figure 21-7; the fast average is shown in Figure 21-7.
The individual test results are shown as circles above the slow or fast moving averageperiod in Figure 21-7. For the slow moving average, it is clear that profits are greatest when50 and 60 days are used for the calculation period; however, it is not as clear which ofthose two values is the best choice. The 60-day moving average posted profits higher thanthe 50-day average, but also showed profits below the lowest 50-day returns. A closer lookshows that the average returns of the moving averages over those periods is about thesame, but the 60-day moving average has a higher variance. Normally, the bestchoice is theresult that is most stable (the 50-day calculation), although in this example we must be cau-tious that the small number of tests do not make the 50-day results look stable just bychance,
The results of the faster moving average, seen in Figure 21-76, show much greater vari-ance. The 7-period results have a very wide range including the highest returns; however,the averages of the 7th through 15th periods are very similar. With variance in mind, the13-day calculation seems to be the best candidate. We can be confident that this method isrobust, because nearly all combinations of fast and slow moving averages gave profitablereturns, net of a $50 commission.
Standardizing Test Results “
When testing a single system, the results of one set of parameters can easily be comparedwith other sets because the test period, commissions, and other basic values are all thesame. However, over time you will test many different systems and variations of those sys-tems, and you will want to compare them to decide which’ strategy is best. This requiressome advanced planning. -
Standardizing test results is the best way to increase the usefulness of extensive testing.This should include;VISUALIZING AND nvTERPRETING THE RESULTS 515
 
 
FIGURE 21-6 Three-dimensional map of crossover model.
IMM Swiss franc, 1986-1996
Net profit/loss
 
 
1, Annualizing all values, Tests that you perform 6 months or 1 year apart are likelyto use different time periods. Annualizing the results will make comparisons easier.
2, Risk adjusting. A profit of $10,000 with a drawdown of $5,000 is effectively thesame as a profit of $20,000 and a drawdown of $10,000, given adequate reserves,Expressing the results as profits divided by risk can be useful. -
3, Adjusting for standard error. The important difference between two tests that yieldthe same results, one with 2 trades and one with 50 trades, is that the one with only2 trades isa less reliable result. If both posted 50% profitable trades, another losingtrade would make the reliability of the second test drop to 49%, while the first casewould drop to 33%, Instead, results should be presented at the same confidencelevel, which means subtracting the standard error from the current result. Analystsdeveloping relatively slow trading systems will find that this procedure has a star-ting effect on expectations.
Averaging the Resuits _
ecause the testing process is computer dependent, it is desirable to make the selection ofthe best parameters automatic. This can usually be done effectively by averaging either theProfits or other results displayed in-the test map. In the case of a moving average system,first determine how broad the area of success must be so that it is not a spike, an anomalyin the results. For example, the results of a 3-, 4-, 5-, and 6-day test show profits of 1,000,8,000, 3,000, and 4,000, respectively. The 4-day case is clearly a profit spike and could beminimized by replacing its value with the average of the three other values or by. the inter-polated value of the two adjacent points (Figure 21-8). In general, the substitution processin which PL, replaces PL,,sié TESTING
 
SS ,FIGURE 21-7 Visualizing the performance of multiple parameters. (a) The slow moving aver-age peaks at 50-60 days, (b) The fast average may be best at any value from 7 through 15.
 
Alternate display of 2-parameter testSwiss franc 1986-1996
 
80,000 8°60,000 g g 22 Oog 8S 3 840,000 ° 8 g 2 e3 8 ° ° ° Q3 ° g ° 8 E3 °3 20,000& ° 6s 3°zZ
 
~20,000_L L L 1 fo"20 "40 60. 80 100
Slow-moving average
(2)
Alternate display of 2-parameter testIMM Swiss franc 1986-1996
 
 
 
 
80,000. ° O°°60,000 8 9 ° 9g g§ § 8 @ |& 98 8 3° 3° ° 8& 40,000 é ° 8 8© g 0 © g 8g 9 ° 8 6 8= 8 g@ g o 8 85 20,000 3 8g _Ss - °2 ° o° S .20,000 1 ! L \ 1 | ‘“oO 5 1000 7 15 20
Fast-moving average(b)
 STEP-FORWARD TESTING AND OUT-OF-SAMPLE DATA 517
 
 
FIGURE 21-8 Replacing a spike with the average A or an interpolation B.
 
 
PL, + PL, + Phi.3
' Py, =will smooth the performance. The best parameter set is the highest value of PL, once allaverages have been substituted for the raw test results.
It is more likely that a spike or erratic results will appear in areas that test shorter peri-ods than in those that test longer trends. The difference between the results of a 2- and 3-day moving average is greater than that of a 30- or 31-day average or even a 30- or 40-daytest. If the optimization was scaled properly, leaving more space between tests of highervalue in the manner ofa percent difference, the simple averaging technique will be a goodsubstitute for testing every value.
Two-Parameter Averaging. -
The results of a 2-parameter test may also be averaged. Using the form discussed earlier(Figure 21-9), the results of each test will be denoted as PL,, the profit/loss associated withrow i and column j of a 2-dimensional display. The object is to replace each PL, with PL,,the average value of its surroundings. This is done, as seen in Figure 21-9a, by taking anaverage of the eight test results adjacent to the ith value as well as the center value. Thereare special cases when the #th result is not fully surrounded but on the perimeter of thetest map. Figure 21-9b-d shows the averaging technique used when the ith box is a top,side, or corner value. The 9-box average shown here is comparable to the 3-point averagein the 1-parameter test. If the test has relatively small increments between calculation val-ues and more test cases, an average of a larger area would be appropriate.
STEP-FORWARD TESTING AND OUT-OF-SAMBPLE DATA
The correct procedure for testing always includes reserving some data to be used after-ward to validate the results. The data used during testing is cailed in-sample data, and thereserved portion for validation is called out-of-sample data. This concept should be famil-iar to everyone involved in the design or development of a system.
Traditionally, analysts use as much data as possible to determine whether their pro-posed strategy is sound. The use of long test periods assures a good sample of price pat-terns, including long periods of sideways movement, bull and bear markets, and a good518 TESTING
 
FIGURE 21-9 Averaging of map output results. (a) Center average (9-box). (b) Top-edge aver~age (6-box). (c) Left-edge average (6-box). (¢) Upper left corner average (4-box).!
 
 
 
     
—_—— FL,
   
|
 
 
LEHH ms
Plaa -
 
(a)
 
number of price shocks of various sizes. When more data are used, it is most likely that theresults will show larger profits and larger losses. ~
When there is, for example, 10 years of data available for testing, a standard approachwould be to test the oldest 9 years, find the best set of parameters, then test the system onthe most recent year using the selected values. If the results of the out-of-sample period aresimilar to the in-sample performance, the system is considered validated. —
One serious problem occurs when the results of out-of-sample testing fail to performas expected. The results are inspected closely to find how this period differed from thepast, and if successful, a rule change is incorporated into the program and the in-sampledata retested. This repeated use of out-of-sample data is called a feedback process. Unfor-tunately, there is no longer any out-ofsample data to be used for validation. The improvedresults are the product of fitting the data and you are left uncertain about the ability to pro-duce profits in the future. This is a problem that can only be resolved by a broad view ofrobust testing procedures discussed throughout this chapter.CHANGING RULES. 519
 
Step-Forward Testing Procedure
The concept of choosing a parameter set from a test of in-sample data, then applying theresults to out-of-sample data, has been developed into a test process called step-forwardtesting’ (also walk-forward testing). This process goes as follows:
1. Select the total test period, for example 10 years of daily data from 1988 through 1997.
. Select the size of the individual test intervals, for example 2 years.
. Begin testing with 1988 and 1989 data.
. Select the best parameters from those results.
. Find the performance of the next 6 months of data (the first half of 1990) using theparameters selected in step 4. Accumulate this out-ofsample performance through-out the test process.
6. If there is more data to test, move the test period forward by 6 months (the secondtest will be from the second half of 1988 through the first half of 1990) and test thedata; otherwise, go to step 8.
7. Go to step 4.
8. The results of the step-forward test are the accumulated results of the individualout-of-sample tests in step 5.
Wok wb
This process clearly simulates the traditional approach to test design, using in-sampleand out-of-sample data in its proper order. Unfortunately, it does not correct the prob-lems of reusing the out-of-sample data, which transforms that step to another piece of in-sample data.
Step-forward testing can introduce a bias that favors faster trading models. Because thein-sample test has only 2 years of data in our example, a long-term trading model may onlypost a few trades during this period, and one of the longer trades may be interrupted bythe end of the data window and discarded. This makes the results ofthe longer-term mod-els unreliable. In addition, these slower trading approaches will often start a trade at thebeginning of the test data window at a point that is abnormal, that is, one that would notoccur if the data were continuous.
The problem of short-term bias can be identified by the optimum parameters switchingfrom short-term to long-term in successive test periods. For example, ifyou_test moving aver-ages from 5 to 50 days, you will find one period shows 10 days as the best followed by aperiod in which 50 days is best, followed again by a period in which 15 days is best. Thiserratic behavior is a sign that the testing specifications are too limited. It can be corrected byextending the test intervals from 2 years to 5 years and making the out-ofsample period 1year. None of these solutions, however, correct for the use of out-of-sample data more thanonce. More on this problem can be found in the following section “Retesting Procedure.”
CHANGING RULES
As testing progresses, it is inevitable that the analyst will want to modify a rule in the tradingstrategy. This is usually the result of inspecting the results of tests and noting that a specificpattern was not treated properly (or profitably). After some work, the analyst introduces aspecial rule, which tums a previously losing situation into a profit. Figure 21-10 shows twopossible patterns in the test performance pattern based on this change.
If the rule change improves one price pattern at the cost of others, the compiete testresults will appear as shown in Figure 21-10a. Higher profits at the previous peak andgreater losses at the ends result in the same cumulative test profitability, Because the new
2 A thorough presentation of both step-forward testing and optimum search techniques can be found in RobertPardo, Design, Testing and Optimization of Trading Systems (John Wiley & Sons, 1992).520 TESTINGee
eeFIGURE 21-10 Patterns resulting from changing rules. (a) A rule change that improves onesituation at the cost of others. (b) A rule change resulting in general improvement.
 
 
(b)
 
tule caused the fitting of a specific pattern at the cost of added losses in other patterns, itshould not be considered an improvement.
In Figure 21-108, the new rule improves performance in all cases. This type of patternis desirable in optimizations. It is possible, of course, that the improvement was caused bya tule that corrects one case only in a way that is so specific that it does not affect any othertrade. That type of rule fitting, on a sample of one pattern, is likely to harm results in thelong run. - ‘
ARRIVING AT VALID TEST RESULTS
It is not unusual for the results of an optimization, especially one with many parameters,to appear perfect. All the trades could be profitable, the equity drops might be small, andthe strategy might perform in changing markets, and yet, the final system might have noreal expectation of profit. To create a system with predictive qualities, rather than one thatis historically ficted, requires preparation in advance of testing.
You Can't Prove a System (or Indicator) Works by One Case
You can always select a moving average or indicator gave a sell signal in the S&P just beforethe 1987 crash. That’s not difficult when there are thousands (or millions) of combinationsof trend speeds, filters, momentum indicators, divergence rules, and so on. However, whatworks in a specific case, and often an unusual one, is not likely to work in general. Manip-ulating the rules is only a waste of time. Showing that a combination of technical indicatorswould havé profited from the last market crash has no bearing on the next drop. -
The addage that “there are lies, half lies, and statistics” is too true. Statistics can provea point or prove just the opposite, depending on how the numbers are presented. A mar-ARRIVING AT VAL.~ TEST RESULTS 521
 
ket can be in an uptrend and a downtrend at the same time, based on the time interval overwhich you view it. You must be constantly aware of those problems that you cannot see—the omissions.
It will be necessary for you to draw your own picture of system performance. What isthe chance of a big profit this year or this week? What is the worst-case scenario? Will themethod survive a sharp change in market volatility? Can I choose a long-term or short-termhorizon with more confidence? Is an 8-day moving average really better than a 10-daybreakout? The following guidelines will get you started by giving you one way to look attest results. From this you will be able to develop, over time, your own method of evalua-tion. As Jack Schwager has stated, “The mistake is extrapolating probable future perfor-mance on the basis of an isolated and well-chosen example from the past.”
Searching for Robustness
A robust system is one that performs consistently in a wide variety of situations, includingpatterns that are still unforeseen. From a practical view, this translates into a method thathas the fewest parameters and is tested over the most data. In a test by Futures Truth, thebest performing systems commonly have four or less optimized variables.’ The character-istics of a system with forecasting ability are: .
1. It must be logical. Each rule and formula must be designed to capitalize on a realfundamental or price phenomenon. Discovering a price pattern or cycle through op-timization may seem to be a revelation, but it is more likely to be an illusion. By test-ing enough patterns, it is statistically probable that one of them will seem to fit.Without a fundamental reason for the existence of that pattern, it is not safe to use it.
2. It must adjust to changing market conditions. A system that assumes that marketsdon’t change, or that everything has been seen in past data, will suffer large equityswings. Self-adjusting features might include an inflator or deflator, stop-loss val-ues that change with volatility, and rule shifts based on seasonal or nonseasonalyears,
3. It must be tested properly. The principles of statistics state that the best tests use themost data. More data includes bull, bear, and sideways markets; large and smallprice shocks; and periods of instability and doldrums. There is no substitute formore data. Proper test procedures also suggest that you reserve some unseen datauntil the very end, so that you can validate your work on an out-of-sample period.If that is not available, you can always paper trade until you can compare the accu-mulated trading profile with the expectations defined by your tests.
Some analysts prefer a technique called blind simulation, commonly known as step-forward testing. In this procedure, discussed in a previous section, a series of fixed-lengthtests are defined and the parameters found to be optimal are used to trade using data inthe next period forward. The process then moves forward by dropping off old data andadding an equal amount of new data, The results of this second test are again traded onforward data. The method is continued until the out-ofsample results inciude all of theavailable test data, This is an important concept and is also discussed in the section “Reop-timization” later in this chapter.
From time to time each of the three characteristics above will seem to be the mostimportant. In the long run they must all be satisfied to have a robust program.
5 Jack D. Schwager, Schwager on Futures: Technical Analysis (john Wiley & Sons, 1996, p. 673).* John Hill, “Simple vs. complex,” Futures (March 1996, p. 57).522 TESTING
 
Performance Criteria
Assuming that the proper principles have been followed, are the test results good enough?Although the strategy shows steady profits with a low equity drawdown, could a naive sys-tem have done better? Did the famous skeptic on television throw darts at buy and sell sig-nals on a board and show a 75% return while your system only netted 25%?
BenchmarksIt is necessary to have a benchmark that provides a way of measuring success. It is best ifthis is a well-documented indicator, such as the S&P Index, the Lehman Brothers TreasuryIndex, a class of Fund Managers, or the Commodity Trading Advisor Index. Newspaper arti-cles that highlight the spectacular profits or losses of one manager is not a good bench-mark, but simply ex post selection (hindsight).
It may seem desirable to have achieved the recognition for having the largest gain duringa single month, but you should focus on having very good gain over each year, The largestgain can only come with high risk. You should not be surprised if those investment advisorsPosting remarkably high returns in one month have had very erratic performance overall.
Measuring Test Results
A number of performance criteria are needed to evaluate any trading strategy during thetest phase and to compare results under actual market conditions.
1, Net profits or losses. Although not the most interesting statistic, the net profit is themotivation for trading. While you would not select a system that produces a net loss,it is the other statistics that will tell which of the better results, if any, are realistic.
2. Number of trades. This simple value indicates whether your test was long enoughto depend on the results. A few trades can appear very profitable, but the tradingprofile is not yet clear.
3. Percentage of profitable trades. Also called reliability, a high value of 60% tellsyou that the method captures profits regularly. A trend system will be working cor-rectly if its reliability is near 40%.
4. Average net return per trade. With or without commissions and slippage costs,the average return per trade gives you an indication of how difficult it will be to re-alize the system returns. A theoretical average of $50 per trade fora currency islikely to net less than one-half after slippage in normal markets, and when a U.S.economic report is released, slippage alone could be $500 on a single trade.
5. Maximum drawdown. The largest equity swing from peak to valley, this mea-surement can be very erratic and is not likely to be the largest drawdown seen inthe future; however, it gives you some idea of the minimum capital needed totrade this market, If the value is very small, it is likely to be based on a smallamount of data, too many specific rules, or a narrow range of test parameters.
6. Annualized rate of return. The rate of return is the profit for a predetermined in-vestment. The investment can be calculated in reverse by knowing the funds-at-risk to equity ratio, the maximum drawdown,-or standard deviation of expectedreturns. These returns should be annualized to compare one test, market, orbenchmark with another.
7. Total profits to total losses. This simple ratio gives a reasonable measure of thesmoothness of the equity curve. As the ratio increases, the proportion of losses de-clines and the equity curve becomes smoother. Any value over 2.0 is very good.
8. Tine to recovery. A large drawdown may be inevitable in a realistic system, but ashorter time to recovery is most desirable. A larger drawdown with a much fasterrecovery seems to be a better trade-off for most investors.ARRIVING AT VALID EST RESULTS 523
 
9. Time in the market. All else being equal, a trading system that is in the market lessthan another system is preferable. If two systems have approximately the same re-turns and risk, then the one that has more time out of the market is actually ex-posed to less risk.
10. Smoothness of returns, In addition to the individual measurements describedabove, some form of traditional equity smoothness is desirable. This can be thestandard deviation of the residuals when the linear regression of the returns is cal-culated, or it can be some weighted combination of the previous measurement.This measurement should reflect the type of equity profile you seek.
Having decided the measurements needed to evaluate the results of the testing, someother procedural issues must be resolved before testing begins. This will help to avoidbending the rules to fit the results.
Defining the test range. Each optimization run should represent a broad sampling oftests over a wide range of parameter values. This range should be established inadvance based on expectations of what is reasonable. If the returns in this range arenot profitable, you must first understand why your ideas have not worked before look-ing for other periods in which the system might show profits. A thorough review ofyour trading rules may show that you have defined them wrong, not limited them tothe best situations, or simply programmed them incorrectly. ,Use the average of all test results. Most tests will show both profits and losses, withsome areas of very attractive profits. If you tested 100 cases and 30 showed profits ofabout 30% per year, 30 showed break-even results, and the last 40 showed variouslosses, you might say that the 30 profits represented a successful system. That assumesthat the market will continue to perform in a way that allows those 30 parameter com-binations to generate profits during the next year. It is better to assume that marketchanges will make it difficult to tell which combination of parameters will be the best inthe future, Then you want a system that can generate profits with any parameter set, notjust 30 out of 100. Perhaps that’s too ambitious; nevertheless, one system is better thanthe other if the average of all tests is higher than the average of all tests in the other sys-tem. To be more precise, we could say that the best system is the one in which:
Adjusted Returns = Average of all tests — 1 standard deviation of all test returns
is greatest. From the discussion of standard deviations in Chapter 2, we know that areturn greater than the adjusted returns will occur 68% of the time.
Set aside data for final validation. To give yourself a way of knowing whether youhave overfitted the data, do not use all the data in your tests, This is needed for out-of-sample verification at the end. If the results of using this data are not good, you willneed to reassess the performance. If it is good, you have some assurance that your test-ing was done correctly, but there are no guarantees other than time.
Stability overtime. Be sure that the test data was long enough to have included a sub-stantial change in price, a major bull and bear market, and an extended sideways periodat low volatility. By viewing performance over a number of smaller time intervals, insteadof only one average or total figure, it is possible to assess the stability of the strategy.
Elimination of outlier periods. Realistically, a system should not base its profitabilityon a single major market move over the test interval. In particular, if the move that gen-erated the profit was actually a price shock, the system had only a 50% chance of post-
- ing that profit, and has an equal chance of a loss in the future. It may be practical toremove the profits generated on the first day or two of a move that began with a priceshock, allowing for the possibility of holding the wrong position in the future,524 TESTING
 
Concentrate on large profits and large losses. The largest profits and losses are criti-cal to the net performance of any system; therefore, you should make a special effortto study those trades. In doing that, you may find that the largest profits were theresults of price shocks, as discussed in the preceding point, or that there is an oddpiece of data that was not obvious before. If there are a number of large profits and nolarge losses, you may have excellent risk control, but more likely you have overfit thedata. There is an unfortunate tendency to look for problems when results are bad, butto accept them when they are good.
Reading between the Lines
It is easy to show specific examples of rules that produce large profits, while the overallstrategy is bad. Many books base intuitive proof or justification on specific examples. Thiscan work if the general theory is sound, when there is fundamental or economic substan-tiation, such as expecting a seasonal low in the U.S. crops at harvest. For patterns and manypurely technical indicators, such as those using momentum, this is not as clear. Results arelikely to show that the situations that fail greatly outnumber the fewer exceptionally goodexamples. :
Systems That Work in Only One Market
From time to time, all traders receive mail offers for a highly specialized system called “