end-following approach captures less of the price move but keeps clear of the sharp reac-tions that stop out the middle-speed trends. The selection of parameters to use in real trad-ing follows the same reasoning applied to Figures 21-3 and ¢. The third case, shown inFigure 21-5¢, is one of erratic profits and losses. The absence of a consistent pattern in theperformance indicates that the trading strategy does not apply to the data.
 
4
Alternate Ways of Visualizing ResultsWith software such as MathCad, you can see the 2-parameter test as a contour or topolog-ical relief map. For some users this form can make it much easier to see the peaks and val-leys of performance and is highly recommended. Any spreadsheet program will allow asurface plot, which will give you a rough look at the continuity of a 2-parameter test, as- shown in Figure 21-6. This 3-dimensional graph gives the net profits of a moving average~ crossover system applied to the Swiss franc. The slower trend is shown along the frontscale and the faster one along the right. There is a ridgé running from the front to the back,~ indicating that a slow trend of about 50 to 60 days dominates the results. Other than thefastest trends of 3 days, it is easy to sec that there are many combinations of fast and slowmoving averages that are historically profitable.
1
‘Moving average (in days}
Marginally
successful
 
Visualizing with Scatter Diagrams _ |“To see the performance of more than two variables, you can plot the results of all testsagainst each parameter (see Figure 21-7) using a scatter diagram (also called an XY Plot).
    
 
 
sia TESTING VISUALIZING AND ny TERPRETING THE RESULTS
 
FIGURE 21-5 (Continued) FIGURE 21-6 Three-dimensional map of crossover model.
‘Unsuccessful
3 Ak”
           
       
IMM Swiss franc, 1986-1996
  
Net profivioss
 
ii'1
‘This will give you a better understanding of how each value affects the total picture andallow you a way to visualize the robustness of each variable. Using a 2-variable example, arange of fast and slow moving averages were tested for the Swiss franc based on a crossoverstrategy. The results of the slow moving averages are plotted for all combinations of the fastmoving average in Figure 21-7; the fast average is shown in Figure 21-7b.
"The individual test results are shown as circles above the slow OF fast moving averageperiod in Figure 21-7. For the slow moving average, itis clear that profits are greatest when50 and 60 days are used for the calculation period; however, it is not as clear which ofthose two valties is the best choice. The 60-day moving average posted profits higher thanthe 50-day average, but also showed profits below the lowest 50-day returns. A closer lookshows that the average returns of the moving averages Over those periods is about thesame, but the 60-day moving average has a higher variance. Normally, the beso.choice is theresult that is most stable (the 50-day calculation), although in this example we must be cau-tious that the small number of tests do not make the 50-day results look stable just bychance.
The results of the faster moving average, seen in Figure 21-76, show much greater vari-ance. The 7-period results have a very wide range including the highest returns; however,the averages of the 7th through 15th periods are very similar. With variance in mind, the13-day calculation seems to be the best candidate. We can be confident that this method isrobust, because nearly all combinations of fast and slow moving averages gave profitablereturns, net of a $50 commission. . -
 
 
1. Annualizing all values, Tests that you perform 6 months or 1 year apart are lito use different time periods. Annualizing the results will make Te enone easiee
2. Risk adjusting. A profit of $10,000 with a drawdown of $5,000 is effectively thesame asa profit of $20,000 and a drawdown of $10,000, given adequate reservesExpressing the results as profits divided by risk can be useful. - :
3. Adjusting for standard error. The important difference between two tests that yieldthe same results, one with 2 trades and one with 50 trades, is that the one with only2 trades is a less reliable result. If both posted 50% profitable trades, another losin,trade would make the reliability of the second test drop to 49%, while the first casewould drop to 33%. Instead, results should be presented at the same confidencelevel, which means subtracting the standard error from the current result. Analysts
developing relatively slow tradin; s‘ 1g systems will find that th -ting effect on expectations. ac this Procedure as a Sie
 
Averaging the Results
Because the testing process is computer dependent, it is desirable to make the stlection of
aram a i i 4
When testing a single system, the results of one set of parameters can easily be compared profits hee secalte display ahs can usually be done effectively by averaging either the
When resting & pecause the test period, commissions, and other basic values are all the a eoe ecemmine bow beoad the area the test map. In the case of a moving average system,
wnne, However, over time you will test many different systems and variations of those sys) = aes ncsulte For example, the res of success must be so that it is not a spike, an anomaly
tems, and you will want to compare them to decide which’ strategy is best. This requires * “2 $1000, 5,000, and 4.000 respec results of a 3-, 4, 5-, and 6-day test show profits of 1,000,
em Advanced planning. " . Q 8,000, 3,000, and 4,000, respectively. The 4-day case is clearly a profit spike and could beStandardizing test results is the best way to increase the usefulness of extensive testing. -#¥ polated value ofthe a a the average of the three other values or by. the inter-
. + = alu oints (Fi : orb‘This should include: OO EE enero. points (Figure 21-8). In general, the substitution process
Standardizing Test Results
   
  
TESTINGsié
 
FIGURE 21-7 Visualizing the performance of multiple parameters. (a)The slow mov ng aver-age peaks at 50-60 days. (b) The fast average may be best at any value from 7 through 15.
Alternate display of 2-parameter test
 
 
 
 
 
 
 
 
 
Swiss franc 1986-199680,000 3r °60,000 - g g 83 L 8 °= ° ° 8 8 ° °40,000 - Q ° ° 5 533 °S ;2 °S ° }3 20,000 }- 8 853 8 °2fg Seot L L L po 1 1 120100036 40 60 20 100Slow-moving average(aAlternate display of 2-parameter testIMM Swiss frane 1986-199680,0005 o 6r °60,000 |- 8 8 8 8 8 -2 F 8 o 0 ° 8s o 3= cr) 8S 40,000 |- ° 8 }5 6 oO2 [ g 8 8 B 68s ® g§ g 2 © 83 20,000|- a _& °zB ° ° °0 S |1 1 4 i - } 420.0005 “Ss 0 7 15 20Fast-moving average
()
 
STEP-FORWARD TESTING AND OUT-OF-SAMPLE DATA su7
FIGURE 21-8 Replacing a spike with the average A or an interpolation B.
 
' Piy= PL At Plt
will smooth the performance. The best parameter set is the highest value of PL, once allaverages have been substituted for the raw test results.
It is more likely that a spike or erratic results will appear in areas that test shorter peri-ods than in those that test longer trends. The difference between the results of a 2- and 3-day moving average is greater than that of a 30- or 31-day average or even a 30- or 40-daytest. If the optimization was scaled properly, leaving more space between tests of highervalue in the manner of a percent difference, the simple averaging technique will be a goodsubstitute for testing every value.
Two-Parameter Averaging. -
The results of a 2-parameter test may also be averaged. Using the form discussed earlier(igure 21-9), the results of each test will be denoted as PLy, the profivloss associated withrow i and column j of a 2-dimensional display. The object is to replace each PL, with PL,the average value of its surroundings. This is done, as seen in Figure 21-9a, by taking anaverage of the eight test resuits adjacent to the ith value as well as the center value. Thereare special cases when the ith result is not fully surrounded but on the perimeter of thetest map. Figure 21-9b-d shows the averaging technique used when the ith box is a top,side, or corner value. The 9-box average shown here is comparable to the 3-point averagein the 1-parameter test. If the test has relatively small increments between calculation val-ues and more test cases, an average of a larger area would be appropriate.
STEP-FORWARD TESTING AND OUT-OF-SAMPLE DATA
The correct procedure for testing always includes reserving some data to be used after-ward to validate the results. The data used during testing is cailed in-sample data, and thereserved portion for validation is called out-of-sample data. This concept should be famil-iar to everyone involved in the design or development of a system.
‘Traditionally, analysts use as much data as possible to determine whether their pro-posed strategy is sound. The use of long test periods assures a good sample of price pat-terns, including long periods of sideways movement, bull and bear markets, and a good4
”
 
 
TESTING
518
FIGURE 21-9 Averaging of mapage (6-box). (c) Left-edge average (|
‘output results. (a) Center average (9-box). (b) Top-edge aver-6-box). (d) Upper left corner average (4-box).
 
   
Plisin | Plind | Peisaer
 
3
Phi jes oS
 
Pliss.iet| __J(a)
 
|
Plait
||
 
jet
 
 
L
(0)
 
{o)
x
—>Pla,
  
of price shocks of various sizes. When more data ar used, it is most likely that theber of price shocks of various sizes. ta are it is most linum
i d larger losses. ;Fe a ae rote, 10 years of data available for testing, 2 standard approacheI D a
re £ parameters, then test the system ondest 9 years, find the best set of p:Fe oe eee year using the selected values. If the results of the ouvofssmple period arem ii nsidere jidated, —
imi inveample performance, the system is co i aSL ae in a Pen occurs when the results of out-ofsample testing fil to performSoected. <The results are inspected closely to find how this period diflered from thesc and if successful, a rule change is incorporated into the program and the insamaleseca setested, This repeated use of oucotsample data scaled 2 feedback process Unforis no longer any out-ofsample data improved
manately, thee one ang the data and you are left uncertain about the ability t0 P
re
duce profits in the future. This is a problem that can only be rerobust testing procedures discussed throughout this chapter.
esolved by a bfoad view of
 
=&Za
 
CHANGING RULES 519
Step-Forward Testing Procedure
The concept of choosing a parameter set from a test of in-sample data, then applying theresults to out-ofsample data, has been developed into a test process called step-forwardtesting? (also walk-forward testing). This process goes as follows:
- Select the total test period, for example 10 years of daily data from 1988 through 1997.Select the size of the individual test intervals, for example 2 years.
. Begin testing with 1988 and 1989 data.
. Select the best parameters from those results.
. Find the performance of the next 6 months of data (the first half of 1990) using theparameters selected in step 4. Accumulate this out-of-sample performance through-out the test process.
6. If there is more data to test, move the test period forward by 6 months (the secondtest will be from the second half of 1988 through the first half of 1990) and test thedata; otherwise, go to step 8.
7. Go to step 4.
8. The results of the step-forward test are the accumulated results of the individual
out-of-sample tests in step 5.
RW
This process clearly simulates the traditional approach to test design, using in-sampleand out-of-sample data in its proper order. Unfortunately, it does not correct the prob-lems of reusing the out-of sample data, which transforms that step to another piece of in-sample data.
Step-forward testing can introduce a bias that favors faster trading models. Because thein-sample test has only 2 years of data in our example, a long-term trading model may onlypost a few trades during this period, and one of the longer trades may be interrupted bythe end of the data window and discarded. This makes the results ofthe longer-term mod-els unreliable. In addition, these slower trading approaches will often start a trade at thebeginning of the test data window at a point that is abnormal, that is, one that would notoccur if the data were continuous.
The problem of short-term bias can be identified by the optimum parameters switchingfrom short-term to long-term in successive test periods. For example, if you test moving aver-ages from 5 to 50 days, you will find one period shows 10 days as the best followed by aperiod in which 50 days is best, followed again by a period in which 15 days is best. Thiserratic behavior is a sign that the testing specifications are too limited. It can be corrected byextending the test intervals from 2 years to 5 years and making the out-ofsample period 1year. None of these solutions, however, correct for the use of out-of-sample data more thanonce. More on this problem can be found in the following section “Retesting Procedure.”
CHANGING RULES
As testing progresses, it is inevitable that the analyst will want to modify a rule in the tradingstrategy. This is usually the result of inspecting the results of tests and noting that a specificPattern was not treated properly (or profitably). After some work, the analyst introduces aspecial rule, which tums a previously losing situation into a profit. Figure 21-10 shows twopossible patterns in the test performance pattern based on this change.
If the rule change improves one price pattern at the cost of others, the complete testresults will appear as shown in Figure 21-10a. Higher profits at the previous peak andgreater losses at the ends result in the same cumulative test profitability, Because the new
* A thorough presentation of both step-forward testing and optimum search techniques can be found in RobertPardo, Design, Testing and Optimization of Trading Systems (John Wiley & Sons, 1992). 
'
 
TESTING520
1-10 Patterns resulting from changing rules. (a) A rule change that improves one
FIGURE 2 (b) A rule change resulting in general improvement.
situation at the cost of others.
 
 
£ added losses in other patterns, it
rule caused the fitting of a specific pattern at the cost o!
idered an improvement. ; , mempo gure 21-108, the new rule improves performance in all cases, This type of pate =is desirable in optimizations. It is possible, of course, that the impro’
cade. That type of rule fitting, on a sample of one pattern, is likely ‘o harm results in thele fitti le of tte likely ttrade. That f rule Ss ‘
long run. -
ESULTSARRIVING AT VALID TEST RI seusIt is not unusual for the results of an optimization, especially one with many Params ersto appear perfect. All the trades could be profitable, che equity drops migh ce small, andi in changing markets, and yet, the hnalce ne eo create a system With predictive qualities, rather than one thatex . systeis historically fitted, requires preparation in advance of testing.’t Prove a System (or Indicator) Works by One Caseaverage or indicator gave a sell signal in the S&P just before‘alt when there are thousands (or millions) of combination: sn i . If trend speeds, filters, momentum indicators, divergence rules, and soon However, whatoeorre ime speeific case, and often an unusual one, is not likely to wo! n general. ManiSons Jthe rales is onlya waste of time. Showing that a combination of tech nicaould have profited from the last market crash has no bearing on the next dro P rovewon The addage that “there are lies, half lies, and statistics” is too true. istics can provea point or prove just the opposite, depending on how the numbers are pr .
You Can!
‘You can always select a movingthe 1987 crash. That’s not diffic
 
ARRIVING ATVAL.~ EST RESULTS 521
ket can be in an uptrend and a downtrend at the same time, based on the time interval overwhich you view it. You must be constantly aware of those problems that you cannot see—the omissions.
It will be necessary for you to draw your own picture of system performance. What isthe chance of a big profit this year or this week? What is the worst-case scenario? Will themethod survive a sharp change in market volatility? Can I choose a long-term or short-termhorizon with more confidence? Is an 8-day moving average really better than a 10-daybreakout? The following guidelines will get you started by giving you one way to look attest results. From this you will be able to develop, over time, your own method of evalua-tion. As Jack Schwager has stated, “The mistake is extrapolating probable future perfor-mance on the basis of an isolated and well-chosen example from the past.”3
Searching for Robustness
A robust system is one that performs consistently in a wide variety of situations, includingpatterns that are still unforeseen. From a practical view, this translates into a method thathas the fewest parameters and is tested over the most data. In a test by Futures Truth, thebest performing systems commonly have four or less optimized variables.’ The character.istics of a system with forecasting ability are:
1. It must be logical. Each rule and formula must be designed to capitalize on a realfundamental or price phenomenon. Discovering a price pattern or cycle through op-timization may seem to be a revelation, but it is more likely to be an illusion. By test-ing enough patterns, it is statistically probable that one of them will seem to fit.Without a fundamental reason for the existence of that pattern, it is not safe to use it.
2. It must adjust to changing market conditions. A system that assumes that marketsdon’t change, or that everything bas been seen in past data, will suffer large equityswings. Self-adjusting features might include an inflator or deflator, stop-loss val-ues that change with volatility, and rule shifts based on seasonal or nonseasonalyears.
3. It must be tested properly. The principles of statistics state that the best tests use themost data. More data includes bull, bear, and sideways markets; large and smallprice shocks; and periods of instability and dotdrums. There is no substitute formore data. Proper test procedures also suggest that you reserve some unseen datauntil the very end, so that you can validate your work on an out-of-sample period.If that is not available, you can always paper trade until you can compare the accu-mulated trading profile with the expectations defined by your tests.
Some analysts prefer a technique called blind simulation, commonly known as step-forward testing. In this procedure, discussed in a previous section, a series of fixed-lengthtests are defined and the parameters found to be optimal are used to trade using data inthe next period forward. The process then moves forward by dropping off old data andadding an equal amount of new data. The results of this second test are again traded onforward data. The method is continued until the outofsample results include afl of theavailable test data. This is an important concept and is also discussed in the section “Reop-timization” later in this chapter.
From time to time each of the three characteristics above will seem to be the mostimportant. In the long run they must all be satisfied to have a robust program.
*Jack D. Schwager, Schwager on Futures: Technical Analysis john Wiley & Sons, 1996, p. 673).* John Hill, “Simple vs. complex,” Futures (March 1996, p. 57). 
 
 
 
 
TESTING522
Performance Criteria
inci d, are the test results good enough?ing that the proper principles have been followed, a n‘Aichough the werategy vhows steady profits with a low equity drawdown, could a naive system have done better? Did the famous skeptic on television throw darts at buy and sell sig-nals on a board and show a 75% return while your system only netted 25%?
penne i asuring success. It is best ifi is mecenary oN 2 ncienon tha Pee uP nds, the Lehman vores TenryInde as eae apectaculat An or losses of One aeaget is not a good bench-mi aks ev achive’ the cecopntion for having ie ages gain dng2 single month, Dut you ho ou should ot be 8004 an oreice investment adsa ea emckably high bigii ns in one month have had very erratic performance overall.
Measuring Test Results /‘A number of performance criteria are needed to evaluate any trading strategy during thetest phase and to compare results under actual market conditions.
i i ting statistic, the net profit is the. Net profits or losses. Although not the most interes', Me en for trading. While you would not select a system that produces a net loss,it is the other statistics that will tell which of the better results, if any, are realistic.2. Number of trades. This simple value indicates ‘whether your test was long enough, to depend on the results. A few trades can appear very profitable, but the tradingrofile is not yet clear. |Percentage | profitable trades. Also called reliability, a high value of cow tells"you that the method captures profits regularly. A trend system will be workingif i iability is near 40%. |4 sect erenarn per trade. With or without commissions and slippage costs: i indication of how difficult it will be to re-the average return per trade gives you an in . .it is f $50 per trade fora currencyflize the system returns. A theoretical average © 3slikely to net less than one-half after slippage in normal markets, and when a US.economic report is released, slippage alone could be $500 on a single trad . nes:5, Maximum drawdown. The largest equity swing f from pea to val hey, wh is me. i i ikely to e largestsurement can be very erratic and is not likely u v seenit gi i f the minimum capital nee: tofuture; however, it gives you some idea of the,teade this market. If the value is very small, it is likely to be based on a smallamount of data, too many specific rules, or a narrow range of test Parameters. iAnnualized rate of return. ‘The rate of return is the profit fers a Pre fini i owing the vataie nn, The investment can be calculated in reverse by loot’isk to equity ratio, the maximum drawdown,-or ‘standard deviation of expectedreturns. These returns should be annualized to compare one test, market, 01benchmark with another. .7. Total profits to total losses. This simple ratio gives a reasonable imeasure of we: smoothness of the equity curve. ‘As the ratio increases, the proportion Oo! lossesclines and the equity curve becomes smoother. Any value over 20 is we 8 an8 inevi ina reali ,Time to recovery. A large drawdown may be inevitable i ;shorter time to recovery is most desirable. A larger drawdown with a much fasterrecovery seems to be a better trade-off for most investors.
w
a
@
 
ARRIVING AT VALID" EST RESULTS 523
9. Time in the market. All else being equal, a trading system that is in the market lessthan another system is preferable. If two systems have approximately the same re-turns and risk, then the one that has more time out of the market is actually ex-posed to less risk.Smoothness of returns. In addition to the individual measurements describedabove, some form of traditional equity smoothness is desirable. This can be thestandard deviation of the residuals when the linear regression of the returns is cal-culated, or it can be some weighted combination of the previous measurement.This measurement should reflect the type of equity profile you seek.
10.
Having decided the measurements needed to evaluate the results of the testing, some
other procedural issues must be resolved before testing begins. This will help to avoidbending the rules to fit the results.
Defining the test range. Each optimization run should represent a broad sampling oftests over a wide range of parameter values. This range should be established inadvance based on expectations of what is reasonable. If the returns in this range arenot profitable, you must first understand why your ideas have not worked before look-ing for other periods in which the system might show profits. A thorough review ofyour trading rules may show that you have defined them wrong, not limited them tothe best situations, or simply programmed them incorrectly. .Use the average of all test results. Most tests will show both profits and losses, withsome areas of very attractive profits. If you tested 100 cases and 30 showed profits ofabout 30% per year, 30 showed break-even results, and the last 40 showed variouslosses, you might say that the 30 profits represented a successful system. That assumesthat the market will continue to perform in a way that allows those 30 parameter com-binations to generate profits during the next year. It is better to assume that marketchanges will make it difficult to tell which combination of parameters will be the best inthe future. Then you want a system that can generate profits with any parameter set, notjust 30 out of 100. Perhaps that’s too ambitious; nevertheless, one system is better thanthe other if the average of all tests is higher than the average of all tests in the other sys-tem. To be more precise, we could say that the best system is the one in which:
Adjusted Returns = Average of all tests — 1 standard deviation of all test returns
is greatest. From the discussion of standard deviations in Chapter 2, we know that areturn greater than the adjusted returns will occur 68% of the time.
Set aside data for final validation. To give yourself a way of knowing whether youhave overfitted the data, do not use all the data in your tests. This is needed for out-of-sample verification at the end. If the results of using this data are not good, you willneed to reassess the performance. If it is good, you have some assurance that your test-ing was done correctly, but there are no guarantees other than time.
Stability overtime. Be sure that the test data was long enough to have included a sub-stantial change in price, a major bull and bear market, and an extended sideways periodat low volatility. By viewing performance over a number of smaller time intervals, insteadof only one average or total figure, it is possible to assess the stability of the strategy.Elimination of outlier periods. Realistically, a system should not base its profitabilityona single major market move over the test interval. In particular, if the move that gen-erated the profit was actually a price shock, the system had only a 50% chance of post-ing that profit, and has an equal chance of a Joss in the future. It may be practical toremove the profits generated on the first day or two of a move that began with a priceshock, allowing for the possibility of holding the wrong position in the future.||||||
 
524 TESTING
 
Concentrate on large profits and large losses. The largest profits and losses are critcal to the net performance of any system; therefore, you should make a special effortto study those trades. In doing that, you may find that the largest profits were theresults of price shocks, as discussed in the preceding point, or that there is an oddpiece of data that was not obvious before. If there are a number of large profits and nolarge losses, you may have excellent risk control, but more likely you have overfit thedata, There is an unfortunate tendency to look for problems when results are bad, butto accept them when they are good.
Reading between the Lines
s that produce large profits, while the overall
It is easy to show specific examples of ruleples. This
strategy is bad. Many books base intuitive proof or justification on specific examycan work if the general theory is sound, when there is fundamental or economic substan-tiation, such as expecting a seasonal low in the U.S. crops at harvest. For patterns and manypurely technical indicators, such as these using imomentum, this is not as clear. Results arelikely to show that the situations that fail greatly ‘outnumber the fewer exceptionally good
examples.
Systems That Work in Only One Market
From time to time, all traders receive mail offers for a highly specialized system called “TheCattle Trader,” the “Silver Day-Trading System,” or “Easy Profits through Stock Index Trad-ing.” It is most likely that each of these systems has been finely tuned with rules unique tone one market, Once aware of the optimization process and its results, these offers mustbe viewed more critically; after all, with optimization techniques able to test combinationsof indicators and rules, you could just as easily create a sysst that seemed to make asovach on 2 single market. To prove that you have actually found something that worksrequires an understanding of the rules and the way it W2s tested, or time to watch the sys-tem operate in the real market, If possible, a compariso® of the real trading profile com-pared with the expected performance based on testing should be available.
X
Student t-Test
Among tests that help determine whether the results are significant, the student ¢-test isone of the most useful. It tells you whether there is systematic bias in the data by show-ing whether the mean of the data is significantly different from zero. This is @ useful pieceof information when you try to decide whether the results of only a few trades represent agood system, or whether a series of losing trades implies that a system bas no value. For asingle set of trades produced by computer testing or by actual trading, the /-test is:
te average trade results
x Vaumberoftrades  -[standard deviation of trade results number .
‘The critical value of f, which cari be found in Appendix A, depends on the number of tradesin the sample. For the case of a series of trades produced from the same market and samesystem, the degrees of freedom = number of trades - 1. Using the table in Appendix A, ifthere were only 10 trades, the degrees of freedom would be 9, and the value of ¢ must begreater than 1.833 to reach the 95% confidence level. .
‘Where you want to know if wo systems are producing results that are significantly dif-ferent from each other, the technique is more complicated. It requires comparing the
 
 
3 Ben Warwick, Event Trading (irwin, 1996).
 
 
POINT-AND-FIGURE TESTING525
mean, i:ean,