cannot be predicted, and does it allat the expense of consistent trading profits.
DATA MINING AND OVEROPTIMIZATION
‘Advances in computing power and testing software Have made it very easy to test tradingstrategies using historic data. It is oniy logical that you should prove that your ideas wouldhave worked in the past. It is also sensible that you understand the amount of risk that youmust take to gain the results, For more sophisticated analysts, a relationship may be foundbetween fundamental factors, volatility, and risk. For fundamental analysts, the expectedprice of oil is greatly dependent upon the crude inventories; for many stock issues, the percapita disposable income is the key element in predicting revenues; and, the anticipatedvolatility of soybeans is often found in the size of the carryover stocks. Although past prices
 
DATA MINING AND OVEROPTIMIZATION say
 
TABLE 21-15 Test 2: January 2, 1990, throughJanuary 16, 1991*
 
 
 
Period NetPrft ROA MaxDD Trds5 ~6010 32.3 ~18610 3210 3660 22.3 16430 2015 12350 103.8 = 11895 1420 19810 345.4 5735, 1025 8410 105.5 -7970 1030 8420 868 9705 835, 18075 1348.9 -1340 540 tise 308.9 —3610 645 9335 304.1 3070 550 8265 246.7 -3350 3SS 2485 55.0 4515 760 9075, 184.8 4910 565 7155 100.4 -7130 770 3815 53.3 -7160 775 7195, 83.9 8580 980 —8540 -82.6 —10340 1285 3860 54.2 -7525 890 1960 35.7 5485 495 440 4 95 —4615 4100 3950 85.6 4615 2Trade Detail from TestingProfit! CumTrade Date B/S Price Loss PiMay 24,90 Buy 23.58May 25, 90 Sell 23.55 55 —55Jul 12,90 Buy 23.49 35 -20Oct 19,90 Sell 40.10 16585 16565Oct 30,90 Buy 41.23 LSS 15410Oct 31,90 Sell 41.92 665 16075Nov 19,90 Buy 38.39 3508 19580Nov 21,90 Sell 37.30 US, 18465Nov 26,90 Buy 40.62 -3345 15120Nov 27,90 Sell 40.53 115 15005Jan 7,91 Buy 35. 4805 19810
 
- * Testing stopped the day before the USS. retaliation.
‘Test selects 20-day moving average.Total profits are $3,225 without price shock. -Holding long when US. attacks Iraq,
and their relationships to economic data cannot be ignored, they can also be misleading,or even erroneous, if used carelessly.
The most important factor in the success of a trading program is a sound premise; thatis, you must find a real-life relationship between the price and the way participants react.For example, when the monetary authority of any country raises interest rates, the price ofequities tends to fall. Or, when there is had news for the economy, investors shift fromequities to guaranteed interest rates, driving the stock market lower and the bond markethigher. Once these relationships have been identified, it is perfectly sensible to look 
 
 
$50 TESTING
TABLE 21-16 Test 3:January 2, 1990, throughMarch 28, 1991*
 
 
 
 
Period NetPrft ROA MaxDD Trds5 ~17255 64.0 ~26975 4510 ~13195 45.0 -29295 3115 4480 26.6 —16840 2020 8430 67.6 12475 1625 4690 -22.6 20725 2030 =1395 ~6.4 -21775 1735 28335 21146 =1340 9)40 =360 =25 =14420 1245 18245 594.3 -3070 750 8265 246.7 ~3350 355 2485 55.0 4515 760 9075 1848 4910 565 7155 100.4 -7130 770 3815 53.3 -7160 775 7195 33.9 8580 980 ~18225 813 22410 1385 —12945 -70.4 ~18390 990 4065 40.3 —fo080 595 8645 -56.0 —15430 5100 =3485 -755 4615 5Trade Detail from TestingProfie/ CumTrade Date BIS Price Loss PILMay 24,90 Buy 23.58May 25,90 Sell 23.55 55 -35* Jat 16,90 Buy 24,16 -635 -690Nov 9,90 Sell 40.58 16395 15705Jan 14,91 ~ Buy 38.76 1795 (7500fan 16,91 Sell 39.36 575 18075Mar 8,91 Buy 29.89 9445, 27520Mar 13,90 Sell 30.89 975 28495Mar 19,90 Buy 30.75 115 28610Mar 28,90 Sell 30.50 -275 28335
—_——————* Test period includes both price shocks.‘Test selects 35-day moving average. .‘oral profits are $2,495 without price shocks.Optimization profits include both Iraq! invasion of Kuwait and the US. attack on iraq.
 
through historic data to uncover the size of the intereSt rate increase needed to move themarket, or the type of pattern shift from equities to bonds that could yield a profitableeen we begin looking for a relationship between inventories and price we are reasonably confident that there is a solution, and-we can even identify those elements thatform the solution. The process is one of careful reasoning and a small amount of testing,or validation. However, when we test a set of trading rules on historic data, we do not actu-ally know that this method will yield a profitable result.
 
DATA MINING AND UVEROPTIMIZATION 554
For example, take the simplest case of a trading system that only uses a moving aver-age trendline. A long position is signaled when the trendline turns up, and a short is gen-erated when the trendline turns down. Is this a sound premise? How do you decide that aprofitable test result will become profitable real-time trading? The process for understand-ing this is far from trivial.
Reali:
 
ic Assumptions
After you have decided on your underlying strategy, and before you begin testing, there area number of decisions to make. These must be made without the benefit of hindsight, butaccording to your own reasoning. They include:
1. The amount of data to be tested
2. The commission cost per unit traded
3. The slippage cost per unit traded
4, The range of each parameter (variable) to be tested
This last item, the range of each parameter, means that you must decide, in advance,which moving average periods will be tested. If this is to be a long-term program trying 'tocapitalize on the way governments manipulate interest rates or money supply, then youmight choose periods beginning with 50 days, up to 300 days. If you believe that the long-term is too risky and most trends are médium-term, then a range of 20 to 100 days mightbe best. The very fast day trader, looking to downplay the trend, would want to test peri-ods from 5 minutes to 1 day. It is important that the periods chosen for testing correspondto how you perceive the price movement. It is not enough to say, “I'll trade anything thatmakes money.” :
Commissions and slippage are also vital to the results, but more significant for theshort-term trader. As a system focuses on shorter holding periods, the profits per trademust become smaller; therefore, the commissions and slippage can become the differencebetween profits and losses. Because a computer assumes that orders can be executed per-fectly (at the signal price), even during a 5-minute bar where prices jumped 1% with virtu-ally no trading, it takes some effort to decide on a realistic level of slippage. Normally, it issafe to overestimate the cost of entering and exiting the market, but for a short-term trader,even a small overestimation could eliminate a potentially profitable trading technique.Accuracy, as near as possible, is the best approach. :
By making clear assumptions in advance, you gain the advantage of validating yourideas. You thought that medium-speed trends were likely to be best, but test results showthat only a narrow range of speeds are profitable. You must now rethink why thisoccurred. Using negative feedback to learn more about the market is a very healthyprocess.
Statistics Can Say Whatever You Want
Most computer software that allow historic testing give results that emphasize the highestreturns. Given a choice, most users of these programs want to see the highest returns. Asingle test result, however, should not be considered representative of the worth of the sys-tem being tested. -
If we test the same 10-year period of historic data for 1,000 different parameter values,spread over a wide range, we are likely to be pleased if 50 of those test results were veryattractive. Statistically, we might say that 50 are significant at the 5% level. But that is notnecessarily true. In a well-distribution, or candom, set of results, we can expect 5% to bestatistically significant. If the trading system is truly sound, then we would like to see muchmore than 5% successes.h
 
552 TESTING
Robustness -
Practically speaking, robustness translates into getting the most realistic results by usingthe fewest parameters and the most data. Robustness is a term used to describe a systemor method that works under many market conditions, one in which we have confidence; itis associated with a successful result using an arbitrary set of parameters for a system. Thisis most likely to occur when only a few parameters are tested over a long time period. Forexample, if you claim that a long-term moving average system is robust, then someoneunfamiliar with the system should be able to choose the moving average period of 130 daysand have reasonable expectations of a profit. This would be most likely if a large percent-age of all time intervals in the long-term test were profitable. ;
When fiest testing a system, it is best to avoid looking at the largest profits, but instead,study only the average values of all tests, that is, the average net profit, average number oftrades, average drawdown, and average profit per trade. In addition, looking at the stan-dard deviation of profits for all tests will give a very good idea of the consistency of returnsover the range of tests. _-
The following is a checklist along with a brief explanation that might help when tyingto follow a procedure to get robust test results:'*
"1. Deciding what to test. Testing should be used for validation of an existing idea, notfor discovery. ,a. Is the strategy logical? It should be based on careful observation of the market,
or an awareness of relationships between different factors that drive the market,or between different markets themselves.
b. Can you program all the rules? Programming a system guarantees that all of thecontingencies have been thought out. While not everything can be programmed,in particular situations requiring crisis intervention, these cases are few.
c. Does the strategy make sense only under certain conditions? A trading programcan be limited to a specific situation, and testing should reflect those limitations.
d. Take a guess as to the expected results. It is important to know if your estimateof thé system success is correct. Getting a good result for the wrong reason re-quires just as much rethinking of the problem as getting bad test results.
Deciding how to test. There are a number of steps necessary before yu can actu-
ally begin the test process.
a. Choose the testing tools and methods. Select a test platform, such as Trade-Station or MetaStock, which can speed up the test process and keep it objective.
b. Do you have enough of the right data? More data is best, but if there is less data
available, it should have a wide range of market conditions, such as bull and bear
markets, price shocks, and sustained sideways periods.
Will you test a full range of parameters? The nature of your trading strategy, for
example, long-term or short-term breakouts, may determine the parameters that
are tested, Test only the range that makes sense for the strategy. This serves to val-idate the concept and avoids discovering better techniques.
d. In what order will the parameters be tested? Test the most important parame-
ters first, such as the time period. More subtle parameters are often used for re-
fining the results, and may not be heipful at all. You can usually tell the mostimportant parameters because they cause the widest range of results. ;
Are the parameters distributed properly? Testing equal intervals of trend peri-
ods can be misleading. The difference between a 3- and 4-day moving average is
N
©
°
11.4 more detailed explanation of testing robustness can be found in Perry Kaufman, Smarter Trading (McGraw:Hill, 1995).
Sees una aeataiie'
 
DATA MINING AND v yEROPTIMIZATION 553
 
33%, while the difference between a 99- and 100-day average is only 1%. Whenyou test periods from 1 to 100 days, your average results are heavily weightedtoward the slow end, which has similar performance.
. Have you defined the evaluation criteria? You must know whether highest prof-its, highest reliability, or some other combination will be used to evaluate the re-sults. Risk-adjusted performance can be a very good objective criteria.
8. How will the output be presented? Results can look very different when pre-sented in a plain table. Two- and 3-dimensional contour maps can give a betterview of the way performance varies when two parameters change.
3. Evaluating the results. Careful review of the test results can avoid many hours ofunnecessary work. Looking at the detail of the performance, and even the individ-ual trades of a few selected results, will give you insight into the good and bad as-pects of your program.
a. Are the calculations correct? Just because calculations were done by a computerdoesn’t mean they are always correct. A person had to tell the computer what todo, and people make mistakes.
b. Were there enough trades to be significant? We have uncertainty when the re-sults show excellent performance on only a few trades. When there is notenough data, there is no way to be sure that the results are robust. The systemcan only be used if you have complete confidence in the underlying premise, andthe sparse results simply confirm that logic.
c. Does the trading system produce profits for most combinations of parameters?‘The best assurance of robustness is when most parameter combinations produceprofitable results, even though they vary.
d. Did logic changes improve overall test performance? When you improve the re-sults by adding a special rule, the average of all tests must increase, while thestandard deviation remains unchanged. Otherwise, you may have simply overfitpart of the data.
e. How did it perform on out-of-sample data?
nm
4. Choosing the specific parameters to trade. Choosing the parameters to be used intrading is another unique and difficult problem. Does the system that showed thepeak historic profits give you the best chance of getting the peak future profits, orwas it just chance?
a. Did the last test include the most recent data? If you have left out data to per-form an out-of-sample test, then respecify the parameters using the most recentdata. You must be as current as possible before trading.
b. Did you choose from a broad area of success? Parameters should be chosenfrom an area that showed consistent profits, but should not be determined by theone that had peak profits. It is best to look for average returns using average pa-rarneters.
©. Are profits distributed relatively evenly over the tested history? Although there
_may be attractive tétal profits, sustained periods of loss would not survive realtrading. Returns should be reasonably steady.
: Did the bistoric results show any large losses due to price shocks? It is not pos-sible to avoid large price shocks in real trading because you never know whenthey will come. Historic results that do not show large equity jumps due to priceshocks are not realistic, and result from overfitting. Even worse, those results thatonly profit from price shocks are benefiting from hindsight. In real trading, 50%of the price shocks should produce a windfall profit, while’ the other 50% willgive equal losses. 
 
554 TESTING
€. Have you risk-adjusted the returns to your acceptable risk level? The best com-parison between tests is to annualize all returns, tO avoid comparing unequaltime periods, and to risk-adjust the results. Risk adjusting changes each result sothat it has the same unit risk as every other test.
5. Trading and performance monitoring. The development of a system is never com-plete. There is a continuous evolution in the markets, seen in the changing partici-pation, volatility, interrelationships between markets, and the introduction of newprarkets, The only way to see how these changes affect your trading program is tocarefully monitor the system signals and the corresponding fills. From this evalua-tion you can reallocate funds, change the way you execute signals, and even findbetter rules. Without monitoring your trading, and comparing the results with yourexpectations, there is no basis for saying that your system is performing correctly.2. Are you following the same rules that were tested? tis often convenient to pro-
gram a simple version of your ideas, but trade a more complex set of rules. If so,you have no basis for comparison.
b. Are you trading the same data that was tested? The use of an artificial, continu-ous contract for testing futures will not give you the same results as testing eachcontract, beginning at the time it becomes the most active of the delivery months.Normally, artificial continuous prices are smoother than real prices and appear toperform better.
c. Are you monitoring the difference between the system and actual entries andexits? Slippage, the difference between the system price and the actual marketcnecution, can change performance from theoretically profitable to a real tradingloss. Monitoring this difference will tell you the realistic execution costs, themaximum volume that can be traded, and the best time of day to trade.
SUMMARY
‘This chapter has covered a lot of material vital to the successful development of a tradingprogram. Because there are so many issues, the following is a brief summary of the mostimportant concepts:
| Select a trading strategy with a sound underlying premise; complexity isnot necessary.
” Reserve out-ofsample data for validation. -
* Select the range of parameters to test that is logical, favoring the faster or slowertrading approach.
| perform the longest reasonable test to include as many market situations as possible.
” Test as many different markets as possible looking for a common pattern in the results.
” Standardize the results to facilitate comparisons between systems and test periods.
"Evaluate the robustness of the method rather than the peak performance; avoid finetuning.
8. Assess the effects of past price shocks on profits and risk.
9. Beta-test (paper trade) the system until you are certain that it is working properly:
UAW R BNE
You might also keep in mind that testing uses historié data and works best in real tradingwhen the current market has the same patterns seen during the test data. The premise of opti-mization is that the markets will continue to behave in the future as they have in the past. Thismay be why the index markets traditionally have been among the poorest system perform-ers—-because they are trading at price levels and volatility that have never been seen before.
Other areas of interest to readers actively testing and evaluating test performance willbe genetic algorithms, a new search technique, discussed in Chapter 20, and most ofChapter 23, especially the section “Measuring Risk.”
 
 
i\
 
22
Practical Considerations
 
his chapter discusses areas of technical and mathematical analysis that are not systemsby themselves, yet essential to the successful development of a system. The first partconcerns the use of the computer. Technical analysis and computers have become syn-onymous with each other and, with the simple steps provided by development software suchas TradeStation, MetaStock, and spreadsheets, it would be rare to trade a new technical sys-tem without first running it through a computer to simulate its past performance. Those whodo not use a computer directly are strongly influenced by the computer systems of others.The second section, “Price Shocks,” is the most underestimated deterrent in findingthe right system and generating trading profits. This idea has been discussed in different
» places during this book, specifically in Chapter 21 in the section “Anatomy of an Optimiza-
tion”; yet it cannot be overstated. Unpredictable events, once they become part of pricehistory, are often treated as though they could be anticipated. This can inflate performancereturns and minimize risk during strategy testing with serious consequences to actual trad-ing. This section will try to explain how to close the-gap between testing and trading.
‘The Theory of Runs is actually an application of gambling techniques, primarily Mar-tingales, to trading. What would seem more reasonable than assuming that the odds areagainst you in the futures market or as a short-term trader and treating it as a gambling sit-uation? The third section is about combining systems together that focus on different mar-ket characteristics, and the understanding of the trade-offs inherent in each type of analyticmethod. There is also a short analysis of trading limits to show some of the problems andflaws of moving from a test to an operating environment. The last section reviews the sim-larity of systems and the impact of technical systems on futures markets.
USE AND ABUSE OF THE COMPUTER
Make sure your present report system is reasonably clean and effective before you auto-mate. Otherwise your new computer will just speed up the mess.Robert Townsend’
Computers are not a substitute for thinking. They excel in performing the same tedioustask, over and over again, quickly and accurately—provided, of course, that correct infor-mation was entered. But even though technology has not yet reached the stage depicted inStar Wars, the computer is the only practical tool for evaluating trading ideas. This sectionwill consider both good and bad ways to approach a computer problem, none of whichcan be credited to or blamed on the machine. As a powerful tool, a computer can’t be beat;many of the systems, advancements, and refinements presented in this book could nothave been considered without it.
Computers come in many shapes and sizes, and can be purchased, leased, and bor:rowed. Their packaging has become second only to the car. During the 1980s, the averagelife of a computer generation was about 10 years; however, advances in technology have
4 Robert Townsend, Up the Organization (Knopf, New York, 1970, p. 36) 
556 PRACTICAL CONSIDERATIONS
 
reduced that to about 3. years, if you don’t mind a year when you feel obsolete. The pastfew years have seen a remarkable evolution in the computer industry. From an acceptanceof the personal computer (PC) to the point at which some households would rather giveup their television than their computer, the PC has revolutionized data processing andbrought the Internet into tens of millions of homes. Because not everyone knows how toprogram a computer, software companies have developed turn-key packages, programsthat are user-friendly, operated primarily by menu selection. The combination of these twofactors, cheap hardware and sophisticated software, has reached far into the realm of indi-vidual and institution investors.
One of the first and foremost organizations to provide convenient computer researchtools for the trader was the Technical Analysis Group (TAG), which began in 1975 as a user’sgroup. It provided menu-driven capabilities for analysis of daily price data and program-ming capability similar to a spreadsheet using its CompuTrac software. It provided dailyautomatic updates of data at a time when that luxury was the state of the art. The companythat breaks ground is not always the one that survives the race, and those analysts develop-ing systems now expect much more. Generally, development software falls into two groups:
1, Daily market analysis, with price updates at the end of the day2. Intraday trading platform, providing users with on-line data feeds to execute theirstrategies
Both types of software provide the ability to enter your trading strategies and test themon historic data. The software that operates on daily data is considered “off-line” and isused by traders who prefer to evaluate the market after it closes and prepare orders thatcan be executed the next day without having to constantly watch the market. The intradaytrading platform is much more complex and more expensive. It requires active data feedsand provides both real-time graphics and generates trading signals at the close of eachprice tick or bar, as defined by the user.
Acquiring Data
Daily historic data, essential to the development of trading systems, is readily available andmost often comes with the purchase of the software. If is common to see 25 years of his-toric data on all U.S. markets, but not as common to have European and Asian data pro-
“vided as well. Intraday data is available from a few major vendors and can be easily
obtained for at least 15 years on U.S. markets, but is also much more difficult to get on mar-kets outside the United States. Occasionally, there are sales on this data, and it can be pur-chased at a reasonable price. The Chicago Mercantile Exchange has made most of its tickdata available on compact disk, but it is necessary to convert their form into one that canbe read by your trading strategy software.
Of the non-U.S. firms, London’s LIFFE has been providing tick data for the longesttime, and 10 to 15 years is available for their oldest markets. France's MATIF is just nowproviding tick data, as is Germany's DTB, Singapore’s SIMEX, and many other exchanges.The form of the data provided is very similar, but all of them are incompatible with theteading strategy software and’ require what is called data reduction. Because theexchanges have taken the policy that providing data facilitates trading on their markets,there is usually no cost for it; this gives you some latitude to pay a programmer to write aprogram to convert this into your own format.
  
Cleaning the Dato . -
Occasionally, you will find that your strategy was immensely profitable, or lost muchmore than you expected. When you test 20 years of data, you may not look at each trade
 
 
USE AND ABUSE Or [HE COMPUTER 557
 
but accept the results as correct and rationalize the outcome as a sequence of bad orgood trades. You might be surprised to learn that there can be many erroneous prices inyour data. It is an incredible waste of time to use data without first reviewing it to finderrors.
Tick data is notorious for including bad data with the good, simply because of themagnitude of the problem. Most daily data is already clean. The easiest approach tochecking your data is to display it as a chart and scan through all of the markets that youplan to use. Bad data points are obvious once seen on a screen. A low price of zero willshow up as a vertical line going to the bottom of the chart; this will result in the correctdata points being scaled down to look as though they are just a horizontal line. You canthen find the date of this point and correct it (provided your software has a correctionfeature).
If you can program the computer yourself, you will want to scan the data looking foroutliers, those data points that are at least 10% away from the previous data points. Thedata provider should have prescanned the data to be sure that there are no intraday pricesthat were outside the end-of-day high and fow prices transmitted by the exchanges, but thisis not usually the case, and you will find that the most obvious errors have been over-looked. There are more subtle errors that can be found and corrected, but the largest onesmust be fixed before you use the data. For testing, even one efror per year is enough tomake all of the tests useless. Don’t forget that, if you remove a price instead of correctingit, you must reduce the tick volume as well.
Data Updates
Iris not difficult to remember when all traders kept a copy of the Wall Street Journal andclipped the price pages so that data could be entered into their computer. These pages,kept in a file drawer, became their historic database and absolute reference. This doesn'tmake sense any more. Automatic end-of-day updates are available to anyone at very littlecost. They provide complete and accurate data as well as correcting any past data thatmight have been wrong. Some exchanges post daily prices on their Web sites and can beretrieved for free, but this requires added effort to convert that data into a form acceptableby your database. Most data providers will accommodate a number of different formats sothat data received each day simply gets added to your historic data and is immediatelyready to use. Don’t do it manually. Your time is best spent finding a profitable trading stcat-egy rather than fiddling with data.
 
Not only must you have a complete database of accurate, historic data, but you must keepthat data current. Even if you have a live data feed, the data management of your systemcannot store an unlimited amount of ticks or days before it begins to discard the oldest val-ues. Most software will have a specific number of data points that it will hold for intradaydata, and much more for daily data. For example, it is common to have only 3 months oftick or 15-minute data, or the computer may hold 4,000 entries each of 1-minute,5-minute, 15-minute, and daily data, which mean about 16 years of daily data but only 4months of 15-minute data. This design may be important when you choose the system youuse. By being aware of how much data is held, you will need to remember to archive the15-minute data before each 4-month period elapses; otherwise, you will lose the data andyour history will not be continuous.
Combining Standard Techniques into a SystemStrategy development programs provide a wide assortment of standaid techniques in theform of indicators and systems. You can pick a stochastic oscillator, a moving average, or 
 
558 1 .ACTICAL CONSIDERATIONSan N-day breakout and combine them in various ways to evaluate markets over differenttime horizons, such as a"1