test and a confidence interval to lead to different conclu-sions about a rule’s expected return. This prospect stems from the factthat a hypothesis test focuses on the right tail of the sampling distributionwhereas the confidence interval focuses on the left tail of the samplingdistribution. This means it is possible for the lower bound of a 90 percentconfidence interval to imply that there is a 5 percent probability that arule’s expected return is less than zero, while a hypothesis test conductedat the .05 significance level rejects H,,. In other words, it is possible for theHypothesis Tests and Confidence Intervals 251
confidence interval to tell us that the rule does not have predictive powerwhile the hypothesis tells us that it does. In theory, the hypothesis testand the confidence interval should come to the same conclusion on thisissue. That is to say, if the lower bound of a 90 percent confidence inter-val tells us that the rule’s expected return is less than 0 percent, then ahypothesis test at the .05 significance level would presumably not rejectthe null hypothesis.
Conflicting conclusions can arise when the sampling distribution isnot symmetrical (i.e., is skewed to the right or left). This is illustrated inFigure 5.22. The sampling distribution, which is clearly skewed to the left,is shown in two positions. In the lower portion of Figure 5.22, the sam-pling distribution is positioned as it would be for conducting a test of H,.Because less than 5 percent of the sampling distribution’s right tail liesabove the mean return of the back-tested rule, the test indicates the rule issignificant at the .05 level. In other words, H, can be rejected in favor ofthe alternative hypothesis, H,, that claims the rule has an expected returnthat is greater than zero.
The upper portion of Figure 5.22 shows the sampling distribution asit would be positioned to construct a 90 percent confidence interval us-ing the bootstrap percentile method. Note that the lower bound of theconfidence interval is below a zero rate of return. This tells us there isgreater than a .05 probability that the rule’s true rate of return is lessthan zero. In other words, the 90 percent confidence interval leads to aconclusion that is opposite to the conclusion of the hypothesis test. The
 
Mean Back-Test ReturnHistorical Sample
 
 
 
Lower Bound i90% Confidence i Sampling DistributionInterval Below 0 i Positioned forConfidence Interval
 
 
    
 
 
 
 
 
 
 
 
 
 
0Sampling Null iDistribution Hypothesis | : Less than .05Positioned for i of Sampling DistributionTest of A Above Mean ReturnNull Hypothesis ;0 10
FIGURE 5.22 Potentially conflicting conclusions: hypothesis test versus confi-dence interval.252 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
figure shows that the ambiguity is due to the asymmetrical shape of thesampling distribution.
Fortunately this problem does not afflict the research conducted inthis book, which uses the sample mean as a performance statistic. Theall-important Central Limit Theorem assures us that the sampling distri-bution of the mean will not be seriously skewed (asymmetrical) as longas the sample size is large. To recap, the Central Limit Theorem tells usthat, as sample size increases, the sampling distribution of the meantends toward a symmetrical bell shape. Other performance statisticsmay not behave this way. In situations where the sampling distributionis not symmetrical, there are bootstrap techniques, which involve pivot-ing the sampling distribution to alleviate this problem. Unfortunately,these other methods can have problems of their own. In any case, thisall makes the mean return an attractive performance statistic to use inrule testing.
Confidence Intervals for the TT-4-91 Rule. This section gives anexample of the confidence interval for the rule TT-4-91. Figure 5.23 showsthe 80 percent confidence interval superimposed on the bootstrapped
 
0.12 T
+— 80% —>
0.10 eee
0.08 ferret fa
0
0.04 [nnn
0.02 |——-----
 
 
 
0.15  -0.10 -0.05 0 0.05 0.10 0.15
FIGURE 5.23 > Sampling distribution and 80 percent confidence interval forrule TT-4-91.Hypothesis Tests and Confidence Intervals 253
sampling distribution positioned at the rule’s observed back-tested returnof +4.84 percent. The lower bound of the 80 percent confidence interval is+0.62 percent. The upper bound is +9.06 percent. This tells us that if ruleTT-4-91 were to be back tested on 100 independent samples of data and an80 percent confidence interval were to be placed around the mean returnin each sample, in approximately 80 of the samples, the true expected re-turn of the rule would be enclosed by the confidence interval.Data-MiningBias: The Fool’s
Gold ofObjective TA
 
 
observed performance is selected. That is to say, data mining involves a
performance competition that leads to a winning rule being picked. Theproblem is that the winning rule’s observed performance that allowed it tobe picked over all other rules systematically overstates how well the rule islikely to perform in the future. This systematic error is the data-mining bias.
Despite this problem, data mining is a useful research approach. Itcan be proven mathematically that, out of all the rules tested, the rulewith the highest observed performance is the rule most likely to do thebest in the future, provided a sufficient number of observations are usedto compute performance statistics.! In other words, it pays to data mineeven though the best rule’s observed performance is positively biased.This chapter explains why the bias occurs, why it must be taken into ac-count when making inferences about the future performance of the bestrule, and how such inferences can be made.
I begin by introducing this somewhat abstract topic with several anec-dotes, only one of which is related to rule data mining. They are appetizersintended to make later material more digestible. Readers who want to starton the main course immediately may choose to skip to the section titled“Data Mining.”
The following definitions will be used throughout this chapter and areplaced here for the convenience of the reader.
|: rule data mining, many rules are back tested and the rule with the best
¢ Expected performance: the expected return of a rule in the immedi-ate practical future. This can also be called the true performance ofthe rule, which is attributable to its legitimate predictive power.256 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
¢ Observed performance: the rate of return earned by a rule ina backtest.
¢ Data-mining bias: the expected difference between the observedperformance of the best rule and its expected performance. Expecteddifference refers to a long-run average difference that would be ob-tained by numerous experiments that measure the difference be-tween the observed return of the best rule and the expected return ofthe best rule.
¢ Data mining: the process of looking for patterns, models, predictiverules, and so forth in large statistical databases.
¢ Best rule: the rule with the best observed performance when manyrules are back tested and their performances are compared.
¢ In-sample data: the data used for data mining (i.e., rule back testing).
¢ Out-of-sample data: data not used in the data mining or back-testingprocess.
¢ Rule universe: the full set of rules back tested in a data miningventure.
¢ Universe size: the number of rules comprising the rule universe.
FALLING INTO THE PIT: TALES OF THEDATA-MINING BIAS
 
This following account is apocryphal. Some years ago, before I took upthe study of statistics, I was approached by an impresario seeking backersfor a show business venture that he claimed would be enormously prof-itable. The show was to feature a monkey that could write Shakespearianprose by dancing on the keyboard of a word processor.
At each show, the literate monkey, whom the promoter had named theBard, would be placed at a keyboard and a large screen would display toan audience what the monkey wrote, as it was being written. Surely, peo-ple would rush to see the Bard, and my share of ticket sales would yield ahandsome return on the required investment of $50,000. At least, that wasthe promoter’s claim. “It can’t miss!” was his refrain.
I was intrigued, but wanted some proof that the monkey could actu-ally produce Shakespearian prose. Any investor would. I was given proof... of asort. It was what accountants call a cold-comfort letter. The lettersaid, “We have examined the Bard’s previous works and he has in factwritten the words ‘To be or not to be, that is the question.’ We are, how-ever, unfamiliar with the circumstances under which these words werewritten.”
What I really wanted was a live demonstration. Regrettably, myData-Mining Bias: The Fool’s Gold of Objective TA 257
request could not be accommodated. The impatient promoter explainedthe monkey was temperamental and besides, there were many otheranxious investors clamoring to buy the limited number of shares beingoffered. So I seized the opportunity and plunked down $50,000. I wasconfident it was just a matter of time before the profits would startflowing in.
The night of the first show arrived. Carnegie Hall was packed to ca-pacity with a crowd that anxiously awaited the first words. With every-one’s eyes glued to the big screen, the Bard’s first line of text appeared.
lkas1dlk5jf wo44iuldjs skOek 123pwkdzsdidip’adipjasdopiksd
Things went downhill quickly from there. The audience beganscreaming for refunds, I threw up, and the Bard defecated on the key-board before scampering off the stage. My investment went up in a cloudof smoke.
What happened? Thinking it unimportant, the promoter failed todisclose a key fact. The Bard had been chosen from 1,125,385 othermonkeys, all of whom had been given the opportunity to dance on akeyboard every day for the past 11 years, 4 months, and 5 days. A com-puter monitored all their gibberish to flag any sequence of letters thatmatched anything ever written by Shakespeare. The Bard was the firstmonkey to ever do so.
Even in my state of statistical illiteracy, I doubt I would have investedhad I known this. Mere common sense would have told me that chancealone favored the occurrence of some Shakespearian quote in such alarge mass of nonsense. The freedom to data mine the trillions of lettersgenerated by an army of monkeys raised the probability of a lucky se-quence of letters to a virtual certainty. The Bard was not literate, he wasjust lucky.
The fault, Dear Brutus, lay with the sincere but statistically naivepromoter. He was deluded by the data-mining bias and attributed toomuch significance to a result obtained by data mining. Despite my loss,I’ve tried not to judge the promoter too harshly. He sincerely believedthat he had found a truly remarkable monkey. He was simply misled byintuition, a faculty inadequate for evaluating matters statistical andprobabilistic.
By the way, the promoter has kept the Bard as a pet and still allowshim to dance on that once-magical keyboard in hopes of new evidence ofliteracy. In the meanwhile, to keep body and soul together, he is now sell-ing technical trading systems developed along similar lines. He has athrong of dancing monkeys developing rules, some of which seem to workquite well, in the historical data.258 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Proving the Existence of God with Baseball Stats
Collectors of sports statistics have also been seduced by the data-miningbias. For example, there is Norman Bloom, who concluded that interest-ing and unusual patterns found in baseball statistics prove the existenceof God. After thousands of searches through his database, the dedicateddata miner found patterns he believed to be so amazing they could only beexplained by a universe made orderly by God.
One of Bloom’s patterns was as follows: George Brett, the third base-man for Kansas City, hit his third home run in the third game of the play-offs, to tie the score 3-3. Bloom reasoned that, for the number three to beconnected in so many ways, compelled the conclusion it was the handi-work of God. Another interesting pattern discovered by Bloom had to dowith the stock market: The Dow Jones Industrial Average crossed the1,000 level 13 times in 1976, miraculously similar to the fact that therewere 13 original colonies that united in 1776 to form the United States.
As pointed out by Ronald Kahn,? Bloom committed several errors onthe way to his unjustified conclusions. First, he did not understand therole of randomness and that seemingly rare coincidences are in fact quiteprobable if one searches enough. Bloom found his mystical patterns byevaluating thousands of possible attribute combinations. Second, Bloomdid not specify what constituted an important pattern before he began hissearches. Instead, he took the liberty of using an arbitrary criterion of im-portance defined after the fact. Whatever struck his fancy as interestingand unusual was deemed to be important. Kahn points out that one isguaranteed to discover “interesting” patterns when they are searched forin such an undisciplined manner.
Discovering Hidden Predictions inthe Old Testament
Even Bible scholars have fallen into the data mining pit. In this instance,the well intentioned but statistically unsophisticated researchers foundpredictions of major world events encoded in the text of the Old Testa-ment. Knowledge of the future would certainly imply that the words hadbeen inspired by an omniscient Creator. However, there was one smallproblem with these predictions, known as Bible Codes. They were alwaysdiscovered after the predicted event had taken place. In other words, thecodes predict with 20/20 hindsight.’
The Bible Codes are clusters of words imbedded in the text that arediscovered by linking together letters separated by a specific number ofintervening spaces or other letters. These constructed words are referredto as equal letter sequences or ELS. Code researchers grant themselvesData-Mining Bias: The Fool’s Gold of Objective TA 259
the freedom to try any spacing interval they wish and allow the wordscomprising the cluster to be arranged in virtually any configuration solong as the cluster occurs in what the researcher deems! to be a compactregion of the original text. What constitutes a compact region and whatwords constitute a prediction are always defined after a code has beendiscovered. Note the use of an evaluation criterion defined after the fact.This is not scientifically kosher.
The Bible Code scholars contend that the occurrence of a code is sostatistically unlikely that it can only be explained by its having been putthere by God. Their fundamental error—the error made by all naive dataminers—is the failure to understand that given enough searching (datamining), the occurrence of such patterns is actually highly probable. Thus,it is likely that researchers will find codes that correspond to names,places, and events of historical importance. For example, the word 1990in the same region of text as Saddam Hussein and war are not rareevents requiring a metaphysical explanation. However, when found in1992, after the first Iraq war had taken place, the word pattern seeminglypredicted the 1990 Iraq war. Bear in mind that the words Bush, Iraq, inva-sion, and desert storm would serve just as nicely as a code that also ap-pears to predict the 1990 Iraq war. Indeed, there are a huge number ofword combinations that would correspond to the 1990 war, after the par-ticulars of that historical event are known.
In his 1997 book, The Bible Code, author Michael Drosnin, a journalistwith no formal training in statistics, describes the research of Dr. EliyahuRips. Dr. Rips is an expert in the mathematics of group theory, a branchof mathematics that is not particularly relevant to the problem of data-mining bias. Though Drosnin claims that the Bible Codes have been en-dorsed by a roster of famous mathematicians, 45 statisticians whoreviewed Rips’s work found it to be totally unconvincing.’ Data-miningbias is, at its heart, a problem of faulty statistical inference.
Statisticians take a dim view of the unconstrained searching prac-ticed by Bible Code researchers. It commits a mathematical sin called theexcessive burning of degrees of freedom. To the statistical sophisticate,the stench produced by this incineration is most foul. As pointed out byDr. Barry Simon, “A Skeptical Look at the Torah Codes,”® in Chumash,just one of the 14 books comprising the Torah, approximately 3 billionpossible words can be produced from the existing text when the ELSspacing interval is allowed to vary from 1 to 5,000. Searching this set ofmanufactured words for interesting configurations is no different thansearching through tons of gibberish written by an army of monkeys danc-ing on keyboards.
The foolishness of the Bible Code scholars’ search algorithms be-comes apparent when they are applied to non-Biblical texts of similar260 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
length, such as the Sears catalogue, Moby Dick, Tolstoy’s War and Peace,or the Chicago telephone directory. When these texts are searched, codedafter the fact predictions of historical events are also found. This suggeststhe codes are a by-product of the search method and not of the text beingsearched.
In a more recent book by Drosnin, The Bible Code II: The Count-down, he shows how the codes “predicted” the terrible events of 9/11/01.Why, you ask, didn’t he warn us before the events happened? He did notbecause he could not. He discovered the predictions after the tragedy oc-curred. Drosnin is another example of a well-intentioned but naive re-searcher fooled by the data-mining bias.
Data-Mining Newspaper Reporters
Newspaper reporters have also been duped by the data-mining bias. In themid 1980s, they reported the story of Evelyn Adams, who had won theNew Jersey state lottery twice in four months.’ Newspaper accounts putthe probability of such an occurrence at | in 17 trillion. In fact, the proba-bility of finding a double winner was far higher and the story far lessnewsworthy than the reporters had thought.
The before-the-fact (a priori) probability that Ms. Adams or any otherindividual will win the lottery winner twice is indeed 17 trillion to one.However, the after-the-fact probability of finding someone who has al-ready won twice by searching the entire universe of all lottery players isfar higher. Harvard statisticians Percy Diaconis and Frederick Mostellerestimated the probability to be about 1 in 30.
The qualifier after-the-fact is the key. It refers to a perusal of data afteroutcomes are known. Just as the probability that any individual monkeywill in the future produce a Shakespearian quote is extremely small, theprobability that there exists some monkey, among millions of monkeys,that has already produced some literate prose, is substantially higher.Given enough opportunity, randomness produces some extraordinary out-comes. The seemingly rare is actually quite likely.
Mining the UN Database for Goldand Finding Butter
David J. Leinweber, on the faculty of California Institute of Technologyand formerly a managing partner at First Quandrant, a quantitative pen-sion management company, has warned financial market researchersabout the data-mining bias. To illustrate the pitfalls of excessive search-ing, he tested several hundred economic time series in a UN database toData-Mining Bias: The Fool’s Gold of Objective TA 261
find the one with the highest predictive correlation to the S&P 500. Itturned out to be the level of butter production in Bangladesh, with a cor-relation of about 0.70, an unusually high correlation in the domain of eco-nomic forecasting.
Intuition alone would tell us a high correlation between Bangladeshbutter and the S&P 500 is specious, but now imagine if the time serieswith the highest correlation had a plausible connection to the S&P 500.Intuition would not warn us. As Leinweber points out, when the totalnumber of time series examined is taken into account, the correlationbetween Bangladesh butter and the S&P 500 Index is not statisticallysignificant.
The bottom line: whether one searches sport statistics, the Bible, therandom writings of monkeys, the universe of lottery players, or financialmarket history, data mining can lead to false conclusions if the data min-ing bias is not taken into account.
THE PROBLEM OF ERRONEOUS KNOWLEDGEIN OBJECTIVE TECHNICAL ANALYSIS
 
TA is comprised of two mutually exclusive domains—subjective and ob-jective. To recap, objective TA is confined to methods that are sufficientlywell defined that they can be reduced to a computerized algorithm andback tested. Everything else is, by default, subjective TA.
Both domains are contaminated with erroneous knowledge, but thefalsehoods are of very different types. The loosely defined propositionsof subjective TA carry no cognitive freight, generate no testable predic-tions,* and are, therefore, shielded from empirical challenge. Withoutthis crucial procedure for excising worthless ideas, falsehoods accumu-late. As a result, subjective TA is not a legitimate body of knowledge buta collection of folklore resting on a flimsy foundation of anecdote andintuition.
Objective TA methods have the potential to be valid knowledge butonly if back-test results are considered in light of randomness (sam-pling variability) and data-mining bias. Because many objective practi-tioners are unaware of these effects, falsehoods accumulate in thisdomain as well.
Erroneous objective TA manifests as out-of-sample’ performance de-terioration—a rule that performs well in the sample used for back testingbut then performs worse in out-of-sample data. This problem is illustratedin Figure 6.1.262 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
CumulativeGains
           
ObservedPerformance50%. ROI
   
In-Sample Out-of-sample
 
FIGURE 6.1 Out-of-sample performance deterioration.
Explanations for Out-of-Sample Deterioration:Old and New
Out-of-sample performance deterioration is a well-known problem.” Ob-jective technicians have proposed a number of explanations. I propose arelatively new one, based on data-mining bias.
The least plausible explanation attributes the problem to random vari-ation. Although it is true that a rule’s performance will vary from one sam-ple of history to another due to sampling variability, this explanation doesnot even fit the evidence. If random variation were responsible, then out-of-sample performance would be higher than in-sample performanceabout as frequently as it is lower. Anyone experienced with back testingknows that out-of-sample performance is inferior far more often.
A second rationale is that the market’s dynamics changed when theout-of-sample period began. It is reasonable to assume that financialmarkets are nonstationary systems.'! However, it is not reasonable toassume that each time a rule fails out of sample it is because market dy-namics have changed. It would be odd, almost fiendish, for the marketto always change its ways just when a rule moves from the technician’slaboratory to the real world of trading. It is simply implausible to sug-gest that a market’s dynamics change as frequently as rules fail out ofsample.
An even more elaborate explanation, also based on changing marketdynamics, invokes the additional assumption that the change occurred be-cause the rule has been adopted by too many traders. This story assertsthat the buying and selling of the traders using the rule has destroyed theData-Mining Bias: The Fool’s Gold of Objective TA 263
market pattern that accounted for the rule’s back-test profitability. This ra-tionale also lacks plausibility. Given that there are an almost infinite num-ber of rules that could be formulated, it would be unlikely that a largeenough number of participants would adopt any particular rule. Evenwhen numerous traders adopt similar rules, as is the case with futurestrading funds that employ objective trend-following methods, reducedrule performance seems to be due to changes in market volatility that arenot related to the usage of technical systems.”
A more plausible explanation’ of out-of-sample performance break-down is based on randomness. It asserts that market dynamics are a com-bination of systematic or patterned behavior and randomness or noise.Valid rules exploit the market’s systematic features. Because the system-atic component should continue to manifest in out-of-sample data, it is adependable source of rule profits. In contrast, the random component is anonrecurring phenomenon that will manifest differently in each sample ofdata. It is reasoned that a portion of a rule’s in-sample performance wasluck—a coincidental correspondence between the rule’s signals and themarket’s nonrecurring noise. These lucky profits will not manifest out ofsample, and so future performance will fall below past performance. Thisexplanation is closer to the mark, but it is incomplete. Randomness is onlyone villain.
Amore complete account of out-of-sample performance deteriorationis based on the data-mining bias. It names two villains: (1) randomness,which is a relatively large component of observed performance and (2)the logic of data mining, in which a best-performing rule is selected afterthe back-tested performances of all tested rules are available for the dataminer's examination. When these two effects combine, they cause the ob-served performance of the best rule to overstate its future (expected) per-formance. Thus, it is likely that the best rule’s future performance will beworse than the level of performance that caused it to be selected from theuniverse of rules tested.
The data-mining bias explanation for out-of-sample performancedeterioration is superior to the one based on changed market dynamics.Both explanations fit the data in the sense that they are both consistentwith the fact that out-of-sample performance tends to be worse. How-ever, the latter explanation invokes the assumption that market dynam-ics have changed. The data-mining bias explanation does not. It simplysays that the process of data mining favors the selection of rules thatbenefited from good luck during the back test. When choosing among aset of explanations that are equally adept at accounting for a phenome-non, it is wisest to choose the one making the fewest assumptions. Thisis the principle of simplicity known as Occam’s Razor, which was dis-cussed in Chapter 3.264 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
As will be shown, the observed performance of a rule can be decom-posed into two independent components: randomness and the rule’s in-herent predictive power. Of these two, randomness is by far the moredominant component. Thus, the rule with the best-observed performancewas most likely the beneficiary of good luck. That is to say, randomnessboosted the rule’s performance above the level attributable to its true pre-dictive power, if it possessed any. The opposite is the case for the rulewith the worst observed performance. It was most likely negatively im-pacted by luck. By selecting the rule with the highest observed perfor-mance, the data miner winds up picking a rule that experienced a largecomponent of good luck. Because luck cannot be counted on to repeat,this rule’s out-of-sample performance is likely to settle back toward a levelthat is representative of its inherent predictive power. This makes it prob-able that out-of-sample performance will be lower than the level of perfor-mance that allowed the rule to win the performance competition. Thebottom line: out-of-sample performance deterioration of the best rule ismost probably a fall from an unrealistically high expectation rather thanan actual decline in the rule’s predictive power.
DATA MINING
 
Data mining is the extraction of knowledge, in the form of patterns,rules, models, functions, and such, from large databases. The limita-tions and biases of human intelligence make this task nearly impossiblefor the unaided mind when the knowledge involves multiple variables,nonlinear relationships, or high levels of noise. Thus, data mining relieson computerized algorithms for knowledge extraction. An excellent dis-cussion of the principal methods used in data mining can be found inElements of Statistical Learning,’ Predictive Data Mining: A Practi-cal Guide,’ and Data Mining: Practical Machine Learning Tools andTechniques.'®
Data Mining as a Multiple Comparison Procedure
Data mining is based on a problem solving approach called a multiplecomparison procedure’ (MCP). The basic idea behind an MCP is to testmany different solutions to the problem and pick the one that performsthe best according to some criterion. Three elements are required to applyan MCP: (1) a well-defined problem, (2) a set of candidate solutions, and(3) a figure of merit or scoring function that quantifies the goodness ofeach candidate’s performance. After all scores are in, they are comparedData-Mining Bias: The Fool's Gold of Objective TA 265
and the candidate with the highest score (best-observed performance) isselected as the best solution for the problem.
Consider how this problem-solving paradigm applies in the context ofrule data mining:
1. The problem: timing long and short positions in a financial market togenerate profits.
2. A set of candidate solutions (solution universe or solution space):a set of rules proposed by the objective technician.
3. Figure of merit: a measure of financial performance such as the av-erage rate of return over a historical test period, Sharpe ratio, returnto Ulcer Index.'®
The performances of all rules are determined by back testing, and therule achieving the highest performance is selected.
Rule Data Mining as a Specification Search
Data mining can be understood as a specification search. That is to say, itis a hunt for the specifications of the rule that produces the highest perfor-mance. The specifications are a set of mathematical and/or logical opera-tions applied to one or more market data series thus transforming theminto a time series of market positions dictated by the rule.
Suppose the rule defined below turned out to be the best perform-ing rule:
Hold a long position in the S&P 500 if the ratio of Dow Jones Trans-portation Average close divided by the S&P 500 close is greater thanits 50-day moving average, else hold a short position.
The rule is specified by two mathematical operators: ratio and mov-ing average; two logical operators: the inequality operator greater-than,the conditional if, then, else; a single numerical constant: 50; and two dataseries: Dow Jones Transports and S&P 500. Data mining discovered thatthis set of specifications produced better performance than any alterna-tive set of specifications that was tested.
Types of Searches
Data mining searches range from the simple to the sophisticated. One ofthe ways in which they differ is how broadly the search universe is de-fined. This section considers three definitions of the search universe,266 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
starting with the narrowest search and progressing to the broadest. All ap-proaches to data mining, whether they search simply and narrowly orbroadly using the most advanced methods are subject to data-mining bias.
Parameter Optimization. The narrowest form of data mining is pa-rameter optimization. Here, the search universe is confined to rules withthe same form differing only in terms of their parameter values. Thus,the search is restricted to finding the parameter value(s) of the highest-performing rule of a particular form.
An example of a rule form would be the dual-moving-average crossoverrule. It is specified by two parameters; look-back periods for the short-termand long-term moving average. Signals are given when the short-term mov-ing average crosses above (buy) and below (sell) a longer-term