moving av-erage. Parameter optimization searches for the pair of parameter valuesthat yields the highest performance.
The maximum number of dual-moving-average crossover rules thatcan be searched is equal to the product of the number of values tested forthe short-term parameter and number of values tested for the long-termparameter. A search that considers all these combinations is referred to asan exhaustive or brute-force search.
There are more intelligent search methods that restrict the search tocombinations that are more likely to yield good results. This is accom-plished by using the performances of parameter combinations tested earlyin the search to guide the combinations that will be tested at later stages.One such intelligent search method is the genetic algorithm, a techniquebased loosely on the principles of biological evolution. Its demonstratedability to find parameter combinations that are close to optimal relativelyquickly makes it especially useful when the number of possible parametercombinations is high. Genetic algorithms have also proven to be effectivewhen performance is impacted strongly by randomness, which rendersmore conventional guided search methods based on calculus impractical.Excellent discussions of optimization methods can be found in Pardo,"and a review of various advanced methods can be found in Katz and Mc-Cormick” and Kaufman.”!
Rule Searching. A broader version of data mining is rule searching.Here, the universe of rules differ in their conceptual form as well as theirparameter values. The dual-moving-average crossover rule is one formal-ism for trend-following. Thus, it is simply one form in the broader cate-gory of trend-following rules that also include channel breakouts,moving-average bands, and so forth. The general category of trend-following rules is merely one technical analysis category among othersthat include counter-trend (mean-reversion) rules,”” extreme value rules,Data-Mining Bias: The Fool's Gold of Objective TA 267
divergence rules,”’ diffusion rules, and so forth. Each category can be real-ized with numerous specific rule forms.
Part Two of this book presents a data-mining case study that is basedon rule searching. The study will focus on three rule categories or themes:trends, extreme-values and transitions, and divergence. Each theme is re-alized with a specific rule form, which is defined in Chapter 8.
Though rule searching considers a multitude of rule forms, each rule’scomplexity remains fixed throughout the course of the search. Complex-ity refers to the number of parameters required to specify the rule. Inother words rule searching, as defined herein, does not involve combiningsimple rules to produce more complex rules
Rule Induction with Variable Complexity. The broadest andmost ambitious form of data mining is rule induction. Here the searchconsiders rules of undefined complexity. As the search proceeds, rulesof ever-greater complexity are considered. A complex rule might bethought of as a composition of a simpler rule conjoined by logical oper-ators or combined with a mathematical function as in a multivariatemodel. Thus rule-induction data mining is concerned with finding therule of optimal complexity.
In contrast to less ambitious forms of data mining in which a rule'scomplexity is defined at the outset of the search, rule induction uses ma-chine learning (autonomous induction) to find the degree of complexitythat produces the best performance. One scheme for rule inductionstarts by testing individual rules. Next, pairs of rules are considered tosee if their performance is better than the best individual rule. Gradu-ally, progressively more complex rules are tested and evaluated. In ef-fect, rule-induction methods learn how to combine rules to optimizeperformance.
Genetic algorithms, neural networks, recursive partitioning, kernel re-gression, support-vector machines, and boosted trees are some of themethods used for these most ambitious of data-mining ventures. An excel-lent discussion of various methods and supporting statistical theory canbe found in The Elements of Statistical Learning by Hastie, Tibshirani,and Friedman.”
OBJECTIVE TA RESEARCH
 
The goal of the objective technician is the discovery of a rule(s) that willbe profitable in the future. The research method is back testing, whichproduces an observable measure of performance. On the basis of this268 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
statistic, an inference is made about a population parameter, the rule’sfuture or expected performance. It can be said, therefore, that theessence of objective TA is statistical inference.
Why Objective Technicians Must Data Mine
The problem of out-of-sample performance failure has encouraged somepractitioners of objective TA to reject data mining. This is neither a wisenor viable position. Today, an objective technician who refuses to datamine is like the taxi driver who refuses to abandon the horse-drawn car-riage—a charming relic but no longer suitable for getting to one’s destina-tion efficiently.
Several factors compel the adoption of data mining as the preferredmethod of knowledge acquisition. First, it works. Experiments presentedlater in this chapter will show that under fairly general conditions, thegreater the number of rules back tested the greater the likelihood of find-ing a good rule.
Second, technological trends favor data mining. The cost effective-ness of personal computers, the availability of powerful back-testing anddata-mining software, and the availability of historical databases nowmake data mining practical for individuals. As recently as a decade ago,the costs limited data mining to institutional investors.
Third, at its current stage of evolution, TA lacks the theoreticalfoundation that would permit a more traditional scientific approach toknowledge acquisition. In developed sciences, such as physics, a singlehypothesis can be deduced from established theory and its predictionscan be tested against new observations. In the absence of theory, adata-mining approach is called for, in which a multitude of hypotheses(rules) are proposed and tested. The risk of this shotgun approach isfool’s gold—rules that fit the data by accident. Steps that can minimizethis risk are discussed later in this chapter.
Single-Rule Back-Testing versus Data Mining
Not all back testing is data mining. When just a single rule is proposedand back tested, there is no data mining. This mode of research is illus-trated in Figure 6.2. If the rule’s back test proves to be unsatisfactory, theresearch stops and other more practical means of earning a living areconsidered.
Data mining involves the back testing of many rules and picking onebased on its superior performance. This process is illustrated in Figure6.3. Note that as depicted here an initial unsatisfactory performance byData-Mining Bias: The Fool's Gold of Objective TA 269
 
Rule Idea
y
Program Rule
 
 
 
 
 
 
 
 
 
 
 
 
  
vBack TestyvObserved PerformanceGetJobSatisfactory 7->NO—) Working
at
YES McDonalds
 
 
 
      
Stop ResearchStart Trading
FIGURE 6.2 Single rule back testing.
 
 
Rule Idea
 
_YProgram Rule }«#——____|
v ; >ss Data Mining
Observed Performance
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
StartSatisfactory 7-> No————>} DataMiningYESv
 
Stop ResearchStart Trading
 
 
 
FIGURE 6.3 Rule Data mining.
the first rule tested does not stop the research. The rule is refined or a newrule is defined, and it is tested and its performance is evaluated. The cyclecontinues until a rule with good performance is produced. This processmay involve testing tens, hundreds, thousands, or even greater numbersof rules.270 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Legitimate Uses of Observed Performance
For this discussion, the assumed performance statistic is the mean rateof return over the back-test period. The observed performance statisticcan play a legitimate role both in single-rule back testing and in datamining. However, the roles are different. In single-rule back testing, ob-served performance serves as an estimator of future performance. Indata mining, observed performance serves as a Selection criterion. Prob-lems arise for the data miner when observed performance is asked toplay both roles.
In single-rule back testing, the tested rule’s mean return can legiti-mately be used as an unbiased estimate of the rule’s expected return. Inother words, the tested rule’s most likely future return is its back testedreturn. This is merely a restatement of a something covered in Chapter4. There, we learned that a sample mean provides an unbiased estimateof the mean of the parent population that gave birth to the sample. Thesample mean may err due to the random variation of sampling. It maybe greater than the population mean or less than it, but neither error ismore likely. This principle pertains to the case of a single-rule back test.The rule’s mean return in a back test is an unbiased estimate of its ex-pected return in the future. And although its historic mean return is in-deed its most likely return in the future, its performance may prove tobe higher or lower, with neither being more likely. This is illustrated inFigure 6.4.
In data mining, the back-test performance statistic plays a very dif-ferent role than it does in single-rule back testing. In data mining, back-
            
CumulativeGains$ExpectedPerformance
  
ObservedPerformanceBack-Test
   
+/-Random Variation
Future—»>
 
Time
FIGURE 6.4 Expected performance for single rule back test.Data-Mining Bias: The Fool’s Gold of Objective TA 271
tested performance serves as a selection criterion. That is to say, it isused to identify the best rule. The mean returns of all back-tested rulesare compared and the one with the highest return is selected. This, too,is a perfectly legitimate use of the back test (observed) performancestatistic.
It is legitimate in the sense that the rule with the highest back-testedmean return is in fact the rule that is most likely to perform the best in thefuture. This is certainly not guaranteed, but it is the most reasonable infer-ence that can be made. A formal mathematical proof of this statement isoffered by White.”°
The Data Miner’s Mistake: Misuseof Observed Performance
Let’s recap two key points. In a single-rule back test, past performancecan be used as an unbiased estimate of future performance. In multiple-rule back testing (i.e., data mining), past performance can be used as a se-lection criterion.
The data miner's mistake is using the best rule’s back-tested perfor-mance to estimate its expected performance. This is not a legitimate useof back-tested performance because the back-tested performance of thebest-performing rule is positively biased. That is, the level of performancethat allowed the rule to win the performance competition overstates itstrue predictive power and its expected performance. This is the data-miningbias. This concept is illustrated in Figure 6.5.
CumulativeGains
     
    
 
     
 
 
Observed cePerformance ' ExpectedBest-Rule ; Performance\ of Best-Rule'1I'IIn-Sample ini Future >>Time
 
JRE 6.5 Expected performance of best data-mined rule.272 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
The best rule’s performance does not truly deteriorate out of sample.It only appears that way as the out-of-sample performance assumes alevel that reflects the rule’s true predictive power without the good luckthat allowed it to beat the other rules. Its in-sample performance was acombination of some level of predictive power, possibly zero, and a bigcomponent of good luck. The good luck that shined on the rule duringthe back test is now shining elsewhere. In a like manner, the monkey didnot suffer a loss of literary skill on the night of the big show. His perfor-mance merely reflected his true literary ability minus the good luck thatallowed him to produce letters that happened to match a segment ofShakespearian prose.
DATA MINING AND STATISTICAL INFERENCE
 
This section discusses the connection between data-mining bias and sta-tistical inference. This section will cover the following points: (1) the dis-tinction between biased and unbiased estimators, (2) the distinctionbetween random error and systematic error (i.e., bias), (3) the fact thatunbiased estimators are afflicted with random error but biased estimatorsare afflicted with both random error and systematic error, (4) statisticalstatements hold true for a large number of observations, such as a largenumber of estimates, and (5) the data-mining bias is an effect that showsup generally over many instances of data mining; thus we cannot say thatany particular data-mined result is biased.
Unbiased Error and Systematic Error
All scientific observations are subject to error. Error is defined as the dif-ference between an observed value and the true value:
Error = observed - true
A positive error is said to occur when the observed value is greaterthan true value. Negative error is the reverse. If a scale indicates a personweighs 140 lbs but they actually weight 150, the error is negative 10 Ibs.
There are two distinct types of error: unbiased and biased (system-atic). All observations are infected with some degree of unbiased error.No measuring instrument or technique is perfect. This type of error has anexpected value of zero. This means that if a large number of observationsare made on some phenomenon, such as the return of a rule, and thoseobservations are afflicted only with unbiased error, the observed valuesData-Mining Bias: The Fool's Gold of Objective TA 273
will be haphazardly distributed about the true value. If an average were tobe computed for all these errors the average would be approximatelyzero. See Figure 6.6.
In contrast, observations afflicted with systematic error tend to lie onone side of the truth. Such observations are said to be biased. When theerrors of many biased observations are averaged, the average error will bedistinctly nonzero. See Figure 6.7. In the illustration the observations arepositively biased and their average error (observed — true) is positive.
Suppose a chemist observes the weight of a residue left by a chemicalreaction on one hundred separate instances. Unbiased error might be attrib-utable to random variations in laboratory humidity over the course of theone hundred weighings. The amount of moisture in the residue affects itsweight. Systematic error, if present, could be due to an imperfection in thescale, always causing the observed weight to be lower than the true weight.
Unbiased and Biased Statistics
Interpreting a large sample of observations is difficult. As was discussedin Chapter 4, a sensible first step is data reduction. This reduces the large
 
        
 
 
 
 
edlyTrue pe ake Probability- “Most Likely ee~ Observed ObservedValue -..., .. ValuesjJFIGURE 6.6 > Unbiased observations.7)7 ProbabilityMost DistributionLikely Of_Observed ObservedValue Values
 
 
 
 
 
TrueValue
FIGURE 6.7 Observations with systematic error.274 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
set of measurements to a smaller set of summary statistics: sample mean,sample variance, and other computed measures that describe the entireset of observations.
Chapter 4 also pointed out that a sample statistic, such as the mean, issubject to a particular type of random error called sampling error. This er-ror is unbiased. Here, the term error refers to the deviation between asample mean and the mean of population from which the sample wastaken. Because the sample does not represent the population perfectly,the mean of the sample will deviate to some degree from the mean of thepopulation.
Relating this back to objective TA, a rule back test produces a largesample of observations—the rule’s daily, weekly, or monthly returns.This sample is made more intelligible by reducing it to a performancestatistic (e.g., annualized average return, Sharpe ratio, and such). As istrue of any sample statistic, the performance statistic is subject to ran-dom error. However, the statistic may also be subject to systematic er-ror or bias.
To state the obvious, a historical performance statistic cannot be putin the bank or be used to buy a Ferrari. Its sole economic utility is the in-ference it allows us to make about future performance of the rule thatproduced it. Objective technicians use back-tested performance to makean inference about a rule’s expected performance in the form of a confi-dence interval or a hypothesis test. In either case, the accuracy of the in-ference will depend on the type of error: unbiased or systematic, and itsmagnitude.
Biased statistics are afflicted with systematic error. As stated previ-ously in a single-rule back test, the mean return is an unbiased statistic.Therefore, inferences about the rule’s expected return based on back-testperformance will be subject only to a form of unbiased error called sam-pling variability.
This, however, is not the case for the best rule found via data mining.The observed average return of the best-performing rule is a positivelybiased statistic. As a result, inferences based on it will be systematicallyin error. This means that when conducting a hypothesis test, one wouldbe prone to reject the null hypothesis more frequently than the signifi-cance level would suggest. For example, at a significance level of 0.05one expects to reject the null hypothesis in error only 5 times in 100.However, if the observed performance is positively biased, then H, willbe rejected more frequently than it should, perhaps far more frequently.This will result in trading using rules that appear to have predictivepower but in fact do not. The question is: What causes the back-testedreturn of the best rule, the rule picked by the data miner, to overstate itstrue predictive power?Data-Mining Bias: The Fool's Gold of Objective TA 275
The Mean versus the Maximum
Single-rule back testers and data miners are looking at two entirely differ-ent statistics. The single-rule back tester is observing the mean of a singlesample. The data miner is observing the maximum mean among a multi-tude of sample means. It is easy to overlook the fact that these are two en-tirely different statistics.
To be clear, in the case of a single-rule back test there is one set of re-sults—the rule’s daily returns generated over the back-test period. Theyare summarized by a performance statistic (e.g., mean daily return). Indata mining, many rules are back tested, and so there are many sets of re-sults and many performance statistics. If 50 rules have been back testedthe data miner has the opportunity to observe 50 mean returns prior to se-lecting the rule that produced the maximum mean return.
The set of 50 means returns can be considered as a set of observa-tions, which, in turn, could be summarized with a statistic. For exampleone could compute the mean of the means. That is the mean return of all50 rules. Another statistic that could be computed for this set of observa-tions would be the minimum mean—the mean return of the rule that didthe worst. Yet another statistic would be the maximum mean—the meanreturn of the rule that did the best.
The statistic observed by the data miner is the maximum mean fromamong the 50 rules. This is illustrated in Figure 6.8, where all rules are as-sumed to be useless and have an expected return equal to zero. Each dotrepresents the observed mean return of a different rule. Note that themean return of the rule that had the maximum return (37 percent) is notat all representative of that rule’s expected return (0 percent). It simplygot lucky in the back test. When many rules are back tested, the one with
    
 
Maximum MeanAmongFifty Means
 
 
 
Observed Rate of Return
FIGURE 6.8 Observed performance of 50 rule back tests.276 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
the highest average return will almost always have been the beneficiary ofgood luck. This is why its back-tested performance is likely to overesti-mate its expected performance. In a different sample, with less luck, therule is likely to do worse.
This is the key lesson to take away from this: The observed perfor-mance of the best-performing rule, amongst a large set of rules, is a posi-tively biased estimate of that best rule’s expected performance because itincorporates a substantial degree of good luck. Data miners who don’t un-derstand this are likely to be disappointed with this rule’s out-of-sampleperformance.
Sound Inference Requires the CorrectSampling Distribution
Sound statistical inference depends on using the correct sampling distri-bution. Each test statistic has a sampling distribution that is appropriatefor testing its statistical significance. The sampling distribution thatwould be correct for testing the significance of a single sample mean orconstructing its confidence interval would not be correct if the test statis-tic being observed were, in fact, a maximum mean among a multitude ofsample means.
Therefore, to make sound inferences, the data miner requires thesampling distribution of the maximum mean among a multitude of meansbecause that is the statistic being considered when evaluating the bestrule found by data mining. The central tendency of the sampling distribu-tion of the maximum mean reflects the role that good luck can play in datamining. The central tendency of a sampling distribution of a single samplemean does not.
Now let’s consider how all this impacts a test of significance. As dis-cussed in Chapter 5, in a traditional test of significance, the null hypothe-sis asserts that the trading rule has an expected return equal to zero orless. The test statistic is the rule’s observed mean return. In this case let’sassume the return was +10 percent annualized. The sampling distributionof the test statistic is centered at the hypothesized value of zero. The p-value is the area of the sampling distribution that is equal to or greaterthan a 10 percent return. This area represents the probability that, if therule’s expected return were truly equal to zero, the rule could have pro-duced a return of +10 percent or higher by chance. If the p-value is smallerthan a preset value, such as 0.05, the null hypothesis would be rejectedand the alternative hypothesis, that the rule’s expected return is greaterthan zero, would be accepted. So long as the performance of only one rulewas being evaluated this is all well and good.
Now consider significance testing in the data mining case. ContinuingData-Mining Bias: The Fool's Gold of Objective TA 277
with the data mining example from the prior section, assume that 50rules have been back tested. The rule with best performance earned an-nualized return of +37 percent. A traditional test of significance, with asignificance level of 0.05, would look like the one in Figure 6.9. The sam-pling distribution shown is centered at zero, reflecting the null’s assump-tion that the expected return of a rule with no predictive power wouldgenerate a return of zero. This sampling distribution does not account forthe data-mining bias. Experimental results presented later in this chapterwill show that this assumption is wrong. They will demonstrate that evenwhen all rules tested during data mining have expected returns equal tozero, the best performing rule is likely to display a much higher perfor-mance than zero.
Note in Figure 6.9 that the observed performance of +37 percentfalls far out in the right tail of the sampling distribution giving a p-valueof less than 0.05. On the basis of this evidence, the null hypothesiswould be rejected and the inference would be that the rule has an ex-pected return greater than zero (i.e., has predictive power). This con-clusion is wrong!
If, however, the observed performance of this best rule of 50 wereto be subjected to a more advanced test of statistical significance thatdoes take into account the data-mining bias, the picture would lookconsiderably different. This is shown Figure 6.10, where the observedmean return for best-rule-of-50 is compared to the correct sampling dis-tribution. This is the sampling distribution of the statistic maximum-mean among 50 means. This sampling distribution properly reflectsthe biasing effects of data mining. Note that the sampling distribution isno longer centered at zero. Rather, it is centered at +33 percent. Against
 
Null
HypothesisObservedf Expected = 0% Performance
Best Rule of5037%
 
 
 
 
      
 
Relative | Sampling DistributionOf A Single SampleMean
 
Freq.
 
 
 
p-value < 0.05
-70 -60 -50 -40 -30-20-10 O 10 20 30 40 50 60 70
 
 
 
 
FIGURE 6.9 Traditional sampling distribution (does not account for data-miningbias).278 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
NullHypothesisExpected = 33%
 
 
 
                 
 
ObservedPerformance37%
f
 
 
 
 
 
 
 
RelativeSampling DistributionFreq. Of Multi-Mean p-value > 0.45Maximum
J
-70 -60 -50 -40 -30-20-10 0 10 20 30 40 50 60 70
FIGURE 6.10 Correct sampling distribution (accounts for data-mining bias).
this backdrop, the best rule’s performance no longer appears signifi-cant. The fraction of the sampling distribution that is equal to or greaterthan the observed return of 37 percent is nearly half (0.45) of the distri-bution total area. In other words, if the expected return of each of the50 rules were zero, then there is a 0.45 probability that the mean returnof the best rule would be greater than or equal to 37 percent due toluck. From this, it is clear that 37 percent is not a statistically significantresult. The particular sample of data in which the rule excelled just hap-pened to favor the rule.
Figure 6.10 shows that randomness (good luck) can inflate the per-formance of a rule with no predictive power. It turns out that random-ness is only one of two factors that jointly cause the data-mining bias.The other cause is the selection principle that underlies all multiplecomparison procedures: picking the candidate with the best-observedperformance.
DATA-MINING BIAS: AN EFFECT WITH TWO CAUSES
 
The data-mining bias is the result of a conjoint effect: (1) randomnessand (2) the selection imperative of data mining or any multiple compari-son procedure—picking the candidate with the best-observed perfor-mance. This section will examine how these two factors combine tocause the observed performance of the best rule to overstate its expectedperformance.Data-Mining Bias: The Fool’s Gold of Objective TA 279
Two Components of Observed Performance
The observed performance of a rule can be factored into two components.One component of observed performance is attributable to the rule’s truepredictive power, if it has any. This is the component of performance dueto a recurring feature of market behavior that is exploited by the rule andthat should continue to manifest in the immediate practical future. This isthe rule’s expected performance.
The second component of observed performance is attributable torandomness. Randomness can manifest as either good luck or bad luck.Good luck boosts observed performance above expected performancewhereas bad luck pushes observed performance below expected. Thecomponent of observed performance attributable to randomness cannotbe expected to repeat in the immediate practical future.
This discussion is summed up by the equation in Figure 6.11.
The Spectrum of Randomness
It is useful to think of a spectrum of randomness. This concept is depictedin Figure 6.12.°° At one end of the spectrum, observed performance is
Observed _ | Expected
= - |RandomnessPerformance Performance +/
FIGURE 6.11 The two components of observed performance.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Monkeys.: asMathematical AuthorsProofsPhysics Laws BibleCodesConcert Level
Musician Lottery
Players
 
 
 
 
TA RulesMerit LuckDominates Dominates
FIGURE 6.12 Spectrum of randomness—relative contributions of merit versusluck in observed outcomes.280 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
dominated by randomness. At this extreme, all performances are purely amatter of luck. Here we find literary works of monkeys dancing on wordprocessors and outcomes of lottery players. At the other end of the spec-trum, observed performance is dominated by merit or systematic behav-ior. At this extreme, it is only merit that matters. Producing a valid proofof a mathematical theorem resides here. Very nearby are the observedperformances of concert-level musicians. Also, in this neighborhood wefind the the laws of physics whose highly accurate predictions are madepossible by the orderly behavior of certain aspects of nature.
Toward the middle of the randomness spectrum lie the most interest-ing data mining problems. The farther to the random end of the spectrumone goes, the greater the risk of data-mining bias. Toward the right end ofthe continuum, we find a region that contains the TA rules. The complex-ity and randomness of financial markets assure us that even the most po-tent rules will be weakly predictive. In this zone, the magnitude of thedata-mining bias will be large.
Now we come to an important principle. The larger the contributionof randomness (luck) relative to merit in observed performance, thelarger will be the magnitude of the data-mining bias. The reason is this:The greater the role of luck relative to merit, the greater the chance thatone of the many candidate rules will experience an extraordinarily luckyperformance. This is the candidate that will be selected by the data miner.However, in situations where observed performance is strictly or primar-ily due to a candidate’s true merit, the data-mining bias will be nonexistentor very small. In these cases, a candidate’s past performance will be a reli-able predictor of future performance and the data miner will rarely fill thehopper with fool's gold.
Because financial markets are extremely difficult to predict, most of arule’s observed performance will be due to randomness rather than itspredictive power. Thus the equation previously given in Figure 6.11 mightbe better illustrated as it is in Figure 6.13. The large role of randomnessrelative to merit (i-e., predictive power) makes it likely that TA rule datamining will be infected with a large bias.
The range of problems to which multiple comparison proceduresmight be applied can be visualized as lying along the spectrum of random-ness. Toward the random end of the spectrum is a range associated with
Observed ;Performance "tower ) +/-| Randomness
FIGURE 6.13 Relative contributions of randomness and predictive power in ob-served performance.Data-Mining Bias: The Fool's Gold of Objective TA 281
TA rule data mining. I used a range to represent rule data mining ratherthan a single location to indicate that the level of randomness will varyfrom one mining venture to the next. As will be demonstrated later in thischapter, the level of randomness in any given data mining venture is deter-mined by five independent factors.”’ When these factors are taken intoconsideration, it becomes possible to develop statistical significance pro-cedures that are better able to deal with the data-mining bias. In otherwords, it becomes possible to compute the statistical significance of orconfidence interval for a rule that was discovered by data mining. Theseprocedures alleviate the key problem facing objective TA practitioners us-ing data mining for knowledge discovery.
The Effectiveness of Multiple ComparisonProcedures under Differing Conditionsof Randomness
Multiple comparison procedures are applied to problems where a best so-lution is sought. A universe of candidate solutions is proposed. A figure ofmerit quantifies the observed performance of each candidate and the onewith the best-observed performance is selected.
