|
A 20%
Ss
10%
 
 
 
 
 
 
1 2 10 50 400Number of Rules Tested (log scale)
FIGURE 6.32  Data-mining bias versus number of rules tested (annualized per-cent return): 24-month performance history.
best-of 2, 10, 50, and 400 ATRs. The vertical axis represents the magnitudeof the data-mining bias—the differential between the observed returns forthe best performing ATR and its expected return. The horizontal axis rep-resents the number of ATRs back tested to find the best. The relationshipis summarized by fitting a curve through the four data points. It is nearlylinear when number of rules tested is plotted in log terms—log (base10).The main point to take away from these experiments is this: The larger thenumber of rules tested to find a best rule, the greater the data-mining bias.298 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
It should be pointed out that the particular magnitudes of data-miningbias shown in the preceding tests are valid only for the particulars of thistest: a particular set of S&P 500 monthly returns, a 24-month perfor-mance history, all rules have independent returns, and all rules having anexpected return equal to zero. In a different data-mining venture, withdifferent particulars, the same principle would apply (more rules pro-duce a bigger bias) but the specific levels of the data-mining bias wouldbe different.
For example, had a longer performance history been used to computemean return—48 months instead of 24—the distribution of mean ATR re-turns would have clustered more tightly around the expected return ofzero. This is merely a manifestation of the Law of Large Numbers. Conse-quently, the bias associated with the highest performing ATR would havebeen less. This tells us that the number of observations used to computethe performance statistic is an important factor in determining the magni-tude of the data-mining bias.
Let’s consider what was said in the previous sentence. The greater thenumber of monthly observations used to compute the performance statis-tic, the smaller the dispersion of the statistic’s sampling distribution. Inother words, the larger the sample size used to compute the performancestatistic, the less the degree of randomness in observed performance andthe less the opportunity for an extraordinarily lucky performance. What-ever reduces the degree of randomness in observed performance reducesthe data-mining bias.
The importance of sample size on data-mining bias can be seen inFigure 6.33. It is similar to the preceding Figure 6.32. It is a plot of data-mining bias, represented on the vertical axis, as a function of number ofATR’s compared to find the best one, on the horizontal axis. However, inthis plot there are four curves instead of one. Each curve is based on us-ing a different number of monthly observations to compute each rule’smean return; 10, 24, 100, and 1,000 months. The dotted line for 24months is the same curve seen in Figure 6.32. Two points are worthy ofnote. First, all curves rise as the number of rules back tested is in-creased. This is consistent with the finding discussed in the precedingsection—the data-mining bias increases as the number of ATR’s tested isincreased. Second, and perhaps most important, is the fact that the mag-nitude of the bias is reduced by increasing the number of months used tocompute the mean return. For example, when only 10 months of dataare used, the bias for the best-of-1,024 ATRs is approximately 84 per-cent. When 1,000 months of data are used, the bias for the best of 1,024rules shrinks to less than 12 percent. In the next section, we will see whythe number of observations is so important in determining the size of thedata-mining bias.Data-Mining Bias: The Fool’s Gold of Objective TA 299
96 -—
               
 
 
M 10 mos.M 84|—721 Or 24 mos.48A5 36;-al Le 100 mos.% —_-_- 70 .ye fer TT eee 1000 mosT T T |' 2 16 64 256 1024
Number of ATR s (log scale)
FIGURE 6.33 Data-mining bias versus number of ATRs tested for different sam-ple sizes.
Factor 2: Number of Observations Used to Compute the Per-formance Statistic. Figure 6.33 tells us that increasing the numberof observations used to compute the performance statistic reduces themagnitude of the data-mining bias. In fact, of all the factors impacting thesize of the bias, sample size may be the most important. The more obser-vations used, the less opportunity there is for a few lucky observations(profitable months) to result in a high mean return. We saw this in Chap-ter 4, where it was demonstrated that the sampling distribution is re-duced in width when a larger number of observations are used tocompute the sample statistic. With only a few observations, a samplemean can stray considerably above or below the true population mean.The message conveyed by a wide sampling distribution is that there is agreater opportunity for a rule to generate a very profitable return in aback test by luck rather than by predictive power. This is illustrated inFigure 6.34, which shows two sampling distributions for the same rule,for which expected return is zero. Note the sampling distribution basedon fewer observations is wider. A mean return computed from a shortperformance history is more likely to produce a very lucky result than amean computed from a larger number of observations (i.e., a longer per-formance history).
The next set of experiments examine the effect of the number ofmonthly observations used to compute mean ATR return on the size ofthe data-mining bias. The number of months was varied from one monthto 1,000 months, in 50-month increments, with data-mining bias com-puted at each increment. This was done for two cases: the best-of-10 and300 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Sample Mean Computed FromLarge Number of ObservationsLucky RuleAppearsModestly Profitable
   
+ ele10% OQ +10%
Sample Mean Computed from
Small Number of Observations Lucky RuleAppears
Highly Profitable
          
     
+ + + + + +-30% 10% OQ +10% +30%
FIGURE 6.34 Narrow versus wide sampling distributions.
the best-of-100 ATRs. As in prior experiments all ATRs were set to haveexpected returns equal to zero.
Figure 6.35 shows the relationship between the data-mining bias onthe vertical axis versus the number of observations used to computeATR mean returns on the horizontal axis. Because the expected returnfor all ATRs is equal to zero, the data-mining bias is equal to the averageobserved performance of the best. Thus, the vertical axis, which is la-beled data-mining bias, could as easily have been labeled average per-formance of the best rule. Note the steep decline in the magnitude of
84 +
   
 
 
Observed PerformanceBest-of-100 ATRs
 
 
 
 
 
 
1 200 400 600 800 1000Number of Months Used To Compute Mean ATR Return
FIGURE 6.35  Data-mining bias versus number observations.Data-Mining Bias: The Fool’s Gold of Objective TA 301
the bias as the number of observations used to compute the mean returnis increased. This is the Law of Large Numbers working in favor of thedata miner.
The lesson here is this: Be very skeptical of performance statisticscomputed from a small number of signals or time intervals. Another im-portant message from Figure 6.35 is that when the number of observa-tions (months) becomes quite large, 600 or so, the difference in the biasbetween best-of-10 and the best-of-100 ATRs becomes tiny. This findinghas very important implications for the data miner—when the number ofobservations is sufficient, one can data mine a much larger number ofrules without significantly increasing the data-mining bias. The Law ofLarge Numbers rules!
Factor 3: Degree of Rule Correlation. The third factor affectingthe size of the data-mining bias is the degree of similarity among the rulestested. Rules are said to be similar when they generate performance histo-ries that are strongly correlated. That is to say, their monthly or daily re-turns are correlated. The stronger the correlation between the rulestested, the smaller will be the magnitude of the bias. Conversely, the lowerthe correlation (i.e., the greater the degree of statistical independence) be-tween rules returns, the larger will be the data-mining bias.
This makes sense because increased correlation among the rules hasthe consequence of shrinking the effective number of rules being backtested. Imagine a large set of rules that are completely identical. Naturally,they will generate perfectly correlated performance histories. In effect,this large set of rules is really just one rule, and we already know that thedata-mining bias shrinks to zero when only one rule is back tested. Back-ing off from this extreme case, when rules are highly similar, and thushave highly correlated returns, the chance of an extraordinarily lucky per-formance is reduced. The more dissimilar the rules, the greater will be theopportunity for one to have a great coincidental fit to the historical dataand achieve high performance. So high correlation amongst the rulesshrinks the effective number of rules back tested and hence shrinks thedata-mining bias.
In practical terms, rule correlation is most likely to be high when datamining involves optimizing the parameters of a specific rule form. Sup-pose a dual moving average crossover rule is being optimized. This meansthat every rule tested is the same except for the parameter values—thenumber of days used to compute the short-term and long-term moving av-erages. The rule using values of 26 days and 55 days will produce returnsthat are highly correlated with a rule using values of 27 and 55.
Figure 6.36 shows the relationship between the magnitude of the data-mining bias for the best rule (vertical axis) and the degree of correlationBOZ METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
 
0.2 #04 O06 O08 1.0Rule Correlation
FIGURE 6.36 Data-mining bias versus rule correlation (annualized percent re-turn number of monthly observations = 100).
(horizontal axis) among the rules. Rule correlation was simulated as fol-lows: An initial ATR performance history was generated. In the generationof asecond ATR history, a biased random device was consulted to deter-mine the monthly returns of ATR history 2. The bias was set to the desiredlevel of correlation. For example, if a .7 correlation was desired, a coin flipwas simulated with the probability of heads = 0.70. If the coin landed onheads, then the monthly return for the second ATR would be the same asthe monthly return of the initial ATR. This was continued for subsequentATR histories.
Each ATR was simulated over a 100-month history. Three differenttests were run, each based on a different number of ATRs: 10, 100, 1,000.In other words, one test measures the data-mining bias associated withthe best ATR rule out of 100. As in prior tests, all rules had expected re-turns equal to zero. The vertical axis of the plot represents the data-mining bias. The factor being examined here, rule correlation, was variedfrom 0 to 1.0. Note that the bias remains high until rule correlations ap-proach a level of 1.0. Thus, rule correlation does not have a major impactin reducing the data-mining bias until there is a high level of correlation intheir returns. Also note that the data-mining bias is higher for the best of1,000 rules than it is for best of 10. This is simply a manifestation of factor1, the greater the number of rules from which the best is selected, thegreater the data-mining bias.
Factor 4: Presence of Positive Outliers in Rule Returns. Asample of rule returns (daily, weekly, or monthly) that contains a few ex-tremely large positive observations has the potential to create a large dataData-Mining Bias: The Fool's Gold of Objective TA 303
mining bias. The size of the bias will depend on how extreme the valuesare and the number of observations used to compute the mean. If a smallnumber of observations are used, one extremely positive value can boostthe sample mean dramatically. A large number of observations will lessenthe impact of an occasional extreme value.
Distributions that contain extreme values are said to have heavy tails.The tails of a distribution are the outer right and left zones where the ex-treme but rare observations lie. Heavy-tailed distributions extend fartheraway from the distribution’s center than light-tailed distributions. The nor-mal distribution has light tails. Figure 6.37 illustrates this distinction. Thedistribution in the upper portion of Figure 6.37 has the normal bell shape.Note how quickly the tails disappear as one moves away from the distrib-ution’s center. This tells us that extreme observations are rare to the pointof being virtually nonexistent. People’s heights are normally distributed. Itis rare to see someone over seven feet tall. No one has ever seen a personover 10 feet. The distribution in the lower portion of the figure is heavytailed. Although extreme observations are also rare, they occur more fre-quently than in light-tailed distributions. Were people’s heights distributedthis way, you might meet someone 20 feet tall!
As mentioned earlier, when the distribution of daily (weekly, monthly)returns contains extreme observations, a sample mean, computed fromthese observations, can also take on an extreme value. This occurs whenone or more of the extreme values wind up in the sample and the samplesize is small. From this it follows that heavy tails in the return distributiontend to cause tail heaviness of the sampling distribution of the mean.
 
 
 
 
 
 
 
 
 
 
 
0; Outliers:Heavy Tails Extreme PositiveMonths, Days, Etc.c Y0
FIGURE 6.37 Light- versus heavy-tailed return distributions.304 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Nevertheless, the value of a sample mean is never as extreme as the mostextreme value found in a sample. This is simply the effect of averaging.Therefore, the sampling distribution of the mean return will always havelighter tails than the distribution of interval returns. This is illustrated inFigure 6.38. Two return distributions are depicted. On the left is a returndistribution with heavy tails. The sampling distribution of the mean basedon this distribution also has heavy tails, although they are lighter than thereturn distribution itself. On the right is a thin-tailed return distribution. Asampling distribution of the mean based on it necessarily has light tails.
Note that both of the sampling distributions in the Figure 6.38 arebased on the same sample size. Sample size is a separate factor influenc-ing tail heaviness of the sampling distribution. I wanted to hold samplesize constant so as not to confuse the issue.
Now let’s consider the effect of sample size on the sampling distribu-tion’s tail heaviness. When more observations are used to compute thesample mean, the effect of tail heaviness in the return distribution is mini-mized. The greater the number of observations used to compute the sam-ple mean, the lighter will be the tails of the sampling distribution. This isillustrated in Figure 6.39. The return distribution clearly has heavy tails.On the left is the sampling distribution of the mean based on a small sam-ple size. Its tails, though not as heavy as the return distribution itself, arestill heavy. On the right is a sampling distribution of the mean derivedfrom the same return distribution, except that a larger sample size is used
Return Distribution: Heavy Tails Return Distribution: Light Tails
 
 
0Resulting ResultingSampling Distribution of Sampling Distribution ofthe Mean the Mean0 )
FIGURE 6.38 Impact of extremes in return distribution on sampling distribu-tion: equal sample size.Data-Mining Bias: The Fool’s Gold of Objective TA 305
Return Distribution: Heavy Tails
 
 
 
0Sampling Distribution Sampling DistributionSmall Sample Size Large Sample Size(Tails Still Heavy) (Lighter Tails)0 0
FIGURE 6.39 Impact of sample size on tail heaviness of the samplingdistribution.
to compute the sample mean. The tails of this sampling distribution arelightened as a result.
What is the relevance of all this to the data-mining bias? A heavy-tailed sampling distribution is saying that randomness is playing a largerole in the mean return. Randomness is always bad news for the dataminer.*! Imagine two rules. Both have expected returns of zero. One has aheavy-tailed sampling distribution. The other has a sampling distributionwith light tails. The heavy tails tell us that there is a greater probabilitythat the rule will produce a mean return, in a back test, that is muchhigher than its expected return. If this rule is picked by the data miner, itsdata-mining bias will be large and its out-of-sample performance is likelyto be disappointing. This is illustrated in Figure 6.40. The heavy-tailedsampling distribution shows that it is within the realm of probability forthe rule to earn mean return of greater than 30 percent in a back test. Inthe case of the sampling distribution with light tails, a return that highwould have a probability of almost zero.
It is widely recognized that the distribution of stock market returnsand the returns of other financial asset have relatively heavy tails com-pared to a normal distribution.’ A rule that was short, by coincidence,when the market crashed may win the data-mining-performance competi-tion, but it will have won for the wrong reason.
The issue of tail heaviness and its resultant impact on the data-mining306 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
Expected Return
 
 
 
   
   
 
Cy Z-10% O +10%
 
 
 
 
 
 
 
 
ExtraordinarilyExpected Return LuckyBack-Tests+ F F“30% -10% O +10%  — +30%
FIGURE 6.40 Light- versus heavy-tailed sampling distributions.
bias was examined experimentally. The data-mining bias was comparedfor two sets of ATRs. One set used the S&P 500 monthly returns, a distrib-ution with tails that are heavier than those of the normal distribution, togenerate performance histories. The other set of ATRs used a return dis-tribution with intentionally light tails. As in the other experiments, theATR with the best-observed performance was selected and its data-miningbias was measured. The results of these experiments were clear. The data-mining bias was much less pronounced for the ATRs that were based onthe light-tailed return distribution. This finding is of practical value onlywhen data-mining rules exit at fixed targets. This has the effect of elimi-nating extreme outcomes. This in turn has the effect of reducing tail heav-iness in the sampling distribution of the performance statistic. In mostinstances, however, rule researchers are stuck with the return distribu-tions produced by financial market behavior. Because markets are subjectto extreme events, mean rule returns will tend to be extreme as well.
Factor 5: Variation in Expected Returns among the ATRs.This factor refers to the degree of variation in the expected returns amongthe set of rules considered during data mining. The ATR experiments dis-cussed thus far have considered ATRs with uniform predictive power. Allhave had expected returns set at zero. When all rules are of equal merit,be that merit none, low, or high, any differences in their observed perfor-mances are due entirely to luck. In such a case, the winner of the data-mining performance competition will simply be the luckiest rule. Thus,the difference between its observed mean return and its expected returnwill be large. Therefore, when all rules possess equal expected returns,the data-mining bias will be large.Data-Mining Bias: The Fool's Gold of Objective TA 307
However, if the universe of rules explored during data mining differswith respect to their expected returns, the data-mining bias will tend to besmaller. Under certain conditions, it will be very small. The following ex-periments investigate how variation in expected returns with the rule uni-verse impacts the data-mining bias. This is accomplished by creating auniverse of ATRs that have differing levels of expected return. Recall, thatan ATR’s expected return is entirely determined by its designated proba-bility of a profitable month.
Let’s first consider what the following experiments should show. Sup-pose a universe of ATRs contains one that is truly superior to all the oth-ers. It has an expected return of +20 percent, whereas the others all haveexpected returns of zero. We would expect its true superiority to reveal it-self by producing the highest-observed performance most of the time (onany given test it is not guaranteed to do so). Consequently, most of thetime the superior ATR will get picked and the data-mining bias associatedwith it will tend to be small. The bias will tend to be small because its ob-served performance was earned the old-fashioned way, by dint of predic-tive power rather than luck.
Experiment 2: Data Mining ATRs with DifferingExpected Returns
The following tests investigate the data-mining bias within a universe pop-ulated by ATRs whose expected returns differ. In these tests, the majorityof ATRs have expected returns equal to zero or close to zero. However, afew ATRs with returns significantly greater than zero are sprinkled in.This is intended to replicate the world a rule data miner can reasonablyhope to encounter—there really is some TA gold in the mountains of dataalthough it is rare.
In addition to investigating the magnitude of the data-mining bias, thisset of tests also addresses another question: Does data mining rest on asound premise? That is, does picking the rule with the best-observed per-formance do a better job of identifying authentic predictive power thanrandomly selecting a rule from the universe of rules. Data mining must, atthe very least, pass this minimal test of efficacy to establish that it is areasonable research approach. Figure 6.41 shows the expected return foran ATR given its specified probability of a profitable month. Bear in mindthat this graph is specific to the monthly SP returns used in these simula-tion experiments.
Recall that an ATR’s expected return is determined entirely by itsprobability-of-profitable-month (PPM). The following tests assume thatrules with high expected returns are rare. To simulate this assumption, thePPM assigned to each ATR was drawn at random from a distribution of3OB_ METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
60%50%40%30%20%10%0%+ T T T0.50 0.60 0.70 0.80 0.90 1.0
Probability of Profitable Month
 
 
 
 
 
FIGURE 6.41 Expected return (f) probability of profitable month.
PPMs. The distribution used for this purpose has a shape similar to thedistribution of wealth in a society. It has a long tail out to the right, indi-cating that wealthy individuals, like Bill Gates, are extremely rate while agreat many are poor. The distribution of expected returns used for theseexperiments was defined such that an ATR with an expected return of 19percent per year or greater had a frequency of occurrence of about | in10,000. The expected return for the average ATR in the universe was +1.4percent per year. The approximate shape of the distribution used is illus-trated in Figure 6.42.
Although the specific assumptions underlying this distribution maynot be accurate, they are probably correct to within one order of magni-tude. This is based on the performance of the best hedge funds using
Frequency
O 1.4% 19%Expected ATR Return
FIGURE 6.42 Distribution of ATR expected returns.Data-Mining Bias: The Fool's Gold of Objective TA 309
technical analysis. However, for purposes of these experiments, thespecifics are not crucial. The real purpose here is to measure the data-mining bias in a universe of rules whose expected returns differ and to in-vestigate how the bias can thwart the objective of data mining, findingmerit in a universe where it is rare.
 
Is Data Mining Based on a Sound PremiA Qualified Yes!
This section addresses the important fundamental question: Is data min-ing based on a sound premise? This is really two questions. First, is ob-served mean return a useful indicator of rule merit in the sense that therule with the highest observed mean return has a higher expected returnthan a rule picked at random? If a randomly selected rule is as likely toperform well in the future as a rule that performed well in a back test,then data mining is a pointless activity. Second, is more data mining bet-ter? That is, does testing a larger number of rules lead to the discovery ofa rule with higher expected returns? If so, then back testing 250 ruleswould be a better idea than back testing only ten.
Fortunately for the field of data mining, the answer to both questionsis a qualified yes. The qualification relates to the number of observationsthat are used to compute the rule’s mean return or other performance sta-tistics. If a sufficient number of observations is used, then the answer toboth questions is yes. This is very good news for the data miner. However,when the number of observations is small, data mining does not work.One would do just as well picking a rule by guess, and data mining a largernumber of rules will not produce better rules.
Because the number of observations used is crucial, no doubtthe reader would like to know how many are sufficient. Unfortunately,there is no simple answer. The requisite number depends on the perfor-mance statistic and the nature of the raw data. The best I can do is toprovide representative charts that give a general sense of requisite sam-ple sizes.
In prior plots, the vertical axis represented the data-mining bias in an-nualized percent return per year. Take note, however, that in Figure 6.43,the vertical axis represents the true merit or expected return of the rulewith the highest-observed performance—the rule that would be selectedby the data miner. Clearly, such a plot could not be produced for actual TArules because their expected returns are never known. The horizontalaxis represents the number of ATRs tested to find the one with the best-observed performance. The set of numbers considered was: 1 (no datamining), 2, 4, 8, 16, 32, 64, 128, and 256. There are three curves on the plot.Each curve is based on a different number of monthly observations used310 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
              
ExpectedReturn% / Yr.
Universe:AverageExpectedReturn1.4%/yr
 
  
 
tt tt1 32 64 128 256Number of ATRs Tested To Find The Best
FIGURE 6.43 Expected return of best performing ATR versus number of ATRsback tested.
to compute the ATR’s mean annual return. The number of monthly obser-vations used for these three cases were: 2, 100, and 1,000.
The second question posed was the following: Does testing morerules lead to the discovery of a rule with a higher expected return? If so,the curve of expected return should rise as the number of ATRs exam-ined to find the best one goes up. In other words, the expected return ofthe ATR with the best-observed performance should be positively cor-related with the number tested to find the best. On the other hand, if thecurve does not rise, it would be telling us that more searching does notlead to rules of higher merit. This would mean that data mining is afruitless activity.
In Figure 6.43, two of the three curves rise—the one based on 100monthly observations and the one based on 1,000. This tells us that backtesting more rules leads to the discovery of higher merit rules. The curvebased on 1,000 rises more steeply than the one based on 100. This istelling us that when performance statistics are computed from a largernumber of observations, data mining more reliably leads to better rules.This is good news for data miners who pay attention to the issue of sam-ple size.
The curve in Figure 6.43 that does not rise also tells a story: When toofew observations are used to compute the performance statistic, data min-ing does not work. That is to say, testing more rules does not lead to thediscovery of better rules. The flat curve shows that when only two monthswere used to compute the mean return of each ATR, the expected returnof the best performing ATR out of 256 ATRs was no higher than the bestData-Mining Bias: The Fool's Gold of Objective TA 3it
performing ATR when only one was tested (i.e., no data mining). The im-portance of sample size cannot be over emphasized!
Now let’s go back and address the first question: Does data mining se-lect rules with higher expected returns than simply picking one at randomfrom the universe? Figure 6.43 speaks to this question as well. Note thecurve based on two observations remains stuck at +1.4 percent as the num-ber of ATRs tested is increased. A return of +1.4 percent is the expected re-turn of the average ATR in the ATR universe. In other words, if one were tograb an ATR at random from the universe and note its expected return andrepeat this many times, on average the expected return of the randomly se-lected rule would be +1.4 percent. Figure 6.41 tells us that when only twomonths of observations are used, picking the ATR with the best-observedperformance is no better than picking an ATR at random. The fact that thecurve is flat says that it does not matter how many ATRs are tested. How-ever, the curves based on 100 and 1,000 observations rise well above theuniverse average. For example, when a best is picked from 256 ATRs using1,000 months of history to compute the observed mean return, its expectedreturn is +10 percent a year, far above the universe average of +1.4 percent.The bottom line: When there are a sufficient number of observations arule’s observed rate of return is a useful, though positively biased, indi-cator of its expected rate of return. However, when there are too few ob-servations, the observed rate of return is virtually useless.
Data-Mining Bias as a Function of Universe Size:In a Universe of Variable Merit
The previous section established that the payoff from data mining in-creases as the number of rules tested is increased, given sufficient samplesize. This shows that observed performance of the best rule is a useful,though positively biased, gauge of its expected return. The question is:How large is this bias associated with the best rule (highest observed re-turn) when the universe is composed of rules that differ in terms of theirexpected returns?
The figures that follow (Figures 6.44, 6.45, 6.46, 6.47) are plots of thedata-mining bias (vertical axis) versus the number of rules back tested ina rule universe where merit is distributed according to the distributiondisplayed in Figure 6.42. Each plot is based on a different number ofmonthly observations for computing mean return: 2, 100, and 1,000months, respectively. First, each curve is shown individually (Figures6.44, 6.45, and 6.46). Figure 6.47 shows the three curves placed on top ofone another. It gives a sense of the relative magnitude of the data-miningbias at different