sample sizes.3#2Z_ METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
       
 
 
240-DataMining 180BiasOfBest 120-ATRAnnual Length ATR Performance% Return 60 History: 2 Months
 
 
1 32 64 128 256Number of ATRs Tested To Find Best
FIGURE 6.44 Data-mining bias versus number of ATRs tested in variable meritrule universe using a two-month performance history.
9.6+
 
DataMiningBias
Of LfBest + Length ATR PerformanceATR 2.44+ History: 100 Months
Annual% Return
 
 
ttt tt1 32 64 128 256Number of ATRs Tested To Find The Best
FIGURE 6.45  Data-mining bias versus number of ATRs tested in variable meritrule universe using a 100 month performance history.
Figure 6.44 is based on using only two months of observations to com-pute each ATR’s observed return. The figure clearly shows the problemcreated by using a small sample size. For example, the observed perfor-mance of the best performing ATR out of 256 ATRs overstates its expectedreturn by over 200 percent per year. For short performance histories, thedata-mining bias is extreme. However, for the 100-observation case,shown in Figure 6.43, and for the 1,000 observation case shown in Figure6.44, the bias is dramatically smaller. For example, the bias associatedwith the best ATR of 256 is approximately 18 percent when 100 monthsare used and less than 3 percent when 1,000 months are used.Data-Mining Bias: The Fool’s Gold of Objective TA
 
 
3.0 -—DataMiningBias :Of 1.8BestATR 1.25 Length ATR PerformanceAnnual History: 1000 Months% Retun gg L
 
Ip} -j—-—_ ; — —_ —__-1 32 64 128 256Number of ATRs Tested To Find The Best
FIGURE 6.46 Data-mining bias versus number of ATRs tested in variable meritrule universe using a 1,000-month performance history.
       
 
240 -—210DataMining 180Bias 150OfBest 120ATR 90Annual eocei months=10030
 
 
 
    
 
  
1 32 64 128 256Number of ATRs Tested To Find The Best
FIGURE 6.47 Data-mining bias versus number of ATRs tested in variable meritrule universe.
There are two messages expressed by Figures 6.44, 6.45, and 6.46.First, the number of observations has a major impact on data-mining bias.Second, when the number of observations used to compute rule perfor-mance is large, the bias levels out quickly. This is seen Figure 6.46 basedon 1,000-month performance histories. The data-mining bias of the best of16 ATRs is not much worse than the best of 256. Figure 6.47 superimposesthe prior three figures. It shows that searching a larger number of ATRs tofind a best (i.e., more data mining) does not increase the magnitude of thebias once the number of ATRs tested exceeds a modest number. In otherwords, the penalty for testing more rules does not increase, once a rather34 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
low threshold number of rules has been exceeded, provided, of course,that a sufficient number of observations has been used to compute theperformance statistic.
How quickly the data-mining bias stabilizes once this threshold ispassed can be seen more clearly in Figure 6.48, which is a portion of thecurve for the 100-month observation case magnified. It shows the regionalong the horizontal axis where the number of ATRs tested is between 1and 40. After approximately 30 ATRs have been examined, increasing thenumber tested has a minimal effect on the size of the data-mining bias.This is good news for the data miner who utilizes an adequate number ofobservations. The bottom line: testing a larger number of rules is a goodthing, and the penalty for doing so drops off quickly if a sufficient numberof observations are used to compute the performance statistic that is usedfor rule selection.
Data-Mining Bias as a Function of Number ofObservations: In a Universe of Variable Merit
This section presents a series of plots showing the relationship betweenthe magnitude of the data-mining bias (vertical axis) and the number ofmonthly observations used to compute the observed performance statis-tic (horizontal axis). This effect was visible on prior plots by showing sep-arate curves of the bias versus number of ATRs for distinct values of 2,100, and 1,000 months. However, the plots presented in this section showthe relationship between the bias and number of observations along acontinuum of the number of months.
16.84DataMining 47Bias 12.07OtBestATRAnnual% Return
    
 
Length ATR PerformanceHistory: 100 Months
 
1 10 20 30 40Number of ATRs Tested To Find The Best
FIGURE 6.48 Data-mining bias stabilizes quickly when sample size is sufficient.Data-Mining Bias: The Fool’s Gold of Objective TA 315
Each plot shows the data-mining bias as a function of number of ob-servations, with that number ranging from 1 month to 1,024 months. Whatwe would expect to see, on the basis of previous findings, is for the bias tobe large when the number of observations is small. This is indeed what oc-curs. The message to the data miner is clear: Heed the Law of Large Num-bers. Figures 6.49, 6.50, and 6.51 show the data-mining bias versus thenumber of observations used to compute mean ATR return for three dif-ferent cases: best ATR of 2, best of 10, and best of 100, respectively. Figure6.52 shows the three curves superimposed.
DataMiningBiasOfBestATRAnnual% Return
 
] 200 400 600 800 1000Number of Observations to Compute Mean Return
FIGURE 6.49 Data-mining bias versus number of observations for best ATR of 2.
60
% Return
 
] 200 400 600 800 1000Number of Observations to Compute Mean Return
FIGURE 6.50 Data-mining bias versus number of observations for best ATR of 10.3G METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
DataMiningBiasOfBestATRAnnual% Return
 
1 200 400 600 800 1000
Number of Observations to Compute Mean Return
FIGURE 6.51 Data-mining bias versus number of observations for best ATRof 100.
 
 
 
 
84DataMining = 72Bias 60Of ;Best 48ATR 36Annual% Return 24
 
1 200 400 600 800 1000
 
Number of Observations to Compute Mean Return
FIGURE 6.52 Data-mining bias versus number of observations for best ATR of 2,10, and 100.
Observed Performance and Expected Performanceversus Number of Observations in a Universeof Variable Merit
Figures 6.53 and 6.54 display two curves. The upper curve is the observedperformance of the best-performing ATR amongst, say, 10 ATRs. Thelower curve shows the expected return for that best-performing ATR. Thevertical axis represents annualized return, both observed and expected.The horizontal axis represents the number of monthly observations. So,for example, in Figure 6.53, point A on the upper curve represents the ob-served performance of the best ATR out of 10 when 400 observations wereData-Mining Bias: The Fool's Gold of Objective TA 317
 
 
 
 
  
36 7Observed% eo PerformanceReturn 24 Best ATR of 10er: ExpectedYear 185 | Performance ;12 4 Best ATR of 10 :6
 
 
{____,_____,___,_____,_____-] 200 400 600 800 1000Number of Observations to Compute Mean Return
FIGURE 6.53 Observed and expected performance best ATR versus number ofobservations based on 10 ATRs.
 
Observed% 60 PerformanceBest ATR of 500
 
 
 
   
 
Return 4gPer SoperExpectedYear 36 i Performance :24 : Best ATR of 500:12
 
1 200 400 600 800 1000Number of Observations to Compute Mean Return
FIGURE 6.54 Observed and expected performance best ATR versus number ofobservations.
used to compute the ATR’s observed mean return. Point B represents thatATR’s expected return. The vertical distance between them is the data-mining bias, given these specific conditions: best of 10 ATRs and 400monthly observations.
Here is what we should anticipate, given the results of prior experi-ments. When the number of observations is small, randomness will domi-nate merit, and observed performance will be much greater than expectedperformance—a large data-mining bias. This will show up on the plot asthe two curves being far apart. The large vertical gap between the curvesoccurs when the performance statistic is based on few observations be-cause this permits one ATR to get very lucky. However, as more observa-3B METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
tions are used to compute mean return, the curves should converge as ob-served performance more truly represents expected performance. How-ever, the curve of observed performance will always lie above the curve ofexpected performance because a best-of-selection criterion will always in-duce some positive bias when luck plays a role in observed performance.
These tests were run for two cases. Figure 6.53 shows the curves forthe best ATR of 10. Figure 6.54 shows them for the best ATR of 500. As ex-pected, the bias shrinks as the number of observations is increased (thecurves converge). Also, as expected, the bias is greater for the 500 ATRcase than for the 10 ATR case due to the greater opportunity for luckwhen more ATRs are examined to find a best.
Data-Mining Bias as a Function of RuleCorrelation: In a Universe of Variable Merit Basedon 500 ATRs
We have previously seen that when the returns of the candidate rules arecorrelated, it reduces the data-mining bias. Correlation shrinks the biasbecause correlation reduces the effective number of rules being exam-ined. In the extreme case, where all rules have perfectly correlated re-turns, there is really only one rule being tested. In this situation, no datamining takes place, and so the data-mining bias shrivels to zero. These re-sults were for a universe of rules of equal merit; all had expected returnsof zero. Now results for a variable-merit universe are presented.
Figure 6.55 plots the expected return (vertical axis) of the uni-verse’s best-performing ATR as the number of ATRs examined (horizon-tal axis) is varied from 1 to 256. There are four curves, each
ExpectedReturn% / Year
 
 
 
 
 
] 4 16 6 4 256Number of ATRs Tested To Find Best (log scale)
FIGURE 6.55 Expected return of best ATR versus number of ATRs. Tested atfour levels of correlation: 0, 0.3. 0.6. 0.9. Number of observations= 100.Data-Mining Bias: The Fool's Gold of Objective TA 319
representing a different level of correlation in returns amongst theATRs from which the best is selected. The correlation levels are: 0.0,0.3, 0.6, and 0.9. The number of monthly observations used to computeATR mean return is set at 100 months.
The first thing to note about Figure 6.55 is that the higher the level ofcorrelation, the less the expected return of the best-performing one. This isconsistent with the previous finding that when fewer ATRs are back tested,there is less opportunity to discover better ones (see Figure 6.43). Highercorrelation means we are effectively searching through a smaller universethan the actual number of ATRs tested would suggest. As pointed out earlier,in the extreme case in which correlation is 1.0, there is no data mining at all.When the correlation among the ATR returns is nearly perfect, 0.9, the ex-pected return of the best of 256 ATRs is only +1.4 percent. This is the same asan ATR picked at random. In other words, at a correlation of 0.9 the effectivenumber of ATRs being tested is quite small (little data mining). Less datamining means less opportunity to find good rules. Conversely, when correla-tion among rule returns is zero, the effective size of the universe is maxi-mized—we really are searching among 256 different ATRs. Consequently, theexpected return of the best ATR out of 256 is nearly 4 percent, considerablyabove the universe average of +1.4 percent (data mining works!).
Summary of Findings Regarding theData-Mining Bias
This section has presented the results of numerous experiments. It isworthwhile, at this point, to summarize our findings. They make clear thenature of the problem that must be solved by the methods presented inthe next section.
1. The observed performance of a rule discovered by data mining—thatis, the best performer in a set of back-tested rules—is positively bi-ased. Its expected future return is less than its observed historicalperformance. Its likely performance in any sample of data from the fu-ture will be lower than the back-tested performance that allowed it towin the performance competition.
2. The magnitude of the data-mining bias is profoundly affected by thenumber of observations used to compute the performance statisticused to select the best rule. The number of observations affects boththe width and tail thickness of the performance statistic’s samplingdistribution.
a. The larger the number of observations, the narrower the samplingdistribution and the smaller the data-mining bias.32O METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
b. The larger the number of observations, the lighter the tails of thesampling distribution and the smaller the data-mining bias.
3. The larger the number of rules searched, the larger the data-miningbias.
4, The smaller the variability in true quality (expected return) among therules tested, the larger will be the data-mining bias. In other words,the more equivalent the predictive power of the rules back tested, thegreater will be the bias. This point was not explored experimentally.
5. Data mining works when the sample size is large. The more rulestested, the higher the expected performance of the rule ultimately se-lected. Data mining does not work when the sample size is too small.
SOLUTIONS: DEALING WITH THE DATA-MINING BIAS
 
Data mining works but its results are positively biased. Hence, if the set ofcandidate rules examined by the data miner contains any rules with supe-rior expected returns, data mining can be an effective way to find them.However, their future performance will, in all probability, be less thantheir back-tested performance. This out-of-sample performance deteriora-tion is simply a fact of life in the data-mining business.
A worse problem occurs when all candidate rules are worthless—allhave expected returns of zero or less. In this instance, the data miner runsthe risk of being fooled by the data-mining bias and choosing a rule whoseexpected return is zero. This is the fool’s gold of the objective technician.This section discusses some ways to reduce this risk.
Three approaches have been proposed: out-of-sample testing, a data-mining correction factor proposed by Markowitz and Xu, and randomiza-tion methods. Out-of-sample testing involves excluding one or moresubsets of the historical data from the data mining (out-of-sample). Thisdata is then used to evaluate the best rule discovered in the mined data(in-sample). The selected rule’s performance on the data that was insu-lated from mining operations provides an unbiased estimate of its ex-pected return in future data. A number of schemes for partitioning thehistorical data into in-sample and out-of-sample segments have been pro-posed. I summarize several of them later.
A second approach, one that has not been widely discussed outsidethe academic literature, is based on randomization methods like boot-strapping and Monte Carlo. This approach offers a number of advantages.First, it allows the data miner to test as many rules as desired. Testing agreater number of rules increases the chance of finding superior rules.Data-Mining Bias: The Fool’s Gold of Objective TA 321
Second, it does not require holding data aside, as is done in out-of-sampletesting, thereby allowing all available historical data to be used for datamining. Third, it permits significance testing, and, sometimes, approxi-mate confidence intervals can be produced.
A third approach, the data-mining correction factor developed byMarkowitz and Xu, deflates the observed performance of the rule that didthe best. The reader is directed to source material for this approach. As apoint of interest, limited experiments conducted with this method suggestit can work surprisingly well, but it can also fail miserably, depending onthe conditions.
Out-of-Sample Testing
Out-of-sample testing is based on the valid notion that the performance ofa data-mined rule,” in out-of-sample data, provides an unbiased estimateof the rule’s future performance. Of course, due to sampling variation, therule may perform differently in the future than it does in its out-of-sampletest, but there is no reason to presume it will do worse. Because out-of-sample performance is unbiased, it makes sense to reserve data for out-of-sample testing.
This raises the issue of how to best segment the historical data into in-sample and out-of-sample subsets. Various schemes have been proposed.The simplest is to create two subsets; the early portion of the historicaldata is used for data mining while the later portion is reserved for out-of-sample testing. See diagram A in Figure 6.56.4 A more sophisticated seg-mentation scheme breaks the historical data up in a checkerboard patternsuch that both in-sample and out-of-sample data come from all parts ofthe history. See diagrams B and C in Figure 6.56.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c. [ts [o-s [ts] os [+S [0s] tS] 0-5] 1S]bFfB. [- In-Sample Out-of-Sample | In-Sample |A. In-Sample | Out-of-SampleTime >
 
FIGURE 6.56 = In-sample and out-of-sample segmentation of historical data.B22Z_ METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Walk-Forward Testing. Another data segmentation approach spe-cific to financial market trading applications is walk-forward testing. It isdescribed in Pardo,®® De La Maza,** Katz and McCormick,” and Kaufman.*It employs a moving data window, which itself is divided into an in-sampleand out-of-sample segment. The terminology used for the in-sample andout-of-sample segments is somewhat different, however. Because walk-forward testing has a dynamic aspect, in which the rule is being modifiedover time as the market evolves, the terminology alludes to a rule that islearning from its experience. Thus the in-sample segment is referred to asthe training data set because the best parameter values for the rule arelearned in this portion of the data. The out-of-sample segment is referredto as the testing data set, because the parameter values that were learnedin the training test are tested for performance in this data segment.
The notion of a moving data window is well known in time seriesanalysis and technical analysis. Moving averages, moving max-min pricechannels, and rate-of-change indicators, to name a few, employ a datawindow that slides along the time axis. Mathematical operations, such asdetermining the maximum value or the average value, are confined to thedata in the window.
In the context of rule testing, the data window (training set + testingset) that is walked forward is sometimes referred to as a fold (see Figure6.57). An out-of-sample performance estimate is computed for each fold.Because there are typically numerous folds and hence numerous out-of-sample estimates, it becomes possible to determine the variance of theperformance statistic. This allows a confidence interval to be computed.Moreover, the moving data window adds a dynamic adaptive aspect torule testing. A new rule can be formulated in each new realization of thedata window. This makes walk-forward testing especially attractive fornonstationary phenomena such as financial markets. Hsu and Kuan foundadaptive trading rules (they referred to them as learning strategies) effec-tive in their test of over 36,000 objective rules."
In a walk-forward scheme, the training segment is used for data min-ing, and then the best rule found there is tested on the testing segment.This yields an unbiased estimate of its out-of-sample performance. Then,the entire window is moved forward and the process is repeated.
The walk-forward process is illustrated in Figure 6.57. Note, the win-dow is moved forward by a sufficient amount so that testing data seg-ments in separate folds do not to overlap. In this way, the of out-of-sampleperformance estimates are independent of each other.
Limitations of Data Out-of-Sample Testing Methods. As an an-tidote to the data-mining bias, out-of-sample testing suffers from severaldeficiencies. First and foremost, the virginal status of the data reservedData-Mining Bias: The Fool’s Gold of Objective TA 323
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Fold,——_———Train Test [|- Time —>Fold,Train Test [| Time —»Fold,4Train Test = _-——_____ Time —»Fold,7,+ Train Test |_——______ Time —»
 
 
 
 
 
FIGURE 6.57 Walk-forward data testing two data segments.
for out-of-sample testing has a short life span. It is lost as soon as it is usedone time. From that point forward, it is no longer able to provide unbiasedestimates of rule performance.
A second deficiency is that it eliminates certain portions of the datafrom mining operations. Thus it reduces the amount of data available tofind patterns. When noise is high and information is low, more observa-tions are extremely valuable.
Third, the decision about how to apportion the data between the in-sample and out-of-sample subsets is arbitrary. There is no theory thatsuggests what fraction of the data should be assigned to training andtesting. Results can be very sensitive to these choices. Often it’s a seat-of-the-pants call.
Markowitz/Xu Data Mining Correction Factor
A formula for estimating the data-mining bias has been proposed byMarkowitz and Xu" (MX) and described for practical implementation inan Excel spreadsheet by De La Maza."! Markowitz is well known for his pi-oneering work in modern portfolio theory, for which he shared the 1990Nobel prize in economics.
The MX method does not require data segmentation. What is re-quired, though, is the complete set of interval returns (e.g., daily, weekly,monthly, and such) for all rules examined by the data miner en route toselecting a best-performing rule. The method corrects for the data-miningbias by shrinking the observed performance of the best rule back towardthe average performance of all rules tested. The degree of shrinkage isdetermined by several factors including: (1) the degree of variation ineach rule’s daily returns around the grand mean daily return for all rules324 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
tested, (2) the degree of variation in each rule’s mean return around themean return for all rules tested, (3) the number of rules examined, and(4) the number of time intervals (e.g., days) of returns.
The article by De La Maza, which was referred to earlier, lays out thedetails for constructing a spreadsheet that computes the value of a shrink-age factor, B, which is a number between zero and one. This value is thenplugged into the formula
H’ = R+ B(H- R)Where
H’ is the expected return for the best-performing rule after adjust-ment for the data-mining bias.
Ris the average return of all the tested rules.H is the observed performance of the best rule.B is the shrinkage factor.
The corrected or shrunken performance estimate for the best rule, H’,will lie somewhere between its observed performance and the average forall rules tested. When B equals zero, then the expected return for the bestrule is simply equal to the average return of all rules (R). As B approachesa value of 1, the expected return of the best rule approaches its back-tested return. This occurs in a case where no data-mining bias was de-tected and thus no need to shrink the observed return of the best rule.
The result given by MX is very approximate. Under some conditions,the shrunken performance estimate can be quite good. However, underother conditions easily encountered in practice, its results can be seri-ously in error. MX is best used as a rough guideline.
Randomization Methods
Chapter 5 introduced two randomization methods that can be used to testthe statistical significance of a rule’s observed performance in a back test.They were bootstrapping and Monte Carlo permutation. As pointed out inChapter 5, these significance tests were only valid for a single rule back test,that is, where there was no data mining used to discover the rule. However,with modifications described in this section, these procedures can also beused to test the statistical significance of a rule discovered with data mining.
Both randomization methods allow the data miner to use all historicaldata for rule discovery. This circumvents the arbitrary decisions requiredwith out-of-sample testing: how to partition the data into in-sample andData-Mining Bias: The Fool’s Gold of Objective TA 325
out-of-sample, how much to use for training versus testing, and so on.However, the randomization methods require that certain informationabout each rule tested during data mining be retained. Because the spe-cific information that must be retained depends on whether Monte Carloor bootstrapping is used, the information requirements of each methodwill be discussed separately.
A brief review is in order to explain how randomization methods canbe used for significance testing. The sampling distribution of a test-statisticis the foundation of statistical inference. It describes the degree of ran-dom variation in a statistic. This distribution is vital in judging the statisti-cal significance of a rule’s observed performance. Chapter 5 pointed outthat the sampling distribution can be derived by bootstrapping or theMonte Carlo permutation.
Bootstrapping the Sampling Distribution: White’s RealityCheck. White’s Reality Check (WRC) was invented and patented byeconomist Halbert White, a professor at the University of California at SanDiego. It uses the bootstrap” method to derive a sampling distributionthat is appropriate to test the statistical significance of the best rule foundby data mining. The software can be acquired from Quantmetrics,” a firmowned by Dr. White.
Prior to WRC, bootstrapping could be used to generate the samplingdistribution to test the significance of a single rule. White's innovation, forwhich he was granted a patent, allows the bootstrap to be applied to thebest rule found by data mining. Specifically, WRC permits the data minerto develop the sampling distribution for the best of N-rules, where N is thenumber of rules tested, under the assumption that all of the rules have ex-pected returns of zero. In other words, WRC generates the sampling distri-bution to test the null hypothesis that all the rules examined during datamining have expected returns of zero.
Let's take a highly simplified example to explain the procedure WRCuses. Suppose the data miner has back tested only two rules (N = 2) over10 days. I will refer to the rules as R, and R,. Over the 10-day period R,earned a mean daily return of 1 percent, whereas R, earned a mean dailyreturn of 2 percent. The data miner picks rule 2 as the best performer, butthen wonders, as one should, if the 2 percent was the result of data-miningbias and if its true expected return is actually zero. To generate the sam-pling distribution that would be appropriate for this data-mining venture,the following steps are taken:
1. The rules are tested over 10 trading days. Thus 10 slips of paper areused to represent each date of trading. They are numbered from 1through 10 and placed in a bucket for sampling.B2G_ METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
2.
on
6.
The slips of paper are then sampled with replacement 10 times. Eachtime a date is withdrawn from the bucket, the date number is noted.The number of samplings is 10 because that was the length of eachrule’s performance history. This is a requirement of the BootstrapTheorem. Suppose the following dates were selected: 5, 3, 5, 5, 7, 2,5, 8, 1, 2.
Using the dates obtained in step 2, a pseudo-track record based onthe actual daily returns associated with these dates is created for R,.This procedure is also carried out for R,. We now have a randomlygenerated performance history of length 10 for each rule.
. Recall that our goal is to generate the sampling distribution for the
best-performing rule in a universe of rules which have expected re-turns equal to zero. Because it is possible, in fact it is our hope, thatat least one of the rules tested has an expected return greater thanzero, the raw historical returns of R, and R, must be adjusted to havemean returns of zero. To accomplish this, WRC determines the meandaily return for each rule tested and subtracts this amount from eachindividual daily return.“ In this example, R,’s mean daily return is 1percent and R,’s is 2 percent. Thus 1 percent is deducted from eachof R,’s daily returns whereas 2 percent is deducted from each of R,'sdaily returns. This has the effect of shifting the distribution of dailyreturns for each rule so that it is centered over zero. These adjusteddaily returns are now used.
The adjusted returns of R, for days 5, 3, 5, 5, 7, 2, 5, 8, 1, 2 are aver-aged. The adjusted returns for R, are also averaged.
The larger of these two mean values is held aside as the first valueused to form the sampling distribution of the maximum mean returnof N rules. In this case N = 2.
Steps 2 through 6 are repeated many times (e.g., 500 or more times).
8. The sampling distribution of the statistic, maximum mean return
10.
in a universe of N (2) rules whose expected returns are zero isformed from these 500 values