toreject or not to reject H, may not always be the same. However, extensivesimulations conducted by Dr. Timothy Masters, developer of the MonteCarlo permutation method, show that both methods generally do agreewhen they are applied to detrended market data. For this reason, the hy-pothesis tests conducted in this book use detrended market data for boththe bootstrap and Monte Carlo methods.
The final distinction of the Monte Carlo permutation method is that itis in the public domain, whereas the bootstrap method that is suitable forrule testing is a patented product that is available only from its developer,Quantmetrics.'°
The Bootstrap
The bootstrap method was first described by Efron" in 1979 and then re-fined in several later publications cited in Eric Noreen’s Computer Inten-sive Methods for Testing Hypotheses.” The bootstrap derives a samplingdistribution of the test statistic by resampling with replacement from anoriginal sample.236 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
The bootstrap is based on a truly amazing mathematical fact, thebootstrap theorem. A mathematical theorem deduces a previously unrec-ognized truth from the established theorems and foundational assump-tions (axioms) of a mathematical system. Assuming that certainreasonable conditions are satisfied, the bootstrap theorem assures us thatit will converge to a correct sampling distribution as the sample size goesto infinity. From a practical standpoint, this means that given a single sam-ple of observations bootstrapping can produce the sampling distributionneeded to test the significance of a TA rule.
In its basic form, the bootstrap is not suitable for evaluating the statis-tical significance of rules discovered by data mining. However, a modifica-tion invented and patented by Dr. Halbert White, professor of economicsat the University of California, San Diego, extended the application of thebootstrap to rules discovered by data mining. This modification to thebootstrap, which is incorporated in software called “Forecaster’s RealityCheck,” is discussed in Chapter 6 and utilized in Chapter 9 to evaluate thestatistical significance of over 6,000 rules for trading the S&P 500.
Bootstrap Procedure: White’s Reality Check. The descriptionthat follows pertains to the use of bootstrapping in the context of testingthe statistical significance test of a single TA rule. Thus, the following de-scription does not address the issue of data mining. Figure 5.11 illustrates
         
        
Resample 21,231 ObservationsBS Mean = -1.8
Resample 31,231 ObservationsBS Mean = 1.9
          
Resample 11,231 ObservationsBS Mean = 0.8
                       
Returns from Tested Rule
ssOriginal Sample1,231 ObservationsSample Mean afterAdjustment = 0%
         
Resample 5,000
1,231 Observations J" \
BS Mean = -2.6
      
  
5000Resampled Means
 
 
 
 
le 0.8 Mean of ea Means
2 1.8
Be 1.9
°
°
°
5000--- -2.6 Bootstrapped Sampling DistributionOf The Mean
FIGURE 5.11 > How bootstrapping produces the sampling distribution for thesample mean return.Hypothesis Tests and Confidence Intervals 237
the bootstrap procedure. The double arrows between each resample andthe original sample indicate that the sampling is being done with replace-ment (explained below).
There are several things worthy of note in Figure 5.11. First, the origi-nal sample, represented by the large oval, is comprised of the daily returnsearned by the rule on detrended data. As discussed in Chapter 1, the de-trended market data has an average daily change of zero.
Second, before the resampling operation begins, the daily returns ofthe rule are adjusted by a procedure called zero-centering, not to be con-fused with the detrending. The zero-centering adjustment makes the meandaily return of the rule equal to zero. In other words, if the rule was able toearn a nonzero return on detrended data, its returns must be zero cen-tered. This serves the purpose of bringing the daily returns into confor-mity with the H,, which asserts that their average value is equal to zero.This step is accomplished by first computing the mean daily return of therule and then subtracting that mean value from each daily rule return.Once the rule’s daily returns have been zero centered in this fashion weare in a position to generate a sampling distribution that conforms to H,’sassumption.
Third, the number of daily observations comprising each resamplemust be exactly equal to the number of observations in the original sample.The bootstrap theorem only holds true if the number of observations ineach resample is equal to the number of observations in the original sam-ple. In the figure, the original sample is composed of 1,231 observations.Thus, each bootstrapped resample is also comprised of 1,231 observations.
Fourth, each resample is produced by sampling with replacement.This means that after a daily rule return has been selected at random fromthe original sample and its value has been noted, it is then replaced backinto the original sample before another daily return is selected at random.This makes it possible for an individual daily return to be selected morethan once or for it never to be selected at all for a given resample. It is thiselement of randomness that enables the bootstrap procedure to model thevariability in the sample statistic.
Fifth, the diagram shows 5,000 resamples being taken. A mean is com-puted for each resample. These 5,000 means are used to construct thesampling distribution of the mean.
The sequence of steps involved in bootstrapping the sampling distrib-ution of the mean is:
1. Calculate the mean daily return for the rule over the observations inthe original sample (1,231 in Figure 5.11).
2. Zero centering: Subtract the mean daily return from each day’s re-turn in the original sample.238 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
»
a
Place the zero-centered data in a bucket.
Select a daily return at random from the bucket and note its value.Place that return back in the bucket, and then thoroughly stir thebucket (some statisticians prefer their samples shaken rather thanstirred).
Perform steps 4 and 5 exactly N — 1 more times (e.g., 1,230) creatinga total of N (1,231) randomly selected observations. This completesthe first bootstrapped sample.
Compute the mean return for the N (1,231) observations in the firstresample. This is one value of the bootstrapped mean.
Perform steps 6 through 9 a large number of times (5,000) yielding alarge number of bootstrapped means.
. Form the sampling distribution of the means.. Compare the observed mean return of the rule to the sampling distri-
bution and determine the fraction of the 5,000 means comprising thesampling distribution that exceed the observed mean return of therule to determine the p-value. See Figure 5.12.
Monte Carlo Permutation Method [MCP]
Monte Carlo simulation, invented by Stanislaw Ulam (1909-1984), is ageneral method for solving mathematical problems by random sampling.The Monte Carlo permutation method for rule testing was developed byDr. Timothy Masters. He is the first to propose this approach as a way to
 
Null Hypothesis0%
 
 
 
 
AverageReturnofTestedRule
lll,
0
 
 
p-value = 0.04
 
 
 
 
FIGURE 5.12 Comparing the rule’s mean return to the bootstrapped samplingdistribution of the mean.Hypothesis Tests and Confidence Intervals 239
produce the sampling distribution to test the statistical significance of arule’s back-tested performance. It is presented as an alternative to White’sreality check.
Although the Monte Carlo method has been in existence for a longtime, it had not been previously applied to rule testing. This was madepossible by Dr. Masters’ insight that the Monte Carlo method could gener-ate the sampling distribution of a rule with no predictive power. This is ac-complished by randomly pairing or permuting the detrended daily returnsof the market (e.g., S&P 500) with the ordered" times series representingthe sequence of daily rule output values. Recall that the H, tested by theMonte Carlo permutation method asserts that the returns of the rule beingevaluated are a sample from a population of returns that were generatedby a rule with no predictive power. The daily returns of such a rule can besimulated by randomly pairing the rule’s output values (+1 and —1) withthe market's price changes. The random pairing of the rule output valueswith market changes destroys any predictive power that the rule mayhave had. I refer to this random pairing as a noise rule.
The process of randomly pairing market-price changes with rule out-put values is illustrated in Figure 5.13. The daily time series of rule outputvalues are simply those that were produced by the rule that is being evalu-ated in their original order. After the sequence of output values has beenrandomly paired with what is effectively a scrambled version of the mar-ket’s history, the mean daily return of the noise rule can be computed.This value appears in the grey box at the end of each row. To produce thesampling distribution we need many such values.
   
                                 
Time Series ofRule Output Values
Randomized SP500Return, -0.8} +0.3 | -0.9 | -2.6 |+3.1] +1.7] -0.8 | -2.6 |+1.2] -0.4
 
  
Randomized RuleReturn,
Randomized SP500 | -0.Return,
 
Randomized RuleReturn,
  
Randomized SP500Return,
 
Randomized RuleReturn,
 
 
 
 
 
 
 
 
 
 
 
 
 240 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
To produce additional values of noise rule mean returns, the sametime series of actual rule output values is paired with (permuted with) nu-merous scrambled (randomized) versions of market-price changes. The il-lustration shows only three Monte Carlo permutations but in practice itwould be done a large number of times, perhaps 5,000. The 5,000 valuesfor average return are then used to form the sampling distribution of themean return earned by a noise rule—a rule with no predictive power.
Procedure. The sequence of steps to generate the sampling distribu-tion by the Monte Carlo permutation method is as follows:
1. Obtain a sample of one-day market-price changes for the period oftime over which the TA rule was tested, detrended as described inChapter 1.
2. Obtain the time series of daily rule output values over the back-testperiod. Assume for this illustration that there were 1,231 such val-ues, one rule output value for each day on which the rule was tested.
3. Place the market’s detrended one-day-forward price changes on apiece of paper. Place them in a bin and stir.
4. Randomly select a market-price change from the bin and pair it withthe first (earliest) rule output value. Do not put the price changeback in the bin. In other words this sampling is being done withoutreplacement.
a
. Repeat step 4 until all the returns in the bin have been paired with arule output value. In this example there will be a total of 1,231 suchpairings.
6. Compute the return for each of the 1,231 random pairings. This isdone by multiplying the rule’s output value (+1 for long, —1 for short)by the market’s one-day-forward price change.
7. Compute the average return for the 1,231 returns obtained in step 6.
. Repeat steps 4 through 7 a large number of times (e.g., 5,000).
©
Form the sampling distribution of the 5,000 values obtained in step 8.
10. Place the tested rule’s rate of return on the sampling distribution andcompute the p-value (the fraction of random rule returns equal to orgreater than the tested rule’s return).
Application of Computer Intensive Methodsto Back-Test of a Single Rule
This section demonstrates the application of the two computer-intensivehypothesis testing methods to a single rule: 91 day channel breakout’* us-Hypothesis Tests and Confidence Intervals 241
ing the Dow Jones Transportation Index (input series 4) as the rule’s inputseries. This rule, which is designated as TT-4-91, and all others tested inPart Two of this book are described in detail in Chapter 8. The main pointfor current purposes is to demonstrate the hypothesis test. The rule wasused to generate long and short signals on the S&P 500 index from No-vember 1980 through June 2005. Over this period of time, the rule earnedan annualized return of 4.84 percent using detrended S&P 500 data tocompute the rule’s daily returns. The expected annual return of a rule withno predictive power is zero on detrended data.
Both the bootstrap and the Monte Carlo permutation methods wereused to test the null hypothesis that the rule has no predictive power. Thequestion is this: Is TT-4-91’s +4.84 percent return sufficient to reject thenull hypothesis?
Testing Rule Performance Using Bootstrap: White’s RealityCheck. To generate the sampling distribution of the average return, thespecific steps taken starting with the detrended S&P 500 data are:
1. Zero Centering the Daily Rule Returns: Because the rule generates apositive return (+4.84 percent) on the detrended market data, the av-erage daily return of the rule (approximately 0.0192 percent per day)is subtracted from the return earned by the rule each day. This trans-formation creates a set of daily returns whose average value is zero,thereby making the data conform to H,. Note—this is not to be con-fused with the detrending of the S&P 500 data.
2. Resampling the Daily Returns: The zero-centered daily returns ascomputed in the preceding step are sampled with replacement. Thismust be done exactly 6,800 times (the number of observations in theoriginal sample) for the Bootstrap Theorem to hold true.
3. Compute the Mean Return: The mean daily return is computed for the6,800 resampled returns. This is the first bootstrapped mean.
4. Repeat steps 2 and 3 5,000 times. This obtains 5,000 values for the re-sampled mean.
. Create the bootstrapped distribution of resampled means.
ao vl
. Compare rule’s +4.84 percent return to the sampling distribution todetermine the fraction of the sampling distribution’s area that lies atvalues equal to or greater than +4.84 percent per annum. This is doneby counting the fraction of the 5,000 bootstrapped means that havevalues equal to or greater than this return.
Results: H, Rejected—Rule Possibly Has Predictive Power.Figure 5.14 shows the bootstrapped sampling distribution with the actual242 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
0.12
TT- 4-910.1 0 i +4.84% [nanan nnnn nnn:
   
 
0.08
0.06 |--
 
 
 
0.04
 
0.02 |-
 
 
 
-0.15  -0.10 -0.05 0 0.05 0.10 0.15FIGURE 5.14 Bootstrapped sampling distribution for rule TT-4-91 and its p-value.
performance of rule TT-4-91 superimposed. The p-value of 0.0692 indi-cates that .069 of the 5,000 bootstrapped means were equal to or greaterthan +4.84 percent. This means that if the rule’s expected return weretruly equal to zero, about 7 times out of 100, the rule would earn a returnof +4.84 percent or greater simply due to chance (sampling variability).Statisticians would regard such a result as possibly significant. As will beseen in Chapter 6, the amount of searching that led to the discovery of therule, specifically whether the rule was found amongst a large group ofback-tested rules, can impact the p-value and the rule's significance. Con-sequently, the results quoted here assume that this was the only rule pro-posed for back testing.
Testing Rule Performance with Monte Carlo PermutationMethod. The following steps describe the process of applying theMonte Carlo Permutation Method to a rule’s back-tested performance:
1. The time series of daily rule output values is laid out in its proper se-quence over the time period of the rule back test. As stated earlierthere will be 6,800 of these daily +1 and —-1 values.
2. Each of the 6,800 detrended one day forward price changes is in-scribed on a ball. These 6,800 balls are placed in a bin.Hypothesis Tests and Confidence Intervals 243
3. The bin is shaken, and then, one at a time, the balls are drawn at ran-dom and paired with an individual rule output value. Thus, each of therule’s daily output values is matched with a single one day forwardS&P 500 return. This is done without replacement until all balls havebeen paired with a rule output value. Because there are exactly 6,800daily market returns and 6,800 rule output values, the bin will beempty when this step is complete.
4. Multiply each rule value (+1 or —-1) by the associated S&P 500 return.This gives the return that would be earned by the noise rule over thenext day. This step will produce 6,800 daily rule returns.
5. Average the 6,800 values obtained in step 4. This is the first MonteCarlo permuted mean return for a noise rule.
6. Steps 3 through 5 must be repeated 5,000 times.7. Produce the sampling distribution from the 5,000 Monte Carlo means.
8. Compare the return of the TT-4-91 rule with the sampling distributionand determine the fraction of Monte Carlo means that equal or exceedthe return earned by the rule. This is the p-value.
Monte Carlo Results: Rule Possibly Has Predictive Power.
Confirming the bootstrapped result, the Monte Carlo method gives a p-valuethat is almost identical.
ESTIMATION
 
Estimation is the other form of statistical inference. In contrast to hypoth-esis testing, which is oriented to the acceptance or rejection of a claimmade about the value of a population parameter, estimation’s purpose isto approximate the value of the population parameter. In our case, it willbe used to estimate the expected return of a rule.
Point Estimates
There are two kinds of estimates: point and interval. A point estimate is asingle value that approximates the population parameter, for example therule has an expected return of 10 percent. An interval estimate is a rangeof values within which the population parameter lies with a given level ofprobability. The following statement would exemplify this: The rule’s ex-pected return lies within the range 5 percent to 15 percent with a proba-bility of 0.95.
Actually, we already have been making point estimates, but they have244 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
not been described as such. Every time we compute a sample mean anduse it to approximate the population mean, we are making a point esti-mate. This fact is easily overlooked. Some commonly used point estima-tors are: the mean, median, standard deviation, and the variance. Theestimate is computed from a sample of observations taken from the popu-lation. In other words, a point estimate is a sample statistic. The formulato compute a sample mean is shown in Figure 5.15.
The use of means (averages) is so ubiquitous in TA that it is taken forgranted, yet the sample mean turns out to be an elegant and powerful esti-mator. It provides a single value that is, in an important sense, the best(most informative) estimate of the population’s mean. This is an impor-tant fact.
Just how informative the sample mean is becomes clear when weconsider the criteria used to judge the quality of an estimator. Good esti-mators should be: unbiased, consistent, efficient, and sufficient. In termsof these four criteria, it can be shown that the sample mean is the best es-timator of the population mean.”
An estimator is unbiased if its expected value is equal to the popula-tion value. Said differently, if an estimator is unbiased its deviations fromthe true population value have an average value of zero. The samplemean’s deviations from the population mean are unbiased. This allows usto say that a rule’s mean return in a historical sample is an unbiased esti-mate of its mean return in the immediate practical future.
Another criterion of a point estimator’s goodness is its consistency.An estimator is said to be consistent if its value converges to the value of
Sample Mean of Variable X
— XX) +X +X3 +. . .xx =?
n__ bsXx = —
n
 
 
Where X, is an individual observation on variable X
FIGURE 5.15 Sample mean of variable X.Hypothesis Tests and Confidence Intervals 245
the population parameter as sample size is increased. The Law of LargeNumbers tells us that this is so for the sample mean.
Estimators should also be efficient. This criterion relates to the widthof its sampling distribution. As mentioned earlier, an estimator is a samplestatistic, and thus has a sampling distribution. The most efficient estima-tor is the one that produces the narrowest sampling distribution. In otherwords, the most efficient estimator has the smallest standard error.'® Boththe sample mean and the sample median are unbiased and consistent esti-mators of the population mean for populations that are distributed sym-metrically. However, the sample mean is more efficient than the samplemedian. For large samples, the standard error of the mean is about 80 per-cent smaller than the sample median’s standard error.“
The final trait of a good point estimator is called its sufficiency. “Anestimator is sufficient if it makes such use of all the available sample datathat no other estimator would add any information about the parameterbeing estimated.”'* The sample mean is sufficient in this sense.
Interval Kstimates—The Confidence Interval
More informative than the point estimate is the interval estimate, alsoknown as a confidence interval. It is described in this section.
What Do Confidence Intervals Tell Us? A point estimate has lim-ited value because it conveys no sense of the uncertainty in the estimatedue to sampling error. The confidence interval solves this problem bycombining the information of the point estimate with the informationabout the estimator’s sampling distribution.
A confidence interval is a range of values that surround the point esti-mate. The interval is defined by upper and lower values called bounds. Inaddition, the interval is accompanied by a probability number that tells ushow likely it is that the true value of the population parameter falls withinthe bounds of the confidence interval. By convention the probability isstated as a percentage rather than a fraction. Thus, a 90 percent confi-dence interval for the mean has a 0.90 probability of enclosing within itsbounds the population’s true mean value.
When thinking about what a confidence interval tells us, it is best tothink of what would happen if one were to construct a large number of 90percent confidence intervals, each based on an independent sample ofdata taken from the population. If this were to be done, about 90 percentof the intervals would actually encompass the value of the population’sparameter. By extension, about 10 percent of the confidence intervalswould fail to include the population parameter. This is illustrated in Fig-ure 5.16 for 10 confidence intervals. Ten is a small number, which I used to246 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
True Population Mean
wu
 
Means H—e—& e :90% Confidence e ErrorIntervals Le 90% CIFor Fl Fails Toi _—e—_10 Independent | + -@ — <7 EncloseSamples [-—_e Population
 
 
 
e+ Mean
FIGURE 5.16 90 percent confidence intervals (.90 probability correct).
keep the figure simple. The true, but unknown, population mean is indi-cated by the Greek letter » (mu). The sample mean is identified by the dotwithin each confidence interval. Note that one of the confidence intervalsfails to enclose the population mean.
The researcher can choose whatever confidence level is desired. Forexample a 99 percent confidence level contains the true population meanwith a probability of 0.99. Of course, there is a price to be paid for thehigher level of confidence—the interval is wider. In other words, the pricefor higher confidence is reduced precision. Note in Figure 5.17, by increas-ing the confidence level to 99 percent, the error made by the 90 percent
True Population Mean
wu
 
 
 
 
Means t-—_o———+ 99% CI& e Thad Correct99% e WhereConfidence _¢ 90% ClIntervals H Ke WasFor 10 } ie e 1 WrongIndependent KeSamples [I——_e——_
FIGURE 5.17 99 percent confidence intervals (.99 probability correct).Hypothesis Tests and Confidence Intervals 247
confidence interval in Figure 5.16 has been eliminated. The reduced errorrate was accomplished by using a wider (less precise) confidence interval.
If arule were to be back tested on one hundred independent samplesof data and a 90 percent confidence interval were constructed about themean return observed in each sample, approximately 90 percent of theseconfidence intervals would contain the rule’s true or expected return inthe population. Figure 5.18 shows a 90 percent confidence interval for arule that earned a 7 percent return in a back test. We can be certain, with aprobability of 0.90, that the rule has an expected return between 2 percentand 12 percent.
The Confidence Interval and Its Connection to the SamplingDistribution. Confidence intervals are derived from the same sam-pling distribution that is used to compute the p-values for a hypothesistest. Given what we have already learned about sampling error (samplingvariability), it can be said that the value of a sample mean is equal to theunknown value of the population mean, plus or minus its sampling error.This relationship is shown by the upper formula in Figure 5.19. By rear-ranging the terms of this formula we get the lower formula in Figure 5.19.It says that the value of the population mean is equal to the known valueof the sample mean plus or minus sampling error.
The bottom formula tells us that, although we do not know the pre-cise value of the population mean, we can take the value of the samplemean, which we do know, and the sampling distribution of the mean,which we also know, and obtain a range of values that contains the popu-lation mean with a specified level of probability. In operational terms,this tells us that if we were to repeat the following procedure 1,000times—compute a sample mean and a 90 percent confidence interval—
 
Rule ReturnWas 7% InHistorical Sample
 
 
 
 
 
Rate of Return
FIGURE 5.18 90 percent confidence interval for rule back test.248 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
The known sample mean isthe unknown population mean with error
X = Lu +/- Sampling Error
The unknown population mean isthe known sample mean with error
Lu = X +/- Sampling Error
FIGURE 5.19 The known sample mean is the unknown population mean witherror.
the population mean would lie within approximately 900 of the 1,000 con-fidence intervals. This concept is illustrated when the procedure is re-peated only 10 times in Figure 5.20. Note that one of the 10 confidenceintervals fails to include the population mean. The point of this section isthat the confidence interval’s width is derived from the width of the sam-pling distribution.
As was said earlier, the confidence interval is based on the same sam-pling distribution that is used in the hypothesis test. However, in the caseof the confidence interval, the sampling distribution is simply shifted fromthe position it occupies in a hypothesis test. In the hypothesis test, thesampling distribution is centered at the hypothesized value of the popula-tion mean, for example, zero. In the case of a confidence interval, the sam-pling distribution is centered over the sample mean, for example, 7percent. This concept is illustrated in Figure 5.21.
Generating Confidence Intervals with the Bootstrap. Boot-strapping can be used to derive confidence intervals. The procedure is al-most identical to the one used to generate the sampling distribution for ahypothesis test.
There are numerous methods for computing bootstrap confidence in-tervals. The one presented here, the bootstrap percentile method, is popu-lar, easy to use, and generally gives good results. More sophisticatedmethods are beyond the scope of this text.
It should be pointed out that the Monte Carlo permutation methodcannot be used to generate confidence intervals. This is because theHypothesis Tests and Confidence Intervals 249
Sampling90% DistributionArea of the Mean
 
 
 
 
 
Confidence IntervalsDerived fromSampling Distribution
Means&
90% ConfidenceIntervalsfor10 IndependentSamples
 
FIGURE 5.20 The connection between the confidence interval and the sampling
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
distribution.Hypothesized Mean ReturnValue .Population inHistorical SampleParameterSampling Distribution Sampling DistributionPositioned for Positioned forHypothesis Test Confidence Interval0% 7%FIGURE 5.21 The sampling distribution positioned for a hypothesis test and
positioned for a confidence interval.250 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
method has nothing to do with estimating the value of a population para-meter or testing a claim made about its value. As previously mentioned,the Monte Carlo permutation method tests a claim about the informationcontent of the rule’s signals. Specifically, the H, asserted under the MonteCarlo permutation method is that the long and short positions dictated bythe rule are devoid of useful information about future market changes. Be-cause there is no reference to a population parameter (e.g., mean rule re-turn) there is nothing to create a confidence interval for!
The bootstrap percentile procedure for constructing the confidenceintervals works as follows: Suppose the rule’s returns have been resam-pled 5,000 times and a mean is computed for each resample. This wouldresult in 5,000 different values for the resampled mean return. We knowthat because of sampling variability, these means will differ. Next supposethat the set of 5,000 values is arranged in rank order from highest mean tolowest. Then, depending on the confidence interval desired, the highest xpercent and lowest >x percent of values are removed from the orderedlist, where
_ 100-Confidence Interval Desired7 2
x
So if a 90 percent confidence interval is desired, one would removethe highest 5 percent and the lowest 5 percent of the values in the 5,000 re-sampled means. This would require removing the highest 250 and the 250lowest values of the resampled mean. After these extreme values are re-moved, the highest remaining resampled mean would be the upper boundof the 90 percent confidence interval and the lowest remaining resampledmean is the lower bound.The 99 percent confidence interval would resultby removing only the highest 25 (top .5 percent) and the lowest 25 (bot-tom .5 percent) from the set of 5,000 resampled means comprising thesampling distribution.
Hypothesis Tests versus Confidence Intervals: Potential Con-flict. Some astute readers may have envisioned a problem. It is possiblefor a hypothesis