hypothe-sis. Though this line of reasoning is counterintuitive, it is logically correct(denial of the consequent) and extremely powerful. It is the logical basisof scientific discovery.
To give a concrete example, suppose I view myself as an excellent so-cial tennis player. My hypothesis is David Aronson is an excellent socialtennis player. I join a tennis club with members whose age and years ofplay are similar to mine. On the basis of my hypothesis, I confidently pre-dict to other club members that I will win at least three-quarters of mygames (predicted win rate = 0.75). This prediction is merely a deductiveconsequence of my hypothesis. I test the hypothesis by keeping track ofmy first 20 games. After 20 games I am shocked and disappointed. Not onlyhave I not scored a single victory (observed win rate = 0), but most losseshave been by wide margins. This outcome is clearly inconsistent with theprediction deduced from my hypothesis. Said differently, my hypothesisimplied that this evidence had a very low probability of occurrence. Suchsurprising evidence forcefully calls for a revision (falsification) of my hy-pothesis. Unless I prefer feel-good delusions to observed evidence, it istime to abandon my delusions of tennis grandeur.”
In the preceding situation, the evidence was overwhelmingly clear. Ilost every one of 20 games. However, what if the evidence had been am-biguous? Suppose I had won two-thirds of my games. An observed winrate of 0.66 is below the predicted win rate of 0.75 but not dramatically so.Was this merely a random negative deviation from the predicted win rateor was the deviation of sufficient magnitude to indicate the hypothesisabout my tennis ability was faulty? This is where statistical analysis be-comes necessary. It attempts to answer the question: Was the differencebetween the observed win rate (0.66) and the win rate predicted by my hy-pothesis (0.75) large enough to raise doubts about the veracity of the hy-pothesis? Or, alternatively: Was the difference between 0.66 and 0.75merely random variation in that particular sample of tennis matches? Thehypothesis test attempts to distinguish prediction errors that are smallenough to be the result of random sampling from errors so large that theyindicate a faulty hypothesis.
Dueling Hypotheses: The Null Hypothesis versusthe Alternative Hypothesis
A hypothesis test relies on the method of indirect proof. That is, it estab-lishes the truth of something by showing that something else is false. There-fore, to prove the hypothesis that we would like to demonstrate as correct,we show that an opposing hypothesis is incorrect. To establish that hy-pothesis A is true, we show that the opposing hypothesis Not-A is false.222 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
A hypothesis test, therefore, involves two hypotheses. One is calledthe null hypothesis and the other the alternative hypothesis. The namesare strange, but they are so well entrenched that they will be used here.The alternative hypothesis, the one the scientist would like to prove, as-serts the discovery of important new knowledge. The opposing or null hy-pothesis simply asserts that nothing new has been discovered. Forexample, Jonas Salk, inventor of the polio vaccine, put forward the alter-native hypothesis that his new vaccine would prevent polio more effec-tively than a placebo. The null hypothesis asserted that the Salk vaccinewould not prevent polio more effectively than a placebo. For the TA rulestested in this book, the alternative hypothesis asserts the rule has an ex-pected return greater than zero. The null hypothesis asserts that the ruledoes not have an expected return greater than zero.
For purposes of brevity, I will adopt the conventional notation: H, forthe alternative hypothesis, and H, for the null hypothesis. A way to re-member this is the null hypothesis asserts that zero new knowledge hasbeen discovered, thus the symbol Hy.
It is crucial to the logic of a hypothesis test that H, and H, be definedas mutually exclusive and exhaustive propositions. What does this mean?Two propositions are said to be exhaustive if, when taken together, theycover all possibilities. H, and H, cover all possibilities. Either the poliovaccine has a preventive effect or it does not. There is no other possibility.Either a TA rule generates returns greater than zero or it does not.
The two hypotheses must also be defined as mutually exclusive. Mu-tually exclusive propositions cannot both be true at the same time, so if H,is shown to be false, then H, must be true and vice versa. By defining thehypotheses as exhaustive and mutually exclusive statements, if it can beshown that one hypothesis is false, then we are left with the inescapable
Null AlternativeHypothesis HypothesisRule Return Rule ReturnLess Than Greater ThanOr Equal to Zero
Zero
 
FIGURE 5.1 Mutually exclusive and exhaustive hypotheses.Hypothesis Tests and Confidence Intervals 223
conclusion that the other hypothesis must be true. Proving truth in thisfashion is called the method of indirect proof. These concepts are illus-trated in the Figure 5.1.
RATIONALE OF THE HYPOTHESIS TEST
 
Two aspects of the hypothesis test warrant explanation. First, why thetest is focused on the null hypothesis. Second, why the null hypothesis isassumed to be true rather than the alternative hypothesis. This section ex-plains the reasoning behind both aspects.
Why Is the Null Hypothesis the Targetof the Test?
As discussed in Chapter 3, evidence can be used to logically deduce thata hypothesis is false, but it cannot be used to deduce that it is true.?Therefore, hypothesis testing must be about trying to falsify something.The question is: Which of the hypotheses, H, or H,, should be the targetof this effort?
Of the two competing claims, H, presents a better target for falsifica-tion because it can be reduced to a single claim about the value of the pa-rameter. This means that only one test must be performed. If that singlevalue can be successfully challenged with evidence, H, will have been fal-sified. In contrast, the alternative hypothesis represents an infinite num-ber of claims about the parameter’s value. With no unique value to shootat, an infinite number of tests would have to be performed to falsify the al-ternative hypothesis.
In fact, both H, and H, represent an infinite number of claims aboutthe rule’s expected return, but H, can be reduced to a single claim. First,let’s consider why H, represents an infinite number of claims. In assert-ing that the rule’s return is greater than zero, H, effectively says that therule’s expected return, over the immediate practical future, might be anyone of an infinite set of values greater than zero: + 0.1 percent, or +2 per-cent or +6 percent or any other positive value. This is illustrated in Figure5.2. Given that H, makes an infinite number of claims, an infinite numberof tests would have to be conducted to conclusively refute it. Clearly thisis impractical.
H, also makes an infinite number of claims about the value ofthe population parameter. It asserts that the rule’s average return isequal to zero or some value less than zero. However, only one of theseclaims really matters—that the rule’s average return is equal to zero.224 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Etc.
     
 
Negative Returns Positive Returns
H, Asserts:Rule’s Expected Return > 0%.
FIGURE 5.2 The alternative hypothesis makes an infinite number of claimsabout the rule’s expected rate of return.
The attempt to falsify a single claim is a practical goal. If the most posi-tive of these claims (return = 0) can be called into question by the rule’sback-test profitability, then all lesser claims (e.g., the rule’s return = -1.3percent) are contradicted but to an even greater degree. This is how H,reduces to the single claim that the rule’s return is zero. This is illus-trated in Figure 5.3.
 
 
Positive Returns
Negative Returns
 
Ho Asserts:Rule’s Return < or = 0%.
FIGURE 5.3 The null hypothesis makes an infinite number of claims about thepopulation mean, but only one matters.Hypothesis Tests and Confidence Intervals 225
Why Is the Null Hypothesis Assumed to Be True?
The hypothesis test assumes H, is true on two grounds: scientific skepti-cism and the principle of simplicity (parsimony).
Skepticism. The assumption that H, is true is consistent with science’sskeptical attitude toward all new claims of knowledge. As explained inChapter 3, this conservative stance is justified because making claims iseasy but making genuine discoveries is hard. The burden of proof shouldbe on the claimant.
Science has a legitimate concern with protecting the storehouse ofknowledge from contamination with falsehoods and weird beliefs. A use-ful analogy can be made to the criminal justice system. Free societieshave a legitimate concern with protecting the lone citizen from the vastprosecutorial power of the state. For this reason the defendant in a crimi-nal prosecution starts off with a presumption of innocence. This placesthe burden of proving guilt—falsifying the assumption of innocence—onthe state, and it is indeed a substantial burden. To gain a conviction, thestate must provide evidence of guilt beyond a reasonable doubt, a veryhigh threshold. The initial assumption of innocence and the high thresh-old of proof can be seen as the legal system’s way of preventing the jailsand gallows from being contaminated with innocent citizens.
Following the scientific tradition, the hypothesis test places the bur-den of proof on those asserting new knowledge. In fact, whenever theback-tested performance of a rule is greater than zero, the sample evi-dence actually favors H,—that the rule’s expected return is greater thanzero. However, the hypothesis test demands more than merely reasonableevidence. It demands compelling evidence before abandoning the as-sumed truth of H,. Those asserting that a TA rule has predictive power(H,) must meet a large burden of proof before they can reasonably expecttheir claim to be accepted by scientific practitioners of TA.
Simplicity. An additional reason to grant priority to H, is the principleof simplicity. This fundamental precept of science says that simpler theo-ries are more likely to capture the true patterns of nature than more elab-orate ones. This principle, known as Occam’s Razor, says that if aphenomenon can be explained by more than one hypothesis, then the sim-plest hypothesis is more likely to be correct. Hj, which explains a rule’spast success as luck, is simpler than H,, which asserts that profits stemfrom a recurring market pattern with predictive power.
Simpler explanations (theories, rules, hypotheses, models, and soforth) are more likely to be correct because they are less likely to fit databy chance. The more complex an explanation—that is, the more numerous226 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
its assumptions, its conditions, and constraints—the more likely it is to fita set of observations by chance. This is illustrated when fitting a mathe-matical function to a set of data. A function can be thought of as a mathe-matical hypothesis that explains a given set of observations. In Figure 5.4,two mathematical functions have been fitted to the same set of data. Onefunction is linear. It is relatively simple in that it is defined by only two co-efficients or degrees of freedom: its slope and where it cuts the verticalaxis (Y-intercept). In other words, the line’s fit to the data can be im-proved by manipulating only these two factors.
The other function is a complex polynomial with 10 coefficients.Each coefficient allows the curve to make an additional bend. With thisdegree of flexibility, the curve can weave and bend so that it touchesevery data point. In fact, when a function is fit to data using a methodcalled least-squares regression, a perfect fit is guaranteed if the functionis allowed to contain as many degrees of freedom (coefficients) as thereare data points. Although the linear function does not manage to touchevery observed point, it does describe the general tendency of the data;increasing values of X are associated with increasing values of Y. In otherwords, the simple function captures the essential feature of the data,making it more likely that it represents the real relationship between Xand Y. In contrast, the complex function is most likely a detailed descrip-tion of the data’s random fluctuations in addition to the positive correla-tion between X and Y.
 
 
FIGURE 5.4 The superiority of simplicity—all else being equal.Hypothesis Tests and Confidence Intervals 227
This is not to suggest that complex curves are never justified. If thereis enough data and it was generated by a complex process, an elaboratemodel may very well be justified. However, all else being equal, the sim-pler explanation is more likely to be correct.
Strong and Weak Decisions
A hypothesis test leads to one of two decisions; reject H, or retain H,. Thedecisions are qualitatively different: the first is a strong decision while thesecond is a weak one.‘ Rejection is a strong decision because it is com-pelled by improbable, informative evidence that forcefully contradicts H).In contrast, the decision to retain H, is a relatively weaker choice becausethe evidence is nothing more than consistent with what we had alreadyexpected and assumed to be so, that H, is true. In others words, the ob-served value of the test statistic is unsurprising and hence uninformative.
Had tests of Salk’s polio vaccine shown that recipients of the real vac-cine had the same risk of getting the disease as recipients of the placebo,H, would be left intact, and no one would have been surprised by the vac-cine’s failure. Nothing had worked up to that point, so it would have beenjust another frustrating day at the lab. Of course this is not what actuallyhappened, and the course of medical history was changed. Salk’s decisionto reject H, was compelled by a rate of infection in the treated group thatwas surprisingly less than the placebo group.
It is in this way that a decision to reject H, is fundamentally a strongerdecision than one to retain H,, It is a decision forced by evidence that isstrong enough to rebut an initial and entirely reasonable doubt that newknowledge has been discovered. In contrast, the decision to retain H, isdue to an absence of compelling evidence. The absence of compelling evi-dence does not mean the null is necessarily true or even probably true.° Itsimply means that it could be true. Because science takes a conservativestance toward new knowledge, in the absence of compelling evidence, themore reasonable conclusion is that nothing new has been discovered.
HYPOTHESIS TESTING: THE MECHANICS
 
“Three ingredients are usually necessary for a hypothesis test: (1) a hy-pothesis, (2) a test statistic, and (3) some means of generating the proba-bility distribution (sampling distribution) of the test statistic under theassumption that the hypothesis is true.”’ The term test statistic refers tothe sample statistic that is being used to test a hypothesis. Thus, the termsare interchangeable.228 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
To recap these items as they pertain to rule testing: (1) the hypothesisH, is that the rule has an expected return of zero or less, (2) the test statis-tic is the rule’s mean return obtained by back testing it in a historical sam-ple of data, and (3) the sampling distribution represents the randomvariation in the rule’s mean return if it were to be tested in many indepen-dent samples. The sampling distribution is centered at a mean return ofzero, reflecting the assumption asserted by H).
As discussed in Chapter 4, the sampling distribution can be derived intwo ways: the analytical approach of classical statistics and via computersimulation. There are two computer-based approaches: Monte-Carlo per-mutation and the bootstrap. Both will be used in the case study presentedin Part Two.
The Question: Is the Test Statistic Improbable?
The basic idea of a hypothesis test is simple: an outcome (observation)that would rarely happen under the condition that the hypothesis weretrue is good evidence that the hypothesis is not true.”’ If my hypothesisthat [am a good tennis player were true, then it would be rare (improba-ble) for me to lose 20 games in a row. To my surprise and embarrassment Idid lose 20 games in a row. That evidence implies my hypothesis is false.Suppose a TA rule called MAS5O is back tested. The MA50 rule is de-fined as follows: If S&P 500 close is greater than a 50-day moving average,hold a long position in the S&P 500, otherwise hold a short position. Thealternative hypothesis (H,) asserts that the rule has predictive power andis, therefore, expected to earn a rate of return greater than zero on de-trended data.* Zero is the rate of return expected for a rule with no pre-dictive power. The null hypothesis (H,) asserts that the MA50’s expectedreturn is equal to zero or less. H,'s assertion is depicted in Figure 5.5.
 
Value of the Population ParameterAsserted by H,
 
 
 
 
Negative Returns oO Positive Returns
FIGURE 5.5 Value of population parameter asserted by H,.Hypothesis Tests and Confidence Intervals 229
The horizontal axis represents expected return over the immediate prac-tical future.
MA50’s mean return is obtained by back testing the rule in a sample ofhistorical data. When this value is plotted on the same axis as the hypoth-esized value for population parameter, we get Figure 5.6.
Note that in Figure 5.6 there is a positive deviation between the valuepredicted by H, and the performance of the rule obtained in the back test.This raises the question: Is the positive deviation so surprising that H,should be rejected as an implausible hypothesis?
There are two possible explanations for the positive deviation. Itcould simply be due to sampling error—the rule got lucky in the particularsample of data used for back-testing—or, it could be because the hypothe-sized value of zero is wrong—the rule does have predictive power and itsexpected return is indeed greater than zero. The objective of the hypothe-sis test is to determine if the evidence, specifically the size of positive de-viation, is sufficiently rare, surprising, or improbable that it would warranta rejection of H,,
To assess the improbability of the evidence, the observed averagereturn, in particular its deviation from the hypothesized value, is evalu-ated in light of the sampling distribution. Recall, the sampling gives theprobability for various sized deviations between the observed value ofthe sample statistic and its expected value due to sampling error. If theobserved value’s deviation is greater than what could reasonably be at-tributed to sampling error, then H, is rejected and the alternative hy-pothesis, H,, is adopted. In other words, we conclude that the rule haspredictive power.
Figure 5.7 gives us an intuitive sense of the values of the sample statistic
 
Value of the Population ParameterAsserted by H,
 
 
 
       
 
Sample StatisticBack-TestedPerformance
of the rule
 
 
 
Deviation
  
vNegative Returns oO Positive Returns
FIGURE 5.6 Hypothesized value of population parameter compared to back-tested performance.230 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
 
Value of the Population ParameterAsserted by H,
 
 
 
 
 
 
 
Sample Statistic
Sampling Distribution
for |Mean Return Deviation
 
 
 
Negative Returns 0 Positive Returns
FIGURE 5.7 Unsurprising evidence lies well within the range of sampling varia-tion; H, not rejected.
that could occur due to sampling variability. Note that the sampling distri-bution is positioned so that the most likely value of the return is zero. Thismerely reflects the value of the population parameter asserted by H,. Alsonote that the observed (back tested) value of the rule’s average return fallswell within the range of random variation allowed by the sampling distrib-ution. This is an unsurprising result. In other words, the deviation be-tween the observed value of the test statistic and the hypothesized(predicted) value could easily be due to sampling error. Hence the evi-dence is not strong enough to warrant a rejection of H,.
Figure 5.7 makes it clear that the width of the sampling distribution iscritical in deciding if the deviation between the observed return and thehypothesized return is large enough to warrant a rejection of H,. We sawin Chapter 4 that the width of a statistic’s sampling distribution is deter-mined by two factors: (1) the amount of variation within the parent popu-lation which gave rise to the sample, and (2) the number of observationscomprising the sample. With respect to the first factor, the greater thevariability of the data comprising the population, in this case daily rule re-turns, the larger the width of the sampling distribution. With respect to thesecond factor, the larger the number of observations comprising the sam-ple, the smaller the width of the sampling distribution.
In Figure 5.8 the sampling distribution is relatively narrow. The ob-served value of the sample statistic lies in the outer right tail of the sam-pling distribution. This would be considered an improbable or surprisingobservation and one that would be incompatible with the hypothesizedvalue. Such evidence would warrant a rejection of H,.Hypothesis Tests and Confidence Intervals 231
 
Value of the Population ParameterAsserted by H,
 
 
 
           
 
 
 
 
Sample StatisticSampling Distribution :for
Mean ReturnDeviation
 
Negative Returns 0 Positive Returns
FIGURE 5.8 Surprising (improbable) evidence at the outer edge of the range ofsampling variation; H, rejected.
These diagrams convey an intuitive sense of how the size of the devia-tion between an observed value and a hypothesized value (the value pre-dicted by the hypothesis) is used to falsify a hypothesis. To be rigorous,this intuition must be quantified. This is done by translating the observedvalue’s deviation into a probability— specifically the probability of observ-ing a deviation that large under the condition that the hypothesized valueis true. A probability that is contingent on the existence of a specifiedcondition, in this case that the hypothesized value is true, is called a condi-tional probability. Said differently, a conditional probability is a probabil-ity that is conditional upon some other fact being true.
In a hypothesis test, this conditional probability is given the specialname p-value. Specifically, it is the probability that the observed value ofthe test statistic could have occurred conditioned upon (given that) thehypothesis being tested (H,) is true. The smaller the p-value, the greater isour justification for calling into question the truth of H,, If the p-value isless than a threshold, which must be defined before the test is carried out,H, is rejected and H, accepted. The p-value can also be interpreted as theprobability H, will be erroneously rejected when H, is in fact true. P-valuealso has a graphical interpretation. It is equal to the fraction of the sam-pling distribution’s total area that lies at values equal to and greater thanthe observed value of the test statistic.
Let’s consider how all of this pertains to the test of a rule. For exam-ple, if a rule’s return in a back test was +3.5 percent, we mark the value+3.5 percent on the horizontal axis upon which the sampling distributionsits. We then determine the fraction of the sampling distribution’s area232 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
occupying values equal to or greater than +3.5 percent. Suppose that thisarea equals 0.10 of the sampling distribution’s total area. The value 0.10 isthe sample statistic’s p-value. This fact is equivalent to saying that if therule’s true return were zero, there is a 0.10 probability that its return in aback test would attain a value as high as +3.5 percent or higher due tosampling variability (chance). This is illustrated in Figure 5.9.
p-value, Statistical Significance, andRejecting the Null Hypothesis
Asecond name for the p-value of the test statistic is the statistical signif-icance of the test. The smaller the p-value, the more statistically signifi-cant the test result. A statistically significant result is one for which thep-value is low enough to warrant a rejection of H,.
The smaller the p-value of a test statistic, the more confident we canbe that a rejection of the null hypothesis is a correct decision. The p-valuecan be looked upon as the degree to which the observed value of the teststatistic conforms to the null hypothesis (H,). Larger p-values meangreater conformity, and smaller values mean less conformity. This is sim-ply another way of saying that the more surprising (improbable) an obser-vation is in relation to a given view of the world (the hypothesis), themore likely it is that world view is false.
How small does the p-value need to be to justify a rejection of theH,? This is problem specific and relates to the cost that would be in-curred by an erroneous rejection. We will deal with the matter of errorsand their costs in a moment. However, there are some standards that
 
Null Hypothesis&Sampling DistributionMean Return
 
 
 
      
 
Test Statistic: +3.5%p-value = 0.10
 
 
 
 
Area = 0.10 of total sampling distribution
FIGURE 5.9 P-Value: fractional area of sampling distribution greater than +3.5%,conditional probability of +3.5% or more given that H, is true.Hypothesis Tests and Confidence Intervals 233
are commonly used. A p-value of 0.10 is often called possibly signifi-cant. A p-value of 0.05 or less is typically termed statistically significantand is usually considered to be the largest p-value that would give a sci-entist license to reject H,. When the p-value is 0.01 or less it is calledcalled very significant and values of 0.001 or less are termed highlysignificant.
Test Conclusions and Errors
A hypothesis test leads to one of two possible conclusions: (1) reject H,,or (2) accept H,. What is actually true also has two possibilities: (1) H, istrue, that is, the expected return of the rule is zero or less); or (2) H, is infact false, that is, the rule’s expected return is greater than zero because itpossesses some degree of predictive power. Given that the test has twopossible conclusions, and the truth has two possible states, there are fourpossible outcomes. They can be illustrated by a table with four cellsshown in Figure 5.10.
The hypothesis test can err in two ways. A type I error is said to oc-cur when a low p-value leads us to reject H,, but in reality H, is true. Thiswould be the case when a rule is truly devoid of predictive power, but byluck it generates a sufficiently profitable back test that its p-value is lowenough to justify rejecting H,. This is a case of the rule researcher beingfooled by randomness. The second type of error, called a type II error,occurs when a high p-value leads us to retain H,, when it is in fact false.In other words, the back test fooled us into concluding the rule has nopredictive power, but it actually does and its expected return is greater
 
 
 
 
 
 
 
than zero.Truth(The Reality Only Known to God)H, True H, FalseRule Return < = 0 Rule Return > 0High Correct Decision Type Il ErrorP-value | TA Rule Useless TA Rule GoodH, We Discard It We Don't Use ItTest Not Opportunity LossResult Rejected& | a: Low | Type | Error Correct DecisionConclusion | | iP iue | TA Rule Useless TA Rule GoodH We Use it We Use Rule &me Earn Zero Return Make ProfitsRejected | "a Take Risk
 
 
 
 
 
FIGURE 5.10 Possible outcomes of a hypothesis test.234 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
At the time the hypothesis test is conducted, only God knows for cer-tain if an error has occurred and of which type. Mere mortals relying onstatistical inference must accept the reality that the test’s conclusion maybe in error.
From the standpoint of an objective technician, the two types of er-rors have different consequences. A type I error, where H, is mistakenlyrejected, leads to the use of a worthless rule. This exposes trading capitalto risk without the prospect of compensation. A type II error causes a use-ful rule to be ignored, resulting in lost trading opportunities. Of the twokinds of error, type I is more serious. Lost trading capital is worse thanlost opportunities. When capital is exhausted one is out of the game,whereas there will always be other trading opportunities.
The hypothesis test can also be right in two ways: a correct rejectionof H,, when the rule does have merit, and a correct acceptance of the H,,when the TA rule is useless.
COMPUTER-INTENSIVE METHODS FOR GENERATINGTHE SAMPLING DISTRIBUTION
 
As previously mentioned, hypothesis testing requires a method for esti-mating the shape of the sampling distribution of the test statistic. Thereare two ways to do this: the traditional approach of mathematical statis-tics, and the more recently developed computer-intensive randomizationmethods. This section discusses two computer-based methods: bootstrap-ping and the Monte Carlo permutation.
Both the traditional and computer-intensive approaches solve theproblem of estimating the degree of random variation in a test statisticwhen there is only a single sample of data and, therefore, only a singlevalue of the test statistic. As previously stated, a single value of a statisticcannot convey a sense of its variability.
Computer-intensive methods estimate the sampling distribution’sshape by randomly resampling (reusing) the original sample of observa-tion so as to produce new computer- generated samples. A test statistic isthen computed for each resample. This procedure can be repeated asmany times as desired, perhaps thousands of times, thus producing a largeset of values for the sample statistic. The sampling distribution is devel-oped from this large set of computer-generated values. It might seemstrange that reusing the original sample of observations over and overagain would allow one to approximate the variability of a sample statistic,yet it does! Not only does this work quite well in practice, the approach isgrounded in sound mathematical theory.Hypothesis Tests and Confidence Intervals 235
The two computer-intensive methods, the bootstrap and MonteCarlo permutation, are similar in that they both rely on randomization.That is, they randomly resample the original sample. However, the twomethods are different in several important respects. First, they testslightly different versions of H,. Although, in both, H, asserts that therule being tested has no predictive power, they do so in slightly differ-ent ways. The H, tested by the bootstrap asserts that the population dis-tribution of rule returns has an expected value of zero or less. Incontrast, the H, tested by the Monte Carlo permutation method assertsthat the rule’s output values (+1 and -1) are randomly paired? with fu-ture market price changes. In other words, it asserts that the rule’s out-put is uninformative noise that could have just as easily been generatedby a roulette wheel.
Because the bootstrap and the Monte Carlo methods test somewhatdifferent versions of the null hypothesis, they require different data. Thebootstrap utilizes a daily history of rule returns. The Monte Carlo uses adaily history of the rule’s output values (i.e., a sequence of +1 and —1’s)and a daily history of price changes for the market being traded.
The two methods also use different random sampling methods. Thebootstrap uses a randomization method called resampling with replace-ment, whereas the Monte Carlo randomly pairs rule output values withmarket returns without replacement. This distinction will be clarified inthe description of each method’s algorithm.
Because of these differences, the methods generate somewhat differ-ent sampling distributions. Therefore, it is possible that the conclusion