that the value of thepopulation parameter F-G was contained within the range 0.40 to 0.65. Italso allowed us to conclude, although with somewhat diminished confi-dence, that F-G was more accurately pinned in the range between 0.50and 0.60. It was the degree of dispersion in f-g’s sampling distribution thatpermitted these statements to be made about the value of the populationparameter F-G.
The dispersion of the sampling distribution quantifies the uncertaintyof our knowledge about the population parameter F-G. The distribution’scentral tendency conveys information about the most likely value of F-G,approximately 0.55. Knowing this is nice, but it is not enough. It is also im-portant to know about the reliability of the value 0.55. In other words,how accurately does the sampling distribution’s central tendency of 0.55portray the true value of F-G?
The reliability (degree of certainty) is conveyed by the dispersion ofthe sampling distribution. The greater the distribution’s dispersion aroundits central value, 0.55, the less certain we can be that 0.55 accurately in-forms us about the true value F-G, the proportion of grey beads in the en-tire box.
To illustrate this point consider the two following sampling distribu-tions. Both have central values of 0.55 but their dispersions are quite dif-ferent. The first shown in Figure 4.27 is a narrow dispersion therebydelivering a strong impression that F-G is in the vicinity of 0.55. The distri-bution in Figure 4.28 is wider thus conveying a less certain message aboutF-G value. It is saying that the true value F-G may be considerably differ-ent from the central value of the sampling distribution.
In summary, certainty is directly related to the dispersion of the sam-pling distribution. And the certainty of a conclusion about a populationparameter depends upon the width of the statistic’s sampling distribu-tion—greater width means greater uncertainty.
Suppose it is hypothesized that a rule’s expected return is equal tozero. Also suppose the rule’s back-tested return turns out to be greaterthan zero. Is the positive return sufficient evidence to conclude that thehypothesized value of zero is false? The answer will depend on how farabove zero the back-tested return is relative to the width of the sam-pling distribution. If the relative deviation is large the hypothesis can berejected. Quantifying the relative deviation and the reasoning behind aStatistical Analysis
 
°N
 
0.18
 
0.16
 
0.14
 
0.12
 
0.1
 
0.08
 
0.06
 
0.04
 
0.02
 
 
 
 
~NZmMCOMmMmAmMrM<—-AHAPrMD
0.1 0.2 03 04 05 06 07 08 09 1
fg
FIGURE 4.27 Relative frequency distribution: f-g. (sample size = 20, number of
samples=50).
 
°N
 
0.18
 
0.16
 
0.14
 
0.12
 
0.1
 
0.08
 
0.06
 
0.04
 
0.02
 
 
 
<~<NZmCHPMmATMDM<-AYTMD”
0
 
0.1 02 03 04 05 06 07 08 09 1
f-g
FIGURE 4.28 Relative frequency distribution: f-g. (sample size = 20, number of
samples=50).206 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
decision to reject or not to reject the hypothesis will be treated in Chap-ter 5. The important point for now is that the width of the sampling dis-tribution is critically important in answering this question.
Sampling Distribution of Trading Performance
A sampling distribution can be formed for any statistic: the average(mean), the median, the standard deviation, and many other statisticsused in statistical inference.”’ Each has its own sampling distribution.
This discussion here is confined to the sampling distribution of themean because the performance statistic used to evaluate TA rules in thisbook is a rule’s mean rate of return. However, there are many other perfor-mance measures that might be used: the Sharpe ratio,” the profit factor,*!the mean return divided by the Ulcer Index,” and so forth. It should bepointed out that the sampling distributions of these alternative performancestatistics would be different from the sampling distribution of the mean.
It should also be pointed out that the methods used in this book togenerate the sampling distribution of the mean may be of limited value ingenerating sampling distributions for performance statistics with elon-gated right tails. This can occur with performance statistics that involveratios such as the Sharpe ratio, the mean-return-to-Ulcer-Index ratio, andthe profit factor. However, this problem can be mitigated by taking the logof the ratio to shorten the right tail.
The Three Distributions of Statistical Inference
Statistical inference actually involves three different distributions, one ofwhich is the sampling distribution. They are easy to confuse. This sectionis intended to clarify their differences. In the context of rule testing thethree distributions are:
1. Data distribution of the population: an infinite sized distributioncomprised of all possible daily rule returns, which we assume extendsinto the immediate practical future.
2. Data distribution of the sample: a distribution comprised of a finitenumber (N) of daily rule returns from the past.
3. Sampling distribution: an infinite-sized distribution of the samplestatistic—in this case it is the rule’s mean return. It represents themean returns of the rule, if it were to be tested in an infinite numberof random samples of size N extracted from the population.
There are two points to emphasize. First, the observations comprisingthe data distribution of the population and the data distribution of theStatistical Analysis 207
sample (1 and 2 in the preceding list) are comprised of single-day rule re-turns. In contrast, the observations comprising the sampling distributionare sample statistics in which each observation represents the mean re-turn of the rule computed over a sample of days. Second, both the popula-tion distribution and the sampling distribution are theoretical in the sensethat they refer to an infinite number of possible observations. In contrast,the data distribution of the sample is composed of a finite number of ob-servations from a historical back test.
The relationship between these three distributions can be visualizedby imagining an experiment similar to the box-of-beads experiment. Inthis case, however, you are to imagine a population distribution of dailyrule returns that is infinite in size. Imagine taking 50 independent samplesfrom the population, where each sample is comprised of a substantialnumber of daily rule returns. Next, determine the mean return for eachsample and then plot the sampling distribution of the 50 means. This is il-lustrated in Figure 4.29.
The Real World: The Problem of One Sample
The preceding discussion was theoretical in the sense that we imaginedwhat it would be like if we could observe 50 independent samples from a
 
Parentistration Population Meanin thePopulation(daily returns)
 
 
 
   
| —Sample #1 of n Sample #2 ofn @ @ @ @ @ Sample #50 of nData Daily returns _ Daily returns Daily Returns
Distribution : :in the ib lily is!Sampleof Size n aes(daily returns) eoceevSamplingDistribution50Sample Means Distribution of the Meansof Sample Size n
 
 
 
           
 
 
 
 
 
FIGURE 4.29 The three distributions of statistical inference.208 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
PopulationMean(not known)
       
Distribution of Daily ReturnsComprising the Population
 
 
Daily Returns
 
One Value for SampleMean(known)
Distribution of Daily ReturnsComprising the Sample
 
 
 
Daily Returns
 
‘IGURE 4.30 > The real world: one sample and one value of the test statistic.
parent population. In most real-world statistical problems, there is onlyone sample of observations and thus one value for the sample mean. Theproblem is that with only one sample mean available we have no notion ofthe sample statistic’s variability. The problem of one sample and one meanis illustrated in Figure 4.30.
Fortunately, we can learn a lot about a sample statistic’s variabilityand its sampling distribution by looking at the single available sample.The discovery of how to do this, early in the twentieth century, is whatmade the field of statistical inference possible. In fact, statisticians havedeveloped two distinct approaches for estimating the sampling distribu-tion of a statistic from a single sample of observations: classical and com-puter intensive. Although both will be discussed, the computer-intensiveapproach will be used for the rule tests performed in Part Two.
DERIVING THE SAMPLING DISTRIBUTION:THE CLASSICAL APPROACH
 
This classical approach is the one most often taught in basic statisticscourses. It is attributed to the two fathers of mathematical statistics, SirRonald Fisher (1890-1962) and Jerzy Neyman (1894-1981). It utilizesprobability theory and integral calculus to derive the sampling distribu-tion on the basis of a single observed sample. It provides estimates of thesampling distribution’s dispersion, its mean, and its basic shape (normal).Statistical Analysis 209
In other words, it provides everything needed to quantify the reliability ofan inference based on a single sample of observations.
The Sampling Distribution of the Sample Mean
Every statistic has its own sampling distribution. This discussion will fo-cus on the sampling distribution of the mean (average), because it is thestatistic used in Part Two to evaluate TA rules.
Classical statistical theory tells us several things about the samplingdistribution of the mean.
1. The mean of a large sample is a good estimate of the mean of the pop-ulation distribution from which the sample was obtained. The largerthe sample, the more closely the sample mean conforms to the popu-lation mean.
2. The dispersion of the sampling distribution depends on the size of thesample. For a given population, the larger the sample, the narrowerthe sampling distribution.
3. The dispersion of the sampling distribution also depends on theamount of variation within the parent population data. The bigger thevariation in the parent population, the wider the sampling distribution.
4. Under the conditions that apply to the evaluation of TA rules, theshape of the sampling distribution of the mean will tend towards theso-called normal or bell shape, with the conformity to the normalshape increasing as the sample size increases. Other sample statistics,such as ratios, can be profoundly nonnormal.
Each of these concepts is now explained.
The Mean of a Large Sample Is a Good Estimate of the Popu-lation Mean. This is a manifestation of The Law of Large Numbers,which we met previously in the context of a coin-flipping experiment. Torecap, The Law of Large Numbers tells us that the larger the number ofobservations comprising the sample, the more closely the sample meanwill approximate the population mean. There are some qualifying condi-tions* attached to this promise, but they need not concern us here.
The graph of the coin-flipping experiment illustrated the action of theLaw of Large Numbers (Figure 4.18). As the number of coin tosses in-creased, the proportion of heads gradually converged to its theoreticallycorrect value of 0.50. In the early stages of the experiment, when the sam-ple size was small, there were large departures from 0.50. These depar-tures illustrate the large role that chance plays in small samples. In four210 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
tosses, though values of 0.75 or 0.25 are not the most probable, they arequite common. However, when the number of coin tosses reaches 60, theprobability of 0.75 or 0.25 is less than 1 in 1,000. The important lesson: In-creased sample size diminishes the role of chance.
The Dispersion of the Sampling Distribution Depends on theSize of the Sample: the Larger the Sample the Less the Dis-persion. Imagine a very large population of observations on some vari-able whose standard deviation is 100. Usually, we do not know thepopulation standard deviation, but in this case we will assume that we do.
In Figure 4.31, I show the parent population distribution at the bot-tom. Above it are two sampling distributions for the variable’s mean forsample sizes of 10 and 100. Notice that the width of the sampling distribu-tion is cut by approximately one-third each time the sample size is in-creased by a factor of 10. One of the lessons from classical statistics isthat the standard deviation of the sampling distribution of the mean is in-versely proportional to the square root of the sample size.*° Thus, if thesample size is increased by a factor of 10, the width of the sampling distri-bution is reduced a factor of 3.16, the square root of 10. The parent distrib-
 
Sampling Distributionof the MeanSample Size = 100Standard Error = 1010
Sampling Distributionof The MeanSample Size = 10St. Error = 31.62
 
 
 
Distribution of Datain the PopulationSt. Deviation =100
 
 
100
FIGURE 4.31 > The narrowing of the sampling distribution with increased sam-ple size.Statistical Analysis 2t1
ution can be thought of as a sampling distribution with a sample size ofone (one mean—the population mean). The sampling distribution for asample size of 100 is one-tenth the width of the parent population. The es-sential message is the larger the sample size, the less the uncertaintyabout the value of a sample statistic. Incidentally, the standard deviationof the sampling distribution of the mean is given a special name in statis-tics, the standard error of the mean (see later explanation).
With some qualifications,”* the Law of Large Numbers tells us that thelarger the sample size, the more accurate our knowledge of the popula-tion’s average. With some caveats,” therefore, we would like to work withsamples that are as large as possible so as to minimize the width of thesampling distribution as much as possible.
The Dispersion of the Sampling Distribution of the Mean AlsoDepends On the Amount of Variation within the Data Compris-ing the Parent Population. There is another factor that impacts thewidth of the sampling distribution—the amount of variation within theparent population from which the sample was taken. The greater the vari-ation within the population data, the greater will be the variation (disper-sion) of the sampling distribution.
You can visualize this by thinking about a population that has no vari-ation at all. For example, if all people in the population weighed 150 lbs,every sample is going to have an average weight of 150 lbs. Therefore,zero variation among the members of the population would lead to zerovariation in the sample average, and consequently a sampling distributionthat has no dispersion whatsoever. Conversely, if there is great variationamong the individuals comprising the population, there is going to begreater variation in the sample averages, which translates to a fat sam-pling distribution.
Shape of the Sampling Distribution Tends Toward Normal.The Central Limit Theorem, a foundational principle of classical statistics,states that as the size of a sample gets larger, the sampling distribution ofthe mean, with some qualifications,** converges toward a specific shape ir-respective of the shape of the population distribution. In other words, nomatter how weirdly shaped the distribution of the data in the parent popu-lation, the shape of the sampling distribution approaches a specific shapethat statisticians refer to as the normal distribution, also known as theGaussian distribution.
The normal distribution is the shape of the most common probabil-ity distribution. Often referred to as the bell curve because it has the sil-houette of the Liberty Bell, the normal distribution characterizescontinuous variables that describe many real-world phenomena.212 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
There are a few points about the normal distribution that should beunderstood:
e It is completely described by its mean and its standard deviation. Ifthese two facts are known, you know every thing that there is to knowabout the distribution.
e About 68 percent of the observations lie within one standard devia-tion of the mean, and about 95 percent lie within two standard devia-tions of the mean.
¢ The tails of the distribution become quite thin beyond two standarddeviations. Thus values beyond three standard deviations are rare andbeyond four are extremely rare.
The normal distribution, which is illustrated in Figure 4.32, is so com-mon that it has been called a fundamental feature of the natural world. Itsshape is caused by the additive effects of many independent factors actingon the same situation. For example, systolic blood pressure is affected bygenetic factors, diet, weight, lifestyle, aerobic conditioning, and a multi-tude of other factors. When they interact, the probability distribution ofblood pressures in a large group of randomly selected people will have thebell-shaped distribution.
Figure 4.33 shows several very nonnormal population distributionsalong with the shape of the sampling distribution of the mean. Note howthe sampling distribution converges to a normal shape as a sample size isincreased irrespective of the shape of the distribution of the data in the
 
 
 
 
Mean
 
 
20 10 +16 +20u
       
 
 
68% of the Area
 
 
 
95% of the Area
FIGURE 4.32 The normal distribution.Statistical Analysis 213
Parent Sampling Sampling SamplingPopulation Distribution: n=2 Distribution: n=5 Distribution: n=30
~ A~A -
 
 
 
crik:Reee
FIGURE 4.33 > Given sufficient sample size the sampling distribution of themean approaches normal shape irrespective of the variable’s distributional shape.
parent population. There is nothing magical about the number 30. The rateat which the sampling distribution converges to a normal shape as a func-tion of sample size depends on the shape of the parent distribution. Notethat for three of the cases in the figure, a sample size of five produces anearly normally shaped sampling distribution of the mean. However, forthe bottom case in the figure, the sampling distribution of the mean doesnot become normal-like until a sample size of 30.
The Standard Error of the Mean
We are now in a position to take the final step toward defining the sam-pling distribution of the mean as it is done in classical statistics. So far, ithas been established that:
1. The sampling distribution of the mean converges to the shape of anormal distribution as the sample size is increased.
2. A normal distribution is fully described by its mean and its standarddeviation.214 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
3. The standard deviation of the sampling distribution of the mean, alsoknown as the standard error of the mean, is directly related to thestandard deviation of the population from which the samples weretaken.
4. The standard error of the mean is inversely related to the square rootof the sample size.
The standard error of the mean is equal to the standard deviation ofthe population divided by the square root of the sample size. This is a truestatement but not of practical use because the standard deviation of thepopulation is not known. This is simply a consequence of the fact that thefull population is not observable. However, the standard deviation of thepopulation can be estimated from the sample. This estimate, which is des-ignated by sigma-hat, is shown in equation in Figure 4.34. The little roof orhat over the sigma is statistical symbolism for a quantity that has been es-timated. You will note that the divisor within the radical sign is n —- 1,rather than n. This modification compensates for the fact that the samplestandard deviation understates the standard deviation of the population.
This expression is used to estimate the standard deviation of the par-ent population, which is then used in the equation in Figure 4.35 to esti-mate the standard error of the mean.
With the standard error of the mean in hand, and the assumption thatthe sampling distribution is normal, it is now possible to form a good esti-
Estimate of Population Standard DeviationBased on a Sample of Observations
 
Where:
Is an individual observation on variable X in the population
XjX Is the sample mean on variable X
N's number of observations in the sample
FIGURE 4.34 Estimate of population standard deviation based on a sample ofobservations.Statistical Analysis 215
Standard GError ~ —“of |the Mean a
FIGURE 4.35 The standard error of the mean.
mate of the sampling distribution of the mean. And recall all this was onthe basis of a single sample.
There is one problem with the traditional approach. It assumes thatthe sampling distribution is normally shaped (bell curve). If this provesnot to be so, the conclusions reached will be inaccurate. For this reason, Ihave chosen to use an alternative method for estimating the sampling dis-tribution of the mean, based on computer simulation.
DERIVING THE SAMPLING DISTRIBUTION WITHTHE COMPUTER-INTENSIVE APPROACH
 
Until about 30 years ago there was no alternative to the classical approachfor deriving the sampling distribution from a single sample. Now there is—the computer-intensive approach. By systematically resampling the singleavailable sample many times, it is possible to approximate the shape of thesampling distribution. This procedure is described in this section.
The next chapter will present two computer-based methods: boot-strap resampling and Monte Carlo permutation. Although each methodhas its limitations, as is true for any statistical method, they are useful al-ternatives to the classic approach.
The bootstrap is so named because it seemingly lifts itself up, as if byits own bootstraps, to approximate the shape of the sampling distribu-tion. In this book, it will be used to approximate the shape of the sam-pling distribution of the average return for rules that have no predictivepower. The Monte Carlo permutation method is named after the famedEuropean gambling casino because it uses the computer as if it were aroulette wheel.
By whatever means it is generated, the sampling distribution servesthe same purpose. It tells us how much the back-tested return of a non-predictive rule can vary above or below zero due to the random effects ofsampling. Thus, it serves as the benchmark for evaluating rule perfor-mance. If the rule’s mean return is too high to be due to sampling variabil-ity, the rule is judged to have predictive power.216 > METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
PREVIEW OF NEXT CHAPTER
 
We have already made a number of references to using statistical inferenceto test claims. The official title for this activity is hypothesis testing. Thenext chapter discusses the rationale and mechanics of hypothesis testing.
It will also consider another use of statistical inference, estimatingconfidence intervals. A confidence interval is a range that contains thetrue value of the population parameter with a specified level of confi-dence. For example, if the observed mean return of a rule is 15 percentthe 95 percent confidence interval might indicate that the rule’s true re-turn lies between 5 percent return and 25 percent return. The width of theconfidence interval, in this case 20 percentage points, is derived from thesame sampling distribution that is used for hypothesis testing.Hypothesis Tests
and ConfidenceIntervals
 
 
TWO TYPES OF STATISTICAL INFERENCE
 
Statistical inference encompasses two procedures: hypothesis testing andparameter estimation. Both are concerned with the unknown value of apopulation parameter. A hypothesis test determines if a sample of data isconsistent with or contradicts a hypothesis about the value of a popula-tion parameter, for example, the hypothesis that its value is less than orequal to zero. The other inference procedure, parameter estimation, usesthe information in a sample to determine the approximate value of a pop-ulation parameter.' Thus, a hypothesis test tells us if an effect is present ornot, whereas an estimate tells us about the size of an effect.
In some ways both forms of inference are similar. Both attempt todraw a conclusion about an entire population based only on what hasbeen observed in a sample drawn from the population. In going beyondwhat is known, both hypothesis testing and parameter estimation take theinductive leap from the certain value of a sample statistic to the uncertainvalue of a population parameter. As such, both are subject to error.
However, important differences distinguish parameter estimationfrom hypothesis testing. Their goals are different. The hypothesis testevaluates the veracity of a conjecture about a population parameterleading to an acceptance or rejection of that conjecture. In contrast, es-timation is aimed at providing a plausible value or range of values forthe population parameter. In this sense, estimation is a bolder endeavorand offers potentially more useful information. Rather than merelytelling us whether we should accept or reject a specific claim such as a
217218 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
rule’s average return is less than or equal to zero, estimation approxi-mates the average return and provides a range of values within whichthe rule’s true rate of return should lie at a specified level of probability.For example, it may tell us that the rule’s estimated return is 10 percentand there is a 95 percent probability that it falls within the range of 5percent to 15 percent. This statement contains two kinds of estimates; apoint estimate, that the rule’s return is 10 percent, and an interval esti-mate, that the return lies in the range 5 percent to 15 percent. The rulestudies discussed in Part Two use estimation as an adjunct to the hy-pothesis tests.
HYPOTHESIS TESTS VERSUS INFORMAL INFERENCE
 
If a rule has been profitable in a sample of historical data, this sample sta-tistic is an indisputable fact. However, from this fact, what can be inferredabout the rule’s future performance? Is it likely to be profitable because itpossesses genuine predictive power or are profits unlikely because itspast profits were due to chance? The hypothesis test is a formal and rigor-ous inference procedure for deciding which of these alternatives is morelikely to be correct, and so can help us decide if it would be rational to usethe rule for actual trading in the future.
Confirmatory Evidence: It’s Nice, It’s Necessary,but It Ain’t Sufficient
Chapter 2 pointed out that informal inference is biased in favor of confir-matory evidence. That is to say, when we use common sense to test thevalidity of an idea, we tend to look for confirmatory evidence—facts con-sistent with the idea’s truth. At the same time, we tend to ignore or givetoo little weight to contradictory evidence. Common sense tells us, andrightly so, that if the idea is true, instances where the idea worked (confir-matory evidence) should exist. However, informal inference makes themistake of assuming that confirmatory evidence is sufficient to establishits truth. This is a logical error. Confirmatory evidence does not compelthe conclusion that the idea is true. Because it is consistent with the idea’struth, it merely allows for the possibility that the idea is true.
The crucial distinction between necessary evidence and sufficient evi-dence was illustrated in Chapter 4 with in the following example. Supposewe wish to test the truth of the assertion: The creature I observe is a dog. Weobserve that the creature has four legs (the evidence). This evidence is con-sistent with (i.e., confirmatory of) the creature being a dog. In other words, ifHypothesis Tests and Confidence Intervals 219
the creature is a dog, then it will necessarily have four legs. However, fourlegs are not sufficient evidence to establish that the creature is a dog. It mayvery well be another four-legged creature (cat, rhino, and so forth).
Popular articles on TA will often try to argue that a pattern has predic-tive power by presenting instances where the pattern made successfulpredictions. It is true that, if the pattern has predictive power, then therewill be historical cases where the pattern gave successful predictions.However, such confirmatory evidence, while necessary, is not sufficient tologically establish that the pattern has predictive power. It is no more ableto compel the conclusion that the pattern has predictive power than thepresence of four legs is able to compel the conclusion that the creatureis a dog.
To argue that confirmatory instances are sufficient commits the fal-lacy of affirming the consequent.
If p is true, then q is true.q is true.Invalid Conclusion: Therefore, p is true.
If the pattern has predictive power, then past examples of successshould exist.
Past examples of success exist, and here they are.Therefore, the pattern has predictive power.
Chapter 3 showed that although confirmatory evidence is not suffi-cient to prove the truth of an assertion, contradictory evidence—evidencethat is incompatible with an assertion’s truth—is sufficient to establishthat the assertion is false. The fact that a creature lacks four legs is suffi-cient to falsify the assertion that the creature is a dog. This is the validform of argument called denial of the consequent.
If p is true, then q is true.
q is not true.
Therefore, p is not true (i.e., p is false).
If the creature is a dog, then the creature has four legs.Creature does not have for legs.
Therefore, creature is not a dog.
The logical basis of the hypothesis test is falsification of the conse-quent. As such, it is a potent antidote to the confirmation bias of informalinference and an effective preventative of erroneous belief.220 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
What Is a Statistical Hypothesis?
A statistical hypothesis is a conjecture about the value of a population pa-rameter. Often this is a numerical characteristic, such as the average re-turn of a rule. The population parameter’s value is unknown because it isunobservable. For reasons previously discussed, it is assumed to havevalue equal to or less than zero.
What an observer does know is the value of a sample statistic for asample that has been drawn from the population. Thus, the observer isfaced with a question: Is the observed value of the sample statistic consis-tent with the hypothesized value of the population parameter? If the ob-served value is close to the hypothesized value, the reasonable inferencewould be that the hypothesis is correct. If, on the other hand, the value ofthe sample value is far away from the hypothesized value, the truth of thehypothesis is called into question.
Close and far are ambiguous terms. The hypothesis test quantifiesthese terms making it possible to a draw a conclusion about the veracityof the hypothesis. The test’s conclusion is typically given as a number be-tween 0 and 1.0. This number indicates the probability that the observedvalue of the sample statistic could have occurred by chance under thecondition that (given that or assuming that) the hypothesized value istrue. For example, suppose it is hypothesized that a rule’s expected returnis equal to zero, but the back test produced a return of +20 percent. Theconclusion of the hypothesis test may say something like the following: Ifthe rule’s expected rate of return were truly equal to zero, there is a 0.03probability that the back-tested return could be equal to or greater than+20 percent due to chance. Because there is only a 3 percent probabilitythat a 20 percent return could have occurred by chance if the rule weretruly devoid of predictive power, then we can be quite confident that therule was not simply lucky in the back test.
Falsifying a Hypothesis with Improbable Evidence
A hypothesis test begins by assuming that the hypothesis being tested istrue. Based on this assumption, predictions are deduced from the hypoth-esis about the likelihood of various new observations. In other words, ifthe hypothesis is true, then certain outcomes would be probable to occurwhereas other outcomes would be improbable. Armed with this set of ex-pectations an observer is in a position to compare the predictions withsubsequent observations. If predictions and observations agree, there isno reason to question the hypothesis. However, if low probability out-comes are observed—outcomes that would be inconsistent with the truthof the hypothesis—the hypothesis is deemed falsified. Thus, it is the oc-Hypothesis Tests and Confidence Intervals 221
currence of unexpected evidence that is the basis for refuting a