universe of observations. In the typical statistical inference problem, wewant to learn some fact about the population, but it is either impracticalor impossible to observe the population in its entirety. In the box-of-beadsexperiments, the population consisted of the set of beads filling a box.The color was the random variable. With respect to testing a TA rule, thepopulation consists of all conceivable daily returns!’ that would be earnedby the rule’s signal over the immediate practical future.
To what does the term immediate practical future refer? It would beunreasonable to assume that the dynamics of financial markets are sta-tionary, and so it would be unreasonable to expect that the profitability ofStatistical Analysis 187
a rule will endure in perpetuity. For this reason, the population with re-spect to TA rules cannot refer to returns occurring over an infinite future.
A more reasonable notion is what I refer to as the immediate practi-cal future. The immediate practical future refers to a finite future time pe-riod, over which it would be reasonable to expect a useful rule’sprofitability to persist, even though markets are nonstationary. Any en-deavor to find predictive patterns must make some assumption about thecontinuity of predictive power. In other words, unless one is willing to as-sume some persistence in predictive power, all forms of TA are pointless.The assumption being made here is that a rule will continue to work longenough to compensate the researcher for the effort of discovering it. Thisis consistent with the position taken by Grossman and Stiglitz in “On theimpossibility of informationally efficient markets.”" It is also consistentwith the idea that profitable rules signal opportunities to earn a risk pre-mium," a topic discussed in Chapter 7.
Therefore, the immediate practical future refers to all possible ran-dom realizations of market behavior over a finite future. It is as if therewere an infinite number of parallel universes, where all universes are anexact duplicate except for the random component of the market’s behav-ior. In each realization, or universe, the pattern that accounts for the rule’sprofitability is the same, but the random component of the market is dif-ferent. This idea is illustrated in Figure 4.12.
Infinite NumberofParallel Universes
 
 
   
 
 
Sell a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Buy>a Immediate TimeHistorical Practical FutureSamplefor Back-Testing Now
 
 
 
FIGURE 4.12 Different realizations of randomness in an infinitude of paralleluniverses.188 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
The Sample. The sample is a subset of the population that is availablefor observation. In the box-of-beads case, we were able to observe 50 in-dependent samples. In the case of a TA rule, we typically observe a singlesample of the rule’s performance by back testing it over a segment of mar-ket history. This sample is composed of a sequence of daily returns gener-ated by the rule’s signals. The sequence of returns is reduced to a singlenumber, a sample performance statistic.
The Population Parameter. The population parameter is a fact orcharacteristic about the population that we would like to know. It is typi-cally numerical, but it need not be.
Unfortunately, the population parameter is unknown because the en-tire population cannot be observed. The essence of statistical inference isthe attempt to increase our knowledge about a population parameter, de-spite the impossibility of observing the population in its entirety. In thebox-of-beads case, the population parameter of interest was the fractionof grey beads in the box (i.e., F-G, and F-G 2). In the case of a TA rule, thepopulation parameter is the rule’s expected performance over the immedi-ate practical future. Performance can be defined in many ways. Somecommon measures include average rate of return, Sharpe ratio, profit fac-tor, and so forth. In this book, our measure of performance is the annual-ized average daily return on detrended market (zero-centered) data.
In many statistical problems, it is safe to assume the population para-meter never changes (i.e., it is a constant). In the box-of-beads experi-ment, the proportion of grey beads remained constant. Statisticalproblems in which the population parameter remains fixed are said to bestationary.'° Now imagine a wrinkle on that situation. Suppose, unbe-knownst to the experimenter, an invisible demon secretly removes oradds grey beads between samplings. Now the population parameter, F-G,is unstable, or nonstationary.”"”
Earlier in this chapter, I said that it is best to start with an assumptionthat any rule we test has no predictive power. That is, we assume that ithas an expected return equal to zero. Or, to put it in statistical terms, thepopulation parameter is assumed equal to be zero.
The Sample Statistic. A sample statistic is a measurable attribute ofa sample.'* Its value is known because it has been observed. In this book,the term sample statistic is restricted to numerical facts, for example aproportion, a percentage, a standard deviation, an average rate of return, atrimmed average,'* a Sharpe ratio, and so forth. In a statistical inferenceproblem, the sample statistic typically refers to the same measurable at-tribute as the population parameter. In the case of the box-of-beads, thesample statistic, f-g, referred to the fraction of grey beads in an individualStatistical Analysis 189
sample. The population parameter, F-G, referred to the fraction of greybeads in the whole box.
The bottom line is this: A sample statistic is important because itsheds light on the population parameter. Beyond this, it is a fact of no par-ticular importance. Some market historians seem to be unmindful of thisessential truth.
If a back test should result in a positive rate of return, it raises the fol-lowing question: Is the positive performance a random deviation abovezero due to sampling variability or is it attributable to the rule’s predictivepower (i.e., the rule has an expected return greater than zero)? Answeringthis question requires the tools of statistical inference. This is illustratedin Figure 4.13.
An Inference. Statistical inference is the inductive leap from the ob-served value of a sample statistic, which is known with certainty butwhich is true only for a specific sample of data, to the value of a popula-tion parameter, which is uncertain but which is deemed to hold true for awide, perhaps infinite, number of unobserved cases.
When a rule’s positive past performance can reasonably be attributedto sampling variability, the reasonable inference is that its expected returnover the immediate practical future is zero or less. However, if the positiveperformance is too high to be reasonably attributed to sampling variability(luck), the reasonable inference would be that the rule possesses genuinepredictive power and has a positive expected return over the immediatepractical future.
  
ExpectedReturn2222
    
 
      
Sample StatisticAverage Return = +10%
Immediate
Observable Past Practical Future
 
 
 
 
TimeDate of Date ofEarliest Most RecentData Data
 
 
 
 
FIGURE 4.13 Is the parameter’s value greater than zero?190 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
A Statement about the Reliability of the Inference. Because asample can not represent the population perfectly, inferences based on asample statistic are subject to uncertainty. In other words, it is possiblethat the inference is wrong. We have already seen how sample statisticsvary randomly around the true value of the population parameter value.The science of statistics goes beyond simply admitting that its inferencesare uncertain. It quantifies their reliability. This makes statistical conclu-sions considerably more useful than those reached by informal methodsthat do not provide such information.
The inference can be wrong in two ways. One error is concluding therule has predictive power when it does not. This is a case of good luckshining on a worthless rule. This type of mistake translates into takingmarket risks that will not be compensated. The other error is concludingthe rule has no predictive ability when it really does. This mistake resultsin missed trading opportunities.
DESCRIPTIVE STATISTICS
 
The field of statistics subdivides into two main areas: descriptive and in-ferential. The most important for TA is statistical inference, which wasdiscussed in the preceding sections. However, before making an infer-ence, we must describe the sample data in a succinct and informative way.The tools of descriptive statistics serve this purpose.
The goal of descriptive statistics is data reduction, that is, reducing alarge set of observed values to a smaller, more intelligible set of numbersand plots. Descriptive statistics tell the story of the forest rather than theindividual trees. Three descriptive tools will be important for the workthat lies ahead: (1) frequency distributions, (2) measures of central ten-dency, and (3) measures of variation.
Frequency Distributions
Frequency distributions have already been discussed in connection withthe box-of-beads experiments. They were used to reduce a set of 50 obser-vations on random variable f-g into an informative plot.
With only 50 observations, one might be able to form an overall im-pression of the data simply by looking at a table of numbers. Then again, ifthe number of observations were 500 or 5,000, a table would not be as in-formative as a frequency distribution.
Plotting a frequency distribution is usually the first step in analyzing aStatistical Analysis 191
set of data. It provides a quick visual impression of two key features of asample of observations: the central tendency (e.g., average value) and thedegree of dispersion or variation about the central value. For example,Figure 4.11 showed that the two boxes of beads had different central val-ues and different degrees of dispersion.
A visual impression of a sample is useful, but quantification improvesmatters. For our purposes, we will need to quantify two features of thefrequency distribution: its central tendency and its variability or disper-sion about its central tendency.
Statistics That Measure Central Tendency
There are many measures of central tendency. Three of the most commonare the average, the median, and the mode. The average, also known asthe arithmetic mean, is used in many TA applications and is the summarystatistic used in this book. It is the sum of the observed values divided bythe number of observations.
It is important to distinguish the population mean from the samplemean. The sample mean is a statistic, a known quantity that is com-puted from observed values and varies randomly from one sample tothe next. The population mean, in contrast, is unknown and does notvary in stationary problems. The formula for the mean of a sample isgiven in Figure 4.14 in two forms.
Sample Mean for Variable X
= X, +X> +X3 +. . Xx= ] 23 n
 
Where x; is an individual observation on variable X
FIGURE 4.14 Sample mean for variable X.192 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Statistical Measures of Variability (Dispersion)
Measures of variability describe the degree to which the observations in asample are dispersed about their central tendency. In other words, mea-sures of variability quantify the width of a frequency distribution.
Among the widely used measures of dispersion are the variance, themean absolute deviation, and the standard deviation. They are importantin the traditional approach to statistics (i.e., classical statistics). The stan-dard deviation is the square root of the average squared deviation of eachobservation from the mean of the data. The formula for the standard devi-ation of a sample is shown in Figure 4.15.
An intuitive notion of dispersion is best conveyed with pictures. Fig-ure 4.16 shows a number of idealized frequency distributions with differ-ing degrees of variation and different central tendencies. They areidealized in the sense that the stair-step look that characterizes real fre-quency distributions has been smoothed away. The key point of Figure4.16 is that central tendency and variation (dispersion) are independentcharacteristics of a frequency distribution. In row 1 of Figure 4.16, thefour distributions have the same degree of variation but different centralvalues. In row 2 the distributions have the same central value but differentdegrees of variation. In row 3 all distributions have different central val-ues and different degrees of variation.
Standard Deviation of a Sample of Observations
» (X, - x)?
   
nWhere:X; Is an individual observation on variable XXxX Is the sample mean on variable X
n Is number of observations in the sample
FIGURE 4.13 Standard deviation of a sample of observations.Statistical Analysis 193
 
 
Same Degree of Dispersion but Different Means
 
 
0 5 10 15
 
 
 
Equal Means but Different Degrees of Dispersion
 
 
 
Different Means and Different Degrees of Dispersion
ge ee
FIGURE 4.16 Central tendency and dispersion are distinct attributes ofa distribution.
 
 
 
PROBABILITY
 
The notion of probability is important in statistics because it is used toquantify uncertainty. A conclusion reached via statistical inference is un-certain; that is, there is a chance of error. Thus, an inference about thevalue of a population parameter has some chance of being wrong. Thischance is given in terms of a probability.
We routinely use informal notions of probability to form expectationsand make choices. Is the person I find so wonderful now, likely to remainso after marriage? What is the chance I will find gold if I dig in a certainspot? Is it probable I will feel fulfilled pursing a career in law? Should Ibuy that stock, and if I do, how much profit am I likely to make?
The informal usage of probability is connected to a cluster of inter-changeable terms illustrated in Figure 4.17.
For our purposes, something more definitive is required. The defini-tion of probability that makes the most sense for the work that lies aheadis based on the notion of relative frequency. The relative frequency of anevent is the number of times the event actually occurred divided by the to-tal number of opportunities on which the event could have occurred. Rel-ative frequency is given as a fraction between 0 and 1. It could rain every194 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Uncertainty
Risk Chance
 
Probability
 
 
 
Luck Likelihood
Random
FIGURE 4.17 Common notions of probability.
day, but in the month of April the relative frequency of rain has actuallybeen 0.366 (11 out of 30 days). If rain never occurred, then its relative fre-quency would be 0. If rain occurred every day, its relative frequency is 1.0.Thus, the relative frequency of an outcome is equal to:
Number of occurrences of an eventNumber of possible opportunities for the event to occur
 
In the box-of-beads experiment the value of f-g was measured 50times. This was the maximum number of opportunities for any specific f-gvalue to occur. In fact, the value f-g = 0.65 occurred on 5 out of 50. There-fore, it had a relative frequency of 5/50 or 0.10.
Probability is the relative frequency of an event over the long run—the very long run. That is to say, the probability of an event is its relativefrequency of occurrence given an infinite number of opportunities for itsoccurrence. Probability is stated as a number that lies in the range of 0 to1 inclusive. A probability value of 0 means the event never occurs,whereas a value of 1.0 means it always occurs.
Probability is a theoretical notion because it is never possible to ob-serve an infinite number of anything. For practical purposes, however,when the number of observations becomes very large, relative frequencyapproaches theoretical probability.
The Law of Large Numbers
The tendency of relative frequencies to converge to theoretical probabili-ties, as the number of observations becomes large, is called the Law ofLarge Numbers.” The operation of the Law of Large Numbers can be illus-trated with coin tossing. The possible outcomes (events) are heads or tailsand the probability of heads is known to be 0.50. The Law tells us that theStatistical Analysis 195
relative frequency of heads will converge to its theoretical value, 0.5, asthe number of coin tosses becomes large. However, even for a large num-ber of tosses, a departure from 0.50 can still occur, though the likely sizeof the departures decreases as the number of tosses increases. For smallsamples, however, The Law of Large Numbers warns us that the relativefrequency of heads can differ substantially from 0.50. When the number oftosses is only three, a value of 1.0 can easily occur—three heads in threetosses.
Figure 4.18 shows the fraction of heads in a coin-toss experiment asthe number of tosses grows from one to one thousand. When the numberof tosses (observations) is less than 10, the fraction of heads experiencedtwo large deviations from 0.50. At 3 tosses the fraction of heads reached0.66 (2 out of 3). At 8, the fraction was 0.375 (3 out of 8). However, as thenumber of tosses increased the random variable, fraction heads, experi-enced progressively smaller deviations from the expected value of 0.50.This is exactly what the Law of Large Numbers predicts.
Now imagine a naive coin flipper. Upon observing five heads in fivetosses, he loudly proclaims, “I’ve found the holy grail of coins. It alwayscomes up heads.” The Law of Large Numbers says that the poor fellow’soptimism is unwarranted. This is also likely to be true for the TA re-searcher who finds a rule with five historical signals, all of which werecorrect. Optimism is most likely unwarranted. Chapter 6 will show that ifthis rule was selected because it was the best performing rule out of alarge universe of back-tested rules, that is to say, it was discovered by datamining, there would be even less reason to be optimistic about its futureperformance.
1.00.80
0.60
ZO-ANPaAn
0.40
0.20
 
nOrPmr
1 10 100 1,000
Number of Coin Flips (log scale)
FIGURE 4.18 Law of Large Numbers.196 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Theoretical versus Empirical Probability
There are two kinds of probabilities: theoretical and empirical. “Theo-retical probabilities can be determined with a good degree of confi-dence purely on logical grounds. They are derived independent of priorexperience, most often based on arguments of symmetry.””! In otherwords, they can be deduced without any reference to the world of expe-rience. The probability of a head in a coin toss is a theoretical probabil-ity. There is no need to toss it ten thousand times to predict that overthe long run the probability of heads is 0.50. Simply the fact that thecoin is fair, that it has two faces, and that it is unlikely to land on itsedge is sufficient grounds for suspecting heads and tails are equallylikely. The probabilities for being dealt a royal flush in poker, the occur-rence of a six in a dice roll, and the chance of winning Lotto are alsotheoretical because each can be determined by a logical analysis of thesituation.
Empirical probabilities are based on observed frequencies. Techni-cal analysis is concerned with this type of probability. The chance ofnew snowfall on Mount Hood during the month of July or the likelihoodof a rise in the stock market following a 2 percent drop in interest ratesare examples of empirical probabilities. They can only be determinedby observing numerous past instances of the specified conditions(Mount Hood in July), and determining the relative frequency of theevent (new snow).
When determining empirical probabilities, it is vitally importantthat each instance be characterized by the same set of conditions. In TAthis is a practical impossibility. Each past instance of a 2 percent dropin interest rates is similar with respect to that specific condition, butthere are numerous other conditions that may not be the same. For ex-ample, the level of interest rates prior to the 2 percent drop may differfrom observation to observation. In one instance, interest rates were at5 percent prior to the drop whereas in another they were 10 percent. Ofcourse, the level of interest rates could be added to the set of condi-tions that define each instance, but that has the downside of reducingthe number of comparable instances. Moreover, there will always beother conditions that are not part of the specified condition set. The in-ability to control all potentially relevant variables is an unfortunate factof life in nonexperimental/observational sciences. In contrast, the ex-perimental scientist enjoys the supreme advantage of being able to holdconstant all or nearly all relevant variables except the one being investi-gated. Imagine being able to do that in TA! I am told this is how it is inthe afterlife.Statistical Analysis 197
PROBABILITY DISTRIBUTIONS OF RANDOM VARIABLES
 
A probability distribution shows how often we can expect the differentpossible values of a random variable to occur (i.e., their relative frequen-cies). The probability distribution of a random variable is a relative fre-quency distribution built from an infinite number of observations.
The concept of a probability distribution can be understood bythinking of a sequence of relative frequency distributions, each builtfrom increasing numbers of observations and progressively narrowerbins or intervals. Gradually, the relative frequency distribution becomesa probability distribution. With increasing numbers of observations,more discrete intervals of decreasing width can be created.” As thenumber of observations approaches infinity, the number of intervalsapproaches infinity and their widths shrink to zero. Thus, the rela-tive frequency distribution morphs into what is called a probabilitydistribution, more technically referred to as a probability densityfunction.”
Figures 4.19 though 4.23 show how a relative frequency distributionevolves into a probability density function as the number of intervals isincreased and their interval widths are decreased. The height of eachbar represents the relative frequency of events falling within the bar’s in-terval. The figures presume that the random variable is a measure ofprice change.
This succession of diagrams shows that a probability density functionis, in effect, a relative frequency distribution composed of an infinite num-ber of observations whose intervals are infinitely narrow. Here is wherethings get a bit strange. If the intervals of a probability distribution have
RelativeFrequency
Price Change Intervals
FIGURE 4.19 Relative frequency distribution based on five intervals.198 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
RelativeFrequency
 
Price Change Intervals
FIGURE 4.20 Relative frequency distribution based on seven intervals.
RelativeFrequency
 
Price Change Intervals
FIGURE 4.21 Relative frequency distribution based on fifteen intervals.
RelativeFrequency
 
Price Change Intervals
FIGURE 4.22 Relative frequency distribution based on twenty nine intervals.Statistical Analysis 199
ProbabilityDensity
 
No Specific Intervals
FIGURE 4.23 Probability density function.
zero width, then there are zero observations per interval. This seems tomake no sense! However, it is quite common for mathematical conceptsto be at odds with common sense. In geometry, a point has location butoccupies no space (i.e., length, width, and breadth equal zero), a line haslength but zero width, and so forth.
The fact that an interval has zero observations has a strange implica-tion—the probability that any single value of a continuous random vari-able will ever occur is equal to zero. For this reason, it only makes senseto speak of the probability that a random variable will assume a valuewithin a specified range of values. In other words, it makes sense to speakof the probability that a random variable will assume a value betweenspecified minimum and maximum values. It also makes sense to speak ofthe probability that a random variable will assume a value that is equal toor greater than some specified value or less than or equal to a specifiedvalue. For example, we can speak of the probability that a value of 3.0 orgreater will occur. However, it would make no sense to speak of the prob-ability of a value of exactly 3.0.74
This somewhat counterintuitive idea fits nicely with rule testing.When testing the statistical significance of a rule’s past rate of return, wewill be concerned with the probability that a rate of return of +10 percentor higher could have occurred by chance under the condition that the rulehas no predictive power. Probability density functions can provide suchinformation.200 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
RELATIONSHIP BETWEEN PROBABILITY ANDFRACTIONAL AREA OF THE PROBABILITY DISTRIBUTION
 
Let’s recap what has been established up to this point:
1. The probability of an event is its relative frequency of occurrencegiven an infinite number of instances in which the event could haveoccurred.
2. A probability density function is a relative frequency distribution builtfrom an infinite number of observations and intervals of zero width.
3. The relative frequency of a random variable taking on a value within agiven interval is equal to the fractional area of the frequency distribu-tion sitting on top of that interval. See Figure 4.24.
We are now ready to take the final step in this sequence of thinking. Itis analogous to point 3 in the preceding list, except that the term relativeSrequency distribution is replaced with probability density function. Theprobability that a continuous random variable will assume a value withina specified range is equal to the fraction of the probability density func-
 
Bars Represent 0.344 ofDistribution’s Total Area
 
 
0.234
 
RelativeFrequency
0.09]
 
0.019
 
 
 
 
 
 
      
Values of RandomVariable
Relative Frequencyof a Value in This Range = 0.344
 
FIGURE 4.24 The correspondence of relative frequency and fractional area ofa distribution.Statistical Analysis 201
tion encompassed by (sitting above) that range. This concept is illustratedin Figure 4.25. It shows the probability that X, a continuous random vari-able, will assume a value within the range A-B is equal to 0.70.
In many instances, the interval in question refers to the extreme ortail of the distribution. For example, Figure 4.26 shows the probabilitythat random variable X will assume a value of B or greater. The probabil-ity is 0.15.
The probability distribution of a random variable is a very useful con-cept. Even though an individual future observation on random variable Xis not predictable, a large number of observations form a highly pre-dictable pattern. This pattern is the random variable’s probability densityfunction.
Now, here is where these ideas start to come together for us. Chapter5 will show that when a hypothesis is represented by the probability dis-tribution of a random variable we will be able to use an observed value ofthe random variable to test the veracity of the hypothesis. This will enableus to conclude if a rule’s back-tested profitability was due to luck or gen-uine predictive power.
The probability distribution used for this purpose is a specific type. Itis called a sampling distribution—perhaps the most important one in all ofstatistics and certainly the most important for TA analysts.
Area AboveInterval ABIsAkolapyesaal ee lered aksBola E 1)
<H-r-wpwornv<AH-uzm0
 
A BRange of Random Variable X
FIGURE 4.25 The probability random variable X will assume a value between Aand B equals the fraction of the distribution’s total area above the interval A,B.202 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
       
of Area
PpR
OD
BE
AN
Bos
I Fraction ofLT Distributionly Beyond “B" = 0.15T
Y
 
BRange of Random Variable X
 
 
 
FIGURE 4.26 The probability random variable X will assume a value equal to orgreater than B equals the fraction of the distribution’s total area equal to or greaterthan B.
THE SAMPLING DISTRIBUTION: THE MOST IMPORTANTCONCEPT IN STATISTICAL INFERENCE
 
To recap:
1. A random variable is the outcome of a probability experiment.
2. Taking a sample of a given number of observations from a populationand computing the value of a statistic such as the sample’s mean is aprobability experiment.”
3. A sample statistic”® is a random variable in the sense that it fluctuatesunpredictably from one sample of observations to the next. The sam-ple statistic fluctuates randomly because chance determines whichspecific observations wind up in a particular sample, and it is that par-ticular set of observations that determines the value of the statistic forthat sample. In the box-of-beads experiment, the value f-g varied ran-domly from one 20-bead sample to the next because the particular setof beads that wound up in a given sample was determined by chance.
4. A relative frequency distribution describes how often the possiblevalues of a random variable occur over a very large number ofobservations.
We are now ready to meet the sampling distribution, the most impor-tant concept in statistical inference.Statistical Analysis 203
The Sampling Distribution Defined
The sampling distribution is the probability distribution of a random vari-able, and that random variable happens to be a sample statistic. In otherwords, the sampling distribution shows the various possible values thesample statistic can assume, and their associated probabilities.”” For exam-ple, “the sampling distribution of the sample mean refers to the probabilitydistribution of all possible means for all possible random samples of agiven size from some population.””’ Here, the sample statistic is the mean.
The sample statistic in the box-of-beads experiments was f-g. Its ran-dom variation across 50 samples, each comprised of 20 beads, was shownby the relative frequency distribution in Figure 4.8. If, instead of 50 valueswe were to take all possible 20-bead samples—a very large number—thistheoretical distribution would be f-g’s sampling distribution.
In rule back testing, the sample statistic is a measure of performanceobserved in a back test. In this book that sample statistic will be the rule’saverage rate of return. A back test typically produces a single observedvalue for the performance statistic because we have a single sample ofmarket history. Now imagine what it would be like if we were able to testa rule in an infinite number of independent samples of market history.This would provide an infinite number of values for the performance sta-tistic. If this set of data were then converted into a relative frequency dis-tribution, it would be the statistic’s exact sampling distribution. This isobviously not possible because we do not have an infinite number of inde-pendent samples of historical data. However, statisticians have developedseveral methods to get something that approximates the exact samplingdistribution closely enough to be useful, despite the fact that we have onlyone sample of historical data and one value of the sample statistic. Two ofthese methods will be discussed later in this chapter.
The Sampling Distribution Quantifies Uncertainty
The sampling distribution of a statistic is the foundation of statistical in-ference because it quantifies the uncertainty caused by the randomness ofsampling (sampling variability).
As stated above, the sampling distribution displays the relative fre-quencies of a statistic if it were to be measured in an infinite number ofrandom samples of the same size, drawn from the same parent popula-tion. The box-of-beads experiments showed that the value of a statistic, f-gfluctuated randomly from sample to sample. Figure 4.8 showed that thesevalues tend to fall into a well-behaved pattern, not a random chaotic mess.The fact that the pattern of random variation is well behaved is whatmakes statistical inference possible.204 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
It may seem ironic that a sample statistic would show such a regu-lar pattern given that it is, in fact, a random variable. Thankfully for sta-tisticians, it does. Figure 4.8 revealed a central value (i.e., the mean ofthe sampling distribution) of approximately 0.55. It also showed a well-defined pattern of dispersion about this central value. This pattern al-lowed us to conclude, with a fair degree of assurance,