this constraint that creates the needfor statistical inference.
Although we are not allowed to examine the box’s contents in its en-tirety, we are permitted to take samples of 20 beads at a time from the boxand observe them. So our strategy for acquiring knowledge about F-G willbe to observe the fraction of grey beads in a multitude of samples. In thisexample 50 samples will be taken. The lowercase f-g stands for the frac-tion of grey beads in a sample.
A sample is obtained as follows: The bottom of the box contains asliding panel with 20 small depressions, sized so that a single bead is cap-tured in each depression. The panel can be slid out of the box by pushingit with a similar panel that takes its place. This keeps the remainingbeads from dropping out of the bottom. Consequently, each time thepanel is removed, we obtain a sample of 20 beads and have the opportu-nity to observe the fraction grey (f-g) in that sample. This is illustrated inFigure 4.5.
After the fraction of grey beads (f-g) in a given sample has been deter-mined, the sample beads are placed back into the box and it is given athorough shaking before taking another sample of 20. This gives eachbead an equal chance of winding up in the next sample of 20. In the par-lance of statistics, we are making sure that each sample is random. Theentire process of taking a sample, noting the value f-g, putting the sampleback in the box, and shaking the box is repeated 50 times. At the end ofthe whole procedure we end up with 50 different values for f-g, one valuefor each sample examined.1°74 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
         
What FractionAreGray?
F-G Sliding Panel
 
FIGURE 4.5 Determining f-g for each sample.
By placing each sample back in the box before taking another sam-ple, we are maintaining a stable concentration for the fraction of greybeads in the box. That is to say, the value for F-G is kept constant overthe course of the 50 samplings. Problems in which the statistical charac-teristics remain stable over time are said to be stationary. If the beadswere not replaced after each sample the value F-G would change over thecourse of the 50 samplings, as groups of beads were permanently re-moved from the box. A problem in which the statistical characteristicschange over time is said to be nonstationary. Financial markets may in-deed be nonstationary, but for pedagogical purposes, the box-of-beadsproblem is designed to be stationary.
It is important to keep in mind the distinction between F-G and f-g.F-G refers to the fraction of grey beads in the entire box. In the languageof statistics, all the observations in which we are interested are called apopulation. In this example, the term population refers to the color of allthe beads in the box. The term sample refers to a subset of the popula-tion. Thus, F-G refers to the population while f-g refers to the sample.Our assigned task is to gain as much knowledge as possible about thevalue of F-G by observing the value f-g over 50 separate samples.
It is also important to keep clear the distinction between two num-bers: the number of observations comprising a sample—in this case 20beads—and the number of samples taken—in this case 50.
PROBABILITY EXPERIMENTS AND RANDOM VARIABLES
 
Probability is the mathematics of chance.!’ A probability experiment isan observation on or a manipulation of our environment that has an un-certain outcome.!! This would include actions such as noting the pre-Statistical Analysis 175
cise time of arrival after a long trip, observing the number of inches ofsnow that falls in a storm or the face of a coin that appears after thetossing of a coin.
The quantity or quality observed in a probability experiment is calleda random variable such as the face of a coin after a flip, the number ofinches of snow that fell, or a value that summarizes a sample of observa-tions (e.g., a sample average). This quantity or quality is said to be randombecause it is affected by chance. Whereas, an individual observation of arandom variable is unpredictable, by definition, a large number of obser-vations made on a random variable may have highly predictable features.For example, on a given coin toss, it is impossible to predict head or tails.However, given one thousand tosses, that the number of heads will liewithin a specified range of 500 is highly predictable.
Depending on how it is defined, a random variable assumes at leasttwo different values, though it can assume more—perhaps even an infinitenumber. The random variable in a coin toss, the face visible after the toss,can assume two possible values (heads or tails). The random variable, de-fined as the temperature at noon taken at the base of the Statue of Liberty,can assume a very large number of values, with the number limited onlyby the precision of the thermometer.
Sampling: The Most ImportantProbability Experiment
The most important probability experiment in statistical analysis is sam-pling. It involves the extraction of a subset of observations from a popula-tion. Here the random variable at issue is called a sample statistic. It is anycomputable characteristic of the sample. The value f-g is an example of asample statistic.
Sampling can be done in a number of different ways, but it is crucialthat the observations that are selected for the sample be chosen in a ran-dom manner and that they be selected independently of each other. Thismeans that. all observations that could possibly end up in the sample havean equal chance of doing so. Because no particular observation has a bet-ter chance of appearing in the sample than any other, the observationsthat wind up in the sample do so by chance. Samples must be constructedin this manner because the principles of probability, upon which statisti-cal reasoning rests, assume that the observations that wind up in the sam-ple got there randomly.
Consider the test of a new medicine on a sample of volunteers. Thesample must be constructed so that anyone who might eventually take themedicine has an equal chance of being selected for the test group. If thesubjects in the experiment are not selected this way, the results of the test176 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
will not lead to a sound conclusion. A medical experiment that selects testsubjects who are expected to respond favorably to the medication willproduce a biased conclusions. In other words, the estimates of the med-ication’s efficacy in the general population will be too optimistic.
Imagine taking a single bead at random from the box with your eyesclosed and then observing its color. This is a sampling experiment with asample size of one. The bead’s color, grey or white, is the random variable.Now imagine randomly selecting 20 beads from the box. Here the samplesize is 20 and the fraction grey in that sample (f-g) is a sample statistic thatis also a random variable.
As discussed earlier, we will call this random variable f-g. It is a ran-dom variable because its value is affected by chance. It can assume 21 dif-ferent values {0, 0.05, 0.10, 0.15, ..., 1.0}. Now let’s get back to the goal:increasing our knowledge about the value of F-G.
The Knowledge Gleaned from One Sample
Suppose the first sample contains 13 grey beads out of 20. The value of thesample statistic f-g is 0.65 (13/20). What does this tell us about F-G? Withonly this information, some might be tempted to conclude that we havesolved the problem, and that 0.65 is the fraction grey for the entire box(i.e., F-G = 0.65). This view tacitly and naively assumes that a single sam-ple can provide perfect knowledge. At the opposite extreme are thosewho just as wrongly conclude that the sample is too small to supply anyinformation.
Both conclusions are wrong. First, there is no basis for thinking thatwe could acquire perfect knowledge from one sample. Though it is pos-sible that a single sample of 20 beads might be a perfect replica of theentire contents of the box, it is not likely. For example, if the true valueof F-G were 0.568, a 20-bead sample could never produce such a value.An f-g value of 0.55 (11 grey of 20) is as close as one could get. Ten greybeads in the sample would give an f-g of 0.50, 11 would give 0.55, and 12would give 0.60.
Those who assert that nothing has been learned are also mistaken.On the basis of this sample alone, two possibilities can be eliminated.We can reject with absolute certainty that F-G = 1.0 (i.e., all beads grey)because there were 7 white beads in the sample. With equal certainty,we can reject the proposition that F-G is 0, because there were 13 greybeads in the sample.
However, more precise estimates of F-G on the basis of the single ob-served value of f-g are subject to uncertainty. Even if we were to take nu-merous samples and have numerous values of f-g, the value of F-G wouldremain uncertain. This is because a sample is, by definition, a partial rep-Statistical Analysis 177
resentation of the full contents of the box. Only by observing every beadin the population could all uncertainty about F-G be eliminated. And thisis prohibited.
However, with what has been learned from the single sample, someintelligent guesses can be made that go beyond the certain knowledge thatF-G is neither 0 nor 1.0. For example, although a claim that F-G is as lowas 0.10 cannot be conclusively ruled out; no one would take it seriously.The observed f-g value of 0.65 obtained from the first sample is too farabove 0.10 for it to be a credible estimate of F-G. If F-G really were as low0.10, the probability of getting a sample with an f-g of 0.65 would seem un-likely just based on common sense. We could apply the same kind of logicto dispense with a claim that F-G = 0.95. An f-g reading of 0.65 would seemtoo low if the entire box were composed of 95 percent grey beads. Thebottom line: It is fair to say that a single sample does provide some knowl-edge about F-G.
What Can Be Learned from 50 Samples?
Greater knowledge about F-G can be obtained by analyzing more samples.Assume another 49 samples are taken and the value f-g for each sample ismeasured. The first thing we would notice is that f-g varies in an unpre-dictable fashion from sample to sample. This particular form of randombehavior is one of the most important phenomena in all of statistics. It iscalled sampling variability or sampling variation.
Sampling variation is important because it is responsible for the un-certainty in statistical conclusions. Greater sampling variability translatesto greater uncertainty. The greater the fluctuations in f-g from one sampleto the next , the greater will be the uncertainty about the value of F-G.
Regrettably, this important phenomenon, sampling variability, is unfa-miliar to many people who analyze data. This is understandable becausereal world problems do not offer the luxury of looking at more than oneindependent sample. The box-of-beads provides this opportunity.
As will be seen, by conducting 50 sampling experiments, the randomvariation in f-g becomes obvious. This is quite different from the situationfaced by rule researchers. They typically have a single historical sample ofthe market, and if all that data is used to test a rule, only one observedvalue of the performance statistic is obtained. This provides no opportu-nity to see how the rule’s performance would vary if it were tested in nu-merous independent samples. The bottom line is this: Sampling variationis an important fact that may not be obvious to data analysts unfamiliarwith statistical analysis. This is a huge blind spot!
Sampling variation can be seen in a table of the 50 f-g values. SeeTable 4.1.178 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
TABLE 4.4 = Fraction of Grey Beads in 50 Different Samples
Sample Sample Sample
Number f-g Number f-g Number f-g
1 0.65 18 0.60 35 0.602 0.60 19 0.40 36 0.553 0.45 20 0.60 37 0.504 0.60 21 0.45 38 0.505 0.45 22 0.55 39 0.456 0.45 23 0.45 40 0.507 0.55 24 0.50 41 0.558 0.40 25 0.70 42 0.459 0.55 26 0.55 43 0.5510 0.40 27 0.60 44 0.6011 0.55 28 0.70 45 0.6012 0.50 29 0.50 46 0.6013 0.35 30 0.70 47 0.6514 0.40 31 0.50 48 0.5015 0.65 32 0.90 49 0.5016 0.65 33 0.40 50 0.6517 0.60 34 0.75
 
Not surprisingly, the value of f-g, a random variable, fluctuates fromsample to sample. This is an example of chance in operation. However,f-g’s value is not entirely determined by chance. Its value is alsostrongly influenced by the value of F-G, the proportion of grey beads inthe entire box.
Thus, it can be said that each observed value of f-g is the result of twoinfluences; the underlying phenomenon, F-G, and randomness caused bysampling. F-G acts like a center of gravity, always keeping f-g within a cer-tain range. In some samples, randomness nudged the value f-g above F-G.In other samples, randomness nudged f-g below F-G. From sample to sam-ple the value of f-g oscillates randomly around this center of gravity. Thisis sampling variability.
Another important aspect of this situation is that the number of obser-vations impacts our level of uncertainty about the value of F-G. Thegreater the number of observations comprising the sample, the more pre-cisely the value f-g will reflect the value F-G. Suppose that, instead of tak-ing a 20-bead sample, we took a 200-bead sample. The random variation inf-g around F-G would be smaller. In fact, it would be about one third ofwhat it is for a 20-bead sample. This is a very important point: the largerthe sample size, the smaller the impact of randomness. Two hundredbeads dampen the ability of any single bead to push f-g away from F-G. InStatistical Analysis 179
a one-bead sample, the single selected bead will produce an f-g of either 0or 1.0. In a two-bead sample, f-g can be 0, 0.50 or 1.0. In a three-bead sam-ple, f-g can be 0, 0.33, 0.66, or 1.0. Thus, the larger the sample size, thesmaller will be the magnitude of random variation in f-g. Large samplesgive F-G, the truth that we wish to know, the ability to reveal itself. This isan effect of the Law of Large Numbers: Large samples reduce the role ofchance. In other words, it can be stated that the larger the size of a sam-ple, the more tightly the values of f-g will cluster about the value F-G. Thisis one of the most important principles in statistics.
We have learned an important concept of statistics that can bestated as follows: even though the value of F-G does not change overthe course of the 50 sampling experiments, the value f-g can vary con-siderably from sample to sample. The phenomenon is called samplingvariability. It is present whenever a random sample of observations isused to form a conclusion about a larger universe (i.e., population).Sampling variability is the source of the uncertainty that is addressedby statistical inference.'”
Frequency Distribution of the Sample Statistic f-g
In the box-of-beads experiment, the random variable f-g can assume21 possible values ranging from zero, when there are no grey beads ina sample, to 1.0, when the sample is composed entirely of grey beads.The 50 observed values of f-g are shown in Table 4.1. A casual examina-tion shows that some values of f-g occurred with greater frequency thanothers. Values in the range of 0.40 to 0.65 occurred quite often, whereasvalues less than 0.40 and greater than 0.65 almost never occurred. Inthe 50 samples taken, f-g assumed the value 0.50 nine times, 0.55 eighttimes, and 0.60 ten times. Note that the value 0.65, which characterizedthe first sample, appeared only five times. Therefore, that value wasnot among the most common values, but it was not particularly unusualeither.
A plot called a frequency distribution or frequency histogram commu-nicates this information more forcefully than words or a table. It displayshow frequently each of f-g’s possible values occurred over the 50 samplingexperiments. The term distribution is apt, because it depicts how a set ofobservations on a random variable are distributed or sprinkled across thevariable’s range of possible values.
Arrayed along the horizontal axis of the frequency distribution is a se-quence of intervals, or bins, one for each possible value of f-g. The heightof the vertical bar over each interval represents the number of times that aspecific value of f-g occurred. Figure 4.6 shows the frequency distributionof the 50 f-g values.180 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Sample Size = 20 Number of Samples = 50
 
 
 
 
 
 
 
 
 
 
 
 
 
10974F 8-R; 7Q 6U- 5N 4Ccy 321oe
 
0 0.1 0.2 03 04 0.5 06 0.7 08 0.9 1fg
FIGURE 4.6 Frequency Distribution: (f-g).
The Equivalence of Frequency and Area
Frequency distributions depict an important relationship between thefrequency of a particular value or set of values and the area covered bythe bars representing that value or set of values. This concept can beunderstood if we first consider the entire distribution. In Figure 4.6, allthe grey bars comprising the frequency distribution represent all 50of the observed values of f-g. Thus, the area covered by all the verti-cal bars corresponds to 50 observations. You can verify this by sum-ming the counts associated with all of the vertical bars. They will addup to 50.
The same principle applies to any fraction of the distribution’s area. Ifyou were to determine the fraction of the frequency distribution’s totalarea covered by the area of a single bar, you would find that the bar's frac-tional area is equal to the fraction of the total observations represented bythat bar. For example, the vertical bar associated with the f-g value 0.60shows a count (frequency) of 10. Thus, the bar represents 20 percent(10/50) of the observed values. If you were to then measure the area cov-ered by this vertical bar, you would find that its area represents 0.20 ofthe total area covered by the entire distribution. This idea is illustrated inFigure 4.7.Statistical Analysis 181
 
 
 
 
 
 
 
 
 
 
 
 
 
    
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10 This Bar Has 0.20 |____>»|9 of Distribution’sF 8-H Total ObservationsR l¢——_| This Bar Has 0.20E 7 of Distribution’sQ 6 Total AreaUa)N 4Ccy 3act]0 —0 0.1 0.2 03 04 0.5 06 0.7 0.8 0.9 1f-gFIGURE 4.7 Proportion of observations equals proportion of distribution’s to-tal area.
Although this idea may seem obvious, even trivial, it is an essentialaspect of statistical reasoning. Ultimately, we will use the fractional areaof a distribution to measure the probability that the back-test profits of arule could have occurred by chance under the assumption that the rulehas no predictive power. When that probability is small, we are led to theconclusion that the rule does have predictive power.
Relative Frequency Distribution of f-g
The relative frequency distribution is similar to the ordinary frequencydistribution discussed earlier. The height of a bar in an ordinary frequencydistribution represents an absolute number or count of the number oftimes a particular value of a random variable was observed to occur. Inthe relative frequency distribution, the height of a bar represents the num-ber of observed occurrences relative to (divided by) the total number ofobservations comprising the distribution. For example, f-g assumed thevalue 0.60 on 10 out of 50 observations. Thus the value 0.60 had a relativefrequency of 10/50 or 0.20. The distribution’s bar for the value 0.60 wouldbe drawn to a height of 0.20 along the vertical scale. This is illustrated inFigure 4.8.#82 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Sample Size = 20 Number of Samples = 50
 
0.20.180.160.140.12
0.10.080.060.040.02
———0 0.1 0.2 03 04 0.5 06 0.7 0.8 09 1
fg
FIGURE 4.8 Relative frequency distribution: (f-g).
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
<~<ONOZmMCOMAmM M<e—-APrmMDA
The equivalence between frequency and area that applies to ordinaryfrequency distributions also applies to relative frequency distributions.The relative frequency of all bars is equal (adds up) to 1.0. This simplysays that a random variable always (1.0 of the time) assumes a valuesomewhere in the range encompassed by all bars. If you were to add upthe relative frequencies associated with all of the individual bars compris-ing the distribution, they would sum to 1.0 (100 percent of the observa-tions). This has to be so because each bar represents the fraction ofobservations falling into that interval, and, by definition, all the bars to-gether represent all (1.0) the observations.
The relative frequency of any single bar or group of contiguous bars isequal to their proportion of the distribution’s total area. Thus the relativefrequency of an f-g value of 0.65 and larger is equal to 0.10 + 0.06 + 0.02 +0.02 = 0.20. This is equivalent to saying that the combined area of the con-tiguous bars associated with f-g values of 0.65 and larger is 20 percent ofthe distribution’s total area. We will be making statements similar to thiswhen testing claims about TA rules. Based on this, it can be said that therelative frequency of f-g values equal to or greater than 0.65 is 0.20. This isillustrated in Figure 4.9.Statistical Analysis 183
Sample Size = 20 Number of Samples = 50
 
 
 
    
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
R
E 0.2 5
L
, 0.18
T 0.16 Relative
| . Frequency
y 0.14 f-g = or > 0.65
E 0.12
F 0.1
R
E 0.08 7
6 0.06
E 0.04
N 0.02
Cc
Y 0 —— 1 —0 0.1 0.2 03 04 05 06 0.7 08 0.9 1
f-gFIGURE 4.9 Relative frequency distribution: (f-g).
What Knowledge Has Been Gained about F-Gfrom Sampling?
So far, sampling has increased our level of knowledge about F-G. In thefirst 20-bead sample, f-g had a value of 0.65. Based on this observation,two possibilities were conclusively ruled out: that F-G is equal to 0, andthat F-G is equal to 1.0.
Not being content with these meager morsels of wisdom, another 49samples were taken, and the value f-g was observed in each. These 50 val-ues showed that f-g varies randomly from sample to sample. However, de-spite the unpredictability of any single f-g value, an organized patternemerged from the randomness that was informative. The f-g values coa-lesced about a central value forming a well-organized hump. We suspect.that this central tendency is related to the value F-G, yet F-G’s precisevalue remains uncertain because of the random variation in f-g. However,in light of the hump’s relatively narrow width, it seems reasonable to con-jecture that F-G lies somewhere in the range 0.40 to 0.65.
Given that we started with no knowledge of F-G and given that wewere precluded from examining the contents of the box in its entirety, alot was learned from these 50 samples.184 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
Sample Size = 20 Number of Samples = 50
 
 
 
 
 
 
 
 
 
RE 0.3LAT 0.25|Vv 0.2EFo 0.15REQ 0.1UE 0.05NCcY 00 0.1 0.2 03 04 05 06 0.7 08 0.9 |fgFIGURE 4.10 Relative frequency distribution box 2: (f-g).
A Second Box of Beads
Now we will use sampling to learn the color proportions of a second boxof beads. It too is a mixture of grey and white. The objective is the same—to learn the relative proportion of grey beads to the total number of beadsin a second box. We call this quantity F-G 2.
As before, we are not allowed to examine the full contents of box 2,but we are permitted to take 50 samples, each composed of 20 beads. Thevalue f-g 2, which refers to the proportion of grey beads in a sample frombox 2, is measured in each sample. Again, note the distinction betweenthe terms F-G 2 and f-g 2. F-G 2 refers to the proportion of grey beads inthe entire box 2 whereas f-g 2 refers to the proportion of grey beads in anindividual sample. F-G 2 is not observable while f-g 2 is.
Figure 4.10 is the relative frequency distribution for 50 values of f-g 2.There are several things to notice:
1. The general shape of the distribution for box 2 is similar to that of box1, ahump clustering about a central value.
2. The central value of the distribution for box 2 is different than that ofbox 1. The distribution for box 1 was centered near a value of 0.55.The central value for box 2’s distribution is near 0.15. This is clearlyStatistical Analysis 185
seen in Figure 4.11, which shows both distributions on the same hori-zontal scale. An arrow has been placed above each distribution attheir approximate central values. Thus the arrows represent averagevalues for f-g and f-g 2. From this we can conclude that box | has ahigher concentration of grey beads than box 2.
. Even though the samples from box | and box 2 were both affected by
random variation, the degree of random variation differs. Box 2’s re-sults are less variable than those of box 1. This is evidenced by box 2’snarrower clustering about its central value. Given the lower degree ofsampling variation in box 2, it would be fair to say that we know thevalue of F-G 2 with greater certainty than we know F-G.
What the Box Experiments Taught Usabout Statistics
Yogi Berra, former manager of the New York Yankees, said you can ob-serve a lot just by looking. If he were a statistician, he might have said thatyou can learn a lot just by sampling. Even though a sample is only a por-tion of a larger universe (population), it can teach us a lot about that pop-
0.25 Box |
0.3
 
 
 
 
 
 
 
0.2
 
 
 
 
 
0.1
 
0.05
0 i =
0 O.. 0.2 03 04 05 06 0.7 08 0.9 1
f-g
 
 
 
<~<NZmMcCOMmMmAMN M<—-APrMDA
 
 
 
 
 
 
 
 
 
 
 
 
 
 
FIGURE 4.11 Relative frequency distributions compared.986 METHODOLOGICAL, PSYCHOLOGICAL, PHILOSOPHICAL, STATISTICAL FOUNDATIONS
ulation. However, there are limits to what a sample can tell us, limits im-posed by randomness.
A fundamental task of statistics is to quantify the uncertainty due tosampling variability. This makes it possible to quantify the reliability ofstatements of knowledge based on samples. Without such quantification,such statements are of limited value. Statistics cannot eliminate uncer-tainty, but it can inform us about it, and in so doing, it tempers assessmentsof how much we know or how accurately we know it. Thus, statisticalanalysis is a powerful antidote to the tendency to be overconfident.
STATISTICAL THEORY
 
The box-of-beads experiments touched on many of the key ideas of statis-tical inference. We will now mix in some theory to extend those ideas tothe evaluation of TA rules.
The Six Elements of a StatisticalInference Problem
Evaluating a claim that a TA rule has predictive power, that a new vaccinecan prevent illness, or any other claim of knowledge are examples of astatistical inference problem. Generally, such problems can be reduced tosix key elements: (1) a population, (2) a sample consisting of a set of ob-servations randomly selected from the population, (3) a population para-meter, (4) a sample statistic, (5) an inference, and (6) a statement aboutthe reliability of the inference. Each of these elements will be discussed asthey relate to the box-of-beads experiments and then to the evaluation ofTA rules.
The Population. A population consists of all possible observations ofa random variable. It is a large—perhaps infinitely large—but a well-defined