RISE
7
IFTHEN
unemploymentp/cap spend
FALLRISE
IFTHEN
unemploymentp/cap spend
RISEFALL
8
IFTHEN
inventoriesproduction
FALLRISE
IFTHEN
inventoriesproduction
RISEFALL
9
IFTHENIFTHEN
productionGNP
RISERISE
productionGNP
FALLFALL
Fed money supinterest
ADDFALL
IFTHENIFTHEN
Fed money supinterest
DECRISE
23
6
10
'
Variable Name
Meaning
InterestStocksU.S. dollarInflationGNPGer Bund rateP/cap spendInventoriesProductionFed money sup
U.S. interest ratesU.S. stock marketU.S. dollar exchange rateRate of U.S. inflationThe Gross National ProductGerman Bund interest ratePer capita spendingU.S. manufacturers' inventoriesTotal U.S. manufacturing productionFederal Reserve money supply target
If the news is:
UNEMPLOYMENT RISESStart at Rule 7b,
IF unemployment RISES, THEN p/cap spend FALLS,
Then to Rule 6b,
IF p/cap spend FALLS, THEN inventories RISE,
To Rule 8b,
IF inventories RISE, THEN production FALLS,
To Rule 9b,
IF production FALLS, THEN GNP FALLS,
To Rule 4a,
IF GNP FALLS, THEN interest FALLS,
Finally to Rule 1
IF interest FALLS, THEN stocks RISE.
Computer Learning, Neural Networks, and New Technology
163
By chaining one rule to another, the computer should reach the sameconclusion as an expert. The only problem is that this answer does notmake sense. In this case, although all the rules are perfectly correct, theconclusion unemployment rises, therefore the stock market rises is wrong.To be fair, it is not technically wrong. What is missing is the time delay.Unemployment will cause interest rates to be lowered, which will movethe stock market higher. But not on the same day. First, the market willdrop on the news. Each sequence, represented by a rule, must be assigneda reaction time or completion criterion. Rule 7b should really read:IF unemployment RISES, THEN p/cap spending FALLSover the next 3 months.
By adding time to each rule, we come closer to an expert system.Drawing on the Knowledge BaseOnce the knowledge base has been established, many different questions can be asked: "What is the effect on the stock market when the Fedwants to increase the money supply? What is the effect of an increase inunemployment on the U.S. dollar?"The reverse of many items in the knowledge base, but not all, mayalso be used: "IF interest rates DO NOT FALL and the GNP IS NOTPOSITIVE, THEN the stock market WILL NOT RISE."Resolving Conflicts ofMultiple EventsAs easy as it is to show sequences stemming from single events, it is notrealistic. Two or more significant factors are usually present and oftenconflict with one another. Which is more important, if any? The following rules can be added to help determine which government statistic ismost important:W1. IF actual statistic minus expectations IS MOST EXTREME THENmost important.W2. IF cumulative difference of last 3 statistics minusexpectations IS MOST EXTREMETHEN most important.W3. IF actual statistic minus year ago IS MOST EXTREMETHEN most important.W4. IF statistic minus long-term mean IS MOST EXTREMETHEN most important.
164
Using Old and New Trading Tools for Reasonable Objectives
But it is still not complete. Two or more events may be extreme in different ways. They may be cumulative or offsetting in their effect on thestock market. Rules must be written to resolve these factors. This conflict resolution can be the weak point of expert systems.ValidationWhen it is all in the computer and an answer pops out, how do youknow it is correct? Because the process is logical, the individual stepscan be traced to prove the answer. But the example of "UNEMPLOYMENT RISES" gave the right answer but not the right time. In the end,the final decision is yours. The answer must seem right, and satisfy thetest of reasonableness.
Neural NetworksAlthough the idea and words for a computerized neural network comefrom the biological ideal of the human brain, an artificial neural networkis not a model of a brain, nor does it "learn" in the human sense. It issimply very good at finding patterns. In fact, it can be so good that it"overfits" the data, finding patterns that exist only by chance. In thatway, nearly all methods for finding the "best" performing systemsshare the same problems, whether very simple or complex techniques.A wide selection of neural network software is available for the personal computer. Many of these can be found through trade magazinessuch as Technical Analysis of Stocks & Commodities and Futures.Terminology of Neural NetworksThe brain is composed of cells called neurons, which process and storeinformation. They are unique in the human system because they do notdie, which is why we are able to remember. Neurons function in groupscalled networks, which have thousands of interconnected neurons, andthose networks are connected to other neural networks.Information is received through dendrites and goes directly into a neuron. The data can be passed to other neurons through an output connector called an axon. As the information passes from one neuron toanother neuron or neural network, it may pass through a synapse, whichcan inhibit or enhance the importance of the data going in differentdirections. A synapse may also be considered a "selector." Figure 9-4shows a biological neural network and its components.The human neural network is remarkable in its ability to receive vast
Computer Learning, Neural Networks, and New Technology
165
Figure 9-4. A biological neural network. Information is receivedthrough dendrites and passed to a neuron for storage. Data are sharedby other cells by moving through the output connector, called an axon.A synapse may be located on the path between some individual neurons or neural networks; they select the relevant data by inhibiting orenhancing the flow.
amounts of data, store it away, and make you aware of only the mostimportant items. This sophisticated selection process can vary for eachindividual and situation. For example, you may no longer hear the ticking of a clock in your bedroom but are instantly aware of the smallestunusual sound coming from a child's room. We actually hear the clock,but the sound becomes routine and our neural networks do not alert usto action. When the clock stops, an efficient human system will notice.Artificial Neural NetworksUsing the same structure as the biological neural network in Figure 9-4,we can show how economic and price information passes through acomputerized, artificial, neural network (ANN) to produce a decisionon the direction of stock prices.The first step is shown in Figure 9-5. The system receives a wideassortment of input data, as does the human system. It must selectwhich of these are relevant in finding the answer and assign weightingfactors to represent their value. Because "Domestic Health" influencesinterest rates and subsequently stock prices, we select how each of thefive inputs will be used. Two are discarded as irrelevant by giving aweighting factor of zero to their importance; however, all items remainstored in "neurons" for later use.Unemployment, GDP, and inventories are all considered to contribute to Domestic Health. The value placed on their importance willbe decided by the computer neural net program. We can expect theweighting factors for GDP and inventories to be positive, because risingGDP and inventories indicate strong economic activity. Inventories are
\ 66
Inputs
Using Old and New Trading Tools for Reasonable Objectives
DataStored inLevel 1Neurons
SelectionbyWeighting#1
CombinedValvesStored inLevel 2Neurons
SelectionbyWeighting#2
CombinedValvesStored inLevel 3Neurons
Outputresult
Figure 9-5. A three-layer artificial neural network. Economic and price dataare received and stored in Level 1 neurons. Information Is passed to Level 2through synapses that reduce or Increase the importance of each item, formingnew composite values representing the economic health of the country and theanticipated direction of the U.S. dollar. These values are then combined intoLevel 3, an expected movement in U.S. stocks. In most artificial neural networkseach neuron in Level 1 is connected to all neurons In Level 2.
not as clear and get a smaller value. Unemployment gets a negative factor. When unemployment rises, domestic health declines.The Three-Layer SystemDomestic health is only one element needed to forecast stock marketprices. Current and anticipated interest rates may be the greatest factor.But interest rates are used to achieve an economic growth target, whichmay be measured by domestic health. Therefore, each feeds on theother. To make matters more complicated, world political events causemoney to move to safety. If Eastern Europeans are buying the U.S. dollar regardless of current interest rates, then rates can go lower. On theother hand, if the United States needs to attract foreign investors to support its debt, then U.S. interest rates must rise to attract buyers, regardless of domestic health.Each neuron in Level 2 and 3 depends on the weighting factors of eachof the neurons that feed it. Unlike the expert system, it is not told the effect
Computer Learning, Neural Networks, and New Technology
167
Figure 9-6. Learning by feedback. A neural network program uses theknown answers as feedback to find the weighting factors that work.
of each piece of input data, but determines its use by comparing historicexamples. For example, the value assigned to domestic health in Figure9-6 is the result of many inputs, each with unique weighting factors.The Training ProcessThe ANN arrives at an answer through a computer-intensive process ofpattern recognition. The most popular method is called a genetic algorithm, because random selection in determining the importance of eachinput causes one solution to be better than another. This "mutation" isused in the same way that natural selection would allow a better speciesto survive.The neural network uses the genetic algorithm to "learn" how toarrive at the best answer in a feedback process called training. Thismethod compares the random use of input data and indicators with aknown answer until it finds the combination that comes closest to beingcorrect. Figure 9-6 shows the "feedback loop" that mutates weightingfactors, using random numbers, until the answers match a large sampleof historic situations.Because this is a trial-and-error process, rather than an analyticapproach, the best results could be caused by a coincidental occurrence
168
Using Old and New Trading Tools for Reasonable Objectives
of data, rather than cause and effect. Keep in mind the famous warning,Post hoc ergo prompter hoc (literally, "After this, therefore because ofthis"), which refers to the error in thinking which assumes that becauseone event followed another, the second event was caused by the first.A Training ExampleWe would like to train a neural net to tell usShould we buy or sell the stock market?
using only the five inputs shown in Table 9-2. To make it simpler:1. Each input is given an adjusted value, from +100 to -100, indicatingwhether the current state is strong or weak, high or low, or neutral.2. Buy signals are given when the total value is above 120; sell signalsoccur when the value is below -120.3. Any value from +120 to -120 is considered neutral.The initial test has only two training cases, that is, the computer is giventwo sets of input data and the correct answers. History tells us that, whenevaluated correctly, Case 1 will give a strong signal and Case 2 a weaksignal. To begin, the weighting factors are all set to 1.0. As seen in Table9-2(a), both Case 1 and Case 2 produce neutral results, which are wrong.Trial and ErrorThrough the process of random assignment of weighting factors (whichwas chosen as the "genetic algorithm"), the network "mutates" the pattern. When the ANN arrives at a wrong answer, by comparing it againstthe historic facts, an error signal is produced and the neural net is toldto try again. This is the feedback process. The system then changes theweighting factors until it stumbles on the reverse sign for interest rateand unemployment values (minus instead of plus). The neural net hadstarted by assuming that "low" meant negative and "high" meant positive. But the effect of low or high interest rates is the reverse in the stockmarket. By changing the weighting factor to -1.0, a correct answer isgenerated in Table 9-2(b).Because there were only two cases, many combinations of weightingfactors would have given the correct answers. For example, interestrates could have been assigned a factor of -5.0 while all other inputswere given a value of zero. These ambiguities are quickly removedwhen the number of training cases increases.
Computer Learning, Neural Networks, and New Technology
169
Table 9-2. Two training cases(a) First Test with Unit WeightsCasel
Case 2
Input
Relative Value Wgt
GNPUnemploymentInventoriesU.S. dollarInterest ratesTotal of all test valuesBuy/sell threshold levelsComputer training answerActual answer should be
StrongLowLowVryStrgFalling
50-25-5075-25
1.01.01.01.01.0
Net
Relative
Value
Wgt Net
50-25-5075-2575±125NeutralStrong
WeakHighNeutralNeutralRising
-604015045
1.01.01.01.01.0
-60401504540±125NeutralWeak
(b) Training Cases with Mutated Weighting FactorsCase 2
CaselInput
Relative Value Wgt
GNPUnemploymentInventoriesU.S. dollarInterest ratesTotal of all valuesBuy/sell threshold levelsComputer training answerActual answer should be
StrongLowLowVryStrgFalling
50-25-5075-25
Net
Relative
Value
Wgt Net
5025-507525125±125Up signalStrong
WeakHighNeutralNeutralRising
-604015045
1.0-1.01.01.0-1.0
1.0-1.01.01.0-1.0
-60-40150-45-130±125Down signalWeak
Relative values assigned in a range from -100 to 100. WGT is the weighting factor, set to 1.0 tostart. Actual is the response you want to get, or the actual price change over the next 10 days.
Specifying a Neural Network TestTraining the neural network can be a very long process. Allowing thecomputer to assign weighting factors to countless data items, and continually comparing the answers with the correct one, can take longerthan we are prepared to wait. To control the process, it is necessary toput limits on the training and help it along.Preprocessing. Rather than giving the system all the data possible,select the most significant information. Rather than using automobile anddepartment store sales separately, use a single retail sales figure. Eliminatesimilar items; each piece of information will continue to be analyzed by thecomputer over and over again without distinguishing whether two itemsare the same. Combine some items into indicators and eliminate the less
170
Using Old and New Trading Tools for Reasonable Objectives
important elements as redundant. If an index is more complicated than asimple weighting of its elements, then the neural network will not includeit properly. Include a trend of prices; the ANN cannot create one itself.Break the Problem into Clear Steps. Solving the problem in a singlestep may be overly complicated. More important, it becomes very difficultto validate the results. For example, separate forecasts into components.Before looking at the expected move in the stock market, forecast interestrates one week out, or forecast a cut or rise in the prime rate. If the stockmarket direction is dependent on interest rates, verifying the decisionprocess for rate changes should be a necessary step.Choose the Number of Decision Levels. Two or more inputs aregiven weights and may be combined into a single item in a new decisionlevel. If all the inputs can be taken two at a time, combined, and then usedwith the combination of two other inputs, the computer creates an excessive number of "hidden layers." More hidden layers allow more combinations and increase processing time. They allow the solution to be morespecific and require much more data to offset possible overfitting. A fourlayer system will also take much longer to process than a three-layer one,therefore the three-layer is highly recommended.Choosing the Smallest Number of Neurons. Although limiting thenumber of "hidden" layers will make the solution faster and more general, the number of neurons that hold intermediate results also can be specified. Just as fewer layers create a more general and faster solution, a smallnumber of neurons in each layer has the same effect. Fewer neurons meana more general solution.Trade-Offs. As with conventional optimization, neural networks canproduce a result that is overfit. Too much data, much of it irrelevant, andtoo many hidden layers and neurons allow the computer to find spuriouspatterns. Too few data items, levels, and neurons may make the result sogeneral that it is useless. The analyst must find the proper compromise.Forever LearningThe neural network responds to combinations of events in the mannerin which it was "taught." A drop in interest rates without the associated poor economic news (e.g., a flight to the safety of the U.S. dollar)results in a signal to buy equities. The neural network learns that thissituation is still good for stocks. But one day, interest rates drop sharply
Computer Learning, Neural Networks, and New Technology
1T1
when investors move their money fearing a plunge in stock prices. Youare a buyer of stocks because the move to lower interest rates satisfiesthe rules. The trade is a large loss as more stock is liquidated. The neural network adds a rule to bypass trades that begin under high volatilityconditions.The system continues to learn. There is no way to know how many different situations will be added to the list of conditions that build a complete network. A synthetic neural network is a technical achievement oflarge proportions. It can find patterns that cannot be identified by conventional methods (such as multiple regression used in econometrics). Itcan train itself to determine the importance of each input. But there aremany problems it cannot resolve. It will recognize only those inputs thatit expects and may not respond properly to combinations of inputs thatit has not "seen" before. If there are too many inputs, the network mayrespond correctly, but to the wrong events. It is the best so far, but it doesnot guarantee the results within the limits of your investment.
Fuzzy LogicFuzzy logic is not a brand name, or a description of scatterbrain thinking;it is a formal area of mathematics. Along with neural nets, fuzzy logicpushes the bounds of science. The idea of "fuzziness" describes the lackof precision in normal human conversation and thought. The conceptwill allow human uncertainty to be introduced to artificial intelligencemethods. Think about most casual conversations:'There were a lot of people at the game.""Most of them were tall.""It was really cold last night.""The market was strong yesterday.""The dollar collapsed when the trade deficit was higher than expected."Although these conversations do not include specific numbers, weaccept and understand what the other person is saying. In fuzzy logic,all is not true or false, 0 or 1, there or not there. It will answer questionssuch as "If a half-eaten apple is still an apple, how much do you have toeat before it stops being an apple?"The fuzziness concept includes fuzzy numbers, such as "small," "about8," "close to 5," and "much larger than 10," as well as fuzzy quantifiers,such as "almost," "several," and "most." The phrase "Surprise government reports cause big moves" is a common fuzzy expression.
172
Using Old and New Trading Tools for Reasonable Objectives
Fuzzy Reasoning"Fuzzy events" and "fuzzy statistics" are combined into fuzzy reasoning.Surprisingly, answers to the following examples are remarkably clear toa brain, but not to a machine:Example 1:
X is a small price move.Y is much smaller than X.How small is Y?
Example 2:
Most price moves are small.Most small price moves are up.How many price moves are up?
Example 3:
It is not quite true that the quarterly earnings were very bad.It is not true that quarterly earnings were good.How bad were the quarterly earnings?*
Practical SolutionsFuzziness is not intended to describe the same concepts that can beexplained precisely by probability (referred to a crisp logic.) Becausefuzzy logic and possibility theory are new, mathematicians believe thatpractical applications will use both (fuzzy) possibilities and (crisp) probabilities. Up to now, the fuzzy part has been assigned ranges to representcommonly used values. For example, in expressing the S&P pricechange, we might have the following:Description of Changeunchangedsmallmedium/normallarge
Value of an S&P Rise or Fall± 20 points± 145 points± 150 to 400 pointsmore than ± 400 points
pvq (£) pue 'fsoiu (z) '\\mus Hon. (;) sajdurexa avp o; siaMsue av[x«
Computer Learning, Neural Networks, and New Technology
173
The advantage of being human is that we can express these ranges ina vague but sophisticated way. We have defined "unchanged" to be aclose within 20 points of the previous close; yet following a few veryvolatile days, we might call a move of ± 50 points "unchanged." We allseem to have the same understanding of relative volatility when wespeak to one another, but trying to put i* on paper evokes disagreement.Suboptimization orDeoptimizationFuzzy logic may bring a better solution than other methods that are nowbeing used to develop trading models. By its very nature, a fuzzy solution must be general. Writing fuzzy rules for a trading program will nothave to be as precise as traditional specifications. That also means thereshould be fewer rules and a more robust solution. No matter how hardwe try, it may be impossible to overfit the solution using fuzzy data.State of the ArtFuzzy systems have been combined with neural networks and expertsystems, which provide a framework for "learning." Neural nets provide the behavioral structure so that correct answers are reinforced andincorrect ones are rejected. Expert systems give the program a knowledge base.Japanese firms have led the financial industry in the application offuzzy expert systems. It is said that programs already exist for financialdealing, especially stock market trading. These models have been basedentirely on price information, but may soon include expectations, or"feelings," about political outcome. It seems that, once the idea is planted, the technology moves forward at a furious pace.
PART 3Making aTrading StrategyRobust
10
Testing forRobustnessA trading strategy is robust if it is successful under many different conditions. It is especially good if it works under situations that are verydifferent from those used in testing, for example, a more volatile moveto new high prices.Many users can blame the speed of the computer for trading systemsthat do not work. Combined with strategy-testing and statistical software,the computer has made it too easy to simulate thousands of trading rulesand techniques. Preprogrammed strategies, countless indicators, and theability to create your own variations often draw inexperienced users intoan indiscriminating and unfocused approach to testing. In the end, thecomputer has tested too much and used too little as criteria for success.Often, the resulting trading programs appear to be remarkably profitablebut in reality are complete failures.
OverfittingA system that has been tailored to work on a specific period of historicdata is called overfit. Everyone who develops a trading system will usepast data to verify the results. It would be irresponsible to define a set oftrading rules, open an account, and begin trading without knowingwhether those rules would have worked in the past. The historic risk willgive you an indication of the investment size needed to achieve yourgoal and survive the interim losses required to reach your objective.A careful study of historic results will often point out an area of highrisk. Sometimes a simple rule is all that is needed to reduce the risk to acomfortable level. For example,177
178
Making a Trading Strategy Robust
Reduce the size of the entry position as the market increases in volatility.Further analysis could lead to other rules:Close out all positions if the S&P drops more than 1500 points in 3 days.
or,Close out all long positions on Friday if the S&P has dropped more than10% during the previous week.
These rules move from a general, logical risk control to very specific guidance intended to isolate one or two past problems. Where do thesechanges stop being reasonable and start being manipulative? The answeris rarely clear.In this chapter, we will consider a system robust if it does not dependon a narrow set of conditions. A profitable 10-day moving average system will not be used if similar 8-day and 12-day systems generate losses. The best system is one that is profitable for a broad set of parameters,including trend speeds, risk control, profit-taking, and filters. The trader gains a more dependable program when nearly any choice of parameters is likely to give profits.
Separating Robustnessfrom Parameter SelectionThere is a clear separation between determining that a trading strategyis valid, and being able to use that method to produce profits in thefuture. Historic testing can verify a premise and show which sets ofparameters, or variables, were successful in the past. But this does notmean the parameters that generated the most profits in the past willlead to future profits. And, when there are two tests with similar historic success, which will be the best?Comprehensive testing, called optimization, results in some cases thatshow profits and others that have losses. The profits show that the logicbehind the strategy is sound. The more cases that are profitable, the moreconfidence we have in the trading method. However, the parametersthat gave profits in historic tests do not always generate profits duringreal trading. The ability to choose, in advance, which parameters will givefuture profits is a separate problem from defining robust trading rules.Most of this chapter will focus on how to build a robust trading system;the more robust it is, the less it will depend on picking the right parameters. We can assume that an arbitrary choice of parameters will yield the
Testing for Robustness
179
average performance; therefore, we should be sure that the average isgood.PrinciplesThe development of a trading model that is independent of parameter selection is always the ideal solution. But it is elusive. It is the opposite conceptof an arbitrage, which takes advantage of distinct economic anomalies. Thischapter will set down a procedure that will greatly improve the robustnessof any trading model. It is not a simple process, but it can be implementedone part at a time. The more that is done, the better the results.No model is without some limitations or restrictions. Each has a purpose that requires some definitions of its operating environment. It iscertainly reasonable to exclude a 3-day moving average from a longterm trading system. Both trading frequency and risk can be narrowedto ranges that make sense for the strategy as well as for the type of business. Exploiting a price pattern is valid if the patterns can be identifiedin advance. It is within this framework that the program can be robust.Some basic principles of parameter selection assure a choice thatyields superior results. These include slower trends and fewer artificialrisk controls. This chapter will draw on previous conclusions to suggesta set of rules and a testing approach that will give test results similar toreal trading.Example of O p t i m i z e dPerformanceTable 10-1 compares test results of a moving average system with a percentage stop-loss. The speed of the moving average (from 5 to 50 days)and the percentage stop-loss (from 0 percent to 2.0 percent) are the twoparameters needed to trade. A TeleTrac optimization was used to findthe returns for each combination of speed and stop for the Hang SengIndex during each of the calendar years 1991 and 1992.If we had used the best results of the 1-year test on 1991 data to selectthe parameters to be traded in 1992, we would have tended toward theslower moving averages. The highest return of 17.4 percent was givenby the 45-day trend. Speeds from 5 to 30 days showed erratic results andonly a few small profits.The 1992 results show nearly the opposite. Moving averages from 5 to20 days had large returns, while speeds from 40 to 50 days had the worstresults. Had we chosen the parameters that posted the highest profits in1991, the 1992 performance would have been only 1.3 percent (if executed perfectly).
Making a Trading Strategy Robust
180Table 10-1. Moving AverageOptimization of the Hang Seng IndexParametersMovingAvg Days510152025
30
35404550
%Stop-Loss
1991 P / L
1992 P / L
Changein P / L
0.01.02.00.01.02.00.01.02.00.01.02.00.01.02.00.01.02.00.01.02.00.01.02.00.01.02.00.01.02.0
4.54.01.6-0.1-0.110.2-11.3-11.5-11.13.43.41.7-3.7-7.0-7.0-13.3-16.4-16.41.01.01.013.012.2-2.917.417.417.4-3.7-3.7-4.6
29.229.229.235.135.135.126.926.926.916.619.819.84.30.91.47.25.95.920.720.720.7-15.9-7.5-7.51.31.31.31.81.81.8
+24.7+25.2+28.0+35.2+35.2+24.9+38.2+38.4+38.0+13.2+ 16.4+18.1+8.0+7.9+8.4+20.5+22.3+22.3+21.7+21.7+21.7-28.9-19.7-4.6-16.1-16.1-16.1+5.5+5.5+5.5
This simple moving average test shows the typicalinconsistency in the performance of the "best" choiceswhen a test uses only a small amount of data. The area ofhighest profits in 1991 produced the worst results in 1992.
The Underlying Method forDetermining RobustnessWe can improve the selection of parameters by focusing on systems thathave the broadest success. If all test cases are profitable, we would havethe perfect robust system and any selection of parameters should returnprofits.
Testing for Robustness
181
To measure which strategies are better than others, we will define testprocedures that measure the results using the average and standard deviation of all tests. The highest average alone is not enough. A smaller standard deviation shows the consistency of performance for all tests. A BestChoice Index combines both values by subtracting the standard deviationfrom the average:Best Choice Index = Average returns - 1 Standard deviation of returns
Because one standard deviation represents a grouping of 68 percentof all data, the Best Choice Index tells us that this system gives an 84 percent chance of a return greater than or equal to the Best Choice Indexvalue. Remember that losses are half the probability, on the left part ofthe distribution curve. For example, if all combinations of test returnsaveraged a rate of return (ROR) of 14 percent with a standard deviationof 6 percent, we have• An 84 percent chance that any choice will yield returnsgreater than 8 percent (the average minus 1 standard deviation).• A 97.5 percent chance that any choice will yield returnsgreater than 2 percent (the average minus 2 standard deviations).• A 