76D-8-33-10-15 | 0.068802 | 0.025 0.023 -0.06302 0.20157D-10-23-20-60 | 0.068752 | 0.0145 | 0.02 -0.06307 0.20152D-7-37-20-30 | 0.068628 | 0.02 0.021 -0.0632 0.2014
E-2-1-10-30 0.068546 | 0.016 0.018 -0.06328 0.20132D-8-31-10-30 | 0.06831 | 0.019 0.023 -0.06351 0.20108D-9-32-10-15 | 0.068112 | 0.0175 | 0.022 -0.06371 0.20088E-5-20-10-30 | 0.068035 | 0.0185 | 0.011 -0.06379 0.2008
 Case Study Results and the Future of TA 447
TABLE 9.1 (Continued)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
E-3-36-10-30 | 0.067922 | 0.0205 | 0.019 -0.0639 0.20069TI-23-41 0.067856 | 0.0225 | 0.019 -0.06397 0.20063E-2-9-10-15 0.067833 | 0.0185 | 0.0215 -0.06399 0.2006
E-7-36-20-60 | 0.067749 | 0.0255 | 0.0205 -0.06408 0.20052TI-21-5 0.067723 | 0.022 0.0205 -0.0641 0.20049E-6-38-10-30 | 0.067453 | 0.0225 | 0.024 -0.06437 0.20022D-6-33-10-15 | 0.067088 | 0.023 0.022 -0.06474 0.19986D-8-34-20-60 | 0.066988 | 0.025 0.023 -0.06484 0.19976E-12-21-20-60 | 0.066859 | 0.0175 | 0.025 -0.06496 0.19963E-1-18-10-60 | 0.066816 | 0.027 0.0235 -0.06501 0.19959D-5-15-10-30 | 0.066604 | 0.024 0.017 -0.06522 | 0.19937
 
 
 
 
 
 
The three inference methods that take data-mining bias into accountwere used:
1. B (boot) is an improved version of WRC, which incorporates the en-hancement suggested by Wolf and Romano’ designed to improve thepower of the test (reduce the chance of a type-II error) as discussed inChapter 6.
2. RC (reality check) is the version of WRC that is currently availablefrom Quantmetrics, which does not incorporate the Wolf and Romanoenhancement.
3. MC is the Monte Carlo permutation method, developed by Masters,which incorporates the Wolf and Romano enhancement.
The upper and lower bounds of the 80 percent confidence intervalswere derived on the basis of the mathematical theory first proposed byWhite,’ using an algorithm that is described in Wolf and Romano. Dr. TimMasters implemented the computer code for this. The confidence inter-vals presented take full account of the data-mining bias. That is to say,they are joint confidence intervals for which upper and lower bounds con-tain the true return of all rules, with a probability of 0.80. This means thatif we were to redo the case study on 1,000 independent sets of data, an ob-vious impracticality given the existence of only one set of historical mar-ket data, in 800 of the tests the 6,402 confidence intervals would containthe expected returns for all rules.
It can also be said that, in 950 of these cases, we would not be fooledinto an erroneous rejection of the null hypothesis, but in 50 we would. Itso happens that in the particular experiment run for the case study, no448 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
rule showed up as significant. However, by setting the significance level to.05, the case study did in fact have a 1 in 20 chance of a type-1 error—thatarule with no actual predictive power would have displayed a p-value lessthan 0.05, thereby resulting in a false rejection of the null hypothesis.
CRITIQUE OF CASE STUDY
 
Positive Attributes
Use of Benchmarks in Rule Evaluation. One thing the case studydid right was using a benchmark to evaluate rule performance. Aspointed out in Chapter 1, the back-tested performance of a rule onlymakes sense in relation to a benchmark. Absolute levels of perfor-mance are uninformative.
The case study used the lowest reasonable benchmark, the perfor-mance of a rule with no predictive power. Under this standard, a rule isdeemed effective if and only if it beats the performance of a nonpredictiverule by a statistically significant margin. Clearly, there are higher bench-marks that could make sense in specific cases. For example, suppose aclaim is made that a superior version of the double moving averagecrossover rule has been developed. A sensible benchmark would be theperformance of a conventional version of the rule.
Controlled for the Effect. of Market Trends. As described inChapter 1, the case study used detrended market data to compute rule re-turns to eliminate performance distortions. As explained, distortions re-sult when a rule with a long- or short-position bias has its returnscomputed on market data that has a net upward or downward trend overthe back-test period.
Controlled for Look-Ahead Bias. Rule studies that assume theavailability of information that is not yet truly available at the time marketpositions are entered or exited are infected with look-ahead bias, alsoknown as future information leakage. For example, if closing price infor-mation is needed to compute a rule’s signals, it would not be legitimate toassume an entry or exit at the closing. The first legitimate price at whichan entry or exit could legitimately be assumed is the next price. If the datais of a daily frequency, as was true for the case study, the first legitimateexecution price does not occur until the market opens the day followingthe signal.
Look-ahead bias can also occur if a rule uses data series that are re-ported with a lag, such as mutual fund cash statistics, or that are subjectCase Study Results and the Future of TA 449
to revisions, such as government economic statistics. When this is thecase, lagged values must be used that take into account reporting delaysor revisions.
The case study avoided look-ahead bias by assuming entries and exitson the open price of the day following a position-reversal signal. More-over, none of data series used were subject to reporting lags or revisions.
Controlled for Data-Mining Bias. Few rule studies in popular TAapply significance tests of any sort. Thus, they do not address the possi-bility that rule profits may be due to ordinary sampling error. This is aserious omission, which is easily corrected by applying ordinary hypoth-esis tests.
However, ordinary tests of significance are only appropriate whenonly one rule has been back tested. When many rules have been testedand a best is selected, the ordinary hypothesis test will make the best ruleappear more statistically significant than it really is (false rejection of thenull hypothesis). Avoiding such type-I errors is the motivation for usingadvanced hypothesis tests such as WRC and Monte Carlo permutationmethod, as described in Chapter 6. The case study used such tests. Aspointed out earlier, both WRC and MCPM revealed that the best-performingrule of the 6,402 tested was not statistically significant.
Controlled for Data-Snooping Bias. Data-snooping bias, whichmight be more properly named prior-research-snooping bias, occurswhen data miners use the results of prior research to choose which rulesto test. In other words, they use rules that have been previously found tobe successful. This is an insidious problem because it is unknown howmany rules were tested to find the successful rules. Because the numberof rules tested is an important factor contributing to the magnitude of thedata-mining bias it is impossible to evaluate the actual statistical signifi-cance of a rule that was included in a new data-mining venture because ithad been successful in prior research efforts.
The case study mitigated the prior-research-snooping bias by not ex-plicitly including any rules that were discovered by other researchers.Though many of the rules examined with the study’s 6,402 rules wereprobably similar to those examined by other researchers, the similaritywas coincidental. As explained in Chapter 8, the set of rules tested werearrived at by combinatorial enumeration of all rules possible within a setof specified parameter values. For example, trend rules were based on 11possible values for the channel breakout look-back parameter and 39 pos-sible time series. All 429 possible rules (11 « 39) were included in the casestudy. If one of these happened to be a rule discovered by some prior ruleresearch, its inclusion was by happenstance and not a perusal of the prior450 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
research. Thus, the case study accounted for the full number of rules ex-amined leading to the discovery of the best-performing rule.
Negative Attributes
Complex Rules Were Not Considered. Perhaps the biggest defi-ciency of the case study was the failure to consider complex rules. Com-plex rules result from combining and condensing, in some fashion, theinformation contained in a multiplicity of simple rules. This is a severe de-fect for several reasons. First, few TA practitioners, subjective or objec-tive, restrict themselves to decisions based on one rule. In this sense, thecase study did not replicate what subjective or objective analysts actuallydo. However, as commented on later in this chapter, subjective analystsare severely constrained in their ability to properly interpret the informa-tion pattern embodied in a multiple rule configuration.
Second, complex rules should produce superior performance on diffi-cult prediction problems, such as financial markets. A nonlinear combina-tion of simple rules allows the complex rule to be more informative thanthe summed information contained in its individual constituents. This al-lows the rule to comply with Ashby’s Law of Requisite Variety,’ which stip-ulates that a problem and its solution must have similar degrees ofcomplexity. In the context of prediction, this implies that a model (rule)intended to predict the behavior of a complex system (e.g., a financialmarket) must also be complex. The general superiority of complex ruleseven of a linear form, was demonstrated in an analysis" of 39,832 rules,both simple and complex. This rule set was tested on four stock marketindexes: Dow Jones, S&P 500, NASDAQ, and Russell 2000 over the period1990 through 2002. Of the entire rule set, 3,180 or 8 percent were complexrules, but of the 229 rules that generated statistically significant profits,188, or 82 percent, were complex rules. This study took account of thedata-mining bias by using WRC to compute significance levels. Inciden-tally, the study showed that none of the rules produced statistically signif-icant gains on either the Dow Jones Industrials or the S&P 500 Index.
The case study was restricted to simple rules to keep its scope man-ageable. However, I did so in the belief that at least a few of the 6,402 ruleswould prove significant. Clearly, I was overconfident.
Only Long/Short Reversal Rules Considered. Also in the interestof keeping the case study manageable, the rule set was restricted to onetype of binary rule; long/short reversal. As discussed in Chapter 1, this re-striction can distort the TA concept a rule is intended to express. This isan acknowledged limitation of binary rules in general and of long/short re-versal rules in particular. The requirement that a rule always hold a mar-Case Study Results and the Future of TA 451
ket position, which is true of reversal rules, rests on the unlikely assump-tion that the market is in a perpetual state of inefficiency and continuallypresents profit opportunities. In contrast, rules that are more selective inidentifying when market exposure is warranted are consistent with themore reasonable assumption that markets are occasionally inefficient.Therefore, tri-state rules, which would allow for long/short/neutral posi-tions, or binary rules, which are long/neutral or short/neutral, may be su-perior. No such rules were examined by the case study.
Rules Limited to S&P 500 Trading. Also in the interest of limitingthe scope of the case study, rules were applied to only one market, theS&P 500 Index. The large rule study’ referred to earlier did not find anyrules, either simple or complex, which were effective on the S&P 500.When I selected S&P 500 as the market to trade, I was unaware of thelarger rule study, which did find useful rules for the NASDAQ and Russell2000. Had I known of the study and decided to use either NASDAQ or Rus-sell 2000 as the target market, I would have been guilty of snooping priorresearch, thereby compromising the validity of the p-values that were ob-tained in the case study. On the other hand, had I chosen the NASDAQ orRussell 2000 without knowledge of that study and found successful rules,they would have been discovered legitimately and the reported p-valueswould have been accurate.
POSSIBLE CASE STUDY EXTENSIONS
 
The aforementioned limitations suggest several possible extensions to thecase study.
Application to Less Seasoned Stock Indexes
The Hsu and Kuan study referred to earlier that found statistically sig-nificant rules for the NASDAQ and Russell 2000 suggested that TA rulesmay be more successful on indexes of less seasoned stocks. Thus, oneextension would be to apply the case study’s rule set to stock market in-dexes of other countries with securities of less-seasoned companies.This would be practical if the raw data series needed to construct theindicators were available, such as advance/decline statistics, up anddown volume, new highs and lows, and such. A literature search uncov-ered a number of studies® that lend credibility to the notion that newerstock markets may be more amenable to prediction than more devel-oped and presumably more efficient markets. One practical limitation of452 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
a newer stock market would be shorter market history and fewer histor-ical observations.
Improved Indicator and Rule Specification
The profitability of a rule is determined by the information content ofthe indicators on which the rule is based. The more effectively an indi-cator quantifies an informative feature of market behavior the higherthe rule’s profit potential. The 6,402 rules used in the case study consid-ered three TA themes: trends, extremes, and divergences. The indica-tors used were thought to be reasonable ways to quantify these themes.No doubt there is room for improvement. As pointed out in a subse-quent section, the most important future role of the TA practitioner willbe developing indicators that quantify market behavior in a more infor-mative manner.
For example, as mentioned in Chapter 8, divergences might be bet-ter measured with indicators based on conintegration.’ Indicators in-tended to measure extremes might prove more useful if they employedthe Fisher transform, discussed by Ehlers.'” He points out that the fre-quency distribution of an indicator based on channel normalization hasan undesirable property, specifically, extreme values are considerablymore likely than values in the middle of the indicator’s range. This ishardly the behavior one would want in an indicator designed to detectrare extremes. However, by applying the Fisher transform to channel-normalized data, Ehlers claims that the resultant indicator is close tonormally distributed. An indicator with a normal distribution has the de-sirable characteristic that its extreme values are rare. This may, in turn,make them more informative. Because Ehlers offers no data supportingevidence of the Fisher transform’s efficacy, at this time it is only an inter-esting conjecture worthy of further investigation.
Consideration of Complex Rules
Complex rules are derived by combining the information in two or moresimple rules in order to augment predictive power. There are two key is-sues in deriving complex rules: Which rules should be combined? andHow should they be combined? The how refers to the mathematical orlogical form used to combine the simple rules’ outputs to produce thecomplex rule’s output.
The simplest method of combining is linear. A linear combination isbased on summing rule the outputs of simple rules. It treats the output ofeach simple rule as if it makes an independent contribution to the outputof the complex rule. That is to say, the contribution of each simple rule isCase Study Results and the Future of TA 453
unaffected by the output of any other rule. The simplest linear combina-tion gives the same weight to each constituent. Thus, a simple linear com-bination of two binary reversal rules would assume a value of —2, whenboth are short; zero, when one is short and one is long; and +2 when bothrules are long. A more sophisticated, though not necessarily superior, lin-ear combination assigns varying weights to each rule, depending upon itspredictive power. The relative weight of each rule is represented by a co-efficient, designated by the letter a. The set of weights—which are best ac-cording to some figure of merit, such as minimum predictive error ormaximum financial performance—can be found by optimization. The gen-eral form of a weighted linear combination is illustrated in the followingequation:
Linear Combining Is Additive
Y = Gg + Ary + A2r2 + wn + Ann
Where:Yis Output of complex linear rulerj is the output of it” ruleaj is the weight for the i” ruledo is a constant - y intercept
In contrast to a linear combination, a nonlinear combination consid-ers interactions between the rules. In other words, the combined value, Y,is not simply a weighted sum of the individual rule values. In a nonlinearcombination, the contribution of any individual rule to the combined out-put Y depends, in some way, on the output values of the other rules. Inalinear combination, each rule’s contribution to Y is unaffected by (inde-pendent of) the value of any other rule. The advantage of a nonlinear com-bination is that it can express more complex information patterns.Thiscreates a disadvantage, however, in that a nonlinear combination may notbe expressible as a neat easily understood equation. Because we are leftin the dark about how the rule outputs are being combined (i.e., interact-ing), nonlinear models are sometimes referred to as black-box formulas.Nonlinear combinations produced by neural network software would bean example of a black-box formula.
The distinction between linear and nonlinear combining is more eas-ily portrayed when one considers continuous variables rather than bi-nary rule outputs. Continuous variables can assume any value within a454 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
range rather than being restricted to discrete values such as those of bi-nary rule (-1 and +1). When a linear model is illustrated graphically, asin Figure 9.4, the response surface, which represents the complex rule’soutput (Y), assumes a flat shape. There are no hills and valleys. Anotherway to say this is that the slope of the response surface, with respect toeach input (X, and X,), remains constant throughout the entire range ofthe input. The slope of the surface with respect to input X, is designated bythe coefficient a,, and the slope with respect to input X, is given by coef-ficient a,. The values a, and a, are typically discovered from a set of databy a statistical technique called regression analysis. The values a, and a,in combination with a third coefficient a,, called the Y intercept, orientthe plane so as to best slice through the set of data points used to esti-mate the model's coefficients. The fact that the slope of the responsesurface is constant (flat) tells us that there are no nonlinear interactionsbetween the inputs. Said differently, the inputs X, and X, are interactingin an additive manner.
In anonlinear model, nonadditive interactions between the inputs areallowed. This means that the value Y is not simply a weighted sum of theindividual input values. In a nonlinear model, the contribution of any in-put (indicator) depends, in some arbitrary way, on the values of the otherinputs. This manifests as a response surface with hills and valleys. Inother words, the slope of the model's response surface with respect to anyindividual input, Xi, will not be constant throughout its range. This varia-tion in the slope over the range of the inputs can produce a response sur-face like the one shown in Figure 9.5.
 
FIGURE 9.4 Y=4,+ 4, x, + a, x,.Case Study Results and the Future of TA 455
 
FIGURE 9.5 Y= F(X, X).
Linear Combinations within a Theme. Despite the simplicity ofcombining rules via addition, complex rules with a linear form haveproven useful. The study conducted by Hsu and Kuan" tried three meth-ods for creating complex rules. Two of the methods were based on linearforms: voting rules and fractional position rules. The third type, whichthey referred to as a learning rule, did not combine rules, but rather wasbased on selecting the simple rule from a set of similar rules that had thebest recent performance. The complex rules based on voting and frac-tional positions were essentially additive combinations of all simple ruleswithin a given theme. Examples of themes would be all rules based onchannel breakouts or all rules based on a moving average of on-balancevolume. The output of a complex rule based on a voting scheme was de-rived by polling all rules within a theme (channel breakout) and taking aone-unit position that reflected the majority vote. For example, if the ma-jority of rules were indicating a short position, a one-unit short positionwas taken. The output of a complex rule based on the fractional-positionconcept was also based on polling all the simple rules within a theme,however, the size of the position taken was proportional to the net per-centage of rules that were long or short ({number long minus numbershort] divided by total number of rules). For example, Hsu and Kuan con-sidered 2,040 on-balance volume rules. If 1,158 were signaling a short po-sition and 882 were signaling a long position, a 0.135 unit short positionwould be taken ((882 — 1,158]/2,040 = —0 .135).
Thus one logical extension of the case study would be to create linearcomplex rules based on voting or fractional-position schemes. This couldbe done for rules within a given theme, for example all divergence rules.456 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
An indicator commonly used in TA, called a diffusion indicator, is basedon a similar idea. An example of a well known diffusion indicator is thepercentage of New York Stock Exchange stocks that are over their 200-day moving averages. In this instance the percentage is essentially an un-weighted linear combination.
The diffusion indicator would simply be the net percentage of diver-gence rules that were long or short. Using a voting scheme, if more than50 percent of the divergence indicators were in a +1 state, a long positionwould be held, else a short position. Using a fractional-position scheme, apartial long or short position would be held and adjusted as the netlong/short percentage changed. It is important to note that the creation ofdiffusion indicators from the 6,402 rules used in the case study would re-quire that inverse rules be eliminated. Their inclusion would cause a diffu-sion indicator to always assume a value of zero. Thus, a diffusionindicator based on the trend-rule category would be constructed onlyfrom TT rules and TI rules would be excluded. Likewise, a diffusion indi-cator based on rules in the extreme and transition category would bebased only on types 1, 2, 3, 4, 5, and 6. Types 7 through 12 would have tobe excluded inasmuch as they are the inverse of types 1 through 6.
Machine-Induced Nonlinear Combinations. Complex rules thatcombine simple rules in a nonlinear or nonadditive fashion could also bederived from the case study’s rules. However, this would require the use ofautonomous machine learning (data-mining) systems such as neural net-works,” decision trees and decision tree ensembles, multiple regressionsplines, kernel regression, polynomial networks,” genetic program-ming,'!° support-vector machines,!” and so on. These systems employ in-ductive generalization to synthesize a complex nonlinear rule (model) bylearning from a large set of historical examples. Each example is a a case,characterized by a set of indicator values as of a given date and the valueof an outcome variable that the model is intended to predict.
Complex nonlinear rules could be derived in two ways. One would beto submit the rules in binary form, as the candidate inputs. A second ap-proach would be to submit the indicators used by the rules.
To the best of my knowledge, there is no product currently availablefor generating complex nonlinear rules that incorporates a significancetest that is robust to the data-mining-bias problem. In the absence of thisprotection, data miners searching for complex nonlinear rules must relyon the protection provided by a sophisticated form of out-of-sample test-ing that involves three data sets (training, testing, and evaluation). This isdescribed later in this chapter.
Allowing the search to optimize a rule’s complexity, rather than re-stricting it to finding the optimal parameter values for a rule whose com-Case Study Results and the Future of TA 457
plexity is fixed, has an upside and a downside. The upside is that allowingcomplexity optimization increases the chance of discovering a superiorrule. The downside is that the added search dimension can greatly in-crease the number of rules considered before a best is selected. This in-creases the chance that the best-performing rule is overfitted. This refersto a rule that not only expresses the sample data’s valid patterns but thatalso describes its random effects or noise. Overfitted rules do not gener-alize well. That is to say, their out-of-sample performance is typicallypoor relative to their in-sample performance. This is an example of data-mining bias.
To guard against the risk of overfitting when complexity optimizationis allowed, three data segments are often used: training, testing, and val-idation. Recall from Chapter 6 that only two data segments are requiredwhen data mining is restricted to finding the optimal parameter values of afixed-complexity" rule: training and testing.
In its search for the nonlinear rule of optimal complexity, machinelearning algorithms cycle through the training and testing sets in two dis-tinct loops. The inner loop searches for the optimal parameters at a givenlevel of complexity. The outer loop searches for the optimal level of com-plexity. Once the best complex rule has been found, it is evaluated in thethird data segment, called the validation set. This data has been held inreserve pending the discovery of the parameter values and complexity de-gree of the best-performing rule. Because the validation set was not uti-lized in this process of discovery, the performance of the best rule in thisdata set is an unbiased estimate of its future performance. This is shownin Figure 9.6.
This three-segment scheme can be conducted on a walk-forward ba-sis. After the best rule has been found in the first data fold (train/ test/vali-dation), the tripart data window slides forward and the entire process isrepeated in the next fold. This is illustrated in Figure 9.7.
The motivation for the tripart data window warrants some additionalexplanation. Market behavior is presumed to be a combination of system-atic behavior (recurring patterns) and random noise. It is always possibleto improve the fit of a rule to a given segment of data by increasing itscomplexity. In other words, given enough complexity, it is always possibleto fashion a rule that buys at every market low point and sells at everymarket high point. This is a bad idea.!® Perfect timing on past data canonly be the result of a rule that is contaminated with noise. In otherwords, perfect signals or anything approaching them almost certainlymeans the rule is, to a disturbing degree, a description of past random be-havior (i.e., overfitted).
Overfitting manifests when the rule is applied to the test data seg-ment. There, its performance will be worse than in the training data. This4538 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
Foldj
Ce aSearchLoop YL > Best
Parameter RuleSearchLoopTraining Set Test Set Validation Set -/—»>Optimal Rule’sPerformanceEvaluated Here[rime ee -
FIGURE 9.6 Searching for optimal parameters and complexity of best rule.
 
Fold4(a Y
|—rime Training Set Test Set Validation
 
 
Fold3
(az )
Fold2
 
Training Set Test Set Validation -/-———————_>
 
 
Fold
FIGURE 9.7  Walk-forward complexity search.Case Study Results and the Future of TA 459
is because the legitimate patterns found in a training set recur in the testset, but the noise in the training set does not. It can be inferred that prof-itability in the training set that does not repeat in the testing set was mostlikely a consequence of overfitting.
During the process of complexity optimization, the first sign that therule has become overfit is when its performance reaches a peak in thetesting set and then starts to degrade. Complexity optimization proceedsas follows. It begins with a low complexity rule, which is run repeatedlyover the training data set using different parameter values. Each run pro-duces a performance figure (e.g., rate of return). The parameter(s) thatyields the highest performance in the training data, let’s call it best-rule 1,is then run on the test data set and its test set performance is noted. Thatcompletes the first complexity loop. The second complexity loop beginswith a rule that is a notch higher in complexity than best-rule 1. That is tosay, it has a greater number of parameters. Again, there is a search for thebest parameter values of this more complex rule in the training set. Thebest rule at this complexity level, let’s call it best-rule 2, will almost cer-tainly perform better in the training set than best-rule 1 because of its ad-ditional complexity. Best-rule 2’s performance is then measured in the testset. This process continues as a succession of rules of increasing com-plexity are mined from the training set and evaluated in the testing set.Typically, performance on the test improves as the rule’s complexity is in-creased. The improvement is an indication that ever-more-intricate butvalid patterns in the training set have been discovered. However, at a cer-tain point in the rule’s complexity growth, the performance in the testingset will start to decline. This indicates that the most recent increment incomplexity has started to describe the random effects (nonrecurringnoise) in the training set. In other words the rule has now become overfit-ted. Had the rule’s complexity growth been stopped