-volved pairing the S&P 500 with 38 other time series. Given that theseseries had varying degrees of co-movement with the S&P 500, the fluctua-tion range of the divergence indicator would vary considerably from onepair to the next. This would make it impractical to use the same thresholdfor all pairings. This problem is illustrated in Figure 8.29. Note that thehigh threshold displacement that would be suitable for a companion se-Case Study of Rule Data Mining for the S&P 500 437
+100
Companion Series Has High Degree of Co-Movement with S&P 500
 
Companion Series Has Low Degree ofCo-Movement with S&P 500
   
-100
FIGURE 8.29 Divergence indicator: inconsistent volatility.
ries with a low degree of co-movement with the S&P 500 would never pro-duce a signal for a companion series with a high degree of co-movementto the S&P 500. For this reason, the initial formulation of the divergenceindicator was deemed impractical.
This problem was addressed with a modified formulation of the diver-gence indicator illustrated in the equation that follows. It employs thechannel normalization operator twice. That is to say, the indicator is achannel-normalized version of the initial divergence indicator.
Divergence Indicator(Double Channel Normalization)
= CN{CN (Series 1, n) —- CN (S&P 500, n), 10n}
Where:CN = Channel normalization operatorSeries | = Companion seriesn= Look-back span of the first channel normalization438 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
The second layer of channel normalization takes into account thefluctuation range of the initial formulation of the divergence indicator.This solves the problem of inconsistent fluctuation ranges across the 38pairs of time series. As a result, the modified version of the divergence in-dicator will have roughly the same fluctuation range irrespective of theparticular pair of time series being used, making it practical to use uni-form thresholds.
The look-back span for the second level of channel normalization wasset at 10 times the look-back interval used for first level. Thus, if the chan-nel normalization used a look-back span of 60 days, the second layer ofchannel normalization used a look-back span of 600 days. It was assumedthat a 10-fold look-back span would be sufficient to establish the fluctua-tion range of the basic divergence indicator. Note that a modified diver-gence indicator has a potential fluctuation range of 0 to 100, similar to anychannel-normalized variable. See Figure 8.30.
Divergence Rule Types. Upper and lower threshold were applied tothe modified divergence indicator to generate signals. A positive or bull-ish divergence was in effect when the divergence indicator was aboveits upper threshold. This occurred when the companion series wasmoving up more or moving down less than the S&P 500. This was evi-denced by the companion series having a higher relative position withinits channel than the S&P 500. Conversely, a negative or bearish diver-gence existed when the divergence indicator was below the lowerthreshold. This occurred when the companion series was moving downmore or moving up less than the S&P 500, resulting in its having a lowerrelative position within its channel than the S&P 500. This is illustratedin Figure 8.30.
Positive Divergences
 
Negative Divergences
FIGURE 8.30 Modified divergence indicator.Case Study of Rule Data Mining for the S&P 500 439
The question was how to create signaling rules from the modified di-vergence indicator. Two rules seemed obvious: a bullish divergence rule,which would call for long positions in the S&P 500 when the divergenceindicator was above the upper threshold, and a bearish divergence rule,which would call for short positions in the S&P 500 when the divergenceindicator was below its lower threshold.
Both rules assumed that the companion series had leading informa-tion about the S&P 500. If the companion series was stronger than theS&P 500 (positive divergence), then a long position would be justified,whereas if the companion series was weak (negative divergence) a shortposition would be justified. However, this assumption may be incorrect. Itmay be the case that negative divergences are predictive of higher pricesfor the S&P 500, whereas positive divergences are predictive of lowerprices. In other words, perhaps it is the S&P 500 that has the leading infor-mation. This suggests that inverse versions of each rule should be tried. Inthe end, it was determined that there are 12 possible divergence ruletypes, and all were tested.
These 12 rule types are exactly the same set used for the extremevalue and transition rules. This makes sense because the modified diver-gence indicator is similar to the indicator used for the E rules because ithas a fluctuation range of 0 to 100 and has two thresholds.
The 12 divergence rule types, presented in Table 8.4, include the basicbullish divergence (type 6), the bearish divergence (type 7) and their in-versions (types 12 and 1). The 12 types are not illustrated because the il-lustrations would be redundant of those presented in the section onExtreme and Transition Rules.
Parameter Combinations and Naming Convention for Diver-gence Rules. Each divergence rule is defined by four parameters:type, companion series, threshold displacement, and channel normaliza-tion look-back span. There are 12 types of the divergence rules (see Table8.4), 38 companion data series, 2 threshold displacement values—10 and20, and 3 look-back spans—15, 30, and 60 days. This gives a total of 2,736divergence rules (12 x 38 x 2 x 3)
The naming convention used for reporting results of divergence orD rules in Chapter 9 is as follows: (D)-type-companion series-thresholddisplacement-channel normalization look back span. Thus, a rule isnamed
D-3-23-10-30
Divergence rule, type 3, companion series 23 (positive volume in-dex 30-day moving average), threshold displacement = 10 (upper440 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
TABLE 8.4 Divergence Rules and Associated Threshold Events
 
 
 
 
 
 
 
 
 
 
 
 
Divergence Rule Types Long Entry/Short Exit Short Entry/Long Exit1 Down Cross Upper CrossLower Threshold Lower ThresholdDown Cross Upper Cross2 Lower Threshold Upper ThresholdDown Cross Down Cross3 Lower Threshold Upper ThresholdUp Cross Up Cross4 Lower Threshold Upper ThresholdUp Cross Down Cross5 Lower Threshold Upper ThresholdUp Cross Down Cross6 Upper Threshold Upper ThresholdUp Cross Down Cross7 Lower Threshold Lower ThresholdUp Cross Down Cross8 Upper Threshold Lower ThresholdDown Cross Down Cross9 Upper Threshold Lower ThresholdUp Cross Up Cross10 Upper Threshold Lower ThresholdDown Cross Up Cross11 Upper Threshold Lower ThresholdDown Cross Up Cross12 Upper Threshold Upper Threshold
 
 
threshold = 60, lower threshold = 40), 30-day channel normalizationlook-back span.
This completes the description of the rules tested in the case study.Results are presented in Chapter 9. 
Case StudyResults and theFuture of TA
 
PRESENTATION OF RESULTS
 
The primary objective of the case study was to demonstrate the applica-tion of two statistical inference methods suitable for the evaluation ofrules discovered by data mining. As explained in Chapter 6, traditional sig-nificance tests are not suitable because they do not take into account thebiasing effect of data mining. The two methods used were White’s realitycheck (WRC) and the Monte Carlo permutation (MCP).
A secondary objective of the case study was the possible discovery ofrules with statistically significant returns when applied to the S&P 500 In-dex. Toward this end, a set of 6,402 rules described in Chapter 8 wereback tested and evaluated.
With respect to the primary objective, the case study resoundinglydemonstrated the importance of using significance tests designed to copewith data-mining bias. With respect to the second objective, no rules withstatistically significant returns were found. Specifically, none of the 6,402rules had a back-tested mean return that was high enough to warrant a re-jection of the null hypothesis, at a significance level of 0.05. In otherwords, the evidence was insufficient to reject a presumption that none ofthe rules had predictive power.
The rule with the best performance, E-12-28-10-30,! generated a meanannualized return of 10.25 percent, on detrended market data. In Figure9.1, the rule’s return is compared to the sampling distribution produced byWRC. The p-value of the return is 0.8164, far above the 0.05 level set as thesignificance threshold. Figure 9.1 makes it clear that the performance of
441442 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
Best Rule0.10Return 10.25%}
 
0.09
 
0.08
mma
 
0.070.060.05
| |sos |_—__/
7,
7,
Oman
0.03
0.020.01
 
era.  —s|
8% 10% 12% 14% 16%Annualized Return
FIGURE 9.1 Sampling distribution for best of 6,402 worthless rules usingWhite’s reality check.
the best rule out of the 6,402 examined falls well within the range of ordi-nary sampling variability. In fact, it even falls below the sampling distribu-tion’s central value of 11 percent. Quite disappointing!”
The sampling distribution was generated by 1,999 replications of thebootstrap procedure. Had a larger number of replications been used, thedistribution’s shape would have been smoother, but the conclusion wouldhave been the same: the rule’s performance was not high enough to rejectthe null hypothesis.
Figure 9.2 shows E-12-28-10-30 plotted on the sampling distributionproduced by MCP. It produced a similar p-value of 0.8194.
Note that both sampling distributions are centered at approximatelyan 11 percent mean return. This particular value was a consequence of thespecifics of the case study; the particular set of 6,402 rules, their returncorrelations, the number of observations used to compute their mean re-turns, and the particular set of S&P data. In other words, the expected re-turn for the best rule of 6,402 competing rules with no predictive powerunder these specific conditions is approximately 11 percent. It is not 0percent.
To recap a point made in Chapter 6, the sampling distribution for themean return of the best-performing rule out of N rules (e.g., N = 6,402) isCase Study Results and the Future of TA 443
 
0.125Best RuleReturn 10.25%
R 0.10EL
0.075FR£0.05Q
0.025
P-value: 0.8194
 
8% 10% 12% 14% 16% 18%
Annualized Return
FIGURE 9.2 Sampling distribution for best of 6,402 worthless rules usingMonte Carlo permutation.
not centered at zero. Rather, it is situated at a more positive value that re-flects the sampling distribution for a statistic defined as the maximummean of N means.
Figures 9.1 and 9.2 give an indication of what rate of return wouldhave been required to be statistically significant. Returns in excess of 15percent would have been significant at the .05. Returns in excess of 17percent would have been highly significant (p-value < .001).
Ironically, the failure of any rule to generate statistically significantreturns, after adjustment for data-mining bias, underscores the huge im-portance of using statistical inference methods that take the biasing ef-fects of data mining into consideration. Had I used an ordinarysignificance test, which pays no attention to data-mining bias, the meanreturn of the best rule would have appeared to be highly significant(a p-value of 0.0005). This is dramatically illustrated in Figure 9.3. Itshows the bootstrapped sampling distribution appropriate for a singlerule back test (data-mining bias ignored). In contrast to the previoustwo Figures 9.1 and 9.2, the sampling distribution in Figure 9.3 iscentered at zero. The arrow represents the mean return of the best ruleE-12-28-10-30.444 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
 
Best RuleReturn: 10.25%R P-value: 0.0005ELFREQ
 
-15% -10% -5% 0% 5% 10% 15%
FIGURE 9.3 Sampling distribution appropriate for a single rule back test.
Had a conventional test of significance been used, about 320 of the6,402 rules would have appeared to be significant at the 0.05 level. This isexactly what would be predicted to occur by chance. The naive dataminer using a conventional test of significance would have concluded thatmany rules with predictive power had been discovered. In reality, miningoperations conducted in this fashion would have produced nothing butfool’s gold.
The full set of rule test results can be found on the web sitewww.evidencebasedta.com. The results for the 100 rules with the high-est mean returns are presented in Table 9.1. The columns are: column1—the rule’s code according to the naming convention established inChapter 8, column 2—the rule’s mean return on detrended S&P dataover the back-test period, columns 3 and 4—the single-rule p-values viabootstrap and Monte Carlo (note these p-values do not take account ofthe data-mining bias), columns 5, 6, and 7—significance at the 0.05level, using three different inference methods in which data-mining biasis taken into account. Had any of the rules been significant, it wouldhave been indicated by an asterisk. None were significant, hence thereare no asterisks. Columns 8 and 9 are the lower and upper 80 percentconfidence intervals.TABLE 9.1 The Best Performing 100 Rules
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Rule ID Return SR Boot| SR MC |B RC | MC | Lower 80%) Upper 80%p-value | p-value| Sig | Sig | Sig | Cl clE-12-28-10-30 | 0.102501 | 0.0015 | 0.002 -0.02932 0.23527D-8-4-10-60 0.102395 | 0.001 0.0015 -0.02943 0.23516E-1-28-10-15 0.100562 | 0.0025 | 0.0005 -0.03126 0.23333E-12-21-20-15 | 0.098838 | 0.0005 | 0.0015 -0.03299 0.23161E-12-28-10-15 | 0.097502 | 0.001 0.0025 -0.03432 0.23027E-12-24-20-15 | 0.095904 | 0.0005 | 0.001 -0.03592 0.22867E-11-28-10-15 | 0.094635 | 0.003 0.002 -0.03719 0.2274E-11-39-10-60 | 0.092282 | 0.001 0.0015 -0.03954 0.22505D-7-37-10-30 | 0.092078 | 0.0025 | 0.003 -0.03975 0.22485D-7-37-10-15 | 0.090196 | 0.0045 | 0.004 -0.04163 0.22296TT-37-5 0.089024 | 0.0045 | 0.003 -0.0428 0.22179TI-20-41 0.08862 0.0045 | 0.0045 -0.0432 0.22139E-1-22-10-60 | 0.088591 | 0.0055 | 0.007 -0.04323 0.22136TI-21-8 0.086963 | 0.0035 | 0.0075 -0.04486 0.21973D-10-28-10-30 | 0.086939 | 0.004 0.0025 -0.04489 0.21971D-11-28-10-15 | 0.085266 | 0.0055 | 0.0055 -0.04656 0.21804D-7-38-10-30 | 0.085206 | 0.005 0.006 -0.04662 0.21798TI-28-12 0.084912 | 0.0035 | 0.0035 -0.04691 0.21768TI24-8 0.084021 | 0.007 0.005 -0.0478 0.21679E-2-28-10-15 0.083267 | 0.0075 | 0.0025 -0.04856 0.21604E-5-36-10-30 0.083087 | 0.0055 | 0.0065 -0.04874 0.21586D-9-34-10-30 | 0.081577 | 0.008 0.009 -0.05025 0.21435E-11-15-20-60 | 0.08058 | 0.007 0.0075 -0.05124 0.21335D-1-21-10-30 | 0.080499 | 0.008 0.008 -0.05132 0.21327D-8-32-10-15 | 0.08048 | 0.006 0.0085 -0.05134 0.21325D-7-36-10-30 | 0.079943 | 0.0065 | 0.0055 -0.05188 0.21271D-8-27-10-30 | 0.079601 | 0.007 0.0085 -0.05222 0.21237E-12-22-10-30 | 0.077796 | 0.009 0.008 -0.05403 0.21057D-1-24-10-30 | 0.077582 | 0.01 0.0125 -0.05424 0.21035D-7-36-20-30 | 0.077312 | 0.0095 | 0.0135 -0.05451 0.21008D-8-33-20-15 | 0.077101 | 0.0095 | 0.0085 -0.05472 0.20987E-1-19-10-60 0.077092 | 0.0075 | 0.0125 -0.05473 0.20986D-8-32-20-15 | 0.076434 | 0.0105 | 0.009 -0.05539 0.2092TI-8-18 0.076144 | 0.009 0.01 -0.05568 0.20891TI-26-137 0.075767 | 0.009 0.0105 -0.05606 0.20854TI-24-5 0.075732 | 0.008 0.009 -0.05609 0.2085D-7-36-10-60 | 0.0755 0.0105 | 0.0105 -0.05632 0.20827E-1-28-20-15 0.07541 0.013 0.0135 -0.05641 0.20818E-6-38-20-30 | 0.074831 | 0.0085 | 0.0125 -0.05699 0.2076E-12-24-10-15 | 0.074809 | 0.0115 | 0.0135 -0.05701 0.20758TT-37-3 0.074756 | 0.0125 | 0.01 -0.05707 0.20753TT-38-5 0.074579 | 0.0135 | 0.012 -0.05724 0.20735(Continued)
445446 CASE STUDY: SIGNAL RULES FOR THE S&P 500 INDEX
TABLE 9.1 (Continued)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
E-11-21-10-15 | 0.074168 | 0.0145 | 0.011 -0.05766 0.20694D- 10-60 0.07409 | 0.0125 | 0.0125 -0.05773 0.20686E-1-25-10-60 | 0.073966 | 0.0155 | 0.01 -0.05786 0.20674E-1-18-20-60 | 0.073439 | 0.015 0.0155 -0.05838 0.20621E-11-25-10-15 | 0.073131 | 0.01 0.018 -0.05869 0.2059
D-10-32-20-15 | 0.072548 | 0.0205 | 0.0145 -0.05928 0.20532E-5-38-10-30 | 0.07245 | 0.014 0.014 -0.05937 0.20522D-11-28-10-30 | 0.07245 | 0.0185 | 0.013 -0.05937 0.20522E-11-15-20-15 | 0.072363 | 0.012 0.018 -0.05946 0.20513E-2-24-20-15 | 0.072286 | 0.018 0.011 -0.05954 0.20505D-9-34-10-60 | 0.072065 | 0.012 0.016 -0.05976 0.20483D-7-11-10-15 | 0.071859 | 0.0135 | 0.0115 -0.05996 0.20463D-8-37-20-60 | 0.071562 | 0.0205 | 0.022 -0.06026 0.20433E-2-21-10-15 0.071546 | 0.0175 | 0.015 -0.06028 0.20432E-6-37-20-15 0.071509 | 0.016 0.0135 -0.06031 0.20428TI-12-12 0.071403 | 0.0135 | 0.0095 -0.06042 0.20417E-12-21-10-15 | 0.07114 | 0.016 0.0195 -0.06068 0.20391TI-23-5 0.071013 | 0.0155 | 0.0115 -0.06081 0.20378E-2-1-20-30 0.070897 | 0.016 0.013 -0.06093 0.20367TT-36-5 0.070709 | 0.0235 | 0.017 -0.06112 0.20348E-6-38-20-15 | 0.070681 | 0.017 0.021 -0.06114 0.20345TH21-12 0.070672 | 0.017 0.0105 -0.06115 0.20344E-1-21-10-15 | 0.070672 | 0.02 0.016 -0.06115 0.20344D-8-38-20-30 | 0.070308 | 0.02 0.0225 -0.06152 0.20308D-10-4-10-60 | 0.070298 | 0.0145 | 0.016 -0.06153 0.20307TI-32-61 0.070203 | 0.0215 | 0.0135 | -0.06162 0.20297E-2-19-10-60 | 0.070192 | 0.015 0.016 -0.06163 0.20296TI-18-8 0.069985 | 0.019 0.0135 -0.06184 0.20275TT-36-3 0.069962 | 0.0185 | 0.0195 -0.06186 0.20273D-8-23-20-60 | 0.069884 | 0.0175 | 0.019 -0.06194 0.20265D-7-16-10 -30 | 0.069749 | 0.013 0.0145 -0.06207 0.20252D-7-34-10-60 | 0.069721 | 0.016 0.0235 -0.0621 0.20249E-2-28-20-5 0.069572 | 0.017 0.02 -0.06225 0.20234D-9-37-10-30 | 0.069551 | 0.0235 | 0.0195 -0.06227 0.20232D-9-29-10-60 | 0.069538 | 0.013 0.0185 -0.06229 0.20231D-9-36-20-30 | 0.069506 | 0.0225 | 0.018 -0.06232 0.20228D-8-36-10-60 | 0.068993 | 0.018 0.022 -0.06283 0.201