is well positioned to play this role.
Indicator design and target specification is a huge topic suitable forentire books. There are many books that cover indicators that are specificto TA.» There is no need to duplicate that here. There are other books thattreat the more general problem of designing inputs (predictive variables)that are suitable for specific types of data-mining algorithms. Differentalgorithms have different requirements. For example, although it is cru-cial that inputs to a neural net be properly scaled, inputs to decision-treealgorithms need not be. Masters,” Pyle,** and Weiss and Indurkhya™ coverthese issues nicely.
Indicator design refers to the specification of transformations appliedto raw market data before it is presented to the data-mining tool. Dataminers often refer to this as preprocessing. The task of proposing a list ofinformation-rich and properly designed candidate indicators for a data-mining problem requires the domain expertise and inventive talents of thehuman analyst.
Anyone who has gotten their hands dirty in the data-mining trenchesknows the following truth: a necessary condition for success in data min-ing is a good set of candidate indicators. Good does not mean that all in-puts have to be informative. However, some of the proposed variablesmust contain useful information about the target variable. According toDorian Pyle, a noted authority on data mining, “after finding the rightproblem to solve, data preparation is often the key to solving the problem.It can easily be the difference between success and failure, between use-able insights and incomprehensible murk, between worthwhile predic-tions and useless guesses.” Dr. Timothy Masters, another data-miningauthority, says, “Preprocessing is the is the key to success... even a prim-itive prediction model can perform well if the variables have been pre-processed in such as way as to clearly reveal the important information.”®!The message is amplified by Weiss and Indurkhya, in their book Predic-tive Data Mining. “The greatest gains in performance are often made bydescribing more predictive features (indicators). It is the human whospecifies the set of features . . . and who investigates how to transform theoriginal features into better features.” “The computer is smart about delet-ing weak features, but relatively dumb in the more demanding task ofCase Study Results and the Future of TA 473
composing new features or transforming raw data into more predictiveforms ... the composition of features is a greater determining factor inthe quality of results than the specific prediction methods used to producethose results. In most instances, feature composition is dependent onknowledge of the application.”
The TA practitioner who understands data-mining methods and theimportant issues in indicator design will be well positioned to play thiscrucial role in the human-machine partnership of twenty-first-centurytechnical analysis. With the computer as an intelligence amplifier and ascientific orientation as the mindset, the EBTA practitioner is uniquelywell positioned to push the frontiers of market knowledge forward at anaccelerated pace. Given that human intelligence is essentially unchanging,but computer intelligence is increasing at an exponential rate, no otherapproach to TA makes sense. 
Proof ThatDetrending IsEquivalent to
Benchmarking Basedon Position Bias
 
one should compute the expected return of a random system with thesame long-short bias, and subtract this bias-related return from the re-turn of the candidate system.The expected value of a single raw return drawn from the populationof historical raw returns is, by definition, the mean of these returns.
[: order to compensate for market bias interacting with long-short bias,
iEw =, DR, (AL)ist
Let P, signify the position of a trading system at opportunity 7. Thiswill be long, short, or neutral. The expected return of a trading system inwhich positions are random is:
Eandom = ¥ Enaw — ¥ Enaw (A.2)
P.=Long P,=Short
The total return of the candidate system is the sum of the raw returnsat those times when the candidate system is long, minus the sum of theraw returns when the system is short:
Total = Y R- YR (A3)
iP,=Long Pj=Short
475476 DETRENDING IS EQUIVALENT TO BENCHMARKING BASED ON POSITION BIAS
The total return of the candidate system (Equation (A.3)) is correctedfor long/short prejudice by subtracting the expectation of a similarly prej-udiced random system (Equation (A.2)). Subtraction distributes over thesummation, giving:
Corrected = y (R, — Egay) — XY (R, — Enay ) (AA)
Pj=Long P,=Short
Notice that the corrected return of Equation (A.4) is identical to theuncorrected return of Equation (A.3) except that the mean of the raw re-turns is subtracted from each individual raw return. In other words, to re-move the effect of long/short imbalance in a biased market, all you needto do is center the raw returns. 
Notes
 
 
 
INTRODUCTION
1.
ee
-1S)
_wo
—=
15.
 
FH Somrn amy
Data typically considered by TA includes prices of financial instruments;trading volume; open interest, in the case of options and futures; as well asother measures that reflect the attitudes and behavior of market participants.J. Hall, Practically Profound: Putting Philosophy to Work in Everyday Life(Lanham, MD: Rowman & Littlefield Publishers, 2005).
Ibid., 4.
Ibid., 5.
Ibid., 81.
R.D. Edwards and J. Magee, Technical Analysis of Stock Trends, 4th ed.(Springfield, MA: John Magee, 1958).
. For a complete description of Elliott wave theory see R.R. Prechter and A.J.
Frost, Elliott Wave Principle (New York: New Classics Library, 1998).
. Any version of these methods that has been made objective to the point
where it is back testable would negate this criticism.
The professional association of technical analysts, the Market TechniciansAssociation (MTA), requires compliance with the National Association ofSecurities Dealers and the New York Stock Exchange. These self-regulatingbodies require “that research reports have a reasonable basis and no un-warranted claims.” Going even further, the MTA requires of its membersthat they “shall not publish or make statements concerning the technicalposition of a security, a market or any of its components or aspects un-less such statements are reasonable and consistent in light of the avail-able evidence and the accumulated knowledge in the field of technicalanalysis.”
Some peer-reviewed academic journals include Journal of Finance,Financial Management Journal, Journal of Financial Economics, Jour-nal of Financial and Quantitative Analysis, and Review of FinancialStudies.
477478
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
NOTES
Outside of academia, there has been a move to greater emphasis on objec-tive methods of TA, but often the results are not evaluated in a statisticallyrigorous manner.
ED. Arditti, “Can Analysts Distinguish Between Real and Randomly Gener-ated Stock Prices?,” Financial Analysts Journal 34, no. 6 (November/December 1978), 70.
J.J. Siegel, Stocks for the Long Run, 2nd ed. (New York: McGraw-Hill, 1998),243.
G.R. Jensen, R.R. Johnson, and J.M. Mercer, “Tactical Asset Allocation andCommodity Futures: Ways to Improve Performance,” Journal of PortfolioManagement 28, no. 4 (Summer 2002).
C.R. Lightner, “A Rationale for Managed Futures,” Technical Analysis of Stocks& Commodities (2003). Note that this publication is not a peer-reviewed jour-nal but the article appeared to be well supported and its findings were con-sistent with the peer-reviewed article cited in the prior note.
P.-H. Hsu and C.-M. Kuan, “Reexamining the Profitability of Technical Analy-sis with Data Snooping Checks,” Journal of Financial Economics 3, no. 4(2005), 606-628.
R. Gency, “The Predictability of Security Returns with Simple TechnicalTrading Rules,” Journal of Empirical Finance 5 (1998), 347-349.
N. Jegadeesh, “Evidence of Predictable Behavior of Security Returns,” Jowr-nal of Finance 45 (1990), 881-898.
N. Jegadeesh and S. Titman, “Returns to Buying Winners and Selling Losers:Implications for Stock Market Efficiency,” Journal of Finance 48 (1993),65-91.
T.J. George and C.-Y. Hwang, “The 52-Week High and Momentum Investing,”Journal of Finance 59, no. 5 (October 2004), 2145-2184.
B.R. Marshall and R. Hodges, “Is the 52-Week High Momentum StrategyProfitable Outside the U.S.?” awaiting publication in Applied FinancialEconomics.
C.L. Osler, “Identifying Noise Traders: The Head and Shoulders Pattern inU.S. Equities,” Staff Reports, Federal Reserve Bank of New York 42 (July1998), 39 pages.
L. Blume and D. Easley, “Market Statistics and Technical Analysis: The Roleof Volume,” Journal of Finance 49, no. 1 (March 1994), 153-182.
V. Singal, Beyond the Random Walk: A Guide of Stock Market Anomaliesand Low-Risk Investing (New York: Oxford University Press, 2004). Theseresults are discussed in Chapter 4, “Short Term Price Drift.” The chapter alsocontains an excellent list of references of other research relating to thistopic.
A.M. Safer, “A Comparison of Two Data Mining Techniques to Predict Abnor-mal Stock Market Returns,” /ntelligent Data Analysis 7, no. 1 (2003), 3-14;G. Armano, A. Murru, and F. Roli, “Stock Market Prediction by a Mixture ofGenetic-Neural Experts,” International Journal of Pattern Recognition &Artificial Intelligence 16, no. 5 (August 2002), 501-528; G. Armano, M.Marchesi, and A. Murru, “A Hybrid Genetic-Neural Architecture for Stock In-Notes 479
dexes Forecasting,” Information Sciences 170, no. 1 (February 2005), 3-33;T. Chenoweth, Z.O. Sauchi, and S. Lee, “Embedding Technical Analysis intoNeural Network Based Trading Systems,” Applied Artificial Intelligence 10,no. 6 (December 1996), 523-542; S. Thawornwong, D. Enke, and C. Dagli,“Neural Networks as a Decision Maker for Stock Trading: A Technical Analy-sis Approach,” International Journal of Smart Engineering System Design5, no. 4 (October/December 2003), 313-325; A.M. Safer, “The Application ofNeural-Networks to Predict Abnormal Stock Returns Using Insider TradingData,” Applied Stochastic Models in Business & Industry 18, no. 4 (October2002), 380-390; J. Yao, C.L. Tan, and H.-L. Pho, “Neural Networks for Techni-cal Analysis: A Study on KLCI,” International Journal of Theoretical & Ap-plied Finance 2, no. 2 (April 1999), 221-242; J. Korczak and P. Rogers, “StockTiming Using Genetic Algorithms,” Applied Stochastic Models in Business& Industry 18, no. 2 (April 2002), 121-135; Z. Xu-Shen and M. Dong, “CanFuzzy Logic Make Technical Analysis 20/20?,” Financial Analysts Journal60, no. 4 (July/August 2004), 54-75; J.M. Gorriz, C.G. Puntonet, M. Salmeron,and J.J. De la Rosa, “A New Model for Time-Series Forecasting Using RadialBasis Functions and Exogenous Data,” Neural Computing & Applications13, no. 2 (2004), 100-111.31. This firm was acquired by Goldman Sachs in September 2000.
CHAPTER 1 Objective Rules andTheir Evaluation
1. A long position in a security means the investor owns the security and hopesto benefit by selling it in the future at a higher price.
2. A short position in a security means the investor has sold the security with-out owning it but is obligated to buy it back at a later point in time. Theholder of a short position therefore benefits from a subsequent price decline,thereby permitting the repurchase at a lower level than the selling price,earning a profit.
3. Neutral refers to the case where the investor holds no position in the mar-ket.
4. It is assumed that all market information required by the method is knownand publicly available at the time the method produces the recommendation.
5. The possibility of the time series being equal to the moving average is elimi-nated by computing the value of the moving average to a greater degree ofprecision than the price level.
6. J. Bollinger, Bollinger on Bollinger Bands (New York: McGraw-Hill, 2002).
7. T. Hayes, The Research Driven Investor: How to Use Information, Data andAnalysis for Investment Success (New York: McGraw-Hill, 2001), 63.
8. A diffusion indicator is based on an analysis of numerous market time serieswithin a defined universe (e.g., all NYSE stocks). The same rule, such as amoving average cross, is applied to all the series comprising the universe.Each series is rated as in an uptrend or downtrend, depending on its position480
10.
ll.12.13.14.
15.
NOTES
relative to its moving average. The value of the diffusion indicator is the per-centage of time series that are in an upward trend. The indicator is confinedto the range 0 to 100.
This analysis does not make clear how many rule variations were exploredto attain this level of discrimination. As will be pointed out in Chapter 6, it isimpossible to evaluate the significance of these findings without informationon the amount of searching that led to the discovery of a rule.
Many of the rules tested in this book use data series other than the S&P 500to generate signals on the S&P 500.
This law of statistics is discussed in detail in Chapter 4.
(0.9) X 0.035% — (0.1) X 0.035% = 0.028% per day or 7.31% annualized.
(0.60) x 0.035% — (0.40) X 0.035% = 0.007% per day or 1.78% annualized.
The look-ahead bias is discussed by Robert A. Haugen, The Inefficient StockMarket: What Pays Off and Why (Upper Saddle River, NJ: Prentice-Hall,1999), 66.
The percentage of fund portfolios invested in interest-bearing cash instru-ments such as T-bills, commercial paper, and so forth.
CHAPTER 2 The Illusory Validity of Subjective
1.
2.
Technical Analysis
J. Baron, Thinking and Deciding, 3rd ed., (Cambridge, UK: Cambridge Uni-versity Press, 2000), 12.
This figure, redrawn by the author, is similar to and inspired by one found inG. Gigerenzer, Calculated Risks: How to Know When Numbers Deceive You(New York: Simon & Schuster, 2002). That book cites as the original sourceof the illustration R.N. Shepard, Mind Sights: Original Visual Illusions(New York: W.H. Freeman & Company, 1990).
T. Gilovich, How We Know What Isn't So: The Fallibility of Human Reasonin Everyday Life (New York: Free Press, 1991).
See “Defying Psychiatric Wisdom, These Skeptics Say Prove It,” New YorkTimes (March 9, 2004), F1-F3.
Alpha refers to the capital asset pricing model term, which represents theY-intercept obtained by regressing the returns of an investment strategyon the returns of a market index. It represents the portion of return that isnot attributable to the volatility of the strategy (its beta). Meritorious in-vestment strategies have alphas that are positive to a statistically signifi-cant degree.
J. Murphy, Technical Analysis of the Financial Markets: A ComprehensiveGuide to Trading Methods and Applications (New York: New York Instituteof Finance, 1999), 20, where market trends are said to be clearly visible.
HV. Roberts, “Stock Market ‘Patterns’ and Financial Analysis: Methodologi-cal Suggestions,” Journal of Finance 14, no. 1 (March 1959), 1-10.
FD. Arditti, “Can Analysts Distinguish Between Real and Randomly Gen-erated Stock Prices?,” Financial Analysts Journal 34, no. 6 (November/December 1978), 70. An informal test of the same nature is discussed byNotes 4381
11.12.13.14.15.16.17.18.19.20.
21.
22.
23.
24.
26.
J.J. Siegel, Stocks for the Long Run, 3rd ed. (New York: McGraw-Hill,2002), 286.
Gilovich, How We Know.
Shermer, M., Why People Believe Weird Things: Pseudoscience, Supersti-tion, and Other Confusions of Our Time (New York: W.H. Freeman, 1997).Gilovich, How We Know.
Ibid., 10.
H.A. Simon, “Invariants of Human Behavior,” Annual Review of Psychology41 (January 1990), 1-20.
Shermer, Why People Believe, 26.
Ibid., 26.
C. Sagan, The Demon-Haunted World: Science as a Candle in the Dark(New York: Random House, 1995), 6.
C. Sagan.
U. Neisser, Cognitive Psychology (Englewood Cliffs, NJ: Prentice-Hall, 1967).D. Kahneman, P. Slovic, and A. Tversky, Judgment under Uncertainty:Heuristics and Biases (Cambridge, UK: Cambridge University Press, 1982).H.A. Simon, Models of Man: Social and Rational (New York: John Wiley &Sons, 1957).
G.A. Miller, “The Magical Number Seven, Plus or Minus Two: Some Limits onOur Capacity for Processing Information,” Psychological Review 63 (1956),81-97.
J.R. Hayes, Human Data Processing Limits in Decision Making, Report No.ESD-TDR-62-48, Massachusetts: Air Force System Command, ElectronicsSystem Division, 1962.
The term algebraic in this context refers to the fact that numbers with bothpositive and negative values are being combined.
This is a simple linear combination. There are more sophisticated methodsof linear combining where the variables are assigned weights (weighted alge-braic sum).
. R. Hastie and R.M. Sawes, Rational Choice in an Uncertain World: The Psy-
chology of Judgment and Decision Making (Thousand Oaks, CA: Sage Pub-lications, 2001), 52.
J.E. Russo and P.J.H. Schoemaker, Decision Traps: The Ten Barriers to Bril-liant Decision-Making and How to Overcome Them (New York: Doubleday,1989), 137. The authors cite 10 studies comparing the accuracy of intuitiveexpert prediction with predictions produced by objective linear models. Thestudies include prediction of academic performance, life-expectancy of can-cer patients, changes in stock prices, psychological diagnosis, bankruptcy,student ratings of teacher effectiveness, sales performance, and IQ based onRorschach test. The average correlation between prediction and outcome forexpert judgment was 0.33 on a scale of 0 to 1.0. The average correlation forthe objective model was 0.64. In a meta-analysis of over 100 peer-reviewedstudies comparing expert judgment with statistical rules, statistical ruleswere more accurate in 96 percent of the cases. See J.A. Swets, R.M. Dawes,and J. Monahan, “Psychological Science Can Improve Diagnostic Decisions,”Psychological Science in the Public Interest 1 (2000).482 NOTES
27. If each variable is assigned a value of 0 for low and 1 for high, the possiblevalues of a sum are: 0 (all have values of zero), 1 (one variable has a value of1 and two are zero), 2 ( two variables have values of 1 and one has a value ofzero), and 3 (all have values of 1).
28. If financial markets are complex nonlinear systems, a predictive modelwould have a nonlinear, functional form. Thus indicators would need to becombined in a configural (nonlinear) manner rather than a simple sequential-linear fashion.
29. B. Fischhoff, P. Slovic, and S. Lichtenstein, "Knowing with Certainty: The Ap-propriateness of Extreme Confidence,” Journal of Experimental Psychology:Human Perception and Performance 3 (1977), 552-564; L.A. Brenner, D.J.Koehler, V. Liberman, and A. Tversky, “Overconfidence in Probability and Fre-quency Judgments: A Critical Examination,” Organizational Behavior andHuman Decision Processes 65 (1996), 212-219; R. Vallone, D.W. Griffin, S. Lin,and L. Ross, “Overconfident Prediction of Future Actions and Outcomes by Selfand Others,” Journal of Personality and Social Psychology 58 (1990),.582-592;A. Cooper, C. Woo, and W. Dunkeelberg, “Entrepreneurs’ Perceived Chances forSuccess,” Journal of Business Venturing 3, no. 97 (1988), 97-108
30. D.G. Myers, Intuition, Its Power and Perils (New Haven: Yale UniversityPress, 2002).
31. J. Metcalfe, “Cognitive Optimism: Self-Deception of Memory Based Process-ing Heuristics,” Personality and Social Psychology Review 2 (1998),100-110, as referenced by D.G. Myers, in Intuition, Its Powers and Perils(New Haven: Yale University Press, 2002).
32. S. Lichtenstein, B. Fischhoff, and L. Phillips, “Calibration of Probabilities:The State of the Art to 1980,” in Judgment under Uncertainty: Heuristicsand Biases, D. Kahneman, P. Slovic, and A. Tversky (Eds.) (Cambridge, UK:Cambridge University Press, 1982), 306-334.
33. B. Fischoff and P. Slovic, “A Little Learning . . . Confidence in Multi-CueJudgment Tasks,” in R. Nickerson (Ed.), Attention & Performance VIII(Hillsdale, NJ: Earlbaum, 1980), the original source, as mentioned in Judg-ment under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic,and A. Tversky (Eds.) (Cambridge, UK: Cambridge University Press,1982).
34. L. Goldberg, “Simple Models or Simple Processes? Some Research on Clini-cal Judgments,” American Psychologist 23 (1968), 338-349.
35. J.J.J. Christensen-Szalanski and J.B. Bushyhead, “Physicians’ Use of Proba-bilistic Information in a Real Clinical Setting,” Journal of Experimental Psy-chology: Human Perception and Performance 7 (1981), 928-935.
36. A.A. DeSmet, D.G. Fryback, and J.R. Thornbury, “A Second Look at the Util-ity of Radiographic Skull Examination for Trauma,” American Journal ofRadiology 132 (1979), 95-99.
37. Russo and Schoemaker, Decision Traps, 72.
38. Ibid.
39. D. Dreman and M. Berry, “Analysts Forecasting Errors and Their Implica-tions for Security Analysis,” Financial Analysts Journal 51 (May/June1995), 30-41.Notes 483
40.
Al.
42.
43.
44,
45.
46.
47.
B. Barber and T. Odena, “Boys Will Be Boys: Gender, Overconfidence, andCommon Stock Investment,” Working Paper, University of California, Davis,1998a.
Hersh Shefrin, Beyond Greed and Fear: Understanding Behavioral Financeand the Psychology of Investing (Boston: Harvard Business School Press,2000), 51.
D. Kahneman and M. W. Riepe, “Aspects of Investor Psychology,” Journal ofPortfolio Management (Summer 1998).
T. Gilovich cites the following studies: N.D. Weinstein, “Unrealistic Optimismabout Future Life Events,” Journal of Personality and Social Psychology 39(1980), 806-820; N.D. Weinstein, “Unrealistic Optimism about Susceptibilityto Health Problems,” Journal of Behavioral Medicine 5 (1982), 441-460;N.D. Weinstein and E. Lachendro, “Eco-centrism and Unrealistic Optimismabout the Future,” Personality and Social Psychology Bulletin 8 (1982),195-200.
Hulbert Digest, www3.marketwatch.conVStore/products/hfd.aspx?siteid=mktw.
The self-attribution bias is discussed in the following studies: D.T. Millerand M. Ross, “Self-Serving Biases in the Attribution of Causality: Fact orFiction?” Psychological Bulletin 82 (1975), 213-225; R. Nisbettand L.Ross, Human Inference: Strategies and Shortcomings of Social Judg-ment (Englewood Cliffs, NJ: Prentice-Hall, 1980); P.E. Tetlock and A. Levi,“Attribution Bias: On the Inconclusiveness of the Cognition-MotivationDebate,” Journal of Experimental Social Psychology 18 (1982), 68-88;R.R. Lau and D. Russel, “Attributions in the Sports Pages,” Journal of Per-sonality and Social Psychology 39 (1980), 29-38; C. Peterson, “Attribu-tions in the Sports Pages: An Archival Investigation of the CovariationHypothesis,” Social Psychology Quarterly 43 (1980), 136-141; R.M. Arkinand G. M. Maruyama, “Attribution, Affect and College Exam Perfor-mance,” Journal of Education Psychology 71 (1979), 85-93; H.M. Dawesand W.G. Stephan, “Attributions for Exam Performance,” Journal of Ap-plied Social Psychology 10 (1980); T.M. Gilmour and D.W. Reid, “Locus ofControl and Causal Attribution for Positive and Negative Outcomes onUniversity Examinations,” Journal of Research in Personality 13 (1979),154-160; R.M. Arkin, H. Cooper, and T. Kolditz, “A Statistical Review of theLiterature Concerning Self-Serving Attribution Bias in Interpersonal Situa-tions,” Journal of Personality 48 (1980), 435-448; D.T. Miller and M. Ross,“Self-Serving Biases in the Attribution of Causality: Fact or Fiction?” Psy-chological Bulletin 82 (1975), 213-225; R. Nisbett and L. Ross, Human In-ference: Strategies and Shortcomings of Social Judgment (EnglewoodCliffs, NJ: Prentice-Hall, 1980).
T. Gilovich, “Biased Evaluation and Persistence in Gambling,” Journal ofPersonality and Social Psychology 44 (1983), 1110-1126; T. Gilovich and C.Douglas, “Biased Evaluations of Randomly Determined Gambling Out-comes,” Journal of Experimental Social Psychology 22 (1986), 228-241.
R. Nisbett and L. Ross, Human Inference: Strategies and Shortcomings ofSocial Judgment (Englewood Cliffs, NJ: Prentice-Hall, 1980).484
51.
53.
55.
57.
58.
NOTES
. John R. Nofsinger, Investment Madness (Upper Saddle River, NJ: Prentice
Hall, 2001).
. D. Dreman, Contrarian Investment Strategies: The Next Generation (New
York: Simon & Schuster, 1998) 80, regarding a study by Paul Slovic discussedin a speech before the IGRF, May 1973, entitled “Behavioral Problems Adher-ing to a Decision Policy.”
. J.E. Russo and P.J.H. Schoemaker refer to the following studies: E. Langer,
“The Illusion of Control,” Journal of Personality and Social Psychology 32(1975), 311-328; L.C. Perlmuter and R.A. Monty, “The Importance of Per-ceived Control: Fact or Fantasy?,” American Scientist 65 (November-December 1977), 759-765.
John R. Nofsinger references P. Presson and V. Benassi, “Illusion of Control:A Meta-Analytic Review,” Journal of Social Behavior and Personality 11,no. 3 (1996), 493-510.
B