end lines, Fibonacci retracements, etc.).• Historical back testing capabilities.• Real time system monitoring capabilities,For further information, we would point you to Futures Magazine'sGuide to Computerized Trading. This guide gives a brief description ofthe different software packages and contact information.The two largest software providers for the Apple Macintosh and PCplatform are:
166
THE ULTIMATE TRADING GUIDE
Apple Macintosh PlatformProTA GoldBeeSoft311 Summerdale LaneAlgonquin, IL 601025upport@beesiift.net(847)854-3960
PC PlatformTradeStation™Omega Research8700 W. Flagler Street Suite 250Miami, FL 33174www.omegaresearch.com13051 485-7000
Trendsetter Software2024 N. Broadway, Suite 310Santa Anna, CA 92706(600) 825-1H52
MetastockEquis International Inc.3950 South 700 East. Suite 100Salt Lake City, UT 84107w w w. equis. com
(800)882-3040
DATA
It doesn't matter how fast your computer is or how robust your tradingsystem is, your trading is not going to go anywhere without good data.Data is the fuel to your system just as gas is to your car. Data basicallycomes in two different formats: real lime or historical. Real time data isavailable within seconds from the action in the pit on the exchangefloor. Historical data is all data that is not real time—Fifteen-minutedelayed, end-of-day, end-of-week, and so on. As with anything, timeliness and speed comes with a premium. End-of-day data will typicallycoal $30 a month, whereas real time data can cost up to $800l a month.End-of-day data is sufficient to profitably trade a system. In manycases, real time data can lead to information overload. We have seentraders with real lime data quotes make a sound and logical trade, andthen prematurely exit because the market goes quickly against them. Inthe end, their first decision ends up being profitable. The end-of-daydata trader, would not have known about the initial adverse move, andwould have stayed with the trade. Figure 9.1 shows the difference between what the end-of-day trader sees and what a real lime trader sees.If you plan to day trade (enter and exit a trade in the same day),then real time data is required. When day trading markets such as theS&P 500, you only have small windows of opportunity, Real time data isnecessary to capitalize on those opportunities and to protect yourselffrom adverse market movements.The search for good data can be as difficult as the search for a goodtrading system. Two different traders trading the same market with thesame system, but with data from different vendors, can end up withcompletely different results. How can that be if the data comes from one
WHERE TO START
FIGURE 9.1
167
Different view points. Intraday view versus end-of-day view.
source, the exchange? Sheldon Knight did an extensive analysis of different data vendors in the September 1999 issue of Futures magazine.In this article, Mr. Knight answered that question, "To understand howdata differences happen, we first have to look at how different vendorsprocess activity on the exchange floor to get end-of-day open, high, low,and closing prices. During the trading day. some traders in the pit arebidding to buy futures contracts while others are offering to sell thesame contracts at slightly higher prices. An actual trade occurs when abid or an offer is accept by another trader. . . . When exchanges reportthe high and low prices, they use the highest bid and lowest offer even ifno trade actually took place at those prices. This convention is usedpartly because a stop order is triggered by a bid or offer at the stopprice, and a limit order will be filled if a bid or offer occurs at the limitprice. So a system test will miss some trades unless bids and offers areincluded in the test data. On the other hand, discretionary traders prefer to study actual trades on their charts.These conflicting requirements of different traders lead some endof-day data vendors to include bids and offers in their reported highs
168TABLE 9.1
THE ULTIMATE TRADING GUIDEHow Clean Is Your Data
SoybeansDays TestedDays missing in vendor databaseNo. of errors in open, high or low
Bridge/CRB*
CSI
Dial/Data*
Omega
748026
748018
748064
7232568
759036
75809
7590257
758038
S&P 500
Days TestedDays missing in vendor databaseNo. of errors in open, high or low
* Bridge/CRB, Dial/Data, and Reuters/Equis comparisons are for combined overnightand day sessions. Others are for day sesssions only. Data cover 1996-1998- Reprintedwith permission from Futures Magazine. September 1997.
Bridge/CRB30 S. Wucker Dr.. Suite 1810Chicago, II 60606(312)454-1801
MJK Associates1289 Park Victoria Dr.. Suite 205Milpitas, CA 95035(4 OS I 941-3400
Commodity Systems Inc. (CSI)200 W. Palmetto Park Hd.Boca Raton, FL 33432(66D392-B663
Genesis Financial Oats Services425 E. Woodmen Rd.Colorado Springs, CO H09191719)260-6111
Dial 'Data division of Tract Data Corp95 Rockwell PlaceBrooklyn, NY 1121712121 422-4300
Pinnacle Data Corp,1016 Plank Rd.Webster. NY 14580(716) B72-0845
Omega Research Inc.STOQ W. Flagler St.. Suite 250Miami. FL 33174(3051485-7000
Prophet Financial Systems, Inc.430 Cambridge Ave.Palo Alto, CA 94306(650i322-4163
Reuters DataLink (Marketed byEquis in Glance Market DataService, Inc.)3950 South 700 East. Suite 100Salt Lake City, LIT 84107(8011265-886634(1 Bruoksbank Ave., Suite 208North Vancouver, B.C.Canada V7T2C11604} 983-0200
WHERE TO START
169
and lows, while others try to include only actual trades. However, vendors are not consistent in applying their own rules. Some days includebids and offers while other days from the same vendor do not."Similar problems, like the ones described by Mr. Knight, are alsofound in the opening and closing prices. Table 9.1 gives informationabout 10 of the more prominent data vendors and their respective datareporting accuracy.Continuous versus Individual ContractsFaulty and/or inconsistent reporting la not the only problem with data.Futures data, unlike cash or equity data, expires every few months andthis causes problems for most testing software. The most widely usedtesting software cannot handle rolling out of an existing position in onecontract, and rolling into the same position in another contract. Becauseof this limitation, data vendors offer their subscribers continuous contracts of the various markets. A continuous contract is one long flow ofdata made up of the individual contracts while they are the front month.Due to the nature of futures, there are usually large gaps in price between the new and expiring contracts. Since you can't accurately testdata with large price gaps, most data vendors offer software that allowsthe price gaps to be smoothed out. The end result is an artificial continuous contract. This type of contract includes prices that may have neverexisted. The debate between continuous contracts versus actual contracts has been going on for years. On one hand, most software packageswill only test continuous data. On the other hand, most traders only tradethe individual contracts. What is a trader to do? There are three solutions to this conundrum.
170
THE ULTIMATE TRADING GUIDE
The first solution is to test, with continuous contracts and tradewith actual contracts. The majority of system traders trade in thismanner (not necessarily the best way) because they don't believe that itmakes that much of a difference; they believe that historical analysis oftrading systems gives only a ballpark view of a system's potential. If afew trades are off, so be it. Everything should wash out in the end.A second solution is to test with continuous contracts and tradewith continuous contracts. More traders (especially long termers) aretrading in this manner for two reasons.Testing long-term systems on individual contracts can sometimesgive somewhat erroneous information. If you want to test a 100-daymoving average system on a Value Line contract, you would find thatthere simply is not 100 days of historical data. A futures contract is considered top step or front month when it is the most actively traded contract for that particular market. Most futures contracts are frontmonth for less than sixty days. Many times you will find that there simply isn't 100 days of data to load or the data that you are loading is sothin in the early stages that the output of your system has no relationto the current actual market. Creating trade signals on a continuouscontract, in some cases, can give more accurate results.The second reason to trade and test in this manner is the security inknowing that you are trading a system the exact way that you tested it. Ifyou decide on this solution, you have one more problem to overcome; notall continuous data contracts are created equal. The different data vendors use different smoothing formulas and algorithms to remove theprice gaps between contracts. In Mr. Knight's article, he tested a simplesystem on three S&P 500 continuous contracts from different vendors.Even though the data came from different vendors, you would think thatsince the teats used the same system and same market, you would getsomewhat similar results. We were surprised by Mr. Knight's findings:
Test lTest 2Test 3
Total Profit ($)
Maximum Drawdown ($)
127,50096,27595,500
25,37527,000
45,475
You may think that this throws a monkey wrench into the works,but really it doesn't. Consistency is the key. If you test with a certainvendor's continuous data, then trade with that same vendor's data. Bydoing this, you hopefully won't have any unexpected surprises.Finally, the third solution is to test on actual contract data and tradeon actual contract data. We believe that this is the best solution; it solves
WHERE TO START
171
the problem of testing systems on artificial data and you trade exactlywhat you have tested and developed. We know it sounds like we are contradicting ourselves due to our discussion on generating long term trading signals on actual contract data, but the majority of trading systemsdon't have such a long term view. If your trading system needs less thanforty days of historical data, there shouldn't be a great difference between signals generated on actual versus continuous data. However, ifyour system needs more historical data, then by all means use continuouscontracts. As with anything dealing with accuracy, testing with actualcontracts is very difficult. We know of only one software package thatcan test in this manner and that is our own Excalibur™. Excalibur cantest over long periods of time and rollover from one contract to another.This type of testing is very accurate, because as in real trading, rollovercosts are incorporated into the system performance accounting. If youdon't have Excalibur, you can still test individual contracts with yoursoftware package; it just requires a lot more work. Instead of one big testover the entire time period, you have many small testa over each contract.In addition, it, is up to you to keep track of the systems performance statistics, such as total profit, maximum draw down, percent wins, averagewin. and so on. A l l the tests that were carried out in this book were donewith Excalibur testing on individual contracts.When you decide on a data vendor and a format, it is very importantthat you know exactly what you are getting. Does your vendor includebid and ask prices in the highs and lows? What is the smoothing formulas used to create continuous contracts? Once you have educated yourself on your data, then their should be no surprise when you start realtime trading.INDICATORSAny mechanical trading system must have some consistent method forentering and e x i t i n g the market. Most of the time this method or trigger is based on some type of indicator or mathematically based statisticThat has price forecasting capability. Anything that indicates what thefuture may hold is an indicator. Some of the more popular indicatorstoday include moving average, rate or change, momentum, stochastic,rsi, Donchian breakout, Bollinger bands, Keltner bands. Most chartingsoftware has more than 50 built in indicators. Of all of the indicatorsthat are out there, only a h a n d f u l are worth more than a grain of salt.Indicators cannot stand alone: they should be used in concert with otherideas and logic. To demonstrate the viability of pure indicator basedsystems, we have tested a few of the more popular indicators and showthe results on the following pages.
172
THE ULTIMATE TRADING GUIDE
StochasticsThis oscillator type indicator was developed by George Lane in the 1950sand compares the current market close to the range of prices over a specified time period. Stochastics try to indicate when a market is eitheroverbought or oversold. It is based on the assumption that when an uptrend/downtrend approaches a turning point, the closing prices start tomove away from the high/low price of a specific range.The overall purpose of the stochastic oscillator is to give warningto supply overtaking demand in an up trend and demand overtakingsupply in a downtrend. The number that is generated by this indicatorcomes in the form of a percentage oscillating between 100 and 0. Areading of 70 or more indicates the close is near the high of the range.On the other side of the scale, a reading of 30 or less indicates the closeis near the law of the range. The "raw" stochastic values oscillate soquickly that they aren't of much value, therefore most of the timethese values are smoothed with a moving average. In most eases,traders take these smoothed values and smooth them even further byapplying another moving average. These end values are called alowstochastics.The most widely used stochastic based system is to buy when the14-day slow stochastic reaches the oversold territory (30% or less) andthen retraces back and to sell when the overbought territory (70% orgreater) is reached and then retraces back. (See Table 9.2.)
Relative Strength IndexWelles Wilder introduced this oscillator indicator in his 1978 book, NewConcepts in Technical Trading Systems (Greensboro, NC: Trend Research). Similarly to stochastics, RSI oscillates between 0 and 100 andtries to determine overbought and oversold situations. The RSI calculates the ratio of up closes to down closes over a specified, range ofprices. The more up closes or stronger up closes the higher the ratio andvise versa. Usually a reading of 70 or more indicates overbought and areading of 30 or leas indicates oversold.Most traders like to buy when the RSI enters the overbought zone of70 or more and then retraces and sell when the RSI enters the oversoldzone of 30 or less and then retraces. Table 9.3 shows if the RSI had anybetter luck than the stochastic.MACDMoving average convergence divergence (MACD) was developed byGerald Appel in 1979. MACD, another price oscillator, is derived fromthree different exponentially smoothed moving averages. This indicator
WHERE TO STARTTABLE 9.2
Stochastic Test
ONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
is usually plotted as two different lines. The first line (MACD line)being the difference between the two moving averages (long-term mashort-term ma ), and the second line (signal or trigger line ) being an exponentially smoothed moving average of the MACD line.The purpose of the MACD is to try and eliminate the lag that is associated with moving average type systems. This is accomplished by anticipating the moving average crossover and taking action before theactual crossover. Unlike the stochastic and RSI, this oscillator doesn'thave fixed upper and lower extremes.The typical MACD system buys when the MACD line crosses thesignal line from below and sell when the MACD line crosses the signalline from above. We haven't had much luck with pure indicator basedsystems, but let's see how MACD works out in Table 9.4.
174TABLE 9.3
THE ULTIMATE TRADING GUIDERelative Strength Index
ONE CONTRACT PER TRADE
TEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
Commodity Channel IndexDonald Lambert introduced this oscillator type indicator in 1980. Unlike the RSI and stochastics, CCI is a trend following indicator. In addition to this major difference, the formula for this indicator is much morecomplex. CCI expresses how far current prices have deviated from amoving average. If prices have deviated considerably, then there is agood chance a new trend has developed. This indicator sounds in partlike the standard deviation based bollinger bands, but it is different.First off, it uses the typical price ((high + low + close )/3) in its movingaverages, instead of just the close. Secondly, the author added a constant multiplier (.015) to the formula 30 that 70-80% of the valueswould fall between the values of +100% and -100%.Donald Lambert suggested that long positions should be initiatedwhen the CCI exceeded +100 and then liquidated when the CCI retraced
WHERE TO START
TABLE 9,4
175
Moving Average Convergence Divergence
ONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
back to this point. Short positions are initiated and liquidated in a similar manner except the trigger value is -100.As with the other indicators, we put this one to the test. You maysee a pattern evolving as you look at Table 9.5.In their book The New Technical Trader (New York: John Wiley &Sons), Tushar Chande and Stanley Kroll explain how most traders tryto overcome the failure rate of indicators:Even the best indicator does not work 100 percent of the time, hence, usingindicators is a game of percentages. Since each indicator has a significant
176
THE ULTIMATE TRADING GUIDE
TABLE 9.5 Commodity Channel IndexONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
failure rate, traders have developed many indicators to analyze prices, therandom nature of price changes being one reason why indicators fail. Therefore, traders use multiple indicators to confirm the signal of one indicatorwith another. They believe that the consensus is more likely to be correct.
Chande and Kroll go on to explain that this is not a viable approachdue to the strong similarities that exist between price based momentumindicators, using multiple ones simultaneously creates redundancy ( Figure 9.2). A trader does not achieve diversification or confirmation byusing different indicators that are basically first cousins.
WHERE TO START
177
FIGURE 9.2 First cousins. Chart created with ProTA Gold" software at www.beesnft.net.So far we have concentrated on momentum-based indicators. Wehave basically proven that as pure systems, these indicators fail most ofthe time. They seem to he based on sound market principles, so why dothey fail? I believe that Chande and Kroll summed this answer up bestin their book. The New Technical Trader.• None of them is a "pure" momentum oscillator that measures momentum directly.• The time period of the calculations is fixed, giving a different picture of market action for different time periods.• They all mirror the price pattern; hence, you may benefit more directiy trading prices themselves.• They do not consistently show extremes in prices because theyuse a constant time period.• The smoothing mechanism introduces lags and obscures shortterm price extremes that are actually valuable for trading.So far it probably seems that indicators are a waste of time. But before we drive the last nail into the indicator coffin, let's look at somenon oscillating longer term momentum indicators.
178
THE ULTIMATE TRADING GUIDE
Bollinger BandsJohn Bollinger, a leading analyst, has made this indicator famous.Bollinger bands are also known as alpha-beta bands. This indicatorusually uses 20 or more days in it calculations and does not oscillatearound a fixed point. Bollinger bands, when graphed, create an envelopeusually above and below the current prices. Unlike the oscillator basedindicators, this indicator and the other ones that we will discuss in theremainder of this chapter have the same scale as price charts.Bollinger bands consists of three lines and are very simple to derive; the middle line is a simple moving average and the outside linesare plus 2 standard deviations above the moving average or minus 2standard deviations below the moving average. The number of standarddeviations can be varied.A typical Bollinger band type system buys when price reaches thebottom hand and liquidates as the price moves up past the moving average. The sell side is simply the apposite. Statistically speaking, when aprice goes beyond two standard deviations it should work its way backto the moving average. In our testing, we used this logic and in additionwe used a $1500 money management stop loss and instead of simply liquidating when the price moved to the moving average line, we forced theprice to penetrate this line and them retrace back from it. Table 9.6shows the results of the testing.Overall WE got the same results with this test as we did with the oscillators, but we noticed something promising with the results of theBollinger band test. The system lost consistently across most marketsand traded on average less than ten times a year per market. Sometimestraders come up with great ideas that don't initially work the way theywere designed. In the case of the Bollinger hand test, the performancenumbers indicate a good idea applied incorrectly. Systems that arebased seemingly on sound market principles and trades sparingly andloses consistently are occasionally considered good candidates for logicreversal. Instead of buying when prices reached the lower band, we decide to sell. We did the same thing for the upper band; instead of sellingwe bought. We used the same money management stop, but this time weeliminated the profit targets. Let's see how reversing a systems entrylogic can sometimes create a winning system (Table 9.71.We know what you're saying: Why not reverse the logic on the previous indicator tests? We could have but we didn't have the correct recipe.Systems that trade frequently and lose consistently are not good candidates far logic reversal. The problem with such systems is they can'tovercome the commission and slippage that is levied against each trade.They may be profitable before execution costs, but when commissionand slippage is taken into account, each trade is usually a net loss.
WHERE TO START
179
TABLE 9.6 Bollinger Band (Test A)ONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
Instead of using the Bollinger bands to determine overbought/oversold situations, we used them to determine trend. In the long run,the trend-following type systems will outperform the counter trendand short-term oscillator based system. The next two tests will involvetrend following indicators.Moving Average CrossoverThis is probably the most utilized indicator out there. It is very simple,and as you will see, quite effective. The calculation for a simple moving
180
THE ULTIMATE TRADING GUIDE
TABLE 9.7 Bollinger Band (Test B)ONE CONTRACT PER TRADETEST PERIOD l/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
average is simply to sum up the closes (opens, highs, lows or midpoints)for the past x days and divide by x. Do this every day and soon you willhave a moving average line or curve. A cross-over system involves two ormore moving averages usually consisting of a longer-term and shorterterm average. When the short-term moving average crosses from belowthe long-term moving average, this usually indicates a buying opportunity. Selling opportunities occur when the shorter-term moving averageentases from above the longer-term moving average. Moving averagescan be calculated in basically three different manners: simple (as described earlier), exponential, and weighted. Exponential and weighted
WHERE TO START
TABLE 9.8
181
Simple Moving Average
ONE CONTRACT PER TRADETEST PERIOD l/l/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
moving averages tend to skew the moving averages toward the most recent prices. In our testing, we will demonstrate the effectiveness ofmoving average crossovers and in addition show that there really isn'tthat much difference between the three different types of moving overages (Tables 9.8, 9.9, 9.10).The overall net results from the three different moving average testswere quite similar. Each moving average made more money in some markets and loss more in others. Each moving average is as good as the others, and research time should not be spent on trying to determine whichis best. One thing should stand out in our testing, however. This was the
182
THE ULTIMATE TRADING GUIDE
TABLE 9.9 Exponential Moving AverageONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
second teat that actually made money and it was based on trend followinglogic. Another pattern is developing.Donchian BreakoutIf you approach the market from a long-term standpoint, this envelopeindicator should be right up your alley. Richard Donchian popularizedthis method back in the 1960s. This indicator involves two lines thatare plotted above and below the market, similarly to Bollinger bands.The top line represents the highest high of n days hack (or weeks) and
WHERE TO START
TABLE 9.10
183
Weighted Moving Average
ONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/3/99$75 ALLOWED FOR COMMISSION AND SLIPPAGE
conversely the bottom line represents the lowest low of n days back ( orweeks). Richard Donchian felt if price penetrated either band then atrend was fully underway; buy on strength and sell on weakness.Let's test the idea of buying when the day's high penetrates thehighest high of four weeks back and selling when the day's low penetrates the lowest low of four weeks back (Table 9.11).Again, we haw a winning approach. It seems as if the trend followingindicators are by far superior to the overbought/oversold type indicators.Because systematic trading has no room for subjectivity, we feel thattread following type systems are hard to beat. The advantages of a trend
THE ULTIMATE TRADING GUIDE
184
TABLE 9.11
Donchian
Breakout
ONE CONTRACT PER TRADETEST PERIOD 1/1/83-8/31/99£75 ALLOWED FOR COMMISSION AND SLIPPAGE
following systems are: (1) 100% objective. (2) Trading infrequently reduces execution costs. (3) Captures the big swings where most of the bigmoney is. The disadvantages are: (1) Whipsaws are inevitable. Most trendfollowing systems have less than 50% wins. (2) Signals are usually latebecause they are not issued until a trend has developed. (3) if a market 19in a trading range, there is a good chance of having a high number of consecutive losers before a trend is initiated.Weighing the advantages and the disadvantages, we still feel thatthe systematic trader will have a better chance with a trend followingmethodology. We have only scratched the surface on the subject ofindicators, but feel that further research on the hundreds of different
WHERE TO START
185
indicators (overbought/oversold and trend following) would be a waste ofyour time and ours. The testing that we have shown exemplifies what wehave found over the past 15 years.
FIVE APPROACHES USED BY THE BESTTRADING SYSTEMS
The top winning trading systems that we track basically fall into fivedifferent categories. That's it! All of the profitable trading systems thatwe have analyzed, has evolved from five basic approaches. And they are:1. Donchian channel.2. Moving average cross over.3. Short-term volatility based open range breakout.4. S&P day trade.5. Pattern recognition.In this section we are going to present complete turnkey systemsbased on these five approaches. The logic of these systems as well as theperformance numbers will be fully discussed.
ANATOMY OF A TRADING SYSTEMAll trading systems are basically identical when it comes to their overall make up and objectives; all systems enter the market and exit themarket and all systems strive to be profitable. Most system traders anddevelopers spend 90% of their time developing entry techniques and therest of their time is dedicated to the decision that ultimately determinesprofitability. It doesn't seem rational that traders spend much moretime with an entry technique when it is the exit that determines theentry's effectiveness. We feel that research time should be split 50/50between the entry and exit technique.One of the great things about futures and stocks is that you canmake as much money when the market goes down as when it goes up. AllSystems should have a long and short entry technique to cover the onlytwo directions the market can go. When we discuss long and short entrytechniques with traders, the one question that seems to always come upis: Should my short signal be the complete opposite of my buy signal? Atfirst, this question my seem simple to answer. If all markets go up anddown, then you should enter the market long and short based on the samepremise, intimately, we feel that this is the right answer, but the fact is
186
THE ULTIMATE TRADING GUIDE
all markets don't go up and down in the same manner. We have neverseen an extended bear market in the S&P futures for example. Whyshould you bang your head against the wall by shorting a bull market?Also, any system designed and tested with historical S&P futures dataautomatically has a bullish bias. In most cases, if you look at the profitdistribution between long and short positions on a S&P system, you willfind that long profits out number short profits two to one. if marketsseem to have a bias, then why shouldn't we enter a long position differently than we do a short position. There are two basic answers to thisquestion: (1) A lot of markets don't have a bias and those that do can shifttheir bias and systems should be prepared for this. Most systems have ashort term memory (i.e., look back over a very short term history) anddon't know what the overall long term trend is. (2) Having different longand short entry techniques increases the number of parameters andtherefore increases the likelihood of over optimization. As