to know the total profit but also how the system made the money. A system maybe extremely profitable, but if it does not match the trading style or risk aversion ofthe trader using it, he will abandon it.There are a number of statistical metrics that can be used to evaluate the efficacyof a trading system. Each has its advantages and disadvantages. The discussion ofcriteria selection begins with five core criteria, and develops a total of 13 evaluativeconcepts. These concepts are:1 The percentage of profitable trades (P): This is the ratio of the profitable tradesdivided by the total trades and it is sometimes also called the Probability of a ProfitableTrade. The main disadvantage of this criterion is that it tells us nothing about themagnitude of the profitable trades.2 The profit factor (PF): This is the ratio of gross winnings to gross losses. Valuesabove one indicate a profitable system.3 Average profit/average loss (Rwl): This is the average winning trade divided bythe average losing trade.4 Total net profit: This is the total net profit that the system makes during the testingperiod. If this is a negative number, it indicates a loss. Commissions and interestearned when out of a trade are taken into account.An important limitation of this metric is that because the total net profit showsonly the bottom line result of trading a particular system, it doesn’t reveal the pathtaken to attain that net profit. It does not show the losing trades and maximum equitydrawdown experienced in the course of trading. There is another obvious problemwhen comparing trading systems over time periods of different lengths using onlythe total net profit criterion: more trades can occur over a longer period of time andthus higher net profits are possible.5 The average profit per trade: This is the total net profit divided by the numberof trades. This evaluation metric is appropriate for comparing performance over timeperiods of differing lengths. This criterion, however, is not perfect either, as a fewlarge values that are well above or below the majority of the observed data can “pull”the average up or down, diminishing the usefulness of this metric in predicting theexpected value.
156
Intermarket Trading Strategies
6 Profitabilty: This metric was first introduced by Michael Harris in an article inTechnical Analysis of STOCKS & COMMODITIES (see Bibliography) and combinesboth the percentage of profitable trades and the ratio of the average winning/losingtrade into a single number according to the following formula:Profitability = 10 · P −
11 + Rwl

where P = % winners; and Rwl = ratio of average winning trade to the average losingtrade.To eliminate decimal places and make the formula comparable with the profitfactor I modified Harris’s formula by multiplying the result by a factor of 10.The interpretation of the results is as follows:• the higher the value, the more profitable the system;• values above 2 indicate a tradable system and above 4 a very good system;• a negative value indicates that the trading system produced a net loss;• profitability is always a little less optimistic than the profit factor.An example of calculating all the metrics described above is included in the Excelspreadsheet in Table 10.1. The profitability of that particular system is depicted inrow 55.7 Profit/loss index: This is not a very popular metric but I decided to include ithere as it is included in MetaStock’s profit and loss report. MetaStock calculatesthe profit/loss index by combining the winning and losing trades into one value thatranges from −100 to +100.8 Buy and hold profit: This is the profit resulting from a buy and hold strategy. Abuy and hold strategy assumes that you buy on the first day of the test and hold theposition until the end. The profit is calculated by using the price on the first and lastdays of the test, taking into account entry and exit commissions.Most testing software use the stock price of the first date of the data loaded in thechart in calculating the buy and hold profit. This results in unrealistic comparisonwith the trading profit result, as the system test cannot produce signals before allindicators are calculated. This can make a big difference where indicators requiring alarge amount of data are involved, like the regression divergence indicators (used inthe next chapters) which require more than three years of data. In these cases, the buyand hold profit should be calculated manually from the date that your system startedproducing trades and not from the first day of the loaded data.9 The buy/hold index: This is another metric reported only by MetaStock. Thisshows the percentage of the system’s profit compared to a buy and hold strategy.
Trading System Design
157
10 The luck coefficient: As the name suggests, this metric attempts to determinehow “lucky” a system is by comparing the largest to the average trade. It is calculatedaccording to the following formula:LC = MAXW/AVGWwhere MAXW = largest winning trade profit; AVGW = average winning trade profit.11 Pessimistic rate of return (PRR): This metric effectively increases the numberof losers in a system by the square root of total losers, and decreases the number ofwinners in a system by the square root of total winners. The result is then computedby multiplying the new number of winners by the average amount won, and dividingthis by the new number of losers multiplied by the average amount lost.The PRR is calculated according to the following formula, which is very similarto the one used for calculating the profit factor:√ W− W∗ AVGWNPRR =  √ L− L∗ ABS(AVGL)N

where W = number of winning trades; L = number of losing trades; N = total numberof trades; AVGW = average winning trade profit; AVGL = average losing trade profit.A value greater than 2 is good; a value greater than 2.5 is excellent.For more information on this statistic see also Vince’s Portfolio ManagementFormulas: Mathematical Trading Methods for the Futures, Options and Stock Markets(see Bibliography).12 Efficiency: By comparing the entry and exit prices to the highs and lows ofthe move from entry to exit, you can get measures of how good your entry andexit technique is. Entry and exit efficiency is measured according to the followingformula:For long trades: Entry Efficiency = (Highest Price − Entry Price)/(Highest Price − Lowest Price).For short trades: Entry Efficiency = (Entry Price − Lowest Price)/(Highest Price − Lowest Price).Entry efficiency is therefore designed to measure how well a system enters into atrade, or how close the long entry was to the lowest point within the trading periodand the reverse for a short signal.Exit efficiency, on the other hand, measures how well a system exits a trade, andin the case of a long entry, how close an exit is to the highest point within the trading
158
Intermarket Trading Strategies
period range according to the following formula:Exit Efficiency = (Exit Price − Lowest Price)/(Highest Price − Lowest Price)for long trades andExit Efficiency = (Highest Price − Exit Price)/(Highest Price − Lowest Price)for short trades.Total efficiency is defined as the percentage of the realized profit in relation to thetotal profit potential during that trade according to the formula:Total Efficiency = (Exit Price − Entry Price)/(Highest Price − Lowest Price)for long trades and for short trades:Total Efficiency = (Entry Price − Exit Price)/(Highest Price − Lowest Price).The total efficiency can also be expressed in terms of entry and exit efficiency:Total Efficiency = Entry Efficiency + Exit Efficiency − 100.As a general rule, entries and exits should capture 55 % or more of the high–lowrange, while overall efficiency should be more than 30 %. The average efficiency ofa system is of course the average of all trade efficiencies.13 The t-test: Statistical analysis can be used to evaluate the trade results and provideadditional information and probability estimates. I haven’t seen this method used veryoften, probably because it involves time-consuming calculations or because statisticsis not a popular subject among traders. Statistics can also help the trader determinewhether the performance of a system is due to chance, or if the trading model willhold in the future.In this case a simple t-test is used to compare the mean of the simulated tradeprofit/loss results to some benchmark and to assess whether they are statisticallydifferent from each other. The statistical analysis involved makes certain assumptionsabout the trade sample and populations to which they belong, which may be violatedwhen dealing with trading models. In any case it is a good idea to check the followingassumptions:• Normality. An important assumption in the t-test is that the underlying distributionof the data is normal. The distribution of profit/loss figures of a trading systemmay not be normal, especially if a stop-loss condition is used, as a lot of tradeswill cluster around the maximum loss. A visual inspection of the distribution oftest results will often determine whether it conforms to the bell-shaped normalcurve. More rigorous methods, such as the Kolmogorov-Smirnov normality test orShapiro-Wilk’s W-test can be used to determine whether the distribution is normal.In case of minor violations, the distribution can be modified by removing some
Trading System Design
159
outliers or transforming the data using a squashing function. The central limittheorem, which states that as the number of cases in the sample increases, thedistribution of the sample mean approaches normality, can also come to the rescuein cases of large sample sizes.• Autocorrelation. This violation is essentially one of serial dependence, whichoccurs when trade cases are not statistically independent of one another. Thisreduces the effective sample size. The extent of autocorrelation can be determinedby computing the correlation between the profit and loss results with the sameresults shifted by n days in the past.• Sample size. The system should produce more than 30 trades in order for thestatistical results to be of any significance.Some violations may be ignored, while others may be compensated for. Where thereare more serious violations this test should be avoided.To illustrate the use of the t-test in evaluating a system, I exported trade resultsfrom the SPY regression long-short test, presented in Chapter 12, into an Excelspreadsheet. The test data together with the t-test and an example of calculating someof the other test metrics described above is presented in Table 10.1.Before performing a t-test comparison, the Kolmogorov-Smirnov normality testwas used to check whether the distribution was normal. The test statistic was foundto be 0.1 and the associated significance 0.2 which is greater than 0.05 (row 66),suggesting that the current distribution was not perfectly normal but not significantlydifferent from a normal distribution. To check on the second assumption of autocorrelation, I copied the same results one row forward (from column C to column D)and calculated the correlation between columns C and D using Excel’s CORRELfunction. In the current case, the autocorrelation was 0.28 (row 65), which suggestsa weak correlation, but not as low as I would prefer.The sample was not perfect but since none of the assumptions were violated wecould proceed to calculate the t-test to compare the sample profitability against noprofitability, by subtracting zero from the sample mean profit. To compare the meanagainst the benchmark, which in this case is the S&P 500 buy and hold performance, Isubtracted the S&P performance from the mean. Next, I divided the resultant numbersby the estimated standard deviation to obtain the value of the t-statistic (in rows 59and 61). Finally, I calculated the probability that I would get a t-statistic as largeby chance alone by using the TDIST function included in Excel. This works out tobe 0.02 % (in row 60). Since this figure is very small, it is highly unlikely that theresults were profitable due to chance alone. When the system is compared to the S&P,however, the probability is 21.6 % (row 62) which suggests that if the system weretested on a number of different samples, about 22 % of the tests would not fare betterthan the S&P buy and hold method.It should be clear by now that each criterion mentioned above has at least onedrawback that keeps it from being the sole method by which to evaluate a tradingsystem. In addition, drawdown or risk has to be considered separately as none of
160
Intermarket Trading Strategies
Table 10.1 The profit/loss trade results for the SPY long-short system presented in Chapter12 were used as an example of evaluating test results using a variety of statistical andempirical metrics.A
B
C
D
E
23
Date
Profit/lossper trade
Profit/loss(lag=1)
Profit/losscumulative
4567891011121314151617181920212223242526272829303132333435363738394041
31/01/9615/03/9621/08/9713/10/9730/10/9717/02/9823/07/9810/11/9826/05/9908/06/9921/07/9929/11/9929/03/0026/06/0031/08/0029/01/0109/03/0129/05/0126/09/0104/12/0112/06/0225/10/0227/03/0308/08/0306/10/0326/03/0429/04/0416/09/0427/10/0422/11/0427/01/0531/03/0522/06/0529/09/0529/11/0505/01/0623/08/0607/02/07
−987.26366.25−830846.45864.116681134.73572.49764.07−638.64681.79693.361160.17−185.2464.16144.6419312.842949.62448.72076.7147.56258.46120.7−652.82554.36−81.6−319.25−84.081027.99235.75470.631013.56224.621197.2−225.67740.0290.4
−987.26366.25−830846.45864.116681134.73572.49764.07−638.64681.79693.361160.17−185.2464.16144.6419312.842949.62448.72076.7147.56258.46120.7−652.82554.36−81.6−319.25−84.081027.99235.75470.631013.56224.621197.2−225.67740.02
−987.3−621.0−1451.0−604.6259.51927.53062.33634.84398.83760.24442.05135.36295.56110.36574.56719.16738.17051.010000.612449.314526.014673.514932.015052.714399.914954.214872.614553.414469.315497.315733.016203.717217.217441.818639.018413.419153.419243.8(Continued)
Trading System Design
161
Table 10.1 (Continued)A
B
C
D
E
23
Date
Profit/lossper trade
Profit/loss(lag=1)
Profit/losscumulative
90.4−156.81121.68
19087.020208.720306.6
424344454647484950515253545556
08/03/0720/04/0717/09/07
−156.81121.6897.92
Average profit /tradeCountWinnersLosersAverage WinnerAverage Loser% of profitable tradesAvg. Winners/Avg. LosersProfitability RatingProfit factor
FORMULA495.3413110789.3−416.175.6 %1.904.115.88
5758596061
Standard DeviationStandard Error of the meant-statistic return vs 0Probability P/L>0t-statistic return vs. S&P
820.9128.23.90.02 %0.8
626364
Probability >S&PCoefficient of VariationMax initial closeddrawdownAutocorrelation (lag=1)Kolmogorov-Smirnov test
21.6 %166 %−1451.0
6566
0.280.099
=AVERAGE(C4:C44)=COUNT(C4:C44)=COUNTIF(C4:C44,“>0”)=COUNTIF(C4:C44,“<0”)=SUMIF(C4:C44,“>0”)/C49=SUMIF(C4:C44,“<0”)/C50=C49/C48=C51/ABS(C52)=10* (C53-1/(1+C51/ABS(C52)))=SUMIF(C4:C44,“>0”)/ABS(SUMIF(C4:C44,“<0”))=STDEV(C4:C44)=C57/SQRT(C48)=C47/(C57/SQRT(C48))=TDIST(C59,C48,1)=(C47-16138/C48)/(C57/SQRT(C48))=TDIST(C61,C48,1)=C57/C47=MIN(E4:E44)=CORREL(C4:C44,D4:D44)Significance=0.2
the profitability criteria take it into account. Therefore the evaluation of each system requires a subjective judgment on the part of the trader, based on his tradingpreferences, experience, and psychological profile.Typically, a trader views a system as a “success” if it maximizes profit – subject tominimizing drawdown, minimizing exposure to the market and minimizing positionrisk – and provides a smooth equity curve. Additionally, trading systems can becompared on a statistical basis by performing the independent-measures single factorANOVA procedure, which effectively quantifies between and within-group variations.ANOVA can be used to compare two or more samples of unequal sizes, although
162
Intermarket Trading Strategies
greatest accuracy is achieved with equal or approximately equal sample sizes. In casesof moderate departure from normality, ANOVA still provides a valid test, especiallywhen the samples are relatively large.
10.3
DRAWDOWN AND OTHER RISK METRICS
The classical performance tests consider in detail the profitability of a technicaltrading system but fail to address its risk reduction properties. Increased risk must becompensated by incremental expected return and vice versa.Drawdown is a very important statistic, since it represents your risk. But what dothose drawdown figures represent? It is important to realize that there are differenttypes of drawdown.Maximum drawdown: Drawdowns are retracements in equity from previous equityhighs in short, losing periods. Maximum drawdown is calculated by finding thehighest peak in the equity curve and subtracting the subsequent lowest trough beforethe next higher peak in the equity curve.There are two different ways to calculate maximum drawdown: open and closedtrade. The maximum open trade drawdown occurs while the trade is open and theclosed trade drawdown only represents the final actual realized loss when the trade isclosed. This is typically the smallest drawdown and some system developers preferto publish this one as it looks the best for marketing purposes. They argue that whatmatters most are realized closed profits and losses, and not what happens during atrade. I believe that this argument may have some value when it comes to reportingnet profits but not in the case of drawdowns, because you will certainly experiencelarger drawdowns than shown by the maximum closed trade drawdown figures.Maximum initial open drawdown: This is the maximum dollar “out of pocket”drawdown that you will first suffer before you finally start making profits.This drawdown is useful in evaluating the minimium account size required when testing tradingsystems on commodities and equity futures.Total daily drawdown: This is the drawdown at market close and it is what willbe reflected on the daily account balance on your statements that you get from yourbroker.Maximum percentage drawdown: This is the maximum drawdown expressed asa percentage of the total equity which may not occur at the same point in time asthe maximum dollar drawdown. For example, for a given equity curve you couldhave a situation where the maximum percentage drawdown of 50 % was causedwhen equity was reduced from a maximum of $10 000 to a minimum of $5000 (ora $5000 reduction in equity). Also from the same equity curve you could have a
Trading System Design
163
maximum peak-to-valley dollar drawdown of $400 000 when the equity was reducedfrom $1 000 000 to $600 000 even though the percentage reduction is only 40 %,less than the 50 % maximum peak-to-valley percentage drawdown caused when theequity was reduced from $10 000 to $5000.Maximum position adverse excursion (MAE): This is defined as the most equitylost intraday by any single position.Ideally we would like to have no drawdown in equity but in reality it cannot be avoided.There will be periods where traders experience a string of losses that outweigh anyrecent gains. The main implication of drawdown is the psychological effect it willhave, as a large string of losses may cause the trader to abandon his trading system.Finally, you should keep in mind that drawdown can sometimes be an unreliablefigure. It should never be used on its own to compare two trading systems but alwaysused in conjunction with profitability criteria.MetaStock reports only the maximum initial drawdown which is usually lower thanthe maximum open drawdown but it is a helpful statistic as it will help determine thesize of the account needed to begin trading the system.In addition to drawdown, the following metrics use drawdown, time in the marketor standard deviation to measure risk:Risk/reward index: This index compares risk to reward and is defined as the ratioof the maximum open drawdown to the total net profits. MetaStock reports thereward/risk index by calculating the ratio of total net profits to maximum initialdrawdown and normalizes it to a range of −100 to +100.Risk adjusted return: A statistical method of comparing systems is to calculate therisk adjusted return. This is calculated by dividing the annualized rate of return onequity by the standard deviation of equity changes. The compounded rate of returnis calculated according to the following formula:Annual compounded rate of return = (ending equity − initial equity)(1/n) − 1where n = test duration in years.The standard deviation of equity changes is usually calculated monthly but canalso be calculated annually depending on the time frame of the system to be analyzed.This method involves time-consuming calculations. A quick method of obtainingthe return/risk of different systems for comparison purposes is to divide the totalprofit by the standard deviation of profits. The test with the greatest return/risk ratiowould be the best as long as the time period is the same.RRindex: Developed by RINASystems, this is a risk adjusted profitabilityRINAmetric. It is the ratio of the net profit divided by the product of the maximum opendrawdown and percent of time in the market.
164
Intermarket Trading Strategies
The number of consecutive losers: This is a test of how strong your stomach mustbe to trade the system. Before making any hasty conclusions, however, the dollarvalue of these strings of losses should be examined as well. A trading system mayexperience a long string of losses but actually lose very little money, compared to ashort string of heavy losses.Sharpe ratio: This is a measure of the mean excess return per unit of risk anddescribes how much excess return you are receiving for the extra volatility of traderesults. It is calculated according to the following simple formula:SR = (r − R) /σwhere r is the asset return; R is the return on a benchmark asset, such as the risk freerate of return of the US T-bill; and σ is the standard deviation of the excess return.When calculating the Sharpe ratio it is important for all the variables to be calculatedfor the same time frame, e.g. if you are comparing annual returns the standarddeviation should be annualized by multiplying by the square root of 365 divided bythe time period (in days) used for the standard deviation calculation. Also all numbers(including the standard deviation) should be expressed in percentage terms.The Sharpe ratio is used widely when evaluating mutual fund performance.These numbers are available for most mutual funds on the Morningstar web siteat http://www.morningstar.com/ (subscription required). A Sharpe ratio of over 1.0is good. Outstanding funds achieve something over 2.0.A serious problem in using this metric is the assumption that asset returns arenormally distributed which in reality is not always the case.Sterling ratio method: Similar to the Sharpe method above, the Sterling ratiomethod measures risk adjusted return and is given by:R(%)/ (D(%) + 10%)where R = three-year average annual return; D = three-year average maximumannual drawdown.Both the Sharpe and Sterling ratio methods compare variability of returns, asopposed to drawdown which measures losses.The coefficient of variation of average profit: The coefficient of variation can beused instead of the average profit per trade in order to account for volatility of results.It is a useful statistic for comparing the degree of variation from one data series toanother, even if the means are significantly different from each other.This is calculated by dividing the standard deviation by the mean profit. It is oftenreported as a percentage (%) by multiplying by 100. The result is a unit-less, relativemeasure and risk adjusted comparisons can be made between different systems.
Trading System Design
165
You should avoid using this metric when the mean profit value is near zero because the coefficient of variation becomes very sensitive to changes in the standarddeviation, limiting its usefulness. A low positive value is the most desirable as thisindicates a high profit with low variability.The Calmar ratio: This is the average percentage yearly gain divided by the largestpercentage drawdown.The Ulcer Index: This risk metric is designed to measure only the negative return (drawdown) fluctuations and it is defined as the root mean square of equitydrawdowns.
10.4
STOP-LOSS
A stop-loss order is used to liquidate a losing position upon the breach of a predefinedmaximum loss limit. When confronted with an equity drawdown on an active trade,a trader must choose between two conflicting actions: liquidating the trade in anattempt to conserve capital, or continuing with the hope of recouping losses on thedrawdown.The ideal stop is unique. Though no such ideal stop exists, a good idea is to placestops just outside the range of random price swings, not so close that it gets you outof a trade too soon but not so far away as to be meaningless. Your time horizon shouldalso determine stop placement. Short-term traders looking for relatively small profitsshould use tight stops, while intermediate to long-term trend traders looking for bigmarket moves should use loose stops placed far away from the current price in orderto allow for random moves.There are many types of stop loss conditions:• Initial stop-loss. This is applied to the initial price when entering the trade. Onentering a trade, the maximum loss, in points or percentage, that will be risked onthe trade is determined. If the market moves against the position by this amountfrom the entry price, the stop is activated and the position is closed. One problemwith this rule is that it doesn’t account for volatility. With volatile stocks or futures,this range may be within the normal price action over a dull week.• Trailing stop-loss. A trailing stop is designed to both limit loss and protect profit.The difference from the initial stop is that it applies to the highest price duringthe trade. If the market initially moves against the trader’s position the stop isactivated and in this case it is identical to the initial stop. If, on the other hand, themarket initially moves in favor of the position, the stop is re-positioned x points orpercentage from the current price.• Price level based stop. This stop-loss is usually set at a price level of technicalsignificance, i.e. a support or resistance. Stops are often placed near a significant
166
Intermarket Trading Strategies
swing high or low in trending markets, while in trading markets, stops may beplaced just above or below the current trading range.• Volatility stops. Instead of defining the stop-loss in terms of a fixed percentage ordollar amount, a standard deviation or average true range threshold can be used toachieve a similar objective. This is my favorite as it seems a good compromise tothe too tight or too loose dilemma.• Indicator based stop. This is triggered by an indicator, the most popular beingWilder’s parabolic stop and reverse (SAR) indicator. The term “parabolic” comesfrom the shape of the curve (resembling a parabola) created on the technical chart.The parabolic SAR is a trend-following indicator that is designed to create a trailingstop. It is primarily used in trending markets so the SAR is really inappropriate forrange bound markets as it will whipsaw in and out of the market.• Time stop. This is triggered after a predetermined period of time has elapsed.There are a number of variants of the stops described above, and stops may also beset on combinations of the above.Despite the many advantages of using them in your strategy, stop-losses are not apanacea. Here are some advantages and disadvantages of using stop-loss conditions:• Risk management. The nature of the stop-loss order allows a certain degree of riskmanagement, according to each trader’s risk aversion, even though it might notimprove overall profitability.• Catastrophic losses. Stop-loss orders are essential to avoid catastrophic losses andreduce the risk of ruin.• Curve-fitting. Stop-loss conditions can reduce the drawdown caused when tradingin real time a bad or a curve-fitted system.• Psychological implications. Stop-loss conditions contribute positively to a trader’semotional and psychological condition. By placing a simple stop-loss it is notnecessary to agonize over unconscionably large equity drawdowns during the lifeof a trade.And some disadvantages:• A stop-loss could liquidate profitable trades prematurely.• The nature of the stop-loss order does not allow the user to absolutely define theamount of risk. As stated in the risk disclosure statement issued by the CommodityFutures Trading Commission (CFTC): Placing contingent orders, such as a “stoploss” or “stop-limit” order will not necessarily limit your losses to the intendedamounts, since market conditions may make it impossible to execute such orders.• A stop-loss might reduce the