Trading
Needless to say, this is not how one should be trading. Recruiting an 18-yearold college freshman (probably hung over from the night before) and asking himto enter important trading orders is downright unprofessional and incompetent.From this trading campaign I learned that following a detailed trading plan basedon sound historical results is the only way to trade effectively. And oh yeah, makesure you’re awake and sober when placing your trades.
CHAPTER
4
Evaluating Trading Strategy PerformanceHow to Correctly Assess Performance
Evaluation of a trading strategy can be a very tricky task because strategy performance can often be misleading. Quantitative traders need the correct tools tograde performance. To many traders, these tools are unknown or misunderstood.This chapter will examine the ideas of reward and risk. In the process, we’ll introduce robust performance measures and expose the multitude of problems with popular performance statistics.
POPPER’S THEORIES APPLIED TO TRADINGWhile Karl Popper is regarded as one of the greatest philosophers of the 20th century, his influence on scientists today might be his most enduring legacy. Despitethe time that has elapsed since his work, it is remarkable that scientists still praisePopper for the practicality of his philosophical work and the influence it has had inshaping their scientific thought processes.Popper’s main scientific theory focuses on the growth of human knowledgeand the methods used in making new discoveries. He was very uneasy with the concept of absolute truth. Instead, he felt that theories, regardless of the scientific discipline, could never be proved. Rather, theories could only be falsified throughexperiment, with those that are not dispelled chosen as the best explanation available until either disproved or a better theory is discovered.For example, most academics believe that the random walk is the best explanation of market behavior. Following Popper’s lead, the best I can do to improve upon this75
Copyright 2003 by Lars Kestner. Click Here for Terms of Use.
PART 1 Structural Foundations for Improving Technical Trading
76
theory is to show that certain quantitative trading strategies have generated tradingprofits in the past. But note that by doing so, I have not disproved the random walktheory. What’s more, I cannot say for sure whether any strategy will be profitable inthe future. If I show that my strategy is profitable over a number of markets and anumber of parameter sets, I can then come closer to accepting a new theory that markets are predictable. Then, until I see more data that would refute my theory, or untila better theory comes along, this is the theory I will act upon. The proof of its efficacyis whether I make money in real-time trading.I believe that Popper’s approach is ideal for developing quantitative tradingstrategies. When developing new strategies, I start with the assumption that theidea will be unprofitable. If rigorous historical back-tests of the idea are profitable,then I’ll reject my initial assumption and look closer at the historical back-testdata. Similar to Popper’s thinking, I can never accept that a strategy is going to beprofitable in the future—the best I can do is assume the strategy will be profitableuntil I find a better strategy or discover problems with historical performance.The entire process follows the scientific method from theory to hypothesis toexperimentation to conclusion. Even when I complete the back-test, I have no guarantee that the strategy will continue to be profitable in the future. I test strategies anddiscard those with problems such as deteriorating performance over time, logicalinconsistencies, or logistical issues in implementing the strategy. If I still have a profitable trading strategy after testing for all these problems, I can expect success in thefuture. Still, while continued success may be likely, I believe that no strategy willcontinue to be profitable forever. Markets may change, theories may no longer hold,and systems that were profitable once may have to be discarded for new and betterideas. I can never accept that my trading theory is truth. Popper would be proud.
FLAWS IN PERFORMANCE MEASURESToday traders use a multitude of performance measures to evaluate trading strategies. Net profit, profit factor (gross profit divided by gross loss), profit to drawdown, and percentage of profitable trades are a few of the many statistics theyemploy. I believe that each of these measures is flawed. In fact, I rarely look at anyof these measures when evaluating my own trading ideas. Let’s look at each ofthese performance measures and critique their characteristics.Net ProfitNet profit is one of the most widely quoted performance statistics. Simply put, netprofit is the dollar profit earned or lost during the life of the back-test. In manyways, it is very important. After all, a strategy that makes money is more desirablethan a strategy that loses money. But beyond simply stating whether a system isprofitable, net profit does not clarify the performance picture. Without measuring
CHAPTER 4 Evaluating Trading Strategy Performance
77
the risk and consistency of the returns, using net profit to judge performance isakin to buying an antique sports car without looking under the hood to determinethe engine’s condition.Consider two trading strategies, depicted in the chart in Figure 4.1, whichproduce the following distributions of weekly returns trading one contract of theS&P 500 futures.Although both strategies produce an average profit of $500 per week,Strategy B is twice as risky as Strategy A when measured by standard deviation ofreturns. Some traders might say, “I can stomach risk, so I only care about returns.”But with the capabilities of leverage in the stock and futures markets, we canleverage returns by factors often as high as 10:1. In the example shown in Figure4.2, we again compare Strategy A and Strategy B, only this time by trading twocontracts of Strategy A versus only one contract of Strategy B.Clearly, Strategy A is preferable. Here, both strategies have the same risk, butStrategy A’s average profit is now $1000 per week, compared with only $500 perweek for Strategy B. The comparison above describes why we must care about riskand why traders who do not respect risk will always run into trouble. Whether weclassify ourselves as traders or investors, we always want to maximize the returnper unit of risk. Leverage then allows us to scale our risk to a desired level of return.
Profit Distribution of Two Strategies
Occurrence
18%16%
Strategy A (thick)
14%
Strategy B (thin)
12%10%8%6%4%2%2000
1500
1000
500
0
–500
–1000
0%
Profit ($)
FIGURE
4.1
Profit Distribution of Two Strategies. Both Strategy A and Strategy B generate an average profit of $500per week. The volatility of each strategy is different.
PART 1 Structural Foundations for Improving Technical Trading
78
Profit Distribution of Modified Strategies
Occurrence
9%8%
Strategy A (2x) – thick
7%
Strategy B – thin
6%5%4%3%2%1%2000
1500
1000
500
0
–500
–1000
0%
Profit ($)
FIGURE
4.2
Profit Distribution of Modified Strategies. When we equalize the volatility by trading Strategy A with 2:1leverage, we see that it is the preferred strategy.
There are many other reasons why absolute returns are not important. For example, consider two strategies that trade the S&P 500. One is always in the market, andthe other attempts to time the market by holding a flat position from time to time. Ifboth produced similar net profit, we would still prefer the latter strategy. It’s less riskythan the first, due to the smaller amount of time it’s exposed to the market.Another example of the flaw in solely considering net profit can be seen incomparing the returns of various futures contracts. A strategy that makes $50,000trading one contract of corn might be better than a system that makes $100,000trading one contract of T-bonds, due to the smaller volatility of the corn futurescontract compared to the bond future. As we see in Figure 4.3, the average dailyhigh-to-low range in corn is $200, compared to $800 for the average daily rangeof the T-bond contract. Since the volatility of the T-bond contract is roughly fourtimes that of corn, one could argue that corn strategy performs much better compared to the overall volatility of each contract.Profit FactorProfit factor has also become a popular measure of trading strategy performance,due to its computational simplicity and its inclusion in popular computer programssuch as TradeStation. Profit factor is calculated by dividing the total profit gained
CHAPTER 4 Evaluating Trading Strategy Performance
79
Average Daily High-to-Low Dollar Range1400Corn range (thin)
Range in Dollars
1200
T-bond range (thick)
1000800600400
FIGURE
8/19/2002
4/11/2002
11/28/2001
7/19/2001
3/12/2001
6/22/2000
10/20/2000
Date
2/14/2000
10/1/1999
5/25/1999
1/14/1999
9/3/1998
4/28/1998
12/16/1997
8/6/1997
0
3/31/1997
200
4.3
Average High-to-Low Dollar Range. The volatility of T-bonds and corn are not equivalent. As such, wemust factor in a market’s volatility when evaluating results.
on winning trades by the total loss on losing trades. If the strategy is profitable,then gross profit will be greater than gross loss and the corresponding profit factor will be greater than one. Unprofitable strategies produce profit factors of lessthan one. Consider the five trades in Figure 4.4:Trade Number
Profit/Loss
12
+500⫺750
345
+250+1000⫺750
Gross ProfitGross LossProfit Factor
+1750⫺15001.17
FIGURE
4.4
Calculating Profit Factor. Profit Factor is calculated by dividing Gross Profit by Gross Loss.
The gross profit, calculated by summing all the winning trades, is $1750.The gross loss on all losing trades is –$1500. These trades result in a net profit of
PART 1 Structural Foundations for Improving Technical Trading
80
$250 (1750 – 1500). The profit factor is calculated by dividing the gross profit bythe gross loss (1750/1500 = profit factor of 1.17).Gross ProfitProfit Factor = ᎏᎏGross LossMany years ago I used the profit factor performance statistic to grade the system performance of over 100 strategies. After ranking each system by the profitfactor statistic, I revisited the best systems and was disappointed with the results ofthe top ones. Very often, all the profits generated were the result of one large winner, while the bulk of the remaining trades were unprofitable. Many of the systemsproduced large profits, but the inconsistency of the results made me question eachsystem’s effectiveness. As a result, I began a search for a better measure of performance. Eventually, I created a new performance measure, the K-ratio, whichwe’ll look at later in the chapter.Profit to DrawdownThe ratio of net profit to maximum drawdown is also a popular measure of performance. A drawdown occurs when net profits falls from its highest point. Thisdrawdown is calculated each trading day, and the maximum value is recorded asthe maximum drawdown.In Figure 4.5, the high in equity and the drawdown from that high is recordedeach day. The maximum of the daily drawdown numbers will be reported as the maximum drawdown of the equity curve.Figures 4.6 and 4.7 detail this process graphically. Each day of trading, wetally the highest profit achieved. Anytime the strategy suffers losses, the distance
Day
Equity
Equity High
Drawdown
012
0⫺5⫺7
000
057
34567
⫺2510125
05101212
20007
8
3
12
9
FIGURE
4.5
Calculating Drawdown. Any day’s drawdown is calculated by subtracting the current day’s equity with thehighest equity up to that point in time.
Cumulative Profit and Maximum Profit1400Profit
1200
Highest Profit10008006004002000–200
1
29
57
85 113 141 169 197 225 253 281 309 337 365 393 421 449 477
–400
FIGURE
4.6
Cumulative Profit and Maximum Profit. We track the highest profit increases the life of the test. Any retreatfrom the highest profit is a drawdown.
Maximum Drawdown500Maximum Drawdown
250
480
450
420
390
360
330
300
270
240
210
180
150
120
90
60
30
0
0
Day
FIGURE
4.7
Maximum Drawdown. The maximum drawdown is computed each day, and it’s increased when today’sdrawdown from an equity high is greater than any prior day’s drawdown.
81
82
PART 1 Structural Foundations for Improving Technical Trading
from current profits and maximum profit is recorded as drawdown. Over the lifeof the test, we keep a running tally of the maximum drawdown since the beginningof trading. Figure 4.6 depicts the current profit and maximum profit over the test.As the current profit declines from the maximum profit, a drawdown occurs. If thedrawdown is greater than any before it, then the maximum drawdown in Figure 4.7increases.Dividing net profit by maximum drawdown creates a measure of reward torisk. Riskier strategies have larger maximum drawdowns and will lead to lowerprofit-to-drawdown ratios. There is, however, an overwhelming problem with usingthe profit-to-drawdown statistic to measure performance across strategies. In a consistent trading strategy, net profit will increase linearly over time. Each periodshould produce, on average, the same profit. Maximum drawdown, however, doesnot increase linearly with time.To show how this works, I used a Monte Carlo simulator to generate theequity curve of 1000 random trials in which the average profit each day was $10,with a standard deviation of $1000. A Monte Carlo simulation involves using arandom number generator to create values subject to a user’s inputs, such as mean,standard deviation, and distribution of values.The average of these 1000 trials is shown in Figure 4.8. Note that as expected, the net profit of the simulation rises linearly with time. On average, the testmakes roughly the same amount of money each trading day when averaged over1000 trials. The maximum drawdown, however, does not increase linearly withtime. Figure 4.9 shows that the same strategy produces much different profit-todrawdown ratios, depending on the number of days in the test. This graph detailsa very large flaw in the profit-to-drawdown ratio. Essentially, using more data inback-tests will lead to higher profit-to-drawdown ratios with all things staying thesame. Traders should be aware of this inconsistency when comparing profit-todrawdown ratios of different performance tests.The profit-to-drawdown ratio will vary depending on the length of time inthe test, despite the static characteristics of the underlying profitability. Due tothis flaw, it becomes meaningless to compare profit-to-drawdown measures ofstrategies tested over differing lengths of time. The profit-to-drawdown ratiosof a 5- and a 10-year back-test cannot be compared in an apples to apples manner. One of my requirements for performance measures is the ability to comparevarious strategies regardless of the time frame studied. Because the profit-todrawdown statistic does not lend itself to this property, it should not be utilizedwhen evaluating trading performance.Percent of Profitable TradesThe percentage of profitable trades is another statistic that many use to gaugestrategy success. It is the number of winning trades divided by the total number of
Average Profit and Average Maximum Drawdown3500Profit
3000Profit and Drawdown
Maximum Drawdown2500200015001000500
300
200
0
100
0Days in TrialFIGURE
4.8
Average Profit and Maximum Drawdown. While our profit increases linearly with time, maximum drawdown does not.
Profit-to-Drawdown Ratio7.006.00
Profit-to-Drawdown Ratio
Ratio
5.004.003.002.001.00
900
800
700
600
500
400
300
200
100
0
0.00
Days in TrialFIGURE
4.9
Profit-to-Drawdown Ratio. The more days, weeks, or months in our strategy test, the higher the profit todrawdown statistic. As such, we cannot compare profit to drawdown statistics for tests using differingnumbers of days, weeks, and months.83
84
PART 1 Structural Foundations for Improving Technical Trading
trades in the back-test. Traders place too much emphasis on the percent of tradesthat are profitable, and I urge you to throw away the preconceived notion of maximizing the percent of profitable trades.As we’ve seen from the studies of behavioral finance, there’s a bias in humanthinking that closing trades with a profit is good and that taking losses is bad.Personally, I am indifferent as to what percent of trades are profitable. It has nobearing on the performance of my portfolio or my strategies. I choose to think ofmy profits and losses in the dimension of time, and examine my profit and losseach day. In some ways, I hold the standards of my strategies to a higher level thanthose who attempt to maximize the probability of winning trades since my goal isto make money every day, rather than in every trade. As such, I use performanceevaluation tools that examine my profit and loss on a daily basis.
BETTER MEASURES OF TRADING PERFORMANCEThe two main tools I use to evaluate trading performance are the Sharpe ratio andthe K-ratio. Both measures compare reward to risk in order to assess strategy performance. Let’s look at them one at a time.The Sharpe RatioDeveloped by Nobel Laureate William Sharpe, the Sharpe ratio is a standard in themoney management industry. The ratio is calculated using two statistical measureswe introduced earlier: mean and standard deviation. The numerator—the returnportion of the formula—is calculated by averaging returns over time using daily,weekly, or monthly returns. For each period’s return, we subtract the return of arisk-free instrument such as short-term Treasury bills. The denominator—the riskportion of the Sharpe ratio—is the standard deviation of returns.If returns are widely dispersed with both large winners and losers, the strategy would have a high standard deviation and would be considered risky. If returnsare wrapped tightly around the mean, the strategy would have a smaller standarddeviation and would be considered less risky.Sharpe ratio = Average return / Standard deviation of returns ⭈ Scaling factorThe quotient of average returns and the standard deviation is then multipliedby a scaling factor equal to the square root of time periods in a year. If daily returnsare used to calculate the Sharpe ratio, then the raw ratio would be multiplied bythe square root of 252 (the approximate number of trading days in a year) to arriveat an annualized value of the Sharpe ratio. If monthly returns are used, the rawratio would be multiplied by the square root of 12 (12 months in a year). This scaling is required because the expected return increases linearly with time, but standard deviation scales proportionally to the square root of time. After applying the
CHAPTER 4 Evaluating Trading Strategy Performance
85
scaling factor to create an annualized Sharpe ratio, the resulting statistic is anapples-to-apples measure of system performance that can compare strategiesregardless of the markets traded or time period studied. Whenever testing strategies, a trader’s first task should be to calculate the Sharpe ratio. Focus on findingstrategies that produce Sharpe ratios greater than positive one.The Sharpe ratio is not without its faults. Critics have argued (Schwager,1995) that the performance statistic does not accurately describe performance ifautocorrelation exists in returns. Positive autocorrelation exists when positivereturns are generally followed by more positive returns and when negative returnsare followed by more negative returns. The opposite situation, negative autocorrelation, exists when returns tend to alternate. That is, positive returns are followedby negative returns and negative returns are followed by positive returns.To understand how this affects performance, consider two systems that bothproduce 10 months of +$1000 returns and 10 months of –$500 returns. The onlydifference between the two is the timing of the returns. The first system produces10 returns of +$1000 followed by 10 returns of –$500—an example of positiveautocorrelation. The second system has alternating returns of +$1000 and–$500—an example of negative autocorrelation.Using just the Sharpe ratio, both systems would be deemed equal. But looking at the graph in Figure 4.10, which system would you rather trade? Clearly, thesecond system is preferable. The first system may have had its day in the sun, asrecent returns have degraded substantially, while the second system is still performing strong.The K-RatioSome years ago I realized that a measure was needed to complement and detect theflaws of the Sharpe ratio such as in the above example. The result was the creationof the K-ratio in 1996. Instead of looking at returns irrespective of when they occur,the K-ratio calculates performance based on the stability of the equity curve.To calculate the K-ratio we first need to create an equity curve— that is, agraph of cumulative profits over time. To calculate the K-ratio correctly, the equitycurve should increase linearly with respect to time. If strategy tests are performedusing a constant number of contracts, shares, or dollar risk, we can cumulativelysum each period’s return to create the equity curve.Most traders test their performance in this manner, so no adjustments need to bemade to the equity curve. Some traders, however, test performance by reinvesting profits. For systems that invest accumulated profits in new trades by adding contracts orshares as profits accumulate, the equity curve should increase exponentially withrespect to time due to the effect of compounding returns. If we take the natural log ofthis exponentially rising equity curve, the result will be a new adjusted equity curvethat will increase linearly with respect to time—exactly what we need for calculating
PART 1 Structural Foundations for Improving Technical Trading
86
Positive and Negative Autocorrelation3500System C (thick)3000
System D (thin)
Profit
2500200015001000500
60
57
54
51
48
45
42
39
36
33
30
27
24
21
18
15
12
9
6
3
0
0Day
FIGURE
4.10
Positive and Negative Autocorrelation. System C suffers from positive autocorrelation, while System D suffers from negative autocorrelation. While System C and System D have equal Sharpe ratios, the autocorrelation in returns should be taken into account for performance evaluation.
the K-ratio. Again, if risk—dollars, shares, or contracts at risk or traded—is kept constant throughout the life of the test, no adjustments need to be made to the equity curve.Both the linear and exponential equity curves are depicted in Figure 4.11.We start by calculating a linear regression of the equity curve to a trend variable. A linear regression is a best fit line that minimizes the squared errors betweenthe forecast and the actual values. The trend variable begins at 0 on the first day (orweek or month) of the performance test and increases by one with each new day (orweek or month). The slope of the regression line—b1 in the equation below—is ourproxy for reward in the K-ratio. It measures how quickly the equity curve rises overtime. Naturally, steeper regression lines indicate a higher level of profitability thanthat indicated by flatter regression lines.The regression equation:Equity Curvei = b0 + b1 ⭈ trendiRisk in the K-ratio is measured by calculating the standard error of the b1regression coefficient. The standard error is a statistic that measures the reliabilityof the b1 estimate calculated from the regression. Large standard errors indicate thatthe slope of the equity curve over time is inconsistent, while small standard errors
CHAPTER 4 Evaluating Trading Strategy Performance
87
Linear and Exponential Equity Curves6050Linear Equity Curve (thick)Exponential Equity Curve (thin)
4030201000
50
FIGURE
4.11
100
150
200
250
300
350
Linear and Exponential Equity Curves. A linear equity curves increases at a fixed rate by unit of time. Anexponential equity curve rises in constant percentage terms and leads to a parabolic rise over time.
indicate a more consistent equity curve. If profits are due to one large winningtrade, then standard error will likely be large due to the lack of steady returns.In the charts following, both strategies produce similar net profit. Figure4.12’s returns are consistently positive over time while the bulk of Figure 4.13’sreturns are produced from strong periods before day 10. As a result, while theregression slope in the first graph (+0.58) is similar to regression slope in the second graph (+0.55), the standard error (0.02) of Figure 4.12 is much less thanFigure 4.13’s (0.07). The lower standard error suggests that the returns associatedwith Figure 4.12 are less risky than the returns associated with Figure 4.13.The K-ratio is calculated by dividing the b1 estimate by both the standarderror of b1 and the number of periods in the performance test. By dividing by thenumber of data points, we normalize the K-ratio to be consistent regardless of theperiodicity used to calculate its components.b1K-ratio = ᎏσb1 ObsThis process can be completed in a strategy-testing software testing programsuch as TradeStation or in a spreadsheet such as Microsoft Excel. Figure 4.14details the Microsoft Excel formulas for calculating the K-ratio.
PART 1 Structural Foundations for Improving Technical Trading
88
Linear Regression of Stable Equity Curve1210
Profit
8
Equity Curve
6Linear regression
4
19
18
17
16
15
14
13
12
11
9
10
8
7
6
5
4
3
2
1
0
0
2
–2DayFIGURE
4.12
Linear Regression of Stable Equity Curve. We fit a best fit line to the path of the equity curve in order togenerate statistics for calculating the K-ratio. The equity curve above appears to rise consistently over time.
Linear Regression of Unstable Equity Curve161412Equity CurveProfit
108
Linear regression
642
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
0DayFIGURE
4.13
Linear Regression of Unstable Equity Curve. The equity curve above performs well in the first half butthen flattens towards the end of the test.
CHAPTER 4 Evaluating Trading Strategy Performance
Day
89
System 1
System 2
0
0.00
0.00
1
⫺ 0.15
1.76
23
0.721.23
3.553.99
4
2.50
5.90
56
2.332.54
7.137.41
7
3.63
7.64
89
4.916.25
8.8410.35
10
6.32
11.05
11
6.60
11.59
1213
6.757.92
10.9011.25
14
8.30
11.60
151617
9.009.229.36
11.9911.1611.67
1819
9.3710.17
10.8910.30
b1s.e. b1
=SLOPE(B2:B21,$A2:$A21)=SLOPE(C2:C21,$A2:$A21)=STEYX(B2:B21,$A2:$A21) / =STEYX(C2:C21,$A2:$A21)/
ObservationsK-ratio
SQRT(DEVSQ($A2:$A21))20=B22/(B23*A24)
SQRT(DEVSQ($A2:$A21))20=C22/(C23*A24)
b1s.e. b1
0.580.02
0.550.07
K-ratio
1.32
0.40
FIGURE
4.14
Excel formulas for calculating the K-ratio. The K-ratio can be calculated in Excel using the above formulas.
The K-ratio is a unitless measure of performance that can be comparedacross markets and time periods. Weekly performance of corn futures can be compared with tick data performance of trading IBM. Traders should search for strategies yielding K-ratios greater than +0.50. Together, the Sharpe ratio and K-ratioare the most important measures when evaluating trading strategy performance.Note: When I created the K-ratio in 1996, I thought I had created a robustmeasure to evaluate performance. In mid-2000, trader Bob Fuchs brought a smallerror to my attention regarding the scaling of the K-ratio. He was correct in his critique and I have corrected the error in this text. Publications prior to 2002 will showa different formula for the K-ratio. The updated formula in this book is correct.
PART 1 Structural Foundations for Improving Technical Trading
90
COMPARISON OF BENCHMARK STRATEGIESWe’ve identified our primary measures of trading strategy performance: theSharpe and K-ratios. In addition to these two, we will use a number of benchmarkstrategies to evaluate new ideas.Channel breakouts and moving average crossover systems are among the mostpopular strategies used by traders today. The performance of both trend-followingmethods has been documented in trade publications and academic journals. Whentesting any new idea, we will always compare performance to these two strategies.In my Comparison of Popular Trading Strategies, I found that an overwhelmingmajority of new strategies actually underperformed the simple channel breakout andmoving average crossover methodologies.For the channel breakout, we use the following rules:Enter long if today’s close is the highest close of the past 40 days.■ Exit long if today’s close is the lowest