our stop, we can work back-ward  from  the  stop  level  that  MAE  supports  to  the  capitalwe’ll need to trade the way we wish. Alternatively, we couldconclude that we just shouldn’t trade the way we tested. Manya trading idea ends right them. MAE will define your stop foryou, but if it’s too big for your wallet, it’s telling you to forgettrading the rules you used. Don’t ignore this message.
6.  Sort  the  trades’  MAEs  into  the  bins.  We  have  the  trades  sepa-rated into winners and losers. Now we break them up by theamount  of  adverse  excursion  they  showed.  Using  the  datafrom Table 3.3, Table 3.4 sorts the winners and losers by thesizes  of  their  MAEs.   Table  3.5  shows  the  frequency  of  win-ners and losers in each bin.
7.  Graph the  summary  table.  Last,  convert  the  summary  tableinto  a  chart.  If  everything  so  far  has  been  done  in  a  spread-sheet, plotting won’t be  difficult6  Figure 3.3 shows our sum-mary  chart.
Figure  3.3  has  some  useful  information  for  us  already.  Tradingthese  rules  in  1983-1984  would  not  have  been  too  difficult  with  a$600  stop  loss;  we  didn’t  see  a  single  trade  that  exceeded  that.  If
6 Unfortunately, MAE is not a technique that’s taken  fire since  its publication in1985. I’m not aware of publicly available software that explicitly does this, thoughthere  are  trading  systems  and  private  software  that  use  the  idea.  MetaStock   forWindows does record the  MAEs in its testing reports.
52
Handling  the  Bad  News
How  Bad  Does  It  Get?
53
TABLE  3.4
BINNING.  The  MAEs  of  trades  in  Table  3.3
are  dumped  into  different  bins  by  their  size.
MAE
O-.30   .31-.60   .61-.90   . 9 1 - 1 . 2 0   1 . 2 1 - 1 . 5 0   1.51+
Winners
Losers
.07,.Ol,.08,
0,.06,
0,.04,.15,.07,.27
.08,.02,.25
.46
this were reliable (and it isn’t, given the low number of trades), we’dbe  comfortable  using  the  $600  figure  and  also  trading  the  marketwith our $30,000. The chart would be solid evidence that our totalcapital and loss amounts per trade were realistic.
Look,  too,  at  the  shape  of  the  two  curves.  The  winning  curveamounts  to  a  single  column  but,  if  things  truly  were  random,  itshould have been smeared all over the different bins.
In a phenomenon we’ll see repeated, the size of the adverse pricemovements  for  winners  is  concentrated  toward  the  left  side  of  thechart  where  the  adverse  excursion,  measured  in  trading  points,  islow  or  zero.  Thus  the  rule  that  good  trades  just  don’t  go  faragainst  us
TABLE  3.5
SUMMARY  TABLE.  The  frequency  of  occurrences  in  each
bin  is  counted.  Graph  this  table  for  convenience  and  analysis.
MAE
O-.30   .31-.60   .61-.90   .91-1.20   1 . 2 1 - 1 . 5 0   1.51+
W i n n e r s   1 0Losers3
1
SUMMARY CHART.  Number  of  trades  versus  adverse
FIGURE  3.3excursion,  1983-l  984.  Not  only  are  there  more  winners  (byhappenstance) in this example, but they also tend to have low adverseprice  movement.  This  is  only  an  illustrative  chart  as  there  aren’t  enoughinstances  for  the  chart’s  information  to  be  reliable  or  significant.
Meanwhile,  the  losing  trades  have  a  broader  distribution  of  ad-verse  excursion,  Although  there  were  only  four  of  them  in  this  ex-ample,  one  of  them  went  46  points  in  the  hole  before  the  tradetapped  out.  This  phenomenon  of  large  adverse  excursions  for  los-ing trades is also common, perhaps necessarily.
To  continue  thinking  about  the  shapes  of  these  curves,  whatwould  it  mean  if  the  two  distributions  were  shaped  like  those  inFigure  3.4?
A  chart  without  discernible,  distinct  patterns  stems  from  tradingrules  that  can’t  discern  winners  from  losers.  After  all,  winners  justshouldn’t  go  very  far  against  you,  whereas  we  expect  losers  to  gothe wrong way, yielding large adverse price movements. Being ableto  discern  these  distinct  patterns  is  the  first  thing  you  want  toachieve when looking at an MAE chart.
Last, Figure 3.3 gives a clue as to when to get out of a trade. Theold adage “When you’re wrong, get out!” gets an operational defi-nition.  You  know  you’re  wrong  if  your  trade  goes  more  than  30ticks  against  you  because,  in  this  experience,  winning  trades  just
5 4
Handling 
the  Bad  News
How  Bad  Does 
It  Get?
5 5
TABLE  3.7  FREQUENCY  BY  BIN.
MAE
Winners L0SWS
o-.7
61
.8-.15 
.16-.22  
.23-.30  
.31-.37  
.3&.45  
.46-.52
21
1
1
1
chart?” Well, one place would be at 30 ticks  (.30).  The rule wouldbe:  Enter  when  both  averages  go  upward  or  downward  togetherand  place  a  stop  at  30  ticks  opposite  the  direction  of  the  trade.Table  3.8  shows 
the  results.
What happened here was the reduction of a large loss. Accord-ing  to  the  rule  of  winning  by  minimizing  the  size  of  your  largestloss,  even  this  simple  cutoff  serves  well.  However,  you  might  wantto cut things even finer. Table 3.9 shows the effect of cutting off thetrades at .08,  or 8 ticks.
The  effect  of  cutting  off  the  trades  at  .08   is  to  reduce  the  netprofit,  but  with  lower  loss  potential  since  the  largest  loss  would  bejust 8 ticks. You end up with 7 winners and 7 losers instead of 11winners and 4 losers, showing the effect of setting stops too close.Don’t  believe  for  a  moment  you’re  going  to  trade  Crude  mechani-cally with an  &tick  stop,  but  the  idea  of  being  able  to  test  the  im-
FIGURE  3.5 QUARTER-SIZE BINS.  Number of trades versus adverseexcursion. Splitting the categories along the X axis in half shows thewinners’extreme   concentration  in  the  low  adverse  excursion  realm  andthe  losers’  greater  spread.
FIGURE  3.4 SMEARED.  Number of trades versus  adverse excursionduring  trade.  Adverse  price  movement  is  spread  randomly  in  this  graph,indicating  little  difference  between  that  of  losing  trades  and  that  ofwinning  trades.  A  display  like  this  indicates  that  the  trading  rules  havelittle  ability  to  distinguish  winners  from  losers.
don’t go more than 30 ticks bad. You could even cut it more finelyby  breaking  down  the  first  bin  (0  to  .30)   into,  say,  four  bins  (0  to.07,   .08  to .15,  .I6 to  .22,  and .23  to .30).  Table 3.6 shows the win-ners’  and  losers’  MAEs  broken  down  into  these  quarter-size  bins,and  Table  3.7  shows  the  frequency  of  winners  and  losers  in  eachbin.  Figure  3.5  presents  these  data  in  a  chart  that  illustrates  thewinners’  high  concentration  of  low  adverse  excursion  and  thelosers’  greater 
spread.
Keeping  mind  that  this  is  an  exemplary  chart,  the  question  nowshould  be  “Exactly  where  would  the  stop  go  if  we  believed  this
TABLE  3.6  TRADES  BY  BIN.
MAE
o-.07
.08-.15
.16-.22  
.23-.30  
.31-.37 
.38-.45  
.46-.52
Winners
.08,
.15
.07,   .Ol,0, .06,.04,   .07
LO%35
0.02
0.08
0.27
0.25
0.46
56
Handling  the  Bad  News
TABLE  3.8
IMPACT  OF  STOPS.
Before  using stopUsing stop at  .30
Net  Profit  or  Loss
4.56 - .58 = 3.984.56 - .48’ = 4.08
pact of adjusting your stop should strike a chord.  Here is  apracti-cal  method of adjusting the stops to the combination of loss  andpro@   level  you  think  is  appropriate.
Typically, profit will decline as you accept less and less chance ofloss,  just  as  you’d  expect.  With  enough  trades,  you  could  picturethe tradeoff. Table 3.10 shows how this calculation would run, andFigure 3.6 presents these data as a chart.
Seen in graphic format, finding the cutoff point between loss andprofit  usually  is  feasible.  By  happenstance,  this  peculiar  section  ofCrude data shows a sharp tradeoff with stops set at .08  and .15.  Af-ter that, there’s not much additional money to be made for absorb-ing adverse excursion beyond 15 ticks,
Results  Using  Crude  Oil
An  example  is  fine  for  learning,  but  what  are  the  results  of  actuallyusing  this  procedure  on  3,069  days  of  perpetual  Crude  data?  Fig-ure 3.7 shows the actual results in the long term. Table 3.11 showsthe frequency of winners and losers in each bin.
TABLE  3.9  CALCULATING  STOP  IMPACT.
Net Profit  or Loss
Before  using stop
4.56  -  .58   =  3.98
Using  stop  at  .08
.OV  +  .7  -  .08 +  .08 + .35 + .34 + .25 - .08+.05-.08-.08-.08-.08-.02=1.36
’ .48 = .58 - C.40 .30).  This reduces the size of the loss on the large loser from anet profit of .40 to .30.
57
58 Handline  the Bad News
How  Bad  Doer  It  Get?
59
TABLE  3.11 MAE  DISTRIBUTION  TRADING  CRUDE  OIL.Bin size indicates a range, for example,  .3 includes all tradeswith  adverse  excursions  rarxine. from  .16  to  .3.
The  overall  message  of  the  example-that  winners  just  don’t  govery  far  against  you-is  borne  out  in  this  larger  sample  of  data.Nearly  all  the  winners  are  clustered  in  the  first  two  bins,  whereasthe  losers, despite a substantial  65   in  the  first  two  bins,  also  havemore  than  half  of  their  number  in  bins  .45   and  larger.  This  gives  usthe  chance  to  make  an  operational  distinction-placing  stops-be-tween winners and losers.
For  example,  what  if  we  said  we’ll  just  stop  out  trades  with  anadverse  excursion  of  more  than  31  ticks  or  roughly  $300?   Clearlywe’d cut off a lot of trades with larger MAEs  but we’d also lose 10trades (4 + 2 + 2 + 1  + 1) that were eventual winners even thoughthey had large  MAEs.  What’s the net effect? Figure 3.8 summarizesthe  effect  of  different  stop  levels  in  our  exemplary  Crude  trading:
TheEdge
This  cutoff  line  beyond  which  winning  trades  rarely  go  may  be  thelong-sought  “edge”  in  trading.  Knowing  where  the  line  is-know-ing  where  to  put  one’s  stops  based  on  actual  experience--doesn’tguarantee  future  success,  but  it’s  a  lot  more  rational  than  arbitrarydollar  figures  (money  management  stops),  percentage  stops,  indi-cator  stops,  or  other  nonexperiential  stops.  Because  the  cutoff  lineis  quantifiable,  it  speeds  up  the  process  of  implementing  experi-ence  you’d  otherwise  gain  over  years  of  experimenting  on  stopplacement.  The  funny  thing,  though,  is  that  the  edge  has  nothingto  do  with  finding  a  good  trade,  only  with  minimizing  the  trade’slosses.
FIGURE  3.6 NET PROFIT VS. STOP LEVEL.  A stop level of .15  seemsto  capture  nearly  all  the  profit  available  from  this  trading  system,  thoughhigher  profits  are  available  at  a  stop  level  of  .3.   Gathering  MAEinformation makes this analysis possible.
FIGURE  3.7 CRUDE  MAE.  Number of trades by maximum adverseexcursion,  1983-l  994,  two-moving-average  trading  rule.  Nearly  allwins showed adverse price movement of no more than  .3 or 30 ticks,whereas more than half the losers showed adverse movement of morethan  30  ticks.  The  sharp  distinction  between  the  two  distributions  is  thebasis for implementing MAE stops.
60
Handline  the Bad News
Exemolarv  Tradine   6 1
ture  will  not  experience  the  same  shocks  that  markets  in  the  pasthave  experienced.  The  result  will  be  adverse  excursions  outsidewhat we expect, and I will take different small losses than I expect.That’s fine as long as they’re small, though. If adverse price move-ment is less than I’ve experienced in the past. all the better.
Admittedly,  letting  experience  set  stop  levels  is  not  complicated,
but  the  very  simplicity  gives  it  durability.
The  charts  of  adverse  movement  also  give  an  immediate  pictureof  whether  a  set  of  trading  rules  can  distinguish  between  goodtrades (i.e., ones that don’t go far bad) and bad trades. If the rulesdon’t  generate  distinctly  different  distributions,  they  are  likely  to  bemeaningless.  They  immediately  answer  the  question  “What  hap-pens when I trade a specific market in a specific way?”
What I really like about this sort of analysis of experience is thatit’s  imperceptible  to  competitive  traders  and  analysts  bound  up  inparametric  statistics  and  charts.  Moreover,  because  it’s  applicableto  any  set  of  trading  rules,  its  results  are  unique  and  my  stops  areless  likely  to  rest  around  the  levels  where  everyone  else  normallyclusters.
EXEMPLARY  TRADING
The chapters ahead use this technique within the context of a cam-paign  of  trading.  Although  there  are  far  too  many  data  to  show  all252 trades from trending alone,  1 do want you to have some sensefor  how  this  underlying  trend  system  trades  so  that  when  we  talkabout refinements later, you’ll have a head start. As a trending sys-tem,  it’s  usually  late  and  subject  to  whipsaws.  Its  good  points  arethat  it  never  misses  a  trend  and  rarely  countertrades  into  a  trend.Figures 3.9 and 3.10 show the good side and the ugly side of ourtrend trader.
Although  you  haven’t  gone  through  all  3,069  days  yourself,  youcan  see  from  these  charts  that  the  normal  amounts  of  misery  andelation are to be had from this system. It will prove to be normallycantankerous  in  trying  to  make  money  as  we  add  refinements  tothis basic approach.
PROFITABILITY. Profitability  versus  size  of  stop,
FIGURE  3.81983-1995, two-moving-average trading rule. As the stop gets bigger,profits  peak  then  decline  steadily  before  going  roughly  flat.  Traders  usingthe  information  in  Figure  3.7  would  use  this  to  choose  between  stops  at.31  and  .46.
The  edge  does  something  else  for  you:  quite  often,  its  use  canconvert a losing  system to a winner or take a set of rules that losesmore  often  than  it  wins  and  make  it  profitable.  There’s  also  atremendous  benefit  from  being  protected  from  wipeout,  a  cata-strophic  loss.  Perhaps  with  the  experiential  evidence  gathered  thisway, you’ll be sufficiently confident to use stops or something oper-ationally  equivalent.  Just  using  MAE  stops  should  materially  im-prove  your  trading.
THE  ARGUMENT  FROM  EXPERIENCE
Stop for a moment and think about what’s been done here.
We’ve  tabulated  the  actual  experience  resulting  from  followingan  explicit  set  of  entry  and  exit  rules.  They  could  have  been  anyrules;  the  analysis  of  adverse  price  movement  would  not  havechanged.  What  expectation  is  there  that  the  experience  will  carryforward  into  the  future?  Certainly,  the  markets  that  trade~in  the fu-
Camoaien   Wrao-Uo
  6 3
CAMPAIGN  WRAP-UP
Just  as  in  a  war  there  will  be  casualties,  so  in  campaigning  therewill  be  losses.  The  campaigning  trader,  having  studied  the  ground,will  have  a  pretty  good  idea  of  the  size  and  frequency  of  thoselosses (and the wins) from his or her study of adverse price move-ment  from  the  point  of  entry.
Before  going  on  to  use  this  technique  on  live  data,  you  shouldfeel  pretty  comfortable  that  you  understand  how  MAE  is  calcu-lated and why you’d use it. Using this concept, the next chapter willstep  directly  into  testing  different  vehicles.
THE  GOOD.  Like  a  good  trending  system,  our  two-
FIGURE  3.9moving-average  example  reliably  catches  every  trend,  though  thingsrarely go this  smoothly in the real world.
FIGURE  3.10experience  would  indicate  during  whipsaw  trading:  badly.
THE UGLY. Our underlying system trades just as
6 2
4Testing
In  this  chapter,  we  pick  what  contracts  to  test  and  subsequentlytrade.  For  stocks  there  is  no  choice,  but  futures  offer  a  variety  ofcontracts we may use. Using New York Light Crude as an example,we  look  at  the  testing  from  the  standpoint  of  loss  control  andprofitability  before  selecting  a  testing  vehicle  for  our  campaign.
Testing  trading  systems  has  been  covered  thoroughly  in  otherpublications’  so  I’ll  hit  only  the  highlights  here,  particularly  withrespect to crude oil.
Testing  in  general  is  suspect  because  we  aren’t  dealing  with  arepetitive process where we expect to find  x that causes  .r.  We arepainfully  aware  that  the  world  is  evolving  and  changing.  Therefore,the markets are evolving and changing. New markets come into be-ing and old markets fade away; new stocks are created and old fa-vorites  die,  dissolve,  or  fall  out  of  favor.  The  ground  on  which  wevalidate  our  trading  rules  won’t  be  the  ground  on  which  they  areused.
I  Tushar  S.  Chande   and  Stanley  Kroll,  The  New  Technical  Trader  (New  York:John  Wiley and Sons,  Inc.; 1994). Charles  Le Beau and David W. Lucas,  TechnicalTraders  Guide  to  Computer  Analysis  t?f  the   Futures   Market  (Homewood:   Irwin,1992).  Bruce  Babcock,  Jr.,  The  Business  One  Irwin  Guide  to  Trading  System(Homewood:  Business One, 1989).
65
66  Testing
Scientific  Process  67
What  may  be  consistent-and  we  have  weak  evidence  of  this-iscrowd  behavior.  Many  speculative  bubbles  have  been  documentedhistorically,  and  these  occur  even  in  laboratory  situations?  It  maybe that the behavior of crowds in times of stress is consistent. Per-sonally,  I  believe  that  fear  and  greed  (mostly  fear),  despair  andhope, drive most adult actions. People in a fearful state are not cal-culating  probabilistic,  discounted,  future  cash  flows!  However,  inlarge numbers, their actions may be consistent. That is my assump-tion, at any rate.
There  is  other  evidence.  Just  in  terms  of  national  and  interna-tional  economics,  it’s  tough  after  experiencing  two  or  three  busi-ness  cycles  to  ignore  the  feeling  that  comes  over  you  that  someevents  are  relatively  predictable.  Cyclical  content  in  economic  timeseries has been analyzed for several centuries and the “business cy-cle” is a workaday business concept. I recall, for example, the as-set/liability  management  committee  of  a  bank  where  I  worked  be-ing  quite  comfortable  with  its  interest  rate  outlook,  the  Fed’sstance,  and  its  consequent  positioning  of  the  bank’s  balance  sheetto exploit the business cycle.
In  the  end,  the  rationale  for  testing,  whether  on  paper  or  withAudiTrack,   is  that  there  is  no  alternative  preparation  for  puttingmoney  at  risk  in  technical  speculation.  So  we  might  as  well  get  onwith  it.
YOUR  PERSONAL  EXPERIENCE
One thing that can be said in testing’s favor is that it accelerates thebuilding  of  our  own  experience  with  the  market.  There  is  certainly
2  Ronald  R.  King,  Vernon  L.  Smith,  Arlington  W  Williams,  and  Mark  Van  Boen-ing,  “The  Robustness  of  Bubbles  and  Crashes  in  Experimental  Stock  Markets,“inR.  Day  and  F!  Chen,  Non-/inear  Dl’namics  and  Evolutionary  Economics  (Oxford:Oxford  Press,  1993):   pp.  182-200.  Vernon Smith  a  the  Economic  Science  Labora-tory, University of Arizona, is an excellent resource on  this topic.’  AudiTrack   is  a  real-time  paper  broker  io  whom  orders  can  be  phoned  for  ac-counting  and  tabulation  of  trading  records.  AUDITRACK;~Inc.,   (407)  393-3876.
craft in trading. We just don’t know what it is, precisely; how to for-malize  it;  or  how  to  hand  it  on.  Institutionally,  it’s  usually  learnedon  the  job  with  more  or  less  supervision  and  guidance  dependingon the house.
For  individual  traders,  there’s  learning  by  observing  for  severalyears,  but  few  follow  that  path.  The  other  way  is  stepping  throughtrades day by day, by hand, with charts. I recommend this ancientpractice because you have many chances to both fool yourself andbe  honest  with  yourself.  The  distinction  will  eventually  dawn  onyou.  You  learn  quite  viscerally  the  difficulties  of  trading  in  themanner you propose and you pick up the patterns that make senseto  you.  Creating  charts  is  much  better  for  learning  patterns  thansimply  reviewing  a  raft  of  them  on  a  screen  or  shuffling  through200  printouts  from  a  screening  program.  These  somehow  fail  tobuild  an  impression  in  one’s  mind,  whereas  manually  plottingcharts day after day does inculcate one, perhaps through some sub-conscious  process,  with  usable  intuition.
When  it  comes  to  testing,  the  alternative  to  personal  experienceis  reading  the  nifty  trading  summaries  produced  by  computers  andtheir software. While these outputs are objective, I suggest that theonly  interesting  things  on  these  summaries  are  the  net  profits  lineand  the  risk  measures  such  as  the  Sharp  Ratio  or  Traynor  Index.I’m in the minority here, most people look at the percentage of suc-cessful  trades,  number  of  winning  or  losing  longs  or  shorts  and soforth.  These  items  are  secondary.  We  know  going  in  that  if  youcontrol  the  size  of  your  losses,  you  can  later  worry  about  beingright  more  often.
SCIENTIFIC  PROCESS
The first step in a scientific process is direct observation and mea-surement,  Shortly,  I’ll  show  you  more  of  this  than  you’ll  want  tosee,  but  it’s  important  to  know  where  we’re  at  regarding  marketphenomena.  At  this  stage  of  market  knowledge,  simply  identifyingand  measuring  distinct  phenomena  in  market  behavior  is  a  stepforward. There may be private practitioners who’ve gone farther in
6 8   Testine
Details  of  Specific  Contracts
69
the scientific process and developed theory, but my sense of the sit-uation  is  that  we  are  at  stage  one  and  no  proper  theory  of  marketbehavior exists. I’m not proposing one, nor do I say that I’m under-taking even the observational process properly. I’m a trader, not aninvestigator.  I  only  hope  to  place  my  observations  in  their  properframe of reference, leave a thorough investigation to others, and ex-ploit their use now for my own gain.
Therefore, when we do undertake testing of a trading system, wedo  so  with  a  grain  of  salt.  Prudence  forces  us  to  estimate  ourchances  of  success,  and  we  have  no  other  means  than  looking  athistorical  performance,  knowing  full  well  that  the  future  campaignwill not be the same as the last one. Our hope is that the reactionsof the market’s mob will be of the type, speed, and nature of its re-actions  in  the  past.  Occasionally  (1987,  for  example)4   this  is  nottrue, but day in and day out, we cannot perceive great differences inbehavior,  so  we  defer  to  necessity  and  test.
IN  SAMPLE  OR  OUT  OF  SAMPLE
It’s thought that, ideally, when testing a trading system, we’d createit using one set of data and test it on a second, third, or fourth set,perhaps  stepping  forward  in  time  as  we  do  so.  This  might  give  usan  opportunity  to  discover  problems  with  the  trading  rules  andmodify  the  rules  somewhat,  always  keeping  in  mind  that  the  moreelaborate the rules, the more rigid the systems and the less likely itis  to  behave  robustly  in  the  future.  Actually,  all  ste