How to evaluate whether the synthetic control is good enough I'm using the synthetic control method to evaluate a state-level policy (using synth package in Stata). I've read the seminal ADH papers on CA tobacco &amp; Germany reunification, but I still don't know exactly how to decide if the constructed synthetic state is a good fit. What we do at the moment is looking at the graph - to see if there's overlap between the synthetic and the real state. However, I feel this is not a very "robust" way... 

Is there any measure that we can use or test on?

Edit:

After reading through reply (and papers mentioned again), I have another related (but maybe a bit nuanced) question on what to expect at the time of (or immediate after) intervention. 

As shown in the two following figures from ADH (2010) and ADH (2015), there is a small effect in the same year of the intervention for CA tobacco Propostion 99, while in the Germany reunification study, the gap only becomes noticeable 2 years later. In some other papers using SC, there is an immediate effect (such as Cavallo et al. (2013)). I suppose as long as there is a consistent story to support the results, the short-term effect size of the intervention will not invalidate the fitness of synthetic control?
This is an excellent question, but unfortunately with no (yet) agreed upon answer. 

Many authors such as in those seminal papers use cross-validation for this. This is basically what is implied in the ADH (2015) paper on German unification. If the synthetic control is good the MAPE (mean absolute prediction error) and RMSPE (root mean square prediction error) should be quite low. Additional cross-validation where the pre-treatment period is split into training and testing period is so far the common way to go, although there is no agreed upon value of how low the prediction errors should be, just that lower is always better. 

Next it is also argued that examining if the trend in the countries from donor pool that are assigned non-zero weights and seeing if the pre-treatment trend in those donor countries seem to match the pre-treatment trend of treated country similar way as you would do for DiD (Difference in Differences) as argued here. 

Lastly, as crazy as it may seem, yes visual inspection is actually recommended as well. 

Beside the above you should also conduct sensitivity analysis by systematically trying to vary the number of countries in the donor pool and see if the resulting synthetic control has still a good fit. That's again also done by ADH 2015 and you will find it in almost all studies as well. Publishing your weights is also considered good practice. 

Of course, doing a proper cross-validation is difficult if you have small pre-treatment sample. I was once working on a synthetic control study at one of the euro-system member's central bank with few colleagues where we had very short pre-treatment period due to data constraints. Instead of doing cross-validation we decided to just test whether the real pre-treatment series is statistically significantly different from the synthetic series using couple of non-parametric tests. When we presented the results and methodology at a workshop no objections to that were raised, and there were some very excellent econometricians present in the room. However, beside central bank's brief the study was not published because the results were not very interesting as placebo test showed results were not significant and we decided to abandon the project since we judged chance of publication to be low so the idea never got proper peer review, so take that as you will. 



Response to edit:

as with classical DiD there is always an issue of potentially delayed effects or anticipation effects that would bias the result. However, in this case the authors argue that the increase after the reunification was actually part of the effect and not delayed effect. As the authors say:


  Our estimate of the effect of the German reunification on per capita GDP in West Germany is given by the
  difference between the actual West Germany and its synthetic version, visualized in Figure 3. We estimate that the
  German reunification did not have much of an effect on
  West German per capita GDP in the first two years immediately following reunification. In this initial period, per
  capita GDP in the synthetic West Germany is even slightly
  lower than in the actual West Germany, which is broadly
  in line with arguments about an initial demand boom
  (see, e.g., Meinbardt et al. 1995). From 1992 onward,
  however, the two lines diverge substantially. While per
  capita GDP growth decelerates in West Germany, for the
  synthetic West Germany per capita GDP keeps ascending
  at a pace similar to that of the prereunification period.


However, arguably this is a weak point of the paper.

Otherwise, if there would be a genuine delay in the effect you would have a problem that could be solved similarly as with DiD by identifying the 'correct' date where the treatment started to have an effect and identify the pre and post treatment period based on that.