How good is a forecast? (Many outputs) I have several macroeconomic models (eg two DSGEs and a VAR). They each produce forecasts for GDP, inflation and unemployment, and I have data to test them on, going back years.
How can I say which model has the &quot;best&quot; overall performance?
I realise this is vague, so here's more detail:
Assume that model X performs better than model Y on GDP, but worse on inflation.
If I'm just looking at GDP, I could just ask something like &quot;which model has the smallest mean square error?&quot; (predicted GDP - actual GDP, squared)
Similarly, if I only care about inflation, I could do the same thing.
But I actually care about forecast accuracy over all three variables (GDP/inf/unemployment).
Is there a way to assess accuracy over all three variables, without just assigning weights to each MSE value and adding them up?
Thanks for your help!
The MSE is essentially a squared Euclidean distance between two vectors, say $\mathbf y$ and $\hat{\mathbf y}$, where $\mathbf y$ is the actual economic data over $T$ periods and $\hat{\mathbf{y}}$ the predicted values. A natural extension of this to matrices $\mathbf Y=(y_{it})$ and $\widehat{\mathbf Y}=(\hat y_{it})$ where $i=1,\dots,n$ and $t=1,\dots,T$ ($n$ variables over $T$ periods) would be
\begin{equation}
\text{MSE}^*=\frac{1}{nT}\sum_{i=1}^n\sum_{t=1}^T(y_{it}-\hat y_{it})^2.
\end{equation}
But beware of the pitfall. Of course there are other matrix norms available as well.