{"word_count": 1109, "character_count": 11462}
This post is the second one of our series on the history and foundations of econometric and machine learning models. Part 1 is online here.Geometric Properties of this Linear ModelLet's define the scalar product in [latex]\mathbb{R}^n[/latex], [latex]⟨\mathbf{a},\mathbf{b}⟩=\mathbf{a}^T\mathbf{b}[/latex], and let's note [latex]\|\cdot\|[/latex] the associated Euclidean standard, [latex]\|\mathbf{a}\|=\sqrt{\mathbf{a}^T\mathbf{a}}[/latex] (denoted [latex]\|\cdot\|_{\ell_2}[/latex] in the next post). Note [latex]\mathcal{E}_X[/latex] the space generated by all linear combinations of the [latex]\mathbf{X}[/latex] components (adding the constant). If the explanatory variables are linearly independent, [latex]\mathbf{X}[/latex] is a full (column) rank matrix and [latex]\mathcal{E}_X[/latex] is a space of dimension [latex]p+1[/latex]. Let's assume from now on that the variables [latex]\mathbf{x}[/latex]  and [latex]y[/latex] are centered here. Note that no law hypothesis is made in this section, the geometric properties are derived from the properties of expectation and variance in the set of finite variance variables.With this notation, it should be noted that the linear model is written [latex] m(\mathbf{x})=⟨\mathbf{x},\beta⟩[/latex]. The space [latex] H_z=\{\mathbf{x}\in\mathbb{R}^{p+1}:m(\mathbf{x})=z\}[/latex] is a hyperplane (affine) that separates the space in two. Let's define the orthogonal projection operator on [latex]\mathcal{E}_X[/latex], [latex]\Pi_X =\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T[/latex]. Thus, the forecast that can be made for it is: [latex display="true"]\widehat{\mathbf{y}}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T\mathbf{y}=\Pi_X\mathbf{y}[/latex]. As, [latex]\widehat{\varepsilon}=\mathbf{y}-\widehat{\mathbf{y}}=(\mathbb{I}-\Pi_X)\mathbf{y}=\Pi_{X^\perp}\mathbf{y}[/latex], we note that [latex]\widehat{\varepsilon}\perp\mathbf{x}[/latex], which will be interpreted as meaning that residuals are a term of innovation, unpredictable in the sense that [latex]\Pi_{X }\widehat{\varepsilon}=\mathbf{0}[/latex]. The Pythagorean theorem is written here: [latex display="true"]\Vert \mathbf{y} \Vert^2=\Vert \Pi_{ {X}}\mathbf{y} \Vert^2+\Vert \Pi_{ {X}^\perp}\mathbf{y} \Vert^2=\Vert \Pi_{ {X}}\mathbf{y}\Vert^2+\Vert \mathbf{y}-\Pi_{ {X}}\mathbf{y}\Vert^2=\Vert\widehat{\mathbf{y}}\Vert^2+\Vert\widehat{\mathbf{\varepsilon}}\Vert^2[/latex]which is classically translated in terms of the sum of squares: [latex display="true"]\underbrace{\sum_{i=1}^n y_i^2}_{n\times\text{total variance}}=\underbrace{\sum_{i=1}^n \widehat{y}_i^2}_{n\times\text{explained variance}}+\underbrace{\sum_{i=1}^n (y_i-\widehat{y}_i)^2}_{n\times\text{residual variance}} [/latex]The coefficient of determination, [latex]R^2[/latex], is then interpreted as the square of the cosine of the angle [latex]\theta[/latex] between [latex]\mathbf{y}[/latex] and [latex]\Pi_X \mathbf{y}[/latex] : [latex display="true"]R^2=\frac{\Vert \Pi_{{X}} \mathbf{y}\Vert^2}{\Vert \mathbf{y}\Vert^2}=1-\frac{\Vert \Pi_{ {X}^\perp} \mathbf{y}\Vert^2}{\Vert \mathbf {y}\Vert^2}=\cos^2(\theta)[/latex]An important application was obtained by Frish &#38; Waugh (1933), when the explanatory variables are divided into two groups, [latex]\mathbf{X}=[\mathbf{X}_1 |\mathbf{X}_2][/latex], so that the regression becomes [latex]y=\beta_0+\mathbf{X}_1 β_1+\mathbf{X}_2 β_2+\varepsilon[/latex]. Frish &#38; Waugh (1933) showed that two successive projections could be considered. Indeed, if [latex] \mathbf{y}_2^\star=\Pi_{X_1^\perp} \mathbf{y}[/latex] and [latex]X_2^\star=\Pi_{X_1^\perp}\mathbf{X}_2[/latex], we can show that [latex display="true"]\widehat{\beta} _2=[{\mathbf{X}_2^\star}^T \mathbf{X}_2^\star]^{-1}{\mathbf{X}_2^\star}^T \mathbf{y}_2^\star[/latex] In other words, the overall estimate is equivalent to the combination of independent estimates of the two models if [latex]\mathbf{X}_2^\star=\mathbf{X}_2[/latex], i.e. [latex]\mathbf{X}_2\in \mathcal{E}_{X_1}^\perp[/latex], which can be noted [latex]\mathbf{x}_1\perp\mathbf{x}_2[/latex] We obtain here the Frisch-Waugh theorem which guarantees that if the explanatory variables between the two groups are orthogonal, then the overall estimate is equivalent to two independent regressions, on each of the sets of explanatory variables. This is a theorem of double projection, on orthogonal spaces. Many results and interpretations are obtained through geometric interpretations (fundamentally related to the links between conditional expectation and the orthogonal projection in space of variables of finite variance).This geometric interpretation might help to get a better understanding of the problem of under-identification, i.e. the case where the real model would be [latex]y_i=\beta_0+ \mathbf{x}_1^T \beta_1+\mathbf{x}_2^T \beta_2+\varepsilon_i[/latex], but the estimated model is [latex]y_i=b_0+\mathbf{x}_1^T \mathbf{b}_1+\eta_i[/latex]. The maximum likelihood estimator of [latex]\mathbf{b}_1[/latex] is [latex display="true"]\widehat{\mathbf{b}}_1=\mathbf {\beta}_1 + \underbrace{ (\mathbf {X}_1^T\mathbf {X}_1)^{-1} \mathbf {X}_1^T \mathbf {X}_{2} \mathbf{\beta}_2}_{\mathbf{\beta}_{12}}+\underbrace{(\mathbf{X}_1^{T}\mathbf{X}_1)^{-1} \mathbf{X}_1^T\varepsilon}_{\nu}[/latex]so that [latex]\mathbb{E}[\widehat{\mathbf{b}}_1]=\beta_1+\beta_{12}[/latex], the bias ([latex] \beta_{12}[/latex]) being null only in the case where [latex]\mathbf{X}_1^T \mathbf{X}_2=\mathbf{0}[/latex] (i. e. [latex]\mathbf{X}_1\perp \mathbf{X}_2 [/latex]): we find here a consequence of the Frisch-Waugh theorem.On the other hand, over-identification corresponds to the case where the real model would be [latex]y_i=\beta_0+\mathbf{x}_1^T \beta_1+\varepsilon_i[/latex], but the estimated model is [latex]y_i=b_0+ \mathbf{x}_1^T \mathbf{b} _1+\mathbf{x}_2^T \mathbf{b}_2+\eta_i[/latex]. In this case, the estimate is unbiased, in the sense that [latex]\mathbb{E}[\widehat{\mathbf{b}}_1]=\beta_1[/latex] but the estimator is not efficient. Later on, we will discuss an effective method for selecting variables (and avoid over-identification).From parametric to non-parametricWe can rewrite equation (4) in the form [latex]\widehat{\mathbf{y}}=\Pi_X\mathbf{y}[/latex] which helps us to see the forecast directly as a linear transformation of the observations. More generally, a linear predictor can be obtained by considering [latex]m(\mathbf{x})=\mathbf{s}_{\mathbf{x}}^T \mathbf{y}[/latex], where [latex]\mathbf{s}_{\mathbf{x}}[/latex] is a weight vector, which depends on [latex]\mathbf{x}[/latex], interpreted as a smoothing vector. Using the vectors [latex]\mathbf{s}_{\mathbf{x}_i}[/latex], calculated from the observations [latex]\mathbf{x}_i[/latex], we obtain a matrix [latex]\mathbf{S}[/latex] of size [latex]n\times n[/latex], and [latex]\widehat{\mathbf{y}}=\mathbf{S}\mathbf{y}[/latex]. In the case of the linear regression described above, [latex]\mathbf{s}_{\mathbf{x}}=\mathbf{X}[\mathbf{X}^T\mathbf{X}]^{-1}\mathbf{x}[/latex], and in that case [latex]\text{trace}(\mathbf{S})[/latex] is the number of columns in the [latex]\mathbf{X}[/latex] matrix (the number of explanatory variables). In this context of more general linear predictors, [latex]\text{trace}(\mathbf{S})[/latex] is often seen as equivalent to the number of parameters (or complexity, or dimension, of the model), and [latex]\nu=n-\text{trace}(\mathbf{S})[/latex] is then the number of degrees of freedom (see Ruppert et al., 2003; Simonoff, 1996). The principle of parsimony says that we should minimize this dimension (the trace of the matrix [latex]\mathbf{S}[/latex]) as much as possible. But in the general case, this dimension is more to obtain, explicitely.The estimator introduced by Nadaraya (1964) and Watson (1964), in the case of a simple non-parametric regression, is also written in this form since[latex display="true"]\widehat{m}_h(x)=\mathbf{s}_{x}^T\mathbf{y}=\sum_{i=1}^n \mathbf{s}_{x,i}y_i[/latex]where[latex display="true"]\mathbf{s}_{x,i}=\frac{K_h(x-x_i)}{K_h(x-x_1)+\cdots+K_h(x-x_n)} [/latex] where [latex]K(\cdot) [/latex] is a kernel function, which assigns a value that is lower the closer [latex]x_i[/latex] is to [latex]x[/latex], and [latex]h&#62;0[/latex] is the bandwidth. The introduction of this metaparameter [latex]h[/latex] is an important issue, as it should be chosen wisely. Using asymptotic developments, we can show that if [latex]X[/latex] has density [latex]f[/latex], [latex display="true"]\text{biais}[\widehat{m}_h(x)]=\mathbb{E}[\widehat{m}_h(x)]-m(x)\sim {h^2}\left(\frac{C_1 }{2}m''(x)+C_2 m'(x)\frac{f'(x)}{f(x)}\right)[/latex]and [latex display="true"]\displaystyle{{\text{Var}[\widehat{m}_h(x)]\sim\frac{C_3}{{nh}}\frac{\sigma(x)}{f(x)}}}[/latex]for some constants that can be estimated (see Simonoff (1996) for a discussion). These two functions evolve inversely with [latex]h[/latex], as shown in Figure 1 (where the metaparameter on the [latex]x[/latex]-axis is here, actually, [latex]h^{-1}[/latex]). Keep in ming that we will see a similar graph in the context of machine learning models.Figure 1. Choice of meta-parameter and the Goldilocks problem: it must not be too large (otherwise there is too much variance), nor too small (otherwise there is too much bias).The natural idea is then to try to minimize the mean square error, the MSE, defined as [latex]bias[\widehat{m}_h (x)]^2+Var[\widehat{m}_h (x)][/latex], and them integrate over [latex]x[/latex], which gives an optimal value for [latex]h[/latex] of the form [latex]h^\star=O(n^{-1/5}) [/latex], and reminds us of Silverman's rule – see Silverman (1986). In larger dimensions, for continuous [latex]\mathbf{x}[/latex] variables, a multivariate kernel with matrix bandwidth [latex]\mathbf{H}[/latex] can be used, and[latex display="true"] \mathbb{E}[\widehat{m}_{\mathbf{H}}(\mathbf{x})]\sim m(\mathbf{x})+\frac{C_1}{2}\text{trace}\big(\mathbf{H}^Tm''(\mathbf{x})\mathbf{H}\big)+C_2\frac{m'(\boldsymbol{x})^T\mathbf{H}\mathbf{H}^T \nabla f(\mathbf{x})}{f(\mathbf{x})}[/latex]while[latex display="true"]\text{Var}[\widehat{m}_{\mathbf{H}}(\mathbf{x})]\sim\frac{C_3}{n~\text{det}(\mathbf{H})}\frac{\sigma(\mathbf{x})}{f(\mathbf{x})}[/latex]If [latex]\mathbf{H}[/latex] is a diagonal matrix, with the same term [latex]h[/latex]  on the diagonal, then [latex]h^\star=O(n^{-1/(4+dim(\mathbf{x}))}[/latex]. However, in practice, there will be more interest in the integrated version of the quadratic error, [latex display="true"] MISE(\widehat{m}_{h})=\mathbb{E}[MSE(\widehat{m}_{h}(X))]=\int MSE(\widehat{m}_{h}(x))dF(x)[/latex]and we can prove that [latex display="true"] MISE[\widehat{m}_h]\sim \overbrace{\frac{h^4}{4}\left(\int x^2k(x)dx\right)^2\int\big[m''(x)+2m'(x)\frac{f'(x)}{f(x)}\big]^2dx}^{\text{bias}^2} +\overbrace{\frac{\sigma^2}{nh}\int k^2(x)dx \cdot\int\frac{dx}{f(x)}}^{\text{variance}}[/latex]as n→∞ and nh→∞. Here we find an asymptotic relationship that again recalls Silverman's (1986) order of magnitude, [latex display="true"] h^\star =n^{-\frac{1}{5}}\left(\frac{C_1\int \frac{dx}{f(x)}}{C_2\int \big[m''(x)+2m'(x)\frac{f'(x)}{f(x)}\big]dx}\right)^{\frac{1}{5}}[/latex]The main problem here, in practice, is that many of the terms in the expression above are unknown. Automatic learning offers computational techniques, when the econometrician used to searching for asymptotic (mathematical) properties.To be continued (references mentioned above are online here)…