# 2.9 Model Selection and the Bias-Variance Tradeoff

All the models described have a *smoothing* or *complexity* parameter that has to be determined:

- the multiplier of the penalty term;

- the width of the kernel

- or the number of basis functions

We cannont use RSS on the training data to determine these parameters, since we would always pick those that gave interpolating fits and have zero residuals.

The kNN regression fit $\hat{f_k}(x_0)$ illustrates the competing forces that effect the predictive ability of such approximations. Suppose the data arise from a model $Y = f(X) + \varepsilon$ with $E(\varepsilon)=0$ and $Var(\varepsilon) = \sigma^2$. We assume that the values of $x_i$ in the sample are fixed. The EPE at $x_0$:

$$
\begin{align}
EPE_k(x_0) &= E[(Y - \hat{f_k}(x_0))^2|X=x_0]\\
&=\sigma^2 + [Bias^2(\hat{f_k}(x_0)) + Var_\tau(\hat{f_k}(x_0))]\\
&=\sigma^2 + \left[f(x_0) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})} \right]^2 + \frac{\sigma^2}{k}
\end{align}
$$

The subscripts in parentheses ($l$)  indicates the sequence of nearest neighbors to $x_0$. There are three terms in this expression:

1. $\sigma^2$ is the *irreducible* error - is beyond our control, even if we know the true $f(x_0)$.

2. The bias term and the expected value of the estimate - $[E_\tau(\hat{f_k}(x_0))-f(x_0)]^2$ - where the expected averages the randomness in the training data. This term increases with $k$ if the function is smooth.

3. The variance term and it decreases as the inverse of k. The expected value of the variance is: 

$$
\begin{align}
Var_\tau(\hat{f_k}(x_0)) &= E_\tau\left[\hat{f_k}(x_0) - E_\tau(\hat{f_k}(x_0))\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k (f(x_{(l)}) + \varepsilon_l) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})}\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l^2\right]\\
&= \frac{\sigma^2}{k}
\end{align}
$$

As the *model complexity* of our procedure is increased, the variance tends to increase and the squared bias tends to decrease. The opposite behavior occurs as the model complexity is decreased.

ee234c4cf8d92ea490c874357a3f6bffac416d94
ipynb
Jupyter Notebook
chapter-02/2.9-model-selection-and-the-bias-variance-tradeoff.ipynb
leduran/ESL
fcb6c8268d6a64962c013006d9298c6f5a7104fe
MIT
2019-01-28T14:05:02.000Z
2022-03-27T00:11:21.000Z
chapter-02/2.9-model-selection-and-the-bias-variance-tradeoff.ipynb
leduran/ESL
fcb6c8268d6a64962c013006d9298c6f5a7104fe
MIT
2020-07-06T16:51:40.000Z
2020-07-06T16:51:40.000Z
chapter-02/2.9-model-selection-and-the-bias-variance-tradeoff.ipynb
leduran/ESL
fcb6c8268d6a64962c013006d9298c6f5a7104fe
MIT
2019-03-21T23:48:35.000Z
2022-03-31T13:05:10.000Z
Qwen/Qwen-72B
1. YES
2. YES