## Chapter 4 -  Training Models

### Regularised Linear Models

Recall that the cost function for:

<b>Least Squares</b>

$$\begin{align}J(\Theta) = \text{RSS} &= \sum_{i=1}^n \begin{pmatrix} y_i -  \hat{y_i} \end{pmatrix}^2
\\&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\end{align}$$

<b>Ridge Regression ($l_2$ normalisation)</b>

$$\begin{align}J(\Theta) = \text{RSS} + \lambda \sum_{j=1}^p\beta^2_j &=\sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p \beta^2_j\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p\beta^2_j\end{align}$$

<b>Lasso Regression ($l_1$ normalisation)</b>

$$\begin{align}J(\Theta) = \text{RSS} + \lambda \sum_{j=1}^p|\beta_j| &=\sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p |\beta_j|\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p|\beta_j|\end{align}$$

and the objective is to solve for $\beta_0, \cdots, \beta_p$ that minimises the cost function.

Another formulation is:

<b>Ridge Regression</b>


$$\underset{\beta_0, \cdots, \beta_p}{\text{Minimise}}\left\{\sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\right\}\text{ s. t. } \sum_{j=1}^p\beta^2_j \leq s$$

<b>Lasso Regression</b>
$$\underset{\beta_0, \cdots, \beta_p}{\text{Minimise}}\left\{\sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\right\}\text{ s. t. } \sum_{j=1}^p |\beta_j| \leq s$$

In other words, for every value of $\lambda$, there is some $s$ that gives the same coefficients for both sets of equations (ridge vs ridge, lasso vs lasso).

<b>Graphical Representation</b>
Consider $p=2$. The ridge minimisation constraint is now a circle $\beta_1^2 + \beta_2^2 \leq s$. Similarly, the Lasso minimisation constraint is now a diamond $|\beta_1| + |\beta_2| \leq s$. The coefficient estimates must be a point in these constraint regions (diamond / circle). 

That means the coefficient estimates now need to satisfy the constraint bounded by $s$. If $s$ is very large, there is no restriction and the coefficient estimates could be the same as the solution without regularisation. If $s$ is small, then the coefficient estimates will be optimised within the constraint.



Graphically, the red contours represent the cost function, $J(\Theta)$. The contour lines all represent the same value of $J(\Theta)$. Recall that it is convex. The black coordinate $\hat{\beta_0}$ represents the least squares estimate of $\beta_1, \beta_2$. The diamond and circle represent that constraints $|\beta_1| + |\beta_2| \leq s$ and $\beta_1^2 + \beta_2^2 \leq s$ respectively. The point (on the edge of the region that is closest to the least squares estimate) is hence the coefficient estimates of the regularised model. 

From here, observe that a large $s$ will eventually result in the least squares estimate to be in the region. That corresponds to a small $\lambda$.

Also observe that the lasso constrains have corners on the axes. When the minimum value of $J(\Theta)$ lies on the corners, then the coefficient estimate is $0$. In this case $\hat{\beta_1}=0$ and the regularised model only contains $\hat{\beta_2}$.

When $p=3$ then the ridge regression constraint is a sphere and the lasso regression constraint is a polyhedron. These key ideas still hold as $p$ increases beyond geometric formulations.

5f3cac29535949132441da32198aab8269734be2
ipynb
Jupyter Notebook
chap04/04-textbook-training-models-03b.ipynb
bryanblackbee/topic__hands-on-machine-learning
3b9a2cfa011099178dd73c3366331958d49ad96f
MIT
chap04/04-textbook-training-models-03b.ipynb
bryanblackbee/topic__hands-on-machine-learning
3b9a2cfa011099178dd73c3366331958d49ad96f
MIT
chap04/04-textbook-training-models-03b.ipynb
bryanblackbee/topic__hands-on-machine-learning
3b9a2cfa011099178dd73c3366331958d49ad96f
MIT
2020-06-20T05:38:43.000Z
2020-06-20T05:38:43.000Z
Qwen/Qwen-72B
1. YES
2. YES