```python
import matplotlib.pylab as plt 
import numpy as np

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
```

# 1. Loss functions

Lets focus on linear regression of the form 

$\mathbf{y} \approx f(\mathbf{X}) = \mathbf{X}\mathbf{w_1} + \mathbf{w_0}.$


#### 1.1 What are the rows of $\mathbf{X}$?

__< Your answer >__

#### 1.2 What are the columns of $\mathbf{X}$?

__< Your answer >__

Often, we write the equation above as

$\mathbf{y} \approx \mathbf{\tilde{X}}\mathbf{w}$

#### 1.3 How does $\mathbf{\tilde{X}}$ look like in this case (i.e., how does the shape of the matrix change compared to $\mathbf{X}$)? (1 point)

__< Your answer >__

For machine learning, we need a cost function. Two common choices are the mean-squared error (MSE, $\mathcal{L}_2$), and the mean-absolute error (MAE, $\mathcal{L}_1$)

\begin{align}
    \mathcal{L}_2 &=& \frac{1}{2N} \sum_{i=1}^N \left(y_i - f(x_i) \right)^2 \\
    \mathcal{L}_1 &=& \frac{1}{2N} \sum_{i=1}^N \left|y_i - f(x_i) \right| 
\end{align}

#### 1.5 In the Jupyter notebook, write a Python function that computes these two cost functions given an error term $\boldsymbol{\epsilon} = \mathbf{y} - \mathbf{\tilde{X}}\mathbf{w}$ (2 points)


```python
def mean_squared_error(error_vector):
    mean_squared_error = # fill your code
    return mean_squared_error
```


```python
def mean_absolute_error(error_vector):
    mean_absolute_error = # fill your code
    return mean_absolute_error
```

Your code should run as follows

```python
mean_squared_error(np.array([0,0,0]))
> returns 0
```

```python
mean_squared_error(np.array([1,1,1]))
> returns 1
```

#### 1.6 What is the shape of these cost functions as a function of the error  (1 point)


```python
x_axis = np.linspace(-10,10,100) # change as you wish for your plot
y_mae = [mean_absolute_error(x) for x in x_axis]
y_mse = [mean_squared_error(x) for x in x_axis]
```


```python
plt.plot(x_axis, y_mse, label='MSE')
plt.plot(x_axis, y_mae, label='MAE')
plt.xlabel('error term')
plt.ylabel('cost function')
```

#### 1.7  Are both loss functions differentiable for all $\boldsymbol{\epsilon}$? (1 point) What implications does this have for gradient based optimization like gradient descent? (1 point)

__< Your answer >__

#### 1.8 Which loss function is more sensitive to outliers (1 point) and why (1 point)?

__< Your answer >__

# 2. Regularization

Assume that the columns of $\mathbf{X}$ are linearly independent.
As a refresher of linear algebra, recall when the linear system $\mathbf{X}\mathbf{w} = \mathbf{y}$ has

#### 2.1 One unique solution (1 point)

__< Your answer >__

#### 2.2 No solution (1 point)

__< Your answer >__

#### 2.3 An infinite number of solutions (1 point)

__< Your answer >__

To calculate the weights, $\mathbf{w}$ we have to solve
\begin{equation}
    \mathbf{w} = \left(\mathbf{\tilde{X}}\mathbf{\tilde{X}}^T\right)^{-1}\mathbf{\tilde{X}}^T\mathbf{y}
\end{equation}

#### 2.4 Will $\left(\mathbf{\tilde{X}}\mathbf{\tilde{X}}^T\right)$ be invertible if the number of features is greater than the number of data points? (1 point)

__< Your answer >__

#### 2.5  What happens if some columns are linearly dependent? (1 point) What is the connection to feature selection? (1 point)

__< Your answer >__

#### 2.6  What is the shape of the parabola as a function of $a$?


```python
def parabola(x, a = 1): 
    return a * x ** 2
```


```python
x_axis_parabola = np.linspace(-10, 10, 100)
```


```python
plt.plot(x_axis_parabola, parabola(x_axis_parabola))
```

__< Your answer >__

#### 2.7 Plot the approximation to the function for different order polynomials ($N \in \{1, 2, 16\}$) and with different regularization strength ($\lambda \in \{0, 10^{-3}, 10^{-2}, 1\}$). What do you observe 


```python
def true_function(X):
    return np.cos(1.5 * np.pi * X)
```


```python
X_test = np.linspace(0, 1, 100) # some grid for us on the x axis
```


```python
n_samples = 10 # the number of points we will sample from true_function
degrees = [1, 2, 16] # the polynomial degrees we will test

X = np.sort(np.random.rand(n_samples))
y = true_function(X) + np.random.randn(n_samples) * 0.1 # add some scaled random noise
```


```python
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

The following code will fit a polynomial regression, you need to fill the degree


```python
polynomial_features = PolynomialFeatures(degree=#FILLEME,
                                             include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([("polynomial_features", polynomial_features),
                     ("linear_regression", linear_regression)])
pipeline.fit(X[:, np.newaxis], y)
```

To plot the result, you can use the following code


```python
plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="model of degree ")
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

Next, we can investigate the effect of the regularization parameter $\lambda$ (function parameter `alpha`), For this, you can use the following code 


```python
polynomial_features = PolynomialFeatures(degree=#fillme,
                                             include_bias=False)
ridge_regression = Ridge(alpha=#fillme)
pipeline_ridge = Pipeline([("polynomial_features", polynomial_features),
                     ("ridge_regression", ridge_regression)])
pipeline_ridge.fit(X[:, np.newaxis], y)

```

For plotting you can reuse the following code


```python
plt.plot(X_test, pipeline_ridge.predict(X_test[:, np.newaxis]), label="model of degree with regularization strength")
plt.scatter(X, y, label='noisy samples')
plt.plot(X_test, true_function(X_test), c='r', label='true function')
plt.legend()
```

#### 2.8 What do you observe if you change the number of samples from the function?

__< Your answer >__

#### 2.9 Why do we need a test set in machine learning? (1 point)

__< Your answer >__

#### 2.10 If we need to optimize hyperparameters, do we use the test set to select the best hyperparameters? (1 point)

__< Your answer >__

cd4871a0fd3450ce52f6fe0c4e0f066574c15c23
ipynb
Jupyter Notebook
ml/ml_theory_assignment.ipynb
lsmo-epfl/ch-315
7cf3bb9c4b422932651ef37eb837a80d9b6a0c26
MIT
ml/ml_theory_assignment.ipynb
lsmo-epfl/ch-315
7cf3bb9c4b422932651ef37eb837a80d9b6a0c26
MIT
2021-12-13T11:32:37.000Z
2021-12-13T11:32:37.000Z
ml/ml_theory_assignment.ipynb
lsmo-epfl/ch-315
7cf3bb9c4b422932651ef37eb837a80d9b6a0c26
MIT
Qwen/Qwen-72B
1. YES
2. YES