**Exercise set 3**
==============


>The goal of this exercise is to investigate some theoretical properties of the solution to
>the (multiple) linear regression problem, and to perform a **least-squares regression**.
>We will also see how we can evaluate our regression model by investigating **residuals**.


**Exercise 3.1**

Multiple linear regression (MLR) solves the regression equation $\mathbf{Y} = \mathbf{X}\mathbf{B}$ with
$\mathbf{B} = (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y}$
where the
dimensions of the data matrix $\mathbf{X}$ is $[N \times M]$.
Will this solution work when (please explain why/why not):


**(a)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) = 0$?

**(b)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) > 0$?

**(c)** $\det \left( \mathbf{X}^\mathrm{T} \mathbf{X} \right) \neq 0$?

**(d)** The variables in $\mathbf{X}$ are correlated?

**(e)** The columns in $\mathbf{X}$ are orthogonal?

**(f)** The rank of $\mathbf{X}$ is $\frac{\min(N, M)}{2}$?

**(g)** We have more variables than samples/objects (more columns than rows in $\mathbf{X}$)?

**(h)** When we have more samples/objects than variables (more rows than columns in $\mathbf{X}$)?

**Your answer to question 3.1:** *Double click here*


**Exercise 3.2**

As stated in the previous problem, the MLR solution of the equation
$\mathbf{Y} = \mathbf{X}\mathbf{B}$ is

\begin{equation}
\mathbf{B} = (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y}.
\end{equation}

If we let $\hat{\mathbf{Y}}$ be the values calculated by the MLR solution, we can write
this as,
\begin{equation}
\hat{\mathbf{Y}} = \mathbf{X}\mathbf{B} =
\mathbf{X} \left[ (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T} \mathbf{Y} \right] = 
\left[\mathbf{X} (\mathbf{X}^\mathrm{T} \mathbf{X})^{-1} \mathbf{X}^\mathrm{T}\right] \mathbf{Y} =
\mathbf{H} \mathbf{Y},
\tag{1}\end{equation}

where we have defined the projection matrix, $\mathbf{H}$, as,
\begin{equation}
\mathbf{H} = \mathbf{X} \left(\mathbf{X}^\mathrm{T} \mathbf{X}\right)^{-1} \mathbf{X}^\mathrm{T}.
\label{eq:projectionmatrix}
\tag{2}\end{equation}

This means that we can write the residual, $\mathbf{E}$, as,
\begin{equation}
\mathbf{E} = \mathbf{Y} - \hat{\mathbf{Y}} = \mathbf{Y} - \mathbf{H} \mathbf{Y} =
(\mathbf{I} -\mathbf{H}) \mathbf{Y},
\tag{3}\end{equation}

where $\mathbf{I}$ is the identity matrix.

In this exercise, we will show two properties of $\mathbf{H}$ that enables us to simplify
the squared error, $\mathbf{E}^\mathrm{T} \mathbf{E}$, as follows,

\begin{equation}
\mathbf{E}^\mathrm{T} \mathbf{E} = \mathbf{Y}^\mathrm{T} (\mathbf{I} -\mathbf{H})^\mathrm{T}
(\mathbf{I} -\mathbf{H}) \mathbf{Y} = 
\mathbf{Y}^\mathrm{T} (\mathbf{I} -\mathbf{H}) \mathbf{Y}.
\tag{4}\end{equation}

In this equation, the last equality follows from the following two properties of $\mathbf{H}$:

**(a)**  $\mathbf{H}$ is *symmetric*: $\mathbf{H}^\mathrm{T} = \mathbf{H}$.

**(b)**  $\mathbf{H}$ is *idempotent*: $\mathbf{H}^{k} = \mathbf{H}$ where $k > 0$ is an integer.

Show these two properties for $\mathbf{H}$. (Hint: For the idempotency, begin by showing that $\mathbf{H}^{2} = \mathbf{H}$.)

**Your answer to question 3.2:** *Double click here*

**Exercise 3.3**

In the regression problem $\mathbf{y} = \mathbf{X}\mathbf{b}$
we find the least-squares solution assuming that $\mathbf{X}^\mathrm{T} \mathbf{X}$ is
non-singular. If you are given the information that $\mathbf{X}$ is symmetric
and non-singular, is there another
simpler formula for estimating the regression coefficients ($\mathbf{b}$)?



**Your answer to question 3.3:** *Double click here*

**Exercise 3.4**

Assume that we have recorded data as shown in Fig. 1.


**Fig. 1:** Example data.

To model this data, we suggest a third-order polynomial in $x$:
\begin{equation}
\hat{y} = b_0 + b_1 x + b_2 x^2 + b_3 x^3 .
\end{equation}

Explain how you can formulate this on a form suitable for least-squares regression,
$\mathbf{y} = \mathbf{X} \mathbf{b}$.
What do the vectors $\mathbf{y}$ and $\mathbf{b}$ contain? What does the matrix $\mathbf{X}$ contain?



**Your answer to question 3.4:** *Double click here*

**Exercise 3.5**

The temperature (°C) is measured continuously over time at a high altitude
in the atmosphere using a
weather balloon. Every hour a measurement is made and sent to an on-board computer.
The measurements are 
shown in Fig. 2 and can be found in [the data file](Data/temperature.txt) (located at 'Data/temperature.txt').


**Fig. 2:** Measured temperature as a function of time.

**(a)**  Create a Python script that performs polynomial
fitting to the data using a first, second, third, fourth,
and fifth order polynomial model. Hint: Make use of `numpy`, `matplotlib`
and `pandas`.


```python
# Your code here
```

**(b)**  Plot the fitted curves for the five models to the raw data.




```python
# Your code here
```

**(c)** Plot the residual curves for the five models and determine,
from a visual inspection, the best polynomial order to use for modeling the
temperature as a function of time. 




```python
# Your code here
```

**Your answer to question 3.5(c):** *Double click here*

**(d)**  Obtain the sum of squared residuals for each polynomial. Plot this as a function
of the degree of the polynomial and determine from visual inspection
the best polynomial order to use for modeling the
temperature as a function of time. Does this agree with your conclusion in point **3.5(c)**?



```python
# Your code here
```

**Your answer to question 3.5(d):** *Double click here*




d402b55165b9946c0fb9667cb3e1b2aa13376229
ipynb
Jupyter Notebook
exercises/03_Exercise_Set_3.ipynb
sroet/chemometrics
c797505d07e366319ba1544e8a602be94b88fbb6
CC-BY-4.0
2020-02-04T12:09:19.000Z
2021-08-06T12:28:24.000Z
exercises/03_Exercise_Set_3.ipynb
sroet/chemometrics
c797505d07e366319ba1544e8a602be94b88fbb6
CC-BY-4.0
2020-01-06T10:24:33.000Z
2022-01-21T10:37:46.000Z
exercises/03_Exercise_Set_3.ipynb
sroet/chemometrics
c797505d07e366319ba1544e8a602be94b88fbb6
CC-BY-4.0
2020-01-09T12:04:26.000Z
2021-01-19T10:06:14.000Z
Qwen/Qwen-72B
1. YES
2. YES