# Linear Regression from scratch
> Linear regression from scratch using Pytorch on Swedish Auto Insurance dataset.
- author: "<a href='https://www.linkedin.com/in/axel-mendoza-298608121/'>Axel Mendoza</a>"
- categories: [linear-regression, insurance-data, pytorch, from-scratch]
- toc: false
- comments: true
- badges: true
- image: images/lines.jpg



This blog post is about implementing linear regression from scratch using Pytorch.
We will train the model on insurance data to predict the total payment received by a customer in regards to his number of claims.

For more details, the swedish auto insurance dataset contains the following attributes:
- X = number of claims
- Y = total payment for all the claims in thousands of Swedish Kronor for geographical zones in Sweden


```python
import os
import sys
import torch
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
```

### Reading Excel file with Pandas


```python
data_path = os.path.join('data', 'slr06.xls')
df = pd.read_excel(data_path, encoding_override="cp1252")
df.head()
```

    *** No CODEPAGE record, no encoding_override: will use 'ascii'





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>X</th>
      <th>Y</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>108</td>
      <td>392.5</td>
    </tr>
    <tr>
      <th>1</th>
      <td>19</td>
      <td>46.2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>13</td>
      <td>15.7</td>
    </tr>
    <tr>
      <th>3</th>
      <td>124</td>
      <td>422.2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>40</td>
      <td>119.4</td>
    </tr>
  </tbody>
</table>
</div>



### Data Visualization Using Seaborn


```python
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
ax1.set_title('Distribution of number of claims')
ax2.set_title('Distribution of total payment')

sns.distplot(df.X, bins=50, hist=True, ax=ax1)
sns.distplot(df.Y, ax=ax2)
```

Two outliers are present between 100 and 125 number of claims.
We will keep them in the training set.


```python
sns.heatmap(df.corr(), annot=True)
```

The correlation heatmap and the plot below highlight that the number of claims and total payment are very highly correlated.


```python
plt.xlabel('Number of claims')
plt.ylabel('Total payment')
plt.scatter(df['X'], df['Y'])
```

### Training a Least Square Regressor

Convert the data to pytorch and separate the data into train & test.


```python
data = df.to_numpy()
X = torch.from_numpy(data[:, 0]).float().unsqueeze(1)
y = torch.from_numpy(data[:, 1]).float().unsqueeze(1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=50)
```

We will train a least square regressor on one dimension using the above formula:
$$
    \begin{align}
    y &= b_0 + b_1x \\
    b_1 &= \frac{\text{Cov}(x, y)}{Var(x)} = \frac{\sum_i{(x_i - \mathbb{E}[x]) * (y_i - \mathbb{E}[y])}}{\sum_i{(x_i - \mathbb{E}[x])^2}}
    \end{align}
$$
where the expectation $\mathbb{E}[x]$ of a vector of random variables is its mean.


```python
def linear_regression_1d(X, y):
    """Trains a linear regression on 1D data
    
    Args:
        X: A numpy array for the training samples
        y: A numpy array for the labels of each sample
    """
    X_m = X.mean(dim=0)
    y_m = y.mean(dim=0)
    
    X_c = (X - X_m)
    
    # Compute covariance and variance
    covar = (X_c * (y - y_m)).sum(dim=0)
    var = X_c.pow(2).sum(dim=0)

    # Divide covariance by variance
    b_1 = covar / var
    
    # Get bias
    b_0 = y_m - b_1 * X_m.sum(dim=0)
    return b_0, b_1
```

We are using Pytorch for the matrix calculus.

### Plot Regression Line 


```python
b_0, b_1 = linear_regression_1d(X_train, Y_train)
plt.scatter(X_train, Y_train, marker='*')
plt.scatter(X_test, Y_test, marker='.', color='red')

x = [int(elt) for elt in range(0, int(X.cpu().numpy().max()))]
y = b_0.numpy() + b_1.numpy() * x
plt.plot(x, y, color='red')

ax = plt.gca()
ax.set_xlabel('Number of claims')
ax.set_ylabel('Total payment')
plt.show()
```

Because both the data and target are very highly correlated, the regression line fits pretty well the data distribution.

### Compute Mean Square Error


```python
pred_train = b_0 + b_1 * X_train
err_train = Y_train - pred_train
mse_train = err_train.T.mm(err_train) / Y_train.shape[0]

pred_test = b_0 + b_1 * X_test
err_test = Y_test - pred_test
mse_test = err_test.T.mm(err_test) / Y_test.shape[0]

print('Train MSE:\t', mse_train.item())
print('Test MSE:\t', mse_test.item())
```

    Train MSE:	 1241.5714111328125
    Test MSE:	 1279.6766357421875


The train and test MSE are close to each other.
The model neither underfit nor overfit.

### Conclusion
Linear regression is a simple and interpretable model that fits very well the purpose of a baseline.
As we have seen during this experiment, if the features are highly correlated to the target, linear regression performs pretty well.

a92d8bf4464bc8c19947a1e98e4a00416caeb0ec
ipynb
Jupyter Notebook
_notebooks/2020-09-07-Linear-Regression.ipynb
ConsciousML/blog
e3f8fb858ebbf840313f192e33cc41a291e04f7b
Apache-2.0
_notebooks/2020-09-07-Linear-Regression.ipynb
ConsciousML/blog
e3f8fb858ebbf840313f192e33cc41a291e04f7b
Apache-2.0
2021-09-28T05:35:20.000Z
2021-09-28T05:35:20.000Z
_notebooks/2020-09-07-Linear-Regression.ipynb
ConsciousML/blog
e3f8fb858ebbf840313f192e33cc41a291e04f7b
Apache-2.0
2021-02-18T16:04:25.000Z
2021-02-18T16:04:25.000Z
Qwen/Qwen-72B
1. YES
2. YES