# Linear regression
- It's a way to model relationship between 2 models
- Also known as the slope formula
- equation form: \begin{align}Y = a + bX\end{align}
  - where X is the independent variable and Y is dependent
  - b = slope
  - a = y-intercept

#### Equations
\begin{align}
a = \dfrac{(\sum y)(\sum x^2)-(\sum x)(\sum xy)}{n(\sum x^2) - (\sum x)^2}
\end{align}

\begin{align}
b = \dfrac{n(\sum xy)-(\sum x)(\sum y)}{n(\sum x^2) - (\sum x)^2}
\end{align}



```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
```


```python
import os
path = os.getcwd() + '\data\ex1data1.txt'
data = pd.read_csv(path, header=None, names=['Population', 'Profit'])
data.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Population</th>
      <th>Profit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>6.1101</td>
      <td>17.5920</td>
    </tr>
    <tr>
      <th>1</th>
      <td>5.5277</td>
      <td>9.1302</td>
    </tr>
    <tr>
      <th>2</th>
      <td>8.5186</td>
      <td>13.6620</td>
    </tr>
    <tr>
      <th>3</th>
      <td>7.0032</td>
      <td>11.8540</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5.8598</td>
      <td>6.8233</td>
    </tr>
  </tbody>
</table>
</div>




```python
data.describe()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Population</th>
      <th>Profit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>97.000000</td>
      <td>97.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>8.159800</td>
      <td>5.839135</td>
    </tr>
    <tr>
      <th>std</th>
      <td>3.869884</td>
      <td>5.510262</td>
    </tr>
    <tr>
      <th>min</th>
      <td>5.026900</td>
      <td>-2.680700</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>5.707700</td>
      <td>1.986900</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>6.589400</td>
      <td>4.562300</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>8.578100</td>
      <td>7.046700</td>
    </tr>
    <tr>
      <th>max</th>
      <td>22.203000</td>
      <td>24.147000</td>
    </tr>
  </tbody>
</table>
</div>



> Let's plot it together to get a better idea of what the data looks like


```python
plt.scatter(x=data['Population'], y=data['Profit'])
plt.show()
```

> Implementing linear regression using gradient descent to minimize cost function

> Creating a function to compute the cost of a given solution


```python
def compute_cost(X, y, theta):
    inner = np.power(((X * theta.T) - y), 2)
    return np.sum(inner) / (2 * len(X))
```

Inserting a column of ones to use a vectorized solution to computing cost and gradients.


```python
data.insert(0, 'Ones', 1)
```

Checking the new table


```python
data.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Ones</th>
      <th>Population</th>
      <th>Profit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>6.1101</td>
      <td>17.5920</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>5.5277</td>
      <td>9.1302</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>8.5186</td>
      <td>13.6620</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>7.0032</td>
      <td>11.8540</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>5.8598</td>
      <td>6.8233</td>
    </tr>
  </tbody>
</table>
</div>



Variable initialization


```python
# set X (training data) and y (target variable)
cols = data.shape[1]
X = data.iloc[:, 0:cols-1]
y = data.iloc[:, cols-1:cols]
```

Checking values


```python
X.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Ones</th>
      <th>Population</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>6.1101</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>5.5277</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>8.5186</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>7.0032</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>5.8598</td>
    </tr>
  </tbody>
</table>
</div>




```python
y.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Profit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>17.5920</td>
    </tr>
    <tr>
      <th>1</th>
      <td>9.1302</td>
    </tr>
    <tr>
      <th>2</th>
      <td>13.6620</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11.8540</td>
    </tr>
    <tr>
      <th>4</th>
      <td>6.8233</td>
    </tr>
  </tbody>
</table>
</div>



Convert to numpy matrices


```python
X = np.matrix(X.values)
y = np.matrix(y.values)
theta = np.matrix(np.array([0, 0]))
```


```python
theta
```




    matrix([[0, 0]])



Shape of matrices


```python
X.shape, y.shape, theta.shape
```




    ((97, 2), (97, 1), (1, 2))



Compute cost for initial solution (0 values for theta)


```python
compute_cost(X, y, theta)
```




    32.072733877455676



---
Defining a function to perform gradient descent on paramaters theta using the update rules defined in the text


```python
def gradient_descent(X, y, theta, alpha, iters):
    temp = np.matrix(np.zeros(theta.shape))
    parameters = int(theta.ravel().shape[1])
    cost = np.zeros(iters)
    
    for i in range(iters):
        error = (X * theta.T) - y
        
        for j in range(parameters):
            term = np.multiply(error, X[:, j])
            temp[0, j] = theta[0, j] - ((alpha / len(X)) * np.sum(term))
            
        theta = temp
        cost[i] = compute_cost(X, y, theta)
        
    return theta, cost
```

Init some additional variables: learning rate $\alpha$ (alpha), and number of iterations to perform


```python
alpha = 0.01
iters = 1000
```

Running the gradient descent algo to ift our parameters theta to the training set


```python
g, cost = gradient_descent(X, y, theta, alpha, iters)
g
```




    matrix([[-3.24140214,  1.1272942 ]])



Now we can compute our cost (error) of trained model using our fitted parameters.


```python
compute_cost(X, y, g)
```




    4.5159555030789118



Let's plot the linear model along with the data to see how it fits


```python
x = np.linspace(data.Population.min(), data.Population.max(), 100)
f = g[0, 0] + (g[0, 1] * x)

fig, ax = plt.subplots(figsize=(12, 8))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')
fig
```

Observations:
- The cost always decreases. This is an example of convex optimization problem
- function outputs a vector with the cost at each iteration, so we can plot that as well


```python
fig, ax = plt.subplots(figsize=(12,8))
ax.plot(np.arange(iters), cost, 'r')
ax.set_xlabel('Iterations')
ax.set_xlim([0, iters])
ax.set_ylabel('Cost')
ax.set_title('Error vs. Training Epoch')
fig
```

---
# Mulitple variables
---

Let's try 2 variables. 

Example: a housing price data set with 2 variables 
- size of house, no. of bedrooms and 
- target price of house.


```python
path = os.getcwd() + '\data\ex1data2.txt'
data2 = pd.read_csv(path, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Size</th>
      <th>Bedrooms</th>
      <th>Price</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2104</td>
      <td>3</td>
      <td>399900</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1600</td>
      <td>3</td>
      <td>329900</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2400</td>
      <td>3</td>
      <td>369000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1416</td>
      <td>2</td>
      <td>232000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>3000</td>
      <td>4</td>
      <td>539900</td>
    </tr>
  </tbody>
</table>
</div>



> add another step: normalizing the features


```python
data2 = (data2 - data2.mean()) / data2.std()
data2.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Size</th>
      <th>Bedrooms</th>
      <th>Price</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.130010</td>
      <td>-0.223675</td>
      <td>0.475747</td>
    </tr>
    <tr>
      <th>1</th>
      <td>-0.504190</td>
      <td>-0.223675</td>
      <td>-0.084074</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.502476</td>
      <td>-0.223675</td>
      <td>0.228626</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.735723</td>
      <td>-1.537767</td>
      <td>-0.867025</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1.257476</td>
      <td>1.090417</td>
      <td>1.595389</td>
    </tr>
  </tbody>
</table>
</div>



Repeating steps from previous part


```python
# Add ones column
data2.insert(0, 'Ones', 1)
```


```python
# set X(training data) and y(target)
cols = data2.shape[1]
X2 = data2.iloc[:, 0:cols-1]
y2 = data2.iloc[:, cols-1:cols]
```


```python
# convert to np array
X2 = np.matrix(X2.values)
y2 = np.matrix(y2.values)
theta2 = np.matrix(np.array([0,0,0]))
```


```python
# perform linear regression
g2, cost2 = gradient_descent(X2, y2, theta2, alpha, iters)

# compute cost
compute_cost(X2, y2, g2)
```




    0.13070336960771892




```python
fig, ax = plt.subplots(figsize=(12,8))
ax.plot(np.arange(iters), cost2, 'r')
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.set_title('Error vs. Training Epoch')
fig
```

---
# Using scikit learn's linear regression function
---


```python
from sklearn import linear_model
```


```python
model = linear_model.LinearRegression()
model.fit(X, y)
```




    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)




```python
x = np.array(X[:, 1].A1)
f = model.predict(X).flatten()

fig, ax = plt.subplots(figsize=(12,8))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')
fig
```

f41b1ce13adf64bbefa6764fd25187526413b278
ipynb
Jupyter Notebook
ML/classification/Linear Regression Python.ipynb
manparvesh/ml-dl
d1eae6b4d8346582303718601394557b5be744c0
MIT
ML/classification/Linear Regression Python.ipynb
manparvesh/ml-dl
d1eae6b4d8346582303718601394557b5be744c0
MIT
ML/classification/Linear Regression Python.ipynb
manparvesh/ml-dl
d1eae6b4d8346582303718601394557b5be744c0
MIT
Qwen/Qwen-72B
1. YES
2. YES