### Logistic Regression From Scratch Using Iris Data Set
Logistic Regression is a classification algorithm that aims at predicting the probability that a certain observation $X_i$ belongs to the class $y=1$.

Here we will first give the formulas that we will implement LR with using Gradient descent algorithm (Following the Likelihood approach).

Generally,we want to find a function that gives the probability of a linear combination of the input variable/features $X$ i.e. $$p=\theta_0+\theta_1x_1+ .... +\theta_nx_n$$. We know that a probability a probaility $0\leq p\leq 1$, from the function $p$ it is evident that the probility might be great than $1$ or even less than $0$.

We can therefore, instead of assignig $p$ to the function we assign the odds i.e $\frac{p}{1-p}$. Such that ,
 $$\frac{p}{1-p}=\theta_0+\theta_1x_1+ .... +\theta_nx_n$$
 
 We also know that $\theta_0+\theta_1x_1+ .... +\theta_nx_n$ might sometimes be negative, to ensure that then negatives are taken care of we apply $\log$ on $\frac{p}{1-p}$.This leads to,
 $$\log\bigg(\frac{p}{1-p}\bigg)=
 \theta_0+\theta_1x_1+ .... +\theta_nx_n=\theta^TX \quad \text{(In vector notation)}$$
 
 Equation above ensures that the output is within the range of $0$ and $1$. To simplify things further and solve for $p$, we can exponentiate both sides of the equation above to obtain,
 \begin{align}
 &\frac{p}{1-p}=e^{\theta^TX}\\
 &p=e^{\theta^TX}-pe^{\theta^TX}\\
 &p+pe^{\theta^TX}=e^{\theta^TX}\\
 &p\bigg(1+e^{\theta^TX}\bigg)=e^{\theta^TX}\\
 &p=\frac{e^{\theta^TX}}{1+e^{\theta^TX}}\\
 &p=\frac{1}{1+e^{-\theta^TX}}
 \end{align}
The last equation is the famous sigmoid/logistic function.

Since we have already found the probility $p$, which gives the probability that a certain observation $X_i$ belongs to a certain class, in this case $1$, then the probaility that that the obsercation $X_i$ belongs to the class $0$ is $1-p$. This is because we have only two classes $\{1,0\}$. This is represented as:
$$P(y=1|X;\theta)=p\\
P(y=0|X;\theta)=1-p$$
For such an obsertion $X_i=x$, the probability above can be written in a more compact form as,$$P(y|x)=p^{y}(1-p)^{1-y}$$

And since we have many observations in $X$, the to obtain the probality for all the observations, we multiply the individual probability with each other i.e.
$$P(y|x)=p^{y_1}(1-p)^{1-y_1}\times p^{y_2}(1-p)^{1-y_2}\times...\times p^{y_n}(1-p)^{1-y_n
}$$
For $n$ observations. The product above is called the Likelihood (which needs to be minimized/optimized)of a certain outcome given inputs. It written as,
$$L(y;x)=\prod_{i=1}^{n}p^{y_i}(1-p)^{1-y_i}$$

But we know that GradienT Descent (GD) uses derivatives, therefore it is wise not find derivatives of products (Not easy). We therefore apply logarithm on the likelihood to obtain the log-likelihood (l)
$$l=\log(L(y;x))=\log\bigg(\prod_{i=1}^{n}p^{y_i}(1-p)^{1-y_i}\bigg)=\frac{1}{n}\sum_{i=1}^{n}y_i\log(p)+(1-y_i)\log(1-p)$$

Recall that $$p=\frac{1}{1+e^{-\theta^TX}}$$
We can therefore find the partial derivatives of $\theta_j$, since we want to to find the parameters $\theta_j$ that optimizes the log-likelihood function. 

After applying chain rule on the log-likelihood function,we obtain
$$\frac{\partial l}{\partial \theta_j}=\frac{1}{n}\sum_{i=1}^{n}X_i(p-y_i)$$

To find the optimum $\theta_j$ we use GD, which iteratively updates the values of \theta_j untill convergence using:
$$\theta_j \text(new)=\theta_j \text(old)-\alpha \frac{\partial l}{\partial \theta_j}$$
where alpha is learning rate choosen experimentaly.

Once the optimum values of $ \theta_j$ is found can use $p=\frac{1}{1+e^{-\theta^TX}}$ to classify unseen data $X$.

Let us now implement Binary Logistic Regression from scratch and also using inbuilt package. We will use data from kaggle Iris data but with two categories instead of three. Kindly download the data here. If you want to use the same data.


```python
## Import the necessary packages
import numpy as np # performing array calculations
import pandas as pd # Load and manipulating the dataset
import seaborn as sns # visualization
import matplotlib.pyplot as plt # visualization
```


```python
#LOad the data which is local in my machine
data=pd.read_csv('Iris.csv')
#check the first 5 rows
data.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Id</th>
      <th>SepalLengthCm</th>
      <th>SepalWidthCm</th>
      <th>PetalLengthCm</th>
      <th>PetalWidthCm</th>
      <th>Species</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>5.1</td>
      <td>3.5</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>4.9</td>
      <td>3.0</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>4.7</td>
      <td>3.2</td>
      <td>1.3</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>4.6</td>
      <td>3.1</td>
      <td>1.5</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>5.0</td>
      <td>3.6</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
  </tbody>
</table>
</div>




```python
#Check the shape of our dataset
data.shape
```




    (150, 6)



#### The data has four features $X=\{SepalLengthC,SepalWidthCm,PetalLengthCm PetalWidthCm\}$  and one Target variable $y=Species$,   with $n=150$. 


```python
#The species has how namy unique categories
data['Species'].unique()
```




    array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'], dtype=object)



#### As seen above,There are three categories ('Iris-setosa', 'Iris-versicolor', 'Iris-virginica'). We remove one so that its binary classification.


```python
#we remove one species using pandas
data1=data[data['Species']!='Iris-virginica']
print(data1.Species.unique() )#Two classes are left.
print(data1.shape)
```

    ['Iris-setosa' 'Iris-versicolor']
    (100, 6)


#### Now $n=100$


```python
#Next we drop the id column and check whether there missing values in the dataset
data2=data1.drop('Id', axis=1)
data2.info() 
#No null /or missing values
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 100 entries, 0 to 99
    Data columns (total 5 columns):
     #   Column         Non-Null Count  Dtype  
    ---  ------         --------------  -----  
     0   SepalLengthCm  100 non-null    float64
     1   SepalWidthCm   100 non-null    float64
     2   PetalLengthCm  100 non-null    float64
     3   PetalWidthCm   100 non-null    float64
     4   Species        100 non-null    object 
    dtypes: float64(4), object(1)
    memory usage: 4.7+ KB


#### The next step we convert the species column to 0 and 1  since it is of onject datatype using the Label Encoder from scikit learn


```python
#encode the Species/Target Vaiable
from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
data2['Species']=le.fit_transform(data2['Species'])
data2.Species.unique()
```




    array([0, 1])



#### Here we see that Iris-setosa was encoded as zero and Iris-versicolor as 1.

#### The data appears to be recorded in a certain order we shuffle to mix the data well.  Uisng the sample method.


```python
data2=data2.sample(frac=1)
```


```python
# we can now assing X and y their respective values.
X=data2.drop('Species',axis=1).values
y=data2.Species.values
```

#### Since the dataset isn't that large we shall only train and check the score on the training data. Remember that it is of paramount to have a testing set in order to have a better model. In this tutorial, we just want to show how Logistic regression works with GD


```python
#hstack a column onf ones in X
ones=np.ones([100,1])
X=np.hstack([ones,X])
```


```python
#initialize the weights
weights=np.zeros(5)
weights
```




    array([0., 0., 0., 0., 0.])




```python
epochs=1000
cost=[]
def sigmoid(p):
    return 1/(1+np.exp(-p))
def gradient(X,y,p):
    return 1/X.shape[0]*(np.dot(X.T,p-y))

```


```python
def logistic(X,y,epochs,alpha):
    weights=np.zeros(5)
    for i in range(epochs):
        product=np.dot(X,weights)
        #print(product)
        p=sigmoid(product)
        l=-1/X.shape[0]*np.sum(y*np.log(p)+(1-y)*np.log(1-p))
        #print(l)
        cost.append(l)
        weights=weights-alpha*gradient(X,y,p)
    return weights
```


```python
w1=logistic(X,y,500,0.02)
diff=np.round(sigmoid(X@w1))-y 
(len(y)-np.count_nonzero(diff))/len(y)*100
```




    100.0




```python
np.linalg.pinv?
```


```python

```

96fd512a15bb89383934fe20db7a5028ec31839a
ipynb
Jupyter Notebook
Logistic_Regression_from_scratch.ipynb
Joemuthui/ML_theory_and_code
ec5ae52809044fca3daf4064f1e3d09224f60ae4
Apache-2.0
Logistic_Regression_from_scratch.ipynb
Joemuthui/ML_theory_and_code
ec5ae52809044fca3daf4064f1e3d09224f60ae4
Apache-2.0
Logistic_Regression_from_scratch.ipynb
Joemuthui/ML_theory_and_code
ec5ae52809044fca3daf4064f1e3d09224f60ae4
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES