```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

```

# Using a known function

<h5> We will optimize the following equation:
\begin{equation} y = (x-5)^2 + 3 \end{equation} 

Which we know the minimum is:

\begin{equation} x = 5 \end{equation} 
\begin{equation} y = 3 \end{equation} 


```python
X = np.linspace(0,10, 100) # Get 100 datapoints equally spaced from 0 to 10
y = 3 + np.power((X-5),2)
```

Let's plot our data to check the relation between X and y


```python
plt.plot(X,y,'red')
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.xlim((0,10))
```

# Gradient Descent

## Cost Function & Gradients

<h4> Remember that in optimization methods we optimize a given cost. In our case it will be the quadratic equation below. Just in case calculating gradients is not your thing, I already precalculated the gradient in this case. 



<b>Cost</b>
\begin{equation}
y = (X-5)^2 + 3
\end{equation}

<b>Gradient</b>

\begin{equation}
2*(X-5)
\end{equation}


```python
def cost(X):
    return 3 + np.power((X-5), 2)
```


```python
def gradient(X):
    return 2*(X-5)
```


```python
def gradient_descent(X,learning_rate=0.01,iterations=100, print_all=True):
    costs = []
    for it in range(iterations):
        X = X -learning_rate*gradient(X)
        if it % 50 == 0 and print_all:
            print("The iteration is {} and the current value of X is {}".format(it, X))
        costs.append(cost(X))
    return X, costs
    
```

<h3> Let's start from a random number between 0 and 1, with 200 iterations and a learning rate of 0.01.


```python
lr =0.01
n_iter = 200
X = np.random.random()*10
X, costs = gradient_descent(X,lr,n_iter)
print("Minimum is at (x,y) = ({}, {})".format(X, cost(X)))
```

    The iteration is 0 and the current value of X is 7.399074672684869
    The iteration is 50 and the current value of X is 5.873670256056753
    The iteration is 100 and the current value of X is 5.3181642176498185
    The iteration is 150 and the current value of X is 5.115865761356701
    Minimum is at (x,y) = (5.043055915557471, 3.001853811864492)


<h3> Let's plot the cost history over iterations


```python
fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),costs,'b.')
```

<h3> After around 120 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. And if the learning rate is bigger?


```python
new_lr =0.1
new_n_iter = 50
X_b = np.random.random()
X_new, costs_new = gradient_descent(X_b,new_lr,new_n_iter)
print("Minimum is at (x,y) = ({}, {})".format(X_new, cost(X_new)))


fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
_=ax.plot(range(new_n_iter),costs_new,'b.')
```

<b>Clearly with a higher learning rate we got to the optimal point in 10 iterations. Let's see how this changes with different learning rates


```python
learning_rates = [0.01,0.1,1,10]
iterations = 200
X_0 = np.random.random()
plt.figure(figsize=(20,5))
all_costs = []
for lr in learning_rates:
    X, costs = gradient_descent(X_0,lr,iterations, print_all=True)
    print("Minimum is at (x,y) = ({}, {})".format(X, cost(X)))
    all_costs.append(costs)

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
plt.xlim((0,200))
plt.ylim((0,50))
plt.scatter(range(iterations), all_costs[0], color='red')
plt.scatter(range(iterations), all_costs[1], color='blue')
plt.scatter(range(iterations), all_costs[2], color='green')
plt.scatter(range(iterations), all_costs[3], color='black')
plt.legend(['LR = 0.01', 'LR = 0.1', 'LR = 1', 'LR = 10'])
plt.show()
```


```python

```

1594ef62f2fcc2989de09969f8fe088f18019e6b
ipynb
Jupyter Notebook
Module 3/demo_module3_gd_sgd.ipynb
axel-sirota/interpreting-data-with-advanced-models
699dd8281c78d88b31b59f43bdf2f29a57f3f94c
MIT
2021-12-10T13:15:01.000Z
2021-12-10T13:15:01.000Z
Module 3/demo_module3_gd_sgd.ipynb
axel-sirota/interpreting-data-with-advanced-models
699dd8281c78d88b31b59f43bdf2f29a57f3f94c
MIT
Module 3/demo_module3_gd_sgd.ipynb
axel-sirota/interpreting-data-with-advanced-models
699dd8281c78d88b31b59f43bdf2f29a57f3f94c
MIT
2021-04-13T06:07:56.000Z
2021-12-10T13:15:09.000Z
Qwen/Qwen-72B
1. YES
2. YES