### Gradient Descent Algorithm

\begin{align}
w(0) & = \text{some initial value for} \; \tau = 1, 2,\dots \; w(\tau) = w(\tau-1) - \eta \nabla E(w(\tau-1))
\end{align}


| Symbol           | Meaning                                                        |
|:-----------------|:---------------------------------------------------------------|
| $$\tau$$         | Iteration number                                               |
| $$w$$            | Coefficients vector                                            |
| $$w(\tau)$$      | Coefficients vector at **$\tau$** iteration                    |
| $$\eta$$         | Learning rate                                                  |
| $$\nabla$$       | Gradient function                                              |
| $$E(w(\tau-1))$$ | Error function with **w** coefficients at **$\tau$** iteration |

---

\begin{align}
\theta = \theta - \eta \nabla_{\theta} J(\theta)
\end{align}


```python
%matplotlib inline
import matplotlib.pylab as plt

import time
from IPython import display
import numpy as np


#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x = np.array([10., 8., 13., 9., 11., 14., 6., 4., 12., 7., 5.])
N = len(x)

# Design matrix
A = np.vstack((np.ones(N), x)).T
# A = np.vstack((np.ones(N), x, x**2, x**3)).T

# Learning rate
eta = 0.01
              
# Initial parameters
w = np.array([2., 1.,])
# w = np.array([2., 1., 0.1, 0.001])

# Chart
f = A.dot(w)

# loop
for epoch in range(10):
    # Error
    err = y - A.dot(w)
    
    # Total error
    E = np.sum(err ** 2) / N
    
    # Gradient
    dE = -2. * A.T.dot(err) / N
    
    if epoch % 1 == 0: 
        print(epoch, ':', E)    

    # Perfom one descent step
    w = w - eta * dE

```

    0 : 15.9938818182
    1 : 11.6346546383
    2 : 8.57455118197
    3 : 6.4263659935
    4 : 4.91830088281
    5 : 3.85956758638
    6 : 3.11624261595
    7 : 2.5943186121
    8 : 2.2278072717
    9 : 1.97038821883


df229249f058233cf37f6c29e5c9858fc8e7d9b0
ipynb
Jupyter Notebook
notebooks/GradientDescent.ipynb
mustafagonul/machine-learning-examples
42eccd17d14c6ababba5c9c35dd868232614f616
MIT
2017-12-15T06:32:10.000Z
2018-01-31T07:22:14.000Z
notebooks/GradientDescent.ipynb
mustafagonul/machine-learning-examples
42eccd17d14c6ababba5c9c35dd868232614f616
MIT
notebooks/GradientDescent.ipynb
mustafagonul/machine-learning-examples
42eccd17d14c6ababba5c9c35dd868232614f616
MIT
Qwen/Qwen-72B
1. YES
2. YES