# "[Draft] Optimizer Code Supplement Using fastai"
> "Building the Optimizers explained by Sebastian Ruder using fastai"

- toc: true
- branch: master
- badges: true
- comments: true
- author: Kevin Bird
- categories: [fastai]

## Introduction

This post is meant to be a supporting guide that can be used with the incredibly detailed [post from Sebastian Ruder](https://ruder.io/optimizing-gradient-descent/index.html).  The goal is to implement each of these optimizers using fastai and run them on a small problem.  
  
The sample problem that will be used is the one created by Jeremy and Rachel in the graddesc spreadsheet.  

Because this post is generated using Jupyter Notebooks, you can also interact with any of these concepts and modify them as you see fit.  In as many places as possible, I will try to connect the code back to the concepts by utilizing the greek letters Sebastian (and usually the paper) uses


```python
#hide
from fastai.vision.all import *
```


```python
a_real = 2
b_real = 30
```

`a_real` is a weight and `b_real` is a bias they form the equation $a\_real*x + b\_real= y$  
Both a and b are parameters and will be learned by our optimizer through the loss function.  
In a real world problem, these values would be unknown and there would typically be a lot more weights, but the concept will remain the same whether there are 2 parameters or [175 billion](https://www.google.com/search?client=firefox-b-1-d&q=how+many+parameters+does+gpt+3+have). 


```python
#hide
class MySimpleModel(torch.nn.Module):
    def __init__(self):
        super(MySimpleModel, self).__init__()
        self.linear = nn.Linear(1,1)
        self.linear.weight.data = nn.Parameter(torch.tensor([[1.0]]))
        self.linear.bias.data = nn.Parameter(torch.tensor([1.0]))
        self.a = self.linear.weight
        self.b = self.linear.bias
        
    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.linear(x)
        return x
```


```python
#hide
class OptimizerInsight(Callback):
    def before_batch(self):
        if self.training:
            self.a_guess+=[float(self.model.a)]
            self.b_guess+=[float(self.model.b)]
            #plt.plot(theta_real,b_real, 'ro', markersize=5)
            #print(self.model.theta.tolist())
    def before_epoch(self):
        try:
            self.a_guess = [self.a_guess[-1]]
        except:
            self.a_guess = []
        try:
            self.b_guess = [self.b_guess[-1]]
        except:
            self.b_guess = []
        
    def after_epoch(self):
        plt.plot(self.a_guess, self.b_guess)
```


```python
#hide
x_list = torch.tensor([14.0,86,28,51,28,29,72,62,84,15,42,62,47,35,9,38,44,99,13,21,28,20,8,64,99,70,27,17,8])
y_list = torch.tensor([58.0,202,86,132,86,88,174,154,198,60,114,154,124,100,48,106,118,228,56,72,86,70,46,158,228,170,84,64,46])
b_list = [(x,y) for x,y in zip(x_list, y_list)]
```


```python
#hide
dl = DataLoader(b_list, bs=4)
#setting validation set equal to training dataset.  For this example, we are only interested in the training dataset
dls = DataLoaders(dl,dl)
```

## Gradient Descent Variants

### [Batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#batchgradientdescent)

Forumula:  
$\theta = \theta - \eta \cdot \nabla_\theta J( \theta)$
  
Code:  
```python
def sgd_step(p, lr, **kwargs):
    p.data.add_(p.grad.data, alpha=-lr)
```

Translation:  
  $\theta$ = p  
  $\eta$ = lr  
  $\nabla_\theta J( \theta)$ = p.grad.data


```python
def _do_epoch_BGD(self):
    self._do_epoch_train()
    self._step()
    self('after_step')
    self.opt.zero_grad()
    self._do_epoch_validate()
```


```python
def _do_one_batch_BGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
```


```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.00005
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_BGD,learn)
learn._do_one_batch = partial(_do_one_batch_BGD,learn)
```


```python
learn.fit(10)
```

### [Stochastic Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#stochasticgradientdescent)

Formula:  
$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i)}; y^{(i)})$  
  
Code:  
```python
def sgd_step(p, lr, **kwargs):
    p.data.add_(p.grad.data, alpha=-lr)
```

Translation:  
  $\theta$ = p  
  $\eta$ = lr  
  $\nabla_\theta J( \theta; x^{(i)}; y^{(i)})$ = p.grad.data


```python
def _do_epoch_SGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
```


```python
def _do_one_batch_SGD(self):
    #pdb.set_trace()
    for i,b in enumerate(zip(self.xb[0][None].T,self.yb[0][None].T)):
        x,y = b
        self.pred = self.model(x)
        self('after_pred')
        if len(y): self.loss = self.loss_func(self.pred, y)
        self('after_loss')
        if not self.training or not len(self.yb): return
        self('before_backward')
        self._backward()
        self('after_backward')
        self._step()
        self('after_step')
        self.opt.zero_grad()
```


```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_SGD,learn)
learn._do_one_batch = partial(_do_one_batch_SGD,learn)
```


```python
learn.fit(10)
```

### [Mini-batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#minibatchgradientdescent)

$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i:i+n)}; y^{(i:i+n)})$

Comments about Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent:  
All three of these use itentical code for the optimizer itself.  The huge difference between these three Optimizers is when the step actually occurs.  


```python
def _do_epoch_MBGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
```


```python
def _do_one_batch_MBGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
    self._step()
    self('after_step')
    self.opt.zero_grad()
```


```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_MBGD,learn)
learn._do_one_batch = partial(_do_one_batch_MBGD,learn)
```


```python
learn.fit(10)
```

The SGD Optimizer can do all three SGD($\theta$, $\eta$) above with the same parameters.  

**Note**: `mom`, `wd`, and `decouple_wd` are all 0 or don't influence things with their default values.  

## Beyond Vanilla Gradient Descent

### [SGD with Momentum](https://ruder.io/optimizing-gradient-descent/index.html#momentum)

$
\begin{align} 
\begin{split} 
v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta) \\ 
\theta &= \theta - v_t 
\end{split} 
\end{align}
$


```python

```

### [Nesterov Accelerated Gradient](https://ruder.io/optimizing-gradient-descent/index.html#nesterovacceleratedgradient)

$
\begin{align} 
\begin{split} 
v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta - \gamma v_{t-1} ) \\ 
\theta &= \theta - v_t 
\end{split} 
\end{align}
$


```python

```

### [Adagrad](https://ruder.io/optimizing-gradient-descent/index.html#adagrad)

$g_{t, i} = \nabla_\theta J( \theta_{t, i} )$  
$\theta_{t+1, i} = \theta_{t, i} - \eta \cdot g_{t, i}$  
$\theta_{t+1, i} = \theta_{t, i} - \dfrac{\eta}{\sqrt{G_{t, ii} + \epsilon}} \cdot g_{t, i}$  
$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_{t} + \epsilon}} \odot g_{t}$


```python

```

### [Adadelta](https://ruder.io/optimizing-gradient-descent/index.html#adadelta)

$E[g^2]_t = \gamma E[g^2]_{t-1} + (1 - \gamma) g^2_t$  
$
\begin{align} 
\begin{split} 
\Delta \theta_t &= - \eta \cdot g_{t, i} \\ 
\theta_{t+1} &= \theta_t + \Delta \theta_t \end{split} 
\end{align}
$  
$\Delta \theta_t = - \dfrac{\eta}{\sqrt{G_{t} + \epsilon}} \odot g_{t}$  
$\Delta \theta_t = - \dfrac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_{t}$  
$\Delta \theta_t = - \dfrac{\eta}{RMS[g]_{t}} g_t$  
$E[\Delta \theta^2]_t = \gamma E[\Delta \theta^2]_{t-1} + (1 - \gamma) \Delta \theta^2_t$  
$RMS[\Delta \theta]_{t} = \sqrt{E[\Delta \theta^2]_t + \epsilon}$  
$
\begin{align} 
\begin{split} 
\Delta \theta_t &= - \dfrac{RMS[\Delta \theta]_{t-1}}{RMS[g]_{t}} g_{t} \\ 
\theta_{t+1} &= \theta_t + \Delta \theta_t 
\end{split} 
\end{align}
$  


```python

```

### [RMSprop](https://ruder.io/optimizing-gradient-descent/index.html#rmsprop)


```python

```


```python

```

### [Adam](https://ruder.io/optimizing-gradient-descent/index.html#adam)


```python

```


```python

```

### [AdaMax](https://ruder.io/optimizing-gradient-descent/index.html#adamax)


```python

```


```python

```

### [Nadam](https://ruder.io/optimizing-gradient-descent/index.html#nadam)


```python

```


```python

```

### [AMSGrad](https://ruder.io/optimizing-gradient-descent/index.html#amsgrad)


```python

```


```python

```

### AdamW


```python

```


```python

```

### QHAdam


```python

```


```python

```

### AggMo


```python

```


```python

```

df49f251d7e759b1f1c1f150f0c1f0c598288317
ipynb
Jupyter Notebook
_notebooks/2020-09-07-Optimizer_Code_Supplement.ipynb
kevinbird15/Blog
98cd08dc40dc5c7becd501e545920a7ccdd2d517
Apache-2.0
_notebooks/2020-09-07-Optimizer_Code_Supplement.ipynb
kevinbird15/Blog
98cd08dc40dc5c7becd501e545920a7ccdd2d517
Apache-2.0
2020-09-05T17:39:18.000Z
2022-02-26T09:53:45.000Z
_notebooks/2020-09-07-Optimizer_Code_Supplement.ipynb
kevinbird15/PersonalBlog
98cd08dc40dc5c7becd501e545920a7ccdd2d517
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES