# The resilu linearity / non-linearity

The function $resilu(x)=\frac{x}{1-e^{-x}}$ can be written as the sum of a linear funciton and a function that limits to relu(x).

By using resilu(x) and 'non'-linearity in neural networks, the effect of a skip-connection should thus be included 'for free'.


```python
import copy
import numpy as np
import matplotlib.pyplot as plt
import math
import sympy
```


```python
x=np.arange(-20,20,0.01)
```


```python
def resilu(x):
    return x/(1.0-np.exp(x*-1.0))

def relu(x):
    y=copy.copy(x)
    y[y<0]=0.0
    return y
    
```

Comparing $resilu(x)=\frac{x}{1-e^{-x}}$ with the $relu(x)$ function.

By introducing a parameter $a$, $relu(x)$ is a limit of $resilu(x,a)$:
    
$$\lim_{a\to\infty}\frac{x}{1-e^{-x a}}=relu(x)$$


```python
fig=plt.figure()
ax=fig.add_axes([0,0,1,1])
ax.plot(x,resilu(x),label='resilu')
ax.plot(x,relu(x),linestyle=':',label='relu')
ax.legend()
```


```python
def resilu_nonlin(x):
    return x/(np.exp(x)-1.0)

def resilu_lin(x):
    return x
```

Resilu(x) can be split into a sum of two functions:

* linear part $f_1(x)=x$
* non-linear part $f_2(x)=\frac{x}{e^x-1}$, with exactly similar limit properties as the complete resilu function: $lim_{a\to\infty}\frac{x}{e^{x a}-1}=relu(-x)$

$$
resilu(x)=f_1(x)+f_2(x)=\frac{x}{e^x-1} + x = \frac{x}{1-e^{-x}}
$$

This should show that using resilu non-linearity should be äquivalent to using a 'standard' non-linearity and a skip connection that ommits the nonlinearity as additive residual connection. The only difference are arbitrary linear transformations which the net can easily cancel out.


```python
fig=plt.figure()
ax=fig.add_axes([0,0,1,1])
ax.plot(x,resilu_nonlin(x),label='resilu (non-lin part)')
ax.plot(x,resilu_lin(x),label='resilu (lin part)')
ax.plot(x,resilu_nonlin(x)+resilu_lin(x),label='non-lin + lin (= complete resilu)')
ax.plot(x,relu(x),linestyle=':',label='relu(x)')
ax.plot(x,relu(-x),linestyle=':',label='relu(-x)')
ax.legend()
```

## Numerical instability at $x=0$

$\frac{x}{1-e^{-x}}$ is obviously numerically unstable at $x=0$, which could be seen as a phase transition or an unstable fixed point.

In order to use the resilu(x) function, the function itself and it's derivate (which are both not numerically stable around $x=0$ are approximated with $\mathcal{O}(4)$ taylor expansions for a small interval $-h<0<h$ in the actual implementation. [See `Resilu` and `dResilu()` implementation](https://github.com/domschl/syncognite/blob/master/cpneural/layers/cpl-nonlinearity.h)

## Series expansion to work with instabilities

### Taylor series approximation for resilu(x)


```python
x=sympy.Symbol('x')
sympy.series(x/(1-sympy.exp(-x)),x,n=10)
```




$\displaystyle 1 + \frac{x}{2} + \frac{x^{2}}{12} - \frac{x^{4}}{720} + \frac{x^{6}}{30240} - \frac{x^{8}}{1209600} + O\left(x^{10}\right)$



### $d/dx$ $resilu(x)$ and it's Taylor approximation


```python
sympy.diff(x/(1-sympy.exp(-x)),x)
```




$\displaystyle - \frac{x e^{- x}}{\left(1 - e^{- x}\right)^{2}} + \frac{1}{1 - e^{- x}}$




```python
sympy.series(sympy.diff(x/(1-sympy.exp(-x)),x),x,n=10)
```




$\displaystyle \frac{1}{2} + \frac{x}{6} - \frac{x^{3}}{180} + \frac{x^{5}}{5040} - \frac{x^{7}}{151200} + \frac{x^{9}}{4790016} + O\left(x^{10}\right)$




```python
sympy.diff(sympy.series(x/(1-sympy.exp(-x)),x,n=10),x)
```




$\displaystyle \frac{1}{2} + \frac{x}{6} - \frac{x^{3}}{180} + \frac{x^{5}}{5040} - \frac{x^{7}}{151200} + O\left(x^{9}\right)$




```python

```

bf4e7f2861021168f6f07e5100ee6901829e2704
ipynb
Jupyter Notebook
doc/resilu-linearity.ipynb
domschl/syncognite
cc4c2e0f4d4207a97e81406b50f154bbae90d524
MIT
2017-02-03T15:47:42.000Z
2021-09-24T13:30:47.000Z
doc/resilu-linearity.ipynb
domschl/syncognite
cc4c2e0f4d4207a97e81406b50f154bbae90d524
MIT
doc/resilu-linearity.ipynb
domschl/syncognite
cc4c2e0f4d4207a97e81406b50f154bbae90d524
MIT
2019-04-08T21:31:09.000Z
2019-04-08T21:31:09.000Z
Qwen/Qwen-72B
1. YES
2. YES