```python
import numpy as np
import matplotlib.pyplot as plt
```

# 8-3. Activation functions

## Sigmoid function

$$
\sigma (x) = \frac{1}{1 + \exp(-x)}
$$


```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```


```python
sigmoid(0)
```




    0.5




```python
sigmoid(-10)
```




    4.5397868702434395e-05




```python
sigmoid(10)
```




    0.99995460213129761




```python
x = np.arange(-5, 5, 0.1)
y = sigmoid(x)

plt.plot(x, y)
plt.show()
```

## Derivative of sigmoid function

\begin{align}
\sigma (x) = \frac{1}{1 + \exp(-x)}
\end{align}

Let

\begin{align}
f(x) &= 1 + \exp(-x) \\
g(u) &= u^{-1}
\end{align}

Now we can write the sigmoid function using $f(x)$ and $g(u)$

\begin{align}
\sigma (x) = g(f(x))
\end{align}

As derivatives of $f(x)$ and $g(u)$ are as follows,

\begin{align}
f'(x) &= - \exp(-x) \\
g'(u) &= - u^{-2}
\end{align}

and by using the chain rule

\begin{align}
\sigma' (x) &= g'(f(x)) \cdot f'(x) \\
&= g'(1 + \exp(-x)) \cdot (- \exp(-x)) \\
&= -(1 + \exp(-x))^{-2} \cdot (- \exp(-x)) \\
&= - \frac{- \exp(-x)}{(1 + \exp(-x))^{-2}} \\
&= \frac{\exp(-x)}{(1 + \exp(-x))^{-2}} \\
&= \frac{1}{1 + \exp(-x)} \cdot \frac{\exp(-x)}{1 + \exp(-x)} \\
&= \frac{1}{1 + \exp(-x)} \cdot \left(  1 - 1 + \frac{\exp(-x)}{1 + \exp(-x)} \right) \\
&= \frac{1}{1 + \exp(-x)} \cdot \left( 1 - \frac{1 + \exp(-x)}{1 + \exp(-x)} + \frac{\exp(-x)}{1 + \exp(-x)} \right) \\
&= \frac{1}{1 + \exp(-x)} \cdot \left( 1 - \frac{1}{1 + \exp(-x)} \right)
\end{align}

Becase $\sigma (x) = \frac{1}{1 + \exp(-x)}$

\begin{align}
\sigma' (x) &= \sigma(x) (1 - \sigma(x))
\end{align}


```python
def sigmoid_prime(x):
    s = sigmoid(x)
    return s * (1 - s)
```


```python
sigmoid_prime(0)
```




    0.23500371220159449




```python
sigmoid_prime(-1)
```




    0.19661193324148185




```python
sigmoid_prime(1)
```




    0.19661193324148185




```python
x = np.arange(-5, 5, 0.1)
y = sigmoid_prime(x)

plt.plot(x, y)
plt.show()
```

## Tanh function

Tanh function or hyperbolic tangent function is defined as follows

$$
\tanh (x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}
$$

Also as other hyperbolic functions (hyperbolic sine and hyperbolic cosine) defined as following,

\begin{align}
\sinh (x) = \frac{\exp(x) - \exp(-x)}{2} \\
\cosh (x) = \frac{\exp(x) + \exp(-x)}{2}
\end{align}

tanh function can be express as below (that is the origina definition)

$$
\tanh (x) = \frac{\sinh (x)}{\cosh (x)}
$$


```python
def tanh(x):
    return np.tanh(x)
```


```python
tanh(0)
```




    0.0




```python
tanh(-10)
```




    -0.99999999587769273




```python
tanh(5)
```




    0.99990920426259511




```python
x = np.arange(-5, 5, 0.1)
y = tanh(x)

plt.plot(x, y)
plt.show()
```

## Derivative of tanh function

$$
\tanh (x) = \frac{\sinh (x)}{\cosh (x)}
$$

Because derivatives of sinh and cosh are as below

\begin{align}
\sinh'(x) = \cosh (x)
\cosh'(x) = \sinh (x)
\end{align}

and by using quotient rule

\begin{align}
\tanh'(x) &= \frac{\sinh' (x) \cosh (x) - \sinh (x) \cosh' (x)}{\cosh^2 (x)} \\
&= \frac{\cosh^2 (x) - \sinh^2 (x)}{\cosh^2 (x)} \\
&= 1 - \tanh^2 (x)
\end{align}





```python
def tanh_prime(x):
    t = tanh(x)
    return 1 - np.power(t, 2)
```


```python
tanh_prime(0)
```




    1.0




```python
tanh_prime(-1)
```




    0.41997434161402614




```python
tanh_prime(1)
```




    0.41997434161402614




```python
x = np.arange(-5, 5, 0.1)
y = tanh_prime(x)

plt.plot(x, y)
plt.show()
```

## ReLU function

ReLU or rectified linear function is defined as following:

$$
f(x) = \max(0, x)
$$


```python
def relu(x):
    return np.maximum(0, x)
```


```python
relu(2)
```




    2




```python
relu(10)
```




    10




```python
relu(-1.5)
```




    0.0




```python
x = np.arange(-5, 5, 0.1)
y = relu(x)

plt.plot(x, y)
plt.show()
```

## Derivative of ReLU function

$$
f'(x)= \begin{cases}
      1 & (x > 0) \\
      0 & (x \leq 0)
\end{cases}
$$


```python
def relu_prime(x):
    return 1 * (x > 0)
```


```python
relu_prime(2)
```




    1




```python
relu_prime(10)
```




    1




```python
relu_prime(-1.5)
```




    0




```python
x = np.arange(-5, 5, 0.1)
y = relu_prime(x)

plt.plot(x, y)
plt.show()
```

## Leaky ReLU

$$
f(x) = max(\alpha x, x)
$$

where $\alpha$ is a small constnt such as 0.01.


```python
def lrelu(x, alpha=0.05):
    return np.maximum(alpha * x, x)
```


```python
lrelu(2)
```




    2.0




```python
lrelu(10)
```




    10.0




```python
lrelu(-1.5)
```




    -0.075000000000000011




```python
x = np.arange(-5, 5, 0.1)
y = lrelu(x)

plt.plot(x, y)
plt.show()
```

## Derivative of Leaky ReLU function

$$
f'(x)= \begin{cases}
      1 & (x > 0) \\
      \alpha & (x \leq 0)
\end{cases}
$$


```python
def lrelu_prime(x, alpha=0.05):
    return 1 * (x > 0) + alpha * (x <= 0)
```


```python
lrelu_prime(2)
```




    1.0




```python
lrelu_prime(10)
```




    1.0




```python
lrelu_prime(-1.5)
```




    0.05




```python
x = np.arange(-5, 5, 0.1)
y = lrelu_prime(x)

plt.plot(x, y)
plt.show()
```

## Comparison


```python
x = np.arange(-3, 3, 0.1)
y_sigmoid = sigmoid(x)
y_tanh = tanh(x)
y_relu = relu(x)
y_lrelu = lrelu(x)

plt.plot(x, y_sigmoid, 'b', label='sigmoid')
plt.plot(x, y_tanh, 'r', label='tanh')
plt.plot(x, y_relu, 'g', label='relu')
plt.plot(x, y_lrelu, 'c', label='lrelu')
plt.legend()
plt.show()
```


```python
x = np.arange(-6, 6, 0.1)
y_sigmoid = sigmoid_prime(x)
y_tanh = tanh_prime(x)
y_relu = relu_prime(x)
y_lrelu = lrelu_prime(x)

plt.plot(x, y_sigmoid, 'b', label="sigmoid'")
plt.plot(x, y_tanh, 'r', label="tanh'")
plt.plot(x, y_relu, 'g', label="relu'")
plt.plot(x, y_lrelu, 'c', label="lrelu'")
plt.legend()
plt.show()
```


```python

```

1b0fced15790495f6d20bc275867fe14dbfc270c
ipynb
Jupyter Notebook
08_3_activation_functions.ipynb
takaiwai/deep-learning-notes
d55b0c8b33238791a08b5ff0b2ef85bff4496624
MIT
08_3_activation_functions.ipynb
takaiwai/deep-learning-notes
d55b0c8b33238791a08b5ff0b2ef85bff4496624
MIT
08_3_activation_functions.ipynb
takaiwai/deep-learning-notes
d55b0c8b33238791a08b5ff0b2ef85bff4496624
MIT
Qwen/Qwen-72B
1. YES
2. YES