# Activation Functions

This function introduces activation functions in TensorFlow

We start by loading the necessary libraries for this script.


```python
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
# from tensorflow.python.framework import ops
# ops.reset_default_graph()
tf.reset_default_graph()
```

### Start a graph session


```python
sess = tf.Session()
```

### Initialize the X range values for plotting


```python
x_vals = np.linspace(start=-10., stop=10., num=100)
```

### Activation Functions:

ReLU activation


```python
print(sess.run(tf.nn.relu([-3., 3., 10.])))
y_relu = sess.run(tf.nn.relu(x_vals))
```

    [ 0.  3. 10.]


ReLU-6 activation


```python
print(sess.run(tf.nn.relu6([-3., 3., 10.])))
y_relu6 = sess.run(tf.nn.relu6(x_vals))
```

    [0. 3. 6.]


ReLU-6 refers to the following function

\begin{equation}
    \min\left(\max(0, x), 6\right)
\end{equation}


Sigmoid activation


```python
print(sess.run(tf.nn.sigmoid([-1., 0., 1.])))
y_sigmoid = sess.run(tf.nn.sigmoid(x_vals))
```

    [0.26894143 0.5        0.7310586 ]


Hyper Tangent activation


```python
print(sess.run(tf.nn.tanh([-1., 0., 1.])))
y_tanh = sess.run(tf.nn.tanh(x_vals))
```

    [-0.7615942  0.         0.7615942]


Softsign activation


```python
print(sess.run(tf.nn.softsign([-1., 0., 1.])))
y_softsign = sess.run(tf.nn.softsign(x_vals))
```

    [-0.5  0.   0.5]


softsign refers to the following function

\begin{equation}
    \frac{x}{1 + |x|}
\end{equation}

<br>



Softplus activation




```python
print(sess.run(tf.nn.softplus([-1., 0., 1.])))
y_softplus = sess.run(tf.nn.softplus(x_vals))
```

    [0.31326166 0.6931472  1.3132616 ]


Softplus refers to the following function

\begin{equation}
    \log\left(\exp(x) + 1\right)
\end{equation}


Exponential linear activation


```python
print(sess.run(tf.nn.elu([-1., 0., 1.])))
y_elu = sess.run(tf.nn.elu(x_vals))
```

    [-0.63212055  0.          1.        ]


ELU refers to the following function

\begin{equation}\label{eq:}
    f = \begin{cases}
        \exp(x) - 1 &(x < 0 )\\
        0 &(x \geq 0 )\\
        \end{cases}
\end{equation}


### Plot the different functions


```python
plt.style.use('ggplot')
plt.plot(x_vals, y_softplus, 'r--', label='Softplus', linewidth=2)
plt.plot(x_vals, y_relu, 'b:', label='ReLU', linewidth=2)
plt.plot(x_vals, y_relu6, 'g-.', label='ReLU6', linewidth=2)
plt.plot(x_vals, y_elu, 'k-', label='ExpLU', linewidth=0.5)
plt.ylim([-1.5,7])
plt.legend(loc='upper left', shadow=True, edgecolor='k')
plt.show()

plt.plot(x_vals, y_sigmoid, 'r--', label='Sigmoid', linewidth=2)
plt.plot(x_vals, y_tanh, 'b:', label='Tanh', linewidth=2)
plt.plot(x_vals, y_softsign, 'g-.', label='Softsign', linewidth=2)
plt.ylim([-1.3,1.3])
plt.legend(loc='upper left', shadow=True, edgecolor='k')
plt.show()
```






```python

```

4aecf5e2aea19df62d47768b5fcf2465a61c4c86
ipynb
Jupyter Notebook
Chap01/06_activation_functions.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
2021-02-27T16:16:02.000Z
2021-02-27T16:16:02.000Z
01_Introduction/06_Implementing_Activation_Functions/06_activation_functions.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
2018-03-07T14:31:22.000Z
2018-03-07T15:04:17.000Z
Chap01/06_activation_functions.ipynb
haru-256/tensorflow_cookbook
18923111eaccb57b47d07160ae5c202c945da750
MIT
Qwen/Qwen-72B
1. YES
2. YES