# Entropy and Cross-Entropy

Created by Andres Segura-Tinoco  
Created on Mar 15, 2021

Source: https://en.wikipedia.org/wiki/Entropy_(information_theory)


```
import math
```

**Entropy** is a measure of unpredictability, so its value is inversely related to the compression capacity of a chain of symbols.
\begin{align}
    Entropy(X) = H(X) = \sum_{i=1}^n P(x_i) \log_{2} \frac{1}{P(x_i)} \tag{1}
\end{align}

The **Cross-Entropy** of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows:
\begin{align}
    Cross-Entropy(p, q) = H(p, q) = \sum_{i=1}^n P(x_i) \log_{2} \frac{1}{Q(x_i)} \tag{2}
\end{align}


```
# Probability distributions P of x set
p_x = [0.1, 0.2, 0.15, 0.15, 0.4]
sum(p_x)
```




    1.0




```
# Calculate entropy of Px
h_x = 0
for x in p_x:
  h_x -= x * math.log2(x)
h_x
```




    2.1464393446710153




```
# Probability distributions Qgu of x set
q_x = [0.08, 0.24, 0.13, 0.17, 0.38]
sum(q_x)
```




    1.0




```
# Calculate cross-entropy of Qx relative to Px
ce_x = 0
for p, q in zip(p_x, q_x):
  ce_x += p * math.log2(1/q)
ce_x
```




    2.1595073003443446




```
# Validation
h_x < ce_x
```




    True



<hr>
You can contact me on <a href="https://twitter.com/SeguraAndres7" target="_blank">Twitter</a> | <a href="https://github.com/ansegura7/" target="_blank">GitHub</a> | <a href="https://www.linkedin.com/in/andres-segura-tinoco/" target="_blank">LinkedIn</a>

4dc954b4ebf4c9bbad99b7d2d0be1f8ea55af74c
ipynb
Jupyter Notebook
data-science/DS - Cross-Entropy.ipynb
ansegura7/DS_ML_DL_Examples
ccfaa3d84e6d0314f3a826f476cfd20e5e7b73d1
MIT
2021-05-17T17:08:04.000Z
2022-03-31T04:55:40.000Z
data-science/DS - Cross-Entropy.ipynb
ansegura7/DS_ML_DL_Examples
ccfaa3d84e6d0314f3a826f476cfd20e5e7b73d1
MIT
data-science/DS - Cross-Entropy.ipynb
ansegura7/DS_ML_DL_Examples
ccfaa3d84e6d0314f3a826f476cfd20e5e7b73d1
MIT
2021-05-17T17:08:06.000Z
2021-09-28T03:06:10.000Z
Qwen/Qwen-72B
1. YES
2. YES