Hi,


$P\_1$, $P\_2$, $P\_3$ are probability distributions defined on the same support.


Knowing that $H(P\_1) < H(P\_2) < H(P\_3)$, can we compare $D\_{KL}(P\_2,P\_1)$ and $D\_{KL}(P\_3,P\_1)$ ?


(H is the Shannon Entropy and $D\_{KL}$ is the Kullback–Leibler divergence)


Thank you.



KL divergence(s) comparison,
In general there is no relation between the two divergences. In fact, both of the divergences may be either finite or infinite, independent of the values of the entropies.


To be precise, if $P\_1$ is not absolutely continuous w.r.t. $P\_2$, then $D\_{KL}(P\_2,P\_1)=\infty$. Similarly, $D\_{KL}(P\_2,P\_1)=\infty$. This fact is independent of the entropies of $P\_1$, $P\_2$ and $P\_3$. Hence, by continuity, the ratio $D\_{KL}(P\_2,P\_1)/D\_{KL}(P\_3,P\_1)$ can be arbitrary.


