Suppose that $D\_{KL}(p\_1\parallel q)<1$ and $D\_{KL}(p\_2\parallel q)<1$. I'm trying to show that either $D\_{KL}(p\_1\parallel p\_2)$ or $D\_{KL}(p\_2\parallel p\_1)$ will have an upper bound close to 1 provided ${q}$ is fixed. It seems intuitive that if $p\_1$ and $q$ are similar enough and if $p\_2$ and $q$ are also similar enough then $p\_2$ can reasonably approximate $p\_1$ or vice versa. Is this actually true?



An inequality relating the Kullback-Leibler divergence of two discrete distributions with constant reference distribution
The answer is no. E.g., let $p\_1,p\_2,q$ be pdf's on $[0,1]$ such that $q=1$, $p\_1=\frac1{1-h}\,1\_{[0,1-h]}$, and $p\_2=\frac1{1-h}\,1\_{[h,1]}$, for some $h\in(0,1)$. Then
$$D\_{KL}(p\_1\parallel q)=D\_{KL}(p\_2\parallel q)=\ln\frac1{1-h}\to0$$
as $h\downarrow0$, whereas
$$D\_{KL}(p\_1\parallel p\_2)=D\_{KL}(p\_2\parallel p\_1)=\infty$$
for all $h\in(0,1)$.


