The Kullback-Leibler Divergence (KLD) of two PMF's $P(x)$ and $Q(x)$ is $D(P||Q)=\sum\_x P(x)\log(P(x)/Q(x))$, with the provisos that $0\cdot \log (0/p)=0$ and $p\cdot \log (p/0)=+\infty$ whenever $p>0$.


It is known that KLD is continuous at $(P,Q)$ if $Q$ is *strictly positive over all $x$'s*. What can be said otherwise?


To be more specific, assume we are given a sequence of PMF $\{(P\_n,Q\_n)\}\_{n\geq 0}$ s.t. $(P\_n,Q\_n)\rightarrow (P,Q)$ in the simplex of PFM's (with the topology induced by, say, norm-1 distance).


Is it correct to deduce that


$\lim \inf\_{n\rightarrow \infty} D(P\_n||Q\_n) \geq D(P||Q)$ ?


This would follows if KLD is lower-semicontinuous, right?


Many thanks.



Lower semicontinuity of Kullback-Leibler divergence
In addition to the conventions you have mentioned, it is also assumed that $0\log(0/0)=0$.


With these conventions, I think, in the finite case, it is always true that $$\lim\_{n\to \infty} D(P\_n||Q\_n)=D(P||Q)$$ As you said, if $Q(x)>0$ for all $x$, its immediate from the Dominated Convergence theorem.
The problem is only when for some $y$, $Q(y)=0$ whereas $P(y)>0$.


In which case $P(y)\log(P(y)/Q(y))=\infty$ and $D(P\|Q)=\infty$


But since $(P\_n,Q\_n)\to (P,Q)$, we have $P\_n(y)\to P(y)$ and $Q\_n(y)\to Q(y)$, whence $$P\_n(y)\log(P\_n(y)/Q\_n(y))\to P(y)\log(P(y)/Q(y))=\infty.$$ Hence $D(P\_n||Q\_n)\to \infty$


So in any case we have $\lim\_{n\to \infty} D(P\_n||Q\_n)=D(P||Q)$.


In a general measurable space (i.e., if $P\_n, Q\_n, P, Q$ are probability measures on some general measure space say $(\mathbb{X}, \mathcal{X})$), I think, we have only lower semicontinuity.


Pardon me, if something is wrong.


