There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.


Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:


$$E(\mu|\omega)=\int\_M\log\phi\_\mu \ d\mu,$$


where $\displaystyle \phi\_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi\_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since


\begin{align\*}
E(\mu|\omega)&=\int\log\phi\_\mu d\mu=\int\_{\phi\_\mu>0}-\log\frac{d\omega}{d\mu} d\mu
\\\&\ge-\log\int\_{\phi\_\mu>0}\frac{d\omega}{d\mu} d\mu
=-\log\omega(\lbrace\phi\_\mu=0\rbrace)\ge0.
\end{align\*}


In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.


Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$


As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.




---


A more interesting statement I want to know if that, if $\mu\_n\ll\omega$ such that $\mu\_n\to\mu\not$$\ll\omega$, will we always have $E(\mu\_n|\omega)\to+\infty$? Thank you!


Here $\mu\_n\to\mu$ in the sense that $\mu\_n(f)\to\mu(f)$ for all continuous functions $f$.




---


The [paper](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1055416&tag=1) given by Ashok below provides an equivalent definition: $\displaystyle E(\mu|\omega)=\sup\_{\alpha}\sum\_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}$, where the supremum is taken over all finite, Borel partitions $\alpha$ with $\omega(A)>0$. 


In particular if $\mu\not$$\ll\omega$, we can take open sets $A\_k$ with $\mu(A\_k)\ge2\delta$ and $\omega(A\_k)\to 0$. So $E(\mu|\omega)=+\infty$.


Now let's make a better choice of $A\_k$'s such that $\mu(\partial A\_k)=0$. Then if $\mu\_n\to\mu$, $\mu\_n(A\_k)\to\mu(A\_k)$ for all $k$. Hence we can pick $n\_k$ such that $\mu\_{n}(A\_k)\ge\delta$ for all $n\ge n\_k$. So for all $n\ge n\_k$, we have
$$E(\mu\_{n}|\omega)\ge\mu\_{n}(A\_k)\log\frac{\mu\_{n}(A\_k)}{\omega(A\_k)}
\ge\delta\cdot\log\frac{\delta}{\omega(A\_k)}\to\infty.$$



Convexity and semicontinuity of the relative entropy function
If $M$ is complete and separable, then $E(\mu|\omega)$ is lower semicontinuous in $\mu$ on the set of all probability measures on $M$ with respect to the weak convergence of probability measures, see Theorem 1 in section III of this [paper](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1055416).


Once we have lower semicontinuity, we have $$ \liminf\_{n\to \infty}E(\mu\_n|\omega)\ge E(\mu|\omega) $$


Since $\mu$ is not absolutely continuous with respect to $\omega$, we have $E(\mu|\omega)=\infty$, hence the left hand side is also $\infty$.


