The question concerns a very general setting and a very general inequality about KL divergence. I'm writing this thread to verify whether my intuition is correct.


Let $E\_1, E\_2$ be two measurable spaces, $f, g: E\_1 \rightarrow E\_2$ two measurable functions, and $A, B$ two random variables taking values in $E\_1$. Is the following inequality:


$D\_{KL}(f(A) || g(B)) \le D\_{KL}(A || B)$


always true?


In case it isn't, I'd like to ask whether the following inequality is always true:


$D\_{KL}(f(A) || f(B)) \le D\_{KL}(A || B)$



A general inequality for KL divergence of functions of variables
$\newcommand{\si}{\sigma}\newcommand{\F}{\mathcal F}\newcommand{\G}{\mathcal G}\newcommand{\pa}{\parallel}$The first inequality is obviously false in general, e.g. when $A=B$ but $f(A)$ differs from $g(B)=g(A)$ in distribution.


The second inequality is true. A bit more generally, let $\mu$ and $\nu$ be probability measures defined on a $\si$-algebra $\F$. Let $\mu\_\G$ and $\nu\_\G$ be the restrictions of $\mu$ and $\nu$ to a sub-$\si$-algebra $\G$ of $\F$.



> 
> Then
> \begin{equation\*}
>  D\_{KL}(\mu\_\G\pa\nu\_\G)\le D\_{KL}(\mu\pa\nu). \tag{1}\label{1}
> \end{equation\*}
> 
> 
> 


Indeed,
\begin{equation\*}
 D\_{KL}(\mu\pa\nu)=\int d\nu\,h\Big(\frac{d\mu}{d\nu}\Big),
\end{equation\*}
where $h(u):=u\ln u$ for $u\in(0,\infty)$, with $h(0):=0$ and $h(\infty)=\infty$.
Next, letting $E\_\nu(\cdot|\G)$ denote the conditional expectation given $\G$ w.r.t. $\nu$, for any nonnegative $\G$-measurable function $q$ we have
\begin{equation\*}
 \int d\nu\_\G\,q\, E\_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big)
 = \int d\nu\,q\, E\_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big)\, 
 =\int d\nu\,q\, \frac{d\mu}{d\nu}\, 
 = \int d\mu\,q= \int d\mu\_\G\,q; 
\end{equation\*}
the second equality in the latter display holds by the definition of the conditional expectation. So,
\begin{equation\*}
 \frac{d\mu\_\G}{d\nu\_\G}=E\_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big). 
\end{equation\*}
So, by Jensen's inequality applied to the convex function $h$,
\begin{multline\*}
 D\_{KL}(\mu\_\G\pa\nu\_\G)
 =\int d\nu\,h\Big(\frac{d\mu\_\G}{d\nu\_\G}\Big)
 =\int d\nu\,h\Big(E\_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big)\Big) \\ 
 \le \int d\nu\,E\_\nu\Big( h\Big(\frac{d\mu}{d\nu}\Big)\Big|\G\Big)
 =\int d\nu\,h\Big(\frac{d\mu}{d\nu}\Big) 
 =D\_{KL}(\mu\pa\nu), 
\end{multline\*}
so that \eqref{1} follows.


Your second inequality now obtains by letting $\mu$ and $\nu$ be the distributions of $A$ and $B$, respectively, over the measurable space $(E\_1,\F)$ and letting $\G$ be the sub-$\si$-algebra of $\F$ generated by the measurable map $f$.
$\quad\Box$


