I have 2 normal distributions $\mathcal{N}(\mu\_1, \mathbb{I}\_d)$ where $\mu\_1$ is a fixed vector in $\mathbb{R}^d$ and $\mathcal{N}(\mu\_2, \mathbb{I}\_d)$, where $\mu\_2$ is $\mu\_1 + V$, where $V$ is uniformly distributed and orthogonal to $\mu\_1$, that is $V^\top \mu\_1 = 0$. (In 3 dimensions $V$ would be uniformly distributed in the unit circle of the $y-z$ plane if $\mu\_1$ is the unit vector in the $x$ axis.)
My question is what would be the KL divergence between $\mathcal{N}(\mu\_1, \mathbb{I}\_d)$ and $\mathcal{N}(\mu\_2, \mathbb{I}\_d)$?


Is the question even valid, and if so how to proceed.



KL divergence between gaussian with uniform prior
$\newcommand{\R}{\mathbb R}\newcommand{\KL}{{\operatorname{KL}}}$For $j=1,2$, let $P\_j:=N(\mu\_j,I\_d)$, where $\mu\_2=\mu\_1+v$ and $v$ is a unit vector. So, for the pdf's $p\_j$ of $P\_j$ we have
\begin{equation\*}
 p\_j(x)=(2\pi)^{-d/2} e^{-|x-\mu\_j|^2/2}
\end{equation\*}
for all $x\in\R^d$, where $|\cdot|$ is the Euclidean norm. Let also $\cdot$ denote the dot product.


Then the KL divergence between $P\_1$ and $P\_2$ is
\begin{equation\*}
\begin{aligned}
 \KL(P\_1,P\_2)&=\int\_{\R^d}p\_1\ln\frac{p\_1}{p\_2} \\ 
 &=\int\_{\R^d}dx\,p\_1(x)\,\tfrac12(|x-\mu\_2|^2-|x-\mu\_1|^2) \\ 
 &=\int\_{\R^d}dx\,p\_1(x)\,\big((\mu\_1-\mu\_2)\cdot x+\tfrac12(|\mu\_2|^2-|\mu\_1|^2)\big) \\ 
 &=(\mu\_1-\mu\_2)\cdot\mu\_1+\tfrac12(|\mu\_2|^2-|\mu\_1|^2) \\ 
 &=(\mu\_1-\mu\_2)\cdot\mu\_1-\tfrac12(\mu\_1-\mu\_2)\cdot(\mu\_1+\mu\_2) \\ 
 &=\tfrac12\,(\mu\_1-\mu\_2)\cdot(\mu\_1-\mu\_2)=\tfrac12\,(-v)\cdot(-v)=\tfrac12. 
 \end{aligned}
\end{equation\*}


Therefore and because $V$ is a unit random vector, the expected KL divergence between $N(\mu\_1,I\_d)$ and $N(\mu\_1+V,I\_d)$ is $\tfrac12$ as well:
\begin{equation\*}
 \mathsf E\,\KL\big(N(\mu\_1,I\_d),N(\mu\_1+V,I\_d)\big)=\tfrac12. 
\end{equation\*}
(The condition that $V$ is uniformly distributed and orthogonal to $\mu\_1$ was not needed or used here.)


