I am working on an optimization problem which I am stuck on towards the end. 


Essentially, I have two probability density functions in $\mathbb{R}^2$, call them $q(x,y)$ and $p(x,y)$, now I define the objective functional to be:


$C(p,q) = \int p(x,y) \ln\left[\frac{p(x,y)}{q(x,y)}\right] dxdy \cdots (1)$


Now assume we already know what $q(x,y)$ is, ie, it is pre-determined, so my aim is to minimise $(1)$ with respect to the unknown density $p(x,y)$ subject to the following constraints:


$\int\_{-\infty}^{\infty} \int\_{X^x\_d}^{\infty} p(x,y) dx dy = PoD^x$


$\int\_{-\infty}^{\infty} \int\_{X^y\_d}^{\infty} p(x,y) dy dx = PoD^y$


$\int\_{-\infty}^{\infty} \int\_{-\infty}^{\infty} p(x,y) dx dy = 1$


$p(x,y) \ge 0 \ \forall (x,y) \in \mathbb{R}^2$ 


Now if we define:


$ I\_{[X^x\_d, \infty)} = \begin{cases} 1 & \text{if} x \ge X^x\_d \\ 0 & \text{if} x < X^x\_d \end{cases}$


$ I\_{[X^y\_d, \infty)} = \begin{cases} 1 & \text{if} y \ge X^y\_d \\ 0 & \text{if} y < X^y\_d \end{cases}$


Then our constraints become:


$\int \int p(x,y) I\_{[X^x\_d, \infty)} dx dy = PoD^x \cdots (2)$


$\int \int p(x,y) I\_{[X^y\_d, \infty)} dy dx = PoD^y \cdots (3)$


$\int \int p(x,y) dx dy = 1 \cdots (4)$


So I set up the Lagrangian as follows:


$L(p,q) = \int \int p(x,y) \ln[p(x,y)] dxdy - \int \int p(x,y) \ln[q(x,y)] dxdy + \lambda\_1 \left[ \int \int p(x,y) I\_{[X^x\_d, \infty)} dx dy - PoD^x\right] + \lambda\_2 \left[\int \int p(x,y) I\_{[X^y\_d, \infty)} dydx - PoD^y \right] + \mu \left[\int \int p(x,y) dxdy -1\right]$


which can further be simplified:


$L(p,q) = \int \int p(x,y)\left[\ln[p(x,y)] - \ln[q(x,y)]\right] dxdy + \int \int p(x,y) \left[\lambda\_1 I\_{[X^x\_d, \infty)} + \lambda\_2 I\_{[X^y\_d, \infty)} + \mu \right] dxdy - \lambda\_1 PoD^x - \lambda\_2 PoD^y - \mu$


Now to solve this for $p(x,y)$ I was told to use the calculus of variations and the optimal solution would be:


$\widehat{p(x,y)} = q(x,y) \exp\{-\left[1+\hat{\mu} + \left(\hat{\lambda\_1} I\_{[X^x\_d, \infty)} \right) + \left(\hat{\lambda\_2} I\_{[X^y\_d, \infty)} \right) \right] \}$


Now I am not very familiar with calculus of variations, I only really know how to optimize functionals with respect to algebraic constraints, however I am not sure what to do when the constraints are (multiple) integrals, any help would be appreciated :)



Probability Density Optimization
Just differentiate $L(p,q)$ with respect to $p(x,y)$, assuming regularities, to get


$$\frac{\partial}{\partial p(x,y)}L(p,q)=1+\ln p(x,y)-\ln q(x,y)+\lambda\_1 I\_{[X^x\_d, \infty)} + \lambda\_2 I\_{[X^y\_d, \infty)}+\mu$$ and equate to zero (the necessary condition in Lagrange multiplier method).


Now using the constraint that $\int \int p(x,y) dx dy = 1$, you see the optimising $p$ is given by $$p(x,y) = c\cdot q(x,y) \exp{\left(-(\lambda\_1 I\_{[X^x\_d, \infty)}+\lambda\_2 I\_{[X^y\_d, \infty)})\right)}$$


where $c$ is the normalising constant.


To confirm the above is indeed the solution, (if you are exposed to measure theory) you can have a look into the nice Ann. Probab. paper (Theorem 3.1) by I.Csiszar <http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aop/1176996454>.


