I have a (given) real-valued function defined over an area, $b(x)$, with $x\in \Omega \subset \mathbb{R}^2$.
I would like to find a smooth real-valued function $a(x)$ that maximises
$J = \int\_{x \in \Omega} a^3(x) + a(x)b(x)~dx$ with the constraint $\int\_{\Omega} a(x) ~dx=0$. If it helps prevent unbounded or irregular solutions I am happy to introduce further constraints or a higher-order penalty, eg,
$J = \int\_{x\in\Omega} a^3(x) + a(x)b(x) - \epsilon a^4(x)~dx$ (with the same constraint).
How would you solve this? Do I need to specify boundary conditions on $a$?



extremal problem (calculus of variations?)
If we choose as a domain of the functional e.g. the space $L^3(\Omega)$ (with the constraint), and we assume $b\in L^{3/2}(\Omega),$ then $J$ is smooth, and if $\sup b(x)<\infty$ it has a bunch of critical points, all of the form described in the first answer, that is, all the $a(x)$ satisfying $3a(x)^2+b(x)=c$, where the constant $c\geq\sup b$ is arbitrary while the sign of $a(x)$ only has to satisfy the constraint condition $\int\_\Omega a(x)\ dx=0$. The second variation of $J$ at a critical point $a(x)$ is $D^2J(a)[h]^2= 6\int\_\Omega a(x)h^2(x)\ dx$. Thus, any critical point is neither a local minimizer nor a local maximizer (unless $a=0$, that only happens if $b$ is a constant and the problem is 1 dimensional). Moreover, $J$ is unbounded (choose a function $a(x)$ with $\int\_\Omega a(x)dx=0$ and 
$\int\_\Omega a^3(x)dx\neq 0$, then for real $t$, $J(ta)$ is a third degree polynomial in $t$, thus unbounded both from above and from below). 
If you take the perturbation with $\varepsilon>0$ the situation becomes more delicate . The natural domain is $L^4(\Omega)$ (with the constraint); assuming $b\in L^{4/3}(\Omega)$, $J$ is again smooth, unbounded from below, but now it is bounded from above; again it has a bunch of critical points, characterized by $3a(x)^2-4\varepsilon a^3(x)+b(x)=c$. There are no local minimizers (unless $b$ is constant) for the second variation 
$6\int\_\Omega \left( a(x)-4\varepsilon a^2(x)\right) h^2(x)\ dx$ is not negative. It's not immediate to decide whether there are local or global maxima; certainly they can't be continuous since they have both to change sign and to verify $a(x)\leq 4\varepsilon a^2(x)$.


