I have the following functional 
$$F(y)=\left[\int \frac{1}{y(x)+A}\cos(x)dx\right]^2$$
How do I find the functional derivative $dF$?


(I never encountered the square of an integral before when I did functional derivatives)



Functional derivative of the square of an integral
Replace $y$ by $y+t\delta y$ and then compute


$$\lim\_{t\to 0} \Bigl(\; F(y+t\delta y)-F(y)\;\Bigr). $$


Here are the details. Observe first that


$$\frac{1}{y+ A+ t\delta y}-\frac{1}{y+A} = -\frac{1}{(y+A)^2} t\delta y+ O(t^2),$$


so that


$$F(y+t\delta y)= \left(\int\;\Bigl(\; \frac{1}{y+A} -\frac{1}{(y+A)^2} t\delta y+O(t^2)\;\Bigr) \cos(x) dx\right)^2 $$


$$= F(y) -2t\left(\int\frac{\cos (x)}{y+A}\right)\left(\int\frac{(\delta y)\cos(x)}{\bigl(y+A)^2} dx\right) +O(t^2). $$


From here you can read that


$$\frac{\delta F}{\delta y} =-2 \left(\int\frac{\cos (x)}{y+A}dx\right)\frac{\cos(x)}{\bigl(\;y(x)+A \;\bigr)^2} $$


**Update 1.** Here are some rules that will help you solve the problem you mentioned in your comment.


Suppose that $I$ is an interval. If


$$F(y) =\int\_I f(y(x)) w(x) dx.$$


Then


$$\frac{\delta F}{\delta y}= f'(y(x)) w(x). \tag{1} $$


If 


$$ F(y)=\left| \int\_I f(y) w(x) dx\right|^2, $$


$w$ complex valued, then $\DeclareMathOperator{\re}{\boldsymbol{Re}}$


$$\frac{\delta F}{\delta y}= 2\re\left(\; f'(y(x)) \overline{w(x)} \int\_I f(y) w(x) dx \;\right) \tag{2} $$


Let me set 


$$ F\_1(y)=\int\_I \frac{A}{y(x)+ A} dx,\;\;I=(-\kappa \pi,\kappa\pi), $$


$$ F\_2(y) =\left|\int\_I \frac{A}{y(x)+ A} e^{- i x} dx\right|^2, $$


$$ E(y) = F\_1-\frac{F\_2}{F\_1} $$


Then


$$ \frac{\delta E}{\delta y} =\frac{\delta F\_1}{\delta y} -\frac{ \frac{\delta F\_2}{\delta y} F\_1- F\_2\frac{\delta F\_1}{\delta y} }{ F\_1(y)^2 }. $$


Now compute the various derivatives using (1) and (2). 


Your problem reduces via Lagrange multipliers to the system


$$\frac{\delta E}{\delta y} =\lambda,\;\;\int\_I y dx=1. $$


**Update 2.** Introduce a new variable


$$ y=\frac{A}{G+A},\;\;G= \frac{A}{y}-A. $$


The constraint 


$$ \int G = const $$ 


becomes 


$$C(y)= \int\frac{1}{y} =const $$


and the energy functional becomes


$$E(y)=\int y -\frac{\left|\int ye^{-i x}\right|^2}{\int y }. $$


The constrained Euler-Lagrange equations have the form


$$\frac{\delta E}{\delta y}=\lambda\frac{\delta C}{\delta y}=-\frac{\lambda}{y^2} \tag{3} $$


Which translates into an equality of the form


$$C\_1(y) y^2+ C\_2(y) \cos x +C\_3(y) \sin x= C\_4(y)-\lambda C\_1(y)^2, $$


where $C\_i(y)$ are constants that depend explicitly on $y$, e.g.,


$$C\_1(y)=\left(\int y\right)^2. $$


Now at least you know that $y$ must be of a rather special form


$$ y =\sqrt{ A\_1+A\_2\cos x+ A\_3\sin x}. $$ If you play with (3) some more I bet that you can extract more precise info.


