I'm trying to derive some weights expression for a boosting algorithm on a L2-ISE loss function, and i have trouble with the taylor expension. 


Suppose that $f$ and $g$ are two densities from $\mathbb{R}^d$ to $R$. Denote by $L$ the loss from estimating $g$ by $f$ i.e the functional : 


$$L(f) = \int \left(f(x) - g(x)\right)^2\; dx$$


I am interested in the first order taylor expension for $L((1-\epsilon) f\_1 + \epsilon f\_2) = L(f\_1 + \epsilon (f\_2 - f\_1)) $ for small $\epsilon > 0$, so i write : 


$$L(f\_1 + \epsilon (f\_2 - f\_1)) \approx L(f\_1) + \epsilon\frac{dL}{df\_1}(f\_1) (f\_2 - f\_1)$$


Then i write the derivative of the loss as : 


$$\frac{dL}{df\_1}(f\_1) = \int 2(f\_1(x) - g(x)) \; dx$$


Which clearly gives $0$ since $f\_1$ and $g$ are densities ! So the loss function is flat everywhere ? Where am i wrong ? 



Taylor expension of a simple integral
If you want to be mathematically precise, you'd have to say a bit more about what space you're doing this on and whether you're thinking of a Frechet derivative or a Gateaux derivative, etc., but for practical purposes, you can think of functions like you would of components of a vector, just labeled by a continuous index instead of a discrete one. Then you have the basic functional derivative
$$
\frac{\delta f(x) }{\delta f(y)} = \delta (x-y)
$$
i.e., a Kronecker $\delta $ is correspondingly replaced by a Dirac $\delta $. So then your Taylor expansion is
\begin{eqnarray}
L[f\_1 +\epsilon (f\_2 -f\_1)] &=& L[f\_1 ] +\epsilon \int dy \frac{\delta L[f\_1 ]}{\delta f\_1 (y) } (f\_2 (y) -f\_1 (y)) \\
&=& L[f\_1 ] + \epsilon \int dy\ 2(f\_1 (y) - g(y)) (f\_2 (y) -f\_1 (y))
\end{eqnarray}


