I have the following function for two matrices ${\bf A}$ and ${\bf B}$:


$f({\bf A}, {\bf B}) = \| {\bf Y - XAB} \|\_F^2 = trace\{({\bf Y - XAB)}^T({\bf Y - XAB)}\}$


where matrices ${\bf X}\_{n \times p}$ and ${\bf Y}\_{n \times q}$ are fixed, and matrices ${\bf A}\_{p \times r}$ and ${\bf B}\_{r \times q}$ are the variables, with $r<\min(p,q)$. I'd like to know whether this function is convex in *both* ${\bf A}$ and ${\bf B}$. (I am quite sure if either ${\bf A}$ or ${\bf B}$ is fixed, then, $f$ is convex for the other one.)


If it is not convex, can I impose some extra constraints on any of these matrices to make $f$ convex?



Convexity of the Frobenius norm of the product of two matrices
$f$ is convex separately for $A$ or $B$ but is not (in general) for the couple $(A,B)$.


Proof: The Hessian of $f$ in $X,Y$ is the following QUADRATIC form: 


$Q(H,K)=2(||XHB+XAK||^2+2trace((XAB)^TXHK))-4trace(Y^TXHK)$.


Then $Q(H,0)=2||XHB||^2$ and $f$ is convex for $A$ (and similarly for $B$) - One must say a little more if there is $H\not=0$ s.t. $XHB=0$ -


The problem is clear when, as Robert wrote above, $Y=0$. Moreover, take $H=A,K=-B$ and assume that $XAB\not=0$. Then $Q(A,-B)=-4trace((XAB)^T(XAB))<0$. Thus $Q$ is not non-negative and $f$ is not convex.


EDIT (answer to Mkl). $H=0$ iff $XHB=(X\bigotimes B^T)(H)=0$ that is $X\bigotimes B^T$ is one to one, or $rank(X\bigotimes B^T)=pr$ (necessarily $pr\leq nq$). Let $(\sigma\_i)\_{i\leq \alpha},(\tau\_j)\_{j\leq\beta}$ be the non-zero singular values of $X,B$ where $rank(X)=\alpha,rank(B)=\beta$. The non-zero singular values of $X\bigotimes B^T$ are the $(\sigma\_i\tau\_j)\_{i,j}$. Then the condition is $\alpha\beta=pr$. Note that $\beta\leq r,\alpha\leq p$. Necessarily $\beta=r,\alpha=p$ and then $p\leq n$.


Conclusion: The NSC is: $X,B$ have full rank and $p\leq n$.


