I would like to solve the following optimization problem in $k$-vector $w\_i$


$$ \min\_{w\_i} \quad \left\|P\_i - X \mbox{diag} (w\_i) Y^T \right\|\_F^2 $$


where $P\_i$ is a $6 \times 6$ matrix, $X$ and $Y$ are $6 \times k$ matrices, and $\mbox{diag}(w\_i)$ is a (square) diagonal matrix whose main diagonal is $w\_i$. How to optimize over $\mbox{diag} (w\_i)$? Does anyone know how to take derivative wrt a diagonal matrix?


Or would it work if treat $\mbox{diag} (w\_i)$ as a square matrix, solve it, and then set off-diagonal entries to zeros?



How do I optimize over (or take derivative wrt) a square diagonal matrix?
Your notation is somewhat confusing, in that you apply the subscript $i$ to $w$, and have a vector $w\_{i}$, but don't use $i$ in any meaningful way in your problem. I'm going to take the liberty of rewriting the problem as 


$\min\_{w} \| P-X \mbox{diag}(w) Y^{T} \|\_{F} $.


You may have a whole bunch of these problems to solve as $i$ varies over some index set, but each can be solved separately.


This is a linear least squares problem in disguise. 


The key to seeing this is to recognize that the Frobenius norm of a matrix $Z$ is the two norm of the vector $\mbox{vec}(Z)$ obtained from the matrix $Z$ by stacking the columns of $Z$ one on top of another. 


Also note that 


$X \mbox{diag}(w) Y^{T}=\sum\_{j=1}^{k} w\_{j} X\_{j}Y\_{j}^{T}$


where $X\_{j}$ is the $j$th column of $X$, and $Y\_{j}$ is the $j$th column of $Y$.


Now, your problem can be written as 


$\min\_{w} \| P- \sum\_{j=1}^{k} w\_{j} X\_{j}Y\_{j}^{T} \|\_{F}$.


Let $H\_{j}=X\_{j}Y\_{j}^{T}$, for $j=1, 2, \ldots, k$. We now have


$\min\_{w} \| P - \sum\_{j=1}^{k} w\_{j} H\_{j} \|\_{F}. $


Transforming this into vector form, this becomes


$\min\_{w} \| \mbox{vec}(P) - \sum\_{j=1}^{k} w\_{j} \mbox{vec}(H\_{j}) \|\_{2}$.


Let $A$ be the matrix whose columns are given by


$A\_{j}=\mbox{vec}(H\_{j})$. 


Then the optimization problem can be written as 


$\min\_{w} \| \mbox{vec}(P) - Aw \|\_{2} $.


which is a conventional linear least squares problem. 


