A matrix, $\mathbf{A}(\theta)\in\mathbb{R}^{n\times n}$, has elements that depend on a parameter $\theta$. The $j$-th eigenvalues and eigenvectors of the matrix are denoted as $\lambda\_j$ and $\mathbf{x}\_j$, respectively.
I would like to know the requirements to obtain finite backward derivatives of eigenvectors in degenerate case.


Let me elaborate.
The forward derivative of the eigendecomposition is given by
$$
\delta\mathbf{x}\_j = -\sum\_{i\neq j} (\lambda\_i - \lambda\_j)^{-1} \mathbf{x}\_i \left[\mathbf{x}\_i^T (\delta \mathbf{A}) \mathbf{x}\_j\right],
$$
where $\delta\mathbf{A}$ and $\delta \mathbf{x}\_j$ are small changes of each element in $\mathbf{A}$ and $\mathbf{x}\_j$, respectively.


If there is a degeneracy, $\lambda\_d = \lambda\_j\ \forall\ d\in\mathrm{degen}(j)$, where $\mathrm{degen}(j)$ is the set of degenerate indices with the same eigenvalues of $\lambda\_j$, **excluding** $j$.
The requirement to get finite $\delta \mathbf{x}\_j$ is
$$
\mathbf{x}\_d^T(\delta\mathbf{A})\mathbf{x}\_j = 0\ \forall\ d\in \mathrm{degen}(j).
$$


The backward derivative, on the other hand, is given by
$$
\frac{\partial \mathcal{L}}{\partial \mathbf{A}} = -\sum\_j\sum\_{i\neq j}(\lambda\_i - \lambda\_j)^{-1}\mathbf{x}\_i \mathbf{x}\_i^T \frac{\partial \mathcal{L}}{\partial \mathbf{x}\_j} \mathbf{x}\_j^T,
$$
where $\mathcal{L}$ is a loss value, $\frac{\partial \mathcal{L}}{\partial \mathbf{x}\_j}\in\mathbb{R}^{n\times 1}$ and $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}\in\mathbb{R}^{n\times n}$ are the change in the loss value w.r.t. each element in $\mathbf{x}\_j$ and $\mathbf{A}$, respectively.


In case of degeneracy, $\mathrm{degen}(j)\neq \emptyset$, is it possible to get finite $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$? If so, what are the requirements?
If we don't care about $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$, can we relax the requirements just to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$ ($\theta$ is the parameter the elements of $\mathbf{A}$ depends on)?



Requirements for finite backward derivatives of degenerate eigenvectors
Let's take $m \in \mathrm{degen}(n)$. The terms involving $m$ and $n$ in $\frac{\partial \mathcal{L}}{\partial \mathbf{A}}$ are
$$
\frac{\partial\mathcal{L}}{\partial \mathbf{A}} =-(\lambda\_m -\lambda\_n)^{-1}\left[\mathbf{x}\_m\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\mathbf{x}\_n^T - \mathbf{x}\_n\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\mathbf{x}\_m^T\right] + \ ...
$$
Therefore, to get finite $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, one needs
$$
\mathbf{x}\_m\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\mathbf{x}\_n^T = \mathbf{x}\_n\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\mathbf{x}\_m^T\ \forall\ m\in\mathrm{degen}(n)
$$


If we don't care about $\frac{\partial\mathcal{L}}{\partial \mathbf{A}}$, only $\frac{\partial\mathcal{L}}{\partial \theta}$, we should write the expression for $\frac{\partial\mathcal{L}}{\partial \theta}$ first,


$$
\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[\left(\frac{\partial\mathcal{L}}{\partial \mathbf{A}}\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right]
$$


The term in $\frac{\partial\mathcal{L}}{\partial \theta}$ involving $m$ and $n$ are
$$
\frac{\partial\mathcal{L}}{\partial \theta} = \mathrm{tr}\left[(\lambda\_m-\lambda\_n)^{-1}\left(\mathbf{x}\_m\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\mathbf{x}\_n^T - \mathbf{x}\_n\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\mathbf{x}\_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] + \ ...
$$
Therefore, to get finite $\frac{\partial \mathcal{L}}{\partial \theta}$, the following condition must be satisfied
$$
\begin{align}
\mathrm{tr}\left[\left(\mathbf{x}\_m\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\mathbf{x}\_n^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] &= 
\mathrm{tr}\left[\left(\mathbf{x}\_n\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\mathbf{x}\_m^T\right)^T\frac{\partial \mathbf{A}}{\partial \theta}\right] \\
\mathrm{tr}\left[\left(\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\right)\mathbf{x}\_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}\_n\right] &= \mathrm{tr}\left[\left(\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\right)\mathbf{x}\_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}\_m\right] \\
\left(\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n}\right)\mathbf{x}\_m^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}\_n &= \left(\mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}\right)\mathbf{x}\_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}\_m
\end{align}
$$


If the matrix $\mathbf{A}$ is always symmetric for all values of $\theta$, i.e. $\frac{\partial\mathbf{A}}{\partial \theta}$ is always symmetric, then there are two conditions that can satisfy the condition above:


$$
\begin{align}
\mathbf{x}\_n^T\frac{\partial \mathbf{A}}{\partial \theta}\mathbf{x}\_m &= 0\\
& \mathrm{or} \\
\mathbf{x}\_m^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_n} &= \mathbf{x}\_n^T\frac{\partial\mathcal{L}}{\partial \mathbf{x}\_m}
\end{align}
$$


