There are a few standard notions of matrix derivatives, e.g.


* If *f* is a function defined on the entries of a matrix *A*, then one can talk about the matrix of partial derivatives of *f*.
* If the entries of a matrix are all functions of a scalar *x*, then it makes sense to talk about the derivative of the matrix as the matrix of derivatives of the entries.


In the second case, it makes sense to talk about higher order derivatives, but in the first example the derivative provides a matrix from a scalar function, so you have to massage it a little bit to define a higher order derivative (e.g. take the trace of the resulting matrix).


I was wondering what other notions of matrix differentiation might exist out there, particularly any notions that allow for higher-order differentiation. I am also interested in any connections between various forms of matrix derivatives. As this is related to an undergraduate research project, I am mostly looking for answers that include a minimum of advanced terminology, but a discussion of how more general concepts (e.g. differential forms, matrix exponentials, etc.) relate to matrix derivatives would also be helpful.



Notions of Matrix Differentiation
There is another interpretation of Elisha's question that I think has not yet been addressed: How, and to what extent, can you do differential calculus with functional expressions of square matrices? For instance, how do you differentiate $\exp(A)$, which is defined for all square matrices $A$?


There is a good answer to this question for polynomials and an even better answer for the trace of a univariate polynomial. Both of these good answers extend automatically to analytic functions $f(z)$ evaluated at matrices by means of their Taylor series. (Which of course includes exponentiation.) I like to write the answer in terms of differentials. The differential of matrices $A$, $B$, etc., is $dA$, $dB$, etc., which you can take to mean that the matrices are unspecified functions of some dummy parameter $t$, and $dA$ then represents the formal numerator of the matrix-valued derivative $dA/dt$.


In this formalism, the Leibniz rule still holds, as long as you remember that matrix multiplication is non-commutative: $d(AB) = (dA)B + A(dB)$. Indeed, the matrices don't actually need to be square. The sum rule holds trivially. The negative power rule has a creative correct answer: $d(A^{-1}) = A^{-1}(dA)A^{-1}$. You can then differentiate exponentiation:
$$d\exp(A) = dA + \frac{(dA)A + A(dA)}2 + \frac{(dA)A^2 + A(dA)A+A^2(dA)}6 + \cdots.$$
As you can see from this example, you can differentiate a power series, but nothing all that great happens because $dA$ might not commute with $A$. However, if you're computing the differential of $\mathrm{Tr}(f(A))$, then something very nice happens: You can cyclically permute each term of the differentiated power series to put $dA$ at the end. Thus, you get the very nice formula
$$d\mathrm{Tr}(f(A)) = \mathrm{Tr}(f'(A)dA).$$
I have only derived this formula when $A$ is within the radius of convergence of the Taylor series of $f$. However, by continuity it applies generally when $f(A)$ and $f'(A)$ are well-defined. (For instance, if $A$ and $dA$ are both real symmetric or Hermitian, then it is enough for $f$ to exist as a real function and have one continuous derivative near the eigenvalues of $A$.)


Higher derivatives work basically the same way as first derivatives. Again, you should take $A$ to be a function of a dummy parameter $t$. To get the simplest expressions for higher derivatives, you should assume that $A$ is a linear function of $t$ (including a constant). Then for example:
$$d^2\exp(A) = (dA)^2 + \frac{(dA)^2A + (dA)A(dA) + A(dA)^2}3 + \cdots.$$ 
The trace of this looks good at first, but the quadratic term in $A$ has both $\mathrm{Tr}((dA)^2A^2)$ and $\mathrm{Tr}((A(dA))^2)$, and I don't think that the rest of the traced Taylor series simplifies the way that it did for the first derivative.


A final remark: All of this works the best for functions $f(x)$ that are entire (in the sense of complex analysis, i.e., an infinite radius of convergence). One of the definitions of an entire function is one whose Taylor series decays superexponentially, and this is also a good condition for a non-commutative multivariate Taylor series, in the sense that it will converge for any matrices that you plug in.


