The Hessian matrix $\{\partial\_i \partial\_j f \}$ of a function $f:\mathbb{R}^n \to \mathbb{R}$ depends on the coordinate system you choose. If $x\_1,\cdots,x\_n$ and $y\_1,\cdots,y\_n$ are two sets of coordinates (say, in some open neighborhood of a manifold), then $\frac{\partial f(y(x))}{\partial x\_i} = \sum\_{k} \frac{\partial f}{\partial y\_k} \frac{\partial y\_k}{\partial x\_i}$. Differentiating again, this time with respect to $x\_j$, we get $\frac{\partial^2 f(y(x))}{\partial x\_i \partial x\_j} = \sum\_{k} \sum\_{l} \frac{\partial^2 f}{\partial y\_k \partial y\_l} \frac{\partial y\_l}{\partial x\_j} \frac{\partial y\_k}{\partial x\_i}+\frac{\partial f(y(x))}{\partial y\_k}\frac{\partial^2y}{\partial x\_i \partial x\_j}$. At a critical point, the second term goes away, so we will consider such a case.


In other words, if the derivative is a differential $1$-form, i.e. $\sum\_{i} \frac{\partial f}{\partial x\_i} dx\_i$, a section of the cotangent bundle, then the second derivative should be $\sum\_{k,l} \frac{\partial^2 f(y(x))}{\partial y\_k \partial x\_l} dy\_k \otimes dy\_l$. This makes sense since $dy\_k=\sum\_{i} \frac{\partial y\_k}{\partial x\_i} dx\_i$, and $dy\_l=\sum\_{j} \frac{\partial y\_l}{\partial x\_j} dx\_i$, meaning that $\sum\_{k,l} \frac{\partial^2 f(y(x))}{\partial y\_k \partial x\_l} dy\_k \otimes dy\_l = \sum\_{k,l} \frac{\partial^2 f(y(x))}{\partial y\_k \partial x\_l} (\sum\_{i} \frac{\partial y\_k}{\partial x\_i} dx\_i) \otimes (\sum\_{j} \frac{\partial y\_l}{\partial x\_j} dx\_j) = \sum\_{i,j,k,l} \frac{\partial^2 f(y(x))}{\partial y\_k \partial x\_l} \frac{\partial y\_k}{\partial x\_i} \frac{\partial y\_l}{\partial x\_j} dx\_i dx\_j = \sum\_{i,j} \frac{\partial^2 f}{\partial x\_i \partial x\_j}$, making it coordinate independent. Note that I did not use exterior powers, I used tensor powers, since I wanted to actually find a way to make sense of second derivatives, rather than having $d^2=0$. This means the Hessian should be a rank $2$ tensor ((2,0) or (0,2), I can't remember which, but definitely not (1,1)).


Does this make sense? Can we then express the third, etc, derivative as a tensor? More interestingly, how can this help us make sense of Taylor's formula? Can we come up with a coordinate-free Taylor series of a function at a point on a manifold?


EDIT: An in general, if the first $n$ derivatives vanish, then the $n+1$ derivative should be a rank $n+1$ tensor, right?



Hessian as a tensor, multi-dimensional taylor series, and generalizations
No, no, no! You left out a term involving $\frac{df(y(x))}{dy}\frac{d^2y}{dx^2}$. This term vanishes at critical points -- points where $df=0$ -- so that indeed at such a point the Hessian define a tensor -- a symmetric bilinear form on the tangent space at that point -- independent of coordinates. Paying attention to what kind of bilinear form it happens to be is the beginning of Morse theory, but it's only intrinsically defined as a tensor if you're at a critical point.


Notice that even the question of whether the Hessian is zero or not is dependent on coordinates. Even in a one-dimensional manifold.


Taylor polynomials don't live in tensor bundles, but in something more subtle called jet bundles.


