```python
import numpy as np
import matplotlib.pyplot as plt
import torch
```

$\textbf{Definitions:}$ $\\$

$\mbox{---Gradient---}$ $\\$ $\\$
Vector formed by partial derivatives of scalar function, f(x) in which $x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\x_n \end{bmatrix}$ $\\$
Gradient maps $\mathbb{R}^n \rightarrow \mathbb{R}$ $\\$

$$\nabla f(\mathbf{x})=\frac{\partial f(\mathbf{x})}{\partial x_1}\hat{x}_1+\frac{\partial f(\mathbf{x})}{\partial x_2}\hat{x}_2+\ldots+\frac{\partial f(\mathbf{x})}{\partial x_n}\hat{x}_n$$

$$\nabla f(x) = \left[\frac{\partial f}{\partial x_1}\frac{\partial f}{\partial x_2}\dots\frac{\partial f}{\partial x_n}\right]$$

Note: Input is a column vector, outputs a row vector. $\\$
Gradient is the rate of change wrt each dimension/component and corresponds to steepest slope due to linear independence used for gradient descent. $\\$

$\mbox{---Jacobian---}$ $\\$
Matrix formed by partial derivatives of vector function of scalar functions, maps $\mathbb{R}^n \rightarrow \mathbb{R}^m$ $\\$

$$J_\mathbf{f} = \frac{\partial (f_1,\ldots,f_m)}{\partial(x_1,\ldots,x_n)} = \left[
\begin{matrix}
\frac{\partial f_1}{\partial x_1} &amp; \cdots &amp; \frac{\partial f_1}{\partial x_n} \\
\vdots &amp; \ddots &amp; \vdots \\
\frac{\partial f_m}{\partial x_1} &amp; \cdots &amp; \frac{\partial f_m}{\partial x_n}
\end{matrix}
\right]$$
The Jacobian is the gradient applied to multiple rows, commonly used as a change of basis/unit conversion:

$$\iiint_R f(x,y,z) \,dx\,dy\,dz = \iiint_S f(x(u,v,w),y(u,v,w),z(u,v,w))\left|\frac{\partial (x,y,z)}{\partial(u,v,w)}\right|\,du\,dv\,dw$$
Note: Gradient = Jacobian if $m = 1$.

$\mbox{---Hessian---}$ $\\$
Gradient applied to Gradient, Double Gradient: $\\$
$$\begin{align}D[\nabla f(\mathbf x)] &amp;= D[D[f(\mathbf x)]]\\
&amp;=\left(D\left[\frac{\partial f}{\partial x_1}\right]^T, \ldots, D\left[\frac{\partial f}{\partial x_n}\right]^T\right)\end{align}$$
Which expands to give us the Hessian matrix:
$$D^2[f(\mathbf x)]=\left(\begin{matrix}\frac{\partial^2 f}{\partial x_1^2} &amp; \ldots &amp; \frac{\partial^2 f}{\partial x_1\partial x_n}\\
\vdots &amp; \ddots &amp; \vdots \\
\frac{\partial^2 f}{\partial x_n\partial x_1}&amp; \ldots &amp; \frac{\partial^2 f}{\partial x_n^2}\end{matrix}\right)$$

Note: Inputs are column vectors, outputs are row vectors. (Transposed first because the first gradient outputs a row vector) $\\$
The Hessian represents the rate of change of gradient, analogous to curvature. Used to computationally determine the position of a min/max point in optimization, which is darn impossible to visualize past 2 dimensions.

$\textbf{Analytic Gradient:}$ $\\$
$\mbox{---Linear Form---}$ $\\$
$f(x) = a^T x$ $\\$
Component-wise derivative yields corresponding dot-product coefficent of each component k. $\\$
Assemble each partial derivatives into vector: $\\$
$\nabla f(x) = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} = a$

-General Linear Form: $\\$
$f(x) = a^Tx + b$ $\\$
$\nabla f(x) = a$


$\mbox{---Quadratic Form---}$ $\\$
$f(x) = x^T A x$ $\\$
Tracing through 2x2 example: $\\$
$\nabla f(x) = (A + A^T)x$ $\\$
For pd matrices, $A = A^T$ so: $\\$
$\nabla f(x) = 2Ax$

-General Quadratic Form, which builds from gradient of general linear form: $\\$
$f(x) = \frac{1}{2}x^T A x + b^Tx + c$ $\\$
$\nabla f(x) = \frac{1}{2}(A^T + A)x + b$ $\\$
For symmteric matrix A: $\\$
$\nabla f(x) = Ax + b$

-Mixed Quadratic Form: $\\$
$f(x,y) = x^T A y$ $\\$
Wrt x: $\nabla_x f(x,y) = Ay$ $\\$
Wrt y: $\nabla_y f(x,y) = A^Tx$ $\\$
Taking the right partial derivative, transpose. $\\$
Matrices are pd, so wrt y: $\nabla_y f(x,y) = Ax$

$\textbf{Analytic Hessian:}$ $\\$
Tracing through 2x2 example again: $\\$
$\mbox{---Linear Form---}$ $\\$
$f(x) = a^T x$ $\\$
$\nabla f(x)$ does not depend on x, so $\nabla^2 f(x) = 0$.

$\mbox{---Quadratic Form---}$ $\\$
$f(x) = x^T A x$ $\\$
$\nabla^2 f(x) = A + A^T$ $\\$
For symmteric matrix A: $\\$
$\nabla^2 f(x) = 2A$

Mixed Quadratic Form: $\\$
$f(x,y) = x^T A y$ $\\$
Wrt xx, yy: $H_{xx} = H_{yy} = 0$ $\\$
Wrt xy, yx: $H_{xy} = H_{yx} = 2A$


Simultaneous gradient descent (continuous time):
    $\dot x = -D_1f_1(x,y),\ \dot y = -D_2f_2(x,y)$, simgrad Jacobian
    $J(x,y) = \begin{bmatrix} D_1^2f_1(x,y) & D_{12}f_1(x,y) \\ D_{21}f_2(x,y) & D_2^2f_2(x,y) \end{bmatrix}$
    
(discrete time): 
$x^+ = x - \gamma_x D_1f_1(x,y),\ 
y^+ = y - \gamma_y D_2f_2(x,y)$


```python

```


```python
m = 2
n = 2

#Random pd matrices, Cholesky form:
np.random.seed(0)
A1 = np.random.randn(n,n)
A1 = A1.T @ A1

A2 = np.random.randn(n,n)
A2 = A2.T @ A2

#Random matricies, not pd:
B1 = np.random.randn(n,m)
B2 = np.random.randn(n,m)
C1 = np.random.randn(m,m)
C2 = np.random.randn(m,m)

#Define e,h vectors:
e1 = np.random.randn(n)
e2 = np.random.randn(m)
h1 = np.random.randn(m)
h2 = np.random.randn(n)

#Convert Matrices into tensors
A1 = torch.tensor(A1, dtype = torch.float)
A2 = torch.tensor(A2, dtype = torch.float)
B1 = torch.tensor(B1, dtype = torch.float)
B2 = torch.tensor(B2, dtype = torch.float)
C1 = torch.tensor(C1, dtype = torch.float)
C2 = torch.tensor(C2, dtype = torch.float)
e1 = torch.tensor(e1, dtype = torch.float)
e2 = torch.tensor(e2, dtype = torch.float)
h1 = torch.tensor(h1, dtype = torch.float)
h2 = torch.tensor(h2, dtype = torch.float)

x1 = torch.ones((n, 1), requires_grad=True)
x2 = torch.ones((m, 1), requires_grad=True)

#Generic Quadratic Cost:
#B_ij, C_ij still rather vague
def f1(x1,x2):
    return (0.5 * x1.t() @ A1 @ x1) + (x1.t() @ B1 @ x2) + (0.5 * x2.t() @ C1 @ x2) + (e1.t() @ x1) + (h1.t() @ x2)

def f2(x1,x2):
    return (0.5 * x2.t() @ A2 @ x2) + (x2.t() @ B2 @ x1) + (0.5 * x1.t() @ C2 @ x1) + (e2.t() @ x2) + (h2.t() @ x1)
```


```python
#Analytical Gradient:
#D wrt x1:
def D1f1(x1,x2):
    return (A @ x1) + 0.5 * (B1 @ x2) + e1

def D1f2(x1,x2):
    return 0.5 * (B2.t() @ x2) + (C2 @ x1) + h2

#D wrt x2:
def D2f1(x1,x2):
    return 0.5 * (B1.t() @ x1) + (C1 @ x2) + h1

def D2f2(x1,x2):
    return (A2 @ x2) + 0.5 * (B2 @ x1) + e2

#Analytical Hessian:
#H wrt x1:
def H11f1(x1, x2):
    return A1

def H11f2(x1, x2):
    return C2

#H wrt x2:
def H22f1(x1, x2):
    return C1

def H22f2(x1, x2):
    return A2

```


```python
#Computational Gradient:
#tensors = [tensor.zero_grad() for tensor in tensors]

'''
-Possible solutions:
-Make functions just expressions
-use backward
-Seeing an example would be nice
'''
print(f1(x1,x2).grad)


#Computational Hessian:
#print(torch.autograd(x1).autograd(x1))
#print(torch.autograd(x2).autograd(x2))
```

    None


    /projects/ac66b0d1-d01e-42cc-9d9b-3799ed5ac4d1/.local/lib/python3.6/site-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
      warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "



```python

```

cbc162fb4c70f94f6a95f3db2635b7c9805eed2a
ipynb
Jupyter Notebook
Derivatives/Derivative Tests.ipynb
JWongDude/FruitLoops
f4346d9db16ba619d71ce5bb819f5da08a88a120
MIT
Derivatives/Derivative Tests.ipynb
JWongDude/FruitLoops
f4346d9db16ba619d71ce5bb819f5da08a88a120
MIT
2020-07-09T19:52:36.000Z
2020-07-09T19:52:36.000Z
Derivatives/Derivative Tests.ipynb
JWongDude/FruitLoops
f4346d9db16ba619d71ce5bb819f5da08a88a120
MIT
Qwen/Qwen-72B
1. YES
2. YES