## Review Calculus using by Python 

Consider a sequence of n numbers $x_0, x_1, \cdots x_{n-1}$. We will start our index at 0, to remain in accordance with Python/Numpy's index system. $x_0$ is the first number in the sequence, $x_1$ is the second number in the sequence, and so forth, so $x_j$ is the general $j+1$ number in the sequence. We will utilize this $j$ index in our summation notation. Suppose we were to calculate the sum of all $n$ of these numbers in the sequence. We write that this sum is equivalent to 



Let's parse this equation. $\Sigma$ is the summation sign, indicating that we are summing a sequence. $j$ is the index of summation that is being iterated over; $j$ is being used to subscript $x$. Then, we start our summation at 0 because the lower bound on our sum is set as $j=0$. The upper bound for our sum, or our stopping point is written as $n-1$, which is equivalent to writing $j=n-1$ on top of the $\Sigma$ symbol. Then, $x_j$ indicates the quantitiy that we are summing. One way to think about this is in terms of a for-loop. This summation concept is equivalent to a for loop for all integers in the range from the lower bound to the upper bound, indexed into the sequence $x$. Then, the code for our previous sum is:


```python

```

Then, you can run through a few more examples of summation notation.

Example1:




```python

```

Example2:




```python

```

Example3:




```python

```

Then, we can also sum over several different sequences. Consider the sequences A and B, where A consists of m values $a_0, a_1, \cdots, a_{m-1}$ and B contains n values $b_0, b_1, \cdots b_{n-1}$. Then, we can calculate the sum of the product of each value in A with each value in B (a sum over $m\times n$ products). Because the $i$ index only appears in association with A, and the $j$ index with B, we can group these summations. 




```python
import numpy as np
```


```python
A = np.random.rand(5)
```


```python
B = np.random.rand(7)
```


```python

```

Note that the following does not hold 



because this effectively treats the index $i$ in the first term independently from the $i$ in the second term of the product. Notice that in the right side of the equation, we could have interchanged the index $i$ in the second summation with $j$, without changing any of the mathematics. The sum on the left represents the summation of $m$ terms of $a_{i}^2$, whereas the sum on the right represents the summation of $m \times m$ terms - products between all possible pairs of A's terms.


```python
import numpy as np
```


```python
A = np.random.rand(5)
```


```python

```

Note that typically when someone writes $\sum_{i} x_i$, this is just the sum of all values in the sequence X, and is the same as writing $\sum_{i=0}^{n-1} x_i$, where you'll usually know the value of n so you can still compute the sum.


## Matrices 


Then suppose we have a m by n matrix that contains all products of the values in both sequence A and sequence B such that the matrix value of $M$ at index $(i,j)$ is $a_i \cdot b_j$.



You can form this matrix in numpy via an "outer product": 


```python
import numpy as np
```


```python

```


```python

```


```python

```


```python

```

You can use this matrix to visualize several possible uses of summations. For example, suppose $m=n$. Then, if we express the sequences A and B as column vectors, the dot product of the two vectors would be the sum of the diagonal of M, or $\sum_{k=0}^{n-1} a_k \cdot b_k$. Furthermore, we can take the sum of a row using the sum $\sum_{i=0}^{n-1} a_k \cdot b_i$ for the $k^{th}$ row of the matrix, and similarly take the sum of a column with $\sum_{i=0}^{n-1} a_i \cdot b_k$ for the $k^{th}$ column of the matrix.


```python
k = 2
```


```python
# sum over the kth column:
```


```python
# sum over the kth row:
```

## Kronecker Delta

To make notation for working with summations (particularly in considering the matrix format) even simpler, we can use the Kronecker delta function, named after Prussian mathematician Leonard Kronecker. We use $\delta_{i,j}$ to denote the Kronecker delta function, defined as: \begin{equation} \begin{cases} \delta{i, j}= 0 & \text{if } i \neq j\
\delta{i,j}=1 & \text{if } i=j
\end{cases} \end{equation}

See that a Kronecker-delta can "collapse" a sum. Let $j$ be an integer between 0 and $n-1$: 



See also that the identity matrix, $I$, can be written as $I_{i,j}=\delta_{i,j}$.



As an exercise, write a python function that behaves as a kronecker delta, and include it in a for-loop that is computing a sum. Verify that it does indeed collapse the sum.

## Partial Derivatives

Partial derivatives are used in multivariable functions, in which we essentially derive with respect to one of these variables and treat the remaining variables as constants. For example: 

$$ f(x,y) = 6 x^2 y^3 $$ $$ \frac{\partial}{\partial x} f(x,y) = 12x y^3 $$ $$ \frac{\partial}{\partial y} f(x,y) = 18 x^2 y^2 $$

So, what if we want to take a partial derivative of a sum?

Suppose that we have two vectors containing variables:





Then, in our previous section we saw that $\vec{x} \cdot \vec{y} = \sum_{i=0}^{n-1} x_i \cdot y_i$. What happens when we take the partial derivative of this sum with respect to the variable $x_{j}$? Let $f = \vec{x} \cdot \vec{y}$, then 



It is critical to see that, because $x_i$ and $x_j$ are distinct variables (unless $i = j$), \begin{equation} \begin{cases} \frac{\partial x_i}{\partial x_j} = 0 & \text{if } i \neq j\
\frac{\partial x_i}{\partial x_j} = 1 & \text{if } i=j
\end{cases} \end{equation} and thus $\frac{\partial x_i}{\partial x_j} = \delta_{i,j}$.

Thus we can simplify our sum by collapsing it via the kronecker-delta: 



Defining $\frac{\partial f}{\partial \vec{x}} = [\frac{\partial f}{\partial x_0}, \cdots , \frac{\partial f}{\partial x_j}, \cdots, \frac{\partial f}{\partial x_{n-1}}]$, we can see from our above result that 



Take a little bit of time to think through the above expression, and make sure you understand why it is true, writing out matrices to help your understanding. This is just one example of how we can apply some of the calculus we already know to vectors, leading to vector calculus, a pillar for linear algebra! This is especially important for deriving expressions used in back-propagation in machine learning.

The end of document

cbbbc7299cec9e3770640e58361d751d4d0ad25f
ipynb
Jupyter Notebook
aip2-dgs/other/reference.ipynb
nurseiit/comm-unist
e7a122c910bf12eddf5c0ffc2c666995b4989408
MIT
2019-07-03T00:57:01.000Z
2020-12-11T23:06:11.000Z
aip2-dgs/other/reference.ipynb
nurseiit/comm-unist
e7a122c910bf12eddf5c0ffc2c666995b4989408
MIT
2019-10-19T17:42:42.000Z
2019-10-19T17:42:42.000Z
aip2-dgs/other/reference.ipynb
nurseiit/comm-unist
e7a122c910bf12eddf5c0ffc2c666995b4989408
MIT
2019-11-05T04:14:08.000Z
2019-11-05T04:14:08.000Z
Qwen/Qwen-72B
1. YES
2. YES