#  Review of Basics of Linear Algebra
---
**Agenda**
>1. Matrix Vector Operations using NumPy
>1. Vector Spaces and Matrices: Four fundamental fubspaces
>1. Motivating Examples: Image and text manipulations
>1. Eigen-decomposition, determinant and trace
>1. Special Matrices: Orthogonal Matrices
>1. Norms
>


```python
# Following lines are for Python 2.x to 3.xx compatibility
from __future__ import print_function
from __future__ import division 
```


```python
#IMPORT
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline

## Set a seed for the random number generator
np.random.seed(100)
```

## NumPy Arrays: Vectors, Matrices and Tensors

### Creating some special vectors and matrices
***
>-  Fixed ones: having a set of given elements
>-  Random ones
>-  Reshaping vectors to get matrices
>-  Zero matrix
>-  One matrix
>-  Identity matrix
>-  Permutation matrix


```python
## Create a vectors of  length 4
v1 =  np.array([1, 2, 3, 4])
v2 = np.array([3, 2,1,-1])

print("v1:", v1)
print ("v2:", v2)
```

    v1: [1 2 3 4]
    v2: [ 3  2  1 -1]



```python
# Create a random vector of Integers
v3 = np.random.randint(0, high=15, size=(4,))
print ("v3: ",v3)
```

    v3:  [8 8 3 7]



```python
# Create a random COLUMN vector of Integers
v31 = np.random.randint(0, high=8, size=(4,1))
print ("v31: \n",v31)
```

    v31: 
     [[7]
     [7]
     [0]
     [2]]



```python
v4 = np.arange(5,9)
print ("v4: ",v4)
```

    v4:  [5 6 7 8]



```python
## Create a matrix of order 4x3
A = np.array([[1, 2, 3],
              [2, 1, 4],
              [2, 4, 7],
              [1, 2, 3]], dtype=float)

## Following is no longer recommeded
matA = np.matrix([[1, 2, 3],
              [2, 1, 4],
              [2, 4, 7],
              [1, 2, 3]])

matA2 = np.matrix("1, 2; 3, 4")

## Create a random matrix of order 4x3 whose elements are chosen uniformly randomly 

## CHANGES NEEDED WITH Python 3.x for compatibilty

B = np.random.rand(4,3)

## Create a matrix of order 4x3 made of all zeros 
zero_43 = np.zeros((4,3), dtype=float)

## Create a matrix of all ones of order 3x5
ones_35 = np.ones((3,5), dtype=float)

## Create an identity matrix of order 4x4
# eye_4 = np.identity(4)
eye_4 = np.eye(4) # This is more general. See the documentation.
```


```python
print("The random matrix B is:\n", B)

print ("\n The identity matrix of order 4: ",eye_4)
```

    The random matrix B is:
     [[ 0.81168315  0.17194101  0.81622475]
     [ 0.27407375  0.43170418  0.94002982]
     [ 0.81764938  0.33611195  0.17541045]
     [ 0.37283205  0.00568851  0.25242635]]
    
     The identity matrix of order 4:  [[ 1.  0.  0.  0.]
     [ 0.  1.  0.  0.]
     [ 0.  0.  1.  0.]
     [ 0.  0.  0.  1.]]



```python
## Create a vector of order 12 (such as [3,5,7,...])and 
## Rearrange its elements to create a matrix of order 4-by-3.
v12 = np.arange(3,26,2)


A43 = v12.reshape(4,3) # or v12.reshape(4,-1)
print( "An array v12=",v12, "\n Reshaped into a matrix:\n", A43)
```

    An array v12= [ 3  5  7  9 11 13 15 17 19 21 23 25] 
     Reshaped into a matrix:
     [[ 3  5  7]
     [ 9 11 13]
     [15 17 19]
     [21 23 25]]



```python
## Create a vector of order 12 (such as [3,5,7,...])and 
## Rearrange its elements to create a matrix of order 4-by-3.
v12 = np.arange(3,42,2)


A43 = v12.reshape(4,-1) # or v12.reshape(4,-1)
print( "An array v12=",v12, "\n Reshaped into a matrix:\n", A43)
```

    An array v12= [ 3  5  7  9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41] 
     Reshaped into a matrix:
     [[ 3  5  7  9 11]
     [13 15 17 19 21]
     [23 25 27 29 31]
     [33 35 37 39 41]]



```python
v12.shape
```




    (20,)




```python

```


```python

```


```python

```

### Basic array operations
---
Revise the following operation on matrices. Study the properties.
>- Transpose of a matrix
>- Addition of two matrices
>- Elementwise product of two matrices and other elementwise operations
>- Multiplication of two matrices (dot product)
>- Finding Submatrices
>- Broadcasting in NumPy


```python
## Add the two vectors above
v3 = v1 + v2
## np.add(v1,v2)

## Multiply the two vectors (element-wise)
v4=v1*v2

#Dot product
dotp = np.dot(v1,v2)


print("The sum of the vectors", v1,"+", v2, "=",v3)

print("The elementwise product of the vectors is:", v4)

print("The dot product of the vectors is A NUMBER: ", dotp)

```

    The sum of the vectors [1 2 3 4] + [ 3  2  1 -1] = [4 4 4 3]
    The elementwise product of the vectors is: [ 3  4  3 -4]
    The dot product of the vectors is A NUMBER:  6



```python
## Adding two matrices
A_plus_B = A + B  #np.add(A,B)
print ("A:\n", A)
print ("\n B:\n", B)
print ("\n The sum is: \n", A_plus_B)

## Can you multiply the two matrices, A and B? How?
multAB = A*B
print ("\n The element wise product is: \n", multAB)

```

    A:
     [[ 1.  2.  3.]
     [ 2.  1.  4.]
     [ 2.  4.  7.]
     [ 1.  2.  3.]]
    
     B:
     [[ 0.81168315  0.17194101  0.81622475]
     [ 0.27407375  0.43170418  0.94002982]
     [ 0.81764938  0.33611195  0.17541045]
     [ 0.37283205  0.00568851  0.25242635]]
    
     The sum is: 
     [[ 1.81168315  2.17194101  3.81622475]
     [ 2.27407375  1.43170418  4.94002982]
     [ 2.81764938  4.33611195  7.17541045]
     [ 1.37283205  2.00568851  3.25242635]]
    
     The element wise product is: 
     [[ 0.81168315  0.34388203  2.44867425]
     [ 0.54814749  0.43170418  3.76011928]
     [ 1.63529876  1.3444478   1.22787318]
     [ 0.37283205  0.01137701  0.75727906]]



```python
print ('A: \n',A)


print ('Transpose of A, A.T \n ', A.T)

# Let us create a matrix 
C = np.random.randint(0, high=2, size=(3,5))
print ("\n Random matrix C:\n",C)


## MULTIPLICATION (4,3) and (3,5): Find the transpose of B and Multiply to A in appropriate order
AtimesC = np.dot(A, C)

print ('\n Product of A and C in the sense of dot product : \n', AtimesC)

```

    A: 
     [[ 1.  2.  3.]
     [ 2.  1.  4.]
     [ 2.  4.  7.]
     [ 1.  2.  3.]]
    Transpose of A, A.T 
      [[ 1.  2.  2.  1.]
     [ 2.  1.  4.  2.]
     [ 3.  4.  7.  3.]]
    
     Random matrix C:
     [[0 1 1 1 0]
     [0 0 0 0 0]
     [1 0 1 0 1]]
    
     Product of A and C in the sense of dot product : 
     [[ 3.  1.  4.  1.  3.]
     [ 4.  2.  6.  2.  4.]
     [ 7.  2.  9.  2.  7.]
     [ 3.  1.  4.  1.  3.]]



```python
# This has been depricated. No longer advised to be used.
mat1 = np.matrix('1 0; 0 2')
mat2 = np.matrix('1 2 3; -1 0 1')

print ("Product of two matrices of data type matrix:\n",mat1*mat2) # no need of np.dot(mat1, mat2)

```

    Product of two matrices of data type matrix:
     [[ 1  2  3]
     [-2  0  2]]


### Submatrices by slicing


```python
# Random Example Matrix
S = np.random.rand(4,4)

print("S:\n",S)

print ("S[:,[1,3]]")
print (S[:,[1,3]])

print ("S[[1,3],:]")
print (S[[1,3],:])

E = S[:,[1,3]][0:2] # Submatrix by slicing

print ('A submatrix is E: ----------------------------')
print (E)



print ("S[:,[1,2]][0:2]")
print (S[:,[1,2]][0:2])
```

    S:
     [[0.00607005 0.82080849 0.26320965 0.50106643]
     [0.71096644 0.02219788 0.53582866 0.24812327]
     [0.55302567 0.81867102 0.04103166 0.56728301]
     [0.2774666  0.75213614 0.62362505 0.81281704]]
    S[:,[1,3]]
    [[0.82080849 0.50106643]
     [0.02219788 0.24812327]
     [0.81867102 0.56728301]
     [0.75213614 0.81281704]]
    S[[1,3],:]
    [[0.71096644 0.02219788 0.53582866 0.24812327]
     [0.2774666  0.75213614 0.62362505 0.81281704]]
    A submatrix is E: ----------------------------
    [[0.82080849 0.50106643]
     [0.02219788 0.24812327]]
    S[:,[1,2]][0:2]
    [[0.82080849 0.26320965]
     [0.02219788 0.53582866]]


### Broadcasting
<hr>
Can you add the following?
$$
A = \left( \begin{array}{ccc} 1 & 1 & 2 \\
1 & -1 & 3
\end{array}
\right) +
\left( \begin{array}{cc} 5 \\
-5
\end{array}
\right)
$$

OR

$$
B = \left( \begin{array}{ccc} 1 & 1 & 2 \\
1 & -1 & 3
\end{array}
\right) +
\left( \begin{array}{ccc} 1 & 1 &1\\
\end{array}
\right)
$$


```python
A = np.array([[1, 1, 2],[1,-1,3]]) + np.array([[5],[-5]])
B = np.array([[1, 1, 2],[1,-1,3]]) + np.array([1,1,1])
print("A is\n",A)
print("B is\n",B)
```

    A is
     [[ 6  6  7]
     [-4 -6 -2]]
    B is
     [[2 2 3]
     [2 0 4]]


## Vector Spaces

### Independence, Orthogonality and Subspaces
---
<h5 style="margin-bottom:10px"> What does it mean for a set of vectors $\{ v_1, v_2, \cdots, v_r\}$ to be linearly independent?</h5>

<div class='eqnbox2'>
$$ \alpha_1 v_1 + \alpha_2 v_2 + \cdots + \alpha_r v_r \implies \alpha_i = 0,\  i=1,\cdots,r.$$
</div>
In other words these vectors can not be combined to form a zero vector, or, one of these vectors can not be given as a linear combination of the remaining.<br>

<h5 style="margin-bottom:10px"> What conditions two vectors that are perpendicular (orthogonal) to each other satisfy?</h5>
<div class='eqnbox2'>
$$ u \perp v \iff \langle u, v\rangle = v^Tu = 0$$
</div>


<h5 style="margin-bottom:10px"> Provide the conditions so that two subspaces of a vector space are mutually orthogonal.</h5>

<div class='eqnbox2'>
$$ U \perp V \iff \langle u, v\rangle = v^Tu = 0,\ \forall (u \in U, v \in V).$$
</div>

---

## The Four Fundamental Subspaces
---

The matrix $A \in \mathbb{R}^{m \times n}$ represents a linear  transformation,  $T_A(x): \mathbb{R}^n \to \mathbb{R}^m$ given by $T_A(x) = A x$.

 We can define the following four space with respect to matrix $A$.

  >1. Row space of $A$ is a subspace of $\mathbb{R}^n$: 
    $$ \mathcal{R}(A) =\mathcal{C}(A^T) = \left\{ x \in \mathbb{R}^n : x = A^T y \textrm{ for some }y \in \mathbb{R}^n \right\}.$$
  >1. Null space of $A$ is a subspace of $\mathbb{R}^n$: 
    $$ \mathcal{N}(A) = \left\{ x \in \mathbb{R}^n : Ax = 0 \right\}.$$
 >1. Column space of $A$ is a subspace of $\mathbb{R}^m$: 
    $$ \mathcal{C}(A) = \left\{ y \in \mathbb{R}^m : y = Ax \textrm{ for some }x \in \mathbb{R}^n \right\}.$$
 >1. Left null space of $A$, or null space of $A^T$, is a subspace of $\mathbb{R}^m$: 
    $$ \mathcal{N}(A^T) = \left\{ y \in \mathbb{R}^m : A^T y = 0 \right\}.$$

<br>
 
<br>
The above four subspaces satisfy the following properties:

 >1. $ \mathcal{R}(A) \perp \mathcal{N}(A) $, or $ \mathcal{C}(A^T) \perp \mathcal{N}(A) $.
>1. $ \mathcal{C}(A) \perp \mathcal{N}(A^T) $.
>1. $ \mathcal{R}(A) \oplus \mathcal{N}(A) =\mathbb{R}^n $.
>1. $ \mathcal{C}(A) \oplus \mathcal{N}(A^T) =\mathbb{R}^m $. 
      
Here $\perp $ stands for perpendicular, and  $\oplus$ means that every element $x \in \mathbb{R}^n$ could be written  as $x = x_n+x_r$, where $x_n \in \mathcal{N}(A)$ and $x_r \in \mathcal{R}(A)$.
      
      
 If rank$(A)$ = r, then    
 
 >- dim $\mathcal{R}(A) = $ dim $\mathcal{C}(A) = r$,
 
 >- dim $\mathcal{N}(A)=n-r$, 
 
 >- and dim$\mathcal{N}(A^T)=m-r$.


**Group-work**: Consider matrix A and its echelon form to answer the following
$$
A = \begin{bmatrix}
    1 & 2 & 0 & 1\\
    0 & 1 & 1 & 0\\
    1 & 2 & 0 & 1
\end{bmatrix}
\Rightarrow U = \begin{bmatrix}
    1 & 2 & 0 & 1\\
    0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0
\end{bmatrix}
$$
>1. Find all its four subspaces as spans of vectors.


### Motivating Example: An image as a tensor, a matrix, and a vector
***
Do the following
 1. Read an image from your computer (I downloaded a 'Baby Yoda')
 1. Verify the datatype and check the dimensions of the data.
 1. Convert the image to a monochrome image (a matrix) by using weighted addition of the RGB values.
 1. Put a frame around the image.
 1. Add some Gaussian-noise to the image.


```python
pix = mpimg.imread("./images/BabyYodaDoll.jpg") # OR BabyYodaDoll.jpg, BabyYodaXmas

plt.axis('off')
plt.imshow(pix)
plt.show()
```


```python
type(pix)
```




    numpy.ndarray




```python
dim = pix.shape
print ("Original order of the image tensor:", dim)


X = pix.reshape(-1,3)/255.0
print("After vectorizing the image, the dimensions are:", X.shape )


```

    Original order of the image tensor: (368, 700, 3)
    After vectorizing the image, the dimensions are: (257600, 3)


#### Conversion to a black & white image

[Check this out](https://www.prasannakumarr.in/journal/color-to-grayscale-python-image-processing
)


```python
## Converting a color image to black-n-white
### Read this blog: https://www.prasannakumarr.in/journal/color-to-grayscale-python-image-processing
color_weight = [0.2125, 0.7154, 0.0721]; # LUMA-REC.709
#color_weight = [1, 0, 0]
pix_gray = np.dot(pix[..., 0:3], color_weight) # Note conversion to gray-scale is not unique
print ("Order of the gray-scale image matrix:", pix_gray.shape)
plt.axis('off')
plt.imshow(pix_gray, cmap='gray')
plt.show()
```

#### Submatrices by Slicing
***


```python
# Create a frame around the image
pix_framed = np.zeros(np.add(pix_gray.shape, tuple([20,20])))
print ("Order of the framed gray-scale image matrix:", pix_framed.shape)
pix_framed[10:pix_gray.shape[0]+10, 10:pix_gray.shape[1]+10] = pix_gray
plt.axis('off')
plt.imshow(pix_framed, cmap='gray')
plt.show()
```

#### Adding Gaussian noise to a black-&-white image.


```python
#classwork: Make the frame lighter and wider.
# Add gaussian noise to the image
#size = pix_gray.shape
gauss_noise =  np.random.normal(0.0,10.0, (368,700))
noisy_pix= pix_gray + gauss_noise
plt.axis('off')
plt.imshow(pix)
#plt.imshow(gauss_noise.reshape(368,700))
plt.imshow(noisy_pix, cmap='gray')
plt.show()
```

#### Normal (Gaussian) Distribution
$$\large
p(x) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}
                 e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} },
                 $$
                 where $\mu$ is the mean and $\sigma$ is the standard deviation of the  distribution.

### Motivating Example: Term-document Matrix
---
AIM: Load some text data in a term-document  matrix.
You can try removing the stop-words.


```python
from sklearn.feature_extraction.text import CountVectorizer,ENGLISH_STOP_WORDS
```


```python
vectorizer = CountVectorizer()
```


```python
document1 = " I bought this game as a gift for my 8 year old daughter who loves games. I was expecting lots of gross foods--but I was surprised at the inappropriate cards--eyeball, human burger, blood salsa, and fresh brains. Those are not foods that typical people find in their refrigerators. We do not practice cannibalism. She was very upset when I suggested that we just take out those cards. I seriously wonder who thinks that those cards are appropriate for kids. The rest of the game is funny, but I wish I would have looked through the cards before I gave it to her."
document2 = "Absolutely love Taco vs burrito. I️ bought it as a kickstarter. I️ originally bought this game because my husband and I️ love to play games with friends but most of them are not targeted to children so I️ got this to add to our collection so we had options when our friends with kids came. I’m not gonna lie I️ did No have high expectations for this to be a game for adult but I️ was Sooooo wrong!!!!!! We have now played with several different groups of friends and it’s a hit!!!!! With adults it becomes a major strategy game. I️ have Now bought it as a Christmas present bc it was so well received!!!!"
document3 = " Unlike several of the reviewers here, I didn't purchase this originally for when kids are around. I bought it because of the reviews that said the adults all loved it too! I'm always on the lookout for games playable by 2 people and this was a great one. It's incredibly simple, but brings a lot of laughs with the competition and sabotage. I'm really glad I gave this game a chance."
doc_list = [document1, document2, document3]
```


```python
# Fit a bag of words
bow = vectorizer.fit_transform(doc_list)
print(type(bow))
print ("Feature (terms) Names: \n",vectorizer.get_feature_names())
```

    <class 'scipy.sparse.csr.csr_matrix'>
    Feature (terms) Names: 
     ['absolutely', 'add', 'adult', 'adults', 'all', 'always', 'and', 'appropriate', 'are', 'around', 'as', 'at', 'bc', 'be', 'because', 'becomes', 'before', 'blood', 'bought', 'brains', 'brings', 'burger', 'burrito', 'but', 'by', 'came', 'cannibalism', 'cards', 'chance', 'children', 'christmas', 'collection', 'competition', 'daughter', 'did', 'didn', 'different', 'do', 'expectations', 'expecting', 'eyeball', 'find', 'foods', 'for', 'fresh', 'friends', 'funny', 'game', 'games', 'gave', 'gift', 'glad', 'gonna', 'got', 'great', 'gross', 'groups', 'had', 'have', 'her', 'here', 'high', 'hit', 'human', 'husband', 'in', 'inappropriate', 'incredibly', 'is', 'it', 'just', 'kickstarter', 'kids', 'laughs', 'lie', 'looked', 'lookout', 'lot', 'lots', 'love', 'loved', 'loves', 'major', 'most', 'my', 'no', 'not', 'now', 'of', 'old', 'on', 'one', 'options', 'originally', 'our', 'out', 'people', 'play', 'playable', 'played', 'practice', 'present', 'purchase', 'really', 'received', 'refrigerators', 'rest', 'reviewers', 'reviews', 'sabotage', 'said', 'salsa', 'seriously', 'several', 'she', 'simple', 'so', 'sooooo', 'strategy', 'suggested', 'surprised', 'taco', 'take', 'targeted', 'that', 'the', 'their', 'them', 'thinks', 'this', 'those', 'through', 'to', 'too', 'typical', 'unlike', 'upset', 'very', 'vs', 'was', 'we', 'well', 'when', 'who', 'wish', 'with', 'wonder', 'would', 'wrong', 'year']



```python
# Check the matrix
print("Bag of words sparse matrix (data structure CSR-compressed sparse row):\n",bow, "\n To an array: \n", bow.toarray())
```

    Bag of words sparse matrix (data structure CSR-compressed sparse row):
       (0, 18)	1
      (0, 129)	1
      (0, 47)	2
      (0, 10)	1
      (0, 50)	1
      (0, 43)	2
      (0, 84)	1
      (0, 149)	1
      (0, 89)	1
      (0, 33)	1
      (0, 143)	2
      (0, 81)	1
      (0, 48)	1
      (0, 139)	3
      (0, 39)	1
      (0, 78)	1
      (0, 88)	2
      (0, 55)	1
      (0, 42)	2
      (0, 23)	2
      (0, 120)	1
      (0, 11)	1
      (0, 125)	4
      (0, 66)	1
      (0, 27)	4
      :	:
      (2, 35)	1
      (2, 102)	1
      (2, 9)	1
      (2, 108)	1
      (2, 110)	1
      (2, 4)	1
      (2, 80)	1
      (2, 133)	1
      (2, 5)	1
      (2, 90)	1
      (2, 76)	1
      (2, 98)	1
      (2, 24)	1
      (2, 54)	1
      (2, 91)	1
      (2, 67)	1
      (2, 115)	1
      (2, 20)	1
      (2, 77)	1
      (2, 73)	1
      (2, 32)	1
      (2, 109)	1
      (2, 103)	1
      (2, 51)	1
      (2, 28)	1 
     To an array: 
     [[0 0 0 0 0 0 1 1 2 0 1 1 0 0 0 0 1 1 1 1 0 1 0 2 0 0 1 4 0 0 0 0 0 1 0 0
      0 1 0 1 1 1 2 2 1 0 1 2 1 1 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 1 1 1 0
      1 0 0 1 0 0 1 0 0 1 0 0 1 0 2 0 2 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 1 0
      0 0 0 1 1 0 1 0 0 0 0 1 1 0 1 0 3 4 1 0 1 1 3 1 1 0 1 0 1 1 0 3 2 0 1 2
      1 0 1 1 0 1]
     [1 1 1 1 0 0 2 0 1 0 2 0 1 1 1 1 0 0 3 0 0 0 1 2 0 1 0 0 0 1 1 1 0 0 1 0
      1 0 1 0 0 0 0 2 0 3 0 3 1 0 0 0 1 1 0 0 1 1 3 0 0 1 1 0 1 0 0 0 0 5 0 1
      1 0 1 0 0 0 0 2 0 0 1 1 1 1 2 2 2 0 0 0 1 1 2 0 0 1 0 1 0 1 0 0 1 0 0 0
      0 0 0 0 0 1 0 0 3 1 1 0 0 1 0 1 0 0 0 1 0 3 0 0 5 0 0 0 0 0 1 2 2 1 1 0
      0 4 0 0 1 0]
     [0 0 0 1 1 1 2 0 1 1 0 0 0 0 1 0 0 0 1 0 1 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1
      0 0 0 0 0 0 0 2 0 0 0 1 1 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 3 0 0
      1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 0 3 0 1 1 0 1 0 0 1 0 1 0 0 0 1 1 0 0 0 1
      1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 0 1 5 0 0 0 3 0 0 0 1 0 1 0 0 0 1 0 0 1 0
      0 1 0 0 0 0]]



```python
print ("Data Type of variable bow:",type(bow))
# Find the index of specific words
print (vectorizer.vocabulary_.get("children"))
print (vectorizer.vocabulary_.get("salsa"))
print (bow.getcol(vectorizer.vocabulary_.get("great")))
```

    Data Type of variable bow: <class 'scipy.sparse.csr.csr_matrix'>
    29
    111
      (2, 0)	1


### Inverse and genralized inverse
***
A square matrix is said to have an inverse if all its rows or all its columns are linearly independent. Inverse of square matrix $A$ is denoted by $A^{-1}$. It has the following property:
$$ A^{-1} A = A A^{-1} = I$$
**Review Gauss-Jordan Elimination** for finding inverse of a matrix.
Ref: __[WikiPedia](https://en.wikipedia.org/wiki/Gaussian_elimination)__



```python
# Using linear algebra module for finding inverse
# Random Example Matrix
S = np.random.rand(4,4)
inv_S = np.linalg.inv(S)
print ("Inverse of S: \n",inv_S)
print ("Verify inverse:\n",np.dot(S,inv_S))

```

    Inverse of S: 
     [[ 6.29223122e-02  4.65377320e-01  1.11496530e+00 -1.38531775e+00]
     [ 5.39687908e-01 -9.77003966e-01  1.14702973e+00  2.37923981e-01]
     [-6.96882874e-01  4.52175993e-01 -9.25863481e-01  1.51010216e+00]
     [ 9.12693835e-01  3.45607950e-01 -5.85756832e-01 -3.29197827e-04]]
    Verify inverse:
     [[ 1.00000000e+00 -4.16333634e-17  4.16333634e-17 -1.38777878e-17]
     [ 1.11022302e-16  1.00000000e+00 -1.11022302e-16  2.77555756e-17]
     [ 5.55111512e-17  0.00000000e+00  1.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  5.55111512e-17  0.00000000e+00  1.00000000e+00]]


### Eigenvalues and eigenvectors
<hr>
An eigenvector of a square matrix $A$ is a special non-zero vector such that
$$ A v  = \lambda v$$
where $\lambda$ is called the associated eigenvalue.

**Example**: Find the eigenvalues and eigenvectors of 
$
A = \left( \begin{array}{cc} 2 & 4  \\
1 & -1 
\end{array}
\right) 
$
### Eigen-decomposition

If the square matrix $A\in \mathbb{C}^{n\times n}$ has $n$ linearly independent eigenvectors  then $A$ can be given in the following factorized form as
$$
A = Q \Lambda Q^{-1},
$$
where $\Lambda = $ diag$(\lambda_1, \cdots, \lambda_n)$ and columns of matrix $Q$ are made of the eigenvector $q_i$ of $A$ $(i=1,\cdots,n)$, arranged in the same order as the eigenvalues in $\Lambda$.
>- When $A$ is a real and symmetric matrix: $A = Q \Lambda Q^T$, where $Q$ is orthogonal $(Q^TQ = I = QQ^T)$ and $\Lambda$ is made of real diagonal entries.
>- If a function $f(x)$ has power series expansion in $x$, then $f(A) = Q f(\Lambda) Q^{-1}$.

### Daterminant
---

Laplace's Formula for determinant of a square matrix
$$\det(A) = \sum_{j=1}^n (-1)^{i+j} a_{ij} M_{ij}\ \textrm{ for a fixed row } i,$$
$$\det(A) = \sum_{i=1}^n (-1)^{i+j} a_{ij} M_{ij} \textrm{ for a fixed column } j .$$
where minor $M_{ij}$ si the determinant of the submatrix without the $i$-th row and $j$-th column.

<br>

>- ##### Determinant of a matrix is the product of all its eigenvalues.
>- ##### $\det\left(I_n\right) = 1$, where $I_n$ is the $n\times n$  identity matrix.
>- #####  $\det\left(A^\textsf{T}\right) = \det(A)$, where $A^\textsf{T}$ denotes the transpose of $A$.
>- #####  $\det\left(A^{-1}\right) = \frac{1}{\det(A)} = [\det(A)]^{-1}.$

>- #####  For square matrices $A$ and $B$ of equal size,
 $$\det(AB) = \det(A) \det(B).$$
>- #####  $\det(cA) = c^n\det(A)$, for an $n\times n$ matrix $A$. 

### Trace
---
Trace of a square matrix is the sum of its diagonal elements
$$
\operatorname{tr}(\mathbf{A}) = \sum_{i=1}^n a_{ii} = a_{11} + a_{22} + \dots + a_{nn}
$$

>- Trace is the sum of the eigenvalues of a matrix.
>- For any nonsingular matrix $\mathbf{S}$: $\operatorname{tr}(\mathbf{S} \mathbf{A} \mathbf{S}^{-1})  = \operatorname{tr}(\mathbf{A}) $ (i.e., invariance under change of base)

>- Following holds
$$
\begin{align}
  \operatorname{tr}(\mathbf{A}^T) &= \operatorname{tr}(\mathbf{A}),\\\\
  \operatorname{tr}(\mathbf{A} + \mathbf{B}) &= \operatorname{tr}(\mathbf{A}) + \operatorname{tr}(\mathbf{B}), \\\\
     \operatorname{tr}(c\mathbf{A}) &= c \operatorname{tr}(\mathbf{A}), \\\\
     \operatorname{tr}(\mathbf{A}\mathbf{B}) &= \operatorname{tr}(\mathbf{B}\mathbf{A}),\\\\
     \operatorname{tr}(\mathbf{A}\mathbf{B}) &\ne \operatorname{tr}(\mathbf{A})\operatorname{tr}(\mathbf{B})
\end{align}
$$

>- $$  \operatorname{tr}\left(\mathbf{A}^\mathsf{T}\mathbf{B}\right) =
  \operatorname{tr}\left(\mathbf{A}\mathbf{B}^\mathsf{T}\right) =
  \operatorname{tr}\left(\mathbf{B}^\mathsf{T}\mathbf{A}\right) =
  \operatorname{tr}\left(\mathbf{B}\mathbf{A}^\mathsf{T}\right) =
  \sum_{i,j}A_{ij}B_{ij}.
  $$

---
# DISREGARD THE REMAINING
---
## The Review continues in Notebook 2.

### Some Special Matrices
***
- Symmetric ans Skew-symmetric Matrics
- Upper and Lower Triangular Matrices
- Banded Matrices
- Orthogonal and Unitary Matrices
- Positive definite, positive semidefinite matrices
- Negative definite, negative semidefinite matrices
- Indefinite Matrices
- Permutation Matrix
- Diagonally Dominant Matrices
- Nonnegative Matrices

### Derivatives of Matrices of functions of some variables
***
#### Review the basics of derivatives, partial derivatives and gradients.
<div class="alert alert-block alert-info">
<b> Definition of derivative of a matrix of functions.</b>
$$ 
\frac{d}{d \alpha} C(\alpha) = \dot{C}(\alpha) = [\dot{c}_{i,j}(\alpha)].
$$
<b>Product Rule</b>
<div class='eqnbox'>
$$
\frac{d}{d \alpha} [ A(\alpha)\, B(\alpha)] = \dot{A}(\alpha)\,B(\alpha) + A(\alpha)\,\dot{B}(\alpha).
$$
</div>
</div>

[See Matrix Calculus on WikiPedia](https://en.wikipedia.org/wiki/Matrix_calculus)
<br>

**Exercise: ** If $\phi(x) = \frac{1}{2} x^T A x - b^T x$, show that the gradient is given by
$$
\nabla \phi(x) = \frac{1}{2} (A^T + A)x - b.
$$
This result will be quite useful as we move along.
<div class='eqnbox'>
$$\large 
\nabla_x \left( \frac{1}{2} x^T A x - b^T x \right) = \frac{1}{2} (A^T + A)x - b.
$$
</div>

### Norms
---
Please see Text-5 for a comprehensive review.

>- What is the norms of a vector?
>- $\|\cdot\|_p$ norms: Euclidean norm, 1-norms, Manhattan distance.
>- Matrix Norms, subordinate matrix norms
>- Unit circles in different norms.





[Image Source: WikiMedia](https://upload.wikimedia.org/wikipedia/commons/f/f8/L1_and_L2_balls.svg)


```python
# NORM : Euclidean, Frobenius

D1 = np.array([[1,1],[1,-1]])
print D1
print np.linalg.norm(D1)

D2 = np.array([[1,2, -1],[3,4, -6]])
print D2
print np.linalg.norm(D2)

#print np.linalg.norm(D2, ord=np.inf)
```


```python

```


```python

```


```python

```

### Derivatives of Matrices of functions of some variables
***
#### Review the basics of derivatives, partial derivatives and gradients.
<div class="alert alert-block alert-info">
<b> Definition of derivative of a matrix of functions.</b>
$$ 
\frac{d}{d \alpha} C(\alpha) = \dot{C}(\alpha) = [\dot{c}_{i,j}(\alpha)].
$$
<b>Product Rule</b>
<div class='eqnbox'>
$$
\frac{d}{d \alpha} [ A(\alpha)\, B(\alpha)] = \dot{A}(\alpha)\,B(\alpha) + A(\alpha)\,\dot{B}(\alpha).
$$
</div>
</div>

[See Matrix Calculus on WikiPedia](https://en.wikipedia.org/wiki/Matrix_calculus)
<br>

**Exercise: ** If $\phi(x) = \frac{1}{2} x^T A x - b^T x$, show that the gradient is given by
$$
\nabla \phi(x) = \frac{1}{2} (A^T + A)x - b.
$$
This result will be quite useful as we move along.
<div class='eqnbox'>
$$\large 
\nabla_x \left( \frac{1}{2} x^T A x - b^T x \right) = \frac{1}{2} (A^T + A)x - b.
$$
</div>

### Linear Systems of Equations
---
Example: Solve the following system by using inversion of the coefficient matrix.
> $2x+3y-z = 4$

> $-x+y+2z = 2$

> $3+2x-4z = 1$


```python
# FIND the solution using coding
A1 =  np.array([[2, 3, -1],
               [-1, 1, 2],
               [3, 2, -4]], dtype=float)
b = np.array([4, 2, 1]).reshape(3, -1)

# X = inverse(A) b
X = np.dot(np.linalg.inv(A1), b)
print X
```


```python
%%html
<style>
.eqnbox{
    margin:auto;width:500px;padding:20px;
    border: 3px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}

.eqnbox2{
    margin:auto;width:500px;padding:20px;
    border: 1px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}
</style>
```


<style>
.eqnbox{
    margin:auto;width:500px;padding:20px;
    border: 3px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}

.eqnbox2{
    margin:auto;width:500px;padding:20px;
    border: 1px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}
</style>



```python

```

8be5d78b0f2ba1a9f5853cfbcf145f7f7fedb4f2
ipynb
Jupyter Notebook
course_notes/MA544 Share/NB1 MA544 updated.ipynb
jschmidtnj/ma544-final-project
61fb57d344ad4f693eb697015ed926988402186f
MIT
2021-03-23T01:48:51.000Z
2022-02-01T22:49:47.000Z
course_notes/MA544 Share/NB1 MA544 updated.ipynb
jschmidtnj/ma544-final-project
61fb57d344ad4f693eb697015ed926988402186f
MIT
course_notes/MA544 Share/NB1 MA544 updated.ipynb
jschmidtnj/ma544-final-project
61fb57d344ad4f693eb697015ed926988402186f
MIT
2021-05-05T01:35:11.000Z
2021-05-05T01:35:11.000Z
Qwen/Qwen-72B
1. YES
2. YES