# Functions as Real-valued Circuits

## Base Case: Single Gate in the Circuit

### Single Multiply Gate

The first thing we are going to compute is a simple function of two variables $F = f(X,Y)=X*Y$.



The circuit takes two real-valued inputs $X$ and $Y$, computes the product $X*Y$ defined by the function $f$ and stores it in the variable $F$.


```python
def f(X,Y):
    return X*Y
```


```python
F = f(-2,3)
print (f'The output F is: {F}')
```

    The output F is: -6


### <font color=blue>*How should one tweak the inputs $(X,Y)$ slightly to increase the output of the function __f__?*</font> 

### Strategy #1 - Numerical Gradient

The partial derivative of the function $f$ with respect to $X$ can be computed as:

\begin{align}
\frac{\partial f (X,Y)}{\partial X} = \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\
\end{align}

This can be simulated in code by choosing $h$ to be a very small number:


```python
h = 0.0001
```

#### Computing a partial derivative with respect to $X$


```python
X = -2; Y = 3
```


```python
X_derivative = (f(X+h, Y)-f(X, Y))/h
print (f'the derivative in respect to X is: {X_derivative}')
```

    the derivative in respect to X is: 3.00000000000189


Positive derivative value indicates that $X$ should be increased in order to increase $f$, let's try that:


```python
print (f'the original output is: {f(-2,3)}')
# increase X for a small value, for example 0.2
print (f'the new output is: {f(-1.8,3)}')
```

    the original output is: -6
    the new output is: -5.4


$-5.4$ is larger than $-6$, it works!

#### compute a partial derivative with respect to $Y$


```python
Y_derivative = (f(X, Y+h)-f(X, Y))/h
print (f'the derivative in respect to Y is: {Y_derivative}')
```

    the derivative in respect to Y is: -2.0000000000042206


Negative derivative value indicates that $Y$ should be decreased in order to increase $f$, let's try that:


```python
print (f'the original output is: {f(-2,3)}')
# decrease Y for a small value, for example 0.1
print (f'the new output is: {f(-2,2.9)}')
```

    the original output is: -6
    the new output is: -5.8


$-5.8$ is larger than $-6$, it works!

The __gradient__ of a function is made up of all the partial derivatives of this function concatenated in a vector

\begin{align}
\nabla f(X,Y)=\left[\frac{\partial f (X,Y)}{\partial X},\frac{\partial f (X,Y)}{\partial Y}\right]
\end{align}

#### Gradually minimizing a function by using derivatives

In order to gradually maximize the function $f$ (step-by-step) towards our desired result, we need to update our parameters. This is done by adding to the parameter's value the value of its partial derivative. To achieve this gradually in small steps, we multiply the value of the partial derivative by a a small number (step).


```python
step_size = 0.01
F = f(X,Y)
print (f'The original output F is: {F}')
```

    The original output F is: -6



```python
X = X + step_size * X_derivative
Y = Y + step_size * Y_derivative
print (f'X is: {X}, Y is: {Y}')
```

    X is: -1.969999999999981, Y is: 2.979999999999958



```python
F_new = f(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: -5.87059999999986


The new output is larger than the old, thanks to partial derivatives.

This approach, however, is still expensive because we need to compute the circuit’s output as we tweak every input value independently a small amount. 

### Strategy #2 - Analytic Gradient

We can use calculus to compute partial derivatives of the function $f(X,Y)$.

\begin{align}
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X} = \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\
\end{align}

A partial derivative of $F=f(X,Y)$ in respect to $X$ is:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X} &= \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+h)Y -XY}{h} \\
&=\lim_{h\to 0}\frac{XY+Yh-XY}{h} \\
&=\lim_{h\to 0}\frac{Yh}{h} \\
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X}&=Y
\end{align*}

A partial derivative of $F=f(X,Y)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial f (X,Y)}{\partial Y} &= \lim_{h\to 0} \frac{f(X,Y+h)-f(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X(Y+h) -XY}{h} \\
&=\lim_{h\to 0}\frac{XY+Xh-XY}{h} \\
&=\lim_{h\to 0}\frac{Xh}{h} \\
\frac{\partial F}{\partial Y}=\frac{\partial f (X,Y)}{\partial Y} &=X
\end{align*}

Here are both partial derivatives:

\begin{align*}
\frac{\partial F}{\partial X}=Y ; \frac{\partial F}{\partial Y}=X \\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla f(X,Y)=\left[Y,X\right]
\end{align}

We can now use this information to increase the output of the function __f__:


```python
X = -2; Y = 3
F = f(X,Y)
print (f'the output F is: {F}')
```

    the output F is: -6





```python
X_gradient = Y
Y_gradient = X
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient}')
```

    X-gradient: 3 
    Y-gradient: -2





```python
step_size = 0.001
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'X is now: {X}, \nY is now: {Y}')
```

    X is now: -1.997, 
    Y is now: 2.998



```python
F_new = f(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: -5.987006000000001


The new output $-5.8706$ is larger than the old: $-6$.

***

### Single Add Gate

The second function we're going to compute is  $G=g(X,Y)=X+Y.$



The circuit takes two real-valued inputs $X$ and $Y$ and computes the sum $X+Y$.


```python
def g(X,Y):
    return X+Y
```


```python
X = -2; Y = 3
G = g(X,Y)
print (f'The output is: {G}')
```

    The output is: 1




As we did before, we can use calculus to compute partial derivatives for the function $G=g(X,Y)$:

\begin{align}
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X} = \lim_{h\to 0} \frac{g(X+h,Y)-g(X,Y)}{h} \\
\end{align}

A partial derivative of $G=g(X,Y)$ in respect to $X$ is:

\begin{align*}
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X} &= \lim_{h\to 0} \frac{g(X+h,Y)-g(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X+h+Y -X-Y}{h} \\
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X}&=\lim_{h\to 0}\frac{h}{h} =1 \\
\end{align*}

A partial derivative of $G=g(X,Y)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial G}{\partial Y}=\frac{\partial g (X,Y)}{\partial Y} &= \lim_{h\to 0} \frac{g(X,Y+h)-g(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X+Y+h -X-Y}{h} \\
\frac{\partial G}{\partial Y}=\frac{\partial g (X,Y)}{\partial Y}&=\lim_{h\to 0}\frac{h}{h} =1 \\
\end{align*}

Both partial derivatives $\frac{\partial G}{\partial X}$ and $\frac{\partial G}{\partial Y}$ in this case are equal to $1$.

We can use this information to maximize the function __g__:


```python
X_gradient = 1
Y_gradient = 1
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient}')
```

    X-gradient: 1 
    Y-gradient: 1





```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'X is: {X} \nY is: {Y}')
```

    X is: -1.99 
    Y is: 3.01



```python
F_new = g(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: 1.0199999999999998


## Recursive Case: Circuits with Multiple Gates

The expression we are computing now is $M = m(X,Y,Z)=(X+Y)*Z$.




```python
def m(X,Y,Z):
    return (X+Y)*Z
```


```python
X = -2; Y = 5; Z = -4
```

this is equal to $M=m(-2,5,-4)=(-2+5)-4=3*-4=-12$


```python
M = m(X,Y,Z)
print (f'the output M is: {M}')
```

    the output M is: -12




As we did before, we can use calculus to derive partial derivatives for the function $M=m(X,Y,Z)$.

\begin{align}
\frac{\partial M}{\partial X}=\frac{\partial m (X,Y,Z)}{\partial X} = \lim_{h\to 0} \frac{m(X+h,Y,Z)-m(X,Y,Z)}{h} \\
\end{align}

A partial derivative of $M=m(X,Y,Z)$ in respect to $X$ is:

\begin{align*}
\frac{\partial M}{\partial X}=\frac{\partial m(X,Y,Z)}{\partial X} &= \lim_{h\to 0} \frac{f(X+h,Y,Z)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+h+Y)*Z -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{ZX+Zh+ZY-ZX-ZY}{h} \\
\frac{\partial M}{\partial X}=\frac{\partial m(X,Y,Z)}{\partial X}&=\lim_{h\to 0}\frac{Zh}{h} =Z \\
\end{align*}

 

Similarly, partial derivative of $M=m(X,Y,Z)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial M}{\partial Y}=\frac{\partial m(X,Y,Z)}{\partial Y} &= \lim_{h\to 0} \frac{f(X,Y+h,Z)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+Y+h)*Z -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{ZX+ZY+Zh-ZX-ZY}{h} \\
\frac{\partial M}{\partial Y}=\frac{\partial m(X,Y,Z)}{\partial Y}&=\lim_{h\to 0}\frac{Zh}{h} =Z \\
\end{align*}

 

A partial derivative of $M=m(X,Y,Z)$ in respect to $Z$ is:

\begin{align*}
\frac{\partial M}{\partial Z}=\frac{\partial m(X,Y,Z)}{\partial Z} &= \lim_{h\to 0} \frac{f(X,Y,Z+h)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+Y)*(Z+h) -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{XZ+Xh+YZ+Yh-XZ-YZ}{h} \\
&=\lim_{h\to 0}\frac{Xh+Yh}{h} \\
&=\lim_{h\to 0}\frac{h(X+Y)}{h}\\
\frac{\partial M}{\partial Z}=\frac{\partial m(X,Y,Z)}{\partial Z}&=X+Y \\
\end{align*}

Here are all three partial derivatives:

\begin{align*}
\frac{\partial M}{\partial X}=Z ; \frac{\partial M}{\partial Y}=Z ;\frac{\partial M}{\partial Z}=X+Y \\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla m(X,Y,Z)=\left[Z,Z,X+Y\right]
\end{align}

We can now use this information to maximize the output of the function __m__:


```python
X_gradient = Z
Y_gradient = Z
Z_gradient = X+Y
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient} \nZ-gradient: {Z_gradient}')
```

    X-gradient: -4 
    Y-gradient: -4 
    Z-gradient: 3





```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
Z = Z + step_size * Z_gradient
print (f'X is: {X} \nY is: {Y} \nZ is: {Z}')
```

    X is: -2.04 
    Y is: 4.96 
    Z is: -3.97



```python
M_new = m(X,Y,Z)
print (f'old output: {M}\nnew output: {M_new}')
```

    old output: -12
    new output: -11.5924


***

### Backpropagation

Instead of working with the function $m(X,Y,Z)=(X+Y)*Z$, we can simplify the computation by composing two new simpler functions: <br>$G=g(X,Y)=X+Y$, and <br>$F=f(G,Z)=G*Z$, into: <br>$F=f(g(X,Y),Z)$



Here, we can apply the chain rule for derivation, because $F$ is a function of $G$ and $G$ is a function of $X$ and $Y$.<br>
So instead of computing $\frac{\partial M}{\partial X}$, $\frac{\partial M}{\partial Y}$  and $\frac{\partial M}{\partial Z}$ which gets more complicated to compute with more complex expressions, we compute instead $\frac{\partial F}{\partial X}$, $\frac{\partial F}{\partial Y}$ and $\frac{\partial F}{\partial Z}$, which can be decomposed:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial X}
\end{align*}

Here $\frac{\partial F}{\partial G}$ is a simple multiplication gate, whose derivate we have already computed:$\frac{\partial F}{\partial G}$ =$\frac{\partial f(G,Z)}{\partial G}=Z$

Also $\frac{\partial G}{\partial X}$ is a simple addition gate, whose derivate we have already computed: $\frac{\partial G}{\partial X}$ =$\frac{\partial g(X,Y)}{\partial X}=1$, thus:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial X} = Z*1 = Z
\end{align*}

***

The same applies when computing the partial $\frac{\partial F}{\partial Y}$:

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial Y}
\end{align*}

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial Y} = Z*1 = Z
\end{align*}

***

The partial derivative $\frac{\partial F}{\partial Z}$ does not need to be decomposed:

\begin{align*}
\frac{\partial F}{\partial Z}=G=X+Y
\end{align*}

We can now use this information to maximize the output of the function __m_decomposed__:


```python
def m_decomposed(X,Y,Z):
    G = g(X,Y)
    F = f(G,Z)
    return F
```


```python
X = -2; Y = 5; Z = -4
```


```python
F = m_decomposed(X,Y,Z)
print (f'the output is: {F}')
```

    the output is: -12





```python
X_gradient = Z
Y_gradient = Z
Z_gradient = X+Y
print (f'X-gradient: {X_gradient}, \nY-gradient: {Y_gradient}, \nZ-gradient: {Z_gradient}.')
```

    X-gradient: -4, 
    Y-gradient: -4, 
    Z-gradient: 3.




Here, we can observe, that in the backward pass, the multiplication gate switches the values of outputs: what used to be (3,-4) in the forward pass, it becomes (-4,3) in the backward pass. The addition gate, on the other hand just passes its input value to the ouput without changing it.


```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
Z = Z + step_size * Z_gradient
print (f'X is: {X}\nY is: {Y}\nZ is: {Z}')
```

    X is: -2.04
    Y is: 4.96
    Z is: -3.97



```python
F_new = m(X,Y,Z)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -12
    new output: -11.5924


***

### Example: More complex functions

Here is a seemingly complicated function:

\begin{align}
l(A,B,C,X,Y)&= \frac{1}{1+e^{-(AX+BY+C)}}\\
&or \\
l(A,B,C,X,Y)&= \sigma (AX+BY+C)\\
\end{align}


The function $\sigma$ is called a *sigmoid function*:

\begin{align}
\sigma&= \frac{1}{1+e^{-x}}\\
\end{align}

and it was used a lot in machine learning before.

The derivative of the sigmoid function is:

\begin{align}
\frac{d\sigma(x)}{dx}= \sigma(x) * (1-\sigma(x))
\end{align}



which means that once we compute the final activation $F=\sigma(AX+BY+C)$, we can simply calculate the derivative as $F*(1-F)$.

We can compute the forward pass of this function without a problem, however, directly computing the partial derivatives $\frac{\partial L}{\partial A}$, $\frac{\partial L}{\partial B}$, $\frac{\partial L}{\partial C}$, ... could be tricky. It is much better to use the chain rule and compose multiple functions togeher.

We can create 4 simple functions:

\begin{align*}
G=g(A,X)&=A*X \\
H=h(B,Y)&=B*Y \\
K=k(G,H,C)&=G+H+C \\
F=f(K)&= \frac{1}{1+e^{-x}}\\
\end{align*}

and compose them as:

\begin{align*}
F=f(k(g(A,X),h(B,Y),C))
\end{align*}

This looks much simpler on a diagram:



Let's compute all the partial derivatives by using the chain rule:

\begin{align*}
\frac{\partial F}{\partial A}&=\frac{\partial F}{\partial K}*\frac{\partial K}{\partial G}*\frac{\partial G}{\partial A} \\
\frac{\partial F}{\partial A}&= F(1-F)*1*X \\
\frac{\partial F}{\partial A}&= XF(1-F)
\end{align*}

\begin{align*}
\frac{\partial F}{\partial X}&=\frac{\partial F}{\partial K}*\frac{\partial K}{\partial G}*\frac{\partial G}{\partial X} \\
\frac{\partial F}{\partial X}&= F(1-F)*1*A \\
\frac{\partial F}{\partial X}&= AF(1-F)
\end{align*}

By following the exact same procedure for $B$, $Y$, and $C$ we get:

\begin{align*}
\frac{\partial F}{\partial B}&=YF(1-F)\\
\end{align*}

\begin{align*}
\frac{\partial F}{\partial Y}&=BF(1-F)\\
\end{align*}

\begin{align*}
\frac{\partial F}{\partial C}&=F(1-F)\\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla l(A,B,C,X,Y)=\left[XF(1-F),YF(1-F),F(1-F),AF(1-F),BF(1-F) \right]
\end{align}

We can now use this information to maximize the output of the function __l__:


```python
import numpy as np
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```


```python
def l(A,B,C,X,Y):
    G = f(A,X)
    H = f(B,Y)
    K = G + H + C
    F = sigmoid(K)
    return F
```


```python
A = 1.0; B = 2.0; C = -3.0; X = -1.0; Y = 3.0
```


```python
F = l(A,B,C,X,Y)
print (f'the output F is: {F}')
```

    the output F is: 0.8807970779778823




Since every partial derivative involves $F(1-F)$, we will compute it first as `F_K`


```python
gradient_end = 1
F_K = (F * (1 - F)) * gradient_end
print (f'F_K = {F_K}')
```

    F_K = 0.10499358540350662



```python
A_gradient = X*F_K
B_gradient = Y*F_K
C_gradient = F_K
X_gradient = A*F_K
Y_gradient = B*F_K

print (f'A-gradient: {A_gradient} \nX-gradient: {X_gradient} \nB-gradient: {B_gradient} \nY-gradient: {Y_gradient}\nC-gradient: {C_gradient}')
```

    A-gradient: -0.10499358540350662 
    X-gradient: 0.10499358540350662 
    B-gradient: 0.31498075621051985 
    Y-gradient: 0.20998717080701323
    C-gradient: 0.10499358540350662





```python
step_size = 0.01
A = A + step_size * A_gradient
B = B + step_size * B_gradient
C = C + step_size * C_gradient
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'A is: {A}, \nB is: {B}, \nC is: {C}, \nX is: {X}, \nY is: {Y}')
```

    A is: 0.998950064145965, 
    B is: 2.0031498075621053, 
    C is: -2.9989500641459648, 
    X is: -0.998950064145965, 
    Y is: 3.00209987170807



```python
F_new = l(A,B,C,X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: 0.8807970779778823
    new output: 0.8825501816218984


The new output is higher than the old one!

21539df46e81ddcbd9698251dc885159f10bf975
ipynb
Jupyter Notebook
04. Functions as Real Valued Circuits.ipynb
Mistrymm7/machineintelligence
7629d61d46dafa8e5f3013082b1403813d165375
Apache-2.0
2019-09-23T11:25:41.000Z
2022-03-29T22:56:10.000Z
04. Functions as Real Valued Circuits.ipynb
Iason-Giraud/machineintelligence
b34a070208c7ac7d7b8a1e1ad02813b39274921c
Apache-2.0
04. Functions as Real Valued Circuits.ipynb
Iason-Giraud/machineintelligence
b34a070208c7ac7d7b8a1e1ad02813b39274921c
Apache-2.0
2019-09-30T16:08:46.000Z
2022-02-19T10:29:07.000Z
Qwen/Qwen-72B
1. YES
2. YES