# Introduction to Neural Networks 



## The Perceptron


To get an intuitive idea about Neural Networks, we will code an elementary perceptron. In this example we will illustrate some of the concepts you have just seen, build a small perceptron and make a link between Perceptron and linear classifier.

### Generating some data

Before working with the MNIST dataset, you'll first test your perceptron implementation on a "toy" dataset with just a few data points. This allows you to test your implementations with data you can easily inspect and visualise without getting lost in the complexities of the dataset itself.


Start by loading two basic libraries: `matplotlib` and numpy




```python
# Load the libraries ...
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

```

Then let us generate some points in a 2D space that will form our dataset (you can add points later if you'd like)


```python
crosses = np.array([[0.5, 1.0], [1.0, 1.5], [1.5, 1.5], [2.0, 1.2], [3.0, 1.7], [1.5, 1.1],[2.1, 1.7]])
circles = np.array([[3.0, 0.5], [4.0, 1.0], [5.0, 0.7], [4.0, 0.2], [5.1, 0.3], [4.2, 0.7]])
```

### Visualising the data

Using `matploblib`, you can display the crosses as crosses (use `marker='x'`) and the circles as circles (use `marker='o'`). You will need to specify that you don't want a line using `linestyle='none'`. You can observe that the points are very easily separable. 


```python
plt.plot(crosses[:,0], crosses[:,1], marker='x', linestyle='none')
plt.plot(circles[:,0], circles[:,1], marker='o', linestyle='none')
plt.ylim((0, 2))
plt.xlim((0, 6))

```

### Computing the output of a Perceptron


Let us consider the problem of building a classifier that for a given **new** point will return whether it belongs to the crosses (class 1) or circles (class 0). So for example it would take `(2, 1.5)` and return `1`. 

Define a function `outPerceptron` which takes a 2d vector `x`, a 2d weight vector `w` and a bias `b` and returns the output following the step rule:

$$
\text{output} = \left\{\begin{align} 1\,\, &\text{if}\,\, \langle x, w\rangle -b \, >\,0 \\ 0\,\, &\text{otherwise}\end{align}\right.
$$


```python
def outPerceptron(x, w, b):
    innerProd = np.dot(x, w)
    output = 0
    if innerProd > b:
        output = 1
    return output

```

You can then enrich the function so that it can take a **sequence of inputs** (in the form of a matrix where each line of the matrix is one input vector) and return the corresponding **sequence of outputs**. 

One way of doing this is to loop over the rows of `X` and for each of them, use the function `outPerceptron` that you just wrote. Store the results in an array `outputs` and return that. Call that function `multiOutPerceptron`/

Once you have that, you can try optimising the function by using a matrix-vector product; call the resulting function `multiOutPerceptron2` (and make sure it leads to the same results!)


```python
def multiOutPerceptron(X,w,b):
    nInstances = X.shape[0]
    outputs    = np.zeros(nInstances)
    for i in range(0,nInstances):
        outputs[i] = outPerceptron(X[i,:],w,b)
    return outputs

def multiOutPerceptron2(X,w,b):
    return (np.dot(X,w)>b).astype(float)

```

## = checkpoint 1 =

here, you should copy-paste the following code. If it returns `True` you're good to go on.

```python
np.random.seed(1234)
X = np.random.randn(10, 5)
w = np.random.randn(5)
b = np.random.randn()
np.all(multiOutPerceptron2(X, w, b) == np.array([ 1.,  1.,  0.,  0.,  0.,  1.,  0.,  0.,  1.,  0.]))
```


### Trying different weights and biases

You now have a method that can compute the outputs predicted by an **untrained** perceptron. Can you try picking different weights and biases and see how well you can classify the crosses and circles? 

**Note**: to join the crosses and circles into one `instances` matrix, you can use `np.concatenate((crosses, circles), axis=0)`.

You can maybe start with `w=[1, 1]` and `b=1` and output the result of `multiOutPerceptron`. What is your analysis?


```python
test_w1 = [1., 1.]
test_b1 = 1.
instances = np.concatenate((crosses, circles), axis=0)
print(multiOutPerceptron(instances, test_w1, test_b1))

```

With the suggested weights and biases (`([1, 1],1)`), you should see something like 

> `[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]` 

which is clearly not great! Now try with `w=[-0.5, 1]` and `b=-0.2`, what do you observe? 


```python
test_w2 = [-0.5, 1.]
test_b2 = -0.2
print (multiOutPerceptron(instances, test_w2, test_b2))

```

### How did we get there?

This is much better (100% correct on the training data). 
To obtain these values, we found a **separating hyperplane** (here a line) between the points. 
The equation of the line is 

$ y = 0.5x-0.2 $


**Quiz**
- **Can you explain why this line corresponds to the weights and bias we used?**
- **Is this separating line unique? does it matter?**

### Illustrating the output of the Perceptron and the separating line

Copy-paste your code to visualise the crosses and circles above and overlay the separating line in red. 

Can you modify the parameters of the line a little bit and still find a separating line that "works"? 


```python
xx  = np.linspace(0, 6)
yy1 = 0.5 * xx - 0.2
yy2 = 0.4 * xx - 0.3

plt.plot(xx, yy1, color='red')
plt.plot(xx, yy2, color='orange')
plt.plot(crosses[:, 0], crosses[:, 1], marker='x', linestyle='none', label='sp1')
plt.plot(circles[:, 0], circles[:, 1], marker='o', linestyle='none', label='sp2')
plt.ylim((0, 2))
plt.xlim((0, 6))
plt.legend()

```

### Testing a few new points

Can you add the following `testPoints` on the plot and discuss what happens to them? 


```python
testPoints = np.array([[1, 0.5],[5, 1.5],[3, 1.1]])
```


```python
plt.plot(xx, yy1, color='red')
plt.plot(xx, yy2, color='orange')
plt.plot(crosses[:, 0], crosses[:, 1], marker='x', linestyle='none')
plt.plot(circles[:, 0], circles[:, 1], marker='o', linestyle='none')
plt.ylim((0, 2))
plt.xlim((0, 6))

# the points
plt.plot(1, 0.5, marker='s', color='blue', markersize=10) 
plt.plot(5, 1.5, marker='s', color='green', markersize=10)
plt.plot(3, 1.1, marker='^', color='black', markersize=10)

```

## = checkpoint 2 =


# Gradient Descent (remember?)

## Coding a simple gradient descent

### Considering some function

Let's consider the following arbitrary function and its gradient

$f(x) = \exp(-\sin(x))x^2$

$f'(x) = -x \exp(-\sin(x)) (x\cos(x)-2)$

It is convenient to define python functions which return the value of the function and its gradient at an arbitrary point $x$. Can you define a function `function` and a function `gradient`? 

* use `np.exp`, `np.sin`, `np.cos` and remember that `x**2` is the squared of `x`


```python
def function(x):
    return np.exp(-np.sin(x))*(x**2)

def gradient(x):
    return -x*np.exp(-np.sin(x))*(x*np.cos(x)-2)

```

### Visualising the function

Can you write a simple code that shows what the function looks like over the interval `[-10,10]`? use at least `100` points in order to have a high enough definition of the line.


```python
x = np.linspace(-10, 10, 500)
plt.plot(x, function(x))

```

### Implementing a simple GD

Now let us implement a simple Gradient Descent that uses constant stepsizes. Define two functions:

1. simplest version which doesn't store the intermediate steps that are taken. 
2. a version which does store the steps (useful to visualize what is going on and explain some of the typical behaviour of GD).

Let's call them `simpleGD` and `simpleGD2`. The parameters of both functions will be the initial point `x0`, the stepsize, and the number of steps to be taken.


```python
def simpleGD(x0, stepsize, nsteps):
    x    = x0
    for k in range(0,nsteps):
        x -= stepsize*gradient(x)
    return x

def simpleGD2(x0, stepsize, nsteps):
    x    = np.zeros(nsteps+1)
    x[0] = x0
    for k in range(0,nsteps):
        x[k+1] = x[k]-stepsize*gradient(x[k])
    return x

```

### Testing different situations

Try your algorithm `simpleGD` in the following cases:

* $x_0=1, \delta=0.1, n=100$
* $x_0=6, \delta=0.1, n=100$
* $x_0=8, \delta=0.01, n=100$

Can you discuss the results you obtained by having a look at the plot of the function? 

### Visualising the cases

We suggest below a function `viz` which shows the path taken by the gradient descent when computed using `simpleGD2`. 

Use it in the different cases above in order to see what the Gradient Descent does. Try to interpret the different cases.


```python
def viz(x, a=-10, b=10):
    xx  = np.linspace(a, b, 100)
    yy  = function(xx)
    ygd = function(x)
    plt.plot(xx, yy)
    plt.plot(x, ygd, color='red')
    plt.plot(x[0], ygd[0], marker='o', color='green', markersize=10)
    plt.plot(x[len(x)-1], ygd[len(x)-1], marker='o', color='red', markersize=10)
    plt.show()

```


```python
x1 = simpleGD2(3, 0.1, 100)
x2 = simpleGD2(6, 0.1, 100)
x3 = simpleGD2(8, 0.01, 100)
x4 = simpleGD2(3, 0.5, 100)

viz(x1)
viz(x2)
viz(x3)
viz(x4)

```

2a65f6fe8f77de38e02f2ae96a91d95bf5c36439
ipynb
Jupyter Notebook
ads8-neuralnets_content_students/d1-introNN-part1_solution.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
2021-06-04T14:26:30.000Z
2021-06-04T14:26:30.000Z
ads8-neuralnets_content_tutors/d1-introNN-part1_solution.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
ads8-neuralnets_content_tutors/d1-introNN-part1_solution.ipynb
JamesOwers/camsparknn
c94b6c8307ff85c819f8efc1f57ab58242bdd7cd
MIT
Qwen/Qwen-72B
1. YES
2. YES