# Single perceptron

We want to implement a single layer perceptron to classify AND, OR and XOR using perceptron learning and delta rule.

In this particular case, we will use the same 4 sets of binary data to train the perceptron. We call this `epoch`.

For the OR case:

<table>
        <thead>
            <th colspan="2">inputs</th>
            <th>target</th>
        </thead>
        <tbody>
            <tr>
                <td>0</td>
                <td>0</td>
                <td>0</td>
            </tr>
            <tr>
                <td>0</td>
                <td>1</td>
                <td>1</td>
            </tr>
            <tr>
                <td>1</td>
                <td>0</td>
                <td>1</td>
            </tr>
            <tr>
                <td>1</td>
                <td>1</td>
                <td>1</td>
            </tr>
        </tbody>
</table>

For the AND case:

<table>
        <thead>
            <th colspan="2">inputs</th>
            <th>target</th>
        </thead>
        <tbody>
            <tr>
                <td>0</td>
                <td>0</td>
                <td>0</td>
            </tr>
            <tr>
                <td>0</td>
                <td>1</td>
                <td>0</td>
            </tr>
            <tr>
                <td>1</td>
                <td>0</td>
                <td>0</td>
            </tr>
            <tr>
                <td>1</td>
                <td>1</td>
                <td>1</td>
            </tr>
        </tbody>
</table>

And for the XOR case:

<table>
        <thead>
            <th colspan="2">inputs</th>
            <th>target</th>
        </thead>
        <tbody>
            <tr>
                <td>0</td>
                <td>0</td>
                <td>0</td>
            </tr>
            <tr>
                <td>0</td>
                <td>1</td>
                <td>1</td>
            </tr>
            <tr>
                <td>1</td>
                <td>0</td>
                <td>1</td>
            </tr>
            <tr>
                <td>1</td>
                <td>1</td>
                <td>0</td>
            </tr>
        </tbody>
</table>

The perceptron will have two inputs $ x_0 $ and $ x_1 $ and an extra "bias" input $ x_2 = 1 $. There will be three weights $ w_0, w_1, w_2 $ and an output $ y $.

The inputs, outputs and bias are related by the relation:

$$ y = \mathbf{Wx}^\text{T} $$

where $ \mathbf{W} = [w_0, w_1, w_2] $ and $ \mathbf{x} = [x_0, x_1, x_2] $

This relation is derived from the following figure:



Explicitly:

$$ y = w_0 \cdot x_0 + w_1 \cdot x_1 + w_2 \cdot x_2 $$

This particular perceptron does not have a threshold unit. We will add a threshold unit to the output defined as:

$$
\text{sgn}(x) = \cases{
    -1, & if $x\lt0$\cr
    1, & if $x\gt0$ \cr
} = \frac{x}{|x|}
$$

We break the process into two steps: training and prediction. In the training phase we show the perceptron example data and expected outputs and let it adjust its weights. In the prediction phase we validate whether the perceptron was able to successfully classify the data. 

The weights and biases form a plane in 3-dimensional space that bisects the 2d plane. The weights $ w_0 $ and $ w_1 $ define the normal of the plane (and thus its orientation) while the weight $ w_2 $ defines its translation from the origin.


```python
# import python packages

%matplotlib inline

from pylab import * # for plotting
from numpy import * # for matrix and numerical libraries

matplotlib.rcParams['figure.figsize'] = (10,5)
```

## Training

We will first implement the training step of the perceptron. The training step will accept the following parameters:

```
inputs (x) : the input patterns (a 2xn matrix containing n examples)
targets (t) : the target patterns (a 1xn vector)
learningRate (eta) : the step size of the gradient descent
epochs : number of times the input and target will be trained
```

### Perceptron learning

As stated earlier, the network learns by adjusting its weights. We need a strategy to update these weights. The simple strategy is to perform a forward pass

$$ y = \text{sgn}(\mathbf{Wx}^T) $$

and compute the error

$$ e = y - t $$

(where t is the target pattern) and calculating

$$ \Delta W = \eta \mathbf{x} e $$.

We can then update the weights with:

$$ \mathbf{W} = \mathbf{W} + \Delta W $$

This is called perceptron learning.

Let us create a simple training function.


```python
def train_pl(inputs, targets, eta=0.01, epochs=200):
    bias = ones((1, inputs.shape[1])) # create a 1xn vector of 1s as our bias
    X = vstack((inputs, bias)) # add bias to the inputs, resulting in a 3xn matrix
    
    weights = random.randn(1, X.shape[0]) # create a 1x3 vector of random weights
    error = zeros(epochs) # create a vector to store error computed at each epoch
    
    for i in range(epochs):
        err = (sign(weights * X) - targets)
        deltaW = -eta * err * X.T
        weights = weights + deltaW # update weights
        error[i] = mean(power(err, 2)) # calculate mean square error for each epoch
    
    return weights, error
```

The training function takes inputs and targets and returns weights of the trained network. It also returns an error so that we can observe the evolution of the network. Let us run it with a OR dataset, note that we replace 0 with a -1 because we want to make the inputs and targets symmetric (to prevent premature termination of learning).

Let us define a function that makes our binary values symmetric.


```python
def symmetric(v):
    """
        if v = 0:
            2 * (0 - 0.5) = 2 * (-0.5) = -1
        if v = 1
            2 * (1 - 0.5) = 2 * (0.5) = 1
    """
    return 2 * (v - 0.5)

OR_X = symmetric(matrix("""
0 1 0 1;
0 0 1 1
"""))

OR_T = symmetric(matrix("""
0 1 1 1
"""))

print(OR_X)
print(OR_T)
```

    [[-1.  1. -1.  1.]
     [-1. -1.  1.  1.]]
    [[-1.  1.  1.  1.]]


We are now ready to train our simple perceptron for the OR dataset.


```python
W, err = train_pl(OR_X, OR_T, eta=0.01, epochs=100) # perform training

# plot error
figure()
plot(err)
xlabel('epoch')
ylabel('mean square error')
title('MSE when training with pereptron learning')

```

Hmm ... we can see the the network was able to classify successfully. But the curve looks really strange, it is difficult to see how the network has drawn the decision boundary (the plane in 3-d space) from this view. A better way is to rearrange our output equation so that we can see the decision boundary in a more clear way.

Given that we have:

$$
\begin{align}
y & = w_0 x_0 + w_1 x_1 + w_2 x_2 \\\\
& = w_0 x_0 + w_1 x_1 + w_2 \ \text{(since $ x_2 = 1 $)}
\end{align}
$$

The definition of a plane is:

$$
0 = ax + by + c
$$

Where a, b and c are the normals of the plane. Since we want to find the plane defined by the weight matrices, we set $ y = 0 $.

Let us rearrange the equation in terms of $ x_0 $ (the decision is arbitrary, we could have chosen $ x_1 $ as well).

$$
\begin{align}
0 & = w_0 x_0 + w_1 x_1 + w_2 \\\\
w_0 x_0 + w_1 x_1 & = -w_2 \\\\
w_0 x_0 & = -w_1 x_1 - w_2 \\\\
x_0 & = -\frac{w_1}{w_0} x_1 - \frac{w_2}{w_0} \\\\
x_0 & = m x + b
\end{align}
$$

In order to draw the decision boundary we need to first plot our data points.


```python
pos = OR_X[:,where(OR_T > 0)[1]] # find points where T > 0
neg = OR_X[:,where(OR_T < 0)[1]] # find points where T < 0

figure()
scatter([pos[0,:]], [pos[1,:]], marker='o', c='g') # plot positive points with a green circle
scatter([neg[0,:]], [neg[1,:]], marker='x', c='r') # plot negative points with a red cross
xlim([-2, 2])
ylim([-2, 2])
grid()
```

Now we can calculate the decision boundary using the weights and plot it on top of our plot to get a better understanding of how the training went.


```python
x = linspace(-2, 2, 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary

figure()
scatter([pos[0,:]], [pos[1,:]], marker='o', c='g') # plot positive points with a green circle
scatter([neg[0,:]], [neg[1,:]], marker='x', c='r') # plot negative points with a red cross
plot(x, boundary)
xlim([-2, 2])
ylim([-2, 2])
grid()
```

It is now much easier to see the decision boundary. However, it is quite clear that this is not the optimal decision boundary and it looks as if the network stopped learning a bit early as the line is closer to the (1,-1) point than the (-1, 1) point. We know it is not an issue of not enough epochs because the MSE graph shows that the error converged to 0.

We need a better solution.

### Delta Rule (Widrow-Hoff rule)

When using perceptron learning we calculated the error by comparing the thresholded output to the targets. A better strategy is to use the unthresholded output to the target as this allows a much better estimate of the error surface allowing us to find the optimal decision boundary for the single-perceptron case.

We change the error function from:

$$
e = \text{sgn}(\mathbf{Wx}^T) - t
$$

to:

$$
e = \mathbf{Wx}^T - t
$$

This requires only a single change in our training function.


```python
def train_dr(inputs, targets, eta=0.01, epochs=200):
    bias = ones((1, inputs.shape[1])) # create a 1xn vector of 1s as our bias
    X = vstack((inputs, bias)) # add bias to the inputs, resulting in a 3xn matrix
    
    weights = random.randn(1, X.shape[0]) # create a 1x3 vector of random weights
    error = zeros(epochs) # create a vector to store error computed at each epoch
    
    for i in range(epochs):
        err = ((weights * X) - targets) # NOTE CHANGE HERE!!
        deltaW = -eta * err * X.T
        weights = weights + deltaW # update weights
        error[i] = mean(power(err, 2)) # calculate mean square error for each epoch
    
    return weights, error
```

Training and plotting the decision boundary and error curve:


```python
W, err = train_dr(OR_X, OR_T, eta=0.01, epochs=100) # perform training

# plot error
figure()
plot(err)
xlabel('epoch')
ylabel('mean square error')
title('MSE when training with delta rule')
```

We see that the error curve is much smoother because we are not thresholding the output.


```python
x = linspace(-2, 2, 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary

figure()
scatter([pos[0,:]], [pos[1,:]], marker='o', c='g') # plot positive points with a green circle
scatter([neg[0,:]], [neg[1,:]], marker='x', c='r') # plot negative points with a red cross
plot(x, boundary)
xlim([-2, 2])
ylim([-2, 2])
grid()
```

The decision boundary separates the two datasets well. Let us now use the network to classify the AND function.


```python
AND_X = symmetric(matrix("""
0 1 0 1;
0 0 1 1
"""))

AND_T = symmetric(matrix("""
0 0 0 1
"""))

pos = AND_X[:,where(AND_T > 0)[1]] # find points where T > 0
neg = AND_X[:,where(AND_T < 0)[1]] # find points where T < 0

print(AND_X)
print(AND_T)
```

    [[-1.  1. -1.  1.]
     [-1. -1.  1.  1.]]
    [[-1. -1. -1.  1.]]



```python
W, err = train_dr(AND_X, AND_T, eta=0.01, epochs=100) # perform training

x = linspace(-2, 2, 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary

# plot error
subplot(2, 1, 1)
plot(err)
xlabel('epoch')
ylabel('mean square error')
title('MSE when training with delta rule')

subplot(2, 1, 2)
scatter([pos[0,:]], [pos[1,:]], marker='o', c='g') # plot positive points with a green circle
scatter([neg[0,:]], [neg[1,:]], marker='x', c='r') # plot negative points with a red cross
plot(x, boundary)
xlim([-2, 2])
ylim([-2, 2])
grid()
```

# Assignment Part I

## Classification with single-layer perceptron

### Generate linearly separable data

We can generate a sample of normally distributed multivariate data centered at `mu` and with covariance `cov` using the following functions. 


```python
mu = [0, 0]
cov = [[1, 0], [0, 100]]
x, y = random.multivariate_normal(mu, cov, 100).T
scatter(x, y)
```

Let us package it into a function so that we can generate labels and data.


```python
def gen_data(mu, label, n=100):
    cov = matrix("8 3; 3 8")
    data = random.multivariate_normal(mu, cov, n).T
    labels = label * ones((1, n))
    return vstack((data, labels))

data = hstack((gen_data(random.randint(0, 20, size=2), 0), gen_data(random.randint(-30, 0, size=2), 1)))
colors = ['red', 'green']
random.shuffle(data.T)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
```

Let us implement sequential learning for both delta and perceptron learning. Instead of creating new functions, let us define a function `train` that takes different parameters.

One important thing we must consider is the scale of the inputs and targets. So far, we have defined our inputs to be in the range $ x \in [-1, 1] $ however if we want to generalise our learning algorithm it must be able to handle inputs of any scale. A simple solution is to normalise our data into the symmetric $ [-1, 1] $ range, if we don't the error function for the delta rule:

$$
e = \mathbf{w}\mathbf{x}^\text{T} - \mathbf{t}
$$

the term $ \mathbf{w}\mathbf{x}^\text{T} $ will grow arbitrarily large (due to $ \mathbf{x} $ being non-normalised) and as a result the weights will grow really large after each training. 

Our `train` function should take the following parameters:

```
inputs : the inputs to train
targets : the targets we are trying to match
eta : the learning rate
activation : the activation function (such as sgn)
batch : whether to train in batch mode or not
```


```python
def train(inputs, targets, eta=0.01, activation=lambda x: x, batch=True, epochs=200):
    bias = ones((1, inputs.shape[1])) # create a 1xn vector of 1s as our bias
    input_min = ndarray.min(inputs)
    input_range = ndarray.max(inputs) - input_min
    normalised_inputs = 2 * (((inputs - input_min) / input_range) - 0.5) # we normalize the inputs between -1 and 1
    X = vstack((normalised_inputs, bias)) # add bias to the inputs, resulting in a 3xn matrix

    weights = random.randn(1, X.shape[0]) # create a 1x3 vector of random weights
    error = zeros(epochs) # create a vector to store error computed at each epoch
    
    if batch:    
        for i in range(epochs):
            err = (activation(dot(weights, X)) - targets)
            deltaW = -eta * dot(err, X.T)
            weights = weights + deltaW # update weights
            error[i] = mean(power(err, 2)) # calculate mean square error for each epoch
    else:
        for i in range(epochs):
            for j in range(X.shape[1]):
                err = (activation(dot(weights, X[:,j])) - targets[j])
                deltaW = -eta * err * X[:,j]
                weights = weights + deltaW # update weights
            
            err = (activation(dot(weights, X)) - targets)
            error[i] = mean(power(err, 2)) # calculate mean square error for each epoch
    return weights, error

W, err = train(data[0:2,:], data[2,:], batch=False, epochs=20)
print(err[-1:])
subplot(1, 2, 1)
plot(err)
subplot(1, 2, 2)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
x = linspace(min(data[0,:]), max(data[0,:]), 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary
plot(x, boundary)
ylim((min(data[1,:]) - 5, max(data[1,:]) + 5))
```

#### Perceptron learning batch mode


```python
W, err = train(data[0:2,:], data[2,:], activation=sign, batch=True, epochs=20)
subplot(1, 2, 1)
plot(err)
print(err[-1:])

subplot(1, 2, 2)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
x = linspace(min(data[0,:]), max(data[0,:]), 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary
plot(x, boundary)
ylim((min(data[1,:]) - 5, max(data[1,:]) + 5))
```

#### Perceptron learning sequential mode


```python
W, err = train(data[0:2,:], data[2,:], activation=sign, batch=False, epochs=50)
subplot(1, 2, 1)
plot(err)
print(err[-1:])

subplot(1, 2, 2)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
x = linspace(min(data[0,:]), max(data[0,:]), 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary
plot(x, boundary)
ylim((min(data[1,:]) - 5, max(data[1,:]) + 5))
```

#### Delta Rule batch mode


```python
W, err = train(data[0:2,:], data[2,:])
plot(err)
```

Interestingly we can see that the error increases after each epochs. This is possibly caused by a too-large learning rate which.


```python
W, err = train(data[0:2,:], data[2,:], eta=0.001)
print(err[-1:])
subplot(1, 2, 1)
plot(err)
subplot(1, 2, 2)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
x = linspace(-20, 20, 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary
plot(x, boundary)
```

#### Delta Rule sequential mode


```python
W, err = train(data[0:2,:], data[2,:], batch=False, epochs=50)
print(err[-1:])
subplot(1, 2, 1)
plot(err)
subplot(1, 2, 2)
scatter(data[0,:], data[1,:], c=data[2,:], cmap=matplotlib.colors.ListedColormap(colors))
x = linspace(-20, 20, 2) # generate two values between -2 and 2
boundary = -(W[0,1] / W[0,0]) * x - (W[0,2] / W[0,0]) # calculate the decision boundary
plot(x, boundary)
```


```python

```

451278bc81bf6eabb06bdc78f8e184388ce0ef6c
ipynb
Jupyter Notebook
Lab1/Single_Perceptron.ipynb
shachibista/ANN_Labs
32870aeef0164414a369c5761118195e6243c944
MIT
Lab1/Single_Perceptron.ipynb
shachibista/ANN_Labs
32870aeef0164414a369c5761118195e6243c944
MIT
Lab1/Single_Perceptron.ipynb
shachibista/ANN_Labs
32870aeef0164414a369c5761118195e6243c944
MIT
Qwen/Qwen-72B
1. YES
2. YES