# Assignment 1

The goal of this assignment is to supply you with machine learning models and algorithms. In this notebook, we will cover linear and nonlinear models, the concept of loss functions and some optimization techniques. All mathematical operations should be implemented in **NumPy** only. 


## Table of contents
* [1. Logistic Regression](#1.-Logistic-Regression)
    * [1.1 Linear Mapping](#1.1-Linear-Mapping)
    * [1.2 Sigmoid](#1.2-Sigmoid)
    * [1.3 Negative Log Likelihood](#1.3-Negative-Log-Likelihood)
    * [1.4 Model](#1.4-Model)
    * [1.5 Simple Experiment](#1.5-Simple-Experiment)
* [2. Decision Tree](#2.-Decision-Tree)
    * [2.1 Gini Index & Data Split](#2.1-Gini-Index-&-Data-Split)
    * [2.2 Terminal Node](#2.2-Terminal-Node)
    * [2.3 Build the Decision Tree](#2.3-Build-the-Decision-Tree)
* [3. Experiments](#3.-Experiments)
    * [3.1 Decision Tree for Heart Disease Prediction](#3.1-Decision-Tree-for-Heart-Disease-Prediction) 
    * [3.2 Logistic Regression for Heart Disease Prediction](#3.2-Logistic-Regression-for-Heart-Disease-Prediction)

### Note
Some of the concepts below have not (yet) been discussed during the lecture. These will be discussed further during the next lectures. 

### Before you begin

To check whether the code you've written is correct, we'll use **automark**. For this, we created for each of you an account with the username being your student number. 


```python
import automark as am

# fill in you student number as your username
username = '13372734'

# to check your progress, you can run this function
am.get_progress(username)
```

    ---------------------------------------------
    | Jakub Miksa                               |
    | jakub.miksa@student.uva.nl                |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | not attempted  |
    | tree_split_data_right    | not attempted  |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------


So far all your tests are 'not attempted'. At the end of this notebook you'll need to have completed all test. The output of `am.get_progress(username)` should at least match the example below. However, we encourage you to take a shot at the 'not attempted' tests!

```
---------------------------------------------
| Your name / student number                |
| your_email@your_domain.whatever           |
---------------------------------------------
| linear_forward           | not attempted  |
| linear_grad_W            | not attempted  |
| linear_grad_b            | not attempted  |
| nll_forward              | not attempted  |
| nll_grad_input           | not attempted  |
| sigmoid_forward          | not attempted  |
| sigmoid_grad_input       | not attempted  |
| tree_data_split_left     | not attempted  |
| tree_data_split_right    | not attempted  |
| tree_gini_index          | not attempted  |
| tree_to_terminal         | not attempted  |
---------------------------------------------
```


```python
from __future__ import print_function, absolute_import, division # You don't need to know what this is. 
import numpy as np # this imports numpy, which is used for vector- and matrix calculations
```

This notebook makes use of **classes** and their **instances** that we have already implemented for you. It allows us to write less code and make it more readable. If you are interested in it, here are some useful links:
* The official [documentation](https://docs.python.org/3/tutorial/classes.html) 
* Video by *sentdex*: [Object Oriented Programming Introduction](https://www.youtube.com/watch?v=ekA6hvk-8H8)
* Antipatterns in OOP: [Stop Writing Classes](https://www.youtube.com/watch?v=o9pEzgHorH0)

# 1. Logistic Regression

We start with a very simple algorithm called **Logistic Regression**. It is a generalized linear model for 2-class classification.
It can be generalized to the case of many classes and to non-linear cases as well. However, here we consider only the simplest case. 

Let us consider a data with 2 classes. Class 0 and class 1. For a given test sample, logistic regression returns a value from $[0, 1]$ which is interpreted as a probability of belonging to class 1. The set of points for which the prediction is $0.5$ is called a *decision boundary*. It is a line on a plane or a hyper-plane in a space.



Logistic regression has two trainable parameters: a weight $W$ and a bias $b$. For a vector of features $X$, the prediction of logistic regression is given by

$$
f(X) = \frac{1}{1 + \exp(-[XW + b])} = \sigma(h(X))
$$
where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ and $h(X)=XW + b$.

Parameters $W$ and $b$ are fitted by maximizing the log-likelihood (or minimizing the negative log-likelihood) of the model on the training data. For a training subset $\{X_j, Y_j\}_{j=1}^N$ the normalized negative log likelihood (NLL) is given by 

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

There are different ways of fitting this model. In this assignment we consider Logistic Regression as a one-layer neural network. We use the following algorithm for the **forward** pass:

1. Linear mapping: $h=XW + b$
2. Sigmoid activation function: $f=\sigma(h)$
3. Calculation of NLL: $\mathcal{L} = -\frac{1}{N}\sum_j \Big[ Y_j\log f_j + (1-Y_j)\log(1-f_j)\Big]$

In order to fit $W$ and $b$ we perform Gradient Descent ([GD](https://en.wikipedia.org/wiki/Gradient_descent)). We choose a small learning rate $\gamma$ and after each computation of forward pass, we update the parameters 

$$W_{\text{new}} = W_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial W}$$

$$b_{\text{new}} = b_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial b}$$

We use Backpropagation method ([BP](https://en.wikipedia.org/wiki/Backpropagation)) to calculate the partial derivatives of the loss function with respect to the parameters of the model.

$$
\frac{\partial\mathcal{L}}{\partial W} = 
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial W} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial W}
$$

$$
\frac{\partial\mathcal{L}}{\partial b} = 
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial b} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial b}
$$

## 1.1 Linear Mapping
First of all, you need to implement the forward pass of a linear mapping:
$$
h(X) = XW +b
$$

**Note**: here we use `n_out` as the dimensionality of the output. For logisitc regression `n_out = 1`. However, we will work with cases of `n_out > 1` in next assignments. You will **pass** the current assignment even if your implementation works only in case `n_out = 1`. If your implementation works for the cases of `n_out > 1` then you will not have to modify your method next week. All **numpy** operations are generic. It is recommended to use numpy when is it possible.


```python
def linear_forward(x_input, W, b):
    """Perform the mapping of the input
    # Arguments
        x_input: input of the linear function - np.array of size `(n_objects, n_in)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the output of the linear function 
        np.array of size `(n_objects, n_out)`
    """
    #################
    ### YOUR CODE ###
    #################
    output = np.dot(x_input,W) + b
    return output
```

Let's check your first function. We set the matrices $X, W, b$:
$$
X = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix} \quad
W = \begin{bmatrix}
4 \\
2 \\
\end{bmatrix} \quad
b = \begin{bmatrix}
3 \\
\end{bmatrix}
$$

And then compute 
$$
XW = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix}
\begin{bmatrix}
4 \\
2 \\
\end{bmatrix} =
\begin{bmatrix}
2 \\
-4 \\
6 \\
\end{bmatrix} \\
XW + b = 
\begin{bmatrix}
5 \\
-1 \\
9 \\
\end{bmatrix} 
$$


```python
X_test = np.array([[1, -1],
                   [-1, 0],
                   [1, 1]])

W_test = np.array([[4],
                   [2]])

b_test = np.array([3])

h_test = linear_forward(X_test, W_test, b_test)
print(h_test)
```

    [[ 5]
     [-1]
     [ 9]]



```python
am.test_student_function(username, linear_forward, ['x_input', 'W', 'b'])
```

    Running local tests...
    answer:  [[36.18722714]
     [18.21546223]]
    out:  [[36.18722714]
     [18.21546223]]
    answer:  [[ -99.99895651]
     [-122.23717235]]
    out:  [[ -99.99895651]
     [-122.23717235]]
    answer:  [[ -9.7398563 ]
     [-16.97097483]
     [ 36.71511796]
     [ 26.14365911]
     [-30.60250802]
     [-95.5183084 ]]
    out:  [[ -9.7398563 ]
     [-16.97097483]
     [ 36.71511796]
     [ 26.14365911]
     [-30.60250802]
     [-95.5183084 ]]
    answer:  [[32.43900919]
     [24.32982298]]
    out:  [[32.43900919]
     [24.32982298]]
    answer:  [[17.5421543 ]
     [ 9.37975723]]
    out:  [[17.5421543 ]
     [ 9.37975723]]
    answer:  [[  79.6182454 ]
     [  -8.22356959]
     [  65.50030757]
     [  66.34544673]
     [-165.20150296]]
    out:  [[  79.6182454 ]
     [  -8.22356959]
     [  65.50030757]
     [  66.34544673]
     [-165.20150296]]
    answer:  [[ 80.42907351]
     [ 19.41738223]
     [ 44.11423277]
     [-27.87643685]
     [104.47353246]
     [-44.09783106]]
    out:  [[ 80.42907351]
     [ 19.41738223]
     [ 44.11423277]
     [-27.87643685]
     [104.47353246]
     [-44.09783106]]
    answer:  [[ 3.39406416]
     [22.4063193 ]
     [29.9194296 ]]
    out:  [[ 3.39406416]
     [22.4063193 ]
     [29.9194296 ]]
    answer:  [[-19.55589635]
     [ 14.96421104]
     [ 19.13580666]
     [ -0.7356672 ]]
    out:  [[-19.55589635]
     [ 14.96421104]
     [ 19.13580666]
     [ -0.7356672 ]]
    answer:  [[-31.95603493]]
    out:  [[-31.95603493]]
    answer:  [[ 59.90055127]
     [ 18.85639502]
     [-50.75080043]
     [-53.59230001]
     [ 51.20926734]
     [-16.08529156]]
    out:  [[ 59.90055127]
     [ 18.85639502]
     [-50.75080043]
     [-53.59230001]
     [ 51.20926734]
     [-16.08529156]]
    answer:  [[ 22.88475183]
     [  3.08320411]
     [  7.89054068]
     [-16.48196553]]
    out:  [[ 22.88475183]
     [  3.08320411]
     [  7.89054068]
     [-16.48196553]]
    answer:  [[3.80774067]
     [4.13799583]]
    out:  [[3.80774067]
     [4.13799583]]
    answer:  [[  45.67662723]
     [  55.20087552]
     [-134.2201392 ]
     [  92.5193568 ]]
    out:  [[  45.67662723]
     [  55.20087552]
     [-134.2201392 ]
     [  92.5193568 ]]
    answer:  [[-34.50415168]
     [ -4.75496166]
     [ 74.19158257]]
    out:  [[-34.50415168]
     [ -4.75496166]
     [ 74.19158257]]
    answer:  [[ 12.88288431]
     [ 82.18805061]
     [124.83849769]]
    out:  [[ 12.88288431]
     [ 82.18805061]
     [124.83849769]]
    answer:  [[-119.15482447]
     [  73.92802082]]
    out:  [[-119.15482447]
     [  73.92802082]]
    answer:  [[ 54.73125204]
     [-26.50633751]
     [  1.26780262]
     [  4.62216939]]
    out:  [[ 54.73125204]
     [-26.50633751]
     [  1.26780262]
     [  4.62216939]]
    answer:  [[ -25.0643021 ]
     [  15.13191336]
     [  34.62868869]
     [ -90.30966198]
     [-191.07346043]
     [ 117.33167738]]
    out:  [[ -25.0643021 ]
     [  15.13191336]
     [  34.62868869]
     [ -90.30966198]
     [-191.07346043]
     [ 117.33167738]]
    answer:  [[26.13370094]
     [99.70084656]]
    out:  [[26.13370094]
     [99.70084656]]
    answer:  [[-126.02861859]
     [  87.41843676]
     [  30.3097599 ]]
    out:  [[-126.02861859]
     [  87.41843676]
     [  30.3097599 ]]
    answer:  [[ 40.57359516]
     [ 37.95346978]
     [ -2.56658096]
     [-44.43466575]]
    out:  [[ 40.57359516]
     [ 37.95346978]
     [ -2.56658096]
     [-44.43466575]]
    answer:  [[-32.97870396]]
    out:  [[-32.97870396]]
    answer:  [[-1.34941279]
     [17.09599619]]
    out:  [[-1.34941279]
     [17.09599619]]
    answer:  [[70.6374726]]
    out:  [[70.6374726]]
    answer:  [[ 80.36330001]
     [  0.83715622]
     [-37.92818129]
     [ 21.8618302 ]
     [ 13.53662798]]
    out:  [[ 80.36330001]
     [  0.83715622]
     [-37.92818129]
     [ 21.8618302 ]
     [ 13.53662798]]
    answer:  [[  8.9156438 ]
     [-16.63453777]
     [ 86.50025367]]
    out:  [[  8.9156438 ]
     [-16.63453777]
     [ 86.50025367]]
    answer:  [[ 12.01247803]
     [-28.69987566]
     [-50.94120916]
     [ -9.1652669 ]
     [ -2.10956696]
     [ 42.06592895]]
    out:  [[ 12.01247803]
     [-28.69987566]
     [-50.94120916]
     [ -9.1652669 ]
     [ -2.10956696]
     [ 42.06592895]]
    answer:  [[-16.3076366 ]
     [ -2.42947767]
     [ 53.86542027]
     [ 39.19930732]
     [ 26.71059504]
     [112.8661556 ]]
    out:  [[-16.3076366 ]
     [ -2.42947767]
     [ 53.86542027]
     [ 39.19930732]
     [ 26.71059504]
     [112.8661556 ]]
    answer:  [[ -88.9016559 ]
     [-108.85952711]]
    out:  [[ -88.9016559 ]
     [-108.85952711]]
    answer:  [[ -9.58222276]
     [149.74662738]
     [-42.80068571]
     [ 14.75414008]
     [-59.5536518 ]]
    out:  [[ -9.58222276]
     [149.74662738]
     [-42.80068571]
     [ 14.75414008]
     [-59.5536518 ]]
    answer:  [[-11.86751433]
     [-14.79927049]]
    out:  [[-11.86751433]
     [-14.79927049]]
    answer:  [[48.56116745]]
    out:  [[48.56116745]]
    answer:  [[23.67338069]]
    out:  [[23.67338069]]
    answer:  [[23.83135961]
     [31.487699  ]]
    out:  [[23.83135961]
     [31.487699  ]]
    answer:  [[ -68.45333931]
     [-190.69893694]
     [ -83.93346386]
     [  59.00562504]
     [-170.4801866 ]]
    out:  [[ -68.45333931]
     [-190.69893694]
     [ -83.93346386]
     [  59.00562504]
     [-170.4801866 ]]
    answer:  [[ 18.56993448]
     [-31.85687342]
     [-65.65812974]
     [-20.1555314 ]]
    out:  [[ 18.56993448]
     [-31.85687342]
     [-65.65812974]
     [-20.1555314 ]]
    answer:  [[ 70.50616607]
     [ 16.9314673 ]
     [ 66.58029864]
     [-50.28092279]
     [ 11.42141314]
     [ 26.93924128]]
    out:  [[ 70.50616607]
     [ 16.9314673 ]
     [ 66.58029864]
     [-50.28092279]
     [ 11.42141314]
     [ 26.93924128]]
    answer:  [[-46.8032183]]
    out:  [[-46.8032183]]
    answer:  [[ 43.27807153]
     [ 68.36725641]
     [-31.5487195 ]
     [  0.4043758 ]
     [-15.20882852]
     [ -9.56903696]]
    out:  [[ 43.27807153]
     [ 68.36725641]
     [-31.5487195 ]
     [  0.4043758 ]
     [-15.20882852]
     [ -9.56903696]]
    answer:  [[82.12640298]]
    out:  [[82.12640298]]
    answer:  [[-14.01706042]
     [ 50.87169936]
     [ -3.75934464]
     [-25.59762101]
     [-36.17022721]]
    out:  [[-14.01706042]
     [ 50.87169936]
     [ -3.75934464]
     [-25.59762101]
     [-36.17022721]]
    answer:  [[-48.7229003 ]
     [-17.55803124]
     [-53.77224134]
     [  2.38422181]
     [ 24.16174928]]
    out:  [[-48.7229003 ]
     [-17.55803124]
     [-53.77224134]
     [  2.38422181]
     [ 24.16174928]]
    answer:  [[ 82.404008  ]
     [160.73521021]]
    out:  [[ 82.404008  ]
     [160.73521021]]
    answer:  [[ 56.996151  ]
     [-10.22002439]
     [-58.64424325]
     [ 81.69340959]
     [-25.99745502]]
    out:  [[ 56.996151  ]
     [-10.22002439]
     [-58.64424325]
     [ 81.69340959]
     [-25.99745502]]
    answer:  [[ 13.53759064]
     [ 14.96286634]
     [  6.83387994]
     [-38.88335628]]
    out:  [[ 13.53759064]
     [ 14.96286634]
     [  6.83387994]
     [-38.88335628]]
    answer:  [[ -4.22845226]
     [  5.50946813]
     [ -0.51923427]
     [-34.46568733]
     [ 19.75170406]
     [  8.9369081 ]]
    out:  [[ -4.22845226]
     [  5.50946813]
     [ -0.51923427]
     [-34.46568733]
     [ 19.75170406]
     [  8.9369081 ]]
    answer:  [[ 45.88936147]
     [  3.94609269]
     [ -5.67755552]
     [-43.36611011]
     [ 27.04834049]
     [ -5.15748913]]
    out:  [[ 45.88936147]
     [  3.94609269]
     [ -5.67755552]
     [-43.36611011]
     [ 27.04834049]
     [ -5.15748913]]
    answer:  [[-39.99720963]]
    out:  [[-39.99720963]]
    answer:  [[-70.20370491]
     [-29.80754561]]
    out:  [[-70.20370491]
     [-29.80754561]]
    answer:  [[   7.08698368]
     [  -6.16872731]
     [ -45.54838137]
     [-126.60475254]
     [ -46.30063323]]
    out:  [[   7.08698368]
     [  -6.16872731]
     [ -45.54838137]
     [-126.60475254]
     [ -46.30063323]]
    answer:  [[107.02114787]
     [ 32.16301096]]
    out:  [[107.02114787]
     [ 32.16301096]]
    answer:  [[ 38.87126059]
     [-16.72532817]
     [125.41652689]
     [ 56.1216772 ]]
    out:  [[ 38.87126059]
     [-16.72532817]
     [125.41652689]
     [ 56.1216772 ]]
    answer:  [[100.62509915]
     [146.79816362]
     [ 17.53826395]]
    out:  [[100.62509915]
     [146.79816362]
     [ 17.53826395]]
    answer:  [[ 40.3677585 ]
     [ 21.32485563]
     [-20.60395558]
     [ 85.35579702]
     [  9.25869747]
     [ 68.10236856]]
    out:  [[ 40.3677585 ]
     [ 21.32485563]
     [-20.60395558]
     [ 85.35579702]
     [  9.25869747]
     [ 68.10236856]]
    answer:  [[-40.13321323]
     [ -7.31052887]
     [-90.26444308]]
    out:  [[-40.13321323]
     [ -7.31052887]
     [-90.26444308]]
    answer:  [[-51.52983232]
     [ -6.03156653]
     [  0.66032489]
     [-50.812921  ]
     [ 16.06728859]]
    out:  [[-51.52983232]
     [ -6.03156653]
     [  0.66032489]
     [-50.812921  ]
     [ 16.06728859]]
    answer:  [[-111.06847432]
     [  -1.56474617]
     [  32.93618567]
     [  20.32254981]
     [  -0.70303137]
     [ -81.97112417]]
    out:  [[-111.06847432]
     [  -1.56474617]
     [  32.93618567]
     [  20.32254981]
     [  -0.70303137]
     [ -81.97112417]]
    answer:  [[ 90.64893445]
     [-59.17359047]
     [ 58.5600365 ]]
    out:  [[ 90.64893445]
     [-59.17359047]
     [ 58.5600365 ]]
    answer:  [[-56.96022424]
     [ 40.93821447]
     [-64.68087709]
     [-31.85609599]]
    out:  [[-56.96022424]
     [ 40.93821447]
     [-64.68087709]
     [-31.85609599]]
    answer:  [[  13.64585157]
     [ -36.45321576]
     [ -60.10027323]
     [-112.69564144]]
    out:  [[  13.64585157]
     [ -36.45321576]
     [ -60.10027323]
     [-112.69564144]]
    answer:  [[128.5426273 ]
     [-42.42034847]
     [ 20.99980663]
     [-43.58104795]]
    out:  [[128.5426273 ]
     [-42.42034847]
     [ 20.99980663]
     [-43.58104795]]
    answer:  [[-40.05347695]
     [-63.97010678]
     [ 25.19162246]
     [-10.80073771]
     [-44.13621803]]
    out:  [[-40.05347695]
     [-63.97010678]
     [ 25.19162246]
     [-10.80073771]
     [-44.13621803]]
    answer:  [[ 111.44886395]
     [  62.1380012 ]
     [ -21.26559513]
     [ -30.64010537]
     [-181.39847848]
     [ -15.1361745 ]]
    out:  [[ 111.44886395]
     [  62.1380012 ]
     [ -21.26559513]
     [ -30.64010537]
     [-181.39847848]
     [ -15.1361745 ]]
    answer:  [[  3.12457614]
     [-47.15363319]
     [ 45.38352129]
     [-64.28429548]]
    out:  [[  3.12457614]
     [-47.15363319]
     [ 45.38352129]
     [-64.28429548]]
    answer:  [[-21.08164661]
     [ 26.01725366]]
    out:  [[-21.08164661]
     [ 26.01725366]]
    answer:  [[ 91.4070133 ]
     [ 77.34260496]
     [-16.30946121]
     [-12.14916642]
     [ 38.37402577]
     [  3.35003843]]
    out:  [[ 91.4070133 ]
     [ 77.34260496]
     [-16.30946121]
     [-12.14916642]
     [ 38.37402577]
     [  3.35003843]]
    answer:  [[ 26.2087638 ]
     [-53.32387641]
     [ 87.28194337]
     [-26.12306796]
     [ 86.98683493]]
    out:  [[ 26.2087638 ]
     [-53.32387641]
     [ 87.28194337]
     [-26.12306796]
     [ 86.98683493]]
    answer:  [[-40.00215565]
     [  6.02230084]]
    out:  [[-40.00215565]
     [  6.02230084]]
    answer:  [[-56.76870442]
     [ -7.92107947]]
    out:  [[-56.76870442]
     [ -7.92107947]]
    answer:  [[-7.38814405]
     [35.71179849]
     [25.27710542]]
    out:  [[-7.38814405]
     [35.71179849]
     [25.27710542]]
    answer:  [[ 119.27765486]
     [  24.10363955]
     [ -38.16347368]
     [ 130.55428762]
     [-156.20491609]
     [  31.1076515 ]]
    out:  [[ 119.27765486]
     [  24.10363955]
     [ -38.16347368]
     [ 130.55428762]
     [-156.20491609]
     [  31.1076515 ]]
    answer:  [[24.55430769]]
    out:  [[24.55430769]]
    answer:  [[-44.92940658]]
    out:  [[-44.92940658]]
    answer:  [[-17.90128571]
     [  9.78614055]
     [-27.69313846]
     [-38.8131829 ]]
    out:  [[-17.90128571]
     [  9.78614055]
     [-27.69313846]
     [-38.8131829 ]]
    answer:  [[-51.59411059]]
    out:  [[-51.59411059]]
    answer:  [[-65.85075227]]
    out:  [[-65.85075227]]
    answer:  [[ 15.67690351]
     [  1.36204812]
     [ 15.55122816]
     [ 17.57315262]
     [  9.34664861]
     [-13.72843926]]
    out:  [[ 15.67690351]
     [  1.36204812]
     [ 15.55122816]
     [ 17.57315262]
     [  9.34664861]
     [-13.72843926]]
    answer:  [[-0.86375657]]
    out:  [[-0.86375657]]
    answer:  [[-30.21237776]
     [-31.58388817]
     [ 11.74383485]
     [-29.47659362]
     [-24.04404196]
     [ 51.55230616]]
    out:  [[-30.21237776]
     [-31.58388817]
     [ 11.74383485]
     [-29.47659362]
     [-24.04404196]
     [ 51.55230616]]
    answer:  [[ -55.83316518]
     [ -69.293224  ]
     [  98.70031924]
     [  56.57730699]
     [  31.94247044]
     [-104.82925385]]
    out:  [[ -55.83316518]
     [ -69.293224  ]
     [  98.70031924]
     [  56.57730699]
     [  31.94247044]
     [-104.82925385]]
    answer:  [[ 0.54873855]
     [35.71972839]]
    out:  [[ 0.54873855]
     [35.71972839]]
    answer:  [[-92.85678206]]
    out:  [[-92.85678206]]
    answer:  [[ 15.96526502]
     [-60.40466765]
     [ -5.34511418]
     [ 38.05145976]
     [-26.00319461]
     [ 11.31497374]]
    out:  [[ 15.96526502]
     [-60.40466765]
     [ -5.34511418]
     [ 38.05145976]
     [-26.00319461]
     [ 11.31497374]]
    answer:  [[-109.94120491]
     [ -59.85566336]]
    out:  [[-109.94120491]
     [ -59.85566336]]
    answer:  [[  -6.92379277]
     [  58.89928515]
     [  66.1682408 ]
     [-118.33116495]
     [  26.27183054]]
    out:  [[  -6.92379277]
     [  58.89928515]
     [  66.1682408 ]
     [-118.33116495]
     [  26.27183054]]
    answer:  [[-21.86190814]]
    out:  [[-21.86190814]]
    answer:  [[ -9.35562429]
     [ 45.18937739]
     [ 22.48723601]
     [-35.98218465]
     [ -6.49605882]]
    out:  [[ -9.35562429]
     [ 45.18937739]
     [ 22.48723601]
     [-35.98218465]
     [ -6.49605882]]
    answer:  [[-17.79814908]
     [-43.15925297]
     [-91.44637216]]
    out:  [[-17.79814908]
     [-43.15925297]
     [-91.44637216]]
    answer:  [[-26.70033742]
     [-36.37143196]
     [-45.87658607]
     [ -7.55617598]
     [-22.80888872]
     [-57.29103509]]
    out:  [[-26.70033742]
     [-36.37143196]
     [-45.87658607]
     [ -7.55617598]
     [-22.80888872]
     [-57.29103509]]
    answer:  [[175.86963575]]
    out:  [[175.86963575]]
    answer:  [[97.5127206 ]
     [15.50171833]]
    out:  [[97.5127206 ]
     [15.50171833]]
    answer:  [[ 13.24416027]
     [ 37.80492351]
     [-23.95257264]
     [-64.08753256]
     [-76.1633833 ]]
    out:  [[ 13.24416027]
     [ 37.80492351]
     [-23.95257264]
     [-64.08753256]
     [-76.1633833 ]]
    answer:  [[-2.72701014]
     [-1.98838646]
     [-1.68567376]
     [-4.35188336]
     [-2.82301516]]
    out:  [[-2.72701014]
     [-1.98838646]
     [-1.68567376]
     [-4.35188336]
     [-2.82301516]]
    answer:  [[-20.61746998]
     [ 78.74548792]
     [ 11.99344604]
     [114.69145158]
     [ 69.24961794]]
    out:  [[-20.61746998]
     [ 78.74548792]
     [ 11.99344604]
     [114.69145158]
     [ 69.24961794]]
    answer:  [[-60.78372766]
     [ -2.69446866]
     [ 32.74935683]]
    out:  [[-60.78372766]
     [ -2.69446866]
     [ 32.74935683]]
    answer:  [[-35.92867211]
     [-50.85766359]
     [-35.63817166]
     [ 89.02674033]]
    out:  [[-35.92867211]
     [-50.85766359]
     [-35.63817166]
     [ 89.02674033]]
    answer:  [[ 70.70188019]
     [ -9.34219218]
     [ 33.77498703]
     [165.26654595]
     [ 83.01694231]]
    out:  [[ 70.70188019]
     [ -9.34219218]
     [ 33.77498703]
     [165.26654595]
     [ 83.01694231]]
    answer:  [[ 21.24133161]
     [-13.31601921]
     [ 22.62083516]]
    out:  [[ 21.24133161]
     [-13.31601921]
     [ 22.62083516]]
    answer:  [[-18.96580027]
     [ 75.24149996]
     [  2.16031128]
     [-39.10367445]]
    out:  [[-18.96580027]
     [ 75.24149996]
     [  2.16031128]
     [-39.10367445]]
    linear_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!


Now you need to implement the calculation of the partial derivative of the loss function with respect to the parameters of the model. As this expressions are used for the updates of the parameters, we refer to them as gradients.
$$
\frac{\partial \mathcal{L}}{\partial W} = 
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial W} \\
\frac{\partial \mathcal{L}}{\partial b} = 
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial b} \\
$$


```python
def linear_grad_W(x_input, grad_output, W, b):
    """Calculate the partial derivative of 
        the loss with respect to W parameter of the function
        dL / dW = (dL / dh) * (dh / dW)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with 
            respect to the ouput of the dense layer (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss 
        with respect to W parameter of the function
        np.array of size `(n_in, n_out)`
    """ 
    #################
    ### YOUR CODE ###
    #################    
    grad_W = np.dot(x_input.T,grad_output)
    print(grad_W)
    return grad_W
```


```python
am.test_student_function(username, linear_grad_W, ['x_input', 'grad_output', 'W', 'b'])
```

    Running local tests...
    [[-42.58872055]
     [-41.11412973]
     [ 27.77954586]]
    answer:  [[-42.58872055]
     [-41.11412973]
     [ 27.77954586]]
    out:  [[-42.58872055]
     [-41.11412973]
     [ 27.77954586]]
    [[  18.02311308]
     [-235.91383617]
     [  -3.06571139]
     [ 164.78656206]]
    answer:  [[  18.02311308]
     [-235.91383617]
     [  -3.06571139]
     [ 164.78656206]]
    out:  [[  18.02311308]
     [-235.91383617]
     [  -3.06571139]
     [ 164.78656206]]
    [[42.72230582]]
    answer:  [[42.72230582]]
    out:  [[42.72230582]]
    [[-69.42267128]
     [  9.85908252]]
    answer:  [[-69.42267128]
     [  9.85908252]]
    out:  [[-69.42267128]
     [  9.85908252]]
    [[ 30.99466241]
     [ 88.5281123 ]
     [ 28.54676195]
     [-61.36322267]
     [ 31.24849848]
     [-86.67186636]]
    answer:  [[ 30.99466241]
     [ 88.5281123 ]
     [ 28.54676195]
     [-61.36322267]
     [ 31.24849848]
     [-86.67186636]]
    out:  [[ 30.99466241]
     [ 88.5281123 ]
     [ 28.54676195]
     [-61.36322267]
     [ 31.24849848]
     [-86.67186636]]
    [[-37.72321569]
     [  8.38159974]
     [ -9.37411529]
     [-42.9079016 ]]
    answer:  [[-37.72321569]
     [  8.38159974]
     [ -9.37411529]
     [-42.9079016 ]]
    out:  [[-37.72321569]
     [  8.38159974]
     [ -9.37411529]
     [-42.9079016 ]]
    [[  44.10861087]
     [  83.13819074]
     [ -15.11413706]
     [  85.98565319]
     [-113.21570379]
     [  54.76359964]]
    answer:  [[  44.10861087]
     [  83.13819074]
     [ -15.11413706]
     [  85.98565319]
     [-113.21570379]
     [  54.76359964]]
    out:  [[  44.10861087]
     [  83.13819074]
     [ -15.11413706]
     [  85.98565319]
     [-113.21570379]
     [  54.76359964]]
    [[ 25.13930622]
     [-36.83132485]
     [ 49.61347852]
     [  1.86629071]]
    answer:  [[ 25.13930622]
     [-36.83132485]
     [ 49.61347852]
     [  1.86629071]]
    out:  [[ 25.13930622]
     [-36.83132485]
     [ 49.61347852]
     [  1.86629071]]
    [[-13.46690752]
     [ 73.25280961]
     [-60.11491132]
     [103.44837323]]
    answer:  [[-13.46690752]
     [ 73.25280961]
     [-60.11491132]
     [103.44837323]]
    out:  [[-13.46690752]
     [ 73.25280961]
     [-60.11491132]
     [103.44837323]]
    [[-21.90726064]
     [-40.06712974]
     [ 43.41053162]
     [ 64.45967918]
     [-70.8161849 ]]
    answer:  [[-21.90726064]
     [-40.06712974]
     [ 43.41053162]
     [ 64.45967918]
     [-70.8161849 ]]
    out:  [[-21.90726064]
     [-40.06712974]
     [ 43.41053162]
     [ 64.45967918]
     [-70.8161849 ]]
    [[-80.51787022]
     [-35.20177247]
     [-46.46840374]]
    answer:  [[-80.51787022]
     [-35.20177247]
     [-46.46840374]]
    out:  [[-80.51787022]
     [-35.20177247]
     [-46.46840374]]
    [[ -9.53400236]
     [-18.27059975]
     [ 18.02220437]
     [  6.06868586]
     [-26.87751497]]
    answer:  [[ -9.53400236]
     [-18.27059975]
     [ 18.02220437]
     [  6.06868586]
     [-26.87751497]]
    out:  [[ -9.53400236]
     [-18.27059975]
     [ 18.02220437]
     [  6.06868586]
     [-26.87751497]]
    [[ -9.56515333]
     [-50.5589575 ]]
    answer:  [[ -9.56515333]
     [-50.5589575 ]]
    out:  [[ -9.56515333]
     [-50.5589575 ]]
    [[  75.49976964]
     [-104.51907719]]
    answer:  [[  75.49976964]
     [-104.51907719]]
    out:  [[  75.49976964]
     [-104.51907719]]
    [[-92.02789449]
     [-30.05122709]
     [ 66.97133894]
     [ 86.89333781]]
    answer:  [[-92.02789449]
     [-30.05122709]
     [ 66.97133894]
     [ 86.89333781]]
    out:  [[-92.02789449]
     [-30.05122709]
     [ 66.97133894]
     [ 86.89333781]]
    [[  6.71717074]
     [-52.48075605]
     [-44.49307702]
     [  0.40411961]
     [-56.52261011]
     [-29.97030926]]
    answer:  [[  6.71717074]
     [-52.48075605]
     [-44.49307702]
     [  0.40411961]
     [-56.52261011]
     [-29.97030926]]
    out:  [[  6.71717074]
     [-52.48075605]
     [-44.49307702]
     [  0.40411961]
     [-56.52261011]
     [-29.97030926]]
    [[ -6.56788687]
     [-10.44910346]
     [  0.51819564]
     [  3.99914008]
     [  7.49504118]
     [ -8.11809805]]
    answer:  [[ -6.56788687]
     [-10.44910346]
     [  0.51819564]
     [  3.99914008]
     [  7.49504118]
     [ -8.11809805]]
    out:  [[ -6.56788687]
     [-10.44910346]
     [  0.51819564]
     [  3.99914008]
     [  7.49504118]
     [ -8.11809805]]
    [[-30.68520793]
     [  5.15871044]]
    answer:  [[-30.68520793]
     [  5.15871044]]
    out:  [[-30.68520793]
     [  5.15871044]]
    [[ 19.72500005]
     [ 54.61110254]
     [-35.65103528]
     [-57.15322288]]
    answer:  [[ 19.72500005]
     [ 54.61110254]
     [-35.65103528]
     [-57.15322288]]
    out:  [[ 19.72500005]
     [ 54.61110254]
     [-35.65103528]
     [-57.15322288]]
    [[ 70.65050981]
     [-30.29311942]
     [-22.02050565]
     [-30.86341764]]
    answer:  [[ 70.65050981]
     [-30.29311942]
     [-22.02050565]
     [-30.86341764]]
    out:  [[ 70.65050981]
     [-30.29311942]
     [-22.02050565]
     [-30.86341764]]
    [[-61.52636174]
     [  2.69559215]
     [-74.43317451]
     [ 75.9561601 ]
     [-60.73025221]
     [ 67.76685142]]
    answer:  [[-61.52636174]
     [  2.69559215]
     [-74.43317451]
     [ 75.9561601 ]
     [-60.73025221]
     [ 67.76685142]]
    out:  [[-61.52636174]
     [  2.69559215]
     [-74.43317451]
     [ 75.9561601 ]
     [-60.73025221]
     [ 67.76685142]]
    [[-47.96714023]
     [-81.67017479]]
    answer:  [[-47.96714023]
     [-81.67017479]]
    out:  [[-47.96714023]
     [-81.67017479]]
    [[  56.49378062]
     [ -65.82990687]
     [  60.30881188]
     [-144.53451314]
     [  50.24574652]]
    answer:  [[  56.49378062]
     [ -65.82990687]
     [  60.30881188]
     [-144.53451314]
     [  50.24574652]]
    out:  [[  56.49378062]
     [ -65.82990687]
     [  60.30881188]
     [-144.53451314]
     [  50.24574652]]
    [[ 147.96216327]
     [  11.72313531]
     [  92.11861781]
     [-222.56228357]
     [  56.44972703]]
    answer:  [[ 147.96216327]
     [  11.72313531]
     [  92.11861781]
     [-222.56228357]
     [  56.44972703]]
    out:  [[ 147.96216327]
     [  11.72313531]
     [  92.11861781]
     [-222.56228357]
     [  56.44972703]]
    [[  43.03119541]
     [-159.57900449]
     [  -9.42167844]
     [  43.81458803]]
    answer:  [[  43.03119541]
     [-159.57900449]
     [  -9.42167844]
     [  43.81458803]]
    out:  [[  43.03119541]
     [-159.57900449]
     [  -9.42167844]
     [  43.81458803]]
    [[-134.8502261 ]
     [  -3.6782941 ]
     [ -45.07111313]
     [ 117.41814768]]
    answer:  [[-134.8502261 ]
     [  -3.6782941 ]
     [ -45.07111313]
     [ 117.41814768]]
    out:  [[-134.8502261 ]
     [  -3.6782941 ]
     [ -45.07111313]
     [ 117.41814768]]
    [[-65.83664772]
     [-47.25569414]
     [-23.07525845]
     [-70.16890961]]
    answer:  [[-65.83664772]
     [-47.25569414]
     [-23.07525845]
     [-70.16890961]]
    out:  [[-65.83664772]
     [-47.25569414]
     [-23.07525845]
     [-70.16890961]]
    [[  36.39628778]
     [-121.9132598 ]
     [  75.09663635]
     [ -86.15881918]
     [  53.19380051]
     [  24.46924039]]
    answer:  [[  36.39628778]
     [-121.9132598 ]
     [  75.09663635]
     [ -86.15881918]
     [  53.19380051]
     [  24.46924039]]
    out:  [[  36.39628778]
     [-121.9132598 ]
     [  75.09663635]
     [ -86.15881918]
     [  53.19380051]
     [  24.46924039]]
    [[-25.04523921]
     [-45.08753556]
     [-25.92467268]
     [-89.33689626]
     [  9.31382349]
     [  2.79260509]]
    answer:  [[-25.04523921]
     [-45.08753556]
     [-25.92467268]
     [-89.33689626]
     [  9.31382349]
     [  2.79260509]]
    out:  [[-25.04523921]
     [-45.08753556]
     [-25.92467268]
     [-89.33689626]
     [  9.31382349]
     [  2.79260509]]
    [[-4.24675112]]
    answer:  [[-4.24675112]]
    out:  [[-4.24675112]]
    [[ 34.39943535]
     [104.40390405]]
    answer:  [[ 34.39943535]
     [104.40390405]]
    out:  [[ 34.39943535]
     [104.40390405]]
    [[67.7960515 ]
     [19.69769659]]
    answer:  [[67.7960515 ]
     [19.69769659]]
    out:  [[67.7960515 ]
     [19.69769659]]
    [[ 16.04818671]
     [ 45.86036258]
     [ 32.13833542]
     [ 38.43300436]
     [-39.6613166 ]]
    answer:  [[ 16.04818671]
     [ 45.86036258]
     [ 32.13833542]
     [ 38.43300436]
     [-39.6613166 ]]
    out:  [[ 16.04818671]
     [ 45.86036258]
     [ 32.13833542]
     [ 38.43300436]
     [-39.6613166 ]]
    [[-15.55651498]
     [ 13.92870664]
     [-33.52936991]
     [ 33.27980854]]
    answer:  [[-15.55651498]
     [ 13.92870664]
     [-33.52936991]
     [ 33.27980854]]
    out:  [[-15.55651498]
     [ 13.92870664]
     [-33.52936991]
     [ 33.27980854]]
    [[ 12.13046422]
     [-13.51201178]]
    answer:  [[ 12.13046422]
     [-13.51201178]]
    out:  [[ 12.13046422]
     [-13.51201178]]
    [[-85.00343897]]
    answer:  [[-85.00343897]]
    out:  [[-85.00343897]]
    [[ 66.90860277]
     [-17.47897777]
     [  8.4540484 ]]
    answer:  [[ 66.90860277]
     [-17.47897777]
     [  8.4540484 ]]
    out:  [[ 66.90860277]
     [-17.47897777]
     [  8.4540484 ]]
    [[-18.80151852]
     [ 23.08719722]]
    answer:  [[-18.80151852]
     [ 23.08719722]]
    out:  [[-18.80151852]
     [ 23.08719722]]
    [[-2.86645437]
     [-2.02611715]]
    answer:  [[-2.86645437]
     [-2.02611715]]
    out:  [[-2.86645437]
     [-2.02611715]]
    [[  99.54067767]
     [ -64.83898033]
     [-182.16732756]]
    answer:  [[  99.54067767]
     [ -64.83898033]
     [-182.16732756]]
    out:  [[  99.54067767]
     [ -64.83898033]
     [-182.16732756]]
    [[107.13642467]
     [ 68.63834146]]
    answer:  [[107.13642467]
     [ 68.63834146]]
    out:  [[107.13642467]
     [ 68.63834146]]
    [[57.48220948]
     [14.66228014]
     [40.5847695 ]
     [38.23892326]
     [78.2512095 ]
     [34.7649858 ]]
    answer:  [[57.48220948]
     [14.66228014]
     [40.5847695 ]
     [38.23892326]
     [78.2512095 ]
     [34.7649858 ]]
    out:  [[57.48220948]
     [14.66228014]
     [40.5847695 ]
     [38.23892326]
     [78.2512095 ]
     [34.7649858 ]]
    [[-32.51576136]
     [-41.03848964]
     [108.31079278]
     [ -4.36238666]]
    answer:  [[-32.51576136]
     [-41.03848964]
     [108.31079278]
     [ -4.36238666]]
    out:  [[-32.51576136]
     [-41.03848964]
     [108.31079278]
     [ -4.36238666]]
    [[-27.72665598]
     [ 44.97513332]
     [ 64.64445213]
     [-19.7476551 ]]
    answer:  [[-27.72665598]
     [ 44.97513332]
     [ 64.64445213]
     [-19.7476551 ]]
    out:  [[-27.72665598]
     [ 44.97513332]
     [ 64.64445213]
     [-19.7476551 ]]
    [[-30.25377365]
     [-31.04053481]
     [ 44.14691997]
     [ 39.18860562]
     [-93.48965679]]
    answer:  [[-30.25377365]
     [-31.04053481]
     [ 44.14691997]
     [ 39.18860562]
     [-93.48965679]]
    out:  [[-30.25377365]
     [-31.04053481]
     [ 44.14691997]
     [ 39.18860562]
     [-93.48965679]]
    [[ 84.48139251]
     [ 22.94504848]
     [-86.80871947]
     [-75.71966865]]
    answer:  [[ 84.48139251]
     [ 22.94504848]
     [-86.80871947]
     [-75.71966865]]
    out:  [[ 84.48139251]
     [ 22.94504848]
     [-86.80871947]
     [-75.71966865]]
    [[ 60.16971864]
     [-53.73637825]]
    answer:  [[ 60.16971864]
     [-53.73637825]]
    out:  [[ 60.16971864]
     [-53.73637825]]
    [[-117.80991742]
     [ -22.29044415]
     [  89.1639108 ]
     [  54.26470268]
     [ 119.598676  ]]
    answer:  [[-117.80991742]
     [ -22.29044415]
     [  89.1639108 ]
     [  54.26470268]
     [ 119.598676  ]]
    out:  [[-117.80991742]
     [ -22.29044415]
     [  89.1639108 ]
     [  54.26470268]
     [ 119.598676  ]]
    [[-5.34222083]]
    answer:  [[-5.34222083]]
    out:  [[-5.34222083]]
    [[-72.18402769]
     [-61.57842059]
     [ 46.7088649 ]
     [ 73.60226021]
     [ 30.99712322]]
    answer:  [[-72.18402769]
     [-61.57842059]
     [ 46.7088649 ]
     [ 73.60226021]
     [ 30.99712322]]
    out:  [[-72.18402769]
     [-61.57842059]
     [ 46.7088649 ]
     [ 73.60226021]
     [ 30.99712322]]
    [[  20.00441006]
     [-105.91714547]
     [ -22.4672807 ]
     [ -27.46238193]]
    answer:  [[  20.00441006]
     [-105.91714547]
     [ -22.4672807 ]
     [ -27.46238193]]
    out:  [[  20.00441006]
     [-105.91714547]
     [ -22.4672807 ]
     [ -27.46238193]]
    [[-4.57107153]]
    answer:  [[-4.57107153]]
    out:  [[-4.57107153]]
    [[-165.57296011]
     [-103.85167031]
     [  16.39631406]
     [  -6.51880849]
     [ -98.35986751]]
    answer:  [[-165.57296011]
     [-103.85167031]
     [  16.39631406]
     [  -6.51880849]
     [ -98.35986751]]
    out:  [[-165.57296011]
     [-103.85167031]
     [  16.39631406]
     [  -6.51880849]
     [ -98.35986751]]
    [[-94.6341112 ]
     [ 24.59731237]
     [-56.01874367]]
    answer:  [[-94.6341112 ]
     [ 24.59731237]
     [-56.01874367]]
    out:  [[-94.6341112 ]
     [ 24.59731237]
     [-56.01874367]]
    [[ -6.53984507]
     [-45.12634874]
     [ 41.78341485]
     [ -8.83058222]]
    answer:  [[ -6.53984507]
     [-45.12634874]
     [ 41.78341485]
     [ -8.83058222]]
    out:  [[ -6.53984507]
     [-45.12634874]
     [ 41.78341485]
     [ -8.83058222]]
    [[33.12958922]]
    answer:  [[33.12958922]]
    out:  [[33.12958922]]
    [[ 88.28663458]
     [ -1.3288362 ]
     [ 31.39542944]
     [-26.30076009]
     [-78.8518734 ]]
    answer:  [[ 88.28663458]
     [ -1.3288362 ]
     [ 31.39542944]
     [-26.30076009]
     [-78.8518734 ]]
    out:  [[ 88.28663458]
     [ -1.3288362 ]
     [ 31.39542944]
     [-26.30076009]
     [-78.8518734 ]]
    [[-17.86170204]
     [  5.80665474]]
    answer:  [[-17.86170204]
     [  5.80665474]]
    out:  [[-17.86170204]
     [  5.80665474]]
    [[-115.08819823]
     [   5.54481082]
     [  15.18371781]
     [ -21.6926247 ]
     [  54.03892117]]
    answer:  [[-115.08819823]
     [   5.54481082]
     [  15.18371781]
     [ -21.6926247 ]
     [  54.03892117]]
    out:  [[-115.08819823]
     [   5.54481082]
     [  15.18371781]
     [ -21.6926247 ]
     [  54.03892117]]
    [[-18.65257381]
     [-89.51869568]
     [-42.47344103]
     [-51.43794191]]
    answer:  [[-18.65257381]
     [-89.51869568]
     [-42.47344103]
     [-51.43794191]]
    out:  [[-18.65257381]
     [-89.51869568]
     [-42.47344103]
     [-51.43794191]]
    [[-163.8416404 ]
     [ -11.64045935]
     [ -80.82448227]]
    answer:  [[-163.8416404 ]
     [ -11.64045935]
     [ -80.82448227]]
    out:  [[-163.8416404 ]
     [ -11.64045935]
     [ -80.82448227]]
    [[132.10344286]]
    answer:  [[132.10344286]]
    out:  [[132.10344286]]
    [[ 42.8906504 ]
     [  0.81606313]
     [  9.63728173]
     [ -6.28839561]
     [-25.53479498]]
    answer:  [[ 42.8906504 ]
     [  0.81606313]
     [  9.63728173]
     [ -6.28839561]
     [-25.53479498]]
    out:  [[ 42.8906504 ]
     [  0.81606313]
     [  9.63728173]
     [ -6.28839561]
     [-25.53479498]]
    [[33.67744955]]
    answer:  [[33.67744955]]
    out:  [[33.67744955]]
    [[-99.08445226]
     [  6.97798986]]
    answer:  [[-99.08445226]
     [  6.97798986]]
    out:  [[-99.08445226]
     [  6.97798986]]
    [[-133.91064315]
     [-115.22374588]
     [  76.69764566]
     [  36.42236798]]
    answer:  [[-133.91064315]
     [-115.22374588]
     [  76.69764566]
     [  36.42236798]]
    out:  [[-133.91064315]
     [-115.22374588]
     [  76.69764566]
     [  36.42236798]]
    [[153.00697242]
     [ 40.28435123]
     [-85.74236524]
     [-23.17865583]
     [-85.64105013]
     [ 94.55245348]]
    answer:  [[153.00697242]
     [ 40.28435123]
     [-85.74236524]
     [-23.17865583]
     [-85.64105013]
     [ 94.55245348]]
    out:  [[153.00697242]
     [ 40.28435123]
     [-85.74236524]
     [-23.17865583]
     [-85.64105013]
     [ 94.55245348]]
    [[  -9.75991681]
     [   2.19678946]
     [-119.02334283]
     [   1.08812632]
     [ -95.8128157 ]]
    answer:  [[  -9.75991681]
     [   2.19678946]
     [-119.02334283]
     [   1.08812632]
     [ -95.8128157 ]]
    out:  [[  -9.75991681]
     [   2.19678946]
     [-119.02334283]
     [   1.08812632]
     [ -95.8128157 ]]
    [[-6.24607346]]
    answer:  [[-6.24607346]]
    out:  [[-6.24607346]]
    [[-63.59922781]
     [-94.25945586]]
    answer:  [[-63.59922781]
     [-94.25945586]]
    out:  [[-63.59922781]
     [-94.25945586]]
    [[ -49.35777749]
     [ -23.89267391]
     [  71.64913929]
     [  58.79791995]
     [-158.02243564]
     [ -56.39043824]]
    answer:  [[ -49.35777749]
     [ -23.89267391]
     [  71.64913929]
     [  58.79791995]
     [-158.02243564]
     [ -56.39043824]]
    out:  [[ -49.35777749]
     [ -23.89267391]
     [  71.64913929]
     [  58.79791995]
     [-158.02243564]
     [ -56.39043824]]
    [[-111.30985555]
     [ 124.80817218]
     [-114.98240786]
     [ 132.92454061]
     [ 158.53255762]]
    answer:  [[-111.30985555]
     [ 124.80817218]
     [-114.98240786]
     [ 132.92454061]
     [ 158.53255762]]
    out:  [[-111.30985555]
     [ 124.80817218]
     [-114.98240786]
     [ 132.92454061]
     [ 158.53255762]]
    [[ 32.90218409]
     [ -0.47799157]
     [-25.17365493]
     [-37.02864058]
     [-50.19245129]]
    answer:  [[ 32.90218409]
     [ -0.47799157]
     [-25.17365493]
     [-37.02864058]
     [-50.19245129]]
    out:  [[ 32.90218409]
     [ -0.47799157]
     [-25.17365493]
     [-37.02864058]
     [-50.19245129]]
    [[ 47.76426478]
     [ 30.41630837]
     [ 39.37469934]
     [-35.67462924]
     [ 62.55371086]
     [-63.12256319]]
    answer:  [[ 47.76426478]
     [ 30.41630837]
     [ 39.37469934]
     [-35.67462924]
     [ 62.55371086]
     [-63.12256319]]
    out:  [[ 47.76426478]
     [ 30.41630837]
     [ 39.37469934]
     [-35.67462924]
     [ 62.55371086]
     [-63.12256319]]
    [[-12.94983173]
     [-14.19694018]
     [-16.17935411]
     [ 31.27532494]
     [-16.17329053]]
    answer:  [[-12.94983173]
     [-14.19694018]
     [-16.17935411]
     [ 31.27532494]
     [-16.17329053]]
    out:  [[-12.94983173]
     [-14.19694018]
     [-16.17935411]
     [ 31.27532494]
     [-16.17329053]]
    [[ 159.47920048]
     [ -23.99262206]
     [-117.7123707 ]
     [  76.9577079 ]
     [ -84.000728  ]
     [  60.40649581]]
    answer:  [[ 159.47920048]
     [ -23.99262206]
     [-117.7123707 ]
     [  76.9577079 ]
     [ -84.000728  ]
     [  60.40649581]]
    out:  [[ 159.47920048]
     [ -23.99262206]
     [-117.7123707 ]
     [  76.9577079 ]
     [ -84.000728  ]
     [  60.40649581]]
    [[-33.86190578]
     [ -4.51969535]
     [  8.36406883]
     [  6.59626717]
     [ -9.12137572]]
    answer:  [[-33.86190578]
     [ -4.51969535]
     [  8.36406883]
     [  6.59626717]
     [ -9.12137572]]
    out:  [[-33.86190578]
     [ -4.51969535]
     [  8.36406883]
     [  6.59626717]
     [ -9.12137572]]
    [[  64.92258011]
     [-102.88573334]]
    answer:  [[  64.92258011]
     [-102.88573334]]
    out:  [[  64.92258011]
     [-102.88573334]]
    [[-130.30724882]
     [-153.32224704]
     [  55.45256424]
     [  21.79758526]
     [  81.08617993]]
    answer:  [[-130.30724882]
     [-153.32224704]
     [  55.45256424]
     [  21.79758526]
     [  81.08617993]]
    out:  [[-130.30724882]
     [-153.32224704]
     [  55.45256424]
     [  21.79758526]
     [  81.08617993]]
    [[ 4.44893683]
     [-3.58301881]
     [ 6.96490785]
     [ 1.48148677]
     [-9.93004603]]
    answer:  [[ 4.44893683]
     [-3.58301881]
     [ 6.96490785]
     [ 1.48148677]
     [-9.93004603]]
    out:  [[ 4.44893683]
     [-3.58301881]
     [ 6.96490785]
     [ 1.48148677]
     [-9.93004603]]
    [[-33.08553105]
     [-36.9472321 ]
     [ -7.44847993]]
    answer:  [[-33.08553105]
     [-36.9472321 ]
     [ -7.44847993]]
    out:  [[-33.08553105]
     [-36.9472321 ]
     [ -7.44847993]]
    [[  0.52766309]
     [-10.5207864 ]
     [-48.98269336]
     [ 49.4494929 ]
     [ 49.97340685]]
    answer:  [[  0.52766309]
     [-10.5207864 ]
     [-48.98269336]
     [ 49.4494929 ]
     [ 49.97340685]]
    out:  [[  0.52766309]
     [-10.5207864 ]
     [-48.98269336]
     [ 49.4494929 ]
     [ 49.97340685]]
    [[  0.60970999]
     [-22.43338426]
     [-86.71280835]
     [ 28.28682762]
     [-40.29613507]
     [102.68773678]]
    answer:  [[  0.60970999]
     [-22.43338426]
     [-86.71280835]
     [ 28.28682762]
     [-40.29613507]
     [102.68773678]]
    out:  [[  0.60970999]
     [-22.43338426]
     [-86.71280835]
     [ 28.28682762]
     [-40.29613507]
     [102.68773678]]
    [[ 17.25846921]
     [ 89.83769477]
     [-36.51302935]]
    answer:  [[ 17.25846921]
     [ 89.83769477]
     [-36.51302935]]
    out:  [[ 17.25846921]
     [ 89.83769477]
     [-36.51302935]]
    [[ -3.80608433]
     [ -2.50260648]
     [  9.26715017]
     [-31.54092321]
     [-23.40690954]
     [-30.13615758]]
    answer:  [[ -3.80608433]
     [ -2.50260648]
     [  9.26715017]
     [-31.54092321]
     [-23.40690954]
     [-30.13615758]]
    out:  [[ -3.80608433]
     [ -2.50260648]
     [  9.26715017]
     [-31.54092321]
     [-23.40690954]
     [-30.13615758]]
    [[40.96068162]
     [46.74482293]]
    answer:  [[40.96068162]
     [46.74482293]]
    out:  [[40.96068162]
     [46.74482293]]
    [[-19.04803004]
     [ 11.53138628]
     [ 11.35753176]
     [ -7.24788771]
     [ 36.27728129]]
    answer:  [[-19.04803004]
     [ 11.53138628]
     [ 11.35753176]
     [ -7.24788771]
     [ 36.27728129]]
    out:  [[-19.04803004]
     [ 11.53138628]
     [ 11.35753176]
     [ -7.24788771]
     [ 36.27728129]]
    [[7.90827216]
     [1.44328639]]
    answer:  [[7.90827216]
     [1.44328639]]
    out:  [[7.90827216]
     [1.44328639]]
    [[ 39.3057981 ]
     [-39.5345167 ]
     [-57.90184745]
     [-70.94700976]
     [-46.52337516]]
    answer:  [[ 39.3057981 ]
     [-39.5345167 ]
     [-57.90184745]
     [-70.94700976]
     [-46.52337516]]
    out:  [[ 39.3057981 ]
     [-39.5345167 ]
     [-57.90184745]
     [-70.94700976]
     [-46.52337516]]
    [[ 20.4544198 ]
     [-45.37908089]
     [151.76233264]
     [ 40.43816541]
     [ -0.39923434]
     [-79.15142609]]
    answer:  [[ 20.4544198 ]
     [-45.37908089]
     [151.76233264]
     [ 40.43816541]
     [ -0.39923434]
     [-79.15142609]]
    out:  [[ 20.4544198 ]
     [-45.37908089]
     [151.76233264]
     [ 40.43816541]
     [ -0.39923434]
     [-79.15142609]]
    [[-21.30005523]
     [114.9504853 ]
     [ 18.1447458 ]
     [ 51.42633373]]
    answer:  [[-21.30005523]
     [114.9504853 ]
     [ 18.1447458 ]
     [ 51.42633373]]
    out:  [[-21.30005523]
     [114.9504853 ]
     [ 18.1447458 ]
     [ 51.42633373]]
    [[-5.47476183]
     [25.37547688]
     [73.82015552]]
    answer:  [[-5.47476183]
     [25.37547688]
     [73.82015552]]
    out:  [[-5.47476183]
     [25.37547688]
     [73.82015552]]
    [[-47.89109967]]
    answer:  [[-47.89109967]]
    out:  [[-47.89109967]]
    [[ 35.41666099]
     [-18.58182349]
     [-24.08603873]
     [-41.40116141]]
    answer:  [[ 35.41666099]
     [-18.58182349]
     [-24.08603873]
     [-41.40116141]]
    out:  [[ 35.41666099]
     [-18.58182349]
     [-24.08603873]
     [-41.40116141]]
    [[ 79.03844003]
     [ -3.31969936]
     [ 82.78699454]
     [ 96.37627465]
     [-96.6055837 ]
     [ 53.34233852]]
    answer:  [[ 79.03844003]
     [ -3.31969936]
     [ 82.78699454]
     [ 96.37627465]
     [-96.6055837 ]
     [ 53.34233852]]
    out:  [[ 79.03844003]
     [ -3.31969936]
     [ 82.78699454]
     [ 96.37627465]
     [-96.6055837 ]
     [ 53.34233852]]
    [[ -9.35222802]
     [-18.09376895]
     [ 36.30294886]
     [ -3.20709957]]
    answer:  [[ -9.35222802]
     [-18.09376895]
     [ 36.30294886]
     [ -3.20709957]]
    out:  [[ -9.35222802]
     [-18.09376895]
     [ 36.30294886]
     [ -3.20709957]]
    [[ -57.82459536]
     [  98.30966383]
     [ -15.7147712 ]
     [ 140.93971508]
     [-115.67967437]]
    answer:  [[ -57.82459536]
     [  98.30966383]
     [ -15.7147712 ]
     [ 140.93971508]
     [-115.67967437]]
    out:  [[ -57.82459536]
     [  98.30966383]
     [ -15.7147712 ]
     [ 140.93971508]
     [-115.67967437]]
    [[145.25612264]]
    answer:  [[145.25612264]]
    out:  [[145.25612264]]
    [[-110.25264458]
     [ -40.9963723 ]
     [  30.50628411]
     [ -16.36484   ]
     [ -60.5817182 ]]
    answer:  [[-110.25264458]
     [ -40.9963723 ]
     [  30.50628411]
     [ -16.36484   ]
     [ -60.5817182 ]]
    out:  [[-110.25264458]
     [ -40.9963723 ]
     [  30.50628411]
     [ -16.36484   ]
     [ -60.5817182 ]]
    [[ 81.0956946 ]
     [ 52.18155068]
     [-20.27549477]]
    answer:  [[ 81.0956946 ]
     [ 52.18155068]
     [-20.27549477]]
    out:  [[ 81.0956946 ]
     [ 52.18155068]
     [-20.27549477]]
    linear_grad_W successfully passed local tests
    Running remote test...
    [[ 79.6967553 ]
     [-49.39660447]
     [ 42.14816672]]
    Test was successful. Congratulations!



```python
def linear_grad_b(x_input, grad_output, W, b):
    """Calculate the partial derivative of 
        the loss with respect to b parameter of the function
        dL / db = (dL / dh) * (dh / db)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with 
            respect to the ouput of the linear function (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss 
        with respect to b parameter of the linear function
        np.array of size `(n_out,)`
    """
    #################
    ### YOUR CODE ###
    #################
    grad_b = np.sum(grad_output)
    return grad_b
```


```python
am.test_student_function(username, linear_grad_b, ['x_input', 'grad_output', 'W', 'b'])
```


```python
am.get_progress(username)
```

    ---------------------------------------------
    | Jakub Miksa                               |
    | jakub.miksa@student.uva.nl                |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | not attempted  |
    | nll_grad_input           | not attempted  |
    | sigmoid_forward          | not attempted  |
    | sigmoid_grad_input       | not attempted  |
    | tree_gini_index          | not attempted  |
    | tree_split_data_left     | not attempted  |
    | tree_split_data_right    | not attempted  |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------


## 1.2 Sigmoid
$$
f = \sigma(h) = \frac{1}{1 + e^{-h}} 
$$

Sigmoid function is applied element-wise. It does not change the dimensionality of the tensor and its implementation is shape-agnostic in general.


```python
import math
def sigmoid_forward(x_input):
    """sigmoid nonlinearity
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
    # Output
        the output of relu layer
        np.array of size `(n_objects, n_in)`
    """
    #################
    ### YOUR CODE ###
    #################
    output = 1 / (1 + np.e**(-x_input))
    return output
```


```python
am.test_student_function(username, sigmoid_forward, ['x_input'])
```

    Running local tests...
    answer:  [[2.05268995e-04]
     [9.99785310e-01]
     [8.57070658e-04]]
    out:  [[2.05268995e-04]
     [9.99785310e-01]
     [8.57070658e-04]]
    answer:  [[0.98800511]
     [0.63353511]
     [0.96767821]]
    out:  [[0.98800511]
     [0.63353511]
     [0.96767821]]
    answer:  [[3.72130625e-02]
     [6.84718144e-05]
     [1.96965009e-04]
     [9.99939313e-01]]
    out:  [[3.72130625e-02]
     [6.84718144e-05]
     [1.96965009e-04]
     [9.99939313e-01]]
    answer:  [[0.9995883 ]
     [0.00114112]
     [0.99979863]]
    out:  [[0.9995883 ]
     [0.00114112]
     [0.99979863]]
    answer:  [[5.51483448e-05]]
    out:  [[5.51483448e-05]]
    answer:  [[4.27894631e-04]
     [7.97747125e-03]
     [7.33514002e-03]
     [4.30715933e-01]
     [2.94879795e-01]
     [3.60834589e-04]]
    out:  [[4.27894631e-04]
     [7.97747125e-03]
     [7.33514002e-03]
     [4.30715933e-01]
     [2.94879795e-01]
     [3.60834589e-04]]
    answer:  [[6.15631495e-04]
     [1.76551273e-03]
     [9.98663580e-01]
     [9.55352239e-01]
     [2.43722003e-02]]
    out:  [[6.15631495e-04]
     [1.76551273e-03]
     [9.98663580e-01]
     [9.55352239e-01]
     [2.43722003e-02]]
    answer:  [[0.05703716]]
    out:  [[0.05703716]]
    answer:  [[0.46334567]
     [0.99839969]
     [0.19530476]
     [0.1620233 ]]
    out:  [[0.46334567]
     [0.99839969]
     [0.19530476]
     [0.1620233 ]]
    answer:  [[0.94748273]
     [0.07110621]
     [0.20242959]]
    out:  [[0.94748273]
     [0.07110621]
     [0.20242959]]
    answer:  [[0.67630722]
     [0.89419932]
     [0.94322001]
     [0.99844905]
     [0.98489843]]
    out:  [[0.67630722]
     [0.89419932]
     [0.94322001]
     [0.99844905]
     [0.98489843]]
    answer:  [[0.00295923]
     [0.4882748 ]
     [0.99989424]
     [0.33342965]]
    out:  [[0.00295923]
     [0.4882748 ]
     [0.99989424]
     [0.33342965]]
    answer:  [[9.11732907e-01]
     [9.84042462e-01]
     [9.99166171e-01]
     [1.68633472e-04]
     [1.53430588e-04]
     [9.56320135e-01]]
    out:  [[9.11732907e-01]
     [9.84042462e-01]
     [9.99166171e-01]
     [1.68633472e-04]
     [1.53430588e-04]
     [9.56320135e-01]]
    answer:  [[0.99929755]
     [0.94668713]
     [0.00153522]
     [0.10095264]
     [0.99963544]]
    out:  [[0.99929755]
     [0.94668713]
     [0.00153522]
     [0.10095264]
     [0.99963544]]
    answer:  [[9.99914006e-01]
     [9.27821791e-01]
     [5.00701211e-04]
     [9.16672303e-01]]
    out:  [[9.99914006e-01]
     [9.27821791e-01]
     [5.00701211e-04]
     [9.16672303e-01]]
    answer:  [[0.97926447]
     [0.88335297]
     [0.99884186]
     [0.41690382]]
    out:  [[0.97926447]
     [0.88335297]
     [0.99884186]
     [0.41690382]]
    answer:  [[9.99490709e-01]
     [9.32345736e-04]
     [9.99940293e-01]
     [9.99425186e-01]
     [9.98566738e-01]]
    out:  [[9.99490709e-01]
     [9.32345736e-04]
     [9.99940293e-01]
     [9.99425186e-01]
     [9.98566738e-01]]
    answer:  [[0.72647751]
     [0.006998  ]
     [0.99974272]
     [0.00287895]
     [0.00139148]]
    out:  [[0.72647751]
     [0.006998  ]
     [0.99974272]
     [0.00287895]
     [0.00139148]]
    answer:  [[0.99680187]
     [0.24444563]
     [0.00169318]
     [0.995807  ]
     [0.999864  ]]
    out:  [[0.99680187]
     [0.24444563]
     [0.00169318]
     [0.995807  ]
     [0.999864  ]]
    answer:  [[0.99960157]
     [0.99955026]
     [0.01283319]]
    out:  [[0.99960157]
     [0.99955026]
     [0.01283319]]
    answer:  [[0.99968338]
     [0.99944205]
     [0.97146409]
     [0.27641222]
     [0.99986987]]
    out:  [[0.99968338]
     [0.99944205]
     [0.97146409]
     [0.27641222]
     [0.99986987]]
    answer:  [[0.995439]]
    out:  [[0.995439]]
    answer:  [[8.52835483e-01]
     [3.98889765e-04]
     [1.76776902e-03]
     [5.98245067e-04]]
    out:  [[8.52835483e-01]
     [3.98889765e-04]
     [1.76776902e-03]
     [5.98245067e-04]]
    answer:  [[0.72821487]
     [0.1544598 ]
     [0.38961763]
     [0.01489064]]
    out:  [[0.72821487]
     [0.1544598 ]
     [0.38961763]
     [0.01489064]]
    answer:  [[0.0008244 ]
     [0.0103292 ]
     [0.00011482]]
    out:  [[0.0008244 ]
     [0.0103292 ]
     [0.00011482]]
    answer:  [[0.88971181]
     [0.99368288]
     [0.99977238]]
    out:  [[0.88971181]
     [0.99368288]
     [0.99977238]]
    answer:  [[0.949244  ]
     [0.05137086]]
    out:  [[0.949244  ]
     [0.05137086]]
    answer:  [[4.24237281e-04]
     [4.81503768e-04]
     [2.17976498e-02]
     [7.30767587e-01]]
    out:  [[4.24237281e-04]
     [4.81503768e-04]
     [2.17976498e-02]
     [7.30767587e-01]]
    answer:  [[0.00043113]]
    out:  [[0.00043113]]
    answer:  [[9.88701671e-01]
     [7.64462779e-05]
     [9.95503392e-01]]
    out:  [[9.88701671e-01]
     [7.64462779e-05]
     [9.95503392e-01]]
    answer:  [[2.38450049e-03]
     [9.99315161e-01]
     [5.88278111e-01]
     [5.78978573e-04]
     [9.99212255e-01]
     [9.78946188e-01]]
    out:  [[2.38450049e-03]
     [9.99315161e-01]
     [5.88278111e-01]
     [5.78978573e-04]
     [9.99212255e-01]
     [9.78946188e-01]]
    answer:  [[0.2596962 ]
     [0.99989757]
     [0.00480776]]
    out:  [[0.2596962 ]
     [0.99989757]
     [0.00480776]]
    answer:  [[1.77175451e-03]
     [7.80817497e-04]
     [9.81165880e-01]
     [2.76790456e-03]
     [8.56120984e-01]
     [9.97308478e-01]]
    out:  [[1.77175451e-03]
     [7.80817497e-04]
     [9.81165880e-01]
     [2.76790456e-03]
     [8.56120984e-01]
     [9.97308478e-01]]
    answer:  [[9.01206649e-02]
     [1.01353516e-01]
     [6.26372314e-01]
     [9.87034971e-01]
     [4.28200103e-04]
     [1.29380641e-02]]
    out:  [[9.01206649e-02]
     [1.01353516e-01]
     [6.26372314e-01]
     [9.87034971e-01]
     [4.28200103e-04]
     [1.29380641e-02]]
    answer:  [[0.99976704]
     [0.00285544]
     [0.99942565]
     [0.99923024]]
    out:  [[0.99976704]
     [0.00285544]
     [0.99942565]
     [0.99923024]]
    answer:  [[0.87100404]]
    out:  [[0.87100404]]
    answer:  [[0.97838822]]
    out:  [[0.97838822]]
    answer:  [[0.1386186 ]
     [0.99831676]
     [0.99884767]
     [0.99298555]]
    out:  [[0.1386186 ]
     [0.99831676]
     [0.99884767]
     [0.99298555]]
    answer:  [[0.00032374]
     [0.21307368]]
    out:  [[0.00032374]
     [0.21307368]]
    answer:  [[9.98332355e-01]
     [1.30974182e-03]
     [6.47569387e-03]
     [2.41427820e-01]
     [3.36415996e-04]
     [6.70388055e-04]]
    out:  [[9.98332355e-01]
     [1.30974182e-03]
     [6.47569387e-03]
     [2.41427820e-01]
     [3.36415996e-04]
     [6.70388055e-04]]
    answer:  [[0.01365281]
     [0.03239814]
     [0.99708961]
     [0.99893771]
     [0.05481965]
     [0.00840359]]
    out:  [[0.01365281]
     [0.03239814]
     [0.99708961]
     [0.99893771]
     [0.05481965]
     [0.00840359]]
    answer:  [[4.66306381e-05]
     [3.27416666e-04]]
    out:  [[4.66306381e-05]
     [3.27416666e-04]]
    answer:  [[9.55543786e-01]
     [1.08159497e-02]
     [9.99518649e-01]
     [1.29767751e-04]
     [7.23244560e-05]]
    out:  [[9.55543786e-01]
     [1.08159497e-02]
     [9.99518649e-01]
     [1.29767751e-04]
     [7.23244560e-05]]
    answer:  [[0.00341761]
     [0.00211404]]
    out:  [[0.00341761]
     [0.00211404]]
    answer:  [[4.52359273e-02]
     [9.97817573e-01]
     [9.44730583e-04]
     [9.88892031e-01]
     [5.79254106e-02]]
    out:  [[4.52359273e-02]
     [9.97817573e-01]
     [9.44730583e-04]
     [9.88892031e-01]
     [5.79254106e-02]]
    answer:  [[6.91483780e-05]
     [9.30199363e-01]
     [9.86269137e-01]]
    out:  [[6.91483780e-05]
     [9.30199363e-01]
     [9.86269137e-01]]
    answer:  [[3.55315151e-01]
     [2.52525577e-04]
     [9.26961129e-01]
     [9.99819318e-01]
     [4.74595379e-03]
     [9.81699039e-01]]
    out:  [[3.55315151e-01]
     [2.52525577e-04]
     [9.26961129e-01]
     [9.99819318e-01]
     [4.74595379e-03]
     [9.81699039e-01]]
    answer:  [[0.00349225]
     [0.99394533]
     [0.98475249]]
    out:  [[0.00349225]
     [0.99394533]
     [0.98475249]]
    answer:  [[1.03115388e-04]
     [8.99570642e-01]
     [4.58200577e-04]
     [1.05680008e-04]
     [5.02999598e-01]]
    out:  [[1.03115388e-04]
     [8.99570642e-01]
     [4.58200577e-04]
     [1.05680008e-04]
     [5.02999598e-01]]
    answer:  [[7.44573439e-01]
     [5.00551059e-04]
     [3.65291091e-04]
     [9.99796246e-01]
     [9.56238616e-01]]
    out:  [[7.44573439e-01]
     [5.00551059e-04]
     [3.65291091e-04]
     [9.99796246e-01]
     [9.56238616e-01]]
    answer:  [[0.99827397]
     [0.99931602]
     [0.99946064]]
    out:  [[0.99827397]
     [0.99931602]
     [0.99946064]]
    answer:  [[3.27517772e-04]
     [9.99727195e-01]
     [6.06576608e-04]
     [4.89590086e-01]
     [2.22323722e-02]]
    out:  [[3.27517772e-04]
     [9.99727195e-01]
     [6.06576608e-04]
     [4.89590086e-01]
     [2.22323722e-02]]
    answer:  [[0.00220215]
     [0.99342421]
     [0.99911075]
     [0.00259965]
     [0.99969738]]
    out:  [[0.00220215]
     [0.99342421]
     [0.99911075]
     [0.00259965]
     [0.99969738]]
    answer:  [[0.17343546]
     [0.97167094]
     [0.99991972]
     [0.81308121]]
    out:  [[0.17343546]
     [0.97167094]
     [0.99991972]
     [0.81308121]]
    answer:  [[0.99895543]
     [0.90572406]
     [0.68781835]
     [0.00508772]
     [0.2044663 ]
     [0.99951705]]
    out:  [[0.99895543]
     [0.90572406]
     [0.68781835]
     [0.00508772]
     [0.2044663 ]
     [0.99951705]]
    answer:  [[0.00157375]
     [0.95176755]
     [0.96098067]
     [0.97259616]]
    out:  [[0.00157375]
     [0.95176755]
     [0.96098067]
     [0.97259616]]
    answer:  [[0.99992748]
     [0.98034976]]
    out:  [[0.99992748]
     [0.98034976]]
    answer:  [[9.99192722e-01]
     [9.99772488e-01]
     [1.88286920e-04]
     [3.78309380e-02]
     [9.41487038e-01]]
    out:  [[9.99192722e-01]
     [9.99772488e-01]
     [1.88286920e-04]
     [3.78309380e-02]
     [9.41487038e-01]]
    answer:  [[8.93372248e-01]
     [4.31871613e-02]
     [4.67794479e-04]
     [9.99898954e-01]]
    out:  [[8.93372248e-01]
     [4.31871613e-02]
     [4.67794479e-04]
     [9.99898954e-01]]
    answer:  [[5.60776828e-02]
     [1.78232031e-04]
     [4.51835105e-03]
     [1.56498273e-03]
     [9.04486634e-04]
     [9.99608851e-01]]
    out:  [[5.60776828e-02]
     [1.78232031e-04]
     [4.51835105e-03]
     [1.56498273e-03]
     [9.04486634e-04]
     [9.99608851e-01]]
    answer:  [[0.00034616]]
    out:  [[0.00034616]]
    answer:  [[6.82094325e-01]
     [6.56255497e-04]]
    out:  [[6.82094325e-01]
     [6.56255497e-04]]
    answer:  [[1.13450337e-03]
     [1.39758921e-04]
     [9.99700548e-01]
     [7.02662957e-05]]
    out:  [[1.13450337e-03]
     [1.39758921e-04]
     [9.99700548e-01]
     [7.02662957e-05]]
    answer:  [[2.61920533e-04]
     [9.18665056e-01]
     [9.95579027e-01]
     [9.50491367e-03]]
    out:  [[2.61920533e-04]
     [9.18665056e-01]
     [9.95579027e-01]
     [9.50491367e-03]]
    answer:  [[0.01338997]
     [0.0345356 ]]
    out:  [[0.01338997]
     [0.0345356 ]]
    answer:  [[5.84854689e-04]
     [7.25675279e-03]
     [9.64691192e-01]
     [6.09851914e-02]]
    out:  [[5.84854689e-04]
     [7.25675279e-03]
     [9.64691192e-01]
     [6.09851914e-02]]
    answer:  [[9.09834300e-01]
     [9.99939576e-01]
     [5.06723605e-05]
     [9.99488211e-01]
     [8.39205655e-05]
     [9.02442150e-01]]
    out:  [[9.09834300e-01]
     [9.99939576e-01]
     [5.06723605e-05]
     [9.99488211e-01]
     [8.39205655e-05]
     [9.02442150e-01]]
    answer:  [[9.99886831e-01]
     [9.99870477e-01]
     [1.59268823e-04]
     [6.51546911e-01]
     [9.99912652e-01]]
    out:  [[9.99886831e-01]
     [9.99870477e-01]
     [1.59268823e-04]
     [6.51546911e-01]
     [9.99912652e-01]]
    answer:  [[0.0135083 ]
     [0.99991652]
     [0.84374568]
     [0.00120615]]
    out:  [[0.0135083 ]
     [0.99991652]
     [0.84374568]
     [0.00120615]]
    answer:  [[0.98598523]
     [0.12898426]
     [0.99989954]]
    out:  [[0.98598523]
     [0.12898426]
     [0.99989954]]
    answer:  [[0.98824301]
     [0.00377513]]
    out:  [[0.98824301]
     [0.00377513]]
    answer:  [[0.00049831]]
    out:  [[0.00049831]]
    answer:  [[0.99985383]
     [0.99963843]
     [0.06434763]
     [0.95591307]]
    out:  [[0.99985383]
     [0.99963843]
     [0.06434763]
     [0.95591307]]
    answer:  [[6.43265209e-05]
     [9.99947629e-01]
     [3.15513043e-01]]
    out:  [[6.43265209e-05]
     [9.99947629e-01]
     [3.15513043e-01]]
    answer:  [[9.46788724e-01]
     [4.61553944e-02]
     [9.99710456e-01]
     [9.82727973e-01]
     [8.87691948e-04]
     [4.60862007e-03]]
    out:  [[9.46788724e-01]
     [4.61553944e-02]
     [9.99710456e-01]
     [9.82727973e-01]
     [8.87691948e-04]
     [4.60862007e-03]]
    answer:  [[0.99963419]
     [0.93594995]]
    out:  [[0.99963419]
     [0.93594995]]
    answer:  [[0.92147172]
     [0.99682962]
     [0.96472824]
     [0.99990125]]
    out:  [[0.92147172]
     [0.99682962]
     [0.96472824]
     [0.99990125]]
    answer:  [[0.00214431]
     [0.00144844]]
    out:  [[0.00214431]
     [0.00144844]]
    answer:  [[0.9988109 ]
     [0.999306  ]
     [0.00123286]]
    out:  [[0.9988109 ]
     [0.999306  ]
     [0.00123286]]
    answer:  [[9.99868546e-01]
     [9.93694970e-01]
     [9.97339532e-01]
     [9.99891776e-01]
     [1.00179635e-04]
     [9.99130758e-01]]
    out:  [[9.99868546e-01]
     [9.93694970e-01]
     [9.97339532e-01]
     [9.99891776e-01]
     [1.00179635e-04]
     [9.99130758e-01]]
    answer:  [[9.96094309e-01]
     [9.68886054e-01]
     [3.06949581e-04]
     [1.10682369e-04]
     [9.82778621e-01]]
    out:  [[9.96094309e-01]
     [9.68886054e-01]
     [3.06949581e-04]
     [1.10682369e-04]
     [9.82778621e-01]]
    answer:  [[0.99980601]]
    out:  [[0.99980601]]
    answer:  [[2.47939078e-01]
     [1.32162694e-01]
     [1.24888894e-04]
     [1.05770441e-02]
     [6.17098092e-03]
     [1.23273026e-03]]
    out:  [[2.47939078e-01]
     [1.32162694e-01]
     [1.24888894e-04]
     [1.05770441e-02]
     [6.17098092e-03]
     [1.23273026e-03]]
    answer:  [[9.89310200e-01]
     [8.85857775e-04]
     [9.97634439e-01]
     [9.99362522e-01]]
    out:  [[9.89310200e-01]
     [8.85857775e-04]
     [9.97634439e-01]
     [9.99362522e-01]]
    answer:  [[1.15386540e-04]
     [9.99906048e-01]
     [7.55706694e-02]
     [9.99844628e-01]
     [1.47460129e-01]]
    out:  [[1.15386540e-04]
     [9.99906048e-01]
     [7.55706694e-02]
     [9.99844628e-01]
     [1.47460129e-01]]
    answer:  [[0.91874566]
     [0.00872616]]
    out:  [[0.91874566]
     [0.00872616]]
    answer:  [[9.89545713e-01]
     [9.83204217e-01]
     [6.72325258e-04]
     [7.48046429e-01]
     [3.43041149e-03]]
    out:  [[9.89545713e-01]
     [9.83204217e-01]
     [6.72325258e-04]
     [7.48046429e-01]
     [3.43041149e-03]]
    answer:  [[0.96513534]
     [0.47818291]
     [0.9993134 ]
     [0.99915849]]
    out:  [[0.96513534]
     [0.47818291]
     [0.9993134 ]
     [0.99915849]]
    answer:  [[9.32073526e-05]
     [9.22917326e-01]
     [9.99908517e-01]]
    out:  [[9.32073526e-05]
     [9.22917326e-01]
     [9.99908517e-01]]
    answer:  [[0.26703175]
     [0.92583562]
     [0.42333683]
     [0.99770626]
     [0.99760522]]
    out:  [[0.26703175]
     [0.92583562]
     [0.42333683]
     [0.99770626]
     [0.99760522]]
    answer:  [[0.02278804]
     [0.04663432]
     [0.49532669]
     [0.09941872]]
    out:  [[0.02278804]
     [0.04663432]
     [0.49532669]
     [0.09941872]]
    answer:  [[0.93891122]]
    out:  [[0.93891122]]
    answer:  [[1.27497724e-01]
     [7.12988471e-01]
     [5.35428365e-05]
     [9.99798617e-01]
     [3.13107508e-04]]
    out:  [[1.27497724e-01]
     [7.12988471e-01]
     [5.35428365e-05]
     [9.99798617e-01]
     [3.13107508e-04]]
    answer:  [[0.01939963]
     [0.05748578]]
    out:  [[0.01939963]
     [0.05748578]]
    answer:  [[0.02555233]
     [0.00030702]]
    out:  [[0.02555233]
     [0.00030702]]
    answer:  [[0.61431616]
     [0.9967905 ]]
    out:  [[0.61431616]
     [0.9967905 ]]
    answer:  [[0.92090008]
     [0.75520605]]
    out:  [[0.92090008]
     [0.75520605]]
    answer:  [[0.99972588]
     [0.00937722]]
    out:  [[0.99972588]
     [0.00937722]]
    answer:  [[0.99839584]
     [0.96453587]]
    out:  [[0.99839584]
     [0.96453587]]
    answer:  [[5.01036101e-05]
     [6.07454196e-02]]
    out:  [[5.01036101e-05]
     [6.07454196e-02]]
    sigmoid_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!


Now you need to implement the calculation of the partial derivative of the loss function with respect to the input of sigmoid. 

$$
\frac{\partial \mathcal{L}}{\partial h} = 
\frac{\partial \mathcal{L}}{\partial f}
\frac{\partial f}{\partial h} 
$$

Tensor $\frac{\partial \mathcal{L}}{\partial f}$ comes from the loss function. Let's calculate $\frac{\partial f}{\partial h}$

$$
\frac{\partial f}{\partial h} = 
\frac{\partial \sigma(h)}{\partial h} =
\frac{\partial}{\partial h} \Big(\frac{1}{1 + e^{-h}}\Big)
= \frac{e^{-h}}{(1 + e^{-h})^2}
= \frac{1}{1 + e^{-h}} \frac{e^{-h}}{1 + e^{-h}}
= f(h) (1 - f(h))
$$

Therefore, in order to calculate the gradient of the loss with respect to the input of sigmoid function you need 
to 
1. calculate $f(h) (1 - f(h))$ 
2. multiply it element-wise by $\frac{\partial \mathcal{L}}{\partial f}$


```python
def sigmoid_grad_input(x_input, grad_output):
    """sigmoid nonlinearity gradient. 
        Calculate the partial derivative of the loss 
        with respect to the input of the layer
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
        grad_output: np.array of size `(n_objects, n_in)` 
            dL / df
    # Output
        the partial derivative of the loss 
        with respect to the input of the function
        np.array of size `(n_objects, n_in)` 
        dL / dh
    """
    #################
    ### YOUR CODE ###
    #################
    sigmoid_derivative = sigmoid_forward(x_input)*(1-sigmoid_forward(x_input))
    return sigmoid_derivative * grad_output
```


```python
am.test_student_function(username, sigmoid_grad_input, ['x_input', 'grad_output'])
```

    Running local tests...
    answer:  [[0.00230761]]
    out:  [[0.00230761]]
    answer:  [[-6.64838231e-05]
     [ 1.38934878e-03]
     [ 2.95415306e-04]
     [ 9.39282683e-02]
     [ 3.30721423e-03]]
    out:  [[-6.64838231e-05]
     [ 1.38934878e-03]
     [ 2.95415306e-04]
     [ 9.39282683e-02]
     [ 3.30721423e-03]]
    answer:  [[0.51101448]
     [0.02277825]]
    out:  [[0.51101448]
     [0.02277825]]
    answer:  [[-0.01144655]
     [ 0.00030077]]
    out:  [[-0.01144655]
     [ 0.00030077]]
    answer:  [[-0.20634018]]
    out:  [[-0.20634018]]
    answer:  [[0.09609206]
     [0.07597397]]
    out:  [[0.09609206]
     [0.07597397]]
    answer:  [[0.10340447]
     [0.000287  ]]
    out:  [[0.10340447]
     [0.000287  ]]
    answer:  [[-0.95973094]
     [ 0.0030641 ]
     [ 0.0225636 ]]
    out:  [[-0.95973094]
     [ 0.0030641 ]
     [ 0.0225636 ]]
    answer:  [[ 0.08985103]
     [ 0.002111  ]
     [-0.04862595]]
    out:  [[ 0.08985103]
     [ 0.002111  ]
     [-0.04862595]]
    answer:  [[-0.00018525]
     [ 0.0533396 ]]
    out:  [[-0.00018525]
     [ 0.0533396 ]]
    answer:  [[0.00171752]]
    out:  [[0.00171752]]
    answer:  [[ 0.00525991]
     [ 0.75989307]
     [-0.38099843]
     [-0.38466608]]
    out:  [[ 0.00525991]
     [ 0.75989307]
     [-0.38099843]
     [-0.38466608]]
    answer:  [[-1.31260033e-04]
     [-8.50481022e-01]]
    out:  [[-1.31260033e-04]
     [-8.50481022e-01]]
    answer:  [[-3.33215568e-05]]
    out:  [[-3.33215568e-05]]
    answer:  [[5.50025708e-05]]
    out:  [[5.50025708e-05]]
    answer:  [[ 0.01055336]
     [-0.01329132]
     [ 0.18480218]
     [ 0.01403653]
     [ 0.13000962]]
    out:  [[ 0.01055336]
     [-0.01329132]
     [ 0.18480218]
     [ 0.01403653]
     [ 0.13000962]]
    answer:  [[-0.04777064]
     [-0.08390344]
     [ 0.25664294]]
    out:  [[-0.04777064]
     [-0.08390344]
     [ 0.25664294]]
    answer:  [[-0.00036572]
     [ 0.01612851]
     [-0.0088985 ]]
    out:  [[-0.00036572]
     [ 0.01612851]
     [-0.0088985 ]]
    answer:  [[0.00026554]]
    out:  [[0.00026554]]
    answer:  [[ 5.84516291e-04]
     [ 5.63952649e-04]
     [ 4.45670648e-01]
     [-2.26085891e-04]
     [ 2.09784836e-02]
     [-1.67936161e-04]]
    out:  [[ 5.84516291e-04]
     [ 5.63952649e-04]
     [ 4.45670648e-01]
     [-2.26085891e-04]
     [ 2.09784836e-02]
     [-1.67936161e-04]]
    answer:  [[-9.39117780e-03]
     [ 1.95541742e-05]
     [ 2.99958574e-02]
     [-1.19136025e-01]
     [-5.17676783e-01]
     [ 7.02286479e-04]]
    out:  [[-9.39117780e-03]
     [ 1.95541742e-05]
     [ 2.99958574e-02]
     [-1.19136025e-01]
     [-5.17676783e-01]
     [ 7.02286479e-04]]
    answer:  [[-0.08719411]
     [-0.02159932]]
    out:  [[-0.08719411]
     [-0.02159932]]
    answer:  [[ 0.02272395]
     [-0.97523255]
     [-0.96817441]]
    out:  [[ 0.02272395]
     [-0.97523255]
     [-0.96817441]]
    answer:  [[-1.73212824e-01]
     [-7.90811345e-05]]
    out:  [[-1.73212824e-01]
     [-7.90811345e-05]]
    answer:  [[ 8.29311024e-01]
     [-4.43591382e-05]
     [-5.70436290e-03]
     [ 3.75304660e-04]
     [ 1.53470636e+00]
     [ 3.58991114e-01]]
    out:  [[ 8.29311024e-01]
     [-4.43591382e-05]
     [-5.70436290e-03]
     [ 3.75304660e-04]
     [ 1.53470636e+00]
     [ 3.58991114e-01]]
    answer:  [[ 0.0004563 ]
     [-0.01391702]
     [ 0.04503274]
     [-0.00624338]
     [-0.00096551]]
    out:  [[ 0.0004563 ]
     [-0.01391702]
     [ 0.04503274]
     [-0.00624338]
     [-0.00096551]]
    answer:  [[-5.25375185e-04]
     [-2.63968268e-01]
     [ 1.02657590e-02]
     [-1.03642049e+00]]
    out:  [[-5.25375185e-04]
     [-2.63968268e-01]
     [ 1.02657590e-02]
     [-1.03642049e+00]]
    answer:  [[ 0.001422  ]
     [-0.52406939]
     [ 0.00170996]
     [ 0.68260825]]
    out:  [[ 0.001422  ]
     [-0.52406939]
     [ 0.00170996]
     [ 0.68260825]]
    answer:  [[-5.26016556e-04]
     [-4.48032732e-01]
     [-2.16203673e-03]
     [-8.22363189e-01]
     [ 4.74359884e-03]]
    out:  [[-5.26016556e-04]
     [-4.48032732e-01]
     [-2.16203673e-03]
     [-8.22363189e-01]
     [ 4.74359884e-03]]
    answer:  [[0.014088]]
    out:  [[0.014088]]
    answer:  [[1.59828444]]
    out:  [[1.59828444]]
    answer:  [[-1.72722258e-03]
     [-8.33241674e-03]
     [ 1.53690003e-02]
     [-8.82796249e-01]
     [ 3.21304305e-04]]
    out:  [[-1.72722258e-03]
     [-8.33241674e-03]
     [ 1.53690003e-02]
     [-8.82796249e-01]
     [ 3.21304305e-04]]
    answer:  [[0.00141842]
     [0.75422818]]
    out:  [[0.00141842]
     [0.75422818]]
    answer:  [[ 0.002205  ]
     [ 0.05029656]
     [-0.00443128]
     [-0.00053174]
     [ 0.30112459]
     [-0.16873886]]
    out:  [[ 0.002205  ]
     [ 0.05029656]
     [-0.00443128]
     [-0.00053174]
     [ 0.30112459]
     [-0.16873886]]
    answer:  [[ 0.21797895]
     [ 0.82272661]
     [-0.10658472]
     [ 0.16848695]]
    out:  [[ 0.21797895]
     [ 0.82272661]
     [-0.10658472]
     [ 0.16848695]]
    answer:  [[1.56311499]]
    out:  [[1.56311499]]
    answer:  [[-1.85508929]
     [-0.16220724]]
    out:  [[-1.85508929]
     [-0.16220724]]
    answer:  [[-1.32804538e-01]
     [ 3.25650609e-01]
     [-4.04369529e-02]
     [-2.05098780e-04]]
    out:  [[-1.32804538e-01]
     [ 3.25650609e-01]
     [-4.04369529e-02]
     [-2.05098780e-04]]
    answer:  [[-0.14928082]
     [-0.00078972]
     [ 0.00167737]]
    out:  [[-0.14928082]
     [-0.00078972]
     [ 0.00167737]]
    answer:  [[-0.00332563]
     [-0.00558278]
     [-0.03970756]
     [ 1.58948499]
     [ 0.01177175]
     [ 0.15052282]]
    out:  [[-0.00332563]
     [-0.00558278]
     [-0.03970756]
     [ 1.58948499]
     [ 0.01177175]
     [ 0.15052282]]
    answer:  [[ 0.34521631]
     [ 0.06767319]
     [-0.03922278]
     [-0.00035858]]
    out:  [[ 0.34521631]
     [ 0.06767319]
     [-0.03922278]
     [-0.00035858]]
    answer:  [[-0.08607301]
     [-0.00643233]
     [ 0.07114168]]
    out:  [[-0.08607301]
     [-0.00643233]
     [ 0.07114168]]
    answer:  [[-0.0010123 ]
     [-0.02380642]
     [ 0.34431377]]
    out:  [[-0.0010123 ]
     [-0.02380642]
     [ 0.34431377]]
    answer:  [[-0.06442839]
     [-0.62507421]]
    out:  [[-0.06442839]
     [-0.62507421]]
    answer:  [[ 0.85060279]
     [ 0.28533543]
     [-0.03962442]
     [-0.00338676]]
    out:  [[ 0.85060279]
     [ 0.28533543]
     [-0.03962442]
     [-0.00338676]]
    answer:  [[-0.00038563]
     [-0.00044679]
     [ 0.07847397]
     [-0.00414688]]
    out:  [[-0.00038563]
     [-0.00044679]
     [ 0.07847397]
     [-0.00414688]]
    answer:  [[0.01918226]]
    out:  [[0.01918226]]
    answer:  [[-9.53072634e-05]
     [-4.83797569e-02]]
    out:  [[-9.53072634e-05]
     [-4.83797569e-02]]
    answer:  [[ 0.00241249]
     [ 0.00063164]
     [ 0.00087688]
     [-0.00433956]]
    out:  [[ 0.00241249]
     [ 0.00063164]
     [ 0.00087688]
     [-0.00433956]]
    answer:  [[-0.00038929]
     [-0.00432968]]
    out:  [[-0.00038929]
     [-0.00432968]]
    answer:  [[-1.51000645e-04]
     [ 1.58572561e-03]
     [ 2.48468861e-02]
     [ 2.91607343e-01]]
    out:  [[-1.51000645e-04]
     [ 1.58572561e-03]
     [ 2.48468861e-02]
     [ 2.91607343e-01]]
    answer:  [[ 5.40869192e-01]
     [-2.91707848e-04]
     [-1.80246814e-03]
     [ 1.20818598e+00]
     [ 1.52493739e+00]
     [-1.15099915e-05]]
    out:  [[ 5.40869192e-01]
     [-2.91707848e-04]
     [-1.80246814e-03]
     [ 1.20818598e+00]
     [ 1.52493739e+00]
     [-1.15099915e-05]]
    answer:  [[-1.80028922]
     [-0.16231016]
     [-0.00837902]
     [-0.06345333]
     [ 0.03669688]
     [-0.5040187 ]]
    out:  [[-1.80028922]
     [-0.16231016]
     [-0.00837902]
     [-0.06345333]
     [ 0.03669688]
     [-0.5040187 ]]
    answer:  [[-3.82978681e-04]
     [-1.98776604e-01]
     [ 2.22855246e-03]
     [-1.02207615e-01]
     [ 5.77348281e-01]]
    out:  [[-3.82978681e-04]
     [-1.98776604e-01]
     [ 2.22855246e-03]
     [-1.02207615e-01]
     [ 5.77348281e-01]]
    answer:  [[0.12281764]]
    out:  [[0.12281764]]
    answer:  [[-0.02979831]
     [-1.18149733]
     [-0.21592933]]
    out:  [[-0.02979831]
     [-1.18149733]
     [-0.21592933]]
    answer:  [[-2.54788448e-05]]
    out:  [[-2.54788448e-05]]
    answer:  [[-0.00569043]
     [ 0.00053956]
     [-0.2586217 ]
     [-0.0100085 ]]
    out:  [[-0.00569043]
     [ 0.00053956]
     [-0.2586217 ]
     [-0.0100085 ]]
    answer:  [[-7.17723799e-02]
     [-1.08757838e-03]
     [-2.90396378e-05]
     [ 1.85980517e+00]
     [ 7.76267575e-01]]
    out:  [[-7.17723799e-02]
     [-1.08757838e-03]
     [-2.90396378e-05]
     [ 1.85980517e+00]
     [ 7.76267575e-01]]
    answer:  [[ 0.00118866]
     [-0.37090244]
     [-0.29306106]]
    out:  [[ 0.00118866]
     [-0.37090244]
     [-0.29306106]]
    answer:  [[-0.00022499]
     [-0.01630216]]
    out:  [[-0.00022499]
     [-0.01630216]]
    answer:  [[ 0.01163941]
     [-0.12744901]
     [ 0.00363262]
     [ 0.00146875]]
    out:  [[ 0.01163941]
     [-0.12744901]
     [ 0.00363262]
     [ 0.00146875]]
    answer:  [[ 0.00151862]
     [-0.00021574]]
    out:  [[ 0.00151862]
     [-0.00021574]]
    answer:  [[-0.79590358]
     [-0.00573288]
     [ 0.00151582]]
    out:  [[-0.79590358]
     [-0.00573288]
     [ 0.00151582]]
    answer:  [[ 2.20347642e-04]
     [ 6.20256767e-03]
     [-1.13438483e+00]
     [ 6.76000795e-06]]
    out:  [[ 2.20347642e-04]
     [ 6.20256767e-03]
     [-1.13438483e+00]
     [ 6.76000795e-06]]
    answer:  [[-1.87756764]
     [ 1.20465708]
     [ 0.03574587]
     [-0.25447086]]
    out:  [[-1.87756764]
     [ 1.20465708]
     [ 0.03574587]
     [-0.25447086]]
    answer:  [[ 0.01474366]
     [-0.07051256]
     [ 0.506335  ]
     [ 0.00060451]]
    out:  [[ 0.01474366]
     [-0.07051256]
     [ 0.506335  ]
     [ 0.00060451]]
    answer:  [[-0.02512012]]
    out:  [[-0.02512012]]
    answer:  [[2.21787298e-01]
     [8.82069778e-04]
     [1.47871640e+00]
     [8.17356301e-01]
     [1.25036632e-02]]
    out:  [[2.21787298e-01]
     [8.82069778e-04]
     [1.47871640e+00]
     [8.17356301e-01]
     [1.25036632e-02]]
    answer:  [[-0.01794399]]
    out:  [[-0.01794399]]
    answer:  [[0.05511928]]
    out:  [[0.05511928]]
    answer:  [[1.62314224e-01]
     [1.46409898e-04]]
    out:  [[1.62314224e-01]
     [1.46409898e-04]]
    answer:  [[-0.3748311 ]
     [-0.13914564]]
    out:  [[-0.3748311 ]
     [-0.13914564]]
    answer:  [[-1.88280347e-01]
     [-1.65949097e+00]
     [-4.53693454e-05]
     [ 7.14795246e-05]
     [-4.03568666e-01]]
    out:  [[-1.88280347e-01]
     [-1.65949097e+00]
     [-4.53693454e-05]
     [ 7.14795246e-05]
     [-4.03568666e-01]]
    answer:  [[0.09960649]]
    out:  [[0.09960649]]
    answer:  [[ 0.003347  ]
     [-0.00332864]
     [-0.00654548]
     [-0.00147714]
     [ 0.01642622]]
    out:  [[ 0.003347  ]
     [-0.00332864]
     [-0.00654548]
     [-0.00147714]
     [ 0.01642622]]
    answer:  [[-8.68426149e-04]
     [ 5.45755260e-05]]
    out:  [[-8.68426149e-04]
     [ 5.45755260e-05]]
    answer:  [[-0.00348254]]
    out:  [[-0.00348254]]
    answer:  [[-0.00035743]]
    out:  [[-0.00035743]]
    answer:  [[-0.00641531]
     [ 2.07098185]]
    out:  [[-0.00641531]
     [ 2.07098185]]
    answer:  [[0.00481503]
     [0.00440269]
     [0.0003992 ]]
    out:  [[0.00481503]
     [0.00440269]
     [0.0003992 ]]
    answer:  [[ 0.00086931]
     [-0.73339532]
     [ 0.13529685]
     [ 0.00386087]]
    out:  [[ 0.00086931]
     [-0.73339532]
     [ 0.13529685]
     [ 0.00386087]]
    answer:  [[ 2.22231502e-01]
     [-1.14570948e+00]
     [-1.62751849e+00]
     [ 2.37658581e-02]
     [ 6.94788738e-01]
     [ 1.06735009e-04]]
    out:  [[ 2.22231502e-01]
     [-1.14570948e+00]
     [-1.62751849e+00]
     [ 2.37658581e-02]
     [ 6.94788738e-01]
     [ 1.06735009e-04]]
    answer:  [[ 7.68331274e-03]
     [ 4.94689743e-05]
     [ 6.26476614e-01]
     [-1.41382297e-03]
     [-4.04608570e-04]
     [ 1.29472166e-04]]
    out:  [[ 7.68331274e-03]
     [ 4.94689743e-05]
     [ 6.26476614e-01]
     [-1.41382297e-03]
     [-4.04608570e-04]
     [ 1.29472166e-04]]
    answer:  [[-8.85599278e-02]
     [ 1.06863483e-03]
     [-4.61248573e-04]
     [-2.73055647e-03]
     [-2.16254268e+00]]
    out:  [[-8.85599278e-02]
     [ 1.06863483e-03]
     [-4.61248573e-04]
     [-2.73055647e-03]
     [-2.16254268e+00]]
    answer:  [[-0.00635035]
     [ 0.13741673]
     [ 0.00501274]
     [-0.01067983]
     [-0.03305881]
     [-0.22306507]]
    out:  [[-0.00635035]
     [ 0.13741673]
     [ 0.00501274]
     [-0.01067983]
     [-0.03305881]
     [-0.22306507]]
    answer:  [[0.98583554]]
    out:  [[0.98583554]]
    answer:  [[0.00351397]]
    out:  [[0.00351397]]
    answer:  [[0.02808003]
     [0.07061347]]
    out:  [[0.02808003]
     [0.07061347]]
    answer:  [[ 5.22472417e-04]
     [-9.88743329e-04]
     [-6.45576700e-03]
     [-1.22054082e+00]]
    out:  [[ 5.22472417e-04]
     [-9.88743329e-04]
     [-6.45576700e-03]
     [-1.22054082e+00]]
    answer:  [[-0.15732573]
     [ 0.15344052]
     [-0.0218198 ]]
    out:  [[-0.15732573]
     [ 0.15344052]
     [-0.0218198 ]]
    answer:  [[-0.02790516]
     [-0.17132318]
     [ 0.01978303]
     [-0.02384776]
     [-0.02427157]]
    out:  [[-0.02790516]
     [-0.17132318]
     [ 0.01978303]
     [-0.02384776]
     [-0.02427157]]
    answer:  [[ 0.00195182]
     [-0.02752949]]
    out:  [[ 0.00195182]
     [-0.02752949]]
    answer:  [[ 1.40733923e+00]
     [-7.99156028e-02]
     [ 1.32514573e+00]
     [ 2.59835183e-02]
     [ 6.87746757e-04]]
    out:  [[ 1.40733923e+00]
     [-7.99156028e-02]
     [ 1.32514573e+00]
     [ 2.59835183e-02]
     [ 6.87746757e-04]]
    answer:  [[0.01305254]]
    out:  [[0.01305254]]
    answer:  [[-3.09869181e-02]
     [ 2.36246201e-04]
     [-1.29107734e+00]
     [ 2.21985939e+00]
     [-4.25558716e-03]]
    out:  [[-3.09869181e-02]
     [ 2.36246201e-04]
     [-1.29107734e+00]
     [ 2.21985939e+00]
     [-4.25558716e-03]]
    answer:  [[-1.22834259]]
    out:  [[-1.22834259]]
    answer:  [[0.00079348]]
    out:  [[0.00079348]]
    answer:  [[-2.72982588e-02]
     [ 2.91942549e-05]
     [-7.12041170e-03]
     [ 2.74251017e-01]
     [-1.05726510e-03]]
    out:  [[-2.72982588e-02]
     [ 2.91942549e-05]
     [-7.12041170e-03]
     [ 2.74251017e-01]
     [-1.05726510e-03]]
    answer:  [[ 3.09822397e-04]
     [ 4.12562117e-02]
     [-1.33844282e+00]]
    out:  [[ 3.09822397e-04]
     [ 4.12562117e-02]
     [-1.33844282e+00]]
    sigmoid_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!


## 1.3 Negative Log Likelihood

$$
\mathcal{L} 
= -\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]
$$

Here $N$ is the number of objects. $Y_j$ is the real label of an object and $\dot{Y}_j$ is the predicted one.


```python
def nll_forward(target_pred, target_true):
    """Compute the value of NLL
        for a given prediction and the ground truth
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the value of NLL for a given prediction and the ground truth
        scalar
    """
    #################
    ### YOUR CODE ###
    #################  
    n = target_true.shape[0]
    nll = (-1/n)*np.sum(target_true*np.log(target_pred) + (1-target_true)*np.log((1-target_pred)))
    return nll
```


```python
am.test_student_function(username, nll_forward, ['target_pred', 'target_true'])
```

    Running local tests...
    answer:  0.294478731620193
    out:  0.294478731620193
    answer:  0.6457859456319661
    out:  0.6457859456319661
    answer:  1.3091010301986676
    out:  1.3091010301986676
    answer:  0.008907215091483118
    out:  0.008907215091483118
    answer:  2.018538276075894
    out:  2.0185382760758945
    answer:  1.4298796144383004
    out:  1.4298796144383004
    answer:  0.3726597753747237
    out:  0.3726597753747237
    answer:  0.7079734798584152
    out:  0.7079734798584152
    answer:  1.0410744722853977
    out:  1.0410744722853977
    answer:  0.5910706538342627
    out:  0.5910706538342627
    answer:  1.0151908615726633
    out:  1.0151908615726633
    answer:  0.11257887616370411
    out:  0.11257887616370411
    answer:  0.8103193059991427
    out:  0.8103193059991426
    answer:  1.6868304173990465
    out:  1.6868304173990465
    answer:  0.3371828528028685
    out:  0.3371828528028685
    answer:  0.4662489469265399
    out:  0.4662489469265399
    answer:  0.42741276618954316
    out:  0.42741276618954316
    answer:  0.10459298419595935
    out:  0.10459298419595935
    answer:  0.5240336025035219
    out:  0.524033602503522
    answer:  2.9703001125937996
    out:  2.9703001125937996
    answer:  0.8062546992591102
    out:  0.8062546992591102
    answer:  1.8396004005574582
    out:  1.8396004005574582
    answer:  0.7400729726511057
    out:  0.7400729726511057
    answer:  0.5759336172512707
    out:  0.5759336172512707
    answer:  0.04127289295470578
    out:  0.04127289295470578
    answer:  1.929103452653353
    out:  1.929103452653353
    answer:  1.3748862210646724
    out:  1.3748862210646724
    answer:  0.3773836587793014
    out:  0.3773836587793014
    answer:  0.691312902173958
    out:  0.691312902173958
    answer:  3.232266230470178
    out:  3.232266230470178
    answer:  1.1919613412061572
    out:  1.1919613412061572
    answer:  0.5523330392510737
    out:  0.5523330392510737
    answer:  0.4855647719680161
    out:  0.4855647719680161
    answer:  1.7778988150313852
    out:  1.7778988150313852
    answer:  0.6399016706184188
    out:  0.6399016706184188
    answer:  0.5863855366687577
    out:  0.5863855366687576
    answer:  4.732131937403078
    out:  4.732131937403078
    answer:  1.007430777861679
    out:  1.0074307778616791
    answer:  0.5546561369650052
    out:  0.5546561369650053
    answer:  0.34497892709372424
    out:  0.34497892709372424
    answer:  2.5213591161543145
    out:  2.521359116154314
    answer:  0.35588322514711934
    out:  0.35588322514711934
    answer:  1.9746044299822587
    out:  1.9746044299822587
    answer:  0.8920188832173994
    out:  0.8920188832173994
    answer:  1.2863266236592863
    out:  1.2863266236592863
    answer:  1.0439258483342515
    out:  1.0439258483342515
    answer:  0.9071465071682489
    out:  0.9071465071682489
    answer:  1.9360755586221554
    out:  1.9360755586221554
    answer:  0.9159333553499727
    out:  0.9159333553499728
    answer:  1.4338383647373674
    out:  1.4338383647373674
    answer:  0.8783706945773327
    out:  0.8783706945773327
    answer:  1.1971584839141023
    out:  1.1971584839141023
    answer:  0.4938570556143446
    out:  0.4938570556143446
    answer:  0.7569980732562938
    out:  0.7569980732562938
    answer:  0.43714419342731625
    out:  0.43714419342731625
    answer:  0.9577314938752208
    out:  0.9577314938752209
    answer:  0.14953252653686347
    out:  0.14953252653686347
    answer:  1.5305036307740885
    out:  1.5305036307740885
    answer:  1.7283049096065168
    out:  1.7283049096065168
    answer:  0.980508193487533
    out:  0.980508193487533
    answer:  0.9718367475758618
    out:  0.9718367475758618
    answer:  0.522060449502922
    out:  0.522060449502922
    answer:  0.9778424121170808
    out:  0.9778424121170808
    answer:  1.493431834437653
    out:  1.493431834437653
    answer:  0.5701722766675419
    out:  0.570172276667542
    answer:  1.1981359479111195
    out:  1.1981359479111195
    answer:  0.8986356616640081
    out:  0.8986356616640081
    answer:  2.1333926333975555
    out:  2.1333926333975555
    answer:  0.764815707001177
    out:  0.764815707001177
    answer:  0.22710601633904992
    out:  0.22710601633904992
    answer:  0.7314215884063483
    out:  0.7314215884063483
    answer:  0.5876952921288678
    out:  0.587695292128868
    answer:  1.4942863554354204
    out:  1.4942863554354204
    answer:  1.1362698770813249
    out:  1.1362698770813249
    answer:  0.8308817121271375
    out:  0.8308817121271375
    answer:  0.34240049909885933
    out:  0.34240049909885933
    answer:  1.31516848830136
    out:  1.3151684883013597
    answer:  0.7225529584364316
    out:  0.7225529584364316
    answer:  0.4515378969954698
    out:  0.4515378969954698
    answer:  0.8534575887948346
    out:  0.8534575887948347
    answer:  0.715284025872319
    out:  0.715284025872319
    answer:  0.37899901355975285
    out:  0.37899901355975285
    answer:  1.5698798984455067
    out:  1.5698798984455067
    answer:  1.0985416433769029
    out:  1.0985416433769029
    answer:  1.1147595926496292
    out:  1.1147595926496292
    answer:  2.0458899857221966
    out:  2.0458899857221966
    answer:  1.2615313170694338
    out:  1.2615313170694338
    answer:  1.6835839239182464
    out:  1.6835839239182462
    answer:  1.6644266455697647
    out:  1.6644266455697647
    answer:  0.8929202748891396
    out:  0.8929202748891396
    answer:  1.8521206252134979
    out:  1.8521206252134979
    answer:  0.7531453527896558
    out:  0.7531453527896558
    answer:  1.9587232690599201
    out:  1.9587232690599201
    answer:  0.9415670661554317
    out:  0.9415670661554317
    answer:  1.0067348745038127
    out:  1.006734874503813
    answer:  1.0076259257827522
    out:  1.0076259257827522
    answer:  0.6696058307883362
    out:  0.6696058307883362
    answer:  0.9865644058748593
    out:  0.9865644058748594
    answer:  0.7560276226514772
    out:  0.7560276226514772
    answer:  2.315626638997756
    out:  2.315626638997756
    nll_forward successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!


Now you need to calculate the partial derivative of NLL with with respect to its input.

$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}}
=
\begin{pmatrix}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_1} \\
\vdots \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_N}
\end{pmatrix}
$$

Let's do it step-by-step

\begin{equation}
\begin{split}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} 
&= \frac{\partial}{\partial \dot{Y}_0} \Big(-\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]\Big) \\
&= -\frac{1}{N} \frac{\partial}{\partial \dot{Y}_0} \Big(Y_0\log \dot{Y}_0 + (1-Y_0)\log(1-\dot{Y}_0)\Big) \\
&= -\frac{1}{N} \Big(\frac{Y_0}{\dot{Y}_0} - \frac{1-Y_0}{1-\dot{Y}_0}\Big)
= \frac{1}{N} \frac{\dot{Y}_0 - Y_0}{\dot{Y}_0 (1 - \dot{Y}_0)}
\end{split}
\end{equation}

And for the other components it can be done in exactly the same way. So the result is the vector where each component is given by 
$$\frac{1}{N} \frac{\dot{Y}_j - Y_j}{\dot{Y}_j (1 - \dot{Y}_j)}$$

Or if we assume all multiplications and divisions to be done element-wise the output can be calculated as
$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}} = \frac{1}{N} \frac{\dot{Y} - Y}{\dot{Y} (1 - \dot{Y})}
$$


```python
def nll_grad_input(target_pred, target_true):
    """Compute the partial derivative of NLL
        with respect to its input
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the partial derivative 
        of NLL with respect to its input
        np.array of size `(n_objects, 1)`
    """
    #################
    ### YOUR CODE ###
    #################  
    n = target_pred.shape[0]
    grad_input = (1/n)*((target_pred-target_true)/(target_pred*(1-target_pred)))
    return grad_input
```


```python
am.test_student_function(username, nll_grad_input, ['target_pred', 'target_true'])
```

    Running local tests...
    answer:  [[ 0.29841421]
     [-0.33465453]
     [ 0.19155277]
     [ 0.36736244]
     [-0.17852187]
     [-0.21752456]]
    out:  [[ 0.29841421]
     [-0.33465453]
     [ 0.19155277]
     [ 0.36736244]
     [-0.17852187]
     [-0.21752456]]
    answer:  [[-1.23923383]
     [-1.38002788]]
    out:  [[-1.23923383]
     [-1.38002788]]
    answer:  [[-0.51874259]
     [ 0.20652031]
     [ 0.7501924 ]
     [ 0.22918272]
     [ 0.16825827]
     [ 0.48544159]]
    out:  [[-0.51874259]
     [ 0.20652031]
     [ 0.7501924 ]
     [ 0.22918272]
     [ 0.16825827]
     [ 0.48544159]]
    answer:  [[0.90063196]
     [1.85346235]]
    out:  [[0.90063196]
     [1.85346235]]
    answer:  [[-0.4293097 ]
     [-0.20796309]
     [ 2.02336044]
     [ 1.4246562 ]
     [ 0.35380357]
     [ 0.20609128]]
    out:  [[-0.4293097 ]
     [-0.20796309]
     [ 2.02336044]
     [ 1.4246562 ]
     [ 0.35380357]
     [ 0.20609128]]
    answer:  [[ 1.13928551]
     [-0.74726513]]
    out:  [[ 1.13928551]
     [-0.74726513]]
    answer:  [[ 0.37383686]
     [-0.41766359]
     [-0.20720614]
     [ 0.34117441]
     [-0.62493844]]
    out:  [[ 0.37383686]
     [-0.41766359]
     [-0.20720614]
     [ 0.34117441]
     [-0.62493844]]
    answer:  [[0.9818511 ]
     [0.87767908]]
    out:  [[0.9818511 ]
     [0.87767908]]
    answer:  [[-0.95925755]
     [ 1.35632045]
     [-0.2619557 ]
     [-0.30558667]
     [-0.25098093]
     [ 0.49696548]]
    out:  [[-0.95925755]
     [ 1.35632045]
     [-0.2619557 ]
     [-0.30558667]
     [-0.25098093]
     [ 0.49696548]]
    answer:  [[-0.69667351]
     [ 1.45937954]
     [-0.46371643]
     [-0.37117459]
     [ 0.23295911]
     [ 0.33987339]]
    out:  [[-0.69667351]
     [ 1.45937954]
     [-0.46371643]
     [-0.37117459]
     [ 0.23295911]
     [ 0.33987339]]
    answer:  [[ 1.26672997]
     [-0.31548053]
     [ 0.33258515]
     [-0.78335037]
     [ 0.19898781]
     [ 0.72973688]]
    out:  [[ 1.26672997]
     [-0.31548053]
     [ 0.33258515]
     [-0.78335037]
     [ 0.19898781]
     [ 0.72973688]]
    answer:  [[-0.31838622]
     [-0.54449317]
     [ 0.78160681]
     [ 2.85704077]
     [-0.2246277 ]
     [ 0.17858765]]
    out:  [[-0.31838622]
     [-0.54449317]
     [ 0.78160681]
     [ 2.85704077]
     [-0.2246277 ]
     [ 0.17858765]]
    answer:  [[-0.30257139]
     [ 0.21136616]
     [-3.44361228]
     [ 1.40539068]
     [38.47805665]]
    out:  [[-0.30257139]
     [ 0.21136616]
     [-3.44361228]
     [ 1.40539068]
     [38.47805665]]
    answer:  [[-107.98000299]]
    out:  [[-107.98000299]]
    answer:  [[ 0.52664532]
     [-0.27818444]
     [ 0.50331287]
     [-0.20254502]
     [ 0.20358298]]
    out:  [[ 0.52664532]
     [-0.27818444]
     [ 0.50331287]
     [-0.20254502]
     [ 0.20358298]]
    answer:  [[-24.83314866]
     [ -1.13057348]]
    out:  [[-24.83314866]
     [ -1.13057348]]
    answer:  [[-2.28104761]
     [ 0.84525687]]
    out:  [[-2.28104761]
     [ 0.84525687]]
    answer:  [[-4.70439813]]
    out:  [[-4.70439813]]
    answer:  [[-6.73330787]
     [ 0.47351493]
     [ 0.3693636 ]]
    out:  [[-6.73330787]
     [ 0.47351493]
     [ 0.3693636 ]]
    answer:  [[-0.18787658]
     [-0.29511787]
     [ 0.1757777 ]
     [ 0.3675202 ]
     [ 0.19581058]
     [ 0.20356258]]
    out:  [[-0.18787658]
     [-0.29511787]
     [ 0.1757777 ]
     [ 0.3675202 ]
     [ 0.19581058]
     [ 0.20356258]]
    answer:  [[-0.75496527]
     [18.58488773]
     [-0.28081939]
     [-0.81091688]]
    out:  [[-0.75496527]
     [18.58488773]
     [-0.28081939]
     [-0.81091688]]
    answer:  [[-0.1717618 ]
     [ 2.72144951]
     [-0.2828699 ]
     [-0.25727799]
     [-0.18688636]
     [ 0.19520072]]
    out:  [[-0.1717618 ]
     [ 2.72144951]
     [-0.2828699 ]
     [-0.25727799]
     [-0.18688636]
     [ 0.19520072]]
    answer:  [[-0.60796235]
     [-0.48052711]
     [-0.44097088]]
    out:  [[-0.60796235]
     [-0.48052711]
     [-0.44097088]]
    answer:  [[ -0.19950724]
     [  0.34670321]
     [ -0.53417512]
     [ -0.39462495]
     [ -0.17764788]
     [-72.92245023]]
    out:  [[ -0.19950724]
     [  0.34670321]
     [ -0.53417512]
     [ -0.39462495]
     [ -0.17764788]
     [-72.92245023]]
    answer:  [[0.22985642]
     [0.30928714]
     [0.25057597]
     [0.39116995]
     [0.57546839]]
    out:  [[0.22985642]
     [0.30928714]
     [0.25057597]
     [0.39116995]
     [0.57546839]]
    answer:  [[ 0.25424739]
     [ 0.28003531]
     [-0.39842311]
     [-0.44263851]
     [12.51756847]
     [ 0.20909007]]
    out:  [[ 0.25424739]
     [ 0.28003531]
     [-0.39842311]
     [-0.44263851]
     [12.51756847]
     [ 0.20909007]]
    answer:  [[-1.23598183]
     [-0.52764574]]
    out:  [[-1.23598183]
     [-0.52764574]]
    answer:  [[ 0.22511093]
     [-0.32164237]
     [-0.59897722]
     [-1.13253466]
     [ 0.22007923]]
    out:  [[ 0.22511093]
     [-0.32164237]
     [-0.59897722]
     [-1.13253466]
     [ 0.22007923]]
    answer:  [[16.19647189]]
    out:  [[16.19647189]]
    answer:  [[ 0.20838305]
     [ 0.21043485]
     [-0.41091752]
     [-0.42503405]
     [-0.24942989]
     [-0.18685701]]
    out:  [[ 0.20838305]
     [ 0.21043485]
     [-0.41091752]
     [-0.42503405]
     [-0.24942989]
     [-0.18685701]]
    answer:  [[-10.65981321]
     [  0.34117355]
     [-24.06783145]]
    out:  [[-10.65981321]
     [  0.34117355]
     [-24.06783145]]
    answer:  [[-1.29388098]
     [42.58155736]
     [-2.52434753]]
    out:  [[-1.29388098]
     [42.58155736]
     [-2.52434753]]
    answer:  [[ 0.33185647]
     [-0.71030565]
     [-0.2293833 ]
     [ 0.32971541]
     [ 0.66630811]
     [-0.66962908]]
    out:  [[ 0.33185647]
     [-0.71030565]
     [-0.2293833 ]
     [ 0.32971541]
     [ 0.66630811]
     [-0.66962908]]
    answer:  [[ 1.75047245]
     [-2.4666835 ]]
    out:  [[ 1.75047245]
     [-2.4666835 ]]
    answer:  [[-0.6134308 ]
     [-0.55061303]
     [ 0.78081234]
     [-0.5429865 ]
     [ 0.26852745]
     [ 0.18133833]]
    out:  [[-0.6134308 ]
     [-0.55061303]
     [ 0.78081234]
     [-0.5429865 ]
     [ 0.26852745]
     [ 0.18133833]]
    answer:  [[-1.40419025]
     [-1.62945813]
     [-0.68155091]]
    out:  [[-1.40419025]
     [-1.62945813]
     [-0.68155091]]
    answer:  [[ 0.67624028]
     [-9.13633336]
     [-0.34574901]]
    out:  [[ 0.67624028]
     [-9.13633336]
     [-0.34574901]]
    answer:  [[ 2.21644903]
     [-0.31383371]
     [ 0.35913734]
     [-0.2081932 ]
     [ 0.53251232]]
    out:  [[ 2.21644903]
     [-0.31383371]
     [ 0.35913734]
     [-0.2081932 ]
     [ 0.53251232]]
    answer:  [[ 0.33600869]
     [ 0.23646374]
     [ 1.74265395]
     [ 0.26068628]
     [-0.76744817]
     [ 0.72206491]]
    out:  [[ 0.33600869]
     [ 0.23646374]
     [ 1.74265395]
     [ 0.26068628]
     [-0.76744817]
     [ 0.72206491]]
    answer:  [[-1.85967745]]
    out:  [[-1.85967745]]
    answer:  [[-1.89541396]
     [-0.36777629]
     [ 5.93636958]]
    out:  [[-1.89541396]
     [-0.36777629]
     [ 5.93636958]]
    answer:  [[-1.01914374]
     [-0.90546112]]
    out:  [[-1.01914374]
     [-0.90546112]]
    answer:  [[-2.53041745]]
    out:  [[-2.53041745]]
    answer:  [[ 0.20711303]
     [ 0.5179022 ]
     [-0.42520374]
     [-1.23179613]
     [-2.12508591]]
    out:  [[ 0.20711303]
     [ 0.5179022 ]
     [-0.42520374]
     [-1.23179613]
     [-2.12508591]]
    answer:  [[ 0.41198032]
     [-0.39326099]
     [-0.39388104]]
    out:  [[ 0.41198032]
     [-0.39326099]
     [-0.39388104]]
    answer:  [[-1.25691987]
     [ 3.1570257 ]]
    out:  [[-1.25691987]
     [ 3.1570257 ]]
    answer:  [[ 2.63378521]
     [ 0.363845  ]
     [-0.23418599]
     [ 0.23617404]
     [ 1.86496298]]
    out:  [[ 2.63378521]
     [ 0.363845  ]
     [-0.23418599]
     [ 0.23617404]
     [ 1.86496298]]
    answer:  [[2.77858598]]
    out:  [[2.77858598]]
    answer:  [[-1.47135789]
     [ 1.55020674]]
    out:  [[-1.47135789]
     [ 1.55020674]]
    answer:  [[-2.19460861]
     [ 0.81934741]]
    out:  [[-2.19460861]
     [ 0.81934741]]
    answer:  [[-0.17003187]
     [-0.3093962 ]
     [ 0.16913873]
     [-0.26807724]
     [-3.70507096]
     [-0.18772724]]
    out:  [[-0.17003187]
     [-0.3093962 ]
     [ 0.16913873]
     [-0.26807724]
     [-3.70507096]
     [-0.18772724]]
    answer:  [[2.9828058]]
    out:  [[2.9828058]]
    answer:  [[ 0.61646111]
     [ 1.13014207]
     [-0.98059025]
     [-0.38946889]]
    out:  [[ 0.61646111]
     [ 1.13014207]
     [-0.98059025]
     [-0.38946889]]
    answer:  [[3.54008487]]
    out:  [[3.54008487]]
    answer:  [[ 0.21943044]
     [ 0.39685733]
     [-4.64797106]
     [ 0.31657707]
     [ 0.2197619 ]]
    out:  [[ 0.21943044]
     [ 0.39685733]
     [-4.64797106]
     [ 0.31657707]
     [ 0.2197619 ]]
    answer:  [[-0.47011291]
     [-0.2443725 ]
     [-0.20590104]
     [ 0.19960627]
     [-0.78171892]
     [ 0.84411893]]
    out:  [[-0.47011291]
     [-0.2443725 ]
     [-0.20590104]
     [ 0.19960627]
     [-0.78171892]
     [ 0.84411893]]
    answer:  [[8.78465195]]
    out:  [[8.78465195]]
    answer:  [[ 0.47233168]
     [ 0.55975304]
     [-0.35233451]]
    out:  [[ 0.47233168]
     [ 0.55975304]
     [-0.35233451]]
    answer:  [[ 0.62854642]
     [-0.61934007]]
    out:  [[ 0.62854642]
     [-0.61934007]]
    answer:  [[-0.3758779 ]
     [ 1.0569975 ]
     [-0.34594441]]
    out:  [[-0.3758779 ]
     [ 1.0569975 ]
     [-0.34594441]]
    answer:  [[0.83313507]
     [2.62520595]]
    out:  [[0.83313507]
     [2.62520595]]
    answer:  [[-0.39791047]
     [-1.03548214]
     [-0.25425122]
     [-0.33201382]
     [-0.454807  ]]
    out:  [[-0.39791047]
     [-1.03548214]
     [-0.25425122]
     [-0.33201382]
     [-0.454807  ]]
    answer:  [[1.26086038]]
    out:  [[1.26086038]]
    answer:  [[-2.36485226]]
    out:  [[-2.36485226]]
    answer:  [[-1.23297624]
     [-0.32234885]
     [-1.23238453]
     [ 0.22058976]
     [ 0.47882827]]
    out:  [[-1.23297624]
     [-0.32234885]
     [-1.23238453]
     [ 0.22058976]
     [ 0.47882827]]
    answer:  [[ 0.28461639]
     [-0.24730242]
     [-1.62488268]
     [-0.76576989]
     [ 0.31858288]]
    out:  [[ 0.28461639]
     [-0.24730242]
     [-1.62488268]
     [-0.76576989]
     [ 0.31858288]]
    answer:  [[3.9342409]]
    out:  [[3.9342409]]
    answer:  [[ 0.62203668]
     [-1.84741428]
     [ 0.35323023]
     [-0.48270699]]
    out:  [[ 0.62203668]
     [-1.84741428]
     [ 0.35323023]
     [-0.48270699]]
    answer:  [[ 0.25274832]
     [-0.25739419]
     [-0.41153188]
     [ 0.32905   ]]
    out:  [[ 0.25274832]
     [-0.25739419]
     [-0.41153188]
     [ 0.32905   ]]
    answer:  [[ 0.37479544]
     [-0.34652176]
     [-1.52254908]]
    out:  [[ 0.37479544]
     [-0.34652176]
     [-1.52254908]]
    answer:  [[-1.45308752]
     [ 0.27228027]
     [-0.50356757]
     [ 0.26155933]
     [ 0.50496289]]
    out:  [[-1.45308752]
     [ 0.27228027]
     [-0.50356757]
     [ 0.26155933]
     [ 0.50496289]]
    answer:  [[-8.3814388 ]
     [ 0.52021738]
     [-0.50869756]]
    out:  [[-8.3814388 ]
     [ 0.52021738]
     [-0.50869756]]
    answer:  [[-2.0554369]]
    out:  [[-2.0554369]]
    answer:  [[2.66196575]]
    out:  [[2.66196575]]
    answer:  [[0.79532527]
     [0.26916739]
     [0.31595314]
     [0.54655781]]
    out:  [[0.79532527]
     [0.26916739]
     [0.31595314]
     [0.54655781]]
    answer:  [[ 0.87858286]
     [ 0.41848784]
     [ 0.25803399]
     [-0.48176482]]
    out:  [[ 0.87858286]
     [ 0.41848784]
     [ 0.25803399]
     [-0.48176482]]
    answer:  [[ 0.51012278]
     [ 0.23414517]
     [-0.41123828]
     [-0.53324707]
     [ 0.25463932]]
    out:  [[ 0.51012278]
     [ 0.23414517]
     [-0.41123828]
     [-0.53324707]
     [ 0.25463932]]
    answer:  [[-0.23051091]
     [-0.44525354]
     [ 0.3560292 ]
     [-0.52347735]
     [-0.16870094]
     [-0.64824686]]
    out:  [[-0.23051091]
     [-0.44525354]
     [ 0.3560292 ]
     [-0.52347735]
     [-0.16870094]
     [-0.64824686]]
    answer:  [[-3.48889688]
     [ 0.62948431]
     [ 4.61715783]]
    out:  [[-3.48889688]
     [ 0.62948431]
     [ 4.61715783]]
    answer:  [[ 0.61946571]
     [-0.44674123]
     [-0.7488963 ]]
    out:  [[ 0.61946571]
     [-0.44674123]
     [-0.7488963 ]]
    answer:  [[-0.49337922]
     [ 0.44354941]
     [ 0.30840402]
     [ 0.81280844]
     [ 0.38107689]
     [-0.84928944]]
    out:  [[-0.49337922]
     [ 0.44354941]
     [ 0.30840402]
     [ 0.81280844]
     [ 0.38107689]
     [-0.84928944]]
    answer:  [[ 0.37571076]
     [-0.20732891]
     [-0.18205064]
     [ 0.47648762]
     [-0.21264777]
     [ 0.59518312]]
    out:  [[ 0.37571076]
     [-0.20732891]
     [-0.18205064]
     [ 0.47648762]
     [-0.21264777]
     [ 0.59518312]]
    answer:  [[-0.59494456]
     [-0.35219503]
     [ 4.67217579]
     [-0.32678897]]
    out:  [[-0.59494456]
     [-0.35219503]
     [ 4.67217579]
     [-0.32678897]]
    answer:  [[-13.00057297]
     [  0.49392315]
     [  2.05162838]]
    out:  [[-13.00057297]
     [  0.49392315]
     [  2.05162838]]
    answer:  [[ 0.69694481]
     [ 0.36958747]
     [ 0.47673502]
     [-0.5665501 ]
     [ 0.27010076]
     [ 0.31715981]]
    out:  [[ 0.69694481]
     [ 0.36958747]
     [ 0.47673502]
     [-0.5665501 ]
     [ 0.27010076]
     [ 0.31715981]]
    answer:  [[ 0.53048278]
     [ 0.4841883 ]
     [-0.2322285 ]
     [-0.2324706 ]
     [-0.23028224]]
    out:  [[ 0.53048278]
     [ 0.4841883 ]
     [-0.2322285 ]
     [-0.2324706 ]
     [-0.23028224]]
    answer:  [[ 0.82587053]
     [ 0.91877135]
     [-3.06525646]]
    out:  [[ 0.82587053]
     [ 0.91877135]
     [-3.06525646]]
    answer:  [[1.58375282]]
    out:  [[1.58375282]]
    answer:  [[-4.19469709]]
    out:  [[-4.19469709]]
    answer:  [[ 0.36056324]
     [ 0.33225572]
     [-0.67324817]
     [ 1.25762787]]
    out:  [[ 0.36056324]
     [ 0.33225572]
     [-0.67324817]
     [ 1.25762787]]
    answer:  [[-0.33950803]
     [ 0.49189453]
     [-1.14951512]]
    out:  [[-0.33950803]
     [ 0.49189453]
     [-1.14951512]]
    answer:  [[ 0.27713741]
     [-3.81833819]
     [-0.56299185]
     [ 0.20510877]
     [-0.27983376]]
    out:  [[ 0.27713741]
     [-3.81833819]
     [-0.56299185]
     [ 0.20510877]
     [-0.27983376]]
    answer:  [[ 0.37996834]
     [ 7.15550143]
     [-0.5341049 ]]
    out:  [[ 0.37996834]
     [ 7.15550143]
     [-0.5341049 ]]
    answer:  [[-0.28376013]
     [-0.54485035]
     [-0.29549225]
     [-0.33816829]]
    out:  [[-0.28376013]
     [-0.54485035]
     [-0.29549225]
     [-0.33816829]]
    answer:  [[1.43374656]]
    out:  [[1.43374656]]
    answer:  [[-0.25961698]
     [ 1.50855592]
     [-1.29114568]
     [ 0.37295383]]
    out:  [[-0.25961698]
     [ 1.50855592]
     [-1.29114568]
     [ 0.37295383]]
    answer:  [[-1.44152426]]
    out:  [[-1.44152426]]
    answer:  [[-1.04635687]]
    out:  [[-1.04635687]]
    answer:  [[-1.14204367]
     [-0.56635234]
     [ 8.95102742]]
    out:  [[-1.14204367]
     [-0.56635234]
     [ 8.95102742]]
    answer:  [[ 0.25846501]
     [ 0.29164992]
     [-0.30135405]
     [-0.29016782]]
    out:  [[ 0.25846501]
     [ 0.29164992]
     [-0.30135405]
     [-0.29016782]]
    nll_grad_input successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!



```python
am.get_progress(username)
```

    ---------------------------------------------
    | Jakub Miksa                               |
    | jakub.miksa@student.uva.nl                |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | not attempted  |
    | tree_split_data_left     | not attempted  |
    | tree_split_data_right    | not attempted  |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------


## 1.4 Model

Here we provide a model for your. It consist of the function which you have implmeneted above


```python
class LogsticRegressionGD(object):
    
    def __init__(self, n_in, lr=0.05):
        super().__init__()
        self.lr = lr
        self.b = np.zeros(1, )
        self.W = np.random.randn(n_in, 1)
        
    def forward(self, x):
        self.h = linear_forward(x, self.W, self.b)
        y = sigmoid_forward(self.h)
        return y
    
    def update_params(self, x, nll_grad):
        # compute gradients
        grad_h = sigmoid_grad_input(self.h, nll_grad)
        grad_W = linear_grad_W(x, grad_h, self.W, self.b)
        grad_b = linear_grad_b(x, grad_h, self.W, self.b)
        # update params
        self.W = self.W - self.lr * grad_W
        self.b = self.b - self.lr * grad_b
```

## 1.5 Simple Experiment


```python
import matplotlib.pyplot as plt
%matplotlib inline
```


```python
# Generate some data
def generate_2_circles(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = 1.1 * np.array([np.sin(phi), np.cos(phi)])
    X2 = 3.0 * np.array([np.sin(phi), np.cos(phi)])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.hstack([X1,X2]).T
    return X, Y


def generate_2_gaussians(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = np.random.normal(loc=[1, 2], scale=[2.5, 0.9], size=(N, 2))
    X1 = X1 @ np.array([[0.7, -0.7], [0.7, 0.7]])
    X2 = np.random.normal(loc=[-2, 0], scale=[1, 1.5], size=(N, 2))
    X2 = X2 @ np.array([[0.7, 0.7], [-0.7, 0.7]])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.vstack([X1,X2])
    return X, Y

def split(X, Y, train_ratio=0.7):
    size = len(X)
    train_size = int(size * train_ratio)
    indices = np.arange(size)
    np.random.shuffle(indices)
    train_indices = indices[:train_size]
    test_indices = indices[train_size:]
    return X[train_indices], Y[train_indices], X[test_indices], Y[test_indices]

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4))


X, Y = generate_2_circles()
ax1.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax1.set_aspect('equal')


X, Y = generate_2_gaussians()
ax2.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax2.set_aspect('equal')

```


```python
X_train, Y_train, X_test, Y_test = split(*generate_2_gaussians(), 0.7)
```


```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)
    
    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))
    
    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

    
print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

    Step: 0 	 Loss: 0.522 	 Acc: 77.9%
    [[-0.92764698]
     [-0.01113851]]
    Step: 1 	 Loss: 0.480 	 Acc: 80.0%
    [[-0.86746426]
     [-0.01061837]]
    Step: 2 	 Loss: 0.444 	 Acc: 82.9%
    [[-0.81055189]
     [-0.01149613]]
    Step: 3 	 Loss: 0.412 	 Acc: 85.0%
    [[-0.75724237]
     [-0.01341063]]
    Step: 4 	 Loss: 0.384 	 Acc: 88.6%
    [[-0.70772329]
     [-0.01603633]]
    Step: 5 	 Loss: 0.360 	 Acc: 89.3%
    [[-0.66204704]
     [-0.01909625]]
    Step: 6 	 Loss: 0.339 	 Acc: 91.4%
    [[-0.62015142]
     [-0.02236729]]
    Step: 7 	 Loss: 0.320 	 Acc: 92.9%
    [[-0.58188578]
     [-0.02567919]]
    Step: 8 	 Loss: 0.304 	 Acc: 92.9%
    [[-0.54703794]
     [-0.02890938]]
    Step: 9 	 Loss: 0.289 	 Acc: 95.7%
    [[-0.51535851]
     [-0.03197552]]
    Step: 10 	 Loss: 0.276 	 Acc: 97.1%
    [[-0.48658091]
     [-0.03482735]]
    Step: 11 	 Loss: 0.265 	 Acc: 98.6%
    [[-0.46043652]
     [-0.03743891]]
    Step: 12 	 Loss: 0.254 	 Acc: 98.6%
    [[-0.43666537]
     [-0.03980171]]
    Step: 13 	 Loss: 0.245 	 Acc: 98.6%
    [[-0.41502294]
     [-0.04191917]]
    Step: 14 	 Loss: 0.236 	 Acc: 98.6%
    [[-0.39528393]
     [-0.04380229]]
    Step: 15 	 Loss: 0.229 	 Acc: 98.6%
    [[-0.3772439 ]
     [-0.04546652]]
    Step: 16 	 Loss: 0.222 	 Acc: 98.6%
    [[-0.36071936]
     [-0.04692944]]
    Step: 17 	 Loss: 0.215 	 Acc: 98.6%
    [[-0.34554689]
     [-0.04820933]]
    Step: 18 	 Loss: 0.209 	 Acc: 98.6%
    [[-0.33158174]
     [-0.04932411]]
    Step: 19 	 Loss: 0.204 	 Acc: 98.6%
    [[-0.31869615]
     [-0.05029083]]
    Step: 20 	 Loss: 0.199 	 Acc: 98.6%
    [[-0.30677754]
     [-0.05112531]]
    Step: 21 	 Loss: 0.194 	 Acc: 98.6%
    [[-0.29572686]
     [-0.05184198]]
    Step: 22 	 Loss: 0.189 	 Acc: 98.6%
    [[-0.28545688]
     [-0.05245392]]
    Step: 23 	 Loss: 0.185 	 Acc: 98.6%
    [[-0.27589082]
     [-0.05297283]]
    Step: 24 	 Loss: 0.181 	 Acc: 98.6%
    [[-0.26696093]
     [-0.05340917]]
    Step: 25 	 Loss: 0.178 	 Acc: 98.6%
    [[-0.25860741]
     [-0.05377221]]
    Step: 26 	 Loss: 0.174 	 Acc: 98.6%
    [[-0.25077733]
     [-0.05407021]]
    Step: 27 	 Loss: 0.171 	 Acc: 98.6%
    [[-0.24342376]
     [-0.05431044]]
    Step: 28 	 Loss: 0.168 	 Acc: 98.6%
    [[-0.23650501]
     [-0.05449937]]
    Step: 29 	 Loss: 0.165 	 Acc: 98.6%
    [[-0.22998396]
     [-0.05464269]]
    
    
    Testing...
    Acc: 96.7%



```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()
    
plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```


```python
# Now run the same experiment on 2 circles
```

# 2. Decision Tree
The next model we look at is called **Decision Tree**. This type of model is non-parametric, meaning in contrast to **Logistic Regression** we do not have any parameters here that need to be trained.

Let us consider a simple binary decision tree for deciding on the two classes of "creditable" and "Not creditable".



Each node, except the leafs, asks a question about the the client in question. A decision is made by going from the root node to a leaf node, while considering the clients situation. The situation of the client, in this case, is fully described by the features:
1. Checking account balance
2. Duration of requested credit
3. Payment status of previous loan
4. Length of current employment

In order to build a decision tree we need training data. To carry on the previous example: we need a number of clients for which we know the properties 1.-4. and their creditability.
The process of building a decision tree starts with the root node and involves the following steps:
1. Choose a splitting criteria and add it to the current node.
2. Split the dataset at the current node into those that fullfil the criteria and those that do not.
3. Add a child node for each data split.
4. For each child node decide on either A. or B.:
    1. Repeat from 1. step
    2. Make it a leaf node: The predicted class label is decided by the majority vote over the training data in the current split.

## 2.1 Gini Index & Data Split
Deciding on how to split your training data at each node is dominated by the following two criterias:
1. Does the rule help me make a final decision?
2. Is the rule general enough such that it applies not only to my training data, but also to new unseen examples?

When considering our previous example, splitting the clients by their handedness would not help us deciding on their creditability. Knowning if a rule will generalize is usually a hard call to make, but in practice we rely on [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor) principle. Thus the less rules we use, the better we believe it to generalize to previously unseen examples.

One way to measure the quality of a rule is by the [**Gini Index**](https://en.wikipedia.org/wiki/Gini_coefficient).
Since we only consider binary classification, it is calculated by:
$$
Gini = \sum_{n\in\{L,R\}}\frac{|S_n|}{|S|}\left( 1 - \sum_{c \in C} p_{S_n}(c)^2\right)\\
p_{S_n}(c) = \frac{|\{\mathbf{x}_{i}\in \mathbf{X}|y_{i} = c, i \in S_n\}|}{|S_n|}, n \in \{L, R\}
$$
with $|C|=2$ being your set of class labels and $S_L$ and $S_R$ the two splits determined by the splitting criteria.
While we only consider two class problems for decision trees, the method can also be applied when $|C|>2$.
The lower the gini score, the better the split. In the extreme case, where all class labels are the same in each split respectively, the gini index takes the value of $0$.


```python
def tree_gini_index(Y_left, Y_right, classes):
    """Compute the Gini Index.
    # Arguments
        Y_left: class labels of the data left set
            np.array of size `(n_objects, 1)`
        Y_right: class labels of the data right set
            np.array of size `(n_objects, 1)`
        classes: list of all class values
    # Output
        gini: scalar `float`
    """
    gini = 0.0
    all_inst = len(Y_left)+len(Y_right)
    
    gini += (len(Y_left)/all_inst)*(1-((len(Y_left[Y_left == classes[0]])/len(Y_left))**2 + 
                                       (len(Y_left[Y_left == classes[1]])/len(Y_left))**2))
    gini += (len(Y_right)/all_inst)*(1-((len(Y_right[Y_right == classes[0]])/len(Y_right))**2 + 
                                       (len(Y_right[Y_right == classes[1]])/len(Y_right))**2))
    
    #################
    ### YOUR CODE ###
    #################
    return gini
```


```python
am.test_student_function(username, tree_gini_index, ['Y_left', 'Y_right', 'classes'])
```

    Running local tests...
    answer:  0.4285714285714286
    out:  0.4285714285714286
    answer:  0.4098877980364656
    out:  0.4098877980364656
    answer:  0.4004884004884005
    out:  0.4004884004884005
    answer:  0.3717532467532467
    out:  0.3717532467532467
    answer:  0.4115384615384615
    out:  0.4115384615384615
    answer:  0.5
    out:  0.5
    answer:  0.461244019138756
    out:  0.461244019138756
    answer:  0.4977272727272727
    out:  0.4977272727272727
    answer:  0.27210884353741494
    out:  0.27210884353741494
    answer:  0.40877192982456134
    out:  0.40877192982456134
    answer:  0.48033126293995854
    out:  0.48033126293995854
    answer:  0.4822916666666666
    out:  0.4822916666666666
    answer:  0.3774104683195592
    out:  0.3774104683195592
    answer:  0.4926984126984127
    out:  0.4926984126984127
    answer:  0.49411764705882355
    out:  0.49411764705882355
    answer:  0.4617604617604619
    out:  0.4617604617604619
    answer:  0.4556277056277057
    out:  0.4556277056277057
    answer:  0.48674443266171785
    out:  0.48674443266171785
    answer:  0.4797979797979799
    out:  0.4797979797979799
    answer:  0.48033126293995854
    out:  0.48033126293995854
    answer:  0.48958333333333326
    out:  0.48958333333333326
    answer:  0.49586776859504145
    out:  0.49586776859504145
    answer:  0.464021164021164
    out:  0.464021164021164
    answer:  0.4666666666666667
    out:  0.4666666666666667
    answer:  0.46469622331691296
    out:  0.46469622331691296
    answer:  0.4587412587412587
    out:  0.4587412587412587
    answer:  0.45999999999999985
    out:  0.45999999999999985
    answer:  0.4740740740740741
    out:  0.4740740740740741
    answer:  0.4907407407407407
    out:  0.4907407407407407
    answer:  0.4913194444444445
    out:  0.4913194444444445
    answer:  0.4763285024154589
    out:  0.4763285024154589
    answer:  0.4688979039891818
    out:  0.4688979039891818
    answer:  0.4956521739130435
    out:  0.4956521739130435
    answer:  0.46315789473684205
    out:  0.46315789473684205
    answer:  0.4716117216117216
    out:  0.4716117216117216
    answer:  0.44323671497584544
    out:  0.44323671497584544
    answer:  0.4688644688644689
    out:  0.4688644688644689
    answer:  0.4778325123152709
    out:  0.4778325123152709
    answer:  0.49857549857549854
    out:  0.49857549857549854
    answer:  0.3120879120879121
    out:  0.3120879120879121
    answer:  0.46380308880308885
    out:  0.46380308880308885
    answer:  0.4782668312080076
    out:  0.4782668312080076
    answer:  0.43508771929824563
    out:  0.43508771929824563
    answer:  0.37972350230414736
    out:  0.37972350230414736
    answer:  0.4151260504201681
    out:  0.4151260504201681
    answer:  0.49760765550239244
    out:  0.49760765550239244
    answer:  0.44544997486173954
    out:  0.44544997486173954
    answer:  0.5
    out:  0.5
    answer:  0.38716577540106956
    out:  0.38716577540106956
    answer:  0.3356643356643357
    out:  0.3356643356643357
    answer:  0.47272727272727266
    out:  0.47272727272727266
    answer:  0.4591251885369533
    out:  0.4591251885369533
    answer:  0.4820261437908497
    out:  0.4820261437908497
    answer:  0.4798797409805735
    out:  0.4798797409805735
    answer:  0.4978354978354979
    out:  0.4978354978354979
    answer:  0.4827586206896552
    out:  0.4827586206896552
    answer:  0.48995148995148996
    out:  0.48995148995148996
    answer:  0.4841269841269841
    out:  0.4841269841269841
    answer:  0.465564738292011
    out:  0.465564738292011
    answer:  0.48674443266171785
    out:  0.48674443266171785
    answer:  0.419047619047619
    out:  0.419047619047619
    answer:  0.4708545557441992
    out:  0.4708545557441992
    answer:  0.40993788819875776
    out:  0.40993788819875776
    answer:  0.4461038961038961
    out:  0.4461038961038961
    answer:  0.4746977160770265
    out:  0.4746977160770265
    answer:  0.48484848484848486
    out:  0.48484848484848486
    answer:  0.36458333333333326
    out:  0.36458333333333326
    answer:  0.48615384615384616
    out:  0.48615384615384616
    answer:  0.4820261437908497
    out:  0.4820261437908497
    answer:  0.31999999999999984
    out:  0.31999999999999984
    answer:  0.48823529411764705
    out:  0.48823529411764705
    answer:  0.4318488529014845
    out:  0.4318488529014845
    answer:  0.4774114774114774
    out:  0.4774114774114774
    answer:  0.49109730848861294
    out:  0.49109730848861294
    answer:  0.48943488943488955
    out:  0.48943488943488955
    answer:  0.4591836734693877
    out:  0.4591836734693877
    answer:  0.44871794871794873
    out:  0.44871794871794873
    answer:  0.4835164835164835
    out:  0.4835164835164835
    answer:  0.4799999999999999
    out:  0.4799999999999999
    answer:  0.4022988505747126
    out:  0.4022988505747126
    answer:  0.4857142857142858
    out:  0.4857142857142858
    answer:  0.4117647058823529
    out:  0.4117647058823529
    answer:  0.49350649350649367
    out:  0.49350649350649367
    answer:  0.43333333333333335
    out:  0.43333333333333335
    answer:  0.37687969924812026
    out:  0.37687969924812026
    answer:  0.4444444444444444
    out:  0.4444444444444444
    answer:  0.3671328671328671
    out:  0.3671328671328671
    answer:  0.39470588235294124
    out:  0.39470588235294124
    answer:  0.38787878787878777
    out:  0.38787878787878777
    answer:  0.4895833333333334
    out:  0.4895833333333334
    answer:  0.49047619047619045
    out:  0.49047619047619045
    answer:  0.4946360153256705
    out:  0.4946360153256705
    answer:  0.48749999999999993
    out:  0.48749999999999993
    answer:  0.4666666666666667
    out:  0.4666666666666667
    answer:  0.4982698961937716
    out:  0.4982698961937716
    answer:  0.4163742690058479
    out:  0.4163742690058479
    answer:  0.4939740655987796
    out:  0.4939740655987796
    answer:  0.48347107438016523
    out:  0.48347107438016523
    answer:  0.47733580018501387
    out:  0.47733580018501387
    answer:  0.4817813765182186
    out:  0.4817813765182186
    tree_gini_index successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!


At each node in the tree, the data is split according to a split criterion and each split is passed onto the left/right child respectively.
Implement the following function to return all rows in `X` and `Y` such that the left child gets all examples that are less than the split value and vice versa. 


```python
def tree_split_data_left(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is less than `split_value` then return it as part of the left group.
    
    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at 
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_left, Y_left = None, None
    #################
    ### YOUR CODE ###
    #################
    X_left = X[X[:,feature_index] < split_value]
    Y_left = Y[X[:,feature_index] < split_value]
            
    XY_left = np.concatenate([X_left, Y_left], axis=-1)
    return XY_left


def tree_split_data_right(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is greater or equal than `split_value` then return it as part of the right group.
    
    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_right, Y_right = None, None
    
    X_right = X[X[:,feature_index] >= split_value]
    Y_right = Y[X[:,feature_index] >= split_value]
    
    XY_right = np.concatenate([X_right, Y_right], axis=-1)
    return XY_right
```


```python
am.test_student_function(username, tree_split_data_left, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    answer:  [[-2.37556118 -2.60844971  1.        ]
     [ 4.26463158 -1.65238926  0.        ]]
    out:  [[-2.37556118 -2.60844971  1.        ]
     [ 4.26463158 -1.65238926  0.        ]]
    answer:  [[ 1.94588175 -1.3034951   1.        ]
     [-0.63283282 -2.97400198  1.        ]
     [-1.7153846   0.74070777  0.        ]
     [-5.79411975 -0.40943116  0.        ]
     [-2.71633746 -1.64726742  0.        ]]
    out:  [[ 1.94588175 -1.3034951   1.        ]
     [-0.63283282 -2.97400198  1.        ]
     [-1.7153846   0.74070777  0.        ]
     [-5.79411975 -0.40943116  0.        ]
     [-2.71633746 -1.64726742  0.        ]]
    answer:  [[ 1.35448839 -0.2099724   0.        ]
     [ 7.77480498 -3.34950436  1.        ]]
    out:  [[ 1.35448839 -0.2099724   0.        ]
     [ 7.77480498 -3.34950436  1.        ]]
    answer:  [[ 0.87634404 -5.06310781  0.        ]
     [-6.03751724  1.05850821  1.        ]
     [-7.1926914  -6.50000432  0.        ]
     [ 8.51710716 -4.21724322  1.        ]]
    out:  [[ 0.87634404 -5.06310781  0.        ]
     [-6.03751724  1.05850821  1.        ]
     [-7.1926914  -6.50000432  0.        ]
     [ 8.51710716 -4.21724322  1.        ]]
    answer:  [[-9.36607929  3.73666788  1.        ]
     [-7.78355718  6.5810372   1.        ]
     [-6.11148523  7.90649653  0.        ]
     [-9.72034328  3.36824612  1.        ]
     [-8.94318009  0.07507696  0.        ]
     [-1.63738251  0.47235638  1.        ]
     [-9.03206994  8.00379159  1.        ]]
    out:  [[-9.36607929  3.73666788  1.        ]
     [-7.78355718  6.5810372   1.        ]
     [-6.11148523  7.90649653  0.        ]
     [-9.72034328  3.36824612  1.        ]
     [-8.94318009  0.07507696  0.        ]
     [-1.63738251  0.47235638  1.        ]
     [-9.03206994  8.00379159  1.        ]]
    answer:  [[-5.26073174 -1.49986052  1.        ]
     [-8.42770937  7.5915227   1.        ]]
    out:  [[-5.26073174 -1.49986052  1.        ]
     [-8.42770937  7.5915227   1.        ]]
    answer:  [[-9.10827565 -9.54824574  1.        ]
     [-9.96200291  2.5183704   0.        ]]
    out:  [[-9.10827565 -9.54824574  1.        ]
     [-9.96200291  2.5183704   0.        ]]
    answer:  [[ 2.65302163e+00  9.35304542e+00  0.00000000e+00]
     [ 1.87233797e+00  3.45427003e+00  0.00000000e+00]
     [-1.35570851e+00  3.76509218e+00  0.00000000e+00]
     [-4.42484811e+00  7.86466080e-01  1.00000000e+00]
     [-8.17879884e-03 -4.52946960e+00  1.00000000e+00]]
    out:  [[ 2.65302163e+00  9.35304542e+00  0.00000000e+00]
     [ 1.87233797e+00  3.45427003e+00  0.00000000e+00]
     [-1.35570851e+00  3.76509218e+00  0.00000000e+00]
     [-4.42484811e+00  7.86466080e-01  1.00000000e+00]
     [-8.17879884e-03 -4.52946960e+00  1.00000000e+00]]
    answer:  [[-4.37185642 -4.87225659  0.        ]
     [-9.9353802   9.33881489  1.        ]
     [ 5.04790996  6.30033027  1.        ]]
    out:  [[-4.37185642 -4.87225659  0.        ]
     [-9.9353802   9.33881489  1.        ]
     [ 5.04790996  6.30033027  1.        ]]
    answer:  [[-7.28596597 -5.57235412  0.        ]
     [-5.92654563  4.87763988  1.        ]
     [-8.39325723 -7.21426803  0.        ]
     [-2.37405564  4.65203543  1.        ]]
    out:  [[-7.28596597 -5.57235412  0.        ]
     [-5.92654563  4.87763988  1.        ]
     [-8.39325723 -7.21426803  0.        ]
     [-2.37405564  4.65203543  1.        ]]
    answer:  [[ 0.21113109 -8.82799123  0.        ]
     [-2.61553065  9.23804143  1.        ]
     [-1.78688084  9.37506723  1.        ]]
    out:  [[ 0.21113109 -8.82799123  0.        ]
     [-2.61553065  9.23804143  1.        ]
     [-1.78688084  9.37506723  1.        ]]
    answer:  [[ 6.31354453 -8.18307638  0.        ]
     [-1.19612965 -9.87256673  0.        ]
     [ 4.81831431 -3.75757405  1.        ]]
    out:  [[ 6.31354453 -8.18307638  0.        ]
     [-1.19612965 -9.87256673  0.        ]
     [ 4.81831431 -3.75757405  1.        ]]
    answer:  [[ 4.44463904 -4.65117093  1.        ]
     [-5.61413303 -6.88834569  1.        ]
     [-1.95748231 -2.88653308  0.        ]
     [ 3.70540254 -3.001772    0.        ]
     [-4.390432   -3.67733266  0.        ]
     [-3.36110821 -9.23287607  1.        ]]
    out:  [[ 4.44463904 -4.65117093  1.        ]
     [-5.61413303 -6.88834569  1.        ]
     [-1.95748231 -2.88653308  0.        ]
     [ 3.70540254 -3.001772    0.        ]
     [-4.390432   -3.67733266  0.        ]
     [-3.36110821 -9.23287607  1.        ]]
    answer:  [[-5.77153704 -0.76438147  1.        ]
     [ 5.00316785  0.118027    0.        ]
     [ 1.26612951  2.20084614  0.        ]
     [ 9.74702944  0.29475347  1.        ]
     [ 2.03368119 -8.67896275  0.        ]
     [ 3.61945011 -0.8396783   0.        ]]
    out:  [[-5.77153704 -0.76438147  1.        ]
     [ 5.00316785  0.118027    0.        ]
     [ 1.26612951  2.20084614  0.        ]
     [ 9.74702944  0.29475347  1.        ]
     [ 2.03368119 -8.67896275  0.        ]
     [ 3.61945011 -0.8396783   0.        ]]
    answer:  [[ 0.58533874  1.6610768   0.        ]
     [-8.35178568 -9.86183915  0.        ]
     [-3.21501887  9.26737555  1.        ]
     [ 0.74656272 -2.10365632  1.        ]
     [ 1.54680626 -8.67605197  1.        ]]
    out:  [[ 0.58533874  1.6610768   0.        ]
     [-8.35178568 -9.86183915  0.        ]
     [-3.21501887  9.26737555  1.        ]
     [ 0.74656272 -2.10365632  1.        ]
     [ 1.54680626 -8.67605197  1.        ]]
    answer:  [[-7.23122502  3.62082515  1.        ]
     [-5.75659807  0.20713906  1.        ]
     [-4.70347676 -2.66121154  0.        ]]
    out:  [[-7.23122502  3.62082515  1.        ]
     [-5.75659807  0.20713906  1.        ]
     [-4.70347676 -2.66121154  0.        ]]
    answer:  [[ 1.38041332 -7.90304597  0.        ]
     [-2.49852907 -1.09573896  0.        ]
     [-2.30090537 -7.65231574  1.        ]
     [-7.3139662  -0.81838309  0.        ]]
    out:  [[ 1.38041332 -7.90304597  0.        ]
     [-2.49852907 -1.09573896  0.        ]
     [-2.30090537 -7.65231574  1.        ]
     [-7.3139662  -0.81838309  0.        ]]
    answer:  [[ 9.64559172 -9.00872846  0.        ]
     [-0.05324152 -6.87058815  0.        ]
     [-0.89077297 -6.65015207  0.        ]
     [ 1.48407158 -8.26599853  0.        ]]
    out:  [[ 9.64559172 -9.00872846  0.        ]
     [-0.05324152 -6.87058815  0.        ]
     [-0.89077297 -6.65015207  0.        ]
     [ 1.48407158 -8.26599853  0.        ]]
    answer:  [[-7.3148247  -3.80817133  1.        ]
     [-3.16647982 -8.34950305  1.        ]
     [-3.44523248  3.47640827  1.        ]
     [-5.52210163  7.6644342   1.        ]
     [-0.34789753 -1.19014071  0.        ]]
    out:  [[-7.3148247  -3.80817133  1.        ]
     [-3.16647982 -8.34950305  1.        ]
     [-3.44523248  3.47640827  1.        ]
     [-5.52210163  7.6644342   1.        ]
     [-0.34789753 -1.19014071  0.        ]]
    answer:  [[ 8.98015199 -8.86684139  0.        ]
     [ 7.09268214  5.29217057  1.        ]
     [-4.45964554 -7.99198627  1.        ]
     [-5.68531485 -3.64304732  1.        ]
     [ 4.21270522  4.32527494  1.        ]
     [ 4.7601182  -6.18756936  0.        ]]
    out:  [[ 8.98015199 -8.86684139  0.        ]
     [ 7.09268214  5.29217057  1.        ]
     [-4.45964554 -7.99198627  1.        ]
     [-5.68531485 -3.64304732  1.        ]
     [ 4.21270522  4.32527494  1.        ]
     [ 4.7601182  -6.18756936  0.        ]]
    answer:  [[ 6.66435037 -3.87849513  1.        ]
     [-4.36138513 -2.43366449  0.        ]
     [ 6.05228506  0.68493948  1.        ]
     [-8.44336263  1.07215902  1.        ]]
    out:  [[ 6.66435037 -3.87849513  1.        ]
     [-4.36138513 -2.43366449  0.        ]
     [ 6.05228506  0.68493948  1.        ]
     [-8.44336263  1.07215902  1.        ]]
    answer:  [[ 5.84066412 -6.09674715  1.        ]
     [ 6.04593831 -8.91448727  0.        ]
     [ 5.94609503 -4.94677465  0.        ]
     [-6.67897898 -4.14387699  1.        ]]
    out:  [[ 5.84066412 -6.09674715  1.        ]
     [ 6.04593831 -8.91448727  0.        ]
     [ 5.94609503 -4.94677465  0.        ]
     [-6.67897898 -4.14387699  1.        ]]
    answer:  [[-2.25284259  0.98189791  0.        ]
     [-7.90012135  1.90272944  0.        ]
     [ 3.45246897  6.28514938  0.        ]]
    out:  [[-2.25284259  0.98189791  0.        ]
     [-7.90012135  1.90272944  0.        ]
     [ 3.45246897  6.28514938  0.        ]]
    answer:  [[-2.91808204  0.03218935  0.        ]
     [-4.44021341 -9.04475902  0.        ]
     [-5.97246031 -1.54516754  0.        ]
     [ 3.80843555 -2.55212295  1.        ]]
    out:  [[-2.91808204  0.03218935  0.        ]
     [-4.44021341 -9.04475902  0.        ]
     [-5.97246031 -1.54516754  0.        ]
     [ 3.80843555 -2.55212295  1.        ]]
    answer:  [[-2.5560739  -2.60592515  0.        ]
     [-0.5263865  -6.42367064  0.        ]
     [ 8.74711608 -8.6728326   1.        ]
     [ 6.6724728  -4.36396978  0.        ]]
    out:  [[-2.5560739  -2.60592515  0.        ]
     [-0.5263865  -6.42367064  0.        ]
     [ 8.74711608 -8.6728326   1.        ]
     [ 6.6724728  -4.36396978  0.        ]]
    answer:  [[-6.96564368 -0.27322537  0.        ]
     [-9.9980993  -7.84847771  0.        ]
     [-7.27905356  0.74250288  1.        ]]
    out:  [[-6.96564368 -0.27322537  0.        ]
     [-9.9980993  -7.84847771  0.        ]
     [-7.27905356  0.74250288  1.        ]]
    answer:  [[-0.38360825 -9.57747752  1.        ]
     [-5.48225289  0.47083727  0.        ]]
    out:  [[-0.38360825 -9.57747752  1.        ]
     [-5.48225289  0.47083727  0.        ]]
    answer:  [[-5.45927576 -4.54719023  1.        ]
     [-5.4209009  -5.67326749  0.        ]
     [-6.71441608 -3.85110805  1.        ]
     [-2.87490704  4.41993717  1.        ]]
    out:  [[-5.45927576 -4.54719023  1.        ]
     [-5.4209009  -5.67326749  0.        ]
     [-6.71441608 -3.85110805  1.        ]
     [-2.87490704  4.41993717  1.        ]]
    answer:  [[-4.90506179 -5.33466372  0.        ]
     [-2.53472208  4.29138841  0.        ]]
    out:  [[-4.90506179 -5.33466372  0.        ]
     [-2.53472208  4.29138841  0.        ]]
    answer:  [[-0.59299695 -3.2942172   0.        ]
     [-5.44373928 -7.32666865  1.        ]
     [-2.79712023 -6.83112458  1.        ]]
    out:  [[-0.59299695 -3.2942172   0.        ]
     [-5.44373928 -7.32666865  1.        ]
     [-2.79712023 -6.83112458  1.        ]]
    answer:  [[-3.68718498  4.09794462  1.        ]
     [-2.31102569 -5.55203413  1.        ]
     [-5.85750672 -5.59810463  0.        ]
     [-3.9142254  -6.4061488   1.        ]
     [-4.34621776  7.54341135  1.        ]]
    out:  [[-3.68718498  4.09794462  1.        ]
     [-2.31102569 -5.55203413  1.        ]
     [-5.85750672 -5.59810463  0.        ]
     [-3.9142254  -6.4061488   1.        ]
     [-4.34621776  7.54341135  1.        ]]
    answer:  [[ 2.00909198  9.35851907  1.        ]
     [-8.77972521 -9.68834571  0.        ]
     [-4.31129695 -9.7187875   1.        ]
     [ 2.33509167 -4.27897808  0.        ]
     [-6.56337944 -7.17549142  1.        ]
     [-3.2334858  -2.20819085  1.        ]
     [-8.63337174  7.00434138  0.        ]]
    out:  [[ 2.00909198  9.35851907  1.        ]
     [-8.77972521 -9.68834571  0.        ]
     [-4.31129695 -9.7187875   1.        ]
     [ 2.33509167 -4.27897808  0.        ]
     [-6.56337944 -7.17549142  1.        ]
     [-3.2334858  -2.20819085  1.        ]
     [-8.63337174  7.00434138  0.        ]]
    answer:  [[ 9.12897885 -4.36342744  0.        ]
     [-0.34338788 -0.59414767  0.        ]
     [ 0.43096634 -2.69557846  0.        ]]
    out:  [[ 9.12897885 -4.36342744  0.        ]
     [-0.34338788 -0.59414767  0.        ]
     [ 0.43096634 -2.69557846  0.        ]]
    answer:  [[ 5.06496341 -6.77222192  1.        ]
     [-1.74686603 -5.8720616   0.        ]]
    out:  [[ 5.06496341 -6.77222192  1.        ]
     [-1.74686603 -5.8720616   0.        ]]
    answer:  [[-8.28359946 -1.19299617  1.        ]
     [-3.63763375  6.71175893  1.        ]
     [-6.56600439  4.8218078   1.        ]
     [-8.55821171  9.28315334  0.        ]]
    out:  [[-8.28359946 -1.19299617  1.        ]
     [-3.63763375  6.71175893  1.        ]
     [-6.56600439  4.8218078   1.        ]
     [-8.55821171  9.28315334  0.        ]]
    answer:  [[-0.94286557 -4.08069731  0.        ]
     [-2.73686439 -2.68371506  1.        ]
     [-3.65253517  8.56299482  1.        ]
     [-5.66654458  9.39629416  1.        ]
     [ 0.59048795 -9.2088475   0.        ]]
    out:  [[-0.94286557 -4.08069731  0.        ]
     [-2.73686439 -2.68371506  1.        ]
     [-3.65253517  8.56299482  1.        ]
     [-5.66654458  9.39629416  1.        ]
     [ 0.59048795 -9.2088475   0.        ]]
    answer:  [[-9.48821184  4.82324189  1.        ]
     [-8.62008202  0.49974124  1.        ]]
    out:  [[-9.48821184  4.82324189  1.        ]
     [-8.62008202  0.49974124  1.        ]]
    answer:  [[ 2.47058779 -5.91974156  0.        ]
     [-4.10266291 -9.14134166  0.        ]]
    out:  [[ 2.47058779 -5.91974156  0.        ]
     [-4.10266291 -9.14134166  0.        ]]
    answer:  [[-3.72601191 -6.27172069  0.        ]
     [ 8.78643858 -6.59278394  1.        ]]
    out:  [[-3.72601191 -6.27172069  0.        ]
     [ 8.78643858 -6.59278394  1.        ]]
    answer:  [[-4.72032533  9.18458634  0.        ]
     [-1.61920226  9.05952796  1.        ]
     [-8.06234418 -6.27702704  0.        ]
     [-9.45923646  2.52972819  0.        ]]
    out:  [[-4.72032533  9.18458634  0.        ]
     [-1.61920226  9.05952796  1.        ]
     [-8.06234418 -6.27702704  0.        ]
     [-9.45923646  2.52972819  0.        ]]
    answer:  [[-0.45162584 -3.2335316   1.        ]
     [-1.28831511 -6.84237011  0.        ]
     [-0.15308094 -7.77799513  1.        ]]
    out:  [[-0.45162584 -3.2335316   1.        ]
     [-1.28831511 -6.84237011  0.        ]
     [-0.15308094 -7.77799513  1.        ]]
    answer:  [[-8.19713698 -3.63722183  0.        ]
     [-7.97172673 -3.22334808  1.        ]
     [-6.41521615 -1.18134938  1.        ]
     [-9.80017337 -5.53944806  1.        ]]
    out:  [[-8.19713698 -3.63722183  0.        ]
     [-7.97172673 -3.22334808  1.        ]
     [-6.41521615 -1.18134938  1.        ]
     [-9.80017337 -5.53944806  1.        ]]
    answer:  [[-4.63433067 -8.81983161  0.        ]
     [-3.84617438 -6.36269861  1.        ]]
    out:  [[-4.63433067 -8.81983161  0.        ]
     [-3.84617438 -6.36269861  1.        ]]
    answer:  [[-6.53397699 -2.79657874  0.        ]
     [ 0.17705281 -4.60140242  0.        ]]
    out:  [[-6.53397699 -2.79657874  0.        ]
     [ 0.17705281 -4.60140242  0.        ]]
    answer:  [[-6.60118327  1.63110532  0.        ]
     [ 4.68722109  0.81532126  0.        ]]
    out:  [[-6.60118327  1.63110532  0.        ]
     [ 4.68722109  0.81532126  0.        ]]
    answer:  [[ 8.9204833  -5.9258933   0.        ]
     [-6.4066338  -1.57470596  1.        ]]
    out:  [[ 8.9204833  -5.9258933   0.        ]
     [-6.4066338  -1.57470596  1.        ]]
    answer:  [[-2.73179573 -4.19069926  1.        ]
     [-3.41452237 -6.67316056  1.        ]]
    out:  [[-2.73179573 -4.19069926  1.        ]
     [-3.41452237 -6.67316056  1.        ]]
    answer:  [[-3.4961022  -9.69119261  1.        ]
     [ 9.85718459 -7.11974872  0.        ]
     [-2.93010878 -6.38764807  1.        ]
     [ 6.54003422 -8.22089358  1.        ]
     [ 2.50436642 -5.72641979  1.        ]]
    out:  [[-3.4961022  -9.69119261  1.        ]
     [ 9.85718459 -7.11974872  0.        ]
     [-2.93010878 -6.38764807  1.        ]
     [ 6.54003422 -8.22089358  1.        ]
     [ 2.50436642 -5.72641979  1.        ]]
    answer:  [[ 2.04304953 -2.66691743  0.        ]
     [ 2.11998695 -6.95996898  0.        ]
     [ 7.39505486 -8.4452957   0.        ]]
    out:  [[ 2.04304953 -2.66691743  0.        ]
     [ 2.11998695 -6.95996898  0.        ]
     [ 7.39505486 -8.4452957   0.        ]]
    answer:  [[-9.26440348  4.94472464  1.        ]
     [-7.1319719  -2.45027692  1.        ]
     [-0.96638402 -4.84270652  1.        ]
     [-8.04817681 -8.98948527  0.        ]]
    out:  [[-9.26440348  4.94472464  1.        ]
     [-7.1319719  -2.45027692  1.        ]
     [-0.96638402 -4.84270652  1.        ]
     [-8.04817681 -8.98948527  0.        ]]
    answer:  [[ 8.00383431 -9.87060869  0.        ]
     [-1.6114016  -6.58899118  1.        ]
     [-6.8729875  -4.4155576   1.        ]]
    out:  [[ 8.00383431 -9.87060869  0.        ]
     [-1.6114016  -6.58899118  1.        ]
     [-6.8729875  -4.4155576   1.        ]]
    answer:  [[-2.38212539 -3.02213617  1.        ]
     [-4.57920761  2.91650932  1.        ]
     [ 3.23766286  4.55335858  1.        ]
     [-7.2921603   3.45951084  1.        ]
     [-8.97675941  5.09661811  0.        ]
     [ 0.03473972  7.26624062  0.        ]]
    out:  [[-2.38212539 -3.02213617  1.        ]
     [-4.57920761  2.91650932  1.        ]
     [ 3.23766286  4.55335858  1.        ]
     [-7.2921603   3.45951084  1.        ]
     [-8.97675941  5.09661811  0.        ]
     [ 0.03473972  7.26624062  0.        ]]
    answer:  [[-8.38217673 -1.45517537  0.        ]
     [-5.70689777 -5.04001407  0.        ]]
    out:  [[-8.38217673 -1.45517537  0.        ]
     [-5.70689777 -5.04001407  0.        ]]
    answer:  [[-3.87393233  9.35165637  0.        ]
     [-5.93469357  9.79240232  1.        ]]
    out:  [[-3.87393233  9.35165637  0.        ]
     [-5.93469357  9.79240232  1.        ]]
    answer:  [[ 0.12488474 -8.71189933  0.        ]
     [-7.53680306 -5.88116334  1.        ]
     [-5.08998958  3.797166    0.        ]]
    out:  [[ 0.12488474 -8.71189933  0.        ]
     [-7.53680306 -5.88116334  1.        ]
     [-5.08998958  3.797166    0.        ]]
    answer:  [[-7.24691362  6.97272631  0.        ]
     [-7.43552393  8.12300635  0.        ]
     [-8.18648773  0.63600758  1.        ]]
    out:  [[-7.24691362  6.97272631  0.        ]
     [-7.43552393  8.12300635  0.        ]
     [-8.18648773  0.63600758  1.        ]]
    answer:  [[-5.85207908 -0.59122037  0.        ]
     [ 1.61606968 -6.87914746  1.        ]
     [ 2.77498938 -0.46787719  0.        ]
     [-5.36533543 -8.65924707  0.        ]
     [ 2.02037997 -4.76943075  1.        ]]
    out:  [[-5.85207908 -0.59122037  0.        ]
     [ 1.61606968 -6.87914746  1.        ]
     [ 2.77498938 -0.46787719  0.        ]
     [-5.36533543 -8.65924707  0.        ]
     [ 2.02037997 -4.76943075  1.        ]]
    answer:  [[-6.33287127  9.53456732  0.        ]
     [-7.10420299 -2.95333293  0.        ]
     [-0.08443264  5.84655359  0.        ]
     [-1.58121719  7.27356233  1.        ]
     [-2.94707113 -1.84666023  0.        ]]
    out:  [[-6.33287127  9.53456732  0.        ]
     [-7.10420299 -2.95333293  0.        ]
     [-0.08443264  5.84655359  0.        ]
     [-1.58121719  7.27356233  1.        ]
     [-2.94707113 -1.84666023  0.        ]]
    answer:  [[-2.87398601 -8.99548391  0.        ]
     [-3.7649271  -2.50289884  1.        ]
     [-2.68129205 -0.01257792  1.        ]
     [-4.62628445  6.03429738  1.        ]
     [-7.49375538  2.40902251  0.        ]
     [-7.18118428 -1.2023141   1.        ]]
    out:  [[-2.87398601 -8.99548391  0.        ]
     [-3.7649271  -2.50289884  1.        ]
     [-2.68129205 -0.01257792  1.        ]
     [-4.62628445  6.03429738  1.        ]
     [-7.49375538  2.40902251  0.        ]
     [-7.18118428 -1.2023141   1.        ]]
    answer:  [[-3.66970987 -4.54154617  1.        ]
     [-0.04433954  2.65642204  1.        ]
     [-8.0981359   0.87803743  1.        ]
     [-4.76792146 -7.00028031  1.        ]
     [-4.99180092 -5.35301201  0.        ]
     [ 8.19457303  2.59606365  1.        ]]
    out:  [[-3.66970987 -4.54154617  1.        ]
     [-0.04433954  2.65642204  1.        ]
     [-8.0981359   0.87803743  1.        ]
     [-4.76792146 -7.00028031  1.        ]
     [-4.99180092 -5.35301201  0.        ]
     [ 8.19457303  2.59606365  1.        ]]
    answer:  [[-4.84935715  6.7547206   0.        ]
     [-8.01174707  8.93445182  1.        ]]
    out:  [[-4.84935715  6.7547206   0.        ]
     [-8.01174707  8.93445182  1.        ]]
    answer:  [[-5.50090019 -6.71965793  0.        ]
     [-6.2562681   5.3800424   0.        ]
     [-1.96759371  8.82984571  0.        ]]
    out:  [[-5.50090019 -6.71965793  0.        ]
     [-6.2562681   5.3800424   0.        ]
     [-1.96759371  8.82984571  0.        ]]
    answer:  [[ 1.74431094  6.95163073  1.        ]
     [-3.82743019  6.64104529  0.        ]
     [-0.02901706  8.48311679  0.        ]]
    out:  [[ 1.74431094  6.95163073  1.        ]
     [-3.82743019  6.64104529  0.        ]
     [-0.02901706  8.48311679  0.        ]]
    answer:  [[-7.58259261 -9.91977316  1.        ]
     [-2.40194889  3.84509431  1.        ]
     [-7.93730542 -4.5259871   0.        ]
     [-1.96065237  7.60936619  0.        ]
     [-6.99162731 -8.87091189  1.        ]]
    out:  [[-7.58259261 -9.91977316  1.        ]
     [-2.40194889  3.84509431  1.        ]
     [-7.93730542 -4.5259871   0.        ]
     [-1.96065237  7.60936619  0.        ]
     [-6.99162731 -8.87091189  1.        ]]
    answer:  [[-6.77785272 -6.54742583  1.        ]
     [-0.34029011 -0.23541973  0.        ]
     [ 2.78476849 -6.03841927  0.        ]]
    out:  [[-6.77785272 -6.54742583  1.        ]
     [-0.34029011 -0.23541973  0.        ]
     [ 2.78476849 -6.03841927  0.        ]]
    answer:  [[-3.96488507 -0.35617297  1.        ]
     [-2.59579459 -6.02516011  1.        ]]
    out:  [[-3.96488507 -0.35617297  1.        ]
     [-2.59579459 -6.02516011  1.        ]]
    answer:  [[ 8.58922241 -8.04297604  0.        ]
     [-1.71552812 -6.81573813  0.        ]]
    out:  [[ 8.58922241 -8.04297604  0.        ]
     [-1.71552812 -6.81573813  0.        ]]
    answer:  [[-9.63881272  4.00267662  1.        ]
     [-5.03307598 -0.10750789  1.        ]
     [-7.29440437 -5.83795654  1.        ]
     [-9.36169689  8.10434713  0.        ]]
    out:  [[-9.63881272  4.00267662  1.        ]
     [-5.03307598 -0.10750789  1.        ]
     [-7.29440437 -5.83795654  1.        ]
     [-9.36169689  8.10434713  0.        ]]
    answer:  [[-2.50417291 -8.36174898  0.        ]
     [-5.95913587 -9.5684707   0.        ]
     [-1.05512041  7.00617975  0.        ]
     [ 0.94759405  6.00070173  1.        ]]
    out:  [[-2.50417291 -8.36174898  0.        ]
     [-5.95913587 -9.5684707   0.        ]
     [-1.05512041  7.00617975  0.        ]
     [ 0.94759405  6.00070173  1.        ]]
    answer:  [[-9.41290257  5.23155346  0.        ]
     [-7.22303734  2.60855153  0.        ]
     [-9.74225545  2.29896043  0.        ]]
    out:  [[-9.41290257  5.23155346  0.        ]
     [-7.22303734  2.60855153  0.        ]
     [-9.74225545  2.29896043  0.        ]]
    answer:  [[-6.55566772  4.81986286  0.        ]
     [ 6.05011921 -4.6320378   1.        ]
     [ 4.73741385  2.59345509  1.        ]]
    out:  [[-6.55566772  4.81986286  0.        ]
     [ 6.05011921 -4.6320378   1.        ]
     [ 4.73741385  2.59345509  1.        ]]
    answer:  [[-6.78707299  2.57737964  1.        ]
     [-3.22153838 -2.06931252  0.        ]]
    out:  [[-6.78707299  2.57737964  1.        ]
     [-3.22153838 -2.06931252  0.        ]]
    answer:  [[-9.55202319 -9.94102357  1.        ]
     [ 4.27491433 -7.78196405  1.        ]
     [ 6.78600825 -3.51659557  1.        ]]
    out:  [[-9.55202319 -9.94102357  1.        ]
     [ 4.27491433 -7.78196405  1.        ]
     [ 6.78600825 -3.51659557  1.        ]]
    answer:  [[-0.93970228  4.40791271  0.        ]
     [-4.41035194  4.65295999  1.        ]
     [-7.3572601  -7.07251393  0.        ]]
    out:  [[-0.93970228  4.40791271  0.        ]
     [-4.41035194  4.65295999  1.        ]
     [-7.3572601  -7.07251393  0.        ]]
    answer:  [[ 2.02528737  3.01206302  0.        ]
     [-2.39231441 -6.050712    1.        ]
     [ 0.37863343  5.42265122  1.        ]
     [-8.96346952 -7.6930434   1.        ]]
    out:  [[ 2.02528737  3.01206302  0.        ]
     [-2.39231441 -6.050712    1.        ]
     [ 0.37863343  5.42265122  1.        ]
     [-8.96346952 -7.6930434   1.        ]]
    answer:  [[-6.26624723 -5.16014257  1.        ]
     [-6.70061154  1.29390345  1.        ]]
    out:  [[-6.26624723 -5.16014257  1.        ]
     [-6.70061154  1.29390345  1.        ]]
    answer:  [[-3.62149249 -6.92291217  0.        ]
     [ 1.18257795 -9.63747597  0.        ]]
    out:  [[-3.62149249 -6.92291217  0.        ]
     [ 1.18257795 -9.63747597  0.        ]]
    answer:  [[-9.9347983  -9.22290294  0.        ]
     [-3.43343251  9.22024277  1.        ]
     [-7.54835663  9.39324371  1.        ]
     [-3.00011844 -0.08409379  0.        ]]
    out:  [[-9.9347983  -9.22290294  0.        ]
     [-3.43343251  9.22024277  1.        ]
     [-7.54835663  9.39324371  1.        ]
     [-3.00011844 -0.08409379  0.        ]]
    answer:  [[-8.19588147 -9.37485802  1.        ]
     [ 6.07354263 -3.1779417   1.        ]
     [-5.18229015 -9.53823914  1.        ]]
    out:  [[-8.19588147 -9.37485802  1.        ]
     [ 6.07354263 -3.1779417   1.        ]
     [-5.18229015 -9.53823914  1.        ]]
    answer:  [[ 0.08052513  1.27516653  1.        ]
     [-7.28213357 -7.056836    1.        ]
     [-7.03800875 -5.10133603  0.        ]
     [-6.08436142  5.36328988  0.        ]]
    out:  [[ 0.08052513  1.27516653  1.        ]
     [-7.28213357 -7.056836    1.        ]
     [-7.03800875 -5.10133603  0.        ]
     [-6.08436142  5.36328988  0.        ]]
    answer:  [[ 9.83764274  2.46952373  1.        ]
     [-0.06915623 -0.76587765  0.        ]
     [-8.95868384 -9.99781408  0.        ]]
    out:  [[ 9.83764274  2.46952373  1.        ]
     [-0.06915623 -0.76587765  0.        ]
     [-8.95868384 -9.99781408  0.        ]]
    answer:  [[ 4.72360431 -4.55592201  0.        ]
     [-1.97286413 -3.08672912  1.        ]]
    out:  [[ 4.72360431 -4.55592201  0.        ]
     [-1.97286413 -3.08672912  1.        ]]
    answer:  [[ 1.43576384 -8.99474414  1.        ]
     [-3.36516254 -6.4934647   0.        ]]
    out:  [[ 1.43576384 -8.99474414  1.        ]
     [-3.36516254 -6.4934647   0.        ]]
    answer:  [[-6.10484594  1.90217048  0.        ]
     [-2.1729519   3.07583054  0.        ]
     [-3.56746362 -3.86210911  1.        ]]
    out:  [[-6.10484594  1.90217048  0.        ]
     [-2.1729519   3.07583054  0.        ]
     [-3.56746362 -3.86210911  1.        ]]
    answer:  [[-4.54583766  4.53405796  1.        ]
     [-1.94975453  4.7680293   1.        ]
     [-4.19862534  7.16475208  1.        ]]
    out:  [[-4.54583766  4.53405796  1.        ]
     [-1.94975453  4.7680293   1.        ]
     [-4.19862534  7.16475208  1.        ]]
    answer:  [[-0.51425963  8.2213693   1.        ]
     [-1.955815   -4.00124272  0.        ]
     [-0.33892306  6.15880467  0.        ]]
    out:  [[-0.51425963  8.2213693   1.        ]
     [-1.955815   -4.00124272  0.        ]
     [-0.33892306  6.15880467  0.        ]]
    answer:  [[-0.69470219  0.58731506  1.        ]
     [-5.9652306   0.67996546  1.        ]
     [-8.07856453  9.02889533  0.        ]]
    out:  [[-0.69470219  0.58731506  1.        ]
     [-5.9652306   0.67996546  1.        ]
     [-8.07856453  9.02889533  0.        ]]
    answer:  [[-3.44882369 -5.31594267  1.        ]
     [-0.60422171  8.37542776  1.        ]]
    out:  [[-3.44882369 -5.31594267  1.        ]
     [-0.60422171  8.37542776  1.        ]]
    answer:  [[-2.80057539 -5.55489229  0.        ]
     [-5.040751   -1.55424598  0.        ]
     [-7.60086867 -2.29169235  0.        ]]
    out:  [[-2.80057539 -5.55489229  0.        ]
     [-5.040751   -1.55424598  0.        ]
     [-7.60086867 -2.29169235  0.        ]]
    answer:  [[-0.60327182 -9.94776048  1.        ]
     [-2.41385399  2.91112426  0.        ]
     [ 0.97285143 -7.40061708  1.        ]]
    out:  [[-0.60327182 -9.94776048  1.        ]
     [-2.41385399  2.91112426  0.        ]
     [ 0.97285143 -7.40061708  1.        ]]
    answer:  [[ 0.48706579 -1.61110782  1.        ]
     [-5.25910094 -9.62965388  0.        ]
     [-9.09885416 -2.8767827   1.        ]
     [ 8.43197642  0.65359807  1.        ]]
    out:  [[ 0.48706579 -1.61110782  1.        ]
     [-5.25910094 -9.62965388  0.        ]
     [-9.09885416 -2.8767827   1.        ]
     [ 8.43197642  0.65359807  1.        ]]
    answer:  [[-1.19738089  4.57571159  1.        ]
     [-6.64194726 -5.56365335  1.        ]
     [-0.94392088 -0.88370692  1.        ]
     [-3.6861174  -0.42453922  1.        ]]
    out:  [[-1.19738089  4.57571159  1.        ]
     [-6.64194726 -5.56365335  1.        ]
     [-0.94392088 -0.88370692  1.        ]
     [-3.6861174  -0.42453922  1.        ]]
    answer:  [[-4.71721731 -1.09871374  1.        ]
     [-6.21443348 -2.84728111  0.        ]]
    out:  [[-4.71721731 -1.09871374  1.        ]
     [-6.21443348 -2.84728111  0.        ]]
    answer:  [[ 2.92732003 -1.88865707  0.        ]
     [-1.60631255 -1.22566077  0.        ]
     [-4.06649003 -7.70954635  0.        ]
     [ 6.69107814  0.54028227  0.        ]]
    out:  [[ 2.92732003 -1.88865707  0.        ]
     [-1.60631255 -1.22566077  0.        ]
     [-4.06649003 -7.70954635  0.        ]
     [ 6.69107814  0.54028227  0.        ]]
    answer:  [[-3.00140948 -6.41315406  0.        ]
     [ 5.13893193 -7.60535297  0.        ]
     [-3.70337221 -1.42053089  1.        ]]
    out:  [[-3.00140948 -6.41315406  0.        ]
     [ 5.13893193 -7.60535297  0.        ]
     [-3.70337221 -1.42053089  1.        ]]
    answer:  [[-7.62775463 -0.13606186  0.        ]
     [-5.93502138 -5.20419842  0.        ]
     [ 9.3924202  -8.03294589  0.        ]]
    out:  [[-7.62775463 -0.13606186  0.        ]
     [-5.93502138 -5.20419842  0.        ]
     [ 9.3924202  -8.03294589  0.        ]]
    answer:  [[-9.61449335 -4.18374342  1.        ]
     [-5.79933452  7.52870301  0.        ]]
    out:  [[-9.61449335 -4.18374342  1.        ]
     [-5.79933452  7.52870301  0.        ]]
    answer:  [[-9.20678831  1.74767156  0.        ]
     [ 0.78580093 -5.35266754  1.        ]
     [ 5.13187746 -6.4307061   0.        ]
     [ 2.81195689 -8.06716307  0.        ]]
    out:  [[-9.20678831  1.74767156  0.        ]
     [ 0.78580093 -5.35266754  1.        ]
     [ 5.13187746 -6.4307061   0.        ]
     [ 2.81195689 -8.06716307  0.        ]]
    answer:  [[-6.37269645 -2.28711518  1.        ]
     [-9.89165526 -0.6289442   0.        ]
     [ 2.49739494 -5.01484464  0.        ]]
    out:  [[-6.37269645 -2.28711518  1.        ]
     [-9.89165526 -0.6289442   0.        ]
     [ 2.49739494 -5.01484464  0.        ]]
    answer:  [[ 1.30252665  1.41473139  0.        ]
     [-9.16038639  3.48248068  0.        ]
     [ 8.72096919 -8.20797439  0.        ]
     [ 0.072007   -7.98098299  0.        ]
     [ 2.89977394 -6.74748526  0.        ]
     [-8.01589773 -0.61247194  1.        ]
     [-0.53524827  2.88008634  0.        ]]
    out:  [[ 1.30252665  1.41473139  0.        ]
     [-9.16038639  3.48248068  0.        ]
     [ 8.72096919 -8.20797439  0.        ]
     [ 0.072007   -7.98098299  0.        ]
     [ 2.89977394 -6.74748526  0.        ]
     [-8.01589773 -0.61247194  1.        ]
     [-0.53524827  2.88008634  0.        ]]
    tree_split_data_left successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!



```python
am.test_student_function(username, tree_split_data_right, ['X', 'Y', 'feature_index', 'split_value'])
```

    Running local tests...
    answer:  [[-4.8970061   9.33106578  1.        ]
     [-3.61684513  6.88138293  0.        ]
     [ 9.59490242 -1.37044005  1.        ]
     [ 3.88313946  9.67431321  1.        ]
     [-7.95783043  5.73215627  1.        ]
     [-5.63968121  0.71451153  0.        ]
     [ 9.13443091  7.77997319  0.        ]]
    out:  [[-4.8970061   9.33106578  1.        ]
     [-3.61684513  6.88138293  0.        ]
     [ 9.59490242 -1.37044005  1.        ]
     [ 3.88313946  9.67431321  1.        ]
     [-7.95783043  5.73215627  1.        ]
     [-5.63968121  0.71451153  0.        ]
     [ 9.13443091  7.77997319  0.        ]]
    answer:  [[ 3.09892195  5.10182147  0.        ]
     [-2.94162536  9.12543561  1.        ]
     [-3.39383923 -0.95693534  0.        ]]
    out:  [[ 3.09892195  5.10182147  0.        ]
     [-2.94162536  9.12543561  1.        ]
     [-3.39383923 -0.95693534  0.        ]]
    answer:  [[ 4.6188895  -8.48170152  0.        ]
     [ 8.78391521 -8.90611468  0.        ]
     [-3.35677736 -8.34618243  0.        ]
     [ 3.49127657  4.13005699  0.        ]]
    out:  [[ 4.6188895  -8.48170152  0.        ]
     [ 8.78391521 -8.90611468  0.        ]
     [-3.35677736 -8.34618243  0.        ]
     [ 3.49127657  4.13005699  0.        ]]
    answer:  [[ 8.03431971  6.79778138  1.        ]
     [ 4.36268631  7.43245613  0.        ]
     [ 5.92355384 -6.51094165  1.        ]
     [ 4.66454697  4.70994357  0.        ]]
    out:  [[ 8.03431971  6.79778138  1.        ]
     [ 4.36268631  7.43245613  0.        ]
     [ 5.92355384 -6.51094165  1.        ]
     [ 4.66454697  4.70994357  0.        ]]
    answer:  [[-7.25516115  3.30039108  1.        ]
     [-5.28971255  0.99700341  0.        ]
     [ 1.32133428  1.50817862  0.        ]
     [ 6.6264207  -2.47096301  1.        ]
     [ 2.02061674 -2.22531409  0.        ]]
    out:  [[-7.25516115  3.30039108  1.        ]
     [-5.28971255  0.99700341  0.        ]
     [ 1.32133428  1.50817862  0.        ]
     [ 6.6264207  -2.47096301  1.        ]
     [ 2.02061674 -2.22531409  0.        ]]
    answer:  [[ 2.8018421   2.33616742  0.        ]
     [ 7.50178436 -0.08870158  0.        ]
     [ 8.18996986  1.67731926  0.        ]
     [ 0.85117047 -2.42004946  0.        ]
     [-0.78748731  2.99783644  1.        ]]
    out:  [[ 2.8018421   2.33616742  0.        ]
     [ 7.50178436 -0.08870158  0.        ]
     [ 8.18996986  1.67731926  0.        ]
     [ 0.85117047 -2.42004946  0.        ]
     [-0.78748731  2.99783644  1.        ]]
    answer:  [[-8.44831712  0.44817077  1.        ]
     [-7.88576029  7.96667802  1.        ]
     [ 6.92768408  0.29981485  1.        ]]
    out:  [[-8.44831712  0.44817077  1.        ]
     [-7.88576029  7.96667802  1.        ]
     [ 6.92768408  0.29981485  1.        ]]
    answer:  [[-7.73457299  7.12364863  0.        ]
     [-9.29883231  1.48205656  0.        ]
     [-4.22459431  3.67803792  1.        ]]
    out:  [[-7.73457299  7.12364863  0.        ]
     [-9.29883231  1.48205656  0.        ]
     [-4.22459431  3.67803792  1.        ]]
    answer:  [[3.42616384 0.27009579 0.        ]
     [3.40345965 1.82518871 0.        ]
     [6.31423425 5.96262439 0.        ]]
    out:  [[3.42616384 0.27009579 0.        ]
     [3.40345965 1.82518871 0.        ]
     [6.31423425 5.96262439 0.        ]]
    answer:  [[ 8.61607244 -0.09410341  1.        ]
     [ 2.477783    9.38012521  0.        ]
     [ 9.96434841 -4.21016905  1.        ]
     [ 2.99645167  5.01084941  1.        ]
     [ 5.54543556 -2.30969165  1.        ]]
    out:  [[ 8.61607244 -0.09410341  1.        ]
     [ 2.477783    9.38012521  0.        ]
     [ 9.96434841 -4.21016905  1.        ]
     [ 2.99645167  5.01084941  1.        ]
     [ 5.54543556 -2.30969165  1.        ]]
    answer:  [[3.1668884  6.47883819 1.        ]
     [6.87461464 8.76976251 0.        ]]
    out:  [[3.1668884  6.47883819 1.        ]
     [6.87461464 8.76976251 0.        ]]
    answer:  [[ 0.57702739 -4.40521119  0.        ]
     [ 8.72496486 -3.18661205  1.        ]
     [-2.08817506 -3.31717391  0.        ]
     [ 9.15246413  4.13720904  0.        ]]
    out:  [[ 0.57702739 -4.40521119  0.        ]
     [ 8.72496486 -3.18661205  1.        ]
     [-2.08817506 -3.31717391  0.        ]
     [ 9.15246413  4.13720904  0.        ]]
    answer:  [[ 5.5544141  -2.19652947  0.        ]
     [-3.97551509  5.3816426   1.        ]
     [-1.07022814  1.62987726  1.        ]
     [ 0.30952255 -2.9890526   1.        ]]
    out:  [[ 5.5544141  -2.19652947  0.        ]
     [-3.97551509  5.3816426   1.        ]
     [-1.07022814  1.62987726  1.        ]
     [ 0.30952255 -2.9890526   1.        ]]
    answer:  [[ 6.11792123  8.26434958  1.        ]
     [ 6.63384996  5.08347633  0.        ]
     [-6.43020124 -0.1601782   0.        ]
     [-3.12891988  8.28874037  1.        ]]
    out:  [[ 6.11792123  8.26434958  1.        ]
     [ 6.63384996  5.08347633  0.        ]
     [-6.43020124 -0.1601782   0.        ]
     [-3.12891988  8.28874037  1.        ]]
    answer:  [[-3.10151367  5.01383924  0.        ]
     [-4.89048403  4.27474653  0.        ]]
    out:  [[-3.10151367  5.01383924  0.        ]
     [-4.89048403  4.27474653  0.        ]]
    answer:  [[ 6.05463489 -4.95085799  0.        ]
     [ 0.2831879   5.98829612  1.        ]]
    out:  [[ 6.05463489 -4.95085799  0.        ]
     [ 0.2831879   5.98829612  1.        ]]
    answer:  [[ 0.45210617  5.08816303  1.        ]
     [ 5.80469984  7.74180993  1.        ]
     [ 5.84517092 -8.59928823  1.        ]]
    out:  [[ 0.45210617  5.08816303  1.        ]
     [ 5.80469984  7.74180993  1.        ]
     [ 5.84517092 -8.59928823  1.        ]]
    answer:  [[-4.48599972 -1.72658853  0.        ]
     [-9.2198824   8.46760581  1.        ]
     [ 8.11512235  8.93889845  0.        ]
     [-6.81449467  9.06887991  0.        ]
     [ 7.44278869 -4.66017351  1.        ]]
    out:  [[-4.48599972 -1.72658853  0.        ]
     [-9.2198824   8.46760581  1.        ]
     [ 8.11512235  8.93889845  0.        ]
     [-6.81449467  9.06887991  0.        ]
     [ 7.44278869 -4.66017351  1.        ]]
    answer:  [[-1.33072667  0.87475516  0.        ]
     [ 2.10805231  1.47500904  1.        ]
     [-2.11786836 -8.77933651  1.        ]]
    out:  [[-1.33072667  0.87475516  0.        ]
     [ 2.10805231  1.47500904  1.        ]
     [-2.11786836 -8.77933651  1.        ]]
    answer:  [[-7.40291802  4.19454706  1.        ]
     [-5.21066275 -0.97159405  1.        ]
     [ 5.20516285  7.5532523   0.        ]
     [ 8.22558216  4.13539542  1.        ]]
    out:  [[-7.40291802  4.19454706  1.        ]
     [-5.21066275 -0.97159405  1.        ]
     [ 5.20516285  7.5532523   0.        ]
     [ 8.22558216  4.13539542  1.        ]]
    answer:  [[ 7.87452913 -2.31156204  0.        ]
     [ 5.20338409  1.29379616  1.        ]
     [ 4.84269544  8.21350779  0.        ]
     [ 5.18954384  6.81511806  1.        ]]
    out:  [[ 7.87452913 -2.31156204  0.        ]
     [ 5.20338409  1.29379616  1.        ]
     [ 4.84269544  8.21350779  0.        ]
     [ 5.18954384  6.81511806  1.        ]]
    answer:  [[-9.95300431  6.05163861  1.        ]
     [-5.44351792  5.434457    0.        ]
     [ 3.15568711  5.84456136  1.        ]]
    out:  [[-9.95300431  6.05163861  1.        ]
     [-5.44351792  5.434457    0.        ]
     [ 3.15568711  5.84456136  1.        ]]
    answer:  [[ 7.43292583  2.16775968  0.        ]
     [ 4.33443695  5.82713608  0.        ]
     [ 1.14027055 -5.63666175  1.        ]]
    out:  [[ 7.43292583  2.16775968  0.        ]
     [ 4.33443695  5.82713608  0.        ]
     [ 1.14027055 -5.63666175  1.        ]]
    answer:  [[ 8.99639499  3.92224209  0.        ]
     [ 9.071767   -5.96793577  0.        ]
     [ 8.49225448  8.76733547  1.        ]]
    out:  [[ 8.99639499  3.92224209  0.        ]
     [ 9.071767   -5.96793577  0.        ]
     [ 8.49225448  8.76733547  1.        ]]
    answer:  [[8.94113554 0.91281464 0.        ]
     [2.60098883 4.55896907 0.        ]]
    out:  [[8.94113554 0.91281464 0.        ]
     [2.60098883 4.55896907 0.        ]]
    answer:  [[-0.83139836 -3.15721798  0.        ]
     [-1.24865198  3.00415918  1.        ]
     [-0.80722454  2.80886349  1.        ]
     [ 8.85682662  1.42612628  1.        ]]
    out:  [[-0.83139836 -3.15721798  0.        ]
     [-1.24865198  3.00415918  1.        ]
     [-0.80722454  2.80886349  1.        ]
     [ 8.85682662  1.42612628  1.        ]]
    answer:  [[ 1.65972698 -9.87453949  1.        ]
     [ 0.9419773   3.70892719  1.        ]]
    out:  [[ 1.65972698 -9.87453949  1.        ]
     [ 0.9419773   3.70892719  1.        ]]
    answer:  [[ 8.46430544 -9.95276257  1.        ]
     [ 3.55810535 -2.0040123   1.        ]
     [-0.76867587 -7.96810575  1.        ]
     [ 3.35939191  2.46313188  1.        ]
     [ 0.19432571 -5.19546763  1.        ]
     [-0.09363581  8.57903752  1.        ]]
    out:  [[ 8.46430544 -9.95276257  1.        ]
     [ 3.55810535 -2.0040123   1.        ]
     [-0.76867587 -7.96810575  1.        ]
     [ 3.35939191  2.46313188  1.        ]
     [ 0.19432571 -5.19546763  1.        ]
     [-0.09363581  8.57903752  1.        ]]
    answer:  [[ 7.27150073  2.26471181  1.        ]
     [ 6.24616991  7.0419269   1.        ]
     [-6.64348192  6.25441143  0.        ]
     [-3.7187825   2.21681097  0.        ]
     [-2.20895999  3.97609758  1.        ]
     [-9.18091764  5.34614893  1.        ]]
    out:  [[ 7.27150073  2.26471181  1.        ]
     [ 6.24616991  7.0419269   1.        ]
     [-6.64348192  6.25441143  0.        ]
     [-3.7187825   2.21681097  0.        ]
     [-2.20895999  3.97609758  1.        ]
     [-9.18091764  5.34614893  1.        ]]
    answer:  [[ 4.00917813  0.85040431  1.        ]
     [ 1.62061315  8.26369856  0.        ]
     [-8.72036373  9.48382117  0.        ]]
    out:  [[ 4.00917813  0.85040431  1.        ]
     [ 1.62061315  8.26369856  0.        ]
     [-8.72036373  9.48382117  0.        ]]
    answer:  [[ 4.52926794  6.6271053   0.        ]
     [-7.09215239  7.44626064  0.        ]
     [-6.18519127  7.27819165  1.        ]
     [-9.82156596  5.94685858  1.        ]]
    out:  [[ 4.52926794  6.6271053   0.        ]
     [-7.09215239  7.44626064  0.        ]
     [-6.18519127  7.27819165  1.        ]
     [-9.82156596  5.94685858  1.        ]]
    answer:  [[-1.9804084   6.26696043  0.        ]
     [ 9.02002719 -2.0770321   1.        ]
     [-7.8328739   6.94390594  1.        ]]
    out:  [[-1.9804084   6.26696043  0.        ]
     [ 9.02002719 -2.0770321   1.        ]
     [-7.8328739   6.94390594  1.        ]]
    answer:  [[1.08589934 1.65107942 1.        ]
     [4.29376554 1.45859637 0.        ]]
    out:  [[1.08589934 1.65107942 1.        ]
     [4.29376554 1.45859637 0.        ]]
    answer:  [[ 0.72961486 -2.25701125  1.        ]
     [ 0.82741003  1.76186999  0.        ]
     [ 7.57203414  8.8322453   1.        ]
     [ 9.40544867  5.3857469   0.        ]
     [ 9.85348853  0.83744403  1.        ]
     [ 5.64283334 -8.60490932  0.        ]]
    out:  [[ 0.72961486 -2.25701125  1.        ]
     [ 0.82741003  1.76186999  0.        ]
     [ 7.57203414  8.8322453   1.        ]
     [ 9.40544867  5.3857469   0.        ]
     [ 9.85348853  0.83744403  1.        ]
     [ 5.64283334 -8.60490932  0.        ]]
    answer:  [[ 8.82495619  8.16596633  0.        ]
     [ 7.18725481  7.59167576  0.        ]
     [-3.49054309  8.63834767  0.        ]
     [ 6.75897325  6.35197031  1.        ]]
    out:  [[ 8.82495619  8.16596633  0.        ]
     [ 7.18725481  7.59167576  0.        ]
     [-3.49054309  8.63834767  0.        ]
     [ 6.75897325  6.35197031  1.        ]]
    answer:  [[ 3.24059835 -2.20053265  0.        ]
     [-5.78573755 -9.64639434  0.        ]
     [ 6.1528784  -5.20695329  1.        ]]
    out:  [[ 3.24059835 -2.20053265  0.        ]
     [-5.78573755 -9.64639434  0.        ]
     [ 6.1528784  -5.20695329  1.        ]]
    answer:  [[ 4.31918871 -2.92108918  1.        ]
     [ 6.15539325  7.70520175  1.        ]
     [ 4.61076636  9.51349334  1.        ]
     [ 5.43860654 -6.41966405  0.        ]]
    out:  [[ 4.31918871 -2.92108918  1.        ]
     [ 6.15539325  7.70520175  1.        ]
     [ 4.61076636  9.51349334  1.        ]
     [ 5.43860654 -6.41966405  0.        ]]
    answer:  [[-6.52843567  9.69976878  0.        ]
     [ 0.58175892  5.01960816  0.        ]
     [ 3.99756065 -4.65049918  1.        ]]
    out:  [[-6.52843567  9.69976878  0.        ]
     [ 0.58175892  5.01960816  0.        ]
     [ 3.99756065 -4.65049918  1.        ]]
    answer:  [[-7.07519213  7.36465907  1.        ]
     [-0.52822863  6.46281137  0.        ]
     [-2.14130796  9.60558823  0.        ]]
    out:  [[-7.07519213  7.36465907  1.        ]
     [-0.52822863  6.46281137  0.        ]
     [-2.14130796  9.60558823  0.        ]]
    answer:  [[-9.2739047   6.5170399   0.        ]
     [-1.74541084  9.89474902  1.        ]
     [ 9.09367842  9.9685425   1.        ]
     [ 7.98731714  3.05583856  1.        ]
     [-3.28839896  7.01561481  0.        ]]
    out:  [[-9.2739047   6.5170399   0.        ]
     [-1.74541084  9.89474902  1.        ]
     [ 9.09367842  9.9685425   1.        ]
     [ 7.98731714  3.05583856  1.        ]
     [-3.28839896  7.01561481  0.        ]]
    answer:  [[ 2.68151222  3.50750687  0.        ]
     [ 5.29309554  8.28093569  0.        ]
     [-6.64095787  4.87525175  0.        ]
     [ 6.07703303  6.73966132  1.        ]]
    out:  [[ 2.68151222  3.50750687  0.        ]
     [ 5.29309554  8.28093569  0.        ]
     [-6.64095787  4.87525175  0.        ]
     [ 6.07703303  6.73966132  1.        ]]
    answer:  [[-1.32112968  0.15607983  1.        ]
     [-2.49803942 -1.49424765  0.        ]
     [ 9.84987675  7.9029997   0.        ]]
    out:  [[-1.32112968  0.15607983  1.        ]
     [-2.49803942 -1.49424765  0.        ]
     [ 9.84987675  7.9029997   0.        ]]
    answer:  [[ 4.0760124  -6.53218073  1.        ]
     [ 3.04251766  4.78518863  1.        ]
     [ 5.24613728 -7.28676438  1.        ]]
    out:  [[ 4.0760124  -6.53218073  1.        ]
     [ 3.04251766  4.78518863  1.        ]
     [ 5.24613728 -7.28676438  1.        ]]
    answer:  [[ 2.77146565  6.03688775  1.        ]
     [-4.28520391  8.21242192  1.        ]
     [ 2.52199459  5.02327993  0.        ]
     [-8.76061838  5.56922902  1.        ]]
    out:  [[ 2.77146565  6.03688775  1.        ]
     [-4.28520391  8.21242192  1.        ]
     [ 2.52199459  5.02327993  0.        ]
     [-8.76061838  5.56922902  1.        ]]
    answer:  [[5.6277257  8.58141942 1.        ]
     [0.09347629 7.69284462 1.        ]
     [3.34639007 5.48825465 0.        ]
     [5.89343852 9.01863194 1.        ]]
    out:  [[5.6277257  8.58141942 1.        ]
     [0.09347629 7.69284462 1.        ]
     [3.34639007 5.48825465 0.        ]
     [5.89343852 9.01863194 1.        ]]
    answer:  [[-0.1913795   0.84051076  1.        ]
     [-0.85790613 -5.40170279  1.        ]
     [ 1.27112258 -0.23085763  0.        ]
     [ 2.80148407 -2.30621144  1.        ]]
    out:  [[-0.1913795   0.84051076  1.        ]
     [-0.85790613 -5.40170279  1.        ]
     [ 1.27112258 -0.23085763  0.        ]
     [ 2.80148407 -2.30621144  1.        ]]
    answer:  [[ 2.16364467 -1.42638375  0.        ]
     [ 6.14008412 -4.21439533  0.        ]
     [ 7.1748265  -2.99738365  1.        ]
     [ 2.99752931 -9.13553193  1.        ]
     [ 4.41829509  1.36430977  0.        ]]
    out:  [[ 2.16364467 -1.42638375  0.        ]
     [ 6.14008412 -4.21439533  0.        ]
     [ 7.1748265  -2.99738365  1.        ]
     [ 2.99752931 -9.13553193  1.        ]
     [ 4.41829509  1.36430977  0.        ]]
    answer:  [[ 9.61535448 -7.50109304  1.        ]
     [-3.50248724 -1.24218637  0.        ]
     [ 9.27016199 -2.44144244  0.        ]]
    out:  [[ 9.61535448 -7.50109304  1.        ]
     [-3.50248724 -1.24218637  0.        ]
     [ 9.27016199 -2.44144244  0.        ]]
    answer:  [[-9.4058481   8.33603325  1.        ]
     [-9.59932209  9.78707692  1.        ]
     [-9.26240068  8.61981826  1.        ]]
    out:  [[-9.4058481   8.33603325  1.        ]
     [-9.59932209  9.78707692  1.        ]
     [-9.26240068  8.61981826  1.        ]]
    answer:  [[-3.02018071 -1.79062609  1.        ]
     [ 8.62654179 -1.19764012  1.        ]
     [ 2.51681643  4.42906879  1.        ]
     [-4.62683627 -1.46593256  1.        ]]
    out:  [[-3.02018071 -1.79062609  1.        ]
     [ 8.62654179 -1.19764012  1.        ]
     [ 2.51681643  4.42906879  1.        ]
     [-4.62683627 -1.46593256  1.        ]]
    answer:  [[ 6.19769041 -1.21847536  0.        ]
     [ 6.02783754  7.82682217  1.        ]]
    out:  [[ 6.19769041 -1.21847536  0.        ]
     [ 6.02783754  7.82682217  1.        ]]
    answer:  [[ 7.53580993  7.86389334  1.        ]
     [-0.71737652  8.66013444  1.        ]]
    out:  [[ 7.53580993  7.86389334  1.        ]
     [-0.71737652  8.66013444  1.        ]]
    answer:  [[-6.5572113   5.53892144  1.        ]
     [ 9.83550526  8.81654318  1.        ]
     [ 2.36906219 -3.94275406  1.        ]]
    out:  [[-6.5572113   5.53892144  1.        ]
     [ 9.83550526  8.81654318  1.        ]
     [ 2.36906219 -3.94275406  1.        ]]
    answer:  [[-8.03713961  8.62561787  1.        ]
     [ 5.82767903 -8.12193619  1.        ]
     [ 5.0718485   1.04406712  0.        ]
     [ 1.76003235 -2.14515344  1.        ]]
    out:  [[-8.03713961  8.62561787  1.        ]
     [ 5.82767903 -8.12193619  1.        ]
     [ 5.0718485   1.04406712  0.        ]
     [ 1.76003235 -2.14515344  1.        ]]
    answer:  [[4.19870847 0.36190365 0.        ]
     [7.59878205 4.64763699 0.        ]]
    out:  [[4.19870847 0.36190365 0.        ]
     [7.59878205 4.64763699 0.        ]]
    answer:  [[-5.2448387   0.48336706  1.        ]
     [-2.67255252  6.35359115  1.        ]]
    out:  [[-5.2448387   0.48336706  1.        ]
     [-2.67255252  6.35359115  1.        ]]
    answer:  [[ 3.01261835  5.3603836   0.        ]
     [-5.26931102  9.87895762  1.        ]]
    out:  [[ 3.01261835  5.3603836   0.        ]
     [-5.26931102  9.87895762  1.        ]]
    answer:  [[ 8.50639433  3.63862192  1.        ]
     [ 5.25033968 -4.72781518  0.        ]]
    out:  [[ 8.50639433  3.63862192  1.        ]
     [ 5.25033968 -4.72781518  0.        ]]
    answer:  [[0.76120646 4.85108874 0.        ]
     [8.1072223  3.96100099 1.        ]]
    out:  [[0.76120646 4.85108874 0.        ]
     [8.1072223  3.96100099 1.        ]]
    answer:  [[-0.47881107 -9.64694335  1.        ]
     [ 9.35836132 -1.6333601   1.        ]
     [-2.46507075  8.47684056  0.        ]
     [ 0.85982077  9.59622903  1.        ]
     [ 6.5574399  -3.1997597   1.        ]
     [ 5.78525114 -1.92577749  0.        ]
     [ 7.60136766  8.32330518  1.        ]]
    out:  [[-0.47881107 -9.64694335  1.        ]
     [ 9.35836132 -1.6333601   1.        ]
     [-2.46507075  8.47684056  0.        ]
     [ 0.85982077  9.59622903  1.        ]
     [ 6.5574399  -3.1997597   1.        ]
     [ 5.78525114 -1.92577749  0.        ]
     [ 7.60136766  8.32330518  1.        ]]
    answer:  [[-1.93243759  5.12100741  1.        ]
     [ 0.92690453  9.11757918  1.        ]]
    out:  [[-1.93243759  5.12100741  1.        ]
     [ 0.92690453  9.11757918  1.        ]]
    answer:  [[ 6.07109447  5.31226017  0.        ]
     [-5.3907863   4.77300928  0.        ]]
    out:  [[ 6.07109447  5.31226017  0.        ]
     [-5.3907863   4.77300928  0.        ]]
    answer:  [[6.22851852 0.12247935 0.        ]
     [5.58537856 7.04425164 0.        ]]
    out:  [[6.22851852 0.12247935 0.        ]
     [5.58537856 7.04425164 0.        ]]
    answer:  [[ 9.55467976  1.66554673  1.        ]
     [ 7.17234236 -7.50206859  1.        ]
     [ 7.82833253  4.73203502  1.        ]
     [ 2.53192114 -0.21462453  0.        ]]
    out:  [[ 9.55467976  1.66554673  1.        ]
     [ 7.17234236 -7.50206859  1.        ]
     [ 7.82833253  4.73203502  1.        ]
     [ 2.53192114 -0.21462453  0.        ]]
    answer:  [[6.71220596 6.82151978 0.        ]
     [3.8624762  6.27172571 1.        ]]
    out:  [[6.71220596 6.82151978 0.        ]
     [3.8624762  6.27172571 1.        ]]
    answer:  [[ 4.6260931  -5.46567208  1.        ]
     [ 9.95565609 -5.53152041  0.        ]
     [ 1.90628258  5.1069257   0.        ]]
    out:  [[ 4.6260931  -5.46567208  1.        ]
     [ 9.95565609 -5.53152041  0.        ]
     [ 1.90628258  5.1069257   0.        ]]
    answer:  [[ 7.95318083 -4.07289365  1.        ]
     [ 0.81822254  3.51618085  0.        ]
     [ 1.18169654  1.64735114  1.        ]]
    out:  [[ 7.95318083 -4.07289365  1.        ]
     [ 0.81822254  3.51618085  0.        ]
     [ 1.18169654  1.64735114  1.        ]]
    answer:  [[ 7.511104   -4.99869851  1.        ]
     [ 8.71495273 -8.57688612  0.        ]
     [-1.86066809  5.06404677  0.        ]
     [ 9.44821106 -9.24783162  1.        ]]
    out:  [[ 7.511104   -4.99869851  1.        ]
     [ 8.71495273 -8.57688612  0.        ]
     [-1.86066809  5.06404677  0.        ]
     [ 9.44821106 -9.24783162  1.        ]]
    answer:  [[ 4.12081611  8.10792227  0.        ]
     [ 3.73333377  8.17862077  1.        ]
     [ 8.84877932  6.12570834  0.        ]
     [ 6.83558207  8.60382884  0.        ]
     [ 5.11430154 -3.79421992  1.        ]
     [ 1.70428815  0.51344922  0.        ]]
    out:  [[ 4.12081611  8.10792227  0.        ]
     [ 3.73333377  8.17862077  1.        ]
     [ 8.84877932  6.12570834  0.        ]
     [ 6.83558207  8.60382884  0.        ]
     [ 5.11430154 -3.79421992  1.        ]
     [ 1.70428815  0.51344922  0.        ]]
    answer:  [[ 2.95128603 -0.57833641  0.        ]
     [ 5.05193911  5.61106024  1.        ]
     [ 8.13855103  2.37862274  0.        ]
     [ 9.98554281 -1.34510102  1.        ]
     [ 6.37332144  7.43563615  0.        ]
     [ 0.48392135  9.750935    1.        ]]
    out:  [[ 2.95128603 -0.57833641  0.        ]
     [ 5.05193911  5.61106024  1.        ]
     [ 8.13855103  2.37862274  0.        ]
     [ 9.98554281 -1.34510102  1.        ]
     [ 6.37332144  7.43563615  0.        ]
     [ 0.48392135  9.750935    1.        ]]
    answer:  [[ 4.85491593  2.24083082  0.        ]
     [-8.96248482  8.80327693  1.        ]
     [-5.69541832  1.94293554  1.        ]
     [-3.69283069  7.65277356  1.        ]]
    out:  [[ 4.85491593  2.24083082  0.        ]
     [-8.96248482  8.80327693  1.        ]
     [-5.69541832  1.94293554  1.        ]
     [-3.69283069  7.65277356  1.        ]]
    answer:  [[-5.08709497  4.22891059  1.        ]
     [ 7.42498584  6.48088998  1.        ]
     [ 2.43883905  3.08925091  0.        ]
     [-3.08362171  3.05091932  0.        ]]
    out:  [[-5.08709497  4.22891059  1.        ]
     [ 7.42498584  6.48088998  1.        ]
     [ 2.43883905  3.08925091  0.        ]
     [-3.08362171  3.05091932  0.        ]]
    answer:  [[ 5.31966537 -8.91400948  1.        ]
     [ 7.1191027   4.59002907  0.        ]
     [ 9.68644203 -5.7056459   0.        ]
     [ 9.09276775 -0.35036015  0.        ]]
    out:  [[ 5.31966537 -8.91400948  1.        ]
     [ 7.1191027   4.59002907  0.        ]
     [ 9.68644203 -5.7056459   0.        ]
     [ 9.09276775 -0.35036015  0.        ]]
    answer:  [[ 6.46882477 -9.05345396  0.        ]
     [ 9.8422324  -7.04411071  0.        ]]
    out:  [[ 6.46882477 -9.05345396  0.        ]
     [ 9.8422324  -7.04411071  0.        ]]
    answer:  [[-8.81541352  7.11398451  1.        ]
     [-0.50893337  6.44519477  1.        ]]
    out:  [[-8.81541352  7.11398451  1.        ]
     [-0.50893337  6.44519477  1.        ]]
    answer:  [[8.28967663 6.49418901 1.        ]
     [8.95529879 8.16053273 0.        ]
     [2.63099191 3.25792633 0.        ]]
    out:  [[8.28967663 6.49418901 1.        ]
     [8.95529879 8.16053273 0.        ]
     [2.63099191 3.25792633 0.        ]]
    answer:  [[ 1.9354346   5.66516996  0.        ]
     [ 6.4207053  -9.76566344  1.        ]
     [ 7.65180507  5.21052577  0.        ]]
    out:  [[ 1.9354346   5.66516996  0.        ]
     [ 6.4207053  -9.76566344  1.        ]
     [ 7.65180507  5.21052577  0.        ]]
    answer:  [[ 6.38227486  6.55349475  0.        ]
     [ 8.53945899  8.5620412   1.        ]
     [ 2.11811145 -8.52327226  0.        ]]
    out:  [[ 6.38227486  6.55349475  0.        ]
     [ 8.53945899  8.5620412   1.        ]
     [ 2.11811145 -8.52327226  0.        ]]
    answer:  [[-5.47758213  8.45179762  1.        ]
     [ 4.07988245  8.97239766  0.        ]]
    out:  [[-5.47758213  8.45179762  1.        ]
     [ 4.07988245  8.97239766  0.        ]]
    answer:  [[3.43439631 2.1341125  0.        ]
     [7.64180706 7.54054951 1.        ]
     [9.15949548 9.83534702 1.        ]]
    out:  [[3.43439631 2.1341125  0.        ]
     [7.64180706 7.54054951 1.        ]
     [9.15949548 9.83534702 1.        ]]
    answer:  [[ 3.81578544  7.05848761  1.        ]
     [ 4.93408634  5.30921609  0.        ]
     [ 3.47072233  5.99959475  0.        ]
     [ 1.46547794 -1.54608469  1.        ]]
    out:  [[ 3.81578544  7.05848761  1.        ]
     [ 4.93408634  5.30921609  0.        ]
     [ 3.47072233  5.99959475  0.        ]
     [ 1.46547794 -1.54608469  1.        ]]
    answer:  [[-4.57453177 -3.85019755  0.        ]
     [ 7.74311824  3.58685887  1.        ]
     [-0.67864307  9.35019045  0.        ]
     [-0.18409434  1.35242302  1.        ]]
    out:  [[-4.57453177 -3.85019755  0.        ]
     [ 7.74311824  3.58685887  1.        ]
     [-0.67864307  9.35019045  0.        ]
     [-0.18409434  1.35242302  1.        ]]
    answer:  [[ 6.98537358 -6.75865069  1.        ]
     [ 2.48007593  3.63993104  1.        ]
     [ 7.11797881 -4.89500585  1.        ]
     [ 3.2218666  -4.10145605  1.        ]
     [ 2.443033    3.86721404  1.        ]
     [ 2.70534629 -7.02361119  1.        ]]
    out:  [[ 6.98537358 -6.75865069  1.        ]
     [ 2.48007593  3.63993104  1.        ]
     [ 7.11797881 -4.89500585  1.        ]
     [ 3.2218666  -4.10145605  1.        ]
     [ 2.443033    3.86721404  1.        ]
     [ 2.70534629 -7.02361119  1.        ]]
    answer:  [[ 9.14958862 -8.86989763  1.        ]
     [-3.64968634 -7.73663149  0.        ]]
    out:  [[ 9.14958862 -8.86989763  1.        ]
     [-3.64968634 -7.73663149  0.        ]]
    answer:  [[-5.3698352   7.9223482   0.        ]
     [-1.568122    4.94929849  0.        ]
     [-2.51745177  0.96006421  0.        ]
     [-2.56855908  6.47337974  1.        ]]
    out:  [[-5.3698352   7.9223482   0.        ]
     [-1.568122    4.94929849  0.        ]
     [-2.51745177  0.96006421  0.        ]
     [-2.56855908  6.47337974  1.        ]]
    answer:  [[9.02429411 6.23418999 0.        ]
     [7.43460958 4.86940925 0.        ]
     [7.49400234 4.88878756 1.        ]]
    out:  [[9.02429411 6.23418999 0.        ]
     [7.43460958 4.86940925 0.        ]
     [7.49400234 4.88878756 1.        ]]
    answer:  [[ 9.09848973  9.22791115  1.        ]
     [ 3.25234027  4.98057189  1.        ]
     [ 6.80421415  3.82950724  1.        ]
     [-6.24707416  4.98739036  0.        ]
     [-1.19818098  7.87405967  1.        ]]
    out:  [[ 9.09848973  9.22791115  1.        ]
     [ 3.25234027  4.98057189  1.        ]
     [ 6.80421415  3.82950724  1.        ]
     [-6.24707416  4.98739036  0.        ]
     [-1.19818098  7.87405967  1.        ]]
    answer:  [[-0.12700563  4.33951848  0.        ]
     [-0.68155137  3.09417866  0.        ]
     [ 6.36142969  8.03907017  0.        ]]
    out:  [[-0.12700563  4.33951848  0.        ]
     [-0.68155137  3.09417866  0.        ]
     [ 6.36142969  8.03907017  0.        ]]
    answer:  [[-0.14524453 -1.34207584  1.        ]
     [ 8.12278014  2.59925058  0.        ]
     [ 1.93724536  5.71356373  0.        ]
     [-7.03776161  1.82429742  0.        ]]
    out:  [[-0.14524453 -1.34207584  1.        ]
     [ 8.12278014  2.59925058  0.        ]
     [ 1.93724536  5.71356373  0.        ]
     [-7.03776161  1.82429742  0.        ]]
    answer:  [[ 2.95011817  0.10096367  1.        ]
     [-3.46444981 -0.50045668  0.        ]
     [ 8.82944168  0.74286084  1.        ]
     [ 2.86876012  9.01383258  1.        ]
     [-0.45930309 -0.9470933   0.        ]
     [-1.51768581  7.48842809  1.        ]]
    out:  [[ 2.95011817  0.10096367  1.        ]
     [-3.46444981 -0.50045668  0.        ]
     [ 8.82944168  0.74286084  1.        ]
     [ 2.86876012  9.01383258  1.        ]
     [-0.45930309 -0.9470933   0.        ]
     [-1.51768581  7.48842809  1.        ]]
    answer:  [[-9.11561235  7.10671513  0.        ]
     [-6.85743465  7.3462275   1.        ]]
    out:  [[-9.11561235  7.10671513  0.        ]
     [-6.85743465  7.3462275   1.        ]]
    answer:  [[ 3.98876624  1.50359805  1.        ]
     [ 1.29809498  1.14985556  0.        ]
     [-2.2068076   3.45934253  0.        ]
     [ 9.176637   -1.05907226  0.        ]
     [-3.07734597 -3.50828783  1.        ]
     [ 2.62306499 -6.96221409  1.        ]]
    out:  [[ 3.98876624  1.50359805  1.        ]
     [ 1.29809498  1.14985556  0.        ]
     [-2.2068076   3.45934253  0.        ]
     [ 9.176637   -1.05907226  0.        ]
     [-3.07734597 -3.50828783  1.        ]
     [ 2.62306499 -6.96221409  1.        ]]
    answer:  [[ 4.60189348 -0.72784419  0.        ]
     [ 7.37331616  4.12037993  0.        ]]
    out:  [[ 4.60189348 -0.72784419  0.        ]
     [ 7.37331616  4.12037993  0.        ]]
    answer:  [[ 1.88419739 -5.85156735  1.        ]
     [ 2.82079804 -5.42749181  0.        ]
     [ 4.77113375 -8.11165463  0.        ]]
    out:  [[ 1.88419739 -5.85156735  1.        ]
     [ 2.82079804 -5.42749181  0.        ]
     [ 4.77113375 -8.11165463  0.        ]]
    answer:  [[ 1.6418351   4.75616185  0.        ]
     [ 0.00968571 -9.02364819  1.        ]
     [ 7.91785716  2.75245739  0.        ]
     [-0.94704741 -1.20993417  0.        ]
     [ 6.07660682  1.30045771  1.        ]
     [ 6.58054233 -1.0103582   1.        ]]
    out:  [[ 1.6418351   4.75616185  0.        ]
     [ 0.00968571 -9.02364819  1.        ]
     [ 7.91785716  2.75245739  0.        ]
     [-0.94704741 -1.20993417  0.        ]
     [ 6.07660682  1.30045771  1.        ]
     [ 6.58054233 -1.0103582   1.        ]]
    answer:  [[ 0.91985376  6.84686836  1.        ]
     [ 8.58858098 -5.86482921  0.        ]
     [ 1.98436373 -4.51726439  0.        ]
     [ 2.28281079 -2.4701563   1.        ]]
    out:  [[ 0.91985376  6.84686836  1.        ]
     [ 8.58858098 -5.86482921  0.        ]
     [ 1.98436373 -4.51726439  0.        ]
     [ 2.28281079 -2.4701563   1.        ]]
    answer:  [[-6.3794093   7.42828745  1.        ]
     [-5.72319284  5.26483206  0.        ]
     [ 7.28391111  8.6801265   1.        ]
     [ 1.06423051  6.74915128  0.        ]]
    out:  [[-6.3794093   7.42828745  1.        ]
     [-5.72319284  5.26483206  0.        ]
     [ 7.28391111  8.6801265   1.        ]
     [ 1.06423051  6.74915128  0.        ]]
    answer:  [[ 4.0960118   0.42254812  1.        ]
     [ 8.3478665  -0.09270325  0.        ]
     [ 7.67603709  2.83243765  1.        ]]
    out:  [[ 4.0960118   0.42254812  1.        ]
     [ 8.3478665  -0.09270325  0.        ]
     [ 7.67603709  2.83243765  1.        ]]
    answer:  [[6.82433150e+00 7.93415625e+00 0.00000000e+00]
     [1.45432668e+00 4.39418860e-03 0.00000000e+00]
     [5.60347454e+00 2.96680019e+00 0.00000000e+00]
     [5.93675892e+00 1.08428958e+00 1.00000000e+00]]
    out:  [[6.82433150e+00 7.93415625e+00 0.00000000e+00]
     [1.45432668e+00 4.39418860e-03 0.00000000e+00]
     [5.60347454e+00 2.96680019e+00 0.00000000e+00]
     [5.93675892e+00 1.08428958e+00 1.00000000e+00]]
    answer:  [[-4.87371868  1.86860531  1.        ]
     [-7.76918477  2.3114619   1.        ]
     [ 0.18252979  7.46941585  1.        ]
     [-5.46341003  9.35685055  1.        ]]
    out:  [[-4.87371868  1.86860531  1.        ]
     [-7.76918477  2.3114619   1.        ]
     [ 0.18252979  7.46941585  1.        ]
     [-5.46341003  9.35685055  1.        ]]
    tree_split_data_right successfully passed local tests
    Running remote test...
    Test was successful. Congratulations!



```python
am.get_progress(username)
```

    ---------------------------------------------
    | Jakub Miksa                               |
    | jakub.miksa@student.uva.nl                |
    ---------------------------------------------
    | linear_forward           | completed      |
    | linear_grad_W            | completed      |
    | linear_grad_b            | completed      |
    | nll_forward              | completed      |
    | nll_grad_input           | completed      |
    | sigmoid_forward          | completed      |
    | sigmoid_grad_input       | completed      |
    | tree_gini_index          | completed      |
    | tree_split_data_left     | completed      |
    | tree_split_data_right    | completed      |
    | tree_to_terminal         | not attempted  |
    ---------------------------------------------


Now to find the split rule with the lowest gini score, we brute-force search over all features and values to split by.


```python
def tree_best_split(X, Y):
    class_values = list(set(Y.flatten().tolist()))
    r_index, r_value, r_score =  float("inf"),  float("inf"), float("inf")
    r_XY_left, r_XY_right = (X,Y), (X,Y)
    for feature_index in range(X.shape[1]):
        for row in X:
            XY_left = tree_split_data_left(X, Y, feature_index, row[feature_index])
            XY_right = tree_split_data_right(X, Y, feature_index, row[feature_index])
            XY_left, XY_right = (XY_left[:,:-1], XY_left[:,-1:]), (XY_right[:,:-1], XY_right[:,-1:])
            gini = tree_gini_index(XY_left[1], XY_right[1], class_values)
            if gini < r_score:
                r_index, r_value, r_score = feature_index, row[feature_index], gini
                r_XY_left, r_XY_right = XY_left, XY_right
    return {'index':r_index, 'value':r_value, 'XY_left': r_XY_left, 'XY_right':r_XY_right}
```

## 2.2 Terminal Node
The leaf nodes predict the label of an unseen example, by taking a majority vote over all training class labels in that node.


```python
def tree_to_terminal(Y):
    """The most frequent class label, out of the data points belonging to the leaf node,
    is selected as the predicted class.
    
    # Arguments
        Y: np.array of size `(n_objects)`
        
    # Output
        label: most frequent label of `Y.dtype`
    """
    label = None
    print(Y.shape)
    counts = np.bincount(Y[:])
    label = np.argmax(counts)
    #################
    ### YOUR CODE ###
    #################
    return label
```


```python
am.test_student_function(username, tree_to_terminal, ['Y'])
```


```python
am.get_progress(username)
```

## 2.3 Build the Decision Tree
Now we recursively build the decision tree, by greedily splitting the data at each node according to the gini index.
To prevent the model from overfitting, we transform a node into a terminal/leaf node, if:
1. a maximum depth is reached.
2. the node does not reach a minimum number of training samples.



```python
def tree_recursive_split(X, Y, node, max_depth, min_size, depth):
    XY_left, XY_right = node['XY_left'], node['XY_right']
    del(node['XY_left'])
    del(node['XY_right'])
    # check for a no split
    if XY_left[0].size <= 0 or XY_right[0].size <= 0:
        node['left_child'] = node['right_child'] = tree_to_terminal(np.concatenate((XY_left[1], XY_right[1])))
        return
    # check for max depth
    if depth >= max_depth:
        node['left_child'], node['right_child'] = tree_to_terminal(XY_left[1]), tree_to_terminal(XY_right[1])
        return
    # process left child
    if XY_left[0].shape[0] <= min_size:
        node['left_child'] = tree_to_terminal(XY_left[1])
    else:
        node['left_child'] = tree_best_split(*XY_left)
        tree_recursive_split(X, Y, node['left_child'], max_depth, min_size, depth+1)
    # process right child
    if XY_right[0].shape[0] <= min_size:
        node['right_child'] = tree_to_terminal(XY_right[1])
    else:
        node['right_child'] = tree_best_split(*XY_right)
        tree_recursive_split(X, Y, node['right_child'], max_depth, min_size, depth+1)


def build_tree(X, Y, max_depth, min_size):
    root = tree_best_split(X, Y)
    tree_recursive_split(X, Y, root, max_depth, min_size, 1)
    return root
```

By printing the split criteria or the predicted class at each node, we can visualise the decising making process.
Both the tree and a a prediction can be implemented recursively, by going from the root to a leaf node.


```python
def print_tree(node, depth=0):
    if isinstance(node, dict):
        print('%s[X%d < %.3f]' % ((depth*' ', (node['index']+1), node['value'])))
        print_tree(node['left_child'], depth+1)
        print_tree(node['right_child'], depth+1)
    else:
        print('%s[%s]' % ((depth*' ', node)))
        
def tree_predict_single(x, node):
    if isinstance(node, dict):
        if x[node['index']] < node['value']:
            return tree_predict_single(x, node['left_child'])
        else:
            return tree_predict_single(x, node['right_child'])
        
    return node

def tree_predict_multi(X, node):
    Y = np.array([tree_predict_single(row, node) for row in X])
    return Y[:, None]  # size: (n_object,) -> (n_object, 1)
```

Let's test our decision tree model on some toy data.


```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)

tree = build_tree(X_train, Y_train, 4, 1)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

We print the decision tree in [pre-order](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)).


```python
print_tree(tree)
```


```python
plot_model_prediction(lambda x: tree_predict_multi(x, tree), X_test, Y_test)
```

# 3. Experiments
The [Cleveland Heart Disease](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) dataset aims at predicting the presence of heart disease based on other available medical information of the patient.

Although the whole database contains 76 attributes, we focus on the following 14:
1. Age: age in years 
2. Sex: 
    * 0 = female
    * 1 = male 
3. Chest pain type: 
    * 1 = typical angina
    * 2 = atypical angina
    * 3 = non-anginal pain
    * 4 = asymptomatic
4. Trestbps: resting blood pressure in mm Hg on admission to the hospital 
5. Chol: serum cholestoral in mg/dl 
6. Fasting blood sugar: > 120 mg/dl
    * 0 = false
    * 1 = true
7. Resting electrocardiographic results: 
    * 0 = normal
    * 1 = having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV) 
    * 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria 
8. Thalach: maximum heart rate achieved 
9. Exercise induced angina:
    * 0 = no
    * 1 = yes
10. Oldpeak: ST depression induced by exercise relative to rest 
11. Slope: the slope of the peak exercise ST segment
    * 1 = upsloping
    * 2 = flat 
    * 3 = downsloping 
12. Ca: number of major vessels (0-3) colored by flourosopy 
13. Thal: 
    * 3 = normal
    * 6 = fixed defect
    * 7 = reversable defect 
14. Target: diagnosis of heart disease (angiographic disease status)
    * 0 = < 50% diameter narrowing 
    * 1 = > 50% diameter narrowing
    
The 14. attribute is the target variable that we would like to predict based on the rest.

We have prepared some helper functions to download and pre-process the data in `heart_disease_data.py`


```python
import heart_disease_data
```


```python
X, Y = heart_disease_data.download_and_preprocess()
X_train, Y_train, X_test, Y_test = split(X, Y, 0.7)
```

Let's have a look at some examples


```python
print(X_train[0:2])
print(Y_train[0:2])

# TODO feel free to explore more examples and see if you can predict the presence of a heart disease
```

## 3.1 Decision Tree for Heart Disease Prediction 
Let's build a decision tree model on the training data and see how well it performs


```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance

tree = build_tree(X_train, Y_train, 5, 4)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

How did changing the hyper parameters affect the test performance? Usually hyper parameters are tuned using a hold-out [validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#Validation_dataset) instead of the test set.

## 3.2 Logistic Regression for Heart Disease Prediction

Instead of manually going through the data to find possible correlations, let's try training a logistic regression model on the data.


```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance
```

How well did your model perform? Was it actually better then guessing? Let's look at the empirical mean of the target.


```python
Y_train.mean()
```

So what is the problem? Let's have a look at the learned parameters of our model.


```python
print(model.W, model.b)
```

If you trained sufficiently many steps you'll probably see how some weights are much larger than others. Have a look at what range the parameters were initialized and how much change we allow per step (learning rate). Compare this to the scale of the input features. Here an important concept arises, when we want to train on real world data: 
[Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling).

Let's try applying it on our data and see how it affects our performance.


```python
# TODO: Rescale the input features and train again
```

Notice that we did not need any rescaling for the decision tree. Can you think of why?

9de8f3a0c5a0ccfc51d8262772b14ade795465b7
ipynb
Jupyter Notebook
week_2/ML.ipynb
kmiksa/UVA_AML20
4c6b6b99984f570bc148a73d83828e6f184df9fc
MIT
week_2/ML.ipynb
kmiksa/UVA_AML20
4c6b6b99984f570bc148a73d83828e6f184df9fc
MIT
week_2/ML.ipynb
kmiksa/UVA_AML20
4c6b6b99984f570bc148a73d83828e6f184df9fc
MIT
Qwen/Qwen-72B
1. YES
2. YES