In this video, I will be using neural networks
to predict whether a tumor in a woman’s
breasts is malignant or benign. By the way,
a benign tumor is a tumor that cannot invade
neighboring tissue, which in most cases renders
it harmless. A malignant tumor is the exact
opposite; it can spread to the rest of the
body and is extremely dangerous. The importance
of differentiating between the two is obvious.
I’ll actually be making multiple models
-- some with logistic regression, some with
softmax regression, some simple, some complex,
and more. I will be implementing all of these
models in keras.
This is a prime example of a classification
problem. As the name implies, classification
simply involves identifying to which of a
set of categories or classes a new observation
belongs. In this instance, I will be identifying
if a tumor belongs to one of two classes:
malignant or benign. Before we start building
the model, let’s take a brief look at the
dataset.
There are a total of 32 attributes, of which
30 are input features, 1 is the classification,
and 1 is the id number. The inputs are just
a bunch of features and measurements of the
tumor. Some of these parameters include measurements
of the nuclei of cells in the tumor. Examples
of these measurements include radius, texture,
perimeter, and area. For more information
on the input features, and even the dataset
as a whole, you can visit the UCI machine
learning repository, which is where I got
the dataset from. There will be a link to
the website in the video description. The
output parameter, which is the classification,
comes as m for malignant or b for benign.
In the dataset that I use in this video, I
exchange the b for 0 and m for 1. As usual,
it will also be divided into inputs and outputs
and you will be able to access it on my GitHub,
for which there will be a link to in the video
description. And finally the id number. As
I’m sure you can imagine, it has absolutely
no correlation to the diagnosis, so it is
in our best interest to simply remove it.
If we did include it, then the model would
very likely establish some completely arbitrary
relationship between the id number and diagnosis,
that doesn’t generalize. The datasets in
my GitHub already have this feature removed.
Next, as I did in the last video, I upload
the input and output data by making a new
folder, clicking upload, selecting my files,
clicking open, and finally clicking the blue
upload button. Next, to make a new notebook
I click new then python 3.
From here I begin writing the code. First,
I import all the packages that I’ll be using
later in the script. Next, I assign the input
and output data to df1 and df2, respectively.
After that, I scale the input data. I do this
because some features are orders of magnitude
larger than others, and it is unlikely that
these features are significantly more correlated
than others to the output. In general, scaling
the data is a good practice. I don’t scale
the output though, because this is a classification
problem, and it wouldn’t make sense to scale
categorical data, even if different classes
are represented by some arbitrary numbers,
like, in our case, 0 and 1. In the same way
as in the last video, I do this with the .scale
function from sklearn.preprocessing.
I then split the inputs and outputs into training
and testing data, with training data being
80% of the total data and testing data being
20% of the total data. Note, that the data
is randomly sampled.
From this point, I’ll begin developing a
relatively simple or “shallow”, logistic
regression model. Logistic simply refers to
a sigmoid activation function. You may be
wondering why it is still “regression”
when it is being used in classification. Sigmoid
still assigns any possible input to an output,
which, by the way, is between zero and one.
In order for sigmoid to be used for classification,
we must set a threshold. For example, on this
sigmoid graph, we could set a threshold here,
or, in other words, any y value, or output,
that is less than 0.5 will be 0, and anything
more will be 1. Note, that with logistic regression,
or sigmoid, you can only be predicting between
2 classes. For problems with more classes
than this you will have to use softmax regression,
which I will also be implementing in this
video. With our dataset having two classes,
either approach works equally well, as we
will see. Anyhow, for the logistic regression
model I first define the model in the line
model=sequential(). Then I add an input layer
with 30 neurons. This corresponds to the 30
input features from the dataset. This connects
to a single hidden layer with 13 neurons,
which I chose for a completely arbitrary reason:
it corresponds to my age. Each of the hidden
layer neurons has the relu activation function
applied to them. Why relu, you may ask. Is
it not inherent in logistic regression that
we use sigmoid? Yes it is, but not necessarily
in the hidden layers. Frankly, relu performs
better than sigmoid, and logistic regression
with a threshold can still be used as long
as the output value has the sigmoid activation
function applied to it. These connect to a
single output layer neuron, which WILL have
the sigmoid activation function applied to
it. In this line, the model is compiled. The
loss or cost function that is used is binary
cross-entropy. The only thing that you really
need to know about binary cross-entropy is
that it is used in tandem with the sigmoid
activation function for 2-class classification
problems. I use the adam optimizer, which
as you know from the last video, simply states
how the weights are updated, and in general,
performs better than most other optimizers.
I also state the learning rate which will
be 0.001, and that I want the model to keep
track of the ‘accuracy’ metric. I then
define the earlystopping function, pass several
parameters, assign it to earlystopper, and
add this as a callback in the .fit() function,
which is all assigned to history. In the .fit
function I pass several other parameters including
X_train, y_train, the number of epochs, which
is 2000, the validation split, which is 15%,
and verbose which I set to 0, or false. In
the next line, I assign history.history to
history_dict. I’ll just be using this to
plot the performance of the model. For more
information on the .fit function, earlystopping,
and much more, watch the previous video in
this series, where I predict the number of
views on a youtube video with neural networks,
implemented in Keras. If you aren’t really
familiar with this keras syntax and you are
confused, certainly go back and watch the
previous video. Once I cover a particular
topic, I don’t repeat myself in the same
detail, or else this video series would be
extremely repetitive.
From this point, I plot the training loss
and the validation split loss over the epochs.
As you can see, the early stopping function
did its job--it made the model complete training
much faster, and didn’t allow the model
to overfit. Next, I plot the accuracy of the
model over the epochs. Accuracy is the inverse
of the cost in the sense that you want it
to be as large as possible, whereas you want
the cost to be as low as possible. Once again,
the graph indicates that the model is performing
well. Once I’ve done this, I want to calculate
the loss, and also the accuracy as a fraction
of the number of correct diagnoses out of
the total diagnoses, for the testing data.
I do exactly this in these lines. When I print
out the loss and the accuracy, you can see
that they are pretty good. Next, I use two
other metrics to further reveal the performance
of the model. They go hand in hand, and are
pretty cool. First, I must explain why there
is a need for a metric besides the accuracy,
which is calculated as a fraction of correct
out of incorrect responses. And before I do
that, you must understand the differences
between true positives, true negatives, false
positives and false negatives. I’ll do that
with this chart. True positives are where
you correctly predict something to be true.
This could be correctly predicting that a
tumor is malignant. True negatives are where
you correctly predict that something is false.
This could be correctly predicting that a
tumor is not malignant, or in other words
benign. There are also false positives where
you incorrectly predict that something is
true. This could be predicting that a tumor
is malignant, when in reality is not. False
negatives are where you incorrectly predict
that something is false. This could be predicting
that a tumor is not malignant when in reality
it is. Depending on your intentions, you will
want to maximize certain parameters and minimize
others. To illustrate this concept, I will
use the diagnosis of breast cancer as an example.
A patient, for whom this diagnosis is crucial,
may want to make sure that if she does have
cancer, then it will be identified, even if
it the diagnosis may call for further testing.
This patient would want to maximize true positives
and minimize false negatives, even at the
expense of an increase in false positives
and a decrease in true negatives. On the other
hand, a hospital with plenty of patients may
find it more profitable to raise the diagnosis
threshold, so that there are fewer patients
without cancer that attend costly further
testing. This hospital would want to maximize
true negatives and minimize false positives,
even at the expense of an increase in false
negatives and a decrease in true positives.
Perhaps the hospital seems evil and and even
abhorrent to you. Regardless, it would be
interesting to see how the model would perform
in both of the situations, and many more,
in a single graph. That is exactly what a
ROC curve does. AUC is the area under the
curve, and it is a numerical value that tells
us how well the model would perform in different
situations, like with the patient and the
hospital. The ROC curve essentially plots
the possible thresholds of a trained model.
It is created by plotting the true positive
rate (on the y-axis) against the false positive
rate (on the x-axis). The “rate” for true
positives is the number of correctly predicted
positives out of all the actual positives.
The “rate” for false positives is the
number of incorrectly predicted positives
out of all the actual negatives. Let’s see
how an roc curve would be created. For the
first point, say that the threshold was at
0.3. Any probability below 0.3, and the model
would predict the patient has a benign tumor.
Any probability above 0.3, and the model would
predict the patient has a malignant tumor.
This threshold is, relatively speaking, pretty
sensitive, meaning that if the model’s diagnosis
is benign (and not malignant), then it is
most likely that the correct diagnosis is
also benign. This threshold would best suit
the heartfelt patient, in the scenario from
before. Perhaps with this threshold, 210 out
of the possible 212 actual positives were
correctly diagnosed, as malignant tumors.
So, the true positive rate is 210/212 or about
0.99. Unfortunately, with this threshold,
the model may incorrectly diagnose 55 out
of the 357 actually benign tumors, as malignant.
So, the false positive rate is 55/357 or about
0.15. This coordinate pair would then be plotted
on the roc curve. Next, I will use a threshold
of 0.7, and calculate the corresponding true
and false positive rates. In contrast to 0.3,
0.7 is a specific threshold, meaning that
if the diagnosis by the model is positive,
it is most likely that the correct diagnosis
is also positive. This threshold would best
suit the lucrative and amoral hospital, in
the scenario from before. Perhaps with this
threshold only 190 out of the possible 212
actual malignant tumors are correctly diagnosed.
The true positive rate would then be 190/212,
or about 0.9. In addition, with this threshold,
the model may incorrectly diagnose only 4
of the 357 actually benign tumors, as malignant.
So, the false positive rate is 4/357, or about
0.01. This coordinate pair would then be plotted
on the graph. The process would continue in
a similar fashion, only starting from a threshold
of 0 and working upward in much smaller increments.
As far as AUC, you just need to know that
a perfect model would have a value of 1.
An ideal model would look something like this.
A model with whatever threshold was used to
generate this point wouldn’t have to make
any compromises in order to please both the
patient and the hospital.
By the way the conflict between the patient
and the hospital is known as sensitivity vs.
specificity. In my opinion this is not only
interesting from a technical perspective,
as a programmer or a data scientist, but also
as a moral consideration; seeing it from the
lens of a philosopher.
Anyhow, back to programming. In these lines,
I calculate the AUC and plot the ROC curve.
In order to do this, I first import these
two packages from sklearn.metrics. Then, I
calculate the models predicted outputs for
X_test, as probabilities, and then assign
these values to y_test_pred. It is important
that the predictions are probabilities and
and not just zeros and ones. Remember, in
order to get the AUC and the ROC curve, you
change the threshold. If your predicted outputs
were categories, represented by zeros and
ones, then the threshold was already set,
likely at 0.5. Attaining the AUC and the ROC
curve with different thresholds from that
point would be impossible. However, I actually
didn’t need to include the _proba, as model.predict
defaults to probabilities. I did this simply
to emphasize the fact that probabilities are
needed for the predictions. So, model.predict_classes
would not have worked. In the next line, I
calculate the false positive rates and the
true positive rates at different thresholds.
Note that the thresholds are actually assigned
to a variable. In the next two lines, I calculate
and print the AUC. In these lines I graph
the ROC curve. As you can see, the model is
pretty darn good.
I then repeat this process for the training
data. Once again it performs well.
I can now conclude that I have a functioning
model. Remember that in this implementation
I used sigmoid as the activation function,
and binary cross entropy for the loss. This
model would be considered a logistic regression
model, and is just one approach to a binary
classification problem. As promised earlier
in the video, I will now implement a different
approach to the same problem. This approach
involves the softmax activation function,
and categorical cross entropy as the loss.
By the way, this approach, which would be
known as softmax regression, can also be used
in multiclass classification problems, while
the simple logistic regression can’t. So,
without further ado, let’s implement it.
From the start, things actually start to get
different from the last implementation, and
the last video for that matter. I use the
function to_categorical from Keras.utils,
pass y_train, and assign it all to y_train_cat.
What to_categorical does is it converts a
class vector to a binary class matrix. Ok,
what does that mean? Well, a vector is simply
a one-dimensional array. This is is the original
y_train, a vector. Note that besides the zeros
and ones there is also an index. A binary
class matrix, or what to_categorical makes
y_train, is simply a bunch of rows and columns
made up of binary numbers, where each row
represents a class. Let’s take a look at
what y_train looks like after it was passed
into to_categorical. Here, the first row corresponds
to the first value of the original y_train.
How is 0,1 equal to 1 and 1,0 to 0? Well,
1,0 represents the first class, which is zero,
and 01, represents the second class, which
is 1. The same applies for any two corresponding
data points in the original and current y_train.
To further solidify this concept, let’s
see how this would play out with three possible
classes. Let’s say that the classes were
originally expressed as zeros, ones, and twos,
and that the first 4 data points are 0,2,1,1.
Assuming that zero is the first category,
one is the second, and two is the third, after
passing 0 into to_categorical we would get
100, after passing 2 we would get 001, after
passing 1 we get 010, and again, after passing
1 we get 010.
The reason why I did this is that the activation
function, softmax, and the loss function,
categorical cross entropy, that I use, requires
data in this format. I’ll talk about the
activation function and loss function a bit
more in just a second after we define the
structure of the model.
In these lines I do exactly this. First I
define the model in the line model=sequential().
Then I add an input layer with 30 neurons.
This corresponds to the 30 input features
from the dataset. This connects to a single
hidden layer with 13 neurons, which again
I chose simply because it corresponds to my
age. Each of these neurons then has the softmax
activation function applied to them. These
neurons then connect to two output neurons,
instead of one in the logistic regression
model. With the softmax activation function,
this is simply the necessary formatting. The
format, as you may have noticed, corresponds
to the training data after to_categorical
was applied to it. To_categorical was used
to match the training data to the format that
the models outputs would be in. This allows
us to train the model with the training data,
which is obviously a crucial step in machine
learning. In my code, I don’t apply to_categorical
to y_test and assign it to y_test_cat just
yet, but you very well could. Eventually,
you will need to do this in order to calculate
the loss and cost of the testing data. Before
I train the model, I first define the loss
which is categorical crossentropy, the optimizer,
Adam, and the learning rate, 0.0001. Adam
was already explained in the previous video,
but categorical crossentropy is new. All that
you need to know about it is that it is used
in tandem with softmax for multiclass classification
problems. Next we train the model with the
earlystopping callback, assign it to history,
and assign history.history to history_dict.
After that, I plot the training loss and the
validation split loss over the epochs on one
graph, and the accuracy over the epochs on
another. As I did last time, I also calculate
the loss and accuracy of the model with the
testing data. Note, that this time, I have
to apply to_categorical to y_test, because
we are using softmax regression. Already,
you can see that the model is pretty good,
just as it was with logistic regression. Still,
I continue measuring the performance of the
model with the AUC and ROC curve. I first
find them for the testing data. I do that
in these lines, and once again they are pretty
darn good. When I do so, note that I use the
original y_test, and not y_test_cat, because
this is the required format. I do the same
for the training data. So, I use the original
y_train. Again, the model performed well.
We can now safely say that in this instance,
and with this particular problem, it really
doesn’t matter if we use logistic regression
or softmax regression.
In the script, I proceed to modify the number
of hidden layers, to see if it will improve
the performance of the model, a little bit.
So, I define a model with 30 inputs, 4 hidden
layers each with 13 neurons, and an output
layer with 2 neurons. Each layer neuron has
the softmax activation function applied to
it. I train the model over 3000 epochs with
the early stopping callback, and plot its
loss and accuracy over the epochs. I then
calculate and print the loss and accuracy
as numerical values. I also find the AUC and
the ROC curve. In conclusion, it doesn’t
change the performance of the model.
Before concluding the video, I want address
an issue that someone in the process of developing
a model would face. The problem is the following:
how can you tell if your model would benefit
from having more data? My prefered approach
would involve a visualization, and is known
as learning curves. It involves restricting
the amount of data used a lot to train the
model, then calculating the performance, increasing
the amount data used slightly, then calculating
the performance, increasing the data used,
then calculating the performance, and so on.
This data can than be graphed, with the number
of data points used on the x-axis, and both
the training and testing accuracy on the y-axis.
As the amount of data used increases, the
training accuracy shouldn’t be impacted
much, and the testing accuracy, at least initially,
should increase. If the testing accuracy follows
an upward trajectory in the latter half of
the graph than it would most likely benefit
from more data. If it remains constant or
decreases, then you can assume that model
will not benefit from having more data. I
implement such learning curves in these lines.
BTW, linspace creates a defined number of
equally spaced values in a defined range.
I use it here to determine the different fractions
of the training data that I will use. As you
can see from the resulting graph, increasing
the amount of data used in training would
most likely not have any effect on the model’s
performance.
I mentioned this in the last video, but there
are many ways in which programmers improve
the performance of neural networks. Among
these are batch normalization, dropout, and
weight regularization. For this model and
for this dataset, after implementing these,
there is no noticeable improvement. If interested,
the supplementary code on my github contains
the implementations of each of these.
So there you have it--functioning logistic
and softmax regression neural network classification
models implemented in keras.
In case you didn’t already know, this video
is part of a series in which I find really
cool datasets like this one, and for each
of them I show you how to implement a Neural
Network. In doing this, I hope to both entertain
and educate you. Subscribe and click on the
notification bell to be notified when I release
the next video in this series, and also hit
the like button and leave a comment if you
want this video to reach more people.
