EuroSciPy 2017: Keras
Maggio, Valerio
Ten Steps to Keras Goal of the Tutorial - Introduce main features of Keras APIs to build Neural Networks. - Learn how to implement simple and complex Deep Neural Networks Architectures using Keras. - Discover Keras Implementation and Internals. - Note: examples and hands-on exercises will be provided along the way. - Multi-layer Fully Connected Networks (and the backends) - Bottleneck features and Embeddings - Convolutional Networks - Transfer Learning and Fine Tuning - Residual Networks - Recursive Neural Networks - [Variational] AutoEncoders and Adversarials - Multi-Modal Networks - Custom Layers and Optimisers - Interactive Networks and Callbacks
 Alright, so in the meantime you get the material. I will just give you a few words about this tutorial and... Oh, sorry. What's the idea I have in mind with this tutorial? So... Alright. You should have a list of notebooks here. The original title of this tutorial was 10 Steps to Carousel. So the idea was to introduce the Carousel framework into 10-ish steps. And these steps are going to be notebooks. You have a pre-amble notebook here where you can see all the instructions. In the back, can you read that? Is it fine for you? No, probably it's a bit tiny. Do you want me to... Better? Good. So this is the goal for this tutorial is to introduce the main features of Carousel and to learn how you can actually implement deep networks using Carousel. And one thing I should definitely want you to learn and to know is that Carousel is, at the end of the day, very easy to use but it is not for easy tasks. You can actually do complicated stuff using Carousel. So this is the outline for this tutorial more or less. And these are the topics we're going to go through in the next hour and a half. So multi-layer, fully connected networks, so to introduce the very basic features. And then we go through engine layers, features, embeddings, convolution networks, hyper-projects, hyper-amplitude tuning, how you implement custom layers in Carousel, and how you actually use deep neural networks, in particular convolution networks and residual networks, transfer learning and fine tuning, and if we have time, we can refine your network and auto-encoder and in the end, multi-modal networks. So there's a lot of stuff, as you can see. All the materials are online, you can actually play with it. All the code has been thought to be run on your notebooks. So I actually simplified some data examples. So you should be able to run it even if you don't have GPUs on your machines. But of course, if you have it, it's definitely better. Just run faster. So I'll try to go fast on some topics. But at any time, if you have questions, please feel free to interrupt me. There's no problem at all. You will find the requirements. Basically it's going to be Python 3. 3.4 should work as well. And also Python 2, I don't really know. Just a quick note about setting everything up. A friend of yours, some people from the audience, sent me that actually it may be a bit cumbersome to set up the Keras folder on Windows machine. If you try to Google it, so what I was saying here, the.keras folder where the actual Keras configuration file should stay, even on Windows machines, should be played. So Google it a few minutes ago, it ended up to be that the home folder is what you get in return from the Python function here. So typically something like seed, semicolon, double, backslash, users, double, backslash, and your username. So these is the place where you should create your.keras folder. I know if you experience the same problem, but this should fix the problem. So when I say home folder here, I assume even on Windows machine that is the folder. And so you should have for this tutorial, for this series of notebooks, these configuration files. I will tell you later on the meanings of these configurations, but they're quite straightforward. So, I assume you already tried and verified everything. That's nice. Good shot. After that, so let's start with the first notebook, if you don't mind, which is going to be number one, multi-layer fully connected network. The main goal of this notebook is to provide you an example on how you can actually deal with our.keras and how you can build networks using.keras. So I freely borrow these payoff sentence saying that Keras is from the Django project, saying that the Keras is deep learning library for perfectionists with deadlines. And these means that, as in Django does, dealing with Keras objects and creating networks with Keras is very easy to go. And in fact, one of the main important features of Keras is summarizing these sentences from the official documentation, when they say that being able to go from one idea to results with the least possible delay is the key to doing good research. In a few words, Keras is a high-level neural network API written in Python. It's pure Python, so you can actually take a look at the code and even modify it, which is capable of running on top of different deep learning frameworks, which Keras calls backends. So far, the officially supported backends are TensorFlow, C and TK, which is the deep learning library from Microsoft. And these has been the latest addition. I think it's been added from three or four versions. And the ANO. So historically, Keras supported the ANO TensorFlow, now C and TK. And just for you to know, there's a sort of unofficial support of MXNet as well in the back end. I'm saying it's unofficial. Well, it's not in the master branch of the Keras code. That's because for some reason, the developer of the MXNet back end supported, are now supporting the Keras 1 API. But now we're using Keras 2 API. So there's been an API change some time ago. And for some reason, MXNet is still stuck at the Keras 1 API. So that's why the MXNet is not yet officially supported. But if you use Keras 1 version API, you can actually also have MXNet as a back end. Having multiple backends is a feature that you will appreciate very soon in the next examples. So starting with this very simple example, let me remove just to get some space. It's a Kaggle challenge data. I just read very briefly the problem. So the Hutter group is one of the worst biggest e-commerce company. And the consistent analysis of the performance of product is crucial in this company. And while that do the diverse global infrastructure, many identical products get classified differently. From this competition, we have provided a dataset with 93 features for more than 200,000 products. Of course, for the sake of these projects, I reduced drastically the number of samples. In our data, each row corresponds to a single product. And so we have a total of 93 numerical features, which represents the count of different events. So if you take the data, it should be placed in your data Kaggle auto group folder. For the rest of these examples, we're going to play a very simple model, which is logistic regression. So for those of you who doesn't know, the logistic regression is a very simple supervised learning algorithm characterized by the usage of the logistic function, also known as the sigmoid function, which has the shape, so the S shape, that's the sigmoid name. So after preparing the data and getting the data, so in a few words, what we do here is just loading the data from the CSV file. There's actually some pandas code running in the back end. And then we preprocess the data so that we just scale the features using standard scaler in scikit-learn. And then we preprocess the labels to have one hot encoding of the labels. Are these terms clear to everybody? So I can go safely with that? Okay, fine. Thank you very much. So taking a look at the final results we have, we have nine classes and 93 features as we expected. So these are the labels. Of course, these are anonymized in some way because they're Kaggle data. And just to give you an example of what is the result of one hot encoding, this is something like we just have, for each sample, just one element corresponding to the activated bit, the so-called hot bit. And that's why this sort of encoding of the labels is called one hot encoding because just one single label is enabled. Of course, this is a single class problem. It's not multi class. Every single sample is just one class in our data. So if you want to implement it in pure theano code, I will go very fast about that. But if you want to implement a logistic function using theano, these are the code you have to do. So you basically have to declare these objects. So the tianomatrix and the shared variables here, the shared are those that are actually modified during the training, that's the difference. So you have to implement manually everything basically. So it's just the probability target, the cross-entropy loss function we want to use. But the only thing, which is not only, of course, but one something which is provided from theano is the calculation of the gradient. So theano provides out of the box the calculation of the gradient. This is one reason why you don't want to really implement this code in pure Python code. So you want to rely on these libraries. So finally, you have to create this tianofunction. So that connects every single tensor object you're actually creating. So you provide the input, you provide the output there. And this is the rule for the updates of the shared variables here. So of course, these are the weights and the bias, simple model here. So we compile all the code. In case you don't know, this is based on symbolic execution. So you have to compile it and then you run it. So everything you set up previously has not been executed yet until you compile it and you run it. Then you create a new function for the prediction. Using all the tensors already provided. So this is for the training. This is for the prediction and the rest of the code is just repeating the training process for the number of epochs. So we are actually implementing the training process manually. And this is the way theano does this very simple problem. As you can see, it's very low level, right? So you have to implement it at every each step. If you want to use TensorFlow, as you will immediately understand, the level of obstruction is exactly the same of theano. So again, you have to specify placeholder here. So let me just execute it because I just want to show you something. So you have to create the placeholder here. And the placeholder are going to be the variables that are going to be updated during the learning into the graph. So as I told you, this is symbolic execution. So if you print X, you just got a tensor object with no shape yet because you don't really have passed the data, fed the data to the graph. And the only thing you know is that you have 93 dimensions, so 93 features. And why should will be a tensor having a total number of classes at the mention? That's a second dimension. TensorFlow ships with these tool, which is very useful, which is called TensorFlow. I'm going to execute it just because I want to show you. This is very nice. So this is a sort of a little bit of engineering of the TensorFlow code. So we're going to again create all the variables here. Then TensorFlow provides a little bit more abstraction with respect to the Tiano having this softmax function already implemented. Then the matmul operation is going to be the equivalent of the dot operation in numpy if you come from that backend, the background. And then we are actually adding some additional information that will be used in the tensorboard tool. So we're going to record some histograms during the learning and some reference to the scalar objects we have here. Again, this is very intuitive because we're actually defining different scopes. So we are defining the scope of the model here, the scope of the cost function, the scope of the train process. And so they're going to live in different scopes, so you can actually better manage them. And finally, the accuracy scope, which is going to be the metric scope we want to use to evaluate our model. So then TensorFlow provides you this file writer, which is actually something that stores the results of the graph you have. So the computational graph you have in the backend. And after that, you just run it defining TensorFlow session. So everything living in this TensorFlow session can be interpreted and then run during the learning process. But again, what you have to do is to implement manually all the learning process by actually writing code and providing the data for each of the tensor you want to change during the learning process. So oops, that's something went wrong, of course. That's the demo approach. Oh, I know why. Oh, let me just, sorry, let me just, I just want to clear the memory of the GPU. I'm actually running on some GPUs in a remit backend. So let me run again. Sorry about that. OK. All right. So I have to load the data. All right. Come on. That's a cold start. Very cold one. All right. So let me skip the other part. So let's go to the TensorFlow. All right. OK. Now this is going to work hopefully. Please. All right. So the app box, we have just run it for 25 app box, I think. The number of the box is, I can remember. So we have 25 app box, and this is the value of the accuracy during the different app box, and this is how the plot, we have just added it, how we plot different costs, how the cost changes during the different app box. So this is the trend. All right. So I just want to show you this TensorFlow tool, which is going to be something we're going to see also integrated in Keras. That's why I'm showing it now. So I'm running. So this TensorFlow.tensorboard command runs a server typically on the port 606.006. And what this does is to read the log file that has been written during the execution in particular, the file stored in this directory. All right. So hopefully we're going to see something. Okay. So we have these colors here. So we have the plots for the accuracy automatically provided, the plots for the cost function during the training, and the plots, and all the different plots for the mean weights and the mean buyers. All right. This is very easy to use and very useful when you face real problems. Another thing it may be useful is to take a look at the graph. So this is actually the model that has been implemented inside TensorFlow. So this is how the graph of tensors has been translated by TensorFlow. So you can actually take a look at all the different nodes here. So each of these corresponds to the different scopes we specified. And finally, these histograms we built to show the difference of the different trends of the cost histogram and of the model histogram. Something wrong maybe because I just run it multiple times. I'm safe holder. So that's why you see this so strange. Anyway. All right. So let me stop this. All right. Okay. So let's see how we implement a logistic regression using Keras. Finally, as you know, the logistic regression may be implemented as a neural network, just one layer, which has the sigmoid activation function. And so it is. So you have all the data again. And we are going to use these sequential objects. We're introducing this new sequential object in Keras. The sequential object gives you the idea that you want to build a series of layers stacked one on top of the other. And so you actually instantiate this sequential object and you simply add layers to these objects. So in this case, we had just had a dense layer having a number of units in output corresponding to a number of clusters. So we specify the input shape, so the number of features we have. And the activation is going to be the sigmoid function as we expect because we implemented a logistic regression. Finally we add the decision function, which is the softmax function as we did in TensorFlow. And we in the end compile the model again. So we compile specifying which is the optimizer. In this case, we specify the stochastic-grade descent. We actually did the same using TensorFlow. Let me just go back a bit. So when we are in the train scope here, we specify that the optimizer was the grade descent optimizer with the minimization problem. So it's stochastic-grade descent with these learning rates. And we are doing the same here. So we are implementing the same using the categorical cross-entropy and the stochastic-grade descent optimizer. After that, we just call this method in Keras object, which is called fit. And we provide the data, the training data, and the training labels. And that's it. So you don't have to implement it manually. You don't have to implement the training process manually. You just call the fit method as you do in scikit-learn, for instance. And that's it. And Keras provides you automatically a series of our default parameters. We're going to change during the next steps of these exercises. But in the very beginning, you just create the model, you compile it, and you fit it. These are the three steps you have to do, the only three steps you have to do to make our Keras model up and running. And so as you can see, if you run the fit method, you're going to have a series of 10 epochs by default. And Keras provides you these progress bar during the training, telling you the time it takes, the values of the loss during the training, and how does it change. All right? This is the very beginning. So now let's have a look at it in a more interesting way, let's say, so in more details. So the core of the destruction of Keras is a model, of course, and the main type of model is going to be sequential objects. The sequential object provides you this method, which is called add, and so you have different layers. In this example, we just had one layer, which is the dense layer, and then the activation layer. So the dense layer, this is the dense object, has this parameter. Most of them are default values, and in case you don't know, keep the default values. Keras typically has been designed to have the best default values you can get for the different objects. And this is particularly useful when you run it for the first time, especially for optimizers, when you have to set up the learning rate and depending on the optimizer, different other parameters. The most important things you have to learn and to understand very soon is that the first and the only required parameter for this layer, and this is typically for all the layers, is the first one, which is units. As you can see, it's the only one that has not default values, of course, and this is where you put your insight and your knowledge, and the unit's value is going to be an integer value greater than zero, of course, and this corresponds to the number of neurons that you will get in output from the layer. Keep this in mind, it's going to be the output number in neurons, not the input, right? You have to specify the number of input dimension only in the first layer. The next layers will get, as input, the output of the previous layers automatically, you don't have to specify it. So the units is going to be the number of neurons you want in the output. So back to the example, since we just have one layer in this example, the number of units we want to have in this network is going to be number of classes because it's the output we want to predict. It's just one layer, right? So it's just number of classes, so this case is going to be nine. All right, and as you can actually see, since this is the very first layer of a network, we also specify the input shape, and this is mandatory for the very first layer of the network. You always have to specify what is the input shape. Speaking of which, the input and the output shapes of the layers are going to be this one. In input shape, you get a non-dimensional tensor object with these series of shape. Remember that the very first dimension of the input tensors are going to be the best size. In other words, the number of samples you're going to provide to the network during the training, right? And this parameter is specified when you start the training process, so you call the fit method. You don't have to specify this dimension in the input shape parameter. All right, so let me just go back again to this example. As you can see, the input shape in this case is going to be dims, so the number of features we have. All right, so this is 93 in this case. We don't specify the best size. The best size is indeed automatically specified in the fit method. We're going to use the default parameter here, but we can actually change it. So remember, the input shape is mandatory for the very first layer and has to not include and must not include, sorry, the best size dimension, which is going to be the very first dimension of the tensor you have. And as output, what you get in output form a tensor is the best size again, so the same number of samples, of course, but as they mentioned, the units you specified as the first parameter of the layer, so the output neurons. All right, is everything clear? Questions? All right. So before moving on, some very few notes about the star initializers parameter. So typically in the dense layer, but in most of the layers, you have this kernel underscore initializer and bias initializer parameters. This is just for you to know that in literature, there are lots of methods to initialize the weights of neural networks, and this is particularly useful, and different methods to initialize for the initialization of the weights can drastically change the performance you're going to have. So I just proposed you to take a look, of course, at the keras.initializers to see what are the functions supported by keras. There's lots of them. And read these very interesting article. This is not in, this article does not include all the possible initializer supported by keras, but this is very interesting, and I give you some recommendation paper to read. This is a very interesting topic, and as you can see, the default initialization method for different weights by default will change according to the different layers keras provides. So if you take a look at the default parameter for the convolutional layer, it should be different. I cannot remember for sure, but it should be different, because for convolutional layers, you have different initialization technique by default. Other core layers provided by keras, you have flatten layer, reshape layer, permute layer, these are going to be so-called operational layer. So when you have to, when you have, they're not learning anything here. They're just making operations on the tensors during the flow of the network. The lambda layer, you can actually specify the custom Python function to apply to the tensor. All right? Please keep in mind that what a keras layer gets an import is always a tensor object, which is going to be physically and concretely corresponding to a tensor flow tensor object, or a theano tensor object, or in this case, nowadays also we have a CnTK tensor object, right? So to give you just the idea, when you run the fit and the training process using keras, keras indeed is not performing any real operation on the GPU, all right? So the keras code relies on the Python level, and all the real computation on the GPU and stuff is demanded completely to the back end. The main advantage you get from this approach is that the same code you write, exactly the same code, is supposed and guaranteed to work on all the back ends supported by keras, without changing any single line of code. Keras does all the job for you, even if there are some very slight and tiny differences between the, for instance, the handling of the shapes of tensors between the tiano and tensor flow. Keras does all the job for you, you don't have to change anything. That's power of keras indeed. Another important layer is the activation regularization, so you can actually apply regularizers on the activation functions to, so for instance, L1 or L2 regularization on the activation function of your network. This picture, kindly provided by my friend Ian, shows all the, probably it's a bit tiny, but should give you another view of all the things, all the kind of networks supported by keras. Keras provides support for feed forward network, so the classical network, you provide input on the left and you get output on the right, so the feed forward network in these sets of networks, also the convolution networks fill in. You have recurrent neural network, support for LSTM or gated recurrent unit network, and also for unsupervised learning, you can actually build autoencoders, variation autoencoders and sequence networks using keras code. Finally, keras also provides support for different optimizers, not just the stochastic ring descent, of course, there are many of them, you can have aider grad, aider delta, lm sprop, lots of different optimizer. You can actually take a look at the keras.optimizer package to see all of those optimizers supported by keras. If you want to data science in a bit of this example, we're going to play a bit with the hyper parameter of the network and the parameter of the feed method. I assume you all know what overfitting is, or underfitting is, so something we can do. One method I'm going to show you, because we're going to use it very heavily, a lot, is this summary method for the model object. When you call the summary, you actually have a look on what are the layers, what are the outward shape, how many parameters you have, and in the end, how many total parameters you have, and how many of them are trainable parameters. If these does not make any sense to you, it will make very soon. So far, the total number of trainable parameters are going to be the same. The design thing about this example, we're going to use some sort of train test split on the data. We have not only training data, but also validation data. We want to fed into the training process. We're going to apply a technique which is called early stopping. The early stopping technique is something that is typically used when you have big networks and lots of data. The intuition is that in these case, we're going to monitor this metric, which is called the validation loss. When the validation loss does not change after two epochs and does not change significantly, so you can actually specify the precision you want, the training process is automatically stopped. This is the early stopping, because, differently from other machine learning methods, unless you specify these kind of tricks, when you run a fit of a neural network, you have to specify for how many steps you want the learning to go on, right? So the number of epochs, so-called. These other objects is the model check point. During each epoch, at each step, automatically, this function, this object, records the value of the validation loss again and saves the weights, so the stages of the network at that stage. And automatically, this has been set up to save only the best model. So in the end, you get the best weight configuration, so leading to the best validation loss value. You can plug both of them using this parameter in the fit method here, which is the call backs. So Keras provides this way to plug additional callback objects. They call callbacks because they're automatically called by the framework before the next batch or before the next epoch is going to be processed during the training. You don't have to do anything, you just pass the callbacks. We're going to specify a different number of epochs. We're going to specify what are the validation data in this case. We previously provided just training data, and these are the two mandatory parameters, the very first two, validation data again, and we specify the size of the batch. And we also specify that we want some verbosity, but this is the default one. All right. All right, so we run it for 50 epochs, and these goes on and on. Additionally, we also have a validation loss that, since we have validation data in this case, we're going to also have the computation of the validation loss. So during each step, we have a log of how the loss changes, the training loss changes, and the validation loss changes. Okay, moving on. Okay, very briefly, moving on just a bit. So we want to create a multi-layer fully connected network. When we say fully connected network, we're going to expect to have all dense layers. The very first example of multi-layer fully connected network is the multi-layer perceptron. The multi-layer perceptron is characterized by having one input layer, one hidden layer, one output layer. All right. So technically, when you have this sort of network, so one, with just one hidden layer in the middle, you're calling it multi-layer perceptron. When you have at least more than one hidden layer, you're allowed to call it deep network. All right? So when you have at least two layers in the middle, it's going to be a deep network. So in this case, we're going to have a multi-layer perceptron because we have one input layer and this is just the decision layer. So it's going to be very shallow network yet. And as you may assume, the way you plug additional layer, so you build a multi-layer network in Keras is just by calling the add method to the special object multiple times with different layers. All right? So again, make sure that you specify the input shape in the very first layer. In the second layer, don't have to because it's automatically induced by the previous layer. All right? So again, we compile the model. We print the summary, so we have dense number two, dense number three here, and the final activation, which is the softmax for the decision function. Of course, since we have 100 neurons here in output and nine in the end, the total number of parameters has increased, as you may expect. Then we run the fit model again and the validation loss changes drastically considering the same data. Yes. So this is 1.6 taken just randomly, and this is 0.7. So this is going to be a better model with respect to the previous one with our data. So if you want to play a bit, there's an exercise for you. The idea is to play a bit with it, adding as much layers as you want, better to add a couple of, and run it in your machine to be sure that everything's up and running, and to see that it's very easy to run using Keras. All right? The next steps will be taking a brief look at what is called the Keras backend. I just want to show you the Keras backend because this is how actually Keras allows you to have multiple backends running for your deep learning code. Okay. So to give you some hints, the only way to play with it is just to add additional dense layer. How many neurons you want to add? How many layers you want to add? Nobody knows. You just play with it and you see what, if that makes any difference to the final results. That's the takeaway typically for deep learning. At the end of the day, you don't really know what you're doing. So let's have a look at this Keras backend. The Keras backend is the module which is provided by Keras that integrates to real backends. If you take a look at this dot backend package, right, in Keras, so it's going to be the Keras dot backend package. If you take a look at the code, you will see that in that package, Python package, you will find a file which is called a module, Python module, which is called TensorFlow underscore backend dot pi, Tiano underscore backend dot pi, and now Cntk underscore backend dot pi. So this backend is a sort of wrapper module that calls the real backend depending on the configuration file or the environment variable you have set up in your settings before running your code. So we have actually set up our TensorFlow backend in the configuration file, as you may have assumed, so I will just want to show you. So if I'm going to cat the Keras dot JSON file, all right, among these configurations, we have the backend and we are specifying TensorFlow. If we just switch these directive to Tiano, you're done. You just rerun your code, all right, and Keras automatically switches everything on Tiano backend. All right. How these backend works, so as I said, the Keras dot backend is the wrapper for the other module. All these modules, so basically the Keras backend provides the interface and all the functions provided in this Keras backend module are assumed to have a counterpart in each of the backend supported. So when you're actually calling the, okay, let's have a look at it. So maybe it's easier to do. So we're actually reusing the Keras backend API to reimplement again the logistic regression example, but in this case, we're actually, we're still using Keras code, all right. So let's have a look. So here we're creating placeholder objects using the K module, which is the Keras backend, we just call it K, it's just a sort of naming convention here, all right. So the K is going to be the Keras backend module and we're going to create a placeholder object. The syntax is very TensorFlow-ish, all right, it's very similar to TensorFlow, but what actually happens in the backend is not TensorFlow always, depending on the, it really depends on the backend you're using. So if I'm using TensorFlow as a backend, this instruction is going to create a TensorFlow placeholder object. If I'm going to use theano, that is going to be a creation of theano tensor object, all right. This is the backend API. So we're creating a variable object again. So the level of abstraction has changed drastically, has moved again to the same level of abstraction you have using TensorFlow or theano packages, but with a big difference that this code is supposed to run seemingly using all the backends you want, those supported, of course. So without changing any single line of code, just the configuration fire, all right. So this is going to create a TensorFlow, if you're using TensorFlow, yes. All right. Okay, so I repeat the question. So the question was, why would you actually have support for multiple backends, right? Because of many reasons need, you can, because the different backends have different performances, all right, and so you want to leverage on the benefits of those switching the backends, because sometimes ago the memory management of TensorFlow was really rubbish, so you typically wanted to switch to theano to have a better location memory. Now it's not the case, all right. Because sometimes you want to try different backends having to test if everything is working with the same performances, all right, because of numerical reasons, because of numerical optimization, you may get, all right, for instance, I just read a sort of benchmark article providing a sort of overview of the benefits provided by CNTK. CNTK, among those backends, is the only one using MPI in the backend, so multiple messaging and passing interface, right, so for multiple processing. As far as I know, CNTK is built on top of MPI. So far now TensorFlow supports MPI in the version 1.3, so the latest one. And considering the different network architectures, at the end of the day, CNTK has turned to be the most efficient for the recurrent neural networks, which are going to be the very heavy network to train, the one that takes more time to train, all right. So for instance, for convolutional networks, TensorFlow and CNTK are sort of the same performances according to that benchmark. But CNTK won for on-require neural network, for instance. So maybe you want to switch to CNTK using, without changing any single line of code, that's a benefit. And you don't have to mess with the very tiny details of the different implementation, because Keras does it for you. All the three backends have totally different APIs and totally different syntax for creating tensors. Keras does the job for you to translate all of them, and the way it does is using this Keras backend. So when you're actually running, that's it. This is what I was looking for. You can actually run this function, the K dot function. This is calling matmool for TensorFlow and dot function on the ANO. All right. I don't know the equivalent in CNTK, but TensorFlow decided to call the dot product matmool, matrix multiplication for some reason. Okay. When you call K dot, it's going to be mapped to the corresponding function in the corresponding backend automatically. That's it. So now the question could be, why would you do that? Why would you implement it using the Keras backend? So what's the reason why Keras provides you these APIs to be used? Because this is the same API you're actually using when you're creating custom layers. If you create custom layers using the Keras backend API, you're automatically sure that that implementation is going to work on all the backends provided, because you're using the Keras backend functions. You're not using the TensorFlow or the Tiano real functions. All right. You see what I mean? Okay. So the rest of the notebook is to show you how you basically do the same TensorFlow-ish stuff using the K backend. And there's also an exercise for you, which I'm going to skip, which is the implementation of the linear regression. I'm going to skip also because you indeed have the solutions in your material. So if you load it, you're done. All right. Okay, so in case you don't have any questions, I would just move forward. Yes, please. Yes. You're actually stucked. Sorry. I repeat the question. I'm actually advertising all the benefits of Keras. So the question was, what's the downside of it? I may say that the downside of Keras stands on the downside of the supported frameworks, supported backends. I would say three things. Not all the functions are really supported by Keras backend, because Keras backends, since it has to provide you that the same code works seemingly on all the backends, in case there are some very specific functions supported by one specific backend, you're not going to find it in the Keras backend API. But a sort of note about this is that if this framework, the one that has this tiny function, is going to be TensorFlow, you can actually manage to do it, because Keras is going to be integrated in TensorFlow, and it is better integrated each version by each version. I will show you something in the end. That's one problem you have. And the downside, and not the downside I may say, is that since Keras is not doing any real computation, you can do with Keras everything you can do with all the frameworks. But if, for instance, so far in TensorFlow, which is the one I know the best among the three, there's no official support for dynamic graph computation. So everything is going to be static. So this means that when you create the graph, you have to compile the model indeed. So this means that after you create in the network, you're compiling it. So you're creating the network beforehand, you compile it, and so the resulting graph will be put, all the tensors will be put into the memory of the GPU, typically, if you have it. And so that's graph, it is very difficult to change it during the training. You cannot do that. The PyTorch approach that I believe we will see in the keynote we have in two days, you will see that that approach is totally different. That's called dynamic graph creation. So you create it one step at a time during the execution, and so that's our sort of different approach to that. Keras does not do that because PyTorch is not supported yet. I say yes, I don't know if we'll be supported at some time. All right. Yes. Default backend. Yes. So the default backend was the question, and the answer is the ANO is default one. You can actually go to the configuration file, or so as I showed you, you just, for instance, let me do this. I can show you in a terminal, maybe it's better, the one to risk to waste the notebooks. But indeed, if you, you have two ways. All right. So if I print a cut, keras.json file, you will see that the backend is TensorFlow because I set it. If I do change this directive, all right, if I specify the ANO here, or even CnTK, all right, now the backend is the ANO. If I actually go in Python and import keras, this is using the ANO backend. You see? If I do this just to play a bit, CnTK should be installed. This gives me the opportunity to say a few words about that. Using CnTK backend. I did not include that examples using CnTK, mainly for one reason, because CnTK is still only supported on Windows machines and Linux machines. There's no support for macOS. You can actually have CnTK installed using a Docker image. So I prefer not to include also these into the setup of this environment. It was not really required. The main focus was on keras, of course. But this was the main reason. And the reason why is it, I don't really know. It's Microsoft stuff. You know, in the end, it's always Microsoft. So let me switch back to TensorFlow. Another way you can do it is by setting us a keras underscore. Let me just move a bit so you can reduce it. You can actually export. No, let me do this. Yes, I can do it. Keras underscore backend environment variable, back end, equals to piano. Okay. Let me show this first. If I do this, sorry. If I import keras, now it should be TensorFlow again. All right. It takes sometimes, typically. Okay. If I do, I can do this, keras underscore backend equal piano, Python, minus M, import keras. Oh, sorry. I cannot remember it. Is this, how's the way to do it? Minus A, C, thank you so much. Using the end of backend. You can actually define this keras backend environment variable and you switch the backend without changing the configuration file. This is because the configuration file is just one file. If you want to have multiple keras instances running, it's not a good idea to change the configuration files multiple times. So you just define the environment variable there. The keras underscore backend and you're done. All right. Okay. Are there any other questions? Fine. So, let's move on. Let's talk about these Amnist dataset. I believe all of you already know what the Amnist dataset is about. In scikit-learn word, this is also known as the digits dataset. So the Amnist dataset is one of the widely used dataset for deep learning experiments and at some point they decided that it was fair enough to not use it anymore. But the Amnist database contains images, black and white images of handwritten digits. And so you want a network in our case to be able to identify the corresponding digits taking a look at the image. All right. So this notebook is just to show you that keras, just like scikit-learn, ships with these datasets, dataset module, and these datasets, Amnist database is provided. So you can actually import from keras.datasets, Amnist module, and with the load data, this method downloads data for you in case you don't have it, and the downloaded data will be placed in your dot keras folder, and you can actually get the data by loading it automatically. The rest of the notebook is just for you to play a bit. So I was asking what type of X-train is, and the reason is, the answer is, of course, it's an ampere rate. So once we have loaded, if we type the type of X-train, it's going to be numpn-eray. All right. The type of Y-train is going to be numpn-eray again. So the load data automatically gives you back the data into numpn-erays, which is something you typically want to have when you have to experiment. How many observations in train gate do we have? So again, it's going to be the X-train dot shape of zero, of course, so the number of samples we're going to have 60,000 samples in the training set, and 10,000 I will tell you in the test set. All right. How many numpn-erays? Observation for its digits. So it's going to be train dot shape again, but I will tell you that 60,000, of course. So the dimension of X-train, so if you take a look at the X-train dot shape, we're going to have 60,000 of 28 times 28 images. So we're going to have images of that size. Since the MNIST images are black and white images, you don't have the channel information, so it's black and white, so it's just two-dimensional object, not three-dimensional object. So very quickly, if you want to take a look at these images, you can actually print it. So I'm going to print the X-train of zero here, and this is going to be a sort of handwritten five. Okay. So we're going to use these data sets to play with the rest of the two notebooks. Okay. The first notebook is, again, about fully connected FIFORWAR network. Let's say the more complicated FIFORWAR network, FIFORWAR network, we're going to take a look at are going to be the convolution network. Some recall of the main things we already did, so sequential object and the graph model of the objects. We talked about the dense layer, aka fully connected layer. In this notebook, we're going to introduce a new layer, which is called the dropout layer, and we applied binary cross-entropy or a categorical cross-entropy to our data using the stochastic grid descent optimizer. Here you have the links to have a look at what Keras supports as optimizers and loss functions. We now introduce these new activation function, which is called the relu function, so the rectified linear unit. This function is going to be very useful in convolutional network. This function has lots of mathematical properties, and this is the mathematical definition of the function. This activation is going to return the maximum value between zero and X, so this means that all the negative activations will be automatically dropped out. When you calculate the derivative of this function, it recalls the sigmoid logistic function. The derivative of this function is called the softplus function. This function, despite its simplicity, is one of the coolest things invented in the deep network words many years ago. The note, keep in mind, this function, as it's heavily used in convolutional neural network. The exercise now asks you to build this network. This sort of syntax is quite standard when you have to specify the architecture of the network. In this case, we're going to have a fully connected network, a fully connected layer, with 512 neurons in output, plus the relu function activation, again the same configuration, and finally another fully connected layer, FC, with a total of a number of classes neurons plus the softmax activation function. If you don't want to bother too much about it, you just load it, and that's it. You create this sequential object, and you just plug all these layers using that model again. It's very simple and straightforward, as you can see. You have dense 522 activation relu, and input shape, because it's the first layer, has 784. Can you guess why this number has the input shape? Yeah. It's 28 times 28. We're going to linearize the images and provide it as just one big tensor or vector. This is because dense networks are not supposed to handle multi-dimensional arrays, but you can actually do it. Again dense layer as requested with the activation relu and the last 10 layers, so the number of classes with the softmax activation. We specify the categorical cross-entropy again and the SGD optimizer. Please note that this time, differently from previous examples, when we provided the optimizer parameter, we provided an actual Python object, normal strings. To be honest, the code is exactly the same, because when we provide SGD object here without any additional parameter, we're actually setting a new object with the default parameter values, which is exactly the same we get when we provide the string SGD. This is just to show you that you can plug not only strings, but real objects. If you're curious enough to understand how these string parameters are indeed implemented in Keras, take a look at the code and I will tell you it's a matter of globals in the namespace. You will find in the loss package an object which is called exactly in this way, categorical underscore cross-entropy. When you provide parameters using strings, Keras looks for that string in the global namespace and returns the object. That's the trick. This is the case for reload, softmax, blah, blah, blah. You can indeed provide real Python object or real Python function or strings if it comes to handy. Moreover, we're also adding this additional parameter to the compile method, which is the metrics. In this case, we're going to record the accuracy during the learning phase. Keras, when you run it, Keras automatically stores the accuracy. I was looking that. We create the model here. Let me just run it. We create the model here. We get the data, the Amnist data. Of course, the shape is the one we had previously. We have to do a little bit of reshape because the input shape of our network is going to be 788. We're going to reshape the Numpy arrays. This is pure Numpy operations. Finally, we're going to apply this to two categorical functions provided by Keras Utils to translate the labels in one hot encoding. This is because maybe you didn't get it. We want to have the one hot encoding for the labels because the last layer has 10 neurons in output. This last layer is the one that will be used to calculate the error function on the labels you have. The labels must have 10 dimensions. That's why you want to apply one hot encoding. You see what I mean? Fine. All right. We did it. We just train and test split. We have this one. This is just one images. Of course, this was different because it's been randomly split. As we see, we have this one hot encoding. For this X-train, the label of Y-train of zero is going to be nine. All zero apart from the last one. We take another one from the validation set. This is one. This is the label. This is because this one, as we expected. We train the model here. We just run it for two epochs. We do not expect very big results just to make it running. This is how it's actually working. We have this loss. We have this. Take a look here. We also have the accuracy for training data and for validation data because we recorded that metric in the compile method. We have loss accuracy, validation loss, and validation accuracy. Something which is new now for you maybe is that we recorded the output of the fit method. We call it network history. That's because by default, recalling you the callbacks of Keras, by default, the model.fit method returns you an history callback object. This callback object embeds all the history of the training process for you to take a look after that. If we want to plot the network performance trend, we have this network history object. As you can see, the network history object at this attribute, which is called history, is going to be a dictionary having keys for names corresponding to those returned during the learning phase. We're going to have a loss and a val loss. We're going to have an accuracy, ACC, and a validation accuracy here. Our intention here is to plot the differences of training and validation accuracy. The training and validation loss and the training and validation accuracy. Of course, these are going to be just two epochs. As you can see, so 01, 01 again. This is not very meaningful. It's just to show you how you can do it. After two epochs, we get just more or less 88% of validation accuracy. If you increase the number of epochs, you will get definitely better results. In this case, we've added a new parameter for the stochastic ring descent. We fit it again here. Let me skip it. It's not very important. Let me introduce another important layer, which is the dropout layer. The dropout layer is a very simple one. The idea of the dropout layer is to be used during the training phase to avoid overfitting. The idea is that when you put the dropout layer in the middle of your network, you specify a drop rate. By that drop rate, some connections from the previous layer to the next layer, neurons, will be dropped randomly. That's the idea of the dropout layer, no more than that. Something you may keep in mind, the Keras API expects you to provide the dropout layer. Some frameworks expect you to provide the rate, the retain probability, not the dropping probability. Keras wants the dropping rate. For instance, if you specify a dropout layer of 0.2, this means that you have 20% of probability to drop the connection. Dropping the connection means that you basically put a zero for that neuron from the previous layer to the next one. The intuition is that when you do it randomly, only during the learning phase, of course, you're doing it because you want to have some variability in your data. You're randomly dropping, muting some connections between neurons to have variability in your data to avoid overfitting. Just to show you this, if you import the dropout layer from the Keras layer and you take a look at the code of the dropout, as I told you, the Keras code is Python code. You can actually take a look at it. Let me see if I can make it bigger. No, this is not. Where is it? I made a mess. I was expected. Sorry. Where is it? No, too many things opened. That's why I do too much stuff. All right, I made a mess. I'm sorry about that. I had good intentions, I promise. Let me go back to it. Sorry. Yeah, yeah, that's because. Okay, so if you take a look at it, this is the implementation of the dropout class in Keras code. When you see it during the call method, this is the interesting part. So it returns the returning of this K.intrainFace function. So if you take a look at it, of this K.intrainFace, all right, the meaning of this function is to select X if the network is actually in the training phase, is to select Alt otherwise, so it's in inference phase. So again, back to the dropout, this means that if the network is during the training phase, this layer returns the dropped inputs. Otherwise, it returns the inputs as it is. So you don't have to worry about the fact that the dropout is only applied during the learning phase because it must be only applied, of course, only during the learning phase because it's supposed to avoid overfitting. All right, this is some internals I want to show you. So apart from this, these exercises just ask you to change a bit of the code, you actually have it down there, to change the previous configuration using the dropout layer. And again, see the difference in performances. And these are the difference. So this starts to have more sense on the data you have, especially, but the fact that now the accuracy reaches the 90%, even if you have only three epochs. Of course, if you continue training at some point, you will start overfitting. And so we can try plugging some early stopping into the network to help to skip it. Movi got something more interesting, something you can actually do with the Keras layers, sequential objects. I don't know how much time do you have? Half an hour? Five minutes. Kidding, right? It's not five minutes. It's six minutes. Sorry. I usually have lots of stuff to say. I'm sorry. Anyway, okay, let me just say this. Something you can do in the, using the Keras object is to take a look at it, and it's very handy to see what are the layers, especially when you have a big network, what are the layers in it. So you can actually iterate using the layers attribute inside the model, so you can take a look at it and take a look at the different parameters of the layers. As you can see, there's a lot of them, even with this quite shallow network. Something very interesting is to extract the hidden layer representation of the given network. This means that sometimes it is very useful to see how the network is actually interpreting and seeing the data inside the network. So it's just a way to open the black box of the deep learning. There are many ways to do it. One simple and most intuitive way is to create a network, train the network, create another similar network, and that network is going to be exactly the same network up to the last layer you want to take a look at. So imagine that we want to see how the data changes after the first dropout. So remember, we have a dense dropout, dense dropout, and other dense. So we have model already pre-trained. After the training, we have a set of weights. So we want to see what's the internal presentation of this layer. So one straightforward way to do it is to create a truncated version of this network, initialize the weights for these layers by setting each layer's weights to the same weights of the target model. We compile the truncated model, and then we just run it in inference mode. So we call it predict, not fit in this case. This is one way to do it. This is, of course, the Python way to do it. So you're actually dealing with Python objects here. Another less intuitive but definitely more effective way to do it is to leverage on the underground graph of tensors you have. And so letting tensor flow or the back end you're using to do the actual job, so you can actually manage to implement this function. There's this implementation at the very end of the network, of the notebook. So you implement this function, which is called get-activation. You pass it the model. So the target model you want, the target layer, and the actual data you want to take a look at. And we're actually using the k function here. So we are creating a function that has the input tensor of the first layer as input, and the target layer tensor as the output tensor. This k function is dealing with tensors. So we're passing the tensor of the input layer as input to this function, and the output of this function will be recorded to the output tensor of the target layer. So we call this function on the data, and that's it. And so since these tensors are automatically connected inside the network, when you call the activation function, you pass the data through these tensors, all these tensors are connected. So they're actually calculating the activations of each tensor up to the target layer we want. This is less intuitive, I know, but it's definitely more effective, because we're actually dealing with tensor flow, Othiano, in the very end. Why would you do that? Because you want to take a look at how the network sees the data after the learning phase, and typically you end up in doing something like this. So imagine that we have already processed it, you don't have to imagine because it is what you're seeing, already processed the MNIST data. We want to get that internal representation of features, so 512 features from the 784 we had initially, and we wanted to plot this data in two dimensions. So we applied some manifold learning on it. We call it the TSNE in this piece of code, so we're actually calling it the TSNE from scikit-learn, and so we're transforming our data using these hidden features. In particular, we're just getting the first thousand samples, not all of them. And after that, we want to take a look at how these data are scrambled in the space, and with each color corresponding to the different classes we have, maybe we can see it interactively, using the bokeh on library. So let me reduce a bit the dimensions so we can see the labels, no, maybe not. So this is quite expected in some way, because for instance, zeros are all here, here you have I think all the 7s, but inside them you still have some 9s, which could make sense. You're dealing with handwritten digits, so maybe some 9s, maybe written on some 7, maybe. There's a lot of mess here, and these are the most difficult classes for the network. The black ones are the 6, and the 6 may be messed with the 5, some not really, maybe with these ones, with the 4, well, this is maybe strange. There's some mess here, so you have the tree with some 2, but yeah, this is just to take a look at what's happening inside the network, all right? This is quite handy and useful to do, and this is typically something you want to do when you want to take a look inside and what's happening during the learning, or what happened during the learning after the training phase. So unfortunately there's a lot of stuff I want to show you, but we don't have the time to do that. If you have time and you want to, take a look at the materials. Everything is on GitHub. Please give me feedback if you have it, PR if you have it, if you spot some errors, of course there will be, I will definitely be more than happy to do that. So I'm going to conclude in case you have questions. Yes, please. Yep. Okay, so the question was how the last method to visualize the id and representation. How that method works with non-sequential networks. What do you mean by no sequential networks? Yeah. Yeah. So basically you cannot have not connected graphs in Kera. Otherwise you end up with errors. That's particularly interesting because if you take a look at the implementation and how Kera detects unconnected graphs, it's bottom up approach. So it starts from the last layer and goes up to the input layer and it traverses the graphs. All the tensors it finds during the path, they are all connected to the same graphs. So at the end of the day, you always have connected tensors. You cannot have unconnected ones. And so the sequential object. The sequential object is just a Python wrapper object. At the end of the day, you always have tensor objects connected one after the other. You can indeed have tensors connected to multiple other tensors. You can have it. You can have multiple, and this is the content of this notebook, number 10, the multi-modal network. Sorry, it's very tiny. With the multi-modal networks, you can actually have multiple input networks, so accept network accepting more than one input tensor, and multiple output network, so network having a multi-objective problem, so multiple losses to optimize during the training. But at the end, you always have a connected graph, so you can always reach the target tensor you're interested in from one other tensor. There's no way some tensor is not connected to any other, I think. Yeah. Yes, but you can... Yeah. So you said network topologies may be much more complicated than the one we worked on, but there should always be a way to reach tensor inside the graph. Otherwise, it's unconnected, so you can always do that. Yes, please. So the question was about requiring your network and how you structured the data. So the difference with requiring your network is that you provide data in sequence, and so that layer expects you to provide a sequence of tensors in input, and you specify the length of the sequence. You may specify the length of the input sequence and the length of the output sequence. So what's the number of sequences you want to take into account in input, and what's the dimension of the output prediction? How many tensors you want in output? That's the difference. Yes, please. Sorry. Yeah. So the question was, in my example of the MNIST, the validation accuracy was better than the training accuracy. It shouldn't be the way around. The reason is we actually trained that network for two or three epochs, so it was very few number of epochs to train. So it's totally meaningless. It's just to show that you can watch what you can do in Keras. It's not real performances, of course, but of course, thanks for pointing out. But it's totally meaningless. Don't rely on it. It's off playground. Any other question? Okay, so, again, thank you very, very much for being here. Please let me know if there's any questions or enquiries about the materials. Enjoy with it. Thank you.
 Alright, so in the meantime you get the material.  I will just give you a few words about this tutorial and...  Oh, sorry.  What's the idea I have in mind with this tutorial?  So...  Alright.  You should have a list of notebooks here.  The original title of this tutorial was 10 Steps to Carousel.  So the idea was to introduce the Carousel framework into 10-ish steps.  And these steps are going to be notebooks.  You have a pre-amble notebook here where you can see all the instructions.  In the back, can you read that?  Is it fine for you?  No, probably it's a bit tiny.  Do you want me to...  Better?  Good.  So this is the goal for this tutorial is to introduce the main features of Carousel  and to learn how you can actually implement deep networks using Carousel.  And one thing I should definitely want you to learn and to know is that Carousel is,  at the end of the day, very easy to use but it is not for easy tasks.  You can actually do complicated stuff using Carousel.  So this is the outline for this tutorial more or less.  And these are the topics we're going to go through in the next hour and a half.  So multi-layer, fully connected networks, so to introduce the very basic features.  And then we go through engine layers, features, embeddings, convolution networks, hyper-projects,  hyper-amplitude tuning, how you implement custom layers in Carousel, and how you actually  use deep neural networks, in particular convolution networks and residual networks, transfer  learning and fine tuning, and if we have time, we can refine your network and auto-encoder  and in the end, multi-modal networks.  So there's a lot of stuff, as you can see.  All the materials are online, you can actually play with it.  All the code has been thought to be run on your notebooks.  So I actually simplified some data examples.  So you should be able to run it even if you don't have GPUs on your machines.  But of course, if you have it, it's definitely better.  Just run faster.  So I'll try to go fast on some topics.  But at any time, if you have questions, please feel free to interrupt me.  There's no problem at all.  You will find the requirements.  Basically it's going to be Python 3.  3.4 should work as well.  And also Python 2, I don't really know.  Just a quick note about setting everything up.  A friend of yours, some people from the audience, sent me that actually it may be a bit cumbersome  to set up the Keras folder on Windows machine.  If you try to Google it, so what I was saying here, the.keras folder where the actual Keras  configuration file should stay, even on Windows machines, should be played.  So Google it a few minutes ago, it ended up to be that the home folder is what you get  in return from the Python function here.  So typically something like seed, semicolon, double, backslash, users, double, backslash,  and your username.  So these is the place where you should create your.keras folder.  I know if you experience the same problem, but this should fix the problem.  So when I say home folder here, I assume even on Windows machine that is the folder.  And so you should have for this tutorial, for this series of notebooks, these configuration  files.  I will tell you later on the meanings of these configurations, but they're quite straightforward.  So, I assume you already tried and verified everything.  That's nice.  Good shot.  After that, so let's start with the first notebook, if you don't mind, which is going  to be number one, multi-layer fully connected network.  The main goal of this notebook is to provide you an example on how you can actually deal  with our.keras and how you can build networks using.keras.  So I freely borrow these payoff sentence saying that Keras is from the Django project, saying  that the Keras is deep learning library for perfectionists with deadlines.  And these means that, as in Django does, dealing with Keras objects and creating networks  with Keras is very easy to go.  And in fact, one of the main important features of Keras is summarizing these sentences from  the official documentation, when they say that being able to go from one idea to results  with the least possible delay is the key to doing good research.  In a few words, Keras is a high-level neural network API written in Python.  It's pure Python, so you can actually take a look at the code and even modify it, which  is capable of running on top of different deep learning frameworks, which Keras calls  backends.  So far, the officially supported backends are TensorFlow, C and TK, which is the deep  learning library from Microsoft.  And these has been the latest addition.  I think it's been added from three or four versions.  And the ANO.  So historically, Keras supported the ANO TensorFlow, now C and TK.  And just for you to know, there's a sort of unofficial support of MXNet as well in the  back end.  I'm saying it's unofficial.  Well, it's not in the master branch of the Keras code.  That's because for some reason, the developer of the MXNet back end supported, are now supporting  the Keras 1 API.  But now we're using Keras 2 API.  So there's been an API change some time ago.  And for some reason, MXNet is still stuck at the Keras 1 API.  So that's why the MXNet is not yet officially supported.  But if you use Keras 1 version API, you can actually also have MXNet as a back end.  Having multiple backends is a feature that you will appreciate very soon in the next  examples.  So starting with this very simple example, let me remove just to get some space.  It's a Kaggle challenge data.  I just read very briefly the problem.  So the Hutter group is one of the worst biggest e-commerce company.  And the consistent analysis of the performance of product is crucial in this company.  And while that do the diverse global infrastructure, many identical products get classified differently.  From this competition, we have provided a dataset with 93 features for more than 200,000 products.  Of course, for the sake of these projects, I reduced drastically the number of samples.  In our data, each row corresponds to a single product.  And so we have a total of 93 numerical features, which represents the count of different events.  So if you take the data, it should be placed in your data Kaggle auto group folder.  For the rest of these examples, we're going to play a very simple model, which is logistic  regression.  So for those of you who doesn't know, the logistic regression is a very simple supervised learning  algorithm characterized by the usage of the logistic function, also known as the sigmoid  function, which has the shape, so the S shape, that's the sigmoid name.  So after preparing the data and getting the data, so in a few words, what we do here is  just loading the data from the CSV file.  There's actually some pandas code running in the back end.  And then we preprocess the data so that we just scale the features using standard scaler  in scikit-learn.  And then we preprocess the labels to have one hot encoding of the labels.  Are these terms clear to everybody?  So I can go safely with that?  Okay, fine.  Thank you very much.  So taking a look at the final results we have, we have nine classes and 93 features as we  expected.  So these are the labels.  Of course, these are anonymized in some way because they're Kaggle data.  And just to give you an example of what is the result of one hot encoding, this is something  like we just have, for each sample, just one element corresponding to the activated bit,  the so-called hot bit.  And that's why this sort of encoding of the labels is called one hot encoding because  just one single label is enabled.  Of course, this is a single class problem.  It's not multi class.  Every single sample is just one class in our data.  So if you want to implement it in pure theano code, I will go very fast about that.  But if you want to implement a logistic function using theano, these are the code you have  to do.  So you basically have to declare these objects.  So the tianomatrix and the shared variables here, the shared are those that are actually  modified during the training, that's the difference.  So you have to implement manually everything basically.  So it's just the probability target, the cross-entropy loss function we want to use.  But the only thing, which is not only, of course, but one something which is provided  from theano is the calculation of the gradient.  So theano provides out of the box the calculation of the gradient.  This is one reason why you don't want to really implement this code in pure Python code.  So you want to rely on these libraries.  So finally, you have to create this tianofunction.  So that connects every single tensor object you're actually creating.  So you provide the input, you provide the output there.  And this is the rule for the updates of the shared variables here.  So of course, these are the weights and the bias, simple model here.  So we compile all the code.  In case you don't know, this is based on symbolic execution.  So you have to compile it and then you run it.  So everything you set up previously has not been executed yet until you compile it and  you run it.  Then you create a new function for the prediction.  Using all the tensors already provided.  So this is for the training.  This is for the prediction and the rest of the code is just repeating the training process  for the number of epochs.  So we are actually implementing the training process manually.  And this is the way theano does this very simple problem.  As you can see, it's very low level, right?  So you have to implement it at every each step.  If you want to use TensorFlow, as you will immediately understand, the level of obstruction  is exactly the same of theano.  So again, you have to specify placeholder here.  So let me just execute it because I just want to show you something.  So you have to create the placeholder here.  And the placeholder are going to be the variables that are going to be updated during the learning  into the graph.  So as I told you, this is symbolic execution.  So if you print X, you just got a tensor object with no shape yet because you don't really  have passed the data, fed the data to the graph.  And the only thing you know is that you have 93 dimensions, so 93 features.  And why should will be a tensor having a total number of classes at the mention?  That's a second dimension.  TensorFlow ships with these tool, which is very useful, which is called TensorFlow.  I'm going to execute it just because I want to show you.  This is very nice.  So this is a sort of a little bit of engineering of the TensorFlow code.  So we're going to again create all the variables here.  Then TensorFlow provides a little bit more abstraction with respect to the Tiano having  this softmax function already implemented.  Then the matmul operation is going to be the equivalent of the dot operation in numpy  if you come from that backend, the background.  And then we are actually adding some additional information that will be used in the tensorboard  tool.  So we're going to record some histograms during the learning and some reference to the  scalar objects we have here.  Again, this is very intuitive because we're actually defining different scopes.  So we are defining the scope of the model here, the scope of the cost function, the scope  of the train process.  And so they're going to live in different scopes, so you can actually better manage  them.  And finally, the accuracy scope, which is going to be the metric scope we want to use  to evaluate our model.  So then TensorFlow provides you this file writer, which is actually something that  stores the results of the graph you have.  So the computational graph you have in the backend.  And after that, you just run it defining TensorFlow session.  So everything living in this TensorFlow session can be interpreted and then run during the  learning process.  But again, what you have to do is to implement manually all the learning process by actually  writing code and providing the data for each of the tensor you want to change during the  learning process.  So oops, that's something went wrong, of course.  That's the demo approach.  Oh, I know why.  Oh, let me just, sorry, let me just, I just want to clear the memory of the GPU.  I'm actually running on some GPUs in a remit backend.  So let me run again.  Sorry about that.  OK.  All right.  So I have to load the data.  All right.  Come on.  That's a cold start.  Very cold one.  All right.  So let me skip the other part.  So let's go to the TensorFlow.  All right.  OK.  Now this is going to work hopefully.  Please.  All right.  So the app box, we have just run it for 25 app box, I think.  The number of the box is, I can remember.  So we have 25 app box, and this is the value of the accuracy during the different app box,  and this is how the plot, we have just added it, how we plot different costs, how the cost  changes during the different app box.  So this is the trend.  All right.  So I just want to show you this TensorFlow tool, which is going to be something we're  going to see also integrated in Keras.  That's why I'm showing it now.  So I'm running.  So this TensorFlow.tensorboard command runs a server typically on the port 606.006.  And what this does is to read the log file that has been written during the execution  in particular, the file stored in this directory.  All right.  So hopefully we're going to see something.  Okay.  So we have these colors here.  So we have the plots for the accuracy automatically provided, the plots for the cost function during  the training, and the plots, and all the different plots for the mean weights and the mean buyers.  All right.  This is very easy to use and very useful when you face real problems.  Another thing it may be useful is to take a look at the graph.  So this is actually the model that has been implemented inside TensorFlow.  So this is how the graph of tensors has been translated by TensorFlow.  So you can actually take a look at all the different nodes here.  So each of these corresponds to the different scopes we specified.  And finally, these histograms we built to show the difference of the different trends  of the cost histogram and of the model histogram.  Something wrong maybe because I just run it multiple times.  I'm safe holder.  So that's why you see this so strange.  Anyway.  All right.  So let me stop this.  All right.  Okay.  So let's see how we implement a logistic regression using Keras.  Finally, as you know, the logistic regression may be implemented as a neural network, just  one layer, which has the sigmoid activation function.  And so it is.  So you have all the data again.  And we are going to use these sequential objects.  We're introducing this new sequential object in Keras.  The sequential object gives you the idea that you want to build a series of layers stacked  one on top of the other.  And so you actually instantiate this sequential object and you simply add layers to these  objects.  So in this case, we had just had a dense layer having a number of units in output corresponding  to a number of clusters.  So we specify the input shape, so the number of features we have.  And the activation is going to be the sigmoid function as we expect because we implemented  a logistic regression.  Finally we add the decision function, which is the softmax function as we did in TensorFlow.  And we in the end compile the model again.  So we compile specifying which is the optimizer.  In this case, we specify the stochastic-grade descent.  We actually did the same using TensorFlow.  Let me just go back a bit.  So when we are in the train scope here, we specify that the optimizer was the grade  descent optimizer with the minimization problem.  So it's stochastic-grade descent with these learning rates.  And we are doing the same here.  So we are implementing the same using the categorical cross-entropy and the stochastic-grade  descent optimizer.  After that, we just call this method in Keras object, which is called fit.  And we provide the data, the training data, and the training labels.  And that's it.  So you don't have to implement it manually.  You don't have to implement the training process manually.  You just call the fit method as you do in scikit-learn, for instance.  And that's it.  And Keras provides you automatically a series of our default parameters.  We're going to change during the next steps of these exercises.  But in the very beginning, you just create the model, you compile it, and you fit it.  These are the three steps you have to do, the only three steps you have to do to make  our Keras model up and running.  And so as you can see, if you run the fit method, you're going to have a series of  10 epochs by default.  And Keras provides you these progress bar during the training, telling you the time  it takes, the values of the loss during the training, and how does it change.  All right?  This is the very beginning.  So now let's have a look at it in a more interesting way, let's say, so in more details.  So the core of the destruction of Keras is a model, of course, and the main type of model  is going to be sequential objects.  The sequential object provides you this method, which is called add, and so you have different  layers.  In this example, we just had one layer, which is the dense layer, and then the activation  layer.  So the dense layer, this is the dense object, has this parameter.  Most of them are default values, and in case you don't know, keep the default values.  Keras typically has been designed to have the best default values you can get for the  different objects.  And this is particularly useful when you run it for the first time, especially for optimizers,  when you have to set up the learning rate and depending on the optimizer, different  other parameters.  The most important things you have to learn and to understand very soon is that the first  and the only required parameter for this layer, and this is typically for all the layers,  is the first one, which is units.  As you can see, it's the only one that has not default values, of course, and this is  where you put your insight and your knowledge, and the unit's value is going to be an integer  value greater than zero, of course, and this corresponds to the number of neurons that  you will get in output from the layer.  Keep this in mind, it's going to be the output number in neurons, not the input, right?  You have to specify the number of input dimension only in the first layer.  The next layers will get, as input, the output of the previous layers automatically, you  don't have to specify it.  So the units is going to be the number of neurons you want in the output.  So back to the example, since we just have one layer in this example, the number of units  we want to have in this network is going to be number of classes because it's the output  we want to predict.  It's just one layer, right?  So it's just number of classes, so this case is going to be nine.  All right, and as you can actually see, since this is the very first layer of a network,  we also specify the input shape, and this is mandatory for the very first layer of the  network.  You always have to specify what is the input shape.  Speaking of which, the input and the output shapes of the layers are going to be this  one.  In input shape, you get a non-dimensional tensor object with these series of shape.  Remember that the very first dimension of the input tensors are going to be the best  size.  In other words, the number of samples you're going to provide to the network during the  training, right?  And this parameter is specified when you start the training process, so you call the fit  method.  You don't have to specify this dimension in the input shape parameter.  All right, so let me just go back again to this example.  As you can see, the input shape in this case is going to be dims, so the number of features  we have.  All right, so this is 93 in this case.  We don't specify the best size.  The best size is indeed automatically specified in the fit method.  We're going to use the default parameter here, but we can actually change it.  So remember, the input shape is mandatory for the very first layer and has to not include  and must not include, sorry, the best size dimension, which is going to be the very first  dimension of the tensor you have.  And as output, what you get in output form a tensor is the best size again, so the same  number of samples, of course, but as they mentioned, the units you specified as the first parameter  of the layer, so the output neurons.  All right, is everything clear?  Questions?  All right.  So before moving on, some very few notes about the star initializers parameter.  So typically in the dense layer, but in most of the layers, you have this kernel underscore  initializer and bias initializer parameters.  This is just for you to know that in literature, there are lots of methods to initialize the  weights of neural networks, and this is particularly useful, and different methods to initialize  for the initialization of the weights can drastically change the performance you're going  to have.  So I just proposed you to take a look, of course, at the keras.initializers to see what are  the functions supported by keras.  There's lots of them.  And read these very interesting article.  This is not in, this article does not include all the possible initializer supported by  keras, but this is very interesting, and I give you some recommendation paper to read.  This is a very interesting topic, and as you can see, the default initialization method  for different weights by default will change according to the different layers keras provides.  So if you take a look at the default parameter for the convolutional layer, it should be  different.  I cannot remember for sure, but it should be different, because for convolutional layers,  you have different initialization technique by default.  Other core layers provided by keras, you have flatten layer, reshape layer, permute  layer, these are going to be so-called operational layer.  So when you have to, when you have, they're not learning anything here.  They're just making operations on the tensors during the flow of the network.  The lambda layer, you can actually specify the custom Python function to apply to the  tensor.  All right?  Please keep in mind that what a keras layer gets an import is always a tensor object, which  is going to be physically and concretely corresponding to a tensor flow tensor object, or a theano  tensor object, or in this case, nowadays also we have a CnTK tensor object, right?  So to give you just the idea, when you run the fit and the training process using keras,  keras indeed is not performing any real operation on the GPU, all right?  So the keras code relies on the Python level, and all the real computation on the GPU and  stuff is demanded completely to the back end.  The main advantage you get from this approach is that the same code you write, exactly the  same code, is supposed and guaranteed to work on all the back ends supported by keras,  without changing any single line of code.  Keras does all the job for you, even if there are some very slight and tiny differences  between the, for instance, the handling of the shapes of tensors between the tiano and  tensor flow.  Keras does all the job for you, you don't have to change anything.  That's power of keras indeed.  Another important layer is the activation regularization, so you can actually apply regularizers on the  activation functions to, so for instance, L1 or L2 regularization on the activation  function of your network.  This picture, kindly provided by my friend Ian, shows all the, probably it's a bit tiny,  but should give you another view of all the things, all the kind of networks supported  by keras.  Keras provides support for feed forward network, so the classical network, you provide input  on the left and you get output on the right, so the feed forward network in these sets  of networks, also the convolution networks fill in.  You have recurrent neural network, support for LSTM or gated recurrent unit network,  and also for unsupervised learning, you can actually build autoencoders, variation autoencoders  and sequence networks using keras code.  Finally, keras also provides support for different optimizers, not just the stochastic  ring descent, of course, there are many of them, you can have aider grad, aider delta,  lm sprop, lots of different optimizer.  You can actually take a look at the keras.optimizer package to see all of those optimizers supported  by keras.  If you want to data science in a bit of this example, we're going to play a bit with the  hyper parameter of the network and the parameter of the feed method.  I assume you all know what overfitting is, or underfitting is, so something we can do.  One method I'm going to show you, because we're going to use it very heavily, a lot,  is this summary method for the model object.  When you call the summary, you actually have a look on what are the layers, what are the  outward shape, how many parameters you have, and in the end, how many total parameters  you have, and how many of them are trainable parameters.  If these does not make any sense to you, it will make very soon.  So far, the total number of trainable parameters are going to be the same.  The design thing about this example, we're going to use some sort of train test split  on the data.  We have not only training data, but also validation data.  We want to fed into the training process.  We're going to apply a technique which is called early stopping.  The early stopping technique is something that is typically used when you have big networks  and lots of data.  The intuition is that in these case, we're going to monitor this metric, which is called  the validation loss.  When the validation loss does not change after two epochs and does not change significantly,  so you can actually specify the precision you want, the training process is automatically  stopped.  This is the early stopping, because, differently from other machine learning methods, unless  you specify these kind of tricks, when you run a fit of a neural network, you have to  specify for how many steps you want the learning to go on, right?  So the number of epochs, so-called.  These other objects is the model check point.  During each epoch, at each step, automatically, this function, this object, records the value  of the validation loss again and saves the weights, so the stages of the network at that  stage.  And automatically, this has been set up to save only the best model.  So in the end, you get the best weight configuration, so leading to the best validation loss value.  You can plug both of them using this parameter in the fit method here, which is the call  backs.  So Keras provides this way to plug additional callback objects.  They call callbacks because they're automatically called by the framework before the next batch  or before the next epoch is going to be processed during the training.  You don't have to do anything, you just pass the callbacks.  We're going to specify a different number of epochs.  We're going to specify what are the validation data in this case.  We previously provided just training data, and these are the two mandatory parameters,  the very first two, validation data again, and we specify the size of the batch.  And we also specify that we want some verbosity, but this is the default one.  All right.  All right, so we run it for 50 epochs, and these goes on and on.  Additionally, we also have a validation loss that, since we have validation data in this  case, we're going to also have the computation of the validation loss.  So during each step, we have a log of how the loss changes, the training loss changes,  and the validation loss changes.  Okay, moving on.  Okay, very briefly, moving on just a bit.  So we want to create a multi-layer fully connected network.  When we say fully connected network, we're going to expect to have all dense layers.  The very first example of multi-layer fully connected network is the multi-layer perceptron.  The multi-layer perceptron is characterized by having one input layer, one hidden layer,  one output layer.  All right.  So technically, when you have this sort of network, so one, with just one hidden layer  in the middle, you're calling it multi-layer perceptron.  When you have at least more than one hidden layer, you're allowed to call it deep network.  All right?  So when you have at least two layers in the middle, it's going to be a deep network.  So in this case, we're going to have a multi-layer perceptron because we have one input layer  and this is just the decision layer.  So it's going to be very shallow network yet.  And as you may assume, the way you plug additional layer, so you build a multi-layer network  in Keras is just by calling the add method to the special object multiple times with  different layers.  All right?  So again, make sure that you specify the input shape in the very first layer.  In the second layer, don't have to because it's automatically induced by the previous  layer.  All right?  So again, we compile the model.  We print the summary, so we have dense number two, dense number three here, and the final  activation, which is the softmax for the decision function.  Of course, since we have 100 neurons here in output and nine in the end, the total number  of parameters has increased, as you may expect.  Then we run the fit model again and the validation loss changes drastically considering the same  data.  Yes.  So this is 1.6 taken just randomly, and this is 0.7.  So this is going to be a better model with respect to the previous one with our data.  So if you want to play a bit, there's an exercise for you.  The idea is to play a bit with it, adding as much layers as you want, better to add  a couple of, and run it in your machine to be sure that everything's up and running,  and to see that it's very easy to run using Keras.  All right?  The next steps will be taking a brief look at what is called the Keras backend.  I just want to show you the Keras backend because this is how actually Keras allows  you to have multiple backends running for your deep learning code.  Okay.  So to give you some hints, the only way to play with it is just to add additional dense  layer.  How many neurons you want to add?  How many layers you want to add?  Nobody knows.  You just play with it and you see what, if that makes any difference to the final results.  That's the takeaway typically for deep learning.  At the end of the day, you don't really know what you're doing.  So let's have a look at this Keras backend.  The Keras backend is the module which is provided by Keras that integrates to real backends.  If you take a look at this dot backend package, right, in Keras, so it's going to be the  Keras dot backend package.  If you take a look at the code, you will see that in that package, Python package, you  will find a file which is called a module, Python module, which is called TensorFlow  underscore backend dot pi, Tiano underscore backend dot pi, and now Cntk underscore backend  dot pi.  So this backend is a sort of wrapper module that calls the real backend depending on the  configuration file or the environment variable you have set up in your settings before running  your code.  So we have actually set up our TensorFlow backend in the configuration file, as you  may have assumed, so I will just want to show you.  So if I'm going to cat the Keras dot JSON file, all right, among these configurations,  we have the backend and we are specifying TensorFlow.  If we just switch these directive to Tiano, you're done.  You just rerun your code, all right, and Keras automatically switches everything on  Tiano backend.  All right.  How these backend works, so as I said, the Keras dot backend is the wrapper for the other  module.  All these modules, so basically the Keras backend provides the interface and all the  functions provided in this Keras backend module are assumed to have a counterpart in each  of the backend supported.  So when you're actually calling the, okay, let's have a look at it.  So maybe it's easier to do.  So we're actually reusing the Keras backend API to reimplement again the logistic regression  example, but in this case, we're actually, we're still using Keras code, all right.  So let's have a look.  So here we're creating placeholder objects using the K module, which is the Keras backend,  we just call it K, it's just a sort of naming convention here, all right.  So the K is going to be the Keras backend module and we're going to create a placeholder  object.  The syntax is very TensorFlow-ish, all right, it's very similar to TensorFlow, but what  actually happens in the backend is not TensorFlow always, depending on the, it really depends  on the backend you're using.  So if I'm using TensorFlow as a backend, this instruction is going to create a TensorFlow  placeholder object.  If I'm going to use theano, that is going to be a creation of theano tensor object, all  right.  This is the backend API.  So we're creating a variable object again.  So the level of abstraction has changed drastically, has moved again to the same level of abstraction  you have using TensorFlow or theano packages, but with a big difference that this code is  supposed to run seemingly using all the backends you want, those supported, of course.  So without changing any single line of code, just the configuration fire, all right.  So this is going to create a TensorFlow, if you're using TensorFlow, yes.  All right.  Okay, so I repeat the question.  So the question was, why would you actually have support for multiple backends, right?  Because of many reasons need, you can, because the different backends have different performances,  all right, and so you want to leverage on the benefits of those switching the backends,  because sometimes ago the memory management of TensorFlow was really rubbish, so you typically  wanted to switch to theano to have a better location memory.  Now it's not the case, all right.  Because sometimes you want to try different backends having to test if everything is working  with the same performances, all right, because of numerical reasons, because of numerical  optimization, you may get, all right, for instance, I just read a sort of benchmark  article providing a sort of overview of the benefits provided by CNTK.  CNTK, among those backends, is the only one using MPI in the backend, so multiple messaging  and passing interface, right, so for multiple processing.  As far as I know, CNTK is built on top of MPI.  So far now TensorFlow supports MPI in the version 1.3, so the latest one.  And considering the different network architectures, at the end of the day, CNTK has turned to  be the most efficient for the recurrent neural networks, which are going to be the very heavy  network to train, the one that takes more time to train, all right.  So for instance, for convolutional networks, TensorFlow and CNTK are sort of the same  performances according to that benchmark.  But CNTK won for on-require neural network, for instance.  So maybe you want to switch to CNTK using, without changing any single line of code,  that's a benefit.  And you don't have to mess with the very tiny details of the different implementation,  because Keras does it for you.  All the three backends have totally different APIs and totally different syntax for creating  tensors.  Keras does the job for you to translate all of them, and the way it does is using this  Keras backend.  So when you're actually running, that's it.  This is what I was looking for.  You can actually run this function, the K dot function.  This is calling matmool for TensorFlow and dot function on the ANO.  All right.  I don't know the equivalent in CNTK, but TensorFlow decided to call the dot product matmool, matrix  multiplication for some reason.  Okay.  When you call K dot, it's going to be mapped to the corresponding function in the corresponding  backend automatically.  That's it.  So now the question could be, why would you do that?  Why would you implement it using the Keras backend?  So what's the reason why Keras provides you these APIs to be used?  Because this is the same API you're actually using when you're creating custom layers.  If you create custom layers using the Keras backend API, you're automatically sure that  that implementation is going to work on all the backends provided, because you're using  the Keras backend functions.  You're not using the TensorFlow or the Tiano real functions.  All right.  You see what I mean?  Okay.  So the rest of the notebook is to show you how you basically do the same TensorFlow-ish  stuff using the K backend.  And there's also an exercise for you, which I'm going to skip, which is the implementation  of the linear regression.  I'm going to skip also because you indeed have the solutions in your material.  So if you load it, you're done.  All right.  Okay, so in case you don't have any questions, I would just move forward.  Yes, please.  Yes.  You're actually stucked.  Sorry.  I repeat the question.  I'm actually advertising all the benefits of Keras.  So the question was, what's the downside of it?  I may say that the downside of Keras stands on the downside of the supported frameworks,  supported backends.  I would say three things.  Not all the functions are really supported by Keras backend, because Keras backends,  since it has to provide you that the same code works seemingly on all the backends,  in case there are some very specific functions supported by one specific backend, you're  not going to find it in the Keras backend API.  But a sort of note about this is that if this framework, the one that has this tiny function,  is going to be TensorFlow, you can actually manage to do it, because Keras is going to  be integrated in TensorFlow, and it is better integrated each version by each version.  I will show you something in the end.  That's one problem you have.  And the downside, and not the downside I may say, is that since Keras is not doing any  real computation, you can do with Keras everything you can do with all the frameworks.  But if, for instance, so far in TensorFlow, which is the one I know the best among the  three, there's no official support for dynamic graph computation.  So everything is going to be static.  So this means that when you create the graph, you have to compile the model indeed.  So this means that after you create in the network, you're compiling it.  So you're creating the network beforehand, you compile it, and so the resulting graph  will be put, all the tensors will be put into the memory of the GPU, typically, if you have  it.  And so that's graph, it is very difficult to change it during the training.  You cannot do that.  The PyTorch approach that I believe we will see in the keynote we have in two days, you  will see that that approach is totally different.  That's called dynamic graph creation.  So you create it one step at a time during the execution, and so that's our sort of different  approach to that.  Keras does not do that because PyTorch is not supported yet.  I say yes, I don't know if we'll be supported at some time.  All right.  Yes.  Default backend.  Yes.  So the default backend was the question, and the answer is the ANO is default one.  You can actually go to the configuration file, or so as I showed you, you just, for instance,  let me do this.  I can show you in a terminal, maybe it's better, the one to risk to waste the notebooks.  But indeed, if you, you have two ways.  All right.  So if I print a cut, keras.json file, you will see that the backend is TensorFlow because  I set it.  If I do change this directive, all right, if I specify the ANO here, or even CnTK, all  right, now the backend is the ANO.  If I actually go in Python and import keras, this is using the ANO backend.  You see?  If I do this just to play a bit, CnTK should be installed.  This gives me the opportunity to say a few words about that.  Using CnTK backend.  I did not include that examples using CnTK, mainly for one reason, because CnTK is still  only supported on Windows machines and Linux machines.  There's no support for macOS.  You can actually have CnTK installed using a Docker image.  So I prefer not to include also these into the setup of this environment.  It was not really required.  The main focus was on keras, of course.  But this was the main reason.  And the reason why is it, I don't really know.  It's Microsoft stuff.  You know, in the end, it's always Microsoft.  So let me switch back to TensorFlow.  Another way you can do it is by setting us a keras underscore.  Let me just move a bit so you can reduce it.  You can actually export.  No, let me do this.  Yes, I can do it.  Keras underscore backend environment variable, back end, equals to piano.  Okay.  Let me show this first.  If I do this, sorry.  If I import keras, now it should be TensorFlow again.  All right.  It takes sometimes, typically.  Okay.  If I do, I can do this, keras underscore backend equal piano, Python, minus M, import  keras.  Oh, sorry.  I cannot remember it.  Is this, how's the way to do it?  Minus A, C, thank you so much.  Using the end of backend.  You can actually define this keras backend environment variable and you switch the backend  without changing the configuration file.  This is because the configuration file is just one file.  If you want to have multiple keras instances running, it's not a good idea to change the  configuration files multiple times.  So you just define the environment variable there.  The keras underscore backend and you're done.  All right.  Okay.  Are there any other questions?  Fine.  So, let's move on.  Let's talk about these Amnist dataset.  I believe all of you already know what the Amnist dataset is about.  In scikit-learn word, this is also known as the digits dataset.  So the Amnist dataset is one of the widely used dataset for deep learning experiments  and at some point they decided that it was fair enough to not use it anymore.  But the Amnist database contains images, black and white images of handwritten digits.  And so you want a network in our case to be able to identify the corresponding digits  taking a look at the image.  All right.  So this notebook is just to show you that keras, just like scikit-learn, ships with these  datasets, dataset module, and these datasets, Amnist database is provided.  So you can actually import from keras.datasets, Amnist module, and with the load data, this  method downloads data for you in case you don't have it, and the downloaded data will  be placed in your dot keras folder, and you can actually get the data by loading it automatically.  The rest of the notebook is just for you to play a bit.  So I was asking what type of X-train is, and the reason is, the answer is, of course, it's  an ampere rate.  So once we have loaded, if we type the type of X-train, it's going to be numpn-eray.  All right.  The type of Y-train is going to be numpn-eray again.  So the load data automatically gives you back the data into numpn-erays, which is something  you typically want to have when you have to experiment.  How many observations in train gate do we have?  So again, it's going to be the X-train dot shape of zero, of course, so the number of  samples we're going to have 60,000 samples in the training set, and 10,000 I will tell  you in the test set.  All right.  How many numpn-erays?  Observation for its digits.  So it's going to be train dot shape again, but I will tell you that 60,000, of course.  So the dimension of X-train, so if you take a look at the X-train dot shape, we're going  to have 60,000 of 28 times 28 images.  So we're going to have images of that size.  Since the MNIST images are black and white images, you don't have the channel information,  so it's black and white, so it's just two-dimensional object, not three-dimensional object.  So very quickly, if you want to take a look at these images, you can actually print it.  So I'm going to print the X-train of zero here, and this is going to be a sort of handwritten  five.  Okay.  So we're going to use these data sets to play with the rest of the two notebooks.  Okay.  The first notebook is, again, about fully connected FIFORWAR network.  Let's say the more complicated FIFORWAR network, FIFORWAR network, we're going to take a look  at are going to be the convolution network.  Some recall of the main things we already did, so sequential object and the graph model  of the objects.  We talked about the dense layer, aka fully connected layer.  In this notebook, we're going to introduce a new layer, which is called the dropout layer,  and we applied binary cross-entropy or a categorical cross-entropy to our data using the stochastic  grid descent optimizer.  Here you have the links to have a look at what Keras supports as optimizers and loss  functions.  We now introduce these new activation function, which is called the relu function, so the  rectified linear unit.  This function is going to be very useful in convolutional network.  This function has lots of mathematical properties, and this is the mathematical definition of  the function.  This activation is going to return the maximum value between zero and X, so this means that  all the negative activations will be automatically dropped out.  When you calculate the derivative of this function, it recalls the sigmoid logistic function.  The derivative of this function is called the softplus function.  This function, despite its simplicity, is one of the coolest things invented in the  deep network words many years ago.  The note, keep in mind, this function, as it's heavily used in convolutional neural  network.  The exercise now asks you to build this network.  This sort of syntax is quite standard when you have to specify the architecture of the  network.  In this case, we're going to have a fully connected network, a fully connected layer,  with 512 neurons in output, plus the relu function activation, again the same configuration,  and finally another fully connected layer, FC, with a total of a number of classes neurons  plus the softmax activation function.  If you don't want to bother too much about it, you just load it, and that's it.  You create this sequential object, and you just plug all these layers using that model  again.  It's very simple and straightforward, as you can see.  You have dense 522 activation relu, and input shape, because it's the first layer, has  784.  Can you guess why this number has the input shape?  Yeah.  It's 28 times 28.  We're going to linearize the images and provide it as just one big tensor or vector.  This is because dense networks are not supposed to handle multi-dimensional arrays, but you  can actually do it.  Again dense layer as requested with the activation relu and the last 10 layers, so the number  of classes with the softmax activation.  We specify the categorical cross-entropy again and the SGD optimizer.  Please note that this time, differently from previous examples, when we provided the optimizer  parameter, we provided an actual Python object, normal strings.  To be honest, the code is exactly the same, because when we provide SGD object here without  any additional parameter, we're actually setting a new object with the default parameter values,  which is exactly the same we get when we provide the string SGD.  This is just to show you that you can plug not only strings, but real objects.  If you're curious enough to understand how these string parameters are indeed implemented  in Keras, take a look at the code and I will tell you it's a matter of globals in the namespace.  You will find in the loss package an object which is called exactly in this way, categorical  underscore cross-entropy.  When you provide parameters using strings, Keras looks for that string in the global  namespace and returns the object.  That's the trick.  This is the case for reload, softmax, blah, blah, blah.  You can indeed provide real Python object or real Python function or strings if it comes  to handy.  Moreover, we're also adding this additional parameter to the compile method, which is  the metrics.  In this case, we're going to record the accuracy during the learning phase.  Keras, when you run it, Keras automatically stores the accuracy.  I was looking that.  We create the model here.  Let me just run it.  We create the model here.  We get the data, the Amnist data.  Of course, the shape is the one we had previously.  We have to do a little bit of reshape because the input shape of our network is going to  be 788.  We're going to reshape the Numpy arrays.  This is pure Numpy operations.  Finally, we're going to apply this to two categorical functions provided by Keras Utils  to translate the labels in one hot encoding.  This is because maybe you didn't get it.  We want to have the one hot encoding for the labels because the last layer has 10 neurons  in output.  This last layer is the one that will be used to calculate the error function on the labels  you have.  The labels must have 10 dimensions.  That's why you want to apply one hot encoding.  You see what I mean?  Fine.  All right.  We did it.  We just train and test split.  We have this one.  This is just one images.  Of course, this was different because it's been randomly split.  As we see, we have this one hot encoding.  For this X-train, the label of Y-train of zero is going to be nine.  All zero apart from the last one.  We take another one from the validation set.  This is one.  This is the label.  This is because this one, as we expected.  We train the model here.  We just run it for two epochs.  We do not expect very big results just to make it running.  This is how it's actually working.  We have this loss.  We have this.  Take a look here.  We also have the accuracy for training data and for validation data because we recorded  that metric in the compile method.  We have loss accuracy, validation loss, and validation accuracy.  Something which is new now for you maybe is that we recorded the output of the fit method.  We call it network history.  That's because by default, recalling you the callbacks of Keras, by default, the model.fit  method returns you an history callback object.  This callback object embeds all the history of the training process for you to take a  look after that.  If we want to plot the network performance trend, we have this network history object.  As you can see, the network history object at this attribute, which is called history,  is going to be a dictionary having keys for names corresponding to those returned during  the learning phase.  We're going to have a loss and a val loss.  We're going to have an accuracy, ACC, and a validation accuracy here.  Our intention here is to plot the differences of training and validation accuracy.  The training and validation loss and the training and validation accuracy.  Of course, these are going to be just two epochs.  As you can see, so 01, 01 again.  This is not very meaningful.  It's just to show you how you can do it.  After two epochs, we get just more or less 88% of validation accuracy.  If you increase the number of epochs, you will get definitely better results.  In this case, we've added a new parameter for the stochastic ring descent.  We fit it again here.  Let me skip it.  It's not very important.  Let me introduce another important layer, which is the dropout layer.  The dropout layer is a very simple one.  The idea of the dropout layer is to be used during the training phase to avoid overfitting.  The idea is that when you put the dropout layer in the middle of your network, you specify  a drop rate.  By that drop rate, some connections from the previous layer to the next layer, neurons,  will be dropped randomly.  That's the idea of the dropout layer, no more than that.  Something you may keep in mind, the Keras API expects you to provide the dropout layer.  Some frameworks expect you to provide the rate, the retain probability, not the dropping  probability.  Keras wants the dropping rate.  For instance, if you specify a dropout layer of 0.2, this means that you have 20% of probability  to drop the connection.  Dropping the connection means that you basically put a zero for that neuron from the previous  layer to the next one.  The intuition is that when you do it randomly, only during the learning phase, of course,  you're doing it because you want to have some variability in your data.  You're randomly dropping, muting some connections between neurons to have variability in your  data to avoid overfitting.  Just to show you this, if you import the dropout layer from the Keras layer and you take a look  at the code of the dropout, as I told you, the Keras code is Python code.  You can actually take a look at it.  Let me see if I can make it bigger.  No, this is not.  Where is it?  I made a mess.  I was expected.  Sorry.  Where is it?  No, too many things opened.  That's why I do too much stuff.  All right, I made a mess.  I'm sorry about that.  I had good intentions, I promise.  Let me go back to it.  Sorry.  Yeah, yeah, that's because.  Okay, so if you take a look at it, this is the implementation of the dropout class in  Keras code.  When you see it during the call method, this is the interesting part.  So it returns the returning of this K.intrainFace function.  So if you take a look at it, of this K.intrainFace, all right, the meaning of this function is  to select X if the network is actually in the training phase, is to select Alt otherwise,  so it's in inference phase.  So again, back to the dropout, this means that if the network is during the training  phase, this layer returns the dropped inputs.  Otherwise, it returns the inputs as it is.  So you don't have to worry about the fact that the dropout is only applied during the  learning phase because it must be only applied, of course, only during the learning phase  because it's supposed to avoid overfitting.  All right, this is some internals I want to show you.  So apart from this, these exercises just ask you to change a bit of the code, you actually  have it down there, to change the previous configuration using the dropout layer.  And again, see the difference in performances.  And these are the difference.  So this starts to have more sense on the data you have, especially, but the fact that now  the accuracy reaches the 90%, even if you have only three epochs.  Of course, if you continue training at some point, you will start overfitting.  And so we can try plugging some early stopping into the network to help to skip it.  Movi got something more interesting, something you can actually do with the Keras layers,  sequential objects.  I don't know how much time do you have?  Half an hour?  Five minutes.  Kidding, right?  It's not five minutes.  It's six minutes.  Sorry.  I usually have lots of stuff to say.  I'm sorry.  Anyway, okay, let me just say this.  Something you can do in the, using the Keras object is to take a look at it, and it's very  handy to see what are the layers, especially when you have a big network, what are the  layers in it.  So you can actually iterate using the layers attribute inside the model, so you can take  a look at it and take a look at the different parameters of the layers.  As you can see, there's a lot of them, even with this quite shallow network.  Something very interesting is to extract the hidden layer representation of the given network.  This means that sometimes it is very useful to see how the network is actually interpreting  and seeing the data inside the network.  So it's just a way to open the black box of the deep learning.  There are many ways to do it.  One simple and most intuitive way is to create a network, train the network, create another  similar network, and that network is going to be exactly the same network up to the last  layer you want to take a look at.  So imagine that we want to see how the data changes after the first dropout.  So remember, we have a dense dropout, dense dropout, and other dense.  So we have model already pre-trained.  After the training, we have a set of weights.  So we want to see what's the internal presentation of this layer.  So one straightforward way to do it is to create a truncated version of this network,  initialize the weights for these layers by setting each layer's weights to the same weights  of the target model.  We compile the truncated model, and then we just run it in inference mode.  So we call it predict, not fit in this case.  This is one way to do it.  This is, of course, the Python way to do it.  So you're actually dealing with Python objects here.  Another less intuitive but definitely more effective way to do it is to leverage on the  underground graph of tensors you have.  And so letting tensor flow or the back end you're using to do the actual job, so you  can actually manage to implement this function.  There's this implementation at the very end of the network, of the notebook.  So you implement this function, which is called get-activation.  You pass it the model.  So the target model you want, the target layer, and the actual data you want to take a look  at.  And we're actually using the k function here.  So we are creating a function that has the input tensor of the first layer as input, and  the target layer tensor as the output tensor.  This k function is dealing with tensors.  So we're passing the tensor of the input layer as input to this function, and the output  of this function will be recorded to the output tensor of the target layer.  So we call this function on the data, and that's it.  And so since these tensors are automatically connected inside the network, when you call  the activation function, you pass the data through these tensors, all these tensors are  connected.  So they're actually calculating the activations of each tensor up to the target layer we want.  This is less intuitive, I know, but it's definitely more effective, because we're actually dealing  with tensor flow, Othiano, in the very end.  Why would you do that?  Because you want to take a look at how the network sees the data after the learning phase,  and typically you end up in doing something like this.  So imagine that we have already processed it, you don't have to imagine because it is  what you're seeing, already processed the MNIST data.  We want to get that internal representation of features, so 512 features from the 784 we  had initially, and we wanted to plot this data in two dimensions.  So we applied some manifold learning on it.  We call it the TSNE in this piece of code, so we're actually calling it the TSNE from  scikit-learn, and so we're transforming our data using these hidden features.  In particular, we're just getting the first thousand samples, not all of them.  And after that, we want to take a look at how these data are scrambled in the space,  and with each color corresponding to the different classes we have, maybe we can see it interactively,  using the bokeh on library.  So let me reduce a bit the dimensions so we can see the labels, no, maybe not.  So this is quite expected in some way, because for instance, zeros are all here, here you  have I think all the 7s, but inside them you still have some 9s, which could make sense.  You're dealing with handwritten digits, so maybe some 9s, maybe written on some 7, maybe.  There's a lot of mess here, and these are the most difficult classes for the network.  The black ones are the 6, and the 6 may be messed with the 5, some not really, maybe  with these ones, with the 4, well, this is maybe strange.  There's some mess here, so you have the tree with some 2, but yeah, this is just to take  a look at what's happening inside the network, all right?  This is quite handy and useful to do, and this is typically something you want to do  when you want to take a look inside and what's happening during the learning, or what happened  during the learning after the training phase.  So unfortunately there's a lot of stuff I want to show you, but we don't have the time  to do that.  If you have time and you want to, take a look at the materials.  Everything is on GitHub.  Please give me feedback if you have it, PR if you have it, if you spot some errors, of  course there will be, I will definitely be more than happy to do that.  So I'm going to conclude in case you have questions.  Yes, please.  Yep.  Okay, so the question was how the last method to visualize the id and representation.  How that method works with non-sequential networks.  What do you mean by no sequential networks?  Yeah.  Yeah.  So basically you cannot have not connected graphs in Kera.  Otherwise you end up with errors.  That's particularly interesting because if you take a look at the implementation and  how Kera detects unconnected graphs, it's bottom up approach.  So it starts from the last layer and goes up to the input layer and it traverses the  graphs.  All the tensors it finds during the path, they are all connected to the same graphs.  So at the end of the day, you always have connected tensors.  You cannot have unconnected ones.  And so the sequential object.  The sequential object is just a Python wrapper object.  At the end of the day, you always have tensor objects connected one after the other.  You can indeed have tensors connected to multiple other tensors.  You can have it.  You can have multiple, and this is the content of this notebook, number 10, the multi-modal  network.  Sorry, it's very tiny.  With the multi-modal networks, you can actually have multiple input networks, so accept network  accepting more than one input tensor, and multiple output network, so network having a multi-objective  problem, so multiple losses to optimize during the training.  But at the end, you always have a connected graph, so you can always reach the target  tensor you're interested in from one other tensor.  There's no way some tensor is not connected to any other, I think.  Yeah.  Yes, but you can...  Yeah.  So you said network topologies may be much more complicated than the one we worked on,  but there should always be a way to reach tensor inside the graph.  Otherwise, it's unconnected, so you can always do that.  Yes, please.  So the question was about requiring your network and how you structured the data.  So the difference with requiring your network is that you provide data in sequence, and  so that layer expects you to provide a sequence of tensors in input, and you specify the length  of the sequence.  You may specify the length of the input sequence and the length of the output sequence.  So what's the number of sequences you want to take into account in input, and what's  the dimension of the output prediction?  How many tensors you want in output?  That's the difference.  Yes, please.  Sorry.  Yeah.  So the question was, in my example of the MNIST, the validation accuracy was better than  the training accuracy.  It shouldn't be the way around.  The reason is we actually trained that network for two or three epochs, so it was very few  number of epochs to train.  So it's totally meaningless.  It's just to show that you can watch what you can do in Keras.  It's not real performances, of course, but of course, thanks for pointing out.  But it's totally meaningless.  Don't rely on it.  It's off playground.  Any other question?  Okay, so, again, thank you very, very much for being here.  Please let me know if there's any questions or enquiries about the materials.  Enjoy with it.  Thank you.
EuroSciPy_2017__Keras_DO3tn_I1P-8_0_4715.png EuroSciPy_2017__Keras_DO3tn_I1P-8_4716_5195.png EuroSciPy_2017__Keras_DO3tn_I1P-8_5196_11999.png EuroSciPy_2017__Keras_DO3tn_I1P-8_12000_13187.png EuroSciPy_2017__Keras_DO3tn_I1P-8_13188_13931.png EuroSciPy_2017__Keras_DO3tn_I1P-8_13932_15935.png EuroSciPy_2017__Keras_DO3tn_I1P-8_15936_18911.png EuroSciPy_2017__Keras_DO3tn_I1P-8_18912_23027.png EuroSciPy_2017__Keras_DO3tn_I1P-8_23028_24191.png EuroSciPy_2017__Keras_DO3tn_I1P-8_24192_26219.png EuroSciPy_2017__Keras_DO3tn_I1P-8_26220_26291.png EuroSciPy_2017__Keras_DO3tn_I1P-8_26292_26423.png EuroSciPy_2017__Keras_DO3tn_I1P-8_26424_27467.png EuroSciPy_2017__Keras_DO3tn_I1P-8_27468_27695.png EuroSciPy_2017__Keras_DO3tn_I1P-8_27696_28475.png EuroSciPy_2017__Keras_DO3tn_I1P-8_28476_31991.png EuroSciPy_2017__Keras_DO3tn_I1P-8_31992_32363.png EuroSciPy_2017__Keras_DO3tn_I1P-8_32364_33287.png EuroSciPy_2017__Keras_DO3tn_I1P-8_33288_33419.png EuroSciPy_2017__Keras_DO3tn_I1P-8_33420_33779.png EuroSciPy_2017__Keras_DO3tn_I1P-8_33780_34451.png EuroSciPy_2017__Keras_DO3tn_I1P-8_34452_35087.png EuroSciPy_2017__Keras_DO3tn_I1P-8_35088_36023.png EuroSciPy_2017__Keras_DO3tn_I1P-8_36024_40727.png EuroSciPy_2017__Keras_DO3tn_I1P-8_40728_48215.png EuroSciPy_2017__Keras_DO3tn_I1P-8_48216_48563.png EuroSciPy_2017__Keras_DO3tn_I1P-8_48564_48611.png EuroSciPy_2017__Keras_DO3tn_I1P-8_48612_48659.png EuroSciPy_2017__Keras_DO3tn_I1P-8_48660_49115.png EuroSciPy_2017__Keras_DO3tn_I1P-8_49116_54827.png EuroSciPy_2017__Keras_DO3tn_I1P-8_54828_57791.png EuroSciPy_2017__Keras_DO3tn_I1P-8_57792_57923.png EuroSciPy_2017__Keras_DO3tn_I1P-8_57924_58055.png EuroSciPy_2017__Keras_DO3tn_I1P-8_58056_58331.png EuroSciPy_2017__Keras_DO3tn_I1P-8_58332_58907.png EuroSciPy_2017__Keras_DO3tn_I1P-8_58908_59063.png EuroSciPy_2017__Keras_DO3tn_I1P-8_59064_61379.png EuroSciPy_2017__Keras_DO3tn_I1P-8_61380_64199.png EuroSciPy_2017__Keras_DO3tn_I1P-8_64200_73679.png EuroSciPy_2017__Keras_DO3tn_I1P-8_73680_74747.png EuroSciPy_2017__Keras_DO3tn_I1P-8_74748_74879.png EuroSciPy_2017__Keras_DO3tn_I1P-8_74880_74999.png EuroSciPy_2017__Keras_DO3tn_I1P-8_75000_77879.png EuroSciPy_2017__Keras_DO3tn_I1P-8_77880_78887.png EuroSciPy_2017__Keras_DO3tn_I1P-8_78888_78959.png EuroSciPy_2017__Keras_DO3tn_I1P-8_78960_79451.png EuroSciPy_2017__Keras_DO3tn_I1P-8_79452_80639.png EuroSciPy_2017__Keras_DO3tn_I1P-8_80640_81119.png EuroSciPy_2017__Keras_DO3tn_I1P-8_81120_81299.png EuroSciPy_2017__Keras_DO3tn_I1P-8_81300_85859.png EuroSciPy_2017__Keras_DO3tn_I1P-8_85860_85907.png EuroSciPy_2017__Keras_DO3tn_I1P-8_85908_86279.png EuroSciPy_2017__Keras_DO3tn_I1P-8_86280_86639.png EuroSciPy_2017__Keras_DO3tn_I1P-8_86640_88619.png EuroSciPy_2017__Keras_DO3tn_I1P-8_88620_88991.png EuroSciPy_2017__Keras_DO3tn_I1P-8_88992_89987.png EuroSciPy_2017__Keras_DO3tn_I1P-8_89988_91547.png EuroSciPy_2017__Keras_DO3tn_I1P-8_91548_91727.png EuroSciPy_2017__Keras_DO3tn_I1P-8_91728_91799.png EuroSciPy_2017__Keras_DO3tn_I1P-8_91800_93263.png EuroSciPy_2017__Keras_DO3tn_I1P-8_93264_93683.png EuroSciPy_2017__Keras_DO3tn_I1P-8_93684_94007.png EuroSciPy_2017__Keras_DO3tn_I1P-8_94008_100667.png EuroSciPy_2017__Keras_DO3tn_I1P-8_100668_101411.png EuroSciPy_2017__Keras_DO3tn_I1P-8_101412_101699.png EuroSciPy_2017__Keras_DO3tn_I1P-8_101700_102899.png EuroSciPy_2017__Keras_DO3tn_I1P-8_102900_110327.png EuroSciPy_2017__Keras_DO3tn_I1P-8_110328_115991.png EuroSciPy_2017__Keras_DO3tn_I1P-8_115992_116075.png EuroSciPy_2017__Keras_DO3tn_I1P-8_116076_116555.png EuroSciPy_2017__Keras_DO3tn_I1P-8_116556_124907.png EuroSciPy_2017__Keras_DO3tn_I1P-8_124908_125063.png EuroSciPy_2017__Keras_DO3tn_I1P-8_125064_125111.png EuroSciPy_2017__Keras_DO3tn_I1P-8_125112_127247.png EuroSciPy_2017__Keras_DO3tn_I1P-8_127248_135575.png EuroSciPy_2017__Keras_DO3tn_I1P-8_135576_135683.png EuroSciPy_2017__Keras_DO3tn_I1P-8_135684_135847.png
Alright, so in the meantime you get the material. I will just give you a few words about this tutorial and... Oh, sorry. What's the idea I have in mind with this tutorial? So... Alright. You should have a list of notebooks here. The original title of this tutorial was 10 Steps to Carousel. So the idea was to introduce the Carousel framework into 10-ish steps. And these steps are going to be notebooks. You have a pre-amble notebook here where you can see all the instructions. In the back, can you read that? Is it fine for you? No, probably it's a bit tiny. Do you want me to... Better? Good. So this is the goal for this tutorial is to introduce the main features of Carousel and to learn how you can actually implement deep networks using Carousel. And one thing I should definitely want you to learn and to know is that Carousel is, at the end of the day, very easy to use but it is not for easy tasks. You can actually do complicated stuff using Carousel. So this is the outline for this tutorial more or less. And these are the topics we're going to go through in the next hour and a half. So multi-layer, fully connected networks, so to introduce the very basic features. And then we go through engine layers, features, embeddings, convolution networks, hyper-projects, hyper-amplitude tuning, how you implement custom layers in Carousel, and how you actually use deep neural networks, in particular convolution networks and residual networks, transfer learning and fine tuning, and if we have time, we can refine your network and auto-encoder and in the end, multi-modal networks. So there's a lot of stuff, as you can see. All the materials are online, you can actually play with it. All the code has been thought to be run on your notebooks. So I actually simplified some data examples. So you should be able to run it even if you don't have GPUs on your machines. But of course, if you have it, it's definitely better. Just run faster. So I'll try to go fast on some topics. But at any time, if you have questions, please feel free to interrupt me. There's no problem at all. You will find the requirements. Basically it's going to be Python 3. 3.4 should work as well. And also Python 2, I don't really know. Just a quick note about setting everything up. A friend of yours, some people from the audience, sent me that actually it may be a bit cumbersome to set up the Keras folder on Windows machine. If you try to Google it, so what I was saying here, the.keras folder where the actual Keras configuration file should stay, even on Windows machines, should be played. So Google it a few minutes ago, it ended up to be that the home folder is what you get in return from the Python function here. So typically something like seed, semicolon, double, backslash, users, double, backslash, and your username. So these is the place where you should create your.keras folder. I know if you experience the same problem, but this should fix the problem. So when I say home folder here, I assume even on Windows machine that is the folder. And so you should have for this tutorial, for this series of notebooks, these configuration files. I will tell you later on the meanings of these configurations, but they're quite straightforward. So, I assume you already tried and verified everything. That's nice. Good shot. After that, so let's start with the first notebook, if you don't mind, which is going to be number one, multi-layer fully connected network. The main goal of this notebook is to provide you an example on how you can actually deal with our.keras and how you can build networks using.keras. So I freely borrow these payoff sentence saying that Keras is from the Django project, saying that the Keras is deep learning library for perfectionists with deadlines. And these means that, as in Django does, dealing with Keras objects and creating networks with Keras is very easy to go. And in fact, one of the main important features of Keras is summarizing these sentences from the official documentation, when they say that being able to go from one idea to results with the least possible delay is the key to doing good research. In a few words, Keras is a high-level neural network API written in Python. It's pure Python, so you can actually take a look at the code and even modify it, which is capable of running on top of different deep learning frameworks, which Keras calls backends. So far, the officially supported backends are TensorFlow, C and TK, which is the deep learning library from Microsoft. And these has been the latest addition. I think it's been added from three or four versions. And the ANO. So historically, Keras supported the ANO TensorFlow, now C and TK. And just for you to know, there's a sort of unofficial support of MXNet as well in the back end. I'm saying it's unofficial. Well, it's not in the master branch of the Keras code. That's because for some reason, the developer of the MXNet back end supported, are now supporting the Keras 1 API. But now we're using Keras 2 API. So there's been an API change some time ago. And for some reason, MXNet is still stuck at the Keras 1 API. So that's why the MXNet is not yet officially supported. But if you use Keras 1 version API, you can actually also have MXNet as a back end. Having multiple backends is a feature that you will appreciate very soon in the next examples. So starting with this very simple example, let me remove just to get some space. It's a Kaggle challenge data. I just read very briefly the problem. So the Hutter group is one of the worst biggest e-commerce company. And the consistent analysis of the performance of product is crucial in this company. And while that do the diverse global infrastructure, many identical products get classified differently. From this competition, we have provided a dataset with 93 features for more than 200,000 products. Of course, for the sake of these projects, I reduced drastically the number of samples. In our data, each row corresponds to a single product. And so we have a total of 93 numerical features, which represents the count of different events. So if you take the data, it should be placed in your data Kaggle auto group folder. For the rest of these examples, we're going to play a very simple model, which is logistic regression. So for those of you who doesn't know, the logistic regression is a very simple supervised learning algorithm characterized by the usage of the logistic function, also known as the sigmoid function, which has the shape, so the S shape, that's the sigmoid name. So after preparing the data and getting the data, so in a few words, what we do here is just loading the data from the CSV file. There's actually some pandas code running in the back end. And then we preprocess the data so that we just scale the features using standard scaler in scikit-learn. And then we preprocess the labels to have one hot encoding of the labels. Are these terms clear to everybody? So I can go safely with that? Okay, fine. Thank you very much. So taking a look at the final results we have, we have nine classes and 93 features as we expected. So these are the labels. Of course, these are anonymized in some way because they're Kaggle data. And just to give you an example of what is the result of one hot encoding, this is something like we just have, for each sample, just one element corresponding to the activated bit, the so-called hot bit. And that's why this sort of encoding of the labels is called one hot encoding because just one single label is enabled. Of course, this is a single class problem. It's not multi class. Every single sample is just one class in our data. So if you want to implement it in pure theano code, I will go very fast about that. But if you want to implement a logistic function using theano, these are the code you have to do. So you basically have to declare these objects. So the tianomatrix and the shared variables here, the shared are those that are actually modified during the training, that's the difference. So you have to implement manually everything basically. So it's just the probability target, the cross-entropy loss function we want to use. But the only thing, which is not only, of course, but one something which is provided from theano is the calculation of the gradient. So theano provides out of the box the calculation of the gradient. This is one reason why you don't want to really implement this code in pure Python code. So you want to rely on these libraries. So finally, you have to create this tianofunction. So that connects every single tensor object you're actually creating. So you provide the input, you provide the output there. And this is the rule for the updates of the shared variables here. So of course, these are the weights and the bias, simple model here. So we compile all the code. In case you don't know, this is based on symbolic execution. So you have to compile it and then you run it. So everything you set up previously has not been executed yet until you compile it and you run it. Then you create a new function for the prediction. Using all the tensors already provided. So this is for the training. This is for the prediction and the rest of the code is just repeating the training process for the number of epochs. So we are actually implementing the training process manually. And this is the way theano does this very simple problem. As you can see, it's very low level, right? So you have to implement it at every each step. If you want to use TensorFlow, as you will immediately understand, the level of obstruction is exactly the same of theano. So again, you have to specify placeholder here. So let me just execute it because I just want to show you something. So you have to create the placeholder here. And the placeholder are going to be the variables that are going to be updated during the learning into the graph. So as I told you, this is symbolic execution. So if you print X, you just got a tensor object with no shape yet because you don't really have passed the data, fed the data to the graph. And the only thing you know is that you have 93 dimensions, so 93 features. And why should will be a tensor having a total number of classes at the mention? That's a second dimension. TensorFlow ships with these tool, which is very useful, which is called TensorFlow. I'm going to execute it just because I want to show you. This is very nice. So this is a sort of a little bit of engineering of the TensorFlow code. So we're going to again create all the variables here. Then TensorFlow provides a little bit more abstraction with respect to the Tiano having this softmax function already implemented. Then the matmul operation is going to be the equivalent of the dot operation in numpy if you come from that backend, the background. And then we are actually adding some additional information that will be used in the tensorboard tool. So we're going to record some histograms during the learning and some reference to the scalar objects we have here. Again, this is very intuitive because we're actually defining different scopes. So we are defining the scope of the model here, the scope of the cost function, the scope of the train process. And so they're going to live in different scopes, so you can actually better manage them. And finally, the accuracy scope, which is going to be the metric scope we want to use to evaluate our model. So then TensorFlow provides you this file writer, which is actually something that stores the results of the graph you have. So the computational graph you have in the backend. And after that, you just run it defining TensorFlow session. So everything living in this TensorFlow session can be interpreted and then run during the learning process. But again, what you have to do is to implement manually all the learning process by actually writing code and providing the data for each of the tensor you want to change during the learning process. So oops, that's something went wrong, of course. That's the demo approach. Oh, I know why. Oh, let me just, sorry, let me just, I just want to clear the memory of the GPU. I'm actually running on some GPUs in a remit backend. So let me run again. Sorry about that. OK. All right. So I have to load the data. All right. Come on. That's a cold start. Very cold one. All right. So let me skip the other part. So let's go to the TensorFlow. All right. OK. Now this is going to work hopefully. Please. All right. So the app box, we have just run it for 25 app box, I think. The number of the box is, I can remember. So we have 25 app box, and this is the value of the accuracy during the different app box, and this is how the plot, we have just added it, how we plot different costs, how the cost changes during the different app box. So this is the trend. All right. So I just want to show you this TensorFlow tool, which is going to be something we're going to see also integrated in Keras. That's why I'm showing it now. So I'm running. So this TensorFlow.tensorboard command runs a server typically on the port 606.006. And what this does is to read the log file that has been written during the execution in particular, the file stored in this directory. All right. So hopefully we're going to see something. Okay. So we have these colors here. So we have the plots for the accuracy automatically provided, the plots for the cost function during the training, and the plots, and all the different plots for the mean weights and the mean buyers. All right. This is very easy to use and very useful when you face real problems. Another thing it may be useful is to take a look at the graph. So this is actually the model that has been implemented inside TensorFlow. So this is how the graph of tensors has been translated by TensorFlow. So you can actually take a look at all the different nodes here. So each of these corresponds to the different scopes we specified. And finally, these histograms we built to show the difference of the different trends of the cost histogram and of the model histogram. Something wrong maybe because I just run it multiple times. I'm safe holder. So that's why you see this so strange. Anyway. All right. So let me stop this. All right. Okay. So let's see how we implement a logistic regression using Keras. Finally, as you know, the logistic regression may be implemented as a neural network, just one layer, which has the sigmoid activation function. And so it is. So you have all the data again. And we are going to use these sequential objects. We're introducing this new sequential object in Keras. The sequential object gives you the idea that you want to build a series of layers stacked one on top of the other. And so you actually instantiate this sequential object and you simply add layers to these objects. So in this case, we had just had a dense layer having a number of units in output corresponding to a number of clusters. So we specify the input shape, so the number of features we have. And the activation is going to be the sigmoid function as we expect because we implemented a logistic regression. Finally we add the decision function, which is the softmax function as we did in TensorFlow. And we in the end compile the model again. So we compile specifying which is the optimizer. In this case, we specify the stochastic-grade descent. We actually did the same using TensorFlow. Let me just go back a bit. So when we are in the train scope here, we specify that the optimizer was the grade descent optimizer with the minimization problem. So it's stochastic-grade descent with these learning rates. And we are doing the same here. So we are implementing the same using the categorical cross-entropy and the stochastic-grade descent optimizer. After that, we just call this method in Keras object, which is called fit. And we provide the data, the training data, and the training labels. And that's it. So you don't have to implement it manually. You don't have to implement the training process manually. You just call the fit method as you do in scikit-learn, for instance. And that's it. And Keras provides you automatically a series of our default parameters. We're going to change during the next steps of these exercises. But in the very beginning, you just create the model, you compile it, and you fit it. These are the three steps you have to do, the only three steps you have to do to make our Keras model up and running. And so as you can see, if you run the fit method, you're going to have a series of 10 epochs by default. And Keras provides you these progress bar during the training, telling you the time it takes, the values of the loss during the training, and how does it change. All right? This is the very beginning. So now let's have a look at it in a more interesting way, let's say, so in more details. So the core of the destruction of Keras is a model, of course, and the main type of model is going to be sequential objects. The sequential object provides you this method, which is called add, and so you have different layers. In this example, we just had one layer, which is the dense layer, and then the activation layer. So the dense layer, this is the dense object, has this parameter. Most of them are default values, and in case you don't know, keep the default values. Keras typically has been designed to have the best default values you can get for the different objects. And this is particularly useful when you run it for the first time, especially for optimizers, when you have to set up the learning rate and depending on the optimizer, different other parameters. The most important things you have to learn and to understand very soon is that the first and the only required parameter for this layer, and this is typically for all the layers, is the first one, which is units. As you can see, it's the only one that has not default values, of course, and this is where you put your insight and your knowledge, and the unit's value is going to be an integer value greater than zero, of course, and this corresponds to the number of neurons that you will get in output from the layer. Keep this in mind, it's going to be the output number in neurons, not the input, right? You have to specify the number of input dimension only in the first layer. The next layers will get, as input, the output of the previous layers automatically, you don't have to specify it. So the units is going to be the number of neurons you want in the output. So back to the example, since we just have one layer in this example, the number of units we want to have in this network is going to be number of classes because it's the output we want to predict. It's just one layer, right? So it's just number of classes, so this case is going to be nine. All right, and as you can actually see, since this is the very first layer of a network, we also specify the input shape, and this is mandatory for the very first layer of the network. You always have to specify what is the input shape. Speaking of which, the input and the output shapes of the layers are going to be this one. In input shape, you get a non-dimensional tensor object with these series of shape. Remember that the very first dimension of the input tensors are going to be the best size. In other words, the number of samples you're going to provide to the network during the training, right? And this parameter is specified when you start the training process, so you call the fit method. You don't have to specify this dimension in the input shape parameter. All right, so let me just go back again to this example. As you can see, the input shape in this case is going to be dims, so the number of features we have. All right, so this is 93 in this case. We don't specify the best size. The best size is indeed automatically specified in the fit method. We're going to use the default parameter here, but we can actually change it. So remember, the input shape is mandatory for the very first layer and has to not include and must not include, sorry, the best size dimension, which is going to be the very first dimension of the tensor you have. And as output, what you get in output form a tensor is the best size again, so the same number of samples, of course, but as they mentioned, the units you specified as the first parameter of the layer, so the output neurons. All right, is everything clear? Questions? All right. So before moving on, some very few notes about the star initializers parameter. So typically in the dense layer, but in most of the layers, you have this kernel underscore initializer and bias initializer parameters. This is just for you to know that in literature, there are lots of methods to initialize the weights of neural networks, and this is particularly useful, and different methods to initialize for the initialization of the weights can drastically change the performance you're going to have. So I just proposed you to take a look, of course, at the keras.initializers to see what are the functions supported by keras. There's lots of them. And read these very interesting article. This is not in, this article does not include all the possible initializer supported by keras, but this is very interesting, and I give you some recommendation paper to read. This is a very interesting topic, and as you can see, the default initialization method for different weights by default will change according to the different layers keras provides. So if you take a look at the default parameter for the convolutional layer, it should be different. I cannot remember for sure, but it should be different, because for convolutional layers, you have different initialization technique by default. Other core layers provided by keras, you have flatten layer, reshape layer, permute layer, these are going to be so-called operational layer. So when you have to, when you have, they're not learning anything here. They're just making operations on the tensors during the flow of the network. The lambda layer, you can actually specify the custom Python function to apply to the tensor. All right? Please keep in mind that what a keras layer gets an import is always a tensor object, which is going to be physically and concretely corresponding to a tensor flow tensor object, or a theano tensor object, or in this case, nowadays also we have a CnTK tensor object, right? So to give you just the idea, when you run the fit and the training process using keras, keras indeed is not performing any real operation on the GPU, all right? So the keras code relies on the Python level, and all the real computation on the GPU and stuff is demanded completely to the back end. The main advantage you get from this approach is that the same code you write, exactly the same code, is supposed and guaranteed to work on all the back ends supported by keras, without changing any single line of code. Keras does all the job for you, even if there are some very slight and tiny differences between the, for instance, the handling of the shapes of tensors between the tiano and tensor flow. Keras does all the job for you, you don't have to change anything. That's power of keras indeed. Another important layer is the activation regularization, so you can actually apply regularizers on the activation functions to, so for instance, L1 or L2 regularization on the activation function of your network. This picture, kindly provided by my friend Ian, shows all the, probably it's a bit tiny, but should give you another view of all the things, all the kind of networks supported by keras. Keras provides support for feed forward network, so the classical network, you provide input on the left and you get output on the right, so the feed forward network in these sets of networks, also the convolution networks fill in. You have recurrent neural network, support for LSTM or gated recurrent unit network, and also for unsupervised learning, you can actually build autoencoders, variation autoencoders and sequence networks using keras code. Finally, keras also provides support for different optimizers, not just the stochastic ring descent, of course, there are many of them, you can have aider grad, aider delta, lm sprop, lots of different optimizer. You can actually take a look at the keras.optimizer package to see all of those optimizers supported by keras. If you want to data science in a bit of this example, we're going to play a bit with the hyper parameter of the network and the parameter of the feed method. I assume you all know what overfitting is, or underfitting is, so something we can do. One method I'm going to show you, because we're going to use it very heavily, a lot, is this summary method for the model object. When you call the summary, you actually have a look on what are the layers, what are the outward shape, how many parameters you have, and in the end, how many total parameters you have, and how many of them are trainable parameters. If these does not make any sense to you, it will make very soon. So far, the total number of trainable parameters are going to be the same. The design thing about this example, we're going to use some sort of train test split on the data. We have not only training data, but also validation data. We want to fed into the training process. We're going to apply a technique which is called early stopping. The early stopping technique is something that is typically used when you have big networks and lots of data. The intuition is that in these case, we're going to monitor this metric, which is called the validation loss. When the validation loss does not change after two epochs and does not change significantly, so you can actually specify the precision you want, the training process is automatically stopped. This is the early stopping, because, differently from other machine learning methods, unless you specify these kind of tricks, when you run a fit of a neural network, you have to specify for how many steps you want the learning to go on, right? So the number of epochs, so-called. These other objects is the model check point. During each epoch, at each step, automatically, this function, this object, records the value of the validation loss again and saves the weights, so the stages of the network at that stage. And automatically, this has been set up to save only the best model. So in the end, you get the best weight configuration, so leading to the best validation loss value. You can plug both of them using this parameter in the fit method here, which is the call backs. So Keras provides this way to plug additional callback objects. They call callbacks because they're automatically called by the framework before the next batch or before the next epoch is going to be processed during the training. You don't have to do anything, you just pass the callbacks. We're going to specify a different number of epochs. We're going to specify what are the validation data in this case. We previously provided just training data, and these are the two mandatory parameters, the very first two, validation data again, and we specify the size of the batch. And we also specify that we want some verbosity, but this is the default one. All right. All right, so we run it for 50 epochs, and these goes on and on. Additionally, we also have a validation loss that, since we have validation data in this case, we're going to also have the computation of the validation loss. So during each step, we have a log of how the loss changes, the training loss changes, and the validation loss changes. Okay, moving on. Okay, very briefly, moving on just a bit. So we want to create a multi-layer fully connected network. When we say fully connected network, we're going to expect to have all dense layers. The very first example of multi-layer fully connected network is the multi-layer perceptron. The multi-layer perceptron is characterized by having one input layer, one hidden layer, one output layer. All right. So technically, when you have this sort of network, so one, with just one hidden layer in the middle, you're calling it multi-layer perceptron. When you have at least more than one hidden layer, you're allowed to call it deep network. All right? So when you have at least two layers in the middle, it's going to be a deep network. So in this case, we're going to have a multi-layer perceptron because we have one input layer and this is just the decision layer. So it's going to be very shallow network yet. And as you may assume, the way you plug additional layer, so you build a multi-layer network in Keras is just by calling the add method to the special object multiple times with different layers. All right? So again, make sure that you specify the input shape in the very first layer. In the second layer, don't have to because it's automatically induced by the previous layer. All right? So again, we compile the model. We print the summary, so we have dense number two, dense number three here, and the final activation, which is the softmax for the decision function. Of course, since we have 100 neurons here in output and nine in the end, the total number of parameters has increased, as you may expect. Then we run the fit model again and the validation loss changes drastically considering the same data. Yes. So this is 1.6 taken just randomly, and this is 0.7. So this is going to be a better model with respect to the previous one with our data. So if you want to play a bit, there's an exercise for you. The idea is to play a bit with it, adding as much layers as you want, better to add a couple of, and run it in your machine to be sure that everything's up and running, and to see that it's very easy to run using Keras. All right? The next steps will be taking a brief look at what is called the Keras backend. I just want to show you the Keras backend because this is how actually Keras allows you to have multiple backends running for your deep learning code. Okay. So to give you some hints, the only way to play with it is just to add additional dense layer. How many neurons you want to add? How many layers you want to add? Nobody knows. You just play with it and you see what, if that makes any difference to the final results. That's the takeaway typically for deep learning. At the end of the day, you don't really know what you're doing. So let's have a look at this Keras backend. The Keras backend is the module which is provided by Keras that integrates to real backends. If you take a look at this dot backend package, right, in Keras, so it's going to be the Keras dot backend package. If you take a look at the code, you will see that in that package, Python package, you will find a file which is called a module, Python module, which is called TensorFlow underscore backend dot pi, Tiano underscore backend dot pi, and now Cntk underscore backend dot pi. So this backend is a sort of wrapper module that calls the real backend depending on the configuration file or the environment variable you have set up in your settings before running your code. So we have actually set up our TensorFlow backend in the configuration file, as you may have assumed, so I will just want to show you. So if I'm going to cat the Keras dot JSON file, all right, among these configurations, we have the backend and we are specifying TensorFlow. If we just switch these directive to Tiano, you're done. You just rerun your code, all right, and Keras automatically switches everything on Tiano backend. All right. How these backend works, so as I said, the Keras dot backend is the wrapper for the other module. All these modules, so basically the Keras backend provides the interface and all the functions provided in this Keras backend module are assumed to have a counterpart in each of the backend supported. So when you're actually calling the, okay, let's have a look at it. So maybe it's easier to do. So we're actually reusing the Keras backend API to reimplement again the logistic regression example, but in this case, we're actually, we're still using Keras code, all right. So let's have a look. So here we're creating placeholder objects using the K module, which is the Keras backend, we just call it K, it's just a sort of naming convention here, all right. So the K is going to be the Keras backend module and we're going to create a placeholder object. The syntax is very TensorFlow-ish, all right, it's very similar to TensorFlow, but what actually happens in the backend is not TensorFlow always, depending on the, it really depends on the backend you're using. So if I'm using TensorFlow as a backend, this instruction is going to create a TensorFlow placeholder object. If I'm going to use theano, that is going to be a creation of theano tensor object, all right. This is the backend API. So we're creating a variable object again. So the level of abstraction has changed drastically, has moved again to the same level of abstraction you have using TensorFlow or theano packages, but with a big difference that this code is supposed to run seemingly using all the backends you want, those supported, of course. So without changing any single line of code, just the configuration fire, all right. So this is going to create a TensorFlow, if you're using TensorFlow, yes. All right. Okay, so I repeat the question. So the question was, why would you actually have support for multiple backends, right? Because of many reasons need, you can, because the different backends have different performances, all right, and so you want to leverage on the benefits of those switching the backends, because sometimes ago the memory management of TensorFlow was really rubbish, so you typically wanted to switch to theano to have a better location memory. Now it's not the case, all right. Because sometimes you want to try different backends having to test if everything is working with the same performances, all right, because of numerical reasons, because of numerical optimization, you may get, all right, for instance, I just read a sort of benchmark article providing a sort of overview of the benefits provided by CNTK. CNTK, among those backends, is the only one using MPI in the backend, so multiple messaging and passing interface, right, so for multiple processing. As far as I know, CNTK is built on top of MPI. So far now TensorFlow supports MPI in the version 1.3, so the latest one. And considering the different network architectures, at the end of the day, CNTK has turned to be the most efficient for the recurrent neural networks, which are going to be the very heavy network to train, the one that takes more time to train, all right. So for instance, for convolutional networks, TensorFlow and CNTK are sort of the same performances according to that benchmark. But CNTK won for on-require neural network, for instance. So maybe you want to switch to CNTK using, without changing any single line of code, that's a benefit. And you don't have to mess with the very tiny details of the different implementation, because Keras does it for you. All the three backends have totally different APIs and totally different syntax for creating tensors. Keras does the job for you to translate all of them, and the way it does is using this Keras backend. So when you're actually running, that's it. This is what I was looking for. You can actually run this function, the K dot function. This is calling matmool for TensorFlow and dot function on the ANO. All right. I don't know the equivalent in CNTK, but TensorFlow decided to call the dot product matmool, matrix multiplication for some reason. Okay. When you call K dot, it's going to be mapped to the corresponding function in the corresponding backend automatically. That's it. So now the question could be, why would you do that? Why would you implement it using the Keras backend? So what's the reason why Keras provides you these APIs to be used? Because this is the same API you're actually using when you're creating custom layers. If you create custom layers using the Keras backend API, you're automatically sure that that implementation is going to work on all the backends provided, because you're using the Keras backend functions. You're not using the TensorFlow or the Tiano real functions. All right. You see what I mean? Okay. So the rest of the notebook is to show you how you basically do the same TensorFlow-ish stuff using the K backend. And there's also an exercise for you, which I'm going to skip, which is the implementation of the linear regression. I'm going to skip also because you indeed have the solutions in your material. So if you load it, you're done. All right. Okay, so in case you don't have any questions, I would just move forward. Yes, please. Yes. You're actually stucked. Sorry. I repeat the question. I'm actually advertising all the benefits of Keras. So the question was, what's the downside of it? I may say that the downside of Keras stands on the downside of the supported frameworks, supported backends. I would say three things. Not all the functions are really supported by Keras backend, because Keras backends, since it has to provide you that the same code works seemingly on all the backends, in case there are some very specific functions supported by one specific backend, you're not going to find it in the Keras backend API. But a sort of note about this is that if this framework, the one that has this tiny function, is going to be TensorFlow, you can actually manage to do it, because Keras is going to be integrated in TensorFlow, and it is better integrated each version by each version. I will show you something in the end. That's one problem you have. And the downside, and not the downside I may say, is that since Keras is not doing any real computation, you can do with Keras everything you can do with all the frameworks. But if, for instance, so far in TensorFlow, which is the one I know the best among the three, there's no official support for dynamic graph computation. So everything is going to be static. So this means that when you create the graph, you have to compile the model indeed. So this means that after you create in the network, you're compiling it. So you're creating the network beforehand, you compile it, and so the resulting graph will be put, all the tensors will be put into the memory of the GPU, typically, if you have it. And so that's graph, it is very difficult to change it during the training. You cannot do that. The PyTorch approach that I believe we will see in the keynote we have in two days, you will see that that approach is totally different. That's called dynamic graph creation. So you create it one step at a time during the execution, and so that's our sort of different approach to that. Keras does not do that because PyTorch is not supported yet. I say yes, I don't know if we'll be supported at some time. All right. Yes. Default backend. Yes. So the default backend was the question, and the answer is the ANO is default one. You can actually go to the configuration file, or so as I showed you, you just, for instance, let me do this. I can show you in a terminal, maybe it's better, the one to risk to waste the notebooks. But indeed, if you, you have two ways. All right. So if I print a cut, keras.json file, you will see that the backend is TensorFlow because I set it. If I do change this directive, all right, if I specify the ANO here, or even CnTK, all right, now the backend is the ANO. If I actually go in Python and import keras, this is using the ANO backend. You see? If I do this just to play a bit, CnTK should be installed. This gives me the opportunity to say a few words about that. Using CnTK backend. I did not include that examples using CnTK, mainly for one reason, because CnTK is still only supported on Windows machines and Linux machines. There's no support for macOS. You can actually have CnTK installed using a Docker image. So I prefer not to include also these into the setup of this environment. It was not really required. The main focus was on keras, of course. But this was the main reason. And the reason why is it, I don't really know. It's Microsoft stuff. You know, in the end, it's always Microsoft. So let me switch back to TensorFlow. Another way you can do it is by setting us a keras underscore. Let me just move a bit so you can reduce it. You can actually export. No, let me do this. Yes, I can do it. Keras underscore backend environment variable, back end, equals to piano. Okay. Let me show this first. If I do this, sorry. If I import keras, now it should be TensorFlow again. All right. It takes sometimes, typically. Okay. If I do, I can do this, keras underscore backend equal piano, Python, minus M, import keras. Oh, sorry. I cannot remember it. Is this, how's the way to do it? Minus A, C, thank you so much. Using the end of backend. You can actually define this keras backend environment variable and you switch the backend without changing the configuration file. This is because the configuration file is just one file. If you want to have multiple keras instances running, it's not a good idea to change the configuration files multiple times. So you just define the environment variable there. The keras underscore backend and you're done. All right. Okay. Are there any other questions? Fine. So, let's move on. Let's talk about these Amnist dataset. I believe all of you already know what the Amnist dataset is about. In scikit-learn word, this is also known as the digits dataset. So the Amnist dataset is one of the widely used dataset for deep learning experiments and at some point they decided that it was fair enough to not use it anymore. But the Amnist database contains images, black and white images of handwritten digits. And so you want a network in our case to be able to identify the corresponding digits taking a look at the image. All right. So this notebook is just to show you that keras, just like scikit-learn, ships with these datasets, dataset module, and these datasets, Amnist database is provided. So you can actually import from keras.datasets, Amnist module, and with the load data, this method downloads data for you in case you don't have it, and the downloaded data will be placed in your dot keras folder, and you can actually get the data by loading it automatically. The rest of the notebook is just for you to play a bit. So I was asking what type of X-train is, and the reason is, the answer is, of course, it's an ampere rate. So once we have loaded, if we type the type of X-train, it's going to be numpn-eray. All right. The type of Y-train is going to be numpn-eray again. So the load data automatically gives you back the data into numpn-erays, which is something you typically want to have when you have to experiment. How many observations in train gate do we have? So again, it's going to be the X-train dot shape of zero, of course, so the number of samples we're going to have 60,000 samples in the training set, and 10,000 I will tell you in the test set. All right. How many numpn-erays? Observation for its digits. So it's going to be train dot shape again, but I will tell you that 60,000, of course. So the dimension of X-train, so if you take a look at the X-train dot shape, we're going to have 60,000 of 28 times 28 images. So we're going to have images of that size. Since the MNIST images are black and white images, you don't have the channel information, so it's black and white, so it's just two-dimensional object, not three-dimensional object. So very quickly, if you want to take a look at these images, you can actually print it. So I'm going to print the X-train of zero here, and this is going to be a sort of handwritten five. Okay. So we're going to use these data sets to play with the rest of the two notebooks. Okay. The first notebook is, again, about fully connected FIFORWAR network. Let's say the more complicated FIFORWAR network, FIFORWAR network, we're going to take a look at are going to be the convolution network. Some recall of the main things we already did, so sequential object and the graph model of the objects. We talked about the dense layer, aka fully connected layer. In this notebook, we're going to introduce a new layer, which is called the dropout layer, and we applied binary cross-entropy or a categorical cross-entropy to our data using the stochastic grid descent optimizer. Here you have the links to have a look at what Keras supports as optimizers and loss functions. We now introduce these new activation function, which is called the relu function, so the rectified linear unit. This function is going to be very useful in convolutional network. This function has lots of mathematical properties, and this is the mathematical definition of the function. This activation is going to return the maximum value between zero and X, so this means that all the negative activations will be automatically dropped out. When you calculate the derivative of this function, it recalls the sigmoid logistic function. The derivative of this function is called the softplus function. This function, despite its simplicity, is one of the coolest things invented in the deep network words many years ago. The note, keep in mind, this function, as it's heavily used in convolutional neural network. The exercise now asks you to build this network. This sort of syntax is quite standard when you have to specify the architecture of the network. In this case, we're going to have a fully connected network, a fully connected layer, with 512 neurons in output, plus the relu function activation, again the same configuration, and finally another fully connected layer, FC, with a total of a number of classes neurons plus the softmax activation function. If you don't want to bother too much about it, you just load it, and that's it. You create this sequential object, and you just plug all these layers using that model again. It's very simple and straightforward, as you can see. You have dense 522 activation relu, and input shape, because it's the first layer, has 784. Can you guess why this number has the input shape? Yeah. It's 28 times 28. We're going to linearize the images and provide it as just one big tensor or vector. This is because dense networks are not supposed to handle multi-dimensional arrays, but you can actually do it. Again dense layer as requested with the activation relu and the last 10 layers, so the number of classes with the softmax activation. We specify the categorical cross-entropy again and the SGD optimizer. Please note that this time, differently from previous examples, when we provided the optimizer parameter, we provided an actual Python object, normal strings. To be honest, the code is exactly the same, because when we provide SGD object here without any additional parameter, we're actually setting a new object with the default parameter values, which is exactly the same we get when we provide the string SGD. This is just to show you that you can plug not only strings, but real objects. If you're curious enough to understand how these string parameters are indeed implemented in Keras, take a look at the code and I will tell you it's a matter of globals in the namespace. You will find in the loss package an object which is called exactly in this way, categorical underscore cross-entropy. When you provide parameters using strings, Keras looks for that string in the global namespace and returns the object. That's the trick. This is the case for reload, softmax, blah, blah, blah. You can indeed provide real Python object or real Python function or strings if it comes to handy. Moreover, we're also adding this additional parameter to the compile method, which is the metrics. In this case, we're going to record the accuracy during the learning phase. Keras, when you run it, Keras automatically stores the accuracy. I was looking that. We create the model here. Let me just run it. We create the model here. We get the data, the Amnist data. Of course, the shape is the one we had previously. We have to do a little bit of reshape because the input shape of our network is going to be 788. We're going to reshape the Numpy arrays. This is pure Numpy operations. Finally, we're going to apply this to two categorical functions provided by Keras Utils to translate the labels in one hot encoding. This is because maybe you didn't get it. We want to have the one hot encoding for the labels because the last layer has 10 neurons in output. This last layer is the one that will be used to calculate the error function on the labels you have. The labels must have 10 dimensions. That's why you want to apply one hot encoding. You see what I mean? Fine. All right. We did it. We just train and test split. We have this one. This is just one images. Of course, this was different because it's been randomly split. As we see, we have this one hot encoding. For this X-train, the label of Y-train of zero is going to be nine. All zero apart from the last one. We take another one from the validation set. This is one. This is the label. This is because this one, as we expected. We train the model here. We just run it for two epochs. We do not expect very big results just to make it running. This is how it's actually working. We have this loss. We have this. Take a look here. We also have the accuracy for training data and for validation data because we recorded that metric in the compile method. We have loss accuracy, validation loss, and validation accuracy. Something which is new now for you maybe is that we recorded the output of the fit method. We call it network history. That's because by default, recalling you the callbacks of Keras, by default, the model.fit method returns you an history callback object. This callback object embeds all the history of the training process for you to take a look after that. If we want to plot the network performance trend, we have this network history object. As you can see, the network history object at this attribute, which is called history, is going to be a dictionary having keys for names corresponding to those returned during the learning phase. We're going to have a loss and a val loss. We're going to have an accuracy, ACC, and a validation accuracy here. Our intention here is to plot the differences of training and validation accuracy. The training and validation loss and the training and validation accuracy. Of course, these are going to be just two epochs. As you can see, so 01, 01 again. This is not very meaningful. It's just to show you how you can do it. After two epochs, we get just more or less 88% of validation accuracy. If you increase the number of epochs, you will get definitely better results. In this case, we've added a new parameter for the stochastic ring descent. We fit it again here. Let me skip it. It's not very important. Let me introduce another important layer, which is the dropout layer. The dropout layer is a very simple one. The idea of the dropout layer is to be used during the training phase to avoid overfitting. The idea is that when you put the dropout layer in the middle of your network, you specify a drop rate. By that drop rate, some connections from the previous layer to the next layer, neurons, will be dropped randomly. That's the idea of the dropout layer, no more than that. Something you may keep in mind, the Keras API expects you to provide the dropout layer. Some frameworks expect you to provide the rate, the retain probability, not the dropping probability. Keras wants the dropping rate. For instance, if you specify a dropout layer of 0.2, this means that you have 20% of probability to drop the connection. Dropping the connection means that you basically put a zero for that neuron from the previous layer to the next one. The intuition is that when you do it randomly, only during the learning phase, of course, you're doing it because you want to have some variability in your data. You're randomly dropping, muting some connections between neurons to have variability in your data to avoid overfitting. Just to show you this, if you import the dropout layer from the Keras layer and you take a look at the code of the dropout, as I told you, the Keras code is Python code. You can actually take a look at it. Let me see if I can make it bigger. No, this is not. Where is it? I made a mess. I was expected. Sorry. Where is it? No, too many things opened. That's why I do too much stuff. All right, I made a mess. I'm sorry about that. I had good intentions, I promise. Let me go back to it. Sorry. Yeah, yeah, that's because. Okay, so if you take a look at it, this is the implementation of the dropout class in Keras code. When you see it during the call method, this is the interesting part. So it returns the returning of this K.intrainFace function. So if you take a look at it, of this K.intrainFace, all right, the meaning of this function is to select X if the network is actually in the training phase, is to select Alt otherwise, so it's in inference phase. So again, back to the dropout, this means that if the network is during the training phase, this layer returns the dropped inputs. Otherwise, it returns the inputs as it is. So you don't have to worry about the fact that the dropout is only applied during the learning phase because it must be only applied, of course, only during the learning phase because it's supposed to avoid overfitting. All right, this is some internals I want to show you. So apart from this, these exercises just ask you to change a bit of the code, you actually have it down there, to change the previous configuration using the dropout layer. And again, see the difference in performances. And these are the difference. So this starts to have more sense on the data you have, especially, but the fact that now the accuracy reaches the 90%, even if you have only three epochs. Of course, if you continue training at some point, you will start overfitting. And so we can try plugging some early stopping into the network to help to skip it. Movi got something more interesting, something you can actually do with the Keras layers, sequential objects. I don't know how much time do you have? Half an hour? Five minutes. Kidding, right? It's not five minutes. It's six minutes. Sorry. I usually have lots of stuff to say. I'm sorry. Anyway, okay, let me just say this. Something you can do in the, using the Keras object is to take a look at it, and it's very handy to see what are the layers, especially when you have a big network, what are the layers in it. So you can actually iterate using the layers attribute inside the model, so you can take a look at it and take a look at the different parameters of the layers. As you can see, there's a lot of them, even with this quite shallow network. Something very interesting is to extract the hidden layer representation of the given network. This means that sometimes it is very useful to see how the network is actually interpreting and seeing the data inside the network. So it's just a way to open the black box of the deep learning. There are many ways to do it. One simple and most intuitive way is to create a network, train the network, create another similar network, and that network is going to be exactly the same network up to the last layer you want to take a look at. So imagine that we want to see how the data changes after the first dropout. So remember, we have a dense dropout, dense dropout, and other dense. So we have model already pre-trained. After the training, we have a set of weights. So we want to see what's the internal presentation of this layer. So one straightforward way to do it is to create a truncated version of this network, initialize the weights for these layers by setting each layer's weights to the same weights of the target model. We compile the truncated model, and then we just run it in inference mode. So we call it predict, not fit in this case. This is one way to do it. This is, of course, the Python way to do it. So you're actually dealing with Python objects here. Another less intuitive but definitely more effective way to do it is to leverage on the underground graph of tensors you have. And so letting tensor flow or the back end you're using to do the actual job, so you can actually manage to implement this function. There's this implementation at the very end of the network, of the notebook. So you implement this function, which is called get-activation. You pass it the model. So the target model you want, the target layer, and the actual data you want to take a look at. And we're actually using the k function here. So we are creating a function that has the input tensor of the first layer as input, and the target layer tensor as the output tensor. This k function is dealing with tensors. So we're passing the tensor of the input layer as input to this function, and the output of this function will be recorded to the output tensor of the target layer. So we call this function on the data, and that's it. And so since these tensors are automatically connected inside the network, when you call the activation function, you pass the data through these tensors, all these tensors are connected. So they're actually calculating the activations of each tensor up to the target layer we want. This is less intuitive, I know, but it's definitely more effective, because we're actually dealing with tensor flow, Othiano, in the very end. Why would you do that? Because you want to take a look at how the network sees the data after the learning phase, and typically you end up in doing something like this. So imagine that we have already processed it, you don't have to imagine because it is what you're seeing, already processed the MNIST data. We want to get that internal representation of features, so 512 features from the 784 we had initially, and we wanted to plot this data in two dimensions. So we applied some manifold learning on it. We call it the TSNE in this piece of code, so we're actually calling it the TSNE from scikit-learn, and so we're transforming our data using these hidden features. In particular, we're just getting the first thousand samples, not all of them. And after that, we want to take a look at how these data are scrambled in the space, and with each color corresponding to the different classes we have, maybe we can see it interactively, using the bokeh on library. So let me reduce a bit the dimensions so we can see the labels, no, maybe not. So this is quite expected in some way, because for instance, zeros are all here, here you have I think all the 7s, but inside them you still have some 9s, which could make sense. You're dealing with handwritten digits, so maybe some 9s, maybe written on some 7, maybe. There's a lot of mess here, and these are the most difficult classes for the network. The black ones are the 6, and the 6 may be messed with the 5, some not really, maybe with these ones, with the 4, well, this is maybe strange. There's some mess here, so you have the tree with some 2, but yeah, this is just to take a look at what's happening inside the network, all right? This is quite handy and useful to do, and this is typically something you want to do when you want to take a look inside and what's happening during the learning, or what happened during the learning after the training phase. So unfortunately there's a lot of stuff I want to show you, but we don't have the time to do that. If you have time and you want to, take a look at the materials. Everything is on GitHub. Please give me feedback if you have it, PR if you have it, if you spot some errors, of course there will be, I will definitely be more than happy to do that. So I'm going to conclude in case you have questions. Yes, please. Yep. Okay, so the question was how the last method to visualize the id and representation. How that method works with non-sequential networks. What do you mean by no sequential networks? Yeah. Yeah. So basically you cannot have not connected graphs in Kera. Otherwise you end up with errors. That's particularly interesting because if you take a look at the implementation and how Kera detects unconnected graphs, it's bottom up approach. So it starts from the last layer and goes up to the input layer and it traverses the graphs. All the tensors it finds during the path, they are all connected to the same graphs. So at the end of the day, you always have connected tensors. You cannot have unconnected ones. And so the sequential object. The sequential object is just a Python wrapper object. At the end of the day, you always have tensor objects connected one after the other. You can indeed have tensors connected to multiple other tensors. You can have it. You can have multiple, and this is the content of this notebook, number 10, the multi-modal network. Sorry, it's very tiny. With the multi-modal networks, you can actually have multiple input networks, so accept network accepting more than one input tensor, and multiple output network, so network having a multi-objective problem, so multiple losses to optimize during the training. But at the end, you always have a connected graph, so you can always reach the target tensor you're interested in from one other tensor. There's no way some tensor is not connected to any other, I think. Yeah. Yes, but you can... Yeah. So you said network topologies may be much more complicated than the one we worked on, but there should always be a way to reach tensor inside the graph. Otherwise, it's unconnected, so you can always do that. Yes, please. So the question was about requiring your network and how you structured the data. So the difference with requiring your network is that you provide data in sequence, and so that layer expects you to provide a sequence of tensors in input, and you specify the length of the sequence. You may specify the length of the input sequence and the length of the output sequence. So what's the number of sequences you want to take into account in input, and what's the dimension of the output prediction? How many tensors you want in output? That's the difference. Yes, please. Sorry. Yeah. So the question was, in my example of the MNIST, the validation accuracy was better than the training accuracy. It shouldn't be the way around. The reason is we actually trained that network for two or three epochs, so it was very few number of epochs to train. So it's totally meaningless. It's just to show that you can watch what you can do in Keras. It's not real performances, of course, but of course, thanks for pointing out. But it's totally meaningless. Don't rely on it. It's off playground. Any other question? Okay, so, again, thank you very, very much for being here. Please let me know if there's any questions or enquiries about the materials. Enjoy with it. Thank you.
original title of this tutorial was 10 Steps to goal for this tutorial is to introduce the main our.keras and how you can build networks important features of Keras is summarizing these all of them, and the way it does is using this APIs to be used? Because this is the same API backends provided, because you're using the Keras examples, when we provided the optimizer will be dropped randomly. That's the idea of the it. One simple and most intuitive way is to actually manage to implement this function.