[Code] PyTorch sentiment classifier from scratch with Huggingface NLP Library (Full Tutorial)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper code pytorch bert pretrained lightning live tutorial pip nlp transformers tokenizers sequence sentiment imdb dataset full github
Huggingface released its newest library called NLP, which gives you easy access to almost any NLP dataset and metric in one convenient interface. We will combine this with a BERT model from Huggingface's Transformers library to build a sentiment classifier for IMDB.

OUTLINE:
0:00 - Intro
1:30 - Boilerplate
3:20 - PyTorch Lightning Module
9:50 - Load Dataset
12:15 - Tokenization
20:50 - Torch Tensors
25:50 - Data Loader
28:00 - Create BERT Model
32:00 - Implement Validation and Train Step
47:00 - Run & Recap
50:20 - Epilogue

My Code: https://github.com/yk/huggingface-nlp-demo
NLP Library: https://github.com/huggingface/nlp
Tutorial Colab: https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb
Transformers Library: https://github.com/huggingface/transformers
Pytorch Lightning: https://github.com/PyTorchLightning/pytorch-lightning

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 How did it do? So Hugging Face just released this NLP library right here and this is pretty cool because it allows you access to about a hundred NLP data sets and ten evaluation metrics pre-packaged. So knowing Hugging Face this is going to be a breeze to work with. So what I thought we would do is we would try to use this. I have not used this yet and it's been a while since I've used any Hugging Face stuff. So what we're trying to do is use this to load up the IMDB data set and then use a BERT model maybe to build a sentiment classifier on top of that using PyTorch, specifically PyTorch Lightning. So all of that combined from scratch and basically if I can do it then so can you and we're going to make some mistakes and have to look at the documentation a bit and so on but that's the process. So first of all if you like content like this let me know if you're not subscribed. Let me know in the comments if you have any sort of criticism or tips. I'm always happy for Vim tips honestly. So I have a pretty empty repo, git repo here. I have a git ignore but that's about it. So we'll just dive right in, start up Vim and let's make a file. So first some boilerplate code. I'm terrible at talking and coding at the same time but you know. So I like to use this APSAL library and I'm using as you can see I'm using the tab 9 completion engine with CoC with Neo Vim. This is absolutely great. We maybe need apps, app flags logging. That sounds good. So we'll need Torch probably right and we'll need PyTorch Lightning as PL. We'll need the NLP library of course since we're gonna use that and we'll need the Transformers library. Now I know Hugging Face has this tokenizers library too but there are some tokenizers in the transformer library already and we'll just keep it light like this. So maybe NumPy, maybe not. Let's see. So we'll export, we'll have these flags object here. Maybe we'll do some flags later and the main function. Let's just call hello. Actually let's log that info and alright. Run main. So this is our boilerplate and let's just quickly try it out just to see whether it works. So here we are. Hello. That's fine. Alright so where do we go from here? So in PyTorch Lightning what you'll have to do is you have to build this kind of model class. We'll build an IMDB sentiment classifier and that's going to extend this Lightning module of PyTorch Lightning. So you need different things in the PyTorch Lightning module. First of all you need the init and we'll just do like a very basic init. We'll call super on it and that's about it. And you need a forward method since this is a module. So in the forward method you're going to get a batch and you have to do something with it. What we also need is a training step method. Training step which gets a batch and a batch index and we'll have to output some kind of loss or some kind of training procedure. Then we'll need a train data loader. So all of this you can look up in the documentation of PyTorch Lightning. Basically you implement these methods and it will do the rest for you. So it will do all the training loop and it will do the handling of GPUs and whatnot. The whole looping over epochs. All of that is basically taken care of for you when you use PyTorch Lightning. So last thing we need is maybe a prepare data. Let's put that more up here. Prepare data. That method is optional but it gets called at the beginning and that's going to be pretty good for us. I have downloaded the weights of a BERT model and the data set so we don't need to do that anymore. That's about it. Maybe I've forgotten something. Lightning examples. There's what we're going to do. We're going to look at it like an example of PyTorch Lightning and just to see whether we'll have it. Maybe here domain examples, ImageNet sounds good. We'll have these methods. This is way more than we need but down here. Basically what you do is you instantiate your model and we won't have these hyper parameters here. These will be our flags but then you'll implement this trainer and then you call fit on the model. Let's maybe copy this down here. This is our IMDB sentiment classifier and the trainer. The root here, let's call that logs. GPUs. We'll give it a GPU if CUDA is available. Else zero. Then we'll make a flag for the epochs. We don't need the rest of this. Then at the end we'll call fit model. If we had a classifier this would already run. Now what I like to do is to have this module called SH which gives you some sort of easy shell commands. At the beginning of each run whenever the file loads I remove the logs folder. I have a clean logs folder and then I make it again like this. It just deletes the logs and then runs them again. If we run this right now this is going to give us an error. We don't have an epochs flag. We need to define a flag. Let's call define integer. We'll go for 10 epochs right now. We haven't configured our optimizers. In PyTorch Lightning you need some sort of optimizer configuration. I'll just copy that from an example. I'm going full Siraj here people. We need to configure optimizers. I like the SGD for this. It tends to work well in neural networks. We don't need the scheduler. We don't need any of that. Let's just return the SGD optimizer with the parameters and we'll make a flag for the learning rate and we'll make a flag for the momentum. We don't need any weight decay. Let's put these. We'll make floats for the learning rate. Maybe start off with something like this. I never put help strings if the description is rather clear. Only losers need help. Don't be kidding yourself. If you put the help string you need help. That's how it works. I just don't like that this library forces you to put the help string because it somehow makes me feel bad. It's very opinionated. It says basically you should put something there. We have this and now when we run this we don't have anything to optimize yet. First of all we need the model. Do we need to prepare data first? Let's check. I have this short snippet here that embeds an IPython shell. I just plug this into anywhere so I can see if I reach it. I reach the prepare data. Let's care about the data set first. This NLP library as you can see right here. There's the usage right here. You can load a data set here with the appropriate split. It will basically just give it back. If you don't have it, it will download it. It's pretty cool. We'll just load the data set. I've already checked out what they have and they have the IMDB data set. In this split argument we can say give me the train split. As a string you can say give me whatever the first 5% of the train split. This is just my laptop here. We won't be able to train a super high grade model. We'll go for 5% of the train split. This is to train data set. Now if we run until here. If you had not downloaded this, it would download this. Given the train data set, I hope you can see this. It says it's a data set. It has 1250 rows. Each entry has a text and a label. You can just index this like a data set. That's the first sample. The label is 1 here. It means that we should predict the label that this is a good sentiment. It's either 1 or 0. I think so. Either good sentiment or bad sentiment. Our first task is going to be to get this into a form where BERT can consume it. How do we do this with this NLP library? That's the pretty cool part. Right now you see this is text. In NLP we need to map this text into token IDs. We need to tokenize and we need to map this to IDs. Huggingface of course has very nice libraries for that. They're called tokenizers. We'll have one of these tokenizers. We'll use this from the transformers library. I think this is called BERT tokenizer. That the BERT models can use. Let's check it out. We're at the documentation. BERT tokenizer. There we go. There's a BERT tokenizer. Fast. Yes, okay. We'll take the fast one. Maybe not. Yeah, we'll take the fast one. Come on. Be risky. BERT tokenizer fast. I think we can do this from pre-trained. They have these methods from pre-trained. We'll take this from pre-trained. We'll put the model name here. I want to make this a flag. Such that I'm not bound to a particular model. Oops. Cool. This is called model. This is our model. BERT based on case. We have a tokenizer right now. We can now tokenize these things. Every entry in the data set. In a classic setting we'd have to write a loop for that. With this data set library, with this nlp library, it's pretty cool that we can tokenize each of the samples. We can map this tokenizer function across the training data set. How do we do that? We have this tokenizer. I'm pretty sure it has a tokenizer, an encode or something method. There's forward. This is the BERT model. Where's the BERT tokenizer? Right here. It has this encode or something. Here. Oh yeah. Encode. Where is the definition of that? Can we click on this? This encode takes text and it takes a bunch of other arguments. I hope you can see this. There we go. Whether or not you should add the special tokens or the max length. This is going to be pretty important. And pad to max length. We want everything to be of the same length. If you apply this encode function to a text of these samples. Let's just take the first sample here and let's take the text entry. Then what you're going to get is a list of these IDs. This is exactly what we want. The 101 here is this CLS token that BERT takes in. Then it's just the word pieces. You could also say instead of this say tokenize. I think. That will just give you the word pieces. Not the encodes yet. These are the word pieces right here. This is the tokenized text and with the encode function it does this. Then it maps these two IDs such that BERT can consume that. For this NLP, this library, has this convenient function called map in their data set. What we'll have to do first is define a tokenized function that takes in a single sample. Then it will run the tokenizer encode function across the text entry. We have already seen we need like... Add special tokens is true. This is cool. Max length, yes. We'll make a flag sequence length or something. We are going to pad to max length is true. Every single sample will be of the same size. In this function there's a number of ways what you can return here. One way is to return the original sample and actually just set a new attribute. I think. Set a new attribute on this original sample right here. Let's format this a bit nicer. You see we have this tokenized function. It takes a sample, it takes the text, it tokenizes it, encodes it and puts this as the new attribute input IDs and returns it again. Now what we can do is we can map this function across the training data set. This will go over the training data set and basically for each entry do this thing. Hopefully after this operation we'll have a data set where each sample not only has a text and a label but also an input IDs attribute. We don't have this sequence length thing right here. Let's put that here. Let's just go with 32 since this is just my laptop. 32 samples should be fine. Here it says can't pickle tokenizer objects. What it tries to do right here is it tries to it tries to parallelize basically this thing right here. If we look at this NLP thing, is there documentation to this? We can just look at the data sets maybe. Naming, splits, builder, arrow, data set. Map right here. This function I think it will try to multiprocess and therefore it needs to basically pickle all of the things that go into the function. It pickles all of the things that go into the function which means this tokenizer right here it needs to be pickled. Maybe there's a way to get around this. One thing we can try is we can try another tokenizer. Maybe this one can be pickled. This did library is pretty good but it can't pickle everything. This tokenizer can actually be pickled. I'm not entirely sure what you'd have to do honestly because I don't know the library but what you could do is make a thread or process local variable of this and basically make it a singleton in each process and then basically in here you call the function to get it and it returns the already instantiated object and so on. If you really want to multiprocess all of this. Anyway we have this train data set right now and you see the schema. If you can see this the schema has been extended so there is now text, there is label and there is input IDs which is a list of int64 things. That's pretty cool. So now what we can do since this is still a python list right this is still a python list. Now I know the tokenizers can already output PyTorch tensors but that's kind of cheating. So we want to use this library right here. We want the train data set. There is a method called set format right here and you say type equals torch. What that does and I think you need to say which columns you want. So we want columns. Maybe we should get all columns. Can we output the text? So you can select from the sample which of the columns you want and let's check it out again. For now as long as we're just debugging here I like to do a debug flag. So this is usually one of the first flags I do. It's define boolean debug. What this does is whenever this is active I try to be as fast as possible. So there in this PyTorch lightning trainer there's actually this fast def run argument. Which does the same thing but I can push it a bit harder with this debug here. So let me say this is like one. We'll just load batch size samples if we are in debug mode. We don't actually have a batch size argument yet do we? If flags.debug else 5 percent. So we don't have batch size yet. We're surely gonna need that at some point. So let's go with a batch size of 8 just because we can. Now if we run this in debug we should... Ah okay yes this needs to be a string. Shag-a-boom! Cool so it says it's the fast def run and if we run it in debug it just loads very few data points. So this map function here doesn't take this whole while. Maybe there's a way you can stream that I don't know. For now this is pretty good. So if we look at the train data set again you can see that it has the same entry. So this is still a list of 64 but if you index it right now if you go to the zero data point. Okay then it crashes because it tries to convert these two PyTorch tensors and it can't convert the string so we'll have to say we just want the columns input IDs and we want the label. Label can't spell. Okay let's try it again. So right here you see that what we get out is actually a PyTorch tensors for this and not kind of Python lists anymore. So this is now pretty this is one-to-one so with duck typing maybe it's even subclassed. This is a PyTorch data set right which we can load into a data loader. So this is a perfectly fine data set so we can now say self train data set is this train data set. Now we want to do this for the test as well but in order to do that we would have to write all of this code again which I'm not really in the mood so we'll just loop it. We'll create a function prepare data set and we'll take in the split name. The split name right like this and we'll just go with the split name here. That should do it and we just call it data set. Data set and return that. So now we can say train data set self dot test data set is prepare data set. For train and test. Excellent so now we have a training data set and a testing data set. So here in the train data loader we need to take the training data set and construct a data loader from it. This is super easy so what we'll do is we'll do it in one line. Data loader so what does the data loader need? The data loader needs a data set. So the prepare data is called at the beginning so we have this data set right here and I think we can go with a batch size right here and we already have a flag for that and I think there is like a drop last yes so the drop last will go for true we only want full batches during training and we'll also shuffle. Okay and the same goes for we need a validation data loader for our validation set. So in PyTorch Lightning you have train validation and test and test is really only for like the final final test. If the test data set we have here is the would be called the validation data set in PyTorch Lightning so we false here false we don't want to shuffle particularly. Okay so we have a training data loader and a validation data loader. Now what do we need? We have optimizer very good. Now what do we need? All we need to do is to actually pass our data through the BERT model. So this forward thing here we're just going to leave not implemented. Maybe we can implement it. Okay so we do need a model as you can see right here this batch let's say this batch is going to let's go right here right so if you know if you don't sometimes don't know what to do you just go to where you should be okay at ultimate empty parameter we don't have parameters yet all right so what do we do we go up here and we make a model we need to actually make the BERT model right here so from transformers we can use the BERT model now they have a lot of BERT models and we'll go back right here to the BERT models because they as you know BERT is just an encoder so we need to build a classifier on top of BERT but they already have done this so they have a bunch of BERT different configurations and the one we're looking for here would be this this BERT for sequence classification right this is BERT BERT model transformer with a sequence classification or regression head on top right so this is exactly what we need a classifier on top of BERT and we can um i think we can also load this with this from pre-trained and just put in the same name so we can this BERT for sequence classification and we'll load up the same model that we had okay so um this is our model easy as that so what do we what do we do with this BERT if we put in data what what happens for that we quickly go back again so in the forward method we can in we can input the input ids right which is batch size sequence length tensor we can input the attention mask that basically tells you where there's padding and where there isn't masks to avoid performing attention on padding token mask value selected in zero one one for tokens that are not masks zero for tokens that are masks then we can input the token type ids which we don't have here we just have one sentence but usually in BERT you have the capability of inputting two different types like a question and a paragraph or a first sentence and the second sentence um position ids are optional um blah blah blah blah blah blah blah none of that okay we could also input the labels these are optional and it would already compute a loss for us uh which we we don't this that's almost cheating so let's just focus on putting in the input ids and i think that's gonna be enough since we basically truncate our long text to 32 tokens we don't need to worry about masking right here otherwise you would input a mask for um actually we we can do it we can do it okay so what you could input a mask for basically where um your tokens are not pad tokens and the pad tokens in BERT are zero so basically your mask should just be whatever's non-zero uh but maybe also your model learns um to ignore the pad tokens i might be wrong here and it does it automatically right so in your forward pass what do you do actually let's go to the training step we'll put something here you can see it so if you if you didn't have BERT um it would actually uh BERT you it BERT you up it would download BERT right here but since i have it you can see here this is the smaller BERT model um pytorch lightning i don't have enough space in my console right here but it would give you a nice overview over your model how many parameters it has how what kind of layers it has and so on so uh we also need a validation step if we have a validation data loader validation step and we need the um validation epoch end function so usually in training you don't really care about epochs too much because you just have many batch after mini batch but in validation uh what you want is kind of one single metric across your entire test data set or validation data set and therefore you sort of in the validation step you'll just kind of output things you output local things per batch and then in the epoch end function you aggregate them into one big number so um we'll we'll we'll just put we'll put things into each thing thing thing so i'm pretty sure we're going to end up in the validation step first because if especially if we do this debug run it basically it tries to run a validation first uh at the very start of training so we can look at a batch right here so what's a batch um the batch seems to be a dictionary if you look at its keys we can see um the batch seems to be a dictionary if you look at its keys we have label and input ids okay so that's pretty cool so if we go for the input ids that gives us a tensor and the tensors of shape eight which is our batch size and 32 which is our sequence length and we should be able to pretty much input that into the BERT model that we created boom okay and what do we get out we get out a tuple and the first entry is going to be this looks like logits all right okay let's check the shape and this is eight so this is our batch size and two is the logit so one for the negative class and one for the positive class and this is this we can basically input into a cross entropy loss given our labels so we also have our label here and their label is all ones nice um is this maybe sorted is the data set sorted into good and bad things because that would be that would be bad in any case um so what do we have to do so in the forward method we get the input ids let's let's say we get the input ids and we run this through our model and we can actually construct a mask here and the mask is going to be wherever the input ids are not zero and um that as a what does it need to be so these mask this attention mask is going to be a float tensor okay so we'll put it as a float tensor cool um right like this so our logits are going to be that and yeah tuple with one entry so the comma here is important we're going to return the logits so this is our forward function so in the validation and the training step the first thing we got to do is we got to uh call this forward function with the input ids and these of course are in our batch like this so these are going to be our logits and then in the validation what we want to do is we first of all want to compute our loss right so we have to construct this up here in the init we can actually fold this prepare data um loss is going to be a cross entropy loss yes that exists with read reduction i like to put reduction none i don't think there's like an a deprecated reduce and there is like a reduction where you can put mean or something i like to not reduce the loss at first because then i can agro i can use the same thing for validation and training so in the validation step i just want to compute my loss right here with self so loss loss um and we'll have to cheat a bit so look up the cross entropy loss and come on okay where is the cross entropy loss loss cross entropy loss it takes yes it's reduction ha tada and so the input to the function that we construct is going to be first um n by c first the input and then the targets so first the logits and then the targets right criterion that combines logs of max and nl loss over a single class nice nice nice okay okay cool so first logits and then labels label okay that's our loss so if we check out what our loss is going to be it's probably going to be an vector of size eight because we have reduction none none loss yes c vector of size eight very nice so we can just um basically return i'll say we can return this loss just as is and then in the validation epoch end the outputs here is going to be a list of and every entry in the list is going to be one of these validation steps for for one batch so we can aggregate those so losses is will concatenate them since they're going to be chunks of eight outputs at the dimension zero and then we can calculate the mean right so um we can do that and then we can oh no we need to do more we also want to know the accuracy right so um the accuracy is going to be whether or not the logits dot arg max is go is equal to the label label so the accuracy for each sample is going to be that it's either going to be one or zero and we want that as a float so here let's output a dictionary with loss um and accuracy all right excellent so here then we can aggregate so the loss is going to be and i like to have um like a construction here that aggregates this still so we go out loss for o in outputs so these are now going to be uh entries each one is going to be a dictionary right so our loss losses we have concatenation to the mean okay our accuracy is going to be the same thing for the accuracy nice so our output here is going to be a dictionary and i think in pytorch lightning there there if you put validation accuracy select valac it selects the model according to this but i'm not sure so also in pytorch lightning i can now output this here but also if you have a log entry it will forward this to the logger which is the logger which we can uh actually do and make a tensor board logger out of this so what have we done we have first of all set up the validation step so the the pytorch lightning is going to run through the data loader for each batch do this so we forward it through the bert model to get our log it's and then we compute our loss by the cross entropy loss of the log it's and the labels and we also compute our accuracy by seeing how much the log it's agree with the labels or the maximum log it and then we aggregate all of this over the entire epoch and output that now let's set up a logger so for the logger we can put this i think in the trainer here pytorch lightning logger dot and i think there is a tensor board logger uh pretty sure pytorch lightning is there tensor board no pytorch nying logger i'm pretty sure that exists that's not the newest version i hate these these old docs so latest come on oh this was called logging logger log loggers tensor board logger right here nice so our save dear is going to be called logs and then what we what do we want we want the name imdb and there's also this version thing where if if if you don't put version zero it will just make a new kind of folder each time but i guess we delete the logs anyway we delete the logs folder at the beginning so we don't have this problem but i generally like to overwrite my logs and not make new runs but if you like something different that's you know fine all right so let's run this again and we're cool though this is the bird configuration that we loaded and then we have no attribute logger pytorch lightning loggers loggers loggers okay again loading the weights very cool blah blah blah blah blah blah blah blah and we're in the night python shell and do we have a night python shell remaining only in the training step okay so we're at the training step right here and we can actually can we can check whether or not um ah now we have lightning logs and logs my okay so these appear to be our tensor board logs so we are maybe able to run the tensor board here later um let's run it logs we don't have tensor board okay oh yeah i've uninstalled it because i was angry at it oh come on what's going on um tensor board i should have tensor board somewhere uh it's it's like in um in local bin or something um in local bin or something local bin no it's not in local bin oh oh we'll find it we'll figure it out uh how to get a tensor board maybe we need to install tensor flow well that's gonna take a while okay so back to the training step in the training step we basically need to do the same as in the validation step so we'll need to forward our batch through the model but here we don't need to compute an accuracy but we do need to compute a actually a batch loss that we can back propagate on now in the training step you can either specify how you back propagate um per se or what you can do is you can just output this log loss attribute and then pytorch lightning will basically do the back propagation for you we have the tensor board now please all right there we go and we can we can put this into a different uh thing right here um git uh lp demo yes um okay so this is running and if everything goes correctly 06 shaboom we have a tensor board okay so we need to forward our training step and we need to calculate a loss for the batch so these loss here we do the same thing but here we call mean on it so this is the mean loss from this batch and we can now return um the loss right here and we can also in the training step you can also output a log dictionary and we'll output the loss again here in order so this is our going to be our training loss that we output right here um let's call it train loss and this also will go into the tensor board so if we run this right now we don't have an ipython shell simply by outputting this loss attribute we already instruct pytorch lightning to now run backprop on this loss uh using the optimizer that we have defined okay and by outputting the log we instructed to put this into the tensor board so now we have a scalar entry and um you can see this it only contains the valid no it contains everything very cool very very cool so let's remove the debug flag and we'll just see what happens so to recap right to recap we have um oh now you go see epoch one epoch two go go go go go ah very cool um what we've done is we've set up this pytorch lightning module it needs a bunch of functions but in the init we've basically just set up our bird model from the hogging face transformers library we've loaded in a pre-trained bird model that we're going to fine tune the main thing that the pytorch lightning module needs is a training step function where you define what it should do with the data and this data loader function so in the data loader function um we've loaded up a data set um and we basically specified the batch size this is very easy where does the data set come from we do it here in prepare data this is a special function in pytorch lightning that's basically called after the init but before anything else runs and here we are loading this data set from the nlp library and this is kind of the magic part we specify the split and the size that we want inside of the string and you can do this in percent or in a number of samples that you want i'm sort of sure you can do more things but i haven't explored that then we run map on the data set in order to tokenize it and that's right here and we use a tokenizer again from the pytorch lightning and um just run this encode function this is very simple like if how complicated was this just like a year ago crazy then we need to to put set format and set format tells the data set how it needs to output its samples and we tell it please output torch tensors and um we want these columns right here and we make a train and test data set with from the train and test split accordingly so we have this this goes into a data loader pytorch lightning will take the data loader and run training on it using this train step function in this train step function we get a batch um in the batch there are these two columns that we specified previously input ids and label the input ids will put through the forward function of the model itself this is the forward function we'll construct a mask and run it through the model um we wouldn't actually need to construct a mask but okay and we get back the logits of the classification and then we run this through a cross entropy loss uh get the mean of the batch and there we go in the validation step we do the same thing but also calculate the accuracy but don't calculate the mean we want to keep it per sample and only at the end we want to concatenate everything and calculate the mean if we've done everything correctly you see right here our train loss goes down down down until it is almost zero because we've just and the validation accuracy is super high is this is this because all the labels are equal okay so we have a all the labels are equal like for real um okay so we'll do something else we'll make an integer um with percent and this was five right so that we loaded five percent of the data set um but let's load some more and this might take longer but let's load 50 percent of the data set and just see what happens no present i called it present very good so we'll load up 50 percent of the data set and um we'll do the same thing and we can track in real time what happens in tensorboard and unrecognized instruction format um okay who can we make a format string in a format string this is nasty does it work please work we can make a format string in a format string absolutely bonkers okay so it takes a little bit longer and you could actually i think you can speed this up this mapping of the data set maybe you can stream it um i'm pretty sure you can batch it you can do a batch um processing of this but for our case right here uh we think it's enough so it was like what 1200 if we had five percent so now it should be something like 12 000 um so let's continue with the recap of what we did here we have the train data set the validation data set and on yes so we have everything like this the configure optimizers you can put an optimizer you can also put a learning rate scheduler if you want to and then in the main function we load this pytorch lightning module and we specify a trainer and the trainer we tell it you know the max epochs and so on and we set up the logger and we just run fit on this model and this runs epochs of the model and after each epoch it does a validation epoch and uh minimizes our loss very cool very effective so now if if please if you would all right here we go this is my laptop training burnt oh okay we don't seem to make too much progress let's check the tensor board training loss goes down training loss goes to zero training loss goes down training loss goes to zero i have the sneaking suspicion that this is not entirely shuffled so um is there like a shuffle like a shuffle thing because this seems a bit this seems a bit bit fishy um this imdb data set right here it just seems like you know we could use a bit of shuffling because all the labels yeah the training loss instantly goes to zero so maybe maybe it's not we can we shuffle here let's look at the load data set function load data set batched uh no keeping memory no none of that okay this does not seem to go to continue right here data sets nlp data sets i hope here we know we should find this load data set somewhere builder features load load data set split can we not shuffle anywhere we'll search shuffle builder okay so generate examples this function pre-processed examples key will be hashed okay we are not able to shuffle this um just like that and we can't do that okay we are not able to shuffle this um just like that but i hope this at least gives you an impression i guess if you were to take the full data set and map it then it would actually work we'll just try again with 10% of the data just uh to see it go down tensorboard see this is now good because we always delete the logs folder we don't have any uh remnant uh old tensorflow logs all right come on come on so 10% should be yeah about this about this okay train loss looking good looking good so far looking good looking good so far look at these models how large is that how large is the bert base case hugging face pre-trained models pre-trained models bert based on case that's the one we have 12 layers 110 million parameters easy easy easy oh no it's too large training loss goes to zero again okay so we've determined that this data set very probably isn't entirely shuffled it might just have all the good labels first and all the bad labels last and um yeah just to confirm let's confirm this uh right here let's go with 100% but let's put an ipython shell down um just before we map the data set so we don't have to go through the whole mapping procedure actually that would be here right yes can we not map this asynchronously map i might be doing something really wrong with this library but i think that's that's how it should go so map def map right here we can do batched we could do batched and then i think hugging face has a function to encode batched encode batch encode encode batch no um let's go to the tokenizer build inputs create token type ids get special token mask save where is encode code right here can we have batch encode build inputs no this might be it batch encode yes there is a batch encode where you have batches of these things so okay what if we do the negative one see here's the label zero um i'm pretty sure i'm pretty sure uh batch true let's do that and in our function here we'll say batch encode so let's see how fast this is with 100% where tokenizer has no but we just had batch encode oh but this might be we have batch encode plus batch encode plus or text pairs okay we need this batch encode plus but then that gives us a dictionary right this gives us a dictionary with the fields input ids right here so like this how about that and then we still might want to limit the actual data set um once we have once we have mapped it because we need to train on it as well but i just want to see how fast this batch uh encoding is yes okay reasonably fast but it takes like three minutes um yeah so we won't go on here i will put i will put this as is on um i'll put this as is on github and i will put this as is on github and i hope you can profit from that in any way you want the hugging face site has a tutorial on squad where they also use the metrics so they have basically these pre-defined metrics like blur or rouge i think and you can just use them as you use these data sets so it's very very very convenient to work with these things in nlp so nlp has come a long way absolutely invite you to check out the um the transformers and tokenizers and nlp repos and with that that's it for me i think i hope you enjoyed this again leave a comment if you see improvements or if i maybe should edit this a bit more or if i should add a little bit more see improvements or if i maybe should edit this a bit more i thought the entire process of just going through and making mistakes um would be entertaining to some all right bye bye
 How did it do? So Hugging Face just released this NLP library right here and
 this is pretty cool because it allows you access to about a hundred NLP data
 sets and ten evaluation metrics pre-packaged. So knowing Hugging Face
 this is going to be a breeze to work with. So what I thought we would do is we
 would try to use this. I have not used this yet and it's been a while since
 I've used any Hugging Face stuff. So what we're trying to do is use this to load
 up the IMDB data set and then use a BERT model maybe to build a sentiment
 classifier on top of that using PyTorch, specifically PyTorch Lightning. So all
 of that combined from scratch and basically if I can do it then so can you
 and we're going to make some mistakes and have to look at the
 documentation a bit and so on but that's the process. So first of all if
 you like content like this let me know if you're not subscribed. Let
 me know in the comments if you have any sort of criticism or tips. I'm always
 happy for Vim tips honestly. So I have a pretty empty repo, git repo here. I have
 a git ignore but that's about it. So we'll just dive right in, start up Vim
 and let's make a file. So first some boilerplate code. I'm terrible at
 talking and coding at the same time but you know. So I like to use this APSAL
 library and I'm using as you can see I'm using the tab 9 completion engine
 with CoC with Neo Vim. This is absolutely great. We maybe need apps, app flags
 logging. That sounds good. So we'll need Torch probably right and we'll
 need PyTorch Lightning as PL. We'll need the NLP library of course since
 we're gonna use that and we'll need the Transformers library. Now I know
 Hugging Face has this tokenizers library too but there are some tokenizers in the
 transformer library already and we'll just keep it light like this. So maybe
 NumPy, maybe not. Let's see. So we'll export, we'll have these flags object
 here. Maybe we'll do some flags later and the main function. Let's just call hello.
 Actually let's log that info and alright. Run main. So this is our boilerplate
 and let's just quickly try it out just to see whether it works. So here we are.
 Hello. That's fine. Alright so where do we go from here? So in PyTorch Lightning
 what you'll have to do is you have to build this kind of model class.
 We'll build an IMDB sentiment classifier and that's going to extend this Lightning
 module of PyTorch Lightning. So you need different things in the PyTorch Lightning
 module. First of all you need the init and we'll just do like a very basic
 init. We'll call super on it and that's about it. And you need a forward method
 since this is a module. So in the forward method you're going to get a batch and
 you have to do something with it. What we also need is a training step method.
 Training step which gets a batch and a batch index and we'll have to output
 some kind of loss or some kind of training procedure. Then we'll need a
 train data loader. So all of this you can look up in the
 documentation of PyTorch Lightning. Basically you implement these methods
 and it will do the rest for you. So it will do all the training loop and it
 will do the handling of GPUs and whatnot. The whole looping over epochs. All of
 that is basically taken care of for you when you use PyTorch Lightning. So last
 thing we need is maybe a prepare data. Let's put that more up here. Prepare data.
 That method is optional but it gets called at the beginning and that's going
 to be pretty good for us. I have downloaded the weights of a BERT model
 and the data set so we don't need to do that anymore.
 That's about it. Maybe I've forgotten something.
 Lightning examples. There's what we're going to do. We're going to look at it
 like an example of PyTorch Lightning and just to see whether we'll have it.
 Maybe here domain examples, ImageNet sounds good. We'll have these methods.
 This is way more than we need but down here. Basically what you do is you
 instantiate your model and we won't have these hyper parameters here.
 These will be our flags but then you'll implement this trainer and then you call
 fit on the model. Let's maybe copy this down here.
 This is our IMDB sentiment classifier and the trainer.
 The root here, let's call that logs. GPUs. We'll give it a GPU if CUDA is available.
 Else zero. Then we'll make a flag for the epochs. We don't need the rest of this.
 Then at the end we'll call fit model. If we had a classifier this
 would already run. Now what I like to do is to have this module
 called SH which gives you some sort of easy shell commands. At the beginning
 of each run whenever the file loads I remove the logs folder.
 I have a clean logs folder and then I make it again like this.
 It just deletes the logs and then runs them again. If we run this right now
 this is going to give us an error. We don't have an epochs flag.
 We need to define a flag. Let's call define integer. We'll go for 10 epochs right now.
 We haven't configured our optimizers.
 In PyTorch Lightning you need some sort of optimizer configuration.
 I'll just copy that from an example. I'm going full Siraj here people.
 We need to configure optimizers. I like the SGD for this. It tends to work well in neural networks.
 We don't need the scheduler. We don't need any of that.
 Let's just return the SGD optimizer with the parameters and we'll make a flag for
 the learning rate and we'll make a flag for the momentum. We don't need any weight decay.
 Let's put these. We'll make floats for the learning rate.
 Maybe start off with something like this. I never put help strings if the description is rather clear.
 Only losers need help. Don't be kidding yourself.
 If you put the help string you need help. That's how it works.
 I just don't like that this library forces you to put the help string because it somehow makes me feel bad.
 It's very opinionated. It says basically you should put something there.
 We have this and now when we run this we don't have anything to optimize yet.
 First of all we need the model. Do we need to prepare data first?
 Let's check. I have this short snippet here that embeds an IPython shell.
 I just plug this into anywhere so I can see if I reach it.
 I reach the prepare data. Let's care about the data set first.
 This NLP library as you can see right here. There's the usage right here.
 You can load a data set here with the appropriate split.
 It will basically just give it back. If you don't have it, it will download it.
 It's pretty cool. We'll just load the data set.
 I've already checked out what they have and they have the IMDB data set.
 In this split argument we can say give me the train split.
 As a string you can say give me whatever the first 5% of the train split.
 This is just my laptop here.
 We won't be able to train a super high grade model.
 We'll go for 5% of the train split. This is to train data set.
 Now if we run until here.
 If you had not downloaded this, it would download this.
 Given the train data set, I hope you can see this.
 It says it's a data set. It has 1250 rows.
 Each entry has a text and a label.
 You can just index this like a data set.
 That's the first sample. The label is 1 here.
 It means that we should predict the label that this is a good sentiment.
 It's either 1 or 0.
 I think so. Either good sentiment or bad sentiment.
 Our first task is going to be to get this into a form where BERT can consume it.
 How do we do this with this NLP library? That's the pretty cool part.
 Right now you see this is text.
 In NLP we need to map this text into token IDs.
 We need to tokenize and we need to map this to IDs.
 Huggingface of course has very nice libraries for that.
 They're called tokenizers.
 We'll have one of these tokenizers.
 We'll use this from the transformers library.
 I think this is called BERT tokenizer.
 That the BERT models can use. Let's check it out.
 We're at the documentation.
 BERT tokenizer. There we go. There's a BERT tokenizer.
 Fast.
 Yes, okay. We'll take the fast one.
 Maybe not.
 Yeah, we'll take the fast one. Come on. Be risky.
 BERT tokenizer fast.
 I think we can do this from pre-trained.
 They have these methods from pre-trained.
 We'll take this from pre-trained.
 We'll put the model name here.
 I want to make this a flag.
 Such that I'm not bound to a particular model.
 Oops.
 Cool.
 This is called model.
 This is our model. BERT based on case.
 We have a tokenizer right now.
 We can now tokenize these things.
 Every entry in the data set.
 In a classic setting we'd have to write a loop for that.
 With this data set library, with this nlp library,
 it's pretty cool that we can tokenize
 each of the samples. We can map this
 tokenizer function across the training data set.
 How do we do that?
 We have this tokenizer.
 I'm pretty sure it has a tokenizer, an encode or something method.
 There's forward. This is the BERT model.
 Where's the BERT tokenizer?
 Right here.
 It has this encode or something.
 Here. Oh yeah. Encode.
 Where is the definition of that?
 Can we click on this?
 This encode takes text and it takes a bunch of other arguments.
 I hope you can see this.
 There we go.
 Whether or not you should add the special tokens
 or the max length. This is going to be pretty important.
 And pad to max length. We want everything to be of the same length.
 If you apply this encode function to a text of these samples.
 Let's just take the first sample here and let's take the text entry.
 Then what you're going to get is a list of these IDs.
 This is exactly what we want.
 The 101 here is this CLS token that BERT takes in.
 Then it's just the word pieces.
 You could also say instead of this say tokenize.
 I think. That will just give you the word pieces.
 Not the encodes yet.
 These are the word pieces right here.
 This is the tokenized text and with the encode function it does this.
 Then it maps these two IDs such that BERT can consume that.
 For this NLP, this library, has this convenient function called map in their data set.
 What we'll have to do first is define a tokenized function that takes in a single sample.
 Then it will run the tokenizer encode function across the text entry.
 We have already seen we need like...
 Add special tokens is true. This is cool. Max length, yes.
 We'll make a flag sequence length or something.
 We are going to pad to max length is true.
 Every single sample will be of the same size.
 In this function there's a number of ways what you can return here.
 One way is to return the original sample and actually just set a new attribute.
 I think.
 Set a new attribute on this original sample right here.
 Let's format this a bit nicer.
 You see we have this tokenized function. It takes a sample, it takes the text,
 it tokenizes it, encodes it and puts this as the new attribute input IDs and returns it again.
 Now what we can do is we can map this function across the training data set.
 This will go over the training data set and basically for each entry do this thing.
 Hopefully after this operation we'll have a data set
 where each sample not only has a text and a label but also an input IDs attribute.
 We don't have this sequence length thing right here.
 Let's put that here.
 Let's just go with 32 since this is just my laptop.
 32 samples should be fine.
 Here it says can't pickle tokenizer objects.
 What it tries to do right here is it tries to
 it tries to
 parallelize basically this thing right here.
 If we look at this NLP thing, is there documentation to this?
 We can just look at the data sets maybe.
 Naming, splits, builder, arrow, data set.
 Map right here.
 This function I think it will try to multiprocess and therefore it needs to basically
 pickle all of the things that go into the function.
 It pickles all of the things that go into the function which means this tokenizer right here
 it needs to be pickled.
 Maybe there's a way to get around this.
 One thing we can try is we can try another tokenizer.
 Maybe this one can be pickled.
 This did library is pretty good but it can't pickle everything.
 This tokenizer can actually be pickled.
 I'm not entirely sure what you'd have to do honestly
 because I don't know the library but what you could do is make a thread or
 process local variable of this and basically make it a singleton in each
 process and then basically in here you call the function to get it
 and it returns the already instantiated object and so on.
 If you really want to multiprocess all of this.
 Anyway we have this train data set right now and you see the schema.
 If you can see this the schema has been extended so there is now text, there is
 label and there is input IDs which is a list of int64 things.
 That's pretty cool.
 So now what we can do since this is still a python list right this is still a python list.
 Now I know the tokenizers can already output PyTorch tensors but that's kind of cheating.
 So we want to use this library right here.
 We want the train data set.
 There is a method called set format right here and you say type equals torch.
 What that does and I think you need to say which columns you want.
 So we want columns.
 Maybe we should get all columns.
 Can we output the text?
 So you can select from the sample which of the columns you want and let's check it out again.
 For now as long as we're just debugging here I like to do a debug flag.
 So this is usually one of the first flags I do.
 It's define boolean debug.
 What this does is whenever this is active I try to be as fast as possible.
 So there in this PyTorch lightning trainer there's actually this fast def run argument.
 Which does the same thing but I can push it a bit harder with this debug here.
 So let me say this is like one.
 We'll just load batch size samples if we are in debug mode.
 We don't actually have a batch size argument yet do we?
 If flags.debug else 5 percent.
 So we don't have batch size yet.
 We're surely gonna need that at some point.
 So let's go with a batch size of 8 just because we can.
 Now if we run this in debug we should...
 Ah okay yes this needs to be a string.
 Shag-a-boom!
 Cool so it says it's the fast def run and if we run it in debug it just loads very few data points.
 So this map function here doesn't take this whole while.
 Maybe there's a way you can stream that I don't know.
 For now this is pretty good.
 So if we look at the train data set again you can see that it has the same entry.
 So this is still a list of 64 but if you index it right now if you go to the zero data point.
 Okay then it crashes because it tries to convert these two PyTorch tensors
 and it can't convert the string so we'll have to say we just want the columns input IDs
 and we want the label.
 Label can't spell.
 Okay let's try it again.
 So right here you see that what we get out is actually a PyTorch tensors for this
 and not kind of Python lists anymore.
 So this is now pretty this is one-to-one so with duck typing maybe it's even subclassed.
 This is a PyTorch data set right which we can load into a data loader.
 So this is a perfectly fine data set so we can now say self train data set is this train data set.
 Now we want to do this for the test as well but in order to do that we would have to write
 all of this code again which I'm not really in the mood so we'll just loop it.
 We'll create a function prepare data set and we'll take in the split name.
 The split name right like this and we'll just go with the split name here.
 That should do it and we just call it data set.
 Data set and return that.
 So now we can say train data set self dot test data set is prepare data set.
 For train and test.
 Excellent so now we have a training data set and a testing data set.
 So here in the train data loader we need to take the training data set
 and construct a data loader from it. This is super easy so what we'll do is we'll do it in one line.
 Data loader so what does the data loader need? The data loader needs a data set.
 So the prepare data is called at the beginning so we have this data set right here and I think we
 can go with a batch size right here and we already have a flag for that and I think there is like a
 drop last yes so the drop last will go for true we only want full batches during training and we'll
 also shuffle. Okay and the same goes for we need a validation data loader for our validation set.
 So in PyTorch Lightning you have train validation and test and test is really only for like the
 final final test. If the test data set we have here is the would be called the validation data
 set in PyTorch Lightning so we false here false we don't want to shuffle particularly. Okay
 so we have a training data loader and a validation data loader.
 Now what do we need? We have optimizer very good.
 Now what do we need? All we need to do is to actually pass our data through the BERT model.
 So this forward thing here we're just going to leave not implemented.
 Maybe we can implement it. Okay so we do need a model as you can see right here this batch
 let's say this batch is going to let's go right here right so if you know if you don't sometimes
 don't know what to do you just go to where you should be okay at ultimate empty parameter we
 don't have parameters yet all right so what do we do we go up here and we make a model we need to
 actually make the BERT model right here so from transformers we can use the BERT model now they
 have a lot of BERT models and we'll go back right here to the BERT models because they as you know
 BERT is just an encoder so we need to build a classifier on top of BERT but they already have
 done this so they have a bunch of BERT different configurations and the one we're looking for here
 would be this this BERT for sequence classification right this is BERT BERT model transformer with a
 sequence classification or regression head on top right so this is exactly what we need a classifier
 on top of BERT and we can um i think we can also load this with this from pre-trained and just put
 in the same name so we can this BERT for sequence classification and we'll load up the same model
 that we had okay so um this is our model easy as that so what do we what do we do with this BERT if
 we put in data what what happens for that we quickly go back again so in the forward method we can
 in we can input the input ids right which is batch size sequence length tensor we can input the
 attention mask that basically tells you where there's padding and where there isn't
 masks to avoid performing attention on padding token mask value selected in zero one one for tokens
 that are not masks zero for tokens that are masks then we can input the token type ids which we don't
 have here we just have one sentence but usually in BERT you have the capability of inputting two
 different types like a question and a paragraph or a first sentence and the second sentence
 um position ids are optional um blah blah blah blah blah blah blah none of that okay we could also
 input the labels these are optional and it would already compute a loss for us uh which we we don't
 this that's almost cheating so let's just focus on putting in the input ids and i think that's
 gonna be enough since we basically truncate our long text to 32 tokens we don't need to worry about
 masking right here otherwise you would input a mask for um actually we we can do it we can do it
 okay so what you could input a mask for basically where um your tokens are not pad tokens and the
 pad tokens in BERT are zero so basically your mask should just be whatever's non-zero uh but
 maybe also your model learns um to ignore the pad tokens i might be wrong here and it does it
 automatically right so in your forward pass what do you do actually let's go to the training step
 we'll put something here you can see it so if you if you didn't have BERT um it would actually
 uh BERT you it BERT you up it would download BERT right here but since i have it you can see here
 this is the smaller BERT model um pytorch lightning i don't have enough space in my console
 right here but it would give you a nice overview over your model how many parameters it has how
 what kind of layers it has and so on so uh we also need a validation step if we have a validation
 data loader validation step and we need the um validation epoch end function so
 usually in training you don't really care about epochs too much because you just have many batch
 after mini batch but in validation uh what you want is kind of one single metric across your entire
 test data set or validation data set and therefore you sort of in the validation step you'll just
 kind of output things you output local things per batch and then in the epoch end function you
 aggregate them into one big number so um we'll we'll we'll just put
 we'll put things into each thing thing thing so i'm pretty sure we're going to end up in the
 validation step first because if especially if we do this debug run it basically it tries to
 run a validation first uh at the very start of training so we can look at a batch right here
 so what's a batch um the batch seems to be a dictionary if you look at its keys we can see
 um the batch seems to be a dictionary if you look at its keys we have label and input ids okay so
 that's pretty cool so if we go for the input ids that gives us a tensor and the tensors of shape
 eight which is our batch size and 32 which is our sequence length and we should be able to pretty
 much input that into the BERT model that we created boom okay and what do we get out we get out a tuple
 and the first entry is going to be this looks like logits all right okay let's check the shape
 and this is eight so this is our batch size and two is the logit so one for the negative class
 and one for the positive class and this is this we can basically input into a cross entropy loss
 given our labels so we also have our label here and their label is all ones nice um is this maybe
 sorted is the data set sorted into good and bad things because that would be that would be bad
 in any case um so what do we have to do so in the forward method we get the input ids let's let's
 say we get the input ids and we run this through our model and we can actually construct a mask
 here and the mask is going to be wherever the input ids are not zero and um that as a what does
 it need to be so these mask this attention mask is going to be a float tensor okay so we'll put it
 as a float tensor cool um right like this so our logits are going to be that and yeah tuple with
 one entry so the comma here is important we're going to return the logits so this is our forward
 function so in the validation and the training step the first thing we got to do is we got to
 uh call this forward function with the input ids and these of course are in our batch
 like this so these are going to be our logits and then in the validation what we want to do is we
 first of all want to compute our loss right so we have to construct this up here in the init
 we can actually fold this prepare data um loss is going to be a cross entropy loss yes that exists
 with read reduction i like to put reduction none i don't think there's like an a deprecated reduce
 and there is like a reduction where you can put mean or something i like to not reduce the loss
 at first because then i can agro i can use the same thing for validation and training
 so in the validation step i just want to compute my loss right here with self so loss loss um and
 we'll have to cheat a bit
 so look up the cross entropy loss and
 come on okay where is the cross entropy loss
 loss cross entropy loss it takes yes it's reduction ha tada and
 so the input to the function that we construct is going to be first um
 n by c first the input and then the targets so first the logits and then the targets right
 criterion that combines logs of max and nl loss over a single class nice nice nice okay okay cool
 so first logits and then labels label okay that's our loss so if we check out what our loss is going
 to be it's probably going to be an vector of size eight because we have reduction none
 none loss yes c vector of size eight very nice so we can just um basically return
 i'll say we can return this loss just as is and then in the validation epoch end the outputs here
 is going to be a list of and every entry in the list is going to be one of these validation steps
 for for one batch so we can aggregate those so losses is will concatenate them since they're
 going to be chunks of eight outputs at the dimension zero and then we can calculate the mean right so
 um we can do that and then
 we can oh no we need to do more we also want to know the accuracy right so um the accuracy is
 going to be whether or not the logits dot arg max is go is equal to the label label
 so the accuracy for each sample is going to be that it's either going to be one or zero and
 we want that as a float so here let's output a dictionary with loss um and accuracy all right
 excellent so here then we can aggregate so the loss is going to be and i like to have um like
 a construction here that aggregates this still so we go out loss for o in outputs so these are now
 going to be uh entries each one is going to be a dictionary right so our loss losses we have
 concatenation to the mean okay our accuracy is going to be the same thing for the accuracy
 nice so our output here is going to be a dictionary
 and i think in pytorch lightning there there if you put validation accuracy select valac
 it selects the model according to this but i'm not sure so also in pytorch lightning i can now
 output this here but also if you have a log entry it will forward this to the logger which is
 the logger which we can uh actually do and make a tensor board logger out of this so what have we
 done we have first of all set up the validation step so the the pytorch lightning is going to
 run through the data loader for each batch do this so we forward it through the bert model to
 get our log it's and then we compute our loss by the cross entropy loss of the log it's and the
 labels and we also compute our accuracy by seeing how much the log it's agree with the labels or the
 maximum log it and then we aggregate all of this over the entire epoch and output that now let's
 set up a logger so for the logger we can put this i think in the trainer here pytorch lightning
 logger dot and i think there is a tensor board logger uh pretty sure pytorch lightning is there
 tensor board no pytorch
 nying logger i'm pretty sure that exists that's not the newest version i hate these
 these old docs so latest come on oh this was called logging logger
 log
 loggers
 tensor board logger right here nice so our save dear is going to be called logs and then
 what we what do we want we want the name imdb and there's also this version thing
 where if if if you don't put version zero it will just make a new kind of folder each time but i
 guess we delete the logs anyway we delete the logs folder at the beginning so we don't have
 this problem but i generally like to overwrite my logs and not make new runs but if you like
 something different that's you know fine all right so let's run this again and we're cool
 though this is the bird configuration that we loaded and then we have no attribute logger
 pytorch lightning loggers loggers
 loggers
 okay again loading the weights very cool blah blah blah blah blah blah blah blah and we're in
 the night python shell and do we have a night python shell remaining only in the training step
 okay so we're at the training step right here and we can actually can we can check whether or not
 um ah now we have lightning logs and logs my
 okay so these appear to be our tensor board logs so we are maybe able to run the tensor board here
 later um let's run it logs we don't have tensor board okay
 oh yeah i've uninstalled it because i was angry at it oh come on what's going on um
 tensor board i should have tensor board somewhere uh it's it's like in um in local bin or something
 um in local bin or something local bin no it's not in local bin
 oh oh we'll find it we'll figure it out
 uh how to get a tensor board maybe we need to install tensor flow
 well that's gonna take a while okay so back to the training step in the training step we
 basically need to do the same as in the validation step so we'll need to forward our batch through
 the model but here we don't need to compute an accuracy but we do need to compute a actually a
 batch loss that we can back propagate on now in the training step you can either specify how you
 back propagate um per se or what you can do is you can just output this log loss attribute and then
 pytorch lightning will basically do the back propagation for you we have the tensor board
 now please all right there we go and we can we can put this into a different uh thing right here
 um git uh lp demo yes um
 okay so this is running and if everything goes correctly
 06 shaboom we have a tensor board okay so we need to forward our training step and we need to
 calculate a loss for the batch so these loss here we do the same thing but here we call mean on it
 so this is the mean loss from this batch and we can now return um the loss right here and we can
 also in the training step you can also output a log dictionary and we'll output the loss again
 here in order so this is our going to be our training loss that we output right here um let's
 call it train loss and this also will go into the tensor board so if we run this right now we don't
 have an ipython shell simply by outputting this loss attribute we already instruct pytorch lightning
 to now run backprop on this loss uh using the optimizer that we have defined okay
 and by outputting the log we instructed to put this into the tensor board so now we have a scalar
 entry and um you can see this it only contains the valid no it contains everything very cool very very
 cool so let's remove the debug flag and we'll just see what happens
 so to recap right to recap we have um oh now you go see epoch one epoch two go go go go go
 ah very cool um what we've done is we've set up this pytorch lightning module it needs a bunch of
 functions but in the init we've basically just set up our bird model from the hogging face
 transformers library we've loaded in a pre-trained bird model that we're going to fine tune the main
 thing that the pytorch lightning module needs is a training step function where you define what it
 should do with the data and this data loader function so in the data loader function um we've
 loaded up a data set um and we basically specified the batch size this is very easy
 where does the data set come from we do it here in prepare data this is a special function in
 pytorch lightning that's basically called after the init but before anything else runs and here
 we are loading this data set from the nlp library and this is kind of the magic part
 we specify the split and the size that we want inside of the string and you can do this in percent
 or in a number of samples that you want i'm sort of sure you can do more things but i haven't
 explored that then we run map on the data set in order to tokenize it and that's right here
 and we use a tokenizer again from the pytorch lightning and um just run this encode function
 this is very simple like if how complicated was this just like a year ago crazy then we need to
 to put set format and set format tells the data set how it needs to output its samples and we tell
 it please output torch tensors and um we want these columns right here and we make a train and
 test data set with from the train and test split accordingly so we have this this goes into a data
 loader pytorch lightning will take the data loader and run training on it using this train step
 function in this train step function we get a batch um in the batch there are these two columns
 that we specified previously input ids and label the input ids will put through the forward function
 of the model itself this is the forward function we'll construct a mask and run it through the model
 um we wouldn't actually need to construct a mask but okay and we get back the logits of the
 classification and then we run this through a cross entropy loss uh get the mean of the batch
 and there we go in the validation step we do the same thing but also calculate the accuracy but
 don't calculate the mean we want to keep it per sample and only at the end we want to concatenate
 everything and calculate the mean if we've done everything correctly you see right here our train
 loss goes down down down until it is almost zero because we've just and the validation accuracy is
 super high is this is this because all the labels are equal okay so we have a
 all the labels are equal like for real um okay so we'll do something else we'll make an integer
 um with percent and this was five right so that we loaded five percent of the data set
 um but let's load some more and this might take longer but let's load
 50 percent of the data set and just see what happens
 no present i called it present
 very good
 so we'll load up 50 percent of the data set and um we'll do the same thing and we can track in
 real time what happens in tensorboard and unrecognized instruction format um
 okay
 who can we make a format string in a format string this is nasty does it work
 please work we can make a format string in a format string absolutely bonkers okay so it takes a
 little bit longer and you could actually i think you can speed this up this mapping of the data set
 maybe you can stream it um i'm pretty sure you can batch it you can do a batch um processing of this
 but for our case right here uh we think it's enough so it was like what 1200 if we had five percent so
 now it should be something like 12 000 um so let's continue with the recap of what we did here
 we have the train data set the validation data set and on yes so we have everything like
 this the configure optimizers you can put an optimizer you can also put a learning rate
 scheduler if you want to and then in the main function we load this pytorch lightning module
 and we specify a trainer and the trainer we tell it you know the max epochs and so on
 and we set up the logger and we just run fit on this model and this runs
 epochs of the model and after each epoch it does a validation epoch and uh minimizes our loss
 very cool very effective so now if if please if you would all right here we go this is my laptop
 training burnt oh okay we don't seem to make too much progress
 let's check the tensor board
 training loss goes down training loss goes to zero
 training loss goes down training loss goes to zero i have the sneaking suspicion that this is not
 entirely shuffled so um is there like a shuffle like a shuffle thing
 because this seems a bit this seems a bit bit fishy um
 this imdb data set right here it just seems like
 you know we could use a bit of shuffling because all the labels yeah the training loss instantly
 goes to zero so maybe maybe it's not we
 can we shuffle here let's look at the load data set function
 load data set batched uh no keeping memory no none of that okay
 this does not seem to go to continue right here data sets nlp data sets
 i hope here we know we should find this load data set somewhere builder features load
 load data set split
 can we not shuffle anywhere we'll search shuffle builder
 okay so generate examples this function pre-processed examples key will be hashed
 okay we are not able to shuffle this um just like that and we can't do that
 okay we are not able to shuffle this um just like that but i hope this at least gives you an
 impression i guess if you were to take the full data set and map it then it would actually work
 we'll just try again with 10% of the data just uh to see it go down
 tensorboard see this is now good because we always delete the logs folder we don't have any uh remnant
 uh old tensorflow logs
 all right come on come on so 10% should be yeah about this about this okay
 train loss looking good looking good so far
 looking good looking good so far
 look at these models how large is that how large is the bert
 base case hugging face pre-trained models
 pre-trained models bert based on case that's the one we have 12 layers 110 million parameters easy
 easy easy
 oh no it's too large training loss goes to zero again okay so we've determined that this data set
 very probably isn't entirely shuffled it might just have all the good labels first and all the
 bad labels last and um yeah just to confirm let's confirm this uh right here let's go with 100%
 but let's put an ipython shell down um just before we map the data set so we don't have to go through
 the whole mapping procedure actually that would be here right yes
 can we not map this asynchronously map
 i might be doing something really wrong with this library but i think that's that's how it should go
 so map def
 map right here we can do batched we could do batched and then i think hugging face has a function to
 encode batched encode batch encode
 encode batch no um let's go to the tokenizer
 build inputs create token type ids get special token mask save where is encode
 code
 right here can we have batch encode build inputs no this might be it batch encode yes there is a
 batch encode where you have batches of these things so okay what if we do the negative one
 see here's the label zero um i'm pretty sure i'm pretty sure uh
 batch true let's do that and in our function here we'll say batch encode
 so let's see how fast this is with 100%
 where tokenizer has no
 but we just had batch encode
 oh but this might be we have batch encode plus
 batch encode plus or text pairs okay we need this batch encode plus
 but then that gives us a dictionary right this gives us a dictionary with the fields input ids
 right here so
 like this how about that and then we still might want to limit the actual data set um once we have
 once we have
 mapped it because we need to train on it as well
 but i just want to see how fast this batch uh encoding is
 yes
 okay reasonably fast but it takes like three minutes um yeah so we won't go on here i will put
 i will put this as is on um i'll put this as is on github and i will put this as is on github
 and i hope you can profit from that in any way you want the hugging face site has a tutorial on
 squad where they also use the metrics so they have basically these pre-defined metrics like blur
 or rouge i think and you can just use them as you use these data sets so
 it's very very very convenient to work with these things in nlp so nlp has come a long way
 absolutely invite you to check out the um the transformers and tokenizers and nlp repos
 and with that that's it for me i think i hope you enjoyed this again leave a comment if you
 see improvements or if i maybe should edit this a bit more or if i should add a little bit more
 see improvements or if i maybe should edit this a bit more i thought the entire process
 of just going through and making mistakes um would be entertaining to some all right bye bye