[ML Coding Tips] Separate Computation & Plotting using locals
deep learning machine learning coding research engineering ipython colab notebook locals
Here's a lazy way to separate computation and subsequent analysis in a notebook without the overhead of manually saving local variables.

WARNING: Don't do this in a serious project.

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! So today I just wanted to bring you a quick coding tip that I often encounter in my daily machine learning research or life that might not be super common in let's say traditional software engineering or elsewhere. So often I have a bunch of, let's say I have a bunch of models right, and I use these IPython notebooks or collabs to analyze my data, plot things and so on. So I have a bunch of models, let's say they're called M1 and M2, and I'll usually run my big jobs on a cluster. So let's say I have run some jobs, one for each model, and have some logs that I want to analyze. So I'll go through my models and here I load a bunch of logs and I'll also compute some stuff, some statistics and some just things, right, that I want to have computed. And maybe these things are called A and let's... this and B, right. So now I've computed these things now, I want to analyze them. And let's shortcut and say printing is plotting. So think of these, this might be numbers right, and now I want to plot them here out, let's just print them, which I do this. Now every time I want to, you know, change something here in my printing, maybe I want the separator to be that, I'll have to load all the logs and compute all the stuff right each time I run this cell, which is not super cool, right. So I usually would like to factor out the plotting and stuff like this from the computation. So I could extract one of them into like a function, but then the point of these notebooks is that I can run each of the cells and they'll run right there, right. So functions aren't really cool in these notebooks. So what I'll usually end up doing is you'll have some second loop in here, right, and let's see, you'll have some data dict up here and you'll... here at the end you'll say something like data for this model is A and B or something like this, right, and then down here the first thing I do is I'll get my data and then I'll unpack again. So DA, DB, either I'll unpack or I'll just address them in dictionary notation like this and then I can do my plotting, right. Some people use tuples here, right, they could just go A and B, but then you'll have to do this unpacking. The problem is now if I want to add something here, right, I compute something new. I now need to add something here, right, I need to remember to store it in the data array and then I need to here remember to unpack it in the same order, right, and then I need to produce to put it in in the plotting, right. So this is very cumbersome. This line here and this line here, they're very... because you kind of duplicate your variable names all over the place just because you want to compute them here and use them here. A software engineer would usually tell you let's do something like a data class or in its most simplest form is say of a class and you know there are multiple ways of achieving this but let's just do A and B here. This is more probably the most verbose, you can also do name tuples, adder classes, data classes and so on but ultimately you produce a class like this and then here you say this is a data class A and B and then here down here you can at least address them like this, right, and you don't have to do the dictionary notation or remember the order. But now again if you want to add the C, not only do you have to add it here but you have to add it up here and you have to add it here and beware if there's a doc string and then you can use it here. This is just too cumbersome. So here is a trick and please only use these in like notebooks like this. This will lead to so much memory problems and everything and if you work with a software engineer you might have to get them chocolate for doing this. But alright, so what you can do is you can save your local variables in this dictionary. So this is a built-in that saves the local variables, sorry that gives you a dict of the local variables. The local variables here are A, B, C, M and all a bunch of other things, right. It's not clear in these IPython notebooks they have a bunch of stuff around them such that the meaning is slightly different depending where locals is but in essence you'll get the A, B and C what you want. Now the locals dict refers in order for this to go through the locals dict refers of course to the same objects not newly constructed so we're safe here to make a copy of that right like this and then down here we simply retrieve these local variables and update the current local variables using the local variables we stored and voila we can use A, B and C without ever having defined them, right. If this doesn't work that means that the locals here is a copy basically and this is a Python optimization and I read I don't have to use it because in these IPython notebooks it appears to work but you might want to have an empty exec statement here such that the Python interpreter can never be sure which variables are created in here and therefore can't optimize away. So that that is a horrible horrible trick. Again don't build anything serious upon this but it is very duper super easy you can just add any variable here and then use it down here so easy right and so wrong. Alright this was it I hope you enjoyed this I want to bring these kind of tips every now and then as a as an intermix into the research papers I hope you enjoyed this bye bye
 Hi there! So today I just wanted to bring you a quick coding tip that I often
 encounter in my daily machine learning research or life that might not be super
 common in let's say traditional software engineering or elsewhere. So often I have
 a bunch of, let's say I have a bunch of models right, and I use these IPython
 notebooks or collabs to analyze my data, plot things and so on. So I have a bunch
 of models, let's say they're called M1 and M2, and I'll usually run my big jobs
 on a cluster. So let's say I have run some jobs, one for each model, and have
 some logs that I want to analyze. So I'll go through my models and here I
 load a bunch of logs and I'll also compute some stuff, some statistics and
 some just things, right, that I want to have computed. And maybe these things
 are called A and let's... this and B, right. So now I've computed these things now, I
 want to analyze them. And let's shortcut and say printing is plotting. So
 think of these, this might be numbers right, and now I want to plot them here
 out, let's just print them, which I do this. Now every time I want to, you know,
 change something here in my printing, maybe I want the separator to be that,
 I'll have to load all the logs and compute all the stuff right each time I
 run this cell, which is not super cool, right. So I usually would like to factor
 out the plotting and stuff like this from the computation. So I could
 extract one of them into like a function, but then the point of these notebooks is
 that I can run each of the cells and they'll run right there, right. So
 functions aren't really cool in these notebooks. So what I'll usually end up
 doing is you'll have some second loop in here, right, and let's see, you'll have
 some data dict up here and you'll... here at the end you'll say something like
 data for this model is A and B or something like this, right, and then down
 here the first thing I do is I'll get my data and then I'll unpack again. So DA,
 DB, either I'll unpack or I'll just address them in dictionary notation like
 this and then I can do my plotting, right. Some people use tuples here, right, they
 could just go A and B, but then you'll have to do this unpacking. The problem is
 now if I want to add something here, right, I compute something new. I now need
 to add something here, right, I need to remember to store it in the data array
 and then I need to here remember to unpack it in the same order, right, and
 then I need to produce to put it in in the plotting, right. So this is very
 cumbersome. This line here and this line here, they're very... because you kind of
 duplicate your variable names all over the place just because you want to
 compute them here and use them here. A software engineer would usually tell you
 let's do something like a data class or in its most simplest form is say of a
 class and you know there are multiple ways of achieving this but let's just do
 A and B here. This is more probably the most verbose, you can also do name tuples,
 adder classes, data classes and so on but ultimately you produce a class like this
 and then here you say this is a data class A and B and then here down here
 you can at least address them like this, right, and you don't have to do the
 dictionary notation or remember the order. But now again if you
 want to add the C, not only do you have to add it here but you have to add
 it up here and you have to add it here and beware if there's a doc string and
 then you can use it here. This is just too cumbersome. So here is a
 trick and please only use these in like notebooks like this. This will lead to so
 much memory problems and everything and if you work with a software engineer you
 might have to get them chocolate for doing this. But alright, so what you
 can do is you can save your local variables in this dictionary.
 So this is a built-in that saves the local variables, sorry that gives you a
 dict of the local variables. The local variables here are A, B, C, M and all a bunch
 of other things, right. It's not clear in these IPython notebooks they have a
 bunch of stuff around them such that the meaning is slightly different depending
 where locals is but in essence you'll get the A, B and C what you want. Now the
 locals dict refers in order for this to go through the locals dict refers of
 course to the same objects not newly constructed so we're safe here to make a
 copy of that right like this and then down here we simply retrieve these local
 variables and update the current local variables using the local variables we
 stored and voila we can use A, B and C without ever having defined them, right.
 If this doesn't work that means that the locals here is a copy basically
 and this is a Python optimization and I read I don't have to use it because in
 these IPython notebooks it appears to work but you might want to have an empty
 exec statement here such that the Python interpreter can never be sure which
 variables are created in here and therefore can't optimize away. So that
 that is a horrible horrible trick. Again don't build anything serious upon this
 but it is very duper super easy you can just add any variable here and then use
 it down here so easy right and so wrong. Alright this was it I hope you enjoyed
 this I want to bring these kind of tips every now and then as a as an intermix
 into the research papers I hope you enjoyed this bye bye