Write more decorators (and fewer classes)
Verardi, Antonio
Write more decorators (and fewer classes) In the wake of famous talk “Stop Writing Classes” by Jack Diederich (?v=o9pEzgHorH0), I’d like to present a pattern that can be used to design clean and pythonic interfaces for programmers based on replacing single-method classes with decorated functions. This pattern is already used by some famous frameworks and libraries, like Pyramid, but I believe it isn’t that well-known to many (even experienced) Python developers and is not as widely used as it deserves. I’ll show how this pattern can be employed to improve a programming interface which is used by an internal log processing framework at Yelp. This will demonstrate how a more functional approach, leveraging the power of Python decorators, can lead to simpler, more beautiful and easier to understand code. However, this talk doesn’t suggest giving up classes altogether, but making use of them only when they are truly useful. In fact, the use-case I’m going to analyze will combine classes, functions, and decorators to make the best out of these tools. Given that the presentation is going to be very code-oriented, the talk is intended for an audience of developers who are already familiar with most Python constructs, including decorators, even though the concept will be briefly introduced at the beginning of the talk. But, if you are one of these people, I promise you that the code will speak for itself
 Hello. Thank you very much for coming. My name is Antonio. I'm a software engineer at Yelp. I work on the metrics team. So what I currently do is working on observability and real-time metrics and real-time processing. Yelp mission is to connect people with great local businesses. We have more than 127 million reviews on our website. And on average every month, we do 26 million unique mobile devices, accesses via the Yelp web app, and eight million unique visitors via the desktop website. We have offices all over the world, and we are more than 400 engineers working both on the product and the infrastructure. So a new introduction before to start. This talk has been inspired by the talk, stop writing classes by Jack Deodorich. I recommend you to watch the talk. It's great. And by the blog post, the controller pattern is awful, and other objective-oriented heresy by EV. Again, great blog post, highly recommended. All the code that you're going to see in this presentation, and trust me, is going to be quite a lot, is available on GitHub at that URL. So all the code runs and tests, it's for you. Now I'd like to start from the very end of the talk, with the main two takeaways. So they might make little sense to you at the moment, but keep them in mind, maybe at the end they will be, you know, cool ideas that you can use at your daily job. So the first one is let users utilize all Python features instead of just inheritance. And the second one is go for decorators when your classes have only one method and are instantiated only once. All right. So what's this all about? Why we're here? Well, this is the reason. How many of you have ever used Celery? Quite a few. Nice. Nice. Celery is an asynchronous job queue based on distributed message passing. It is focused on real-time operation, but also scheduling and this kind of stuff. So the way you define a task in Celery is defining an app using the Celery API, and then calling the decorator task on the app object that you just created in order to decorate your task function. So this is the first example on their website. And this is the pattern that I'm going to show you today. This pattern is very, very common, actually. And it's used by a lot of cool frameworks. For example, you know Flask, right? And Flask is a micro framework for web development. And the way you define logic for your route in Flask is exactly the same as you define task in Celery. You define a Flask app, and then you call the route decorator or the app object you just created with some parameters, in this case, the URL you want the function to be executed for. And then you just give it the function that is going to encapsulate the logic you want to execute. This is a very common pattern for web frameworks. In fact, also the Pyramid frameworks use it in order to define views, which are basically the same concept of routes in Flask. As you can see, you just call the at view config decorator, you pass in some arguments, and then the function you decorated is going to be the logic that is going to be executed. This is, again, the first example on their website. Now what about classes? As you notice, the name of the talk is write more decorators and fewer classes. So if you ever try to write a test in Python, it's highly likely that you have read about unit test. Unit test is now in the standard Libin Python 3, so I guess it's the recommended way of writing test in Python. The way you do it is you define a class in everything from unit test.test case, and you define some test methods that are going to have your assertions in order to understand if your code is doing the right thing, at least with the aspect. And in case you want to execute some setup with your down methods, what you do is that you override a method with a specific signature, in this case, setup, and here you may do all your setup logic. In this case, for example, what the example in the Python documentation does is instantiated a widget object and attaching it on the fly to the test class. Now PyTest, which is an alternative framework to write tests, avoid the usage of classes, substituting them with the decorator. So if you use the PyTest.fixure decorator in order to decorate your setup logic, then you don't need the class anymore. And in order to access the object that is returned by your setup function, what you do is that you pass the name of the function to your test functions, and it works magically. So as you can see, the interface is much more linear. You can use all the concept from functional programming and everything, and, well, at the end of the day, it's just shorter. So that's cool, at least, that's what I think. So I want you to understand how you implement this stuff is like how do you write these decorators in order to avoid using classes or to build clean interfaces? Well, what you do when you want to understand how some code works, you go on GitHub and read the source, right? So I went to the PyTest project, and this was the result. It is kind of complicated. So I say to myself, well, PyTest does a lot of magic. Maybe PyTest is going to be easier. Nope. It was even a little bit worse, maybe. Yeah, again, PyRAMid is a mature framework, and it has a lot of business logic, and there's a lot of, you know, corner cases. So Celery. Celery was simple, right? Nope. Celery was the worst of all. Like, as you can see, it's a wall of code. I have no idea why they put all this code, probably because they wanted to handle a lot of corner cases. And so my reaction after seeing the Celery implementation was this one. And I say to myself, decorators are hard. I'm not going to use them. So we believe this sentence is true. No one. That's cool. That's cool. Well, maybe someone there. Well, it's actually not entirely correct. A few months later, I looked more into how decorators work in Python. And basically the way they work is that you define a function which receives as an argument the function you want to decorate, and the result of the decorator function is assigned to the function name. That's pretty easy, right? And in case you want to pass arguments to your decorator, the way it works is that the decorator function is going to be called with the arguments before, and then the result is going to be used as a decorator as we have just seen. And in case you have more than one decorator, the way they get executed is that the closest one to the function that you want to decorate gets executed first, and the result gets passed to the decorator that is upper level. So actually I changed my stance after that, and so I decided decorators are pretty easy to use, but maybe are just hard to write. Well, this is not entirely true as well. So this is the simplest decorator ever made, I guess. As you can see, what it does is just when decorating the function, it prints decorating function with the useless, but it's a decorator, and it works. In fact, if we execute it in the Python interpreter, that's what we get. The first time we import the function, the decorator function is going to be executed, and the decorator function is going to be on the output, and then every time we execute the function, nothing happens. Just execute the regular function. So after this, maybe you can think that decorators are easy to write. Again, that is not entirely true. So in case you want to implement the decorator with arguments, as we saw before, the way you do it is defining an inner function in your decorator function, and the inner function is going to be the real decorator, the one that we saw before, so then we're receiving the function as first argument. While instead, you can use the arguments that you pass to your ideal decorator in any other part of the function itself. So the way it works for this decorator implemented over here is that when the function gets imported, it brings decorating function with bar and buzz, which are the two arguments that we pass to the decorator, and then when the function runs, the function runs without any modification at all. Now if we want to do something more useful with our decorators, like for example changing the way the function is called or saving the result into a cache and the kind of stuff, we need a wrapper function as well that is going to be defined inside your decorator function, and these wrapper functions receive for arguments the arguments of the function you want to decorate. So piece of advice, if you want to write code that is reusable, just use args and vargs, so you need to care about what the function you're going to decorate is taking as parameter. And the way this decorator function works is that again when executed in the interpreter, it decorates the function and at the import time it brings decorating function, and now every time we call the function, the code that we've written in the wrapper function inside the decorator function is going to be executed before the function gets executed for real. Now with that in mind, my stance on decorator is that they're neither easy or hard, they're just a bit tedious to write, you see. Quite a few corner cases to remember, but once you know how to write them, once you have this free example in front of you, that it's kind of okay, I mean it doesn't take much time. And in fact, if you look at the Flask framework, which is a great piece of software in my opinion, this is the way they implement their decorator, which is not much different from what we saw before, and it's much, much simpler than all the other decorators we've seen so far. So basically in the Flask class, which is the one we use to instantiate the object, we define a root method, which takes some parameters, and since it's a decorator that takes some parameters, we need an inner function, and the signal function is the actual decorator, and in the actual decorator, what Flask does is defining an endpoint using the options parameter that we provided, and then adding a rule for the endpoint to the app, and it returns the function, and it returns the inner function. So my talk was supposed to stop here to finish here, but then I noticed I booked a 45-minute slot instead of a 15-minute slot, so I decided to go for a real-life example. So basically we're not going to see decorators for a while. What we're going to do is looking at the class-based interface that these examples implement, and then we're going to look at how we can rewrite it with decorators. So the example I want to take is StutMoster. StutMoster is a real internal framework that we use at TL to extract real-time metrics out of logs, where logs are just stream of events, and this is not really important for the presentation. It's just to give you much more context. So the way StutMoster works is that it consumes logs from Apache Kafka, which is a distributed streaming platform usually used to carry logs around your infrastructure, and then it does some custom logic, which is implemented by the users via an API, and emits some metrics to signal effects, which is a third-party services that we use in order to visualize and analyze our metrics, and to KairosDB, which is a fast distributed, scalable, time-serious database on top of Cassandra that we maintain internally. So to reiterate, the input for StutMoster are logs, and the output are actually metrics. This is an example of what StutMoster does. So in this example, we see that the input is a log line, which is JSON-formatted with some information. When some user code gets executed inside StutMoster, StutMoster emits a timing metrics, so basically just timing the result of a function or something similar, which has a name, a timestamp, a value, and a dictionary of attributes that we call dimensions in the metrics world. Now let's look at the metrics interface. The metrics interface is composed by three little pieces. The first one is an enum defining the two kind of metrics that you can emit, which are basically just counters in order to count events and timers in order to time them. Then we have the metric definition. We use the name tuple here, and a metric, as we say, as always a name, a timestamp, a value, a dictionary of tags, which we call dimensions, and a type. And at the very end, we define two commodity functions in order to give the user the possibility to instantiate a counter and a timer without providing the type using partial. So let's look at the input interface instead, so the logs. So this is where the class is already coming play. Basically, the way a log class is defined is defining the coder function, which is going to use in order to decode your log line. In this case, we provide the JSON decoding as default, but you can just overwrite this default just setting your custom decoding function as a class attribute. And then the base class, which you are seeing at the moment, just define the code class method, which is going to be used to decode the log line. And the second thing is the name of the log you want to tail. So StuttMoster is a dynamic system, so the way it works is just inspect at runtime all the classes defined in the framework, and it checks for the name field, and it just tails from Kafka all the logs named this way. So if we want to implement a log, just say to StuttMoster, tail this log for me, that's the way we do it. It's pretty easy. It's reasonable. So we just subclass the log base class, and we just provide the name, in this case, events. Now in case you want to provide your custom decoding function, the way you do it is just that you define your function at the very beginning, and then you set up the decoding function as a class attribute of the log. Now let's look at the StuttMoster interface. So the real code is. The StuttMoster interface is based on the concept of trigger. A trigger is a class which encapsulates the logic to extract metrics from logs. And this is the base trigger class. It does very few things. So the first one is defining owners for every trigger, so we always know which team is responsible which logic. And what it does is asserting the owner. And this field is going to be a list of strings. There is no way for us to encode this information in Python, and so we decided to start writing a little bit of a tutorial so people could read it and understand what they need to pass to the class. And then every trigger needs to implement a digest method. The digest method is what's used to transform the decoded log line into metrics. Say it's supposed to be a generator, and it's written in the tutorial as well. So this is the core of StuttMoster. This is all the logic that really matters. And that's the process function that receives the log we are consuming from, the line we just consumed, and then it iterates over all the triggers, and it tries to yield from the digest method called on the entry, so the decoded log line. And in case of any exception, it sends an email to the owners. Now this is the way a user basically instruments StuttMoster to consume a log, and then to emit some metrics out of it. So as you can see, you first define the log as we saw before, then you define the owners for your trigger, in this case just me, and then you define your digest function. Your digest method, what it does, at least for this trigger, is just yielding a counter, saying we saw an event with this timestamp, and we just count one. Here it is another example. You define the resource usage log, you define your trigger with your list of owners, and then in the digest method here, we loop over just the custom table as time and you time, and then we emit your timing metrics based on the content of the log line. So this all worked fine. We published a tutorial, and we started to have some users, but then we realized that our users are not just regular users, they are engineers. And you know what the problem with engineers is, right? You give them a Lego set, and then they build a flying DevStar out of it. So the first question we got as soon as we released the tutorial was, how do I inherit from a base log class? Well, I didn't think about it. So I answered, well, let's do so. So you define a module starting with underscore, and here you define your base log class. In this case, what the base log class is doing is providing a coder for Apache log lines. And now, with a bit of magic, we tell Stutmaster not to look for these underscore prefixed modules so we don't start consuming logs that don't exist. Okay, that's cool. So that's the way you use these kind of base classes in your regular module. You just import them, and you set the name of the log you want to consume. Pretty easy. You update the tutorial, and we thought we were okay. Well, that was wrong. Because as soon as we updated the tutorial, we got this other question. How do I inherit from a base trigger class? Because now I can inherit log class, so I want to inherit trigger classes as well. All right. They didn't think about it either. So what we decided to do was basically reusing the same pattern. So in this same underscore module, you define your base trigger with some logic. In this case, for example, you ask the user to set the metric name and the end points that are written in the log line as class attributes. And then you also ask him to implement your get additional dimensions method, for example, where you can actually customize the metrics you are going to meet. And then you provide a digest function that does some logic and calls the get additional dimension method and uses the end points and the metric name that you define as class attributes. And at the end, yield the counter. So with a bit of magic, we say to start monster again, just don't look for these underscore modules for triggers as well. And that's the way you use them. So you just import them. And then you just set those owners, end points, metric name attributes for the subclass. And then you implement the get additional dimensions in order to customize your metrics. And that's it. It works. So again, we updated the tutorial. It started to be a bit long. But users were happy for like 24 hours. Because then I got this question, how do I know it's from two trigger classes at the same time? Well, I didn't think about it either. But you know, Python supports multiple inheritance. So we started to look around it. And our answer was, you just don't. So let's keep things simple. You implement your base class with some logic. And then you inherit from it in your other base class. And you define some other logic. And then what you do is define another base class. Well, you still inherit from it. And you define some other logic. And then in your real module, you just subclass from both of them and provide to them the same parameters as class attributes. So I get it. This is not ideal. But as long as it's documented, it's going to be OK, right? Then after more or less one week, we got the first code review in for adding a new trigger. And this is the first question we got. But how do I test my trigger class? Because you want to write tests, right? Well, guess what? This time I was prepared. You remember PyTest, right? So the way you can test your trigger is just defining a fixture in PyTest when you return the instantiation of the trigger you want to test. And then what you do, you call the digest method on the trigger. And you test a certain metrics. It's in the result of the digest method. And one minute after, he asked me, but how do I test my base trigger class? All right. What's the difference? I don't understand. Well, if you look at this class, it inherits from service-based trigger. And service-based trigger in it asserts that the derived class define a metric name. And at the same time, it inherits from trigger, which in this init asserts that it defines owners as well. So you cannot really instantiate it because the assertion is going to fail. I was very puzzled by this. I stayed one day thinking about this even under the shower. I didn't find any good solution. But then a colleague of mine came up with a very good idea. Let's use in the Python picture the type function that Python provides. So with the type function, you can instantiate a subclass of a certain class at runtime. And also you can provide class attribute as a dictionary. That worked. We updated the tutorial. It started to be long enough that people didn't want to read it. And yes, I guess it's okay. But let's try to make things better, right? So I'm going to demonstrate to you that my face before using decorator was this one. And after switching to a decorator-based interface, it changed it to this one. So looking back at the Satmoster module, we got logs as input and metrics as output. So what's missing here? What is missing is the trigger. What is a trigger? So we say the trigger is a class that encapsulates the logic to transform logs to metrics. Well, just get rid of it. It doesn't have any actual reference to the real model. It's just an abstraction that we constructed ourselves. Why do we do it in the first place? No idea. So let's remove it. And let's not force people to utilize just inheritance for the interface. Let's give them the power to use all the features that Python provides. So in the new interface, the metrics interface stays exactly the same, no modifications. The logs interface started already to change a bit. So instead of using classes only via subclassings, we decided to give the user the ability to just use classes as they were meant to be used by Python. So you can instantiate your class this time, and you pass the name of the log and the decoder function as parameters of the init method. And no more session over here. And the code class method now became a simple method. The only real difference from before is that now every log class defines an empty set of functions. Those functions are basically the equivalence of triggers in the new interface. Now this is the way you define a log in the new stat monster. You just use the class. That's it. You don't even need the file for each log. You just instantiate it with the name and it works. And in case you want to pass in your own decode function, you just call the class and you pass the parameter. That's it. It's already much cleaner. But now let's look at the stat monster interface. So first of all, the process function stayed exactly the same, but became a method of the log class. And again, the only thing we did was substituting the trigger with the functions set that is attached to the log class. Now this is the way we wanted the user to utilize the new interface. The way you define some logic in order to transform your logs to metrics is now importing your log object that you created somewhere else and then calling a decorator method on top of the object. So using another decorator to pass in the owner's method out of the function and just define the function. The function is exactly the digest meat of the first trigger we saw without any modification. You just remove all the class thingy. And basically, the way the decorator is implemented, it's the simplest decorator ever. What it does, it just receives the function as an argument and just adds it to the set that is already attached to the log class. That's it. Now this is even simpler than Flask. And in case we want to provide some more helpful decorators, like for example this one, which makes you able to register the same function for multiple logs at the same time, what we can do is implement a decorator with arguments and the arguments is basically the list of logs. And in this case, what we need is just an inner function where we iterate over the log and we call the register method once again because remember, it's decorator but at the same time it's emitted. And in case we want to implement this owner's decorator that gives some metadata to the function, what we do is exactly the same as before. We pass it to the endlers as parameters of the function and we just attach on the fly the endlers to the function because every function in Python is also an object. So I know that some people are religiously opposed to these. So in case you don't want to monkey patch your function, you can always define a dictionary as a global and then use it in the closure of the upper function. It works exactly the same. Now this is the way this register decorator is used. As you can see, we import two logs and we pass two logs to the decorator. And actually I also want to point out that the owner's decorator removes this ambiguity about oh, is the owner just a string or is the owner a list of strings? If you want to pass in more owners, you just pass in more arguments and the Python interpreter is going to check that for us as we don't need more assertion because the Python interpreter at import time is going to check that we created the two logs we want to register the function for. The code of the function once again is just a digest speed of one decorator we saw before. Now let's go back to the questions we got. So how do I inherit from a base log class? If you want to subclass your log class, you can, but if your idea is just passing the coding functions and then avoiding to pass it over and over and over, what you do is just use partial and then you instantiate the new logs using the result of partial. Next question, how do I inherit from a base trigger class? Well, no more base classes, no more assertions. What we do is just defining a simple function. We define a function, yielding some metrics and having some attributes as the attributes of the class before as parameters, in this case the end points, and even the additional dimensions method is now just a parameter for the function. And then we call it. That's it. So we just yield from the function within our logic. And that's what we do. We don't even need methods. We don't need subclasses. We don't need to remember to assert for class attributes. It's all done by the interpreter for us for free. But I'll do an error from true trigger classes. Double API about this one. What you do is just you define two functions. This is the digest method of the first trigger that we saw when we saw the example of the multiple inheritance. This is the exact same code of the second trigger that we saw before. You just call them both. You don't even need to ask me. Just call functions. Now let's go back to tests. How do I test my trigger class now? Easier as before, even easier. You don't even need the Python's feature anymore. It's just a function. You call it. That's it. And how do I test my base trigger class? That's the thing I'm the most happy about. It's basically another function. You just call it again. And the way you mock yourself is just passing some random strings, like, for example, tests as parameters. You don't need to use the type function that Python provides. You don't need to remember about the inheritance and everything. You just pass in a string. So now that we removed all the assertions, how do I make sure that all functions have owners? Well, we write a test. What the test does is calling the collect function, which is the one used by StatMoster to collect runtime all the logs. We iterate over the logs. And we just check that they have the owners attribute set. So, well, that's even better than before, because we transformed a runtime error to a test that could fail before we even push some function without owners in production. So it's everything we indeed saw implementing these decorators and changing a bit interface only for user's sake. No, actually, for the most part, it was a sem-fishing test, because you remember that they talked a lot about the import system of StatMoster, the thing that makes StatMoster understand which logs it needs to consume, which classes it needs to execute, and the kind of stuff. Well, you don't need to understand the code, but this is the import system in the new StatMoster. Just look at it and compare it with the old one. It doesn't even fit in the slides. And actually, this is a version that I wrote in Python 3.6, so it's much more complicated than the Python 2.7 version that we used, and that makes me very, very happy. Now, closing up, the main takeaways of the talk were let user utilize old Python feature instead of just inheritance, as we saw, and go for decorators when your classes have only one method, for example, the trigger class has only the digest method, and they are instantiated only once. Actually, we never even instantiated trigger class ourselves. The framework was doing that for us. And now, instead, what you do is you instantiate your log class whenever you want. You even want to subclass it. You are free to do it. You don't want to. Don't do it. And all the triggers disappeared. They are just functions. So just to clarify, I'm not saying that we should move entirely out of classes. It's just use classes when they're useful. In this case, the log class is still a class, because it has some states and it has some behavior, and it's used as class as supposed to be used in Python, not only for inheritance. So when it makes sense, go for decorator. When it makes sense, go for classes. And that's the end of the call. Remember that we are hiring, especially for our offices in London and Hamburg. So if you want to work with Python and on big real-time processing streams and kind of stuff, just come to talk to us at our booth just in the big hall. And don't forget to follow us on every social network possible, Facebook, Twitter, and GitHub. Yes, nowadays, GitHub is a social network. And please read our engineering blog, where a lot of smart engineers, why better than me, speak about all the cool stuff that we build every day. Questions? Thanks, Antonio, for your presentation. We have time for a few questions. No questions? Please. Hi. Hi. Did you run into any specific situations where switching to using decorators made it hard and practice to debug certain problems? So where the decorator obscured the actual problem that you were trying to get to? That's a very good question, actually. So you may have used decorators that are very complicated, like to do a lot of stuff, and they are then very difficult to debug because they change the way the function is executed, and then you got these weird stack traces, and you really understand what's going on. Well, my true sense of this is that as in any, while always writing code, you want to keep it simple. So my true sense is decorators should always return the function untouched. It's like the function should be able to be called without using the decorator, and the decorator should stay really small. So the frameworks that I showed you before are very complex decorators, but they are very much a framework with a good open source community, and a lot of features just came into it. So my true sense, especially for your own production code, is keep your decorator simple and avoid to do a lot of stuff into them. I hope you answered the question. More questions? Do you want to go? No? Well, thank you, Antonio, for your presentation. Thank you for coming.
 Hello.  Thank you very much for coming.  My name is Antonio.  I'm a software engineer at Yelp.  I work on the metrics team.  So what I currently do is working on observability and real-time metrics and real-time processing.  Yelp mission is to connect people with great local businesses.  We have more than 127 million reviews on our website.  And on average every month, we do 26 million unique mobile devices, accesses via the Yelp  web app, and eight million unique visitors via the desktop website.  We have offices all over the world, and we are more than 400 engineers working both on  the product and the infrastructure.  So a new introduction before to start.  This talk has been inspired by the talk, stop writing classes by Jack Deodorich.  I recommend you to watch the talk.  It's great.  And by the blog post, the controller pattern is awful, and other objective-oriented heresy  by EV.  Again, great blog post, highly recommended.  All the code that you're going to see in this presentation, and trust me, is going to be  quite a lot, is available on GitHub at that URL.  So all the code runs and tests, it's for you.  Now I'd like to start from the very end of the talk, with the main two takeaways.  So they might make little sense to you at the moment, but keep them in mind, maybe at  the end they will be, you know, cool ideas that you can use at your daily job.  So the first one is let users utilize all Python features instead of just inheritance.  And the second one is go for decorators when your classes have only one method and are  instantiated only once.  All right.  So what's this all about?  Why we're here?  Well, this is the reason.  How many of you have ever used Celery?  Quite a few.  Nice.  Nice.  Celery is an asynchronous job queue based on distributed message passing.  It is focused on real-time operation, but also scheduling and this kind of stuff.  So the way you define a task in Celery is defining an app using the Celery API, and then calling  the decorator task on the app object that you just created in order to decorate your  task function.  So this is the first example on their website.  And this is the pattern that I'm going to show you today.  This pattern is very, very common, actually.  And it's used by a lot of cool frameworks.  For example, you know Flask, right?  And Flask is a micro framework for web development.  And the way you define logic for your route in Flask is exactly the same as you define  task in Celery.  You define a Flask app, and then you call the route decorator or the app object you  just created with some parameters, in this case, the URL you want the function to be  executed for.  And then you just give it the function that is going to encapsulate the logic you want  to execute.  This is a very common pattern for web frameworks.  In fact, also the Pyramid frameworks use it in order to define views, which are basically  the same concept of routes in Flask.  As you can see, you just call the at view config decorator, you pass in some arguments,  and then the function you decorated is going to be the logic that is going to be executed.  This is, again, the first example on their website.  Now what about classes?  As you notice, the name of the talk is write more decorators and fewer classes.  So if you ever try to write a test in Python, it's highly likely that you have read about  unit test.  Unit test is now in the standard Libin Python 3, so I guess it's the recommended way of  writing test in Python.  The way you do it is you define a class in everything from unit test.test case, and you  define some test methods that are going to have your assertions in order to understand  if your code is doing the right thing, at least with the aspect.  And in case you want to execute some setup with your down methods, what you do is that  you override a method with a specific signature, in this case, setup, and here you may do all  your setup logic.  In this case, for example, what the example in the Python documentation does is instantiated  a widget object and attaching it on the fly to the test class.  Now PyTest, which is an alternative framework to write tests, avoid the usage of classes,  substituting them with the decorator.  So if you use the PyTest.fixure decorator in order to decorate your setup logic, then  you don't need the class anymore.  And in order to access the object that is returned by your setup function, what you  do is that you pass the name of the function to your test functions, and it works magically.  So as you can see, the interface is much more linear.  You can use all the concept from functional programming and everything, and, well, at the  end of the day, it's just shorter.  So that's cool, at least, that's what I think.  So I want you to understand how you implement this stuff is like how do you write these  decorators in order to avoid using classes or to build clean interfaces?  Well, what you do when you want to understand how some code works, you go on GitHub and  read the source, right?  So I went to the PyTest project, and this was the result.  It is kind of complicated.  So I say to myself, well, PyTest does a lot of magic.  Maybe PyTest is going to be easier.  Nope.  It was even a little bit worse, maybe.  Yeah, again, PyRAMid is a mature framework, and it has a lot of business logic, and there's  a lot of, you know, corner cases.  So Celery.  Celery was simple, right?  Nope.  Celery was the worst of all.  Like, as you can see, it's a wall of code.  I have no idea why they put all this code, probably because they wanted to handle a lot  of corner cases.  And so my reaction after seeing the Celery implementation was this one.  And I say to myself, decorators are hard.  I'm not going to use them.  So we believe this sentence is true.  No one.  That's cool.  That's cool.  Well, maybe someone there.  Well, it's actually not entirely correct.  A few months later, I looked more into how decorators work in Python.  And basically the way they work is that you define a function which receives as an argument  the function you want to decorate, and the result of the decorator function is assigned  to the function name.  That's pretty easy, right?  And in case you want to pass arguments to your decorator, the way it works is that the  decorator function is going to be called with the arguments before, and then the result  is going to be used as a decorator as we have just seen.  And in case you have more than one decorator, the way they get executed is that the closest  one to the function that you want to decorate gets executed first, and the result gets passed  to the decorator that is upper level.  So actually I changed my stance after that, and so I decided decorators are pretty easy  to use, but maybe are just hard to write.  Well, this is not entirely true as well.  So this is the simplest decorator ever made, I guess.  As you can see, what it does is just when decorating the function, it prints decorating  function with the useless, but it's a decorator, and it works.  In fact, if we execute it in the Python interpreter, that's what we get.  The first time we import the function, the decorator function is going to be executed,  and the decorator function is going to be on the output, and then every time we execute  the function, nothing happens.  Just execute the regular function.  So after this, maybe you can think that decorators are easy to write.  Again, that is not entirely true.  So in case you want to implement the decorator with arguments, as we saw before, the way  you do it is defining an inner function in your decorator function, and the inner function  is going to be the real decorator, the one that we saw before, so then we're receiving  the function as first argument.  While instead, you can use the arguments that you pass to your ideal decorator in any other  part of the function itself.  So the way it works for this decorator implemented over here is that when the function gets imported,  it brings decorating function with bar and buzz, which are the two arguments that we  pass to the decorator, and then when the function runs, the function runs without any modification  at all.  Now if we want to do something more useful with our decorators, like for example changing  the way the function is called or saving the result into a cache and the kind of stuff,  we need a wrapper function as well that is going to be defined inside your decorator  function, and these wrapper functions receive for arguments the arguments of the function  you want to decorate.  So piece of advice, if you want to write code that is reusable, just use args and vargs,  so you need to care about what the function you're going to decorate is taking as parameter.  And the way this decorator function works is that again when executed in the interpreter,  it decorates the function and at the import time it brings decorating function, and now  every time we call the function, the code that we've written in the wrapper function  inside the decorator function is going to be executed before the function gets executed  for real.  Now with that in mind, my stance on decorator is that they're neither easy or hard, they're  just a bit tedious to write, you see.  Quite a few corner cases to remember, but once you know how to write them, once you  have this free example in front of you, that it's kind of okay, I mean it doesn't take  much time.  And in fact, if you look at the Flask framework, which is a great piece of software in my opinion,  this is the way they implement their decorator, which is not much different from what we saw  before, and it's much, much simpler than all the other decorators we've seen so far.  So basically in the Flask class, which is the one we use to instantiate the object,  we define a root method, which takes some parameters, and since it's a decorator that  takes some parameters, we need an inner function, and the signal function is the actual decorator,  and in the actual decorator, what Flask does is defining an endpoint using the options  parameter that we provided, and then adding a rule for the endpoint to the app, and it  returns the function, and it returns the inner function.  So my talk was supposed to stop here to finish here, but then I noticed I booked a 45-minute  slot instead of a 15-minute slot, so I decided to go for a real-life example.  So basically we're not going to see decorators for a while.  What we're going to do is looking at the class-based interface that these examples implement, and  then we're going to look at how we can rewrite it with decorators.  So the example I want to take is StutMoster.  StutMoster is a real internal framework that we use at TL to extract real-time metrics  out of logs, where logs are just stream of events, and this is not really important for  the presentation.  It's just to give you much more context.  So the way StutMoster works is that it consumes logs from Apache Kafka, which is a distributed  streaming platform usually used to carry logs around your infrastructure, and then it does  some custom logic, which is implemented by the users via an API, and emits some metrics  to signal effects, which is a third-party services that we use in order to visualize  and analyze our metrics, and to KairosDB, which is a fast distributed, scalable, time-serious  database on top of Cassandra that we maintain internally.  So to reiterate, the input for StutMoster are logs, and the output are actually metrics.  This is an example of what StutMoster does.  So in this example, we see that the input is a log line, which is JSON-formatted with  some information.  When some user code gets executed inside StutMoster, StutMoster emits a timing metrics,  so basically just timing the result of a function or something similar, which has a name, a  timestamp, a value, and a dictionary of attributes that we call dimensions in the metrics world.  Now let's look at the metrics interface.  The metrics interface is composed by three little pieces.  The first one is an enum defining the two kind of metrics that you can emit, which are  basically just counters in order to count events and timers in order to time them.  Then we have the metric definition.  We use the name tuple here, and a metric, as we say, as always a name, a timestamp,  a value, a dictionary of tags, which we call dimensions, and a type.  And at the very end, we define two commodity functions in order to give the user the possibility  to instantiate a counter and a timer without providing the type using partial.  So let's look at the input interface instead, so the logs.  So this is where the class is already coming play.  Basically, the way a log class is defined is defining the coder function, which is going  to use in order to decode your log line.  In this case, we provide the JSON decoding as default, but you can just overwrite this  default just setting your custom decoding function as a class attribute.  And then the base class, which you are seeing at the moment, just define the code class  method, which is going to be used to decode the log line.  And the second thing is the name of the log you want to tail.  So StuttMoster is a dynamic system, so the way it works is just inspect at runtime all  the classes defined in the framework, and it checks for the name field, and it just  tails from Kafka all the logs named this way.  So if we want to implement a log, just say to StuttMoster, tail this log for me, that's  the way we do it.  It's pretty easy.  It's reasonable.  So we just subclass the log base class, and we just provide the name, in this case, events.  Now in case you want to provide your custom decoding function, the way you do it is just  that you define your function at the very beginning, and then you set up the decoding  function as a class attribute of the log.  Now let's look at the StuttMoster interface.  So the real code is.  The StuttMoster interface is based on the concept of trigger.  A trigger is a class which encapsulates the logic to extract metrics from logs.  And this is the base trigger class.  It does very few things.  So the first one is defining owners for every trigger, so we always know which team is responsible  which logic.  And what it does is asserting the owner.  And this field is going to be a list of strings.  There is no way for us to encode this information in Python, and so we decided to start writing  a little bit of a tutorial so people could read it and understand what they need to pass  to the class.  And then every trigger needs to implement a digest method.  The digest method is what's used to transform the decoded log line into metrics.  Say it's supposed to be a generator, and it's written in the tutorial as well.  So this is the core of StuttMoster.  This is all the logic that really matters.  And that's the process function that receives the log we are consuming from, the line we  just consumed, and then it iterates over all the triggers, and it tries to yield from the  digest method called on the entry, so the decoded log line.  And in case of any exception, it sends an email to the owners.  Now this is the way a user basically instruments StuttMoster to consume a log, and then to emit  some metrics out of it.  So as you can see, you first define the log as we saw before, then you define the owners  for your trigger, in this case just me, and then you define your digest function.  Your digest method, what it does, at least for this trigger, is just yielding a counter,  saying we saw an event with this timestamp, and we just count one.  Here it is another example.  You define the resource usage log, you define your trigger with your list of owners, and  then in the digest method here, we loop over just the custom table as time and you time,  and then we emit your timing metrics based on the content of the log line.  So this all worked fine.  We published a tutorial, and we started to have some users, but then we realized that  our users are not just regular users, they are engineers.  And you know what the problem with engineers is, right?  You give them a Lego set, and then they build a flying DevStar out of it.  So the first question we got as soon as we released the tutorial was, how do I inherit  from a base log class?  Well, I didn't think about it.  So I answered, well, let's do so.  So you define a module starting with underscore, and here you define your base log class.  In this case, what the base log class is doing is providing a coder for Apache log lines.  And now, with a bit of magic, we tell Stutmaster not to look for these underscore prefixed  modules so we don't start consuming logs that don't exist.  Okay, that's cool.  So that's the way you use these kind of base classes in your regular module.  You just import them, and you set the name of the log you want to consume.  Pretty easy.  You update the tutorial, and we thought we were okay.  Well, that was wrong.  Because as soon as we updated the tutorial, we got this other question.  How do I inherit from a base trigger class?  Because now I can inherit log class, so I want to inherit trigger classes as well.  All right.  They didn't think about it either.  So what we decided to do was basically reusing the same pattern.  So in this same underscore module, you define your base trigger with some logic.  In this case, for example, you ask the user to set the metric name and the end points  that are written in the log line as class attributes.  And then you also ask him to implement your get additional dimensions method, for example,  where you can actually customize the metrics you are going to meet.  And then you provide a digest function that does some logic and calls the get additional  dimension method and uses the end points and the metric name that you define as class  attributes.  And at the end, yield the counter.  So with a bit of magic, we say to start monster again, just don't look for these underscore  modules for triggers as well.  And that's the way you use them.  So you just import them.  And then you just set those owners, end points, metric name attributes for the subclass.  And then you implement the get additional dimensions in order to customize your metrics.  And that's it.  It works.  So again, we updated the tutorial.  It started to be a bit long.  But users were happy for like 24 hours.  Because then I got this question, how do I know it's from two trigger classes at the  same time?  Well, I didn't think about it either.  But you know, Python supports multiple inheritance.  So we started to look around it.  And our answer was, you just don't.  So let's keep things simple.  You implement your base class with some logic.  And then you inherit from it in your other base class.  And you define some other logic.  And then what you do is define another base class.  Well, you still inherit from it.  And you define some other logic.  And then in your real module, you just subclass from both of them and provide to them the  same parameters as class attributes.  So I get it.  This is not ideal.  But as long as it's documented, it's going to be OK, right?  Then after more or less one week, we got the first code review in for adding a new trigger.  And this is the first question we got.  But how do I test my trigger class?  Because you want to write tests, right?  Well, guess what?  This time I was prepared.  You remember PyTest, right?  So the way you can test your trigger is just defining a fixture in PyTest when you return  the instantiation of the trigger you want to test.  And then what you do, you call the digest method on the trigger.  And you test a certain metrics.  It's in the result of the digest method.  And one minute after, he asked me, but how do I test my base trigger class?  All right.  What's the difference?  I don't understand.  Well, if you look at this class, it inherits from service-based trigger.  And service-based trigger in it asserts that the derived class define a metric name.  And at the same time, it inherits from trigger, which in this init asserts that it defines  owners as well.  So you cannot really instantiate it because the assertion is going to fail.  I was very puzzled by this.  I stayed one day thinking about this even under the shower.  I didn't find any good solution.  But then a colleague of mine came up with a very good idea.  Let's use in the Python picture the type function that Python provides.  So with the type function, you can instantiate a subclass of a certain class at runtime.  And also you can provide class attribute as a dictionary.  That worked.  We updated the tutorial.  It started to be long enough that people didn't want to read it.  And yes, I guess it's okay.  But let's try to make things better, right?  So I'm going to demonstrate to you that my face before using decorator was this one.  And after switching to a decorator-based interface, it changed it to this one.  So looking back at the Satmoster module, we got logs as input and metrics as output.  So what's missing here?  What is missing is the trigger.  What is a trigger?  So we say the trigger is a class that encapsulates the logic to transform logs to metrics.  Well, just get rid of it.  It doesn't have any actual reference to the real model.  It's just an abstraction that we constructed ourselves.  Why do we do it in the first place?  No idea.  So let's remove it.  And let's not force people to utilize just inheritance for the interface.  Let's give them the power to use all the features that Python provides.  So in the new interface, the metrics interface stays exactly the same, no modifications.  The logs interface started already to change a bit.  So instead of using classes only via subclassings, we decided to give the user the ability to  just use classes as they were meant to be used by Python.  So you can instantiate your class this time, and you pass the name of the log and the decoder  function as parameters of the init method.  And no more session over here.  And the code class method now became a simple method.  The only real difference from before is that now every log class defines an empty set of  functions.  Those functions are basically the equivalence of triggers in the new interface.  Now this is the way you define a log in the new stat monster.  You just use the class.  That's it.  You don't even need the file for each log.  You just instantiate it with the name and it works.  And in case you want to pass in your own decode function, you just call the class and you  pass the parameter.  That's it.  It's already much cleaner.  But now let's look at the stat monster interface.  So first of all, the process function stayed exactly the same, but became a method of the  log class.  And again, the only thing we did was substituting the trigger with the functions set that is  attached to the log class.  Now this is the way we wanted the user to utilize the new interface.  The way you define some logic in order to transform your logs to metrics is now importing  your log object that you created somewhere else and then calling a decorator method on  top of the object.  So using another decorator to pass in the owner's method out of the function and just  define the function.  The function is exactly the digest meat of the first trigger we saw without any modification.  You just remove all the class thingy.  And basically, the way the decorator is implemented, it's the simplest decorator ever.  What it does, it just receives the function as an argument and just adds it to the set  that is already attached to the log class.  That's it.  Now this is even simpler than Flask.  And in case we want to provide some more helpful decorators, like for example this one, which  makes you able to register the same function for multiple logs at the same time, what we  can do is implement a decorator with arguments and the arguments is basically the list of  logs.  And in this case, what we need is just an inner function where we iterate over the log and  we call the register method once again because remember, it's decorator but at the same time  it's emitted.  And in case we want to implement this owner's decorator that gives some metadata to the  function, what we do is exactly the same as before.  We pass it to the endlers as parameters of the function and we just attach on the fly  the endlers to the function because every function in Python is also an object.  So I know that some people are religiously opposed to these.  So in case you don't want to monkey patch your function, you can always define a dictionary  as a global and then use it in the closure of the upper function.  It works exactly the same.  Now this is the way this register decorator is used.  As you can see, we import two logs and we pass two logs to the decorator.  And actually I also want to point out that the owner's decorator removes this ambiguity  about oh, is the owner just a string or is the owner a list of strings?  If you want to pass in more owners, you just pass in more arguments and the Python interpreter  is going to check that for us as we don't need more assertion because the Python interpreter  at import time is going to check that we created the two logs we want to register the function  for.  The code of the function once again is just a digest speed of one decorator we saw before.  Now let's go back to the questions we got.  So how do I inherit from a base log class?  If you want to subclass your log class, you can, but if your idea is just passing the  coding functions and then avoiding to pass it over and over and over, what you do is  just use partial and then you instantiate the new logs using the result of partial.  Next question, how do I inherit from a base trigger class?  Well, no more base classes, no more assertions.  What we do is just defining a simple function.  We define a function, yielding some metrics and having some attributes as the attributes  of the class before as parameters, in this case the end points, and even the additional  dimensions method is now just a parameter for the function.  And then we call it.  That's it.  So we just yield from the function within our logic.  And that's what we do.  We don't even need methods.  We don't need subclasses.  We don't need to remember to assert for class attributes.  It's all done by the interpreter for us for free.  But I'll do an error from true trigger classes.  Double API about this one.  What you do is just you define two functions.  This is the digest method of the first trigger that we saw when we saw the example of the  multiple inheritance.  This is the exact same code of the second trigger that we saw before.  You just call them both.  You don't even need to ask me.  Just call functions.  Now let's go back to tests.  How do I test my trigger class now?  Easier as before, even easier.  You don't even need the Python's feature anymore.  It's just a function.  You call it.  That's it.  And how do I test my base trigger class?  That's the thing I'm the most happy about.  It's basically another function.  You just call it again.  And the way you mock yourself is just passing some random strings, like, for example, tests  as parameters.  You don't need to use the type function that Python provides.  You don't need to remember about the inheritance and everything.  You just pass in a string.  So now that we removed all the assertions, how do I make sure that all functions have  owners?  Well, we write a test.  What the test does is calling the collect function, which is the one used by StatMoster  to collect runtime all the logs.  We iterate over the logs.  And we just check that they have the owners attribute set.  So, well, that's even better than before, because we transformed a runtime error to a  test that could fail before we even push some function without owners in production.  So it's everything we indeed saw implementing these decorators and changing a bit interface  only for user's sake.  No, actually, for the most part, it was a sem-fishing test, because you remember that  they talked a lot about the import system of StatMoster, the thing that makes StatMoster  understand which logs it needs to consume, which classes it needs to execute, and the  kind of stuff.  Well, you don't need to understand the code, but this is the import system in the new StatMoster.  Just look at it and compare it with the old one.  It doesn't even fit in the slides.  And actually, this is a version that I wrote in Python 3.6, so it's much more complicated  than the Python 2.7 version that we used, and that makes me very, very happy.  Now, closing up, the main takeaways of the talk were let user utilize old Python feature  instead of just inheritance, as we saw, and go for decorators when your classes have only  one method, for example, the trigger class has only the digest method, and they are instantiated  only once.  Actually, we never even instantiated trigger class ourselves.  The framework was doing that for us.  And now, instead, what you do is you instantiate your log class whenever you want.  You even want to subclass it.  You are free to do it.  You don't want to.  Don't do it.  And all the triggers disappeared.  They are just functions.  So just to clarify, I'm not saying that we should move entirely out of classes.  It's just use classes when they're useful.  In this case, the log class is still a class, because it has some states and it has some  behavior, and it's used as class as supposed to be used in Python, not only for inheritance.  So when it makes sense, go for decorator.  When it makes sense, go for classes.  And that's the end of the call.  Remember that we are hiring, especially for our offices in London and Hamburg.  So if you want to work with Python and on big real-time processing streams and kind  of stuff, just come to talk to us at our booth just in the big hall.  And don't forget to follow us on every social network possible, Facebook, Twitter, and GitHub.  Yes, nowadays, GitHub is a social network.  And please read our engineering blog, where a lot of smart engineers, why better than  me, speak about all the cool stuff that we build every day.  Questions?  Thanks, Antonio, for your presentation.  We have time for a few questions.  No questions?  Please.  Hi.  Hi.  Did you run into any specific situations where switching to using decorators made it hard  and practice to debug certain problems?  So where the decorator obscured the actual problem that you were trying to get to?  That's a very good question, actually.  So you may have used decorators that are very complicated, like to do a lot of stuff, and  they are then very difficult to debug because they change the way the function is executed,  and then you got these weird stack traces, and you really understand what's going on.  Well, my true sense of this is that as in any, while always writing code, you want to  keep it simple.  So my true sense is decorators should always return the function untouched.  It's like the function should be able to be called without using the decorator, and the  decorator should stay really small.  So the frameworks that I showed you before are very complex decorators, but they are  very much a framework with a good open source community, and a lot of features just came  into it.  So my true sense, especially for your own production code, is keep your decorator simple  and avoid to do a lot of stuff into them.  I hope you answered the question.  More questions?  Do you want to go?  No?  Well, thank you, Antonio, for your presentation.  Thank you for coming.
Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_0_203.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_204_287.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_288_347.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_348_2843.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_2844_8555.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_8556_10235.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_10236_10475.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_10476_10907.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_10908_12179.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_12180_13151.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_13152_16139.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_16140_18563.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_18564_18815.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_18816_22043.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_22044_23519.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_23520_24263.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_24264_25475.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_25476_26567.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_26568_28643.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_28644_29279.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_29280_30071.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_30072_31799.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_31800_33227.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_33228_33275.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_33276_34415.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_34416_35579.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_35580_35735.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_35736_36527.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_36528_37319.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_37320_37475.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_37476_39191.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_39192_40859.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_40860_40919.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_40920_45779.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_45780_46463.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_46464_47699.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_47700_48887.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_48888_52175.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_52176_54671.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_54672_54851.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_54852_54959.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_54960_55247.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_55248_55319.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_55320_55667.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_55668_57467.png Antonio_Verardi_-_Write_more_decorators__and_fewer_classes__VLlLIL5ZYLI_57468_57565.png
Hello. Thank you very much for coming. My name is Antonio. I'm a software engineer at Yelp. I work on the metrics team. So what I currently do is working on observability and real-time metrics and real-time processing. Yelp mission is to connect people with great local businesses. We have more than 127 million reviews on our website. And on average every month, we do 26 million unique mobile devices, accesses via the Yelp web app, and eight million unique visitors via the desktop website. We have offices all over the world, and we are more than 400 engineers working both on the product and the infrastructure. So a new introduction before to start. This talk has been inspired by the talk, stop writing classes by Jack Deodorich. I recommend you to watch the talk. It's great. And by the blog post, the controller pattern is awful, and other objective-oriented heresy by EV. Again, great blog post, highly recommended. All the code that you're going to see in this presentation, and trust me, is going to be quite a lot, is available on GitHub at that URL. So all the code runs and tests, it's for you. Now I'd like to start from the very end of the talk, with the main two takeaways. So they might make little sense to you at the moment, but keep them in mind, maybe at the end they will be, you know, cool ideas that you can use at your daily job. So the first one is let users utilize all Python features instead of just inheritance. And the second one is go for decorators when your classes have only one method and are instantiated only once. All right. So what's this all about? Why we're here? Well, this is the reason. How many of you have ever used Celery? Quite a few. Nice. Nice. Celery is an asynchronous job queue based on distributed message passing. It is focused on real-time operation, but also scheduling and this kind of stuff. So the way you define a task in Celery is defining an app using the Celery API, and then calling the decorator task on the app object that you just created in order to decorate your task function. So this is the first example on their website. And this is the pattern that I'm going to show you today. This pattern is very, very common, actually. And it's used by a lot of cool frameworks. For example, you know Flask, right? And Flask is a micro framework for web development. And the way you define logic for your route in Flask is exactly the same as you define task in Celery. You define a Flask app, and then you call the route decorator or the app object you just created with some parameters, in this case, the URL you want the function to be executed for. And then you just give it the function that is going to encapsulate the logic you want to execute. This is a very common pattern for web frameworks. In fact, also the Pyramid frameworks use it in order to define views, which are basically the same concept of routes in Flask. As you can see, you just call the at view config decorator, you pass in some arguments, and then the function you decorated is going to be the logic that is going to be executed. This is, again, the first example on their website. Now what about classes? As you notice, the name of the talk is write more decorators and fewer classes. So if you ever try to write a test in Python, it's highly likely that you have read about unit test. Unit test is now in the standard Libin Python 3, so I guess it's the recommended way of writing test in Python. The way you do it is you define a class in everything from unit test.test case, and you define some test methods that are going to have your assertions in order to understand if your code is doing the right thing, at least with the aspect. And in case you want to execute some setup with your down methods, what you do is that you override a method with a specific signature, in this case, setup, and here you may do all your setup logic. In this case, for example, what the example in the Python documentation does is instantiated a widget object and attaching it on the fly to the test class. Now PyTest, which is an alternative framework to write tests, avoid the usage of classes, substituting them with the decorator. So if you use the PyTest.fixure decorator in order to decorate your setup logic, then you don't need the class anymore. And in order to access the object that is returned by your setup function, what you do is that you pass the name of the function to your test functions, and it works magically. So as you can see, the interface is much more linear. You can use all the concept from functional programming and everything, and, well, at the end of the day, it's just shorter. So that's cool, at least, that's what I think. So I want you to understand how you implement this stuff is like how do you write these decorators in order to avoid using classes or to build clean interfaces? Well, what you do when you want to understand how some code works, you go on GitHub and read the source, right? So I went to the PyTest project, and this was the result. It is kind of complicated. So I say to myself, well, PyTest does a lot of magic. Maybe PyTest is going to be easier. Nope. It was even a little bit worse, maybe. Yeah, again, PyRAMid is a mature framework, and it has a lot of business logic, and there's a lot of, you know, corner cases. So Celery. Celery was simple, right? Nope. Celery was the worst of all. Like, as you can see, it's a wall of code. I have no idea why they put all this code, probably because they wanted to handle a lot of corner cases. And so my reaction after seeing the Celery implementation was this one. And I say to myself, decorators are hard. I'm not going to use them. So we believe this sentence is true. No one. That's cool. That's cool. Well, maybe someone there. Well, it's actually not entirely correct. A few months later, I looked more into how decorators work in Python. And basically the way they work is that you define a function which receives as an argument the function you want to decorate, and the result of the decorator function is assigned to the function name. That's pretty easy, right? And in case you want to pass arguments to your decorator, the way it works is that the decorator function is going to be called with the arguments before, and then the result is going to be used as a decorator as we have just seen. And in case you have more than one decorator, the way they get executed is that the closest one to the function that you want to decorate gets executed first, and the result gets passed to the decorator that is upper level. So actually I changed my stance after that, and so I decided decorators are pretty easy to use, but maybe are just hard to write. Well, this is not entirely true as well. So this is the simplest decorator ever made, I guess. As you can see, what it does is just when decorating the function, it prints decorating function with the useless, but it's a decorator, and it works. In fact, if we execute it in the Python interpreter, that's what we get. The first time we import the function, the decorator function is going to be executed, and the decorator function is going to be on the output, and then every time we execute the function, nothing happens. Just execute the regular function. So after this, maybe you can think that decorators are easy to write. Again, that is not entirely true. So in case you want to implement the decorator with arguments, as we saw before, the way you do it is defining an inner function in your decorator function, and the inner function is going to be the real decorator, the one that we saw before, so then we're receiving the function as first argument. While instead, you can use the arguments that you pass to your ideal decorator in any other part of the function itself. So the way it works for this decorator implemented over here is that when the function gets imported, it brings decorating function with bar and buzz, which are the two arguments that we pass to the decorator, and then when the function runs, the function runs without any modification at all. Now if we want to do something more useful with our decorators, like for example changing the way the function is called or saving the result into a cache and the kind of stuff, we need a wrapper function as well that is going to be defined inside your decorator function, and these wrapper functions receive for arguments the arguments of the function you want to decorate. So piece of advice, if you want to write code that is reusable, just use args and vargs, so you need to care about what the function you're going to decorate is taking as parameter. And the way this decorator function works is that again when executed in the interpreter, it decorates the function and at the import time it brings decorating function, and now every time we call the function, the code that we've written in the wrapper function inside the decorator function is going to be executed before the function gets executed for real. Now with that in mind, my stance on decorator is that they're neither easy or hard, they're just a bit tedious to write, you see. Quite a few corner cases to remember, but once you know how to write them, once you have this free example in front of you, that it's kind of okay, I mean it doesn't take much time. And in fact, if you look at the Flask framework, which is a great piece of software in my opinion, this is the way they implement their decorator, which is not much different from what we saw before, and it's much, much simpler than all the other decorators we've seen so far. So basically in the Flask class, which is the one we use to instantiate the object, we define a root method, which takes some parameters, and since it's a decorator that takes some parameters, we need an inner function, and the signal function is the actual decorator, and in the actual decorator, what Flask does is defining an endpoint using the options parameter that we provided, and then adding a rule for the endpoint to the app, and it returns the function, and it returns the inner function. So my talk was supposed to stop here to finish here, but then I noticed I booked a 45-minute slot instead of a 15-minute slot, so I decided to go for a real-life example. So basically we're not going to see decorators for a while. What we're going to do is looking at the class-based interface that these examples implement, and then we're going to look at how we can rewrite it with decorators. So the example I want to take is StutMoster. StutMoster is a real internal framework that we use at TL to extract real-time metrics out of logs, where logs are just stream of events, and this is not really important for the presentation. It's just to give you much more context. So the way StutMoster works is that it consumes logs from Apache Kafka, which is a distributed streaming platform usually used to carry logs around your infrastructure, and then it does some custom logic, which is implemented by the users via an API, and emits some metrics to signal effects, which is a third-party services that we use in order to visualize and analyze our metrics, and to KairosDB, which is a fast distributed, scalable, time-serious database on top of Cassandra that we maintain internally. So to reiterate, the input for StutMoster are logs, and the output are actually metrics. This is an example of what StutMoster does. So in this example, we see that the input is a log line, which is JSON-formatted with some information. When some user code gets executed inside StutMoster, StutMoster emits a timing metrics, so basically just timing the result of a function or something similar, which has a name, a timestamp, a value, and a dictionary of attributes that we call dimensions in the metrics world. Now let's look at the metrics interface. The metrics interface is composed by three little pieces. The first one is an enum defining the two kind of metrics that you can emit, which are basically just counters in order to count events and timers in order to time them. Then we have the metric definition. We use the name tuple here, and a metric, as we say, as always a name, a timestamp, a value, a dictionary of tags, which we call dimensions, and a type. And at the very end, we define two commodity functions in order to give the user the possibility to instantiate a counter and a timer without providing the type using partial. So let's look at the input interface instead, so the logs. So this is where the class is already coming play. Basically, the way a log class is defined is defining the coder function, which is going to use in order to decode your log line. In this case, we provide the JSON decoding as default, but you can just overwrite this default just setting your custom decoding function as a class attribute. And then the base class, which you are seeing at the moment, just define the code class method, which is going to be used to decode the log line. And the second thing is the name of the log you want to tail. So StuttMoster is a dynamic system, so the way it works is just inspect at runtime all the classes defined in the framework, and it checks for the name field, and it just tails from Kafka all the logs named this way. So if we want to implement a log, just say to StuttMoster, tail this log for me, that's the way we do it. It's pretty easy. It's reasonable. So we just subclass the log base class, and we just provide the name, in this case, events. Now in case you want to provide your custom decoding function, the way you do it is just that you define your function at the very beginning, and then you set up the decoding function as a class attribute of the log. Now let's look at the StuttMoster interface. So the real code is. The StuttMoster interface is based on the concept of trigger. A trigger is a class which encapsulates the logic to extract metrics from logs. And this is the base trigger class. It does very few things. So the first one is defining owners for every trigger, so we always know which team is responsible which logic. And what it does is asserting the owner. And this field is going to be a list of strings. There is no way for us to encode this information in Python, and so we decided to start writing a little bit of a tutorial so people could read it and understand what they need to pass to the class. And then every trigger needs to implement a digest method. The digest method is what's used to transform the decoded log line into metrics. Say it's supposed to be a generator, and it's written in the tutorial as well. So this is the core of StuttMoster. This is all the logic that really matters. And that's the process function that receives the log we are consuming from, the line we just consumed, and then it iterates over all the triggers, and it tries to yield from the digest method called on the entry, so the decoded log line. And in case of any exception, it sends an email to the owners. Now this is the way a user basically instruments StuttMoster to consume a log, and then to emit some metrics out of it. So as you can see, you first define the log as we saw before, then you define the owners for your trigger, in this case just me, and then you define your digest function. Your digest method, what it does, at least for this trigger, is just yielding a counter, saying we saw an event with this timestamp, and we just count one. Here it is another example. You define the resource usage log, you define your trigger with your list of owners, and then in the digest method here, we loop over just the custom table as time and you time, and then we emit your timing metrics based on the content of the log line. So this all worked fine. We published a tutorial, and we started to have some users, but then we realized that our users are not just regular users, they are engineers. And you know what the problem with engineers is, right? You give them a Lego set, and then they build a flying DevStar out of it. So the first question we got as soon as we released the tutorial was, how do I inherit from a base log class? Well, I didn't think about it. So I answered, well, let's do so. So you define a module starting with underscore, and here you define your base log class. In this case, what the base log class is doing is providing a coder for Apache log lines. And now, with a bit of magic, we tell Stutmaster not to look for these underscore prefixed modules so we don't start consuming logs that don't exist. Okay, that's cool. So that's the way you use these kind of base classes in your regular module. You just import them, and you set the name of the log you want to consume. Pretty easy. You update the tutorial, and we thought we were okay. Well, that was wrong. Because as soon as we updated the tutorial, we got this other question. How do I inherit from a base trigger class? Because now I can inherit log class, so I want to inherit trigger classes as well. All right. They didn't think about it either. So what we decided to do was basically reusing the same pattern. So in this same underscore module, you define your base trigger with some logic. In this case, for example, you ask the user to set the metric name and the end points that are written in the log line as class attributes. And then you also ask him to implement your get additional dimensions method, for example, where you can actually customize the metrics you are going to meet. And then you provide a digest function that does some logic and calls the get additional dimension method and uses the end points and the metric name that you define as class attributes. And at the end, yield the counter. So with a bit of magic, we say to start monster again, just don't look for these underscore modules for triggers as well. And that's the way you use them. So you just import them. And then you just set those owners, end points, metric name attributes for the subclass. And then you implement the get additional dimensions in order to customize your metrics. And that's it. It works. So again, we updated the tutorial. It started to be a bit long. But users were happy for like 24 hours. Because then I got this question, how do I know it's from two trigger classes at the same time? Well, I didn't think about it either. But you know, Python supports multiple inheritance. So we started to look around it. And our answer was, you just don't. So let's keep things simple. You implement your base class with some logic. And then you inherit from it in your other base class. And you define some other logic. And then what you do is define another base class. Well, you still inherit from it. And you define some other logic. And then in your real module, you just subclass from both of them and provide to them the same parameters as class attributes. So I get it. This is not ideal. But as long as it's documented, it's going to be OK, right? Then after more or less one week, we got the first code review in for adding a new trigger. And this is the first question we got. But how do I test my trigger class? Because you want to write tests, right? Well, guess what? This time I was prepared. You remember PyTest, right? So the way you can test your trigger is just defining a fixture in PyTest when you return the instantiation of the trigger you want to test. And then what you do, you call the digest method on the trigger. And you test a certain metrics. It's in the result of the digest method. And one minute after, he asked me, but how do I test my base trigger class? All right. What's the difference? I don't understand. Well, if you look at this class, it inherits from service-based trigger. And service-based trigger in it asserts that the derived class define a metric name. And at the same time, it inherits from trigger, which in this init asserts that it defines owners as well. So you cannot really instantiate it because the assertion is going to fail. I was very puzzled by this. I stayed one day thinking about this even under the shower. I didn't find any good solution. But then a colleague of mine came up with a very good idea. Let's use in the Python picture the type function that Python provides. So with the type function, you can instantiate a subclass of a certain class at runtime. And also you can provide class attribute as a dictionary. That worked. We updated the tutorial. It started to be long enough that people didn't want to read it. And yes, I guess it's okay. But let's try to make things better, right? So I'm going to demonstrate to you that my face before using decorator was this one. And after switching to a decorator-based interface, it changed it to this one. So looking back at the Satmoster module, we got logs as input and metrics as output. So what's missing here? What is missing is the trigger. What is a trigger? So we say the trigger is a class that encapsulates the logic to transform logs to metrics. Well, just get rid of it. It doesn't have any actual reference to the real model. It's just an abstraction that we constructed ourselves. Why do we do it in the first place? No idea. So let's remove it. And let's not force people to utilize just inheritance for the interface. Let's give them the power to use all the features that Python provides. So in the new interface, the metrics interface stays exactly the same, no modifications. The logs interface started already to change a bit. So instead of using classes only via subclassings, we decided to give the user the ability to just use classes as they were meant to be used by Python. So you can instantiate your class this time, and you pass the name of the log and the decoder function as parameters of the init method. And no more session over here. And the code class method now became a simple method. The only real difference from before is that now every log class defines an empty set of functions. Those functions are basically the equivalence of triggers in the new interface. Now this is the way you define a log in the new stat monster. You just use the class. That's it. You don't even need the file for each log. You just instantiate it with the name and it works. And in case you want to pass in your own decode function, you just call the class and you pass the parameter. That's it. It's already much cleaner. But now let's look at the stat monster interface. So first of all, the process function stayed exactly the same, but became a method of the log class. And again, the only thing we did was substituting the trigger with the functions set that is attached to the log class. Now this is the way we wanted the user to utilize the new interface. The way you define some logic in order to transform your logs to metrics is now importing your log object that you created somewhere else and then calling a decorator method on top of the object. So using another decorator to pass in the owner's method out of the function and just define the function. The function is exactly the digest meat of the first trigger we saw without any modification. You just remove all the class thingy. And basically, the way the decorator is implemented, it's the simplest decorator ever. What it does, it just receives the function as an argument and just adds it to the set that is already attached to the log class. That's it. Now this is even simpler than Flask. And in case we want to provide some more helpful decorators, like for example this one, which makes you able to register the same function for multiple logs at the same time, what we can do is implement a decorator with arguments and the arguments is basically the list of logs. And in this case, what we need is just an inner function where we iterate over the log and we call the register method once again because remember, it's decorator but at the same time it's emitted. And in case we want to implement this owner's decorator that gives some metadata to the function, what we do is exactly the same as before. We pass it to the endlers as parameters of the function and we just attach on the fly the endlers to the function because every function in Python is also an object. So I know that some people are religiously opposed to these. So in case you don't want to monkey patch your function, you can always define a dictionary as a global and then use it in the closure of the upper function. It works exactly the same. Now this is the way this register decorator is used. As you can see, we import two logs and we pass two logs to the decorator. And actually I also want to point out that the owner's decorator removes this ambiguity about oh, is the owner just a string or is the owner a list of strings? If you want to pass in more owners, you just pass in more arguments and the Python interpreter is going to check that for us as we don't need more assertion because the Python interpreter at import time is going to check that we created the two logs we want to register the function for. The code of the function once again is just a digest speed of one decorator we saw before. Now let's go back to the questions we got. So how do I inherit from a base log class? If you want to subclass your log class, you can, but if your idea is just passing the coding functions and then avoiding to pass it over and over and over, what you do is just use partial and then you instantiate the new logs using the result of partial. Next question, how do I inherit from a base trigger class? Well, no more base classes, no more assertions. What we do is just defining a simple function. We define a function, yielding some metrics and having some attributes as the attributes of the class before as parameters, in this case the end points, and even the additional dimensions method is now just a parameter for the function. And then we call it. That's it. So we just yield from the function within our logic. And that's what we do. We don't even need methods. We don't need subclasses. We don't need to remember to assert for class attributes. It's all done by the interpreter for us for free. But I'll do an error from true trigger classes. Double API about this one. What you do is just you define two functions. This is the digest method of the first trigger that we saw when we saw the example of the multiple inheritance. This is the exact same code of the second trigger that we saw before. You just call them both. You don't even need to ask me. Just call functions. Now let's go back to tests. How do I test my trigger class now? Easier as before, even easier. You don't even need the Python's feature anymore. It's just a function. You call it. That's it. And how do I test my base trigger class? That's the thing I'm the most happy about. It's basically another function. You just call it again. And the way you mock yourself is just passing some random strings, like, for example, tests as parameters. You don't need to use the type function that Python provides. You don't need to remember about the inheritance and everything. You just pass in a string. So now that we removed all the assertions, how do I make sure that all functions have owners? Well, we write a test. What the test does is calling the collect function, which is the one used by StatMoster to collect runtime all the logs. We iterate over the logs. And we just check that they have the owners attribute set. So, well, that's even better than before, because we transformed a runtime error to a test that could fail before we even push some function without owners in production. So it's everything we indeed saw implementing these decorators and changing a bit interface only for user's sake. No, actually, for the most part, it was a sem-fishing test, because you remember that they talked a lot about the import system of StatMoster, the thing that makes StatMoster understand which logs it needs to consume, which classes it needs to execute, and the kind of stuff. Well, you don't need to understand the code, but this is the import system in the new StatMoster. Just look at it and compare it with the old one. It doesn't even fit in the slides. And actually, this is a version that I wrote in Python 3.6, so it's much more complicated than the Python 2.7 version that we used, and that makes me very, very happy. Now, closing up, the main takeaways of the talk were let user utilize old Python feature instead of just inheritance, as we saw, and go for decorators when your classes have only one method, for example, the trigger class has only the digest method, and they are instantiated only once. Actually, we never even instantiated trigger class ourselves. The framework was doing that for us. And now, instead, what you do is you instantiate your log class whenever you want. You even want to subclass it. You are free to do it. You don't want to. Don't do it. And all the triggers disappeared. They are just functions. So just to clarify, I'm not saying that we should move entirely out of classes. It's just use classes when they're useful. In this case, the log class is still a class, because it has some states and it has some behavior, and it's used as class as supposed to be used in Python, not only for inheritance. So when it makes sense, go for decorator. When it makes sense, go for classes. And that's the end of the call. Remember that we are hiring, especially for our offices in London and Hamburg. So if you want to work with Python and on big real-time processing streams and kind of stuff, just come to talk to us at our booth just in the big hall. And don't forget to follow us on every social network possible, Facebook, Twitter, and GitHub. Yes, nowadays, GitHub is a social network. And please read our engineering blog, where a lot of smart engineers, why better than me, speak about all the cool stuff that we build every day. Questions? Thanks, Antonio, for your presentation. We have time for a few questions. No questions? Please. Hi. Hi. Did you run into any specific situations where switching to using decorators made it hard and practice to debug certain problems? So where the decorator obscured the actual problem that you were trying to get to? That's a very good question, actually. So you may have used decorators that are very complicated, like to do a lot of stuff, and they are then very difficult to debug because they change the way the function is executed, and then you got these weird stack traces, and you really understand what's going on. Well, my true sense of this is that as in any, while always writing code, you want to keep it simple. So my true sense is decorators should always return the function untouched. It's like the function should be able to be called without using the decorator, and the decorator should stay really small. So the frameworks that I showed you before are very complex decorators, but they are very much a framework with a good open source community, and a lot of features just came into it. So my true sense, especially for your own production code, is keep your decorator simple and avoid to do a lot of stuff into them. I hope you answered the question. More questions? Do you want to go? No? Well, thank you, Antonio, for your presentation. Thank you for coming.
and real-time processing. Yelp mission is to stop writing classes by Jack Deodorich. I like to start from the very end of the talk, with mind, maybe at the end they will be, you know, on their website. And this is the pattern that I'm going to show you today. This pattern is pattern for web frameworks. In fact, also the Pyramid frameworks use it in order to define decorators and fewer classes. So if you ever try the concept from functional programming and classes or to build clean interfaces? Well, what you do when you want to understand how some code the presentation. It's just to give you much more define the code class method, which is going to beginning, and then you set up the decoding I get it. This is not ideal. But as long as it's inheritance for the interface. Let's give them the power to use all the features that Python be used by Python. So you can instantiate your adds it to the set that is already attached to people are religiously opposed to these. So in I'll do an error from true trigger classes. you define two functions. This is the digest the old one. It doesn't even fit in the slides. entirely out of classes. It's just use classes when they're useful. In this case, the log class supposed to be used in Python, not only for complex decorators, but they are very much a