12 years of Pylint
Popa, Claudiu
Claudiu Popa - 12 years of Pylint (or How I learned to stop worrying about bugs) Given the dynamic nature of Python, some bugs tend to creep in our codebases. Innocents NameErrors or hard-to-find bugs with variables used in a closure, but defined in a loop, they all stand no chance in front of Pylint. In this talk, I'll present one of the oldest static analysis tools for Python, with emphasis on what it can do to understand your Python code. Pylint is both a style checker, enforcing PEP 8 rules, as well as a code checker in the vein of pyflakes and the likes, but its true power isn't always obvious to the eye of beholder. It can detect simple bugs such as unused variables and imports, but it can also detect more complicated cases such as invalid arguments passed to functions, it understands the method resolution order of your classes and what special methods aren't implemented correctly. Starting from abstract syntax trees, we'll go through its inference engine and we'll see how Pylint understands the logical flow of your program and what sort of type hinting techniques are used to improve its inference, including PEP 484 type hints. As a bonus, I'll show how it can be used to help you port your long-forgotten library to Python 3, using its new -py3k mode.
 Thank you guys. It's really nice to see so many people interested in static analysis here and this year is a special year for Pylint because it's 12th birthday. So, yeah, basically it's the oldest static analysis tool in Python that is still maintained, which is really nice. So in this talk, I'm going to talk a little bit about Pylint's history and we'll go through a detour in static analysis and we'll see where that goes. So, what is this Lint thing I'm talking about? Basically, a Lint is like a tool you use to analyze your code in order to find bugs or potential errors or style checks or stuff like that. But saying this about Pylint is an understatement since it can detect a lot more than a normal Linter Linter. It's also a style checker which enforces Pepet roles. It's a structural analyzer looking at your classes and at your special methods and checking if they are implemented correctly. And it's also a type checker. So, Pylint works by using a technique called static analysis, which is basically the act of analyzing code without actually executing it. And if you don't use static analysis right now, you should use it. You should use any other tool, not just Pylint, because it can really help you in your day-to-day job. For instance, you can use static analysis if you have a lot of tests and it takes a lot of time to run. You just want to check that there are no obvious errors in your code or stuff like that. Or if you have big legacy systems which don't have tests at all, in fact, as my first job experience, I worked in a company where we have a lot of big legacy systems, but they didn't have tests. So, using static analysis was paramount for checking that everything works before going in production. And you can use it as a form of doing reviews. Well, of course, it's not equivalent to a manual review, but it's better than no review at all. So, here's a piece of code which has a couple of problems. And I'm going to show how Pylint detects this kind of stuff. For instance, if you can see, I imported OS, but I didn't use it across my program. And I also defined a variable which is just there. It's not used. Of course, this is not a problem per se, but it leads to ugly code or unmaintainable code or stuff like that. And more important is this block here from line 8 to 10. Let's just say that I invented that code incorrectly. And that code now will not run because it's an after-run erase statement. So, basically, it's useless code. Also, at line 10, I intended to call the execute method, but I didn't. So, that's also a problem. And as you can see down here, Pylint detects this kind of stuff and properly emits that there is something fishy in your code. Okay. But we can detect more serious bugs than this, like using undefined variables or trying to access members that don't exist or calling functions that aren't functions like calling an integer. And as you can see here, Pylint detects this kind of stuff. And it's really nice for aesthetic analysis tool to tell you that you made a mistake in your program. And as I was saying before, Pylint is also a structural analyzer checking that your special methods are properly implemented or that your classes are written correctly. And in this case, the dunder exit method needs three parameters in order to work properly. And Pylint detects this kind of stuff. It emits that warning. Right? Another nice error that Pylint can detect is this one. Basically, here I have a function and then I have an if statement with func as a conditional for the if statement. And what I intended here to do initially was to call func, but I forgot to write the parents. And this statement will always be true. And that if statement, if branch will always be taken. So if you write this for this piece of code, you might, how to say, if you write this for this piece of code, you might overlook it if you didn't test it properly. So Pylint can help you in this kind of situation as well. Take a couple of moments to figure out what's the problem here. If you know the answer, you'll really, really loud. Anyone? Sorry? Var is being changed in the loop. So what's the output of this code? Who said that? Yeah, that's it. Sorry. Basically, what we did is we created a closure here and Var was looked up into parent scope when called. And Var was looked up into parent scope and called. And for this particular case, the parent scope was the least comprehension, at least on Python 3. So when the for loop was executed and callable was called, basically, Var was the last value from the for loop, which was 9. So that's a bug. And if you try to do a review, you might have missed this stuff and Pylint can help you deal with this kind of situation. Okay? A little bit of history about Pylint. It was created in 2003 by a French company called Logilab. They used to maintain it for a while for 10 years, but now they're not so involved anymore. In fact, they're not involved anymore in Pylint's development. Even though many still think that about them, which is wrong, I know that Google uses a modified version of Pylint internally for their use cases because they had a maintainer for GPylint that was also maintainer for Pylint and they used to push upstream a lot of changes. And some statistics, according to ohlog.net, we have over 30,000 lines of code, which is pretty big, but not big enough because we can detect all the problems in your code. And unfortunately, Pylint is GPL-licensed because GPL was really cool back then in 2003. It still is. Yeah. And here's my involvement with Pylint. Basically, I started working on it in 2013 when Pylint 1.0 was released and up to Pylint 1.3, I became a committer, a maintainer, the only maintainer, and the guide that pushes the thing forward. I'm planning to release Pylint 2.0 in 2016 if my time allows it and I'm going to have some advanced techniques in it. It's still right now pretty advanced, but it's going to be even more advanced than that. Basically, I'm going to use a technique called abstract interpretation where I'm going to interpret statically your code, not actually executing, but statically. And also, I'm going to add support for PAP484, what Widow stock was about, and changing the things internally, like having control flow graphs and having a symbolic execution engine for it. Yeah. Right. And at this point, you might ask yourself, okay, this is cool, but how it works and why does it work statically? First of all, Pylint is split in actually in two components. In the real checker, which is Pylint, and its inference engine and the component that understands Python, which is Astray. We're following the general pattern of building a linter. Basically, we're using abstract syntax trees, even though our abstract syntax trees are augmented with a couple of functionalities that help us in building the inference engine. And internally, Astray uses the built-in module AST, which kind of looks like this when using it. You import pars and you give it a string, and from there, you get an abstract syntax tree, which basically looks like this. So, if you can see at the top of the tree, we have a lambda, which has a couple of arguments. And the blue stuff out there, the blue thingy is a node, and the rest of them are just attributes of those nodes. So, as you can see, it's pretty structured, and you can, just by looking at it, you can reason about code. Right. Even though AST is great, it's built-in, we don't have to write our own parser, it's not perfect because it's not backwards compatible, not even for minor versions of Python releases like 3.4 and 3.5. Basically, they are going to add new nodes or remove nodes or change things out there. And Astray strives to be a backward compatible wrapper over AST that you can use for Python 2.7, Python 3.5, and as well for PyPy or Jyton or any other interpreter which supports abstract syntax trees. And the API is quite similar with the AST. As you can see, it doesn't differ too much. We have basically the same functions. Yeah. So, if you're using AST right now and you want something more capable, you could use Astray. And as I mentioned, our Astray nodes are quite similar with one from AST, but we augment them with a couple of capabilities specific for our purposes. Like in this example, for instance, you can actually retrieve the parent of a node and you can walk up in the tree starting from one node. Also, you can walk down by using the getchildren method. Right? Also, you can retrieve the lexical scope of a variable. For instance, in this particular case, the lexical scope of foo is the function where it is defined. And down there, at least comprehension, basically I'm doing foo for foo in range 10 and the scope of foo, at least on Python 3, is the least comprehension itself. So foo doesn't leak outside. Yeah. And PyLint can actually know this kind of stuff and Astray. Some nodes are augmented even more than that. Like in this example, we have a couple of classes and we're using a meta class, we're using ABC, ABC meta, and we're defining some slots for our class. And yeah. Let's see what PyLint can do, what Astray can do with this code. So we can retrieve the slots of the class using the slots method or the meta class using the meta class method or using the MRO method. We could retrieve the method resolution order of your classes, which is actually the meta resolution order when running Python over that script or over that class, which is nice. Yeah. But the most important part about Astray is not the AST itself, but the capability of doing inference. Basically, inference means the act of resolving what a node really represents. Like if you have a name and that name represents something like a variable or a function call, you want to infer that and you want to see what's at the end of the tree for that particular node. Our nodes basically implement the Python semantics. For instance, the callfunk inference rule will be the return values of the callfunk or for least comprehensions, its inference rule would be the actual list that is returned when evaluating the least comprehension. And our inference also does some partial abstract interpretation, but it's partial because it's not working for all the cases in your code. And in this particular example, we have a function which adds two values, the argument, and as you can see in this particular case, when in inferring what this function actually returns, in fact, when inferring what's the result of the function called, Astray knows that it's 48 because 24 plus 24, it's 48. This is a more complex example involving binary operators. And if you know, the rules for binary operators in Python goes like that. If you have two different objects and you try to add them, first, the Dunder add method of the left-hand side will be called. And if that doesn't return not implemented, then I think the right-hand side object, no, no, I think that the Dunder add method of the right-hand side object will be called. And in this particular example, we're having in the left-hand side a supertype, which is A. As you can see, A is a supertype for B because B has, as its base class, A. And in the right-hand side, we have B, which is a subtype of A. And the rules are a little bit different in this particular case because if the semantics would be the same as for the different classes rule, then the first rule, then the first method, A, Dunder add, will always be called, which is wrong. And in this particular example, what will happen is that first the Dunder add method of B will be called. In this particular example, it returns not implemented, so it will fall back to the other one, which is A, the Dunder add method from A. And if you do the calculations, you can see that SRAD is right. The result of that operation is actually 45. Now, SRAD is great, but they can't really understand your code. I mean, they can't really understand your full code, so we have to deal with this kind of situation. And we're providing a couple of APIs that could help you, like having node transforms. Basically, with a node transform, we can modify a part of the AST to be something else, like you have a function call, and instead of that function call, you want to do inlining, replacing the AST node with the result of the function call or anything you'd like. And we can do that with this API. Basically, that API is a function which should accept one parameter, that parameter being the original node, and should return either the node modified or a new node. And you just register that function with an internal manager, some implementation detail, but anyway, you just register your transform function, and when doing SRAD dot parse, that transform function will be called for that particular type of node. In this example, I'm registering a transform for the class node, and also you can apply a filter function because you don't actually want to change all class nodes in your code. You want to change something in particular. And as you can see there, I'm just giving it filter functions for filtering any other class that is not 6 add meta class. The same thing can be used for inference rules because at some point, really, you want to have different inference semantics than the Python offers you. So using the same API, you can provide a custom inference rule for, I don't know, you want to infer these comprehensions differently, or you want to infer function calls differently. So you're using this inference rule, and you're changing basically the semantics of Python in your AST. And we're using our own dog food in this example, not in this example, we're using our dog food with inference rules because we have inference rules for built-ins, as I'll show you immediately. Basically, with inference rules, we're understanding existence, is subclass, we're understanding the other, has other type, callable, list, set, whatever. We're understanding, not in particular with inference rules, binary arithmetic operations, really, really good logical operators or comparisons. Also, we're understanding context managers and list, set, string indexing, slicing, whatever. Yeah. As earlier, take a couple of moments to see where are the bugs in this particular code. There should be like three bugs, but if you can find more, join Pylinx thing. Also, if you know the answer, yell really loud. And as a hint, how super works, basically, the first argument of super specifies the object from which the method resolution order will be retrieved. Yeah, that's good. Because in that particular case, super C and self would be, the method resolution order in this particular case, I don't know what it is, but super C self will call B because A doesn't have Boo, yeah, and it's multiple cooperative inheritance, if I'm not mistaken, and B Boo will be called and B Boo doesn't accept two arguments. Anyone? Yeah, there's no spa and there is a full. Sorry, sorry, there are too many voices. One at a time. Full is not color. Exactly. So, Pylinx detects this stuff because asteroid knows Python really well. And as you can see, what Pylinx says, there are too many position arguments at line 14, which is actually true. Full is not callable, of course, because full is an integer. And super V has no spam number, which is actually true because it's spam, not spam. Yeah. Here's a more complex example of asteroid capabilities. And as you can see here, asteroid understands, least indexing understands has outer, callable, get outer. And at that particular line where met is retrieved, what will happen is that has outer call will return true because A has the method called met. And A met is callable. And get out from A, we'll return the met object. So it will be like true and true and A dot met. And in this particular case, the last value, which is also true, will be return. So met will be A dot met. And then the context manager is invoked. And the context manager will return real func, which has no argument whatsoever. And we're going to call that. And Pylinx will say that, hey, you use too many position arguments in your method call. You should change that. Yeah. So this kind of stuff that asteroid knows about Python code and Pylinx knows about it as well. Yeah. I don't know. I'm going really fast or how much time do I have? Okay. Thank you. Pylinx is not so complicated. Astrid is the most complicated thing. Basically, it's like a fancy walker of the AST. And it has a couple of patterns, more than a couple, because Pylinx can detect almost 180 type of errors or verifications. And basically, we are using the visitor pattern to visit each node because that pattern really decouples your data structures from your algorithms. And a small example of how visitor pattern works. Let's say we implement in our own checker, visit call func. And in this particular example, we're importing collections. And we're obtaining the default attribute from collections. There's no default attribute in collections, by the way, it's default dict. And what will happen is that visit get other will be called with the node, which will be the get other node. Afterwards, we'll infer what that node really represents. I mean, what the parent of the statement really represents, the parent will be the name, the name collections. And we need to infer it because we should know what that node represents at the end of the AST. Afterwards, we're calling the data method with the result from the inference. And if that doesn't raise any error, such as not found that yeah, collections has default to default attribute. Else, we're going to emit a no member error. But before that, we're going to have a couple of filters like the owner is a mixing class because you might not have that attribute in the mixing class. Or I don't know, the owner is a class with unknown bases, like bases from extension modules. And we currently understand extension modules, by the way. Yeah. I'll go really fast from now on because I don't have much time left. Basically, abstract interpretation will have piling for this kind of code. In this particular example, the dunder date is updated with the dictionary of, I don't know, what kind of arguments will be passed. And at line five, we will call some arguments set in dunder in it or whatever other attribute. And that attribute might not exist. And what happens right now is that piling says, hey, you don't have that attribute. But we're actually having that attribute because we just inserted it at line two. So that's what abstract interpretation is going to fix because you will just interpret every line, every logical line in your code and it will know at the end what side effects each line had. Okay. We have multiple type of checkers and errors, such as conventions for Pepe rules, refectorings, various warnings which aren't necessarily bugs in your code. And actual bugs like no member or not global, things like that. And we have two types of checkers, AST based and token based. The token based checkers are for, I don't know, line is too long or bad indentation or other similar examples. Piling has a really vibrant community. There are a lot of external packages for improving in-furnace, for improving piling checkers out there. You could write your own if you want to. The problem is that they are pure Python, so you need to write Python code in order to have a custom checker. Yeah, it comes with a couple of features, extra features like you could generate UML diagrams from your code or you could spell check your doc strings or comments. These are disabled by default. Or you could use the Python 3 porting checker, which is a checker when activated, all other checkers are disabled and it will emit stuff that it's not going to work in Python 3 anymore. Like using remove syntax or remove built-ins or special methods. Or my favorite is using map filter or reduce in non-interval contexts. As you can see in this particular example, on Python 3, map is lazy evaluated. In this particular example, download URL will never be called because at least on Python 3 because you should evaluate it first. Yeah, it will return a generator, stuff like that. Okay. Oh, there are a lot of similar tools like Pyflex, Mypy, Pychecker. Pychecker is the forefather of PyLint. I should say a couple of things. Even Pychecker is now defunct and dead for a couple of years. It was way more advanced than many static analysis tools that exist currently in Python. As you can see in this example, we are unpacking three items in two variables or we are having a constant check or we are catching an exception, which is not really an exception, true. And Pyflex, as you can see, detected almost nothing while Pychecker detected all of these errors. Yeah, there's also Jedi and Mypy. Okay. Yeah. One final note is that users, at least a part of them, actually expects this kind of code to be understood. But really, that's not actually possible. So if you want static analysis tools to understand this, just don't do it. Yeah. Yeah, and this is basically the future of PyLint. We'll have PyLint 2.0 next year. We'll have full flow control analysis, better data model, like understanding the scriptors or having the proper attribute access logic. It's not the same in PyLint as it is in Python. And I'm interested in bringing more contributors into the project. And my final slide is, hey, what your talk title is, how do I stop worrying and start loving the bugs and so on? Well, PyLint helps if you're going to use it and if you're going to write as many tests as possible. So use PyLint. Thank you. Thank you. Are we going to any questions or many questions? That's really odd. Yeah, there are a lot of questions. Hi. I actually have a comment or a word of caution and a question. The comment is, if you're going to start PyLint, it will take you some time to configure it to make it shut up about things which don't interest you. But it is well worth the time to do so. And the question was, how did you get into this kind of parsing Python and ASTs? I mean, I tried to write PyLint plugins because there isn't really much documentation for PyLint and that's really hard to get into this topic. I was a user of PyLint before and it had a bug. It didn't detect stuff like it didn't detect unbalanced topo unpacking, like unpacking three items into two items. And I had a bug in production with that stuff and I just wanted PyLint to, hey, tell me about this. So I implemented that check and I started from there. Okay. Thank you. One other question. What are your thoughts about PyLama? PyLama? Yeah. If I'm not mistaken, it's a wrapper over multiple linters like PyLint and Flake8. Yeah. So it just takes PyLint and other stuff. I never use it actually. Okay. More questions? There's one here. I'm not sure what is. Can you be more visible? Here. At the front, okay. Okay. Hello. So regarding the first question, the most common complaint about PyLint is that it has too many checks active. So maybe what if there will be like some sort of configuration wizard that will show you examples and rescue if you want this or not? Yeah. That would be nice to have. But I don't have time to write it myself actually. Sorry. Yeah. That's the question. Is there a sprint on PyLint? Yeah, there is. There is Saturday and Sunday. Thank you. And can you join if you're a beginner? Sorry. Can you join to the sprint if you're a beginner? Yeah, of course. Great. More questions? All right. I think it's fair enough. Thank you again. Closing. Thank you. Okay.
 Thank you guys. It's really nice to see so many people interested in static analysis  here and this year is a special year for Pylint because it's 12th birthday. So, yeah, basically  it's the oldest static analysis tool in Python that is still maintained, which is really  nice. So in this talk, I'm going to talk a little bit about Pylint's history and we'll  go through a detour in static analysis and we'll see where that goes. So, what is this  Lint thing I'm talking about? Basically, a Lint is like a tool you use to analyze your  code in order to find bugs or potential errors or style checks or stuff like that. But saying  this about Pylint is an understatement since it can detect a lot more than a normal Linter  Linter. It's also a style checker which enforces Pepet roles. It's a structural analyzer looking  at your classes and at your special methods and checking if they are implemented correctly.  And it's also a type checker. So, Pylint works by using a technique called static analysis,  which is basically the act of analyzing code without actually executing it. And if you  don't use static analysis right now, you should use it. You should use any other tool, not  just Pylint, because it can really help you in your day-to-day job. For instance, you  can use static analysis if you have a lot of tests and it takes a lot of time to run.  You just want to check that there are no obvious errors in your code or stuff like that. Or  if you have big legacy systems which don't have tests at all, in fact, as my first job  experience, I worked in a company where we have a lot of big legacy systems, but they  didn't have tests. So, using static analysis was paramount for checking that everything  works before going in production. And you can use it as a form of doing reviews. Well,  of course, it's not equivalent to a manual review, but it's better than no review at  all. So, here's a piece of code which has a couple of problems. And I'm going to show  how Pylint detects this kind of stuff. For instance, if you can see, I imported OS, but  I didn't use it across my program. And I also defined a variable which is just there. It's  not used. Of course, this is not a problem per se, but it leads to ugly code or unmaintainable  code or stuff like that. And more important is this block here from line 8 to 10. Let's  just say that I invented that code incorrectly. And that code now will not run because it's  an after-run erase statement. So, basically, it's useless code. Also, at line 10, I intended  to call the execute method, but I didn't. So, that's also a problem. And as you can see  down here, Pylint detects this kind of stuff and properly emits that there is something  fishy in your code. Okay. But we can detect more serious bugs than this, like using undefined  variables or trying to access members that don't exist or calling functions that aren't  functions like calling an integer. And as you can see here, Pylint detects this kind of  stuff. And it's really nice for aesthetic analysis tool to tell you that you made a mistake  in your program. And as I was saying before, Pylint is also a structural analyzer checking  that your special methods are properly implemented or that your classes are written correctly.  And in this case, the dunder exit method needs three parameters in order to work properly.  And Pylint detects this kind of stuff. It emits that warning. Right? Another nice error that  Pylint can detect is this one. Basically, here I have a function and then I have an if statement  with func as a conditional for the if statement. And what I intended here to do initially was  to call func, but I forgot to write the parents. And this statement will always be true. And  that if statement, if branch will always be taken. So if you write this for this piece  of code, you might, how to say, if you write this for this piece of code, you might overlook  it if you didn't test it properly. So Pylint can help you in this kind of situation as  well. Take a couple of moments to figure out what's the problem here. If you know the answer,  you'll really, really loud. Anyone? Sorry? Var is being changed in the loop. So what's  the output of this code? Who said that? Yeah, that's it. Sorry. Basically, what we did is  we created a closure here and Var was looked up into parent scope when called. And Var  was looked up into parent scope and called. And for this particular case, the parent  scope was the least comprehension, at least on Python 3. So when the for loop was executed  and callable was called, basically, Var was the last value from the for loop, which was  9. So that's a bug. And if you try to do a review, you might have missed this stuff  and Pylint can help you deal with this kind of situation. Okay? A little bit of history  about Pylint. It was created in 2003 by a French company called Logilab. They used to  maintain it for a while for 10 years, but now they're not so involved anymore. In fact,  they're not involved anymore in Pylint's development. Even though many still think that about them,  which is wrong, I know that Google uses a modified version of Pylint internally for  their use cases because they had a maintainer for GPylint that was also maintainer for Pylint  and they used to push upstream a lot of changes. And some statistics, according to ohlog.net,  we have over 30,000 lines of code, which is pretty big, but not big enough because we  can detect all the problems in your code. And unfortunately, Pylint is GPL-licensed  because GPL was really cool back then in 2003. It still is. Yeah. And here's my involvement  with Pylint. Basically, I started working on it in 2013 when Pylint 1.0 was released  and up to Pylint 1.3, I became a committer, a maintainer, the only maintainer, and the  guide that pushes the thing forward. I'm planning to release Pylint 2.0 in 2016 if my time allows  it and I'm going to have some advanced techniques in it. It's still right now pretty advanced,  but it's going to be even more advanced than that. Basically, I'm going to use a technique  called abstract interpretation where I'm going to interpret statically your code, not actually  executing, but statically. And also, I'm going to add support for PAP484, what Widow  stock was about, and changing the things internally, like having control flow graphs  and having a symbolic execution engine for it. Yeah. Right. And at this point, you might  ask yourself, okay, this is cool, but how it works and why does it work statically? First  of all, Pylint is split in actually in two components. In the real checker, which is  Pylint, and its inference engine and the component that understands Python, which is Astray.  We're following the general pattern of building a linter. Basically, we're using abstract  syntax trees, even though our abstract syntax trees are augmented with a couple of functionalities  that help us in building the inference engine. And internally, Astray uses the built-in module  AST, which kind of looks like this when using it. You import pars and you give it a string,  and from there, you get an abstract syntax tree, which basically looks like this. So,  if you can see at the top of the tree, we have a lambda, which has a couple of arguments.  And the blue stuff out there, the blue thingy is a node, and the rest of them are just attributes  of those nodes. So, as you can see, it's pretty structured, and you can, just by looking at  it, you can reason about code. Right. Even though AST is great, it's built-in, we don't  have to write our own parser, it's not perfect because it's not backwards compatible, not  even for minor versions of Python releases like 3.4 and 3.5. Basically, they are going  to add new nodes or remove nodes or change things out there. And Astray strives to be  a backward compatible wrapper over AST that you can use for Python 2.7, Python 3.5, and  as well for PyPy or Jyton or any other interpreter which supports abstract syntax trees. And the  API is quite similar with the AST. As you can see, it doesn't differ too much. We have  basically the same functions. Yeah. So, if you're using AST right now and you want something  more capable, you could use Astray. And as I mentioned, our Astray nodes are quite similar  with one from AST, but we augment them with a couple of capabilities specific for our  purposes. Like in this example, for instance, you can actually retrieve the parent of a  node and you can walk up in the tree starting from one node. Also, you can walk down by using  the getchildren method. Right? Also, you can retrieve the lexical scope of a variable.  For instance, in this particular case, the lexical scope of foo is the function where  it is defined. And down there, at least comprehension, basically I'm doing foo for foo in range 10  and the scope of foo, at least on Python 3, is the least comprehension itself. So foo  doesn't leak outside. Yeah. And PyLint can actually know this kind of stuff and Astray.  Some nodes are augmented even more than that. Like in this example, we have a couple of  classes and we're using a meta class, we're using ABC, ABC meta, and we're defining some  slots for our class. And yeah. Let's see what PyLint can do, what Astray can do with this  code. So we can retrieve the slots of the class using the slots method or the meta class  using the meta class method or using the MRO method. We could retrieve the method resolution  order of your classes, which is actually the meta resolution order when running Python over  that script or over that class, which is nice. Yeah. But the most important part about Astray  is not the AST itself, but the capability of doing inference. Basically, inference means  the act of resolving what a node really represents. Like if you have a name and that name represents  something like a variable or a function call, you want to infer that and you want to see  what's at the end of the tree for that particular node. Our nodes basically implement the Python  semantics. For instance, the callfunk inference rule will be the return values of the callfunk  or for least comprehensions, its inference rule would be the actual list that is returned  when evaluating the least comprehension. And our inference also does some partial abstract  interpretation, but it's partial because it's not working for all the cases in your code.  And in this particular example, we have a function which adds two values, the argument,  and as you can see in this particular case, when in inferring what this function actually  returns, in fact, when inferring what's the result of the function called, Astray knows  that it's 48 because 24 plus 24, it's 48. This is a more complex example involving binary  operators. And if you know, the rules for binary operators in Python goes like that.  If you have two different objects and you try to add them, first, the Dunder add method  of the left-hand side will be called. And if that doesn't return not implemented, then  I think the right-hand side object, no, no, I think that the Dunder add method of the  right-hand side object will be called. And in this particular example, we're having  in the left-hand side a supertype, which is A. As you can see, A is a supertype for B  because B has, as its base class, A. And in the right-hand side, we have B, which is a  subtype of A. And the rules are a little bit different in this particular case because  if the semantics would be the same as for the different classes rule, then the first rule,  then the first method, A, Dunder add, will always be called, which is wrong. And in this  particular example, what will happen is that first the Dunder add method of B will be called.  In this particular example, it returns not implemented, so it will fall back to the other  one, which is A, the Dunder add method from A. And if you do the calculations, you can  see that SRAD is right. The result of that operation is actually 45. Now, SRAD is great,  but they can't really understand your code. I mean, they can't really understand your  full code, so we have to deal with this kind of situation. And we're providing a couple  of APIs that could help you, like having node transforms. Basically, with a node transform,  we can modify a part of the AST to be something else, like you have a function call, and instead  of that function call, you want to do inlining, replacing the AST node with the result of  the function call or anything you'd like. And we can do that with this API. Basically,  that API is a function which should accept one parameter, that parameter being the original  node, and should return either the node modified or a new node. And you just register that  function with an internal manager, some implementation detail, but anyway, you just register your  transform function, and when doing SRAD dot parse, that transform function will be called  for that particular type of node. In this example, I'm registering a transform for the  class node, and also you can apply a filter function because you don't actually want to  change all class nodes in your code. You want to change something in particular. And as  you can see there, I'm just giving it filter functions for filtering any other class that  is not 6 add meta class. The same thing can be used for inference rules because at some  point, really, you want to have different inference semantics than the Python offers  you. So using the same API, you can provide a custom inference rule for, I don't know,  you want to infer these comprehensions differently, or you want to infer function calls differently.  So you're using this inference rule, and you're changing basically the semantics of Python  in your AST. And we're using our own dog food in this example, not in this example, we're  using our dog food with inference rules because we have inference rules for built-ins, as  I'll show you immediately. Basically, with inference rules, we're understanding existence,  is subclass, we're understanding the other, has other type, callable, list, set, whatever.  We're understanding, not in particular with inference rules, binary arithmetic operations,  really, really good logical operators or comparisons. Also, we're understanding context managers  and list, set, string indexing, slicing, whatever. Yeah. As earlier, take a couple of moments  to see where are the bugs in this particular code. There should be like three bugs, but  if you can find more, join Pylinx thing. Also, if you know the answer, yell really loud.  And as a hint, how super works, basically, the first argument of super specifies the  object from which the method resolution order will be retrieved.  Yeah, that's good. Because in that particular case, super C and self would be, the method  resolution order in this particular case, I don't know what it is, but super C self  will call B because A doesn't have Boo, yeah, and it's multiple cooperative inheritance,  if I'm not mistaken, and B Boo will be called and B Boo doesn't accept two arguments. Anyone?  Yeah, there's no spa and there is a full. Sorry, sorry, there are too many voices. One  at a time. Full is not color. Exactly. So, Pylinx detects this stuff because asteroid  knows Python really well. And as you can see, what Pylinx says, there are too many position  arguments at line 14, which is actually true. Full is not callable, of course, because full  is an integer. And super V has no spam number, which is actually true because it's spam,  not spam. Yeah. Here's a more complex example of asteroid capabilities. And as you can see  here, asteroid understands, least indexing understands has outer, callable, get outer.  And at that particular line where met is retrieved, what will happen is that has outer call will  return true because A has the method called met. And A met is callable. And get out from  A, we'll return the met object. So it will be like true and true and A dot met. And in  this particular case, the last value, which is also true, will be return. So met will  be A dot met. And then the context manager is invoked. And the context manager will return  real func, which has no argument whatsoever. And we're going to call that. And Pylinx will  say that, hey, you use too many position arguments in your method call. You should change that.  Yeah. So this kind of stuff that asteroid knows about Python code and Pylinx knows about  it as well. Yeah. I don't know. I'm going really fast or how much time do I have? Okay.  Thank you. Pylinx is not so complicated. Astrid is the most complicated thing. Basically,  it's like a fancy walker of the AST. And it has a couple of patterns, more than a couple,  because Pylinx can detect almost 180 type of errors or verifications. And basically,  we are using the visitor pattern to visit each node because that pattern really decouples  your data structures from your algorithms. And a small example of how visitor pattern  works. Let's say we implement in our own checker, visit call func. And in this particular  example, we're importing collections. And we're obtaining the default attribute from  collections. There's no default attribute in collections, by the way, it's default dict.  And what will happen is that visit get other will be called with the node, which will be  the get other node. Afterwards, we'll infer what that node really represents. I mean,  what the parent of the statement really represents, the parent will be the name, the name collections.  And we need to infer it because we should know what that node represents at the end  of the AST. Afterwards, we're calling the data method with the result from the inference.  And if that doesn't raise any error, such as not found that yeah, collections has default  to default attribute. Else, we're going to emit a no member error. But before that,  we're going to have a couple of filters like the owner is a mixing class because you might  not have that attribute in the mixing class. Or I don't know, the owner is a class with  unknown bases, like bases from extension modules. And we currently understand extension  modules, by the way. Yeah. I'll go really fast from now on because I don't have much  time left. Basically, abstract interpretation will have piling for this kind of code. In  this particular example, the dunder date is updated with the dictionary of, I don't know,  what kind of arguments will be passed. And at line five, we will call some arguments  set in dunder in it or whatever other attribute. And that attribute might not exist. And what  happens right now is that piling says, hey, you don't have that attribute. But we're actually  having that attribute because we just inserted it at line two. So that's what abstract interpretation  is going to fix because you will just interpret every line, every logical line in your code  and it will know at the end what side effects each line had. Okay. We have multiple type  of checkers and errors, such as conventions for Pepe rules, refectorings, various warnings  which aren't necessarily bugs in your code. And actual bugs like no member or not global,  things like that. And we have two types of checkers, AST based and token based. The  token based checkers are for, I don't know, line is too long or bad indentation or other  similar examples. Piling has a really vibrant community. There are a lot of external packages  for improving in-furnace, for improving piling checkers out there. You could write your own  if you want to. The problem is that they are pure Python, so you need to write Python code  in order to have a custom checker. Yeah, it comes with a couple of features, extra features  like you could generate UML diagrams from your code or you could spell check your doc  strings or comments. These are disabled by default. Or you could use the Python 3 porting  checker, which is a checker when activated, all other checkers are disabled and it will  emit stuff that it's not going to work in Python 3 anymore. Like using remove syntax  or remove built-ins or special methods. Or my favorite is using map filter or reduce  in non-interval contexts. As you can see in this particular example, on Python 3, map  is lazy evaluated. In this particular example, download URL will never be called because at  least on Python 3 because you should evaluate it first. Yeah, it will return a generator,  stuff like that. Okay. Oh, there are a lot of similar tools like Pyflex, Mypy, Pychecker.  Pychecker is the forefather of PyLint. I should say a couple of things. Even Pychecker is  now defunct and dead for a couple of years. It was way more advanced than many static analysis  tools that exist currently in Python. As you can see in this example, we are unpacking  three items in two variables or we are having a constant check or we are catching an exception,  which is not really an exception, true. And Pyflex, as you can see, detected almost nothing  while Pychecker detected all of these errors. Yeah, there's also Jedi and Mypy. Okay. Yeah.  One final note is that users, at least a part of them, actually expects this kind of code  to be understood. But really, that's not actually possible. So if you want static analysis tools  to understand this, just don't do it. Yeah. Yeah, and this is basically the future of  PyLint. We'll have PyLint 2.0 next year. We'll have full flow control analysis, better  data model, like understanding the scriptors or having the proper attribute access logic.  It's not the same in PyLint as it is in Python. And I'm interested in bringing more contributors  into the project. And my final slide is, hey, what your talk title is, how do I stop worrying  and start loving the bugs and so on? Well, PyLint helps if you're going to use it and  if you're going to write as many tests as possible. So use PyLint. Thank you.  Thank you. Are we going to any questions or many questions? That's really odd. Yeah, there  are a lot of questions. Hi. I actually have a comment or a word of caution and a question.  The comment is, if you're going to start PyLint, it will take you some time to configure it  to make it shut up about things which don't interest you. But it is well worth the time  to do so. And the question was, how did you get into this kind of parsing Python and ASTs?  I mean, I tried to write PyLint plugins because there isn't really much documentation for  PyLint and that's really hard to get into this topic.  I was a user of PyLint before and it had a bug. It didn't detect stuff like it didn't  detect unbalanced topo unpacking, like unpacking three items into two items. And I had a bug  in production with that stuff and I just wanted PyLint to, hey, tell me about this. So I implemented  that check and I started from there. Okay. Thank you.  One other question. What are your thoughts about PyLama?  PyLama? Yeah.  If I'm not mistaken, it's a wrapper over multiple linters like PyLint and Flake8.  Yeah. So it just takes PyLint and other stuff.  I never use it actually. Okay.  More questions?  There's one here.  I'm not sure what is. Can you be more visible?  Here.  At the front, okay.  Okay.  Hello. So regarding the first question, the most common complaint about PyLint is that  it has too many checks active. So maybe what if there will be like some sort of configuration  wizard that will show you examples and rescue if you want this or not?  Yeah. That would be nice to have.  But I don't have time to write it myself actually.  Sorry.  Yeah.  That's the question. Is there a sprint on PyLint?  Yeah, there is. There is Saturday and Sunday.  Thank you. And can you join if you're a beginner?  Sorry.  Can you join to the sprint if you're a beginner?  Yeah, of course.  Great.  More questions? All right. I think it's fair enough.  Thank you again.  Closing.  Thank you.  Okay.
Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__0_527.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__528_743.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__744_839.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__840_2543.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__2544_2723.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__2724_6911.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__6912_7163.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__7164_20339.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__20340_22475.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__22476_26591.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__26592_29951.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__29952_30299.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__30300_30347.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__30348_30731.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__30732_34931.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__34932_34991.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__34992_35087.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__35088_35387.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__35388_36527.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__36528_36671.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__36672_37091.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__37092_37139.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__37140_45767.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__45768_46055.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__46056_47039.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__47040_48743.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__48744_49091.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__49092_49415.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__49416_49487.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__49488_49955.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__49956_52175.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__52176_53207.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__53208_53327.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__53328_53507.png Claudiu_Popa_-_12_years_of_Pylint__or_How_I_learned_to_stop_worrying_about_bugs__53508_54205.png
Thank you guys. It's really nice to see so many people interested in static analysis here and this year is a special year for Pylint because it's 12th birthday. So, yeah, basically it's the oldest static analysis tool in Python that is still maintained, which is really nice. So in this talk, I'm going to talk a little bit about Pylint's history and we'll go through a detour in static analysis and we'll see where that goes. So, what is this Lint thing I'm talking about? Basically, a Lint is like a tool you use to analyze your code in order to find bugs or potential errors or style checks or stuff like that. But saying this about Pylint is an understatement since it can detect a lot more than a normal Linter Linter. It's also a style checker which enforces Pepet roles. It's a structural analyzer looking at your classes and at your special methods and checking if they are implemented correctly. And it's also a type checker. So, Pylint works by using a technique called static analysis, which is basically the act of analyzing code without actually executing it. And if you don't use static analysis right now, you should use it. You should use any other tool, not just Pylint, because it can really help you in your day-to-day job. For instance, you can use static analysis if you have a lot of tests and it takes a lot of time to run. You just want to check that there are no obvious errors in your code or stuff like that. Or if you have big legacy systems which don't have tests at all, in fact, as my first job experience, I worked in a company where we have a lot of big legacy systems, but they didn't have tests. So, using static analysis was paramount for checking that everything works before going in production. And you can use it as a form of doing reviews. Well, of course, it's not equivalent to a manual review, but it's better than no review at all. So, here's a piece of code which has a couple of problems. And I'm going to show how Pylint detects this kind of stuff. For instance, if you can see, I imported OS, but I didn't use it across my program. And I also defined a variable which is just there. It's not used. Of course, this is not a problem per se, but it leads to ugly code or unmaintainable code or stuff like that. And more important is this block here from line 8 to 10. Let's just say that I invented that code incorrectly. And that code now will not run because it's an after-run erase statement. So, basically, it's useless code. Also, at line 10, I intended to call the execute method, but I didn't. So, that's also a problem. And as you can see down here, Pylint detects this kind of stuff and properly emits that there is something fishy in your code. Okay. But we can detect more serious bugs than this, like using undefined variables or trying to access members that don't exist or calling functions that aren't functions like calling an integer. And as you can see here, Pylint detects this kind of stuff. And it's really nice for aesthetic analysis tool to tell you that you made a mistake in your program. And as I was saying before, Pylint is also a structural analyzer checking that your special methods are properly implemented or that your classes are written correctly. And in this case, the dunder exit method needs three parameters in order to work properly. And Pylint detects this kind of stuff. It emits that warning. Right? Another nice error that Pylint can detect is this one. Basically, here I have a function and then I have an if statement with func as a conditional for the if statement. And what I intended here to do initially was to call func, but I forgot to write the parents. And this statement will always be true. And that if statement, if branch will always be taken. So if you write this for this piece of code, you might, how to say, if you write this for this piece of code, you might overlook it if you didn't test it properly. So Pylint can help you in this kind of situation as well. Take a couple of moments to figure out what's the problem here. If you know the answer, you'll really, really loud. Anyone? Sorry? Var is being changed in the loop. So what's the output of this code? Who said that? Yeah, that's it. Sorry. Basically, what we did is we created a closure here and Var was looked up into parent scope when called. And Var was looked up into parent scope and called. And for this particular case, the parent scope was the least comprehension, at least on Python 3. So when the for loop was executed and callable was called, basically, Var was the last value from the for loop, which was 9. So that's a bug. And if you try to do a review, you might have missed this stuff and Pylint can help you deal with this kind of situation. Okay? A little bit of history about Pylint. It was created in 2003 by a French company called Logilab. They used to maintain it for a while for 10 years, but now they're not so involved anymore. In fact, they're not involved anymore in Pylint's development. Even though many still think that about them, which is wrong, I know that Google uses a modified version of Pylint internally for their use cases because they had a maintainer for GPylint that was also maintainer for Pylint and they used to push upstream a lot of changes. And some statistics, according to ohlog.net, we have over 30,000 lines of code, which is pretty big, but not big enough because we can detect all the problems in your code. And unfortunately, Pylint is GPL-licensed because GPL was really cool back then in 2003. It still is. Yeah. And here's my involvement with Pylint. Basically, I started working on it in 2013 when Pylint 1.0 was released and up to Pylint 1.3, I became a committer, a maintainer, the only maintainer, and the guide that pushes the thing forward. I'm planning to release Pylint 2.0 in 2016 if my time allows it and I'm going to have some advanced techniques in it. It's still right now pretty advanced, but it's going to be even more advanced than that. Basically, I'm going to use a technique called abstract interpretation where I'm going to interpret statically your code, not actually executing, but statically. And also, I'm going to add support for PAP484, what Widow stock was about, and changing the things internally, like having control flow graphs and having a symbolic execution engine for it. Yeah. Right. And at this point, you might ask yourself, okay, this is cool, but how it works and why does it work statically? First of all, Pylint is split in actually in two components. In the real checker, which is Pylint, and its inference engine and the component that understands Python, which is Astray. We're following the general pattern of building a linter. Basically, we're using abstract syntax trees, even though our abstract syntax trees are augmented with a couple of functionalities that help us in building the inference engine. And internally, Astray uses the built-in module AST, which kind of looks like this when using it. You import pars and you give it a string, and from there, you get an abstract syntax tree, which basically looks like this. So, if you can see at the top of the tree, we have a lambda, which has a couple of arguments. And the blue stuff out there, the blue thingy is a node, and the rest of them are just attributes of those nodes. So, as you can see, it's pretty structured, and you can, just by looking at it, you can reason about code. Right. Even though AST is great, it's built-in, we don't have to write our own parser, it's not perfect because it's not backwards compatible, not even for minor versions of Python releases like 3.4 and 3.5. Basically, they are going to add new nodes or remove nodes or change things out there. And Astray strives to be a backward compatible wrapper over AST that you can use for Python 2.7, Python 3.5, and as well for PyPy or Jyton or any other interpreter which supports abstract syntax trees. And the API is quite similar with the AST. As you can see, it doesn't differ too much. We have basically the same functions. Yeah. So, if you're using AST right now and you want something more capable, you could use Astray. And as I mentioned, our Astray nodes are quite similar with one from AST, but we augment them with a couple of capabilities specific for our purposes. Like in this example, for instance, you can actually retrieve the parent of a node and you can walk up in the tree starting from one node. Also, you can walk down by using the getchildren method. Right? Also, you can retrieve the lexical scope of a variable. For instance, in this particular case, the lexical scope of foo is the function where it is defined. And down there, at least comprehension, basically I'm doing foo for foo in range 10 and the scope of foo, at least on Python 3, is the least comprehension itself. So foo doesn't leak outside. Yeah. And PyLint can actually know this kind of stuff and Astray. Some nodes are augmented even more than that. Like in this example, we have a couple of classes and we're using a meta class, we're using ABC, ABC meta, and we're defining some slots for our class. And yeah. Let's see what PyLint can do, what Astray can do with this code. So we can retrieve the slots of the class using the slots method or the meta class using the meta class method or using the MRO method. We could retrieve the method resolution order of your classes, which is actually the meta resolution order when running Python over that script or over that class, which is nice. Yeah. But the most important part about Astray is not the AST itself, but the capability of doing inference. Basically, inference means the act of resolving what a node really represents. Like if you have a name and that name represents something like a variable or a function call, you want to infer that and you want to see what's at the end of the tree for that particular node. Our nodes basically implement the Python semantics. For instance, the callfunk inference rule will be the return values of the callfunk or for least comprehensions, its inference rule would be the actual list that is returned when evaluating the least comprehension. And our inference also does some partial abstract interpretation, but it's partial because it's not working for all the cases in your code. And in this particular example, we have a function which adds two values, the argument, and as you can see in this particular case, when in inferring what this function actually returns, in fact, when inferring what's the result of the function called, Astray knows that it's 48 because 24 plus 24, it's 48. This is a more complex example involving binary operators. And if you know, the rules for binary operators in Python goes like that. If you have two different objects and you try to add them, first, the Dunder add method of the left-hand side will be called. And if that doesn't return not implemented, then I think the right-hand side object, no, no, I think that the Dunder add method of the right-hand side object will be called. And in this particular example, we're having in the left-hand side a supertype, which is A. As you can see, A is a supertype for B because B has, as its base class, A. And in the right-hand side, we have B, which is a subtype of A. And the rules are a little bit different in this particular case because if the semantics would be the same as for the different classes rule, then the first rule, then the first method, A, Dunder add, will always be called, which is wrong. And in this particular example, what will happen is that first the Dunder add method of B will be called. In this particular example, it returns not implemented, so it will fall back to the other one, which is A, the Dunder add method from A. And if you do the calculations, you can see that SRAD is right. The result of that operation is actually 45. Now, SRAD is great, but they can't really understand your code. I mean, they can't really understand your full code, so we have to deal with this kind of situation. And we're providing a couple of APIs that could help you, like having node transforms. Basically, with a node transform, we can modify a part of the AST to be something else, like you have a function call, and instead of that function call, you want to do inlining, replacing the AST node with the result of the function call or anything you'd like. And we can do that with this API. Basically, that API is a function which should accept one parameter, that parameter being the original node, and should return either the node modified or a new node. And you just register that function with an internal manager, some implementation detail, but anyway, you just register your transform function, and when doing SRAD dot parse, that transform function will be called for that particular type of node. In this example, I'm registering a transform for the class node, and also you can apply a filter function because you don't actually want to change all class nodes in your code. You want to change something in particular. And as you can see there, I'm just giving it filter functions for filtering any other class that is not 6 add meta class. The same thing can be used for inference rules because at some point, really, you want to have different inference semantics than the Python offers you. So using the same API, you can provide a custom inference rule for, I don't know, you want to infer these comprehensions differently, or you want to infer function calls differently. So you're using this inference rule, and you're changing basically the semantics of Python in your AST. And we're using our own dog food in this example, not in this example, we're using our dog food with inference rules because we have inference rules for built-ins, as I'll show you immediately. Basically, with inference rules, we're understanding existence, is subclass, we're understanding the other, has other type, callable, list, set, whatever. We're understanding, not in particular with inference rules, binary arithmetic operations, really, really good logical operators or comparisons. Also, we're understanding context managers and list, set, string indexing, slicing, whatever. Yeah. As earlier, take a couple of moments to see where are the bugs in this particular code. There should be like three bugs, but if you can find more, join Pylinx thing. Also, if you know the answer, yell really loud. And as a hint, how super works, basically, the first argument of super specifies the object from which the method resolution order will be retrieved. Yeah, that's good. Because in that particular case, super C and self would be, the method resolution order in this particular case, I don't know what it is, but super C self will call B because A doesn't have Boo, yeah, and it's multiple cooperative inheritance, if I'm not mistaken, and B Boo will be called and B Boo doesn't accept two arguments. Anyone? Yeah, there's no spa and there is a full. Sorry, sorry, there are too many voices. One at a time. Full is not color. Exactly. So, Pylinx detects this stuff because asteroid knows Python really well. And as you can see, what Pylinx says, there are too many position arguments at line 14, which is actually true. Full is not callable, of course, because full is an integer. And super V has no spam number, which is actually true because it's spam, not spam. Yeah. Here's a more complex example of asteroid capabilities. And as you can see here, asteroid understands, least indexing understands has outer, callable, get outer. And at that particular line where met is retrieved, what will happen is that has outer call will return true because A has the method called met. And A met is callable. And get out from A, we'll return the met object. So it will be like true and true and A dot met. And in this particular case, the last value, which is also true, will be return. So met will be A dot met. And then the context manager is invoked. And the context manager will return real func, which has no argument whatsoever. And we're going to call that. And Pylinx will say that, hey, you use too many position arguments in your method call. You should change that. Yeah. So this kind of stuff that asteroid knows about Python code and Pylinx knows about it as well. Yeah. I don't know. I'm going really fast or how much time do I have? Okay. Thank you. Pylinx is not so complicated. Astrid is the most complicated thing. Basically, it's like a fancy walker of the AST. And it has a couple of patterns, more than a couple, because Pylinx can detect almost 180 type of errors or verifications. And basically, we are using the visitor pattern to visit each node because that pattern really decouples your data structures from your algorithms. And a small example of how visitor pattern works. Let's say we implement in our own checker, visit call func. And in this particular example, we're importing collections. And we're obtaining the default attribute from collections. There's no default attribute in collections, by the way, it's default dict. And what will happen is that visit get other will be called with the node, which will be the get other node. Afterwards, we'll infer what that node really represents. I mean, what the parent of the statement really represents, the parent will be the name, the name collections. And we need to infer it because we should know what that node represents at the end of the AST. Afterwards, we're calling the data method with the result from the inference. And if that doesn't raise any error, such as not found that yeah, collections has default to default attribute. Else, we're going to emit a no member error. But before that, we're going to have a couple of filters like the owner is a mixing class because you might not have that attribute in the mixing class. Or I don't know, the owner is a class with unknown bases, like bases from extension modules. And we currently understand extension modules, by the way. Yeah. I'll go really fast from now on because I don't have much time left. Basically, abstract interpretation will have piling for this kind of code. In this particular example, the dunder date is updated with the dictionary of, I don't know, what kind of arguments will be passed. And at line five, we will call some arguments set in dunder in it or whatever other attribute. And that attribute might not exist. And what happens right now is that piling says, hey, you don't have that attribute. But we're actually having that attribute because we just inserted it at line two. So that's what abstract interpretation is going to fix because you will just interpret every line, every logical line in your code and it will know at the end what side effects each line had. Okay. We have multiple type of checkers and errors, such as conventions for Pepe rules, refectorings, various warnings which aren't necessarily bugs in your code. And actual bugs like no member or not global, things like that. And we have two types of checkers, AST based and token based. The token based checkers are for, I don't know, line is too long or bad indentation or other similar examples. Piling has a really vibrant community. There are a lot of external packages for improving in-furnace, for improving piling checkers out there. You could write your own if you want to. The problem is that they are pure Python, so you need to write Python code in order to have a custom checker. Yeah, it comes with a couple of features, extra features like you could generate UML diagrams from your code or you could spell check your doc strings or comments. These are disabled by default. Or you could use the Python 3 porting checker, which is a checker when activated, all other checkers are disabled and it will emit stuff that it's not going to work in Python 3 anymore. Like using remove syntax or remove built-ins or special methods. Or my favorite is using map filter or reduce in non-interval contexts. As you can see in this particular example, on Python 3, map is lazy evaluated. In this particular example, download URL will never be called because at least on Python 3 because you should evaluate it first. Yeah, it will return a generator, stuff like that. Okay. Oh, there are a lot of similar tools like Pyflex, Mypy, Pychecker. Pychecker is the forefather of PyLint. I should say a couple of things. Even Pychecker is now defunct and dead for a couple of years. It was way more advanced than many static analysis tools that exist currently in Python. As you can see in this example, we are unpacking three items in two variables or we are having a constant check or we are catching an exception, which is not really an exception, true. And Pyflex, as you can see, detected almost nothing while Pychecker detected all of these errors. Yeah, there's also Jedi and Mypy. Okay. Yeah. One final note is that users, at least a part of them, actually expects this kind of code to be understood. But really, that's not actually possible. So if you want static analysis tools to understand this, just don't do it. Yeah. Yeah, and this is basically the future of PyLint. We'll have PyLint 2.0 next year. We'll have full flow control analysis, better data model, like understanding the scriptors or having the proper attribute access logic. It's not the same in PyLint as it is in Python. And I'm interested in bringing more contributors into the project. And my final slide is, hey, what your talk title is, how do I stop worrying and start loving the bugs and so on? Well, PyLint helps if you're going to use it and if you're going to write as many tests as possible. So use PyLint. Thank you. Thank you. Are we going to any questions or many questions? That's really odd. Yeah, there are a lot of questions. Hi. I actually have a comment or a word of caution and a question. The comment is, if you're going to start PyLint, it will take you some time to configure it to make it shut up about things which don't interest you. But it is well worth the time to do so. And the question was, how did you get into this kind of parsing Python and ASTs? I mean, I tried to write PyLint plugins because there isn't really much documentation for PyLint and that's really hard to get into this topic. I was a user of PyLint before and it had a bug. It didn't detect stuff like it didn't detect unbalanced topo unpacking, like unpacking three items into two items. And I had a bug in production with that stuff and I just wanted PyLint to, hey, tell me about this. So I implemented that check and I started from there. Okay. Thank you. One other question. What are your thoughts about PyLama? PyLama? Yeah. If I'm not mistaken, it's a wrapper over multiple linters like PyLint and Flake8. Yeah. So it just takes PyLint and other stuff. I never use it actually. Okay. More questions? There's one here. I'm not sure what is. Can you be more visible? Here. At the front, okay. Okay. Hello. So regarding the first question, the most common complaint about PyLint is that it has too many checks active. So maybe what if there will be like some sort of configuration wizard that will show you examples and rescue if you want this or not? Yeah. That would be nice to have. But I don't have time to write it myself actually. Sorry. Yeah. That's the question. Is there a sprint on PyLint? Yeah, there is. There is Saturday and Sunday. Thank you. And can you join if you're a beginner? Sorry. Can you join to the sprint if you're a beginner? Yeah, of course. Great. More questions? All right. I think it's fair enough. Thank you again. Closing. Thank you. Okay.
oldest static analysis tool in Python that is this talk, I'm going to talk a little bit about Pylint's history and we'll go through a detour in static analysis and we'll see where that goes. understatement since it can detect a lot more than a normal Linter Linter. It's also a style at your special methods and checking if they are implemented correctly. And it's also a type review, but it's better than no review at all. problems. And I'm going to show how Pylint exist or calling functions that aren't functions you that you made a mistake in your program. And Pylint can help you in this kind of situation as which is Pylint, and its inference engine and the component that understands Python, which is abstract syntax trees, even though our abstract slots of the class using the slots method or the resolution order of your classes, which is meta class. The same thing can be used for built-ins, as I'll show you immediately. Basically, with inference rules, we're and self would be, the method resolution order in don't know, what kind of arguments will be Python code in order to have a custom checker. example, on Python 3, map is lazy evaluated. In analysis tools that exist currently in Python. As how do I stop worrying and start loving the bugs like some sort of configuration wizard that will