Unleash the power of C++ in Python
Maureira-Fredes, Cristián
One of the main Python aspects is to have a clear syntax and be an easy-to-understand language, which compared to others like C++ (depending of the kindness of the developer) can make a huge difference. Not only the readability is in play, having a dynamically typed and interpreted language improves the development workflow. This improvement comes with a cost, performance. In most of the cases, a raw comparison of simple routines will leave Python behind, but there is a partial solution to this problem, connecting a powerful performant C++ library with the simplicity of Python. In this talk, we will go through the process of generating language bindings to allow you to take any amazing C or C++ project and bring it to Python land. We will briefly discuss one of the many success stories, and we will analyze how the Qt project managed to expose its whole framework to Python, with the help of a self-made binding generator called Shiboken. After the talk, you will be able to decide which option is better for the projects you have in mind, and with a bit of luck you will be responsible of exposing the next popular C++ library that makes the life of us all Pythonistas easier.
 Hello, everyone. This is one room more that I was expecting for, because I know that there's also amazing talks that I would like to go to at this moment. So if you want, we can all go to the other talks, or we can stay here for some fun with C++, of course. And I know it's just the last talk from all the tracks, so bear with me. There will be no weird things here, and we will not like be trying to understand obfuscated C++ or something else. So it's a really light talk with advanced topics. So let's go for it. So as you heard, my name is Christian. Currently I'm working as a software engineer in the Acute Company, which one of the companies that support the Acute project, maybe you heard about it. Later I will have one slide to explain the motivation behind this talk, and why I'm talking about C++ on Python. So first of all, don't worry about the examples or the slides or whatever. Just go to that GitHub repository. There is a link to the slides, and you can get all the source code. Oh, my God, I feel like a star. Everyone taking pictures. I'm like, stop it. You can check it out. At the end of the talk, also, is my GitHub username, so you can check it out. It's called UnleashedCBP. Okay. So first of all, the Acute project in general is a C++ framework, huge framework, and with starting to do some Python stuff, you need to have the mandatory comparison between two languages. First of all, we know that both languages are general purpose. Also, you have multi-paradigm. You can do many things with them. And then we start to notice some difference that I'm pretty sure that most of you are aware of, dynamically type, static-ally type, compile, interpret, and then memory management and about also code reliability. So I know this is kind of if we have just raised your hand, who is currently writing C or C++ code? Okay. Good. So, I feel your pain, too. So I know that sometimes C++ could be complicated, but it's just one to show you that it's not and you can even make things in Python better. So first of all, everyone knows that Python is beautiful. I remember the first time that I typed that on a console when I was studying computer science and I was, wow, everything works. But if you think about it, I mean, C++ is not so different, right? I mean, you can write something. Like maybe you need to run an external compiler or something, but you achieve the same thing, right? So, okay. After all, C++ is simple. It's not. Because then you have things like meta-programming and you start, you know, templates over templates and this is one of the reasons that people get scared of C++. But C++ is not so ugly and this is some extreme cases of people with a lot of time, a lot of effort and try to do these complicated things like compile time if. But let's go about one of the goodies, a couple of goodies that C++ and the latest standards being 11 on has. I mean, if you're familiar with C++, now there is this amazing auto type that kind of detects the variable that you have, which is kind of a similar, like a more dynamic language, but not. You can even use some functions to get the declarative time of a different variable to declare another one and stuff. And then again, we have the for loops that with time it has been improved. I mean, this is kind of the more dumb old version that we have in C++ to kind of like go in a vector. I know that you can do also with an iterator too, but it's kind of like going element by element, but it's too complicated. So then you can do more fun things like this. And if you remember the other one, then you can kind of go, which is not so complicated to understand. Also if you're a fan of Python lambdas, you can also do it in C++ with the latest standards and so on and so forth. And you can have this for each handling a nice lambda there to just sum it up some values. But the most interesting things about C++ are coming, in my opinion, in C++ 20, which is, I don't know, I have this relationship with love and hate with C++ 20. There are some things that are really nice and some things that I have no idea why people is doing it. But one of the things that I like is this one. So you have this amount, sorry, this type of ranges that you can use now. And if you see the four, the kind of like the type of the four there is kind of similar that you can have there. So after all, I mean, it's not so crazy different, right? The same with a more example, I mean, I know that this mod way maybe is the first line that is commented out, but you can write the function maybe like the second line that you see there, which is also quite similar what you can achieve in C++ 20. So after all, I mean, it's just adding some semicolon sentient, right? So the idea is that now it's not about like which language is best, but how to make both languages to improve each other and try to collaborate in a more efficient way. So I want to propose now a new language that is, no, no, it's not. But I'll just talk it to extending Python with C++. And for this, we will go through a couple of examples, really simple ones. And the first of all, who here didn't know before the conference that Python was written in C? We had some, okay, we had some people. Okay. So this is something that also was kind of shocking to me because I thought that it was some black magic assembly stuff, but it was written in C. So if you learn some C, you could understand. And then you could understand other things like this joke, for example. I know that is really bad, but it's the only joke that I have on the slides. So I'm really sorry, but I needed to do it. So let's take a look at how we can create an amazing module based in pure C Python. Okay. Let's go for it. So the first thing that you need to know is that you just need to see C file. And we will go through these examples with the most simple thing that I would think. There was a hello world function that just returns the char star or string or whatever. So I hope that everyone can read. If you don't know C or C++, it's kind of that you get the idea. It's kind of a function declaration that returns some message, which is hello, EuroPython 2019. Okay. So for building a C++, I will just sum out. Don't be afraid just to show you. So that's all the code that we required to create a new Python module written in C Python, which one function that is just printing hello world. It's not so scary, right? I mean, I need to confess the first time that I started to play around with this, I was just copying all these weird structures that you have around and all these widers, null, null, zero and other things. Just copy some stuff and it works. So if you are brave enough after this talk, just go to the C Python directory and play with it and maybe you can write other sophisticated models with it. So this is a simple case, right? And this is just one example, one simple function and we have this. So what's the motivation behind this Qt thing? So Qt is just a C++ framework that is like huge and one of the main functionalities was to offer the way of developing user interface, graphical user interfaces for people. And also Qt implements many of the things that we have for free in Python in C++, which is like a lot of abstractions for different classes to interact with more databases or maybe doing command line or scripting alike in C++ and even other advanced things that different way of creating user interfaces, actions, notification, threats, all these things. So they are providing a lot of goodies to the C++ world and at some point the project was, but you know what? We should, I mean, this thing need to be maybe a value in other languages. So people started to wonder about Python. So maybe you are aware there is already a set of binding which is really old, it's called PyQt and this was developed by a different company, some people doing a lot of work there and this was fantastic. And in the Qt project they wanted to kind of have an official adoption of these things so they decided maybe we can create a new set of bindings and just put it there and see it's an open market so people can decide which to choose. At the moment they are at the same level, one is really old and there's a lot of examples and tutorials. But the Qt project at that point had a couple of options to do this with this huge C++ framework. So they could write raw C Python, they could write a Sweak, I mean maybe you heard about Sweak, or they could use boost Python. So if you take a look at the Sweak implementation, so with Sweak we just for you to believe me we have here our difficult function to understand and the only things that we need to do for the Sweak is just to create an interface. So this interface is in an i-file that also nothing really scary or something at the moment. So we have a couple of instructions to build it, I mean we run Sweak and then we compile some stuff and then we can just use a normal simple import the module and execute it. And then we have the message, nothing too fancy. With boost Python it's kind of a similar story. We go there again, we have the simple Cpp file with our function there and then we just need to define boost Python module, the name, the namespace and then we define the function. Nothing too scary at the moment, right? So this was looking at a really nice solution, right? I mean we were achieving the same thing but the problem lied that many things that we do in C++ need to be specially treated inside Python. For example ownership of objects. If you are wrapping an interface that is in C++ who owns the objects? Some cases will be Python, some cases will be C++. So you need to deal with all these things. So the option that at that time the developer had was okay, this is just one of the generators and we can maybe modify the wrapper generated code. So let's take a look at the code that Sweak generates. So just for you to have an idea. So this is kind of the code that generates and I will start to go down. We are in 2%, 3%. Okay, we'll just page down. So I understand it. It's a little bit too much with this small function, right? So that's okay. That's all. And there is kind of a motivation to say, okay, I know that they need to set up many things to get for granted and to automate many processes and it's a really smart solution. But if you want to modify this file, it will be embeddable. I mean you cannot do it. So there was a lot of motivation behind this Cpython and Sweak and Boost Python. I don't have the source code from the Boost Python here but the shared library is roughly the same size so you can imagine that there are a lot of black magic there. So what did they say? Okay, they started the development of this new tool based in Boost at the moment and they said, no, it's too heavy. Let's write our own thing. Okay, everything was released properly. Then there was the project to continue the development and the good news was that last year officially was released this new set of bindings of Qt in Python and then the name of the new project was called Qt for Python. So maybe you heard about this or PySide tool, it was all this story behind. But okay, this is just a story but what was the important part, at least in my opinion, how do we do it? So this tool that you see here in the center called Gvogan 2 is kind of like the response for writing our own code generator tool to expose all this huge C++ framework to Python. So there is a module inside that kind of extracts all the API information from Qt, it's based in Clang, of course, if you want to do smart things in C++, you need to use Clang. Then we have a support type system which is nothing else than XML file. Then you get, wait, something is being weird, an XML file where you can define all these things that I told you about ownership or, for example, what do we do if a function has a void star argument in Python? We need to have a special or type or treatment or do some casting or something around. So this tool graphs this type system, the information from the framework and generates some wrapper, the one that you saw from SWIG but of course more reduced and it makes more sense and it's more clean in my opinion. And then with this we can just compile it and have the same Python module that we have. So this tool is called Gvogan, the documentation is there and I don't know what is happening, I don't know if I'm doing anything and the slides are coming. Okay, so the Japanese Ganges doesn't mean anything, if there is, we have some Japanese speaker there here, it's just three words that make no sense but let's take a look of the Gvogan thing. Okay. So again, we will go first for you to believe me that we have here the implementation and we are like cheating a little bit because of course we are working with strings from the standard library, then we define complex header for this which is just that and for this type system that I just told you, it's nothing else than this. So we've had HTML, we said this would be a package called simple, it will have a primitive type, there's a string from standard library and a hello function. After all the compiling and stuff, we will get something like this. Let's go to build and it's inside simple and here you have the module wrapper. I will zoom out just for you to show you that I'm not lying. It's long but it's not so long as the other one. So here you have it, I don't expect you to read the code, I just want to show you the magnitude of the code. So at least this is being code automatically generated, calling this C++ function and exposing everything with Python to be able to use it from Python. So it's way shorter than the option that we had. Of course this translates into having a more lightweight shared libraries and you can achieve the same things. So there are other nice options out there and for this I kind of tend to recommend people to take a look of all the solutions that you have out there because it's unfair to say yeah, this is the best solution. No, it's not. I mean there are many things that you can complement to each other. I don't know if you have the chance or maybe know one of these ones. The last one for example is the one that the other set of binding called PyQt uses which is called SIP. PyBind 11, also it's a really nice project that appeared I think a couple of years ago, even I think two EuroPythons ago there was also a talk about PyBind 11. So let's take a look of like how do you achieve the same things with the other options. I think I am good with the time at the moment. So this is the case of PyBind 11. So let's open again the file. If you see the motivation here is clearly from boost Python. I mean it's again a kind of a macro that kind of defines a module, then you have the definition of the function, some documentation and so on and so forth. And then you can do the same thing. I mean after you compile it and everything you have your simple main that you can achieve the same. So it's kind of like same idea as boost Python but they are doing way more things and there is a lot of nice support there that I encourage you to check it out. CFFI. Well this is again not really fair comparison because CFFI, we had some talks already that we explained about this thing, it's not kind of like generating code but it's just kind of floating something. So we can have this is the one here. This is the code that is being generated. So we go to the simple build and then you have like inline raw string that contains the function that we want to expose. And then we just compile this thing in some share library that we can easily load from Python. So if you check here for example, the main is the same thing that we were seeing in the other examples, just calling this nice function and in this case we are casting it to a string and so on and so forth. But it's kind of the same idea. But again, I mean it's not like you can write inline code, kind of read lots of C++ code to kind of expose it in this way. The other option that I had there, ah, of course. If CFFI is kind of focused on C, they support C89 and I don't remember which one, all their standard. But for playing with C++ you have a similar idea which is CPPYY. Kind of fine, right? CPPYY. And then it's kind of the same idea. So you declare everything like in a string and then you have this declaration that you can use this from the GBL. Again, same idea and nothing too scary. The last one is kind of similar. It was a zip that I showed you, the one that is using the PyCute bindings. I will just show you, this is just there. So here we have the simple, again, same idea but the only difference that they require to define a zip file. And it's nothing like the other set this. I mean it's also quite simple. I mean you just need to define the module and then you have some include and then the signature function. So there are many options out there to achieve the same thing, right? But what's the idea of doing this, right? I have been having some conversation with people sometimes that they say, ah, yeah, but I mean I use Python for everything. I mean you don't need C++. And they say, okay, what do you use? No, I do everything with numpy because it's amazing and it's fast and it's Python, you know? And then it's like, yeah, I mean, you know, it's written in C... Oh, no, it's not. Okay, there's even some people that is not aware that many of the popular libraries out there are using C++ or C. So in this case you have a case of numpy, I just downloaded the source code. The first line is all the Python files that you have that are not test and then you have the C files there, which is 96, which is still a lot of things that they are doing. And if you want to maybe refer to a more modern module, the case of PyTorch is a little bit more extreme. And if you look there, we have... No, this is... This was... This is... So here we have 547 C++ files inside the PyTorch because of course it's based in Torch and Torch is C++, so everything is C++ inside. So it's kind of a good... And I use the motivation of using all these tools to start to write things to improve the Python. We are all here. If we know C or C++, we kind of have some responsibility in our hands to improve things. And one of the examples that just out of curiosity and please don't blame me about the things that I will show you is that simple case, I was helping someone that I was listing some files in Python and getting all different absolute path and what do you use there? Glob, right? How do you use Glob? Or if you are more like up-to-date person, you use the bat leaf and then also you have the Glob access there. So for this, you know the deal, I mean writing Glob is... Things are kind of simple. This is a hard-coded thing just to play with the recursion thing, but you know import Glob and then you need to double stars there and specify if it's recursive or not. The same way with bat leaf is kind of the same idea, right? And then I thought, I mean let's look at the implementation and see if we can kind of do something smarter in C++. So the first thing that came to my mind that in C++ 17, maybe you know, there is a new adoption in the C++ standard which is the module called file system. This is based in the boost, it's file system. So I thought, okay, I will just copy paste the hell word example there and I will put the file system call to list directories. So again, same idea, I will just replace the old simple and hello. And the only difference is this. If there is any C Python core developer in the audience, please forgive my C Python, but this is just to show how simple we can achieve things. Of course it's just and save. There's memory leaks there because when you append that increase the rev counts and I'm not taking care of anything, but just taking one argument which is the recursive to see through or false and create an empty list and then just doing some dumb appends. So the magic here from the C++ site is that I'm using the file system module, I'm using C++ 17 and there is luckily some directory that you can kind of do recursively or not. And that's it. I mean, if you don't know C Python, this is just a function that will take here some arguments. We create an empty list and then if it's recursive enough, we are appending, well, you know how an append works, some stuff and then we are returning this list. Nothing else. So I thought, okay, let's see how this thing was working. And then I wrote a really, really simple bash script because, yeah, I was growing up with bash and I thought I would just have this scenario when I have 1,000 directories with 1,000 files inside of each empty files and just to list them. So I just create a shell script to measure this. As you can see here, the non-recursive option, which is just listing the directories inside is fake environment, it's.05. I was expecting love to take so long. But I think it should be something around 20 something seconds. I mean, in the meantime, I can tell you that I was afraid that maybe I was using user being time and maybe it's not the proper way of doing it. So okay, there you have it. 23 seconds, we're talking about all these files. Then I tried a bad leave. Again, it was a little bit more, it was slower than the Glob file. Then the other case is the recursive way. It will be roughly, I guess, the same if I'm not wrong. I should have this prepared. Okay, there you have it. 20 seconds. And then the fast glove implementation that I have there, it's less than a second. I mean, I know. You know, taking care of cache or, you know, the, I don't know, releasing the GIL to use parallel computer or something fancy, just the simplest thing that you can think. And then you have it. I mean, it's, and then I thought maybe it's the same thing I am doing it wrong. So let's do the same performance in Python, which, you know, the time module or something. And after all the tries, it was more or less the same result. And it's not like I want to say, yeah, this should be the new glove, but this is how easy you can improve things there. And if you are thinking in the next numpy, the next pandas, the next by torture, something like that, well, I will not wait for the results there, but it's roughly the same, believe me. So, yeah, this is a summary. It's okay if you cannot read everything because you can check the slides afterwards about the version that you can use, the licenses, the compatibility with Python. Here I just want to highlight Cheebokan, the thing that I am working with, and also SIP because there is something that is called stable ABI. Maybe you heard about it. Yes? SIP ABI? Good. So this is just for developers to release wheels that are compatible with Python 3.5 onwards. And I don't need to have different wheels with different Python versions. So this is really tricky. It's just the way that you create objects that are more dynamic way, but it's really hard and it's implemented in those two options. So there you can have information from the Python project. You can find me here also. You can check my social networks and all the information and you can just type make. I add some make files there just for you to know the process of building all these wrappers so you can start playing around today about it. And just as a PSA that it's always good to support your local groups. So if you, at any moment, you come to Berlin, I mean, we have, I think, one of the most amazing Python communities out there. We have PyLadies, Python users Berlin, PyBerlin, and PyData Berlin. So that's it. Thank you very much. All right, thank you very much. We have time for some questions. We have actually five minutes. So a couple of questions for sure. If you don't like your voice on a microphone, maybe you can find me outside. I will be here until Friday or we can hang up there a couple of years. We can discuss the truth behind C++. So if you have questions, yes. Did you ever look into performance overhead differences between those options? No, I haven't and I wanted to, but then I thought, that's not a half an hour talk. So I was really considering for the meetups or conference to kind of do this kind of thing because I want to measure code length performance, which is also critical. I mean, I have noticed some stuff different between the SIP and the Shiboken, for example, that there was an issue that Shiboken was way slower because we were using lists in C++ and the SIP was using burqtors. And then we have, for example, in some tables or cases like this, they were kind of different. But yes, good question. And I will try to maybe next year, I will have one performance. Thank you. Any more questions? Excellent. It does work. You can shout if you want. So I can use my side too. One thing I was missing was any typing. So what I have for my findings is just a bunch of taking arcs, equal arcs and parts. Yes. Lately, there was something that fixed that because I wanted to implement breaking the QTBI, having, for example, for QT settings to have explicitly cast in types and I needed to do the name arguments. So now it's implemented and it's out there. So you can do it. And now there's an annotation, which is the at. So you use the at between the name that you want to use and then you have nine arguments and you can parse it like a normal dictionary for the arcs and the... So yeah. You can ask me later if you want. I can show you how to do it. Okay. Any more questions? Do you hear me? Yes. Does the Shiboken uses some... Does it have some CMake integration? Yes. It's CMake. It's CMake and you can make a C++ project and use CMake integration in your project. Okay. Yes. So it's... For example, you want to override LAN or some other Python features. Can you override it in XML file for bindings? Can you name one... I mean, the only override that came to my mind is that, for example, you can kind of like remove functions and replace it with a whole new Python equivalent. Okay. So signature-wise or functions. We can remove arguments sometimes. Yeah. So sometimes, like square brackets, override... Operators. Operators. Yes. I mean, we have some special treatment for a few operators. I don't know how much you can modify this thing, but at least I know that you can manually modify, for example, the lower, lower operator in C++ and in Python to do different things. So you kind of like override the Python's default stuff. Yeah. So for example, I have C++ method size and the other one is operator for accessing it and I want in Python to just use square brackets for operating. As long as you define this operator inside the type of the variable, it will be possible because underneath, it will call this specific rewriting that you have. Okay. Thank you. You're welcome. Any other question? I will ask you a question now. Okay. So how is it to wrap a C++ class? It's kind of the same thing that you saw there. You need to just... The class will be an object. So you declare a new object and then inside, you have any function that you want to modify. If you don't want to modify, you just close the tag and that automatically go to your header. So you can just take all the signature and expose everything. If everything is normal in the sense of like there is no Boyd star or weird, you know, kind of like the pointers working around. So yeah. Can you then subclass that class from within Python as you would normally... Yes, totally. Because you are exposing some... A new type in Python. So for example, the simple things that you can have in Qt is a QWidget and then this is C++, you're exposed to Python and the recommended way of doing it is to use this as a meta class for your new class in Python that is called MyWidget or something like that. So you can do it. Very nice. Thank you. Thank you. Go to donuthand handOK.com for more fun and structure Advisory breeching.
 Hello, everyone.  This is one room more that I was expecting for, because I know that there's also amazing  talks that I would like to go to at this moment.  So if you want, we can all go to the other talks, or we can stay here for some fun with  C++, of course.  And I know it's just the last talk from all the tracks, so bear with me.  There will be no weird things here, and we will not like be trying to understand obfuscated  C++ or something else.  So it's a really light talk with advanced topics.  So let's go for it.  So as you heard, my name is Christian.  Currently I'm working as a software engineer in the Acute Company, which one of the companies  that support the Acute project, maybe you heard about it.  Later I will have one slide to explain the motivation behind this talk, and why I'm talking  about C++ on Python.  So first of all, don't worry about the examples or the slides or whatever.  Just go to that GitHub repository.  There is a link to the slides, and you can get all the source code.  Oh, my God, I feel like a star.  Everyone taking pictures.  I'm like, stop it.  You can check it out.  At the end of the talk, also, is my GitHub username, so you can check it out.  It's called UnleashedCBP.  Okay.  So first of all, the Acute project in general is a C++ framework, huge framework, and with  starting to do some Python stuff, you need to have the mandatory comparison between two  languages.  First of all, we know that both languages are general purpose.  Also, you have multi-paradigm.  You can do many things with them.  And then we start to notice some difference that I'm pretty sure that most of you are  aware of, dynamically type, static-ally type, compile, interpret, and then memory management  and about also code reliability.  So I know this is kind of if we have just raised your hand, who is currently writing  C or C++ code?  Okay.  Good.  So, I feel your pain, too.  So I know that sometimes C++ could be complicated, but it's just one to show you that it's not  and you can even make things in Python better.  So first of all, everyone knows that Python is beautiful.  I remember the first time that I typed that on a console when I was studying computer science  and I was, wow, everything works.  But if you think about it, I mean, C++ is not so different, right?  I mean, you can write something.  Like maybe you need to run an external compiler or something, but you achieve the same thing,  right?  So, okay.  After all, C++ is simple.  It's not.  Because then you have things like meta-programming and you start, you know, templates over templates  and this is one of the reasons that people get scared of C++.  But C++ is not so ugly and this is some extreme cases of people with a lot of time, a lot of  effort and try to do these complicated things like compile time if.  But let's go about one of the goodies, a couple of goodies that C++ and the latest standards  being 11 on has.  I mean, if you're familiar with C++, now there is this amazing auto type that kind of detects  the variable that you have, which is kind of a similar, like a more dynamic language,  but not.  You can even use some functions to get the declarative time of a different variable to  declare another one and stuff.  And then again, we have the for loops that with time it has been improved.  I mean, this is kind of the more dumb old version that we have in C++ to kind of like  go in a vector.  I know that you can do also with an iterator too, but it's kind of like going element by  element, but it's too complicated.  So then you can do more fun things like this.  And if you remember the other one, then you can kind of go, which is not so complicated  to understand.  Also if you're a fan of Python lambdas, you can also do it in C++ with the latest standards  and so on and so forth.  And you can have this for each handling a nice lambda there to just sum it up some values.  But the most interesting things about C++ are coming, in my opinion, in C++ 20, which  is, I don't know, I have this relationship with love and hate with C++ 20.  There are some things that are really nice and some things that I have no idea why people  is doing it.  But one of the things that I like is this one.  So you have this amount, sorry, this type of ranges that you can use now.  And if you see the four, the kind of like the type of the four there is kind of similar  that you can have there.  So after all, I mean, it's not so crazy different, right?  The same with a more example, I mean, I know that this mod way maybe is the first line  that is commented out, but you can write the function maybe like the second line that you  see there, which is also quite similar what you can achieve in C++ 20.  So after all, I mean, it's just adding some semicolon sentient, right?  So the idea is that now it's not about like which language is best, but how to make both  languages to improve each other and try to collaborate in a more efficient way.  So I want to propose now a new language that is, no, no, it's not.  But I'll just talk it to extending Python with C++.  And for this, we will go through a couple of examples, really simple ones.  And the first of all, who here didn't know before the conference that Python was written  in C?  We had some, okay, we had some people.  Okay.  So this is something that also was kind of shocking to me because I thought that it was  some black magic assembly stuff, but it was written in C. So if you learn some C, you  could understand.  And then you could understand other things like this joke, for example.  I know that is really bad, but it's the only joke that I have on the slides.  So I'm really sorry, but I needed to do it.  So let's take a look at how we can create an amazing module based in pure C Python.  Okay.  Let's go for it.  So the first thing that you need to know is that you just need to see C file.  And we will go through these examples with the most simple thing that I would think.  There was a hello world function that just returns the char star or string or whatever.  So I hope that everyone can read.  If you don't know C or C++, it's kind of that you get the idea.  It's kind of a function declaration that returns some message, which is hello, EuroPython 2019.  Okay.  So for building a C++, I will just sum out.  Don't be afraid just to show you.  So that's all the code that we required to create a new Python module written in C Python,  which one function that is just printing hello world.  It's not so scary, right?  I mean, I need to confess the first time that I started to play around with this, I was  just copying all these weird structures that you have around and all these widers, null,  null, zero and other things.  Just copy some stuff and it works.  So if you are brave enough after this talk, just go to the C Python directory and play  with it and maybe you can write other sophisticated models with it.  So this is a simple case, right?  And this is just one example, one simple function and we have this.  So what's the motivation behind this Qt thing?  So Qt is just a C++ framework that is like huge and one of the main functionalities was  to offer the way of developing user interface, graphical user interfaces for people.  And also Qt implements many of the things that we have for free in Python in C++, which  is like a lot of abstractions for different classes to interact with more databases or  maybe doing command line or scripting alike in C++ and even other advanced things that  different way of creating user interfaces, actions, notification, threats, all these things.  So they are providing a lot of goodies to the C++ world and at some point the project  was, but you know what?  We should, I mean, this thing need to be maybe a value in other languages.  So people started to wonder about Python.  So maybe you are aware there is already a set of binding which is really old, it's called  PyQt and this was developed by a different company, some people doing a lot of work there  and this was fantastic.  And in the Qt project they wanted to kind of have an official adoption of these things  so they decided maybe we can create a new set of bindings and just put it there and  see it's an open market so people can decide which to choose.  At the moment they are at the same level, one is really old and there's a lot of examples  and tutorials.  But the Qt project at that point had a couple of options to do this with this huge C++ framework.  So they could write raw C Python, they could write a Sweak, I mean maybe you heard about  Sweak, or they could use boost Python.  So if you take a look at the Sweak implementation, so with Sweak we just for you to believe me  we have here our difficult function to understand and the only things that we need to do for  the Sweak is just to create an interface.  So this interface is in an i-file that also nothing really scary or something at the moment.  So we have a couple of instructions to build it, I mean we run Sweak and then we compile  some stuff and then we can just use a normal simple import the module and execute it.  And then we have the message, nothing too fancy.  With boost Python it's kind of a similar story.  We go there again, we have the simple Cpp file with our function there and then we just need  to define boost Python module, the name, the namespace and then we define the function.  Nothing too scary at the moment, right?  So this was looking at a really nice solution, right?  I mean we were achieving the same thing but the problem lied that many things that we  do in C++ need to be specially treated inside Python.  For example ownership of objects.  If you are wrapping an interface that is in C++ who owns the objects?  Some cases will be Python, some cases will be C++.  So you need to deal with all these things.  So the option that at that time the developer had was okay, this is just one of the generators  and we can maybe modify the wrapper generated code.  So let's take a look at the code that Sweak generates.  So just for you to have an idea.  So this is kind of the code that generates and I will start to go down.  We are in 2%, 3%.  Okay, we'll just page down.  So I understand it.  It's a little bit too much with this small function, right?  So that's okay.  That's all.  And there is kind of a motivation to say, okay, I know that they need to set up many  things to get for granted and to automate many processes and it's a really smart solution.  But if you want to modify this file, it will be embeddable.  I mean you cannot do it.  So there was a lot of motivation behind this Cpython and Sweak and Boost Python.  I don't have the source code from the Boost Python here but the shared library is roughly  the same size so you can imagine that there are a lot of black magic there.  So what did they say?  Okay, they started the development of this new tool based in Boost at the moment and  they said, no, it's too heavy.  Let's write our own thing.  Okay, everything was released properly.  Then there was the project to continue the development and the good news was that last  year officially was released this new set of bindings of Qt in Python and then the name  of the new project was called Qt for Python.  So maybe you heard about this or PySide tool, it was all this story behind.  But okay, this is just a story but what was the important part, at least in my opinion,  how do we do it?  So this tool that you see here in the center called Gvogan 2 is kind of like the response  for writing our own code generator tool to expose all this huge C++ framework to Python.  So there is a module inside that kind of extracts all the API information from Qt, it's  based in Clang, of course, if you want to do smart things in C++, you need to use Clang.  Then we have a support type system which is nothing else than XML file.  Then you get, wait, something is being weird, an XML file where you can define all these  things that I told you about ownership or, for example, what do we do if a function has  a void star argument in Python?  We need to have a special or type or treatment or do some casting or something around.  So this tool graphs this type system, the information from the framework and generates  some wrapper, the one that you saw from SWIG but of course more reduced and it makes more  sense and it's more clean in my opinion.  And then with this we can just compile it and have the same Python module that we have.  So this tool is called Gvogan, the documentation is there and I don't know what is happening,  I don't know if I'm doing anything and the slides are coming.  Okay, so the Japanese Ganges doesn't mean anything, if there is, we have some Japanese  speaker there here, it's just three words that make no sense but let's take a look of  the Gvogan thing.  Okay.  So again, we will go first for you to believe me that we have here the implementation and  we are like cheating a little bit because of course we are working with strings from  the standard library, then we define complex header for this which is just that and for  this type system that I just told you, it's nothing else than this.  So we've had HTML, we said this would be a package called simple, it will have a primitive  type, there's a string from standard library and a hello function.  After all the compiling and stuff, we will get something like this.  Let's go to build and it's inside simple and here you have the module wrapper.  I will zoom out just for you to show you that I'm not lying.  It's long but it's not so long as the other one.  So here you have it, I don't expect you to read the code, I just want to show you the  magnitude of the code.  So at least this is being code automatically generated, calling this C++ function and exposing  everything with Python to be able to use it from Python.  So it's way shorter than the option that we had.  Of course this translates into having a more lightweight shared libraries and you can achieve  the same things.  So there are other nice options out there and for this I kind of tend to recommend people  to take a look of all the solutions that you have out there because it's unfair to say  yeah, this is the best solution.  No, it's not.  I mean there are many things that you can complement to each other.  I don't know if you have the chance or maybe know one of these ones.  The last one for example is the one that the other set of binding called PyQt uses which  is called SIP.  PyBind 11, also it's a really nice project that appeared I think a couple of years ago,  even I think two EuroPythons ago there was also a talk about PyBind 11.  So let's take a look of like how do you achieve the same things with the other options.  I think I am good with the time at the moment.  So this is the case of PyBind 11.  So let's open again the file.  If you see the motivation here is clearly from boost Python.  I mean it's again a kind of a macro that kind of defines a module, then you have the definition  of the function, some documentation and so on and so forth.  And then you can do the same thing.  I mean after you compile it and everything you have your simple main that you can achieve  the same.  So it's kind of like same idea as boost Python but they are doing way more things and there  is a lot of nice support there that I encourage you to check it out.  CFFI.  Well this is again not really fair comparison because CFFI, we had some talks already that  we explained about this thing, it's not kind of like generating code but it's just kind  of floating something.  So we can have this is the one here.  This is the code that is being generated.  So we go to the simple build and then you have like inline raw string that contains the function  that we want to expose.  And then we just compile this thing in some share library that we can easily load from  Python.  So if you check here for example, the main is the same thing that we were seeing in the  other examples, just calling this nice function and in this case we are casting it to a string  and so on and so forth.  But it's kind of the same idea.  But again, I mean it's not like you can write inline code, kind of read lots of C++ code  to kind of expose it in this way.  The other option that I had there, ah, of course.  If CFFI is kind of focused on C, they support C89 and I don't remember which one, all their  standard.  But for playing with C++ you have a similar idea which is CPPYY.  Kind of fine, right?  CPPYY.  And then it's kind of the same idea.  So you declare everything like in a string and then you have this declaration that you  can use this from the GBL.  Again, same idea and nothing too scary.  The last one is kind of similar.  It was a zip that I showed you, the one that is using the PyCute bindings.  I will just show you, this is just there.  So here we have the simple, again, same idea but the only difference that they require  to define a zip file.  And it's nothing like the other set this.  I mean it's also quite simple.  I mean you just need to define the module and then you have some include and then the  signature function.  So there are many options out there to achieve the same thing, right?  But what's the idea of doing this, right?  I have been having some conversation with people sometimes that they say, ah, yeah, but I mean  I use Python for everything.  I mean you don't need C++.  And they say, okay, what do you use?  No, I do everything with numpy because it's amazing and it's fast and it's Python, you  know?  And then it's like, yeah, I mean, you know, it's written in C...  Oh, no, it's not.  Okay, there's even some people that is not aware that many of the popular libraries out  there are using C++ or C.  So in this case you have a case of numpy, I just downloaded the source code.  The first line is all the Python files that you have that are not test and then you have  the C files there, which is 96, which is still a lot of things that they are doing.  And if you want to maybe refer to a more modern module, the case of PyTorch is a little bit  more extreme.  And if you look there, we have...  No, this is...  This was...  This is...  So here we have 547 C++ files inside the PyTorch because of course it's based in Torch and  Torch is C++, so everything is C++ inside.  So it's kind of a good...  And I use the motivation of using all these tools to start to write things to improve the  Python.  We are all here.  If we know C or C++, we kind of have some responsibility in our hands to improve things.  And one of the examples that just out of curiosity and please don't blame me about the things  that I will show you is that simple case, I was helping someone that I was listing some  files in Python and getting all different absolute path and what do you use there?  Glob, right?  How do you use Glob?  Or if you are more like up-to-date person, you use the bat leaf and then also you have  the Glob access there.  So for this, you know the deal, I mean writing Glob is...  Things are kind of simple.  This is a hard-coded thing just to play with the recursion thing, but you know import Glob  and then you need to double stars there and specify if it's recursive or not.  The same way with bat leaf is kind of the same idea, right?  And then I thought, I mean let's look at the implementation and see if we can kind of do  something smarter in C++.  So the first thing that came to my mind that in C++ 17, maybe you know, there is a new  adoption in the C++ standard which is the module called file system.  This is based in the boost, it's file system.  So I thought, okay, I will just copy paste the hell word example there and I will put  the file system call to list directories.  So again, same idea, I will just replace the old simple and hello.  And the only difference is this.  If there is any C Python core developer in the audience, please forgive my C Python, but  this is just to show how simple we can achieve things.  Of course it's just and save.  There's memory leaks there because when you append that increase the rev counts and I'm  not taking care of anything, but just taking one argument which is the recursive to see  through or false and create an empty list and then just doing some dumb appends.  So the magic here from the C++ site is that I'm using the file system module, I'm using  C++ 17 and there is luckily some directory that you can kind of do recursively or not.  And that's it.  I mean, if you don't know C Python, this is just a function that will take here some arguments.  We create an empty list and then if it's recursive enough, we are appending, well, you know how  an append works, some stuff and then we are returning this list.  Nothing else.  So I thought, okay, let's see how this thing was working.  And then I wrote a really, really simple bash script because, yeah, I was growing up with  bash and I thought I would just have this scenario when I have 1,000 directories with  1,000 files inside of each empty files and just to list them.  So I just create a shell script to measure this.  As you can see here, the non-recursive option, which is just listing the directories inside  is fake environment, it's.05.  I was expecting love to take so long.  But I think it should be something around 20 something seconds.  I mean, in the meantime, I can tell you that I was afraid that maybe I was using user being  time and maybe it's not the proper way of doing it.  So okay, there you have it.  23 seconds, we're talking about all these files.  Then I tried a bad leave.  Again, it was a little bit more, it was slower than the Glob file.  Then the other case is the recursive way.  It will be roughly, I guess, the same if I'm not wrong.  I should have this prepared.  Okay, there you have it.  20 seconds.  And then the fast glove implementation that I have there, it's less than a second.  I mean, I know.  You know, taking care of cache or, you know, the, I don't know, releasing the GIL to use  parallel computer or something fancy, just the simplest thing that you can think.  And then you have it.  I mean, it's, and then I thought maybe it's the same thing I am doing it wrong.  So let's do the same performance in Python, which, you know, the time module or something.  And after all the tries, it was more or less the same result.  And it's not like I want to say, yeah, this should be the new glove, but this is how easy  you can improve things there.  And if you are thinking in the next numpy, the next pandas, the next by torture, something  like that, well, I will not wait for the results there, but it's roughly the same, believe  me.  So, yeah, this is a summary.  It's okay if you cannot read everything because you can check the slides afterwards about  the version that you can use, the licenses, the compatibility with Python.  Here I just want to highlight Cheebokan, the thing that I am working with, and also SIP  because there is something that is called stable ABI.  Maybe you heard about it.  Yes?  SIP ABI?  Good.  So this is just for developers to release wheels that are compatible with Python 3.5 onwards.  And I don't need to have different wheels with different Python versions.  So this is really tricky.  It's just the way that you create objects that are more dynamic way, but it's really  hard and it's implemented in those two options.  So there you can have information from the Python project.  You can find me here also.  You can check my social networks and all the information and you can just type make.  I add some make files there just for you to know the process of building all these wrappers  so you can start playing around today about it.  And just as a PSA that it's always good to support your local groups.  So if you, at any moment, you come to Berlin, I mean, we have, I think, one of the most  amazing Python communities out there.  We have PyLadies, Python users Berlin, PyBerlin, and PyData Berlin.  So that's it.  Thank you very much.  All right, thank you very much.  We have time for some questions.  We have actually five minutes.  So a couple of questions for sure.  If you don't like your voice on a microphone, maybe you can find me outside.  I will be here until Friday or we can hang up there a couple of years.  We can discuss the truth behind C++.  So if you have questions, yes.  Did you ever look into performance overhead differences between those options?  No, I haven't and I wanted to, but then I thought, that's not a half an hour talk.  So I was really considering for the meetups or conference to kind of do this kind of thing  because I want to measure code length performance, which is also critical.  I mean, I have noticed some stuff different between the SIP and the Shiboken, for example,  that there was an issue that Shiboken was way slower because we were using lists in  C++ and the SIP was using burqtors.  And then we have, for example, in some tables or cases like this, they were kind of different.  But yes, good question.  And I will try to maybe next year, I will have one performance.  Thank you.  Any more questions?  Excellent.  It does work.  You can shout if you want.  So I can use my side too.  One thing I was missing was any typing.  So what I have for my findings is just a bunch of taking arcs, equal arcs and parts.  Yes.  Lately, there was something that fixed that because I wanted to implement breaking the  QTBI, having, for example, for QT settings to have explicitly cast in types and I needed  to do the name arguments.  So now it's implemented and it's out there.  So you can do it.  And now there's an annotation, which is the at.  So you use the at between the name that you want to use and then you have nine arguments  and you can parse it like a normal dictionary for the arcs and the...  So yeah.  You can ask me later if you want.  I can show you how to do it.  Okay.  Any more questions?  Do you hear me?  Yes.  Does the Shiboken uses some...  Does it have some CMake integration?  Yes.  It's CMake.  It's CMake and you can make a C++ project and use CMake integration in your project.  Okay.  Yes.  So it's...  For example, you want to override LAN or some other Python features.  Can you override it in XML file for bindings?  Can you name one...  I mean, the only override that came to my mind is that, for example, you can kind of like  remove functions and replace it with a whole new Python equivalent.  Okay.  So signature-wise or functions.  We can remove arguments sometimes.  Yeah.  So sometimes, like square brackets, override...  Operators.  Operators.  Yes.  I mean, we have some special treatment for a few operators.  I don't know how much you can modify this thing, but at least I know that you can manually  modify, for example, the lower, lower operator in C++ and in Python to do different things.  So you kind of like override the Python's default stuff.  Yeah.  So for example, I have C++ method size and the other one is operator for accessing it  and I want in Python to just use square brackets for operating.  As long as you define this operator inside the type of the variable, it will be possible  because underneath, it will call this specific rewriting that you have.  Okay.  Thank you.  You're welcome.  Any other question?  I will ask you a question now.  Okay.  So how is it to wrap a C++ class?  It's kind of the same thing that you saw there.  You need to just...  The class will be an object.  So you declare a new object and then inside, you have any function that you want to modify.  If you don't want to modify, you just close the tag and that automatically go to your  header.  So you can just take all the signature and expose everything.  If everything is normal in the sense of like there is no Boyd star or weird, you know,  kind of like the pointers working around.  So yeah.  Can you then subclass that class from within Python as you would normally...  Yes, totally.  Because you are exposing some...  A new type in Python.  So for example, the simple things that you can have in Qt is a QWidget and then this  is C++, you're exposed to Python and the recommended way of doing it is to use this as a meta class  for your new class in Python that is called MyWidget or something like that.  So you can do it.  Very nice.  Thank you.  Thank you.  Go to donuthand handOK.com for more fun and structure Advisory breeching.
Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_0_179.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_180_995.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_996_1103.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_1104_1151.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_1152_1307.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_1308_8423.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_8424_9299.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_9300_9383.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_9384_9719.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_9720_10259.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_10260_11507.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_11508_12659.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_12660_13223.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_13224_13823.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_13824_13979.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_13980_15995.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_15996_19067.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_19068_20243.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_20244_20675.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_20676_24203.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_24204_24395.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_24396_25955.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_25956_26351.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_26352_26963.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_26964_27923.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_27924_28355.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_28356_30407.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_30408_30623.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_30624_31559.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_31560_31631.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_31632_42779.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_42780_44099.png Cristian_Maureira-Fredes_-_Unleash_the_power_of_C___in_Python_02-IYuDQdTE_44100_44154.png
Hello, everyone. This is one room more that I was expecting for, because I know that there's also amazing talks that I would like to go to at this moment. So if you want, we can all go to the other talks, or we can stay here for some fun with C++, of course. And I know it's just the last talk from all the tracks, so bear with me. There will be no weird things here, and we will not like be trying to understand obfuscated C++ or something else. So it's a really light talk with advanced topics. So let's go for it. So as you heard, my name is Christian. Currently I'm working as a software engineer in the Acute Company, which one of the companies that support the Acute project, maybe you heard about it. Later I will have one slide to explain the motivation behind this talk, and why I'm talking about C++ on Python. So first of all, don't worry about the examples or the slides or whatever. Just go to that GitHub repository. There is a link to the slides, and you can get all the source code. Oh, my God, I feel like a star. Everyone taking pictures. I'm like, stop it. You can check it out. At the end of the talk, also, is my GitHub username, so you can check it out. It's called UnleashedCBP. Okay. So first of all, the Acute project in general is a C++ framework, huge framework, and with starting to do some Python stuff, you need to have the mandatory comparison between two languages. First of all, we know that both languages are general purpose. Also, you have multi-paradigm. You can do many things with them. And then we start to notice some difference that I'm pretty sure that most of you are aware of, dynamically type, static-ally type, compile, interpret, and then memory management and about also code reliability. So I know this is kind of if we have just raised your hand, who is currently writing C or C++ code? Okay. Good. So, I feel your pain, too. So I know that sometimes C++ could be complicated, but it's just one to show you that it's not and you can even make things in Python better. So first of all, everyone knows that Python is beautiful. I remember the first time that I typed that on a console when I was studying computer science and I was, wow, everything works. But if you think about it, I mean, C++ is not so different, right? I mean, you can write something. Like maybe you need to run an external compiler or something, but you achieve the same thing, right? So, okay. After all, C++ is simple. It's not. Because then you have things like meta-programming and you start, you know, templates over templates and this is one of the reasons that people get scared of C++. But C++ is not so ugly and this is some extreme cases of people with a lot of time, a lot of effort and try to do these complicated things like compile time if. But let's go about one of the goodies, a couple of goodies that C++ and the latest standards being 11 on has. I mean, if you're familiar with C++, now there is this amazing auto type that kind of detects the variable that you have, which is kind of a similar, like a more dynamic language, but not. You can even use some functions to get the declarative time of a different variable to declare another one and stuff. And then again, we have the for loops that with time it has been improved. I mean, this is kind of the more dumb old version that we have in C++ to kind of like go in a vector. I know that you can do also with an iterator too, but it's kind of like going element by element, but it's too complicated. So then you can do more fun things like this. And if you remember the other one, then you can kind of go, which is not so complicated to understand. Also if you're a fan of Python lambdas, you can also do it in C++ with the latest standards and so on and so forth. And you can have this for each handling a nice lambda there to just sum it up some values. But the most interesting things about C++ are coming, in my opinion, in C++ 20, which is, I don't know, I have this relationship with love and hate with C++ 20. There are some things that are really nice and some things that I have no idea why people is doing it. But one of the things that I like is this one. So you have this amount, sorry, this type of ranges that you can use now. And if you see the four, the kind of like the type of the four there is kind of similar that you can have there. So after all, I mean, it's not so crazy different, right? The same with a more example, I mean, I know that this mod way maybe is the first line that is commented out, but you can write the function maybe like the second line that you see there, which is also quite similar what you can achieve in C++ 20. So after all, I mean, it's just adding some semicolon sentient, right? So the idea is that now it's not about like which language is best, but how to make both languages to improve each other and try to collaborate in a more efficient way. So I want to propose now a new language that is, no, no, it's not. But I'll just talk it to extending Python with C++. And for this, we will go through a couple of examples, really simple ones. And the first of all, who here didn't know before the conference that Python was written in C? We had some, okay, we had some people. Okay. So this is something that also was kind of shocking to me because I thought that it was some black magic assembly stuff, but it was written in C. So if you learn some C, you could understand. And then you could understand other things like this joke, for example. I know that is really bad, but it's the only joke that I have on the slides. So I'm really sorry, but I needed to do it. So let's take a look at how we can create an amazing module based in pure C Python. Okay. Let's go for it. So the first thing that you need to know is that you just need to see C file. And we will go through these examples with the most simple thing that I would think. There was a hello world function that just returns the char star or string or whatever. So I hope that everyone can read. If you don't know C or C++, it's kind of that you get the idea. It's kind of a function declaration that returns some message, which is hello, EuroPython 2019. Okay. So for building a C++, I will just sum out. Don't be afraid just to show you. So that's all the code that we required to create a new Python module written in C Python, which one function that is just printing hello world. It's not so scary, right? I mean, I need to confess the first time that I started to play around with this, I was just copying all these weird structures that you have around and all these widers, null, null, zero and other things. Just copy some stuff and it works. So if you are brave enough after this talk, just go to the C Python directory and play with it and maybe you can write other sophisticated models with it. So this is a simple case, right? And this is just one example, one simple function and we have this. So what's the motivation behind this Qt thing? So Qt is just a C++ framework that is like huge and one of the main functionalities was to offer the way of developing user interface, graphical user interfaces for people. And also Qt implements many of the things that we have for free in Python in C++, which is like a lot of abstractions for different classes to interact with more databases or maybe doing command line or scripting alike in C++ and even other advanced things that different way of creating user interfaces, actions, notification, threats, all these things. So they are providing a lot of goodies to the C++ world and at some point the project was, but you know what? We should, I mean, this thing need to be maybe a value in other languages. So people started to wonder about Python. So maybe you are aware there is already a set of binding which is really old, it's called PyQt and this was developed by a different company, some people doing a lot of work there and this was fantastic. And in the Qt project they wanted to kind of have an official adoption of these things so they decided maybe we can create a new set of bindings and just put it there and see it's an open market so people can decide which to choose. At the moment they are at the same level, one is really old and there's a lot of examples and tutorials. But the Qt project at that point had a couple of options to do this with this huge C++ framework. So they could write raw C Python, they could write a Sweak, I mean maybe you heard about Sweak, or they could use boost Python. So if you take a look at the Sweak implementation, so with Sweak we just for you to believe me we have here our difficult function to understand and the only things that we need to do for the Sweak is just to create an interface. So this interface is in an i-file that also nothing really scary or something at the moment. So we have a couple of instructions to build it, I mean we run Sweak and then we compile some stuff and then we can just use a normal simple import the module and execute it. And then we have the message, nothing too fancy. With boost Python it's kind of a similar story. We go there again, we have the simple Cpp file with our function there and then we just need to define boost Python module, the name, the namespace and then we define the function. Nothing too scary at the moment, right? So this was looking at a really nice solution, right? I mean we were achieving the same thing but the problem lied that many things that we do in C++ need to be specially treated inside Python. For example ownership of objects. If you are wrapping an interface that is in C++ who owns the objects? Some cases will be Python, some cases will be C++. So you need to deal with all these things. So the option that at that time the developer had was okay, this is just one of the generators and we can maybe modify the wrapper generated code. So let's take a look at the code that Sweak generates. So just for you to have an idea. So this is kind of the code that generates and I will start to go down. We are in 2%, 3%. Okay, we'll just page down. So I understand it. It's a little bit too much with this small function, right? So that's okay. That's all. And there is kind of a motivation to say, okay, I know that they need to set up many things to get for granted and to automate many processes and it's a really smart solution. But if you want to modify this file, it will be embeddable. I mean you cannot do it. So there was a lot of motivation behind this Cpython and Sweak and Boost Python. I don't have the source code from the Boost Python here but the shared library is roughly the same size so you can imagine that there are a lot of black magic there. So what did they say? Okay, they started the development of this new tool based in Boost at the moment and they said, no, it's too heavy. Let's write our own thing. Okay, everything was released properly. Then there was the project to continue the development and the good news was that last year officially was released this new set of bindings of Qt in Python and then the name of the new project was called Qt for Python. So maybe you heard about this or PySide tool, it was all this story behind. But okay, this is just a story but what was the important part, at least in my opinion, how do we do it? So this tool that you see here in the center called Gvogan 2 is kind of like the response for writing our own code generator tool to expose all this huge C++ framework to Python. So there is a module inside that kind of extracts all the API information from Qt, it's based in Clang, of course, if you want to do smart things in C++, you need to use Clang. Then we have a support type system which is nothing else than XML file. Then you get, wait, something is being weird, an XML file where you can define all these things that I told you about ownership or, for example, what do we do if a function has a void star argument in Python? We need to have a special or type or treatment or do some casting or something around. So this tool graphs this type system, the information from the framework and generates some wrapper, the one that you saw from SWIG but of course more reduced and it makes more sense and it's more clean in my opinion. And then with this we can just compile it and have the same Python module that we have. So this tool is called Gvogan, the documentation is there and I don't know what is happening, I don't know if I'm doing anything and the slides are coming. Okay, so the Japanese Ganges doesn't mean anything, if there is, we have some Japanese speaker there here, it's just three words that make no sense but let's take a look of the Gvogan thing. Okay. So again, we will go first for you to believe me that we have here the implementation and we are like cheating a little bit because of course we are working with strings from the standard library, then we define complex header for this which is just that and for this type system that I just told you, it's nothing else than this. So we've had HTML, we said this would be a package called simple, it will have a primitive type, there's a string from standard library and a hello function. After all the compiling and stuff, we will get something like this. Let's go to build and it's inside simple and here you have the module wrapper. I will zoom out just for you to show you that I'm not lying. It's long but it's not so long as the other one. So here you have it, I don't expect you to read the code, I just want to show you the magnitude of the code. So at least this is being code automatically generated, calling this C++ function and exposing everything with Python to be able to use it from Python. So it's way shorter than the option that we had. Of course this translates into having a more lightweight shared libraries and you can achieve the same things. So there are other nice options out there and for this I kind of tend to recommend people to take a look of all the solutions that you have out there because it's unfair to say yeah, this is the best solution. No, it's not. I mean there are many things that you can complement to each other. I don't know if you have the chance or maybe know one of these ones. The last one for example is the one that the other set of binding called PyQt uses which is called SIP. PyBind 11, also it's a really nice project that appeared I think a couple of years ago, even I think two EuroPythons ago there was also a talk about PyBind 11. So let's take a look of like how do you achieve the same things with the other options. I think I am good with the time at the moment. So this is the case of PyBind 11. So let's open again the file. If you see the motivation here is clearly from boost Python. I mean it's again a kind of a macro that kind of defines a module, then you have the definition of the function, some documentation and so on and so forth. And then you can do the same thing. I mean after you compile it and everything you have your simple main that you can achieve the same. So it's kind of like same idea as boost Python but they are doing way more things and there is a lot of nice support there that I encourage you to check it out. CFFI. Well this is again not really fair comparison because CFFI, we had some talks already that we explained about this thing, it's not kind of like generating code but it's just kind of floating something. So we can have this is the one here. This is the code that is being generated. So we go to the simple build and then you have like inline raw string that contains the function that we want to expose. And then we just compile this thing in some share library that we can easily load from Python. So if you check here for example, the main is the same thing that we were seeing in the other examples, just calling this nice function and in this case we are casting it to a string and so on and so forth. But it's kind of the same idea. But again, I mean it's not like you can write inline code, kind of read lots of C++ code to kind of expose it in this way. The other option that I had there, ah, of course. If CFFI is kind of focused on C, they support C89 and I don't remember which one, all their standard. But for playing with C++ you have a similar idea which is CPPYY. Kind of fine, right? CPPYY. And then it's kind of the same idea. So you declare everything like in a string and then you have this declaration that you can use this from the GBL. Again, same idea and nothing too scary. The last one is kind of similar. It was a zip that I showed you, the one that is using the PyCute bindings. I will just show you, this is just there. So here we have the simple, again, same idea but the only difference that they require to define a zip file. And it's nothing like the other set this. I mean it's also quite simple. I mean you just need to define the module and then you have some include and then the signature function. So there are many options out there to achieve the same thing, right? But what's the idea of doing this, right? I have been having some conversation with people sometimes that they say, ah, yeah, but I mean I use Python for everything. I mean you don't need C++. And they say, okay, what do you use? No, I do everything with numpy because it's amazing and it's fast and it's Python, you know? And then it's like, yeah, I mean, you know, it's written in C... Oh, no, it's not. Okay, there's even some people that is not aware that many of the popular libraries out there are using C++ or C. So in this case you have a case of numpy, I just downloaded the source code. The first line is all the Python files that you have that are not test and then you have the C files there, which is 96, which is still a lot of things that they are doing. And if you want to maybe refer to a more modern module, the case of PyTorch is a little bit more extreme. And if you look there, we have... No, this is... This was... This is... So here we have 547 C++ files inside the PyTorch because of course it's based in Torch and Torch is C++, so everything is C++ inside. So it's kind of a good... And I use the motivation of using all these tools to start to write things to improve the Python. We are all here. If we know C or C++, we kind of have some responsibility in our hands to improve things. And one of the examples that just out of curiosity and please don't blame me about the things that I will show you is that simple case, I was helping someone that I was listing some files in Python and getting all different absolute path and what do you use there? Glob, right? How do you use Glob? Or if you are more like up-to-date person, you use the bat leaf and then also you have the Glob access there. So for this, you know the deal, I mean writing Glob is... Things are kind of simple. This is a hard-coded thing just to play with the recursion thing, but you know import Glob and then you need to double stars there and specify if it's recursive or not. The same way with bat leaf is kind of the same idea, right? And then I thought, I mean let's look at the implementation and see if we can kind of do something smarter in C++. So the first thing that came to my mind that in C++ 17, maybe you know, there is a new adoption in the C++ standard which is the module called file system. This is based in the boost, it's file system. So I thought, okay, I will just copy paste the hell word example there and I will put the file system call to list directories. So again, same idea, I will just replace the old simple and hello. And the only difference is this. If there is any C Python core developer in the audience, please forgive my C Python, but this is just to show how simple we can achieve things. Of course it's just and save. There's memory leaks there because when you append that increase the rev counts and I'm not taking care of anything, but just taking one argument which is the recursive to see through or false and create an empty list and then just doing some dumb appends. So the magic here from the C++ site is that I'm using the file system module, I'm using C++ 17 and there is luckily some directory that you can kind of do recursively or not. And that's it. I mean, if you don't know C Python, this is just a function that will take here some arguments. We create an empty list and then if it's recursive enough, we are appending, well, you know how an append works, some stuff and then we are returning this list. Nothing else. So I thought, okay, let's see how this thing was working. And then I wrote a really, really simple bash script because, yeah, I was growing up with bash and I thought I would just have this scenario when I have 1,000 directories with 1,000 files inside of each empty files and just to list them. So I just create a shell script to measure this. As you can see here, the non-recursive option, which is just listing the directories inside is fake environment, it's.05. I was expecting love to take so long. But I think it should be something around 20 something seconds. I mean, in the meantime, I can tell you that I was afraid that maybe I was using user being time and maybe it's not the proper way of doing it. So okay, there you have it. 23 seconds, we're talking about all these files. Then I tried a bad leave. Again, it was a little bit more, it was slower than the Glob file. Then the other case is the recursive way. It will be roughly, I guess, the same if I'm not wrong. I should have this prepared. Okay, there you have it. 20 seconds. And then the fast glove implementation that I have there, it's less than a second. I mean, I know. You know, taking care of cache or, you know, the, I don't know, releasing the GIL to use parallel computer or something fancy, just the simplest thing that you can think. And then you have it. I mean, it's, and then I thought maybe it's the same thing I am doing it wrong. So let's do the same performance in Python, which, you know, the time module or something. And after all the tries, it was more or less the same result. And it's not like I want to say, yeah, this should be the new glove, but this is how easy you can improve things there. And if you are thinking in the next numpy, the next pandas, the next by torture, something like that, well, I will not wait for the results there, but it's roughly the same, believe me. So, yeah, this is a summary. It's okay if you cannot read everything because you can check the slides afterwards about the version that you can use, the licenses, the compatibility with Python. Here I just want to highlight Cheebokan, the thing that I am working with, and also SIP because there is something that is called stable ABI. Maybe you heard about it. Yes? SIP ABI? Good. So this is just for developers to release wheels that are compatible with Python 3.5 onwards. And I don't need to have different wheels with different Python versions. So this is really tricky. It's just the way that you create objects that are more dynamic way, but it's really hard and it's implemented in those two options. So there you can have information from the Python project. You can find me here also. You can check my social networks and all the information and you can just type make. I add some make files there just for you to know the process of building all these wrappers so you can start playing around today about it. And just as a PSA that it's always good to support your local groups. So if you, at any moment, you come to Berlin, I mean, we have, I think, one of the most amazing Python communities out there. We have PyLadies, Python users Berlin, PyBerlin, and PyData Berlin. So that's it. Thank you very much. All right, thank you very much. We have time for some questions. We have actually five minutes. So a couple of questions for sure. If you don't like your voice on a microphone, maybe you can find me outside. I will be here until Friday or we can hang up there a couple of years. We can discuss the truth behind C++. So if you have questions, yes. Did you ever look into performance overhead differences between those options? No, I haven't and I wanted to, but then I thought, that's not a half an hour talk. So I was really considering for the meetups or conference to kind of do this kind of thing because I want to measure code length performance, which is also critical. I mean, I have noticed some stuff different between the SIP and the Shiboken, for example, that there was an issue that Shiboken was way slower because we were using lists in C++ and the SIP was using burqtors. And then we have, for example, in some tables or cases like this, they were kind of different. But yes, good question. And I will try to maybe next year, I will have one performance. Thank you. Any more questions? Excellent. It does work. You can shout if you want. So I can use my side too. One thing I was missing was any typing. So what I have for my findings is just a bunch of taking arcs, equal arcs and parts. Yes. Lately, there was something that fixed that because I wanted to implement breaking the QTBI, having, for example, for QT settings to have explicitly cast in types and I needed to do the name arguments. So now it's implemented and it's out there. So you can do it. And now there's an annotation, which is the at. So you use the at between the name that you want to use and then you have nine arguments and you can parse it like a normal dictionary for the arcs and the... So yeah. You can ask me later if you want. I can show you how to do it. Okay. Any more questions? Do you hear me? Yes. Does the Shiboken uses some... Does it have some CMake integration? Yes. It's CMake. It's CMake and you can make a C++ project and use CMake integration in your project. Okay. Yes. So it's... For example, you want to override LAN or some other Python features. Can you override it in XML file for bindings? Can you name one... I mean, the only override that came to my mind is that, for example, you can kind of like remove functions and replace it with a whole new Python equivalent. Okay. So signature-wise or functions. We can remove arguments sometimes. Yeah. So sometimes, like square brackets, override... Operators. Operators. Yes. I mean, we have some special treatment for a few operators. I don't know how much you can modify this thing, but at least I know that you can manually modify, for example, the lower, lower operator in C++ and in Python to do different things. So you kind of like override the Python's default stuff. Yeah. So for example, I have C++ method size and the other one is operator for accessing it and I want in Python to just use square brackets for operating. As long as you define this operator inside the type of the variable, it will be possible because underneath, it will call this specific rewriting that you have. Okay. Thank you. You're welcome. Any other question? I will ask you a question now. Okay. So how is it to wrap a C++ class? It's kind of the same thing that you saw there. You need to just... The class will be an object. So you declare a new object and then inside, you have any function that you want to modify. If you don't want to modify, you just close the tag and that automatically go to your header. So you can just take all the signature and expose everything. If everything is normal in the sense of like there is no Boyd star or weird, you know, kind of like the pointers working around. So yeah. Can you then subclass that class from within Python as you would normally... Yes, totally. Because you are exposing some... A new type in Python. So for example, the simple things that you can have in Qt is a QWidget and then this is C++, you're exposed to Python and the recommended way of doing it is to use this as a meta class for your new class in Python that is called MyWidget or something like that. So you can do it. Very nice. Thank you. Thank you. Go to donuthand handOK.com for more fun and structure Advisory breeching.
talk it to extending Python with C++. And for this, we will go through a couple of examples, with the most simple thing that I would think. C++ framework that is like huge and one of the lot of examples and tutorials. But the Qt project the same thing but the problem lied that many generator tool to expose all this huge C++ framework to Python. So there is a module inside would be a package called simple, it will have a library and a hello function. After all the function and exposing everything with Python to be able to use it from Python. So it's way to take a look of all the solutions that you have for example, the main is the same thing that we that is using the PyCute bindings. I will just libraries out there are using C++ or C. So in simple. This is a hard-coded thing just to play only difference is this. If there is any C Python C Python, but this is just to show how simple we here, the non-recursive option, which is just there just for you to know the process of amazing Python communities out there. We have to maybe next year, I will have one performance. It's CMake. It's CMake and you can make a C++ and replace it with a whole new Python will be an object. So you declare a new object