Get up to speed with Cython 3.0
Behnel, Stefan
Processing lots of data, in need of more speed, or struggling to make use of native code? The Cython compiler for Python solves all of these problems in a pythonic way, by compiling Python code to optimised C code. Cython has a very long history of faithfully helping users to solve their Python performance problems and integrating native code into Python in production critical settings, while stimulating somewhat less confidence with its 0.x versioning scheme. For the next Cython version, a stable 3.0 release, we are planning to clean up several historically grown issues in the language to make the Cython compiler more friendly for modern Python users to work with. In this talk, I will show how Cython can be used for compiling and speeding up Python code, using fast native data structures and libraries. I will then give an overview of the changes that are on their way for the 3.0 release and how they will affect the user experience.
 Hey, everyone. I'm one of the site and core developers. Before I start this talk, I'm going to show you a lot of examples of site and how to use it in different fields. I'd like to know a bit what you're more interested in. Who's never used site and before? That is some... It's true. The youngest non-site user. We'll introduce some to site today. That's about one-third of you. Who is interested in integrating and accelerating numpy code, computational code? Accelerating computational code with numpy and all that. That seems a bit less than the group we just had. Talking to external libraries, integrating native code into Python, is that a topic for you? Raise your hands. A couple of people. Anyone using C++? Yeah. Surprisingly, not so few. I'll try to focus a bit on what you just said. Okay. I'll start with myself. So, I'm a software developer, data engineer, whatever you want to name it these days. I've been using Python since 2002. I'm one of the founders of the Python project, not of the original inventors, but when we forked the project from Pyrex, that was in 2007. I was already on board. And I'm also a CPython core dev since this year. I do training and consulting in-house if you want to know anything about Python, have it taught in your company at home, you can contact me. I might come over and teach you something. I'll definitely have something to teach you or look in your code and approve it. I've been working for TrustU since 2017. And I use that as a little introductory example because TrustU is a data company. And that fits very well in the field. So what we do is we have terabytes of hotel guest reviews that we collect from all over the Internet. And when I say guest reviews, that literally brain dumps of arbitrary people writing stuff on the Internet, right? So we collect them, analyze them in 24 languages. There's a lot of NLP involved. Have surveys that we send out for the hotels, collect them from portals, from partners that we have to text and data analysis on them, and then find out what people are talking about, what kind of different categories they are talking about, and analyze trends, find out this kind of information. So that back to the hotels to tell them how they can improve. So we can tell the hotels, you know, if you renovate your pool or hire someone new for the reception, you'll probably improve your score by 10%. That's what we tell them. So this might also be interesting for you. So if you go to trust.com and type in the name of hotels, next time you look for a hotel somewhere, type it in there, and we'll tell you exactly what people actually think about this hotel, right? Notebooking.com wants you to think about your hotel in order to book it. We're independent, and so we can give you the actual opinions based on actual reviews, actual data. So go there, remember this, and you'll see it's all cut down into different categories and all that, so it's really relevant information that you want to know before you book that hotel. Why do I tell you all this? Well, these are some tools that we use at Trust.io. It's a data company, so we use NumPy, SciPy, we use Scikit-learn for things. We use Pinellis for data analysis, use NLTK and Spacey for text analysis, Alex and Mel whenever we have to deal with XML, PyYaml for configuration and all this, and it's all based on Hadoop processing and Spark. Some MapReduce, some whatever Hadoop provides, Hive and all these technologies. So this is mostly what we do. And now most of these tools are actually partly or even entirely written in Python. Right? Who didn't know this? Raise your hands. Interesting. NLTK is no longer entirely true because we're working on patch for them. NLTK is a pure Python toolkit. So I've written a PR for them to start compiling certain of their modules so that they just run faster if you install the binary instead of the Python package. So we're working on that. Hadoop is straight out because most written Java. But Trust is on the list. Who's seen this? Has anyone not seen it yet? Okay. So that's the image that the EHT project took off a black hole a couple of months ago. And the interesting thing about this here is that these are the tools that they use for calculating this image out of the raw data. And as you may notice, they're all Python tools. They use pandas, numpy, astropy, scikit image over there. Scipy, all these tools, all Python tools for aggregating the, again, I think they start with petabytes of data coming from eight different telescopes being taken across 10 days in a row. Right? Like huge telescopes all around the world. Like the data cut it down into manageable data sizes and then calculated this image out of it. And that was all done in Python. And now if you look at the tools that they used, all of them actually use Siphon inside. Right? So that was a huge Siphon-based project, although they didn't, you know, largely advertise that, but it was. Okay. So what is Siphon actually used for? It's used for integrating native code with Python. It's used to speed up Python code in C Python and simply even use it to write C, you know, without having to write C. So basically we write C so you don't have to. Okay? Skip that. Gradle typing. Does anyone know what gradle typing is? I heard the term before, maybe in context with PEP484, the typing annotations in Python. So the basic idea is there are two main different ways to type languages. One is statically typed, which tends to be, you know, fast languages, but it tends to become a sum because you have to type everything. It's very annoying. And then the dynamic languages like Python, which are easy to use, but tend to be slow kind of. It's not that simple, but more or less those are the two fields. And gradle typing is somewhere between. It says, you know, you can use typing, but it helps. It's documenting, but it makes things faster. That's where you should use typing. And for everything else, it's fine if you don't use it and don't need it there. So it was termed coined in 2006. And there's a blog post, just look for what is gradle typing and you'll find it. And it was the basis for PEP484 type annotations in Python. It's really the best mix of static and dynamic typing. You can use dynamic typing for the ease of development and optional static typing for safety, speed and documentation. And you should really only use static types where they help. Right? That's important. And so don't type all your things. Right? There are limits to what you should use types in your code for. And this is especially important when it comes to Python because what Python gives you is types in Python. And it uses them to accelerate your code. So Python is a pragmatic programming language. It's gradle typing for Python and it's an optimizing compiler. So it takes your Python code or you type annotated Python code and translates it to C and that compiles into a native extension module for Python that you can just import like any other binary module. It's production proven. It's widely used as you've seen. There are lots of tools using it. And it's really all about getting things done in the same way that Python is. It helps you keep your focus on functionality, removes the boilerplate for writing native modules. It allows you to move freely between Python and C. You'll see that in a minute. So it gives you a language that is as Pythonic as you want and as low level as you need it. Okay. So it's a demo. I'll make that bigger. Okay. Who's never used Jupyter Notebook? Again, the kid over there. See? Teach him. So yeah. So pretty much almost everyone else did and it's a very nice and interactive way to use Python to jump around in your code to show stuff. So what I have here is a Jupyter Notebook. And as you probably also know, Jupyter supports lots of different languages. I remember that it was 14 different languages like years ago. It's probably 20, 30 something in that order now. And it supports Python. So it's included in there. If you just say load XSython and have it installed, obviously, that teaches it how to run Python code, how to compile it in a Jupyter cell. So I'll just show you what I'm using here. Python 37. Fairly old version. But anyway, Python 3, fairly new version. NumPy GCC7. Oh, yeah. And that's something I should mention. Since Python compiles your translator code to see what you need now is a C compiler. Python is nice and easy. You write your Python code to run Python, run my code, and it runs it. In Scython, there's a compilation step before that. Because in the end, you get a natively compiled module, a native code, and that requires a C compiler. But that's it. That's all you pay. Okay. So very simple example. I'm going to take the Python module, import the sign function, and calculate sign of 5. Okay. It's a bit boring. I can do the same thing in Scython. And one difference you notice here, so the first thing is this is how I declare a Scython cell. I just tell Jupyter, you know, this cell is no longer running in Python. It needs to be passed through Scython. This compiled module for me. And then what Jupyter does in the back is it imports that module and executes it along the way. And that's also the reason why there's a print statement down here, rather than just the sign of 5 as you would know from Jupyter. So in Jupyter, cells, in Python cells, the last result of an expression kind of falls out of the notebook and gets displayed. And in a Scython cell, since the cell is compiled to native code, that doesn't work anymore. So Jupyter can't look into native code. It just executes some module and does nothing for no other good. So I have to explicitly say print. But that's fine, I think. Okay. It gives the same result. Now, as I said, since Scython compiles a C code here, that translates to C code, I can start using C functions now, or C functionality. And what I do here is I take the sign function from the libc math header. So this is the C sign function. And for now, I'll just assign it to a variable. And then I can call that from Jupyter. Who's surprised? Anyone? No one? A couple of people over there? What's surprising here? Yeah, no compilation. Nothing you can see. So the Scython annotation up here makes it a Scython cell. It compiles it for me. But the funny thing is, just by assigning the C sign function here, I can actually call it from Python. So now Python knows how to call a C function. That's interesting. The reason for that is that Scython is a type language, since C is a type language. So the sign function is something that knows that it's a function, a C function that takes a double floating point value in and outputs a double, or returns a double floating point value. So it knows the input and output values. And that allows Scython to directly wrap this function as a Python object. And when I assign it to global Python name here, it does it for me. Because that's an obvious assignment. I assign a C function to a Python variable. What else should it do than wrap it for me? Obvious, right? And does it. So this is kind of the quickest way to wrap a simple C function for Python. Here's the same thing spelled out. So again, asking Scython to learn what the C math header is. And then I write a Python function. You can see the extension here allows me, so Scython allows me to declare C argument types as I would in a C function. And then I can just call the sign function. And this is me manually wrapping this C function call here in a Python function. Which essentially does the same as above here. So it's not automatic. So when I run this and down here, now I can call the Python function and it calls the C function internally. Okay. That's still a bit boring because as you've seen here, Scython can also do this happily on its own. It doesn't need my code writing for this. So when it becomes more interesting to do this, that's when you can move more functionality into the Scython layer. I can easily now call this C sign function up there that I just wrote with x squared as input. But that would calculate the x square in Python. Then take the result of that, pass it into a C function, calculate that in C space and return it for me and return that as a Python object again. If I do the squaring also in Scython, then that gets translated into C code as well. And now the whole expression, the whole function body itself runs in C at C speed rather than me doing something in Python then in C and then passing it back to Python. So the nice thing about Scython is that it allows me to move functionality freely between Python space, Scython generated C space and at the end also the low level C written code. And it's up to me as a developer to decide where I want to put my functionality, how I want to implement this. In some cases I have an existing library, a C library for example that I want to call. I can just do that. I can declare it in Scython, call it from Scython code, do some more stuff in Scython space to keep it at the below the Python level. And when I'm done with it, then I return to Python. So I have all three levels available in one programming language and that gives me a lot of freedom for moving around optimizing code in these three levels of performance also. So here's an example I'll go through somewhat quickly. It's about speeding up NumPy code. Any questions so far on what I presented? Anything you would like to know or clarify? I didn't understand. I'm not always as clear as I want to be. I hope it was. Apparently it was. Thanks. Next example, NumPy. So the idea is we're going to calculate the average tax rate for Germany. I only have the numbers from 2016. So at the time there were 44 million people working. So paying income tax, let's put that way. And the average at the time was 3703 euros per month. So the average income was at the time 44,000 euros. That was 2016, it probably didn't change that much. It's still an average. So problem is I then tried to find official data for this and I couldn't find anything really probably due to data protection or something. Like there are probably a couple of people who aren't that much that you would identify them by their income. So there's no official data for this and what I did instead was I just took a look, normal distribution, fit it to the one data point that I have, made it kind of look good like this. And it's kind of not too far from what you would expect for an income distribution. So just assume this is actual data. Okay. So let's calculate everyone's taxes. When you look up the tax rate calculation in Wikipedia, what it gives you is a beautiful Excel formula. There's actually German Excel, right? So it says WIN and that's if. And then it gives you lots of formulas. So there are apparently different income ranges that change against each other and that have different formulas for them to calculate the tax rate. So the income tax for that income range. Okay. So you can easily translate that into Python and it becomes this, which I think is a bit more readable, especially because it allows you to ignore the right sides of the formula here and just go, ah, look at the different income ranges and then it does something. Just ignore the formulas. So this is how the income tax is calculated. And then in order to get the average tax rate, we take the sum of the taxes divided by the sum of the incomes. Okay. So I'll set my, you know, kind of a fake income list. I think I cut it down a bit. I'm not taking the 44 million, which is a fourth or something. And now, yeah, whenever you do a timings, don't forget to disable the energy management on your laptop because otherwise you get funny numbers. Okay. So it's going to take a while. And I can already show this a bit. So what we're going to do here is since we're optimizing this code, I have a little function that just collects different timings from different implementations. And so I remember the timing here and then we'll use this function to show the differences. So it took three seconds to calculate this whole thing in Python. Okay. So it's 3.2. Okay. So that's our baseline. Python has factor one. Now you can implement the same thing in NumPy and that gives you something like this. So we're slicing the income array, then doing some computations on them, and then build the sums on it. Okay. So it's kind of heavy NumPy code, but that's how you can do it in NumPy. And now we can calculate the whole thing in NumPy. It should be faster. So we're down to 62.1. And that is 50 times faster than Python. Okay. There's a different way to do this. So you can take the Python function of written and wrap it for NumPy so that it can apply it through the whole array, one item at a time. And then you can do the same formula, sum up the taxes and the incomes and divide one by the other. That again is going to take a while. That is slower than the slicing version, but it should still be faster than the Python version we had. And it's faster by quite a bit. So that gives us 49 milliseconds, and that is four times faster than Python still, but it's completely dwarfed by the NumPy version. Okay. Enter siphon. Here's a plain copy from the Python code I had above. It's doing the same thing. And now what I change is I compile the whole thing in siphon. And then do the calculation again. I shouldn't show this yet. Okay. Yeah. As you can see, it takes a while to calculate this. Actually, it already takes a while to... Yeah, it's running here. Okay. So it takes 2.74 seconds. Yep. And as you can see, that is 70% faster than the Python version, which is nice, right? I mean, all we really added is this little line up here. And so how much is that, so that's 8 characters for 17% speed up? That's certainly areas where that's acceptable, right? Okay. But we can certainly do better. Now, where siphon shines is static typing, and I told you about gradual typing. So what we're going to do now here is we'll gradually start typing the siphon code. And I'm going to use the PEP for notation for it. So the income that the text function here is calculating is definitely some double, right? C double value. It's perfectly fine to represent that because some thought in part in doubles is just fine for these calculations. Okay. And returns. Well, it does some calculation afterwards, and so it definitely also returns a double safely. Now next thing I notice is this function is only used internally inside of my module. I'm not exposing it anywhere. I'm really just using it down here. So what I can do is I can convert it to a C function, which is faster to call than a Python function because it has different call semantics. In C code, you can call C functions pretty much without overhead. In Python code, calling a Python function is a lot of overhead. Also in siphon code, calling a Python function is a lot of overhead because they are called with argument tuples, maybe keyboard arguments even. And so creating these objects, even just for calling the function, takes a lot of time. In C, it just passed by a stack or a register, so that's very, very fast as fast as it gets. So one thing I can do is I can declare this function as a C function. And now when I compile this, then siphon compiles it for me and I can time it again. And we're down to 205 milliseconds. So when I compare this to what we had before, that is now quite a bit faster than the Python version by 15 times. And the compiled version that we had before is still 13 times slower. So this was a speedup by 13 times compared to the version we had before. Why is that? Because one thing is the call overhead that I completely removed. So this is, you know, that's kind of the inner function of my loop when I'm doing a lot of work, right? And the call overhead for that function was removed, so that calling that function, going into that function is basically low time now. But the second thing happened. By typing the input and output arguments, siphon understood this function better and managed to generate plain C code for it because it knew now that income, you know, that's a C double. It didn't have to do any object comparisons anymore. It can calculate this whole thing in C space now. So by typing some variables, by typing some arguments in there, siphon takes decisions about my code and adapts it to the variable types. And you can see that with a functionality called annotate. So siphon-a gives me HTML output. And in here, it replicates, so it outputs an HTML snippet into my notebook that replicates my code. And in here, sorry, in here you can see, when I click on one line, this is just plain C code, right? Income greater than something. Same thing here. Take it formula, go to something, right? And in Python, that would look a lot more involved. It would do lots of Python C API operations, lots of object operations along the way. And this now really generated plain C code. And it gives me a speed of 13 times. Okay. So I would then continue this. There's a lot more I can do at this front. And I think the final speed up in the end that I usually have is, do I have it here? Yeah. So in the end, I usually manage to get it down to something like 11 milliseconds from the current 200. Right? So that is another factor of almost 20. Okay? I'm going to switch to a different example here. So any questions regarding this topic so far? Okay. Yeah. I think you were first over there. Is the decorator C func same as C def? Yes. Well, C func is actually explicitly saying this is a C function, which is the same as declaring as a C def function. Right? So there's more meanings than that, but this is more specific. For the others, the question was, there's a different syntax in Scythian, which is not Python compatible. And there's a special file type also, which is PIX instead of PY, which allows you to use the syntax. And I've only been using Python syntax here. You can do the same thing in a special syntax, which is more relevant when you start talking to external C code, because that is a ground where you cannot cover the same thing with Python compatible functionality anymore. And that's where we use the second syntax. But as you can see, you can get very far by just adding decorators, Python type annotations, and so on and so forth. So that's me making your code a little more C-ish than it used to be. That also helps in a lot of cases. Okay. Question over there? So if you were to actually have that in a code base, would you put this part of the code in a PYX file, or would you have it in a Python file and only annotate? Do you siphonize the file and compile it, then you ship it, or do you ship it, and then if you have siphon on, you go and you have that in a music file? So question was, this is in a notebook, right, where it's nice you can compile a cell or use a Python cell, like, mix them as you wish. How is that when you have code in the module, in the Python module? How do you optimize that? So siphon compiles one module at a time, meaning you would normally pass the whole module through siphon and compile it. It rarely hurts to also speed up a couple of other places in your code without actually annotating them, so if you just want to optimize one function, you can still compile the whole module, right? This might not even be a bad idea. Different choices that you have. One is Python often uses this concept of an accelerator module where you have a Python implementation of something and then you have an external native implementation of it, which you import in the end and if the import works, then you replace some function in the Python module with some faster function, something like this, right? And you can do the same thing in siphon, right? You can externalize one function into a separate module, compile it in siphon or not, and then import it from there, right? And if it's compiled, then it's probably faster. If it's not compiled, then it's as fast as Python will run it. Or as PyPy will run it in that case, for example, right? So that is fairly nice. This syntax actually allows you to optionally compile code, right? You can have it as fast as PyPy can run it and in siphon you can compile it and get it as fast as siphon can run native code. Also nice. Okay. I'm currently looking for a little example that I had, those related. Yeah, okay. I mean, that answers the question, right? Okay. Okay. Anything else? Then the next example that I have is C++. Just a very quick example. So as I said, siphon generates C code by default, but it can also generate the C code into the C++ module and that allows you to use C++ code along the way from your siphon code. And that is very nice from siphon because C++ is an object-oriented language, Python is an object-oriented language, so you can mix two object-oriented languages in the same siphon module and they look totally like Python when you use them. Even C++ looks a lot like Python when you use it from siphon, but it gives you a nice additional standard library, like fast data types, for example, fast container data types, 3D STL or anything you might be able to write in C++ yourself and want to wrap it for siphon. So that is a very nice addition to siphon, actually. Here is a very quick example. One thing you have to change is you have to tell Siphon explicitly that the language it should use is C++. There is more than one way to do this, but this is a common one, especially from a notebook. And then that enables access to the C++ STL declarations that we already ship in Siphon because they are so commonly used. But then you can use this syntax here to say I have variable v, that is C++ vector of ints, and then I just push a value in there and return the vector. What does this do? Returning a C++ vector from a Python function. What do you think happens? It will transfer it to a list. It will copy it into a list because the obvious representation of a C++ vector in Python space would be something list-like. You would use a Python list for that. So simply returning something that Siphon understands as list-like will copy it into a list, Python list automatically. So it will create Python for you, copy all the values from the C++ vector and return that. Okay? All automatic. If you don't want that, you can implement your own little thing in whatever way. You can use a list comprehension to get something out of the C++ vector. Or you can return a generator expression that uses the C++ vector. You can do all these kind of real things by mixing Python features with C++ features. But this is the most simple and straightforward way to do it. So when I call this function, then I get turned back, which is very surprising. This is a very common way to wrap C++ objects for Python. I can use a so-called Cdef class. So Cdef class is an extension type in Python. So it's a native, low-level implemented class, like a Python class, but implemented in C. And in Scythian, these extension types allow me to directly use C++ objects as instance attributes, that's an instance attribute. And I'm using a vector of int as values attribute. And then the lifetime of this C++ object is tied to the lifetime of a Python object. So it's automatically created when I create a Python object. It's automatically de-allocated for me when my Python object goes on a scope. So I don't have to care about any memory management. It's all automatic in this case. Very nice. And so this is just a very tiny example that uses an integer, this like object, wrapper object in Python. I can add values to it, which is the pushback C++ function to push it into the C++ vector. And as you can see from the usage here, that totally looks like I'm using Python code. I'm calling method on something, I don't care if it's a C++ object or a Python object, just call it. I also like the wrapper implementation here. What do you think this does? It runs wrapper on the list. So it does the same thing as we had before in the function. It creates a Python list from the values. So this is actually very expensive. But hey, how often do I actually need wrapper? So what it does is it copies all the values in the Python list, uses wrapper on that, and then gives the string back to Python. If it's only something I occasionally use, why not? It doesn't cost much. Okay. So much for C++. Any more questions on that field? Last time you checked, we didn't have bit vector. You mean in the C++ declarations. The membership might be. Please provide a pull request and we'll happily add it. Anything else? A gil handling. I actually have another example for that as the next and last example that I'm showing. So here's an example of wrapper in external library. That's the last use case that I wanted to show here. And so far we have seen this C port, something, it's a bit of magic. And this is unpacking the magic. What we ship is site declarations of most of the C header set of the C++ header set of the STL. Well, C++ is huge, but much of the STL. And this is how it's spelled out in the end. This is special syntax for that. So what I want to do now is I write a little wrapper for the Lua runtime. So I'll execute Lua code from Python. And for this, I just need to declare a couple of functions that the Lua API defines. And then up here, declare how to link against this. This is a bit of a heavy way to do this writing notebook. There are more nicer ways to do this from the setup file, but this works for now. And then once these are declared, I can use them in my code. So if you find a Python function, it takes a code string, converts it to a native needs to or accepts unicode and byte strings. So you can just use a normal Python string from Python 3 and pass it in there. Then creates a new Lua state as a Lua runtime representation. If that fails, then raises memory error. That is also fairly nice. Do you know what you would have to do in C in order to do this? Happily, there's a C API function for it. But just being able to say, you know, do some C stuff if that fails, if I get an out pointer from some memory or an education back, raise memory error? That is nice. That is what you want in your code. So that is totally the Python way of doing it. So then I have a try and accept. Sorry, try finally. And at the end of the finally, I delete the Lua runtime again. So if anything goes wrong along the way, then I make sure I clean up the memory. Because C requires manual memory management. I am responsible for cleaning everything up that goes wrong. And in Python, I can use try finally and it is just going to work. So here is Lua function to load the code into Lua, execute it from there, and then a tiny bit of return value adaptation to return whatever number Lua wants to return here. Okay. I just quickly run this and notice that I didn't run the beginning. And then run it again. And it fails because I didn't set up my Lua properly. Just believe me that this works. Okay. It is not the first time I show an example. It is just the first time it fails so heavily. I am not going to fix it right now in the last two minutes. Okay. One title of that talk was siphon 3. So those of you who already use siphon will probably know that siphon is always this 0. something version kind of thing. So 0.29 currently. So what is the siphon 3? Where does that come from? So the last version that we have is 0.29. So if you push the dots one digit to the right, then it becomes 3. The next version is 3.0. Okay. So that is kind of frightening. You jump from 0. something to 3.0. What is special about it? First thing is siphon 3 is Python. Now you are going to say, but it was always Python. What is special about it? Well, it is Python 3. And some of you might say, I could always put my Python 3 code in there. I just had to say language level equals 3 and then compiled it. Well, it is Python 3 by default. So the language level is 3 by default. So there are a couple of things that we changed now in siphon 3. It is Python 3 by default. We are going to change mostly the standard configuration of the compiler to make it more modern, to move stuff that is not appropriate anymore these days. And we are going to adapt it to today's Python 3 world. So these are the main changes in there. We are going to adapt to several peps. That came along the way and said Python 3 by default, yes. You can look up the Python 3 milestone. There is still quite a bit to do along the way and we will try to get there. Okay. That is it for my site. Thanks for listening. Thank you.
 Hey, everyone.  I'm one of the site and core developers.  Before I start this talk, I'm going to show you a lot of examples of site and how to use  it in different fields.  I'd like to know a bit what you're more interested in.  Who's never used site and before?  That is some... It's true.  The youngest non-site user.  We'll introduce some to site today.  That's about one-third of you.  Who is interested in integrating and accelerating numpy code, computational code?  Accelerating computational code with numpy and all that.  That seems a bit less than the group we just had.  Talking to external libraries, integrating native code into Python, is that a topic for  you?  Raise your hands.  A couple of people.  Anyone using C++?  Yeah.  Surprisingly, not so few.  I'll try to focus a bit on what you just said.  Okay.  I'll start with myself.  So, I'm a software developer, data engineer, whatever you want to name it these days.  I've been using Python since 2002.  I'm one of the founders of the Python project, not of the original inventors, but when we  forked the project from Pyrex, that was in 2007.  I was already on board.  And I'm also a CPython core dev since this year.  I do training and consulting in-house if you want to know anything about Python, have it  taught in your company at home, you can contact me.  I might come over and teach you something.  I'll definitely have something to teach you or look in your code and approve it.  I've been working for TrustU since 2017.  And I use that as a little introductory example because TrustU is a data company.  And that fits very well in the field.  So what we do is we have terabytes of hotel guest reviews that we collect from all over  the Internet.  And when I say guest reviews, that literally brain dumps of arbitrary people writing stuff  on the Internet, right?  So we collect them, analyze them in 24 languages.  There's a lot of NLP involved.  Have surveys that we send out for the hotels, collect them from portals, from partners that  we have to text and data analysis on them, and then find out what people are talking  about, what kind of different categories they are talking about, and analyze trends, find  out this kind of information.  So that back to the hotels to tell them how they can improve.  So we can tell the hotels, you know, if you renovate your pool or hire someone new for  the reception, you'll probably improve your score by 10%.  That's what we tell them.  So this might also be interesting for you.  So if you go to trust.com and type in the name of hotels, next time you look for a hotel  somewhere, type it in there, and we'll tell you exactly what people actually think about  this hotel, right?  Notebooking.com wants you to think about your hotel in order to book it.  We're independent, and so we can give you the actual opinions based on actual reviews,  actual data.  So go there, remember this, and you'll see it's all cut down into different categories  and all that, so it's really relevant information that you want to know before you book that  hotel.  Why do I tell you all this?  Well, these are some tools that we use at Trust.io.  It's a data company, so we use NumPy, SciPy, we use Scikit-learn for things.  We use Pinellis for data analysis, use NLTK and Spacey for text analysis, Alex and Mel  whenever we have to deal with XML, PyYaml for configuration and all this, and it's all  based on Hadoop processing and Spark.  Some MapReduce, some whatever Hadoop provides, Hive and all these technologies.  So this is mostly what we do.  And now most of these tools are actually partly or even entirely written in Python.  Right?  Who didn't know this?  Raise your hands.  Interesting.  NLTK is no longer entirely true because we're working on patch for them.  NLTK is a pure Python toolkit.  So I've written a PR for them to start compiling certain of their modules so that they just  run faster if you install the binary instead of the Python package.  So we're working on that.  Hadoop is straight out because most written Java.  But Trust is on the list.  Who's seen this?  Has anyone not seen it yet?  Okay.  So that's the image that the EHT project took off a black hole a couple of months ago.  And the interesting thing about this here is that these are the tools that they use  for calculating this image out of the raw data.  And as you may notice, they're all Python tools.  They use pandas, numpy, astropy, scikit image over there.  Scipy, all these tools, all Python tools for aggregating the, again, I think they start  with petabytes of data coming from eight different telescopes being taken across 10 days in a  row.  Right?  Like huge telescopes all around the world.  Like the data cut it down into manageable data sizes and then calculated this image  out of it.  And that was all done in Python.  And now if you look at the tools that they used, all of them actually use Siphon inside.  Right?  So that was a huge Siphon-based project, although they didn't, you know, largely advertise that,  but it was.  Okay.  So what is Siphon actually used for?  It's used for integrating native code with Python.  It's used to speed up Python code in C Python and simply even use it to write C, you know,  without having to write C.  So basically we write C so you don't have to.  Okay?  Skip that.  Gradle typing.  Does anyone know what gradle typing is?  I heard the term before, maybe in context with PEP484, the typing annotations in Python.  So the basic idea is there are two main different ways to type languages.  One is statically typed, which tends to be, you know, fast languages, but it tends to  become a sum because you have to type everything.  It's very annoying.  And then the dynamic languages like Python, which are easy to use, but tend to be slow  kind of.  It's not that simple, but more or less those are the two fields.  And gradle typing is somewhere between.  It says, you know, you can use typing, but it helps.  It's documenting, but it makes things faster.  That's where you should use typing.  And for everything else, it's fine if you don't use it and don't need it there.  So it was termed coined in 2006.  And there's a blog post, just look for what is gradle typing and you'll find it.  And it was the basis for PEP484 type annotations in Python.  It's really the best mix of static and dynamic typing.  You can use dynamic typing for the ease of development and optional static typing for  safety, speed and documentation.  And you should really only use static types where they help.  Right?  That's important.  And so don't type all your things.  Right?  There are limits to what you should use types in your code for.  And this is especially important when it comes to Python because what Python gives you is  types in Python.  And it uses them to accelerate your code.  So Python is a pragmatic programming language.  It's gradle typing for Python and it's an optimizing compiler.  So it takes your Python code or you type annotated Python code and translates it to C and that  compiles into a native extension module for Python that you can just import like any other  binary module.  It's production proven.  It's widely used as you've seen.  There are lots of tools using it.  And it's really all about getting things done in the same way that Python is.  It helps you keep your focus on functionality, removes the boilerplate for writing native  modules.  It allows you to move freely between Python and C. You'll see that in a minute.  So it gives you a language that is as Pythonic as you want and as low level as you need it.  Okay.  So it's a demo.  I'll make that bigger.  Okay.  Who's never used Jupyter Notebook?  Again, the kid over there.  See?  Teach him.  So yeah.  So pretty much almost everyone else did and it's a very nice and interactive way to use  Python to jump around in your code to show stuff.  So what I have here is a Jupyter Notebook.  And as you probably also know, Jupyter supports lots of different languages.  I remember that it was 14 different languages like years ago.  It's probably 20, 30 something in that order now.  And it supports Python.  So it's included in there.  If you just say load XSython and have it installed, obviously, that teaches it how to run Python  code, how to compile it in a Jupyter cell.  So I'll just show you what I'm using here.  Python 37.  Fairly old version.  But anyway, Python 3, fairly new version.  NumPy GCC7.  Oh, yeah.  And that's something I should mention.  Since Python compiles your translator code to see what you need now is a C compiler.  Python is nice and easy.  You write your Python code to run Python, run my code, and it runs it.  In Scython, there's a compilation step before that.  Because in the end, you get a natively compiled module, a native code, and that requires a  C compiler.  But that's it.  That's all you pay.  Okay.  So very simple example.  I'm going to take the Python module, import the sign function, and calculate sign of 5.  Okay.  It's a bit boring.  I can do the same thing in Scython.  And one difference you notice here, so the first thing is this is how I declare a Scython  cell.  I just tell Jupyter, you know, this cell is no longer running in Python.  It needs to be passed through Scython.  This compiled module for me.  And then what Jupyter does in the back is it imports that module and executes it along  the way.  And that's also the reason why there's a print statement down here, rather than just  the sign of 5 as you would know from Jupyter.  So in Jupyter, cells, in Python cells, the last result of an expression kind of falls  out of the notebook and gets displayed.  And in a Scython cell, since the cell is compiled to native code, that doesn't work anymore.  So Jupyter can't look into native code.  It just executes some module and does nothing for no other good.  So I have to explicitly say print.  But that's fine, I think.  Okay.  It gives the same result.  Now, as I said, since Scython compiles a C code here, that translates to C code, I can  start using C functions now, or C functionality.  And what I do here is I take the sign function from the libc math header.  So this is the C sign function.  And for now, I'll just assign it to a variable.  And then I can call that from Jupyter.  Who's surprised?  Anyone?  No one?  A couple of people over there?  What's surprising here?  Yeah, no compilation.  Nothing you can see.  So the Scython annotation up here makes it a Scython cell.  It compiles it for me.  But the funny thing is, just by assigning the C sign function here, I can actually call  it from Python.  So now Python knows how to call a C function.  That's interesting.  The reason for that is that Scython is a type language, since C is a type language.  So the sign function is something that knows that it's a function, a C function that takes  a double floating point value in and outputs a double, or returns a double floating point  value.  So it knows the input and output values.  And that allows Scython to directly wrap this function as a Python object.  And when I assign it to global Python name here, it does it for me.  Because that's an obvious assignment.  I assign a C function to a Python variable.  What else should it do than wrap it for me?  Obvious, right?  And does it.  So this is kind of the quickest way to wrap a simple C function for Python.  Here's the same thing spelled out.  So again, asking Scython to learn what the C math header is.  And then I write a Python function.  You can see the extension here allows me, so Scython allows me to declare C argument  types as I would in a C function.  And then I can just call the sign function.  And this is me manually wrapping this C function call here in a Python function.  Which essentially does the same as above here.  So it's not automatic.  So when I run this and down here, now I can call the Python function and it calls the  C function internally.  Okay.  That's still a bit boring because as you've seen here, Scython can also do this happily  on its own.  It doesn't need my code writing for this.  So when it becomes more interesting to do this, that's when you can move more functionality  into the Scython layer.  I can easily now call this C sign function up there that I just wrote with x squared  as input.  But that would calculate the x square in Python.  Then take the result of that, pass it into a C function, calculate that in C space and  return it for me and return that as a Python object again.  If I do the squaring also in Scython, then that gets translated into C code as well.  And now the whole expression, the whole function body itself runs in C at C speed rather than  me doing something in Python then in C and then passing it back to Python.  So the nice thing about Scython is that it allows me to move functionality freely between  Python space, Scython generated C space and at the end also the low level C written code.  And it's up to me as a developer to decide where I want to put my functionality, how  I want to implement this.  In some cases I have an existing library, a C library for example that I want to call.  I can just do that.  I can declare it in Scython, call it from Scython code, do some more stuff in Scython  space to keep it at the below the Python level.  And when I'm done with it, then I return to Python.  So I have all three levels available in one programming language and that gives me a lot  of freedom for moving around optimizing code in these three levels of performance also.  So here's an example I'll go through somewhat quickly.  It's about speeding up NumPy code.  Any questions so far on what I presented?  Anything you would like to know or clarify?  I didn't understand.  I'm not always as clear as I want to be.  I hope it was.  Apparently it was.  Thanks.  Next example, NumPy.  So the idea is we're going to calculate the average tax rate for Germany.  I only have the numbers from 2016.  So at the time there were 44 million people working.  So paying income tax, let's put that way.  And the average at the time was 3703 euros per month.  So the average income was at the time 44,000 euros.  That was 2016, it probably didn't change that much.  It's still an average.  So problem is I then tried to find official data for this and I couldn't find anything  really probably due to data protection or something.  Like there are probably a couple of people who aren't that much that you would identify  them by their income.  So there's no official data for this and what I did instead was I just took a look,  normal distribution, fit it to the one data point that I have, made it kind of look good  like this.  And it's kind of not too far from what you would expect for an income distribution.  So just assume this is actual data.  Okay.  So let's calculate everyone's taxes.  When you look up the tax rate calculation in Wikipedia, what it gives you is a beautiful  Excel formula.  There's actually German Excel, right?  So it says WIN and that's if.  And then it gives you lots of formulas.  So there are apparently different income ranges that change against each other and that have  different formulas for them to calculate the tax rate.  So the income tax for that income range.  Okay.  So you can easily translate that into Python and it becomes this, which I think is a bit  more readable, especially because it allows you to ignore the right sides of the formula  here and just go, ah, look at the different income ranges and then it does something.  Just ignore the formulas.  So this is how the income tax is calculated.  And then in order to get the average tax rate, we take the sum of the taxes divided by the  sum of the incomes.  Okay.  So I'll set my, you know, kind of a fake income list.  I think I cut it down a bit.  I'm not taking the 44 million, which is a fourth or something.  And now, yeah, whenever you do a timings, don't forget to disable the energy management  on your laptop because otherwise you get funny numbers.  Okay.  So it's going to take a while.  And I can already show this a bit.  So what we're going to do here is since we're optimizing this code, I have a little function  that just collects different timings from different implementations.  And so I remember the timing here and then we'll use this function to show the differences.  So it took three seconds to calculate this whole thing in Python.  Okay.  So it's 3.2.  Okay.  So that's our baseline.  Python has factor one.  Now you can implement the same thing in NumPy and that gives you something like this.  So we're slicing the income array, then doing some computations on them, and then build  the sums on it.  Okay.  So it's kind of heavy NumPy code, but that's how you can do it in NumPy.  And now we can calculate the whole thing in NumPy.  It should be faster.  So we're down to 62.1.  And that is 50 times faster than Python.  Okay.  There's a different way to do this.  So you can take the Python function of written and wrap it for NumPy so that it can apply  it through the whole array, one item at a time.  And then you can do the same formula, sum up the taxes and the incomes and divide one  by the other.  That again is going to take a while.  That is slower than the slicing version, but it should still be faster than the Python  version we had.  And it's faster by quite a bit.  So that gives us 49 milliseconds, and that is four times faster than Python still, but  it's completely dwarfed by the NumPy version.  Okay.  Enter siphon.  Here's a plain copy from the Python code I had above.  It's doing the same thing.  And now what I change is I compile the whole thing in siphon.  And then do the calculation again.  I shouldn't show this yet.  Okay.  Yeah.  As you can see, it takes a while to calculate this.  Actually, it already takes a while to...  Yeah, it's running here.  Okay.  So it takes 2.74 seconds.  Yep.  And as you can see, that is 70% faster than the Python version, which is nice, right?  I mean, all we really added is this little line up here.  And so how much is that, so that's 8 characters for 17% speed up?  That's certainly areas where that's acceptable, right?  Okay.  But we can certainly do better.  Now, where siphon shines is static typing, and I told you about gradual typing.  So what we're going to do now here is we'll gradually start typing the siphon code.  And I'm going to use the PEP for notation for it.  So the income that the text function here is calculating is definitely some double,  right?  C double value.  It's perfectly fine to represent that because some thought in part in doubles is just fine  for these calculations.  Okay.  And returns.  Well, it does some calculation afterwards, and so it definitely also returns a double  safely.  Now next thing I notice is this function is only used internally inside of my module.  I'm not exposing it anywhere.  I'm really just using it down here.  So what I can do is I can convert it to a C function, which is faster to call than a  Python function because it has different call semantics.  In C code, you can call C functions pretty much without overhead.  In Python code, calling a Python function is a lot of overhead.  Also in siphon code, calling a Python function is a lot of overhead because they are called  with argument tuples, maybe keyboard arguments even.  And so creating these objects, even just for calling the function, takes a lot of time.  In C, it just passed by a stack or a register, so that's very, very fast as fast as it gets.  So one thing I can do is I can declare this function as a C function.  And now when I compile this, then siphon compiles it for me and I can time it again.  And we're down to 205 milliseconds.  So when I compare this to what we had before, that is now quite a bit faster than the Python  version by 15 times.  And the compiled version that we had before is still 13 times slower.  So this was a speedup by 13 times compared to the version we had before.  Why is that?  Because one thing is the call overhead that I completely removed.  So this is, you know, that's kind of the inner function of my loop when I'm doing a lot of  work, right?  And the call overhead for that function was removed, so that calling that function, going  into that function is basically low time now.  But the second thing happened.  By typing the input and output arguments, siphon understood this function better and  managed to generate plain C code for it because it knew now that income, you know, that's  a C double.  It didn't have to do any object comparisons anymore.  It can calculate this whole thing in C space now.  So by typing some variables, by typing some arguments in there, siphon takes decisions  about my code and adapts it to the variable types.  And you can see that with a functionality called annotate.  So siphon-a gives me HTML output.  And in here, it replicates, so it outputs an HTML snippet into my notebook that replicates  my code.  And in here, sorry, in here you can see, when I click on one line, this is just plain C  code, right?  Income greater than something.  Same thing here.  Take it formula, go to something, right?  And in Python, that would look a lot more involved.  It would do lots of Python C API operations, lots of object operations along the way.  And this now really generated plain C code.  And it gives me a speed of 13 times.  Okay.  So I would then continue this.  There's a lot more I can do at this front.  And I think the final speed up in the end that I usually have is, do I have it here?  Yeah.  So in the end, I usually manage to get it down to something like 11 milliseconds from  the current 200.  Right?  So that is another factor of almost 20.  Okay?  I'm going to switch to a different example here.  So any questions regarding this topic so far?  Okay.  Yeah.  I think you were first over there.  Is the decorator C func same as C def?  Yes.  Well, C func is actually explicitly saying this is a C function, which is the same as  declaring as a C def function.  Right?  So there's more meanings than that, but this is more specific.  For the others, the question was, there's a different syntax in Scythian, which is not  Python compatible.  And there's a special file type also, which is PIX instead of PY, which allows you to  use the syntax.  And I've only been using Python syntax here.  You can do the same thing in a special syntax, which is more relevant when you start talking  to external C code, because that is a ground where you cannot cover the same thing with  Python compatible functionality anymore.  And that's where we use the second syntax.  But as you can see, you can get very far by just adding decorators, Python type annotations,  and so on and so forth.  So that's me making your code a little more C-ish than it used to be.  That also helps in a lot of cases.  Okay.  Question over there?  So if you were to actually have that in a code base, would you put this part of the  code in a PYX file, or would you have it in a Python file and only annotate?  Do you siphonize the file and compile it, then you ship it, or do you ship it, and then  if you have siphon on, you go and you have that in a music file?  So question was, this is in a notebook, right, where it's nice you can compile a cell or  use a Python cell, like, mix them as you wish.  How is that when you have code in the module, in the Python module?  How do you optimize that?  So siphon compiles one module at a time, meaning you would normally pass the whole module through  siphon and compile it.  It rarely hurts to also speed up a couple of other places in your code without actually  annotating them, so if you just want to optimize one function, you can still compile the whole  module, right?  This might not even be a bad idea.  Different choices that you have.  One is Python often uses this concept of an accelerator module where you have a Python  implementation of something and then you have an external native implementation of it, which  you import in the end and if the import works, then you replace some function in the Python  module with some faster function, something like this, right?  And you can do the same thing in siphon, right?  You can externalize one function into a separate module, compile it in siphon or not, and then  import it from there, right?  And if it's compiled, then it's probably faster.  If it's not compiled, then it's as fast as Python will run it.  Or as PyPy will run it in that case, for example, right?  So that is fairly nice.  This syntax actually allows you to optionally compile code, right?  You can have it as fast as PyPy can run it and in siphon you can compile it and get it  as fast as siphon can run native code.  Also nice.  Okay.  I'm currently looking for a little example that I had, those related.  Yeah, okay.  I mean, that answers the question, right?  Okay.  Okay.  Anything else?  Then the next example that I have is C++.  Just a very quick example.  So as I said, siphon generates C code by default, but it can also generate the C code into  the C++ module and that allows you to use C++ code along the way from your siphon code.  And that is very nice from siphon because C++ is an object-oriented language, Python  is an object-oriented language, so you can mix two object-oriented languages in the same  siphon module and they look totally like Python when you use them.  Even C++ looks a lot like Python when you use it from siphon, but it gives you a nice  additional standard library, like fast data types, for example, fast container data types,  3D STL or anything you might be able to write in C++ yourself and want to wrap it for siphon.  So that is a very nice addition to siphon, actually.  Here is a very quick example.  One thing you have to change is you have to tell Siphon explicitly that the language  it should use is C++.  There is more than one way to do this, but this is a common one, especially from a notebook.  And then that enables access to the C++ STL declarations that we already ship in Siphon  because they are so commonly used.  But then you can use this syntax here to say I have variable v, that is C++ vector of ints,  and then I just push a value in there and return the vector.  What does this do?  Returning a C++ vector from a Python function.  What do you think happens?  It will transfer it to a list.  It will copy it into a list because the obvious representation of a C++ vector in Python space  would be something list-like.  You would use a Python list for that.  So simply returning something that Siphon understands as list-like will copy it into  a list, Python list automatically.  So it will create Python for you, copy all the values from the C++ vector and return  that.  Okay?  All automatic.  If you don't want that, you can implement your own little thing in whatever way.  You can use a list comprehension to get something out of the C++ vector.  Or you can return a generator expression that uses the C++ vector.  You can do all these kind of real things by mixing Python features with C++ features.  But this is the most simple and straightforward way to do it.  So when I call this function, then I get turned back, which is very surprising.  This is a very common way to wrap C++ objects for Python.  I can use a so-called Cdef class.  So Cdef class is an extension type in Python.  So it's a native, low-level implemented class, like a Python class, but implemented in C.  And in Scythian, these extension types allow me to directly use C++ objects as instance  attributes, that's an instance attribute.  And I'm using a vector of int as values attribute.  And then the lifetime of this C++ object is tied to the lifetime of a Python object.  So it's automatically created when I create a Python object.  It's automatically de-allocated for me when my Python object goes on a scope.  So I don't have to care about any memory management.  It's all automatic in this case.  Very nice.  And so this is just a very tiny example that uses an integer, this like object, wrapper  object in Python.  I can add values to it, which is the pushback C++ function to push it into the C++ vector.  And as you can see from the usage here, that totally looks like I'm using Python code.  I'm calling method on something, I don't care if it's a C++ object or a Python object,  just call it.  I also like the wrapper implementation here.  What do you think this does?  It runs wrapper on the list.  So it does the same thing as we had before in the function.  It creates a Python list from the values.  So this is actually very expensive.  But hey, how often do I actually need wrapper?  So what it does is it copies all the values in the Python list, uses wrapper on that,  and then gives the string back to Python.  If it's only something I occasionally use, why not?  It doesn't cost much.  Okay.  So much for C++.  Any more questions on that field?  Last time you checked, we didn't have bit vector.  You mean in the C++ declarations.  The membership might be.  Please provide a pull request and we'll happily add it.  Anything else?  A gil handling.  I actually have another example for that as the next and last example that I'm showing.  So here's an example of wrapper in external library.  That's the last use case that I wanted to show here.  And so far we have seen this C port, something, it's a bit of magic.  And this is unpacking the magic.  What we ship is site declarations of most of the C header set of the C++ header set of  the STL.  Well, C++ is huge, but much of the STL.  And this is how it's spelled out in the end.  This is special syntax for that.  So what I want to do now is I write a little wrapper for the Lua runtime.  So I'll execute Lua code from Python.  And for this, I just need to declare a couple of functions that the Lua API defines.  And then up here, declare how to link against this.  This is a bit of a heavy way to do this writing notebook.  There are more nicer ways to do this from the setup file, but this works for now.  And then once these are declared, I can use them in my code.  So if you find a Python function, it takes a code string, converts it to a native needs  to or accepts unicode and byte strings.  So you can just use a normal Python string from Python 3 and pass it in there.  Then creates a new Lua state as a Lua runtime representation.  If that fails, then raises memory error.  That is also fairly nice.  Do you know what you would have to do in C in order to do this?  Happily, there's a C API function for it.  But just being able to say, you know, do some C stuff if that fails, if I get an out pointer  from some memory or an education back, raise memory error?  That is nice.  That is what you want in your code.  So that is totally the Python way of doing it.  So then I have a try and accept.  Sorry, try finally.  And at the end of the finally, I delete the Lua runtime again.  So if anything goes wrong along the way, then I make sure I clean up the memory.  Because C requires manual memory management.  I am responsible for cleaning everything up that goes wrong.  And in Python, I can use try finally and it is just going to work.  So here is Lua function to load the code into Lua, execute it from there, and then a tiny  bit of return value adaptation to return whatever number Lua wants to return here.  Okay.  I just quickly run this and notice that I didn't run the beginning.  And then run it again.  And it fails because I didn't set up my Lua properly.  Just believe me that this works.  Okay.  It is not the first time I show an example.  It is just the first time it fails so heavily.  I am not going to fix it right now in the last two minutes.  Okay.  One title of that talk was siphon 3.  So those of you who already use siphon will probably know that siphon is always this 0. something  version kind of thing.  So 0.29 currently.  So what is the siphon 3?  Where does that come from?  So the last version that we have is 0.29.  So if you push the dots one digit to the right, then it becomes 3.  The next version is 3.0.  Okay.  So that is kind of frightening.  You jump from 0. something to 3.0.  What is special about it?  First thing is siphon 3 is Python.  Now you are going to say, but it was always Python.  What is special about it?  Well, it is Python 3.  And some of you might say, I could always put my Python 3 code in there.  I just had to say language level equals 3 and then compiled it.  Well, it is Python 3 by default.  So the language level is 3 by default.  So there are a couple of things that we changed now in siphon 3.  It is Python 3 by default.  We are going to change mostly the standard configuration of the compiler to make it more  modern, to move stuff that is not appropriate anymore these days.  And we are going to adapt it to today's Python 3 world.  So these are the main changes in there.  We are going to adapt to several peps.  That came along the way and said Python 3 by default, yes.  You can look up the Python 3 milestone.  There is still quite a bit to do along the way and we will try to get there.  Okay.  That is it for my site.  Thanks for listening.  Thank you.
Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_0_815.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_816_887.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_888_2015.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_2016_2243.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_2244_5279.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_5280_5543.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_5544_6071.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_6072_7427.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_7428_8627.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_8628_10187.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_10188_10727.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_10728_10955.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_10956_11303.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_11304_11411.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_11412_12431.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_12432_12527.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_12528_12635.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_12636_12779.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_12780_13655.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_13656_13835.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_13836_13931.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_13932_14111.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_14112_14555.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_14556_14879.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_14880_15647.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_15648_15875.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_15876_17315.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_17316_18119.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_18120_18407.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_18408_19979.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_19980_20063.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_20064_20711.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_20712_23519.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_23520_30179.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_30180_31559.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_31560_32171.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_32172_32555.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_32556_34451.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_34452_35519.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_35520_36155.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_36156_36419.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_36420_36731.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_36732_36803.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_36804_36995.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_36996_37499.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_37500_37619.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_37620_38207.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_38208_39251.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_39252_39887.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_39888_41495.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_41496_41555.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_41556_42959.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_42960_44663.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_44664_44711.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_44712_45899.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_45900_46475.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_46476_53999.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_54000_63215.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_63216_63263.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_63264_65855.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_65856_65999.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66000_66059.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66060_66107.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66108_66575.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66576_66659.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66660_66959.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_66960_67355.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_67356_67727.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_67728_69287.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_69288_69611.png Stefan_Behnel_-_Get_up_to_speed_with_Cython_3_0_lxh4lsvEBhI_69612_69728.png
Hey, everyone. I'm one of the site and core developers. Before I start this talk, I'm going to show you a lot of examples of site and how to use it in different fields. I'd like to know a bit what you're more interested in. Who's never used site and before? That is some... It's true. The youngest non-site user. We'll introduce some to site today. That's about one-third of you. Who is interested in integrating and accelerating numpy code, computational code? Accelerating computational code with numpy and all that. That seems a bit less than the group we just had. Talking to external libraries, integrating native code into Python, is that a topic for you? Raise your hands. A couple of people. Anyone using C++? Yeah. Surprisingly, not so few. I'll try to focus a bit on what you just said. Okay. I'll start with myself. So, I'm a software developer, data engineer, whatever you want to name it these days. I've been using Python since 2002. I'm one of the founders of the Python project, not of the original inventors, but when we forked the project from Pyrex, that was in 2007. I was already on board. And I'm also a CPython core dev since this year. I do training and consulting in-house if you want to know anything about Python, have it taught in your company at home, you can contact me. I might come over and teach you something. I'll definitely have something to teach you or look in your code and approve it. I've been working for TrustU since 2017. And I use that as a little introductory example because TrustU is a data company. And that fits very well in the field. So what we do is we have terabytes of hotel guest reviews that we collect from all over the Internet. And when I say guest reviews, that literally brain dumps of arbitrary people writing stuff on the Internet, right? So we collect them, analyze them in 24 languages. There's a lot of NLP involved. Have surveys that we send out for the hotels, collect them from portals, from partners that we have to text and data analysis on them, and then find out what people are talking about, what kind of different categories they are talking about, and analyze trends, find out this kind of information. So that back to the hotels to tell them how they can improve. So we can tell the hotels, you know, if you renovate your pool or hire someone new for the reception, you'll probably improve your score by 10%. That's what we tell them. So this might also be interesting for you. So if you go to trust.com and type in the name of hotels, next time you look for a hotel somewhere, type it in there, and we'll tell you exactly what people actually think about this hotel, right? Notebooking.com wants you to think about your hotel in order to book it. We're independent, and so we can give you the actual opinions based on actual reviews, actual data. So go there, remember this, and you'll see it's all cut down into different categories and all that, so it's really relevant information that you want to know before you book that hotel. Why do I tell you all this? Well, these are some tools that we use at Trust.io. It's a data company, so we use NumPy, SciPy, we use Scikit-learn for things. We use Pinellis for data analysis, use NLTK and Spacey for text analysis, Alex and Mel whenever we have to deal with XML, PyYaml for configuration and all this, and it's all based on Hadoop processing and Spark. Some MapReduce, some whatever Hadoop provides, Hive and all these technologies. So this is mostly what we do. And now most of these tools are actually partly or even entirely written in Python. Right? Who didn't know this? Raise your hands. Interesting. NLTK is no longer entirely true because we're working on patch for them. NLTK is a pure Python toolkit. So I've written a PR for them to start compiling certain of their modules so that they just run faster if you install the binary instead of the Python package. So we're working on that. Hadoop is straight out because most written Java. But Trust is on the list. Who's seen this? Has anyone not seen it yet? Okay. So that's the image that the EHT project took off a black hole a couple of months ago. And the interesting thing about this here is that these are the tools that they use for calculating this image out of the raw data. And as you may notice, they're all Python tools. They use pandas, numpy, astropy, scikit image over there. Scipy, all these tools, all Python tools for aggregating the, again, I think they start with petabytes of data coming from eight different telescopes being taken across 10 days in a row. Right? Like huge telescopes all around the world. Like the data cut it down into manageable data sizes and then calculated this image out of it. And that was all done in Python. And now if you look at the tools that they used, all of them actually use Siphon inside. Right? So that was a huge Siphon-based project, although they didn't, you know, largely advertise that, but it was. Okay. So what is Siphon actually used for? It's used for integrating native code with Python. It's used to speed up Python code in C Python and simply even use it to write C, you know, without having to write C. So basically we write C so you don't have to. Okay? Skip that. Gradle typing. Does anyone know what gradle typing is? I heard the term before, maybe in context with PEP484, the typing annotations in Python. So the basic idea is there are two main different ways to type languages. One is statically typed, which tends to be, you know, fast languages, but it tends to become a sum because you have to type everything. It's very annoying. And then the dynamic languages like Python, which are easy to use, but tend to be slow kind of. It's not that simple, but more or less those are the two fields. And gradle typing is somewhere between. It says, you know, you can use typing, but it helps. It's documenting, but it makes things faster. That's where you should use typing. And for everything else, it's fine if you don't use it and don't need it there. So it was termed coined in 2006. And there's a blog post, just look for what is gradle typing and you'll find it. And it was the basis for PEP484 type annotations in Python. It's really the best mix of static and dynamic typing. You can use dynamic typing for the ease of development and optional static typing for safety, speed and documentation. And you should really only use static types where they help. Right? That's important. And so don't type all your things. Right? There are limits to what you should use types in your code for. And this is especially important when it comes to Python because what Python gives you is types in Python. And it uses them to accelerate your code. So Python is a pragmatic programming language. It's gradle typing for Python and it's an optimizing compiler. So it takes your Python code or you type annotated Python code and translates it to C and that compiles into a native extension module for Python that you can just import like any other binary module. It's production proven. It's widely used as you've seen. There are lots of tools using it. And it's really all about getting things done in the same way that Python is. It helps you keep your focus on functionality, removes the boilerplate for writing native modules. It allows you to move freely between Python and C. You'll see that in a minute. So it gives you a language that is as Pythonic as you want and as low level as you need it. Okay. So it's a demo. I'll make that bigger. Okay. Who's never used Jupyter Notebook? Again, the kid over there. See? Teach him. So yeah. So pretty much almost everyone else did and it's a very nice and interactive way to use Python to jump around in your code to show stuff. So what I have here is a Jupyter Notebook. And as you probably also know, Jupyter supports lots of different languages. I remember that it was 14 different languages like years ago. It's probably 20, 30 something in that order now. And it supports Python. So it's included in there. If you just say load XSython and have it installed, obviously, that teaches it how to run Python code, how to compile it in a Jupyter cell. So I'll just show you what I'm using here. Python 37. Fairly old version. But anyway, Python 3, fairly new version. NumPy GCC7. Oh, yeah. And that's something I should mention. Since Python compiles your translator code to see what you need now is a C compiler. Python is nice and easy. You write your Python code to run Python, run my code, and it runs it. In Scython, there's a compilation step before that. Because in the end, you get a natively compiled module, a native code, and that requires a C compiler. But that's it. That's all you pay. Okay. So very simple example. I'm going to take the Python module, import the sign function, and calculate sign of 5. Okay. It's a bit boring. I can do the same thing in Scython. And one difference you notice here, so the first thing is this is how I declare a Scython cell. I just tell Jupyter, you know, this cell is no longer running in Python. It needs to be passed through Scython. This compiled module for me. And then what Jupyter does in the back is it imports that module and executes it along the way. And that's also the reason why there's a print statement down here, rather than just the sign of 5 as you would know from Jupyter. So in Jupyter, cells, in Python cells, the last result of an expression kind of falls out of the notebook and gets displayed. And in a Scython cell, since the cell is compiled to native code, that doesn't work anymore. So Jupyter can't look into native code. It just executes some module and does nothing for no other good. So I have to explicitly say print. But that's fine, I think. Okay. It gives the same result. Now, as I said, since Scython compiles a C code here, that translates to C code, I can start using C functions now, or C functionality. And what I do here is I take the sign function from the libc math header. So this is the C sign function. And for now, I'll just assign it to a variable. And then I can call that from Jupyter. Who's surprised? Anyone? No one? A couple of people over there? What's surprising here? Yeah, no compilation. Nothing you can see. So the Scython annotation up here makes it a Scython cell. It compiles it for me. But the funny thing is, just by assigning the C sign function here, I can actually call it from Python. So now Python knows how to call a C function. That's interesting. The reason for that is that Scython is a type language, since C is a type language. So the sign function is something that knows that it's a function, a C function that takes a double floating point value in and outputs a double, or returns a double floating point value. So it knows the input and output values. And that allows Scython to directly wrap this function as a Python object. And when I assign it to global Python name here, it does it for me. Because that's an obvious assignment. I assign a C function to a Python variable. What else should it do than wrap it for me? Obvious, right? And does it. So this is kind of the quickest way to wrap a simple C function for Python. Here's the same thing spelled out. So again, asking Scython to learn what the C math header is. And then I write a Python function. You can see the extension here allows me, so Scython allows me to declare C argument types as I would in a C function. And then I can just call the sign function. And this is me manually wrapping this C function call here in a Python function. Which essentially does the same as above here. So it's not automatic. So when I run this and down here, now I can call the Python function and it calls the C function internally. Okay. That's still a bit boring because as you've seen here, Scython can also do this happily on its own. It doesn't need my code writing for this. So when it becomes more interesting to do this, that's when you can move more functionality into the Scython layer. I can easily now call this C sign function up there that I just wrote with x squared as input. But that would calculate the x square in Python. Then take the result of that, pass it into a C function, calculate that in C space and return it for me and return that as a Python object again. If I do the squaring also in Scython, then that gets translated into C code as well. And now the whole expression, the whole function body itself runs in C at C speed rather than me doing something in Python then in C and then passing it back to Python. So the nice thing about Scython is that it allows me to move functionality freely between Python space, Scython generated C space and at the end also the low level C written code. And it's up to me as a developer to decide where I want to put my functionality, how I want to implement this. In some cases I have an existing library, a C library for example that I want to call. I can just do that. I can declare it in Scython, call it from Scython code, do some more stuff in Scython space to keep it at the below the Python level. And when I'm done with it, then I return to Python. So I have all three levels available in one programming language and that gives me a lot of freedom for moving around optimizing code in these three levels of performance also. So here's an example I'll go through somewhat quickly. It's about speeding up NumPy code. Any questions so far on what I presented? Anything you would like to know or clarify? I didn't understand. I'm not always as clear as I want to be. I hope it was. Apparently it was. Thanks. Next example, NumPy. So the idea is we're going to calculate the average tax rate for Germany. I only have the numbers from 2016. So at the time there were 44 million people working. So paying income tax, let's put that way. And the average at the time was 3703 euros per month. So the average income was at the time 44,000 euros. That was 2016, it probably didn't change that much. It's still an average. So problem is I then tried to find official data for this and I couldn't find anything really probably due to data protection or something. Like there are probably a couple of people who aren't that much that you would identify them by their income. So there's no official data for this and what I did instead was I just took a look, normal distribution, fit it to the one data point that I have, made it kind of look good like this. And it's kind of not too far from what you would expect for an income distribution. So just assume this is actual data. Okay. So let's calculate everyone's taxes. When you look up the tax rate calculation in Wikipedia, what it gives you is a beautiful Excel formula. There's actually German Excel, right? So it says WIN and that's if. And then it gives you lots of formulas. So there are apparently different income ranges that change against each other and that have different formulas for them to calculate the tax rate. So the income tax for that income range. Okay. So you can easily translate that into Python and it becomes this, which I think is a bit more readable, especially because it allows you to ignore the right sides of the formula here and just go, ah, look at the different income ranges and then it does something. Just ignore the formulas. So this is how the income tax is calculated. And then in order to get the average tax rate, we take the sum of the taxes divided by the sum of the incomes. Okay. So I'll set my, you know, kind of a fake income list. I think I cut it down a bit. I'm not taking the 44 million, which is a fourth or something. And now, yeah, whenever you do a timings, don't forget to disable the energy management on your laptop because otherwise you get funny numbers. Okay. So it's going to take a while. And I can already show this a bit. So what we're going to do here is since we're optimizing this code, I have a little function that just collects different timings from different implementations. And so I remember the timing here and then we'll use this function to show the differences. So it took three seconds to calculate this whole thing in Python. Okay. So it's 3.2. Okay. So that's our baseline. Python has factor one. Now you can implement the same thing in NumPy and that gives you something like this. So we're slicing the income array, then doing some computations on them, and then build the sums on it. Okay. So it's kind of heavy NumPy code, but that's how you can do it in NumPy. And now we can calculate the whole thing in NumPy. It should be faster. So we're down to 62.1. And that is 50 times faster than Python. Okay. There's a different way to do this. So you can take the Python function of written and wrap it for NumPy so that it can apply it through the whole array, one item at a time. And then you can do the same formula, sum up the taxes and the incomes and divide one by the other. That again is going to take a while. That is slower than the slicing version, but it should still be faster than the Python version we had. And it's faster by quite a bit. So that gives us 49 milliseconds, and that is four times faster than Python still, but it's completely dwarfed by the NumPy version. Okay. Enter siphon. Here's a plain copy from the Python code I had above. It's doing the same thing. And now what I change is I compile the whole thing in siphon. And then do the calculation again. I shouldn't show this yet. Okay. Yeah. As you can see, it takes a while to calculate this. Actually, it already takes a while to... Yeah, it's running here. Okay. So it takes 2.74 seconds. Yep. And as you can see, that is 70% faster than the Python version, which is nice, right? I mean, all we really added is this little line up here. And so how much is that, so that's 8 characters for 17% speed up? That's certainly areas where that's acceptable, right? Okay. But we can certainly do better. Now, where siphon shines is static typing, and I told you about gradual typing. So what we're going to do now here is we'll gradually start typing the siphon code. And I'm going to use the PEP for notation for it. So the income that the text function here is calculating is definitely some double, right? C double value. It's perfectly fine to represent that because some thought in part in doubles is just fine for these calculations. Okay. And returns. Well, it does some calculation afterwards, and so it definitely also returns a double safely. Now next thing I notice is this function is only used internally inside of my module. I'm not exposing it anywhere. I'm really just using it down here. So what I can do is I can convert it to a C function, which is faster to call than a Python function because it has different call semantics. In C code, you can call C functions pretty much without overhead. In Python code, calling a Python function is a lot of overhead. Also in siphon code, calling a Python function is a lot of overhead because they are called with argument tuples, maybe keyboard arguments even. And so creating these objects, even just for calling the function, takes a lot of time. In C, it just passed by a stack or a register, so that's very, very fast as fast as it gets. So one thing I can do is I can declare this function as a C function. And now when I compile this, then siphon compiles it for me and I can time it again. And we're down to 205 milliseconds. So when I compare this to what we had before, that is now quite a bit faster than the Python version by 15 times. And the compiled version that we had before is still 13 times slower. So this was a speedup by 13 times compared to the version we had before. Why is that? Because one thing is the call overhead that I completely removed. So this is, you know, that's kind of the inner function of my loop when I'm doing a lot of work, right? And the call overhead for that function was removed, so that calling that function, going into that function is basically low time now. But the second thing happened. By typing the input and output arguments, siphon understood this function better and managed to generate plain C code for it because it knew now that income, you know, that's a C double. It didn't have to do any object comparisons anymore. It can calculate this whole thing in C space now. So by typing some variables, by typing some arguments in there, siphon takes decisions about my code and adapts it to the variable types. And you can see that with a functionality called annotate. So siphon-a gives me HTML output. And in here, it replicates, so it outputs an HTML snippet into my notebook that replicates my code. And in here, sorry, in here you can see, when I click on one line, this is just plain C code, right? Income greater than something. Same thing here. Take it formula, go to something, right? And in Python, that would look a lot more involved. It would do lots of Python C API operations, lots of object operations along the way. And this now really generated plain C code. And it gives me a speed of 13 times. Okay. So I would then continue this. There's a lot more I can do at this front. And I think the final speed up in the end that I usually have is, do I have it here? Yeah. So in the end, I usually manage to get it down to something like 11 milliseconds from the current 200. Right? So that is another factor of almost 20. Okay? I'm going to switch to a different example here. So any questions regarding this topic so far? Okay. Yeah. I think you were first over there. Is the decorator C func same as C def? Yes. Well, C func is actually explicitly saying this is a C function, which is the same as declaring as a C def function. Right? So there's more meanings than that, but this is more specific. For the others, the question was, there's a different syntax in Scythian, which is not Python compatible. And there's a special file type also, which is PIX instead of PY, which allows you to use the syntax. And I've only been using Python syntax here. You can do the same thing in a special syntax, which is more relevant when you start talking to external C code, because that is a ground where you cannot cover the same thing with Python compatible functionality anymore. And that's where we use the second syntax. But as you can see, you can get very far by just adding decorators, Python type annotations, and so on and so forth. So that's me making your code a little more C-ish than it used to be. That also helps in a lot of cases. Okay. Question over there? So if you were to actually have that in a code base, would you put this part of the code in a PYX file, or would you have it in a Python file and only annotate? Do you siphonize the file and compile it, then you ship it, or do you ship it, and then if you have siphon on, you go and you have that in a music file? So question was, this is in a notebook, right, where it's nice you can compile a cell or use a Python cell, like, mix them as you wish. How is that when you have code in the module, in the Python module? How do you optimize that? So siphon compiles one module at a time, meaning you would normally pass the whole module through siphon and compile it. It rarely hurts to also speed up a couple of other places in your code without actually annotating them, so if you just want to optimize one function, you can still compile the whole module, right? This might not even be a bad idea. Different choices that you have. One is Python often uses this concept of an accelerator module where you have a Python implementation of something and then you have an external native implementation of it, which you import in the end and if the import works, then you replace some function in the Python module with some faster function, something like this, right? And you can do the same thing in siphon, right? You can externalize one function into a separate module, compile it in siphon or not, and then import it from there, right? And if it's compiled, then it's probably faster. If it's not compiled, then it's as fast as Python will run it. Or as PyPy will run it in that case, for example, right? So that is fairly nice. This syntax actually allows you to optionally compile code, right? You can have it as fast as PyPy can run it and in siphon you can compile it and get it as fast as siphon can run native code. Also nice. Okay. I'm currently looking for a little example that I had, those related. Yeah, okay. I mean, that answers the question, right? Okay. Okay. Anything else? Then the next example that I have is C++. Just a very quick example. So as I said, siphon generates C code by default, but it can also generate the C code into the C++ module and that allows you to use C++ code along the way from your siphon code. And that is very nice from siphon because C++ is an object-oriented language, Python is an object-oriented language, so you can mix two object-oriented languages in the same siphon module and they look totally like Python when you use them. Even C++ looks a lot like Python when you use it from siphon, but it gives you a nice additional standard library, like fast data types, for example, fast container data types, 3D STL or anything you might be able to write in C++ yourself and want to wrap it for siphon. So that is a very nice addition to siphon, actually. Here is a very quick example. One thing you have to change is you have to tell Siphon explicitly that the language it should use is C++. There is more than one way to do this, but this is a common one, especially from a notebook. And then that enables access to the C++ STL declarations that we already ship in Siphon because they are so commonly used. But then you can use this syntax here to say I have variable v, that is C++ vector of ints, and then I just push a value in there and return the vector. What does this do? Returning a C++ vector from a Python function. What do you think happens? It will transfer it to a list. It will copy it into a list because the obvious representation of a C++ vector in Python space would be something list-like. You would use a Python list for that. So simply returning something that Siphon understands as list-like will copy it into a list, Python list automatically. So it will create Python for you, copy all the values from the C++ vector and return that. Okay? All automatic. If you don't want that, you can implement your own little thing in whatever way. You can use a list comprehension to get something out of the C++ vector. Or you can return a generator expression that uses the C++ vector. You can do all these kind of real things by mixing Python features with C++ features. But this is the most simple and straightforward way to do it. So when I call this function, then I get turned back, which is very surprising. This is a very common way to wrap C++ objects for Python. I can use a so-called Cdef class. So Cdef class is an extension type in Python. So it's a native, low-level implemented class, like a Python class, but implemented in C. And in Scythian, these extension types allow me to directly use C++ objects as instance attributes, that's an instance attribute. And I'm using a vector of int as values attribute. And then the lifetime of this C++ object is tied to the lifetime of a Python object. So it's automatically created when I create a Python object. It's automatically de-allocated for me when my Python object goes on a scope. So I don't have to care about any memory management. It's all automatic in this case. Very nice. And so this is just a very tiny example that uses an integer, this like object, wrapper object in Python. I can add values to it, which is the pushback C++ function to push it into the C++ vector. And as you can see from the usage here, that totally looks like I'm using Python code. I'm calling method on something, I don't care if it's a C++ object or a Python object, just call it. I also like the wrapper implementation here. What do you think this does? It runs wrapper on the list. So it does the same thing as we had before in the function. It creates a Python list from the values. So this is actually very expensive. But hey, how often do I actually need wrapper? So what it does is it copies all the values in the Python list, uses wrapper on that, and then gives the string back to Python. If it's only something I occasionally use, why not? It doesn't cost much. Okay. So much for C++. Any more questions on that field? Last time you checked, we didn't have bit vector. You mean in the C++ declarations. The membership might be. Please provide a pull request and we'll happily add it. Anything else? A gil handling. I actually have another example for that as the next and last example that I'm showing. So here's an example of wrapper in external library. That's the last use case that I wanted to show here. And so far we have seen this C port, something, it's a bit of magic. And this is unpacking the magic. What we ship is site declarations of most of the C header set of the C++ header set of the STL. Well, C++ is huge, but much of the STL. And this is how it's spelled out in the end. This is special syntax for that. So what I want to do now is I write a little wrapper for the Lua runtime. So I'll execute Lua code from Python. And for this, I just need to declare a couple of functions that the Lua API defines. And then up here, declare how to link against this. This is a bit of a heavy way to do this writing notebook. There are more nicer ways to do this from the setup file, but this works for now. And then once these are declared, I can use them in my code. So if you find a Python function, it takes a code string, converts it to a native needs to or accepts unicode and byte strings. So you can just use a normal Python string from Python 3 and pass it in there. Then creates a new Lua state as a Lua runtime representation. If that fails, then raises memory error. That is also fairly nice. Do you know what you would have to do in C in order to do this? Happily, there's a C API function for it. But just being able to say, you know, do some C stuff if that fails, if I get an out pointer from some memory or an education back, raise memory error? That is nice. That is what you want in your code. So that is totally the Python way of doing it. So then I have a try and accept. Sorry, try finally. And at the end of the finally, I delete the Lua runtime again. So if anything goes wrong along the way, then I make sure I clean up the memory. Because C requires manual memory management. I am responsible for cleaning everything up that goes wrong. And in Python, I can use try finally and it is just going to work. So here is Lua function to load the code into Lua, execute it from there, and then a tiny bit of return value adaptation to return whatever number Lua wants to return here. Okay. I just quickly run this and notice that I didn't run the beginning. And then run it again. And it fails because I didn't set up my Lua properly. Just believe me that this works. Okay. It is not the first time I show an example. It is just the first time it fails so heavily. I am not going to fix it right now in the last two minutes. Okay. One title of that talk was siphon 3. So those of you who already use siphon will probably know that siphon is always this 0. something version kind of thing. So 0.29 currently. So what is the siphon 3? Where does that come from? So the last version that we have is 0.29. So if you push the dots one digit to the right, then it becomes 3. The next version is 3.0. Okay. So that is kind of frightening. You jump from 0. something to 3.0. What is special about it? First thing is siphon 3 is Python. Now you are going to say, but it was always Python. What is special about it? Well, it is Python 3. And some of you might say, I could always put my Python 3 code in there. I just had to say language level equals 3 and then compiled it. Well, it is Python 3 by default. So the language level is 3 by default. So there are a couple of things that we changed now in siphon 3. It is Python 3 by default. We are going to change mostly the standard configuration of the compiler to make it more modern, to move stuff that is not appropriate anymore these days. And we are going to adapt it to today's Python 3 world. So these are the main changes in there. We are going to adapt to several peps. That came along the way and said Python 3 by default, yes. You can look up the Python 3 milestone. There is still quite a bit to do along the way and we will try to get there. Okay. That is it for my site. Thanks for listening. Thank you.
to show you a lot of examples of site and how to code into Python, is that a topic for you? Raise so we can give you the actual opinions based on all this, and it's all based on Hadoop processing here is that these are the tools that they use for calculating this image out of the raw data. for? It's used for integrating native code with Python. It's used to speed up Python code in C typing for Python and it's an optimizing compiler. So it takes your Python code or you how to run Python code, how to compile it in a C code here, that translates to C code, I can through somewhat quickly. It's about speeding up Python C API operations, lots of object specific. For the others, the question was, compiled, then it's as fast as Python will run is a very common way to wrap C++ objects for right, then it becomes 3. The next version is Well, it is Python 3 by default. So the language to make it more modern, to move stuff that is not