Deep Postgres Extensions in Rust
Davis, Jeff
Postgres relies heavily on an extension ecosystem, but that is almost 100% dependent on C; which cuts out developers, libraries, and ideas from the world of Postgres. postgres-extension.rs changes that by supporting development of extensions in Rust. Rust is a memory-safe language that integrates nicely in any environment, has powerful libraries, a vibrant ecosystem, and a prolific developer community. Rust is a unique language because it supports high-level features but all the magic happens at compile-time, and the resulting code is not dependent on an intrusive or bulky runtime. That makes it ideal for integrating with postgres, which has a lot of its own runtime, like memory contexts and signal handlers. postgres-extension.rs offers this integration, allowing the development of extensions in rust, even if deeply-integrated into the postgres internals, and helping handle tricky issues like error handling. This is done through a collection of Rust function declarations, macros, and utility functions that allow rust code to call into postgres, and safely handle resulting errors.
 Okay, hi everyone, my name is Jeff. Today I'm going to be talking about writing deep post-scrass extensions, real post-scrass extensions that do interesting stuff beyond just maybe a UDF, something like that, in a language called Rust. Probably a lot of the people in the area have heard about the language Rust. Is there anybody in here that hasn't heard of Rust or don't really know anything about it? Okay, so I'll kind of briefly go over that. I won't be talking as much about the language itself. The code samples should be legible for the most part they're not, except for the last one, but that's for a different reason. I'll get into that. So let me just talk about the motivation here. So post-scrass is more and more reliant on the ecosystem of extensions. There are important extensions out there that do important things for a lot of post-scrass users and customers. And that's a good thing. Having an extensible database system provides a lot of flexibility in terms of things like not needing to wait for release cycles of the main post-scrass product, not needing to meet some of the standards in terms of maybe you depend on libraries that might have an unacceptable license for the post-scrass project, or maybe you're using proprietary code or something like that. And for those reasons, maybe it can't make it into the main post-scrass SQL code base. And it shouldn't. And it's great that we have the flexibility to use those maybe a proprietary library or a GPL library or something that would never be accepted into the main code base, but still provides really important functionality. Post-gis is a good example of that. Very important spatial extension for post-scrass. But it depends on libraries that have different licenses. Post-gis itself has a different license. So anyway, this is all good stuff. And I'd like to expand this beyond C. I'd like to expand this capability of writing extensions to a new set of developers that for whatever reason may not be able to or not want to use C as a programming language. And Rust is offering another option. Right now, everything needs to be C. So this going from supporting one language to supporting two. Hopefully, that's a big step. There aren't really many options out there, aside from C. And I'd like to create a second one. So why Rust? There's a lot of marketing out there. This isn't an advocacy talk about Rust, but it's got some nice features. It's got a nice community. It provides some improved memory safety features above what C can offer. It's used a modern type system that helps catch a lot of errors. But it's also nice and low level. It's suitable for, at least, ostensibly suitable for low level work. So it's not kind of adding code behind your back. It's not got a large runtime or injecting extra fields and destructs or things like that. It really just allows you to work with the memory representation that you already have or really want to use. And then you can use the other language features on top of that representation. It doesn't impose a lot of that on you. And it doesn't kind of inject code or use aspects of the runtime that might conflict with what Postgres wants to do. So Rust, if we're going to support a second language, we really have very few options that actually meet these requirements. So I mean, there's C. And then after that, the options get a little bit thin. C++, Rust might be a couple of obvious ones. I think Rust is different enough. It's kind of a more interesting exercise and might open us up a little bit more to a wider group of developers than trying to say work with C++. It's also got an awesome ecosystem and awesome community. And I'd like to kind of bring all those people into the idea that, hey, you can write a really interesting extension that really expands the functionality of Postgres in interesting ways in their language of choice. Right now, the Postgres world is C. I mean, that's just the way it is. There are some really nice PLs out there. And I've used those to great effect. They're really convenient to use. But they're fairly constrained in that it's pretty much scalar functions, set returning functions. You can do a few things like that. There might be some efficiency costs going in and out of that. But I mean, really, if you want to do things like this, foreign data wrappers, custom data types, index support functions, background workers, UDFs calling lots of internal functions all over the place that might be not really intended to be called from a UDF, then you need to use C. That's the current state of the world. And I'm trying to change that so you can do all of these things with an additional language. So what about procedural languages? I mentioned this briefly before. But procedural language, the infrastructure here is great. You have the ability to write, say, in Perl or Python or JavaScript or PLPG SQL, you have this ability to write functions and even execute queries using the server programming interface. And it's, like I said, a great API. But I'd actually like to go further than this. So this is not like a PL Rust implementation. I'm trying to take it a little bit further than that and make it really kind of co-equal with C in terms of capability. So the other thing, I'd like to see what Rust can do. I mean, it's kind of an exciting language. But that also means there's a lot of marketing that goes overboard. They kind of not intentionally from the Rust developers, but some of the community might get a little bit over excited and maybe overstate some claims or overstate the ease with which those claims can be realized. And so let's really apply it to an interesting project. I mean, Postgres is a big piece of software with a lot of things going on. I think I have another. Oh, yeah. This is a slide. It's got things like memory contexts, doing the allocations. It's dealing lots of global variables, set jump and long jump based error handling. These are non-trivial things. In other words, we're not just kind of plugging it into our Hello World C app and saying, hey, Rust is great. We're plugging it into a real thing like Postgres. And we're going to test some of these ideas and how far they can really go and what we can really accomplish. So what is Postgres extension.rs? This is a crate that I wrote. It's available. You can use it as a dependency today. So if you're writing your own Rust crate that's a library, I've got some examples in GitHub as well. You can, extensions can use this as a dependency using the crate ecosystem, which is the package manager comparable to the package managers in other languages. So basically this dependency that you can pull in called Postgres extension, it's a collection of functions, macros, the kinds of things that you need. Basically speaking, it's like the C header files. So just like if you're developing a C extension, you need to have the Postgres server header files available to develop that. Also with Rust, you would use this. And this is essentially providing definitions that you need to get stuff done. Now you can imagine there's an awful lot of C header files in the server. So this only implements some of these things, I tried to really take on the hardest problems. And right now the boilerplate stuff of just copying struct definitions around, that it's not capturing all the C headers out there. But I tried to take on some of the more challenging problems and see how far Rust could go with that. So I'll talk about those challenging problems and show some demos. Just to clarify, so Rust already has a great client driver. This is not like for developing an application in Rust that talks to Postgres with the client. This is really about actually modifying the server with an extension. So I just like to clarify that. There's already a great Rust Postgres driver out there. If you have an application, you'd like it to talk to Postgres, that's got a great answer. So let me talk a little bit about the features of what I think I actually accomplished with Postgres extension.rs, the crate. So first of all, you can use these C structures directly. So you might, using other languages, if you're using Python or something like that, and you need to access some of the internal structures in Postgres, you might imagine that you would write some kind of translation layer that would take the memory, copy it around, move it in different ways, and then eventually you have something that looks like Python that you can work with, and then you operate on that. This, I don't want to do that. For efficiency reasons and other reasons, you just don't want to be copying and translating all the time. So this can actually work directly on those C structures. If you get a parameter that's a Postgres structure, then you can just immediately use that as though you're writing C code or comparably to writing C code, you're operating directly on that representation. You're not translating, you're not copying. So that's a big point to make because we don't want to discourage people from writing these extensions because they say, oh, well, we're copying it again and again, and how do we, maybe they need to modify that structure, and then that means they need to kind of translate it back to update the changes. It might cost in terms of efficiency. So that's an important point there. Another thing this does is it actually uses Palik and P free for the memory allocations. So if you are using Rust standard library functions of vector strings, these kinds of things, pushing and popping and all that stuff, when that's doing allocations and freeze, those are going through Palik and P free. So that means it's happening in the memory context, just like in Postgres when you're doing these kinds of P string, dupe, or some kind of string function or something like that, then when Postgres is doing those internal allocations, it's using the current memory context, and so will your Rust code. So it's a Postgres kind of feel to it. And just like with C code, if you're using memory context, you have to be careful when you're resetting them, deleting them, that kind of thing because obviously that can leave Rust with dangling references if you cut the memory out from under it. And then this is actually what I spent an awful lot of the time on is the error handling, and this is what has made it difficult for other projects in the past like related to C++ to really be successful in Postgres is handling these error handling situations. So Postgres uses set jump and long jump. For those who don't know what those do, it's kind of about what it sounds like. It jumps across many stack frames. So if you have deeply nested function calls, when Postgres encounters an error rather than kind of returning to each one, it uses this C standard library feature. It just jumps over all these function calls back to a place where it can actually recover from the error. Kind of like an exception, but C doesn't have exceptions, they have set jump long jump, which is a very kind of wild west way of doing this, but it's been very useful in Postgres for getting out of a lot of work and kind of aborting a transaction, it works quite well. So to handle this in Rust, it was a fair amount of effort to one, actually make it work, and then two, make it work somewhat ergonomically with macros that can make these things nicer to program with. And then, so anyway, this ensures that Rust, the structures are properly called, so it's not just kind of jumping over stack frames that Rust expects to be properly unwound. Okay, so I'd like to, just one second here, let me pull a terminal over there. Okay. Oops. Okay. Let me do it. Okay, better. Okay. So this is basically just kind of showing off the error handling. So those who have written Postgres extensions and UDFs before will recognize, you know, PG module magic, that's kind of the Rust way of doing a macro with this bang at the end. I'm pulling in some, you know, C functions that I need access to. I'm pulling in some of the utilities in the Postgres extension crate. So you know, and then I'm doing something unsafe. I'm, you know, going to divide by zero. So you know, that's something that might, you know, happen in, you know, whatever Rust code you're writing, it might generate an error that way. So this div zero, basically it's going to, you know, come into this function here. So basically we're going to see how like an internal error generated can end up, you know, going back out and being properly handled. The next one is a case of you're writing a function and you actually decide on your own. You want to make your own e-report. So that's what this next function is about. And the third is you do something in Rust and then somewhere Rust has, I try not to make this too confusing because Postgres has an ocean of panic and Rust has an ocean of panic. So when you're reading this here, this is basically just using the Rust version of panic. So somewhere in your code, what that means is that it's going to kind of unwind the stack and then get the error and then turn that into a Postgres error. So it's like these three different cases. The first being that you call a Postgres function that throws an error. The second is that you want to throw an error yourself and the third is that, you know, some Rust function you're calling wants to throw an error. And some going to show how each of those are handled. So. So you'll see this is in the examples directory. So this is actually a separate crate that depends on Postgres extension. So it's an example of using the Postgres extension to create these UDS. So build it. We copy the generated shared object file. So Rust is capable of generating shared object files, which are the way that Postgres loads an extension. So we're going to copy that. And then we're going to fire up Postgres. Okay. So. So if we select UDF panic. So what happened is the function, we just take a quick look back here. The function panic. This is a Rust panic. So this is a case where, you know, you have this maybe deep link nest address code throws a panic. And that's again not a Postgres panic. This causes stack unwind and then is translated into a Postgres error. Then the next one. Here is a time when you want to, within your Rust code, generate a nicely formatted e-report. You'd like to provide, you know, detail and, you know, all kinds of things like that. And it takes that e-report in Postgres, or in, sorry, the e-report from Rust. It unwinds the stack and then translates it into a Postgres error. And then the last one, div zero. This is generated inside of Postgres. You call this function, it ends up throwing divide by zero inside of the division function, and it passes that unmodified backup. It catches it in Rust, unwinds the stack correctly, and then passes it unmodified backup into Postgres. Okay. So that's demo number one. Do I have a question over here? An error context, are you talking about like line numbers and stuff? Oh, so like the file and are you talking about the, you're not talking about. Yeah, I think I just didn't do that. But you know, basically, you know, yeah, I don't think there's any reason it can't be done. What are those exclamation marks in this R? Oh, so in Rust, they have a hard distinction between function calls and macros. So every macro ends in exclamation point, and function calls do not. Yeah. So it's just, you know, that part of the language. Okay. So the next demo I have is a little bit more interesting. So let's first look at it. So this is UDFSPI, and it does some SPI stuff. So you know, those familiar with kind of higher level languages might be happy to see that it's a little bit easier than using SPI might be in ordinary C code. So I've got, you know, I've got this connection being made. I execute that on the connection object, and I have some iterators there that, you know, can iterate through the result set tuples, and then also through the attributes in the tuple, in each tuple. So we got this kind of nested for loops, but it's, you know, only, you know, fewer lines of code than it would be in C. So one thing I'd like to point out here is that, you know, I'm doing some things differently. So the SPI API in Postgres is quite a bit uglier, you know, who's, who here has used the C API for SPI? Okay. So there's a couple things you might recall. One of them is that SPI Connect doesn't return anything, so you might be wondering what that thing is. The other thing is SPI does kind of return things from an, you know, query execution, but they end up in these global variables, which you have to kind of copy over, and you have to be sure not to execute another SPI function because it will overwrite those global variables without freeing whatever resources they were pointing to. So you better get the reference to the tuple table while you can. And it's, you know, potentially leads to, you know, it's error prone, let's say. And there have been bugs in Postgres, not even in Postgres extensions, but there have actually been bugs in core Postgres because of that API. And there is a comment pointing out that it is perhaps not the best API, but we're stuck with it. In Rust here, so I, you know, kind of solved some of these problems, I believe. I think I made a nicer, less error prone API. So one of the things I'm doing here is I'm returning this object, this SPI object here, if I'll put the cursor on that variable. And then that is a zero-sized value. So there's no allocation there. It's really just compile time thing. And the reason I do this is because for one thing, that way you can't really call the execute method without having connected first. So that's a potential, you know, those of who have used SPI have probably all made that mistake. And so it's, you know, just nicely there. It also allows, you know, the destructor to call SPI finish. And so it kind of helps you make this symmetric, you know, use the API in this symmetric way consistently. And so it kind of offers these benefits, you know, but it's actually zero-sized type. So that's not doing any allocation there, not even on the stack. The result set here, this is not zero-sized, but it's the same size as if you were to kind of copy those global variables locally, which you really should anyway. So, you know, just, it's, I think, you know, two or three words or whatever to make a copy of those global variables before you lose them from the next call. Question over here? And so that, I also consider zero cost, you know, compared to using the API the way you should, because you really should make copies of those global variables unless, you know, I mean, it's just kind of asking for trouble. It's not really expensive, a couple stack variables, you know, maybe in registers. So I don't really consider that to be a cost. You could get around it if you really wanted to, by the way, it's not like Rust can't support global variables, you know, but you'd almost never want to do that. The iterators, they might have some, you know, kind of transient data costs, you know, maybe on the stack or in registers or something, but it's pretty lightweight. It's basically, you know, just all using the same data and references where appropriate it's not doing a bunch of data copying, it's not doing real allocations. So that's also, you know, zero cost. So you're basically getting, you know, for paying nothing, you're getting this, you know, substantially better API that prevents a lot of mistakes, including, you know, making sure to finish the, you know, SPI finish at the end. So maybe I'll actually run this example. If somebody has a question, please jump in. The destructor for this zero size SPI object will SPI finish, which is the disconnect, right? So that's the other side of the connect. Okay, so let's build it. There we go, build. And then copy, copy the right one, PDF SPI. Okay. And then now let's go back to our server because the way Postgres works, we just need to make a new connection and we'll get the new code. Create or replace. Okay, so first I'll just select star from who and we see we have this wonderful table here, very sophisticated. And then if we also do UDF SPI, which is running the same query, we get these notes. Notices printed out. Now it'll kind of make sense when we look back at the code, you'll see how these notices are generated. It's just returning a one always and it kind of builds up those strings for each tuple. It gets out of SPI. So we can look back at the code here. And you'll see here this is where it's, you know, doing the notices, building up the strings. It's kind of pushing, you know, pushes kind of way to build up strings. These are formatting codes here. The simplest one is just curly brace and curly brace, but there might be formatting options inside of those. So it's like a, it's a formatting string. And then here it's doing the return, which we saw at the end. So that's an SPI example. I think the interesting part there that is different is just that I feel like the API is safer and nicer without imposing a significant cost. David? I believe that is immutable. So I mean, I don't, what would it mean to mutate a tuple that you get back from SPI? I mean, it wouldn't go back to the table, right? I assume that you're changing the, you're mutating the, where, I mean, so this is just kind of a read-only thing. Is there a reason, you know, so, do you just mean that that's like kind of rebound each time through the loop? Yeah. Okay. So Rust isn't really considered mutating it because it's a separate binding each time. It's almost as if you had had like the declared value in this block. So it's just, it's not actually mutating the value. Yeah. So we're a destructor on the couple, do you want to be getting disrupted at the end of the loop? Yes, it would be. So I think in this case it's kind of a reference or immutable. It's the, the actual memory of the tuple is, is, it's just a reference to that. So I mean, there's not much destruction to do. SPI will clean up the result set afterward. The result set does have a destructor which frees the result set. And then if you SPI finish it, frees all the results that's generated in that session. Andres? Can you catch errors to throw in an SPI and then continue it in some way? If so, what exactly, because you need to do weird stuff to actually set up SPI contacts and stuff? So I mean, you have the same problems there you do in C, right? I mean, kind of doing a try, catch around SPI is a little bit questionable, you know, more than a little bit questionable in C as well. So I, I didn't magically solve that just by using Rust. I mean, that's, you solved that in your records, I'm not that way. So I mean that, I mean, in, in general, you've got to be really careful with the try, catch around SPI because it might have changed tuples in, you know, and actually there's NBCC information there. So if then you kind of catch that and end up an error and then end up committing the transaction, then it's possible you would have like, you know, things that you inserted that wrong. Yeah, so I'm trying not to do that level of, of magic here because, you know, it would be something that the developer would really want control over. It's, I want this to be kind of a, at the same level of abstraction as C with perhaps some safer interfaces, but I'm, I'm trying not to do kind of things behind the scenes like save points. You know, a C programmer would have to do their own save points and then so would a Rust programmer. Well, other designers have like much nicer wrapper like Pio, Perl, the public pretty nice wrapper-sized, right? Yeah, I mean, I think there would be, you know, room to do that as like an optional extra layer, but just that, you know, my primary goal for this project is not to be at a higher level unless it's kind of a no-brainer sort of like these SPI iterators and stuff would just make things easier without really changing the level of control you have. So yeah, basically the, the scope of this is not to make it feel like, you know, programming Python or something like that or with kind of magic happening behind the scenes. I just want it to be like, you know, at that same low level that a C programmer would expect. So I'm not sure I understood the question about the error in SPI correctly, but do you have an unwrap on the execute that returns something? Yeah, so the unwrap, this is a Rust thing. It's basically if you have something that can return a result or an option or something like that where, you know, maybe it returns a result, maybe it returns nothing or maybe it returns an error, these, these options. So you can handle these, whoa, I hope this video comes in. Okay, so the unwrap is basically a way to kind of short circuit that and say, I, I know there's a value there, just give it to me and then if there isn't a value there, then panic. So if that panic were to happen, we would end up seeing that as a post-gress error. A, a Rust panic happened. So I, during this whole talk, anytime I say panic, it's not the post-gress panic. It's a Rust panic. So that's what would happen in that case. So it would be handled reasonably well. Here I'm really expecting a result. I could have kind of handled this and wrote in more Rust code to make it more robust, but I just, I just didn't do that for clarity. And that's a pretty common pattern. If you know something's there, then you just unwrap and then, you know, if you're wrong later you're going to get a panic and know where it's from. And then I'm running out of time a little bit here, but I have one more demo that I'd like to show off. I'll show it off a little bit quick. Okay. So this is, let me actually, let me build it first and kind of show it off first. And then we can look at the code. This is a background worker. So here I already set up the configuration file to auto-load this, this background worker. And so it listens on 8080. And then if we use netcat, it just reading lines off the network and executing them as SPI and returning the result. So when I, so, lots of things. So yeah. So if we just do like select one, boom, we get a one. Select star from foo, we get our table back. And then this is a, this is a concurrent system. So and I'll show you that in a second. So you'll notice I had two connections going there. They're both working. So how is that working? Is it two background workers? No, it's one background worker. And it's using a kind of a cool Rust library that's gotten a lot of attention called Tokyo, which is a futures runtime. So this is, futures are a concept in programming its form of abstraction for like, say, asynchronous operations where you are essentially deferring the work. You kind of say, do this later and then when you have the result available, then I'll be able to get that result later. And then if you build these things up enough, you have whole network libraries and things like that that allow you to process many concurrent network requests in either a thread pool, kind of arbitrarily shuffling among them, or in this case here, just a single current thread runtime which runs it all in the current process. So basically, I've got a process request which looks a lot like our previous UDFSPI. But I've also got this set up here to initialize the background worker and then have a main function that does all the unblocked signals and all that other stuff you need to do. And then here is me creating a runtime, got a server that, which is this kind of way of building up futures. So at the beginning when I said all the code would be intelligible, except the last thing, this is what I was talking about. It's not intelligible to me either. So if somebody knows more about this, then they might be happy. I do know a little bit about this, but it's kind of a different style. It's like asynchronous programming. It's supposed to be a higher level than that than kind of the raw state machines. You can program with futures and you basically have this way of getting lines over the network. And then when one thing is doing something, you can do it. When that is kind of waiting on input, say waiting on new data to come over the network, then it just gets kind of yields and the next request in turn comes along and you process what you have. So this allows the concurrency. It also executes everything in one background worker, which is good because executing SPI across a bunch of threads or something would be a very dubious thing to do. So anyway, the kind of point of this example is really to show a non-trivial combination of all these things coming together using something that's not just a UDF, but actually a background worker that does interesting things in Postgres and needing to copy things out of memory contexts and this kind of thing using, and then in addition, so it's doing non-trivial Postgres stuff. It's also doing non-trivial Rust stuff. This is a large dependency doing a lot of stuff and it all kind of comes together and works. So if I might just kind of conclude from here, I think there's a lot of room for developing extensions and Rust and I think it essentially stood up to every challenge that I put forward and I think that I solved a lot of the hard problems in this process for actually writing real extensions and Rust and doing real stuff. I think that Rust kind of came through and withstood the challenges and kind of, in my opinion, it seems like it's a real low-level database worthy language. So I'll take any questions. Yes? It's probably going to be a large debug and everything else. You can with Rust cut out a lot of stuff including the standard library. It's meant to be used even in embedded environments. I can't really speak much to that. Let's see here, to that part of it, but supposedly, it looks like 14 megabytes. So here we have one point, it looks like, yeah. Okay? Yeah, I think it's mostly just code. So yeah, it should be possible to cut it down as much in size as possible, like similar to C. You could cut out the standard library or cut out this library. It tries to inline things and cut out symbols it doesn't need and that kind of stuff. If we did a release build, I think it would go through a link time optimization pass as well and all that. So it should be reasonable. Question? Yeah, it's, I mean, really I just did a few, I took on what I felt like were the hard problems. After that, I think I need to write some more utilities that just do things that maybe everyone would need to do or maybe lots of people would need to use, but that maybe not that terribly hard and then the other things that might be functions I wouldn't guess necessarily would need to be called, you might just need to write some external declarations at the top to pull those in. So I think it's a good start and like I said, I'm just trying to solve the hardest problem so that we have a place to take it from here incrementally. So I think it would be a good thing to hack on. So anyway, yeah, I think I see people kind of lining up here. So I'll take certainly more questions in the hallway or at the party later anytime, happy to. So again, you know, my name is Jeff Davis. If you want to find me, I work for Citus Data Microsoft. And that's all. Thank you very much, everyone. I want to remind you that I'm Hi Mike, happy to set up, please.
 Okay, hi everyone, my name is Jeff.  Today I'm going to be talking about writing deep post-scrass extensions, real post-scrass  extensions that do interesting stuff beyond just maybe a UDF, something like that, in  a language called Rust.  Probably a lot of the people in the area have heard about the language Rust.  Is there anybody in here that hasn't heard of Rust or don't really know anything about  it?  Okay, so I'll kind of briefly go over that.  I won't be talking as much about the language itself.  The code samples should be legible for the most part they're not, except for the last  one, but that's for a different reason.  I'll get into that.  So let me just talk about the motivation here.  So post-scrass is more and more reliant on the ecosystem of extensions.  There are important extensions out there that do important things for a lot of post-scrass  users and customers.  And that's a good thing.  Having an extensible database system provides a lot of flexibility in terms of things like  not needing to wait for release cycles of the main post-scrass product, not needing  to meet some of the standards in terms of maybe you depend on libraries that might have  an unacceptable license for the post-scrass project, or maybe you're using proprietary  code or something like that.  And for those reasons, maybe it can't make it into the main post-scrass SQL code base.  And it shouldn't.  And it's great that we have the flexibility to use those maybe a proprietary library or  a GPL library or something that would never be accepted into the main code base, but still  provides really important functionality.  Post-gis is a good example of that.  Very important spatial extension for post-scrass.  But it depends on libraries that have different licenses.  Post-gis itself has a different license.  So anyway, this is all good stuff.  And I'd like to expand this beyond C. I'd like to expand this capability of writing  extensions to a new set of developers that for whatever reason may not be able to or  not want to use C as a programming language.  And Rust is offering another option.  Right now, everything needs to be C. So this going from supporting one language to supporting  two.  Hopefully, that's a big step.  There aren't really many options out there, aside from C. And I'd like to create a second  one.  So why Rust?  There's a lot of marketing out there.  This isn't an advocacy talk about Rust, but it's got some nice features.  It's got a nice community.  It provides some improved memory safety features above what C can offer.  It's used a modern type system that helps catch a lot of errors.  But it's also nice and low level.  It's suitable for, at least, ostensibly suitable for low level work.  So it's not kind of adding code behind your back.  It's not got a large runtime or injecting extra fields and destructs or things like that.  It really just allows you to work with the memory representation that you already have  or really want to use.  And then you can use the other language features on top of that representation.  It doesn't impose a lot of that on you.  And it doesn't kind of inject code or use aspects of the runtime that might conflict  with what Postgres wants to do.  So Rust, if we're going to support a second language, we really have very few options  that actually meet these requirements.  So I mean, there's C. And then after that, the options get a little bit thin.  C++, Rust might be a couple of obvious ones.  I think Rust is different enough.  It's kind of a more interesting exercise and might open us up a little bit more to a wider  group of developers than trying to say work with C++.  It's also got an awesome ecosystem and awesome community.  And I'd like to kind of bring all those people into the idea that, hey, you can write a really  interesting extension that really expands the functionality of Postgres in interesting  ways in their language of choice.  Right now, the Postgres world is C. I mean, that's just the way it is.  There are some really nice PLs out there.  And I've used those to great effect.  They're really convenient to use.  But they're fairly constrained in that it's pretty much scalar functions, set returning  functions.  You can do a few things like that.  There might be some efficiency costs going in and out of that.  But I mean, really, if you want to do things like this, foreign data wrappers, custom data  types, index support functions, background workers, UDFs calling lots of internal functions  all over the place that might be not really intended to be called from a UDF, then you  need to use C. That's the current state of the world.  And I'm trying to change that so you can do all of these things with an additional language.  So what about procedural languages?  I mentioned this briefly before.  But procedural language, the infrastructure here is great.  You have the ability to write, say, in Perl or Python or JavaScript or PLPG SQL, you have  this ability to write functions and even execute queries using the server programming interface.  And it's, like I said, a great API.  But I'd actually like to go further than this.  So this is not like a PL Rust implementation.  I'm trying to take it a little bit further than that and make it really kind of co-equal  with C in terms of capability.  So the other thing, I'd like to see what Rust can do.  I mean, it's kind of an exciting language.  But that also means there's a lot of marketing that goes overboard.  They kind of not intentionally from the Rust developers, but some of the community might  get a little bit over excited and maybe overstate some claims or overstate the ease with which  those claims can be realized.  And so let's really apply it to an interesting project.  I mean, Postgres is a big piece of software with a lot of things going on.  I think I have another.  Oh, yeah.  This is a slide.  It's got things like memory contexts, doing the allocations.  It's dealing lots of global variables, set jump and long jump based error handling.  These are non-trivial things.  In other words, we're not just kind of plugging it into our Hello World C app and saying, hey,  Rust is great.  We're plugging it into a real thing like Postgres.  And we're going to test some of these ideas and how far they can really go and what we  can really accomplish.  So what is Postgres extension.rs?  This is a crate that I wrote.  It's available.  You can use it as a dependency today.  So if you're writing your own Rust crate that's a library, I've got some examples in GitHub  as well.  You can, extensions can use this as a dependency using the crate ecosystem, which is the package  manager comparable to the package managers in other languages.  So basically this dependency that you can pull in called Postgres extension, it's a  collection of functions, macros, the kinds of things that you need.  Basically speaking, it's like the C header files.  So just like if you're developing a C extension, you need to have the Postgres server header  files available to develop that.  Also with Rust, you would use this.  And this is essentially providing definitions that you need to get stuff done.  Now you can imagine there's an awful lot of C header files in the server.  So this only implements some of these things, I tried to really take on the hardest problems.  And right now the boilerplate stuff of just copying struct definitions around, that it's  not capturing all the C headers out there.  But I tried to take on some of the more challenging problems and see how far Rust could go with  that.  So I'll talk about those challenging problems and show some demos.  Just to clarify, so Rust already has a great client driver.  This is not like for developing an application in Rust that talks to Postgres with the client.  This is really about actually modifying the server with an extension.  So I just like to clarify that.  There's already a great Rust Postgres driver out there.  If you have an application, you'd like it to talk to Postgres, that's got a great answer.  So let me talk a little bit about the features of what I think I actually accomplished with  Postgres extension.rs, the crate.  So first of all, you can use these C structures directly.  So you might, using other languages, if you're using Python or something like that, and you  need to access some of the internal structures in Postgres, you might imagine that you would  write some kind of translation layer that would take the memory, copy it around, move  it in different ways, and then eventually you have something that looks like Python that  you can work with, and then you operate on that.  This, I don't want to do that.  For efficiency reasons and other reasons, you just don't want to be copying and translating  all the time.  So this can actually work directly on those C structures.  If you get a parameter that's a Postgres structure, then you can just immediately use that as though  you're writing C code or comparably to writing C code, you're operating directly on that  representation.  You're not translating, you're not copying.  So that's a big point to make because we don't want to discourage people from writing these  extensions because they say, oh, well, we're copying it again and again, and how do we,  maybe they need to modify that structure, and then that means they need to kind of translate  it back to update the changes.  It might cost in terms of efficiency.  So that's an important point there.  Another thing this does is it actually uses Palik and P free for the memory allocations.  So if you are using Rust standard library functions of vector strings, these kinds of  things, pushing and popping and all that stuff, when that's doing allocations and freeze,  those are going through Palik and P free.  So that means it's happening in the memory context, just like in Postgres when you're  doing these kinds of P string, dupe, or some kind of string function or something like  that, then when Postgres is doing those internal allocations, it's using the current memory  context, and so will your Rust code.  So it's a Postgres kind of feel to it.  And just like with C code, if you're using memory context, you have to be careful when  you're resetting them, deleting them, that kind of thing because obviously that can  leave Rust with dangling references if you cut the memory out from under it.  And then this is actually what I spent an awful lot of the time on is the error handling,  and this is what has made it difficult for other projects in the past like related to  C++ to really be successful in Postgres is handling these error handling situations.  So Postgres uses set jump and long jump.  For those who don't know what those do, it's kind of about what it sounds like.  It jumps across many stack frames.  So if you have deeply nested function calls, when Postgres encounters an error rather than  kind of returning to each one, it uses this C standard library feature.  It just jumps over all these function calls back to a place where it can actually recover  from the error.  Kind of like an exception, but C doesn't have exceptions, they have set jump long jump,  which is a very kind of wild west way of doing this, but it's been very useful in Postgres  for getting out of a lot of work and kind of aborting a transaction, it works quite  well.  So to handle this in Rust, it was a fair amount of effort to one, actually make it work, and  then two, make it work somewhat ergonomically with macros that can make these things nicer  to program with.  And then, so anyway, this ensures that Rust, the structures are properly called, so it's  not just kind of jumping over stack frames that Rust expects to be properly unwound.  Okay, so I'd like to, just one second here, let me pull a terminal over there.  Okay.  Oops.  Okay.  Let me do it.  Okay, better.  Okay.  So this is basically just kind of showing off the error handling.  So those who have written Postgres extensions and UDFs before will recognize, you know,  PG module magic, that's kind of the Rust way of doing a macro with this bang at the end.  I'm pulling in some, you know, C functions that I need access to.  I'm pulling in some of the utilities in the Postgres extension crate.  So you know, and then I'm doing something unsafe.  I'm, you know, going to divide by zero.  So you know, that's something that might, you know, happen in, you know, whatever Rust  code you're writing, it might generate an error that way.  So this div zero, basically it's going to, you know, come into this function here.  So basically we're going to see how like an internal error generated can end up, you know,  going back out and being properly handled.  The next one is a case of you're writing a function and you actually decide on your own.  You want to make your own e-report.  So that's what this next function is about.  And the third is you do something in Rust and then somewhere Rust has, I try not to make  this too confusing because Postgres has an ocean of panic and Rust has an ocean of panic.  So when you're reading this here, this is basically just using the Rust version of panic.  So somewhere in your code, what that means is that it's going to kind of unwind the stack  and then get the error and then turn that into a Postgres error.  So it's like these three different cases.  The first being that you call a Postgres function that throws an error.  The second is that you want to throw an error yourself and the third is that, you know,  some Rust function you're calling wants to throw an error.  And some going to show how each of those are handled.  So.  So you'll see this is in the examples directory.  So this is actually a separate crate that depends on Postgres extension.  So it's an example of using the Postgres extension to create these UDS.  So build it.  We copy the generated shared object file.  So Rust is capable of generating shared object files, which are the way that Postgres loads  an extension.  So we're going to copy that.  And then we're going to fire up Postgres.  Okay.  So.  So if we select UDF panic.  So what happened is the function, we just take a quick look back here.  The function panic.  This is a Rust panic.  So this is a case where, you know, you have this maybe deep link nest address code throws  a panic.  And that's again not a Postgres panic.  This causes stack unwind and then is translated into a Postgres error.  Then the next one.  Here is a time when you want to, within your Rust code, generate a nicely formatted e-report.  You'd like to provide, you know, detail and, you know, all kinds of things like that.  And it takes that e-report in Postgres, or in, sorry, the e-report from Rust.  It unwinds the stack and then translates it into a Postgres error.  And then the last one, div zero.  This is generated inside of Postgres.  You call this function, it ends up throwing divide by zero inside of the division function,  and it passes that unmodified backup.  It catches it in Rust, unwinds the stack correctly, and then passes it unmodified backup into  Postgres.  Okay.  So that's demo number one.  Do I have a question over here?  An error context, are you talking about like line numbers and stuff?  Oh, so like the file and are you talking about the, you're not talking about.  Yeah, I think I just didn't do that.  But you know, basically, you know, yeah, I don't think there's any reason it can't be  done.  What are those exclamation marks in this R?  Oh, so in Rust, they have a hard distinction between function calls and macros.  So every macro ends in exclamation point, and function calls do not.  Yeah.  So it's just, you know, that part of the language.  Okay.  So the next demo I have is a little bit more interesting.  So let's first look at it.  So this is UDFSPI, and it does some SPI stuff.  So you know, those familiar with kind of higher level languages might be happy to see that  it's a little bit easier than using SPI might be in ordinary C code.  So I've got, you know, I've got this connection being made.  I execute that on the connection object, and I have some iterators there that, you know,  can iterate through the result set tuples, and then also through the attributes in the  tuple, in each tuple.  So we got this kind of nested for loops, but it's, you know, only, you know, fewer lines  of code than it would be in C.  So one thing I'd like to point out here is that, you know, I'm doing some things differently.  So the SPI API in Postgres is quite a bit uglier, you know, who's, who here has used  the C API for SPI?  Okay.  So there's a couple things you might recall.  One of them is that SPI Connect doesn't return anything, so you might be wondering what that  thing is.  The other thing is SPI does kind of return things from an, you know, query execution,  but they end up in these global variables, which you have to kind of copy over, and you  have to be sure not to execute another SPI function because it will overwrite those global  variables without freeing whatever resources they were pointing to.  So you better get the reference to the tuple table while you can.  And it's, you know, potentially leads to, you know, it's error prone, let's say.  And there have been bugs in Postgres, not even in Postgres extensions, but there have  actually been bugs in core Postgres because of that API.  And there is a comment pointing out that it is perhaps not the best API, but we're stuck  with it.  In Rust here, so I, you know, kind of solved some of these problems, I believe.  I think I made a nicer, less error prone API.  So one of the things I'm doing here is I'm returning this object, this SPI object here,  if I'll put the cursor on that variable.  And then that is a zero-sized value.  So there's no allocation there.  It's really just compile time thing.  And the reason I do this is because for one thing, that way you can't really call the  execute method without having connected first.  So that's a potential, you know, those of who have used SPI have probably all made that  mistake.  And so it's, you know, just nicely there.  It also allows, you know, the destructor to call SPI finish.  And so it kind of helps you make this symmetric, you know, use the API in this symmetric way  consistently.  And so it kind of offers these benefits, you know, but it's actually zero-sized type.  So that's not doing any allocation there, not even on the stack.  The result set here, this is not zero-sized, but it's the same size as if you were to kind  of copy those global variables locally, which you really should anyway.  So, you know, just, it's, I think, you know, two or three words or whatever to make a copy  of those global variables before you lose them from the next call.  Question over here?  And so that, I also consider zero cost, you know, compared to using the API the way you  should, because you really should make copies of those global variables unless, you know,  I mean, it's just kind of asking for trouble.  It's not really expensive, a couple stack variables, you know, maybe in registers.  So I don't really consider that to be a cost.  You could get around it if you really wanted to, by the way, it's not like Rust can't support  global variables, you know, but you'd almost never want to do that.  The iterators, they might have some, you know, kind of transient data costs, you know, maybe  on the stack or in registers or something, but it's pretty lightweight.  It's basically, you know, just all using the same data and references where appropriate  it's not doing a bunch of data copying, it's not doing real allocations.  So that's also, you know, zero cost.  So you're basically getting, you know, for paying nothing, you're getting this, you know,  substantially better API that prevents a lot of mistakes, including, you know, making sure  to finish the, you know, SPI finish at the end.  So maybe I'll actually run this example.  If somebody has a question, please jump in.  The destructor for this zero size SPI object will SPI finish, which is the disconnect,  right?  So that's the other side of the connect.  Okay, so let's build it.  There we go, build.  And then copy, copy the right one, PDF SPI.  Okay.  And then now let's go back to our server because the way Postgres works, we just need to make  a new connection and we'll get the new code.  Create or replace.  Okay, so first I'll just select star from who and we see we have this wonderful table  here, very sophisticated.  And then if we also do UDF SPI, which is running the same query, we get these notes.  Notices printed out.  Now it'll kind of make sense when we look back at the code, you'll see how these notices  are generated.  It's just returning a one always and it kind of builds up those strings for each tuple.  It gets out of SPI.  So we can look back at the code here.  And you'll see here this is where it's, you know, doing the notices, building up the  strings.  It's kind of pushing, you know, pushes kind of way to build up strings.  These are formatting codes here.  The simplest one is just curly brace and curly brace, but there might be formatting options  inside of those.  So it's like a, it's a formatting string.  And then here it's doing the return, which we saw at the end.  So that's an SPI example.  I think the interesting part there that is different is just that I feel like the API  is safer and nicer without imposing a significant cost.  David?  I believe that is immutable.  So I mean, I don't, what would it mean to mutate a tuple that you get back from SPI?  I mean, it wouldn't go back to the table, right?  I assume that you're changing the, you're mutating the, where, I mean, so this is just  kind of a read-only thing.  Is there a reason, you know, so, do you just mean that that's like kind of rebound each  time through the loop?  Yeah.  Okay.  So Rust isn't really considered mutating it because it's a separate binding each time.  It's almost as if you had had like the declared value in this block.  So it's just, it's not actually mutating the value.  Yeah.  So we're a destructor on the couple, do you want to be getting disrupted at the end of  the loop?  Yes, it would be.  So I think in this case it's kind of a reference or immutable.  It's the, the actual memory of the tuple is, is, it's just a reference to that.  So I mean, there's not much destruction to do.  SPI will clean up the result set afterward.  The result set does have a destructor which frees the result set.  And then if you SPI finish it, frees all the results that's generated in that session.  Andres?  Can you catch errors to throw in an SPI and then continue it in some way?  If so, what exactly, because you need to do weird stuff to actually set up SPI contacts  and stuff?  So I mean, you have the same problems there you do in C, right?  I mean, kind of doing a try, catch around SPI is a little bit questionable, you know,  more than a little bit questionable in C as well.  So I, I didn't magically solve that just by using Rust.  I mean, that's, you solved that in your records, I'm not that way.  So I mean that, I mean, in, in general, you've got to be really careful with the try, catch  around SPI because it might have changed tuples in, you know, and actually there's NBCC information  there.  So if then you kind of catch that and end up an error and then end up committing the  transaction, then it's possible you would have like, you know, things that you inserted  that wrong.  Yeah, so I'm trying not to do that level of, of magic here because, you know, it would  be something that the developer would really want control over.  It's, I want this to be kind of a, at the same level of abstraction as C with perhaps  some safer interfaces, but I'm, I'm trying not to do kind of things behind the scenes  like save points.  You know, a C programmer would have to do their own save points and then so would a  Rust programmer.  Well, other designers have like much nicer wrapper like Pio, Perl, the public pretty  nice wrapper-sized, right?  Yeah, I mean, I think there would be, you know, room to do that as like an optional  extra layer, but just that, you know, my primary goal for this project is not to be at a higher  level unless it's kind of a no-brainer sort of like these SPI iterators and stuff would  just make things easier without really changing the level of control you have.  So yeah, basically the, the scope of this is not to make it feel like, you know, programming  Python or something like that or with kind of magic happening behind the scenes.  I just want it to be like, you know, at that same low level that a C programmer would expect.  So I'm not sure I understood the question about the error in SPI correctly, but do  you have an unwrap on the execute that returns something?  Yeah, so the unwrap, this is a Rust thing.  It's basically if you have something that can return a result or an option or something  like that where, you know, maybe it returns a result, maybe it returns nothing or maybe  it returns an error, these, these options.  So you can handle these, whoa, I hope this video comes in.  Okay, so the unwrap is basically a way to kind of short circuit that and say, I, I know  there's a value there, just give it to me and then if there isn't a value there, then  panic.  So if that panic were to happen, we would end up seeing that as a post-gress error.  A, a Rust panic happened.  So I, during this whole talk, anytime I say panic, it's not the post-gress panic.  It's a Rust panic.  So that's what would happen in that case.  So it would be handled reasonably well.  Here I'm really expecting a result.  I could have kind of handled this and wrote in more Rust code to make it more robust,  but I just, I just didn't do that for clarity.  And that's a pretty common pattern.  If you know something's there, then you just unwrap and then, you know, if you're wrong  later you're going to get a panic and know where it's from.  And then I'm running out of time a little bit here, but I have one more demo that I'd  like to show off.  I'll show it off a little bit quick.  Okay.  So this is, let me actually, let me build it first and kind of show it off first.  And then we can look at the code.  This is a background worker.  So here I already set up the configuration file to auto-load this, this background worker.  And so it listens on 8080.  And then if we use netcat, it just reading lines off the network and executing them as  SPI and returning the result.  So when I, so, lots of things.  So yeah.  So if we just do like select one, boom, we get a one.  Select star from foo, we get our table back.  And then this is a, this is a concurrent system.  So and I'll show you that in a second.  So you'll notice I had two connections going there.  They're both working.  So how is that working?  Is it two background workers?  No, it's one background worker.  And it's using a kind of a cool Rust library that's gotten a lot of attention called Tokyo,  which is a futures runtime.  So this is, futures are a concept in programming its form of abstraction for like, say, asynchronous  operations where you are essentially deferring the work.  You kind of say, do this later and then when you have the result available, then I'll be  able to get that result later.  And then if you build these things up enough, you have whole network libraries and things  like that that allow you to process many concurrent network requests in either a thread pool,  kind of arbitrarily shuffling among them, or in this case here, just a single current  thread runtime which runs it all in the current process.  So basically, I've got a process request which looks a lot like our previous UDFSPI.  But I've also got this set up here to initialize the background worker and then have a main  function that does all the unblocked signals and all that other stuff you need to do.  And then here is me creating a runtime, got a server that, which is this kind of way of  building up futures.  So at the beginning when I said all the code would be intelligible, except the last thing,  this is what I was talking about.  It's not intelligible to me either.  So if somebody knows more about this, then they might be happy.  I do know a little bit about this, but it's kind of a different style.  It's like asynchronous programming.  It's supposed to be a higher level than that than kind of the raw state machines.  You can program with futures and you basically have this way of getting lines over the network.  And then when one thing is doing something, you can do it.  When that is kind of waiting on input, say waiting on new data to come over the network,  then it just gets kind of yields and the next request in turn comes along and you process  what you have.  So this allows the concurrency.  It also executes everything in one background worker, which is good because executing SPI  across a bunch of threads or something would be a very dubious thing to do.  So anyway, the kind of point of this example is really to show a non-trivial combination  of all these things coming together using something that's not just a UDF, but actually  a background worker that does interesting things in Postgres and needing to copy things  out of memory contexts and this kind of thing using, and then in addition, so it's doing  non-trivial Postgres stuff.  It's also doing non-trivial Rust stuff.  This is a large dependency doing a lot of stuff and it all kind of comes together and  works.  So if I might just kind of conclude from here, I think there's a lot of room for developing  extensions and Rust and I think it essentially stood up to every challenge that I put forward  and I think that I solved a lot of the hard problems in this process for actually writing  real extensions and Rust and doing real stuff.  I think that Rust kind of came through and withstood the challenges and kind of, in my  opinion, it seems like it's a real low-level database worthy language.  So I'll take any questions.  Yes?  It's probably going to be a large debug and everything else.  You can with Rust cut out a lot of stuff including the standard library.  It's meant to be used even in embedded environments.  I can't really speak much to that.  Let's see here, to that part of it, but supposedly, it looks like 14 megabytes.  So here we have one point, it looks like, yeah.  Okay?  Yeah, I think it's mostly just code.  So yeah, it should be possible to cut it down as much in size as possible, like similar  to C. You could cut out the standard library or cut out this library.  It tries to inline things and cut out symbols it doesn't need and that kind of stuff.  If we did a release build, I think it would go through a link time optimization pass as  well and all that.  So it should be reasonable.  Question?  Yeah, it's, I mean, really I just did a few, I took on what I felt like were the hard problems.  After that, I think I need to write some more utilities that just do things that maybe everyone  would need to do or maybe lots of people would need to use, but that maybe not that terribly  hard and then the other things that might be functions I wouldn't guess necessarily  would need to be called, you might just need to write some external declarations at the  top to pull those in.  So I think it's a good start and like I said, I'm just trying to solve the hardest problem  so that we have a place to take it from here incrementally.  So I think it would be a good thing to hack on.  So anyway, yeah, I think I see people kind of lining up here.  So I'll take certainly more questions in the hallway or at the party later anytime, happy  to.  So again, you know, my name is Jeff Davis.  If you want to find me, I work for Citus Data Microsoft.  And that's all.  Thank you very much, everyone.  I want to remind you that I'm Hi Mike, happy to set up, please.
Deep_Postgres_Extensions_-_Jef_Davis_0_4379.png Deep_Postgres_Extensions_-_Jef_Davis_4380_7463.png Deep_Postgres_Extensions_-_Jef_Davis_7464_8975.png Deep_Postgres_Extensions_-_Jef_Davis_8976_9983.png Deep_Postgres_Extensions_-_Jef_Davis_9984_11819.png Deep_Postgres_Extensions_-_Jef_Davis_11820_15119.png Deep_Postgres_Extensions_-_Jef_Davis_15120_22199.png Deep_Postgres_Extensions_-_Jef_Davis_22200_22823.png Deep_Postgres_Extensions_-_Jef_Davis_22824_23087.png Deep_Postgres_Extensions_-_Jef_Davis_23088_23147.png Deep_Postgres_Extensions_-_Jef_Davis_23148_24623.png Deep_Postgres_Extensions_-_Jef_Davis_24624_25307.png Deep_Postgres_Extensions_-_Jef_Davis_25308_26939.png Deep_Postgres_Extensions_-_Jef_Davis_26940_27491.png Deep_Postgres_Extensions_-_Jef_Davis_27492_27899.png Deep_Postgres_Extensions_-_Jef_Davis_27900_28007.png Deep_Postgres_Extensions_-_Jef_Davis_28008_28139.png Deep_Postgres_Extensions_-_Jef_Davis_28140_29615.png Deep_Postgres_Extensions_-_Jef_Davis_29616_29723.png Deep_Postgres_Extensions_-_Jef_Davis_29724_30383.png Deep_Postgres_Extensions_-_Jef_Davis_30384_31343.png Deep_Postgres_Extensions_-_Jef_Davis_31344_33515.png Deep_Postgres_Extensions_-_Jef_Davis_33516_33611.png Deep_Postgres_Extensions_-_Jef_Davis_33612_42131.png Deep_Postgres_Extensions_-_Jef_Davis_42132_42311.png Deep_Postgres_Extensions_-_Jef_Davis_42312_43223.png Deep_Postgres_Extensions_-_Jef_Davis_43224_43979.png Deep_Postgres_Extensions_-_Jef_Davis_43980_51719.png Deep_Postgres_Extensions_-_Jef_Davis_51720_51875.png Deep_Postgres_Extensions_-_Jef_Davis_51876_51971.png Deep_Postgres_Extensions_-_Jef_Davis_51972_53495.png Deep_Postgres_Extensions_-_Jef_Davis_53496_53807.png Deep_Postgres_Extensions_-_Jef_Davis_53808_53951.png Deep_Postgres_Extensions_-_Jef_Davis_53952_54047.png Deep_Postgres_Extensions_-_Jef_Davis_54048_54119.png Deep_Postgres_Extensions_-_Jef_Davis_54120_54299.png Deep_Postgres_Extensions_-_Jef_Davis_54300_55007.png Deep_Postgres_Extensions_-_Jef_Davis_55008_55199.png Deep_Postgres_Extensions_-_Jef_Davis_55200_55547.png Deep_Postgres_Extensions_-_Jef_Davis_55548_55871.png Deep_Postgres_Extensions_-_Jef_Davis_55872_56159.png Deep_Postgres_Extensions_-_Jef_Davis_56160_56327.png Deep_Postgres_Extensions_-_Jef_Davis_56328_57515.png Deep_Postgres_Extensions_-_Jef_Davis_57516_57659.png Deep_Postgres_Extensions_-_Jef_Davis_57660_57719.png Deep_Postgres_Extensions_-_Jef_Davis_57720_57887.png Deep_Postgres_Extensions_-_Jef_Davis_57888_58475.png Deep_Postgres_Extensions_-_Jef_Davis_58476_61451.png Deep_Postgres_Extensions_-_Jef_Davis_61452_61535.png Deep_Postgres_Extensions_-_Jef_Davis_61536_62399.png Deep_Postgres_Extensions_-_Jef_Davis_62400_62459.png Deep_Postgres_Extensions_-_Jef_Davis_62460_62555.png Deep_Postgres_Extensions_-_Jef_Davis_62556_62795.png Deep_Postgres_Extensions_-_Jef_Davis_62796_63119.png Deep_Postgres_Extensions_-_Jef_Davis_63120_66344.png
Okay, hi everyone, my name is Jeff. Today I'm going to be talking about writing deep post-scrass extensions, real post-scrass extensions that do interesting stuff beyond just maybe a UDF, something like that, in a language called Rust. Probably a lot of the people in the area have heard about the language Rust. Is there anybody in here that hasn't heard of Rust or don't really know anything about it? Okay, so I'll kind of briefly go over that. I won't be talking as much about the language itself. The code samples should be legible for the most part they're not, except for the last one, but that's for a different reason. I'll get into that. So let me just talk about the motivation here. So post-scrass is more and more reliant on the ecosystem of extensions. There are important extensions out there that do important things for a lot of post-scrass users and customers. And that's a good thing. Having an extensible database system provides a lot of flexibility in terms of things like not needing to wait for release cycles of the main post-scrass product, not needing to meet some of the standards in terms of maybe you depend on libraries that might have an unacceptable license for the post-scrass project, or maybe you're using proprietary code or something like that. And for those reasons, maybe it can't make it into the main post-scrass SQL code base. And it shouldn't. And it's great that we have the flexibility to use those maybe a proprietary library or a GPL library or something that would never be accepted into the main code base, but still provides really important functionality. Post-gis is a good example of that. Very important spatial extension for post-scrass. But it depends on libraries that have different licenses. Post-gis itself has a different license. So anyway, this is all good stuff. And I'd like to expand this beyond C. I'd like to expand this capability of writing extensions to a new set of developers that for whatever reason may not be able to or not want to use C as a programming language. And Rust is offering another option. Right now, everything needs to be C. So this going from supporting one language to supporting two. Hopefully, that's a big step. There aren't really many options out there, aside from C. And I'd like to create a second one. So why Rust? There's a lot of marketing out there. This isn't an advocacy talk about Rust, but it's got some nice features. It's got a nice community. It provides some improved memory safety features above what C can offer. It's used a modern type system that helps catch a lot of errors. But it's also nice and low level. It's suitable for, at least, ostensibly suitable for low level work. So it's not kind of adding code behind your back. It's not got a large runtime or injecting extra fields and destructs or things like that. It really just allows you to work with the memory representation that you already have or really want to use. And then you can use the other language features on top of that representation. It doesn't impose a lot of that on you. And it doesn't kind of inject code or use aspects of the runtime that might conflict with what Postgres wants to do. So Rust, if we're going to support a second language, we really have very few options that actually meet these requirements. So I mean, there's C. And then after that, the options get a little bit thin. C++, Rust might be a couple of obvious ones. I think Rust is different enough. It's kind of a more interesting exercise and might open us up a little bit more to a wider group of developers than trying to say work with C++. It's also got an awesome ecosystem and awesome community. And I'd like to kind of bring all those people into the idea that, hey, you can write a really interesting extension that really expands the functionality of Postgres in interesting ways in their language of choice. Right now, the Postgres world is C. I mean, that's just the way it is. There are some really nice PLs out there. And I've used those to great effect. They're really convenient to use. But they're fairly constrained in that it's pretty much scalar functions, set returning functions. You can do a few things like that. There might be some efficiency costs going in and out of that. But I mean, really, if you want to do things like this, foreign data wrappers, custom data types, index support functions, background workers, UDFs calling lots of internal functions all over the place that might be not really intended to be called from a UDF, then you need to use C. That's the current state of the world. And I'm trying to change that so you can do all of these things with an additional language. So what about procedural languages? I mentioned this briefly before. But procedural language, the infrastructure here is great. You have the ability to write, say, in Perl or Python or JavaScript or PLPG SQL, you have this ability to write functions and even execute queries using the server programming interface. And it's, like I said, a great API. But I'd actually like to go further than this. So this is not like a PL Rust implementation. I'm trying to take it a little bit further than that and make it really kind of co-equal with C in terms of capability. So the other thing, I'd like to see what Rust can do. I mean, it's kind of an exciting language. But that also means there's a lot of marketing that goes overboard. They kind of not intentionally from the Rust developers, but some of the community might get a little bit over excited and maybe overstate some claims or overstate the ease with which those claims can be realized. And so let's really apply it to an interesting project. I mean, Postgres is a big piece of software with a lot of things going on. I think I have another. Oh, yeah. This is a slide. It's got things like memory contexts, doing the allocations. It's dealing lots of global variables, set jump and long jump based error handling. These are non-trivial things. In other words, we're not just kind of plugging it into our Hello World C app and saying, hey, Rust is great. We're plugging it into a real thing like Postgres. And we're going to test some of these ideas and how far they can really go and what we can really accomplish. So what is Postgres extension.rs? This is a crate that I wrote. It's available. You can use it as a dependency today. So if you're writing your own Rust crate that's a library, I've got some examples in GitHub as well. You can, extensions can use this as a dependency using the crate ecosystem, which is the package manager comparable to the package managers in other languages. So basically this dependency that you can pull in called Postgres extension, it's a collection of functions, macros, the kinds of things that you need. Basically speaking, it's like the C header files. So just like if you're developing a C extension, you need to have the Postgres server header files available to develop that. Also with Rust, you would use this. And this is essentially providing definitions that you need to get stuff done. Now you can imagine there's an awful lot of C header files in the server. So this only implements some of these things, I tried to really take on the hardest problems. And right now the boilerplate stuff of just copying struct definitions around, that it's not capturing all the C headers out there. But I tried to take on some of the more challenging problems and see how far Rust could go with that. So I'll talk about those challenging problems and show some demos. Just to clarify, so Rust already has a great client driver. This is not like for developing an application in Rust that talks to Postgres with the client. This is really about actually modifying the server with an extension. So I just like to clarify that. There's already a great Rust Postgres driver out there. If you have an application, you'd like it to talk to Postgres, that's got a great answer. So let me talk a little bit about the features of what I think I actually accomplished with Postgres extension.rs, the crate. So first of all, you can use these C structures directly. So you might, using other languages, if you're using Python or something like that, and you need to access some of the internal structures in Postgres, you might imagine that you would write some kind of translation layer that would take the memory, copy it around, move it in different ways, and then eventually you have something that looks like Python that you can work with, and then you operate on that. This, I don't want to do that. For efficiency reasons and other reasons, you just don't want to be copying and translating all the time. So this can actually work directly on those C structures. If you get a parameter that's a Postgres structure, then you can just immediately use that as though you're writing C code or comparably to writing C code, you're operating directly on that representation. You're not translating, you're not copying. So that's a big point to make because we don't want to discourage people from writing these extensions because they say, oh, well, we're copying it again and again, and how do we, maybe they need to modify that structure, and then that means they need to kind of translate it back to update the changes. It might cost in terms of efficiency. So that's an important point there. Another thing this does is it actually uses Palik and P free for the memory allocations. So if you are using Rust standard library functions of vector strings, these kinds of things, pushing and popping and all that stuff, when that's doing allocations and freeze, those are going through Palik and P free. So that means it's happening in the memory context, just like in Postgres when you're doing these kinds of P string, dupe, or some kind of string function or something like that, then when Postgres is doing those internal allocations, it's using the current memory context, and so will your Rust code. So it's a Postgres kind of feel to it. And just like with C code, if you're using memory context, you have to be careful when you're resetting them, deleting them, that kind of thing because obviously that can leave Rust with dangling references if you cut the memory out from under it. And then this is actually what I spent an awful lot of the time on is the error handling, and this is what has made it difficult for other projects in the past like related to C++ to really be successful in Postgres is handling these error handling situations. So Postgres uses set jump and long jump. For those who don't know what those do, it's kind of about what it sounds like. It jumps across many stack frames. So if you have deeply nested function calls, when Postgres encounters an error rather than kind of returning to each one, it uses this C standard library feature. It just jumps over all these function calls back to a place where it can actually recover from the error. Kind of like an exception, but C doesn't have exceptions, they have set jump long jump, which is a very kind of wild west way of doing this, but it's been very useful in Postgres for getting out of a lot of work and kind of aborting a transaction, it works quite well. So to handle this in Rust, it was a fair amount of effort to one, actually make it work, and then two, make it work somewhat ergonomically with macros that can make these things nicer to program with. And then, so anyway, this ensures that Rust, the structures are properly called, so it's not just kind of jumping over stack frames that Rust expects to be properly unwound. Okay, so I'd like to, just one second here, let me pull a terminal over there. Okay. Oops. Okay. Let me do it. Okay, better. Okay. So this is basically just kind of showing off the error handling. So those who have written Postgres extensions and UDFs before will recognize, you know, PG module magic, that's kind of the Rust way of doing a macro with this bang at the end. I'm pulling in some, you know, C functions that I need access to. I'm pulling in some of the utilities in the Postgres extension crate. So you know, and then I'm doing something unsafe. I'm, you know, going to divide by zero. So you know, that's something that might, you know, happen in, you know, whatever Rust code you're writing, it might generate an error that way. So this div zero, basically it's going to, you know, come into this function here. So basically we're going to see how like an internal error generated can end up, you know, going back out and being properly handled. The next one is a case of you're writing a function and you actually decide on your own. You want to make your own e-report. So that's what this next function is about. And the third is you do something in Rust and then somewhere Rust has, I try not to make this too confusing because Postgres has an ocean of panic and Rust has an ocean of panic. So when you're reading this here, this is basically just using the Rust version of panic. So somewhere in your code, what that means is that it's going to kind of unwind the stack and then get the error and then turn that into a Postgres error. So it's like these three different cases. The first being that you call a Postgres function that throws an error. The second is that you want to throw an error yourself and the third is that, you know, some Rust function you're calling wants to throw an error. And some going to show how each of those are handled. So. So you'll see this is in the examples directory. So this is actually a separate crate that depends on Postgres extension. So it's an example of using the Postgres extension to create these UDS. So build it. We copy the generated shared object file. So Rust is capable of generating shared object files, which are the way that Postgres loads an extension. So we're going to copy that. And then we're going to fire up Postgres. Okay. So. So if we select UDF panic. So what happened is the function, we just take a quick look back here. The function panic. This is a Rust panic. So this is a case where, you know, you have this maybe deep link nest address code throws a panic. And that's again not a Postgres panic. This causes stack unwind and then is translated into a Postgres error. Then the next one. Here is a time when you want to, within your Rust code, generate a nicely formatted e-report. You'd like to provide, you know, detail and, you know, all kinds of things like that. And it takes that e-report in Postgres, or in, sorry, the e-report from Rust. It unwinds the stack and then translates it into a Postgres error. And then the last one, div zero. This is generated inside of Postgres. You call this function, it ends up throwing divide by zero inside of the division function, and it passes that unmodified backup. It catches it in Rust, unwinds the stack correctly, and then passes it unmodified backup into Postgres. Okay. So that's demo number one. Do I have a question over here? An error context, are you talking about like line numbers and stuff? Oh, so like the file and are you talking about the, you're not talking about. Yeah, I think I just didn't do that. But you know, basically, you know, yeah, I don't think there's any reason it can't be done. What are those exclamation marks in this R? Oh, so in Rust, they have a hard distinction between function calls and macros. So every macro ends in exclamation point, and function calls do not. Yeah. So it's just, you know, that part of the language. Okay. So the next demo I have is a little bit more interesting. So let's first look at it. So this is UDFSPI, and it does some SPI stuff. So you know, those familiar with kind of higher level languages might be happy to see that it's a little bit easier than using SPI might be in ordinary C code. So I've got, you know, I've got this connection being made. I execute that on the connection object, and I have some iterators there that, you know, can iterate through the result set tuples, and then also through the attributes in the tuple, in each tuple. So we got this kind of nested for loops, but it's, you know, only, you know, fewer lines of code than it would be in C. So one thing I'd like to point out here is that, you know, I'm doing some things differently. So the SPI API in Postgres is quite a bit uglier, you know, who's, who here has used the C API for SPI? Okay. So there's a couple things you might recall. One of them is that SPI Connect doesn't return anything, so you might be wondering what that thing is. The other thing is SPI does kind of return things from an, you know, query execution, but they end up in these global variables, which you have to kind of copy over, and you have to be sure not to execute another SPI function because it will overwrite those global variables without freeing whatever resources they were pointing to. So you better get the reference to the tuple table while you can. And it's, you know, potentially leads to, you know, it's error prone, let's say. And there have been bugs in Postgres, not even in Postgres extensions, but there have actually been bugs in core Postgres because of that API. And there is a comment pointing out that it is perhaps not the best API, but we're stuck with it. In Rust here, so I, you know, kind of solved some of these problems, I believe. I think I made a nicer, less error prone API. So one of the things I'm doing here is I'm returning this object, this SPI object here, if I'll put the cursor on that variable. And then that is a zero-sized value. So there's no allocation there. It's really just compile time thing. And the reason I do this is because for one thing, that way you can't really call the execute method without having connected first. So that's a potential, you know, those of who have used SPI have probably all made that mistake. And so it's, you know, just nicely there. It also allows, you know, the destructor to call SPI finish. And so it kind of helps you make this symmetric, you know, use the API in this symmetric way consistently. And so it kind of offers these benefits, you know, but it's actually zero-sized type. So that's not doing any allocation there, not even on the stack. The result set here, this is not zero-sized, but it's the same size as if you were to kind of copy those global variables locally, which you really should anyway. So, you know, just, it's, I think, you know, two or three words or whatever to make a copy of those global variables before you lose them from the next call. Question over here? And so that, I also consider zero cost, you know, compared to using the API the way you should, because you really should make copies of those global variables unless, you know, I mean, it's just kind of asking for trouble. It's not really expensive, a couple stack variables, you know, maybe in registers. So I don't really consider that to be a cost. You could get around it if you really wanted to, by the way, it's not like Rust can't support global variables, you know, but you'd almost never want to do that. The iterators, they might have some, you know, kind of transient data costs, you know, maybe on the stack or in registers or something, but it's pretty lightweight. It's basically, you know, just all using the same data and references where appropriate it's not doing a bunch of data copying, it's not doing real allocations. So that's also, you know, zero cost. So you're basically getting, you know, for paying nothing, you're getting this, you know, substantially better API that prevents a lot of mistakes, including, you know, making sure to finish the, you know, SPI finish at the end. So maybe I'll actually run this example. If somebody has a question, please jump in. The destructor for this zero size SPI object will SPI finish, which is the disconnect, right? So that's the other side of the connect. Okay, so let's build it. There we go, build. And then copy, copy the right one, PDF SPI. Okay. And then now let's go back to our server because the way Postgres works, we just need to make a new connection and we'll get the new code. Create or replace. Okay, so first I'll just select star from who and we see we have this wonderful table here, very sophisticated. And then if we also do UDF SPI, which is running the same query, we get these notes. Notices printed out. Now it'll kind of make sense when we look back at the code, you'll see how these notices are generated. It's just returning a one always and it kind of builds up those strings for each tuple. It gets out of SPI. So we can look back at the code here. And you'll see here this is where it's, you know, doing the notices, building up the strings. It's kind of pushing, you know, pushes kind of way to build up strings. These are formatting codes here. The simplest one is just curly brace and curly brace, but there might be formatting options inside of those. So it's like a, it's a formatting string. And then here it's doing the return, which we saw at the end. So that's an SPI example. I think the interesting part there that is different is just that I feel like the API is safer and nicer without imposing a significant cost. David? I believe that is immutable. So I mean, I don't, what would it mean to mutate a tuple that you get back from SPI? I mean, it wouldn't go back to the table, right? I assume that you're changing the, you're mutating the, where, I mean, so this is just kind of a read-only thing. Is there a reason, you know, so, do you just mean that that's like kind of rebound each time through the loop? Yeah. Okay. So Rust isn't really considered mutating it because it's a separate binding each time. It's almost as if you had had like the declared value in this block. So it's just, it's not actually mutating the value. Yeah. So we're a destructor on the couple, do you want to be getting disrupted at the end of the loop? Yes, it would be. So I think in this case it's kind of a reference or immutable. It's the, the actual memory of the tuple is, is, it's just a reference to that. So I mean, there's not much destruction to do. SPI will clean up the result set afterward. The result set does have a destructor which frees the result set. And then if you SPI finish it, frees all the results that's generated in that session. Andres? Can you catch errors to throw in an SPI and then continue it in some way? If so, what exactly, because you need to do weird stuff to actually set up SPI contacts and stuff? So I mean, you have the same problems there you do in C, right? I mean, kind of doing a try, catch around SPI is a little bit questionable, you know, more than a little bit questionable in C as well. So I, I didn't magically solve that just by using Rust. I mean, that's, you solved that in your records, I'm not that way. So I mean that, I mean, in, in general, you've got to be really careful with the try, catch around SPI because it might have changed tuples in, you know, and actually there's NBCC information there. So if then you kind of catch that and end up an error and then end up committing the transaction, then it's possible you would have like, you know, things that you inserted that wrong. Yeah, so I'm trying not to do that level of, of magic here because, you know, it would be something that the developer would really want control over. It's, I want this to be kind of a, at the same level of abstraction as C with perhaps some safer interfaces, but I'm, I'm trying not to do kind of things behind the scenes like save points. You know, a C programmer would have to do their own save points and then so would a Rust programmer. Well, other designers have like much nicer wrapper like Pio, Perl, the public pretty nice wrapper-sized, right? Yeah, I mean, I think there would be, you know, room to do that as like an optional extra layer, but just that, you know, my primary goal for this project is not to be at a higher level unless it's kind of a no-brainer sort of like these SPI iterators and stuff would just make things easier without really changing the level of control you have. So yeah, basically the, the scope of this is not to make it feel like, you know, programming Python or something like that or with kind of magic happening behind the scenes. I just want it to be like, you know, at that same low level that a C programmer would expect. So I'm not sure I understood the question about the error in SPI correctly, but do you have an unwrap on the execute that returns something? Yeah, so the unwrap, this is a Rust thing. It's basically if you have something that can return a result or an option or something like that where, you know, maybe it returns a result, maybe it returns nothing or maybe it returns an error, these, these options. So you can handle these, whoa, I hope this video comes in. Okay, so the unwrap is basically a way to kind of short circuit that and say, I, I know there's a value there, just give it to me and then if there isn't a value there, then panic. So if that panic were to happen, we would end up seeing that as a post-gress error. A, a Rust panic happened. So I, during this whole talk, anytime I say panic, it's not the post-gress panic. It's a Rust panic. So that's what would happen in that case. So it would be handled reasonably well. Here I'm really expecting a result. I could have kind of handled this and wrote in more Rust code to make it more robust, but I just, I just didn't do that for clarity. And that's a pretty common pattern. If you know something's there, then you just unwrap and then, you know, if you're wrong later you're going to get a panic and know where it's from. And then I'm running out of time a little bit here, but I have one more demo that I'd like to show off. I'll show it off a little bit quick. Okay. So this is, let me actually, let me build it first and kind of show it off first. And then we can look at the code. This is a background worker. So here I already set up the configuration file to auto-load this, this background worker. And so it listens on 8080. And then if we use netcat, it just reading lines off the network and executing them as SPI and returning the result. So when I, so, lots of things. So yeah. So if we just do like select one, boom, we get a one. Select star from foo, we get our table back. And then this is a, this is a concurrent system. So and I'll show you that in a second. So you'll notice I had two connections going there. They're both working. So how is that working? Is it two background workers? No, it's one background worker. And it's using a kind of a cool Rust library that's gotten a lot of attention called Tokyo, which is a futures runtime. So this is, futures are a concept in programming its form of abstraction for like, say, asynchronous operations where you are essentially deferring the work. You kind of say, do this later and then when you have the result available, then I'll be able to get that result later. And then if you build these things up enough, you have whole network libraries and things like that that allow you to process many concurrent network requests in either a thread pool, kind of arbitrarily shuffling among them, or in this case here, just a single current thread runtime which runs it all in the current process. So basically, I've got a process request which looks a lot like our previous UDFSPI. But I've also got this set up here to initialize the background worker and then have a main function that does all the unblocked signals and all that other stuff you need to do. And then here is me creating a runtime, got a server that, which is this kind of way of building up futures. So at the beginning when I said all the code would be intelligible, except the last thing, this is what I was talking about. It's not intelligible to me either. So if somebody knows more about this, then they might be happy. I do know a little bit about this, but it's kind of a different style. It's like asynchronous programming. It's supposed to be a higher level than that than kind of the raw state machines. You can program with futures and you basically have this way of getting lines over the network. And then when one thing is doing something, you can do it. When that is kind of waiting on input, say waiting on new data to come over the network, then it just gets kind of yields and the next request in turn comes along and you process what you have. So this allows the concurrency. It also executes everything in one background worker, which is good because executing SPI across a bunch of threads or something would be a very dubious thing to do. So anyway, the kind of point of this example is really to show a non-trivial combination of all these things coming together using something that's not just a UDF, but actually a background worker that does interesting things in Postgres and needing to copy things out of memory contexts and this kind of thing using, and then in addition, so it's doing non-trivial Postgres stuff. It's also doing non-trivial Rust stuff. This is a large dependency doing a lot of stuff and it all kind of comes together and works. So if I might just kind of conclude from here, I think there's a lot of room for developing extensions and Rust and I think it essentially stood up to every challenge that I put forward and I think that I solved a lot of the hard problems in this process for actually writing real extensions and Rust and doing real stuff. I think that Rust kind of came through and withstood the challenges and kind of, in my opinion, it seems like it's a real low-level database worthy language. So I'll take any questions. Yes? It's probably going to be a large debug and everything else. You can with Rust cut out a lot of stuff including the standard library. It's meant to be used even in embedded environments. I can't really speak much to that. Let's see here, to that part of it, but supposedly, it looks like 14 megabytes. So here we have one point, it looks like, yeah. Okay? Yeah, I think it's mostly just code. So yeah, it should be possible to cut it down as much in size as possible, like similar to C. You could cut out the standard library or cut out this library. It tries to inline things and cut out symbols it doesn't need and that kind of stuff. If we did a release build, I think it would go through a link time optimization pass as well and all that. So it should be reasonable. Question? Yeah, it's, I mean, really I just did a few, I took on what I felt like were the hard problems. After that, I think I need to write some more utilities that just do things that maybe everyone would need to do or maybe lots of people would need to use, but that maybe not that terribly hard and then the other things that might be functions I wouldn't guess necessarily would need to be called, you might just need to write some external declarations at the top to pull those in. So I think it's a good start and like I said, I'm just trying to solve the hardest problem so that we have a place to take it from here incrementally. So I think it would be a good thing to hack on. So anyway, yeah, I think I see people kind of lining up here. So I'll take certainly more questions in the hallway or at the party later anytime, happy to. So again, you know, my name is Jeff Davis. If you want to find me, I work for Citus Data Microsoft. And that's all. Thank you very much, everyone. I want to remind you that I'm Hi Mike, happy to set up, please.
ecosystem of extensions. There are important maybe it can't make it into the main post-scrass post-scrass. But it depends on libraries that can use the other language features on top of an awesome ecosystem and awesome community. And state of the world. And I'm trying to change that of not intentionally from the Rust developers, piece of software with a lot of things going on. slide. It's got things like memory contexts, collection of functions, macros, the kinds of not capturing all the C headers out there. But I has a great client driver. This is not like for actually modifying the server with an extension. Postgres is handling these error handling any reason it can't be done. What are those there that, you know, can iterate through the SPI function because it will overwrite those to call SPI finish. And so it kind of helps you of offers these benefits, you know, but it's like that or with kind of magic happening behind these, these options. So you can handle these, this and wrote in more Rust code to make it more things like that that allow you to process many creating a runtime, got a server that, which is declarations at the top to pull those in. So I