Zero-Cost Async IO
@withoutboats
Boats is a member of the language design, library, and cargo teams of the Rust Project. They are Senior Research Engineer working on Rust for Mozilla Research. Contributing to Rust since 2014, they've done design work for significant language extensions like generic associated types and constant generics. For the last year, Boats has been focused on adding async/await syntax for non-blocking I/O to Rust. Follow us on Twitter:
 Hey, everyone. Thanks for having me. Yeah, I'm, my name is Without Boats. As in, yeah, seen Boatess. You can, most people call me Boats, which is actually even more confusing. I am a researcher at Mozilla. I work on Rust. It's my full-time job. It's pretty cool. I think, so this talk is going to be about sort of, it's a feature that I've been working on for about the last year and a half. And before me, other people were working on for, I mean, honestly, since before, the very beginning of Rust, before 1.0. But before that, I just wanted to thank the organizers for having me and for all of their work in putting this conference on. I think, I, maybe someone's already said pointed this out, but I believe this is the first Rust conference outside of either the United States or Europe. And so, as someone who works on Rust, I'm really, like, excited and glad to see our global community, like, thriving and growing. And it's really cool to see all the conferences that are happening this year. Come on to the technical stuff. So, the feature that I've been working on is this thing called Async Await. It's sort of going to be probably the biggest thing that we do in the language this year. We're planning to ship it sometime in the next few months. And it's the solution to this problem that we've been struggling with for a really long time, which is how can we have a zero cost abstraction for asynchronous I.O. in Rust? So, I'm going to explain what zero cost abstraction means in a moment, but first, just to kind of give an overview of the feature. So, Async Await, it's just these two new keywords that we're adding to the language, Async and Await. And so, Async is this modifier that can be applied to, like, functions where now the function, instead of when you call it, it runs all the way through and returns. Instead, it returns immediately and returns this feature that will eventually result in whatever the function would return. And inside of an Async function, you can take the Await operator and apply it to other features, which will pause the function until those features are ready. And so, it's this way of handling asynchronous concurrent operations using these annotations that makes them much easier to write. So, here's a little code sample just to sort of highlight and explain the feature. This is in, like, basically just an adapter on, like, a kind of ORM type of thing, it's handwritten, where you have this getUser method, which takes a string for a username and then returns this, like, user domain object by querying the database for the record for that user. And it does that using Async.io, which means that it's an Async function instead of just a normal function. And so, when you call it, you can Await it. And then, just to walk through the body of this method, you just, the first thing is it creates the SQL query, interpolating the username into, you know, select from user's table. And then, we query the database. And this is where we're actually performing some I.O. So, query also returns a future because it's doing this Async.io. And so, when you query the database, you just add this Await in order to Await for the response. And then, once you get the response, you can parse a user out of it. This user domain object, which is, you know, part of your application. And so, this method is just sort of like a toy example for the talk. But what I wanted to highlight in it is that the only difference really between this and using Blocking.io are these little annotations where you just mark the functions as being Async. And when you call them, you add this Await. And so, it's, you know, relatively little overhead for getting, using Non-Blocking.io instead of Blocking.io. And in particular, in Rust, the really great thing about our implementation that makes me really excited about it is that our Async Await and futures are this zero-cost abstraction. And so, zero-cost abstractions are sort of a really defining feature of Rust. It's one of the things that differentiates us from a lot of other languages is that we really care about when we add new features that they are zero cost. We didn't actually come up with the idea. It's a big thing in C++ also. And so, I think the best explanation is this quote from Bjarne Streestrup, which is that a zero-cost abstraction means that when you don't use it, you don't pay for it. And further, when you do use it, you couldn't hand-code it any better. And so, there's these two aspects to a zero-cost abstraction, which is that the first is that the feature can't affect, can't add cost to people who aren't using the feature. So, we can't add this global cost that will slow down every program in order to enable this feature. And the second is that when you do use the feature, it can't be slower than if you didn't use it and then you feel like, oh, well, I would like to use this really nice feature that makes it easier, but it will make my program slow. And so, I'll just write this thing by hand. It will be a much bigger pain. And so, I'm going to kind of walk through the history of how we've tried to solve async.io and Rust. And some of the steps along the way we had features that failed the zero-cost test on both principles. So, the specific problem that we're trying to solve is async.io. And so, normally, I.O. is blocking. So, when you use I.O., you'll block thread and that will stop your program and then have to be rescheduled by the OS. And the problem with blocking I.O. is it just doesn't really scale when you have been trying to serve a lot of connections from the same program. And so, for these kinds of really high-scale network services, you really need some form of non-blocking or asynchronous I.O. And Rust in particular is supposed to be designed for languages that have these really high performance requirements. You know, it's a systems programming language for people who really care about the computing resources they're using. And so, for Rust to really be successful in the network space, we really need some sort of solution to this asynchronous I.O. problem. But the big problem with async.io is that the way it works is that when you call the I.O. system call, it just returns immediately and then eventually you can continue doing other work. But it's your program's responsibility for figuring out how to, like, schedule, calling back to the task that you had to pause on when you're doing the asynchronous I.O. And so, this makes, you know, writing an async I.O. program much more complex than writing one of these is blocking I.O. And so, a lot of languages that are trying to target these like scalable network services have been trying to come up with solutions for this problem that like take it, make it not the end-users problem, but a part of the language or part of the libraries. And so, the first solution that Rust started with was the study of green threads, which have been successful in a lot of languages. And so, green threads basically look like blocking I.O. They look like spawning threads and then just blocking on I.O. And everything looks exactly the same as if you were using the native OS primitives, but they've been designed as part of the language runtime to be optimized for this use case of having these network services which are trying to spawning, you know, thousands, tens of thousands, maybe millions of green threads at the same time. A language I think right now that is like having a lot of success with this model is Go, where they're called Go routines. And so, it's very normal for a Go program to have tens of thousands of these running at the same time because they're very cheap to spawn unlike OS threads. And so, just the advantage of green threads is that the memory overhead when you spawn an OS thread is much higher because you create this huge stack for each OS thread, whereas green threads, the way they normally work is that you will spawn a thread that starts with a very small stack that can grow over time. And so, spawning a bunch of new threads that aren't using a lot of memory yet is much cheaper. And also, a problem with using like the operating system primitives is that you depend on the operating system for scheduling, which means that you have to switch from your program's memory space into the kernel space and the context switching as a lot of overhead if you want to start having, you know, tens of thousands of threads that are all being like switched between really quickly. And so, by keeping that scheduling in the same program, you avoid these context switches and that really reduces the overhead. And so, green threading is a pretty good model that works for a lot of languages, both Go and Java, I believe, use this model. And Rust had it for a long time, but removed it shortly before 1.0. And the reason that we removed it is that it ultimately was not a zero cost abstraction, specifically because of the first issue that I talked about where it was imposing costs on people who didn't need it. So, if you just wanted to write a Rust program that didn't need to use green threads, it wasn't a network service. You still had to have this language runtime that was responsible for scheduling all of your green threads. And so, this was especially a problem for people who were trying to embed Rust inside of like a larger C application. It's one of the ways that we've seen a lot of success in people adopting Rust is that they have some big C program and they want to start using Rust. And so, they start integrating a bit of Rust into their program. It's just writing one section of the code in Rust. But the problem is that if you have to set up this runtime in order to call the Rust, then it's too expensive to just have a small part of your program in Rust because you have to set up the runtime before you can call the Rust functions. And so, shortly before 1.0, we removed green threads from the language. We removed this language runtime and we now have a runtime that is essentially the same as C. And so, it makes it very easy to call between Rust and C and it's very cheap, which is one of the key things that makes Rust really successful. And having removed green threads, we still needed some sort of solution to async.io. But what we realized was that it needed to be a library-based solution. We needed some sort of with providing a good abstraction for async.io that was not a part of the language, it was not a part of this runtime that came with every program, but was just this library that you could opt into and use when you needed it. The most successful library-based solution in general is this concept called futures. It's also called promises and JavaScript. So, the idea of a future is that it represents a value that may not have evaluated yet. And so, you can manipulate it before you actually have, before the future is actually resolved. Eventually, we'll resolve to something, but you can start running things with it before it's actually resolved. And there's a lot of work done on futures in a lot of different languages and they are a great way for supporting a lot of combinators and especially this async await syntax that makes it much more ergonomic to build on top of this concurrency primitive. And so, futures can represent a lot of different things. So, async.io is kind of the biggest, the most prominent one where you maybe make a network request and you immediately get a future back, which once the network request is finished, will resolve into whatever that network request is returning. But you can also represent things like timeouts where a timeout is just a future that will resolve once that amount of time has passed. And even things that aren't doing in the I.O. or anything like that, where just CPU intensive work, you can run that on like a thread pool and then just get a future that you hold on to that once the thread pool is finished doing that work, the future will resolve. The problem with futures was that the way that they've been represented in most languages is this callback based approach where you have this future and you can schedule a callback to run once the future resolves. And so the future is responsible for figuring out when it resolves and then when it resolves it runs whatever your callback was. And all these distractions are built on top of this model. And this just really didn't work for us because there's a lot of people experiment with it a lot and found that it just was forcing way too many allocations. Essentially every callback that you tried to schedule had to get its own separate like create object heap allocation. And so there were these allocations everywhere, these dynamic dispatches, and this like approach failed the zero cost abstraction on the second principle where now you were, it was not affecting people who weren't using it, but if you did use it, it would just be way slower than if you just had it written something yourself. And so why would you use it? Because if you wrote the thing yourself would be much faster. But I hope it's not lost. Sorry, I'll set up a bit of a cold. This really great alternative abstraction that was pole based. Yes, we've arrived at this model. I really want to give credit to Alex who's here and Aaron Turan who came up with this idea where instead of futures scheduling a callback, there instead you pull them. And so there's this other part of the program called an executor, which is responsible for actually running the futures. And so what the executor does is it pulls the future. And the future either returns pending that it's not ready yet, or once it is ready, it returns ready. And this model has a lot of advantages. One advantage is that you can just cancel futures very simply because all you do to cancel them is you stop polling them. Whereas with this callback based approach, it was really difficult once you've scheduled some work to cancel that work and haven't stopped. It also really enabled us to have this really clean abstraction boundary between different parts of the program. So most futures libraries come with an event loop. And your futures are scheduled across this event loop, but this way of doing I.O. and you really don't have any control over it. But in Rust, we have this really clean boundary between the executor, which schedules your futures, the reactor, which handles all of the I.O. and then your actual code. And so the end user can decide which executor they want to use, which reactor they want to use, giving them the kind of control that is really important in a systems language. But the real advantage, the most important thing about this model is that it enabled us to have this really perfect zero cost way of implementing futures, where each representative is this kind of state machine. And so the way this works is that when the futures, the code that you write gets compiled down to, it gets actually compiled into native code, where it is this kind of like state machine where it has one variant for each pause point, for each I.O. event, essentially. And each variant then has the state that it needs to resume from that I.O. point. And that is represented as this essentially like an enum, where it's this one structure where it's the variant, the discriminant, and then a union of all of the states that it could possibly need. And so this is an attempt to visually represent that abstractly, where this is a, this state machine has, you know, you perform two I.O. events, and so it has these different states. And each state it has this, the amount of like, space it needs to store everything you would need to restore to that state. And the entire future is just a single heap allocation that's that size, where you just allocate that state machine into one place and a heap. And it's just no additional overhead. So you don't have these like, all these boxed callbacks and things like that. You just have this like perfect, really truly zero cost model. So I feel like that is usually a bit confusing to people. So I tried to, this is my best keynote, visually represent what's going on, which is that the, so you spawn a future and that puts the future in the heap in this one location. And then if there's a handle to it, that's you start with the executor, the executor pulls the future until eventually the future needs to perform some sort of I.O. In which case, the future gets handed off to the reactor. And the reactor, which is handling the I.O., registers that the future is waiting on this particular I.O. event. And then eventually when that I.O. event happens, the reactor will wake up the future using the waker argument that you passed in when you pulled it. And so waking the future up, passes it back to the executor. And then the executor will pull it again. And it will just go back and forth like this until eventually the future resolves. And so then when the future finally resolves and evaluates its final result, the executor knows that it's done. And then it drops the handle and drops the future and the whole thing is finished. And so it forms this sort of cycle where you pull the future, wait for I.O., wake it up again, pull it again, on and on until eventually the whole thing is finished. And this model ended up being quite fast. This is the sort of the benchmark that was posted in the first post about futures, where benchmarked features against a lot of different implementations from other languages. Hire is better and futures is the one on the far left. So we had this really great zero cost abstraction that was competitive with the fastest implementations of async I.O. in a lot of other languages. But of course the problem is that you don't want to write these state machines by hand. You have your whole entire application status of state machine is not very pleasant to write. But that's where the future abstraction is really helpful is that we can build these other APIs on top of it. And so the first solution that we had was this idea of futures combinators where you can build up these state machines by applying all of these methods of the future, sort of similar to the way that iterator adapters like filter and map work. And so this function, it just, what it does is it requests rustling.org and then converts that the response to a string. So instead of returning just a string, it returns a future of a string because it's going to be an async function. And it has these futures in the body that it's going to be calling and those are going to actually be the parts that do some I.O. And then they're all sort of combined together using these combinators like and then in map. And we build all these combinators like and then map, filter, map error, like all kinds of different things. And this works. It has some downsides, especially these like nested callbacks, which can be really difficult to read sometimes. And so because you know it has these downsides, we also started at it, tried to implement an async await implementation. And so the first version of async await was not part of the language. Instead, it was this library that provided this through like a syntax plugin. And this is doing the same thing that the previous function did. It just fetches rustling and turns it to a string. But it does so using async await. And so it's much more like straight line, looks much more like the way normal blocking I.O. works, where just like in the example that I showed originally, the only real difference is these annotations. And so the async annotation turns this function into a future instead of just returning immediately. And then the await annotations await on these on the futures that you actually construct inside of the function. And await under this poll model, desugars to this sort of loop where what you do is you just pull in a loop. And every time you get pending back, you yield all the way back up to the executor that you're pending. And so then it waits until it gets woken up again. And then when finally the future that you're awaiting finishes, it finishes with the value and you break out of the loop with the value. And that's what these await expressions evaluate to. So this seemed like a really good solution. You know, you have this async await notation, which is compiling down to these really awesome zero cost futures. And so it was sort of released futures into the wild and got feedback. And that's where we ran into problems. That essentially anyone who tried to use futures quickly ran into very confusing error messages, where it would just kind of complain about how your future isn't static or doesn't implement this trait. And it would be sort of this baffling thing you didn't really understand. And the compiler would like make helpful suggestions, which you would just follow until eventually compiled. And so you would be like annotating closures with move, and you would put things into reference counted pointers and clone things and this and that. And it all felt like you were adding all this overhead to the thing that you didn't seem necessary. You didn't understand why you had to do it. And also when you were done, your code ended up looking like soup. And so tons of people were bouncing off of futures. And it didn't help that the combinators produced these really huge types where your entire terminal would be filled up with the type of one of your combinator chains. Or it would just be like, you know, an end then of an end then of a map error of a TCP stream and so on. And you know, you have to dig through this to try to figure out what the actual error that you encountered was. And I found this quote on Reddit, which I think really beautifully sums up all of the complaints about futures, which is that, you know, when using futures, the error messages are inscrutable. Having to use ref cell or clone everything for each feature leads to over company code. And it makes me wish that Russ just had garbage collection, which is, yeah, not great feedback. So looking at the situation maybe a year, a year and a half ago, it was clear that there were sort of two problems that needed to be solved in order to make futures more usable for people. And the first was we needed better error messages. And so the easiest way to do that is to build the syntax into the language and then they can hook into all of our diagnostics and error message, you know, support so that you can have really good error messages for async await. But the second was that most of these errors people are running into were actually them bouncing off a sort of obscure problem, which is called the borrowing problem, where there was this fundamental limitations in the way that futures were designed that made it something really common pattern was not possible to express. And that problem was that you can't in the original line of futures, you could not borrow across an await point. So if you were to await something, you couldn't have any references that were alive at that time. And so when people were having these problems that they were bouncing off of what they were actually ultimately what they were doing was they were trying to borrow while they were awaiting and they couldn't do that. And so if we could just make it so that was allowed, then most of these errors would go away and everything would be much easier to use. And you could just kind of write normal REST code with async and await and it would all work. And these kinds of borrows during await are extremely common because just the natural API service of Rust is to have references in the API. And so, but the problem with futures is that when you actually compile the future, which has to restore all that state, when you have some references to something else that's in the same stack frame, what you end up getting is the sort of self-referential future. And so here's the some code from the original like get user method where we have this SQL string and then when we call query with it, we use pass a reference to the SQL string. And so the problem here is that this reference to the SQL string is a reference to something else that's being stored in the same future state. And so it becomes the sort of self-referential struct where you have these are the fields of the future in theory if it were a real struct, you would have the database handle that you were in the self aspect of it. Then we'd also have the SQL string and the reference to the SQL string, which is ultimately a reference pointing back to a field of the same struct. And self-referential structs are sort of a really hard problem that we don't have a general solution for. And the reason that we can't allow you to have references to the same struct is that when you move that struct, then what happens is that we make a new copy of the struct in the location that you're moving it to and the old copy becomes invalidated. But when you make that copy, the reference that was self-referential is still pointing to the old copy. And that's becomes a dangling pointer and it's the kind of memory issues that Rust has to prevent. So we can't have self-referential structs because if you move them around, then they become invalidated. But what made this really frustrating in the futures case was that we actually don't really need to move these futures around. So if you remember the model where the future is in the heap and a handle to it is getting passed back and forth between the reactor and the executor, the future itself never actually moves. And so it's totally fine for the future to contain self-references as long as you never move it and you don't need to move it. So we really needed to solve this problem with some way to express in the API of futures that while you're pulling it, you're not allowed to move it around. And then if we just could express that somehow, then we could allow these kinds of self-references in the body of the future and then we could just have these references in your async functions and everything would work. And so we were working on this problem and ultimately we came out with this new API called PIN. And so PIN is a sort of adapter around other pointer types where they become this append reference instead of just a normal reference. And append reference, in addition to whatever other guarantees it has, it guarantees that the reference will never, the value of the reference is pointing to will never be moved again. And so it guarantees that it's going to stay in the same place until eventually it gets deallocated. And so if you have something in your API which says that it has to be taken by PIN, then you know that it will never be moved again and you can have these kinds of self-referential structs. And so we changed the way that futures work so that now there's just being a boxed future, it's a boxed future behind a PIN. So we know that wherever we boxed it up, put it in the heap, it's guaranteed now by part of the PIN API that it will never move again. And then, you know, when you pull the future instead of just passing a normal reference to it, we pass the PIN reference to it. And so the future knows that it can't be moved. And the trick here that makes this all work is that you can only get an un-PIN reference out of a PIN reference in unsafe code. It's an unsafe function to be able to do that. And so the API looks roughly like this, where you have PIN which is just, you know, a wrapper around a pointer type. It doesn't have any runtime overhead or anything. It just like demarcates it as being PIN. And then a PIN to box can be converted into a PIN reference. But the only way to convert a PIN reference into an un-PIN reference is to use an unsafe function. And so what we did was then we just changed the futures API so that instead of taking a normal reference, it takes a PIN reference. And otherwise, the API is the same as it was before. And this is essentially the API that we're going to be stabilizing. And so with that change, this code from the first example just works the way that it's written. And so you can just write code exactly the way you would write it with blocking I.O. Add these async and await annotations. And then what you get is this, you know, async I.O. with this really awesome zero cost abstraction where it's basically as cheap as if you hand wrote the state machine yourself by hand. So the situation today, PINning was stabilized in the last release about a month ago. We're in the process of stabilizing the future API. So probably in 135, maybe it will slip in B136, which is, you know, in about two or three months basically. And then we're hoping sometime this year we'll have async await stabilized, hopefully by the end of summer even, we're going to have this stabilized and so that people will be able to write non-blocking I.O. network services using this syntax that makes it very similar to writing with blocking I.O. Looking beyond that stabilization, we're also already starting to work on these sort of more long-term features like streams, I think is probably the next big one where it's an async future. So a future is just, you know, one value, but a stream is many values being yielded asynchronously. It's essentially like an asynchronous iterator. And so you'll be able to like loop asynchronously over a stream and things like that. And it's very important for like a lot of use cases where you have like, you know, streaming, HTTP, WebSockets, HTTP to push requests, that kind of thing where instead of having a networking, like RPC model where you, you know, make a network request and get a single response, you have streams of responses and requests going back and forth. There's also today a limitation that async fn can't be used in traits. There's a lot of work being done in the compiler to make it so that it would be able to support that. And looking out even beyond these features, someday we want to have generators where, sort of similar to generators in like Python or JavaScript, where instead of just having functions that return, you can also have functions that yield. And so then you can resume them again after the yield and you can use these to write functions as a way of writing iterators and streams, similar to how an async function lets you write a function that's actually a future. But I guess just sort of recapping the real critical insights that led to this sort of zero cost async Ion model where that first was this pull based version of futures, where we were able to compile these futures into these really tight state machines. And then secondly, this way of doing async wait syntax where we're able to have references across the wait points because of pinning. Supposed to say thank you, I don't know how it became an X. Thank you.
 Hey, everyone. Thanks for having me. Yeah, I'm, my name is Without Boats. As in, yeah,  seen Boatess. You can, most people call me Boats, which is actually even more confusing.  I am a researcher at Mozilla. I work on Rust. It's my full-time job. It's pretty cool. I  think, so this talk is going to be about sort of, it's a feature that I've been working  on for about the last year and a half. And before me, other people were working on for,  I mean, honestly, since before, the very beginning of Rust, before 1.0. But before that, I just  wanted to thank the organizers for having me and for all of their work in putting this conference  on. I think, I, maybe someone's already said pointed this out, but I believe this is the first  Rust conference outside of either the United States or Europe. And so, as someone who works on Rust,  I'm really, like, excited and glad to see our global community, like, thriving and growing.  And it's really cool to see all the conferences that are happening this year. Come on to the  technical stuff. So, the feature that I've been working on is this thing called Async  Await. It's sort of going to be probably the biggest thing that we do in the language this  year. We're planning to ship it sometime in the next few months. And it's the solution to this  problem that we've been struggling with for a really long time, which is how can we have a zero  cost abstraction for asynchronous I.O. in Rust? So, I'm going to explain what zero cost  abstraction means in a moment, but first, just to kind of give an overview of the feature. So,  Async Await, it's just these two new keywords that we're adding to the language, Async and Await.  And so, Async is this modifier that can be applied to, like, functions where now the function,  instead of when you call it, it runs all the way through and returns. Instead, it returns  immediately and returns this feature that will eventually result in whatever the function would  return. And inside of an Async function, you can take the Await operator and apply it to other  features, which will pause the function until those features are ready. And so, it's this way of  handling asynchronous concurrent operations using these annotations that makes them much easier to  write. So, here's a little code sample just to sort of highlight and explain the feature. This is  in, like, basically just an adapter on, like, a kind of ORM type of thing, it's handwritten,  where you have this getUser method, which takes a string for a username and then returns this,  like, user domain object by querying the database for the record for that user. And it does that  using Async.io, which means that it's an Async function instead of just a normal function. And  so, when you call it, you can Await it. And then, just to walk through the body of this method,  you just, the first thing is it creates the SQL query, interpolating the username into, you  know, select from user's table. And then, we query the database. And this is where we're actually  performing some I.O. So, query also returns a future because it's doing this Async.io. And so,  when you query the database, you just add this Await in order to Await for the response. And then,  once you get the response, you can parse a user out of it. This user domain object, which is,  you know, part of your application. And so, this method is just sort of like a toy example for  the talk. But what I wanted to highlight in it is that the only difference really between this  and using Blocking.io are these little annotations where you just mark the functions as being Async.  And when you call them, you add this Await. And so, it's, you know, relatively little overhead  for getting, using Non-Blocking.io instead of Blocking.io. And in particular, in Rust,  the really great thing about our implementation that makes me really excited about it is that our  Async Await and futures are this zero-cost abstraction. And so, zero-cost abstractions are  sort of a really defining feature of Rust. It's one of the things that differentiates us from a  lot of other languages is that we really care about when we add new features that they are zero  cost. We didn't actually come up with the idea. It's a big thing in C++ also. And so, I think the  best explanation is this quote from Bjarne Streestrup, which is that a zero-cost abstraction means  that when you don't use it, you don't pay for it. And further, when you do use it, you couldn't  hand-code it any better. And so, there's these two aspects to a zero-cost abstraction, which is  that the first is that the feature can't affect, can't add cost to people who aren't using the  feature. So, we can't add this global cost that will slow down every program in order to enable  this feature. And the second is that when you do use the feature, it can't be slower than if you  didn't use it and then you feel like, oh, well, I would like to use this really nice feature that  makes it easier, but it will make my program slow. And so, I'll just write this thing by hand. It  will be a much bigger pain. And so, I'm going to kind of walk through the history of how we've  tried to solve async.io and Rust. And some of the steps along the way we had features that  failed the zero-cost test on both principles. So, the specific problem that we're trying to solve  is async.io. And so, normally, I.O. is blocking. So, when you use I.O., you'll block thread and  that will stop your program and then have to be rescheduled by the OS. And the problem with  blocking I.O. is it just doesn't really scale when you have been trying to serve a lot of  connections from the same program. And so, for these kinds of really high-scale network services,  you really need some form of non-blocking or asynchronous I.O. And Rust in particular is  supposed to be designed for languages that have these really high performance requirements. You  know, it's a systems programming language for people who really care about the computing  resources they're using. And so, for Rust to really be successful in the network space,  we really need some sort of solution to this asynchronous I.O. problem. But the big problem  with async.io is that the way it works is that when you call the I.O. system call, it just  returns immediately and then eventually you can continue doing other work. But it's your program's  responsibility for figuring out how to, like, schedule, calling back to the task that you had  to pause on when you're doing the asynchronous I.O. And so, this makes, you know, writing an  async I.O. program much more complex than writing one of these is blocking I.O. And so, a lot of  languages that are trying to target these like scalable network services have been trying to  come up with solutions for this problem that like take it, make it not the end-users problem,  but a part of the language or part of the libraries. And so, the first solution that Rust  started with was the study of green threads, which have been successful in a lot of languages. And  so, green threads basically look like blocking I.O. They look like spawning threads and then  just blocking on I.O. And everything looks exactly the same as if you were using the native OS  primitives, but they've been designed as part of the language runtime to be optimized for this use  case of having these network services which are trying to spawning, you know, thousands, tens of  thousands, maybe millions of green threads at the same time. A language I think right now that  is like having a lot of success with this model is Go, where they're called Go routines. And so,  it's very normal for a Go program to have tens of thousands of these running at the same time  because they're very cheap to spawn unlike OS threads. And so, just the advantage of green  threads is that the memory overhead when you spawn an OS thread is much higher because you create  this huge stack for each OS thread, whereas green threads, the way they normally work is that you  will spawn a thread that starts with a very small stack that can grow over time. And so, spawning  a bunch of new threads that aren't using a lot of memory yet is much cheaper. And also, a problem  with using like the operating system primitives is that you depend on the operating system for  scheduling, which means that you have to switch from your program's memory space into the kernel  space and the context switching as a lot of overhead if you want to start having, you know, tens of  thousands of threads that are all being like switched between really quickly. And so, by keeping  that scheduling in the same program, you avoid these context switches and that really reduces the  overhead. And so, green threading is a pretty good model that works for a lot of languages, both  Go and Java, I believe, use this model. And Rust had it for a long time, but removed it shortly  before 1.0. And the reason that we removed it is that it ultimately was not a zero cost  abstraction, specifically because of the first issue that I talked about where it was imposing  costs on people who didn't need it. So, if you just wanted to write a Rust program that didn't  need to use green threads, it wasn't a network service. You still had to have this language  runtime that was responsible for scheduling all of your green threads. And so, this was especially  a problem for people who were trying to embed Rust inside of like a larger C application. It's one  of the ways that we've seen a lot of success in people adopting Rust is that they have some big C  program and they want to start using Rust. And so, they start integrating a bit of Rust into  their program. It's just writing one section of the code in Rust. But the problem is that if you  have to set up this runtime in order to call the Rust, then it's too expensive to just have a small  part of your program in Rust because you have to set up the runtime before you can call the Rust  functions. And so, shortly before 1.0, we removed green threads from the language. We removed this  language runtime and we now have a runtime that is essentially the same as C. And so, it makes it  very easy to call between Rust and C and it's very cheap, which is one of the key things that makes  Rust really successful. And having removed green threads, we still needed some sort of solution  to async.io. But what we realized was that it needed to be a library-based solution. We needed  some sort of with providing a good abstraction for async.io that was not a part of the language,  it was not a part of this runtime that came with every program, but was just this library that  you could opt into and use when you needed it. The most successful library-based solution in  general is this concept called futures. It's also called promises and JavaScript. So, the idea of a  future is that it represents a value that may not have evaluated yet. And so, you can manipulate it  before you actually have, before the future is actually resolved. Eventually, we'll resolve to  something, but you can start running things with it before it's actually resolved. And there's a lot  of work done on futures in a lot of different languages and they are a great way for supporting a  lot of combinators and especially this async await syntax that makes it much more ergonomic to build  on top of this concurrency primitive. And so, futures can represent a lot of different things.  So, async.io is kind of the biggest, the most prominent one where you maybe make a network  request and you immediately get a future back, which once the network request is finished,  will resolve into whatever that network request is returning. But you can also represent things  like timeouts where a timeout is just a future that will resolve once that amount of time has  passed. And even things that aren't doing in the I.O. or anything like that, where just CPU  intensive work, you can run that on like a thread pool and then just get a future that you hold  on to that once the thread pool is finished doing that work, the future will resolve. The problem  with futures was that the way that they've been represented in most languages is this callback  based approach where you have this future and you can schedule a callback to run once the future  resolves. And so the future is responsible for figuring out when it resolves and then when it  resolves it runs whatever your callback was. And all these distractions are built on top of this  model. And this just really didn't work for us because there's a lot of people experiment with  it a lot and found that it just was forcing way too many allocations. Essentially every callback  that you tried to schedule had to get its own separate like create object heap allocation. And  so there were these allocations everywhere, these dynamic dispatches, and this like approach failed  the zero cost abstraction on the second principle where now you were, it was not affecting people  who weren't using it, but if you did use it, it would just be way slower than if you just had  it written something yourself. And so why would you use it? Because if you wrote the thing yourself  would be much faster. But I hope it's not lost. Sorry, I'll set up a bit of a cold. This really  great alternative abstraction that was pole based. Yes, we've arrived at this model. I really want  to give credit to Alex who's here and Aaron Turan who came up with this idea where instead of  futures scheduling a callback, there instead you pull them. And so there's this other part of the  program called an executor, which is responsible for actually running the futures. And so what the  executor does is it pulls the future. And the future either returns pending that it's not ready  yet, or once it is ready, it returns ready. And this model has a lot of advantages. One advantage  is that you can just cancel futures very simply because all you do to cancel them is you stop  polling them. Whereas with this callback based approach, it was really difficult once you've  scheduled some work to cancel that work and haven't stopped. It also really enabled us to have this  really clean abstraction boundary between different parts of the program. So most futures  libraries come with an event loop. And your futures are scheduled across this event loop,  but this way of doing I.O. and you really don't have any control over it. But in Rust, we have  this really clean boundary between the executor, which schedules your futures, the reactor,  which handles all of the I.O. and then your actual code. And so the end user can decide which  executor they want to use, which reactor they want to use, giving them the kind of control that is  really important in a systems language. But the real advantage, the most important thing about  this model is that it enabled us to have this really perfect zero cost way of implementing  futures, where each representative is this kind of state machine. And so the way this works is  that when the futures, the code that you write gets compiled down to, it gets actually compiled  into native code, where it is this kind of like state machine where it has one variant for each  pause point, for each I.O. event, essentially. And each variant then has the state that it needs  to resume from that I.O. point. And that is represented as this essentially like an enum,  where it's this one structure where it's the variant, the discriminant, and then a union of  all of the states that it could possibly need. And so this is an attempt to visually represent  that abstractly, where this is a, this state machine has, you know, you perform two I.O. events,  and so it has these different states. And each state it has this, the amount of like, space it  needs to store everything you would need to restore to that state. And the entire future is just a  single heap allocation that's that size, where you just allocate that state machine into one place  and a heap. And it's just no additional overhead. So you don't have these like, all these boxed  callbacks and things like that. You just have this like perfect, really truly zero cost model.  So I feel like that is usually a bit confusing to people. So I tried to, this is my best keynote,  visually represent what's going on, which is that the, so you spawn a future and that puts the  future in the heap in this one location. And then if there's a handle to it, that's you start with  the executor, the executor pulls the future until eventually the future needs to perform some sort  of I.O. In which case, the future gets handed off to the reactor. And the reactor, which is  handling the I.O., registers that the future is waiting on this particular I.O. event. And then  eventually when that I.O. event happens, the reactor will wake up the future using the waker  argument that you passed in when you pulled it. And so waking the future up, passes it back to  the executor. And then the executor will pull it again. And it will just go back and forth like  this until eventually the future resolves. And so then when the future finally resolves and  evaluates its final result, the executor knows that it's done. And then it drops the handle and  drops the future and the whole thing is finished. And so it forms this sort of cycle where you  pull the future, wait for I.O., wake it up again, pull it again, on and on until eventually the  whole thing is finished. And this model ended up being quite fast. This is the sort of the  benchmark that was posted in the first post about futures, where benchmarked features  against a lot of different implementations from other languages. Hire is better and  futures is the one on the far left. So we had this really great zero cost abstraction that was  competitive with the fastest implementations of async I.O. in a lot of other languages. But  of course the problem is that you don't want to write these state machines by hand. You have your  whole entire application status of state machine is not very pleasant to write. But that's where  the future abstraction is really helpful is that we can build these other APIs on top of it.  And so the first solution that we had was this idea of futures combinators where you can build  up these state machines by applying all of these methods of the future, sort of similar to the  way that iterator adapters like filter and map work. And so this function, it just, what it does  is it requests rustling.org and then converts that the response to a string. So instead of  returning just a string, it returns a future of a string because it's going to be an async function.  And it has these futures in the body that it's going to be calling and those are going to actually  be the parts that do some I.O. And then they're all sort of combined together using these combinators  like and then in map. And we build all these combinators like and then map, filter, map error,  like all kinds of different things. And this works. It has some downsides, especially these  like nested callbacks, which can be really difficult to read sometimes. And so because  you know it has these downsides, we also started at it, tried to implement an async await implementation.  And so the first version of async await was not part of the language. Instead, it was this library  that provided this through like a syntax plugin. And this is doing the same thing that the previous  function did. It just fetches rustling and turns it to a string. But it does so using async await.  And so it's much more like straight line, looks much more like the way normal blocking I.O. works,  where just like in the example that I showed originally, the only real difference is these  annotations. And so the async annotation turns this function into a future instead of just  returning immediately. And then the await annotations await on these on the futures that you actually  construct inside of the function. And await under this poll model, desugars to this sort of loop  where what you do is you just pull in a loop. And every time you get pending back, you yield all  the way back up to the executor that you're pending. And so then it waits until it gets  woken up again. And then when finally the future that you're awaiting finishes, it  finishes with the value and you break out of the loop with the value. And that's what  these await expressions evaluate to. So this seemed like a really good solution.  You know, you have this async await notation, which is compiling down to these really awesome  zero cost futures. And so it was sort of released futures into the wild and got feedback.  And that's where we ran into problems. That essentially anyone who tried to use futures  quickly ran into very confusing error messages, where it would just kind of complain about how  your future isn't static or doesn't implement this trait. And it would be sort of this baffling  thing you didn't really understand. And the compiler would like make helpful suggestions,  which you would just follow until eventually compiled. And so you would be like annotating  closures with move, and you would put things into reference counted pointers and clone things and  this and that. And it all felt like you were adding all this overhead to the thing that you  didn't seem necessary. You didn't understand why you had to do it. And also when you were done,  your code ended up looking like soup. And so tons of people were bouncing off of futures. And it  didn't help that the combinators produced these really huge types where your entire terminal  would be filled up with the type of one of your combinator chains. Or it would just be like,  you know, an end then of an end then of a map error of a TCP stream and so on. And you know,  you have to dig through this to try to figure out what the actual error that you encountered was.  And I found this quote on Reddit, which I think really beautifully sums up all of the  complaints about futures, which is that, you know, when using futures, the error messages are  inscrutable. Having to use ref cell or clone everything for each feature leads to over company  code. And it makes me wish that Russ just had garbage collection, which is, yeah, not great  feedback. So looking at the situation maybe a year, a year and a half ago, it was clear that  there were sort of two problems that needed to be solved in order to make futures more usable  for people. And the first was we needed better error messages. And so the easiest way to do that  is to build the syntax into the language and then they can hook into all of our diagnostics and  error message, you know, support so that you can have really good error messages for async await.  But the second was that most of these errors people are running into were actually  them bouncing off a sort of obscure problem, which is called the borrowing problem, where  there was this fundamental limitations in the way that futures were designed that  made it something really common pattern was not possible to express. And that problem was that  you can't in the original line of futures, you could not borrow across an await point. So if you  were to await something, you couldn't have any references that were alive at that time. And  so when people were having these problems that they were bouncing off of what they were actually  ultimately what they were doing was they were trying to borrow while they were awaiting and  they couldn't do that. And so if we could just make it so that was allowed, then most of these  errors would go away and everything would be much easier to use. And you could just kind of write  normal REST code with async and await and it would all work. And these kinds of borrows during  await are extremely common because just the natural API service of Rust is to have references in the  API. And so, but the problem with futures is that when you actually compile the future, which has  to restore all that state, when you have some references to something else that's in the same  stack frame, what you end up getting is the sort of self-referential future. And so here's the some  code from the original like get user method where we have this SQL string and then when we call  query with it, we use pass a reference to the SQL string. And so the problem here is that  this reference to the SQL string is a reference to something else that's being stored in the same  future state. And so it becomes the sort of self-referential struct where you have these are  the fields of the future in theory if it were a real struct, you would have the database handle  that you were in the self aspect of it. Then we'd also have the SQL string and the reference to  the SQL string, which is ultimately a reference pointing back to a field of the same struct.  And self-referential structs are sort of a really hard problem that we don't have a general solution  for. And the reason that we can't allow you to have references to the same struct is that when you  move that struct, then what happens is that we make a new copy of the struct in the location  that you're moving it to and the old copy becomes invalidated. But when you make that copy, the  reference that was self-referential is still pointing to the old copy. And that's becomes a  dangling pointer and it's the kind of memory issues that Rust has to prevent. So we can't have  self-referential structs because if you move them around, then they become invalidated.  But what made this really frustrating in the futures case was that  we actually don't really need to move these futures around. So if you remember the model  where the future is in the heap and a handle to it is getting passed back and forth between the  reactor and the executor, the future itself never actually moves. And so it's totally fine for the  future to contain self-references as long as you never move it and you don't need to move it. So  we really needed to solve this problem with some way to express in the API of futures that while  you're pulling it, you're not allowed to move it around. And then if we just could express that  somehow, then we could allow these kinds of self-references in the body of the future and then  we could just have these references in your async functions and everything would work.  And so we were working on this problem and ultimately we came out with this  new API called PIN. And so PIN is a sort of adapter around other pointer types where they  become this append reference instead of just a normal reference. And append reference, in addition  to whatever other guarantees it has, it guarantees that the reference will never, the value of the  reference is pointing to will never be moved again. And so it guarantees that it's going to stay in  the same place until eventually it gets deallocated. And so if you have something in your API which  says that it has to be taken by PIN, then you know that it will never be moved again and you can have  these kinds of self-referential structs. And so we changed the way that futures work so that now  there's just being a boxed future, it's a boxed future behind a PIN. So we know that wherever we  boxed it up, put it in the heap, it's guaranteed now by part of the PIN API that it will never move  again. And then, you know, when you pull the future instead of just passing a normal reference to it,  we pass the PIN reference to it. And so the future knows that it can't be moved. And the trick here  that makes this all work is that you can only get an un-PIN reference out of a PIN reference in  unsafe code. It's an unsafe function to be able to do that. And so the API looks roughly like this,  where you have PIN which is just, you know, a wrapper around a pointer type. It doesn't  have any runtime overhead or anything. It just like demarcates it as being PIN. And then  a PIN to box can be converted into a PIN reference. But the only way to convert a PIN reference  into an un-PIN reference is to use an unsafe function. And so what we did was then we just  changed the futures API so that instead of taking a normal reference, it takes a PIN reference. And  otherwise, the API is the same as it was before. And this is essentially the API that we're going  to be stabilizing. And so with that change, this code from the first example just works the way  that it's written. And so you can just write code exactly the way you would write it with  blocking I.O. Add these async and await annotations. And then what you get is this, you know,  async I.O. with this really awesome zero cost abstraction where it's basically as cheap as  if you hand wrote the state machine yourself by hand. So the situation today, PINning was  stabilized in the last release about a month ago. We're in the process of stabilizing the future  API. So probably in 135, maybe it will slip in B136, which is, you know, in about two or three  months basically. And then we're hoping sometime this year we'll have async await stabilized,  hopefully by the end of summer even, we're going to have this stabilized and so that people will  be able to write non-blocking I.O. network services using this syntax that makes it very  similar to writing with blocking I.O. Looking beyond that stabilization, we're also already  starting to work on these sort of more long-term features like streams, I think is probably the  next big one where it's an async future. So a future is just, you know, one value, but a stream  is many values being yielded asynchronously. It's essentially like an asynchronous iterator.  And so you'll be able to like loop asynchronously over a stream and things like that. And it's  very important for like a lot of use cases where you have like, you know, streaming,  HTTP, WebSockets, HTTP to push requests, that kind of thing where instead of having a networking,  like RPC model where you, you know, make a network request and get a single response,  you have streams of responses and requests going back and forth.  There's also today a limitation that async fn can't be used in traits. There's a lot of work  being done in the compiler to make it so that it would be able to support that. And looking  out even beyond these features, someday we want to have generators where, sort of similar to  generators in like Python or JavaScript, where instead of just having functions that return,  you can also have functions that yield. And so then you can resume them again after the yield and  you can use these to write functions as a way of writing iterators and streams, similar to how  an async function lets you write a function that's actually a future. But I guess just sort of  recapping the real critical insights that led to this sort of zero cost async Ion model where  that first was this pull based version of futures, where we were able to compile these  futures into these really tight state machines. And then secondly, this way of doing async  wait syntax where we're able to have references across the wait points because of pinning.  Supposed to say thank you, I don't know how it became an X.  Thank you.
RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_0_47.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_48_371.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_372_563.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_564_2015.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_2016_2111.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_2112_8519.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_8520_8975.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_8976_14063.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_14064_17147.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_17148_17399.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_17400_22811.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_22812_22871.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_22872_31055.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_31056_31139.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_31140_31355.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_31356_34703.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_34704_34787.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_34788_34883.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_34884_35087.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_35088_37955.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_37956_38459.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_38460_44531.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_44532_46199.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_46200_46607.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_46608_47243.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_47244_47471.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_47472_48347.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_48348_48407.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_48408_48479.png RustLatam_2019_-_Without_Boats__Zero-Cost_Async_IO_skos4B5x7qE_48480_48597.png
Hey, everyone. Thanks for having me. Yeah, I'm, my name is Without Boats. As in, yeah, seen Boatess. You can, most people call me Boats, which is actually even more confusing. I am a researcher at Mozilla. I work on Rust. It's my full-time job. It's pretty cool. I think, so this talk is going to be about sort of, it's a feature that I've been working on for about the last year and a half. And before me, other people were working on for, I mean, honestly, since before, the very beginning of Rust, before 1.0. But before that, I just wanted to thank the organizers for having me and for all of their work in putting this conference on. I think, I, maybe someone's already said pointed this out, but I believe this is the first Rust conference outside of either the United States or Europe. And so, as someone who works on Rust, I'm really, like, excited and glad to see our global community, like, thriving and growing. And it's really cool to see all the conferences that are happening this year. Come on to the technical stuff. So, the feature that I've been working on is this thing called Async Await. It's sort of going to be probably the biggest thing that we do in the language this year. We're planning to ship it sometime in the next few months. And it's the solution to this problem that we've been struggling with for a really long time, which is how can we have a zero cost abstraction for asynchronous I.O. in Rust? So, I'm going to explain what zero cost abstraction means in a moment, but first, just to kind of give an overview of the feature. So, Async Await, it's just these two new keywords that we're adding to the language, Async and Await. And so, Async is this modifier that can be applied to, like, functions where now the function, instead of when you call it, it runs all the way through and returns. Instead, it returns immediately and returns this feature that will eventually result in whatever the function would return. And inside of an Async function, you can take the Await operator and apply it to other features, which will pause the function until those features are ready. And so, it's this way of handling asynchronous concurrent operations using these annotations that makes them much easier to write. So, here's a little code sample just to sort of highlight and explain the feature. This is in, like, basically just an adapter on, like, a kind of ORM type of thing, it's handwritten, where you have this getUser method, which takes a string for a username and then returns this, like, user domain object by querying the database for the record for that user. And it does that using Async.io, which means that it's an Async function instead of just a normal function. And so, when you call it, you can Await it. And then, just to walk through the body of this method, you just, the first thing is it creates the SQL query, interpolating the username into, you know, select from user's table. And then, we query the database. And this is where we're actually performing some I.O. So, query also returns a future because it's doing this Async.io. And so, when you query the database, you just add this Await in order to Await for the response. And then, once you get the response, you can parse a user out of it. This user domain object, which is, you know, part of your application. And so, this method is just sort of like a toy example for the talk. But what I wanted to highlight in it is that the only difference really between this and using Blocking.io are these little annotations where you just mark the functions as being Async. And when you call them, you add this Await. And so, it's, you know, relatively little overhead for getting, using Non-Blocking.io instead of Blocking.io. And in particular, in Rust, the really great thing about our implementation that makes me really excited about it is that our Async Await and futures are this zero-cost abstraction. And so, zero-cost abstractions are sort of a really defining feature of Rust. It's one of the things that differentiates us from a lot of other languages is that we really care about when we add new features that they are zero cost. We didn't actually come up with the idea. It's a big thing in C++ also. And so, I think the best explanation is this quote from Bjarne Streestrup, which is that a zero-cost abstraction means that when you don't use it, you don't pay for it. And further, when you do use it, you couldn't hand-code it any better. And so, there's these two aspects to a zero-cost abstraction, which is that the first is that the feature can't affect, can't add cost to people who aren't using the feature. So, we can't add this global cost that will slow down every program in order to enable this feature. And the second is that when you do use the feature, it can't be slower than if you didn't use it and then you feel like, oh, well, I would like to use this really nice feature that makes it easier, but it will make my program slow. And so, I'll just write this thing by hand. It will be a much bigger pain. And so, I'm going to kind of walk through the history of how we've tried to solve async.io and Rust. And some of the steps along the way we had features that failed the zero-cost test on both principles. So, the specific problem that we're trying to solve is async.io. And so, normally, I.O. is blocking. So, when you use I.O., you'll block thread and that will stop your program and then have to be rescheduled by the OS. And the problem with blocking I.O. is it just doesn't really scale when you have been trying to serve a lot of connections from the same program. And so, for these kinds of really high-scale network services, you really need some form of non-blocking or asynchronous I.O. And Rust in particular is supposed to be designed for languages that have these really high performance requirements. You know, it's a systems programming language for people who really care about the computing resources they're using. And so, for Rust to really be successful in the network space, we really need some sort of solution to this asynchronous I.O. problem. But the big problem with async.io is that the way it works is that when you call the I.O. system call, it just returns immediately and then eventually you can continue doing other work. But it's your program's responsibility for figuring out how to, like, schedule, calling back to the task that you had to pause on when you're doing the asynchronous I.O. And so, this makes, you know, writing an async I.O. program much more complex than writing one of these is blocking I.O. And so, a lot of languages that are trying to target these like scalable network services have been trying to come up with solutions for this problem that like take it, make it not the end-users problem, but a part of the language or part of the libraries. And so, the first solution that Rust started with was the study of green threads, which have been successful in a lot of languages. And so, green threads basically look like blocking I.O. They look like spawning threads and then just blocking on I.O. And everything looks exactly the same as if you were using the native OS primitives, but they've been designed as part of the language runtime to be optimized for this use case of having these network services which are trying to spawning, you know, thousands, tens of thousands, maybe millions of green threads at the same time. A language I think right now that is like having a lot of success with this model is Go, where they're called Go routines. And so, it's very normal for a Go program to have tens of thousands of these running at the same time because they're very cheap to spawn unlike OS threads. And so, just the advantage of green threads is that the memory overhead when you spawn an OS thread is much higher because you create this huge stack for each OS thread, whereas green threads, the way they normally work is that you will spawn a thread that starts with a very small stack that can grow over time. And so, spawning a bunch of new threads that aren't using a lot of memory yet is much cheaper. And also, a problem with using like the operating system primitives is that you depend on the operating system for scheduling, which means that you have to switch from your program's memory space into the kernel space and the context switching as a lot of overhead if you want to start having, you know, tens of thousands of threads that are all being like switched between really quickly. And so, by keeping that scheduling in the same program, you avoid these context switches and that really reduces the overhead. And so, green threading is a pretty good model that works for a lot of languages, both Go and Java, I believe, use this model. And Rust had it for a long time, but removed it shortly before 1.0. And the reason that we removed it is that it ultimately was not a zero cost abstraction, specifically because of the first issue that I talked about where it was imposing costs on people who didn't need it. So, if you just wanted to write a Rust program that didn't need to use green threads, it wasn't a network service. You still had to have this language runtime that was responsible for scheduling all of your green threads. And so, this was especially a problem for people who were trying to embed Rust inside of like a larger C application. It's one of the ways that we've seen a lot of success in people adopting Rust is that they have some big C program and they want to start using Rust. And so, they start integrating a bit of Rust into their program. It's just writing one section of the code in Rust. But the problem is that if you have to set up this runtime in order to call the Rust, then it's too expensive to just have a small part of your program in Rust because you have to set up the runtime before you can call the Rust functions. And so, shortly before 1.0, we removed green threads from the language. We removed this language runtime and we now have a runtime that is essentially the same as C. And so, it makes it very easy to call between Rust and C and it's very cheap, which is one of the key things that makes Rust really successful. And having removed green threads, we still needed some sort of solution to async.io. But what we realized was that it needed to be a library-based solution. We needed some sort of with providing a good abstraction for async.io that was not a part of the language, it was not a part of this runtime that came with every program, but was just this library that you could opt into and use when you needed it. The most successful library-based solution in general is this concept called futures. It's also called promises and JavaScript. So, the idea of a future is that it represents a value that may not have evaluated yet. And so, you can manipulate it before you actually have, before the future is actually resolved. Eventually, we'll resolve to something, but you can start running things with it before it's actually resolved. And there's a lot of work done on futures in a lot of different languages and they are a great way for supporting a lot of combinators and especially this async await syntax that makes it much more ergonomic to build on top of this concurrency primitive. And so, futures can represent a lot of different things. So, async.io is kind of the biggest, the most prominent one where you maybe make a network request and you immediately get a future back, which once the network request is finished, will resolve into whatever that network request is returning. But you can also represent things like timeouts where a timeout is just a future that will resolve once that amount of time has passed. And even things that aren't doing in the I.O. or anything like that, where just CPU intensive work, you can run that on like a thread pool and then just get a future that you hold on to that once the thread pool is finished doing that work, the future will resolve. The problem with futures was that the way that they've been represented in most languages is this callback based approach where you have this future and you can schedule a callback to run once the future resolves. And so the future is responsible for figuring out when it resolves and then when it resolves it runs whatever your callback was. And all these distractions are built on top of this model. And this just really didn't work for us because there's a lot of people experiment with it a lot and found that it just was forcing way too many allocations. Essentially every callback that you tried to schedule had to get its own separate like create object heap allocation. And so there were these allocations everywhere, these dynamic dispatches, and this like approach failed the zero cost abstraction on the second principle where now you were, it was not affecting people who weren't using it, but if you did use it, it would just be way slower than if you just had it written something yourself. And so why would you use it? Because if you wrote the thing yourself would be much faster. But I hope it's not lost. Sorry, I'll set up a bit of a cold. This really great alternative abstraction that was pole based. Yes, we've arrived at this model. I really want to give credit to Alex who's here and Aaron Turan who came up with this idea where instead of futures scheduling a callback, there instead you pull them. And so there's this other part of the program called an executor, which is responsible for actually running the futures. And so what the executor does is it pulls the future. And the future either returns pending that it's not ready yet, or once it is ready, it returns ready. And this model has a lot of advantages. One advantage is that you can just cancel futures very simply because all you do to cancel them is you stop polling them. Whereas with this callback based approach, it was really difficult once you've scheduled some work to cancel that work and haven't stopped. It also really enabled us to have this really clean abstraction boundary between different parts of the program. So most futures libraries come with an event loop. And your futures are scheduled across this event loop, but this way of doing I.O. and you really don't have any control over it. But in Rust, we have this really clean boundary between the executor, which schedules your futures, the reactor, which handles all of the I.O. and then your actual code. And so the end user can decide which executor they want to use, which reactor they want to use, giving them the kind of control that is really important in a systems language. But the real advantage, the most important thing about this model is that it enabled us to have this really perfect zero cost way of implementing futures, where each representative is this kind of state machine. And so the way this works is that when the futures, the code that you write gets compiled down to, it gets actually compiled into native code, where it is this kind of like state machine where it has one variant for each pause point, for each I.O. event, essentially. And each variant then has the state that it needs to resume from that I.O. point. And that is represented as this essentially like an enum, where it's this one structure where it's the variant, the discriminant, and then a union of all of the states that it could possibly need. And so this is an attempt to visually represent that abstractly, where this is a, this state machine has, you know, you perform two I.O. events, and so it has these different states. And each state it has this, the amount of like, space it needs to store everything you would need to restore to that state. And the entire future is just a single heap allocation that's that size, where you just allocate that state machine into one place and a heap. And it's just no additional overhead. So you don't have these like, all these boxed callbacks and things like that. You just have this like perfect, really truly zero cost model. So I feel like that is usually a bit confusing to people. So I tried to, this is my best keynote, visually represent what's going on, which is that the, so you spawn a future and that puts the future in the heap in this one location. And then if there's a handle to it, that's you start with the executor, the executor pulls the future until eventually the future needs to perform some sort of I.O. In which case, the future gets handed off to the reactor. And the reactor, which is handling the I.O., registers that the future is waiting on this particular I.O. event. And then eventually when that I.O. event happens, the reactor will wake up the future using the waker argument that you passed in when you pulled it. And so waking the future up, passes it back to the executor. And then the executor will pull it again. And it will just go back and forth like this until eventually the future resolves. And so then when the future finally resolves and evaluates its final result, the executor knows that it's done. And then it drops the handle and drops the future and the whole thing is finished. And so it forms this sort of cycle where you pull the future, wait for I.O., wake it up again, pull it again, on and on until eventually the whole thing is finished. And this model ended up being quite fast. This is the sort of the benchmark that was posted in the first post about futures, where benchmarked features against a lot of different implementations from other languages. Hire is better and futures is the one on the far left. So we had this really great zero cost abstraction that was competitive with the fastest implementations of async I.O. in a lot of other languages. But of course the problem is that you don't want to write these state machines by hand. You have your whole entire application status of state machine is not very pleasant to write. But that's where the future abstraction is really helpful is that we can build these other APIs on top of it. And so the first solution that we had was this idea of futures combinators where you can build up these state machines by applying all of these methods of the future, sort of similar to the way that iterator adapters like filter and map work. And so this function, it just, what it does is it requests rustling.org and then converts that the response to a string. So instead of returning just a string, it returns a future of a string because it's going to be an async function. And it has these futures in the body that it's going to be calling and those are going to actually be the parts that do some I.O. And then they're all sort of combined together using these combinators like and then in map. And we build all these combinators like and then map, filter, map error, like all kinds of different things. And this works. It has some downsides, especially these like nested callbacks, which can be really difficult to read sometimes. And so because you know it has these downsides, we also started at it, tried to implement an async await implementation. And so the first version of async await was not part of the language. Instead, it was this library that provided this through like a syntax plugin. And this is doing the same thing that the previous function did. It just fetches rustling and turns it to a string. But it does so using async await. And so it's much more like straight line, looks much more like the way normal blocking I.O. works, where just like in the example that I showed originally, the only real difference is these annotations. And so the async annotation turns this function into a future instead of just returning immediately. And then the await annotations await on these on the futures that you actually construct inside of the function. And await under this poll model, desugars to this sort of loop where what you do is you just pull in a loop. And every time you get pending back, you yield all the way back up to the executor that you're pending. And so then it waits until it gets woken up again. And then when finally the future that you're awaiting finishes, it finishes with the value and you break out of the loop with the value. And that's what these await expressions evaluate to. So this seemed like a really good solution. You know, you have this async await notation, which is compiling down to these really awesome zero cost futures. And so it was sort of released futures into the wild and got feedback. And that's where we ran into problems. That essentially anyone who tried to use futures quickly ran into very confusing error messages, where it would just kind of complain about how your future isn't static or doesn't implement this trait. And it would be sort of this baffling thing you didn't really understand. And the compiler would like make helpful suggestions, which you would just follow until eventually compiled. And so you would be like annotating closures with move, and you would put things into reference counted pointers and clone things and this and that. And it all felt like you were adding all this overhead to the thing that you didn't seem necessary. You didn't understand why you had to do it. And also when you were done, your code ended up looking like soup. And so tons of people were bouncing off of futures. And it didn't help that the combinators produced these really huge types where your entire terminal would be filled up with the type of one of your combinator chains. Or it would just be like, you know, an end then of an end then of a map error of a TCP stream and so on. And you know, you have to dig through this to try to figure out what the actual error that you encountered was. And I found this quote on Reddit, which I think really beautifully sums up all of the complaints about futures, which is that, you know, when using futures, the error messages are inscrutable. Having to use ref cell or clone everything for each feature leads to over company code. And it makes me wish that Russ just had garbage collection, which is, yeah, not great feedback. So looking at the situation maybe a year, a year and a half ago, it was clear that there were sort of two problems that needed to be solved in order to make futures more usable for people. And the first was we needed better error messages. And so the easiest way to do that is to build the syntax into the language and then they can hook into all of our diagnostics and error message, you know, support so that you can have really good error messages for async await. But the second was that most of these errors people are running into were actually them bouncing off a sort of obscure problem, which is called the borrowing problem, where there was this fundamental limitations in the way that futures were designed that made it something really common pattern was not possible to express. And that problem was that you can't in the original line of futures, you could not borrow across an await point. So if you were to await something, you couldn't have any references that were alive at that time. And so when people were having these problems that they were bouncing off of what they were actually ultimately what they were doing was they were trying to borrow while they were awaiting and they couldn't do that. And so if we could just make it so that was allowed, then most of these errors would go away and everything would be much easier to use. And you could just kind of write normal REST code with async and await and it would all work. And these kinds of borrows during await are extremely common because just the natural API service of Rust is to have references in the API. And so, but the problem with futures is that when you actually compile the future, which has to restore all that state, when you have some references to something else that's in the same stack frame, what you end up getting is the sort of self-referential future. And so here's the some code from the original like get user method where we have this SQL string and then when we call query with it, we use pass a reference to the SQL string. And so the problem here is that this reference to the SQL string is a reference to something else that's being stored in the same future state. And so it becomes the sort of self-referential struct where you have these are the fields of the future in theory if it were a real struct, you would have the database handle that you were in the self aspect of it. Then we'd also have the SQL string and the reference to the SQL string, which is ultimately a reference pointing back to a field of the same struct. And self-referential structs are sort of a really hard problem that we don't have a general solution for. And the reason that we can't allow you to have references to the same struct is that when you move that struct, then what happens is that we make a new copy of the struct in the location that you're moving it to and the old copy becomes invalidated. But when you make that copy, the reference that was self-referential is still pointing to the old copy. And that's becomes a dangling pointer and it's the kind of memory issues that Rust has to prevent. So we can't have self-referential structs because if you move them around, then they become invalidated. But what made this really frustrating in the futures case was that we actually don't really need to move these futures around. So if you remember the model where the future is in the heap and a handle to it is getting passed back and forth between the reactor and the executor, the future itself never actually moves. And so it's totally fine for the future to contain self-references as long as you never move it and you don't need to move it. So we really needed to solve this problem with some way to express in the API of futures that while you're pulling it, you're not allowed to move it around. And then if we just could express that somehow, then we could allow these kinds of self-references in the body of the future and then we could just have these references in your async functions and everything would work. And so we were working on this problem and ultimately we came out with this new API called PIN. And so PIN is a sort of adapter around other pointer types where they become this append reference instead of just a normal reference. And append reference, in addition to whatever other guarantees it has, it guarantees that the reference will never, the value of the reference is pointing to will never be moved again. And so it guarantees that it's going to stay in the same place until eventually it gets deallocated. And so if you have something in your API which says that it has to be taken by PIN, then you know that it will never be moved again and you can have these kinds of self-referential structs. And so we changed the way that futures work so that now there's just being a boxed future, it's a boxed future behind a PIN. So we know that wherever we boxed it up, put it in the heap, it's guaranteed now by part of the PIN API that it will never move again. And then, you know, when you pull the future instead of just passing a normal reference to it, we pass the PIN reference to it. And so the future knows that it can't be moved. And the trick here that makes this all work is that you can only get an un-PIN reference out of a PIN reference in unsafe code. It's an unsafe function to be able to do that. And so the API looks roughly like this, where you have PIN which is just, you know, a wrapper around a pointer type. It doesn't have any runtime overhead or anything. It just like demarcates it as being PIN. And then a PIN to box can be converted into a PIN reference. But the only way to convert a PIN reference into an un-PIN reference is to use an unsafe function. And so what we did was then we just changed the futures API so that instead of taking a normal reference, it takes a PIN reference. And otherwise, the API is the same as it was before. And this is essentially the API that we're going to be stabilizing. And so with that change, this code from the first example just works the way that it's written. And so you can just write code exactly the way you would write it with blocking I.O. Add these async and await annotations. And then what you get is this, you know, async I.O. with this really awesome zero cost abstraction where it's basically as cheap as if you hand wrote the state machine yourself by hand. So the situation today, PINning was stabilized in the last release about a month ago. We're in the process of stabilizing the future API. So probably in 135, maybe it will slip in B136, which is, you know, in about two or three months basically. And then we're hoping sometime this year we'll have async await stabilized, hopefully by the end of summer even, we're going to have this stabilized and so that people will be able to write non-blocking I.O. network services using this syntax that makes it very similar to writing with blocking I.O. Looking beyond that stabilization, we're also already starting to work on these sort of more long-term features like streams, I think is probably the next big one where it's an async future. So a future is just, you know, one value, but a stream is many values being yielded asynchronously. It's essentially like an asynchronous iterator. And so you'll be able to like loop asynchronously over a stream and things like that. And it's very important for like a lot of use cases where you have like, you know, streaming, HTTP, WebSockets, HTTP to push requests, that kind of thing where instead of having a networking, like RPC model where you, you know, make a network request and get a single response, you have streams of responses and requests going back and forth. There's also today a limitation that async fn can't be used in traits. There's a lot of work being done in the compiler to make it so that it would be able to support that. And looking out even beyond these features, someday we want to have generators where, sort of similar to generators in like Python or JavaScript, where instead of just having functions that return, you can also have functions that yield. And so then you can resume them again after the yield and you can use these to write functions as a way of writing iterators and streams, similar to how an async function lets you write a function that's actually a future. But I guess just sort of recapping the real critical insights that led to this sort of zero cost async Ion model where that first was this pull based version of futures, where we were able to compile these futures into these really tight state machines. And then secondly, this way of doing async wait syntax where we're able to have references across the wait points because of pinning. Supposed to say thank you, I don't know how it became an X. Thank you.
researcher at Mozilla. I work on Rust. It's my that I've been working on for about the last year abstractions are sort of a really defining this just really didn't work for us because was this library that provided this through like is to build the syntax into the language and then is a sort of adapter around other pointer types to write non-blocking I.O. network services using