Life is but a Stream
Lessel, Geoffrey
A overview and dive into the Stream module. When should you use it over Enum? What are the pros and cons of using Stream for your day-to-day tasks? How are some of the functions implemented? We'll go into those questions and find some answers together. About Geoffrey: Geoffrey Lessel has been a web developer for over 15 years—first with PHP and Perl and now with Ruby on Rails and Phoenix. He has spoken at ElixirConf.
 So today, we're going to talk about something fun. But first, I want to say that I'm very honored to be here. I remember a couple of years ago when Impax, New York City, happened and I wanted to be there at the time and seeing like videos and feedback online about how awesome the conference was and how amazing like the surroundings were. I was very jealous that I wasn't going to be able to be there. So I am very honored to be here today at the first West Coast Impax and they delivered. Didn't they? I mean, this place is amazing. Everything around, like even the pianist in the background coming in is like, what the heck? We got live piano as we're walking into a conference, a tech conference. How cool is that? It's pretty cool. All right. So I'm Jeffrey Lussell. Today, we're going to talk about Life is But a Stream. We're going to dive a little bit into the stream module. So what I'm hoping to do today for you guys and gals is if, well, basically to compare Enum to stream and give you a couple of examples of when you might want to switch out Enum for stream and sometimes when you may not. So the title of the talk is Life is But a Stream and I apologize. I may murder this analogy over the course of the talk. So I apologize up front, but bear with me. So in the background here, we got this nice looking stream. It looks really nice. You want to kind of spend some time on that bench over there. There's a little bit of water trickling down over a little, I guess you can call it a waterfall, but then we've got another picture here, another stream, kind of fall time. The water is a little bit rushier. The water is getting a little crazier. But still, you wouldn't mind spending some time either in the stream or by the side walking by it. Then we've got this one, which I don't know where that is, but I want to be there. That's pretty amazing. It's a little rockier, so you may, if you try to float down that thing, you may hurt yourself, but it's beautiful surroundings. And then you can imagine that those little streams may go from that first one to the second one to this one to this, right? This could be where your stream ends up, and you're towering over this, or you're falling over this towering waterfall into that huge, maybe like a two-foot thing down there. You're going to die if you go over that thing, right? So this is where stream and enum can lead you. So if we'd like to take a look back at this stream, I'm going to come back to these pictures in a moment, but keep those in mind as we talk about stream and enum, and just the fact of how the water moves through those pictures, okay? All right, so the first thing is enum is awesome. So how many of you in here have written at least one line of Elixir code in your life? Okay, cool. So how many of you have written one thing, one line of Elixir that contained a function from enum, the enum module? Okay, cool. So the Venn diagram of those two groups are basically the same, right? Everything you do in Elixir, not everything, but almost everything you do in Elixir at some point is going to touch the enum module. That's because there's so many helpful things in there, and it's typical in functional programming that you're going to be dealing with collections, and those collections are going to be passed from one thing to the next, and you have to do something with those collections and transform the data somehow. And enum is basically the module where you do that. So I love enum. And stream is kind of like a souped up enum that can be switched out a lot of times directly with enum. But we'll get into that a little bit more in detail in a second. So for right now, I'm going to dive into an IEX session. Hopefully you guys can see this. Do you think it'd be better with the black on white or the white on black? Okay, cool. I'm going to take it just a little bit bigger. All right, so let's say we have, let me clear. Let's say we've got a list, and our list is just going to be three items. Okay, we're going to say Elixir, let's say Impax, and LA. Okay, so that's our collection, right? Just three items, not a big deal, three strings. Now what if we wanted to do something with each one of those items in the list? Now a lot of this is going to be kind of, I know how to use an enum, but we'll get kind of make this a little more complicated as we go on. So let's say we want to capitalize or upcase each one of those items in the list. So we can send that list, and I'm going to do it this way. We send that list to enum.map. And for each one of those, I'm going to use the shorthand notation here. I'm going to upcase every one of those. That happens pretty quickly, right? It takes Elixir. In your view, it's Elixir. So we've got Elixir, Impax, and LA all upcased. So basically what enum.map did is it took that whole list we gave it, all three items at once, and enum kind of put it in its head. It put it in its mind, in its memory, and said, okay, I have this list now on every one of these items, I need to do this one particular function. In this case, we told it to do string.upcase. Now of course, we can take a little further. We can say, okay, we want to do that, then we want to send it to the upcase. Sorry, I forgot to do the post-fix thing here. And then we want to send it to another map. We want to transform it again. And this time, we're going to just take the graph emes and get each individual letter. So now we have a collection of collections, right? Each one of those is broken up into their individual letters. And so that's fine. That works. So that's what enum does. It takes the whole list that goes to the first enum, gets it all in its memory, and says, okay, I'm done now. I'm going to send the new list that I have into the next enum. And then our final enum says, okay, now I'm going to take that string and break it up into its individual characters. So this is fine, right? This is a fast-performant, whatever. This is milliseconds of time to spend in code. Now there's a problem, though, and that's when you get into large collections. So let's go back to this stream, right? So we have this stream. Let's say you want to operate on every single piece of water model queue in this picture, okay, as it's going over the waterfall. We could probably do it. It's not that big a deal. It's just these little trickle. But then let's say you got this, and you want to operate on every single molecule of water that falls over that stream as it comes by. That is a large data collection. So for you guys and gals, if you're in Elixir, maybe on the web, you've got a log file, for example, that's very large, or maybe some user input, maybe a text of some book that you want to analyze. Your data collection can get pretty large. But in fact, even in the stream module definition or story documentation, it says that the data size could be infinite. So we have no idea how much water is going to come over this waterfall in the life of this waterfall, right? We don't know how long, you start your little web project, and you're like, this is going to live forever. And so the log file could grow infinitely big. Hopefully you do things to keep it from doing that. But it could grow infinitely big. You don't know when the end is going to be. So we could be operating on that log file to the end of eternity, right? You could be always, always, always operating. So stream is something where you can operate on that. On enum, it would have to get all of that into its head at once. So if we look at that waterfall, it's not very easy to think of every single piece of water molecule in that waterfall, right? That's just a lot of stuff to put in your head. Same thing with enum. That's a lot of stuff to put in enum's head. So here's your solution, streams, OK? Let's go back to IEX. In this case, I'm just going to do a range. Let's say 1 to 10 million. And for each one of those things, we're going to pipe that into another enum. And we're going to map it. Let's say we're going to change it, change them to strings. To string. OK. So I hit enter, and it's taking a little while. That's because what it has to do is it has to take that entire list of 10 million, put it into its brain, and on each one of those say, OK, the first one's going to be a string of 1, and the second one's going to be a string of 2. All that whole list is going into its memory, and then it's going to spit it back out. So let's say, again, we want to take it one step further, and we want to pipe that into another map that changes into the graphemes. Now this is going to take a little while. So let's pretend I am that final enum.map. OK. I'm sitting here looking up at the pipeline. At the very top, I see the list of 10 million items. It's huge. All right. And then you can kind of hear the sound as it moves to the pipeline. So it goes to enum.map, and we've got the two strings. And two strings, like, I've got this. This is my time to shine. So enum two string goes, here's the strings. I'm piping them out, piping them out. And so it's got that whole thing that's minuses. I'm done. It's your turn. And I'm like, oh crap, here comes this huge list. So I've got to get my mind ready, clear it, and say I've got to allocate all this memory now to figure out how to operate on this huge list. 10 million may mean that huge. But it took a little bit of time to operate on. If your user was waiting for that, they would probably go to some other site and buy some other thing. So that's what enum does. It takes all that into memory, operates on each one of those huge collections at once, and then spits it down to the next one, spits it down to the next one. So stream, what stream does is something a little different. In this case, we've been using enum.map. There is also a stream.map. So let's just go in and replace stream.map, or enum.map with stream.map. Yes, that's a good idea. Oh, you know what? Did that wrong, but that's OK. So one to 10 million, pipe that in to stream.map. And for those, we're going to do the integer.toString. And we're going to pipe that directly into stream.map again. And this time, the string.grapheams. OK, so now, remember how long the enum one took, right? So what happens if I press enter now? Are you guys expecting something real quick? Hopefully so, because boom! That's not what you wanted, is it? You wanted the list. You wanted the list of the graph themes, right? You wanted those characters, but now we've got this weird thing. It's hard to see because it's cut off on the side a little bit. Let me do this. Boom! Yeah, I'll just do that. And then now the bottom's gone a little bit. All right, there we go. That works. OK, so now we have that stream, and we have a list inside stream of enum. It's keyword list. So we have our enum in there. And then we have funds, and that sounds like a fun thing, right? We've got funds. But inside funds is another list, and they are two different things, and they're functions. So what this returns to us is not what we were expecting. At least it wasn't what I was expecting when I first thought, well, I'll just drop in stream, and then nothing came out. And I was like, why not? Stream in order to operate on the pipeline has to have something at the end that actually requests something. So in typical cases, you would use a function from the enum module. In this case, you could use to list or take or something like that. But the very end of your stream has to actually request something. So streams are what's known as lazy. They don't actually execute and sell a need for it to execute. So that's kind of where the life is but a stream is. So you're like streaming, you're like, I don't know, what are those like donut inflatable things called? I'm blinking right now. But anyway, you're in though, and though it's inner tube. You're floating down the stream in an inner tube, right? That's lazy. You're just kind of, you know, whatever. So that's kind of what stream does. It's like, I'm not going to execute anything yet. I'll set all my stuff up. But you don't need anything. Why would I work, right? Forget you guys. I'm just going to sit here. So what if we need to take something? So let's go through that again. To string and then the graph themes. And then let's do enum.toolist. Now so what are you expecting here? I already pressed inner. Sorry, I should have gone up again. I already pressed inner. So you might be expecting that this is going to take hardly any time at all. But in fact, you see, well, this isn't hardly any quicker than the enum. It's still taking a long time. So this is where you have to really discover what you're trying to do with your collections in order to learn how to do stream correctly. Because at first, I thought it was just going to straight drop in and I would get these huge speed bumps. But in this case, the enum.toolist is still requesting the entire list of items, the entire collection of 10 million things, right? It's saying, I just want everything you got, and I'm just going to put it into a list. Now what if we wanted to take only four of those things? We've got a collection of 10 million items, but really we only want four of them. Maybe it's a list of questions or it could be a log file. We just want the last four things. In that case, if we did the enum thing, the whole list, again, we'd go, gjong, gjong. And then it would go, gjong to me. And I'm like, thanks for these 10 million items. I only wanted four. You jerk, right? So you'd like, I got these four. You forget your other ones. What Stream does is, let's say we have a stream at the bottom, or a stream pipeline. So let's pipe these things through again. One through 10 million, two string. We got the graph themes. And from there, we're going to take four. All right. So again, you might think this is going to take forever because an enum it would, but now, wow, right? Real quick. That didn't take hardly any time at all because here's what's happening now. Instead of enum pushing all those lists down the pipeline, Stream goes at the very bottom. This enum take says, I need one. And so at the very top, that range goes, OK, I'm spitting down one. And then it goes down into the first stream function, stream map function, the energy to string. And it goes, OK, one to string is the string one. It passes that down to the next function. And it poops that out, right? All the way down to the enum take one, or take four. And enum take four is like, all right, I got one of my four. I need two. And so the second one comes down, three, four. And once it gets to four, it's like, I'm done. Stream's like, well, I set up all these 999,996 others for you. Are you not going to use them? And I'm like, no, I don't need them. It's like, OK, thanks. You saved us a bunch of work. So that's where Stream actually makes a huge difference in collections like this, is when you only need a subset of the entire collection. So let's take a look again at my slides here. Streams are lazy, again, as I said. So I thought, well, let me take some benchmarks because then I was like, it all depends on your code, right? So let's use some benchmarks. So in this case, I set up something where I had four functions in a pipeline. And I thought, OK, well, I'm going to pipe those through in the functions through each other. And at the very end, I'm going to call enum.toolist and see how long it takes. So I piped in 1,000 items. And the enum iterations per second was almost 1.6,000, almost 1,600 a second. Stream was actually just under 1,200 a second. And so I thought, well, that's weird. So in this case, stream is actually 1.34 times slower than enum. Interesting. So why is that? Let's say, OK, well, maybe I just need a bigger data set. So I got passed in 100,000 items. Enum is 11.53 a second. Stream is 7,000 a second. I'm sorry, that should be 11.53 is the wrong number. But stream actually happens to be 1.57 times slower than enum. Again, the enum, I copied that wrong. Sorry. But it's slower. Enum, the stream is still slower. So let's go to 10 million items. Enum, 0.04 iterations per second. Stream is 0.38 iterations per second. And stream is 1.29 times slower. Even with a large data set, if I'm just at the very end calling to list, there's really a detriment for using stream. And that's where you think, well, why is that? Didn't you just say it was like a drop-in? Kind of. So enum and stream are two different things. You notice that when we created that pipeline, but we didn't actually take anything yet, it returned us this weird data structurey thing, right? And said, here's what I've set up for you, but we haven't operated on it yet. So all that set up actually takes some computational power. And so that's the case where it may not make much sense to do the computation and using the power to set up the stream pipeline if all you're doing is taking the entire list at once. Because the stream is like, well, I'm doing all this work for you, but you could have used enum and actually been a bit quicker. So I thought, OK, well, I've got four functions. That's not a whole lot of functions in the pipeline. What if I send it through eight functions, a pipeline through eight different functions, one with enum.map and another with stream.map? So in this case, I had 1,000. Enum did 581. Stream did 527. Again, stream is 1.1 times slower. 400,000. Stream got a little quicker, so it's 1.07. And then in 10 million, stream went backwards a little bit, but that could just be because of some deviation. So in this case, stream, again, is slower. They're getting closer to where you wouldn't notice much difference between enum and stream. So here's where the big stuff comes in. So we have eight functions that we're going to pass through. So we have a long function list. But at the very end, instead of just calling two lists, what if we just needed to take 10 of those things? So for 1,000 of them, enum can do almost 560 a second, whereas stream can do almost 74,000 a second. So enum is 132 times slower than stream. This is where stream starts to show its usage. For 100,000, enum is 13,000 times slower. And then for 10 million items, enum is almost 2.4 million times slower than stream. That's because, as we mentioned, enum has to contain that whole entire collection of 10 million items in its brain in each step of the way. Another way we can illustrate this, let's go back to our list here. Okay, list equals elixir, impacts, and LA. So what I'm going to do now is, in each step of the pipeline, I'm going to have our pipeline basically spit out what its current state is, like what the current collection is. So we're going to take our list. I'm just going to do it this way. So we're going to enum map our list. And for each one, we're going to just inspect it. And from there, we're going to pipe it into the first thing we did, which was our string.upcase. And then let's pipe that in again to map. And we're going to inspect it again. And then finally, we're going to do our graphemes. Okay, so in this case, you can see here, our first inspect goes through and has the entire collection, right? Our three items are impacts, or sorry, elixir, impacts, and LA. And we see all three of those come out first. That's coming out from this right here, our first thing. Then we upcase them. And you can see again that we have our upcase collection all at once. Now what happens if we change that up to stream? So let's say stream.map. And we're going to spec those. And we're going to pipe that into stream.map. We're going to do our string upcase. I'm sorry? Oh, yeah, thanks. Stream.map, and then we're going to inspect it again, just like we did with the map version. And then we're going to map those to our graphemes. And then at the very end, since stream won't actually operate this until we request it, we're going to call enum.toList. Okay, so in here, my inspect, oh, inspect. Awesome. So let's not inspect. Let's inspect. Upcase, and then we're going to inspect there, and then we're going to at the very end. Okay, so in this case, remember the first time we saw all three lowercase items first, and then all three uppercase items. In this case, stream is saying, I'm ready for the first item, and it sends Elixir down the pipeline, right? It drips it down. So the first one comes Elixir lowercase, which we expected. But the second item is actually our uppercase Elixir. So we can see that that actually did just drip that first one all the way down to us. And then the second item was our impacts, and that was drip down from lowercase to uppercase into our final graphemes, and the same thing with LA. So this is another way to look at the fact that each one of those items are kind of drip down as the final thing says, I'm ready for it. So this is where stream really comes into effect and really comes into its own as something that's very useful for your applications, is when you take a subset of items or you just need to make sure you're doing one at a time through your huge pipeline or in your huge data set. Yeah? Can you show us what it looks like again if you don't take that list again? Yeah, absolutely. So let's go through that again. So we're doing our list, inspect, and then we go to uppercase, and then we do another inspect, and then we go to graphemes. And instead of that, we're just going to end it. Is that what you're talking about? Yeah. All right. Then we have that stream thing. Whatever. Yes. Yeah. So the question was, can you take something that may be that infinite stream that you don't know when it ends, it may never end? Can stream trickle those down as you need them, even if it's not a complete set at the very top? And that answer is yes. So the key to keep in mind is there's, I mentioned that enum kind of finalizes that stream and takes them. There's another function in the stream module called run. And that actually is something where you can use it to execute the stream, but you may not need the final result of the stream. So in this case, let's say it's a log file, you need to write different things. So like, you do something to that log file line, and then write that to another file, maybe two files, maybe three files, maybe send it to an FTP server, whatever you need to do to it. And you don't need to really know the output of that pipeline. In that case, run can be used to do something that you don't really need to know the end result. All right. So that's a run through stream. Again, I am Jeffrey Lessle. I'm writing a book called Phoenix in Action. I should have some discount codes for you guys later today, I'm hoping. But you can buy it there, jeffreylessle.com slash phoenixinaction. It's in Meep right now, Manning's Early Access Program. I think there's two chapters available, but the third should be available sometime hopefully next week, actually. This is where you can find me online, jeffreylessle.com. I do some blogging about Elixir and Ecto and stuff like that. I'm Twitter, Geo-Lessle, and then GitHub at Geo-Lessle as well. So thank you guys so much. Thanks a lot, chef.
 So today, we're going to talk about something fun.  But first, I want to say that I'm very honored to be here.  I remember a couple of years ago when Impax, New York City, happened and I wanted to be  there at the time and seeing like videos and feedback online about how awesome the conference  was and how amazing like the surroundings were.  I was very jealous that I wasn't going to be able to be there.  So I am very honored to be here today at the first West Coast Impax and they delivered.  Didn't they?  I mean, this place is amazing.  Everything around, like even the pianist in the background coming in is like, what the  heck?  We got live piano as we're walking into a conference, a tech conference.  How cool is that?  It's pretty cool.  All right.  So I'm Jeffrey Lussell.  Today, we're going to talk about Life is But a Stream.  We're going to dive a little bit into the stream module.  So what I'm hoping to do today for you guys and gals is if, well, basically to compare  Enum to stream and give you a couple of examples of when you might want to switch out Enum  for stream and sometimes when you may not.  So the title of the talk is Life is But a Stream and I apologize.  I may murder this analogy over the course of the talk.  So I apologize up front, but bear with me.  So in the background here, we got this nice looking stream.  It looks really nice.  You want to kind of spend some time on that bench over there.  There's a little bit of water trickling down over a little, I guess you can call it a waterfall,  but then we've got another picture here, another stream, kind of fall time.  The water is a little bit rushier.  The water is getting a little crazier.  But still, you wouldn't mind spending some time either in the stream or by the side walking  by it.  Then we've got this one, which I don't know where that is, but I want to be there.  That's pretty amazing.  It's a little rockier, so you may, if you try to float down that thing, you may hurt yourself,  but it's beautiful surroundings.  And then you can imagine that those little streams may go from that first one to the  second one to this one to this, right?  This could be where your stream ends up, and you're towering over this, or you're falling  over this towering waterfall into that huge, maybe like a two-foot thing down there.  You're going to die if you go over that thing, right?  So this is where stream and enum can lead you.  So if we'd like to take a look back at this stream, I'm going to come back to these pictures  in a moment, but keep those in mind as we talk about stream and enum, and just the fact of  how the water moves through those pictures, okay?  All right, so the first thing is enum is awesome.  So how many of you in here have written at least one line of Elixir code in your life?  Okay, cool.  So how many of you have written one thing, one line of Elixir that contained a function  from enum, the enum module?  Okay, cool.  So the Venn diagram of those two groups are basically the same, right?  Everything you do in Elixir, not everything, but almost everything you do in Elixir at  some point is going to touch the enum module.  That's because there's so many helpful things in there, and it's typical in functional programming  that you're going to be dealing with collections, and those collections are going to be passed  from one thing to the next, and you have to do something with those collections and transform  the data somehow.  And enum is basically the module where you do that.  So I love enum.  And stream is kind of like a souped up enum that can be switched out a lot of times directly  with enum.  But we'll get into that a little bit more in detail in a second.  So for right now, I'm going to dive into an IEX session.  Hopefully you guys can see this.  Do you think it'd be better with the black on white or the white on black?  Okay, cool.  I'm going to take it just a little bit bigger.  All right, so let's say we have, let me clear.  Let's say we've got a list, and our list is just going to be three items.  Okay, we're going to say Elixir, let's say Impax, and LA.  Okay, so that's our collection, right?  Just three items, not a big deal, three strings.  Now what if we wanted to do something with each one of those items in the list?  Now a lot of this is going to be kind of, I know how to use an enum, but we'll get kind  of make this a little more complicated as we go on.  So let's say we want to capitalize or upcase each one of those items in the list.  So we can send that list, and I'm going to do it this way.  We send that list to enum.map.  And for each one of those, I'm going to use the shorthand notation here.  I'm going to upcase every one of those.  That happens pretty quickly, right?  It takes Elixir.  In your view, it's Elixir.  So we've got Elixir, Impax, and LA all upcased.  So basically what enum.map did is it took that whole list we gave it, all three items  at once, and enum kind of put it in its head.  It put it in its mind, in its memory, and said, okay, I have this list now on every one of  these items, I need to do this one particular function.  In this case, we told it to do string.upcase.  Now of course, we can take a little further.  We can say, okay, we want to do that, then we want to send it to the upcase.  Sorry, I forgot to do the post-fix thing here.  And then we want to send it to another map.  We want to transform it again.  And this time, we're going to just take the graph emes and get each individual letter.  So now we have a collection of collections, right?  Each one of those is broken up into their individual letters.  And so that's fine.  That works.  So that's what enum does.  It takes the whole list that goes to the first enum, gets it all in its memory, and says,  okay, I'm done now.  I'm going to send the new list that I have into the next enum.  And then our final enum says, okay, now I'm going to take that string and break it up  into its individual characters.  So this is fine, right?  This is a fast-performant, whatever.  This is milliseconds of time to spend in code.  Now there's a problem, though, and that's when you get into large collections.  So let's go back to this stream, right?  So we have this stream.  Let's say you want to operate on every single piece of water model queue in this picture,  okay, as it's going over the waterfall.  We could probably do it.  It's not that big a deal.  It's just these little trickle.  But then let's say you got this, and you want to operate on every single molecule of water  that falls over that stream as it comes by.  That is a large data collection.  So for you guys and gals, if you're in Elixir, maybe on the web, you've got a log file, for  example, that's very large, or maybe some user input, maybe a text of some book that  you want to analyze.  Your data collection can get pretty large.  But in fact, even in the stream module definition or story documentation, it says that the data  size could be infinite.  So we have no idea how much water is going to come over this waterfall in the life of  this waterfall, right?  We don't know how long, you start your little web project, and you're like, this is going  to live forever.  And so the log file could grow infinitely big.  Hopefully you do things to keep it from doing that.  But it could grow infinitely big.  You don't know when the end is going to be.  So we could be operating on that log file to the end of eternity, right?  You could be always, always, always operating.  So stream is something where you can operate on that.  On enum, it would have to get all of that into its head at once.  So if we look at that waterfall, it's not very easy to think of every single piece of  water molecule in that waterfall, right?  That's just a lot of stuff to put in your head.  Same thing with enum.  That's a lot of stuff to put in enum's head.  So here's your solution, streams, OK?  Let's go back to IEX.  In this case, I'm just going to do a range.  Let's say 1 to 10 million.  And for each one of those things, we're going to pipe that into another enum.  And we're going to map it.  Let's say we're going to change it, change them to strings.  To string.  OK.  So I hit enter, and it's taking a little while.  That's because what it has to do is it has to take that entire list of 10 million, put  it into its brain, and on each one of those say, OK, the first one's going to be a string  of 1, and the second one's going to be a string of 2.  All that whole list is going into its memory, and then it's going to spit it back out.  So let's say, again, we want to take it one step further, and we want to pipe that into  another map that changes into the graphemes.  Now this is going to take a little while.  So let's pretend I am that final enum.map.  OK.  I'm sitting here looking up at the pipeline.  At the very top, I see the list of 10 million items.  It's huge.  All right.  And then you can kind of hear the sound as it moves to the pipeline.  So it goes to enum.map, and we've got the two strings.  And two strings, like, I've got this.  This is my time to shine.  So enum two string goes, here's the strings.  I'm piping them out, piping them out.  And so it's got that whole thing that's minuses.  I'm done.  It's your turn.  And I'm like, oh crap, here comes this huge list.  So I've got to get my mind ready, clear it, and say I've got to allocate all this memory  now to figure out how to operate on this huge list.  10 million may mean that huge.  But it took a little bit of time to operate on.  If your user was waiting for that, they would probably go to some other site and buy some  other thing.  So that's what enum does.  It takes all that into memory, operates on each one of those huge collections at once,  and then spits it down to the next one, spits it down to the next one.  So stream, what stream does is something a little different.  In this case, we've been using enum.map.  There is also a stream.map.  So let's just go in and replace stream.map, or enum.map with stream.map.  Yes, that's a good idea.  Oh, you know what?  Did that wrong, but that's OK.  So one to 10 million, pipe that in to stream.map.  And for those, we're going to do the integer.toString.  And we're going to pipe that directly into stream.map again.  And this time, the string.grapheams.  OK, so now, remember how long the enum one took, right?  So what happens if I press enter now?  Are you guys expecting something real quick?  Hopefully so, because boom!  That's not what you wanted, is it?  You wanted the list.  You wanted the list of the graph themes, right?  You wanted those characters, but now we've got this weird thing.  It's hard to see because it's cut off on the side a little bit.  Let me do this.  Boom!  Yeah, I'll just do that.  And then now the bottom's gone a little bit.  All right, there we go.  That works.  OK, so now we have that stream, and we have a list inside stream of enum.  It's keyword list.  So we have our enum in there.  And then we have funds, and that sounds like a fun thing, right?  We've got funds.  But inside funds is another list, and they are two different things, and they're functions.  So what this returns to us is not what we were expecting.  At least it wasn't what I was expecting when I first thought, well, I'll just drop in stream,  and then nothing came out.  And I was like, why not?  Stream in order to operate on the pipeline has to have something at the end that actually  requests something.  So in typical cases, you would use a function from the enum module.  In this case, you could use to list or take or something like that.  But the very end of your stream has to actually request something.  So streams are what's known as lazy.  They don't actually execute and sell a need for it to execute.  So that's kind of where the life is but a stream is.  So you're like streaming, you're like, I don't know, what are those like donut inflatable  things called?  I'm blinking right now.  But anyway, you're in though, and though it's inner tube.  You're floating down the stream in an inner tube, right?  That's lazy.  You're just kind of, you know, whatever.  So that's kind of what stream does.  It's like, I'm not going to execute anything yet.  I'll set all my stuff up.  But you don't need anything.  Why would I work, right?  Forget you guys.  I'm just going to sit here.  So what if we need to take something?  So let's go through that again.  To string and then the graph themes.  And then let's do enum.toolist.  Now so what are you expecting here?  I already pressed inner.  Sorry, I should have gone up again.  I already pressed inner.  So you might be expecting that this is going to take hardly any time at all.  But in fact, you see, well, this isn't hardly any quicker than the enum.  It's still taking a long time.  So this is where you have to really discover what you're trying to do with your collections  in order to learn how to do stream correctly.  Because at first, I thought it was just going to straight drop in and I would get these  huge speed bumps.  But in this case, the enum.toolist is still requesting the entire list of items, the entire  collection of 10 million things, right?  It's saying, I just want everything you got, and I'm just going to put it into a list.  Now what if we wanted to take only four of those things?  We've got a collection of 10 million items, but really we only want four of them.  Maybe it's a list of questions or it could be a log file.  We just want the last four things.  In that case, if we did the enum thing, the whole list, again, we'd go, gjong, gjong.  And then it would go, gjong to me.  And I'm like, thanks for these 10 million items.  I only wanted four.  You jerk, right?  So you'd like, I got these four.  You forget your other ones.  What Stream does is, let's say we have a stream at the bottom, or a stream pipeline.  So let's pipe these things through again.  One through 10 million, two string.  We got the graph themes.  And from there, we're going to take four.  All right.  So again, you might think this is going to take forever because an enum it would, but  now, wow, right?  Real quick.  That didn't take hardly any time at all because here's what's happening now.  Instead of enum pushing all those lists down the pipeline, Stream goes at the very bottom.  This enum take says, I need one.  And so at the very top, that range goes, OK, I'm spitting down one.  And then it goes down into the first stream function, stream map function, the energy  to string.  And it goes, OK, one to string is the string one.  It passes that down to the next function.  And it poops that out, right?  All the way down to the enum take one, or take four.  And enum take four is like, all right, I got one of my four.  I need two.  And so the second one comes down, three, four.  And once it gets to four, it's like, I'm done.  Stream's like, well, I set up all these 999,996 others for you.  Are you not going to use them?  And I'm like, no, I don't need them.  It's like, OK, thanks.  You saved us a bunch of work.  So that's where Stream actually makes a huge difference in collections like this, is when  you only need a subset of the entire collection.  So let's take a look again at my slides here.  Streams are lazy, again, as I said.  So I thought, well, let me take some benchmarks because then I was like, it all depends on  your code, right?  So let's use some benchmarks.  So in this case, I set up something where I had four functions in a pipeline.  And I thought, OK, well, I'm going to pipe those through in the functions through each  other.  And at the very end, I'm going to call enum.toolist and see how long it takes.  So I piped in 1,000 items.  And the enum iterations per second was almost 1.6,000, almost 1,600 a second.  Stream was actually just under 1,200 a second.  And so I thought, well, that's weird.  So in this case, stream is actually 1.34 times slower than enum.  Interesting.  So why is that?  Let's say, OK, well, maybe I just need a bigger data set.  So I got passed in 100,000 items.  Enum is 11.53 a second.  Stream is 7,000 a second.  I'm sorry, that should be 11.53 is the wrong number.  But stream actually happens to be 1.57 times slower than enum.  Again, the enum, I copied that wrong.  Sorry.  But it's slower.  Enum, the stream is still slower.  So let's go to 10 million items.  Enum, 0.04 iterations per second.  Stream is 0.38 iterations per second.  And stream is 1.29 times slower.  Even with a large data set, if I'm just at the very end calling to list, there's really  a detriment for using stream.  And that's where you think, well, why is that?  Didn't you just say it was like a drop-in?  Kind of.  So enum and stream are two different things.  You notice that when we created that pipeline, but we didn't actually take anything yet,  it returned us this weird data structurey thing, right?  And said, here's what I've set up for you, but we haven't operated on it yet.  So all that set up actually takes some computational power.  And so that's the case where it may not make much sense to do the computation and using  the power to set up the stream pipeline if all you're doing is taking the entire list  at once.  Because the stream is like, well, I'm doing all this work for you, but you could have  used enum and actually been a bit quicker.  So I thought, OK, well, I've got four functions.  That's not a whole lot of functions in the pipeline.  What if I send it through eight functions, a pipeline through eight different functions,  one with enum.map and another with stream.map?  So in this case, I had 1,000.  Enum did 581.  Stream did 527.  Again, stream is 1.1 times slower.  400,000.  Stream got a little quicker, so it's 1.07.  And then in 10 million, stream went backwards a little bit, but that could just be because  of some deviation.  So in this case, stream, again, is slower.  They're getting closer to where you wouldn't notice much difference between enum and stream.  So here's where the big stuff comes in.  So we have eight functions that we're going to pass through.  So we have a long function list.  But at the very end, instead of just calling two lists, what if we just needed to take  10 of those things?  So for 1,000 of them, enum can do almost 560 a second, whereas stream can do almost 74,000  a second.  So enum is 132 times slower than stream.  This is where stream starts to show its usage.  For 100,000, enum is 13,000 times slower.  And then for 10 million items, enum is almost 2.4 million times slower than stream.  That's because, as we mentioned, enum has to contain that whole entire collection of  10 million items in its brain in each step of the way.  Another way we can illustrate this, let's go back to our list here.  Okay, list equals elixir, impacts, and LA.  So what I'm going to do now is, in each step of the pipeline, I'm going to have our pipeline  basically spit out what its current state is, like what the current collection is.  So we're going to take our list.  I'm just going to do it this way.  So we're going to enum map our list.  And for each one, we're going to just inspect it.  And from there, we're going to pipe it into the first thing we did, which was our string.upcase.  And then let's pipe that in again to map.  And we're going to inspect it again.  And then finally, we're going to do our graphemes.  Okay, so in this case, you can see here, our first inspect goes through and has the entire  collection, right?  Our three items are impacts, or sorry, elixir, impacts, and LA.  And we see all three of those come out first.  That's coming out from this right here, our first thing.  Then we upcase them.  And you can see again that we have our upcase collection all at once.  Now what happens if we change that up to stream?  So let's say stream.map.  And we're going to spec those.  And we're going to pipe that into stream.map.  We're going to do our string upcase.  I'm sorry?  Oh, yeah, thanks.  Stream.map, and then we're going to inspect it again, just like we did with the map version.  And then we're going to map those to our graphemes.  And then at the very end, since stream won't actually operate this until we request it,  we're going to call enum.toList.  Okay, so in here, my inspect, oh, inspect.  Awesome.  So let's not inspect.  Let's inspect.  Upcase, and then we're going to inspect there, and then we're going to at the very end.  Okay, so in this case, remember the first time we saw all three lowercase items first,  and then all three uppercase items.  In this case, stream is saying, I'm ready for the first item, and it sends Elixir down  the pipeline, right?  It drips it down.  So the first one comes Elixir lowercase, which we expected.  But the second item is actually our uppercase Elixir.  So we can see that that actually did just drip that first one all the way down to us.  And then the second item was our impacts, and that was drip down from lowercase to uppercase  into our final graphemes, and the same thing with LA.  So this is another way to look at the fact that each one of those items are kind of drip  down as the final thing says, I'm ready for it.  So this is where stream really comes into effect and really comes into its own as something  that's very useful for your applications, is when you take a subset of items or you  just need to make sure you're doing one at a time through your huge pipeline or in your  huge data set.  Yeah?  Can you show us what it looks like again if you don't take that list again?  Yeah, absolutely.  So let's go through that again.  So we're doing our list, inspect, and then we go to uppercase, and then we do another  inspect, and then we go to graphemes.  And instead of that, we're just going to end it.  Is that what you're talking about?  Yeah.  All right.  Then we have that stream thing.  Whatever.  Yes.  Yeah.  So the question was, can you take something that may be that infinite stream that you  don't know when it ends, it may never end?  Can stream trickle those down as you need them, even if it's not a complete set at the very  top?  And that answer is yes.  So the key to keep in mind is there's, I mentioned that enum kind of finalizes that stream and  takes them.  There's another function in the stream module called run.  And that actually is something where you can use it to execute the stream, but you may  not need the final result of the stream.  So in this case, let's say it's a log file, you need to write different things.  So like, you do something to that log file line, and then write that to another file,  maybe two files, maybe three files, maybe send it to an FTP server, whatever you need  to do to it.  And you don't need to really know the output of that pipeline.  In that case, run can be used to do something that you don't really need to know the end  result.  All right.  So that's a run through stream.  Again, I am Jeffrey Lessle.  I'm writing a book called Phoenix in Action.  I should have some discount codes for you guys later today, I'm hoping.  But you can buy it there, jeffreylessle.com slash phoenixinaction.  It's in Meep right now, Manning's Early Access Program.  I think there's two chapters available, but the third should be available sometime hopefully  next week, actually.  This is where you can find me online, jeffreylessle.com.  I do some blogging about Elixir and Ecto and stuff like that.  I'm Twitter, Geo-Lessle, and then GitHub at Geo-Lessle as well.  So thank you guys so much.  Thanks a lot, chef.
Life_is_but_a_Stream_0_179.png Life_is_but_a_Stream_180_239.png Life_is_but_a_Stream_240_2219.png Life_is_but_a_Stream_2220_2531.png Life_is_but_a_Stream_2532_2999.png Life_is_but_a_Stream_3000_3407.png Life_is_but_a_Stream_3408_3683.png Life_is_but_a_Stream_3684_5003.png Life_is_but_a_Stream_5004_5063.png Life_is_but_a_Stream_5064_5231.png Life_is_but_a_Stream_5232_5303.png Life_is_but_a_Stream_5304_5351.png Life_is_but_a_Stream_5352_8315.png Life_is_but_a_Stream_8316_8363.png Life_is_but_a_Stream_8364_10487.png Life_is_but_a_Stream_10488_10559.png Life_is_but_a_Stream_10560_13427.png Life_is_but_a_Stream_13428_13475.png Life_is_but_a_Stream_13476_13523.png Life_is_but_a_Stream_13524_14507.png Life_is_but_a_Stream_14508_20387.png Life_is_but_a_Stream_20388_25391.png Life_is_but_a_Stream_25392_31355.png Life_is_but_a_Stream_31356_31427.png Life_is_but_a_Stream_31428_32231.png Life_is_but_a_Stream_32232_32387.png Life_is_but_a_Stream_32388_32426.png
So today, we're going to talk about something fun. But first, I want to say that I'm very honored to be here. I remember a couple of years ago when Impax, New York City, happened and I wanted to be there at the time and seeing like videos and feedback online about how awesome the conference was and how amazing like the surroundings were. I was very jealous that I wasn't going to be able to be there. So I am very honored to be here today at the first West Coast Impax and they delivered. Didn't they? I mean, this place is amazing. Everything around, like even the pianist in the background coming in is like, what the heck? We got live piano as we're walking into a conference, a tech conference. How cool is that? It's pretty cool. All right. So I'm Jeffrey Lussell. Today, we're going to talk about Life is But a Stream. We're going to dive a little bit into the stream module. So what I'm hoping to do today for you guys and gals is if, well, basically to compare Enum to stream and give you a couple of examples of when you might want to switch out Enum for stream and sometimes when you may not. So the title of the talk is Life is But a Stream and I apologize. I may murder this analogy over the course of the talk. So I apologize up front, but bear with me. So in the background here, we got this nice looking stream. It looks really nice. You want to kind of spend some time on that bench over there. There's a little bit of water trickling down over a little, I guess you can call it a waterfall, but then we've got another picture here, another stream, kind of fall time. The water is a little bit rushier. The water is getting a little crazier. But still, you wouldn't mind spending some time either in the stream or by the side walking by it. Then we've got this one, which I don't know where that is, but I want to be there. That's pretty amazing. It's a little rockier, so you may, if you try to float down that thing, you may hurt yourself, but it's beautiful surroundings. And then you can imagine that those little streams may go from that first one to the second one to this one to this, right? This could be where your stream ends up, and you're towering over this, or you're falling over this towering waterfall into that huge, maybe like a two-foot thing down there. You're going to die if you go over that thing, right? So this is where stream and enum can lead you. So if we'd like to take a look back at this stream, I'm going to come back to these pictures in a moment, but keep those in mind as we talk about stream and enum, and just the fact of how the water moves through those pictures, okay? All right, so the first thing is enum is awesome. So how many of you in here have written at least one line of Elixir code in your life? Okay, cool. So how many of you have written one thing, one line of Elixir that contained a function from enum, the enum module? Okay, cool. So the Venn diagram of those two groups are basically the same, right? Everything you do in Elixir, not everything, but almost everything you do in Elixir at some point is going to touch the enum module. That's because there's so many helpful things in there, and it's typical in functional programming that you're going to be dealing with collections, and those collections are going to be passed from one thing to the next, and you have to do something with those collections and transform the data somehow. And enum is basically the module where you do that. So I love enum. And stream is kind of like a souped up enum that can be switched out a lot of times directly with enum. But we'll get into that a little bit more in detail in a second. So for right now, I'm going to dive into an IEX session. Hopefully you guys can see this. Do you think it'd be better with the black on white or the white on black? Okay, cool. I'm going to take it just a little bit bigger. All right, so let's say we have, let me clear. Let's say we've got a list, and our list is just going to be three items. Okay, we're going to say Elixir, let's say Impax, and LA. Okay, so that's our collection, right? Just three items, not a big deal, three strings. Now what if we wanted to do something with each one of those items in the list? Now a lot of this is going to be kind of, I know how to use an enum, but we'll get kind of make this a little more complicated as we go on. So let's say we want to capitalize or upcase each one of those items in the list. So we can send that list, and I'm going to do it this way. We send that list to enum.map. And for each one of those, I'm going to use the shorthand notation here. I'm going to upcase every one of those. That happens pretty quickly, right? It takes Elixir. In your view, it's Elixir. So we've got Elixir, Impax, and LA all upcased. So basically what enum.map did is it took that whole list we gave it, all three items at once, and enum kind of put it in its head. It put it in its mind, in its memory, and said, okay, I have this list now on every one of these items, I need to do this one particular function. In this case, we told it to do string.upcase. Now of course, we can take a little further. We can say, okay, we want to do that, then we want to send it to the upcase. Sorry, I forgot to do the post-fix thing here. And then we want to send it to another map. We want to transform it again. And this time, we're going to just take the graph emes and get each individual letter. So now we have a collection of collections, right? Each one of those is broken up into their individual letters. And so that's fine. That works. So that's what enum does. It takes the whole list that goes to the first enum, gets it all in its memory, and says, okay, I'm done now. I'm going to send the new list that I have into the next enum. And then our final enum says, okay, now I'm going to take that string and break it up into its individual characters. So this is fine, right? This is a fast-performant, whatever. This is milliseconds of time to spend in code. Now there's a problem, though, and that's when you get into large collections. So let's go back to this stream, right? So we have this stream. Let's say you want to operate on every single piece of water model queue in this picture, okay, as it's going over the waterfall. We could probably do it. It's not that big a deal. It's just these little trickle. But then let's say you got this, and you want to operate on every single molecule of water that falls over that stream as it comes by. That is a large data collection. So for you guys and gals, if you're in Elixir, maybe on the web, you've got a log file, for example, that's very large, or maybe some user input, maybe a text of some book that you want to analyze. Your data collection can get pretty large. But in fact, even in the stream module definition or story documentation, it says that the data size could be infinite. So we have no idea how much water is going to come over this waterfall in the life of this waterfall, right? We don't know how long, you start your little web project, and you're like, this is going to live forever. And so the log file could grow infinitely big. Hopefully you do things to keep it from doing that. But it could grow infinitely big. You don't know when the end is going to be. So we could be operating on that log file to the end of eternity, right? You could be always, always, always operating. So stream is something where you can operate on that. On enum, it would have to get all of that into its head at once. So if we look at that waterfall, it's not very easy to think of every single piece of water molecule in that waterfall, right? That's just a lot of stuff to put in your head. Same thing with enum. That's a lot of stuff to put in enum's head. So here's your solution, streams, OK? Let's go back to IEX. In this case, I'm just going to do a range. Let's say 1 to 10 million. And for each one of those things, we're going to pipe that into another enum. And we're going to map it. Let's say we're going to change it, change them to strings. To string. OK. So I hit enter, and it's taking a little while. That's because what it has to do is it has to take that entire list of 10 million, put it into its brain, and on each one of those say, OK, the first one's going to be a string of 1, and the second one's going to be a string of 2. All that whole list is going into its memory, and then it's going to spit it back out. So let's say, again, we want to take it one step further, and we want to pipe that into another map that changes into the graphemes. Now this is going to take a little while. So let's pretend I am that final enum.map. OK. I'm sitting here looking up at the pipeline. At the very top, I see the list of 10 million items. It's huge. All right. And then you can kind of hear the sound as it moves to the pipeline. So it goes to enum.map, and we've got the two strings. And two strings, like, I've got this. This is my time to shine. So enum two string goes, here's the strings. I'm piping them out, piping them out. And so it's got that whole thing that's minuses. I'm done. It's your turn. And I'm like, oh crap, here comes this huge list. So I've got to get my mind ready, clear it, and say I've got to allocate all this memory now to figure out how to operate on this huge list. 10 million may mean that huge. But it took a little bit of time to operate on. If your user was waiting for that, they would probably go to some other site and buy some other thing. So that's what enum does. It takes all that into memory, operates on each one of those huge collections at once, and then spits it down to the next one, spits it down to the next one. So stream, what stream does is something a little different. In this case, we've been using enum.map. There is also a stream.map. So let's just go in and replace stream.map, or enum.map with stream.map. Yes, that's a good idea. Oh, you know what? Did that wrong, but that's OK. So one to 10 million, pipe that in to stream.map. And for those, we're going to do the integer.toString. And we're going to pipe that directly into stream.map again. And this time, the string.grapheams. OK, so now, remember how long the enum one took, right? So what happens if I press enter now? Are you guys expecting something real quick? Hopefully so, because boom! That's not what you wanted, is it? You wanted the list. You wanted the list of the graph themes, right? You wanted those characters, but now we've got this weird thing. It's hard to see because it's cut off on the side a little bit. Let me do this. Boom! Yeah, I'll just do that. And then now the bottom's gone a little bit. All right, there we go. That works. OK, so now we have that stream, and we have a list inside stream of enum. It's keyword list. So we have our enum in there. And then we have funds, and that sounds like a fun thing, right? We've got funds. But inside funds is another list, and they are two different things, and they're functions. So what this returns to us is not what we were expecting. At least it wasn't what I was expecting when I first thought, well, I'll just drop in stream, and then nothing came out. And I was like, why not? Stream in order to operate on the pipeline has to have something at the end that actually requests something. So in typical cases, you would use a function from the enum module. In this case, you could use to list or take or something like that. But the very end of your stream has to actually request something. So streams are what's known as lazy. They don't actually execute and sell a need for it to execute. So that's kind of where the life is but a stream is. So you're like streaming, you're like, I don't know, what are those like donut inflatable things called? I'm blinking right now. But anyway, you're in though, and though it's inner tube. You're floating down the stream in an inner tube, right? That's lazy. You're just kind of, you know, whatever. So that's kind of what stream does. It's like, I'm not going to execute anything yet. I'll set all my stuff up. But you don't need anything. Why would I work, right? Forget you guys. I'm just going to sit here. So what if we need to take something? So let's go through that again. To string and then the graph themes. And then let's do enum.toolist. Now so what are you expecting here? I already pressed inner. Sorry, I should have gone up again. I already pressed inner. So you might be expecting that this is going to take hardly any time at all. But in fact, you see, well, this isn't hardly any quicker than the enum. It's still taking a long time. So this is where you have to really discover what you're trying to do with your collections in order to learn how to do stream correctly. Because at first, I thought it was just going to straight drop in and I would get these huge speed bumps. But in this case, the enum.toolist is still requesting the entire list of items, the entire collection of 10 million things, right? It's saying, I just want everything you got, and I'm just going to put it into a list. Now what if we wanted to take only four of those things? We've got a collection of 10 million items, but really we only want four of them. Maybe it's a list of questions or it could be a log file. We just want the last four things. In that case, if we did the enum thing, the whole list, again, we'd go, gjong, gjong. And then it would go, gjong to me. And I'm like, thanks for these 10 million items. I only wanted four. You jerk, right? So you'd like, I got these four. You forget your other ones. What Stream does is, let's say we have a stream at the bottom, or a stream pipeline. So let's pipe these things through again. One through 10 million, two string. We got the graph themes. And from there, we're going to take four. All right. So again, you might think this is going to take forever because an enum it would, but now, wow, right? Real quick. That didn't take hardly any time at all because here's what's happening now. Instead of enum pushing all those lists down the pipeline, Stream goes at the very bottom. This enum take says, I need one. And so at the very top, that range goes, OK, I'm spitting down one. And then it goes down into the first stream function, stream map function, the energy to string. And it goes, OK, one to string is the string one. It passes that down to the next function. And it poops that out, right? All the way down to the enum take one, or take four. And enum take four is like, all right, I got one of my four. I need two. And so the second one comes down, three, four. And once it gets to four, it's like, I'm done. Stream's like, well, I set up all these 999,996 others for you. Are you not going to use them? And I'm like, no, I don't need them. It's like, OK, thanks. You saved us a bunch of work. So that's where Stream actually makes a huge difference in collections like this, is when you only need a subset of the entire collection. So let's take a look again at my slides here. Streams are lazy, again, as I said. So I thought, well, let me take some benchmarks because then I was like, it all depends on your code, right? So let's use some benchmarks. So in this case, I set up something where I had four functions in a pipeline. And I thought, OK, well, I'm going to pipe those through in the functions through each other. And at the very end, I'm going to call enum.toolist and see how long it takes. So I piped in 1,000 items. And the enum iterations per second was almost 1.6,000, almost 1,600 a second. Stream was actually just under 1,200 a second. And so I thought, well, that's weird. So in this case, stream is actually 1.34 times slower than enum. Interesting. So why is that? Let's say, OK, well, maybe I just need a bigger data set. So I got passed in 100,000 items. Enum is 11.53 a second. Stream is 7,000 a second. I'm sorry, that should be 11.53 is the wrong number. But stream actually happens to be 1.57 times slower than enum. Again, the enum, I copied that wrong. Sorry. But it's slower. Enum, the stream is still slower. So let's go to 10 million items. Enum, 0.04 iterations per second. Stream is 0.38 iterations per second. And stream is 1.29 times slower. Even with a large data set, if I'm just at the very end calling to list, there's really a detriment for using stream. And that's where you think, well, why is that? Didn't you just say it was like a drop-in? Kind of. So enum and stream are two different things. You notice that when we created that pipeline, but we didn't actually take anything yet, it returned us this weird data structurey thing, right? And said, here's what I've set up for you, but we haven't operated on it yet. So all that set up actually takes some computational power. And so that's the case where it may not make much sense to do the computation and using the power to set up the stream pipeline if all you're doing is taking the entire list at once. Because the stream is like, well, I'm doing all this work for you, but you could have used enum and actually been a bit quicker. So I thought, OK, well, I've got four functions. That's not a whole lot of functions in the pipeline. What if I send it through eight functions, a pipeline through eight different functions, one with enum.map and another with stream.map? So in this case, I had 1,000. Enum did 581. Stream did 527. Again, stream is 1.1 times slower. 400,000. Stream got a little quicker, so it's 1.07. And then in 10 million, stream went backwards a little bit, but that could just be because of some deviation. So in this case, stream, again, is slower. They're getting closer to where you wouldn't notice much difference between enum and stream. So here's where the big stuff comes in. So we have eight functions that we're going to pass through. So we have a long function list. But at the very end, instead of just calling two lists, what if we just needed to take 10 of those things? So for 1,000 of them, enum can do almost 560 a second, whereas stream can do almost 74,000 a second. So enum is 132 times slower than stream. This is where stream starts to show its usage. For 100,000, enum is 13,000 times slower. And then for 10 million items, enum is almost 2.4 million times slower than stream. That's because, as we mentioned, enum has to contain that whole entire collection of 10 million items in its brain in each step of the way. Another way we can illustrate this, let's go back to our list here. Okay, list equals elixir, impacts, and LA. So what I'm going to do now is, in each step of the pipeline, I'm going to have our pipeline basically spit out what its current state is, like what the current collection is. So we're going to take our list. I'm just going to do it this way. So we're going to enum map our list. And for each one, we're going to just inspect it. And from there, we're going to pipe it into the first thing we did, which was our string.upcase. And then let's pipe that in again to map. And we're going to inspect it again. And then finally, we're going to do our graphemes. Okay, so in this case, you can see here, our first inspect goes through and has the entire collection, right? Our three items are impacts, or sorry, elixir, impacts, and LA. And we see all three of those come out first. That's coming out from this right here, our first thing. Then we upcase them. And you can see again that we have our upcase collection all at once. Now what happens if we change that up to stream? So let's say stream.map. And we're going to spec those. And we're going to pipe that into stream.map. We're going to do our string upcase. I'm sorry? Oh, yeah, thanks. Stream.map, and then we're going to inspect it again, just like we did with the map version. And then we're going to map those to our graphemes. And then at the very end, since stream won't actually operate this until we request it, we're going to call enum.toList. Okay, so in here, my inspect, oh, inspect. Awesome. So let's not inspect. Let's inspect. Upcase, and then we're going to inspect there, and then we're going to at the very end. Okay, so in this case, remember the first time we saw all three lowercase items first, and then all three uppercase items. In this case, stream is saying, I'm ready for the first item, and it sends Elixir down the pipeline, right? It drips it down. So the first one comes Elixir lowercase, which we expected. But the second item is actually our uppercase Elixir. So we can see that that actually did just drip that first one all the way down to us. And then the second item was our impacts, and that was drip down from lowercase to uppercase into our final graphemes, and the same thing with LA. So this is another way to look at the fact that each one of those items are kind of drip down as the final thing says, I'm ready for it. So this is where stream really comes into effect and really comes into its own as something that's very useful for your applications, is when you take a subset of items or you just need to make sure you're doing one at a time through your huge pipeline or in your huge data set. Yeah? Can you show us what it looks like again if you don't take that list again? Yeah, absolutely. So let's go through that again. So we're doing our list, inspect, and then we go to uppercase, and then we do another inspect, and then we go to graphemes. And instead of that, we're just going to end it. Is that what you're talking about? Yeah. All right. Then we have that stream thing. Whatever. Yes. Yeah. So the question was, can you take something that may be that infinite stream that you don't know when it ends, it may never end? Can stream trickle those down as you need them, even if it's not a complete set at the very top? And that answer is yes. So the key to keep in mind is there's, I mentioned that enum kind of finalizes that stream and takes them. There's another function in the stream module called run. And that actually is something where you can use it to execute the stream, but you may not need the final result of the stream. So in this case, let's say it's a log file, you need to write different things. So like, you do something to that log file line, and then write that to another file, maybe two files, maybe three files, maybe send it to an FTP server, whatever you need to do to it. And you don't need to really know the output of that pipeline. In that case, run can be used to do something that you don't really need to know the end result. All right. So that's a run through stream. Again, I am Jeffrey Lessle. I'm writing a book called Phoenix in Action. I should have some discount codes for you guys later today, I'm hoping. But you can buy it there, jeffreylessle.com slash phoenixinaction. It's in Meep right now, Manning's Early Access Program. I think there's two chapters available, but the third should be available sometime hopefully next week, actually. This is where you can find me online, jeffreylessle.com. I do some blogging about Elixir and Ecto and stuff like that. I'm Twitter, Geo-Lessle, and then GitHub at Geo-Lessle as well. So thank you guys so much. Thanks a lot, chef.
little bit into the stream module. So what I'm murder this analogy over the course of the talk. right now, I'm going to dive into an IEX session. long, you start your little web project, and put it into its brain, and on each one of those different. In this case, we've been using items. Enum is 11.53 a second. Stream is 7,000 a different functions, one with enum.map and should have some discount codes for you guys