Okay, so today we're gonna talk about something fun, but first I want to say that I'm very honored to be here. I remember a couple years ago when Impex New York City happened and I wanted to be there at the time and seeing like videos and feedback online about how awesome the

conference was and how amazing like the surroundings were. I was very jealous that I wasn't able to be there, so I am very honored to be here today at the first West Coast Impex and they delivered, didn't they? I mean this place is amazing. Everything around, like even the pianist in the background coming in, he's like, what the heck, we got live piano as we're walking to a conference, a tech conference?

How cool is that? It's pretty cool. Alright, so I'm Jeffrey Lessl. Today we're gonna talk about Life is But a Stream. We're gonna dive a little bit into the stream module. So what I'm hoping to do today for you guys and gals is if, well basically to compare enum to stream and give you a couple of examples of when you might want to

switch out enum for stream and sometimes when you may not. So the title of the talk is Life is But a Stream, and I apologize, I may murder this analogy over the course of the talk, so I apologize up front. But bear with me, okay? So in the background here we've got this nice looking stream, right? It looks really nice, you know?

You kinda wanna spend some time on that bench over there. There's a little bit of water trickling down over a little, I guess you can call it a waterfall. And then we've got another picture here, another stream, kind of fall time. The water's a little bit rushier, you know, the water's getting a little crazier. But still, you wouldn't mind spending some time either in the stream or by the side walking by it.

Then we've got this one, which I don't know where that is, but I wanna be there. That's pretty amazing. It's a little rockier, so you may, you know, if you try to float down that thing, you may hurt yourself, but it's beautiful surroundings. And then you can imagine that those little streams may go from that first one to the second one to this one to this, right?

This could be where your stream ends up and you're towering over this, or you're falling over this towering waterfall into, you know, that huge, it may be like a two foot thing down there. You're gonna die if you go over that thing, right? So this is where stream and Enum can lead you. So if we take a look back at this stream, I'm gonna come back to these pictures in a moment, but keep those in mind as we talk about stream and Enum

and just the fact of how the water moves through those pictures, okay? All right, so the first thing is Enum is awesome. So how many of you in here have written at least one line of Elixir code in your life? Okay, cool. So how many of you have written one thing, one line of Elixir that contained a function from Enum, the Enum module?

Okay, cool. So the Venn diagram of those two groups are basically the same, right? Everything you do in Elixir, not everything, but almost everything you do in Elixir at some point is gonna touch the Enum module. That's because there's so many helpful things in there, and it's typical in functional programming that you're gonna be dealing with collections and those collections are gonna be passed from one thing to the next

and you have to do something with those collections and transform the data somehow. And Enum is basically the module where you do that. So I love Enum. And stream is kind of like a souped-up Enum that can be switched out a lot of times directly with Enum. But we'll get into that a little bit more in detail in a second.

So for right now I'm gonna dive into an IEX session. Hopefully you guys can see this. Do you think it'd be better with the black on white or the white on black? Okay, cool. I'm gonna make it just a little bit bigger. All right, so let's say we have... Let's say we've got a list, and our list is just gonna be three items, okay? We're gonna say Elixir, let's say Impacts, and LA.

Okay, so that's our collection, right? Just three items, not a big deal, three strings. Now what if we wanted to do something with each one of those items in the list? Now a lot of this is gonna be kind of, I know how to use an Enum. But we'll kind of make this a little more complicated as we go on. So let's say we want to capitalize or upcase each one of those items in the list.

So we can send that list, and I'm gonna do it this way. We send that list to Enum.map. And for each one of those, I'm gonna use the shorthand notation here. I'm gonna upcase every one of those. That happens pretty quickly, right? It takes Elixir, in your view it's Elixir. So we got Elixir, Impacts, and LA, all upcased.

So basically what Enum.map did is it took that whole list we gave it, all three items at once, and Enum kind of put it in its head, it put it in its mind, in its memory, and said, okay, I have this list, now on every one of these items, I need to do this one particular function. In this case, we told it to do string.upcase.

Now of course we can take it a little further, we can say, okay, we want to do that, then we want to send it to upcase. Sorry, I forgot to do the postfix thing here. And then we want to send it to another map. We want to transform it again. And this time we're going to just take the graphemes and get each individual letter. So now we have a collection of collections, right?

Each one of those is broken up into their individual letters. And so that's fine, that works. So that's what Enum does. It takes the whole list, it goes to the first Enum, gets it all in its memory, it says, okay, I'm done now, I'm going to send the new list that I have into the next Enum. And then our final Enum says, okay, now I'm going to take that string

and break it up into its individual characters. So this is fine, right? This is a fast, performant, you know, whatever. This is milliseconds of time to spend in code. Now there's a problem though, and that's when you get into large collections. So let's go back to this stream, right? So we have this stream. Let's say you want to operate on every single piece of water model fuel in this picture, okay,

as it's going over the waterfall. You know, we could probably do it. It's not that big a deal. It's just this little trickle. But then let's say you got this, and you want to operate on every single molecule of water that falls over that stream as it comes by. That is a large data collection. So for you guys and gals, if you're in Elixir,

maybe on the web, you've got a log file, for example, that's very large, or maybe some user input, maybe a text of some book that you want to analyze. Your data collection can get pretty large. And in fact, even in the stream module definition, or documentation, it says that the data size could be infinite. So we have no idea how much water

is going to come over this waterfall in the life of this waterfall, right? We don't know how long. You start your little web project, and you're like, this is going to live forever. And so the log file could grow infinitely big. Hopefully you do things to keep it from doing that. But it could grow infinitely big. You don't know when the end is going to be. So we could be operating on that log file to the end of eternity, right?

Could be always, always, always operating. So stream is something where you can operate on that. An enum, it would have to get all of that into its head at once. So if we look at that waterfall, it's not very easy to think of every single piece of water molecule in that waterfall, right? That's just a lot of stuff to put in your head. Same thing with enum. That's a lot of stuff to put in enum's head.

So here's your solution, streams, okay? Let's go back to IEX. In this case, I'm just going to do a range. Let's say one to 10 million. And for each one of those things, we're going to pipe that into another enum. And we're going to map it. Let's say we're going to change it, change them to strings. Okay, so I hit enter,

and it's taking a little while. That's because what it has to do is it has to take that entire list of 10 million, put it into its brain, and on each one of those say, okay, the first one's going to be a string of one, and the second one's going to be a string of two. All that whole list is going into its memory and then it's going to spit it back out. So let's say, again, we want to take it one step further, and we want to pipe that into another map

that changes into the graphemes. Now this is going to take a little while. So let's pretend I am that final enum.map, okay? I'm sitting here looking up at the pipeline. At the very top, I see the list of 10 million items. It's huge, right? And then you can kind of hear the sound as it moves through the pipeline.

So it goes to enum.map, and we got the two string, and two string's like, I've got this. This is my time to shine. So enum2.string goes, here's the strings. I'm piping them out. I'm piping them out. And so it's got that whole thing that's mine. It says, I'm done. It's your turn. And I'm like, oh crap, here comes this huge list, right? So I've got to get my mind ready, clear it, and say, I've got to allocate all this memory now

to figure out how to operate on this huge list, right? 10 million, maybe not huge. But it took a little bit of time to operate on. If your user was waiting for that, they would probably go to some other site and buy some other thing, right? So that's what enum does. It takes all that into memory, operates on each one of those huge collections and sends it once and then spits it down to the next one, spits it down to the next one. So stream, what stream does,

is something a little different. In this case, we've been using enum.map. There is also a stream.map. So let's just go in and replace stream.map or enum.map with stream.map. Yes, that's a good idea. You know what? Did that wrong, but that's okay.

Okay, so one to 10 million. Pipe that in to stream.map. And for those, we're going to do the integer.toString. And we're going to pipe that directly into stream.map again. And this time the stream.graphemes.

Okay, so now, remember how long the enum one took, right? So what happens if I press enter now? Are you guys expecting something real quick? Hopefully so, because boom! That's not what you wanted, is it? You wanted the list of the graphemes, right? You wanted those characters. But now we've got this weird thing. It's hard to see because it's cut off on the side a little bit. In fact, let me do this.

Boop. Yeah. I'll just do that. Okay. And then now the bottom's gone a little bit. Ooh. All right, there we go. That works. Okay, so now we have that stream and we have a list inside stream of enum. It's a keyword list. So we have our enum in there. And then we have funds.

That sounds like a fun thing, right? We've got funds. But inside funds is another list. And they are two different things and they're functions. So what this returns to us is not what we were expecting. At least it wasn't what I was expecting when I first thought, well, I'll just drop in stream and then nothing came out. And I was like, well, why not? Stream, in order to operate on the pipeline, has to have something at the end

that actually requests something. So in typical cases, you would use a function from the enum module. In this case, you could use like to list or take or something like that. But the very end of your stream has to actually request something. So streams are what's known as lazy. They don't actually execute until there's a need for it to execute.

So that's kind of where the life is but a stream is. So you're like streaming, you're like in a, I don't know, what are those like donut inflatable things called? I'm blanking right now. But anyway, you're in one of those, inner tube. You're like floating down the stream in an inner tube, right? That's lazy. You're just kind of, you know, whatever. So that's kind of what stream does. It's like I'm not going to execute anything yet. I'll set all my stuff up,

but you don't need anything. Why would I work, right? Forget you guys. I'm just going to sit here. So what if we need to take something? So let's go through that again. Two string and then the graph beams. And then let's do enum.2 list. Now so what are you expecting here? I already pressed enter. Sorry, I should have gone up again.

I already pressed enter. So you might be expecting that this is going to take hardly any time at all. But in fact you see, well this isn't hardly any quicker than the enum. It's still taking a long time. So this is where you have to really discover what you're trying to do with your collections in order to learn how to use stream correctly. Because at first I thought it was just a straight drop in

and I would get these huge speed bumps. But in this case, the enum.2 list is still requesting the entire list of items. The entire collection of 10 million things, right? It's saying I just want everything you got and I'm just going to put it into a list. Now what if we wanted to take only four of those things? We've got a collection of 10 million items,

but really we only want four of them. Maybe it's, you know, a list of questions or it could be a log file. We just want the last four things. In that case, if we did the enum thing, the whole list again would go kajong, kajong, and then it would go kajong to me and I'm like thanks for these 10 million items. I only wanted four, you jerk, right? So you're like I got these four. You forget your other ones.

But what stream does is, let's say we have a stream at the bottom, or a stream pipeline. So let's pipe these things through again. One through 10 million, two string. We got the graphemes. And from there, we're going to take four. All right, so again, you might think this is going to take forever because an enum it would, but now, wow, right? Real quick.

It didn't take hardly any time at all. Because here's what's happening now. Instead of enum pushing all those lists down the pipeline, stream goes, at the very bottom, this enum take says, I need one. And so at the very top, that range goes, okay, I'm spitting down one. And then it goes down into the first enum, or the first stream function, stream map function, the integer to string,

and goes, okay, one to string is the string one. Passes that down to the next function, and it poops that out, right? All the way down to the enum take one, or take four. The enum take four is like, all right, I got one of my four. I need two. And so the second one comes down, three, four. And once it gets to four, it's like, I'm done. And stream's like, well, I've set up all these 999,996 others for you.

Are you not going to use them? And I'm like, no, I don't need them. It's like, okay, thanks. You saved us a bunch of work. So that's where stream actually makes a huge difference in collections like this, is when you only need a subset of the entire collection. So let's take a look again at my slide here. Streams are lazy, again, as I said. So I thought, well, let me take some benchmarks,

because then I was like, it all depends on your code, right? So let's do some benchmarks. So in this case, I set up something where I had four functions in a pipeline. And I thought, okay, well, I'm going to pipe those through in the functions through each other. And at the very end, I'm going to call enum.toList and see how long it takes. So I piped in 1,000 items, and the enum iterations per second was almost 1.6,000,

almost 1,600 a second. Stream was actually just under 1,200 a second. And so I thought, well, that's weird. So in this case, stream is actually 1.34 times slower than enum. Interesting. So why is that? Okay, well, maybe I just need a bigger data set. So I got past 100,000 items.

Enum is 11.53 a second. Stream is 7,000 a second. I'm sorry, that should be, 11.53 is the wrong number. But stream actually happens to be 1.57 times slower than enum. Again, the enum, I copied that wrong, sorry. But it's slower. A stream is still slower. So let's go to 10 million items. Enum, 0.04 iterations per second.

Stream is 0.38 iterations per second. And stream is 1.29 times slower. So even with a large data set, if I'm just at the very end calling toList, there's really a detriment for using stream. And that's where you think, well, why is that? Didn't you just say it was like a drop-in? Kind of.

So enum and stream are two different things. You notice that when we created that pipeline, but we didn't actually take anything yet, it returned us this weird data structure-y thing, right? It said, here's what I've set up for you, but we haven't operated on it yet. So all that setup actually takes some computational power. And so that's the case where it may not make much sense to do the computation using the power to set up the stream pipeline

if all you're doing is taking the entire list. At once. Because the stream's like, well, I'm doing all this work for you, but you could have used enum and actually been a bit quicker. So I thought, okay, well, I've got four functions. That's not a whole lot of functions in the pipeline. What if I send it through eight functions, piping through eight different functions, one with enum.map, another with stream.map.

So in this case, I had 1,000. Enum did 581. Stream did 527. Again, stream is 1.1 times slower. 400,000. Stream got a little quicker, so it's 1.07. And then in 10 million, stream went backwards a little bit, but that could just be because of some deviation. So in this case, stream, again, is slower.

We're getting closer to where you wouldn't notice much difference between enum and stream. But here's where the big stuff comes in. So we have eight functions that we're gonna pass through. So we have a long function list. But at the very end, instead of just calling two lists, what if we just needed to take ten of those things? So for a thousand of them,

enum can do almost 560 a second, whereas stream can do almost 74,000 a second. So enum is 132 times slower than stream. This is where stream starts to show its usage. For 100,000, enum is 13,000 times slower, okay?

And then for 10 million items, enum is almost 2.4 million times slower than stream. That's because, as we mentioned, enum has to contain that whole entire 10 million, collection of 10 million items in its brain in each step of the way. Another way we can illustrate this, let's go back to our list here.

Say list equals elixir, impacts, and la. Okay, so what I'm gonna do now is, in each step of the pipeline, I'm gonna have our pipeline basically spit out what its current state is, like what the current collection is. So we're gonna take our list, I'm just gonna do it this way, so we're gonna enum map our list,

and for each one, we're going to, let's just inspect it. And from there, we're going to pipe it into, the first thing we did, which was our string.upcase, and then let's pipe that in again to map, and we're gonna inspect it again. And then finally, we're going to do our graphemes.

Okay, so in this case, you can see here, our first inspect goes through and has the entire collection, right? Our three items are impacts, or sorry, elixir, impacts, and la, and we see all three of those come out first. That's coming out from this right here, our first thing. Then we upcase them,

and you can see again that we have our upcase collection all at once. Now what happens if we change that up to stream? So let's say stream.map, and we have, we're gonna inspect those, and we're gonna pipe that into stream.map. We're gonna do our string upcase.

I'm sorry? Oh yeah, sorry, thanks. Stream.map, and then we're gonna inspect it again, just like we did with the map version, and then we're gonna map those to our graphemes, and then at the very end, since stream won't actually operate this until we request it,

we're gonna call enum.toList. Okay, so in here, I inspect impsect, awesome. So let's not inspect, let's inspect upcase, and then we're going to inspect there, and then we're going to the very end. Okay, so in this case,

remember the first time we saw all three lowercase items first, and then all three uppercase items. In this case, stream is saying, I'm ready for the first item, and it sends Elixir down the pipeline, right? It drips it down. So the first one comes Elixir lowercase, which we expected, but the second item is actually our uppercased Elixir. So we can see that that actually did just drip that first one all the way down to us.

And the second item was our impects, and that was dripped down from lowercase to uppercase into our final graphemes, and the same thing with LA. So this is another way to look at the fact that each one of those items are kind of dripped down as the final thing says, I'm ready for it. So this is where stream really comes into effect and really comes into its own as something that's very useful for your applications,

is when you take a subset of items or you just need to make sure you're doing one at a time through your huge pipeline or in your huge data set. Yeah? Yeah, absolutely. So let's go through that again. So we're doing our list, inspect, and then we go to upcase, and then we do another inspect, and then we go to graphemes,

and instead of that, we're just going to end it. Is that what you're talking about? All right, then we have that stream thing. Yes. Yeah, so the question was, can you take something that may be that infinite stream that you don't know when it ends, it may never end, can stream trickle those down as you need them even if it's not a complete set at the very top? And the answer is yes. Another thing to keep in mind is there's,

I mentioned that enum kind of finalizes that stream and takes them. There's another function in the stream module called run, and that actually is something where you can use it to execute the stream, but you may not need the final result of the stream. So in this case, let's say it's a log file, you need to write different things. So like, you know, do something to that log file line and then write that to another file,

maybe two files, maybe three files, maybe send it to an FTP server, whatever you need to do to it, but at the very end, you don't need to really know the output of that pipeline. In that case, run can be used to do something that you don't really need to know the end result. All right, so that's a run-through stream. Again, I'm Jeffrey Lessl. I'm writing a book called Phoenix in Action.

I should have some discount codes for you guys later today. I'm hoping. But you can buy it there, jeffreylessl.com slash phoenixinaction. It's in MEEP right now, Manning's early access program. I think there's two chapters available, but the third should be available sometime, hopefully next week, actually. This is where you can find me online, jeffreylessl.com. I do some blogging about Elixir and Ecto

and stuff like that. Twitter, GeoLessl, and then GitHub at GeoLessl as well. So thank you guys so much. Thanks a lot, Jeff.