Functional Programming in an Imperative World. Maybe
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Part Number | 18 | |
Number of Parts | 48 | |
Author | ||
Contributors | ||
License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/33189 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
DjangoCon US 201718 / 48
2
5
6
14
16
23
26
30
31
32
34
39
43
48
00:00
Functional programmingComputer programmingPell's equationEmailMaizeRecursionFunction (mathematics)Digital filterReduction of orderConcurrency (computer science)CodeChemical equationComputer wormSystem callQuery languageDatabaseCursor (computers)CodeFunctional programmingComputer programmingDifferent (Kate Ryan album)Programming paradigmCore dumpSound effectCartesian coordinate systemQuicksortBitDatabaseReal numberFunctional programmingPoint (geometry)Connected spaceLogicResultantProgramming languageLine (geometry)Row (database)Parameter (computer programming)CASE <Informatik>Software bugWordTotal S.A.Set (mathematics)MereologyOrder (biology)Variable (mathematics)Object-oriented programmingCursor (computers)Loop (music)Parallel computingMathematicsRight angleBoundary value problemTouchscreenLetterpress printingBlogType theoryControl flowLevel (video gaming)Slide ruleQuery languageSystem callMoment (mathematics)Perturbation theoryObject (grammar)Social classEmailView (database)TwitterUniform resource locatorPresentation of a groupEntire functionLaptopMultiplication signData modelOpen setInsertion lossCoprocessorConcurrency (computer science)Computer animation
08:40
Source codeTotal S.A.Object (grammar)RecursionAcoustic couplerDigital filterReduction of orderNatural number3 (number)Equals signCodeData structureAnnulus (mathematics)Hill differential equationType theoryMultiplication signData structureSource codeLink (knot theory)MathematicsCycle (graph theory)Pointer (computer programming)Right angleQuicksortSemiconductor memorySummierbarkeitIntegerPersonal identification numberResultantDatabaseSystem callProduct (business)MereologyBitProgramming languageDescriptive statisticsFunctional programmingFunctional programmingElectronic mailing listKey (cryptography)Set (mathematics)Standard deviationLimit (category theory)WordDifferent (Kate Ryan album)TrailPattern languageEntire functionType theoryCASE <Informatik>Order (biology)Wrapper (data mining)Instance (computer science)Polymorphism (materials science)AutocovariancePressureSequelComplex (psychology)Solid geometryObject-oriented programmingData streamTupleData dictionaryLine (geometry)Loop (music)2 (number)Object (grammar)Computer programmingGoodness of fitMetropolitan area networkComputer animation
17:19
RecursionData structureDigital filterReduction of orderCodeProgrammschleifeRecursionSystem callProgrammschleifeMultiplication signLambda calculusElectronic mailing listFunctional programmingState of matterSurfaceQuicksortData structureVariable (mathematics)Entire functionSemiconductor memoryTrailProgramming languageLoop (music)Limit (category theory)Revision controlChannel capacityDisk read-and-write headSummierbarkeitFrame problemNumberDefault (computer science)Computer programmingReduction of orderCodeContext awarenessMathematical optimizationRoutingPoint (geometry)CASE <Informatik>BitStack (abstract data type)Functional programmingFlow separationLine (geometry)Arithmetic progressionLevel (video gaming)Arithmetic meanLocal ringStatuteBuffer overflowMoment (mathematics)Ising-ModellResultantMassSingle-precision floating-point formatComputer animation
25:59
Digital filterReduction of orderNormed vector spaceComputer wormCodeLemma (mathematics)Manufacturing execution systemMaxima and minimaMenu (computing)Interior (topology)Binary fileExecution unitNormal (geometry)Electronic program guideIterationModulo (jargon)Functional programmingExterior algebraElectronic mailing listQuicksortMappingDifferent (Kate Ryan album)Line (geometry)Type theoryReduction of orderComputer configurationSinc functionParity (mathematics)Initial value problemFront and back endsLoop (music)Right angleFunctional programmingLambda calculusLevel (video gaming)Error messageNumberSystem callMultiplication signMoment (mathematics)Stack (abstract data type)Inheritance (object-oriented programming)Function (mathematics)Parameter (computer programming)IntegerString (computer science)Disk read-and-write headVariable (mathematics)State of matterWordSemantics (computer science)Field (computer science)CASE <Informatik>Group actionTransformation (genetics)Figurate numberOptical disc driveSource codeSet (mathematics)DemosceneMathematicsEmailPlanningRecursionBitComputer animation
34:39
CodeLambda calculusSquare numberElectronic mailing listCausalityLink (knot theory)Different (Kate Ryan album)Set (mathematics)Library (computing)Revision controlReal numberProgramming languageDecision theoryCodeMultiplication signQuicksortBitFunctional programmingProgramming paradigmFunctional programmingObject (grammar)Goodness of fitComputer programmingSystem callCache (computing)Data structureAreaLambda calculusSequenceString (computer science)Initial value problemSummierbarkeitType theoryCountingRecursionMereologyStrategy gameInternetworkingIterationOffice suiteLevel (video gaming)Performance appraisalElectronic mailing listCore dumpMathematicsGroup actionFunction (mathematics)ResultantReduction of orderMappingRight angleCASE <Informatik>Object-oriented programmingMaizeTouch typingBuildingCurveWebsiteDocument Type DefinitionDefault (computer science)Database transactionIntegerParameter (computer programming)Lecture/Conference
43:18
XML
Transcript: English(auto-generated)
00:15
Thanks. I appreciate it. My name is Derek Pell. I'm an engineer at EMA Email Marketing.
00:24
That's me on GitHub and Twitter. Actually, if you go out to my GitHub, if you go out to the URL to the DjangoCon 2017, my entire presentation is being done in a Jupyter notebook and I've uploaded it to GitHub, so if you can't see the slide, you can pull
00:43
it up in GitHub and follow along. It won't run the code in GitHub, but you can look at it all, clone it, run it. It's Python 3. Quick show of hands, how many people showed up hoping I was going to explain monads? No, sorry. As far as I know, they're still just burritos,
01:04
so if you don't know what I'm talking about, Google it. Anyway, so what I want to do is talk a little bit about functional programming. Functional programming has become sort of the hot new buzzword, right? Coming from an object-oriented language like Python,
01:21
sometimes we might see functional programming as something mysterious. I want to kind of break that up a little bit for you and introduce you to some of the tools that help functional programming do its job and recognize that most of these tools exist in Python, not always to the best. They're not always the best tool to use, but they're all there,
01:47
and so I just want to sort of get everybody just an overview of what functional programming is. Why would we use functional programming? Why would we stop talking about object-oriented programming and look at this new style of programming? Well, functional programming is
02:01
very expressive, as we will see through some of the code examples. You can do a lot with very little bit of code. You can shrink your code base pretty considerably in the right places. It can be very efficient. Things run really quickly when your language actively supports
02:22
a functional programming paradigm, and we'll see some places in Python where it does that very well, some places where it doesn't quite so well. It can be safer. Hyphenated that on purpose to sort of emphasize the fact that it's still not safe, like there's still problems, but it can be safer. It can overcome a lot of the problems, a lot of the bugs that we see
02:44
in day-to-day object-oriented programming, and it's easier to work with concurrent and parallel programming. We won't actually directly dig into that, but that's really sort of the reason that functional programming has become so popular the last several years as we've stopped getting core processors that can go faster and faster, and instead using more
03:04
and more cores, being able to effectively take our code and spread that across a couple of different cores is the next step in making our applications run faster. Functional programming can help lead into that. But what is functional programming? It's really
03:28
functional. It utilizes pure functions, which are functions that don't have side effects, and it uses those functions to transform data. So functional programming actually thinks about data in a very different way than object-oriented programming. Object-oriented
03:43
programming, we hide our data away, and we're very particular about how we use it and whatever and who can access it and that sort of thing. Functional programming is a more open data model. It's like, no, there's no need to hide your data away. Let's take it and work on it in a more mathematical sense. That's where we try to have pure functions,
04:04
things that don't have these side effects because they don't always make sense to have side effects like in the middle of your code. On the IO boundaries, of course, you have to have side effects. You have to write to a database, print to a screen, read from a database, that type of thing. But in the middle of your code, there's a lot of places where we do side effects where we don't necessarily have to. And functional
04:24
programming is a way of thinking that helps you sort of reduce those because a lot of times those side effects are places where you have bugs. So what I want to do, though, is start looking through some of the tools that help you get to functional programming. Functional programming is like saying, if you were
04:43
to explain object-oriented programming, you might start with, okay, this is how you build an object. But building an object or a class is not the totality of object-oriented programming. In the same way, there's not one part of functional programming. It's really just a set of tools and a mindset on how you use them.
05:01
So with that, let's start looking at the tools. The first one is higher order functions. If you've never come across this phrase before, and a lot of this stuff you guys may have seen, you just may not realize that it is sort of like hand-in-hand with more functional thinking. But higher order functions just means that functions
05:22
get treated like first-class citizens. They can be passed to other functions as parameters, they can be returned from other functions as values, and they can be stored in variables. So they're just like any other piece of data in your application.
05:42
So let's look at a couple of examples. So here is something that's sort of real-world-ish. We do this kind of thing quite a bit at work where we will make a function that knows how to talk to our database. It knows how to make the initialization
06:01
to the database, or maybe it's just pulling a connection off a pool or something or other. But at the end of the day, we have one function that knows how to grab a cursor to our database. Then we have all these other types of functions that need that cursor. So we may have inserts that will return a row ID, but that's going to
06:21
be different than using a query that's going to return you an entire result set. So instead of trying to make each individual query make its own connection to the database or make some function that connects to the database in a way that's versatile enough
06:41
to return a row ID in this situation, but return an entire result set in this other. What we do is we make a function that knows how to connect to the database, grabs the cursor, and then we make other functions that use that cursor, and each individual function, all it needs to know is how to use the cursor. It doesn't need to know
07:03
how to connect to the database. So what we do is we have something along the lines of call database, which takes a function, somewhere in there it makes the cursor, and then as its return value, it calls the function that we passed in, giving it the cursor. So one example might be where you query a database, the query database function takes
07:27
a cursor, and then it knows how to loop over the result set or grab the result set and pass it back or whatever it needs to do. It just depends on your business logic. But the idea is that we're passing, when we do this, we're going to, like this next
07:44
line down here, we're going to take the results, we're going to get them by calling directly to the call database function and passing it in the query database function. So we can see here, call database is doing its thing to initialize the database, then
08:00
it gets a cursor, and it passes that cursor into whatever function we called it with, which in this case is query database, so it's querying the database with our cursor. But if we also need to do something slightly different, we can do something like returning the new row value. So we can reuse this same call database function because all it
08:24
does is one thing, work on the cursor, and it just passes the cursor in. Whenever we pass it a function, it calls that function and passes in the cursor. So that's a place where we use quite often passing a function into another function. You can also return
08:45
functions. So this is a little more contrived example. This is probably not something that would actually happen quite a bit, but if, for instance, let's say you had some data coming in somewhere, and you had – it could potentially come in from several different
09:00
sources. It could come in from a source that needs to read off S3 or another source that needs to read out of a SQL database. You can create these functions that do whatever they need to do. Like you have an S3 function that knows how to call out to S3 and retrieve the data, or another function that, like we just saw, that calls out to the database
09:20
and retrieves data from the database. But you need to dynamically choose which of those functions based on the source coming in. So this essentially becomes almost like a poor man's polymorphism, but it's a pretty straightforward example. What we do is we have a source that comes in, in this case S3. We ask for a new function
09:45
by calling out to this give me a function. Give me a function takes our source and says, okay, well, it's S3. I'm going to return this complex S3 function to you, right? It doesn't call it. It doesn't do anything with it. It's just passing the function back to us, and then it's up to us, once we get this new function, it's up to us to actually
10:05
call it with whatever we need to call it with. We're essentially saying, we're coming in from S3. What do I do with it? Give me some function that will know how to handle S3, or give me some function that will know how to handle SQL. And because these are
10:25
actually what is passing back are functions, we don't actually have to even do this where we assign it to a value and then call it. You can just call it straight away this way, right? So we're calling into give me a function, passing it in our source
10:42
and whatever comes back we immediately call it just like we would any other function because that's all it is. It's just passing this back a function. Now, anyone who's... Like I said, you may have seen this sort of stuff, especially this type of thing we use a lot when we're doing JavaScript if you have to do that, but we actually use
11:02
it a lot in decorators. Who has worked with, created, looked at decorators? A lot of people? Yeah, oh, yeah, the entire room, right. So yeah, that's all a decorator really is. You're passing in a function to a decorator function, in this case decorate it. That has a nested function, which we typically call wrapper, and
11:27
then it does whatever kind of work you need to do before calling your function, whatever kind of work you need to do after calling your function. So yeah, this should be pretty straightforward to everybody who's ever used
11:40
decorators before, and that's all it is. So without higher order functions, Python couldn't have decorators, so we have to be able to pass those functions around. And so you see it's not just usable in functional programming. We use it all the time in object oriented programming in certain languages. And that's sort of the
12:01
key is like functional programming isn't really a mystery. It's really just a set of tools that you use in a slightly different way. The next set of tools I want to look at is our immutable data or immutable data structures. Now this is something we don't use as much in Python because mutable data is sort of the bread and
12:21
butter of object oriented programming. But for immutable data, it's a structure that never changes itself. So like you have a list, you don't actually add something new to the list. Instead, your list can't be changed. Sorry about that. Your list is sort of set in stone. And we'll look in a little bit about how you
12:43
overcome that because that seems like a limitation, right? Like you create a list, you can't add to it, you can't delete from it. What good is it? How do you make any changes? You have to think about your list in a completely different way because whenever you make some update, we're used to passing in a list, adding something to it, getting that list back. With immutable data, you don't do that.
13:04
You pass in a list, if you add something to it, you get a brand new list back. It's a completely different data structure. Same idea if you take something off the list. You know, you pass in the list to a function, it takes something off that list, what you get back, completely different list.
13:22
That's a very safe way of handling things though. That means that list isn't going to change out from under you, right? How many times have we all done this, tried to track down a bug, where because we've passed off a list, we don't realize that whatever function we're passing it to is mutating that list before it comes back to us. Well, with immutable data structures, you don't have to worry about that. You can pass them a list,
13:43
they can do whatever they want to with it. Your list never changes. Their list gets adjusted to a new list. Now, we don't do this a lot because historically, they have been very inefficient. What you have to do is take that list, it's sitting in memory, the language has to then copy the entire
14:01
list to a new part of memory so that you essentially have two lists. It's very slow to do, but there have been new data structures called persistent data structures that make this much more efficient. If you download this, I've got a link to persistent data structures, to the description in Wikipedia. I'm not going to dive into them, but they're much more efficient at this sort of thing. So it lets you use
14:22
an immutable data structure without paying the additional cost of like all that copying over in memory. If you're not familiar with the problem with mutable data structures, and probably almost all of you have seen something like this, this is like almost every language I've ever looked at has this
14:42
sort of warning of this kind of thing where you create a list and then you create a second list, you point a second list to the first list. Anytime you change that first list, you've automatically changed that second list, right? So that's not a safe way of making sure your list
15:01
doesn't change. You can't do that typically. You can see here, list one and list two, they are the same thing. They're basically the same list because what we have are two pointers to the same piece of memory. And this can be a problem here in that sort of situation, but you can also create some really interesting problems if you're not thinking about it.
15:24
So this is just a quick function that goes through, takes a list of integers and then sums up that list, right? And gives you the results. So if you pass in the list one, two, three, the summation of that is six. But if somewhere in there you accidentally start appending to your list,
15:42
what you're going to get are some really weird results, right? You're in the middle of working your way through a list in a way that we do every day. It's just a for loop. But somehow we've appended to that list in the middle of that loop and now our list is going to continue to grow. I added this if clause in there because if I don't catch for a list of a certain size,
16:03
this will grow until I run out of memory. So that's one of the things that we end up having to track down sometimes when we work with mutable data. But if you work with the mutable data, a lot of those situations disappear. So right here, we're doing the same thing that we did above with the list, right?
16:21
We're creating a tuple instead of we're creating a list. We're creating a tuple, adding some items to it, and then creating another tuple pointer to the same tuple, right? So we've got two pointers to the same tuple just like we did with the list above. But now, when we update the one tuple, it does not update the second one
16:40
because in Python, whenever you try to update, whenever you try to concatenate an item into a tuple, you get back an entirely new data structure, right? So before we do the concatenation, we look at these two, tuple one and tuple two. They are the same. After we do the concatenation, they're no longer the same because Python has created a brand new data structure for us.
17:01
It's a safe way to do it, but it's slow because these were not created as persistent data structures. Typically, if we want an immutable data structure, we go to a tuple. You can do something along the same lines with list or dictionaries. You can use a deep copy, and instead of actually just making two pointers,
17:24
you make a deep copy of the list. That way, when you change one, it does not change the other because you start off with two different lists. But again, it's slow because it has to take that list, copy it over in memory. So those are some of the problems that we can see with immutable data
17:40
that we can hopefully solve with more mutable data structures. But working with immutable data structures means we have to start thinking about things slightly differently often. Like we don't want to do a for loop on an immutable list because it doesn't loop quite as well.
18:03
It'll loop in Python. In a lot of functional programming languages, though, they just pull the loop out of there because if you aren't mutating your list, you don't necessarily need loops. You can come up with other ways of getting through everything in your list. And one of those ways is recursion, or the big way is recursion.
18:22
If you've never worked with recursion, most of you have probably, I assume, seen it, but recursion is when you call a function from within itself. Like I said, it takes the place of loops in many programming languages, and we'll see that here in just a moment, how we can use recursion instead of a loop.
18:41
The problem with recursion is that every time you make a call back to the function, you add a frame to the call stack, and that call stack can grow and grow and grow. In languages that embrace recursion, they often do this thing called tail call optimization, which lets you more efficiently do recursion
19:02
without adding to the call stack. But Python does not have tail call optimization. Guido has, if you've ever looked at it, he's sort of famously come out against tail call optimization in Python because it hides your stack trace. What tail call optimization does is instead of building up every stack, any time you call back, if you do your recursive call properly,
19:23
it just overwrites the existing frame on your stack so that you don't build. You just sort of keep reusing the same frame. It's much more memory efficient, but you do lose your call stack if something happens five or six calls deep. You don't know exactly where it happened. You don't have all the context for it.
19:40
We probably will never get tail call optimizations. Regardless, let's take a quick look at recursion. This is a recursive version of the summation list that we did above. So above, we're giving it the list and looping through the list. Here, what we're doing is we're giving it the list and then instead of looping through each item,
20:03
we send the list back into the function itself. So the function, in this case, has to take a list, but it also has to take the summation. Because we are recursively calling back into itself, it's harder to keep track of the state. You typically end up having to pass the state back into the function along with whatever...
20:26
Well, in this case, we're passing the list in. We also are passing in the cumulative sum. Depending on what algorithm you're using, there are ways to do this without having to pass the state back in. But in this case, we are.
20:41
For recursive functions, you almost always have to have a base case. Base case is the point that tells the recursive function stop recursing, we're done, just return a value, don't call back into the function again. So in this case, we're done when the list is empty. When it's just an empty list,
21:01
we have summed up everything in the list, just return the summation. If that's not the case, what we do is we pop the last item off the list, add it to the current sum as the new sum, return everything back into the loop again. So that's our sum list, our function,
21:23
and here we're calling it. We're calling it with a list one, two, three. We get our result of six. Now, we still have the same issue that we had with the iterative approach, which is if this list gets changed while we're doing the summation, it's going to mess up our recursive calls.
21:42
This does not fix that problem. That's a mutable data structure problem. But if you were to try to implement this in a more functional language, this is the kind of thing you would do. You would loop through it using a recursive call rather than an actual for loop. Now, one of the problems with recursion inside Python
22:05
is if you try to go too big, if you try to recurse too many times, you can get a recursion error. Now, in a lot of languages, you don't have something called a recursion error. Python has built in a limit on the number of frames you can put in the stack,
22:22
which by default is 1,000. You can up that if you need to, but it's sooner or later you're going to run out of memory. If your list is big enough, if you recurse deep enough, you're going to run out of memory because the language has to keep track of everything on the stack. That's where you get a stack overflow. So in most other languages,
22:41
if they don't limit your recursion depth like Python does, then you'll just recur and keep going and going until you run out of memory and get a stack overflow. So without tail call optimization, recursion is actually really dangerous to do. So you don't see it a lot. I've never seen it in the wild in Python,
23:00
and there's a really good reason. So if you don't know how deep your recursion is going to go, come up with a different solution, you can always use an iterative solution. There are no problems you can solve with recursion that you can't solve with just a loop in some capacity. So in Python we definitely tend to favor loops over recursion.
23:24
So anyway, those are some of the ways that recursion gets used, and that's how we kind of get around some of the problems of mutable data. I don't think I pointed that out well, but essentially what you're doing is you're taking that list and sending a new version of the list back through the recursive loop each time.
23:42
In this case, we're popping the last item off, but in other situations, a lot of functional languages have this concept of the head and the tail of the list. The head of the list is the first item. The tail is everything else, and a lot of recursive calls will use the head and pass the tail back along. So that's sort of essentially what we're doing here.
24:04
So yeah, those are, if you've looked into functional programming just sort of at a surface level, those are probably the minor players that we've gone over so far. The big three, the holy trinity of functional programming are map, filter, and reduce.
24:22
And this is where the more functional style really shines out well in Python. If you've never used any of these in Python, if you get nothing else out of this talk, I hope you can walk away with a sense of I need to check into map, filter, and reduce and figure out how to use them because they make your code.
24:40
Once you get used to how they work, they're beautiful. So a lot of times, though, to really use these tools, map, filter, and reduce, we a lot of times end up using lambdas. Lambdas are anonymous functions, and that really just means it's a function that we don't name, right?
25:01
Instead of saying def call db, we just have, well, this is how it falls out in Python. You use the keyword lambda, you pass it whatever variables you need, in this case, x and y, and then there's a colon that separates the variables from the body, right?
25:23
Lambdas are used for typically one-liners. If your lambda gets much more complex than you can fit on one line, you're probably better off just making a named function and going the typical route there. But lambdas are really good for these inline one-offs where it doesn't even make sense to make an entire function
25:41
just to add two numbers together, right? You can just throw it into a lambda. In this case, we're assigning that lambda to a variable named sum. We call sum just like we would any other function, and it sums up whatever we give it, right? You don't usually use lambdas this particular way, but you do use them quite a bit in map, filter, and reduce.
26:02
So let's start looking at those three tools. Filter does exactly what you think it would do. You pass in a filtering function, which we'll go over in a moment, and a collection to operate on. The actual filter function
26:21
takes whatever filtering function you pass in and applies it to every item in the list one at a time. So it takes the first item, runs it through your filtering function, the filtering function needs to evaluate to either true or false. So if it evaluates to true, the filter will then take that item out of the list
26:43
and add it to your new list. If it evaluates to false, that item does not get added to your new list. Technically, map, filter, and reduce all return iterators now. In Python 2, they would return lists, but now they're returning iterators, which is much better. That way you can kind of evaluate that as you want. It doesn't build your entire list at one time.
27:01
It yields back one item at a time so you can loop through it however you need to. But essentially, if you take something like that list, if we look at our filter right here, we're going to create a new list, iterator. Using filter, we pass it in a lambda that's basically just saying,
27:22
looking to whether or not the value that we're looking at, is it even or odd, right? If it's even, then we assume modulo two is equal to zero. Then it gets added into the iterator. Modulo two doesn't equal zero. It doesn't get added to the iterator. We're passing it list one as the collection that we want to iterate over.
27:45
And so it's filtered out all of our... I guess you can say it's filtered out all the odd integers. It's filtered in the even integers. I don't know semantics. But in any case, that's what filter does.
28:02
You can pass it a lambda like this, or you can do something much more complex, as long as the filtering function ends up evaluating to true or false, so filter knows whether or not to add that value into the new list. The same kind of thing is true for map.
28:21
Map takes a mapping function and a collection to operate on, and then it applies that function to every item in the collection one at a time. It does not filter out any items. If you pass in a 10-item list, you will get back a 10-item list, or you should get back a 10-item list. But what it does is
28:43
it says, I have this whole list of users, and I want to transform them in this particular way. You can use map and give it whatever function that's going to transform your list of users and then give it the list, and it will go through and do the transformation for you. It's a simpler way of...
29:01
You just don't have to call it all in a loop. Normally, if you're not using map, you might, just for every item in this list, call out to this function manually. Map does it for you. Map is actually a lot more efficient at it. It will run a lot faster. Map is... If you have one of those simple situations
29:20
where you just need to pass every item in a list one at a time to a function, map is a really good alternative to using a for loop. And same kind of thing here. We're calling map... We're using list one. We're calling map. We're passing it this lambda, this anonymous function that all it does is
29:41
take whatever item I give it and multiply it by two and then put it back in the new list. So there, we've taken list one and we've multiplied it by two, every item in there by two, and get back a new list. And you'll notice, even though we did filtering on the list one earlier, we didn't actually adjust list one at all.
30:02
We got a brand new list back. Again, because this is happening, because these functions were built, map, filter, and reduce, were built to work this way, they're much more efficient than if we tried to reproduce them ourselves. So this is actually a safe, efficient way to go through a list of items
30:21
and do something with it, rather than just sort of manually looping through it, risking the possibility of adjusting your list in line or anything like that. So this is a really safe way of doing this type of thing. And the last of these is reduce.
30:40
Reduce is a little bit different than the other two. So what reduce does, it starts off the same. It takes a reducing function, and it takes a collection to reduce. But the reducing function's a little bit different. It doesn't just act on one item in the list, it acts on two items in the list
31:02
at a time. You can actually send it an optional initial value, so the reducing function, instead of acting on the first and second item, it'll start off acting on the initial value that you send it, and then the first item. And the way it works is this. Your reducing function takes these two items,
31:20
does something to it, so that the output is one item of the same type that comes in. Now, we're not strictly using type, like integer versus string versus whatever. But essentially, what's going to happen is it will take the reducing function, the reducing function will take two values,
31:40
do something with it, spit back out a new value. That new value then becomes the first parameter when it calls the reducing function again, and the next item in the list becomes the second parameter. And so let's take a quick look here.
32:01
And reduce was the one that, for whatever reason, was hardest for me to wrap my head around exactly what it's doing. Map and filter were pretty straightforward for me. Reduce was a little harder to overcome, so I've got a couple of examples that might help it solidify. But in any case, this first one is the sort of standard reducing example.
32:20
We're calling reduce on this anonymous function that just takes two variables and adds them together. So, just like we did above when we were looking at lambdas initially, we're just taking x and y and adding them together and returning that value. What this is going to do, though, is it's going to start with one and two out of the list. That's going to be the first x and y.
32:40
It's going to add them together, and you're going to get three. Three is then passed back in as the x, and the next item in the list, which is also three, gets passed in as y. Those two get added together to get six. Six gets passed back in as the x, and the next item in the list is four, gets passed in as y,
33:01
and at the end of the list, you end up with your sum of ten. Now, this is the same kind of thing that we saw above. We saw an iterative approach and a recursive approach to taking a list of numbers and adding them all together. This is using a reducer
33:21
just to do the same thing, and it's so much better. If this is the kind of thing you need to do, you need to start looking at reduce. For one thing, before, when we tried to use the recursive approach, and we added a thousand, we blew the sack. We had that recursive error.
33:41
Here, because of the way it's built behind the scenes, it adds the first thousand numbers all together without blowing through the stack. It knows what it's doing. But, it also does it really fast. That's the first thousand numbers. It's the first ten thousand numbers. It's the first hundred thousand and one numbers.
34:01
I mean, it's super speedy. It's incredibly efficient. This is probably much faster than we could manually go. It is much faster because when we were doing the recursive approach above, I started it off, and we had to wait for a moment for it to blow through the stack. This is going through it just lightning fast. These things were made to do this type of
34:20
work. They're very efficient on the back end. You really need to start looking at map, filter, and reduce as viable options for whatever kind of, like, wherever you think you can put them. It's probably a good place to put them because they really do make things so much easier and so much faster.
34:42
I want to go through one more quick example. There's a couple of examples you can pull them down and look at them if you want. I like this example. Essentially, what this is going to do is it's going to go through and count. It'll sum up all of the A's in all three of these strings, right? So, this is, imagine this is a DNA sequence.
35:00
It's got three A's here. The next sequence has two, and the last sequence has none. The reducing function in here is taking A and X. In this case, we're starting with the initial value of zero. So, the first time it goes through, the A is zero and the X
35:20
is whatever the next item is. In this particular case, X is that particular string. What the reducing function does is it takes that string, does a count of the A's, adds that count back into whatever the A, whatever the A value is, and kind of sums it up that way, right? This is the same kind of problem
35:42
we were trying to solve above, more or less. Anyway, I forgot which one it was. In any case, this is a really efficient way of, like, you can give this huge list of DNA strings, and it will go through and tell you all these markers in it
36:00
just by using a reduce. You can also mimic, map, and filter inside Python using list comprehensions. List comprehensions are something I'm sure every one of us has done day in and day out. You can throw a lambda inside a list comprehension. If the lambda
36:21
is too ugly, you can pull it out. You can also throw a filtering function more or less inside a list comprehension. Anyway, these are tools that are already built into
36:42
Python that we use day in and day out, but we don't think of them as being functional programming, perhaps, but they really are. At their core, they do the same type of thing that we would do in functional programming. That leads us to the question of, should we bring functional programming into our code? Because it is a really different paradigm.
37:00
I think, of course, you should. Like I think I've shown in a couple of different places, functional programming can be less verbose. It can be much more efficient, and it can help you think through problems in a way that maybe you weren't able to think through before, so maybe solutions become more obvious when you're using a different set of tools. But there are some very real drawbacks, as we've also seen.
37:22
Working with immutable data can be slower, especially in languages like Python. Again, there's a persistent data structure library that's in Python now that someone's building. I don't know anything about it. I haven't had a chance to look at it. If it turns out it works well, then I'm going to start trying to use it more in my code because it's much safer.
37:43
But immutable data is still slow in Python as it's the built-in versions. Recursion can blow through the stack very quickly, as we've seen, so recursion is not a good solution in most cases. And functional code looks sort of weird. I have a coworker who loves to throw functional code into our Python,
38:01
and every time I hit it, even though I like looking at functional code, every time I hit it, it's still a bit of a speed bump. When I first started hitting it, it was a brick wall, so it totally slowed me down and stopped me. But now that I understand what's going on with it, I can scan through it more easily and understand it. So if you're going to bring functional programming into your code, probably something you need to discuss
38:21
with your team or your management, if you're working on a team, and decide as a team, hey, is this something we really want to approach? Because if you just start throwing it in there and your teammates don't know anything about it, they're going to have quite a learning curve, but you can help them learn, and that's kind of fun. At least I think so. So that's our
38:41
speed run through functional programming in Python. How much time do I have? Okay, all right. Does anyone have any questions? Thanks. Thank you for the talk. So you mentioned that in some situations it might be good to bring functional
39:02
programming into your code, but I was wondering, it sounds like you are saying it can be possible to just have a bit sprinkled in here and there, is that right? Is it kind of, do you recommend having it basically be the paradigm, like choosing one or the other, or do you think it can succeed if it's half and half or sprinkled in?
39:21
Yeah, definitely in Python it's better to sprinkle it in. Python, while it supports all of these tools and a whole slew of others, there's the functools library in Python which is really great for doing these things efficiently. Python is at heart an object oriented language and so making that full transition into a
39:41
functional paradigm probably isn't really going to work out. There are a lot of good places where you can put it where you might not think you can. One thing I didn't get a chance to touch on is the idea of data pipelines. So one of the things you can do is compose functions which is sort of like if you've ever done any Unix programming
40:01
where you pipe output from one function call to another or if you've done like maybe node working with promises where you can then one function, the results of one function call into the next. That's composing functions. You can build really great data pipelines doing that and you can do that in Python and Python is really great with
40:21
like data science kind of things. So there are a lot of places where you can use a more functional mindset but it's probably not the paradigm you're going to run with completely inside Python. You might be able to and if you can let me know because I want to know how that works out but I've never been able to do that. Hi, great talk. Thank you.
40:43
You mentioned emulating a map with list comprehension. I was wondering other than considerations of who else is reading the code whether or not they're familiar with functional programming are there technical differences between the two and how should you decide which one to use? I don't know the technical differences.
41:02
There are. I'm just going to assume that there are. I know the difference that you're going to run up against would be efficiency and speed. My guess and again this is just a guess but my guess is now maps are probably
41:22
they're not going to be any slower than doing a list comprehension. I can almost safely say that. They may be a little more efficient and a little faster because of the funk tools that have been brought in. Map and filter are still a core part of Python. Reduce has to be imported
41:40
but I'm going to go out on a limb and say it's a coin toss? Probably more or less. You might get a little more efficiency with one or the other. You're probably not going to notice a huge difference. If I'm wrong about that I'm sure the internet will let me know. Has incorporating functional programming in your code changed your caching strategies at all?
42:05
It's not changed our caching strategies. I can see how it would. We haven't gotten to that place yet. Really there's only one or two other people at my office who start implementing functional
42:20
programming so we're still I'm not even as a company we're building it out. It's just something that some of us know about and like oh yeah this is a really good place to throw a map in or whatever. Our former architect loved functional programming and he's the one that first started putting it in and he's got
42:40
weird functional stuff all over the place and that's where I sometimes hit a brick wall looking at functional code but I can definitely see how it can change our caching strategies especially since a lot of these like map and filter are returning iterators now so they're lazy evaluations. We don't have
43:00
to like cache this whole huge change of a list or whatever. We can sort of slowly build those changes and use them as needed and then kind of get rid of them. Alright great. Thank you very much. Thanks.