Declarative Thinking and Programming
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33710 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017114 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Declarative programmingComputer programmingSoftwareComputer fontMathematicsLevel (video gaming)AbstractionTask (computing)Flow separationSingle-precision floating-point formatRange (statistics)Set (mathematics)WordDeclarative programmingDigital signal processingEmailTransformation (genetics)Query languageElectronic mailing listMultiplication signAddress spaceResultantWave packetAbstractionDigital photographyArithmetic meanNumberDegree (graph theory)Focus (optics)Task (computing)Equivalence relationProjective planeOrder (biology)Field (computer science)Physical lawBitProgramming languageState observerBuildingLevel (video gaming)Computer fontMathematicsComputer programmingFlow separationCodeFunctional programmingSoftware design patternOrder of magnitudeMereologyJava appletBus (computing)CuboidLoop (music)Information engineeringPlanningWeightQuicksortBlock (periodic table)Different (Kate Ryan album)Right angleNatural numberSquare numberMobile WebModulare ProgrammierungForcing (mathematics)CASE <Informatik>Boss CorporationPower (physics)Imperative programmingCommunications protocolWorkstation <Musikinstrument>Scripting languageGroup actionInstance (computer science)State of matterComplete metric spaceMoment <Mathematik>Computer animationLecture/Conference
09:31
Declarative programmingElectronic mailing listComputer configurationFunctional programmingFunction (mathematics)Default (computer science)Equals signComputational complexity theoryProgramming languageComputer fileMarkup languageComputer fontMathematicsNumerical digitVertical directionNumberSquare numberPrime idealSubsetQuery languagePersonal digital assistantInformation securityInformationDisintegrationComputer networkMathematical analysisRule of inferenceStructured programmingFirewall (computing)Level (video gaming)Rule of inferenceLibrary (computing)Integrated development environmentInstance (computer science)SoftwareCodeComputer configurationFunctional programmingTask (computing)Modulare ProgrammierungBitNumberCondition numberComplete metric spaceQuery languageDatabaseCASE <Informatik>String (computer science)State of matterAbstractionPrime numberDeclarative programmingLogic programmingParameter (computer programming)Computer fileMarkup languageData typeElectronic mailing listDefault (computer science)RootOrder (biology)Online helpParallel portHash functionDigitizingAlgorithmMappingMathematicsArc (geometry)Configuration spaceComplex systemProgrammanalyseRight angleWritingSquare numberGoodness of fitData dictionaryUniverse (mathematics)WordSummierbarkeitPower (physics)SubsetEquivalence relationProjective planeMixture modelInformationLie groupMetropolitan area networkResultantComputational complexity theoryCodeCapability Maturity ModelPie chartPhysical lawMultiplication signPattern languageWebsiteStandard deviationProgramming languageMatching (graph theory)Set (mathematics)Data integritySource codeInformation extractionComputer scienceDecimalInformation securityMomentumPatch (Unix)
18:50
DivisorPrime idealMathematicsNumerical digitHand fanAdvanced Boolean Expression LanguageWindowScalable Coherent InterfaceElectronic meeting systemCodeError messageFlow separationComputer programmingAbstractionDeclarative programmingLinear algebraNumberLevel (video gaming)Domain nameDigitizingMereologyAbstractionGraph coloringComputer programmingCASE <Informatik>Metric tensorOperating systemFlow separationPartition (number theory)Prime numberSemiconductor memorySource codeError messageRun time (program lifecycle phase)Operator (mathematics)Electronic mailing listRecursionCondition numberPoint (geometry)State of matterDeclarative programmingRootLine (geometry)SoftwareLimit (category theory)Execution unitDataflowModule (mathematics)Online helpCodeLogic programmingEquals signFrame problemRule of inferenceMultiplication signPiInstance (computer science)Arrow of timeMappingLink (knot theory)Cartesian coordinate systemConstraint (mathematics)LoginDialectMultiplicationLetterpress printingCombinational logicMatrix (mathematics)CuboidLatent heatSet (mathematics)Square numberSeries (mathematics)Right angleHypermediaAlgebraMessage passingDifferent (Kate Ryan album)Moment <Mathematik>System programmingPredicate (grammar)Stress (mechanics)Beta functionTheory of relativityIntegerExpert systemUniform resource locatorMobile appProgrammschleifeDivision (mathematics)Software frameworkCode refactoringQuery language
28:09
Multiplication signMereologyControl flowSubsetPrologLecture/Conference
28:44
Operator (mathematics)NeuroinformatikExpert systemLecture/Conference
Transcript: English(auto-generated)
00:04
Hello everyone, thanks and a few words about me. So my name is Florian Wilhelm and I'm a data scientist at Innovex. Innovex is an IT project house in Germany and its focus is on digital transformation.
00:23
So we have projects in the fields of big data engineering and data science but also web, mobile and so on. In my spare time I like to contribute to the Pi data stack and I also am the creator of Pi Scaffold and I really found it cool that Pi Scaffold was actually mentioned
00:42
a few times in some of the talks. So short hand up, who of you has used Pi Scaffold? Just to, okay only a few. Okay but to the actual talk. So the outline is first give a small motivation
01:01
and about declarative thinking about the concept behind this then I'll show some examples in Python how you can actually apply this way of thinking and in the end I conclude with a little math riddle so everyone likes riddles hopefully. So let's assume you just move to another city
01:23
and after all the moving business you want to make a little housewarming party and invite your friends. So let's say you have two friends, two best friends, Alice and Bob and you know that Alice, she's really good with tech and so on
01:42
so you just send her an email with your new address saying okay please come over, I want to make a little housewarming party and be there at that time at that date and you know she will be all right, she will use Google Maps or whatever to figure out how to get to your new address and to be on time.
02:02
But let's say with good old Bob it's a bit, it's a different kind of story. So he's let's say barely able to write emails so you send him not only an email with your, with a new address and place and date, you also make the whole trip planning for him.
02:21
Let's say you say okay it would be best if you take that train from your city to my new city and then leave the main station, take this tram and the dead bus and finally you will arrive on time. So with this example we directly see, so we interacted completely differently with Alice and Bob.
02:42
So with Alice we more or less stated what we want of her, we declared what we want of her and with Bob we not only told him that he should be at the housewarming party, we also defined how he should actually accomplish the task. But it's not always that easy to tell the how from the what
03:03
and the how clearly is more in imperative way and the what is more in declarative way. So to see this let's take a look again what our actual task was in this case, in this example. So we were, our task was to do,
03:20
to plan like a housewarming dinner. So we have different task like to clean up the mess, find an easy recipe to cook for your friends, put the last remaining boxes in the basement, buy maybe more beer and one of this, only one part of this was to invite your friends. So this is the kind of abstraction layer
03:40
we are dealing with if we want to invite, to make a housewarming party. And we see that if we have given this abstraction layer, so with Alice we stayed actually on that abstraction layer and with Bob it's a completely different kind of story. We went down to actually planning a trip for him.
04:01
So clearly we left this kind of layer of making, of planning a housewarming party. And this is an important aspect actually when talking about declarative thinking that one has to think about the level of abstraction
04:21
and where our tasks actually reside. And we know this from many other examples like if you want to take a picture for instance, then your kind of abstraction layer is that you know how to handle a camera and you don't care about all the inner workings
04:41
of how actually the photo is created. And to be a bit more techy, I mean we're in a more tech-related conference, right? So maybe you remember like five or six years ago there was this big hype about Hadoop, MapReduce and everyone was so hyped about this.
05:01
Yeah, it's so cool, it's just two easy building blocks, MapReduce and you can do basically everything with it. But then later people realized that, wait, so most of the tasks are actually more like query tasks, so more like SQL or SQL-related tasks. So having realized this, people realized,
05:22
okay, so but to do something like queries, to do this in MapReduce, it's a lot of work and it's actually not stating the problem anymore on that level, so people came up with things like Hive and Pig and you name it. So this is also an example that the layer of abstraction
05:45
is really important for this. So again, to sum this a little bit up, so I hope I conveyed the idea behind this imperative and declarative thinking in a way that imperative is more like specializing on the how
06:03
and you can see this if you over-specify things like when we talked to Bob, we said, okay, you should end at a train and stuff like this which is completely unrelated to the actual task of giving a housewarming party and detailed instructions and declarative is the more the what
06:21
and this is also related if we think about programming and also about design patterns. There was one talk about Python design patterns which I found really cool and one of them is separation of concerns and the single level of abstraction and those are also highly connected to declarative thinking and programming,
06:41
meaning that you wanna keep, you wanna modularize your code, you wanna make sure that one part of your code is actually dealing only with one concern and when you're calling, when you write a function, that you stay inside that function on the same level of abstraction,
07:01
not going up and down all the layers. But this of course, as I said, depends on the level of abstraction you're dealing with right now. So one note about abstraction, there's the so-called law of leaky abstractions by Polsky, so this is actually more like an observation
07:21
so Americans tend to call everything a law but this is more like an observation. So he said that all non-trivial abstractions to some degree are leaky and so from my experience, this is absolutely true. So I mean, many of you have surely used SQL
07:41
and maybe have also tuned SQL and you might have then realized that if you start with a slow query, you can sometimes do a lot of equivalent transformation of that query and you end up having the same result but the actual query is some magnitude faster than before
08:02
and this is clearly leaky abstraction because actually SQL should kind of cover the complexity below and so that you can just state what you wanna have but sometimes it's then better to know what's going on beneath to optimize something
08:23
and this is where even a language as old and as good as SQL, so good for this one specific task, is leaky. So this is something one should be aware of when talking about, yeah, when speaking about abstractions.
08:43
So having now the idea of decorative thinking, I wanna give some small examples in Python, some quite easy examples. So let's say our task is to get a list of squared numbers from one to 10
09:01
and imperative and maybe something someone coming from C or Java would maybe do is you would create an empty list and you would just say, you just loop over it and append the squared numbers of one to 10 and this clearly is imperative because we are,
09:21
I mean our task is just saying we want some kind of list and by here we are over specifying, we're giving an order of how we wanna do it and a more declarative approach would surely be just to use list comprehension as you surely all know and love list comprehension because they're so easy to use,
09:42
they're much more readable and actually we are staying more on the level of our task so we're basically transforming the task of getting a list of squared numbers from one to 10 directly to the syntax and this has many advantages so one could think that this could also be paralyzed.
10:03
I mean it's not paralyzed right now in C Python but I mean it could be possibly so someone could say okay, we changed the code in the back so under this abstraction layer and do it in parallel so this is also an advantage that it could be done besides being more readable.
10:23
Another example would be let's say you wanna write some kind of dispatcher so given some argument you wanna call different functions on some value and an imperative way of doing this would be just like you check if arc is option A
10:41
then you call function A with a value and so on and if it's not the case you call default and also here we are giving an order and we are going down to a much lower level actually. So what would be better is to use a dictionary here so some way more complex data type
11:02
and that data type, a dictionary is actually more like a mathematical mapping so you map one thing to another and I mean this is what we wanna do if we wanna write a dispatcher like this so basically with the help of a dictionary we can just say okay, we map one argument to a certain function and then we can just use
11:23
the get method of the dictionary to do this so this would be way more declarative and even if we had a lot of options this would even be faster because in the back it's implemented with some kind of hash map algorithm.
11:40
So maybe a little bit more complicated example let's say you wrote a paper, your paper is called B and you wanna see if someone stole from your paper. So you wanna check your paper against the paper A and the question is how would you do this? So of course the really naive approach
12:00
would be you take each sentence from your paper B and check all the sentences from paper A and see if they're equivalent. So this would be an algorithm if you do it like this with the complexity of O N squared or to the power of two so quite, yeah, quite a huge complexity
12:23
and if you think a little bit more about the problem you see that actually a good abstraction layer could be set theory. So given that Python has also a set data type you could then think of a better way to actually formulate your problem if you just wanna compare the sentences
12:41
and just put all the sentences of work A and work B into dictionaries and then finding the intersection is just as simple as this and this is also a nice example where like if you think a little bit more in advance about what kind of problem you're dealing with
13:02
and if you find the right abstraction layer for it then it's actually much easier to write code and yeah, it's actually way more readable and in this case also way faster because also this will use some hashing magic.
13:21
So another thing you experience quite often in the Python ecosystem are configuration files. I mean it was also mentioned I think in the first lightning talk and in other talks that in Python a lot of libraries like strings and also set up high,
13:41
set up tools, they use Python modules for configuration and this is quite bad I think because a Python module it's kind of hard to check what happens in there, everything can happen. It's a tooling complete way to do something as simple as a configuration
14:01
and for instance in IDE, so it was also mentioned in the PyCharm tutorial of PyCharm talk that an IDE like PyCharm could not check what kind of requirements you put into set up high if you put it there directly. It's much easier to pass something
14:20
like a requirements txt, txt. So I think the takeaway here is that it's way easier to use for configuration, a way declarative approach is to use something like YAML, some markup language or Toml for instance that is less powerful
14:44
so you have to kind of think more about your actual problem, how you wanna set up the structure of your configuration but you earn a lot with it. It's easier to check it and yeah, it's way better and this was also one of the reasons why I chose, I used for my projects PBR
15:04
which is in PyScaffold because there you can, instead of using a set up high, you can use a setup.cfg to do the actual configuration in a more declarative way. Okay, so coming from some really simple Python example,
15:26
let's talk about a little math riddle. So you might know the German newspaper Die Zeit, so the time, and they have this little math riddles called Lorelei, which is some kind of mixture word
15:42
between logic and Lorelei and I think Lorelei is some kind of mermaid that attracts you and then kills you. So this is kind of riddles you start and then you can't stop until you finally solve them or gave up and yeah, so this is a riddle
16:04
and it's like crosswords but with numbers and we see here that for instance we have to make sure that the digit sum of A is horizontal C and that C is a prime number and so on and one of my friends sent me this
16:21
and or showed me this and said okay, how would you solve this with the help of a PC or with the help of Python? And this got me thinking and then I remembered like yeah, back in the days at university someone mentioned prologue and declarative programming and yeah, logical programming.
16:43
So I thought okay, maybe I look this up again and see what I can do and see if I can solve this staying on the same level of abstraction so that I basically just want to state what's in the riddle and in the end come up with a solution.
17:01
So how many of you know prologue? Maybe, ah okay, quite a lot. So Datalogue is actually just a subset of prologue and it's easier so it is used to be able to reason more about Datalogue.
17:20
It's declarative and logical programming language. It's used to query deductive databases so databases where you just specify facts and certain rules and then you can ask this kind of knowledge database for certain facts if it can be deduced from whatever you've given before.
17:42
And it actually has quite many use cases so Microsoft is doing a lot of research in this regarding security, network security, they're checking some kind of firewall rules for this. Then it's also used for data integration, information extraction, networking, program analysis and of course you can do it for,
18:01
also use it for silly riddles. And there's a nice Python library actually called pydatalogue where you can just use it and run it as a Python module. And I just wanna show a little bit how one would go about this
18:20
and how it is possible to stay really at the level of this, yeah, of the conditions in the riddle. So we saw that one of those conditions was that some numbers are squared numbers. So how would you specify this
18:41
with the help of pydatalogue? So you would just say that squared x is, so x is squared if you can take the square root of it and if it's still an integer. And you read the leftmost smaller than equal sign
19:00
just as an if so the operator is overloaded and it's best to just read it as an if. And you can see it's just like you're just stating it. So and this is squared is now some kind of predicate for x and it's some kind of set of some, by this you're defining some set but this is all you need to know.
19:23
And the same goes for divisible if x is divisible by y and we can also check only for the remainder. It's divisible if you do this and then it's equal to zero. So maybe it's something more interesting. How would you check if a number is prime? We know that two is a prime number
19:41
so the plus operator is overloaded. We therefore just check if or state that two is a prime number as a fact and for other numbers and three and for the other numbers we use a rule stating more or less that all numbers larger than three which are odd numbers or not,
20:02
this is the tilde so the curly one, not divisible by two and if there's not any other factor between three and the square root of x and this we define by the second last line
20:20
with some kind of recursion. So in the last line we're basically saying so if y plus two is smaller than some upper bound so some square root and then we check all the numbers and in just a few lines we can state this lazily for all numbers. And since most of our conditions are defined on numbers
20:44
and we are actually looking for the digits we of course need to have some kind of mapping between digits and actual numbers and this is really easy so the digits, the number of two digits is just 10 times the first digit plus on and then plus the second and then we can just recursively define this
21:03
for all digits, for different number of digits. Here again, so, oh sorry, here at that point we would be now able with some more rules and so on to actually just write down one condition
21:24
after another of the riddle. And here is a part where the leaky abstraction bites us. So since what happens below is that what PyDataLag does, it generates a whole list
21:44
of solution candidates and starts eliminating them and so on and if you start with the wrong kind of condition first then it might happen that the list of candidates gets too large at certain points
22:02
and then you can run into an out of memory error or just a runtime is just too long. So this is then some kind of leaky because actually it should play no role if we wanna stay on this level.
22:21
So one thing to deal with this is to define certain parts, so like these partitions, these colored partitions and yeah, using these partitions now in case of this riddle helps to keep the number of solutions low at all time and then you just combine them. So just to give an example how this looks like
22:44
for the upper left corner. So for the blue corner, is this large enough to be red? So the first line a little larger. So this was C and we wanted that the prime numbers,
23:01
the two prime numbers, sorry, the two digits form a prime numbers. So in PyDataLog we can just easily state that A2 is a digit ranging from one to 10 because it's the start of a number so it can't be zero to 10, or sorry, one to nine and not zero to nine
23:23
and the same goes for A3 and then we just can use the prime check, the prime rule that we have defined and by this we can for all conditions in our riddle, we can just write them down
23:41
and actually it's just like as you see here, like each line in the box above is more or less one of the conditions in the riddle and it's really staying at this abstraction level, it's using no loops or over specification, it's just a declarative way of specifying a riddle.
24:02
How would we query this now? It's really as easy as just saying just like print and the riddle and the riddle is a combination of all those four boxes which I have left out here but you can look this up, I'll later show a link and you just say okay, now give me the numbers,
24:22
the digits that fulfill all the constraints I have given you and then it will just come up with a solution and not only the solution, so if there were more than one solution, it would print all solutions so we know now not only one solution,
24:40
we are also sure that it's a unique solution which also a really nice thing and yeah, if you follow the URL below, then you can also get the whole source code and I think this, so for me, this was a really nice experience to work with a logical programming language and yeah, maybe some of you got interested in this
25:02
and wanna try this PyData log out or maybe prolog directly and yeah, I wanna conclude that there are many other applications where actually declarative thinking and programming becomes much more important. So in one of the talks, also Nixo OS was mentioned
25:21
and this is one operation system where you have a declarative way of stating how you want your system to be. It's not like app get install and moving from one state to another, you have a declarative way and this has gained quite some traction over the last year. Also, a tool set, a framework like TensorFlow,
25:41
I mean, they included just the layer sub-module where you can now easily define like in a curious style of way your network because if you wanna build a network, if you're on that layer, you're not interested in really all the details, how you do the matrix multiplication, so on,
26:01
but it now provides like two layers, like this layer sub-module where you can just define layers and of course, also the low-level parts with where you can do like almost everything that's possible with linear algebra and others, yeah. And there's, for instance, also Luigi
26:21
which is kind of a competitor to Airflow and this is also using a more declarative approach. So to make a small summary of the talk, so what are the advantages of declarative way of programming and of thinking
26:41
because actually, it's more like thinking. So in most cases, you can actually improve the readability of your code. You're more like stating what you want, of course, and keep it then to some other parts of the code to do this. It also reduces the number of errors most of the time
27:03
because if you, for instance, if you write some query in SQL, if you would do this really with the help of some map reduce or if you would try to mimic this with your own code, you would surely end up having a lot of error and strange corner cases.
27:21
In many cases, it's also increased performance because, I mean, as a user, you stay at the same level of, at one level of abstraction and what happens below is then can be programmed by some domain expert like, yeah.
27:40
I mean, in case of SQL, it would be the relational algebra and so on. And yeah, and you're following the separation of concern. The big takeaway message here is, so it's kind of my definition for me how I think that declarative programming is, that declarative programming for me means finding the right abstraction layer,
28:01
describing the problem, and then using this layer. So thanks for your attention and yeah, welcome to your questions if we have enough time. Okay, we have about two minutes or so, not much.
28:20
So I will take one question. So who has a question? Ah, that saves us time. Okay, there's one. Hi, you said that Datalog is a subset of Prolog. What is missing? I mean, what are the future or?
28:41
It's kind of those, there's in Prolog this breaking operator where you can kind of stop the computation and this is missing. I think this is one of the most important thing which is missing in Datalog, but I'm also, I'm not an expert in this, yeah. But yeah. Okay, thanks.
29:01
Thanks. All right, that was it. So give him a hand, and remember to.