Next generation configuration mgmt: A small, FRP DSL for distributed systems
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46952 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Programming languageProblemorientierte ProgrammierspracheMereologyResultantBitVideoconferencingQuicksortMultiplication signComputer animation
00:35
MereologyProblemorientierte ProgrammierspracheVideoconferencingBitMultiplication signComputer animation
01:09
State of matterBus (computing)Boom (sailing)Right angleRandom matrixLecture/Conference
01:41
Online helpBoom (sailing)MereologyVideoconferencingLabour Party (Malta)Office suiteQuicksortSoftware bug
02:20
Boom (sailing)Bit rateHacker (term)Data managementBlogLecture/Conference
02:38
Hacker (term)BlogProgramming languageTouch typingMKS system of unitsFamilySoftware bugNoise (electronics)Physical systemError messageProgramming languageComplex (psychology)Real-time operating systemComputer animation
02:54
Problemorientierte ProgrammierspracheBitComputer programmingData centerProgramming languageComputer animation
03:14
Independent set (graph theory)Programming languageRight angleEngineering drawing
03:29
MKS system of unitsIndependent set (graph theory)Programming languageTypsichere SpracheType theoryEndliche ModelltheorieReal-time operating systemWordState of matterMultiplication signTask (computing)Connectivity (graph theory)Physical systemComputer animationEngineering drawing
03:46
Programming languageMultiplication signStructural loadEndliche ModelltheorieBit rateCategory of beingProgramming languageMathematicsComputer animation
04:02
MKS system of unitsProgramming languageDeclarative programmingFunctional programmingRead-only memoryProblemorientierte ProgrammierspracheLatent heatStructural loadDemo (music)Template (C++)Category of beingProgramming languageMultiplication signLibrary (computing)MereologyFunctional programmingStreaming mediaPoint (geometry)Single-precision floating-point formatNumberReal-time operating systemComputer animation
04:32
Structural loadDemo (music)Template (C++)AlgorithmGraph (mathematics)SynchronizationContent (media)Source codeEstimationRead-only memoryComputer fileoutputMetreSystem administratorDeclarative programmingProgramming languageContent (media)String (computer science)Physical systemTemplate (C++)View (database)Demo (music)Different (Kate Ryan album)CASE <Informatik>Right angleHand fanMultiplication signNoise (electronics)Computer animation
05:23
Demo (music)Content (media)MKS system of unitsTunisStructural loadGlass floatLoop (music)Graph (mathematics)NumberServer (computing)Endliche ModelltheorieOperator (mathematics)State of matterCodeCore dumpRevision controlPhysical systemCase moddingReverse engineeringType theoryComputer filePoint (geometry)Functional programming2 (number)Demo (music)Category of beingVirtual machineBitTheoryMultiplication signCASE <Informatik>Computer animation
07:00
Meta elementContent (media)Magneto-optical driveMathematicsCodeRevision controlComputer fileCodeMultiplication signComputer animation
07:16
Total S.A.Operator (mathematics)2 (number)CASE <Informatik>Computer fileRight angleComputer animation
07:39
Graph (mathematics)System callTunisInterior (topology)SynchronizationTotal S.A.MKS system of unitsComputer virusComputer fileMultiplication signReverse engineeringAsynchronous Transfer ModeReal-time operating systemQuicksortCrash (computing)Content (media)Operator (mathematics)BackupVirtualizationVirtual machineComputer configurationCASE <Informatik>State of matterConfidence intervalComputer animation
08:39
Virtual machineDefault (computer science)MKS system of unitsCode2 (number)State of matterMereologyFinite-state machineVariable (mathematics)Statement (computer science)Programming languageBranch (computer science)CASE <Informatik>Demo (music)Graph (mathematics)Multiplication signComputer fileObject-oriented programmingDefault (computer science)Condition numberSet (mathematics)Computer animation
10:01
AlgorithmGraph (mathematics)SynchronizationHard disk driveComputer virusEndliche ModelltheorieCountingAutonomous system (mathematics)MathematicsCodeState of matterComputer-assisted translationComputer fileError messageBitEmailMultiplication signInheritance (object-oriented programming)Server (computing)Intelligent NetworkAnnihilator (ring theory)Physical systemDemo (music)Cellular automatonReal-time operating systemComputer animation
11:05
IterationError messageMultiplicationCodeFunction (mathematics)WindowComputer virusGreatest elementDirectory serviceBootstrap aggregatingServer (computing)outputNetwork topologyBuildingRevision controlIterationBitDemo (music)Functional programmingProgrammschleifeProgramming languageError messageDirection (geometry)MathematicsConstructor (object-oriented programming)NumberTotal S.A.Compilation albumMultiplication signVertex (graph theory)Electronic mailing listImplementationString (computer science)MultiplicationBinary multiplierBefehlsprozessorLevel (video gaming)Revision controlParsingLine (geometry)Goodness of fitRight angleReduction of orderCountingPatch (Unix)IdentifiabilityComputer animation
14:11
MathematicsComputer virusLambda calculusLambda calculusProgramming languageFunctional programmingDifferent (Kate Ryan album)Electronic signatureCodeType theoryPlanningFluid staticsMultiplication signCASE <Informatik>Letterpress printingResultantComputer animation
14:42
Interior (topology)UsabilityPolymorphism (materials science)String (computer science)Type theoryMultiplication signComputer virusPrisoner's dilemmaLecture/Conference
15:14
Electronic mailing listServer (computing)TunisAlgorithmGraph (mathematics)SynchronizationLetterpress printingPolymorphism (materials science)Motion captureVariable (mathematics)Multiplication signProgramming languageThomas BayesComputer animation
15:35
CodeFunction (mathematics)Heat transferDemo (music)Multiplication signTerm (mathematics)Cellular automatonOnline helpMereologyProduct (business)CASE <Informatik>Programming languageFunctional programmingCompilation albumLibrary (computing)ImplementationComputer animation
16:34
MKS system of unitsSoftware testingPatch (Unix)BlogSoftware testingShared memoryTwitterBlogPatch (Unix)Multiplication signSoftwareDemo (music)CodeFreewareComputer animationXML
16:50
Menu (computing)Hacker (term)Multiplication signHacker (term)Computer animation
17:04
WebsiteWebsiteMultiplication signProduct (business)Service (economics)Computer animation
17:25
TwitterFreewareBlogTouchscreenEmailSpherical capBlogElectronic mailing listProcess (computing)TwitterEmailMeeting/InterviewComputer animation
17:45
WebsiteFeedbackFeedbackData managementForm (programming)Link (knot theory)WebsiteSelf-organizationMathematicsDenial-of-service attackComputer animation
18:03
FeedbackLink (knot theory)LaptopFreewareHacker (term)FeedbackForm (programming)2 (number)Computer animationLecture/Conference
18:25
Point cloudFacebookOpen source
Transcript: English(auto-generated)
00:05
The next speaker, James, you also gave a talk last year, right? And it was an excellent talk, so we invited him again. It was fun and everyone leaving is going to miss the fireworks. So if you're leaving me quickly because I have only 20 minutes,
00:20
I'm going to talk really fast. Oh, and I have a microphone. Thank you so much. But everyone leaving is going to miss out. That's okay. I'm just teasing. Hello? That's okay. That's just for the video, right? So I'll start. I'll just show you a quick video from Montreal while we're waiting just to let people get in and out of the room.
00:42
I don't know if I showed this video last year or if I showed the other video. I have some silly videos. It was a little snowy and slippery and some cars had a bit of a fun time on the road because the road slides down ever so slightly. And who has seen this before? Has someone seen this? Okay, only a few people, so that's okay.
01:01
And it's worth the minute that I'm going to take away from my talk. Maybe I can go over two minutes. Nothing, nothing. Have you seen this? It's like this bus that, for a 20-minute talk, it's tight.
01:22
But we have to laugh, right? So the bus thought he could do better. This car is trying to get out of there in a fast way. And here comes the bus. Boom. But then the bus goes. He's still going.
01:41
Can't stop. Boom. So obviously the police can't help and be part of the solution here. Sorry to video. The video is kind of weird because LibreOffice has bugs.
02:01
I love LibreOffice, but there's bugs. Bugs suck. And then, of course, CityWorks comes to help fix the problem. And then after this, I'm going to talk really fast, so sorry for that, but that's all I can do.
02:22
Boom. All right. So who am I? I'm a hacker. I work on config management. I apologize if I'm going really fast, but there's a lot of material online, so you can follow what's happening if you missed it. I write a technical blog called The Technical Blog of James. Who's seen it? Just raise your hand. My blog? If you haven't seen it, just raise your hand, so I seem really popular. Thank you. And just some background.
02:41
This is also LibreOffice bug. It's just beaker showing that everything is kind of bad, and there's some noise. But, sorry about that. So anyways, long story short, everyone says don't reinvent the wheel, but I don't want this like yucky stone wheel. I want the like fancy wheel. And I work in automation tools, and I really believe that like using a DSL
03:00
is the way we want to describe our problems, because we want to build these complex real-time systems, and if you're doing it in a full-fledged programming language, you're going to have bugs, you're going to have off-by-one errors, and if you have an off-by-one error, that might be like a data center that you would destroy. This is really bad. So I talked a bit about this last year. This was my like, do we want a full-fledged language for all our problems?
03:20
I guess this is the right room for this. No, we don't, okay? And every tool that thinks that they can, yeah, we'll just use Python, or even safer languages, anything basically probably short of Haskell, and even then I wouldn't necessarily agree that a full language, even a type-safe language like Haskell is really the right thing. So I have a DSL,
03:40
and we use the DSL to describe these real-time systems that we want to automate. Tools you might be familiar with, like Puppet and Ansible and so on, they look at one state of time. So they say, we're going to set up your infrastructure like this, but they don't actually model the time component, which is what actually changes day-to-day. Load goes up and down, error, failure rates go up and down. And the tool has an engine and a language.
04:01
Again, I talked about this last year, so you'll have to see that talk. And it has some interesting properties. Again, super fast because this is really short. And here's just a simple example. So this is kind of an FRP. It's this functional reactive language that I've manually syntax-highlighted. So imagine you have this datetime function here, which is a stream of numbers, right?
04:20
So everything is a real-time, streams of things instead of single points in time. So this happens to be part of our standard library. And I'll just show you a quick example of this. If I run this on the left, you can see at the end, it actually is taking all these different values, integrating them, and putting them into a text file, this file resource, which is a declaration of, I want this file to have these contents.
04:42
But since this value, this big string template, happens to change, in this case, at least once every second, if we actually watch this file, you can see that it's actually changing over time. Because MGMT notices when the language has reevaluated, it pushes a new copy of the file to the engine,
05:01
the engine switches them very quickly, and you can see it's very fast. And in fact, one of the inputs I have is this VU meter. So if I make noise, you can see it's even reading my microphone right now to take that as another input, right? So it's kind of the joke. Like you could have your infrastructure running, and if you hear a lot of screaming, maybe the sys admins are fighting, so you've said everything read-only for an hour, right? And so that's the basic premise.
05:21
But I want to show, I think I showed that last year, I'm going to show some fancier stuff. So you can do history of values, I'm not going to talk about that. We can do hysteresis, which is super cool, but we're not going to talk about that. I want to show you some different demos. The first is called reversible resources. So the lovely thing about DSLs is you can add into your core resources some cool properties. And one of them is called reversibility.
05:41
So imagine you take the state of something, and you move from state A to state B. Most systems don't model this, but what we should be able to do is model the reverse operation. So an idempotent operation to get us to state B, but a reverse operation, in theory, should be possible for many types of resources. And if that's possible, that would be something called reversible.
06:01
That would be reversible. And in this case, for a file resource, if you create a file, and then you run the reverse operation, it would remove the file, right? And the way this works is because in normal tools, when you push some code in, for example, file so-and-so, and then in the next git commit, that code is just absent, that's when the reverse operation will run.
06:22
So the whole point of putting some code in and then it disappearing, you'll normally have that file or that thing laying around on your machine for whatever reason. So adding this reverse, make sure that the engine says, ah, I knew this thing was here in version one of the code. Version two makes it go away for some reason. Therefore, I'm going to perform the undo operation. And just to show off this a little bit,
06:42
I'm going to show you, instead of actually changing the code each time, I'm going to just actually run some code. I'll show you the code I'm actually running. And this is all git, so you can, so it's the same thing, but I just have this mod function that's basically looking at the date, time, and seconds. So basically every four seconds, it will basically make this value become false.
07:01
Can you see that okay? It's cut off. Oh, that sucks. It's cut off a little bit. It'll make this value become false, and so the file will effectively be gone and the engine will apply. But you could just change the code version and do the same. So let me just run this. There we go. Let's remove date, time.
07:20
And if we run this LS, you can actually see we wait four seconds and the file appears and then four seconds go by and it disappears. And it will just fluctuate back and forth in this case because I had this ping pong thing, right? Does that make sense? Am I talking way too fast? You're supposed to say yes,
07:40
but that's okay. Who's really shy? Anyone shy? Raise your hand so I can pick on you. All right, cool. So that's the idea. I could show you another example where the file mode. So in this case, I won't show this example for time, but basically if you have an existing file and you set the mode to something and then you undo it, it will undo the mode. Same thing for an existing file with some contents that says hello.
08:00
And then you remove the file as your state. The reverse option will actually restore those file contents. So MGMT the engine is actually secretly doing a backup of any necessary data that it needs to restore. Now I use files just because it's a simple example, but in practice, the resources we care about doing this with are things like virtual machines, containers, and other real-time things. So with this sort of thing,
08:21
you can have a, if we have decided to schedule this virtual machine on this physical hardware, create this machine, and when that goes to false for some reason or the machine crashes or something happens, the remove operation can happen automatically. That's not very well explained, but hopefully it's good enough for the time I have. So do you want to see some more demos or are you fed up?
08:42
Demos! Do you want to see some more demos? Yes! All right, I know it's early. Who picked this shitty time slot like 9 in the morning? Come on. Just kidding, just kidding. I have no idea. Okay, I'm just teasing. For me, conferences don't start before 10, so I'm here on a special, special request. So here's something fun. I just want to show you a cyclical state machine.
09:01
So you've all heard about state machines. Here's just the start of the code. So basically we have a state variable which we pull in. The full code is in Git, but just the important parts. And the first basically big if statement is if we're in the default state, so it's undefined kind of, or state one, we run a little timer here, which in this case is just a command
09:20
that sleeps for one second. And then we have a kv resource which will actually set the state that we want, so the next state. In this case, we'll set it to two. And this is just a little edge in our language. It says this first resource in the DAG, so the dependency graph of what runs happens for the next one. And so that will basically mean that this runs and then waits a second and then sets it to state two.
09:42
It's an FRP, so this state variable will end up changing. And so suddenly it will go from being in this condition branch to this conditional branch, which sets it to state three. State three goes to state one and so on, so it keeps looping. Make sense? And the other resource that's in here is just I actually write to a file so you can actually see what's happening.
10:01
Oops. State. I think it's this example. It is so. I'm just cat star. That abracadabra is from the old file. Let's see. Hello. So we can see it's just changing this file. One, two, three. One. Counting with James.
10:20
Two, three, everyone. One, two. This is fun. So obviously a bit of a contrived example, but the idea again is not to model counting because that's dumb. The idea is to model the real-time changes in your infrastructure over time. In a safe DSL so that it's super safe. And the reason is because we want to build autonomous systems.
10:41
As sys admins, as infrastructure people, as just individuals who want to run their mail server at home, we want to build something really robust and the code should take care of things. If there's an error scenario, a hard drive fails, the code will notice this failure and react to it and build the right thing. Make sense?
11:01
You want to see more demos or have you had enough? Does anyone want to see another demo? Okay, I'll show you some boring demos. I have to make it a little bit more boring. So I'm going to show you one more demo iteration. So let's look at this example.
11:21
So this is a little bit more... So we have no looping construct, no direct looping constructs in the language. Why don't we have four loops in our language? What? Yeah, so basically loops are unsafe and can produce a lot of problems like off-by-one errors and many other things in functional languages. Long story, they're not safe. But we still want to do things like iteration, right?
11:42
Because iteration is how we get the CPU to do work. And so there's many ways we can do that. One way is actually edge multiplication. So if you look here, these are the resources. And resources have a unique identifier, which is either a string or a list of strings that identify them. So if this one here... This means we would have one resource, okay?
12:01
So I'm going to borrow someone... Do you want to count? Can you put up one finger? We're going to count for one resource? It's what? Oh, goodness, sorry. Thank you. I should do this. How's that? So we're just going to... Yep, no problem. So this is one resource here at the top.
12:21
Down here we have this names variable, which is a list of strings. So that would be how many resources total now? The first one plus three total, right? So the first one plus three. Down here we have another two. And then down here we have three more, so a total of what?
12:41
Eight, okay? Now here's the cool thing. So now we have the dependencies. So let's count the dependencies, which are these edges between the resources. We're building a DAG. And so the first one here is names. So we have number of names is how many? There's two.
13:00
And there's how many resources here that depend on that? There's two, so it actually multiplies out. So two times two is? That's very good. So we have four. And then down here we have this edge that's in line, that's not in line. So we have how many edges here? Names. Names is two. And this one is how many?
13:21
So that's how many, two times three? Plus the original four from before. So yeah, so we have eight resources and ten edges. I know doing math is fun. And if we look, we'll just run here, you can see it builds the right thing. So eight vertices and ten edges. And you can actually see them all and so on. So that's one way we actually end up using iteration
13:42
as a cool concept. We actually will also have map, reduce, and filter. Funny enough, it turns out functions as values, I'm having a bit of trouble with my implementation and getting it exactly right. So those functions actually don't exist yet, but they're coming pretty soon. And if you're really good at lexing and parsing and compilers and you can help me finish this patch,
14:01
that would be awesome. Because then we can do map, reduce, and stuff too. I've got a few more minutes, so I'm going to show you one more example. I'm going to show you this example. So here is an example of fortunately no syntax highlighting where we build a lambda function x, which is x plus x.
14:21
And so here's the cool thing about the language. We want our language to be very safe. So when the code is running, just after compile time, everything is static. All the types are static, like there's no weird stuff that goes on. However, it's annoying to think about things that way because functions like printf and things where you can have many different function signatures can be quite difficult. So as a result, before compile time,
14:42
everything is polymorphic and it attempts to find out what type would work and then if it finds a single solution, it compiles successfully. So in that case, you can see this add function right here, this add lambda, can work with an integer because plus is also polymorphic before compile time. So two plus two would equal four. And then if it's a string, you can do that here
15:02
and you can see the value here is hello plus hello. Does that make sense? That's kind of like the cool hack where we get the fun user-friendliness of polymorphism but without the danger of string plus int. What should we do? Python, basically disasters. So and I'll show you that example just for fun and you can see basically two plus two equals four
15:22
and then hello plus hello is big hello. Make sense? Not as fun demo-wise but hopefully gets you thinking about the language and so on. I won't have much time so I'll just finish. I've got five minutes left. I'll just go through the end and if we have time at the end, I'll show you one more demo. There's a lot still left to do. So I think I'm starting to use this tool in production now
15:41
and hopefully you can too and if you all are good at functional languages or reactive languages or DSLs in general and compilers, I would love your help because I don't know anything about this stuff. I'm just inventing this as I go along. Like if Connell Elliott like the FRP guy saw my talk, he'd be like, what are you doing? It's not called that.
16:00
But I've renamed terms to make it accessible and friendly to the mainstream audience that I want to meet with this and I think if we want FRP and these kind of DSLs to become more widespread, we have to sort of make them accessible to non-FRP specialists. So if you are an FRP specialist and you want to help me fix the little parts that I got wrong, I'd love to hear from you. So more function in the standard library.
16:22
We'd love to have more resources in the tool itself. New features. There's actually this function is values implementation. It's not complete. I'm still missing a few edge cases which are tricky and it's programming so it's hard. How can you help? This is all about you. You can use this, test it, patch it, share it with your friends, document it, start on GitHub blog.
16:41
You're smiling because you're going to do all these things, right? Tweet it if you have Tweeter. Discuss it. Hack on it. Just hack on this stuff. It's a free software conscience. All the demos, all the codes, all online in Git. So I used to work at this company called Red Hat and I left about a year and a half ago to try and do this full time which kind of sucks because I'm just living off my savings. So if you want to help, donate on Patreon if you like
17:02
because funding a hacker is very sexy. So the other thing to try and make this a full fledged product, this year I'm going to be actually trying to put corporate logos on the website which we are making. So selling support, services, and sponsored by your cool company that's not evil.
17:22
Maybe it's a good time to start asking a couple of questions. Yeah, I'm almost done. So let's just recap. He's putting his cap back on his pen. It's a bad joke. So there's an IRC channel. There's a Twitter account, a mailing list. There's a technical blog of James which you all know about. I'm going to try and blog more so have a look.
17:40
I'm purpleidea on Twitter and GitHub and all these things so you can ping me and say you like this talk. And I'm actually going to Ghent later tonight because there's Config Management Camp which is a cool conference about automation stuff that I'll be giving a longer, more in-depth talk about so if you want to see more. If you really like this talk, please DDoS the organizers by going up and saying, hey, I really like James' talk.
18:02
And if you go on the website, there's actually a link to the feedback form under the talk link. It's kind of hidden. So please go. It takes two seconds. Tell me you liked it or not. And if you promise to use a sticker, I will give you a sticker. They're crazy expensive, but then you can have a cool laptop like me. That's it. Thank you very much.