ncf
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32581 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2014150 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Software frameworkBitMathematicsRight angleLine (geometry)Lecture/Conference
00:40
Event horizonVideo gameConfiguration managementMultiplication signSystem administratorData managementXML
01:29
Link (knot theory)Data managementPresentation of a groupStreaming mediaAbstractionMedical imagingMultiplication signExecution unitMereologyType theoryMathematicsPoint (geometry)Figurate numberIntegrated development environmentSubject indexingFunctional (mathematics)Virtual machineResultantFeedbackCASE <Informatik>SoftwareLevel (video gaming)Different (Kate Ryan album)Pattern languageHypermediaNumberRight angleInformationDemonData recoveryDataflowLocal ringSoftware bugInheritance (object-oriented programming)Physical systemCodeInformation securityCurveOpen sourceRevision controlAnalogyInformation technology consultingComputing platformCuboidWater vaporPower (physics)Basis <Mathematik>Error messageBlogComputer animation
05:52
Family
06:17
Goodness of fitSubsetTerm (mathematics)Software bugProjective planeExpert systemSoftware testingFeedbackBitFormal languageMultiplication signFile formatTable (information)Revision controlLoginInformation technology consultingMereologyType theoryControl flowCodeExecution unitDirectory serviceStandard deviationComputer fileOpen sourceConnectivity (graph theory)Line (geometry)Installation artDifferent (Kate Ryan album)State of matterFrequencyEvent horizonData structureFood energyElectronic mailing listScaling (geometry)Right angleTrailDisk read-and-write headComputer programmingMathematicsBuildingComputer animation
10:27
Module (mathematics)Fiber bundleConfiguration managementSystem callData managementMessage passingSlide ruleSoftware frameworkTask (computing)Computer fileAttribute grammarFeedback1 (number)CodeService (economics)Connectivity (graph theory)Greatest elementMathematicsFlow separationDifferent (Kate Ryan album)Line (geometry)Video game consoleScripting languageSoftware developerInformationWhiteboardWebsiteServer (computing)Physical systemSet (mathematics)Multiplication signEquivalence relationOpen sourceBitQuicksortReading (process)Group actionForm (programming)View (database)CASE <Informatik>Point (geometry)Symbol tableData structureState observerSpeech synthesisEqualiser (mathematics)Endliche ModelltheorieTheoryVideo game
15:35
Projective planeAreaTheoryState of matterNumberMatching (graph theory)CASE <Informatik>Software testingPoint (geometry)Electronic mailing listSpeech synthesisWordRule of inferenceData managementMultiplication signRight angleTwitterSocial classArithmetic meanConfiguration managementOperator (mathematics)Internet service providerAxiom of choiceCodePrime idealSoftware frameworkBoolean algebraGeneric programmingExpert systemInformationService (economics)Software developerOvalUser interfaceDatabaseRevision controlStructural loadFiber bundleSystem administratorWeb 2.0Core dumpParameter (computer programming)Module (mathematics)Computer fileMereologyLibrary (computing)Cartesian coordinate systemRepository (publishing)WritingFeedbackXML
20:44
Integrated development environmentCASE <Informatik>Volume (thermodynamics)Strategy gameDialectRight angleNumberSound effectNatural numberServer (computing)Expert systemFormal languageData managementLine (geometry)Goodness of fitRevision controlPoint (geometry)Process (computing)Software bugDefault (computer science)Lecture/Conference
23:56
Run time (program lifecycle phase)Fiber bundleKey (cryptography)2 (number)SubsetSlide ruleLecture/Conference
24:31
TouchscreenLine (geometry)Social classComputer fileMultiplication sign
25:20
Multiplication signSoftware frameworkRevision controlLatent heatServer (computing)Beta functionPoint cloudLecture/Conference
Transcript: English(auto-generated)
00:00
This speaker is Jonathan Clark, he's going to tell us a bit about NCF. And I'll let him know. Thank you, hi everyone. So yeah, my name is Jonathan Clark. I want to tell you about this framework we created for the NCF. It's to do with CF engines, so just to get started, a quick quick question. Who here has heard of CF engines?
00:20
It's pretty good, pretty good. Who here has tested NCF engines? Especially if you've never tried. Okay, nice. Who here still uses CF engines? Okay. Not everyone, right. So, maybe I can change your mind about that.
00:40
Let me start by introducing myself. I worked at a company called Nalmation and I co-founded in 2009 in Paris. We basically work around config management. I was originally a sysadmin, did lots of infrastructure-y things. Now I kind of run a company, so I don't have as much time to be a sysadmin as I would like, probably.
01:00
Running a company is fun too. I've used various free software, whether it's tools that we've created. I've presented it here at FOSDEM in previous years. CF engine, which you've heard of, like most of you anyway. And OpenLL. That was another life. I also help co-organize some events occasionally.
01:21
There's some DevOps days in Paris. Paris DevOps Group and LDAP. Which is, like I said, another life, but also pretty interesting. So, this presentation is actually about cakes. That might come as a surprise in the company management room at FOSDEM, but hopefully you'll get the link pretty soon. So, for those of you who are not familiar about making cakes,
01:43
the way you do it is you need ingredients. Apparently I hear, I'm told, that you need eggs, flour, butter and sugar and other stuff if you want to do it better. You also need recipes. So you need a recipe book or you need your grandmother or your grandfather or your parents. In general, they'll pass down recipes to you if you're frightened to share them with you.
02:02
People will know how to do it. Okay, so let's put the cakes aside for a second and say who is we? Who is the we that created it? It's a bunch of CF engine consultants. Some guys from my company, Enormation. Some guys that we've worked with. Another company in Canada called Cerro Verneux.
02:21
It's a name for foreigners. We've been working around CF engine for more than four years now since version 3 came around. We've been in companies that are tiny. We've been in big companies. We've been in huge companies. I'll make the difference because one of the companies where we've worked is actually a hundred and twenty system that is hacking into CF engine every day.
02:40
That's probably called a huge company. Then there's the normal regular large ones. We just have ten or so people. And we've seen various uses for it. Applying security policies, provisioning new machines, having them configured automatically, disaster recovery. And every single time, every customer, every person we talk to, we get the same feedback.
03:01
Two pieces of feedback. Number one, CF engine box. CF engine is really powerful. It's really small. It's lightweight. It's written in C. It uses like five or eight megabytes of RAM. It's so cool. So fast. Really quick. Awesome. It runs on pretty much every platform you've ever heard of.
03:21
I had it running on my Android phone. It runs on all the AIX, HP, UX servers, all that kind of thing. Linux, BSD, I should probably mention that. Windows as well. It's an agent-based approach. You don't need to connect over the network to get CF engine to do stuff. It's there. It's sitting there as a demon or an agent that runs. It's local. It's powerful.
03:41
And because it's lightweight, it runs every five minutes. You can get things done quickly. It's resilient. Errors. No network. It's fine. It's tedious. I know some submarines in the world have CF engine on board. They get it running. The submarine goes out under the waters for nine months. It comes back. Oh, great. CF engine's still running. No problem.
04:00
It's pretty cool. And it's open source. You look at that. I think, yeah, let's do CF engine, man. This is awesome. Then we get feedback number two. CF engine is hard. Yeah. Maybe some of you who tested it have experienced that. Maybe some of you who haven't tested it have heard that.
04:22
There's nothing new under the sun, I don't think. Four key points I would point out. There's a steep learning curve. There is for most of these tools. You're learning a new syntax to do new things. But CF engine syntax is kind of weird. It's not procedural. It doesn't follow the flow you write it in.
04:41
When you use CF engine with policy you've just written, you don't actually see any feedback necessarily about what's happening. It doesn't tell you, yeah, I tried this and it didn't work, or I tried this and it's fine. It's quite hard to get that information. You can build that there. And like old software, maybe more open source software
05:00
than other software, there's bugs. And the developers, great guys, they prioritize bugs based on whether there's a workaround or not a workaround. If there's a workaround, it gets lower priority. Which means that sometimes you have a way to work around a bug. But how do you do that every single time in the code you write? The code being CF engine practices. It's pretty hard. You have to go through everything.
05:21
Sometimes you forget. If there's a team of people writing it, you forget. And basically, at the end of the day, it's too much to do it yourself. You get CF engine. It's a very powerful agent. We have to write a lot of things to get it working. We have to build a lot of things. You can compare in this blog post that I linked here to flour, eggs, milk, and butter.
05:42
So if you see where I'm going with the whole cake analogy thing, that CF engine would be the ingredients on this basis. So what this feedback is telling us, really, is that learning to bake your own cakes is pretty frustrating when you don't have a recipe. You can put the ingredients in a bowl. You mix them together. You put them in the oven.
06:01
And you get something like this. It's OK. It kind of tastes a cake, you hope. But it's not really the solid foundation you wanted to build your company or your IT services. So we asked a simple question. Can we fix this? We've worked with lots of different customers, like you said.
06:23
And for every one of these customers, we found a lot of workarounds. So we convinced them, implemented methods, tools, scripts, whatever, to work around these things. We thought, yeah, of course we can fix this. We've done it n times. We can do it once and for all. We can make an open source project that includes most of these things. We can include some best practices.
06:41
We can include some clever workarounds, like everything. So we accepted the challenge. And this is how NTF was built. We actually built NTF as a team of consultants and that we were good friends and things. We also did a lot of pushing from patient customers,
07:01
enthusiastic customers. We really wanted this to work in the way it did. It may or may not have known at the time it was creating NTF what these ideas gained from chatting with them and saying, oh, this should be easier. Those guys I can't name for privacy, contract, legal, nonsense reasons. But anyway, they are also part of the guys that made this happen.
07:23
So how did we do it? What have we done? We looked at each of the downfalls and said, so what can we fix? There's too much do-it-yourself NTF engine. So what we've done with NTF is provide a structured layout to start with. It's very simple. It's just a git checkout or a table to download.
07:41
It's got some nice directories in. You can get a bit of guidance as to where to put your stuff. This is stuff we've learned over the years. There's good ways and bad ways. Some of them work for a bit on break. Some of them work forever. And maybe in ten years, we've proven wrong. But so far, it's a good layout. And what we've done, which is very different from what
08:01
CF Engine provides now, is there's lots of reusable methods in there. A method in CF Engine, which you bundle, you call it using methods. It does something. It does one thing, but it does it well, like command line tools and units. So we'd have a method, for example, that installs a package. It does that well. It's got all the bug workarounds built in.
08:25
To avoid the steep learning curve, what we tried to do is say, OK, there's a lot of weird syntax in CF Engine. Let's put that weird syntax where the CF Engine experts can read it and hack it and change it and optimize it. But let's put everyday user in front of a much simpler
08:45
syntax. So what we've done is actually cure CF Engine. There's no other languages in there. It's not invented in its name in terms of language. But the syntax you use for your everyday policies is just a subset of standard CF Engine syntax.
09:00
You can actually use one promise type. For those who know CF Engine, it's like 10 or 14 promise types. You can just use one. Use the methods promise type. What about bugs? Well, since we have these generic methods that we can call and reuse all the time, it's very simple. It's a bug workaround. We put in a generic method. It's done, sorted. It's dry. Don't repeat yourself.
09:20
You've got it there. You've got it once. Every time you use that method, you have a bug fixed. Next version of CF Engine, the bug is properly fixed in the code. You remove your workaround. Easy. And we've also in UN Dog Food, always preaching, do testing, automatic testing. Yeah, we've got automatic tests for all these bugs, all these things. How should the component work?
09:41
Test it. See what comes out. It comes out. It's fine. As soon as something breaks, as soon as a bug workaround, no longer a workaround, it works around the bug. Well then. And last but not least, the lack of feedback. Because we have, once again, all these generic methods, we can build in feedback to them.
10:03
Every time you run through a package install or a file copy bundle, it can print out a line saying, hey, I checked this. Well, there's nothing to do. Or, hey, I prepared this because it was wrong. And of course, the feedback format is configurable. You don't always want your logs full of, I checked this. Nothing happened. I checked this. Nothing happened.
10:20
So it gets boring. If you're debugging, it can be nice to turn it off. So basically, what we're saying with NTF is food baking. We don't want to go baking. We don't want to learn to bake. Let's just go and buy some cakes. These ones look pretty good, right? I've actually got to tell you, when I was doing the slides yesterday, I ended up again. I really need to eat some cake.
10:42
So bad. So NTF. What is NTF? NTF is a framework. It is not a config management tool. The config management tool underneath there is CF Engine. It's a framework that runs in pure CF Engine code. There is no other code in there. And it helps you structure your CF Engine policy
11:03
by providing these reusable, single-purpose components between the following methods. They're really the body of NTF. And the whole thing, of course, is open source. Otherwise, I wouldn't be talking about it at FOSTM. And I just lied. It's not actually completely pure CF Engine.
11:22
There's also some Python tooling on the side. You don't need to use that. You don't need that on your servers. Just as a development aid, it can be useful. So let's go with Python scripts. Since an example is worth a thousand words, or so I'm told, here's an example. Read that.
11:40
Basically, this is a bit of pure CF Engine code using NTF that installs the NTP package, copies a config file. If the config file was changed, it restarts NTP. So it re-reads its config file. And if whatever happens, it checks and the NTP service is running.
12:00
So there's four lines of code. I hope they're kind of readable. There's a lot of light on the board. Basically, four actions, and it's five lines of code to do. The equivalent without NTF, in pure CF Engine, would look something like this. You go on promise for each of those four actions. You go to where it was at the top.
12:22
There's all sorts of attributes here that I've been doing CF Engine for four years. I still have to read the reference manual every time to figure out what attributes I'm actually supposed to use. And this is even a simplistic example. There's actually no feedback. Nothing comes out. When you put this into CF Engine, you run it. Maybe you'll get a warning message or two,
12:42
and it won't tell you what CF Engine is actually doing. So long version, short version, NTF, no NTF. And I told you there was automatic feedback. When you run this through NTF, you get something like this on your console. This is with debug logging enabled to debug for NTF,
13:01
not CF Engine debug. That's just crazy. Basically, you get one line for each of those promises we saw earlier. So the top one is saying the promise was kept. The package NTP is already installed. We don't need to change anything at all. The second two lines, well, two and three are saying the promise was repaired. Something changed.
13:20
So we actually entered the config file. It was not up to date. And therefore, we restarted NTP. And then the final promise, checked that NTP was running. It was running. Fortunately, it just restarted it. Just an example. So how does this actually work? Well, for those of you who know CF Engine, it's basically bundles with method calls to call the bundles.
13:42
For those of you who don't, bundles in CF Engine are a way of containing a set of promises. I guess they're an equivalent of a module in Puppet, or a recipe in Chef, roughly. And you can pass parameters. That's all we've done. We've created a lot of bundles in CF Engine that you can use. So what we did in this example
14:01
is we have an NTP bundle that's called out to four sub-bundles. I only put three on the slide, but there's four. And they all, in turn, call out to another bundle at the bottom, which is called Logger, which, as you would expect since these log lines, which is pretty simple, but it's pretty powerful in the end.
14:22
As you can see, there's several different layers here. We've actually built the whole framework in a layered approach from top to bottom because the customers we've worked with, usually the guy at the top or the guys at the top, they don't really care that the Apache.com file or the virtual host or whatever is not working.
14:41
What they care about is their corporate website is OK. So we've built this down from the services, corporate website level, down to techniques. How do I do stuff? Do stuff being, for example, set up my Apache server, put my baseline on the system that you were saying earlier.
15:00
We've included this IT ops knowledge level, which is maybe a cool name, but hopefully it gets the point across. The IT ops knowledge includes stuff like, what is the package name for Apache on your current OS? And you can just call out to a variable, and it'll tell you it's HTTPD, it's Apache 2, or whatever. OS has decided to call that now.
15:22
The idea is to build up all of that stuff as possible with as much information. And then we come to really the core, which is the generic methods. All of these unique tasks. Do one thing, but do it well. And then underneath there are some libraries, framework stuff.
15:44
The core principles we've tried to follow are basically avoiding all the nightmares that we've got into at once. So not repeating code, any code, any even like patent code should not be repeated. So we've wanted to try principle,
16:00
don't repeat yourself. KISS as well. I mean, that's a simple one. Keep it simple. The idea being that users would be able to write configuration management policy without being an expert. That was one of the prime use cases actually with the companies I was working for. When I first came up with this idea, it was a web hosting.
16:24
They had a big team of sys admins that dealt with the basic infrastructure. Then they had whole loads of developer teams. Each team developed one application or one part of an application. And the way they communicated was the developer team would provide the code to the ops team. It would also provide config management
16:41
for how to install the application, how to upload it, and go from version 2 to 3 needing to change the dependency to read in the database. I know I'm tearing their hair out and trying to get my attention here. So the point was with this framework, they provide their intention to be simple and shallow.
17:03
You see all these generic methods I keep talking about. One of the goals here is that the generic method, there's only one. There's not a good way and a bad way of doing it. It's not like a library of four modules. We've all seen forges of modules
17:20
in all projects. There's Apache 1, Apache 2, Apache 3, Apache 4, Apache 5, Apache 6, my way, and so on. It's just crazy. So the point is here with these generic methods, because they do one thing, you can already improvise. Creating a file is creating a file, restarting a service is restarting a service. But the restarting a service can be implemented in several ways.
17:40
Upstart, initd, and a couple of other crazy things that people are now using anyway. The point is that creating a new generic method is an objective choice and a subjective choice. It's extensible over this. This is a quick list of some of the generic methods
18:00
that are available already. There's huge numbers of them because we've tried to keep it simple. So if you look at the package ones, package install, package install version, package install version, you'll do the same thing. You'll call the same code in the end. But just you can call a simpler version to get a simpler code to write.
18:20
All this is documented online. Each bundle has this kind of doc. This is just a screenshot I took of the website last night. It explains what it does, what parameters it takes. And each of these bundles provides a return, like feedback information as well. So it defines the class. Class can see if engine is a way of sharing information.
18:40
It's basically like a boolean. It's like a tag. It's set, not set. Each of these bundles will set tags saying whether it performs well or didn't perform well. In my example earlier, we had copy the NTB file and then restart NTB if the config file changed. That was one of those recent classes. So I'm running out of time,
19:01
but I just wanted to share a few words about the status of this project. It's a young project. It started officially the first commit and the repository was like October. It's young, but it's pretty robust because we've built this time and time again for different customers over the past, let's say, two, three years.
19:20
So it's not new to us. It's stuff we've done before. It obviously needs more generic methods. I don't know if you can call them or not, but definitely. I don't know if 13 or 14, but we could do with a lot more. Something to create users would be nice, for example. But it's being used by quite a few different people. Lots of testing going into it.
19:40
Lots of new generic methods here. I included the other statistics, but it is an online tool that I haven't even heard of yet, and it sees what it does. It's pretty objective again about what you do. And according to them, we're a very young product. We have a code base with a very short history. We have a large development team, which I thought was, wow, that's pretty cool.
20:02
Something that we just started. And we have very well commented. So, hopefully I've explained roughly what that was. Lost the cake. There's a fall there somewhere. Hopefully you understand that NTF is a recipe book
20:21
that goes with the flour and eggs. Do you have any questions? Two independent projects at the moment, but one that can actually read in code created by NTF to display it in a web interface automatically.
20:50
My recommendation would be to jump straight into NTF. As a user who's going to write a conflict policy, if you're the conflict management expert in your company,
21:01
probably better to learn the NTF engine first, so you can debug it and fix it and add methods in. Basically, if you remember the different layers I was showing earlier, I'd say the bottom three layers, you need to know the NTF engine to play with them, to hack them, that's the point. And the layers above can be exposed to the NTF engine users. You can basically look at the docs, copy and paste, for example.
21:24
I'm going to be able to do that a little bit here. I'm going to be very happy to be here. I'm going to be starting now, and we're going to be doing a project. We're starting now, basically. We're going to be doing a project,
21:40
and I'm thinking that if you take the language that you're going to use, why wouldn't you just go to the project? Why wouldn't you just go to the project? Why wouldn't you just go to the project?
22:00
So this would be the NTF engine rock slide I had earlier. The really lightweight footprint, multi-OS support, all of the great things. To me, it's the lightweight, the open source, and the performance, the multi-OS support.
22:22
That's really the reason. Yeah, good question. I try. I do. I fix NTF bugs in CF engine. Unfortunately, my job is not fixing bugs in CF engine.
22:40
It's often a lot shorter to write a workman in CF engine policy, and it is to go and hack into the C code, and it takes time, and it's complex. Yeah, I would say 10 at the moment,
23:04
like 5 to 10. If you're using some weird OSes, as soon as you line X enterprise server 10, the default package methods don't work. We've already used this weird thing, which is I don't think we've broken the package manager. So we've fixed that.
23:21
We've patched that to use it, which is another package manager, which is kind of weird, but works better. That kind of thing. Copying files, for example. Sometimes CF engine behaves a bit weird. You've got to just add a few extra checks.
23:48
It costs some, of course. I mean, the more layers you're going through, you've got to have some. We've looked at the impact, and it's really lightweight. Going through layers of bundles in CF engine is very fast.
24:01
Lots of other things in CF engine are a lot slower. So we've seen that running pretty much the same promises with or without, you're going to go from execution time of like three seconds to three more times. I think that's acceptable.
24:26
Which slide do you want to see? So this is the with NCF. That's the one with NCF, so just basically your five lines.
24:42
These are one, two, three, four calls out to generic methods. And this line here is an if statement, basically. You only do this one if this weird class here is being defined by the screen. It's the file from template, the name of the destination file, and prepare to tell what that means.
25:00
And the after is this long thing. I'll leave that and answer another question. So you can read it. Myself personally, we've been using three, zero, three, one, two, three, three, four, three, five, and three, six.
25:24
But this framework, I think, no longer supports three or four. I think we removed something. There's a spot there. And I tested it yesterday on three or six. It was the first beta version that came out, and it works fine. One more question?
25:48
I'm sorry. I quite hate you. There's no specific cloud methods in there now.
26:02
We have been using CF Engine and a similar framework to this in the cloud instances, though, but just managing them like any other server. Nothing specific for now. All right. Thank you, Jonathan. Thank you.