And I'm quite curious to see who he has to say, so, you know, just shut up. Hello, everyone. I couldn't make any of these talks before. I'm just kidding outside. So,

I'm Mark Dwett. I've been at SESami for 16 years. The only thing of everything from racking models to setting up networks to going to custom house and then holding to networks. I was working at Hemrico before where I was one of the founders of Juju and Mas.

And I'm now running a DevOps engineering team in workspace in which we try to help with configuration management policies. I like DevOps and I probably know my free time and walks on the beach. So, Rackspace is the second biggest public cloud in the world.

It's quite a big custom company from the 98. We're quite proud to say we are the open-stack project co-founders with Mas. So, what is metadata? Let's go back to the basics, right? This is not a very metadata joke in a certain way. But in reality, metadata is data about data. So, it's a data used

to define what data is. There's two types of metadata, right? There's structural metadata, which would be, for example, XML, which you use data to define data. Or descriptive metadata, which is the one that we're most interested about in this talk. It would be the kind of data that defines a unique thing like a server.

So, all the unique parameters. So, how does metadata work in Chef? Metadata in Chef is compiled on the client execution time. And this is a very small detail, but it's something that will bite you very hard. Metadata is always compiled at the end of the execution time.

And it's always stored in a JSON dictionary. So, anytime that you use Chef, and you do a live edit node, you're actually editing the JSON dictionary that belongs to that server. So, where you can add metadata in Chef, you can add it

in the attribute files of your kubectl. You can add it in the node itself, which is something that you shouldn't do, but I think a thousand times I've noted afterwards. You can add it in your recipes as well, in your environment set, if that metadata is put into the environment or in the role itself, if it belongs to the role. And the good thing about Chef is that

all these variables in metadata can be overwritten at certain points in time, but the amount of exceptions for that is quite complex. So, I've just added a chart here straight from ops code recommendation to see how complicated that can get. So, you can define the same variable of

metadata in a few different ways. So, in Puppet, metadata is slightly different. So, as you know, Puppet always compiles metadata on the server in execution time. So, that's something that's a good thing, but then the Puppet must start suffering from that when you have a

lot of service. Metadata variables can be re-exited, which is something that I would say it's a deliberate choice, but it makes it execute good. And the way you define metadata is quite tricky, because you can't write any kind of variables, and it's always defined

like that, or with a class before they introduce any comments. So, where you can add metadata in Puppet, you can add it in your classes, you can add it in your notes, or you can add it in the top scope of your Puppet installation if you have a Puppet master. So, variables and metadata in general in Puppet is very tricky,

because, as I said, the scoping in Puppet has been changing with time. So, what we all used to do is to build all the metadata in our definition. You define $server type equal $webserver, and you will have this done. But that's now a bad

thing to do, because the scoping, as I said, in Puppet gets very complex, and this starts to look to me a little bit like Java, to be honest. It's very complicated, very difficult, and wherever you put the variable, you have to have in mind all the context that it has, which makes you, I would say,

quite wary of where you put that definition. So, Ari Pinar came to the rescue. He's the guy who did Hyera. So, he did Hyera as a Puppet plugin, and in there, you can define all your metadata in a very structured way.

It's all based on the Amero JSON dictionaries, and it looks very good. You can actually overwrite values. You can separate, literally, your variables and your metadata outside your execution field, which would be your models and your recipes. So, this, for example, would be a conflict on Hyera, and you can see, especially, the hierarchy section, which is

very interesting, in which you can actually define how these metadata variables will overwrite on each other when you execute. So, as everything, there's bad metadata practices, so you can play with the wheel of fortune of metadata madness. So, in Chef, what kind of things

you shouldn't do in Chef? Chef is a very powerful tool, as his focus, but since you can overwrite definitions and load definitions of the data pretty much everywhere, it can get very tricky, and you can actually put metadata in every single point. You can put it in the top scope, you can put it in the environment,

or you can put it in the node itself. So, at the end of the day, when you execute, you don't really know where variables are coming from, and they start depopulating your node dictionary, and that gets incredibly messy. So, you need to keep consistency of that and make sure that whatever variables, whatever metadata you define, needs to be in the right scope. So, if that belongs to the environment, you need

to predict the environment one. If that belongs to the class, the class one, et cetera, et cetera. If you don't do otherwise, you can start having things like I did, that I had an environment, then a function, then a class, but I could touch metadata in any of those three, and at the end, I had the metadata divided in three different

places, and I didn't even know where I was looking. I was repping my JSON definitions like crazy. Also, the other thing that we always forget is that when you remove a Kupu or a class from a server, all the metadata that it declares stays behind it. So, all that is inside the JSON dictionary, so you need to be very careful, because you can make those dictionaries grow,

and you're very silly, and make your executions very slow. And also, the other thing that we've done, we've all done at some point, is to assign metadata directly inside the Kupu, just go $node name variable equals something, and then suddenly, your node variables start

changing, and you don't know where that comes from. So, that would be slightly madness, but that is always the thing that we do when we're under pressure. So, what kind of things you shouldn't do in Puppet? In Puppet, sending metadata straight into a class without defining a default variable for that is always a bad thing to do. So, that means

having the metadata inside your class itself, in the middle of that, and you don't know where that variable comes from, that's the scope. So, it's always good to, when you define your class, to have the list of variables as the definition of the class. You don't have to fully manage with that. So, then you can override those in the definition of the node,

and you know exactly what all of your variables are. Also, you shouldn't assign variables inside most DPs, which is one thing that I've been very guilty about myself. That was the usual before, but right now, the scoping problems that most DPs have are very, very crazy. So, they are not global scope, they are not global scope, it's something

in between. And as I said, when you run Puppet lint, or you run a key combo, it's always complete. And also, global variables should go inside DPs. So, make sure that if you're running on a Puppet master, not on a headless operation, you should define those variables. So, what are the behaviors that we have?

Because as there's bad behaviors, there's always good behaviors, right? So, for me, the most important behavior is to separate variables from execution, right? Because your variables in your metadata is your brain, where the execution and the definitions are your muscle. So, you shouldn't be merging your brain with

your muscle, because that always creates trouble. And having a high abstraction of your variables, being very clear about it, so, say, being a better programmer than I am personally, makes all of this a lot easier. So, the good behavior is set with the time global, then

that being modified, modified by the environment, and then, that at the same time being modified by the nodes. And this problem just gets bigger, right? Because you have metadata in just one thing, that's shared for public, but then you have more players in the field. You have your provisioners, like cloud init, like coblet, or cardboard, or like

juju. And those provisioners also have a set of mistakes. So, let's say in this case, you have a provisioner, which would be cloud init, and the conflict management is puppet, and both of them like to define what the dog is, and they define it in completely different ways. So, the problem is who is right on that?

Because, again, you have the brain with the muscle all together, so what happens is that you pretty much are not touching yourself in the face, and that's not a good thing to happen, right? So, this is what I call the metadata brain split problem. It's a very hard problem to have, and it starts kicking you very

early when you do your implementations. And not also that, there's also another layer of complexity of things, because the system view varies as well, because the system gives you another set of metadata, like CPU, amount of memory, etc. And it's not the same, the kind of stuff that the buyer says to you,

and the kind of stuff that Coder will see at reading time, but the kind of stuff that Factor or Ojai will see when they are using level execution. So, that also adds some randomness to the whole equation. So, what would be some possible solutions for this? I would say one solution would be, for example, creating a central database using

something very fast, like when it's on MongoDB, if you write replication delays at my home bicycle. Maybe writing some metadata translators to have that central entity translate that back to every kind of metadata that you need from every single provider on execution of the management database. The problem with these is that these

create a new single point of failure. So, this is pretty much like existing, right? You have 14 standards, you say, hey, I found a solution for everything, I'll just create a new one and then, you know, 15 standards come running against one each other, which, again, sucks. So, another solution with me,

which is something that I try to do with Juju, but I wasn't too successful on that, is to try to scope variables from top of the chain. So, whenever you execute, since this is a chain of eventuality, and one thing passes to the other, make sure that all the metadata from the first one gets visible from the second execution, gets visible from the third

execution. So, whatever you have for the application will see all the metadata, all the variables from the previous ones, which would be very keen, but again, that would require all the information between different applications and different companies that are competing and one against each other. That's pretty much it. Any questions you may have from Metadata?

So, the question is, how do you encrypt metadata, because he knows there's some libraries in Cloudant that

can do it. The answer is very difficult. I wrote a plugin for Cloudant in order to do that, because Cloudant itself alternated that it injects into the cloud. It's all plain text, and it's completely encrypted. Now, the problem is the chicken and egg situation, in which, what's first? Is it first

the machine? Is it first the provisioner? Is it first the packet or the packet certificate? So, for the provisioners, the situation is attached to a real solution. I would say the best thing you can do is to create some PSK and hope that nobody knows it. And for Poppit, for example, or mCollective, I wrote a plugin

for mCollective that what it did is to presign a PSK into Poppit to know that the nodes were coming up, so Poppit knew which ones to sign, so it didn't sign anything that it didn't know about, which is as secure as you can get. You need to put a PSK in the middle

and the NSA will know about it. It's just crazy. Any other questions? So, the question is, there's bad ways to solve this big, big problem. If there's any good solution,

the answer is not really. Those are the ones that I think are closer to the solution, but all of them have caveats, and all of them have problems as well. So, I would say, at the end, it's just a question of what's the best for your platform. Are you able to inject $10,000 into making this happen in your company, or do you get, as most of us do,

$100 in the same game? So, it's a question of how much money can you inject into this, and how important is that?

So, the question is, if I'm asking for a central database that is completely agnostic for metadata, and for example, Cobra tried to do that in the past, and yes, that's what I would like to see. I said, what I would like to see,

and my hopes are, go against what the industry thinks. So, everybody thinks that they solve this solution with their own platform and their own product. I said, it's just like the case in the past, it's not just more and more established.

Thank you. Thank you.

Oh, yeah, you can do that. He's great. I know. For the recording? Yeah, okay. Excellent, I think they might be explaining.

100% on that, but they're definitely being recorded.

I have one, as I showed up,

and I realized, I don't have to be using truck-related. That's awesome. I have like 30 minutes left. Wow, good time. All right. The room's full anyway, so yeah.

All right. So, you can take it nicely. Amazing. There he is. All right, so next up is Corey Quinn. He's going to introduce us to Saltstack. Again, I'm quite interested because

I literally have not seen anything about Saltstack. Except that it's interesting. There we go.

25 minutes. We'll keep it, we'll pay it forward. Okay, before I dive into this, can I see a show of hands of who's ever run Salt before?

Okay, roughly half of the room. That's quite reasonable. The reason I ask is just because whenever I give these talks, it seems that the audience is sort of varied as far as their level of experience, level of exposure. Some people have never even heard of it before and just wanted a seat in the room they could actually get into. Other people have been contributing for it for years and just want to ask fun questions in the end.

It's an interesting thing. First off, I'd like to apologize a little bit. They wound up cutting times in these presentations, so I know I promised a demo in the end. But unfortunately we're not going to be able to look at that at this point. But we do what we can.

Sorry for the false advertising on this. Okay, so who am I? To answer the question initially, I do not work for Saltstack, although Dave in the corner does. We'll be fact-checking me on everything I say and stupid, but just most things. I'm a technical consultant at TABS, a company based out of the Greater Bay Area in California. I'm also staff for the Free Note IRC

network in my spare time, where we contribute things like file time. And I was number 15 to contribute to Salt, which is kind of awesome that we're above 500 now. Unless we talk about things I remember way back when. At the moment I'm currently packaging it for Homebrew for Mac, but I spent an interesting year as the Ubuntu packager as well.

And didn't make it onto the slide, but three days ago I wound up passing the newly created Saltstack certified engineering certification. Fantastic. That went on the resume. Oh, yes. Okay, so what is Salt? This is not going to be an in-depth

explanation of everything that Salt does, because at this point the project has thrown and blossomed in such a way that this would take several hours to go through. And frankly, no one has that kind of attention. At least of all me. We're going to be relatively high level, an overview of what it does. And we're going to come back before the end with a couple of the high level features that Salt

does that you may not have heard of. So what we're going to do is we're going to start with a basis of what Salt actually is. It's configuration management meeting remote execution. And as far as what those are, let's break them down one at a time. Configuration management, as I see it, really consists of five primitives.

And there's no configuration management system out there that I'm aware of that doesn't do these five things. Once you end up achieving all of these, you effectively have control of the number. And that's not particularly difficult these days, and it's not hugely interesting. Everyone does it, but

the question is, what else is actually out there? Analyst studies have shown that anywhere between 4-10% of the companies out there are running a configuration management system that is not developed properly. So, far and away, the most common thing people are still using in 2014 is SSH in a form

or something built around RCD. This is somewhat unfortunate given that people are reinventing an awful lot of wheels, and it doesn't necessarily have to be there. Now that we've covered what configuration management is, there's really only one primitive for remote execution, which is to do the thing over there. You can think of it like a flank thrower.

Basically, I want to set that thing over there on fire, done. Problem solved. The nice thing about SALT is that it leverages a message bus to do this, so you can do the thing everywhere, or on a very clearly defined subset of things. So you're not going to do this in series. You're actually going parallel

throughout your entire environment. And at this point, it does scale to tens of thousands of nodes, and this has been tested in a number of environments. LinkedIn, for example, is currently running this at that scale. It's kind of an interesting thing to say. So, one of the things that I recommend to just dive into what makes it a little bit interesting

compared to other contenders in this space is the simplicity of the configuration. It doesn't have anything approaching the DSL. It's pure YAML at this point. In fact, this is not an abbreviated example. This is actually taken out of a running production environment. It starts off by defining... all this does is handles HTTP. Unlike a lot

of entrants in this space, one thing that's probably worth pointing out is that it is preceded... it actually does have a dependency model that goes top to bottom, which is nice. It doesn't randomly allocate that. That was actually added in relatively recent generations. So it starts off with obviously defining the package. It has to be installed. It does that.

It then winds up managing the file itself, obviously the configuration file, and it gives it a source. That source can be a static file on disk. It can also be tempted about with Jinja, which we're not going to dive too heavily into. And one thing that I want to point out as well is that it then defines a service as running.

And I forgot to go over there. I'm kind of going... it watches the file, so it will restart the service when that file changes. And lastly, what I want to point out that makes this a little bit interesting is the require statement at the end. If it goes top to bottom, then why would I bother to put in a require there? Simply put, because if I don't require

it, something goes wrong putting that configuration file into place, it'll just continue iterating through. If that require fails, it will not start the service, which actually winds up providing a nice fail set. Picture a scenario of adding a load balancer, or adding a web server to a load balancer, where you don't actually have static assets in place on the web server. That's how you

tend to have embarrassing production actions. One other interesting thing as well that takes it a bit beyond this is something called the Event Reactor System. And this is where SALT really wants to turn. You just saw the dependency model that I laid out. That was not a random thing.

Because what this does is effectively does the same thing. If a file changes, you start the service. If this, then that. What the Event Reactor winds up buying us is it gives us that same type of dependency model, only we're no longer talking about a single node. We're talking about things like if this web server comes up,

then add it to the load balancer. If that server exceeds a certain threshold, remove it from the load balancer. You essentially wind up being able to map dependencies and have cause and effect relationships throughout your entire environment. This is something that traditional systems, particularly our old friend Arsyk and SSH, tend not to do as well as you would hope.

It's environmental organization, and that's sort of a new and interesting thing. Something else that's been included in SALT for a while as well is called SALT Vert. It's actually a built-in part of SALT. It's not a separate project. And this buys us a few interesting things. Specifically, it lets us deploy virtual machines. Today,

KVM is the number one first-class citizen, although support for LXC is coming up, as well as Xam. At SALTCOMP three days ago, we were also given an interesting presentation on how it integrates well with Docker, which was presented here two docs ago. It's really turning into an interesting space.

What this winds up doing is this adds a great abstraction layer on the business of instantiating and running VMs. Being able to pre-see the image with SALT's configuration management system means you can decide to spin up at the end on a particular hypervisor in your environment, and it comes up automatically populated,

which is rather nice. At that point, that's a neat idea, but it also starts speaking to something that's a little bit higher level. That's called SALT Cloud. What this does is it winds up provisioning into both private and public clouds. Yes, welcome to the cloud. It's where we have ops.

In the next SALT release, this is actually going to be merged in as well, which is a component of SALT, no longer a separate project. It's already been done in the current release candidate, which we're expecting to release this coming week. What this serves as is an interface to cloud providers, which is fairly comprehensive at this point. What this does is it's not just a list of

ooh, look at all the things that we support, but this represents something that we tend to be driving towards as an industry, whether we realize it or not. For example, as a consultant, I wind up speaking to a number of companies that are doing migrations either into or out of AWS. Moving it in because of the rapid provisioning

and instantiation of environments, and moving out because holy crap is it expensive to do it at scale. When you can build a data center for what you're spending in Amazon in less than three months, it's really time to consider maybe doing something else. The problem is, and the reason I pick on Amazon for this, is that they are obviously the market leader with a lot of very

interesting platform specific services. Take RDS, their database service. That's great, but no one else really offers a database service at that same layer. What this really speaks to is what it's going to take going forward as we start building out cloud environments that are truly portable is that we also have to wind up

building services as well rather than relying on the ones that providers give to us. Rather than going with RDS potentially, you instead define some form of new design state within salt for whatever you're using. This doesn't need to be salt specific. You wind up spinning up your database service inside of the container. You wind up restricting your interaction

with AWS or any other cloud that you're using to basic primitives. Stand up an instance, spin down an instance, add it to a load balancing layer, and then have everything else defined in terms of what happens once those instances are up. You've bought yourself tremendous portability. Obviously, it does want to be sort of a fascinating experience, but you've got to figure out

how to scale it sooner rather than later as opposed to once you've tried to evolve and then reinvent the wheel that's already bought you a tremendous amount of technical debt. This is sort of becoming a new best practice in the industry that some of us haven't quite fully realized yet until we wind up smacking us on the face a couple of times. As a consultant, I get to see it a lot, but people

who are embedded in various shops either are just starting to realize it and don't have the luxury of moving on to a new project in six months so they can forget and do it better next time. So it's definitely worth calling out as one of those dependencies. If you manage the targets of common APIs, your migrations wind up hurting a lot less. As I mentioned earlier,

we're definitely looking at a point now where there's a transition in going on in the configuration management space. Note management has been relatively solid for a number of years. It started with CM Engine, then Puppet and Chef wound up addressing that, but now with things that are coming up like Salt and Ansel,

we're really starting to talk a little bit more about the ability to manage in higher environments. When you start thinking more holistically and being able to interface between different services, that really becomes a powerful thing and it drives towards a new way of looking at your environment. The real fun part about this

is, I've talked a little bit about what's coming, but specific to Salt, what's next? What is it that we're looking to wind up doing? It's a hard question to answer because probably in the time I've been giving this talk the renewed pull requests have come in, each with a new feature. To say that Salt moves rapidly is being a little bit understating the case.

There's currently a project in the works that's going to significantly shore up the integration with Docker. It's already working in an early alpha stage, but it's not there yet. And there's a separate project that I want to go into very slightly that's actually, sorry, before I dive into this, one thing that's interesting about Salt today is that it's built on top of ZeroMQ,

which serves as the transport layer. Very soon, that's going to be replaced with a transport that by default uses UDP. It could also be TCP as well. It's currently in development and is just announced this week called Great. This is sort of new. This hasn't really hit the configuration

management space, so you're pretty much here seeing it here first. Hooray! Secrets. What this does is it makes the communication protocol very pluggable. You can drop in ZeroMQ, which is there today. You can use Salt SSH and run the whole thing agentless and just use our old friend SSH. But now you can also drop in this as well, which is sort of forcing a refactor

of how things have been addressed before, which means other transport layers are available in the future. What this does is it means that your actual queues wind up, you have multiple queues per socket, which allows for things like packet prioritization, and it scales up rapidly. This very realistically takes Salt from being able to scale from tens of thousands of nodes

to hundreds of thousands of nodes, which is turning into something relatively interesting. Not a lot of companies are at that space yet, but it's coming, and it's really neat to be able to see it. At this point, it also winds up taking encryption down to the socket layer, which is rather convenient, because at that point, encryption becomes pluggable as well. The

value of that is that it winds up using published crypto libraries, which means that at this point Salt can actually get out of handling the crypto space itself, which in the past has led to some interesting challenges, as I'm sure some of you have heard. At this point, this has been my talk,

and if you have any questions, this is something I've been good at it by, something I've been talking about for over a couple of years now, and it's something that I really hope that people wind up seeing a bit more of. The next talk, just so you're all aware, is David Ludacourt from Puppet Labs, who's going to talk about a project that's near and dear to my heart, provisioning with Razor. So it's definitely worth sticking around if you're on the fence about it. Are there any

questions I can answer for anyone? Yes? I've never heard of Wash. Never heard of it? I'm sorry, your guess is as good as mine. Sorry. Yes?

Well, the lack of separation I hope is going to make it very easy to separate the orchestration there, because all the things

You're right, it is. What's interesting is the way that you can actually wind up structuring your state tree. We can leave them a very clear separation. There are a number of shops as well today that are using salt to orchestrate existing puppet or shaft environments, to use existing other systems entirely for management of nodes, but then turn into salt for the ability to orchestrate

them as environments. At its very basic level, it acts as an incredibly powerful replacement for that. If all you're ever using it for is to kick off runs of your puppet nodes, just so you don't wind up having everything fire at once and destroy your puppet master, that alone is sometimes worth looking into. Does that answer

your question? Thank you all very much for your time. We have 15 more minutes, if you have something you'd like to ignore. I'm not prepared.

I'm still 25 minutes the most. No problem. In that case, we'll Climb on the table. I guess we'll have a little break. Otherwise we're going to be too much ahead of the schedule.

We just took a 15 minute break. We'll continue with the next talk.