Just put a thumbs up or something. I'll try to, oh, if you need a little louder, I'll try to get it going. OK, this talk is called, I can't tell if we're closing the doors or this. Anyways, we can just roll with it. This talk is called Consuming Open Source Infrastructure.

So who am I? Well, my name is Spencer Crum. I work at HP. My primary open source contributions are to the Puppet ecosystem and to the OpenStack ecosystem. What we do is we run a CI system inside HP.

And it turns out that it's a clone of an upstream OpenStack CI system. And Liz will be talking about that directly after this talk. So running somebody else's infrastructure as your own, it turns out, is not super trivial. I thought it would be a lot easier than it ended up actually being. And so this talk is going to cover sort of why that was

and what we've gone to and stuff like that. So the target audience here is really for operations people. There's three main people I want to talk to today. The first are people who are currently pulling down someone else's configuration, consuming someone else's open source configuration. The second is people who are considering doing that, maybe making plans.

And the third group are people who are producing something in the open source and want to learn how to do that a little bit better as somebody who's actually doing it, I can give you some feedback. So this is very much a what are we doing talk. And it will talk about where, as we canoe across the river, we actually are, which does not

mean that I'm going to give you 10 steps to do it easily. It means I'm going to give you a few key takeaways. And that's kind of what you're going to get out of this talk. And I'll try to slow the narrative down a little bit when we get to those takeaways so that we can really focus on them. And also, we're going to be fine on time. So feel free to get loud and ask a question immediately

when you think of it. So we always define our terms when we start up something. So open source. Open source to me means something that has an OSI approved license on it. But it also means a little bit more than that. So when I first learned about open source, I was taught by Bart Massey, who's a professor at PSU.

And what he said is that you can write a piece of open source software, and then put it on a hard drive, and then put that hard drive in the basement. And it's just as open source as it was before. And so there's this element of being able to share it. And that is kind of required. So I consider it only really an open source infrastructure

if you're actively doing your best to share that code with people, not just licensing it and hiding it somewhere. And so that means you put it somewhere visible. There's another concept called open development. And so what that means is that it's not really open source unless it's developed in the open. So for instance, if a company, everybody

goes to the company from 9 to 5, and they have meetings inside the company. They have internal emails in the company. All the code review is internal to the company. And then a cron job runs every Friday that pushes it to GitHub. That's not openly developed, because there's not a forum which anybody can join and communicate. So infrastructure.

Well, this is kind of the beginning of an infrastructure. It's some kind of a plan. And we can take that plan, that diagram of things connected to each other, and using configuration management tools like Puppet, we can put that into code. And that code can be committed to some kind of version control repository, and then that can be used to share. And now that it's code, it can be OSI licensed,

and we can start talking about it, infrastructure as code. So consuming, what does consuming mean? Consuming literally means to eat. But it basically means using something you yourself did not write, that your organization did not write, someplace you got started. I'm also going to use words like upstream and downstream. I'm going to say you, me, us, them, we.

And what I mean by that is that I consider myself a downstream. We at HP, we are a downstream. We consume from upstream. And upstream is OpenStack's continuous integration team. And when I say we, I generally mean downstream. And when I say they, I generally mean upstream.

But we'll talk a little bit more about that later. So it's not just OpenStack that's doing this, not by a long shot. The Wikimedia Foundation has an open source infrastructure. Oh, my bad. Jenkins, R. Tyler's running around here. He's always looking for people to help

with the Jenkins infrastructure. And Mozilla's infrastructure is also open source. So there's some takeaways immediately on why this is good. And the simplest thing to say is that we benefit from consuming infrastructure because we don't have to do all the work ourselves. Just right away, the fact that they wrote a 10 line shell

script and we get to use it means we have to do significantly less than we used to do. It's the classic not invented here. It's the inverse of not invented here. When I was on a robot team at the end of my college career, we were building a quadcopter to fly. And one of our experienced mentors said, you know, you should just start with the Parrot AR drone and then add your features on top of that. We're like, no, we need to write our own. And then 10 months later,

we finally had something that would fly and we could start working on the real project. So we don't have to do all the work ourselves. This is a good thing. But I think there's a more important lesson. We benefit from consuming the infrastructure because we have confidence that the architecture is viable.

And so what that means is when you set out a team of architecture, a team of DevOps people to go build something, they're gonna spend some time planning it. They're gonna spend some time mapping it out. They're gonna spend some time building it. And all that time is essentially risk. There's projects that are waiting for them to complete. There's developer time you're spending on it. And if it doesn't come to something that's useful to you,

that's all lost. But if you know that someone else has used this implementation successfully, that gives you some confidence that you're not gonna end up losing that much, which basically reduces the risk. So the first thing we said was an advantage was that we didn't have to write everything ourself.

And there's a corollary to that, which is that any departure we take from the reference implementation is immediately technical debt. That's something we have to think about when we consume someone else's infrastructure. So about me. Our team at HP is called the Ghostbusters,

and our project is called Gozer. And we consume the OpenStack CI infrastructure. And this is kind of where the rubber meets the road. So there's some tools they use. They use Puppet for configuration management. They use Ansible for orchestration. They use Git for version control. And they run all their infrastructure on OpenStack.

And so they have two OpenStack clouds. They have a Rackspace cloud and HP cloud, and they kind of split their servers between them. So this is kind of what operators do with Git these days.

Just put the Git on everything. Which is a good thing. Back when I was at PSU at Portland State University, we used to tell other people that this directory is managed by Git by touching a file called all caps, use the Git luke. It's a good thing, it's a good thing. It means everything's a Git,

which means it's shareable, which is actually really cool. So if you look at the goals for both of these teams, both teams are building CI infrastructure for OpenStack developers. Upstream is building it for OpenStack, or Upstream developers. And downstream in HP, we're building it for HP developers. So what HP has is an OpenStack product. It's called Helion. You should buy it.

And when we take standard OpenStack, we apply a little bit of HP internal patches, and then we test those HP internal patches with HP internal CI. And it's attempted to be a pretty good mirror reflection of what happens upstream. And so that means you look like this. But if you take a little bit further look,

our goals are actually somewhat divergent. So the upstream OpenStack team has to run a lot of daemons that we don't have to run. They have to run a bunch of infrastructure for publishing packages to PyPI. They have to run a lot of infrastructure for voting and other technical or governance needs of the OpenStack foundation. They have to run a bug tracker. We don't have to run those things,

because in many cases, those don't apply. And in terms of bug tracking, HP uses JIRA, so we don't have to pick these things. But it also sucks downstream, because downstream, we started out as a team that would provide CI infrastructure to OpenStack developers with an HP. But they got so excited about the idea that now we have developers all over HP coming out

of the woodwork saying, run our tests. And so we have .net tests, and we have Docker tests, and we have VMware tests, and we have Java tests. And some of those can map to the models that we learned from upstream. But some of those can't. And every time we roll some custom solution ourselves, we've incurred technical debt.

So it really, I don't know how to make a Venn diagram with open source software. This is like, this is like 45 minutes of my time. It's just unreasonable. It's pretty bad. So what are some takeaways?

So for a downstream, what you can do is as you get started with this project, like early in the project, look at what, you remember that diagram of everything connected to everything else, what do you not need to replicate? Resist the urge to replicate all the things, because that's just work you're going to use, and maybe you don't have to do it. For upstreams, keep track of what you're coupling

to what, and where your assumptions are. And obviously it's really hard to figure out what your assumptions are, but here's some ground rules. So don't imply colocation. So if you have two services that are colocated on the same host, talk over a localhost socket. Don't just go directly to file system access. That way a downstream can split them. Don't assume user ID and group membership.

So if SSH is one user to another user on a different host, don't assume that you have the group permissions, or group write permissions in some directory somewhere. And this is kind of a good one. Firewall rules and network proximity slash layout. It's really easy to think, well, our network is set up like this. And maybe you can't change that. Maybe that's just the way it is,

but you can document it, which will help your downstreams figure out what to do. So let's talk about the network. So upstream, the kind of reference implementation of this CI system has two open stack clouds they work with. One is Rackspace, one is HP Cloud. And so every server they spin up has a public IP address, and they firewall it all the way off,

and then whitelist select hosts. And so what that means is that any host on the network, or any host in the infrastructure that wants to make a connection to any other host can do so as long as it's plus one by the administrators. When we went to do that at HP, we made a mistake of not respecting that network

topology. And so we built it out with two data centers. Well, one data center in a physical data center in Fort Collins in Colorado, an HP data center. And we put some of our infrastructure on HP Cloud. And that represented a different network topology because connections from the data center to the cloud were fine. That works. But the cloud can't make connections

into the data center. And so the reference implementation has push, push, push, push, push, push, push. And we had to hack it to be push, push, pull, push, push, pull, which is hard enough to begin with. But we had to somehow trigger pulls. And so it ended up looking kind of like this.

And there was another issue. So anybody who's ever worked in a corporate data center knows about HTTP underscore proxy as a variable. And so what that meant is that anywhere inside HP's network, if you want to make an outgoing network request, you're allowed to do that as long as you

go through the HTTP proxy. And what that meant was we had to change a little. We had to make these small little changes to add HTTP proxy validation, to add that to the environment, to add that to the daemons or whatever. And so that meant that we had to go upstream. So let's talk about upstream contribution. So downstreams must contribute upstream. This is probably the central thesis of my presentation

today. Every single member of the Ghostbusters team contributes upstream. And we don't do it 50% of our time. But there are some days it's 100% of our time. And there are some days it's 10%. And there are some days we're not at all. There's practical and social reasons why you have to do that. The practical reasons are stuff like this. We just need to, this is the puppet templating language,

by the way. You just have to sneak in there and add a little, if HTTP proxy do something. And if you're a downstream, this is your bread and butter. You're going to submit this to upstream a lot. It's a small little change. It changes behavior slightly, not in a big way. It's off by default. And you could turn it on when that's consumed downstream.

But there's also social reasons. And the social reasons are important. And you can think of the social reasons as an exercise in social capital. And how do you get social capital? Well, you get social capital through trust. It's essentially just a trust exercise.

And so these are the things that we get from upstream. We get testing. They test our stuff for us. They write a bunch of code that we consume from them. They review our code. They support us when we have issues. And we get to read their docs. And so when I first started writing the slide,

I had a long list of things we provide back to them. But then I realized it's the exact same list in terms of what we push upstream. And the reason we have to do all of those things is because when we have an issue with one of them, we need it to be important to the upstream core developers that our issue becomes solved.

And if you show up in an IRC channel or on a mailing list and say, it's all broken for me, and you're someone who is not known to that community, they will spend 10 seconds trying to help you and then get on with their lives. But if you are a consistent contributor, if you're solving problems for them, they're going to go that extra mile for you. And it becomes a two-way street. And it's a trust relationship.

And so the takeaways from this is context matters. We got bit by the network. And I think after this presentation, if any of you sat down to try to emulate something, you'd look really hard at the network. But there might be other assumptions that you're not thinking of when you're trying to consume like that. And so one of the things I have it in my mind that

could be true is geographic location, network latency, and maybe virtualization versus bare metal. But those are just examples. But this is really what comes out. Upstream contribution is not an option. You have to contribute upstream. If you don't contribute upstream, you might as well just stay at home.

There's really no point. So there's different ways to contribute upstream, different levels of upstream contribution. And you can kind of describe it in two camps. You can say successful upstream contribution and unsuccessful contribution upstream.

And these are the same rules for contributing to any public open source project, except this one happens to be infrastructure related. And it's simple things. It's being consistent. It's being reliable and frequent in your reviews. It's submitting small patches instead

of big giant dump patches. It's having a regular cadence. It's showing up to the meetings. And it's doing things that are so fufu, like active listening. And I think we all know what an unsuccessful contribution looks like, right? It's like somebody showing up with a 3,000 line patch saying, hey, I solved all the problems.

Just merge it, and we're good. And the developers are like, hold on, no. And the developers say, I have all these technical complaints with your issue. And the downstream contributor says, no, man, it works in our end. Just merge it. And then they get hostile. And they say, no, it really works.

This doesn't work, blah, blah, blah, blah, blah. And that's the kind of person you don't want to be. And if you are that kind of person, you're not going to be successful in this model of consuming and giving back. And in fact, consuming is really the wrong word, because it's a two-way street. OK, so now we can talk a little bit more technical. And I know this is the configuration management room, not the puppet room.

But we're going to talk a little bit about the puppet situation in OpenStack info right now. So when I first got there about seven or eight months ago, this was the situation. Upstream used Hira for secrets only. They were on puppet 2.7, which is really bad, even for seven months ago. They used a site.pp, so they didn't have an enc.

They had a monolithic repository, which meant all of their kind of modules, all the code they wrote, was in one big Git repository. And they had a modules dir in there, and then 60 or so modules. They were stuck on Apache module version 0.0.4, which is from like 2012.

And you bring that up. So Apache module 0.0.4 is very old. And what that does more so than, because you can get anything you want done in the Apache module version 0.0.4. It's not like there's features missing from that. But because there's an API change and you can't upgrade to a modern Apache version,

that means you can't use any module that depends on a modern Apache version. So that only sends you further down into the system of consuming old crap and writing your own crap. They had a forked VCS repo module. In fact, they still do, which means that there's OpenStack CI dash VCS repo instead of Puppa Labs dash VCS

repo, which pisses off the module installer like nobody's ends. They had a decent role system. And we'll talk about that in a minute. Actually, we'll talk about that next, which turned out to be really good. And they had a shell script for installing puppet modules. So what that means is that rather than using R10K or something, it's just a shell script and it wraps puppet module install.

So for the roles, they had a pretty good role system. So there was an entire module called OpenStack underscore project. And that had classes like OpenStack underscore project colon colon Jenkins. And that's where they tried to put most of their OpenStack

specific stuff. And then they had a module Jenkins they wrote themselves that was more or less free of hardcoded variables. Of course, there's hardcoded variables all over the place. But at least there's a way forward here, right? And the same with Zul, which is a daemon we were at, which Playa will probably tell you about. And so when we were bootstrapping this, when we were first getting it, I should probably

When we were first getting things ready to go, we did a deep copy of OpenStack project to a new module called HP. And so we then did a find or place. Everywhere it said OpenStack project, we replaced it with HP. So now we had our own HP colon colon Jenkins that could go use the kind of public Jenkins module. And what we actually did is two Git repositories.

So we pulled down our config repo, which basically just had the HP module in it. And theirs, which had OpenStack project as well as a bunch of other modules. And we put both of those directories in the module path. And so OpenStack colon colon project, or OpenStack underscore project was still available, but it wasn't referenced anywhere in our site.pp.

So it actually worked pretty well. And that gave us the opportunity to do some really neat stuff. We can change in HP anything we actually need to change without touching upstream. And downstream, we can just run Git pull or Git cherry pick to pull changes from downstream. So that's a really great way of describing

something that actually went pretty terribly. So we fork every repo we use, which you can argue about that decision or not. But we need to be able to control what's in our environment. So we kind of need to control it. So we kind of need to fork it. And we very rarely sync that kind of public from upstream. We are 1,000 commits behind, like no joke.

Which isn't actually as bad as it sounds. But the worst part is that we have the several patches ahead. So there's places where we needed to change it and now that we have changes on our own, we can't just pull anymore. It's going to be merge conflicts. It's pretty rough. And even though OpenStack in front of the upstream

has moved to 3.7, we're still stuck on 2.7 because we can't move fast enough to pull. So what are the takeaways here? Find your intermediary between upstream and downstream. And I'm not saying that you make a module called

OpenStack underscore project. I'm saying that you find someplace that both you, as downstream, and they, as upstream, can hard code variables without stepping on each other's toes. And I think the traditional way, if you were green fielding this, you would probably do that today with high ref. And there's an interesting story about that,

which is that the first time that I tried to do this, I suggested high ref. And I wrote some patches and I sent them up and they weren't merged. And you can argue about why that wasn't happening, but I'll tell you why I think it didn't work. And I think it was because I was a relatively new contributor who had not built that trust relationship, back

and forth trust, codependence relationship, with the upstream cores. And I was proposing this patch that was very complex. It was ripping out a lot of things, changing things that were hard coded into variables, doing high ref lookups. And so if you consider those two factors together, it's a new contributor we don't trust

and a big bite of complexity. And that's an obvious no. I think today, now that I have a fair amount of social capital with these people, I could say, look guys, I think we need to go this higher route. And I know a lot of them and I know how to address their concerns. I know how to talk to these people. And as a result, I could probably get that merged now.

And it's just one example of how you have to be serious about contributing upstream, otherwise this model can't work. The other thing is that use small sinks. And so if you go upstream and you drop a 3,000 line patch bomb, you're going to be told no.

You're going to be sent away. They're going to laugh you out of the building. Similarly, if you wait three months before you sink from upstream, you're going to have the equivalent of a 3,000 line patch that you're trying to dump in your infrastructure. And you're going to be like, no, I don't want to do that either. So the only way forward is to sink often with small little sinks. And what that means is you need some kind of testing framework.

And one of the errors we made is we didn't set up some minimal acceptance test framework at the beginning. And that has put us very far behind. And that's what I mean by reflexive of upstream, is that if you wouldn't accept it upstream, why would they accept it downstream? Or if they wouldn't accept it upstream, why would you accept it downstream?

So this is my model for keeping up with upstream. I don't know if you guys are familiar with the story of Sisyphus. If I recall correctly, Sisyphus was a Greek who pissed off a god. And so he was sentenced for life to push a boulder up a hill. And as soon as he pushed it up the hill,

it would roll down again. And he'd have to push it up. Or he could be a kitten on a sled. So let's talk about what has happened since upstream. And I am pretty serious when I say that the Ghostbusters are a big part of these changes. We've upgraded from Puppet 2.7 to Puppet 3. Also note that I use the word we. I totally consider myself a part

of the OpenStack upstream team. And I think they consider me a part of the team, too. And that's your goal. We've split the data into a separate Git repository. And interestingly enough, we did not actually use Hira. So there was a lot of hard-coded data. There's some config files that Playa will tell you about later. But there are these giant blobs of data, like YAML, and which jobs to run,

and how many projects there are, and who's allowed to merge what where. And as kind of a Puppet fanboy, my inclination is to put that all in Hira. But what the team figured out to do is they can put these flat config files that configure that daemon in a Git repository, and just tell Puppet VCS repo

to just make sure it's over there, and point Zul at it, and point the daemon at the config file, and you're done. You don't actually need the complexity that Hira provides, which turned out to be a really cool result. It also means that there's now, when someone pulls down our configuration and tries to use it downstream, they just have to add their own Git repository

full of config files. It's pretty slick. We've split the monolithic repo into repos for modules. So like yesterday or something, Jeremy sent out an email that said, we announced that we have 61 new modules. And they're all pretty OpenStack specific. That's totally true. But now that they're all split out individually and it can be published to the Forge,

we're hoping to get folks like you involved in maintaining and using them. There's machinery to release modules to the Forge using Puppet Blacksmith, more or less the same way we release PyPy modules right now. So if we want to release a new version of, say, OpenStack Nova, what somebody does who's core on Nova is they make a signed Git tag,

and they push that Git tag into our code review system. And then machinery fires, and it ends up on PyPy. So now you push a signed Git tag of a Puppet module and some machinery fires, and a new version ends up on the Forge, which is a great pipeline. So you said that the Apache 0.0.4 module was pretty good.

OK, well, somebody in our team is really excited about the Puppet 0.0.4 module. And that's because the Puppet Apache 0.0.4 module is weakly modeling.

So a modern Apache module, you have defined types for mod rewrite rules, for example. And so you have to not only understand your Apache configuration, but you have to translate that into the Puppet DSL. What the Apache 0.0.4 module lets you do is just say, here's a template. Put that in my vhost pools. And so that means you, as an administrator,

can write native Apache code, and Puppet will figure it out. Puppet won't have any visibility into the resource it's creating, but it can get the job done. And so some people think that that's feature complete. It's weakly modeled versus strongly modeled. It's just as good. And so what we've actually done is we've done a deep copy and another find and replace. We've created an OpenStack HTTP module,

which means that anybody who's a stickler and wants to use the old one can use that one, but it can coexist in the same module path with a modern Puppet Labs Apache module, which means that we can pull in a new one and there's an actual upgrade path. It also means that anybody who wants a weakly modeled Apache module can use ours. We have made our installmodules.sh script super smart.

It now has three different types of modules. It has modules to install from the Forge, modules to install from Git, and modules to install during integration testing. And that ties into the last thing, which is we've built this integration testing system upstream so that when you propose a change to the Jenkins module,

it tests that with all the other modules and the site.pp and upstream to make sure that everything works. Now, it's not the best testing. It's kind of weak testing, because all it really does is make sure that Puppet compiles, but that's better than we were before. So what's planned upstream,

so what's planned upstream is we're going to put semver on all the modules, and we're going to probably switch out of using the Forge and go to straight Git clone, because we're tired of the transitive dependency problem in Puppet. And we're going to figure out some kind of a fix for our VCS repo. There's a couple ways we can go on this.

We can either add our, we can change our module to provide a new provider for the Puppet Labs VCS repo. We can also try to merge the behaviors of ours into the Git provider of the Puppet Labs VCS repo module. We haven't really figured that out yet. Yeah.

So when you use Puppet module install, you have a transitive dependency problem sometimes. And so transitive dependency problem means, okay, so the question was, why do we want to go to Git instead of using Puppet module install? And the answer is that there's a transitive dependency

problem where you ask for Puppet Labs Postgres, right? And a dependency of that is the concat module and the NTP module. And so it'll pull down the NTP module, but if you already have an NTP module installed, it will fail to install the NTP module,

and you'll get a version mismatch, and maybe the Postgres module won't work. The Puppet module tool is also kind of bad at identifying that this problem has occurred and reporting that to the user in a way that's detectable by script. Does that answer your question? So what's planned downstream is we're gonna start

dogfooding the OpenStack product that HP develops, which means that we're gonna change from a network that does not well reflect the reference architecture to a network that does well reflect the network architecture. That's gonna be huge. We're planning to upgrade to Puppet 3. That's in progress, but we're operators, right?

So we're fixing the machine while it's driving, and so that's scary. We have minimal kind of weak testing, but effective testing upstream that we wanna bring downstream. And we're gonna, now that all the modules are synced out individually, we're 1,000 commits behind across 61 modules, right? But remember, we're only using a subset

of OpenStack infras tooling. So we're not actually 1,000 commits behind. We're more like 80, and 80 is doable, especially when you can do the commits that went to Zul module, and the commits that went to Jenkins module. You can do that one at a time. It's also worth noting that OpenStack infra sets up a lot of daemons.

Have you ever been so mad at Google Hangouts that you set up an asterisk server? These people were, but I love them. So some closing thoughts. I was hanging out with Nigel Kirsten from Puppet Labs the other day, and he was talking about the days of Puppet 0.24.

And he said at that time, everybody wrote code. And what he meant by that was that everybody who was using the product at that time was reading the source code and changing it to get results that they wanted. And so that means that it's, you gotta treat it like a, so that means there's no,

there's no users who aren't reading code. Like LibreOffice, for instance, a user without reading any source code can install and run and click in buttons. That's kinda where these open source infrastructures are right now. You have to consider yourself a developer. You're gonna have to go in there and twiddle some bits. There isn't git clone dot slash install.

It's not there yet. Maybe one day it'll become polished enough that there'll be infrastructures in a box and you can download them and run them. And I think some organizations, like 4J is one of them, are trying to build that. But right now, with OpenStack infra and probably the others that I mentioned at the beginning, you're gonna have to roll up your sleeves

and get in there and write some code. This is a bunch of links that you can use when you look at these slides on the internet. And that's my contact information. Are there any questions? Okay, well thank you for, oh yeah.

Yeah, yeah, yeah.

That is absolutely one of the paths we want our development to take, which is that the modules right now, despite best efforts, have cross dependencies, right? Hidden cross dependencies, the worst kind, right? So what we wanna do over the next several weeks is identify where those are and nip them in the bud.

And if we need to have a clear dependency, we need to call that out in the documentation and stuff. Yeah? Yeah, so usually there's like unit, so unit versus kind of acceptance, so yeah.

I've done some work, so Beaker was brought up. So there's some work right now in OpenStack infra that I sort of spearheaded. So that we can, using the OpenStack infra node pool system, spin up a machine and run Beaker tests on that machine. That was actually kind of difficult to do

and the hack is kind of janky. Because Beaker is an opinionated tool. Beaker wants to be in charge. It wants to identify when the VM spins up and when it spins down. It wants to be in charge of reporting results. And we already have a lot of tooling for that stuff. It's like considerably better. So two daemons that wanna be in charge, or one set of daemons and another run on command tool

is kind of meant to be done on a laptop, don't really get along that well. But I'd be happy to talk about that after this.

So one of the things I did, so I wrote a module called PuppetBoard, right? So there was a question about, it was kind of long, but. The idea was that you don't have real abstraction layers in Puppet. You still have to ask for what you want, even if you're at a high level. And so you can't ask for web server.

You have to basically be specific about Apache versus NGINX, right? So what we did in the PuppetBoard module was I too got tired of forcing dependencies on people. And so there's a class you include to set up the PuppetBoard Whisky application. And then you include either a PuppetBoard colon colon Apache or a PuppetBoard colon colon NGINX.

So that's provided by the public module and you just pick one. That's better than nothing, I guess. Okay, well thank you for your attention, everybody.