Chef Cookbook Testing Like a Pro
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/34573 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
ChefConf 201735 / 45
1
4
5
9
12
16
18
20
21
22
24
25
27
28
35
37
39
42
00:00
Statistical hypothesis testingSmith chartHypermediaCASE <Informatik>BitUnit testingState of matterJSONXMLUMLComputer animation
00:43
Statistical hypothesis testingNP-hardMultiplication signDrag (physics)WindowGame theoryPoint (geometry)Process (computing)BitSoftware developerData managementSystem administratorDevice driverSoftware testingMathematicsStatistical hypothesis testingCodeOperator (mathematics)Web 2.0AreaWordArithmetic meanPower (physics)Right angleElectronic mailing list
02:37
SoftwareMultiplication signSoftwareSystem administratorProcess (computing)Statistical hypothesis testingData structureStandard deviationPoint (geometry)Category of beingWordCombinational logicOrder (biology)Computer animation
03:17
Normed vector spaceCodeMathematicsMultiplication signServer (computing)Instance (computer science)Point (geometry)Configuration spaceComputer animation
03:51
Data managementPhysical systemStatistical hypothesis testingHydraulic jumpServer (computing)DisintegrationExecution unitLogicCodeVelocityMathematicsTerm (mathematics)Line (geometry)Statistical hypothesis testingQuicksortSpacetimeDrop (liquid)Right angleProper mapGodCodeLogicSoftware developerNumberData managementEmailGraph (mathematics)WebsiteArithmetic meanExecution unitCurvature1 (number)Software testingPoint (geometry)Axiom of choiceWeb pageStatistical hypothesis testingNeighbourhood (graph theory)Utility softwareBlogVideoconferencingConfiguration managementOperator (mathematics)Table (information)Single-precision floating-point formatMultiplication signMereologyHydraulic jumpConfiguration spaceServer (computing)BitSurjective functionComputer animationLecture/Conference
08:48
LogicRule of inferenceCodeValidity (statistics)Formal languageProduct (business)Boom (sailing)Process (computing)Block (periodic table)Point (geometry)
09:57
Execution unitVariable (mathematics)Rule of inferenceWeb pageSpacetimeCodeWrapper (data mining)Statistical hypothesis testingInterpreter (computing)Cartesian coordinate systemMereologyRobotFiber bundleStapeldateiComputer animation
11:10
Rule of inferenceCodeLogicRevision controlComputer animation
12:02
Internet service providerDefault (computer science)Host Identity ProtocolDaylight saving timeInfinityCorrelation and dependenceGroup actionRule of inferenceStatistical hypothesis testingDisintegrationExecution unitRule of inferenceCodeDescriptive statisticsValidity (statistics)Sheaf (mathematics)Group actionDefault (computer science)Shared memoryUnit testingMetadataField (computer science)Function (mathematics)Statistical hypothesis testingCASE <Informatik>
13:12
Execution unitStatistical hypothesis testingoutputStatistical hypothesis testingComponent-based software engineeringVariable (mathematics)Source codeCodeGroup actionTemplate (C++)Installation artConic sectionRead-only memoryNatural numberUnit testingOperating systemStatistical hypothesis testingQuicksortStatistical hypothesis testingCartesian coordinate systemSheaf (mathematics)Function (mathematics)CASE <Informatik>Social classDifferent (Kate Ryan album)Right angleVector spaceShift operatorSemiconductor memoryCuboidMereologyPhysical systemConfiguration spaceLevel (video gaming)outputConnectivity (graph theory)Complex (psychology)Extension (kinesiology)BitEndliche ModelltheorieWritingSet (mathematics)Musical ensembleCodeControl flowStandard deviationService (economics)NP-hardComputer animationMeeting/Interview
15:15
Template (C++)Stress (mechanics)Unit testingInjektivitätCodeLogicStatistical hypothesis testingPoint (geometry)MappingCodebuchComputer animation
16:12
LogicStatistical hypothesis testingStatistical hypothesis testingLogicComputing platformPoint (geometry)CASE <Informatik>Statement (computer science)BitDifferent (Kate Ryan album)WordACIDLecture/Conference
16:59
Perfect groupVariable (mathematics)Source codeService (economics)Group actionStatistical hypothesis testingDisintegrationComponent-based software engineeringSoftwareStatistical hypothesis testingForm (programming)Installation artComputing platformSuite (music)Commodore VIC-20Gastropod shellMedical imagingComputing platformPhysical systemProduct (business)Server (computing)FreewareStatistical hypothesis testingSoftwareMereologyCuboidCASE <Informatik>Right angleService (economics)Projective planeWater vaporReal numberRepository (publishing)Statistical hypothesis testingCross-platformScripting languageRevision controlVideo gameArithmetic meanFunction (mathematics)INTEGRALConfiguration spaceFilter <Stochastik>Point (geometry)Suite (music)Validity (statistics)Client (computing)Installation artMultiplicationComputer animation
20:47
Statistical hypothesis testingDirectory serviceLocal GroupProcess (computing)Statistical hypothesis testingStatistical hypothesis testingBitGroup actionRight angleCategory of beingPhysical systemReal numberConfiguration spaceDifferent (Kate Ryan album)Gastropod shellVector spaceCartesian coordinate systemProduct (business)Descriptive statisticsAsynchronous Transfer ModeExpected valueResultantService (economics)SoftwareWindowMultiplication signPower (physics)Directory serviceDatabasePanel painting
22:49
Avatar (2009 film)Statistical hypothesis testingPhysical systemPhysical systemStatistical hypothesis testingStatistical hypothesis testingComputing platformService (economics)Dependent and independent variablesComputer fileAsynchronous Transfer ModeoutputCategory of beingResponse time (technology)Function (mathematics)Functional (mathematics)Cartesian coordinate systemDifferent (Kate Ryan album)Validity (statistics)Point (geometry)Data conversionServer (computing)Confidence intervalRight angleGroup actionComputer animationLecture/Conference
24:25
Statistical hypothesis testingLogicExecution unitAsynchronous Transfer ModeLocal ringMereologyTotal S.A.Data structureMultiplication signPhase transitionStatistical hypothesis testingComputer fileType theoryDifferent (Kate Ryan album)MereologyAutomationMathematicsComputer animation
25:28
Local ringAsynchronous Transfer ModeConfiguration spaceComputer fileStructural loadStatistical hypothesis testingContent (media)Multiplication signPhysical systemRemote procedure callWeb pageSoftware developerPoint (geometry)Computer animationMeeting/Interview
26:26
FlagLocal ringAutomationProduct (business)Control flowLocal ringFunction (mathematics)Statistical hypothesis testingMultiplication signPhase transitionInstallation artOpen sourcePhysical systemServer (computing)AutomationStatistical hypothesis testingGame controllerMathematicsScaling (geometry)Information securityClient (computing)Software developerPoint (geometry)Coefficient of determinationEndliche ModelltheorieRight angleComputer animation
28:28
Statistical hypothesis testingPhysical systemConfiguration spaceLie groupData compressionMobile appBitCuboidMassMedical imagingDifferent (Kate Ryan album)WordComputer animation
29:08
System programmingComputer networkInstallation artRandom numberProcess (computing)Process (computing)Software repositoryRevision controlPhysical systemComputing platformService (economics)Right angleMedical imagingSoftwareComputer animation
29:44
Computing platformPhysical systemComputer-generated imageryInclusion mapInstallation artIntelPhysical systemOrder (biology)BitPrincipal ideal domainMedical imagingWeightService (economics)Real number
30:26
Statistical hypothesis testingInstance (computer science)Physical systemMIDIHacker (term)Local ringInterior (topology)Normed vector spaceInstallation artExecution unitConfiguration spacePhysical systemRepository (publishing)Function (mathematics)MathematicsDifferent (Kate Ryan album)Statistical hypothesis testingCASE <Informatik>Product (business)Workstation <Musikinstrument>Right angleView (database)Stress (mechanics)Statistical hypothesis testingConfidence intervalComputing platformSpacetimeBit rateTable (information)MereologyProcess (computing)Binary codeSweep line algorithmInstance (computer science)Suite (music)Execution unitComputer animation
32:28
Confidence intervalConfidence intervalData structureConfiguration spaceMathematicsSeries (mathematics)Statistical hypothesis testing
33:01
Computer fileStatistical hypothesis testingWritingRepository (publishing)Statistical hypothesis testingData structureBlock (periodic table)Vector spaceCommutator
33:34
Task (computing)Multiplication signGoodness of fitStatistical hypothesis testingDifferent (Kate Ryan album)Right angleFigurate numberServer (computing)Computer fileAttribute grammarCuboidComplex systemLevel (video gaming)Revision controlElectric generatorINTEGRALComplex (psychology)Java appletMassNatural numberStatistical hypothesis testingPoint (geometry)Category of beingTraverse (surveying)VirtualizationCodeShared memoryGene clusterPressureParallel portPhysical systemPower (physics)Presentation of a groupTheoryMereologyVariable (mathematics)Computing platformWordLattice (group)Term (mathematics)Traffic reportingSingle-precision floating-point formatLaptopInstallable File SystemWebsiteMathematicsCASE <Informatik>Installation artTotal S.A.Replication (computing)FeedbackStatement (computer science)Logical constantLinear regressionInstance (computer science)Combinational logicUnit testingPermutationDirectory serviceMedical imagingLogicConfiguration spaceReal numberSuite (music)Lecture/Conference
40:45
JSONXML
Transcript: English(auto-generated)
00:06
Might as well, 3 o'clock, right on. Yeah, so I'm Tim Smith. I'm a media engineer here at Chef. I work on the community cookbook team. You've probably seen some of the stuff I have worked on and maybe used some of the cookbooks.
00:21
So I've been through a bit of a journey of testing, and I kind of want to share everything with you. I do want to throw out just one quick warning. I'm definitely going to start from the absolute beginning. So if you consider yourself a decent tester, like a solid eye, do testing, this might not be the case.
00:43
So the first question I think I want to ask is really just why test at all? It's kind of like the elephant in the room. And I know out there somewhere somebody is saying, like, it's just my servers, it's just some infrastructure, software testing is weird, that's what developers do. And that's totally fine. I started my journey as a sysadmin
01:02
doing Windows administration, and I don't mean like the kind of cool Windows administration we have now, not like PowerShell and software development and that kind of thing. I mean like Windows NT, Windows 2000, like I'd click on stuff and drag things and maybe put a floppy in and try to figure out why device drivers don't work. So my job was very much traditional sysadmin,
01:23
point and click, this and that. And I really got hit with a ton of bricks when the infrastructure is code gaming. I was still doing operations work, and I was even doing web ops work for a pretty important, pretty high traffic website, but it was still Windows, still Windows 2003 and 2008,
01:41
and it was still very much pointy-clicky. There was a little bit of some fancy math scripts, but definitely not the advances that we have now. So I made a lot of excuses. I made excuses for why I didn't need to test that infrastructure. It took a lot of time, and sure, it does. It does take a lot of time. I was hard, also completely valid.
02:02
Testing required you to learn new things, and that is hard work. Quite often, you will spend significantly more time on the tests than you do on the actual code that you're trying to test. And the last one is kind of like me almost putting a bit of a manager hat on, and that was that it slowed us down, right? We were trying to compete with competition
02:20
that was 10 times larger than us, and I just didn't have time for this. I needed to get that code out there, I needed to deploy it, we needed to ship, and if I spent time testing, that was time we weren't making money. But I was really leaving out one thing. Even though these were all completely valid, I was leaving out that it was just quite simply my job. Testing the infrastructure was my job.
02:42
And it doesn't matter what your title is, because I was a sysadmin at the time. If you're a developer, you're writing software that's helping your business. And if you're on the off side of things, you're keeping that software up in order to help your business. And if you are lucky enough to be DevOps-y, then you're really just some combination
03:00
somewhere between those two. But no matter what your job is to keep infrastructure up, to deploy infrastructure, and that always has to help your business. And if we have failures, we're not helping our business. Those failures are hurting our business, and at that point, we're really just not doing our job. And unfortunately, as we move to infrastructure as code, and we move to Chef,
03:21
even though these are fantastic tools, that's really just made our failures faster and uglier. Before, when I was clicking through servers, I might see a change on the third server that caused that problem. But now I push a change out and go and get coffee, and by the time I come back, that change is already deployed to 500 servers. So the ability to roll back your change,
03:43
to see issues and the instincts that goes along with that, doesn't come into play anymore. And I think a good example of that is pre-configuration management, TIM. I made a lot of changes. 98% of them were clearly fantastic, wonderful changes.
04:00
I got some promotions, people were happy with it. 2% of them were bad, and by bad, I mean like, Johnny, drop tables, outage, horrible, horrible, website down, bad. And that seems pretty high, like looking back at it now, I'm like, that's probably about 2%. It might be bad at first glance, but I was only doing five changes a day, right?
04:22
Pointy clicky, you can't do much. So our mean time to failure as a company was very low. Our customers were happy, the website was up, the business was happy, we were making money. But we had to increase our velocity. Our competition was increasing their velocity, and we had to stay relevant in the business world. So we took on configuration management,
04:42
we brought in Chef, we increased our velocity, and instantly, you see that my bad went to 1%. This is exactly what the business wanted. They wanted us to have automation reducing risk. So we automated away risk, right? Half of it, in fact. Because it turned out a lot of what I did was just clicking on the wrong thing,
05:01
forgetting a checkbox, running the wrong command. I mean, that happens, right? When we're replicating the same thing across 50 servers by hand, we make mistakes. But unfortunately, I'm making 50 changes a day now, which means the business is not happy, the customers are not happy, because we're actually failing more. We have more outages than we've ever had before,
05:21
and we have unhappy customers, simply because that increase in velocity came along with an increase in outages. So we have to utilize testing. There's really no choice. We cannot have an increase in velocity that also increases our outages at the same proportion. We want to see our nice, beautiful, up and to the right graph, but we want a nice, flat, horrible graph for outages.
05:45
So we decided it's time to jump in, right? We're gonna start testing things. And the first thing we do is we go on Google and we search for Chef testing and this huge number of blog posts come back, things that Joshua wrote five years ago that aren't really valid at all. Somebody's telling you you should use Minitest,
06:01
and they're like, 2012. And you're like, I don't understand this. This is too many tools. And the first thing we want to do is not do this. Do not search for the tools that we're gonna use. Instead, we want to put our developer hat on, and we want to think about what we actually want in terms of testing. So we have three kinds of testing that we're gonna utilize within Chef,
06:22
within our infrastructure. There's certainly more kinds of software testing if you ever want pain, search for software testing and find some nice IEEE doc that talks about it. But these are three, and these are the three that we're gonna utilize today. And the first one is linting. And linting is when we statically analyze our code,
06:43
and in the Chef world, what we're gonna do is look for style, syntax, and logic mistakes. And when I say statically analyze, what I mean is that we're not actually executing our code. What we're effectively doing is just a giant spell check. We're going to regex for things that we think shouldn't be there.
07:00
And it sounds like, what could I possibly accomplish with that? But linting is really one of the best first things to grasp onto. We can grab a huge amount of value from just two simple tools. So the first part of that is style. And I put style in here first because when I say that I think style linting is important, a lot of people kind of scoff and chuck a little bit.
07:21
They think it's like some OCD thing where I need the proper spaces and tabs and whatnot. And it sort of is a little bit, in all honesty. But I think that style is incredibly important, and I really do believe that style within your code can prevent outages. I put this picture here as a background to illustrate the point. This is a phone trunk coming into a pedestal.
07:41
If you live in the suburbs, you have one somewhere in your neighborhood. Somebody got the unlucky stick, and that's on their lawn. And inside is something like this, right? This is a whole bunch of people's home phone lines, and it works. It is functional, but it's awful at the same time. And if a phone tech comes out to make a change in this, the first thing they have to do
08:01
is they have to grapple with everything that is wrong with this pile of cables. And they have to dig through that, and they have to understand what it is, and our code is exactly the same. And when a person makes a change here, there's a risk in dealing with this mess that they're making a change that's not what they intended to make. They might bump one of these cables,
08:20
they might knock your phone line off. God forbid you don't have a fax machine going anymore. And we don't want that. And someone is gonna inherit your code, no matter what. Even if you're in a single ops person shop or you're just a lone developer doing stuff, someone will inherit that code at some point, and they will have to read it. They will have to interpret it. And we can do things within Ruby and within Chef as well
08:41
that really are sometimes hard to interpret. So we wanna make sure that we're all on the same page in terms of style. Now the second one is syntax. And this is an incredibly important one, but really dead simple. Ruby is a language, Chef is a language. These both have rules to their language. We have to follow those rules. And if we have a missing end or a missing do in a block,
09:03
that's obviously not valid syntax anymore. And the last thing we wanna do is code up a wonderful, nice cookbook, upload it to production, and boom, all of our nodes fail. So we can avoid that super easily. We can syntax check within our linting process. We can make sure that what we have is in fact valid Ruby code.
09:21
Then we'll fail in a different way, but at least our code will run. And then the final one is logic. So this is the one I think is really fun. We have Chef the language, we have Ruby the language. We can actually go through and see what you're doing within this code. And we can give you recommendations on things that might cause you problems.
09:41
You might be just incorrectly using something, you might be using Chef in an unsafe way, or you might be using something that's gonna be deprecated at some point. We can tell you not to do that. So we can give you little nudges to push you towards better code. And we can do that all without, again, ever running that code. So how do we lint within Chef? Well, we have two tools for linting in the Chef world.
10:02
The first one is called Cookstyle. Cookstyle is a tool that we wrote at Chef. It's really a wrapper around RuboCop. RuboCop is a really popular Ruby linter. The problem with it is that it's a Ruby linter. Ruby has a different interpretation of the way some things should be than we think a cookbook should look.
10:22
Chef cookbook is not a Ruby application. So we've bundled RuboCop with a customized rule set, and we think this is a rule set that's really good for writing cookbooks. You can run this and you can get everybody on the same page. You can interpret a community cookbook where somebody is also using Cookstyle and everyone will have similar looking code.
10:43
And the cool thing within Cookstyle is that it comes with auto-correcting. So you're not just gonna get a bunch of warnings that's gonna tell you you're using bad spaces or you need to have extra commas somewhere. It's just gonna fix it for you. And 95% of these things will just be fixed automatically. So you can take your existing code base
11:01
and you can just run Cookstyle dash a against it, and boom, you're all on the same style. And you can just keep that going forward with some testing. Now, then we have Food Critic. Whereas Cookstyle is focused mostly on the styling and on the actual Ruby, Food Critic is actually written to look at what's going on within your cookbook,
11:22
within primarily the logic of the cookbook, but also the syntax of the Chef code. And we have a ton of rules here. We've got about 87, I think, now, shipped about 15 in the last month that you'll get with Chef DK2. And a lot of rules, particularly about doing things that are just gonna be broken,
11:40
incorrectly using custom resources and LWRPs, and then lots and lots of rules about new deprecations that we're introducing. So as you wanna migrate to Chef 12, to Chef 13, to Chef 14, you're gonna see warnings within your code, again, before you ever run it, before you ever install that version of Chef. You'll be able to run Food Critic and see exactly what you have to update.
12:03
And here's an example output of it. It's kinda small, obviously. There's kinda two things you'll see here. I just picked randomly a cookbook on the supermarket. This one was pretty good, because it had a LWRP in it. This is missing a bunch of metadata fields. That's gonna make it a little tricky if you wanna share it on the supermarket. These are fields that we're keying off
12:20
to fill in data there. It's also missing a license, so some people won't even be able to consume this, because you're not properly telling them what the license is. And then it has a LWRP that's simply not doing the right thing. So we're warning you that you're not using inline resources and you're using updated by last action. So this is something that you could write and it is valid Chef code and it would run,
12:40
but it might not actually do what you think it's doing. So we're gonna warn you right off the bat that your code might not give you the desired outcome. And you might look at one of these and go, what is FC74? And luckily, we have nice documentation. Foodcritic.io has a documentation section for each rule. So you can look at it, you can get a description of what it is, and you can see a little example
13:01
of the before and after code. In this case, we're just telling you that you are setting the default action wrong and there's a simpler way to do it. So that's linting. The next one I talk about is unit testing. Unit testing is when you're looking at the individual components of your cookbook and you're really just testing that given a certain set of inputs,
13:21
you'll get a certain set of outputs out. Now this is really, really important when you're writing Ruby applications. You can very easily test classes and methods. It's a little different in the Chef world. We'll talk about that a little bit. ChefSpec, though, brings RSpec testing to Ruby. It's just extensions on top of RSpec,
13:41
allows you to write unit tests like they're just standard RSpec tests and inspect what's going on within your Chef code. So this is just a simple example cookbook. I thought it was actually a Seth Vargo cookbook. Turned out it's actually a Jamie cookbook. Goes way back. And this is my face. So it's really simplified here,
14:00
but this looks a lot like a lot of the cookbooks that we have floating out in the real world. So it's just a package that gets installed, some sort of configuration that happens, and then a service that gets managed and restarted and whatnot. And if we write ChefSpec for this, it would look something like this, right? And the first thing we're gonna see is this is actually larger than the cookbook and also harder to read than the cookbook.
14:21
So I'll break down the individual sections and what is happening here at a high level. We're certainly not gonna dive into ChefSpec because it's crazy complex and hard. The first part is that we're setting up a Chef solo run and we're doing that using mocked data, right? So we might be testing this on my Mac. We're not on an Ubuntu 16.04 box,
14:40
but we need it to look like an Ubuntu 16.04 box. So we have data for each major operating system. We can mock out this Ubuntu 16.04 box, and then we can do an in-memory converge of Chef. So that, again, we're not running Chef. Nothing's happening on your local system. We're just seeing sort of what would happen when Chef runs.
15:00
And then once Chef's run, we can actually do a bunch of different tests on it. So we're gonna describe what we're testing. In the top case there, it's installing the package. And then we can say what we expect. So we expect the package was installed, right? And that's all great, except it's totally not. And I really can't stress enough here that that is the most worthless Chef spec ever.
15:23
And unfortunately, it's what most people use. Chef spec is a really great tool for a lot of people, but unfortunately, a lot of cookbooks simply don't need unit tests. And the example there is this. So we have three resources in that cookbook, and we have three very simple Chef spec tests.
15:43
And that one-to-one mapping tells me right off the bat that we don't actually have any logic within that cookbook. There's nothing to test, because there's no logic. If you write out that a package should be installed, and then you write a Chef spec that says the package was installed, you're not actually testing your cookbook at that point.
16:00
You're actually testing Chef, and we already test Chef. So you should probably just not do that. And I see a lot of people who say, I need 100% code coverage, I have to test. But Chef spec is for the logic. So test the logic. And you should test the logic that you have very well. But if your cookbook lacks logic, if it looks like that previous cookbook,
16:20
then don't test it. Just skip Chef spec entirely. Now what if our MyFace cookbook was a little bit more complex? It's not just supporting Ubuntu now. We need it to support CentOS. And CentOS might have a different package name. Well now we have logic, right? Now we have a different package that has to get installed on a different platform. So we can actually test that at this point.
16:40
And we can converge two different platforms. Now this is a real Chef spec. This will actually deliver some value to our company. If someone did refactoring later on and they messed with that case statement that we had or the if statement that we had that changed what that package name was, this will alert us that it doesn't work. But there's a second warning.
17:01
And that is that even perfect specs, even that spec before, really don't catch most of the failures that we have. And the reason is that when we're writing a cookbook, what we're doing is we're codifying our assumptions. And we're making a lot of assumptions when we first write that cookbook. So in this simple example here, the first thing is does my face the package exist?
17:23
Do we have a repository to even install that? Because if I simply test it, I tried to install the package and then my spec says that the package was installed, I'll never catch the fact that the package doesn't even exist in the repository. Is this a valid configuration even? Did a new version of my face come out
17:41
that no longer accepts some syntax I'm using? Is that service even there? Is there no longer maybe a Sysv init script that I need? Can this even start? So a Chef spec won't catch those real failures, and for that we need integration testing. And this is really the important part within Chef testing. We want to integrate all of our software together,
18:04
not just our cookbook but our dependencies, we wanna test them all together, and we wanna do that in a way that looks as much like production as possible. So that's gonna be a real Chef run, right? This run here, totally not something you can comprehend or read, but we're actually testing that cookbook.
18:20
All its dependencies are coming together, it's being compiled, it's converging. If we're trying to install a package, that package has to exist within the repository. It's a real run, right? And for that we have Test Kitchen. Test Kitchen is a super great tool. Lets us create a simple little YAML like this and converge multiple scenarios while on multiple platforms.
18:41
So in this case, what we're gonna do is we're going to set this system up, we're gonna converge on Vagrant. This could be different things if we want a different driver, so we could converge to an OpenStack or AWS or Azure, but we're gonna converge on Vagrant which is gonna run locally on my Mac with VirtualBox. We're gonna use Chef Zero as our provisioner,
19:00
that's gonna make it look as much like a Chef server as we possibly can. And then we'll talk a little bit later, but we're gonna verify using InSpec when we're done. So as far as platforms go, we're gonna converge on four different platforms here. We have CentOS and Ubuntu, recent releases. These are from our Bento project, so as new releases of free distros come out,
19:21
we're gonna constantly be delivering more Bento images. So you can always say, oh, when CentOS 7.4 comes out, I wanna test on that because that's what I use in production. There will be a 7.4 image here, you can just pull it down automatically. And then, this is the actual bread and butter here, we actually have to test something, right? So we're gonna converge two different suites here,
19:41
one is gonna be a client install of MyFace, whatever that would be, and then the other one is gonna be a server install. And the part to point out here is that we're using test recipes. And really, you wanna do yourself a favor, write a real test recipe. This gives you the real life scenario of how the cookbook actually would come together. If you have LWRPs or custom resources,
20:02
you can utilize them within a test recipe, you can execute them in different scenarios, if you have dependencies that need to be installed first, you can put those all in your test recipe and really test the way that this cookbook would get used in production. But when we put that all together, you get this, right? Simple output, we have eight different converges that we'll be doing here,
20:21
and we could run kitchen converge, that would build all these systems out, we can log into them if we need to troubleshoot them, or we could just say kitchen tests, and it would build them all and tear them down. But did it actually work, right? Because you wanna test your business, not mine. My business is converging Chef nodes, your business is whatever you do.
20:41
And for that node to simply converge doesn't mean it actually works. So for that, we need InSpec. And Kristoff talked about InSpec a lot, so he kinda got a brief description of what we can do with that from a compliance world. It's really a hugely powerful tool when we're talking about kitchen testing.
21:00
And InSpec allows us to write really, really simple RSpec tests. They actually look a lot like the ChefSpec tests, but these are running against a real node, right? We've converged this node in Test Kitchen, and now we can check it and verify it. So we know that the system is actually the way our system in production will work. And it's great because there's 82 built-in resources.
21:22
And those 82 built-in resources allow us to do all kinds of cool testing. So we can test the outcomes of our Chef resources, we can look at all the basic things like packages and directories and groups. Or we can dig a little bit deeper into system configs. We can look at grub and network bonding and Windows features and services.
21:41
And we're also building out a lot more application configuration things. And again, this is what powers our compliance product. So as you see more and more people that want to actually check the compliance of their applications, you're gonna see those resources come into InSpec. We're doing about a release a week of InSpec, and every release is coming with new resources. So as time goes on, you have more and more things,
22:02
more helpers to allow you to simply test and say, hey, can I connect to Postgres? I don't wanna have to do a shell out to that. I wanna just simply say, does Postgres work? Is this database there? So this is what one of our tests would look like. I pulled this out of our Tomcat cookbook.
22:21
It seems like it's a weird thing to test. We're testing a file, and then we're checking to see if it's a directory, if it has an owner, a right group, different mode. This was being used within a custom resource we had in the Tomcat cookbook, and we're actually executing that resource in different ways, and this test is formulated to make sure that all the different properties
22:41
that we're setting on that resource are actually working. So very simple way to make sure that what we wrote, all of our assumptions and expectations are actually coming out. And that gets us something like this. And this is confidence, really. Like this is me converging a full cookbook on a platform and saying it meets maybe 30 or 40 different things that I expect to be there.
23:02
And we wanna make sure that we're writing our tests to actually check for our desired outcomes. And what are those desired outcomes? Because this is one of the most important things, and I see people often do this in their tests, where they don't test the right thing. They've written really elegant tests. Maybe they're even using full automation. They've got Test Kitchen running, InSpec running.
23:21
But what they're testing is not delivering the value that they need. We wanna make sure that we balance both system outcomes and business outcomes. So in that first example I used there, I was testing a bunch of things like file mode and users and groups. That was making sure that my system worked. Making sure that when a person put an input on a property for a custom resource, that I got the desired output.
23:41
But my business doesn't care about who owns the Tomcat file. My business cares that Tomcat works. So we wanna make sure that not only are you writing your kind of sysadminning tests, but also the test to just make sure that this is gonna do the function it has for your business. The Tomcat server, can I get response back? Is that response a 200?
24:01
Is it valid data? If it's an API, like let's hit the API. Let's hit some endpoints. Let's make sure they're performant. Let's check the response times. These are things that actually matter to your business, because they don't care that the application is there. They don't care that the service has started. Quite frankly, you don't even need to check that the service has started. Just hit the URL. So we have all this.
24:21
We have our perfect tests. Where do we stand at this point? We've done a whole bunch of different tools to give us a slew of different kinds of testing. But how do we actually run these tools? I don't expect you to sit down and type cook style every time and food critic every time. We need a nice way to bring this all together. And we need a nice way to automatically do it
24:41
for all of our changes. So the first thing I'm gonna talk about is delivery local. This is part of Chef DK. This is something that our delivery team actually wrote for our cookbook engineering team, because we had exactly this problem. How do I run all my tests over and over again, and how do I do them across all my infrastructure?
25:02
Well, delivery local gives us the ability to write this really simple toml file here. It defines a bunch of phases. It's very similar if you've used automate to what we have in automate, but it actually runs entirely locally. You're not using cookbooks for this. You're just using a simple toml file. And in these phases, you just define the command that you'd wanna run.
25:21
So we say our linting phase is gonna be cook style. Our syntax phase will be food critic. If we don't wanna run the phase, we just hit skip. But you probably have more than one cookbook. I really hope that your business is not one cookbook. So managing these files becomes really burdensome. What if there's a food critic check that you don't think is important to you? Well, you can turn that off in the command line.
25:42
But do you wanna turn that off in, say, 100 cookbooks? Because that's what I used to have to do, and it was really a pain to be shuffling around things like rake files or food critic configs. Well, the really neat thing you get with the delivery local mode is you can delete that entire file and you can just point it out a URL. And then you can put the contents
26:00
of what you want the file to look like in that page. So in the cookbook world, in community cookbooks, we have a single gist that is just what we want it to look like. And any time I run this locally or any time I run it in a CI system or any time anyone runs it, all my other developers in the community, they're always getting the latest updates. So if I wanna change the way people test that cookbook,
26:22
I just do it in the remote file and everyone gets it every time they run. And then if I run it locally, like, this is it. It's delivery local all, and the all is that I'm running all those phases. If I wanted to run just the syntax, delivery local syntax, but I can just deliver local all and get a nice little output of all the tests.
26:41
But I obviously wanna run that automatically, right? I'm not expecting my developers to run this locally on their system every time because they, quite frankly, they won't. Someone will skip it at some point, you'll forget, and that'll be the time that you'll introduce a breaking change. So we need a CI system, right? And we have one.
27:00
You can buy it. It's great, you should get it. I have a mortgage, it needs to be paid. But what if you don't have automate? Because maybe you're open source only, maybe you're not at that scale for an automate server. Not everybody necessarily needs that infrastructure. Well, we can definitely make this work. Becomes a lot harder, but we can do it.
27:22
And we can bring it in Travis CI, but unfortunately, as soon as we get into a SAS CI system we're instantly losing a lot of control, right? We're not running this in automate, we're not running it in Jenkins, we don't have control over what's happening on our builders, and there's a lot of security concerns for that CI system. So they're not gonna let us run everything we wanna run.
27:40
We can't just kitchen vagrant. So instead what we're gonna have to do is we're gonna have to run kitchen docking. Now kitchen docking is something that Sean O'Mara wrote when he was on our community cookbook team, and he wrote this so that we could actually run test kitchen, not just quickly for him locally, but also somewhere where we could run it in Travis. And it's just another driver for test kitchen,
28:03
so it looks a lot like we had with vagrant, but instead of it being vagrant it's basically docking, docking, docking. It controls the whole thing. It actually takes over for Chef. It doesn't even use the full Chef client install. It uses its own container of Chef, which makes that even faster.
28:21
But that lets us do some great things with wonderful, wonderful warnings, and that is that now we're testing in Docker, and Docker containers kinda lie to us. When we're testing on that full Ubuntu VM, that's a 480 meg image, and that is a compressed vagrant box, right? That's not just an uncompressed horrible thing.
28:42
That's 480 megs of packages and configs and everything it takes to be in a Ubuntu system. But now we go to that same thing in a container, and it's 47 megs. So what's the difference? It's not magical Docker compression. That's missing a lot of things, right? That Docker container is not meant to be a VM. It's not a stand-in for a system.
29:01
It was meant for us to run our Node app or our Ruby app or something on. So we need to do a bit of work to kind of hack those containers to look like they're actually VMs. And this is dirty, dirty stuff, but it does work. So what we're gonna do is we're gonna install a whole bunch of stuff, right? We need init systems, we need packet managers,
29:22
we need network tools, process tools, we need a whole bunch of things, and they look different on every platform and every version of that platform. And thankfully, we've already done that. So we have a nice example repo here. I'll show it again at the end. And that example repo has an example of how we set up kitchen docking and all the things we need for it to actually
29:41
be able to start a systemd service. So within that kitchen docking config, again, I said it looks a lot like the kitchen vagrant config, but obviously now we're configuring a container. So what we're doing, just a simple example here, we're setting the CentOS 6 image. We're gonna grab that from Docker Hub.
30:01
We're setting the PID 1 command, and that's basically allowing us to actually start a service and have it run using init. And then the intermediate instruction here is to install a bunch of packages. And that's what I was talking about. We have to make this thing look a little bit more like a real CentOS system. In order to do that, we need simple things like net tools because otherwise we can't check if a port's open.
30:22
But once we have that all, once we put that all together and we hack these systems to look like it, we have really nice, fast kitchen runs. Fast kitchen runs that could be on your desktop, but also fast kitchen runs that could be in Travis. So how does that look in Travis? Well, again, it's a nice, huge config that you should totally just copy from us.
30:40
And there's a few different things that are going on in there, and I'll quickly go over them. Basically, we install ChefDK within Travis. ChefDK is an approved repository with Travis, so you can install it right in your Chef run or right in your Travis run, and that means that you're not testing with gems anymore. You're actually testing the same way you would test on your workstation, and the setup is gonna be the same
31:02
as when you go into production with Chef. We're gonna turn on Docker, because obviously we need Docker. We're going to define each of the different systems that we're gonna run. So in this case, I'm just gonna converge four different instances. It's only one suite on four different platforms. We're gonna do some boring stuff to make sure that IP tables is set up and that all the ChefDK binary is available for us.
31:23
Then we're going to run it all. And then the last part there is you see we inject one more job, and that's because we want to split out all of our non-Test Kitchen stuff into its own job in Travis. And that means that if somebody accidentally puts an extra space in
31:40
and now Cookstyle fails, it doesn't fail every single one of our things. It makes it much more apparent how it failed. And when we put that all together, we get something like this. And I cannot stress how awesome it is to have these come in. This is a PR that someone opened against OpenSSH. They want to make a simple config change,
32:02
but I don't know if that config change introduces breaking problems across other platforms. It might, who knows. So we're gonna converge it on those platforms, right? We're gonna converge it on Ubuntu and Debian and CentOS and even OpenSUSE. And then we're also gonna run all of our Cookstyle and FoodCrick and ChefSpec unit linting tests.
32:21
And that entirety of that testing ran in five minutes. So in five minutes, I have confidence that this is gonna work. And that's really just confidence that I'm not gonna burn my infrastructure to the ground. And as the person writing community cookbooks, I'm not gonna burn your infrastructure to the ground. Because I really hope that doesn't happen. But that also, very importantly,
32:42
just brings value to our business. And at the end of the day, that's what this is actually about, right? We want to be able to know that if we take a simple configuration change that we think of, we can make that, we can take it through a series of testing, we can deploy it with confidence, and we can actually achieve what our business wants.
33:02
So go home, test your infrastructure. I just ask that you don't write bad ChefSpec tests. That's my only ask here. And this is the example repository. There's a ton of great stuff in there. And you can kind of look also at all of our various community cookbooks. They do the same thing. We're testing everything in Travis. And yeah, good luck.
33:32
Does anybody have any questions? Just real quick, what's a test cookbook? You mentioned that before. Yeah, so you can have, if you do, in ChefDK, if you just say Chef generate cookbook,
33:43
you'll get your base cookbook. It's gonna create a recipes folder and attributes folder. It's also gonna create another cookbook within there that's just for testing. So particularly as you start to move to writing resources instead of actual recipes, you need to make sure that that resource actually works, right?
34:01
So in the case of the Tomcat cookbook that we ship, there's no recipe for Tomcat. We just ship you three resources to work with Tomcat. So we need to be able to test that. So we have to create a recipe that can actually go through and set a Tomcat server up. That recipe does an apt update, installs Java, pulls down a war file after Tomcat setup, starts it.
34:21
We can make sure that that's a real site. That's the way somebody would use it. Because not all of our recipes or cookbooks are things that we would directly apply to a node. They might be a cookbook that we are consuming from another cookbook. So we still wanna make sure we test that. And we can kind of fake it out with a full test suite. It also allows you, like in the Tomcat cookbook,
34:40
we're setting it up like three or four different ways. I think we set up like three different Tomcat sites that kind of utilize our resources in different ways and makes it really easy to just converge that all at once test it, take it through a bunch of InSpec tests and make sure everything looks good. Is there a place I can find good examples of ChefSpec tests?
35:01
And are there some cookbooks where I just wouldn't wanna have any tests because they're not needed? There's not a definitive good example of good ChefSpec tests. I think the problem is understanding when you need to use unit tests is kind of tricky. And as soon as I said like don't write ChefSpec,
35:21
all these people were like, but I have this scenario. And like we use them extensively in community cookbooks because we have a lot of logic. Like we're supporting sometimes 15 different platforms and version combinations. So you can get just a simple thing of like what's the name of the package? And it can be a case statement with like nested ifs. And if somebody makes a tiny little change to that, they break the entire thing.
35:40
So it's like that scenario is fantastic for ChefSpec. Off the top of my head, I can't think of some of our cookbooks that have it set up correctly. There certainly are some that are tested very well with ChefSpec where somebody is going through the logic and they're not just copying one to one. The biggest thing I would say just in ChefSpec
36:00
to keep yourself from doing the wrong thing is don't turn on the post-converged report. A lot of people see this and it says, oh you have 20% code coverage. I want 100% code coverage. The only way you get 100% code coverage is just to copy every single resource and say, I'm testing that this directory was created or this package was installed. That's when you really get yourself into writing
36:22
ChefSpec that's just not, it doesn't bring value. It's not gonna hurt you. But there's probably something better in terms of testing you can do with your time. So you mentioned Bento boxes and then with docking, you have to put a few packages on there. Might there be Bento containers in the future? Yeah, Bento containers are definitely something we've thought about.
36:42
We've considered what we could do to actually make basically like fat containers. So instead of shipping you that 47 meg Ubuntu straight from Docker Hub container, we would have an actual Chef slash Ubuntu 16.04 container that was maybe 150 megs.
37:03
Which would actually be a lot faster too because you wouldn't have to do the installs. The problem with that is that's us committing to updating those constantly. And the Bento stuff is already a ton of work. We're cutting out the Bento images for VirtualBox, Parallels, and VMware.
37:21
And getting that all to work, getting those three different VM technologies to all converge on 20 platforms and versions is really hard. There's constant regressions that happen in VMware tools or the VirtualBox tools. And poking Travis, or not Travis, poking Packard to make that all work is hard.
37:42
But I really would like to do that. It's just a matter of having resources. So based on your presentation, what is the point of writing a ChefSpec test as opposed to an InSpec test? You mentioned ChefSpec basically tests to see if your Chef code works,
38:02
but that already, Convergence already does that. So I guess I'm just trying to see the difference. To me it makes much more sense to do integration testing, but I'm just trying to see what the point of ChefSpec would be. Really about the time for your feedback loop. So you running that Convergence,
38:22
even in Travis, that took five minutes, right? But that wouldn't take five minutes on my laptop. And if I was doing that with Vagrant and not with kitchen docking, if I actually wanted to get a full Convergence of a real system in Vagrant, I could be spending a long time. And those are simple examples of packages, like that's five minutes to run OpenSSH
38:41
where SSH-D and a config is dropped. But what if I'm doing something that is a very complex system setup? I could be testing that for like three or four hours. Whereas I can run ChefSpec, and in a minute get back a whole bunch of Convergence. I can set attributes differently. I can go through a bunch of different scenarios,
39:00
different versions of, say, three Ubuntu versions and three CentOS versions with different attributes set here and there and different recipes used. And you could have hundreds of different permutations of what that test looks like that still runs in two or three minutes.
39:22
As far as I remember, maybe I'm wrong, but Test Kitchen allows you to do tests one by one. Is there some way to deploy several instances at the same time if you have clusterization or replication or something like that?
39:43
So we don't have the concept of cluster testing in Test Kitchen. That's something that comes up a lot, and it adds a huge amount of complexity because you kind of get into Test Kitchen needing to be an orchestrator, which is really beyond the scope of Test Kitchen. We thought about how we could do it in a simple way,
40:00
and the problem is as soon as we introduce something that's the simple way, people are gonna want all the different versions that become the complex way, and then we've built this massive orchestration testing solution. But you can, if you want to run more than one test, the same test, but more than one at a time, like in the scenario that I had
40:20
where we had eight different total tests there, you could concurrent that. And that's pretty handy if you're testing against EC2 or something. You can spin up eight boxes at the same time. All right, cool. If anybody else has any questions, hit me up. I'm around. Definitely always on community Slack too.