Beyond the Cookbook: Using Workflow to Bring Continuous Delivery to And Project

Video in TIB AV-Portal: Beyond the Cookbook: Using Workflow to Bring Continuous Delivery to And Project

Formal Metadata

Beyond the Cookbook: Using Workflow to Bring Continuous Delivery to And Project
Title of Series
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
While we all know workflow provides an easy way to do continuous integration/delivery for cookbooks, we also all have other parts that need to be developed and maintained for a successful DevOps environment. This talk will provide information on how to extend the power of delivery to those other projects, and provide the basics needed to understand creating build cookbooks well enough to create one for any project. Topics covered: Understanding the build cookbook, what the phases of the build cookbook are, and how workflow uses them. Using dependencies to tie together both cookbook and non-cookbook items. Demo: Using workflow to build a small web application including building the Web server from source, and delivering to an end environment.
Arithmetic mean Presentation of a group Lie group Programmable read-only memory Independence (probability theory) Information Continuous function Information security Thomas Kuhn Spacetime
Standard deviation Default (computer science) Building Key (cryptography) INTEGRAL Code Continuous integration Building View (database) Disintegration Mereology Directory service Mereology Subset Word Integrated development environment Phase transition Phase transition Formal verification Normal (geometry) Automation Normal (geometry) Analytic continuation Default (computer science)
Mathematics Phase transition Phase transition Moment (mathematics) Formal verification Software testing Product (business)
Revision control Execution unit Integrated development environment Code Phase transition Execution unit Phase transition Software testing Software testing Error message Rule of inference Number
Point (geometry) Server (computing) Building State of matter Code Server (computing) Code Instance (computer science) Directory service Subset Revision control Mathematics Error message Repository (publishing) Computer configuration Phase transition Formal verification Software testing Local ring Information security
Functional (mathematics) Codebuch Client (computing) Instance (computer science) Process (computing) Integrated development environment Different (Kate Ryan album) Phase transition Phase transition Integrated development environment Software testing File viewer Functional (mathematics) Library (computing)
Building Standard deviation Functional (mathematics) State of matter Building Execution unit Sheaf (mathematics) Instance (computer science) Directory service Limit (category theory) Subset Power (physics) Revision control Tablet computer Integrated development environment Root Phase transition Phase transition Revision control Statement (computer science) Vertex (graph theory)
Building Electric generator Code Building Source code Directory service Mereology Subset Product (business) Integrated development environment Phase transition Formal verification Software testing Determinant
Execution unit Building Computer file Code Execution unit Demoscene Goodness of fit Error message Integrated development environment Software repository Compiler Phase transition Phase transition Software repository Configuration space Software testing Configuration space Vertex (graph theory)
Computer file Multiplication sign Source code Execution unit Electronic mailing list Set (mathematics) Branch (computer science) Bit Function (mathematics) Demoscene Number Revision control Mathematics Computer configuration Network topology Royal Navy Phase transition Configuration space Utility software
Building Group action Presentation of a group Computer file State of matter Structural load Moment (mathematics) Execution unit Login Demoscene Mathematics Repository (publishing) Computer configuration Phase transition Configuration space Utility software Compilation album
Point (geometry) Server (computing) Building Distribution (mathematics) Server (computing) Multiplication sign Number Web 2.0 Integrated development environment Compiler Phase transition Phase transition Software testing Information security Information security Resultant
Installation art Execution unit Building Hoax Information Root Key (cryptography) Repository (publishing) Set (mathematics) Convex hull Directory service
Server (computing) Code Set (mathematics) Attribute grammar Client (computing) Variable (mathematics) Subset Web 2.0 Integrated development environment Military operation Phase transition Phase transition Revision control Set (mathematics) Configuration space Integrated development environment Software testing Software testing Functional (mathematics) Local ring Dean number Reverse engineering Default (computer science)
Point (geometry) Server (computing) Functional (mathematics) Computer file Code Source code Maxima and minima Set (mathematics) Function (mathematics) Mereology Number Subset Goodness of fit Insertion loss Term (mathematics) Software testing Drum memory Error message Execution unit Standard deviation Instant Messaging Menu (computing) Bit Directory service Type theory Integrated development environment Video game Configuration space Library (computing)
Root Computer file Block (periodic table) Configuration space Convex hull Local ring Subset
Point (geometry) NP-hard Variety (linguistics) Multiplication sign Source code Execution unit MIDI 1 (number) Set (mathematics) Water vapor Function (mathematics) Mass Mereology Demoscene Field (computer science) Attribute grammar Hardware-in-the-loop simulation Software testing Default (computer science) Execution unit Wrapper (data mining) Block (periodic table) Instance (computer science) Demoscene Type theory Uniform resource locator Integrated development environment Personal digital assistant Phase transition Order (biology) Configuration space Convex hull Library (computing)
Area Default (computer science) Addition Shift operator Building Electric generator Block (periodic table) Multiplication sign Execution unit Codebuch Directory service Instance (computer science) Metadata Number Revision control Frequency Latent heat Phase transition Software testing Error message Form (programming)
Point (geometry) Building Presentation of a group Randomization Group action Divisor Computer file Code Multiplication sign Streaming media Mereology Number Revision control Medical imaging Roundness (object) Different (Kate Ryan album) Natural number Operator (mathematics) Formal verification Gastropod shell Automation Software testing Form (programming) Area Scripting language Standard deviation Forcing (mathematics) Software developer Fitness function Entire function Demoscene Data mining Film editing Process (computing) Integrated development environment Repository (publishing) Factory (trading post) Configuration space Quicksort Writing
Kelly welcome begin to be on the Cookbook using chef workflow to bring this receive project I'm J. Robinson I made DevOps architect with as a
pianist who I'm I'm sure most of you were here saw commons presentation this morning and mean take away here is
that we are a 100 % independent subsidiary of SAP they caters exclusively within the Federal Space for US government and things so and many of you here no
itself workflow is never used it before that some of you for any of you know it's in tinuous integration Continuous delivery pipeline it basically is turnkey solution for if you have chef cookbooks and want to go over them and a ICT pipeline very easily part of chiffon a mate and it actually those of build could be uses the build cookbook so you can point things in a standardized automated way using all the syntax you already know where it's a cookbook and the 1st place but only deploys cookbooks right now you can use you build cookbook induced by things so the bill could tells workflow how to deploy your code delivery trucks the standard try the example we need you shut generate cookbook it by default will start a new project for you with the dot delivery directory any build cookbook that just goes through and calls delivery truck Torrance various phases as I said before normal chef resources and can do just about anything in normal traffic cookbook key so most of you
views workflows for are probably familiar with these phases basically goes through verify build acceptance union worst 1 delivered the idea here is also that verify happens before you merge and there's an approval date for merging it before builds the building is where most CI CD pipelines would say disposed to build your artifacts In this starts out to you acceptance word starts deploying and testing your artifacts once he passed acceptance you have another gate deliver into your production environment passing through union rehearsal and deliver delivered being ultimately your production environment so we can take those and
kind of separate them into 3 distinct
places pre-merge pre-merge everything that happens from the moment you submit your pull request and say this is a change that I want to do post merger is what happens once that has been approved and merged into your master trading and unemployment is the phases that are purely for deploying and production so they deploying they test and that's all they really do verify phase
and we go through what delivery truck does with it it runs the Auspex spec unit tests it as a went check with rule copper cook style make sure that your code is good in passing no errors there and there's a final syntax check with my cookbook test and also checks to make sure that you've updating your version number in the cookbook because delivery works by pinning the various versions within the environment so that delivers to so if you haven't updated your version there's nothing there liver then moves on into the build
phase which has the
verify state again it runs all 3 of those verify stages again post-merged to make sure for instance if you submitted your change in somebody else came along and submitted their change they you having conflicts or anything that breaks the build after that point the quality and security but then delivery truck these are left intentionally bright blank because the idea here is that you should write your own quality and security tests publish publishes the code according to that in Fig JASigning about delivery directory with delivery truck there several options including your local chefs server your local supermarket server or even the global supermarket server gets repository
once it passes through building goes to acceptance the idea here being acceptances
your 1st day of test environment so cookbooks inappropriate environment the it attempts to find anything that has this cookbook assigned to it In deploys it via push jobs to make sure that you have chef-client run isn't it it's deployed out of an environment the smoke and functional stages are also left intentionally blank again so you the idea being you should write your own smoke test for each year projects where on functional test the truth is this phase of these phases get repeated throughout the Union rehearsal and delivered the they're exactly the same going through each of them the only difference is the union also runs these free things depends on this project so this is useful for instance if you got a library cookbook or when developing your own build codebooks if you got something has libraries for instance you can actually tell it there the main project depends on those library projects it will actually rerun with the new libraries to make sure that everything works so which these phases is
really important the the viewer right around build codebook will which of these phases where I wanna start what I wanna do it that's
really up to you you should decide where things should happen within your project if you've got a project for instance that you need to run kitchen for you merge then you should put that obviously into the verify section either in unit winter syntax if you've got projects showing the published after a major update then you should probably check in the Acceptance phase In fail acceptance to make if you haven't done a major version of data minor version of day to make sure that it does not go through to your production environment basically you can also do things for instance if you wanted to say well I wanted not have any dates whatsoever I want to you basically have its everything goes through so somebody submits it you could technically do that by putting everything in the verify stage is probably a bad practice to do it however and the way you're building basically you need just the standard delivery truck for building your cookbooks you need something a little more power or you need to build something entirely different your own vehicle
so some things to keep in mind as you do your own build cookbooks the phases or a flexible you can put whatever you want into any fix so for instance if you want to publish In the provision face you could do that it is they keep in mind what happens in merge versus the post barge and then what happens in each of those deployment faces keeping the phase is close to what they're named for the function will help you later if you come back your project to go where provide but that again and I don't remember some limits on the cookbook it's worth keeping in mind the cookbooks for building are on as a nonprivileged users on your bill notes so that means for instance if you need to install package is to do whatever built you going to do you're probably going to have to provide the user with elevated privileges to do so to do do whatever you use so if you need to use a compile something else entirely as root here after provide it with the ability to do so on the other thing is that the phase each of those phases each of those sections runs on different bill notes so if you have 10 build those you're not likely to end up on the same build them for the next phase and even if you do each of them is isolated into its own directory on the build up so you can't depend that the state immediately after you finish 1 phase is going to be the statement you enter the next phase with it completely checks out a project again so that has a clean and it's a clean slate don't expect it to a member state at all
I'm self building the build cookbook where should I start I've got my project I've got my source code that I want to make it build cookbook for this is a chef cook book and ownership just do a chef generate cookbook ship generate build cookbook you will actually add e . delivery directory with a completely empty cookbook it sets out the recipes named for the phases which is required but each of those phases will be empty won't include delivery truck automatically that I figure out what goes where same way you would building any automation project determine what your build steps or are determine what you need to do pre-merge what you need to actually build and package it for deployment in that how you want to actually do that deployment
basically keep these in mind anything that you want to do before that code is merged into your code base you wanna do in the verify stage anything that you want to do from building your project you probably wanna put into the build phase acceptance is kind of a test run of 2 deployment so it's really a deployment phase but because of that gave there kind of grouped into post March the the next part of it is the deployment phases hello my deploying this into production how my deploying this anti-union play this into my stage environment rehearsal and had deployed alternately into the production environment so I'm going to do a
demo I've got a project build Apache it is basically builds Apache from the Apache git repo based on a configuration file that tells it how I want to build it packages it for deployment in the place it 2 nodes within my demo environment the code for this can be found at my get home which is get about comes less streamlined under the build Apache project so that will be available so what am I doing with
it from my from my hand a very unit phase that will actually a test compile to make sure the before emerges it at least compiles it doesn't compile obviously there's no reason for me the birds that in a it's not good code the windfall actually check my configuration file that I set up 40 years and rejects anything that doesn't have a good configuration file the syntax I've actually set up to skip entirely for right now the reason for that is that configuring the Ulster run through before I merged it so I just wanna show you guys
the cookbook here this is what the
configuration file looks like it will actually go ahead and make a change here to bump it bill number so that will build a new version of it I've got here my required packages which is just literally a list of the API required for to build because a cookbook installs us where to find the Apache source right now I this set is put at the main apache tree you could fought this appointed put into your untrained AP on a pure util sensors are also required to build Apache where they did that source and I've also got a option down here to reduce the stadium as it's building and basically not output anything from the make if I wanted to so and when to make yeah can actually go ahead and remember to branch 1st some agreed branch live demo just like you normally would your projects going to save my changes for that I'm also going to come in here in update to skip the unit phase as well and that's just because compiling Apache takes a bit of time and if I do not come if I do not skip that phase will be sitting here for 5 minutes just watching you compile so the money go ahead and commit that but the and tell it to deliver a review the that so will
automatically pop up to the new change and would start running through the phases which right now it's going to do it go ahead and prove that let it go on to the build phase log over what's actually
doing so as you can see in my unit recipe here I'm basically just including the set up build in perform build the actions that I have here set up build actually walks through loads the configuration file installs the required packages and then sets up checking out each of those git repositories for Apache in its required utilities and then runs the build comps so that it's ready in in a state that I can actually start doing the build for so so that the other cops perform build actually walks through things up anything that maybe there left over for any reason In the runs . such configure with whatever options as specified in my configuration file that red in it then just runs make so that's the that's how it actually performs the build so come back to the presentation from moment to
check the build face verify would actually go through and do another compile post-merger again to make sure nothing has changed since the 1st time we did it the insecurity I've also let intentionally blank I don't have any particular quality a security tests I want to run within this them publish it does the full build so this is now a bill number 3 this time it will actually take the result that build package it up and publish it to a web server that I have in my environment so let's see where this is
reached this point within the build phase is actually running the published now and you can see
here is actually running through spamming up to compile so while it does that so runs again performed build set
up the other thing that will do now is actually the package build recipe that I've set up which will pull from a data bank get some information like I've got in SSH key that's using 2 you copy the file over and publish it does the full build installs it to a fake root directory to install built locally and then just basically does a toric to top that build file and push out it will then go in copy over to the appropriate place on the deployment node for the repository
but said this takes a while has a lot of spam to it um so I'm gonna
go and move on and for the what it does and next faces so again acceptance union reversal delivered all do the same thing it will actually set an environment variable within the appropriate environment for here's where on that web server you need to go to find that new build that we just created the new package it will and do as the smoke phase it will deploy the package locally from the package to make sure that the environment variable setting went OK that the package actually will extract OK and it's doing this locally on the build itself In this starts that Apache up does a GET request just a should get to the local Apache Server and then the functional test will actually going and deploy a node within that environment that is running cookbook to the that will deploy the code for it the deploy code in the custom configuration and then also the build will send a GET request to that server to check and see if it's operational so just
go over there is a little bit more detail so the smoke test actually comes in goes to this effort server which this I with server configure that keeps doing is actually set up in a cookbook called delivery sugar if any of you are familiar with that 1 delivery sugar has a lot of things that when you're writing your own custom built cookbooks will actually make your life a lot easier because it will actually provides a good set of functions for you to work with for working with your actual life Chef Server as opposed to just doing things locally on the build itself so it actually will come in here and figure out what the acceptance name is which is also within delivery sugar the In I'm actually using a helper from delivery truck to which free of that environment back from the chef server so that I can work with it later so I've also got my build room here I'm sorry this is not provision that provision face so I'm doing the same type of thing here with the the create better create the environment and then here's where I'm actually going through in setting the note attributes so I set up the custom Apache attributes for the URL where the dark roof for that it should be and what Bill number it actually is on and it saves that environment so back to smoke test the smoke test like I said creates the directory for pulls back the remote file and then just extract and as you can see this is all the same type of code in same resources you familiar with from writing your own cookbooks anything you're already alternating you can do in here the same type of white it then starts Apache locally with the appropriate environment from libraries sends an HTTP request again with the standard Jeffrey source and as long as all that's correct shuts down the Apache Server very simple thing for to do here so the the last part of that would be the functional test does much of the same thing with going in with server configure pulls back day the bag actually does a node search here same standard way you would do a node search many of your other recipes search for whichever terms you want here recipe custom Apache chef environment whatever environment were currently in for delivery so if that's your acceptance environment everything project it's own acceptance environment if it's union Kullback union as its union environments and then I will find each of those nodes it will then go and use knife ssh here to reach out to those nodes In just run chef-client here to say yes can you not see the full command here on doing is executing chat clients and going through with my customers with my user get into that once it's run chef-client once all those run chef-client then i'll come through for each of those nodes it'll send the test request to the poor there I've specified for to listen on 80 80 years it is too simple GET request that passes the bill passes if any of this fails if any of those nodes fail to deploy it for any reason or anything like that then obviously the C. pipeline fails in it starts at that point go back fix your errors continuous you would normally it so let's go back and check and see how that
is doing this was still back in the publisher it here it's a large output for this is where actually packaging and up here tha creates interfile plays a and then moves
on automatically into acceptance
provision actually vote there's review block for setting the current environment human
in same thing about deployed I'm not actually doing anything in the boy face smoke it actually
plays out with the custom configuration file here so you can see secret the file create the root file extracted creatively configured file for use locally and get request up to local host ch
then the functional test it's just basically running through send it off to knife ssh there's a whole lot of output from that but completes so that point I can do the same I can basically of say deliberate and
it's going to go through and repeat the
same phases for each of the environments that's basically all there is to it you just using whatever chef resource you already know to tell it here's how I want you to build make configure very simple the dual any other thing else so you have to do you can easily do it through here just keeping in mind it runs on the bill node in has some its any questions for me at this point so what to on the in order do that that is a hard set water and actually what it does is it runs all 3 of those phases of same time for units went and syntax this 3 phases all run at same time the all wants the same time if any of them failed you obviously can't move on from verify when it runs in the build phase it also will launch all 3 of those verify faces again at the same time before moves on the quality or publisher anything else but once it moves on from that each of those phases is executed 1 by 1 the the goal of you in this the and if you have cookbooks and depend on each other and even if they're not library type of books like you've got a wrapper cookbook the depends on another cookbook it you're essentially using it as a library in that case even though it's not the chef library so in that case you would want the wrapper cookbook to run with the attributes your setting so that when you push it into union rehearsal delivered you're not just testing OK this cookbook with the default attributes works make sense I this could be a and all the within the . live ReConFig we're defining the build cookbook you can pull that from kid you can pull that I believe you can pull that from the supermarket as well so he can be pulled from a variety of sources of the worst case scenario you can specify a get slash slash URL for the mass of the other questions I'm sorry how many builders at points after this pipeline this is just a demo so it's only running the 1 pipeline I in my production environments I here we have something like 100 projects in there some of them have more than 1 pipelined for instance we seperate out say dev environment pipeline from our QA pipeline so we have quite a few so on so that that you know what it is what it would and so let me see here we go back the time here a the so the question was how do I determine what caused aýý I what caused part of it to fail basically so let me come back to 1 of my older ones in is 1 that field for instance on smoke test when I was writing it originally um I was uh up late writing it in and for some reason but I need to put the whole thing into a Ruby block on which obviously did not work so well when it fails is
just like a chef run failing it will give you a trace back and it will tell you here's where I failed in 1 had failed so you can see it was processing this Ruby what that had set up undefined method directory for ship resource Reebok is obviously a shift resources not Ruby command and it will tell you where he was actually failing so it outputs yeah really block smoke test here's in the Cookbook or I need to look at what it failed and why it failed just like you would for any of your chapter on far and if you ever if you use this to use a set up a cookbook delivery which is the example if you forget to bump that metadata it'll very quickly fill out on the syntax check and it will tell you explicitly in the area that raises median bump the version number you can do that with your own phases as well where if it fails you can you know the specifics of the Ruby block for instance you can raise your own error message or along your own error message on certain failures so that's so just like you would in the shuffle the they have questions the I have 1 so they need to ship on it and I noticed that on the pipeline you every step you have you you recipe for did you specify that the default recipe in a cookbook audio or was did out it automatic pick that up which rest period asking this is in your could working shaft and you're in your built it but you always have you have a default recipe does include all the other recipes you go down the
pipeline addition on it on it and I think that this is my recipe of of stuff on as you can see it's fairly blank on you notice that each of the phases here with in the build codebook In this is set automatically when you do a chef generate build cookbook on each of these phases are named for what the phase actually it's for instance functional when it runs functional it runs the functional recipe where runs unit it runs the unit recipe of the only place I'm doing includes here or for things like the year recipe where I'm actually including my to my own custom recipes CSI divided the building to set up build and form building package build I'm actually including those recipes in if I didn't the only thing that we get wrong here is whatever is in units or whatever is in the next phase smoke published I'm I'm I'm really new to this time trying understand of how much of this so these kinds of book book sir publicly available become chemist you typically have to have to tweak to get run of that is what you're doing if you're doing purely cookbooks then delivery truck
is publicly available as you can see I have it pulled up here a reference to a few times but delivery truck is basically the standard shaft of workflow cookbook have it used to be called chef delivery of the test my on and delivery truck was basically the standard pipeline for move your cookbook for this will deliver cookbooks into environments that will can them for you it makes it very easy to take this is my cookbook I want to delivered into a production environment had pinned so I'm not worried about well diopter diversion or nite which version may actually pulling you will actually take care of that for you but for writing other things that's part of why I'm doing this presentation is to show number 1 that yes workflow is very much able to him but whatever you wanna throw at it you do have to write some code for I like is that mine is available on my git hub to add to idea of slash stream you can take that and modify it as you wish to you know support whatever kind of build you're doing especially if it's a follows the auto Conf configure make make install type of build and if you have something more complex you may have to go ahead and write your own complete code for it on the the if you guys are doing projects I'd encourage you to do similar to play with the supermarket clear cut out there for build cookbooks and let it be out there because we can all point at different build cookbooks for different things um but if you guys say open-source projects so you as a working on and writing a cookbook to compile it deployed put it out there I
have a question regarding the and Canada the early pipeline with that the developers are among the operation side the and the developers we have that push all the artifacts into our factory on so when they go from snapshot or to release can that that release Bush and in our factor can that trigger the action of automate peace and actually start into that verification to go out into the the deployment texts so basically right now the review on deliveries kicked off by that CLI Command delivery review delivery review will actually take whatever the current state of the git repository In push it up to delivery the workflow pushed up to workflow in in that stock ticks off the pipeline at the verify stage yet so now that you work this you know obviously more than a lot of this is the like a natural fit for a good idea for building Apache or building the images or building random thing of using sort of more traditional tools is is is a matter for the force that I think it's a natural fit but partly because I came into things from the very much chef background where I'm writing a lot of cookbooks and things that nature if you're having tools in place the other tools and place in this can replace them but I can understand that trying to get people to change their ways of even trying to get people to adjust to say using chef can be kind of a pain as that everybody's kind of pull those teeth at some point I'm sure on but it is a natural fit if you're writing cookbooks all ready to go ahead and write the build cookbook and use that as the pipeline in to me it makes more sense than using other things where I have to worry about managing say shell scripts or managing you know various different kinds can fake files are you no shaft I can use a very naturally on by that same token as far as a natural fit for your build process I think I think this can build sparse anything you want to of just a day at the partner day there was actually a demonstration of using it they're using it to build an entire demo environment some build a imize with Packer in the play them with an AWS so only it's a very powerful tools and do you have any it expands on note anyway that's making any build cookbooks but with terraformed I need to not know for form um I don't use her terraform specifically I you chef provisioning so I'm a little weak in that area maybe over an area has any expensive but if you could come from that just but it took thanks anyone else OK let's get a really big a round of applause Hall