Strong Practices for Rails Applications Continuous Delivery
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 51 | |
| Number of Parts | 89 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/31562 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201651 / 89
1
2
3
5
7
8
11
12
19
22
28
29
30
31
33
45
47
50
52
53
58
62
67
71
74
76
77
78
79
80
81
84
85
86
88
89
00:00
Type theoryBitSoftwareStack (abstract data type)Dynamical systemMathematicsControl flowComputer hardwareSlide ruleSoftware developerCurveBit rateMultiplication signStapeldateiPhysical systemOperator (mathematics)Well-formed formulaCollaborationismGraph (mathematics)BuildingMobile appProduct (business)Process (computing)VelocityRule of inferenceWeb applicationAnalytic continuationEquivalence relationWeb 2.0Standard deviationMoving averageFrequencyLevel (video gaming)WordSelf-organizationInformation securityWebsiteSystem administratorResultantInstance (computer science)Regulator geneDifferent (Kate Ryan album)Group actionGaussian eliminationCodeCartesian coordinate systemMereologyAutomationQuicksortExecution unitData structurePerspective (visual)Shift operatorDialectState of matterPattern languageComputer animation
08:40
FrequencyStudent's t-testCellular automatonFactory (trading post)Term (mathematics)INTEGRALFormal verificationPhysical systemCategory of beingStapeldateiSoftware testingLevel (video gaming)Goodness of fitSynchronizationOrder (biology)Branch (computer science)Process (computing)Functional (mathematics)MathematicsWebsiteMultiplication signBoss CorporationInformationSoftwareTupleDifferent (Kate Ryan album)Similarity (geometry)Self-organizationContinuous integrationSet theoryUnit testingHierarchyWordVirtual machineServer (computing)Identity managementCartesian coordinate systemRevision controlRight angleBit rateInstance (computer science)Image resolutionFluidMachine visionJust-in-Time-CompilerNeuroinformatikIdeal (ethics)SubsetIntegrated development environmentCodeProduct (business)Reading (process)Software bugConfiguration spaceInformation securityRule of inferenceAnalytic continuation
17:11
MathematicsQuicksortRule of inferenceProduct (business)MereologyPhysical lawMeasurementEvent horizonSet theoryAxiom of choiceSound effectArithmetic meanContinuous integrationReal numberPoint (geometry)Service (economics)Natural numberCuboidIntegrated development environmentRepository (publishing)Right angleDatabaseBuildingSoftware maintenance1 (number)File formatLevel (video gaming)Parameter (computer programming)DialectRevision controlSoftware testingShift operatorInformation securityTheoremExecution unitComputer fileBranch (computer science)Computer programmingAreaUnit testingComplex numberMultiplication signPhase transitionDoubling the cube4 (number)Texture mappingFormal verificationCloningAbsolute valueSoftware developerFiber bundleDemo (music)Vulnerability (computing)Boilerplate (text)CodeMultilaterationHookingShape (magazine)Centralizer and normalizerMessage passingAdditionCartesian coordinate systemGoodness of fitDynamical systemNeuroinformatikEquivalence relationCollatz conjectureFrustrationMobile appBlock (periodic table)Computer animation
25:41
Software testingLogic gateMobile appMultiplication signPhysical systemSubsetLevel (video gaming)Order (biology)Projective planeState of matterFunctional (mathematics)LastteilungCryptographyAxiom of choicePhase transitionInternetworkingConnectivity (graph theory)CASE <Informatik>Error messageBitIntegrated development environmentWeb 2.0Pay televisionFunction (mathematics)Right angleRevision controlBuildingMathematicsCartesian coordinate systemSource codeGroup actionHuman migrationUser interfaceProduct (business)Task (computing)Fiber bundleInstallation artDebuggerParameter (computer programming)IdentifiabilitySoftware developerFront and back endsInteractive televisionPhysicalismVirtual machineControl flowSet theoryPoint (geometry)CodeOpen sourceLimit (category theory)Natural numberOracleWritingRun time (program lifecycle phase)Sound effectUniverse (mathematics)Shift operatorWordExistenceShared memoryAdditionComputer programmingSystem callPower (physics)Focus (optics)Message passingComputer animation
34:11
Computer animation
Transcript: English(auto-generated)
00:12
Welcome to our talk. I'm Nathan Smith from Chef Software. This is Rob Kidd, also from Chef Software.
00:20
And our talk is about strong practices for continuous delivery using Ruby on Rails and Chef Delivery. And I'm an engineer at Chef. I do mostly front-end development. So you can find me working on our web applications at Chef. I do a lot of stuff on the ops side, as many of us do at Chef.
00:48
And how about you, Rob? I am on the community engineering team. I maintain a Rails app that is the Ruby gems equivalent for the Chef community. Supermarket web app, the community site where cookbooks are shared.
01:02
I'm a sysadmin by trade and a software developer by happenstance. So we're going to start out talking a little bit about DevOps and why we do some
01:21
of the things that we do and how we solve some of the problems that we face. So if you look at this graph, this is kind of a graph of engineering anything. And what you want, you want to be at that top corner where you have a high level of quality with whatever quality standards you're reaching for.
01:47
And compliance with either, depending on what you're working on, it might be compliance with regulations or compliance with your company's internal rules. But you want your stuff to be good and you want your stuff to have compliance.
02:01
And you also want to go fast and you want to have a high rate of innovation to be able to try things, break things, do things and create new things. And there's a trade-off here. You can go really fast, not care about quality and make lots of stuff that's bad and you don't want that.
02:23
You can have very strict adherence to all of your compliance guidelines but move very slowly. And if you've ever worked in the kinds of companies that have a lot of compliance requirements, there's a lot of moving very slowly. So how do we take this curve and move it so we can get closer
02:43
to the corner without sacrificing our rate of innovation or the quality of what we're building? So some of these processes, these are some of the challenges that exist in engineering software. So we have manual processes where things can take a long time to spin up a new server, get new hardware, acquire some kind of software you need.
03:10
And so I'm just going to kind of roll through these different challenges and things that we can do to help solve them. So with manual processes, we want to automate. So we want to be able to build and use tools
03:23
that let us get rid of these manual processes and be able to do things in a more automated fashion. So a lot of us are probably using cloud for this. It's a lot easier to spin up an EC2 instance than it is to order and rack and stack new hardware.
03:45
Another thing is moving from legacy systems that are a lot harder to change to dynamic infrastructure. Again, some of that cloud-based infrastructure that lets you do a lot more dynamic management and be a lot more flexible in what you do.
04:03
In our organizations, silos can be a problem that don't let us collaborate in between teams. So what we want there is to build a culture where there's common trust and collaboration between teams and we can work towards those goals to increase our quality and the rate at which we innovate.
04:29
And then if you've ever tried to deploy something where you've made a lot of changes over a long period of time, when it's time to deploy that, it's really painful because so much has changed.
04:41
But if you can decrease the size of the batches that you're sending out and continuously deliver your applications, then it's a lot safer to make those small changes and your customers get them a lot quicker rather than having to wait for a big release. And then we have regulatory burdens where we have these compliance rules that we have to keep up with.
05:06
And the way we can increase that is to integrate what we're doing for our compliance workflow into the very beginning. And we can build safety and security end to end so we don't have to have this stage at
05:21
the end where now we've done our thing, now we have to wait to go through this arduous compliance process. So these are some challenges and some techniques that we can use to deal with those challenges. And then the result of that is we take this curve and we move it so we can
05:43
build higher quality things that are more compliant and we can do it faster by applying all these principles. And we put things like dynamic infrastructure, infrastructure as code, and DevOps, then those are the ingredients we can use to increase the...
06:04
What's DevOps? DevOps. Well, let's talk about DevOps. Why don't you tell me what DevOps is? I got a slide for you right there. At Chef, we have determined... You should always be wary of somebody who's defining DevOps for you, because vendors do it a lot. We're a vendor, and I'm about to give you our definition of DevOps.
06:23
We believe at Chef that it is a cultural movement and professional movement focused on how we build and operate high velocity organizations. And the movement and experiences come from the experiences of the practitioners. That's a mouthful, and it's kind of dense, but we believe that organizations move quickly, not necessarily software.
06:49
Organizations that move quickly move quickly with software. And the formula for that is you have good people, they build... How about this? Let me start over.
07:02
You have great people build great products, and those great products build great companies. If you have sad people, they're going to build sad products, and they're going to make sad companies. So what you want is the ingredients and the people and the culture that will let you build great products and great companies.
07:28
How about diversity? Diversity is super important.
07:49
Well said. Another thing that you're going to hear, if you start learning about DevOps and this movement and the kinds of companies and people who are doing stuff in this way,
08:01
you're going to learn some Japanese, or at least you're going to hear Japanese words. You might not know what they mean. Because a lot of this philosophy comes from, originally from the Toyota production system, and from a lot of work that was done in Toyota and in other companies to increase quality in their processes.
08:25
And so there's a lot of these lean words. So we want to eliminate non-value-added action, so eliminate waste. And Japanese for that is muda, I guess it says right there.
08:40
And then you want to have a system, if you ever used a system like Kanban, where you want to pull the work in rather than pushing it through a system. And kaizen is a word that means continuous improvement. So as you're going through these processes, you want to be looking at what you're doing constantly and finding out how to improve the process over time.
09:06
But then sometimes, instead of making these small changes, you're going to want to do what's called kaikaku, which is a big disruptive change. And you want to do these things in small batches. You want to do experiments and be constantly improving what you're doing.
09:24
And one of my favorite books you should go read is called The Goal by Eli Goldratt. It's a business novel, and it's a novel about a guy who is running a factory, and it's terrible, and he learns how to apply some processes like this in order to improve it.
09:46
And it's a very good lesson on what lean means, and there's lots of information about this. If you don't know about it, it's great stuff to learn about and to think about and to apply toward how we build systems.
10:02
And if you're going to move quickly, you need to embrace failure. You need to prepare for it and treat failure as a learning opportunity. Hold postmortems when there are outages or breakages or breakdowns in the process. And not hold people to blame, but to hold learning reviews so that the organization can learn from that experience and improve.
10:34
And you're going to want to automate everything, because the more manual things you have, those stand in the way of what you're trying to do.
10:43
So you can make donuts, but if you have this donut machine, you can make a lot more donuts. And we want to make a lot of donuts, and we want you to make a lot of donuts. So the rewards of adopting a DevOps culture and workflow is that you will deploy more frequently.
11:02
Because you can deploy more frequently, obviously your shipments are faster, and so your customers get changes faster, and presumably they are happier. You will have a shorter mean time to resolution when a problem is discovered to when it gets resolved, because you have gotten very good at pushing change out into production and in front of users.
11:25
The ability to push new features is also the ability to push security updates or bug fixes. Profits go up, company is happy, users are happy, everybody's happy. And if you want to sell this kind of stuff to your boss and say we should be doing DevOps,
11:43
prove to them that you make more money, and then your boss will be cool with it, I promise. And there's science behind that. Science plus money can pretty much answer that. Alright, so you all, this was a talk on continuous delivery. We will shift from talking about DevOps to a continuous delivery pipeline.
12:03
We still have some practices to talk about to prepare you to continuously deliver change to your users. This is, I don't think this is a legal license plate. It's not. And I think it's pronounced JIT now. Oh, JIT, yes, lay JIT.
12:23
But the tools that you're using matter. You can choose any tools you want, but a good example of how your tools will shape your culture is using a thing like Git. Pretty much everyone here probably uses Git or a similar capable version control system.
12:45
And it's important to use tools that will help you get your stuff done. Get er done. The other tenet here is that absolutely everything should be in version control. One of the, well, you can manage risk by having everything in version control.
13:05
There is an ideal that you can rebuild your systems with what's in version control, what's in your data stores, your databases, and compute that you have asked to exist. Either you get new metal servers, you get new EC2 instances, absolutely everything should be in version control.
13:25
Changes, like we pound this drum a little bit, small batches, to test that you are making the right thing. So you make a small change, you put it into an acceptance environment where ideally a customer or end user can say, yes, that's the thing that I wanted, that validates the work that you're doing.
13:45
It does introduce volatility in the near term, but you get long-term stability and reduce long-term risk. And another thing that we should be practicing is continuous integration.
14:01
So you do not want to have long-lived branches. And we stress that you want master to be, whatever you call master, to be something that you can deploy. And it should always be in a stable state. Because if you have these long-lived branches, they get out of sync, you have an integration problem when you try to integrate these things back in.
14:28
If you keep everything in small changes, merge to master as quickly as possible. If you find a problem, you fix it right away, and you do continuous integration.
14:42
You should also practice, as we do, the four-eye rule. This means two sets of eyes should review every piece of code, every set of changes that are going through the system. Two people should agree on that. It's a pretty simple rule, but we like getting marbled people in.
15:01
We also like telling people what to do, I guess. But this is advice, and I think if you follow this stuff, it'll be to your benefit. Write tests. We're among the Ruby community, so I don't think this needs too much stressing as far as unit testing goes. Ruby has a very good testing culture.
15:22
That's one of the reasons I love Ruby. And also, integration and functional tests that are higher level, more oriented toward the user. I think Ruby's still pretty good at that, but those things are less common. Having all these different kinds of tests should be there and are important in order to ensure the quality of your software.
15:49
There should be one path of change in an organization to move change from idea to production in front of your users. The delivery pipeline that we'll be demonstrating to you shortly is fixed.
16:03
These are the stages that a change goes through, whether it's a configuration change to a server, whether it's a code change to an application. It goes through these stages where you get to be flexible about what does it mean to verify and unit test an application.
16:21
It depends on the application stack, and I'll go into that in a little bit. Today, the rest of our talk, we're going to show how we're going to take an example Rails app, something really dumb and simple, and move it through this pipeline. Well, you didn't write it. You just copied it off the internet, right?
16:41
Rails, new, and then let's do this, basically. We're going to be deploying that app. So this first stage here, we have verify and approval. So one of the things we want to do, the verify stage is going to run
17:02
every time you, if you're using GitHub, it will run every time you open a pull request. Before we go into that, I'm going to pull the room here. How many of you write Ruby? Are you a Rails developer?
17:22
How do you lint a Rails app? You don't, okay? This is what I'll be encouraging you to do. So how do you syntax check a Rails app?
17:43
Also RuboCop. Turns out, in the Rails community, we have a lint and syntax trigger that we can use one tool. So that will become important later. And RuboCop can be kind of a jerk and pull you over and give you a ticket and not be very nice.
18:04
But you can customize how RuboCop works so that you can tell it that you should always use double quotes, because that's the right way to write Ruby. Or you can tell it to use single quotes when you're not interpolating things. It's up to you. It's up to your team. You should define those rules.
18:21
And also to get back to people and having some empathy for others, get RuboCop to be that nitpicker. Like a human shouldn't have to tell another human which style of quoting to use. Get a computer to do that so that everybody can bond in frustration at the computer. It's good team dynamics. All right, so how do you unit test a Rails app?
18:42
This is where we get maybe a little controversial. I don't know. Anybody? Unit testing a Rails app. RSpec. No controversy there. What else? What else can you use? MiniTest, absolutely. It's up to you. You can use RSpec or MiniTest for our purposes. We'll be using RSpec in our demo.
19:00
How about security testing a Rails app? It's totally something we should do, right? Breakband. Breakband, exactly. What else could you do with Breakband in addition to Breakband? Bundler Audit. It is an awesome little gem that the maintainers keep up to date with a database of gems,
19:21
versions that have vulnerabilities in them. So you can very quickly audit your gem file, specifically your gem file lock, the versions of the gems that your application is using and are there known vulnerabilities against them. In our pipeline, my master will go red if I have vulnerable gems. And that's something I have to fix.
19:42
How about quality testing? What would Rake test? I'm sorry? Reek. Looking for code smells. Yes, Reek.
20:01
There's Flog. There is Flog from the same fellow. There's Flay. So there are a few tools. This one I'm not as opinionated about in the Rails community. I don't think the community has come to a consensus of what quality means. And it can be kind of subjective for a team. I will be hand-waving over quality in this pipeline. There is a hook if you decide on how you would measure quality
20:24
and ensure that you don't go below a particular measure. My recommendation is decide on what quality means for a particular application. Maybe don't go below a 3.5 in code climate and then your quality check is the commit above or below 3.5.
20:44
This is sort of a quick glimpse at the Chef ecosystem. We have assessment tools. There's a ChefDK, which is a development environment, a Chef delivery pipeline environment service, centrifuge
21:05
that can control what changes are going into production. We're going to be focusing today on the delivery tool. And this is what the pipeline looks like, that fixed pipeline. This is the shape. You go through these six stages to take a code change into production.
21:26
In the verify stage, this is where if you have a pull request and it needs to be reviewed, this is the equivalent of having, say, Travis run your tests. Delivery would run these three phases in verify. Lint we've already established is RuboCop.
21:41
Syntax we've already established is also RuboCop, so I'm going to turn off syntax highlighting for this particular Rails pipeline. And unit tests are run by RSpec. It is after these three phases succeed that delivery would go nag a person and say, hey, this is worth reviewing because it's passed the tests. So a human needs to decide.
22:02
Oop, nope, I'm getting ahead of myself. Okay. All of you are Rubyists. Have any of you ever written Chef? Okay, a few of you have written Chef. The ones that haven't written Chef, tell me what this does.
22:21
This is not a trick question. It runs RuboCop. If you cross your eyes, you see it's just Ruby. It's an execute method. It takes a parameter. There's a block that does a thing. So this is the lint test in our delivery pipeline. That's the code that does the lint testing.
22:41
This is unit. You see the difference? It runs the spec. Obviously there's a lot of boilerplate there. I could improve this by wrapping it up into a test execute that takes the repetition out, but I wanted to not hide too much of the magic.
23:03
All right. So tests have passed in verify, and a human has been asked, does this code look good? This is the four-eye rule in effect. A human decides whether this change looks good. It looks great. It looks great. We're going to move into the build phase. So something that I would like to point out about lint unit syntax,
23:23
all of these light blue boxes that are running, because we advocate continuous integration, on the build node that's going to be running these tests, it takes the change set, the branch that is under review, and merges it locally to master. So it clones git, merges it locally to master, and then runs your tests.
23:41
Because if it doesn't merge cleanly to master, it's not been integrated against master. Continuous integration. It might run on your feature branch on your laptop, but delivery will tell you, hey, you're out of date. And you have an opportunity very quickly, because your branches don't live long, to catch up to master. So in build, we rerun lint syntax and unit,
24:03
because it might have taken a while for a human to push approve. So master might have changed. So we merge it to master at this point in the central repository, and rerun lint syntax and unit, to make sure that everything has integrated correctly. If those pass, we move into security and quality. I'm hand waving quality, but in security,
24:22
we'll run breakman and bubble audit. And then we move into publish. What is publish? One of the messages I want to, that we would like you to get out of this, is make a release artifact, and promote that into staging and production. You should not be running bundle install and production.
24:44
You should not be running bundle install and staging. You should be building a release artifact, and then testing that in your integration environment. So to that effect, we will be using bundle package, in the publish phase, to cache our gems.
25:01
We precompile our assets, and then we put that into a tarball. We're going to be deploying to Heroku, so the choice of release artifact format, depends on your production environment. So in this example, we're going to use Heroku, so I'm going to give it a tarball that I have precached stuff in,
25:20
so that I don't depend on Heroku being able to reach Ruby gems, or needing to compile my assets. So Chef Delivery, it's a program for deploying apps to Heroku. It could be. Did I use it on EC2?
25:42
Yes. Okay. But not on physical machines? No, you can use it on physical machines. Okay. We're trying to focus on the delivery pipeline, and how you get an app through it, so a simple way to do that is to just use Heroku.
26:01
If you're just using Heroku, as you normally do going through the tutorial, you're probably not going to use these APIs directly. You might not have heard of the Heroku source API, but there's an API on Heroku that you can upload anything to,
26:20
and it'll give you back a URL, which is just an S3 URL, where you can get that artifact. So we're just kind of using that as, hey, we created a tarball of our thing, so we need to put it somewhere. Heroku conveniently has a place, and we can use the URL that it gives us in the Heroku build API. This would be the secret code.
26:40
The peek behind the curtain, when you push Heroku, all the tasks that it takes, whether you're asking Heroku to do the bundle installs or whether you do it ahead of time with this tarball, in the end, when Heroku has gotten your app runnable, what they call that artifact at the end that is the runnable state is called a slug, and this is a way to poke the API with a tarball
27:01
to get a slug out of it. And if you are just using Heroku, you might not need to do this, but to kind of think about the flexibility of how Chef delivery works, say you're using Heroku, and then you get big,
27:20
and Heroku's not economical to use anymore, and you want to switch over to using Chef to spin up EC2 instances and deploy your app that way, then you can use your same workflow and your same pipeline, and for the developer end user, nothing's going to change,
27:41
and everything's going to work the same way. So that's another reason to just, let's take the simplest possible thing and deploy with Heroku. So the simplest possible thing, the thing that I'm going to do if I succeed at making that release artifact, and in our case it's a slug, a Heroku slug, that Heroku's stored and I have an ID that is the identifier for that release artifact
28:03
that Heroku is storing for me, I could move into an acceptance environment, and in the acceptance environment, I'm going to spin up the app at this version that's being tested and test it and then make it available to a human to accept the changes. It's a little bit like a QA environment,
28:21
but we use acceptance. In the provision phase, I'm going to make sure that a Heroku app exists that I could deploy to. If you were in EC2, the provision phase would be spinning up a VPC and a load balancer and your web nodes and your database. It's make sure you have runtime infrastructure
28:42
in place to deploy to. The deploy phase obviously deploys, and then if those two succeed, no errors in either provisioning the infrastructure or deploying this version. Smoke tests are very fast tests to just see if it's worth going into the longer running functional tests. We hit some URL endpoints
29:00
and make sure that we get 200 OKs. If we get 200 OKs, we know it's worth testing with the longer runs. Functional tests in this case were feature specs that are told to run against a remote URL instead of a locally running rack app. Yeah, so copy borrow. Copy borrow. It's a good thing to use. Use cucumber.
29:21
So what does provision... So using the Heroku API in this example, what does provision deploy look like? It says the app name, project stage. Stage is I'm in the acceptance stage right now, so it would make me a project acceptance dino app. I poke Heroku app create with some parameters
29:41
and say I want to make sure that this app exists. This is a simplified version of the code. There's a little thing that if it already exists, it's fine. And then that succeeds. I test that it succeeds. I have an app. It might not be running anything yet. And then in the deploy, I'm using the API again to say, hey, app, you have just come into existence.
30:01
Or maybe you're using... Maybe you still exist from the last time I ran. I want you to run this version. And you poke it with a slug ID. And Heroku handles spinning that thing up. And you're not running bundle install again. And you're not compiling gems, and you're not compiling assets. It's ready to go. So it's a very fast swap out at Heroku.
30:22
And then delivery will poke a human again and say, all right, acceptance is up. It's passed all the automated tests. And now a human needs to go and decide, was this change implemented correctly? So a QA group can now go and poke a thing and confirm that customers would want to see this. And your choice now is, do I deliver this to production?
30:43
You decide to. You hit the little button that says deliver. And it moves into a union environment. And it runs those same four stages. Provision a union environment. Deploy our application to the environment. Smoke test it and functional test it. What is different about union is that union is the environment
31:02
in which all your components run. And your tests will test against other components that are related to this pipeline, but not directly in it. If you have a front end app and a back end app, the front end, a change to the back end would get through acceptance. But then in union, it would be tested against the latest
31:22
front end app. And smoke tests and functional tests for the back end app would also run smoke tests and functional tests for the front end app. So you would find interaction errors in union. Some people call union staging. We call it union because it has a very intentional purpose to be the first opportunity for the delivered versions
31:43
of all of the components of a system to be tested together. If union is green, all the tests pass. It automatically, a rehearsal environment is spun up. I'll explain rehearsal in a moment. If rehearsal is green, all the same tests pass. It moves on to the delivered environment,
32:01
which is probably your production environment that your users are hitting. I'm reading your minds and your question is, what's the purpose of rehearsal? If union breaks, if you discover an interaction error between two components that were tested in isolation and acceptance and they break in union, you need to fix it.
32:21
So union has gone from green to red and now you need to push a change through the pipeline again to bring union from red to green. Rehearsal is where you test going from green to green and ensure that the fix for the error does not assume that the system is broken. My example, my go-to example are schema migrations.
32:41
If you have a schema migration and then your data looks wrong, if your fix is, I assume the data looks wrong and I make it right, in an environment like production where you didn't break yet, if your fix for the data error, the schema error is, the schema is broken, it's not going to pass.
33:00
So rehearsal is an opportunity to go from green to green and catch it there. And we can kind of show a little bit of the Chef Delivery UI and then we'll have a few minutes for questions. I'm not on the internet right now. You're not on the internet? No. Okay. It might take me a minute, but if you stick around we can check. Yeah, yeah, we can check. So pretty much when you're using Chef Delivery,
33:21
which you can get with a Chef Premium subscription, you have a web interface that you can run a delivery cluster that has build nodes and it has a web interface where you can visualize all this stuff happening and you can see the output from your test runs,
33:42
you can click a button to approve things, and it is important that we have those, those manual gates are built into the system in order to ensure that we can have a human look at a thing. So if you want to stick around, we can kind of show you around the UI of delivery,
34:01
but now we can take some questions if you have any. All right, well, thanks for coming everybody. We really appreciate it.