.NET Deployment Strategies: the Good, the Bad, and the Ugly
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 133 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/48930 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
00:00
SoftwareentwicklerKartesische KoordinatenDigital Rights ManagementVisualisierungRechenwerkSoftwareJSONXMLUMLComputeranimationProgramm/Quellcode
00:52
Gewicht <Ausgleichsrechnung>SoftwareentwicklerVollständiger VerbandChecklisteSkriptspracheSoftwareMailing-ListeKartesische KoordinatenTeilbarkeitSoftwareentwicklerMathematikFaktor <Algebra>Prozess <Informatik>Rechter WinkelARM <Computerarchitektur>Quick-SortQuantenzustandProdukt <Mathematik>GarbentheorieSkriptspracheGreen-FunktionMultiplikationsoperatorVersionsverwaltungFrequenzSystemplattformAggregatzustandPhysikalisches SystemServerExogene VariableDigital Rights ManagementVakuumEDV-BeratungGüte der AnpassungSelbst organisierendes SystemBefehl <Informatik>Grundsätze ordnungsmäßiger DatenverarbeitungMereologieSoftwareVollständigkeitCodeURLDienst <Informatik>ClientKontrollstrukturMailing-ListeSoftwaretestBitBenutzerbeteiligungProgrammierumgebungTest-First-AnsatzStrömungsrichtungKonfigurationsraumSoftwarearchitekturElektronische PublikationGewicht <Ausgleichsrechnung>Vorlesung/KonferenzBesprechung/Interview
07:36
SoftwareMailing-ListeSoftwareentwicklerCodeDreiecksfreier GraphProzess <Informatik>Gebäude <Mathematik>Produkt <Mathematik>CodeMaschinenspracheBitExistenzaussageRichtungRuhmasseZahlenbereichServerSoftwareTransaktionSelbst organisierendes SystemDreiecksfreier GraphPunktMereologieFahne <Mathematik>Kartesische KoordinatenDienst <Informatik>Spezifisches VolumenSoftwareentwicklerBenutzerbeteiligungSprachsyntheseCoxeter-GruppeSystemaufrufSkriptspracheOrdnung <Mathematik>Reelle ZahlElektronische PublikationComputerspielSoundverarbeitungDifferenteVersionsverwaltungRechter WinkelGüte der AnpassungTouchscreenVirtuelle MaschineDatenbankFlächeninhaltGruppenoperationMathematikURLMultiplikationsoperatorData MiningFortsetzung <Mathematik>Grundsätze ordnungsmäßiger DatenverarbeitungVorlesung/Konferenz
14:20
Dreiecksfreier GraphSoftwareentwicklerSkriptspracheProzess <Informatik>ProzessautomationProzess <Informatik>Dreiecksfreier GraphSkriptspracheBenutzerbeteiligungProgrammierumgebungKonfigurationsraumSoftwaretestSoftwareentwicklerProdukt <Mathematik>PunktMultiplikationsoperatorBereichsschätzungSichtenkonzeptBitZweiGamecontrollerGüte der AnpassungSoftwareProzessautomationFehlermeldungElektronische PublikationProgrammfehlerSelbst organisierendes SystemIdeal <Mathematik>GruppenoperationRekursive FunktionMAPSpeicherabzugKartesische KoordinatenMessage-PassingTaskARM <Computerarchitektur>TropfenKontrollstrukturMereologieRechter WinkelZahlenbereichEinfache GenauigkeitSkalenniveauFestplatteForcingVersionsverwaltungWarteschlangeDisk-ArrayFreies ProduktComputeranimation
22:25
SoftwareentwicklerSicherungskopieServerSicherungskopieExistenzaussageRoboterCASE <Informatik>Quick-SortProdukt <Mathematik>ProgrammfehlerUnendlichkeitSoftwaretestMultiplikationsoperatorResolventeDateiformatBefehlsprozessorSynchronisierungSoftwareentwicklerProzess <Informatik>CodeDifferenteGruppenoperationVersionsverwaltungKartesische KoordinatenMathematikEinfache GenauigkeitBitFacebookAuswahlaxiomGebäude <Mathematik>MusterspracheSoftwareDatenbankRefactoringDatensatzPunktSkriptspracheReverse EngineeringMereologieMini-DiscInformationKomplex <Algebra>ARM <Computerarchitektur>Rechter WinkelZeichenketteGüte der AnpassungNummernsystemWeb SiteMinkowski-MetrikDienst <Informatik>TransaktionTeilbarkeitComputeranimation
30:30
SoftwareentwicklerVerzweigendes ProgrammAbstraktionsebeneSicherungskopieVersionsverwaltungCodeVersionsverwaltungDifferenteKartesische KoordinatenInterface <Schaltung>Service providerVirtuelle MaschineSoftwareUmkehrung <Mathematik>ServerImplementierungVollständigkeitMultiplikationsoperatorAbstraktionsebeneSkriptspracheVerzweigendes ProgrammKontrollstrukturMigration <Informatik>Framework <Informatik>ClientAggregatzustandEinsDatenbankGleitendes MittelMaßerweiterungProdukt <Mathematik>MathematikIdempotentZurücksetzung <Transaktion>Überlagerung <Mathematik>RechenschieberSicherungskopieZahlenbereichPunktBildschirmfensterSpeicherabzugPatch <Software>SchaltnetzGruppenoperationGarbentheorieElektronische PublikationRechter WinkelARM <Computerarchitektur>Fortsetzung <Mathematik>GamecontrollerEnergiedichteCASE <Informatik>BitInjektivitätNichtlinearer OperatorBildverstehenTotal <Mathematik>Dreiecksfreier GraphInverser Limes
38:35
SoftwareentwicklerCodeVisualisierungProzess <Informatik>Produkt <Mathematik>SoftwareGeradeQuaderVirtuelle MaschineElastische DeformationKartesische KoordinatenCodeDienst <Informatik>SoftwaretestMehrrechnersystemAnalogieschlussBenutzerbeteiligungQuick-SortServerPrinzip der gleichmäßigen BeschränktheitPublic-domain-SoftwareLastteilungEinfacher RingSchießverfahrenMailing-ListeReelle ZahlMomentenproblemPlastikkarteMereologieBetriebssystemExogene VariableStellenringLastExistenzaussageAggregatzustandVersionsverwaltungDatenbankMathematikSprachsyntheseLeistung <Physik>MaschinenschreibenComputerspielSummengleichungObjekt <Kategorie>Ganze FunktionKontrast <Statistik>BeobachtungsstudieUnordnungDruckverlaufRechter WinkelComputeranimation
43:56
SoftwareentwicklerKartesische KoordinatenBildschirmfensterServerLastteilungPrinzip der gleichmäßigen BeschränktheitZentrische StreckungLastBitDiagrammRobotikHardwareVersionsverwaltungEinsGanze FunktionProdukt <Mathematik>PunktProgrammierumgebungBetriebssystemUnordnungSoftwareGewicht <Ausgleichsrechnung>Prozess <Informatik>CodeVirtuelle MaschineSoftwaretestSoundverarbeitungNichtlinearer OperatorZweiAusnahmebehandlungMereologieSkalierbarkeitBenutzerbeteiligungMaschinenschreibenARM <Computerarchitektur>Dienst <Informatik>Web logMultiplikationsoperatorPhysikalisches SystemSystemplattformGeradeSondierungWeb ServicesRechter WinkelMehrschichten-PerzeptronSummengleichungSchreib-Lese-KopfComputerspielEgo-ShooterApp <Programm>Computeranimation
49:17
SoftwareentwicklerIdempotentOpen SourceMereologieVisualisierungSchnitt <Mathematik>SoftwareentwicklerKartesische KoordinatenWiederherstellung <Informatik>VollständigkeitProdukt <Mathematik>Numerische IntegrationInstantiierungServerCASE <Informatik>Prozess <Informatik>FestplatteBildschirmfensterQuick-SortIntegralVersionsverwaltungFacebookZehnBetriebssystemSoftwareProjektive EbeneEreignishorizontBenutzerschnittstellenverwaltungssystemDemoszene <Programmierung>E-MailBildgebendes VerfahrenMomentenproblemMini-DiscPunktImplementierungMultiplikationsoperatorARM <Computerarchitektur>BAYESMathematikSprachsyntheseSelbst organisierendes SystemHIP <Kommunikationsprotokoll>Dienst <Informatik>Public-domain-SoftwareSystemaufrufSicherungskopieVirtuelle MaschineObjekt <Kategorie>Vakuum
56:12
SoftwareentwicklerOrdnung <Mathematik>Produkt <Mathematik>Digital Rights ManagementPRINCE2GeradeElektronische UnterschriftFrequenzProzess <Informatik>Public-domain-SoftwareMultiplikationsoperatorLesen <Datenverarbeitung>Basis <Mathematik>CodeFlickrPrinzip der gleichmäßigen BeschränktheitARM <Computerarchitektur>Rechter WinkelLeistung <Physik>BildverstehenBitrateComputeranimationVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:05
Thank you everyone for coming. It's good to see a full room here, particularly when I looked at the agenda and saw who I was up against, you know, Damien Edwards and David Fowler in another room. By the way, I really hope that nobody gets up and walks out when I say this. Damien Edwards and David Fowler in another room, Hattie Hariri announcing something from
00:24
JetBrains in another room as well. But it looks like all of you guys care about deploying your software, which is a good thing, especially as far as I'm concerned. So that's me. I just realised half of that's wrong, by the way. It's not ALM, it's not MVP in Application Lifecycle Management.
00:44
They renamed it. It's like Visual Studio and Developer Tools or something now. I probably need to learn what it is. But cool. Let's get started. So like I said, welcome. And I'm going to be talking to you today. My name's Damien Brady.
01:00
I work for a company called Octopus Deploy in Brisbane, Australia. Who here, just out of interest, who uses Octopus Deploy? Wow, that's like half the room, which is awesome. You guys are effectively paying me to be here, which is cool. Yeah, that's really good to see. So I'm not going to talk about Octopus really at all, which some of you are
01:25
going, oh, yeah, somebody stood up. I am going to talk about things that I've learned working at Octopus and also prior to Octopus as a consultant as well. So I've had a lot of exposure, obviously, to the way that people deploy software. And I've seen a lot of good stuff and I've seen a lot of horrible stuff as well.
01:45
So that's kind of where I'm coming from today. And we're going to go through a few different things. Before I go much further, though, I want to introduce you to somebody. This is Frank. And Frank is responsible for the deployments at his company. So the developers come and they have a new version of their software and they
02:03
give it to Frank and it's Frank's responsibility to put this in production. So Frank has this document that he's developed over a long period of time. It's got all these steps. You might see some red marks in there. He's made some changes as he's gone through. He hasn't quite got them on the soft copy. It doesn't really matter, though, because he's the only one who does it anyway.
02:25
And his job changed, by the way, or his title changed. He was the deployment manager and now he's the senior DevOps technician. So his job didn't change. He's just have DevOps in the title now, which you have to do. So he gets this document and when there's a new version to deploy,
02:41
he will go through this document and make sure every single step is followed. And the right files are copied to the right location, the right servers are spun up, the right SQL scripts are run, things like that. And he's good at this. He's been doing it for years and years. And he's very good at this process. And it works pretty much all the time, right? Occasionally, though, one or two mistakes, like the last time he deployed,
03:05
there was a bit of an issue where the web config in production got overwritten with the dev version. It wasn't Frank's fault. He told Brian, developer, don't leave it in the package, please. But Brian forgot, and so anyway, it only took him about three or
03:22
four hours to get production up and running again. So that was fine. But that doesn't usually happen, and Frank is pretty happy with this process. It works most of the time. It gets the code into production. Everything's great, right? So Frank's an idiot. That's not a good process. That's a terrible process.
03:40
And this big document that he's holding is essentially, it's just a script, right? It's a list of things that have to happen to deploy the software. And scripts are generally trustworthy if you write scripts and they run automatically and so on. But Frank's not infallible. He makes mistakes.
04:01
It doesn't matter how good this script is. If he's following it and doing it manually, he's going to make a mistake here and there. Now, this is a really, really common deployment scenario. If anybody wants to actually admit to it, whether they do this in their current organization, I've worked with countless companies and countless people who do this kind of process.
04:25
So we're going to go through a little bit why this is bad, some things you can do, and some actual specific tips to make your application deployable in a better way as well. So I'm breaking this down into kind of three sections.
04:40
The first section's really going to be talking about what we try to do as developers and why this whole DevOps deployment automation, all that sort of stuff, why that movement's important and why it's important to get it right. We're going to talk about the idea of making your software actually deployable. So it's all very well and good to say, yeah, deploy frequently,
05:00
do it all automatically. But if your software's not written in that particular way, it's difficult to do. So a few practical tips about how you can change your software architecture to make it easy to actually deploy this way. And then finally, we're going to look at the idea of deploying your application holistically. So accepting the idea that your application doesn't live in a vacuum,
05:24
it does run on servers, it runs on operating systems, it runs on platforms, things like that, and making sure that you account for those when you deploy your software and the advantages of that kind of thing. All right, so let's start. We'll take a step back and look at what we actually do as software developers,
05:41
and that's kind of an easy question, right? We make software. Who here would count themselves as a software developer? Okay, pretty much everybody. Great. So that's a simplistic statement, right? There's a lot more to it. You make software, great. But there's a lot of, like, if that was as simple as it was,
06:02
we wouldn't have a conference like NDC, right? We wouldn't have a room full of people in each of these rooms looking at how to do it better. So I've been thinking about this for a long time because it's difficult to convince some companies and some clients and so on that they're doing things wrong. People get angry about it.
06:21
So I sat down and I kind of thought about writing a complete list of everything that matters in software development. So, like, three-day conference with a couple of days of workshops about things that matter in software development. And I got this list. That's the list, by the way, the entirety of the list. It's not item one of ten.
06:41
What matters is delivering working software to users. Again, very simplistic statement, but there's kind of two parts to this. There's working software and then delivering it to users. Everything else we talk about and argue about as developers and all these sessions in all of these rooms will really focus on these two things.
07:01
Working software is things like test-driven development and having different environments to make sure that your application actually runs correctly and having a test process and all that kind of stuff. And red-green refactor, or as I like to call it, red-green refactor, red, red, red, oh God, revert, usually.
07:21
So all of that stuff is around working software. And then the other side of it, which in the .NET community we've only really started focusing on it with any depth in the last few years, which is actually delivering it to users. That's always kind of an assumed part of the application.
07:40
You write your software and then of course it goes to production. That's not a big deal. But all of this stuff around agile process and DevOps and all of those things really focuses on this second part. Traditionally we've cared very much about the first part, but not so much about the second part. And the second part is just as important as the first, right? Sure, there's no point having software that doesn't do what it's supposed to do,
08:03
but there's no point having it at all if it doesn't make it into production. Because code that doesn't make it into production is effectively 0% done. If you have magnificent working software on your local machine, but your customer or your end user never sees it,
08:20
you may as well have not started. You've been in that situation in some companies, I'm sure, where you're 80% done three months in and the customer hasn't seen a thing. They ask you again in a couple of months, how far are you done, oh, we're 80% done. You said that last time, sorry, we're 85% done.
08:41
But unless it's 100% done, unless it's in production, it's effectively useless, like the customer doesn't care. So getting it into production is just as important as writing good software. So what's wrong with this process, right? Sure, it's a little bit problematic sometimes, it causes problems, but you're putting stuff into production, right?
09:00
You've got some rigour around how you do that. To work out what's wrong with it, it's a good idea to have a look at what might be in this deployment process. So, number one, right-click, publish. This is a massive pet peeve of mine, by the way, and if there's any Microsoft people here, I'm glad Scott's not here because he does it all the time, Scott Hanselman.
09:21
They all do. Right-click, publish, oh my God, it's in production in Azure. Don't deploy your software like that. God, that's horrible, right? Anyway, so you right-click, publish, you have a build server in real life and you dump the files in a particular location, then you RDP into your production servers,
09:41
maybe you have a cool tool like this that lets you RDP into a number of servers, you put some stuff in the web server, you run some SQL scripts in the production database, you do all this other kind of stuff and eventually you get it done. There's also some other steps, like making sure that you don't copy over the web config, and that's a real screenshot, by the way, I didn't make that one up.
10:02
So you have all of these other steps, other things you have to remember when you're doing your deployment. The problem with it is, as I mentioned before, that humans make mistakes, right? That's deliberate. Humans make mistakes, right? They're not as trustworthy as we all think we are.
10:22
It doesn't matter how much awesome process is in your document, you might skip a step, or you might copy something you didn't mean to, and it's not your fault, it's just that we're fallible, we make mistakes. Who's heard of this company, Knight Capital Group? Couple of hands.
10:41
I always like seeing just a few hands, because it's the best story ever. AKA worst story ever. So Knight Capital, essentially they're a trading firm, they have software that does trades on the New York Stock Exchange. Sorry, I'm speaking in present tense, this is past tense. You'll see why. They did trading software, and they did huge numbers,
11:01
like huge high volume of trades, so billions of US dollars a day in trading. And they had a service that did all this stuff, and it was running on eight different servers, and did all this work. In 2012, the developers made a change, and they repurposed a flag in the application. So there was a flag that they could pass that was never ever used anymore,
11:22
and it ran down this code path, which hadn't been touched in ages and ages. So they repurposed that flag, and they put new code behind it, and they tested the hell out of it, right? Billions of dollars of transactions a day, of course they're going to test it. And they knew it worked, and so they deployed it overnight. And in the morning, when the market opened, this happened.
11:44
They lost $440 million in 45 minutes before somebody worked out there was a problem, and hit the kill switch. Now, worse than that, this organisation was big enough that they were a market mover, is what I've been told, which means that they were doing transactions that didn't make sense,
12:04
and artificially moving the market in particular directions. So they got gigantic fines for this as well. All of these other companies were relying on their own kill switches, and something's going wrong, we're not going to trade anymore to get that working. So there's other companies downstream that got hit by this really badly.
12:23
Now, the software they deployed worked. The code that they deployed absolutely 100% worked. But the problem was, the guy who did the deployment deployed it to seven of the eight servers. So that eighth server was running through this code path that hadn't been touched in years,
12:43
and the flow-on effect was effectively instant bankruptcy for this company. So that's an awesome story about working software, but the problem being deployment. And usually what happens when a company has some catastrophic failure like this, hopefully not in the order of $440 million in 45 minutes,
13:03
but usually their reaction is to add more governance. This wouldn't have happened if we had six more people sign off on it, and the developers didn't have access to production and things like that. And maybe it's pretty unlikely though.
13:20
I mean, if 10 other people had signed off on that deployment, the guy still would have forgotten the eighth server. It's not going to change too many things. It actually gives you the illusion of safety when it's not actually adding that much safety anyway. The downside of this as well, adding more governance and slowing down your processes, you get into this deployment cycle which I kind of call the bureaucratic deployment cycle.
13:44
And this is kind of how it goes. If you're in this deployment cycle, it basically runs like this. So our deployments are so far apart, they take so long between deployments, because there's so much stuff that we have to test, because the deployments are so far apart.
14:04
And it feeds on itself, and it makes things worse and worse and worse. Every deployment you do every six months, there's so much stuff you have to test, and if one of those things isn't right, you've got to start the process again, and it feeds on itself and makes things worse.
14:20
And slowing these things down makes things worse as well. There was a government organisation that I worked for back in Australia, and they had huge rigour around their deployment process, which was understandable given the industry, which I won't talk about because it's not a great story. They had huge rigour around their deployment process.
14:42
They were deploying to a test environment, and the DBA running the script accidentally ran it against production. It happened. Terrible situation, but it happened. They ran it against production. The big problem was, to get something into production to fix it,
15:00
officially, their deployment process was gigantic. So effectively, it took 10 seconds to break Prod, about 10 minutes for them to work out a fix for Prod, and then six hours to put that fix into Prod. So production was down for six hours because of a 10-second mistake. And that was not because the developers weren't doing a good job or anything like that.
15:22
It was because their deployment process had got so bulky and so much of a big deal that it was a problem when something went wrong. And that's what you really need to avoid. The first step towards moving towards this is always automation. Automate as much stuff as you can. If you can click a button and it does 20 things,
15:43
it will do those 20 things exactly the same way, every single time. And that's because scripts, for the most part, are repeatable and trustworthy, as long as you write them to do what they're supposed to do. It's not going to skip a step. It's not going to forget to copy the web config or leave the web config out or something like that.
16:01
It's going to do exactly the same thing every time. And if it doesn't, if you make a mistake in this script, you're going to make that mistake once. And then you can fix the script and it won't happen again. But automation, so people usually go to automation. Automation by itself isn't good enough. Automation is not the same as a good process.
16:22
You can automate some terrible stuff. We had a customer of Octopus who was calling us because they were blaming our recent release for really, really slow deployments. They said, these used to take five minutes and now they're taking upwards of 30 minutes. We haven't changed our process. What have you done? You've broken something. The first step in their deployment process, completely automated, was this.
16:44
Does anybody know what that means straight off the bat? Because I didn't. I had to look it up. What that does is that grants permission to the everyone group full permissions to C drive recursively. And the reason they did that is because there were some file permission errors
17:00
once or twice when they were deploying the thing and they went, you know, the easiest thing to do is just make C drive read-write for everybody. As C built up a little bit and more and more stuff got on the hard drive, it made things slower and slower and slower. So half hour deployments now for what was essentially just dropping some files in a folder. So automation is not the solution of itself.
17:23
You still need to know what you're doing when you deploy your software. It's really important. And if you're not automating these things and you're doing it all manually and something goes wrong, something like this happens. If you're doing it manually, it's much easier to see the problem. If you're doing it automatically, it kind of hides the problem from you.
17:42
Like that's why they thought it was our fault that it was a 30 minute deployment. No, it's because you're doing a stupid thing at the start. If you're doing it manually though, there's other problems that can happen. So something goes wrong and you think, oh, it was just this time that I forgot to do this thing. It'll be fine next time.
18:00
If you're working on a script or automation, you'll fix that problem and make sure it won't happen next time because you don't have control over it. Because people in IT are generally liars. Maybe not deliberately. Put your hand up if you have estimated how long it's going to take to complete a task. Yeah, you probably lied, didn't you?
18:21
Like not deliberately set out to mislead, but you know. I did something similar a while ago, but that time it was different because whatever, this time will be twice as fast. We're not good at judging when something is a transient problem or when something is not going to happen again.
18:40
And we have a bit too much confidence in what we do generally. If there's a script that does it the same way every time, it's going to do it the same way every time. So that's really important. The other idea of DevOps and this deployment process and things is something we've had for a long time, which is having your test environment and your staging environment
19:01
and all of the QA and pre-prod and all those pre-production environments to make sure that you've tested the application in ideally the same kind of scenario as it is in production. But it's really important to get this into your deployment process as well. Like if you're deploying to production, that's the first time you've ever done that deployment.
19:22
If something goes wrong, you've broken production. It's much, much better if you can get yourself into a situation where the same deployment happens, regardless of whether it's going to staging, test, pre-prod or prod. So by the time you hit production, you've run those same steps to deploy the same artifacts
19:41
ten times or five times or whatever it is. But it's not the first time. You're much, much less likely to have a problem. It lets you kind of deploy fast and frequently, really, really often. And if there is a problem, you're going to see that problem with your deployment in a pre-production environment. You're not going to break stuff.
20:00
You're going to break pre-prod. And that's fine. Breaking pre-prod is fantastic. It means that you didn't break production. You've caught this problem before it got that far. And if you can get yourself into this cycle, deploying really fast, failing really early, it means that by the time you get to prod, it's really trustworthy. And it kind of gives you this opposite deployment cycle
20:21
as the bureaucratic one. It's this kind of virtuous deployment cycle we refer to it as. And it kind of goes like this. So we deploy frequently, which gives us confidence in our process because it happens over and over again, and it's fine. And so we're comfortable deploying more frequently. The idea being that a production deployment is not a big deal anymore.
20:44
You can deploy to production because you've done it ten times. You know by the time it gets to that point where you can put it in prod, it's been tested. It's fine. We know it works. So I just wanted to, before we move on to the next bit, I just want to hit the highlights because I know I'm kind of skipping through a lot of stuff.
21:02
I just want to make sure that the key things are things that you remember. So number one, get it in prod. It doesn't matter if it's not in production. So that's a key aim of your deployment automation and deploying your software. Get it in production as early as possible. The idea that bureaucracy only gives you the illusion of safety,
21:21
it doesn't actually make your process better. It just means that more people are responsible when it goes wrong, which is not a good thing from a development point of view, software point of view. Automate as much as you can because you won't make mistakes then. The script will run the same way every single time. And then finally, get yourself in a situation where you can fail perfectly happily
21:42
in a pre-production environment and it's not a big deal. Much, much, much better than failing in production. The kind of the core message of all of this though is that production deployments should not be a big deal. Your ideal scenario is that production deployment should not be a fanfare at all.
22:02
You just go, oh, we have a bug, the bug fixes there, cool, let's push it to prod. And just to make sure you know I'm not kind of just saying this stuff and then going back home or going back to Octopus and then we right-click publish. We don't do that. So this is the way we currently deploy at Octopus.
22:22
This is Slack, so this is our chat room stuff. This is one of the channels. There's a lot of stuff in there but the summary of it is, in this case, Shane, so anybody can do this. There's a release bot that we wrote and we just say release Octopus deploy whatever version. And that's a compiled version. We know it works because it was tagged with that version.
22:43
TeamCity built it. We ran API tests against the application running in like 8 or 10 different server arrangements. We ran all of these tests against the thing and we know it works, right. So by the time we can do this, that version is definitely working. So we say release. Release bot says cool, does some stuff and then tells us it's done.
23:05
And that is a live version of Octopus deploy available for download on the server with one command from Shane. Or anybody, really. So our production deployments are not a big deal at all.
23:20
And we've done things, we've deployed to production. Somebody said we've got this weird edge case, I think you've broken something. We look at it and just go, we did make a mistake actually. Do a bug fix, build it, pull the old release, push out a new version and that has taken like 10 minutes. It's a really, really, like there's no fanfare behind it, it's just a really, really nice way of working.
23:45
So that's kind of how we work. So I'm not just telling you, hey you should do this stuff, we don't do it though. Kind of key to this as well is the idea of only moving forwards with your deployments. And this is something that people are really uncomfortable with.
24:01
Not rolling back. So probably the best way to talk about this is to talk about backups. Who here in their deployment process has step one, back up your database, back up your files, back up all that sort of stuff? Okay, only a few hands. Maybe that's because you know where I'm going with this.
24:23
How many of those people test their backups regularly? So back up, run it and then roll back. Yep, I saw a couple of hands. Okay, that's good. How many people, once the deployment is finished, delete that backup? Nobody.
24:41
Cool. I am of the very strong opinion, and people don't like this, that backups are bad. If you are relying on your backup as part of your deployment process, this is my opinion by the way, not Octopus. Although we do say it. If you're relying on your backup, that's a terrible situation to be in.
25:05
Yep, I'm telling you don't do backups. If your backup is part of your deployment process, that's fine. You back up everything, you do your deployment process, it fails halfway, maybe there's not enough disk space, maybe something goes wrong. Yeah, restore, because nobody's hit it at that point. But if you've deployed and then five people have used your application, put five rows in the database, and you go, oh, something's wrong,
25:27
your choices now are kill their data and restore from your backup, which is not a great scenario. Or do some weird backup and then try to replay those transactions, and it's not a good situation to be in.
25:42
A better situation is if you can get yourself in the pattern of being able to release a fix to that into production really, really quickly. So the example I gave before about the guy who ran the script in production and then they needed to fix it, they had to fix it, but then it took them six hours to get a new version out.
26:00
If you can do that in ten minutes, it's not that much of a big deal. Yeah, you have to work out what you did wrong and how to fix it, but actually getting it into prod is quite easy, so it's not quite a big deal. With databases, this is quite difficult as well. I just want to run through a couple of techniques for actually making your software able to do this,
26:22
because obviously you can't just say, oh, fine, we can just move forward constantly. You've got to think about this, you've got to write your software in a way that supports this kind of thing. So the first one I came across was the idea of transitional deployments. Transitional deployments are the idea of accepting that maybe you do need a way to back out of your change.
26:44
So rather than going whole hog, let's delete columns of data, things like that. Acknowledging that you may get it wrong and giving yourself a way of fixing that problem. So the best example is kind of, let's say you've got an application, it's got some customers,
27:01
you've got an ID and name, a date of birth, and somebody in their infinite wisdom back in the 80s decided date of birth should be a string. It's happened to everybody. You need to change this to a date time. Obviously there's a lot of stuff in your code that's going to need to change. You don't want to just parse it, turn it into a date column and then kill the old stuff.
27:22
The transitional deployment should be acknowledging that and adding a new column. Maybe keeping these in sync and even like, triggers a badge, but using triggers to just enter the old data in a nice format just in case you have to kill it later on. If something goes wrong, you can revert back to that old code and deploy a new version which doesn't look at that DoB column
27:47
because you know something's wrong, you just need to work out what's going wrong. You haven't lost any information here. And then once you are confident that the fix you've made is good, you can get rid of that column if you want.
28:04
Now this is obviously a simplistic example. With a more complex database and more complex schema changes, there's obviously problems. But if you can think about making it a reversible change or something that doesn't tie you into a corner,
28:20
that's a good way of doing it as well. This is kind of described as well in a really good book called Refactoring Databases, Evolutionary Database Design by a guy called Scott Ambler. It's not a new book, but in databases nothing's new. But have a look at that if you want to find out about some of these techniques for evolutionary database stuff.
28:43
That's the database stuff, a technique for that. But being able to revert these things, there's a really good technique called feature toggles. Does anyone use feature toggles now? Oh, awesome. That was like a third of the room or something. Oh great, I'm going to get this wrong and you're all going to yell at me.
29:02
The idea of feature toggles is that you have this new feature that you're building and you can deploy it to production and flick it on for one or two people. So this is something that Facebook does actually for every single bit of code that they write. Every single change that they make is feature toggled.
29:22
And they deploy it to production, they just have to make sure it compiles, that's fine. Deploy it to production and then the developer or the team that's responsible for it will just feature toggle it on, they'll just toggle it on for themselves. So not only are they testing their own code, they're testing it in production and not affecting other people ideally, not affecting other people.
29:42
If they find there's a problem with it, let's say they flick it on and all of a sudden the CPU usage goes through the roof and they go, oh God, that's bad, flick it off, it's not going to affect you anymore. Work out what you did, work out what went wrong. If you just can't resolve it, well, Facebook, if they just can't resolve it or if it has performance problems, they just leave it there and never turn it on again.
30:05
I was listening to one of the Facebook speakers at a conference not long ago and he said, yeah, we've got a lot of code in our code base that never runs now because we don't really clean that stuff up, which scared the crap out of me. There's so much stuff in Facebook that is code on the server
30:23
that just never gets hit because they just don't clean it up because they're a bit scared to. That's a bad way of doing it but if you have these feature toggles, the other advantage is that you can turn this stuff on, you can make corrections to it and so on until you think it's right and then you can do these little canary deployments to specific users. So you might say, yeah, we've got this build or it's sitting in production,
30:45
you guys, you're really nice to us, we trust you, we're just going to flick this on for you. Can you tell us whether we've done anything wrong? Does this work for you? Is this right? And if it is and you're confident, you can just start flicking it on for more and more people until it's on for everybody.
31:01
If it's wrong, you can say, sorry guys, I'll flick it back off again, we'll work out what we did, we'll fix the problem, we'll come back to you. It's really good for software that people have to download and install themselves, so Windows software and things like that as well because you can keep releasing new versions of the software constantly
31:21
and nobody sees any changes ideally and then on your three-month cycle, which is really common in large companies, they don't like the idea of a new build every day, on your three-month cycle, you can flick all of these features on and say here are the new features and not only are there new features that you've just released in this cycle, they're features that have been tested in production
31:42
because a few people have been trying them out. You know they work, you're confident in that way as well. So that's really cool, doing feature toggles for every new change that you make. One way of doing that is the idea of branch by abstraction. So this is something that I'd actually done a lot and I didn't know what it was called,
32:00
but Martin Fowler decided to call it branch by abstraction, so apparently that's its official name now. But effectively it means that it's kind of an extension of inversion of control, dependency injection, that kind of stuff. So you might have like a weather provider, and this is a client that I did some work for, they had a weather provider and they just used,
32:20
I don't think it was Google, but they used another one. So they had one implementation of this weather provider that would provide weather forecasts for the next few days or something like that. We worked on a replacement for it, but we were still deploying the whole time. As long as this compiles, we don't have to go down that code path, it doesn't really matter. So we were just doing it, improving the implementation and so on
32:44
until we thought it was ready, and then you just change the IOC container so it points at that guy instead. And you can combine this with feature toggles and say, if this person is asking for an I weather provider, give them the Bing one, and you can test it that way. Flick it off again and it says, okay,
33:01
I'm going to resolve that with the Google weather provider. And this is a really good way of actually implementing this, putting an interface over the top of it, using dependency injection, inversion of control, things like that to control that. So that's pretty handy. You can also, again, if you don't want the technical debt, by the way, feature toggles, awesome way of building up technical debt.
33:22
You can remove the old stuff as well, which is ideal, but it would have been great for Knight Capital if they'd removed that code that never ever got run, because that would have saved them the entire company. So yeah, that's one way of doing it as well.
33:42
So that's a few little techniques that you can use. There's a lot more of this kind of stuff, but the combination of those transitional deployments, feature toggles and using branching by abstraction are really good ways of actually implementing that. So again, I'm doing these little summaries just because I've run through a bunch of stuff. I just want to make sure that the key points stay with you a little bit.
34:03
So number one, backups are bad. Damien Brady from Australia told you not to back up anything. Don't actually quote that. The idea of using transitional deployments and acknowledging that maybe you need a way to back out of a change later on, like you think everything's right, you've tested it, but if something goes wrong, maybe you need to flick it off.
34:24
Think about what you've done again. I'll just go through the summary and then I'll grab a question. Feature toggles and using branching by abstraction to get those things going as well. Yes, so the question was,
34:51
if there's a build that breaks or something that breaks and you need to roll back to a previous version or something like that, you have a previous version of the deployment.
35:01
So in Octopus that would be a previous release that you've used that you can deploy again. The question was, that's not what I mean by backup, and that's correct, that's not what I mean by backup. We, worst case, and one of the things that we recommend as well at Octopus, is rolling the previous version forward. And I'll talk about that literally in the next couple of slides.
35:24
I don't consider that a rollback. It's a roll forward of a previous version. And obviously you can't always do that depending on... Well, there is a difference, trust me. There is a difference. Because rolling back is generally like something went wrong,
35:40
we need the previous version of the code, which this is the same as previous version of the code, the previous version of the database and so on. So you're restoring from your old stuff. Rolling the previous version forward is taking your application and deploying it again over the top. And the big difference, and actually I'll get to this, because that kind of answers the question a little bit better.
36:02
The idea of your deployments being idempotent and complete. So that's really the idea that if you deploy version 1.5 of your software, and maybe something breaks towards the end, you can hit it again and just deploy it again and again and again. It doesn't make a difference. That's the idempotent stuff, right?
36:20
Being a complete deployment means that if everything falls away and you lose a server, you can spin up a new server and just do a deployment of your application. It's everything that's there. If you have an idempotent and complete deployment, rolling forward the previous version of your application is exactly the same as rolling forward a new version of your application.
36:43
You just say, this is the release, dump it on top. The difference is really the way that the application is structured. Traditionally you might do this kind of thing. You have application version 1, and then you have some scripts that run to turn it into version 2, maybe replace some files.
37:01
It's essentially a patch. And then version 3 runs on top of that as well. The difficulty with this is you're kind of relying on the existing state of the application. So if your deployment is things like, run this SQL script, because that's what turns version 1.3 into 1.4.
37:22
That's not a great situation. What you really want to be getting to is this is at version 1, that's version 2, it just clobbers the old stuff and dumps the whole thing. So if you deployed version 2 without having version 1 in place, doesn't matter, same thing. Same thing, version 3, you just clobber version 2.
37:41
Now under the covers it may actually look exactly the same as that. So the idea of database deployments using migrations and things like that, so things like dbup and entity framework migrations and stuff like that, they will manage this for you. So they will say, well the current state is here, I only need to run these three scripts because we've done the previous ones.
38:02
So that's kind of still a big bang deployment. It's not saying version 2 of the application is reliant on version 1 being there and we also do these things. So if your application can be deployed in one hit like this, then rolling forward a previous version is the same as rolling out that previous version to a fresh machine.
38:22
It's the same as rolling out a new version. There is a subtle difference in that and it's really important to understand. Does that answer the question? Cool. I even had two slides prepared. The other side of this as well is that infrastructure matters. We think of our application as just we deploy some code
38:43
and we make some database changes and that's fine. But really the actual server state kind of matters as well. So who's heard this phrase, treating your servers like cattle, not pets? Yeah, a few hands. Somebody advised me not to use this analogy in Europe but I'm going to anyway.
39:03
I'm just going to play the insensitive Aussie card. The idea, so this was coined by an ex-Microsoft engineer called Bill Baker and it was about the way that you treat your servers, so the stuff that your application runs on, that you should treat them like cattle, not like pets.
39:20
So if you treat them like pets, you give them a name, maybe your production server has a name of a Lord of the Rings character or something like that and you treat them really nicely and if they get sick, like if you need updates and things like that, then you nurse them back to health, right? They're really precious to you. By contrast, according to the analogy, if you treat them like cattle,
39:42
you don't really name them individually and if one of your cows out of your herd of a thousand get sick, you'd shoot it and get another one, right? Not the most PC analogy but it gives you the idea, right? If you just treat them as disposable objects,
40:03
treating your cows like disposable objects, I'm making this better, aren't I? Treating your servers like disposable objects, then it gives you a scenario where it doesn't matter if a server dies, you just replace it with another one, that's fine. Which means it kind of gives you this idea of deploying infrastructure alongside your code.
40:22
Now you don't have to deploy everything, you don't have to have your entire server, VM and the operating system and all that sort of stuff in one big package and deploy that, although that is a way of doing it. But it means that if something goes wrong, if you lose a server, not a big deal, you can just replace it with a new one and you know that your application is going to run
40:40
because that's your deployment. You've deployed that whole thing to staging, you've deployed that whole thing to QA, you know that that works. There's no unexpected surprises about, it works on my machine but we deployed it to production and it's complaining about missing a DLL, right? Who's... I've done this.
41:00
Who's deployed or who's installed Visual Studio on a production web server before? Yep, I have. Not recently, but it works fine, it works on the test server where we have VS so we can change a few things. We deployed it to production, it's complaining that something's missing. Let's just put Visual Studio on it because it puts all the DLLs in the place
41:21
that I had it in my local machine, right? That's not a great way of doing it. If you have a scenario where the infrastructure is part of your deployment, you know that when you deploy it to production that that's what you've deployed, the entire thing plus all its dependencies. There's a customer of ours at Octopus that I always talk about called Domain.
41:43
I don't know whether you guys have Domain, you probably have really similar things here but it's a real estate listings company, so a web search for rentals and house buyers and things like that and they have a bunch of services. And this is kind of the way they've worked on this for years to get their DevOps process really, really smooth.
42:01
They've written hundreds of thousands of lines of PowerShell but they have like hundreds of services, microservices and things. But they are awesome at this stuff. So they have like a self-healing, self-balancing, load-balanced thing sitting in AWS. So these green boxes are all VMs that are running in AWS
42:20
and they've got ELB, so Elastic Load Balancing over the top. They use Octopus to deploy the versions of the software. It's not really important what you use. If you're using Octopus, great. If you're using something else, great, that's fine. Slightly less great but still great. So they use Octopus to deploy the software to all those machines.
42:40
If one of those machines stops serving, like the application stops serving responses properly, it effectively dies and the ELB and all their monitoring and stuff will pull it out of the load balancer. Now because it's a load balancer, it says, well now we're under higher load on these servers, we need a replacement server.
43:01
So it hits Octopus and Octopus actually deploys the entire VM. So they use Octopus to deploy a new VM using Azure DSC and it installs a tentacle and then it talks back to Octopus and says, hey, I'm one of these guys, what version should I be running at the moment of this application?
43:24
Octopus will tell it and it will say, okay, great, trigger a new deployment just to me of that version of the application. They'd run some smoke tests and things like that and then they will add it back to the load balancer. Now the important thing here is this is 100% no touch at all.
43:41
If they lose a VM on Saturday night while they're six pints deep, this will happen anyway. Speaking of being in a pub six pints deep, they sometimes six pints deep will just log on to AWS and shoot a server. It's like the ultimate chaos monkey.
44:02
Let's say, check this out, kill, and then they just watch it come back up again. They're so confident in this process working and it's such not a big deal. They're deploying to production effectively but they'll do it at the pub. And this guy's actually at the pub a fair bit.
44:21
This kind of extends as well. They've got this concept of a robot army. If you have a look at the blog as well, so tech.domain.com.au, they describe all of this stuff and it's been an ongoing process. They're kind of up to robot army 3 now, I think. The last thing they talked about was 2.5. To deploy a new version of one of their applications, this is what they do.
44:43
So let's say we've got three web servers running in a load balancer there. Three because it's low season, they still don't like to do this peak. But they can. The first thing they do is a phantom deployment and that really just has the effect of telling Octopus, hey, the latest version in production is this one.
45:02
So if you've used Octopus before, there were a lot of hands, which is good. They basically trigger a deployment and disable every step except for the one that tells Slack that it's done. So they might say, I want to deploy 1.5 but deploy it but just skip every step except for this one. So Octopus now says, right, 1.5 is in production.
45:22
It hasn't actually hit any of these machines yet though. So they're kind of working around the way that Octopus works, which is fine. So what they do then is they take out a server. They'll just shoot it exactly the same as before. The load balancer says we need a new server because we're under high load.
45:40
So they do the exactly the same process as before. They provision a server using Octopus but this time, when it asks what version is in production, it gives it the new version. So this new server gets spun up with the new version of the application and added to the load balancer. Now obviously you'd need these to be able to run side by side and it not be a big deal.
46:00
So they can't do this with all of their applications. If they could, they'd kind of have to do a bit of a switch or at least some sticky sessions and things like that. But that's effectively what they do. And then they can deprovision the other ones and just wait until the load balancer loads up the new ones. Another way they do this is scale out and scale back in. So they'll just say, just scale out to six servers
46:22
and then scale back to three. And the way that their ELB stuff works is it'll provision six new ones and then just remove the older six. So they just move everything across. Again, this is completely no touch for them. They hit a button to say deploy and it will just do all of this stuff.
46:43
They do exactly the same process in their non-production environments. So they spin up VMs for their test environment and QA environment and so on. So they know the entire process from start to finish will work because they've done it in all of their previous environments. Not deployed the same code but done the exact same process.
47:02
So this is an awesome, awesome way of deploying the software. Like I said, it's taken them years to get to this point but it's a really exciting thing for them. This kind of leads on to the idea of containers. So they use entire VMs for this stuff, which is fine.
47:21
But you kind of don't need an entire VM. It's easy in AWS because you just spin up an AMI. You have an AMI and it's not easy. 20 minutes? Oh, there you go. These guys are claiming less than 20 minutes but maybe they're doing some small stuff anyway.
47:42
So the idea of containers is you don't need the entire operating system and all that sort of stuff. This is where the idea of Docker and things like that kind of came from. You can do smaller bits and instead of, like Scott was saying in the keynote, instead of lying to the operating system or lying to the hardware, you're lying to the operating system
48:02
and you're lying to your application and just saying, hey application, you've got your own operating system and hardware. It's probably best with some diagrams and stuff but I'm not going to talk, by the way, too much about Docker and about Windows Server containers and things like that because there are people here who are much, much, much better than me at it.
48:22
So like Michelle Bustamante and Mark Rendell, I think, have talks on those. And there's even a panel later on about microservices and containers and things like that. But the general idea is the way we usually think of deploying our software is we have our application, we have a server, we deploy our application to the server, right?
48:42
It's a simplification because we have an operating system, we have a platform, we have .NET stuff, we have dependencies, we have IIS running on the machine, all these other things that we need and if we're just deploying the application, we're relying on those things being there already. The idea of Docker and containerization and stuff,
49:00
is that this is really what you want to have as part of your deployment. Yes, your application, but you also need to know about all the other things that are required by your application. You don't need to redeploy the operating system or the server. So the way it kind of works, at least in my head, is you have the server there, the hardware, that can be a virtual server,
49:22
it can be whatever. You have your operating system and then like a Docker host or something over the top or a Windows Server, or is it Hyper-V host or something like that. You have your application in these Docker images or in these little packages of things and then you deploy them to a container that sits in the host.
49:44
So your application is just running inside this sort of stuff. If you have a new version of your application, you're in a very similar situation to what I was describing with domain, except you don't have to deploy an entire VM. You can just deploy your new version to the same host and then when you're fine and you've changed the network addressing
50:04
and all that sort of stuff, you just turn off the old one and the new one is the one that's working. You can scale out, you can put other applications in there as well. That's also a really, it's an easy way of doing things. Now, I'll admit I haven't used Docker or Windows Server containers in anger at all.
50:24
I've had a play but I haven't really ever, since they started, had a good reason to use them. So again, I'm not the right person to talk to. That's a way of deflecting questions. Go and speak to Michelle or Mark about those or see those sessions. What I do want you to do though is be aware of it,
50:42
start reading about it, start thinking about how your applications can take advantage of this as well and just be prepared for what's coming. So that's really important. The last few things, again, I'm just going to do this little summary. The last few things I talked about, making your deployments item potent and complete.
51:00
So the entire deployment, it's not a big deal, you don't have to rely on the previous instance of your application to be running so that when something dies, it's a big deal. There was actually the same government organisation, you'll get a picture of what they looked like internally, but they were running a really important application
51:21
and they were running it on a Mac, this is back in 2002, they were running it on a Mac, not a server machine, one of those glassy bubble things, but it was in the server room, so you know, it's a server, and it was sitting there, they had a process of backing up where they put a zip disk in every day
51:40
and it backed up and then they put the zip disk to the side and so on. It died one day, that machine, but it was such a big deal because, well, A, it was so old that they needed to get a hard drive manufactured to replace the one that was there, which cost them a lot of money. And then when they put the hard drive in and they went and restored from that backup,
52:01
it turns out the zip disk was empty, it hadn't actually been backing up for years. They spent a whole lot more money getting some data recovery from the old hard drives, but they did lose a lot of stuff, there was a lot of emails flowing around to replace this application for a few months while they were sorting this all out.
52:21
But big, big deal, the best part of it was, so my boss was responsible for repairing this thing, it took him months, big disaster, finished it, everything was back up and running again, and he said, cool, let's start the project to replace it, and they went, what are you talking about? We spent so much money getting this working, it's working again.
52:43
Here's some other things to do. But if they'd been in a situation where deploying that software was just a matter of, servers died, bang, and admittedly back in those days it wasn't a case of VMs being able to spin something up quickly, but they could have maybe got another Mac, bought one on eBay or something like that, and then dumped it on there.
53:03
All of the dependencies, all of the things that needed, if they were part of that deployment, it would have made a lot more sense, it would have been a couple of days rather than a couple of months. So the infrastructure matters. Your application doesn't run in a vacuum, there are dependencies,
53:21
and being aware of them and including them in what you're doing is really important as well. Treating your servers like cattle, not pets, or less horribly treating them like replaceable objects, things that you don't need to worry about, and then finally looking at containers and containerisation now,
53:44
and at least having a think about how your applications might be able to take advantage of that. Just before we move on, there's one little point I want to make at the end as well, that a lot of people have asked for. But before we move on, does anybody have any questions? I think I've got about 10 minutes and only a couple of minutes left, yeah?
54:09
The question was, is the Slack integration available for the public? Let me just take a drink. Not yet.
54:20
So if you use Octopus, you might know that it's API first. So doing this implementation was relatively easy. We've just got a lot of stuff to make it work exactly the way we want. But it means that anything you can do in the UI in Octopus, you can also do through the API, because that's all the UI does. It just hits our own API. We are doing some work at the moment behind the scenes
54:43
to improve the integration story with Octopus. And that'll mean that there's events that can get raised, there's things that you can subscribe to. So this whole idea of when stuff happens, do this, or when Octopus does this thing, let these things know as well.
55:00
As part of that effort, we're just going to go and start going to places like Slack and HipChat and I would say the team rooms in Visual Studio, VSTS, but nobody uses them. And all these other integration points and being able to plug into those. That will be open source. I don't actually have that from OnHi.
55:22
Admittedly, there's only 12 people in the company, so OnHi is not that high. But I would guarantee that those integrations are going to be publicly available. So it will be there soon. Any other questions? Cool. So there's one thing that I get after this talk a lot,
55:41
which is, yeah, cool, you've talked about all this sort of stuff, but in reality I'm going to go back to my company and we're going to get out that document in three months' time when it's time for our production deployment. It's very nice of you to say that, you're a small company and these other people are very small as well. But you've got to keep in mind that Facebook has tens of thousands of developers
56:03
and they will do a daily cut and push to production every single day. And yeah, it wasn't easy for them. It's not an easy thing to do this stuff. It's very hard, right? Domain spent a couple of years and wrote thousands of lines of PowerShell to get this stuff working,
56:20
but the advantages now are really things that they're loving. They love talking about this stuff because, again, they can do it from the pub and it's not a big deal. So the idea that we can't do it because we're a gigantic company, Flickr as well, they deploy to production like 30 to 50 times a day, I think.
56:42
They're bigger than, well, I'm generalising, but they're bigger than your company, right? That's probably not actually true here. It is in Australia. But yeah, it is difficult and it is a journey. I hate that word, sorry. It is a thing that's not trivial to do. You do have to do a lot of work and it is really hard to do.
57:03
But the benefits far outweigh the costs in this. I also get asked, how do you convince your managers that time should be taken doing this? Because they don't really see the benefits.
57:21
Another conference I was at, I'm stealing this blatantly from somebody else, but another conference that I was at, a guy put up a quote and it was a fantastic quote and it kind of described the way that I've always told people to do this, to try and convince their managers and so on. And it's by Antoine de Saint-Exubery who wrote The Little Prince.
57:42
I'll just let you read it because there's nothing worse than somebody reading while you're trying to read. But the idea is, don't tell people, hey, we're doing DevOps now and therefore you have to do this stuff. That's the wrong way to convince anybody to do it.
58:01
You don't go to your manager and say DevOps is the thing we need to do. Here are some vague advantages and we're going to do it, it's really important. Show them the vision, get them actually seeing the advantages of it. I was speaking to somebody a couple of hours ago actually, who was saying in their company they have an automatic process
58:22
that deploys to QA, they commit their code and it's in QA half an hour later. But the production deployment is a whole different story. It goes through a steering committee and gets signatures and somebody gets a big document out and they just try and reproduce what they did to get it in production.
58:41
But their technique to try and get this process happening automated in production is that they're releasing to QA and requesting a production release on an hourly basis now. And this production release team is going, guys, you can't release that quickly. What are you talking about? We're doing it. We released to QA 10 times today.
59:02
What are you guys doing? So you can show them how easy this stuff is and after a period of time, yes, there's going to be pushback, but after a period of time, even the worst manager is going to go, maybe we can use this same process to put it in production. We haven't had a problem in QA for six months.
59:22
Maybe we can do that. But yeah, if you sell the dream, it's much, much easier to sell this idea of doing this work. So that's the last thing that I really wanted to talk to you about. So that is me. Thank you.
59:41
Now, have we got like one or two minutes, I think, if there are any other questions? Yep. I think I drained people of questions anyway, so... By the way, if anybody wants to talk to me about Octopus as well, I'll be floating around for the next two days. Otherwise, thank you very much. Enjoy the rest of the conference.