DevOps: More than Marketing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 64 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45921 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201128 / 64
3
7
10
11
17
19
21
28
33
34
35
37
40
44
48
49
52
55
57
59
62
63
64
00:00
Turnbull, JamesSystem programmingSoftware testingInformation managementEnterprise architectureInformation Technology Infrastructure LibraryFunctional (mathematics)Configuration spacePoint cloudImplementationSystem administratorContinuous integrationComplex (psychology)Control flowCollaborationismRevision controlState of matterBitQuicksortBlogOperations support systemService (economics)Group actionSystem administratorOpen sourceMereologyWaveSign (mathematics)Perfect groupSoftware developerWindowAuthorizationKey (cryptography)WordPoint cloudSet (mathematics)Goodness of fitComputer animationXMLLecture/Conference
02:50
Normal (geometry)Mobile appProcess (computing)Lattice (order)Data conversionResultantBitAreaProduct (business)MereologyCartesian coordinate systemText editorAxiom of choiceOrder (biology)Key (cryptography)Analytic continuationProcess (computing)Physical systemOperating systemIntegrated development environmentInformation managementSoftware developerTerm (mathematics)Different (Kate Ryan album)QuicksortSet (mathematics)Rule of inferenceBand matrixElectronic program guideMultiplication signPosition operatorMessage passingMathematicsSoftware bugElectronic mailing listVideo gameOperator (mathematics)MassConfiguration spaceMaxima and minimaMoment (mathematics)Fitness functionOperations support systemState of matterTheoryMagnetic stripe cardCASE <Informatik>Computer animationLecture/Conference
07:18
First-person shooterBeat (acoustics)Core dumpProcess (computing)AutomationBuildingDatabase normalizationExpected valueSoftware testingSystem administratorRevision controlOperations support systemCartesian coordinate systemQuicksortBus (computing)Type theory1 (number)MathematicsPhysical systemMainframe computerComputer architectureSoftware developerInformation securityLibrary (computing)Process (computing)Integrated development environmentProduct (business)Demo (music)Structural loadClient (computing)Lattice (order)Radical (chemistry)TouchscreenServer (computing)Term (mathematics)Enterprise architectureCycle (graph theory)Data centerProjective planeVideoconferencingService (economics)Message passingDampingEmailMultiplication signData conversionBitPoint (geometry)NumberBuildingMereologyClassical physicsInternetworking2 (number)ExpressionFormal languageVideo gameVirtual machinePoint cloudConnected spaceKey (cryptography)SpacetimeGodStandard deviationSet (mathematics)Scripting languageOperating systemVirtualizationArithmetic meanAutomationBoss CorporationGraph coloringComputer animation
15:16
AutomationSoftware testingBuildingDatabase normalizationExpected valueProcess (computing)Information managementConfiguration spaceInformation securitySubsetControl flowRevision controlContinuous functionArchitectureBackupCartesian coordinate systemJava appletOperations support systemQuicksortIntegrated development environmentKey (cryptography)ResultantSystem administratorOperator (mathematics)MereologySoftware developerPhase transitionModule (mathematics)Product (business)BuildingDirect numerical simulationProcess (computing)Software testingState of matterGame controllerMultiplication signCycle (graph theory)NumberOpen sourceFrequencyDatabase normalizationComputer architectureExpected valueConfiguration spaceTrailCollaborationismCodeFlickrConfiguration managementRevision controlService (economics)Data centerStructural loadGraph drawingTerm (mathematics)outputAnalytic continuationVideo gameDemoscenePosition operatorPoint (geometry)Moment (mathematics)Group actionInformation securityMoving averageSoftware development kitFunctional (mathematics)VibrationSpacetimeComputer configurationStability theoryInformation managementMathematicsAlgebraWeb 2.0Function (mathematics)Virtual machineComputer animation
22:41
Continuous functionRevision controlControl flowArchitectureSoftware testingMaxima and minimaProduct (business)Range (statistics)Computer wormNP-hardGrand Unified TheoryNormed vector spaceProcess (computing)Right angleCoordinate systemGrand Unified TheoryService (economics)Operations support systemFunctional (mathematics)Software developerCartesian coordinate systemEmailQuicksortIntegrated development environmentMultiplication signBoss CorporationUniform boundedness principleAnalytic continuationLevel (video gaming)Time zoneSelf-organizationTelebankingException handlingResultantSet (mathematics)VirtualizationSystem administratorEndliche ModelltheorieTraffic reportingSoftware testingMathematicsData conversionProduct (business)Physical systemMereologyBitExistential quantificationSpacetimeSystem callSeries (mathematics)Order (biology)Message passingCodeoutputOraclePerspective (visual)CodecWeb pageSatelliteVideo gameComputer animationLecture/Conference
30:07
Self-organizationLocal GroupDependent and independent variablesClassical physicsRoboticsTask (computing)Cartesian coordinate systemSequelSelf-organizationOperator (mathematics)Multiplication signGoodness of fitSystem administratorQuicksortResultantView (database)MereologyGroup actionDifferent (Kate Ryan album)Operations support systemUniqueness quantificationSoftware developerBus (computing)Product (business)Integrated development environmentConfiguration managementCodeLimit (category theory)Arithmetic meanLine (geometry)Mainframe computerSet (mathematics)PlanningOrder (biology)Physical systemLipschitz-StetigkeitService (economics)MathematicsAnnihilator (ring theory)AreaKey (cryptography)BitPerturbation theorySoftware testingEvent horizonLoop (music)Configuration spaceNetwork topologyBookmark (World Wide Web)Server (computing)Speech synthesisStapeldateiScripting languageInformation managementComputer animationLecture/Conference
37:07
Range (statistics)Execution unitSoftware developerMetric systemOperations support systemCodeMereologyCartesian coordinate systemEnterprise architectureMultiplication signGroup actionProduct (business)Hard disk driveSelf-organizationData conversionIncidence algebraDrag (physics)QuicksortUniform boundedness principleNumberSet (mathematics)Java appletCoordinate systemConfiguration spaceMoment (mathematics)Online helpKey (cryptography)Field (computer science)Order (biology)Kanban <Informatik>Scaling (geometry)State observerSemiconductor memoryLibrary (computing)Projective planeExistential quantificationInformationGrand Unified TheoryCollaborationismDifferent (Kate Ryan album)Computer architectureAxiom of choicePoint (geometry)Web 2.0Social classBefehlsprozessorProcess (computing)Computer animationLecture/Conference
43:52
Execution unitLevel (video gaming)Event horizonCartesian coordinate systemoutputGreen's functionNumberField (computer science)QuicksortBitPhysical systemIntegrated development environmentSelf-organizationBefehlsprozessorProcess (computing)Semiconductor memoryConfiguration spaceVirtual machineWindowSoftware developerCASE <Informatik>Response time (technology)Euler anglesDrag (physics)Dependent and independent variablesOrder (biology)Client (computing)Operations support systemDenial-of-service attackMetric systemView (database)MathematicsResultantDampingOpen sourceQuality of serviceRing (mathematics)MiniDiscConfiguration managementMetropolitan area networkMultiplication signVapor barrierReal numberInformation managementComputer animationLecture/Conference
50:37
XML
Transcript: English(auto-generated)
00:07
Thank you very much. Good afternoon. Thanks very much for coming along. It's a very, very big room. It's a lot bigger room than I'm used to speaking in.
00:23
Start off a little bit about me. I've been involved in the open source community for about 20 years now. Since I was a teenager and discovered this Linux thing was a bit cooler than that Windows thing. I work at Puppet Labs. I run the operations and professional services group there. I'm an author. I've written five technical books,
00:40
mostly about Linux, sysadmin and open source sort of things. I'm a big geek. My wife would say I'm very much a big geek. I'm known to buy every single gadget that appears on the market. I usually buy two of them and break them fairly quickly. I should warn you, I do talk funny. I'm from Australia and pretty much every other part of the English-speaking world
01:02
doesn't believe we speak English. So if you find anything you can't understand or find that I'm talking very quickly, wave your hands frantically in the air and go please slow down. Okay. Not a good sign. I'm also a whinging bastard. I've spent a lot of years
01:21
in operations working in some big shops and I'm well known for complaining about things. I don't like things that are broken. I'm a firm believer in perfection. I don't think you can achieve perfection but you can get a bloody well of a long way towards it. So what's DevOps all about?
01:43
This is a topic that comes up a fair bit and I don't think there's a clear answer to it yet. So DevOps is about cooperation for me, cooperation between development teams and operations teams. And particularly important as to why Belgium is so important for DevOps is because the gentleman who coined
02:01
the phrase DevOps, Patrick Dubois is obviously from Belgium and Chris, and I'm going to stuff his surname up, Bayart, also heavily involved in the original DevOps movement. So pretty much the original DevOps thinking emerged out of a bunch of people in Belgium sitting around, I think, drinking beer, which may have something to do with it,
02:20
while they felt the love for one another. And generally the whole movement emerged out of a bunch of Northern European sysadmins. So what comes up when you start to look at DevOps? So I did a little bit of a keyword cloud, and I'm afraid the top bit's cut off there.
02:40
There's some of the phrases that come up when you look at various blog posts around DevOps. And I look at these sort of things and I go, well, what has this become? Is this some kind of buzzword bingo where after a few phrases you all get sitting around together and go, we can talk about DevOps and cooperation and ITIL and we can talk about cross-functional
03:01
teams and then somebody can sit in their meeting and go, bingo, we've managed to parrot out all of the appropriate corporate phrases and phraseology. Is it a pop culture movement? Are we a bunch of fly-by-night guys who are just trying to con you into buying something? I've seen a couple of products so far that claim they're DevOps compliant. I'm not sure
03:21
who they were certified by, but if they buy me beer, I'm happy to. Or bourbon, even better. I don't think we're a pop culture movement. I also don't think that we're going away in a hurry. But I do have some caveats before I talk a bit further.
03:42
So it's very early days. So DevOps is probably about two years old in terms of its sort of thinking and it's probably only about six months old in terms of people actually sort of, I guess, more theoretically considering all of the possibilities that DevOps implies. So, largely speaking,
04:00
the first conversations about DevOps happened two years ago. I think last year they had a DevOps dinner at Fosdem and I believe there was about 10 or 15 people there. Last night they had the same thing again and there were 60 people. So it's early days. No one has all the answers. I don't think anyone's suggesting that it's a fixed movement.
04:20
We certainly don't have a manifesto or a style guide that says, you know, this is how you recognize it's DevOps. It's painted green and it's got little stripes. That's not the case. Nothing's fixed in stone. I'm certainly not advocating a position that there's some sort of doctrinal approach that we need to take.
04:41
And at the moment, for me, this is all about outreach. It's trying to get a message out about what DevOps is and about the community around DevOps. And you may notice a little footer on my thing, all opinions, my own maybe subject change added alcohol. I think a lot of the DevOps movement has been about discussion about people going, well, DevOps is a bit of this, it's a bit of that.
05:02
Let's see how this all fits together. For me, I like lists and DevOps for me is about four things. Simplicity, relationships, process and continuous improvement. And I'm going to talk a little bit about each of those areas and why they're sort of important to me and why I think
05:22
they're an important part of DevOps and why I think that makes DevOps useful to both operations and development communities. Firstly, simplicity. We all work in, if you work with IT, then IT is complex. The simple things, some of the simple things
05:42
in an IT shop, how you build your application stack, what particular operating system you choose, can cause a massive amount of debate. I'm not even going to talk about things like if I sat down and said to you, Vim is the best editor in the world, then all the Emacs people would be stoning me. And it's not an uncommon phenomenon in the IT world to have hugely complex applications linked together
06:01
based on choices that you have to inherit from somewhere, legacy applications. And in order to manage those environments, simplicity is the key. So you need to choose things that are simple and easy to manage. Those things need to be repeatable and reusable. It's particularly particularly in environments where you're managing thousands of hosts.
06:22
You cannot do something that is a one-off thing anymore. You can no longer get away with saying, all of my systems are Unix snowflakes, and they're all different, and they need a different set of rules. You simply do not have the bandwidth or the time to manage a thousand systems that you are considering to be Unix snowflakes. And simplicity implies easy to communicate.
06:41
Whatever message you need to communicate about your systems, whether it be their state, whether it be their configuration, whether it be how to recover them or back them up or secure them, it needs to be something that's simple and easy to communicate. The key for me around this is that if you're sitting at the end of the day, you're sitting in a room full of people trying
07:01
to fix a bug or doing an outage, you want to have the best possible conversation with somebody where the end result is that you solve the problem as fast as possible and you solve the problem in such a way that it never reappears again. And if your environment isn't simple and repeatable, then you're going to have a lot of struggle with that. The next thing for me is relationships.
07:23
So we all work in an industry that's well known for the fact that we're supposedly misanthropic, we're supposedly this bastard operator from hell types, we're sysadmin types, and we're the ones, you pick up the phone and call them and you go, they're going to blame me for breaking something, or the dev team is going to actually cry out and
07:41
say, here's the guys that can't install our latest version of the application and they hate us and we cause them all their problems. But every single thing that we do in the IT world is built around a relationship with other people. Too many of the conversations I've had over the years in IT
08:01
have consisted of what I like to call the blame storm. So the blame storm is where instead of ascertaining what the actual problem is, the first thing you do is work out who can I blame for the problem and who can I make sure, how can I make sure it's not my fault. And there's a famous American phrase, cover your arse, and that's
08:21
an expression that gets used quite a lot in enterprise IT shops, is you're attempting to rather than solve the problem, you're attempting to be the person that is not at fault. The only way to change that mentality, to change that conflict between teams is to build relationships. And there's a lot of
08:42
reading you can do about how to build relationships and a lot of it's touchy feely stuff. I now work in America where everyone talks about their feelings all the time. I'm Australian, we don't. We just get drunk and hit things. But generally speaking, it's about engagement and it's about engaging early and often. So if you imagine the life cycle of a project,
09:01
a life cycle project consists of if you work in a business and the business says we would like to build a product or a service and that product or service is going to talk to a new bunch of customers. And the marketing team come up with a marketing message and the business comes up with all the things they want to do. They throw over
09:21
maybe a few scratchings on a cocktail napkin that the marketing guys came up with between strippers and then hand it over to a bunch of developers. The developers go, we can't build this, it's impossible. That's their first reaction almost every time. They next go, okay, we can build this, but it's going to look a bit like this. They then build the application
09:40
there's a whole bunch of people sitting around and they go, this is really cool, we should use React here. That NoSQL stuff's really cool. And then someone says, oh wow, we better make sure that it runs on this and we better make sure it runs on the latest version of Ruby and oh my god, we can use this really cool library I found and let's install all these gems and vendor them in here and all of a sudden, and I'm picking on Ruby people because I am a Ruby person
10:01
but it's true of almost every environment. The next step that happens is that the team puts together this amazing demo and they demo it for marketing people. The marketing people say we hate it. They hire a couple of UI people and fix that and then the application people say we're done. Hand it over, let's toss it over the fence. And the people hand it over to operations people.
10:21
The operations people take one look at it and go, oh my god, how are we going to make this work? It doesn't run on our standard operating system, it requires all these libraries we don't have, how do we back it up? Oh my god, it's syncing data between three data centers across four continents and how do we secure it? The application architecture is completely different from our existing application architecture.
10:42
And they model through, they spend six weeks trying to work out all of the bits and pieces, they jury rig things, they write scripts, they build systems that, you know, especially unique systems, they stick application, they run it up first day, goes under some load and it falls over. We've all sat in that meeting where the system's fallen over,
11:01
there's 10,000 concurrent connections and the developers say it should support 10,000 concurrent connections, you guys haven't provisioned enough systems, you guys haven't worked out how to handle this problem. And everybody sits around and basically it's an exercise in trying to ascertain whose fault it was rather than fixing the problem and trying to ascertain how do we get here and how do we get out
11:22
of the hole. The key to getting out of that hole is instead of right back at the very start where the business has a particular need, the business says we want to create something. The people they should have gone and have a conversation with are both developers and operations. And if those people are in the same room having that conversation, then all of these little things which are called in IT world, non-functional requirements.
11:43
So non-functional requirements are little things like security and backing things up. And they're equally, ironically called non-functional, but equally as important to the process of building an application. So if we start from the beginning where both sides actually have a conversation, actually get involved
12:00
and talk to one another, then we actually have an opportunity to develop applications and products where everybody gets what they want out of it. So how do we do that? To start to do that, both sides need to have détente. And my French accent is terrible, it sounds much better when you actually have a French accent.
12:21
But détente is about both sides need to understand that they're all in the same problem. We're all in the same problem space. We all have the same issues we have to deal with. We all have the same bosses, the customers. We all work for the same people, we all get paid the same way. We need to actually get stuck into a conversation with one another
12:40
where we actually don't let our prejudices sort of colour the conversation. And you have to have that conversation. You can't send someone an email. I find the classic example of someone is wrong on the internet. And the reason someone is usually wrong on the internet is because someone has written an email
13:00
and someone else has read the email and gone I interpret it like this, or I speak English as a second language and I've read this email in a particular way. Or I miss the smiley face you suck at the end of that. And next thing you know somebody's having a conversation that consists of we're actually aggressively pursuing a point rather than actually having a dialogue.
13:21
And I can't underestimate you the number of times that I've solved simple problems by getting off my ass, walking halfway across a room, or getting onto a video conference and actually talking to the person involved. And building those relationships is a key part of any of the steps that we need to talk about in DevOps. So
13:41
the next important thing for me is process. So I've spent about watching the IT industry change dramatically over the last 20 years. When I first started the one system we worked on was a massive big mainframe, partitioned into production and development. There was a DRP mainframe and there was pretty much
14:00
we had a system and it did stuff. There was no client server, there were dumb terminals. I started to cut my teeth on 3470 green screens, 3270 green screens. That's changed dramatically. We went from a world where all of a sudden client server appeared. You were no longer managing one big system, you were managing thousands of small systems.
14:22
Throw in virtualization, throw in the cloud, if you're happy with the term the cloud. And all of a sudden individual sysadmins have gone from saying a 1 to 50 relationship with machines to a 1 to 5000 relationship with machines. That's how many hours people spend
14:41
doing sysadmin stuff back in the day, but certainly these days you literally, I don't know a sysadmin who isn't overworked. I don't know a sysadmin in a big shop who isn't constantly looking at new technologies, constantly having to look at new challenges, new products and having to deal with projects in far more projects than they're used to handling. The only
15:02
way to handle that is automate. I cannot underemphasize the importance of automation. The only way you're going to scale up to those environments is to actually manage to all of the things that you do should be done at the click of a button. The other thing is built-in redundancy and the expectation of failure.
15:21
For me the classic example of this is that the business comes along and they say we want 99.999 availability. And it needs to be redundant across all of our data centers and if anything goes wrong it needs to be up within an hour, whatever our SLA happens to be. And then they say
15:40
we want this magnificent Rolls-Royce solution but here's a budget that borrows you a Morris Minor. And that's something that we all have to deal with. So my assumption is that instead of starting from a position of let's build the most wonderful architecture in the world let's make the assumption that let's build from a concept
16:00
of this is going to fail at some point. So have the expectation of build a process and your tools around the assumption that things are going to fail and you need to build redundancy around them. The key part of doing that is to test things. Don't do your testing at the last moment
16:21
of your operation. Assume that at any point in time you should pull the plug and say this is broken, this is gone what's happened? How do I resolve this problem? The key part of that is understanding that if this is in a failure state how did it get there
16:40
and how do I reverse it? So you have the processes and the automation to say if it fails this will automatically heal it, make it better again. The other thing too is that remember that process and the various processes and steps across both teams, both development and operations is all part of the same life cycle.
17:01
So all part of the experience of running an IT environment. So if the development team is doing testing in their development environment if they're deploying applications then the overall it's part of an overall life cycle that ends with that application going to production. So any process you build should be something that should be transparent and open
17:20
to everybody. The key around that is tooling. And this is where the open source community has really come to the fore in the DevOps sort of scene. And the reason that that is so prevalent in the open source community is that there are tools like Puppet and CF Engine and Chef in the configuration management space. Control tier, RunDeck,
17:42
M-Collective, monitoring tools, security, testing, a number of people I think believe Lindsey Holmwood who came up with Cucumber Nagios talked to you last year about that. These are all perceived as ops tools. And the reason that I put the ops in the question mark is that
18:00
the value you get from having these tools is not limited to operations teams. Development teams have a key stake in building these tools. And the important part for example that we cite as an example of why use Puppet is that an application developer can build a Puppet manifest or a Chef cookbook or a CF Engine recipe and they can actually
18:20
deliver that recipe from lifecycle to its end. So they've cut their code, they've built their associated Puppet module or their Chef cookbook and it ships all the way through the lifecycle. So it gets the application development team hand that over to a testing team or hand that over to a production operations team who actually implement that live. And it means that you actually
18:40
have that cooperation, have those things like we are building the infrastructure, we're building this is how the DNS looks, this is how the backups work, this is how you secure things, all the way through that lifecycle. Deployment and orchestration, application teams particularly, for example big Java shops, there's a very clear way they like to deploy
19:01
and orchestrate applications. There's an important opportunity for developers and operations people to work together with those tools and actually understand what the particular requirements the development team has and what the requirements the operation team have and why those are important. The other thing is things like monitoring and testing is they're not
19:22
solely limited to one group or another. So the sort of data that you make available with your monitoring environment is a key piece of data that most of your application developers need to understand and troubleshoot problems. So if you actually go and ask them questions about what they want to monitor, what do they want to see, the end result is something that you both work with and both use.
19:50
The other piece of this is there's lots of things developers do that are useful to operations people. So smart sysadmins for a long time have been version controlling things, everything ranging from DNS configurations
20:02
to whatever happens to be that change of state you want to keep track of. The last couple of years you've seen lots more operations people understand tools like Git and Bazaar and the reason they've done that is because there's some useful things developers do about branching, about the way that you can actually have multiple people
20:22
collaborate on tools and collaborate on configuration and collaboration on code that's useful to both operations people. Following it from that there's also things like Agile and XP and concepts like continuous deployment that have emerged out of the development community that are useful things for operations people to know as well.
20:41
That's not to say that those concepts are perfect, not to say those concepts are things that you should immediately adopt in your operations team. John Allspar who was at Flickr and is now at Etsy talks a number of times about how Flickr used to deploy into production 10 times a day. Now a lot of people look at that and go
21:01
those WebOps guys are fucking nuts. There's no way a stable environment is going to be healthy if you deploy 10 times a day into that environment. That may not suit your particular environment but the ability to say go from having a six week deployment cycle and there's a number of banks and financial institutions and insurance companies that literally go weeks and weeks between deployments because
21:21
the processes of deploying an application is so complex, so risky and has such a potential to cause disaster that they're incredibly cautious and risk averse. There's a lot of concepts that come out of that Agile and continuous deployment things that can be adopted to cut down that time period.
21:41
Application, architecture and orchestration. Operations people are classically bad at documentation. I think the only time the network diagram tends to be up to date is usually after the post-mortem for an outage. There's a lot to be said for the way applications people build architecture, the way they
22:01
document architecture that operations people can learn a lot from. Another thing is testing methodology. This is something that's new to a lot of operations people. For most operations people the concept of testing things is monitoring. Is the SMTP port open? Is the DNS service running?
22:22
What's the load on this particular machine? Applications people think about testing in a whole different way. They think about testing in terms of is the application functional? Does the result that I input run through this method and output, does that result mean something? A lot of what someone like
22:41
Lindsay Holmwood has to say, for example, about Kubernetes is about understanding that testing things like is the SMTP service running is not that important. The reason it's not important is because it doesn't tell you anything about the function of the application. It doesn't tell you anything about what the business value of that application does. The SMTP port can be open, but if it's
23:01
not sending email and that email is not shaped the right way, whatever latest marketing blast that Oracle is sending out to everybody, if it's not shaped that way, then the service is valueless. You need to take your testing up to another layer, which is test the business functionality. From an operations perspective, you can learn the lessons that application developers
23:21
have, which is that you test the function and you return the result that you expect the customer to see. In the process of testing that function, in order to send a message via that SMTP, wow, the SMTP port has to be open. So you actually get all of the base level reporting, and I'm not saying you certainly turn off the monitoring, but all of the base level reporting, you get that as well,
23:41
but you also get to understand that what the customer is expecting to see, what the application is supposed to deliver, actually gets delivered. The last piece I want to talk about is continuous improvement. So in the IT industry, as I said before, we all have customers. The business is our customer.
24:02
It doesn't matter what sort of organization you work in, you have somebody who consumes the result of what you produce. Those people don't stop asking for things. When I first started out as a sysadmin, I think we fixed a couple of problems and I was really chuffed. I was really pleased. I fixed this problem.
24:21
It's awesome. The customers have been complaining for months that this thing didn't do what they thought it was doing, and it was generating a report. And they wanted the report by 7.30 in the morning instead of 8 o'clock. And I was really chuffed. I did a bit of fiddling around and the report generated, tweaked some SQL and the report generated much faster. And I sat back and I went, I can have a cup of
24:40
coffee and a cigarette, I've done a really good job. And about 9 o'clock I got an email going, yeah, we've got the report, it looks like this, but now we want it at 7 because we've actually ascertained that it would be really useful if this particular operations team in this time zone had it. And I'm like, fuck you guys! Where's my thanks? Where's my, you know,
25:00
you did a really good job. Where's my pat on the back? It never comes because customers are never happy with the end result. They want the next best thing. And fair enough too because they pay your salary and you work for them. And I get a lot of stick for this because I'm a firm believer in saying that we all deliver a service and the people that pay
25:20
for that service are entitled to get a decent service out of us. Obviously if they pay like crap then that's a whole different thing. But those customers don't stand still. They have requirements that keep changing. You cannot sit back and go, wow we've done something really cool here, that's the last time we have to worry about this. Because they will come back next week and say, well
25:40
deliver me this, I'd like a little piece more. Products don't stand still either. So businesses change, organizations like banks, I spent a lot of time working in banks 20 years ago, the concept of online banking completely non-existent. Nowadays I ask most customers to say with the notable exception of the United States, writing
26:00
a cheque is a foreign concept to most banking customers. As a result things become far more complex, they don't stand still and you need to keep on top of delivering things that are new, that are changed and that have unexpected sets of requirements. Technology definitely doesn't stand still either. Things
26:20
like virtualization have come from a rude shock to a lot of shops. Sysadmins have gone from saying we can manage 50 systems like this to managing that 5,000 system model. And your team doesn't stand still either. People get bored and there's nothing worse than a bunch of people who are bored and disinterested by their jobs. If you provide
26:40
people with the right sort of challenges, the right sort of things to learn, then you move forward quite rapidly and you're able to if you're able to improve on things, move forward and look at things in that sort of continuous improvement light then it's an opportunity
27:00
for both people to learn and people to develop. The strike often strike hard and be aggressive is that you can't sit still in an environment like this, you can't sit still and wait for things to happen to you. So the continuous improvement piece needs to be think about what would go wrong here, think about the failure you're trying to resolve, think about the issues you need to deal with and deal with those first. The more
27:22
problems you solve up front, the more time you spend putting into making things better and my boss Luke Knees has a phrase, the faster you get to the pub. And I'm a firm believer that if you address things aggressively and quickly and actually think about the problems first,
27:40
you're able to get to the pub much faster. The big thing about DevOps for me is that it's a cultural change. It's not a series of technologies and tools and processes alone. And cultural change is hard. People don't like thinking about things differently, they don't like looking at things in a new way.
28:01
The not invented here syndrome is very strong in a lot of organisations. People hate change. Even little changes, people don't like little changes in their life. We like routine, we like things to be normal, we like things to be consistent. And as a result when you make change, people don't like you. So I've worked in a lot of environments
28:20
where we're introducing changes like DevOps where we know that the coordination between teams the first thing that comes out is why do I have to talk to him? That's the guy that wrote that code that really sucked and I got woken up at 4 o'clock in the morning with a pager. And as a result people will respond to you in a fairly aggressive
28:40
way when you make changes like that. That fear of change is largely irrational. It's largely a I don't know where this is leading I'm largely going to be concerned about my future, where this is leading what happens to me. The best way to deal with that fear of change is to listen to people, actually have a conversation with them about what they're worried about
29:00
what's going on, provide concrete examples of how things help, how things change. I think the classic example for a lot of people is presenting to them this solution will actually improve the availability will result in a better SLA actually demonstrate how the change will work. The other thing is you can't just stand up and make pronouncements.
29:23
You have to actually be in the guts of things actually solving problems is part of the problem part of the solution rather than the problem. The worst thing in the world for any IT shop, the classic example is the development team who don't get woken up at 4 o'clock in the morning.
29:41
Who are able to come in at 10 o'clock or whatever happens to be and they see the ticket and they go, oh, something's broken, oh well. Whereas the development team have been the operations team have been awake for the last five hours trying to resolve that issue. If you want to make things like cultural change and that sort of space give developers pages, make them on call
30:00
make them responsible for service levels and availability that they wouldn't otherwise be. So I also wanted to talk a little bit about some of the responses to the DevOps movement. I'm very fond of the classic Danger Will Robinson or Robbie the Robot talking about
30:22
whatever unknown danger is around. It's also my favorite Thomas Paine quote. When we're planning for posterity we ought to remember that virtue is not hereditary. So when I first started out in the IT industry the guys who taught me about ITOps were all in their 50s, they'd all come from that mainframe
30:41
world, they were very serious guys who were like availability is the key, big iron is the only way to do it. And the virtues they taught have not passed on. There's a lot of people who have come into the operations and development world with a very different perception of how availability, how operations works.
31:00
You need to actually provide that concrete set of planning in order for people to continue to understand what's important. So some of the classic responses to DevOps that have been is good system sysadmins have been doing this since they came down from the trees.
31:21
And for me this is really something that I look at in fairly good horror. There is a very small percentage of the sysadmin community and the development community who look at this stuff and go we've always done this. And those people sort of go, shrug, as a result of that I'm not going to make any changes outside of my world. So the other part they completely ignore that 85%
31:42
of the community that has no idea that they're a better way of doing things, no idea that automation will make you free, that there are ways of, different ways of looking at things. The next response is that my organization is different, that can't work here. And that, as I said earlier, that whole unique snowflake world that's gone. There's no
32:01
way that you can look at an environment and say we're going to manage everything differently, we're going to be different from everyone else. That thinking is something that will become, that comes from that whole bastard operator from hell view of the world. And that's no longer something that's feasible in a modern environment.
32:21
The next one is all about one particular group or the other. So I concentrate on operations. That's the area that I work in. And the response here is that operations people are going to have to become like developers. The developer, we're espousing that development methodology is the key, the way forward.
32:41
And for me that's simply not true. The thing for us is that both sides need to learn from one another. So for me it's understanding that both sides need to actually collaborate, both sides can learn from one another. And someone actually posted you're a bunch of European wankers, which I was a bit horrified by because I'd love an EU passport
33:01
and I didn't have one. So the other danger is that where DevOps ends up. And for me as I said earlier, there's been a couple of products that have said they're DevOps certified. The new hotness is DevOps and therefore my product
33:21
does DevOps. I've also seen Forrester and Gartner both have DevOps events. I know speaking to Patrick and Chris, neither of them were invited to that. And it's always fascinating when analysts, some of you may understand the analyst market is very soft serving. The analysts get a question from customers saying I've heard about this DevOps thing, what's it all about? And they all go around and ask all their questions
33:41
of their other customers and they feed the result back and it becomes a self-fulfilling loop. The key thing for me is that around DevOps and understanding how to avoid that for DevOps is to make it more than marketing speak. So actually put this stuff in practice in organisations to actually solve problems using DevOps concepts to solve problems by actually making operations
34:01
and develop work better together. The other thing is it's very easy to pay lip service to this. It's very easy to say we're going to have a role called a DevOps person and we're going to have a DevOps team. It's not a role, it's not a thing you can build a team around. It's about a different way of looking at IT and IT operations. It's also a concept that's
34:20
very easy to cause disenchantment amongst people, mostly because people are uncertain about those changes, they're uncertain about how the world will change for them in that sort of DevOps environment. It's very easy for people to lose faith in the organisation that they work for because they don't feel that they're going to get anything out of this. And that whole disenfranchisement of
34:40
people is something that I think every organisation struggles with and it's something that I think DevOps has a real opportunity to change that frustrated mentality that a lot of sysadmins have and the frustrated mentalities a lot of developers have dealing with those sysadmins. And that change is around that whole learning, understanding
35:01
from each other and understanding what's going on. I've found the last few years that I spent a lot of time cutting code where I didn't used to. I used to spend a lot of time writing batch scripts. I now find myself cutting Ruby code, learning things, learning new things, and I get a lot more pleasure out of my role than I did previously. And the last one, anti-disestablishmentarianism
35:21
I just wanted to put in because I like anti-disestablishmentarianism. So I think I've spoken very quickly and I think rather poorly unfortunately, but I did want to ask if anyone had any questions before we go on?
35:50
When you look at the role of development and operations in your environments what are some of the tangible things that change about their roles?
36:00
Like I know for Agile principles for example I mean things like between Agile and testing tend to blur the lines. What about in DevOps? Does it change the rela- Does it change some of the activities? Does it blur some of the lines between development and operations? So I think classically the whole, I think configuration management is the prime
36:21
example there. So the we used to spend a lot of time doing things like creating users, installing packages, managing services, monitoring things. Those are not cool things. Those are boring tasks. The thing about that DevOps allows is to say let's take all of that stuff let's make it automated, let's make it something
36:41
that we don't have to worry about. Because the really cool thing I'd like to do is learning how to scale this particular application. Or I'd like to learn how the new NoSQL stuff that the developers have introduced works. I think the tangible outcome for a lot of organisations is that you get away with saying that all the boring shit, the stuff that has occupied your time,
37:02
fixing things, installing things, changing basic configuration things that is fairly boring and monotonous, you can actually work on the projects that are actually interesting. The other thing is that for a lot of organisations having that conversation with development teams means that you're actually in the guts of the development process. That for me is a far more interesting activity
37:22
than sitting around being an operator and watching the meters and running. If you run a whole bunch of people through and for example I have a couple of organisations we work for that have embedded operations people into the developers scrum. They can't necessarily cut the same code but they're part of the actual development of the application.
37:41
By the time the application gets to production those guys are like this is really cool and they become evangelists for the applications developed. Not only do they understand the application better they understand the metrics and the basic concept the developers put into the thing but they actually care about it. I think that's a really important I've seen too many applications
38:00
that get thrown over the fence and you go I don't really understand what this does I don't understand why it breaks all the time I don't understand why it performs like shit but if you're actually involved from day one you can go well we made a compromise here, we actually decided to wire it like this instead of this because of this particular reason and you go we can make it better by like this, this, this but here's the caveats we had.
38:20
I think a lot of that sort of knowledge is not that sort of knowledge is not something that gets passed on otherwise. When will your big and explicit book about Puppet will be finished finally? So I'm I'm hugely jet lagged and that
38:40
question has been asked 17 times since I landed on Belgian soil March or April maybe? No unfortunately I think I spend about 20 hour days at the moment and it's almost done almost. Two chapters left and I had the help of a very good colleague
39:00
who was pushing along Hello obviously one of the keys behind DevOps is a better collaboration between the development
39:21
and the IT operations guys. Coming from the IT operations field yourself, what would you expect from a development team in order for you to relate better to it? What could the developers do to ease your way towards this collaboration? So I talked about embedding operations
39:41
people into Scrum or into your Kanban or whatever developers do. Do it the other way around as well. So drag development people into incidents and outages. If you have an operation stand up, drag the developers in. Make them understand the sort of problems you have. A lot of developers operate on the principle that
40:01
we've built something, it's really cool, it's really hot and then we piss off down the pub and the operations people have to manage it. If they actually understand that there are implications and consequences to their choices that if they choose a particular application architecture, it's going to have an impact on the operations team at the end of the day. The number of developers I have a conversation with
40:21
who go, oh you didn't tell us that you don't do things like this or it's hard to package this or this set of Java libraries are like this. If you actually have teams involved on both sides from day one of a project there's a far greater appreciation for how that works. Developers also
40:41
stuff like developers are like you have a particular operations problem we spend to be fairly metric centric people, operations people. We're interested in metrics, we're interested in measuring things. Developers are also interested in that sort of thing and they also have a set of things they'd like to know. When something goes wrong, something's performing badly they have
41:00
that desire for that information. If you have a conversation with a developer that says let me tell you about the metrics I have you can tell me about how your application is instrumented and then I can tell you what I'd like to have added to that instrumentation, you can tell me what data you'd like from my end. Those sort of conversations are really powerful and I found that a lot of developers are sort of stunned by the
41:20
I guess the depth to which operations people use metrics and they sort of have underappreciated our perception of that. So I guess there's some conversations around that are really healthy and promote that sort of cooperation. So this sounds really interesting
41:41
and it sounds like what we did actually, what I've done actually in some of the companies I've worked and it works really well but does this really scale at all, does it scale to bigger teams of operations and development people because that's one of the main problems when the teams become big enough
42:01
it's really hard for them to mangle. And the other thing is mostly if you've done this but I have this very weird observation that development people think for example that CPU, memory and hard drives are infinite and most of their resources are infinite and this is one of the main issues
42:21
with working with deploying their software so if you manage to teach anyone that that's not exactly right. I think the challenge and I certainly, you can see a lot of pushback around DevOps and the fact that it works best in small teams for example people say it's much easier to do this if you haven't
42:41
got, you're not an enterprise and you haven't got 500 people. I think that's true, it's easy to have relationships in small teams. I think the larger teams, you do things like you move whole groups of people through different teams and different organisations, you shuffle people around you keep people actually learning new things, you don't
43:01
silo things, I think the key mistake people make is you build silos, you build a group of people who are all the Java developers and you build another group of people that are all of the WebOps people and then you build another, that is not only an unhealthy way of building teams but it's a way that doesn't promote that sort of cooperation and coordination.
43:28
He's cancelled his question. Hi, I have a quick question over here. When would be the exact perfect time to start doing
43:41
DevOps when you're doing a startup? When you're doing a beginning of a startup and you've found it yourself, you don't have that much capital, I can't see why for example, a whole bunch of things are useful but the whole configuration management and all that might just be a drag in the beginning because you really need to move quickly forward in order to get capital
44:01
and first clients and angel and whatever. So how would you approach that one? So I think Greenfields is far easier. So if you have a Greenfields environment it's far easier to introduce new concepts. People don't have the it's not invented here or we've always done it this way mentality in a Greenfields environment.
44:20
I think though there are real advantages to looking at something like DevOps in legacy environments. Legacy environments, we've all got them, we've all got lots of organizations have their collection of Rel4 machines that have been living there forever and they can't change or upgrade them. They're a bunch of Windows NT machines that they've ring fenced off from the rest of the environment.
44:43
Those things cost lots of money to maintain because they've become boutique or no one wants to change anything. There's real opportunities in that sort of environment to actually if you automate stuff in those environments to save a lot of money and to become a lot more efficient. But it's a lot harder to do because making those changes
45:02
tend to be systems like that tend to be very fossilized. They tend to be things that have the organizations become risk adverse about. We can't change that no one who works here will understand how that works anymore. But as a result the opportunities there are quite large but the risks are quite high. But Greenfields is certainly easier. If you're a startup from day
45:22
one, the classic example is Agile. If you start from day one using Agile it's far easier to go from a legacy development process to an Agile than going from a legacy to an Agile process. If you start from scratch there. Do you have a guarantee?
45:45
I'm in operation myself and we are in this not uncommon situation where all the applications all the applications that we are deploying were not developed in-house.
46:00
So is DevOps still relevant and how do you trigger this cultural change about automating things? Because we can only automate some baseline basic things like SSH configuration, VM provisioning and very basic stuff. All the
46:21
rest is too diverse and too complex to do automatic deployment. I deal a lot with customers that have outsourced stuff to India and China. It's really interesting. Outsourcing in a lot of cases gets a very bad name. It's the lowest bidder
46:40
who you've outsourced your stuff to and they have no interest in quality or changing things. I think that's the wrong attitude to take. The opportunity is there for you to infect those organizations. If they understand, and there's a couple of people here, I know there's one person here from India who works in a big Indian IT company who actually can tell you that those guys are interested in learning new things.
47:02
They want to work on cool technology and if you have an opportunity to infect them with a whole different idea about how to do things, I think those things will be picked up. There's big cultural barriers in any outsourcing arrangement, but I think those are also opportunities. It's also about demonstrating value. If you can demonstrate that it's cheaper for the organization to do it a particular way,
47:22
then they're going to jump on that because they want to deliver the service and make a bigger profit. If you can deliver you get something out of that, you get better availability, you get a better quality of service, you get a better SLA, and they get to do it cheaper or faster or whatever it is. It's a win-win situation for both organizations.
47:52
Hi. I luckily started two weeks ago and I'm responsible for implementing application monitoring
48:01
in a quite very big corporate Belgian environment, which already says a lot for people who know about that. I'm facing the fact that the operation people have a solar system to be able to participate in that, but on the other hand, they're saying
48:20
don't flood us with all your application level events and all the business process metrics that you would like to see on the dashboard for the business. Do you have any comment on that or do you just want to show some empathy? So I'm not sure I understood the question totally, but
48:41
I think again it's about demonstrating what so if you demonstrate was the question if I can paraphrase you've got, they're not interested in looking at the business metrics, they're not interested in looking at the application metrics? Well the developers
49:01
and the business are but the iOS people, they want to limit the number of events that will go through their monitoring systems. So what I'm trying to promote is to help them changing their mind in such a way that they accept that monitoring should go a bit higher level.
49:21
I think it is about demonstrating the value of it, so demonstrating that monitoring the CPU, the memory and the disk is not what's important about the application. What's important about the application is what business results it delivers. You still want to know about the other stuff but if all of those metrics are perfect, if all of those dashboards
49:41
are perfect, but the response time over here and the application doesn't do what it's supposed to do, then it's not working. So you need to demonstrate that if they have another picture of that it may not necessarily be the same metrics that all the dashboards and the business people look at, but a view of that and how it links to those other metrics that they care about, infrastructure metrics
50:02
that's a way that you can say this is the picture your customers get, this is the picture that they're seeing and they pay your salaries, they pay all of us and we have to work together in solving that particular problem or giving that particular view. I think it's probably I think there's time for one more question. Any more questions
50:22
here? Thanks very much.