Devops notes from the field
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 60 | |
Author | ||
License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/37378 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Producer | ||
Production Year | 2018 |
Content Metadata
Subject Area | |
Genre |
1
10
14
18
24
25
26
32
34
40
41
44
46
54
00:00
Configuration spaceVirtual machineDemo (music)Well-formed formulaBitGoodness of fitComputer animation
01:07
Inclusion mapIP addressBootingVirtual machineConfiguration spaceBuildingMedical imagingWindowDomain nameModulare ProgrammierungMultiplication signManufacturing execution systemPhysical systemDemo (music)MereologyBitConnected spaceiSCSILecture/ConferenceComputer animation
02:30
Line (geometry)Computer fileVirtual machineDirect numerical simulationSoftware testingError message2 (number)BitPresentation of a groupPublic key certificateWhiteboardExterior algebraField (computer science)Multiplication signSpacetimeQuicksortGreatest elementFamilyMoment (mathematics)Content (media)Focus (optics)Computer animationLecture/Conference
04:50
Configuration spaceDemo (music)ComputerState of matterService (economics)Data managementCodeMotion captureStack (abstract data type)SoftwareManufacturing execution systemServer (computing)Surjective functionCluster samplingSoftware testingIntegrated development environmentTime domainPoint cloudVisualization (computer graphics)HypercubeComputer-generated imageryStandard deviationSoftware developerVirtual machineServer (computing)Power (physics)Gastropod shellService (economics)DiagramComputer fileFinite-state machineSelf-organizationInformationComputer scienceLine (geometry)SoftwarePhysical systemRow (database)MereologyCausalityMultilaterationMathematical optimizationProjective planeFrustrationCodeData managementFamilyProduct (business)Process (computing)Order (biology)SpreadsheetStack (abstract data type)Different (Kate Ryan album)BitManufacturing execution systemPresentation of a groupStudent's t-testComputer animation
12:46
Stack (abstract data type)SoftwareManufacturing execution systemServer (computing)Cluster samplingSurjective functionSoftware testingTime domainIntegrated development environmentVisualization (computer graphics)HypercubePoint cloudComputer-generated imagerySoftware developerStandard deviationData managementInstallation artConfiguration spaceDependent and independent variablesModul <Datentyp>SQL ServerTime zoneService (economics)Firewall (computing)Local GroupDatabaseMiniDiscComputer fileLocal ringAxiom of choiceProcess (computing)Logical constantState of matterProcess (computing)Logical constantConfiguration spaceServer (computing)WordSummierbarkeitJunction (traffic)Datei-ServerTime zoneVirtual machineCartesian coordinate systemIntegrated development environmentFactory (trading post)Link (knot theory)Domain nameBuildingGene clusterData managementOnline helpState of matterSet (mathematics)Revision controlSoftware testingCodeWindowFamilyBitGame controllerPosition operatorStatement (computer science)Multiplication signSlide ruleMobile appUtility softwarePlanningTask (computing)Medical imagingDirectory serviceStandard deviationWell-formed formulaMathematical optimizationProduct (business)Point cloudComputer animation
20:08
State of matterIntegrated development environmentConfiguration spaceServer (computing)Common Information Model (computing)Social classProcess (computing)Decision theoryVirtual machineFlow separationLogicParameter (computer programming)Installation artRenewal theoryLemma (mathematics)ModemLattice (order)Asynchronous Transfer ModeTable (information)Hash functionInstallable File SystemPatch (Unix)Computer filePoint (geometry)Game controllerComputer programmingNeuroinformatikBitWindows RegistryQuicksortMonster groupWordCASE <Informatik>Bootstrap aggregatingMereologyParameter (computer programming)Configuration spaceType theoryData structureLoop (music)WindowProjective planeRule of inferenceDomain nameVirtual machineDescriptive statisticsCodeService (economics)WebsiteDecision theory2 (number)Firewall (computing)Cartesian coordinate systemScripting languageLatent heatProgrammschleifeFunction (mathematics)Integrated development environmentWritingBuildingProcess (computing)Software testingSoftwarePresentation of a groupModule (mathematics)Server (computing)LaptopMiniDiscSet (mathematics)Computer animation
27:31
Decision theoryParameter (computer programming)Flow separationLogicData conversionLattice (order)Installation artCore dumpComputer fileLine (geometry)ApproximationFunction (mathematics)Common Information Model (computing)Server (computing)Virtual machineNeuroinformatikRow (database)Core dumpInternetworkingWrapper (data mining)Macro (computer science)Computer fileModule (mathematics)Revision controlSet (mathematics)Configuration spaceLine (geometry)Escape characterMereologyBitSound effectComputer fontReading (process)LogicDecision theoryParameter (computer programming)Computer animation
30:03
Core dumpComputer fileCommon Information Model (computing)Function (mathematics)Software testingLatent heatConfiguration spacePublic key certificateMiniDiscBefehlsprozessorLine (geometry)LogicApproximationSystem callAsynchronous Transfer ModeSQL ServerPermianServer (computing)Connectivity (graph theory)Computer networkModemTime domainSanitary sewerDatabaseGroup actionExecution unitMaxima and minimaRandom numberAtomic nucleusRepeating decimalConvex hullComputer fileTime zoneSoftwareComputer programmingDependent and independent variablesConfiguration spaceManufacturing execution systemLine (geometry)BitGame controllerDialectDomain nameServer (computing)Parameter (computer programming)DatabaseVirtual machinePoint (geometry)Scripting languageMathematicsComputer animation
31:48
ApproximationLogicLine (geometry)Common Information Model (computing)Core dumpComputer fileFunction (mathematics)Configuration spaceSoftware testingLatent heatSet (mathematics)Public key certificateBefehlsprozessorRead-only memoryMiniDiscVirtual machineMetreConfiguration spaceComputer fileVirtual machineBuildingPublic key certificateSoftware testingLatent heatSet (mathematics)LogicDescriptive statisticsLink (knot theory)Computer animation
32:45
Virtual machineSQL ServerTime zoneServer (computing)Perturbation theoryDatabaseBitConnectivity (graph theory)Connected spaceWindowComputer virusCartesian coordinate systemComputer fontInternetworkingSet (mathematics)MereologyLocal ring1 (number)Selectivity (electronic)Lecture/ConferenceComputer animation
34:33
Programmable read-only memoryClient (computing)Installation artContext awarenessPower (physics)MathematicsMatrix (mathematics)LogicCommon Information Model (computing)Statement (computer science)Computer fileVirtual machineConfiguration spaceData managementQuicksortComputer animation
35:17
Line (geometry)BuildingConfiguration spaceModule (mathematics)Group actionPasswordState of matterMountain passMessage passingMetreGauge theoryFile formatPartition (number theory)MiniDiscCodeVirtual machineConfiguration spaceVideo gameMiniDiscState of matterComputer fileBitMultiplication signStandard deviationPasswordProcess (computing)Representation (politics)String (computer science)Parameter (computer programming)Set (mathematics)Social classLine (geometry)Object (grammar)CodeFunctional (mathematics)Statement (computer science)FlagDistribution (mathematics)User interfaceWritingAdditionGroup actionFile format1 (number)Point (geometry)Type theoryModule (mathematics)Server (computing)MereologyInformation securityComputer animation
41:27
Configuration spaceModul <Datentyp>Error messageModule (mathematics)Revision controlUniqueness quantificationProcess (computing)Physical systemTotal S.A.Social classScripting languageGroup actionFirewall (computing)Public key certificateHypercubeBuildingComputer-generated imageryComputer fileMiniDiscInstallation artMoment of inertiaBargaining problemBroadcast programmingAddress spaceBootingCommon Information Model (computing)Software testingVertex (graph theory)BlogParsingMessage passingWeb pageSoftwareVirtual machineComputer fileServer (computing)Scheduling (computing)Bootstrap aggregatingPublic key certificateLoginBitVirtual machineProcess (computing)Scripting languageResultantType theoryRevision controlSet (mathematics)IP addressConfiguration spaceShared memoryModule (mathematics)BootingDemo (music)Installation artWeb pageTask (computing)Ferry CorstenSocial classPatch (Unix)Point (geometry)Computer programmingCASE <Informatik>Firewall (computing)SoftwareMiniDiscCodeMultiplication signRemote procedure callEvent horizonMessage passingSoftware testingComputer animation
47:37
Virtual machineSoftware testingVertex (graph theory)BlogParsingScripting languageMessage passingWeb pageSoftwareRevision controlSystem administratorInformationTouchscreenIP addressPasswordLine (geometry)Computer programmingVirtual machinePower (physics)Domain nameComputer animation
48:32
Electronic mailing listClient (computing)Direct numerical simulationVirtual machine1 (number)Domain nameView (database)Server (computing)Object (grammar)Multiplication signStructural loadComputer animation
49:55
Virtual machineScripting languageComputer fileLoop (music)Electronic mailing listProof theoryDemo (music)Module (mathematics)CASE <Informatik>Domain nameElectronic signatureGame controllerAntivirus softwareComputer animation
51:18
Virtual machineSoftware testingVertex (graph theory)ParsingBlogScripting languageMessage passingWeb pageSoftwareRevision controlCommon Information Model (computing)Latent class modelComputing platformParameter (computer programming)CodeParameter (computer programming)Slide ruleCodeError messageComputing platformConfiguration spaceLoginComputer fileService (economics)Computer animation
52:50
Coma BerenicesJSONXML
Transcript: English(auto-generated)
00:10
Welcome. You all having a good conference? Looking forward to the party? Good. So this session is entitled DevOps Notes from the Field.
00:23
I'm James O'Neill. My contact details are up there. I worked for Microsoft for quite a long time, and then I spent a couple of years working for a Formula One team, and I'm now working for myself. So when people ask what the Mobula bit is, Mobula is a thing very similar to a Manta Ray,
00:43
and that's the reason you'll see references to Flatfish around when I run my demos. Now it's traditional that a show starts with an agenda slide. I'm going to break with tradition, and I'm going to go straight into the demo. The reason for this is while you were arriving, I was setting up a brand new virtual machine.
01:10
The earlier rivals will have seen this machine boot up for the very first time. Nobody's ever logged into it. I don't even know what name's been assigned to it in the Windows boot up process.
01:22
I know the image it was built from. I know the unattended XML that was used to build it, and Hyper-V has very kindly told me that it's got this IP address ending in 84. Now I've got another virtual machine over here. It happens to be a domain controller, and I'm using this to push out my DSC configuration.
01:43
So you can see up here we have the name of the machine. It's going to be called Warwick Node. It's going to join a cluster. Mez, when you see that a couple of times in the presentation, the system I've been building is what's known as a manufacturing execution system, so those are the references to Mez.
02:00
We've got some iSCSI connections to make clustering work easily on a demo system, some cluster parameters, and some software packages we're going to install. Now, when we do this for real, there are about 14 packages here. I'm just trying to keep this simple and just install one of the packages that's part of the manufacturing execution system.
02:21
So I'm just going to kick off the DSC part of this, and I'm just going to shrink this part up so that we can see a bit more of it. Now, this contains both a long pause and an error usually, so I'm not going to panic if this stops for a few seconds and says, I'm doing something, but just be patient.
02:48
So what we've done already is we've built a MOF file for this new machine. We've also built a test file to go with it, and we've copied, you can just see about three
03:01
lines up from the bottom there, we've copied about 1,000 files over there that are things that this machine will need, okay? We've noticed that we don't have to update PowerShell. This obviously matters a lot, because if you look there, it comes up three times in the space of five lines, and what is now happening is the machine's actually creating a certificate
03:25
and assigning that certificate to DSC. Now this is the bit that I have to kind of explain a little bit and just stand here. So while that's happening, you may have noticed already there is a bit of a DSC focus.
03:43
I need to tell you a little bit about the background to this session, because Richard Sidway up the front here, our content manager, was reminding me that I had said I would speak at this conference when I was tied up with an awful lot of family strife. And I said to Richard, oh, I can do a session for you.
04:02
What I'm doing at the moment is quite interesting, so I can do something that sort of notes from the field from this DevOps role that I've got, and so I think Richard put in DevOps notes from the field as a placeholder, and some of the alternative and witty titles for this presentation kind of went by the board. So it's not called how I built a monster, though it might very well be.
04:25
It's not called upgrading to mediocre, although it might very well be. It's not even called how I stopped worrying and learned to love thousand line MOF files. And trust me, the MOF files are big.
04:40
So the machine that we're starting with, you can see there, it didn't have a proper DNS. It's figured out where its DNS should be. It's now trying to join the domain, and this is where I said this ends with an error, because if I go and minimize this, we should find that the machine called Warwick has just rebooted.
05:01
You can just see it there in Hyper-V Manager, and you can see it's just coming up again. So it's joined the domain, it's rebooting, it's going to do a bunch of interesting stuff, we hope. So, I've explained how we came by the title of Notes from the Field.
05:27
I said I was involved in a lot of family strife, which is an odd way of leading on to talking about my sister. But my sister is actually a professor of computer science down in California, and she gave me this quote which she tells her students when they're going to do a presentation.
05:44
And this has been worrying me ever since I started putting this presentation together, because yeah, I did something, and I want to show someone, and maybe I just ought to tell my mum. So I hope you're going to care a bit more, because trying to explain this to my mum might be a bit on the tough side.
06:02
So the agenda proper, like most Notes from the Field sessions, we're going to look at what was the problem, how did we solve it, and what did we learn on the back of that. Just a little bit of my opinionated stuff as a preamble to that.
06:23
I keep seeing the same thing in the last four different places I've been. You know there's an old joke, and it goes something like, there's a balloonist and he's lost, and he descends, and he sees some people below playing golf, and he shouts down,
06:42
where am I? And the golfer shouts back, you're halfway down the 17th fairway, and he says, you work in IT don't you? Yes, says the golfer, how did you know? Well, the information you gave me is 100% accurate and totally blooming useless.
07:02
But somebody added a two line appendix to that old joke, and the golfer yelled back, and you work in management don't you? Well, yes, says the balloonist, how did you work that out? Well, says the golfer, you don't know where you are, you don't know where you're going, and now it's my fault.
07:26
I've been in one organisation where if you said to them, what services are we delivering, you've got a blank look. If you said, to whom? Oh, we don't know who uses that file server, you can't shut that one down, it might be used by, and there's a long story I can tell about that.
07:45
I went to another organisation, and I made myself very popular there, because I drew a Visio diagram. And that Visio diagram captured all the servers that were used in delivering the service that these guys were supporting.
08:02
And the previous Visio was about four years old. And the fact that I captured everything, they loved it. If any of you bumped into me earlier on in the week, and I was wearing a t-shirt with some power shell on it, one of the guys there bought me that power shell t-shirt, they really liked it. And we don't know what state machines are in, and how they got there.
08:23
And this is really one of the things that has been the biggest frustration of the place where I'm now working, which is Jaguar Land Rover. How do we get there? Well, if you've heard of optimism bias, we always think we can do it more quickly than we really can.
08:45
And what then happens is we get rewarded by what we seem to deliver. So we have to be seen to deliver everything, and we can't be late, and we know that adding more resources late in a project will just make things later.
09:00
So we carry technical debt forward. One of the first things that goes on the technical debt pile is documentation. I've also seen part of the cause being very poor management of vendors. You tell a vendor to do something, they don't actually give you a proper record of what they did, how, and so on.
09:26
I see lots of nodding around the room. Like I say, IT people just don't like doing the documentation, and there's a running joke in the place that I work, which is the last line up there. We're agile, we don't do documentation.
09:41
And yeah, it is a problem. So, to automation. Jeffrey talked a lot about automation. If you automate a mess, what do you get? You just get a mess more quickly.
10:02
So my kind of agenda when it comes to DevOps is that it needs to be more than just doing it fast. It needs to be doing it better. Now, you all know the old triangle, don't you? You can have good, fast, or cheap.
10:20
I want to do good and fast, and I don't want to be cheap. I joke that I'm not a freelance, I'm a mercenary, so I want those two things. And if we're doing this much-vaunted infrastructure as code, then what we want to be capturing is what we got,
10:41
ideally why and on who says so, and how it got there. And I don't like infrastructure as code, because when I talk to management, management hear code and they all kind of do this. What management, what do management understand? Money, yes?
11:02
No? There's a thing on the end there. No. Excel files. Give it to them as a PowerPoint or in Excel and actually you make some progress. So my joke for this is I do infrastructure as a spreadsheet.
11:26
What brought me into this job was they wanted repeatability. So what repeatability did they want? I mentioned this thing, MeS, this manufacturing execution system. I'm working for Jaguar Land Rover.
11:42
The manufacturing execution system basically takes all the orders for cars and says, right, this car is going down the production line now, it needs this seat put in, it needs this different stereo, it needs this and this and this, and it tells the people beside the production line what they're supposed to do. It's made up of a lot of very complicated bits of software produced by GE
12:04
and then on top of that we have stuff that Jaguar Land Rover code in-house. And we need to put that software stack onto servers that have been built for us by our off-shore vendor. I say off-shore vendor very, very carefully
12:23
because they are also the owner of Jaguar Land Rover. Jaguar is owned by Tata, Tata Motors is the Jaguar bit, Tata Consulting Services is one of the world's big outsourcing vendors. Because of their culture, a lot of their philosophy is not to automate stuff
12:47
but to solve problems by adding labor. And that isn't always helpful. So we need to build that on servers that they've built. We also need to build, and by the way, we're not allowed to build the production servers or to set up clustering.
13:07
Adding the cluster name to Active Directory is something that's reserved for certain people within the IT department. We have to build test environments and we might be building our own domains and our own SQL servers in those test environments.
13:24
The SQL servers might need to be clustered, so might the application servers. And those test environments might be on Hyper-V or VMware or in one of those wonderful master strokes they decided they'd put some of these in the cloud and they'd use Google Cloud.
13:41
Now I don't know why either. Azure I could have understood, Amazon Web Services I could have understood, Google Cloud I still don't understand why but we have to work in that environment. And then those images have to be portable and so they have to follow all the corporate standards.
14:02
And they said to me, we built one of the pre-production environments for one of the factories overseas and it took us four months. And I said, wow, that must have been an awful lot of servers. And they said, yes, it was eight. So when I joke about upgrading to mediocrity,
14:22
you can see I wasn't really trying to improve on a very great position to start with. So what was the scope? I tried to capture as many of the bits that were in scope as possible. So I have to be able to create a test AD domain and when I create one of those test environments,
14:42
I use the domain controller as my file server for everything. It's just the infrastructure server. We need to be able to set up SQL, sometimes on clusters. We need to set up clustering for our applications, set up all the Windows apps, services, firewall, install quite a lot of PowerShell modules, not surprisingly.
15:03
Various utilities to make things helpful, other management tools and about a dozen of these apps from GE plus sundry other bits and pieces. We were joking at the start about my machines being on UK time.
15:21
Actually, that's one of the requirements. We set up a server and it was on Pacific time and the business as usual people said, oh we can't look after that, it's on the wrong time zone. So some of this stuff kind of creeps in. Now I don't know if any of you, some of you must have been in Gail's session earlier on today
15:43
and he had a great quote and I wish I'd written it down about learning stuff as you go and it was from the thing about agile is dead long live agility. That puts this slide a lot better but normally as you get scope creep in the work that you're doing,
16:03
that's a bad thing because you're having to do more stuff and you go oh no, I've already over promised what I can deliver in the time because I suffer from optimism bias and now someone's giving me more stuff to do. Well here scope creep is good because all the things that you learn about the configuration
16:22
if you can get those into your centralized configuration that's one less thing that might not be configured correctly. It's one less thing that somebody else on the team is RDPing into a server and manually configuring. So for me that process of constant little refinements
16:44
and learning about things that we've missed that's actually a really positive thing and that's kind of my take on being agile that I'm always adding little pieces to this because somebody comes along and says oh, could you just make sure that
17:01
and normally anything that begins could you just is like oh no. When I was at the Formula One team we started referring to tasks as Kibbe tasks, C-I-B-Y. C-I-B-Y stands for can I borrow you and Kibbe tasks were the things
17:22
that killed every kind of planned work because somebody would come along and say can I borrow you, I just need some help with this SQL server and you go hang on, I'm supposed to be getting link working why are you asking me about SQL server? So normally it's a bad thing. Here I've learned to embrace it.
17:46
So the mantra is basically describe how it should be. Henley node by the way if you notice the name the other one was called Warwick node these are a pair of servers in my test environment after junctions as you drive up the motorway going to work
18:02
so we're at Warwick and the next one up the road is Henley. It's quite fun when you get some motorway junctions where the names of places have got rude words buried in the middle of the name. So describe how it's supposed to be and isolate this from
18:22
the code that does it. So you can see we've got a statement here and that's very much like what I showed you when I was starting the build process off. Then well I wouldn't be able to get through without at least one Star Trek reference make it so.
18:41
And it doesn't stop there because you need to show that it's been done and how you did it. So you can see this is a snapshot from an earlier version but here I've gone through and I've run some tests to show everything was configured
19:00
the way it was supposed to do. Now just on DSC I hear people talking about DSC in a particular way and it sets my teeth on edge. It's desired state
19:20
not one setting not a couple of settings but it's everything. State is the sum of settings. Try saying that after a few drinks especially if you have badly fitting teeth. Actually going back to my family
19:42
my mother often talks about one of her school teachers who did have badly fitting false teeth and so I like the idea of her saying state is the sum of settings but I think if I do that for too long I'll get stuck like it in a session talking like that.
20:01
So don't use DSC for just poking one thing in because when you then go to the server and you say test DSC configuration it shows you the last thing that it set. So you set one registry entry. Well done. What were all the other settings?
20:24
So it's cumulative. Each additional thing gets added to the one configuration. That's important because it means you have to have somebody managing what that configuration is centrally.
20:40
Oh look, I managed to get another reference in there. The thing for this environment was they didn't want anything that involved installing more agents onto computers. They have an awful lot of firewall problems. We can't actually connect
21:01
to all the domain controllers in our domain because of firewall problems. So the idea of having pool servers on the network filled people with horror and then they got me in to do the job so it was going to be very heavily DSC. Now was anybody in Jeff Hick's session this morning
21:20
on DSC modules? He talked a little bit about writing both the old style modules and the new style modules and having to write DSC resources to work with PowerShell 4. When I got to Jaguar Land Rover I was actually having to use my laptop
21:40
which has obviously been upgraded, well it's Windows 10 so it's PowerShell 5 and the service had PowerShell 4 on them. So the first DSC things I tried to do did not work. I found that a MOF file built on a PowerShell 5 machine that MOF file will not run on a PowerShell 4 machine.
22:02
Now there may be specifics in the MOF file that I was building but I couldn't find what they were. So I then had to take my configuration and compile it on a PowerShell 4 machine and then I found that actually things I was writing into configurations wouldn't compile on PowerShell 4 anyway.
22:21
So the first decision that I made was right, we're going to just make sure we upgrade to PowerShell 5. That does complicate things because the first thing I have to do with the pre-built machines is I have to upgrade PowerShell on them and if I'm working with DSC
22:42
I've got a little bit of a bootstrapping problem there. So in the end we went with that. One of the things that I was told and was part of the initial brief on the project was don't try and introduce too much new software and in fact
23:01
if you can introduce no new software that would be a plus. Upgrading PowerShell wasn't seen as that it was just a Windows patch, so that was okay. And actually being a Windows patch complicates upgrading it even further. Now the next decision was about separating the parameters
23:21
and the logic. So I'll show you this. Don't try and read this as code too much because it's just too small it's too much of an optician's chart. But we have a configuration and there are lots of ifs and loops in that configuration. So I have one big configuration
23:43
that will set up a domain controller SQL clustered or not applications clustered or not and just a generic machine. That's when I talked about oh my word I built a monster. Okay. The other thing
24:01
sometimes when you look at a configuration you'll see that people have said we want to install these Windows features so you get 10 segments in the configuration that say install this one, install this one, install this one. Well I've said take the features that you want added and removed spin them out to the configuration file
24:20
and then just do a for loop to install each one. This kind of structure was what basically didn't work properly without PowerShell 5. Then the configuration data goes in its own file so you can see here we can have simple values,
24:41
arrays, hash tables all sorts of bits and pieces and then I run the configuration and you just say run it with that switch. Now that will spit out a moth file for each machine I then put those moth files through a script
25:01
called convert moth to pester so I read one text file and I spit out PowerShell files at the other end. I've become quite adept over the years at programs that write other programs even to the point of writing PowerShell scripts that output
25:20
Python programs. I should have just learnt Python it would have been easier infrastructure as Excel I actually read the Excel build sheets and spit out a configuration file if I just drop out
25:41
the presentation for a second go on. Last week we built some servers in one of our sites so here's the description of the service we had to build you can see it goes on a bit
26:01
okay. But somebody gave me that which contains the IP addresses the names cluster names they have this lovely naming convention by the way of naming the cluster after the first node in the cluster and I struggle to understand that one but there you go
26:20
what disks we have all of that stuff so I read that and I crunch it this so that very first column in there was that machine and those are all the settings that we're going to apply to it
26:40
so you can see firewall rules in there cluster name cluster quorum disk what type of quorum it's going to be all of this stuff I'm not sure that I can make the font bigger
27:01
but yeah you can see there's an awful lot of stuff goes into the parameter file there are about 70 parameters but firewall rules to add is one parameter okay so you can see in each firewall rule to add there is one of these so the firewall rule
27:22
is a hash table in this case so that's what we actually get from the spreadsheet and the
27:41
ah very good question what am I using to get the settings from Excel there is a very very good module out on the internet it's in the PowerShell gallery and you can get the latest version off GitHub it's simply called import Excel and I've done some work to add to the export part of it
28:02
and basically it allows you to read xlsx files without having Excel on the computer so it's brilliant to put on servers but it also I used parts of it to give me the effect of an Excel macro from PowerShell so I could basically say
28:21
read down to here because in this row you've got all the machine names and then you know that the if you read down the columns you'll find the values for each of those machine names so I used import Excel to do it it's a wrapper for somebody else's dll that processes xlsx files
28:42
it's written by a guy called Doug Fink who I can't speak highly enough of but go and download it if you only take one thing away from this session getting import Excel out of the PowerShell gallery wouldn't be a bad thing to learn so the first decision we made was around
29:01
PowerShell the second one was enforcing this thing of separating the parameters and the logic and being able to just say right go and look for any files that are called config.ps1 and basically if you've got one of those that's been spat out just add that to the nodes in the configuration
29:21
so we ended up with three core files one of them is called DSC baseline I wasn't joking about the size of the MOF files by the way they typically have thousands in them they're typically
29:41
a little over a thousand lines and here is the configuration itself so if you just again I'll make the font a little bit bigger oh did I not press escape come on
30:08
so up here I've got a definition of what all the parameters are that can be used you can see we've got node name and time zone etc. all the way down and then here
30:22
it's broken up into regions but you can see we've got things that are generic are about 400 lines there how to install a domain controller how to install the different bits of the MeS software that software in itself is quite painful to install
30:42
because a lot of the installation relies on response files being fed into a setup program and those response files change from machine to machine so I actually have a script resource in here which will modify those response files so that they're appropriate to the machine that we're going on
31:01
so for example the name of the database server that they need to connect to gets written on the fly so you can see here the configuration starts at line 86 and ends at line 2025 so that's about 2,000 lines of actual configuration I don't know if I were taking this on again
31:22
how I would break this up into smaller pieces there are ways to do it I wouldn't necessarily advocate making it one file like this but the idea of one single configuration managed at one central point is one I'd stick to
31:41
so that was the first one the next one is this thing called convert MOF to PESTA and it does exactly what it says on the tin it reads a MOF file and it spits out a PESTA test at the end of it
32:02
I'll show you one of those in a second and then the last one is called publish MOF and tests and again does pretty much what it says on the tin that sets up the configuration data adds any files that I've exported from Excel links up the machine certificates if they're there
32:23
and if we're building on things like Hyper-V we might have configuration data for Hyper-V in the machine settings so graphically it looks like that logic in one file description in another out come the machine specific files
32:42
now I notice that everything's gone quiet and things have been running for about 30 minutes I want to come back to this one later actually so over here this is my SQL server
33:01
and you can see that it's run a set of PESTA tests let me make the font a little bit bigger so you can see we've got all the basic things these ones that are, this one in yellow it said don't try and check this one because you haven't got internet connectivity so we haven't checked to see
33:21
we don't check to see whether we've got up to date virus signatures we don't check to see whether we're configured for NuGet because we're not connected to the internet but you can see we've checked that we've got British local and time zone we're registered to the right organisation because we've got flack for not setting that
33:40
on servers that were in Google Cloud we've installed Windows components and a select set of hot fixes and then we installed SQL server and somewhere in here it should say that we created a database and I've obviously skipped that bit somewhere
34:08
oh there it is so it created a database user called mezadmin it's created another one based on the local username and it's created a database for the Simplicity application
34:25
so we get all those checks as part of the convert MOF to PESTA and if I edit this just one thing that I'm really finicky about with PESTA the it statement in PESTA
34:41
the rest of it should read like a proper sentence so it ensured that this is not installed it ensured it installed this and so on so I can show that to somebody from management and say look these are the things that I'm checking and it's kind of human readable well it's management readable
35:02
so that is all produced automatically based on the configuration file for the machine so there's no James work gone into that now building any sort of significant configuration
35:23
you've learned very quickly if you went to Missy's session yesterday afternoon she had this great phrase about DSC being a toolkit which meant that you could create your own tools with it but the bad news was you had to well yeah that's about the strength of it
35:41
there are lots of additional resources online and what I've found is there are resources online that somehow or other I managed not to discover so one or two of the resources that I created I could if I'd searched better have found something that might have done the job but some of the resources that you find
36:01
particularly on the PowerShell gallery they have two faults one is somebody was trying to solve a particular problem so they covered the bit that they needed to cover so there's a cluster module out there but it doesn't create cluster groups or cluster resources some of them work in a perverse way
36:24
the late Douglas Adams talked about there was a hitchhiker's guide to the galaxy computer game at one point and he described the user interface of it as user mendacious and I had to look up mendacious but believe me it's a wonderful way of saying
36:40
it's the opposite of friendly some of the tools, SQL does it actually some of the AD ones do the same kind of thing they assume that what you've put in your configuration is how something must be forever and ever so you put a password in if that password is ever changed
37:00
it changes it back actually the SQL one changes it not back to the original password but back to a secure string representation of the original password but the Active Directory Groups one for example will remove everybody that you haven't named from that AD group
37:21
so you want to add somebody to a group well you better know who else is in that group before you start so some of these we had to rewrite or I had to rewrite so you're going to have to get used to the idea that you'll either need to adapt or create some resources now Jeff's session on DSC resources this morning
37:41
he was saying if you know the PowerShell for it already the good news is you're just about all the way there the bad news is that you are probably thinking of one configuration path and you will have to cope with more deviations off that path than you'd imagined
38:01
so again I don't want to dwell too much on this if you want to know about classes Mike Robbins has got a session in the next time slot I think where he's talking about DSC classes I highly recommend that one specifically we provide a set of parameters we've then got a get function
38:20
which returns something of this type we've got a test function that says do we need to do the set which is the next bit it's not just is it in the desired state but if it's not in the desired state do we think we can get it there sometimes it's easier to return
38:41
we're as good as we're going to get from test than say no we're not in the desired state and then have the set part fail and the set bit will as I keep using make it so really important Geoff talked about this and I've just put it here as well absolutely pepper these classes
39:01
with write for both statements because that's your clue as to what's going on now I talked about things not following the happy path one very simple one was that machine that I've just brought up had a brand new disk in it that had just been created before I started the machine so what I want the machine to do
39:21
when it comes up I want it to check that all the disks are online and formatted so they've got to be initialized partitioned formatted away we go and that's great if the disk isn't online or it's read only oh well I better bring it online then and then I started having to deal with clustering
39:43
because there's another flag that you get on a disk object and it says this disk is in a cluster and it's not online for you the other node of the cluster has it you can't bring it online so now I've got to go back and change my code so having initially gone
40:01
oh I hadn't allowed for disks not being online now I've said always put them online now I've got to say always put them online unless you're in a cluster and then of course something fails somewhere along the line and the disks initialized but not partitioned and I hadn't picked that up the drive letters already in use or the prize one of the lot I thought everybody would format
40:23
or initialize disks as GPT disks these days so when I went to do stuff with clustering I assumed everything was a GPT disk offshore friends still partition everything or format everything or do everything as MBR disks
40:41
and the way that you find a disk in a cluster is completely different if it's an MBR disk compared with if it's a GPT disk oh and don't expect your disks to be consistent between one server and the next as to the disk ID so that really simple thing that I started writing has ended up having to deal
41:00
with all these other paths and this is what I was saying earlier on about you learn more and more as you do this it's an iterative process you won't know it all at the beginning you can't possibly so you just add to it as you go some of the pitfalls with resources you would hope that you could actually say
41:21
at the start of the configuration copy these files down because I'm going to need them later no the configuration has a look and says you refer to these types and oh they don't exist bang stop if you're using a pool server and remember we said we couldn't use a pool server there's a great opportunity
41:41
to get something wrong and add something to the configuration I know people have fallen into that hole if we've got the same resource defined in two modules you can't compile it and if you've got two versions on the same server when you go to apply the configuration
42:00
it says you didn't specify which version to use so I'm just going to abandon at this point so we created a process to remove all copies of the modules that we use and make sure we copied on six from the PowerShell gallery six that we built ourselves
42:23
I replaced the failover cluster module I actually did what Mike's describing in the session I was plugging before and I wrote my own for managing disks managing the firewall and one specific to this GE software 45 resource classes in the configuration in total
42:42
obviously we use some classes over and over again and there's about another 15 or so or 20 script items in there that we could turn into classes so it becomes quite complex in case you hadn't picked that up by the way so bootstrapping problems that we hit
43:02
well the first one I mentioned already was upgrading PowerShell if you try and apply a patch in a remote session you'll actually see the I forget what the letters stand for but WUSA which is the thing that
43:20
applies patches starts and then very quickly logs an access denied message in the event log you can't apply a patch from a remote PowerShell session we needed to get all the resource files on the other thing is a lot of these things need credentials to be specified
43:40
they're going to be my credentials unfortunately and so I want them encrypted with a certificate but how do I get a certificate from that machine initially so we needed to sort that one out so when we build on Hyper-V I mount the VHD file I create a machine-specific
44:02
unattend.txt file and copy that in I copy the other files that I need and then the unattend.xml schedules a script called bootstrap dsc and that runs at boot time if I'm presented with a pre-built machine I connect to the C$ share on it
44:20
and I run some commands in a PowerShell session so the bootstrap process basically says if you're not running PowerShell 5 schedule this program to run at boot time
44:45
yes schedule this program to run at boot time and install PowerShell 5 at that point it exits if we haven't got a certificate create one and apply it and export it
45:00
which is the middle bit and then finally we can say start dsc configuration now one of the tricks that I use is I make sure there is no configuration to start if I'm expecting to create a certificate so I can run this once and it will create the certificate I can run it a second time
45:20
and the configuration is there and if it gets to the end this is on 2012 R2 if I can get it to 16 I'll have achieved so many things
45:51
you can but remember anything that starts it if you're in a remote session so I couldn't use
46:01
this particular piece of code push dsc which is what you saw me run at the beginning so this connects to the new machine if necessary it does that it checks the IP address and specifying a set of credentials it checks for the certificate if the certificate is missing then it copies files to the machine
46:22
and creates and exports the certificate and you saw that happening if we're running PowerShell before version 5 we schedule that bootstrap dsc to run and then run it so we get around this you can't run it in the PowerShell session by running it as a scheduled task
46:41
and then while that's running we copy the MOF file and the PESTA tests over to the machine if we didn't have to upgrade to PowerShell 5 we can then just start bootstrap.dsc sorry bootstrapdsc.dsc and
47:01
the nodes get built with a set of PESTA tests so the PESTA tests are placed on the machine I also have a script which parses the dsc logs and I extract the stuff from the dsc logs and the PESTA results to an excel file
47:20
there's a plug by the way for that import excel module so every node writes two pages to a workbook and I've got more auditing that creates a workbook per machine so let's just go back to my demo machine because I wanted to save this for the end and I reckon I'm just beginning to overrun so here's my new machine so
47:40
let's go to log on to it and the first thing that I notice is it's joined the domain called domain.local and I've renamed the administrator to summit you notice that if I put the local administrator in it says oh you want to log on to the local machine no I'm going to log on as the domain administrator because I'm drunk on the power
48:03
my passwords incidentally have become lines from Monty Python sketches and working with a lot of people who are rotated in from India trying to explain the Monty Python based passwords is fun some days this grey screen tells me that I've never logged on to this machine before
48:21
because when I log on I get BG info that does that and it looks like I've got oh look I've got a failover cluster adapter I've got a cluster IP address and if I go to my programs you can see I've got the general electric license client which was in the list of things I've got notepad++
48:41
which was in the list of things I wanted to install so that all looks quite positive and one thing that I've not got to the bottom of that I wish I understood is why the very first time you run PowerShell on a new server you have to wait about a minute for it to actually become responsive
49:04
but I've got something that parses the JSON files that DSC leaves behind and turns them into PowerShell objects and the minus grid there will make that a grid view yes, oh okay
49:22
if you didn't hear that that wait is because it's doing a bunch of stuff in the GAC so that should have run that so here I've got my view of what DSC has done so you can see we saw when it was configuring itself you notice the machine name on this side is one of those randomly generated ones
49:43
set the DNS join the domain rebooted did a load more stuff okay and what I can then do is I can run my PESTA scripts
50:00
what I'm actually going to do is do the PESTA script and send it to an Excel file now because I'm slightly overrun what I've got here is the one that I created on the other machines earlier on so I'm just going to go to setup.xls
50:21
so this is what it looks like when it comes back and the nice thing about this module of DUGS is that it does more than exporting a CSV file you can see that I've got filtering here so I can say just show me the failures so you can see here this one failed to get
50:41
new antivirus signatures you can see also they're colour coded and what I've got here is a list of all the steps that in this case it's the domain controller that I've been running the demo from all the steps that it went through and how long it took them to run so I've got my proof that I've made it so
51:03
I've got my steps that I took to make it so and I can hand that over to somebody so I've now got closed this loop of actually being able to show how I got to where I am and where I am so
51:21
what did we learn that last piece the Excel piece I'm going to stand by really firmly if you can't show what you've done don't bother if you can show
51:45
if you come to download these slides afterwards it will say you can't if you can't show but it amazes me you think of if you went to get on an aeroplane and the maintenance log for that aeroplane was as good as your company's log of what it had done to the service
52:01
you wouldn't fly yet it's not something that's safety critical like that if you check a parcel that's been sent from one side of the country to the other you know every step that it took and probably the name of the guy that delivered it to your house is for a parcel get as much into that config as you can
52:20
I think I've done the thing about it being more a platform than a tool to death by now and the other thing that I learnt was my code is surprisingly robust most of my errors when I set this stuff up is because I've put the parameters in the parameter file wrong the code that seems to be more of a problem than anything else
52:42
so that's it sorry for overrunning thank you for staying