Scripting Big Changes With Small Risk
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 60 | |
Author | ||
License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/37406 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Producer | ||
Production Year | 2018 |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
1
10
14
18
24
25
26
32
34
40
41
44
46
54
00:00
Scripting languageUsabilityInformation technology consultingRevision controlPhishingCybersexGroup actionSelf-organizationOperations researchAddress spaceInformation securityVector potentialCoefficient of determinationChainEmailVector graphicsPoint (geometry)Communications protocolProduct (business)Game theoryData conversionGoodness of fitScaling (geometry)Shared memoryRight angleMultiplication signTwitterMessage passingSinc functionDisk read-and-write headOnline helpOffice suiteService (economics)Enterprise architectureMathematicsSystem administratorScripting languagePhishingSuite (music)Physical systemMoment (mathematics)CodeSelf-organizationBitEmailPresentation of a groupProduct (business)Video gameRevision controlData managementPlastikkarteProcess (computing)Operator (mathematics)Strategy gameChainTouchscreenComputer animation
08:00
Coefficient of determinationVector potentialChainEmailSelf-organizationVector graphicsPoint (geometry)Communications protocolProduct (business)Game theoryScripting languagePhishingOperations researchoutputShape (magazine)Function (mathematics)AerodynamicsProcess (computing)Scripting languageComputer fileService (economics)Process (computing)Absolute valueGame theoryEmailStaff (military)Boss CorporationPoint (geometry)Cloud computingLink (knot theory)Information securityMathematicsHypothesisPoint cloudImplementationFunction (mathematics)Medical imagingoutputFlow separationGoodness of fitRight angleBitHacker (term)Self-organizationCartesian coordinate systemSystem administratorMultiplication signDirectory serviceDependent and independent variablesHand fanFacebookIntegrated development environmentResultantKey (cryptography)FrequencyMobile appLengthSet theoryComputer animationLecture/Conference
15:57
Scripting languagePhishingPower (physics)Mobile appBoss CorporationEmailMathematicsLine (geometry)Scripting languageOffice suiteVector potentialElectronic mailing listGreatest elementInformation securityInheritance (object-oriented programming)AreaCartesian coordinate systemSelf-organizationRight angleRevision controlBitFunction (mathematics)Different (Kate Ryan album)Multiplication sign1 (number)Key (cryptography)Numerical analysisWindowIntegrated development environmentPoint (geometry)Power (physics)CodeSet theoryMotion captureComputer animationSource code
21:04
PhishingProcess (computing)CodeScripting languageMathematicsActive DirectoryComputerState of matterWebsiteComplex (psychology)Data structureBitLaptopIntegrated development environmentHypothesisData structureVideo gameDirectory serviceGodDemo (music)Type theoryVirtual machineComputer animation
22:57
Scripting languageDemo (music)Single-precision floating-point formatRight angleDirectory serviceData structureMathematicsProcess (computing)BitComputer animation
23:35
Scripting languageState of matterStaff (military)Scripting languageLaptopOffice suiteComplex (psychology)Variety (linguistics)Group actionType theoryDifferent (Kate Ryan album)InformationMathematicsRight angleComputer animation
24:25
Scripting languageActive DirectoryMathematicsComputerState of matterWebsiteComplex (psychology)Data structureMathematicsVirtual machineoutputElectronic mailing listObject (grammar)Endliche ModelltheorieTablet computerComputer animationSource code
25:27
Scripting languageMIDIData structureObject (grammar)InformationBitSelf-organizationMathematicsSign (mathematics)Coefficient of determinationVirtual machineDifferent (Kate Ryan album)PlanningBootingGoodness of fitSource code
28:05
BuildingScripting languageDemo (music)Object (grammar)MereologyState of matterLaptopMoment (mathematics)Scripting languageTablet computerGoodness of fitComputer animation
28:37
Scripting languageoutputScripting languageCodeOnline helpMathematicsObject (grammar)Tablet computerLaptopOrder (biology)Point (geometry)Sheaf (mathematics)Type theorySource code
30:01
Power (physics)Scripting languageMIDIType theoryCASE <Informatik>LaptopComputer fileProcess (computing)Different (Kate Ryan album)Data structureExecution unitIdentical particlesComputer animation
30:35
Scripting languageTwin primePower (physics)BuildingoutputFunction (mathematics)Object (grammar)Computer fileInheritance (object-oriented programming)Revision controlComputer animation
31:27
Scripting languageInformation managementExecution unitBuildingElectronic mailing listMathematicsScripting languageoutputComputer fileHydraulic jumpMultiplication signObject (grammar)Computer animationLecture/ConferenceSource code
32:34
MathematicsActive DirectoryScripting languageEstimationProcess (computing)Self-organizationBoom (sailing)MathematicsVolume (thermodynamics)Vector potentialDifferent (Kate Ryan album)CodeMultiplication signPlanningBitModal logicPeer-to-peerComputer fileObject (grammar)Point (geometry)Personal area networkRollback (data management)Right angleEnterprise architecturePerfect groupLevel (video gaming)Direction (geometry)Directory serviceGame controllerInformation securityComputer animationLecture/Conference
37:46
EstimationScripting languageGame controllerMathematicsObject (grammar)System administratorOrder (biology)Capability Maturity ModelMultiplication signAbsolute valueDirectory serviceData managementServer (computing)Numerical analysisFunction (mathematics)Water vaporInformationScripting languagePhysical systemEnterprise architectureProcess (computing)MultiplicationWhiteboardLocal ringRight angleSelf-organizationProper mapInterpreter (computing)State of matterHecke operatorAdditionNeuroinformatikComputer animation
42:59
Coma BerenicesJSONXML
Transcript: English(auto-generated)
00:10
Welcome to Scripting Big Changes with Small Risk. My name is Brad Sterkenberg as it says on the screen. You can find me via email at therealstirk at gmail.com and I can also be found on Twitter at therealstirk. So Scripting Big Changes with Small Risk,
00:25
we're going to get into it. We'll talk about what it really means for our lives with PowerShell and other things that we're doing. But before we do that, I really quick, I just want to let you all know that I'm super excited to be here. The concept that I'm sharing is one which has hugely helped me in my career, one which I think most of
00:45
us, in the back of our minds, like when we talk through this today, most of us are probably going, oh yeah, that makes total sense. But it's one of those things that if it goes unsaid, it oftentimes goes unthought. And I think it's really important that we have this conversation because we need to keep some of these concepts in the forefront
01:03
of our minds that we can really be more effective at the job that we're doing. Like I said, super excited to be here. I am going to jump into this thing. It is, the session is more conceptual in nature. It's not a technical deep dive like many of the sessions at the conference that you've been to. But let's go ahead and jump right
01:20
in. So really quick about me, I've spent 15 years in the private sector doing IT consulting for small to mid-sized nonprofits. About 10 years in the public sector doing everything from large scale operations and help desk, office automation, enterprise solution design and IT strategy. I've worked with PowerShell since version 2. There's probably
01:45
some people in this room that have been using it since it was just in Jeffrey's head. But I'm not one of those people. What's that? No, no, no, I can imagine. So I've been using it since version 2 and I found it to be hugely helpful in my career.
02:05
At first it was one of those things that it was kind of scary, right? You're taking away my gooey, you're taking away my checks, you're taking away a lot of things. And in the early versions, some of the niceties that we've come to love about PowerShell were not present. And so there was a lot of stuff that it was like, wow, all right,
02:23
so I'm going to pull the trigger on this thing. And I have no real clue what I'm about to do, but I'm going to do it because I'm in the battle faction, right? And I just write my code and it's ugly and I'm going to go. So I've worked with Microsoft products primarily, PKI, Active Directory, that's my sweet spot. But I've also done
02:40
some system administration and exchange and worked with the system center suite of products as well. And in the last year and a half of my life, I spent in the CTO's office at the state of Michigan doing strategy and things with the state of Michigan across the entire IT portfolio. A little bit less time hands-on with PowerShell, more time
03:04
really trying to convince the organization leadership how we can do things better from an automation perspective. Enough about me, right? So this phrase right here, this you could say it's for this methodology, when we're good, we're good, but we are
03:24
only human. We have a tendency when things are going well to believe that we're good. It's good. I'm just going to keep going, just plugging away at my computer, making a bunch of changes and everything's great. And then something goes wrong. And that's when we realize that it's not enough to be good because unless we're perfect, that
03:46
doesn't carry us all the time. So if you remember nothing else besides this phrase, when we're good, we're good, but we're only human, when you walk out the door, my guess is that you'll remember the things that you need to remember that we're going to talk about today. It's not about removing risk. It's about reducing risk.
04:04
For a long time in IT service provision, our managers have told us, hey, you can do that thing that you say you need to do as long as you can do it with zero risk. But if there's going to be risk, then don't do that, right? And then what happens? We're crippled by this inability to accept risk. So I would propose that we can't be about removing
04:25
it. We have to be about reducing it so that when we fail, we can fail fast, we can fail forward, and we can fail into recovery. So I've got two scenarios that I want to talk to you about today. The first scenario is about a fictitious phishing campaign. For anybody who was in Lee Holmes' session
04:44
yesterday, Lee talked about phishing being one of those things that's a critical thing that we're facing right now, and talked about how that's playing into the overall landscape. But things have come a long way since the advent of chain letters and the
05:00
initial phishing campaigns and things that we've received. I mean, I think by now, through the Nigerian letter phishing campaign stuff, I think we've given away about $53 trillion in money, right? But we've all seen it. We've all seen these things. And it's not just about data. It's not just about the things that we know. In many
05:22
cases, it's about the money that we have, right? These phishing campaigns are all about getting our money, getting our credit card information, getting our whatever. And occasionally, we face a threat that's a little bit more sinister, a threat to, for some of us in the healthcare industry, for example,
05:41
that threat is more sinister. It's a threat that is potentially against human life. It's a threat potentially against people's livelihood, people's health, people's well-being. So let's believe for a moment, let's make believe for a moment that we work for the same company. All of us work for a company called Hypothetical Inc. And at Hypothetical Inc, we do many awesome things, some of
06:05
which are just extraordinary and world-renowned, and some of which nobody knows about. Because in most companies, that's the way it is. We have stuff that we do that everybody knows about, the stuff that's public knowledge, and then we have the stuff, the R&D stuff, the cool stuff, the whatever stuff that we do
06:22
that isn't so well-known. But here at Hypothetical Inc, we're all in the war room right now, because we've had a situation where we're being attacked by these phishers, and cybersecurity's alerted us to it. They've said, hey, we've got this problem. We need to get everybody together. We need
06:41
to get the ops people. We need to get the dev people in the room. We need infrastructure people. We've all got to get together, and we've got to figure out how we're going to stop this threat. And we've got the director on the phone, and he's saying, hey, get that done. But remember, you can't take any risk, because we can't have any downtime. We can't have
07:02
automation. You guys have all faced, I asked before some of you came in, how many of you are exchange admins? Okay. How many of you have been exchange admins? Okay. So we're facing a risk, and that risk is multifaceted. That risk is one which is going to vary slightly by organization.
07:28
But let's talk about a few of the key impacts that we may face. So people are spoofing our mail. They're saying, hey, I'm going to send mail from you. I'm going to send it to your trusted partners. I'm going to send it to people who think it's coming from you. And this is one of the things
07:41
that Lee Holmes was talking about in his session. You know, it's one thing when you receive a message from somebody that says, hey, click me. I'm this awesome message you should click, and you don't know who the person is. It's a whole different ballgame when that message comes from somebody who we know, somebody who we trust. So spoofing of mail from within the organization by bad actors is a big risk for us. Likewise, capturing of
08:04
credentials is a big risk, because if they get our creds, what can they do? Everything. They can do whatever they want within our organization if they get our creds, especially depending on what creds we have. So that's a huge risk. And then one of the things, for those of you who have faced these situations,
08:22
one of the biggest problems that we have in this situation is that they constantly change the game plan. That the bad actors don't just do the same thing over and over and over again. No. They hit a wall, they change. They don't hit a wall, they change. They don't want to change, they change. Why? Because they change so that we can't keep up with them. So we've got our security
08:44
people in the room. How many of you work in security? Yep, so everybody on the security side is going, we got to find a kill chain. Kill this thing, right? Yeah, absolutely. Absolutely. He said the saying in security is the hackers
09:04
only have to be right once, but security has to be right every time. That's a very true statement. So security is sitting in the room, they're sweating bullets because the organization's at risk. And who's responsible for reducing the risk? Security. Well, we all are, but primarily security. So they're trying to figure out how are we going to fix this
09:21
problem. Let's kill mail relay. That's a good idea. Let's shut off mail relay at these points. Anybody who's done mail administration knows that that's a really scary thought for the mail administrators. Or, hey, here's an idea, let's turn off POP and IMAP on every account. I don't know how many of you in your organization have people who are doing app dev
09:44
stuff and using POP and IMAP to connect to mail. It's a really disturbing practice, yet it happens. And so let's try that. That or, hey, maybe we can unlicense things. That's a little bit less scary. So if
10:02
you're using a cloud-based email service like we are here at Hypothetical Link, well guess what? We just need to shut off the licenses in O365 for everybody who doesn't need to send and receive email. All simple stuff, no risk there at all, right? Yeah, right. So let's talk about that for a minute. So we said we're going to shut off mail relay at several key
10:23
points, maybe at the edge of our cloud environment. If you've had a cloud environment for any length of time and you didn't shut that off on day one, I promise you when you shut that off it's going to have an impact. So that's not a really great thing for us. It's scary for us. Likewise, turning off POP and IMAP for every account in the organization. We
10:44
have no clue. Hypothetical Link has been around for 15 years. We have no clue why some of these accounts exist. I mean, the IT staff has changed 65 times over in 15 years. Nobody's left that knows why some of this stuff is the way that it is. So not to mention the fact that, as I said at the
11:04
beginning, one of the biggest struggles that we face in this whole process is this problem of the bad actors constantly changing their game. So even if we take these steps, that doesn't mean it's going to stop the problem, and at least not entirely. So at this point, everybody
11:26
in the room knows what you'd be thinking. We've talked about this scenario. We know what we'd be thinking. We've got customer-facing applications that could be broken. We've got internal stuff that could be broken. We've got a public image. It doesn't matter whether you're public sector or private sector. You have a public image, period. We all know that
11:41
from Facebook, right? So it doesn't matter how well Facebook did what they did prior to the Cambridge Analytica thing. Now, all that anybody knows about Facebook is Cambridge Analytica. So we all have that concern about our public image. And we know that the change is going to take time.
12:01
Anybody who deals in Active Directory or in Exchange knows that if I want to do a get CAS mailbox on a 50,000 mailbox environment and then do a set CAS mailbox on that same environment, it's going to take a little while. So likewise, in Active Directory, it takes a long time to parse through all the accounts in your directory. And let's go back to what our bosses said.
12:23
Remember, we can't fail. We can't take any risks. We can't do anything that would break anything. Remember, when we're good, we're good. But we're only human. So no matter how well we plan for the changes, inevitably, we're going to make mistakes in our implementation path. And at this point,
12:41
we have to change our game, just a little bit, so that we can avoid extended outages. How do we do it? We do it through something I call inputs, outputs, and putbacks. You've all heard of inputs. You know what inputs are. Inputs are the things that we feed into a script or an automation process to make it more dynamic, to make it do the thing that we want
13:02
it to do. Outputs are the end game of our scripting efforts. They're the thing that comes out of the script when we're done. And putbacks are the thing that you probably have never heard of before, because I just coined the term. Putbacks are the result of an output turned input. And the putback, so originally,
13:24
I was going to call the session inputs, outputs, and putbacks. And you would have not been here, because you would have said, I have no clue what that means. So we've made a change to a little bit more descriptive title. But really, the putback is the critical thing. It's the thing of beauty. It's the crux of the methodology.
13:41
So at this point in our phishing story, what would happen is I would have to turn to one of you and say, OK, we're going to need to make a change. So let's make that change, but let's script big changes with small risk. Of course, it's a shameless plug. I understand that. But it's important.
14:02
So at every point in this process, what am I really saying? At every point in this process, we may need to output something when we intend to effect a change. So at every point in the process where we intend to effect a change, I'm going to read it right off the screen, because this is critical to understanding. At every point in the process
14:22
where we need to make a change, we may need to prepare an output file, which can be easily fed back into the script to revert some or all of our changes. Think about that for just a minute. At every point in this process where we intend to effect a change, we may need to prepare an output file that can be fed
14:44
back into the script to revert our changes. I'm guessing what you're thinking right now is, gee, Brad, that sounds like a lot of work. Sure, it does sound like a lot of work. It also sounds like it's going to be something, in some cases, monumental. Let's talk through it a bit. What's the problem? Oh, the wind. I apologize for that.
15:15
Sorry. I'm fine right here, so I should just stand right here. Do you know why that is?
15:24
I have a fan under there. You guys know my secret. So I'll stand over here, so I'll I'm so glad that we are a room full of problem solvers, because you see, I'm sitting here going, what is taking place next to me here? So you've all solved my concern of self-consciousness
15:47
that comes from not knowing what's going on in the room. That's right, you've reduced my risk, and that's an important thing to do. All right, so I'm going to switch over here, and that thing is really high up, which is fantastic for you guys. For me, it's
16:08
insanely high up, and I can't just point to things and say, hey, check that out. That's an annoying feature of Windows right there. All right, so I'm going to step over here so that I can be out of the way and feel like I can point, because for some reason
16:24
I want to point. So in this situation, we had our security folks tell us that they want us to make a change, they want us to turn off Pop and IMAP for every account that it's turned on for. We know that it's a risk, but we also know that we need to make the change, because right now, what we're facing is a potential for security breach through
16:43
capture of credentials, through a bunch of other things. So there's a number of different ways that we could do this, and in this particular case, we're connecting to our exchange online, and we're going to just do a set CAS mailbox, because someone in my exchange area already gave me a list of everybody who has their Pop and IMAP access turned on
17:04
on their accounts. So this is literally all that I have to do to turn it off. Super simple, right? Six lines of code, everybody's happy, we've made the change, no problem. So what happens when I turn that off? That's an honest question for anybody in the audience.
17:26
Tell me what's going to happen when I turn off Pop and IMAP on accounts, okay? They're all going to be off, and what could it mean? It could mean that people don't have access, it could mean a number of different things, right? And that is a critical problem that
17:44
some organizations email from a copy or not going, there's a lot of things. So in our war room here, we've just identified like 10 things, 5 to 10 things that could be a problem that we may need to address. So let's put this into practice a minute.
18:06
Let's take a look at how this changes when we script big changes with small risk. So as I said, the key is the inputs, the outputs, and most importantly, the put
18:21
backs. So in this new version of the script, what you see happening is we're doing a CAS mailbox. In this case, I'm not even relying on my exchange guys to get me the information in advance. I'm doing a get CAS mailbox against my environment for every account that either has Pop enabled or IMAP enabled. I'm outputting the name and the setting for
18:41
Pop enabled and IMAP enabled for each of those accounts. Super simple stuff. We run through here. Like I said, I output it to a CSV file. Everybody's happy because we know what those settings are. I'm going to run it through. I'm going to say for every one of those accounts, do exactly the same thing. Set it to false. Now, what's
19:01
going to happen when the folks with the copier problem find the copier problem that I just created? What have I done by doing this? I need it turned back on. Whoa. Okay, so your copier people know what the accounts that are impacted or are affected are. What if it's a line of business application that we have that is incredibly huge? Let's
19:23
say our organization has one line of business app. Let's say it has 35 accounts that are associated with it. Let's say the guys who wrote it are gone. Let's say they don't even know what accounts are affected. Well, at Hypothetical Inc., we have thousands of email accounts, and we just turned it off for thousands of people, and we don't
19:42
know which ones to turn it back on for all the reasons that we've already stated. So you're right. I know what I impacted by making my change now just by outputting that file, and the beauty of it, right? The beauty of it, I'm not going to unhighlight it, but the bottom line of code. Just take the second to last line of code and
20:02
the bottom line of code, or third to the last line of code, comment that one out, uncomment the bottom line of code, and guess what? We just put everything back the way it was. It's that simple in this case, and you're going, well, hey, but we didn't have a problem. Everything worked great. That was a colossal waste of time, Brad.
20:24
No, no, it wasn't. Why? Because at the end of the day, what's important? At the end of the day, what's important is that we went home. We're not in the office still. What's important is at the end of the day, we went home. At the end of the day, the bosses were happy. At the end of the day, we got a little
20:42
bit better at reducing risk so that the next time that we have a bigger risk, we're not going to be down and out forever. And we eliminated an attack vector. The security guy is telling me that we succeeded. We succeeded. So, all
21:03
right. No way. It worked. Great. Awesome. So, power show. It really works. It really does the job. So, I had nothing to do with that, and it's a wonderful thing. So, I'm going to move really quickly into our second scenario here
21:20
because we have quite a few steps that I'm going to take. This is going to be a little bit more interactive demo. So, in this particular situation, how many of you work in an environment where compliance matters? Okay, cool. How many of you are lying? Those of you that didn't raise their hand. That's great. I wish I
21:42
worked in a place where compliance was irrelevant because it really makes life harder in some ways. And one of the ways that it can make life harder is when our folks from corporate compliance come to us and say, hey, you know how we've had those laptops and those desktops? Well, now we're going to have those tablets. And I'll go, why? Why does that matter? Well, because
22:04
compliance. And we all know, those of us who deal with compliance issues, all know that's the answer, because compliance. In our organization, Hypothetical Link, we have an IT policy that states that machines have to be named based upon the type. And we also do a lot of our billing and other site-specific stuff
22:23
based on our Active Directory structure. And so, I'm going to jump over to that structure. We're going to take a look at it. If I'm unlucky, it's not still open. You know how the demo gods are. There was a couple people that had
22:43
problems yesterday with the demo gods, I heard. Yeah. Yeah, I was in that session. So, yeah. So, let's take a look here. By the way, really quick, I just want to give a shout out to Missy Januszko and Jason Helmick for their creation
23:02
of AutoLab. AutoLab works really well for people who are trying to set up a really quick demo lab. This is the single DC demo lab that I set up. Works great, because right here, I have everything that I need to be able to demonstrate to you how Active Directory is impacted by a massive
23:22
change and how this process works for us. This is really disturbing when you're not, just so you know, it's really disturbing when you can't see what's up there. So, I mentioned our Active Directory structure is a little bit complex. So, desktops and notebooks. Well, we have offices in many states. We have
23:41
offices that contain staff who do a variety of different types of work. I'm going to key in on the notebooks OU for our Alabama office, and you can see we have marketing, sales, and support staff that exist in each of these offices that we have. Specifically, we have, and this doesn't matter right now,
24:03
but it's going to matter, so I'm going to customize this and turn on some more information in AD. So, we see we have four objects. There's four notebooks. As I said, we have a corporate compliance group that's saying, hey, you need to change a bunch of this stuff. So, we're going to just script a change. No big deal. Let me jump into code, and we will, as soon as we get out of the
24:26
desktop session. So, major Active Directory change that we're about to make. It's on a remote machine. We're going to go ahead and just connect to
24:42
that remote machine, and we have an input list. We got that input list from our corporate compliance people. Actually, it came from our SCCM people by way of a request from our corporate compliance people, and essentially it
25:03
identified all of the machine names for the machine objects that are now going to be considered tablets based on their model. So, that kind of gives you an idea of what we're doing. We're going to input that information, and as you know, if you're an AD administrator, you cannot create an object or move an
25:22
object to an OU that does not exist. So, we're going to go ahead and create that OU for ourselves, and it ran through. It did its thing, and I'm going to kick this off because I want to make the change here because it takes a couple minutes to run, and while it's running, we're going to talk about a few things. So, it's running in the background. You guys can go ahead and let
25:44
me know. You don't have to though. I've got a plan to check on this thing at some point. So, this change is a little bit different. In this change, the first change was kind of just a simple flipping a switch. In this particular change, we're talking about creating a whole new OU structure.
26:04
We're talking about moving a ton of objects, and we're talking about changing information about those objects. I understand that I did say in this scenario that our machine objects are named based on their type. Don't worry, we're not doing that now. We're going to use SCCM to push out a name change because it's a lot more friendly to the users when we can push that out and
26:21
control the reboot. So, I'm not going to do that here, and you don't have to feel like you need to come up and tell me afterwards that I failed to do that. It's in the plan. So, this is different. This is a little bit more work that we're doing, a little bit more potential for problems, and any of you who are AD administrators, you probably have more than one of you at your
26:42
organizations. Am I right? Yeah. So, there's this wonderful thing where everybody does stuff, and they don't ever tell anybody that they're doing it, and that really leads to a whole lot of problems for us as we're going through these processes. Because when I make my change and I just move a bunch of objects from one OU to another, everything's good. It's not a
27:03
problem. But what if you come after me, and you move those objects to a whole different OU, and you did it because you know that you need to make a change to every tablet, and I did it because I knew that I need to say they were tablets, and then our SCCM guy comes back to us, and he says, hey man, so
27:26
I'm really sorry. I used a percent sign instead of an asterisk, or an asterisk instead of a percent sign in my collection query, and I accidentally got a whole lot more machines than I was supposed to. Right? This sounds
27:43
totally reasonable, and I'm not trying to dog on the SCCM folks. It happens to all of us, because when we're good, we're good, but we're only human, right? So our folks gave us the information. They gave us the wrong information, and as we're preparing for this change to be completed, we're actually going to take a
28:02
look at it. I think it would be good to show you what we're doing here. So as I said, script is running, I hope. I really don't know at the moment that it is. It's an assumption on my part. It looks like things are going well. Looks like things are going well, because we have a tablets OU now, and we can actually see that there's 28 states that are identified there, and if
28:23
we look at the state of Alabama in the marketing department, we can see that we moved one of our notebook objects over to that OU. So that's good. That's good. Let's jump back really quick, and we can see that our script is done running. As I said, we ran into a problem here, because our SCCM folks
28:44
came to us and said, hey, and this happened, by the way, six days later, after you made your change, after I made my change, and now everything's changed to the point that I can't just rely upon where the object resides now in order to put it back in the correct OU, and so it's not just
29:03
as simple, because in my script, my script is written with two prongs. It's not just as simple for me to say, well, I know that it's no longer a tablet type. Now it's a notebook type, so just make it a notebook again. That's not good enough, because you moved my object. Dang you. Why did you move my object? So what I've done here, when we talk about how do we change our
29:24
code so that we can script big changes with small risk, what I had to do here, if I scroll up in my code, is I had to have two sections in my code. One section where we do the work if we have a distinguished name. So if I
29:41
know the distinguished name from where the object came from, then I can just input that, and I can say, this is where I want the object to reside. If I don't have that, and I only have what I had at the beginning, which I'm going to show you my files, my input files, because that'll help you to see what I'm talking about, if I only have, yeah, netbios name or a type, because in
30:04
this case I needed to know either the type or the distinguished name. So when I did this thing with notebooks to tablets, I had a CSV file with name, distinguished name, and type. I didn't have a distinguished name for anything, because all that the guy from SCCM said is, hey, here's your objects, and I just
30:23
appended a comma comma tablets, because I just wanted to move them from whatever the organizational unit structure they were in to the new organizational unit structure for tablets, which is a little bit different than what I did as I was running through the process. So as I was running through the process, again, outputs that may need to turn inputs are called putbacks.
30:46
What I did, assuming that I might need to put something back, is I said, okay, let's be sure to grab the current distinguished name for every one of those objects that I'm going to move. Before I make my change, let's grab it so that we can use it and go on. So I did that, and then, as I said,
31:03
my SCCM guy said, hey, Brad, I'm super sorry, but I really messed you up. I gave you a bunch of stuff that you didn't need, and so I decided, because I was angry, that I was going to create this file called crap we shouldn't have changed, and I was going to put the crap back where the crap belonged, and
31:21
so you can see this is just a slightly less voluminous version of the post changes that we made, so the stuff that was changed. We narrowed that list down. We got this list, and we're just going to run this list through exactly the same script. So we hop back. We come right up here, and of
31:46
course, we have our file called crap we shouldn't have changed, and we're going to just input that file, and we don't need to recreate OUs because all that already took place. We're just going to run our exact same script again, and this time it's going to take care of cleaning up the things that I
32:02
don't need anymore because I never needed them in the first place, as well as moving the objects back to where they need to be. So I'm going to let that run while I talk for just a couple minutes about some other things. You guys don't need me to go back to AD and show you the
32:22
change that it actually happened. We believe that it happened because the first one happened, right? So let's jump back to presentation. So I am way back where I didn't mean to be. This is always good as well.
32:46
Let's just keep going the wrong direction. Oh, I'm back in the pan. I apologize, folks. So was it worth the effort? In the second time, was it worth the effort? We created a little bit more code. We have a slightly longer script.
33:04
We all agree it was worth it. It doesn't have to be hard. It doesn't have to be a lot of work. It doesn't have to take forever. What it does have to do is it has to involve a change in the thought process. I'm in the battle
33:20
faction. I just want to make the change. Boom. Enter. Oh crap. What happened? Oh well. Boom. Enter. Did I fix it? No. Boom. Nope. Still didn't fix it. So that's my way of smashing and crashing through the China shop, but that doesn't work. It can't work as well as this process of scripting big changes
33:42
with small risk because I am very human. You guys are just human, but I'm very human. I make a lot of mistakes. Jeffrey Snover even alluded to that. Nobody's perfect. We're all fallible people. We make mistakes, and when we do, things like this can make a big difference. The first time that I had to
34:02
use this, it actually was born out of necessity by me because I made a change to 6,000 objects. I work in a 50,000 object active directory at the time, and I made a change to 6,000 objects, and when the change was made, I didn't take any notes in the process. I just ran it. And sure, that's a real
34:25
noob mistake. We all make those mistakes though, don't we? We all make the mistake, whether it's five objects, whether it's 5,000 or 6,000 or 50,000 objects, we all make the mistakes, and we can all benefit from thinking a little bit in advance. Now, was it worth it? Yeah, it was worth it. What did I
34:43
have to do to make it happen? I had to think about what I was doing. I had to think about what I might want to undo. I had to think about what the potential problems were that I could experience, and I had to take all of that into consideration as I was developing the code. I had to do it before and during, because in some cases, we're already done developing the
35:05
code, and then this idea crosses our mind. It saves our lives, by the way, sometimes. This idea crosses our mind. Oh man, what if that went wrong? Oh, that's not good. So you change your code a little bit more, you adapt the problem that you might have had, and again, I just outputted one
35:21
file. One file that was created in that temp folder of all of the objects I changed. This could get bigger. It doesn't have to be that small. We may be making 16 changes in a row, and if we're making 16 changes in a row, maybe I only care about the third, the seventh, and the thirteenth change. Maybe I care about all of them. Maybe I care about going back at any point in the
35:43
process. Maybe I don't. There's a lot about this that raises questions, but like I said at the very beginning, it's necessary, because when we're good, we're good, but we are only human. All right. Questions? Anyone? Yeah.
36:06
Putbacks is just like a revert. The question was a rollback plan. That's exactly right. So in our change control process, we have a back-out plan that's required. Everybody in an organization that has that level
36:23
of enterprise discipline has something like that, but how many of your organizations actually take that seriously and say, how am I going to get back to where I was? I'm seeing somebody shake their head over here.
36:56
Right. So do you find it useful, the idea? Just even the reminder.
37:09
Just to serve as a reminder, when we make a big change, and sometimes people ask me the question, as I've talked with some of my peers about this, one of the questions that I've often got is, what's a big change?
37:24
Well, it's really situational, isn't it? Is it big because it has tremendous potential for business impact? Is it big because of the volume of change that I'm making? There's a lot of different things. The security guy's nodding his head. There's a lot of things that go into determining risk. How late am I
37:47
going to get home to see my family? Right. Right. There's so many things. There's so many things. Any other questions? I could talk for a long time,
38:02
so I don't want to keep talking. This is time for you guys to ask questions. It took eight hours. We ended up having to use PowerShell to access the dumpster in Active Directory to recover a bunch of stuff, and then use some more
38:21
PowerShell to try and figure out what the heck changed in addition to that that I don't care about. In the end, it was bad because we rolled back the entire change. That's what we had to do. Oftentimes, that's what we have to do. That's not failing forward. That's failing backward, and we don't want to fail backward. It's not just because of the release pipeline. It's not just
38:42
because of this whole idea of the DevOps process and needing to be able to go, go, go, go all the time. We don't want to fail back because we were already there. We didn't want to be there, so we went over here. Nobody wants to go back there because that's not where we need to be. Other questions? Absolutely. StartTranscript is another great...
39:18
That's absolutely right. Thank you. StartTranscript is another great way for
39:22
you to capture what you're doing. It really comes down to, you saw that there were only three columns in the data that I output because there were only three things that I needed to get back to where I might want to be. Because at the outset, I determined all I might need to do is I might need to put them back where they came from because I didn't change anything else about the objects. At the start of the process, you have to
39:43
determine, in some cases, StartTranscript and StopTranscript are going to be enough for you to say, okay, I have everything I need to know in order to be able to undo something that I did. In other cases, where you're interacting with another system or multiple systems, what you may find is similar to me in this scenario. You need to be able to output information that is specific to
40:04
something that happened outside of the PowerShell script itself. Other questions? Yeah. So the question was, have you found that at any of
40:30
the companies that you worked at that may have had a change management process, that that change management process made this better? I'm going to take some liberty in the interpretation of your question and
40:40
correct me if I'm wrong. You may even be asking, is it really necessary if you have proper change control in place? We have proper change control in place. At the organization I currently work for, we have about 54,000 employees. Those 54,000 employees use 60,000 computers and there are 3,000 of those people that are IT people. There are potentially 15 to 30
41:07
Active Directory administrators. There are potentially 30 SCCM administrators. There are, so in any given week we have probably 285 change controls in
41:21
the organization that we're reviewing and there's local change boards and an enterprise change board and all of that's great and it works well except when you don't read and sometimes I just forget to read that and when I forgot to read that I didn't realize that the change they were making they needed to take something else into consideration beyond what they described.
41:41
So I don't think it, in our organization, where I work today at the state of Michigan, this has helped me even without, or even in the presence of a very mature change control process because maturity of your change control process is really only a thing if everybody adheres to it.
42:01
It's a great process but Johnny and the janitors, sorry, in the sanitation department, he didn't, you know, follow the change control process and so we went out to mop the floor, he spilled the water into my server and everything was bad. Stupid example but it's still little things, little
42:23
things like that. Oh, we were making a change in this rack and somebody thought our rack should be numbered, number letter and letter number. So when I went to 10E instead of E10 and I powered off U34,
42:42
oops, right, stuff happens and when it happens you just, you don't know. Any other questions? I think we're almost out of time. Any other questions? If not, I'm going to be hanging out outside. I appreciate it everybody. Thank you so much for your time and thank you for what you do.