So, I have no slides for this session because, well, I didn't have time to write slides. Thank you very much for coming in. Today we're going to kind of talk about using

Operations Management Suite to generate alerts, send them over to Azure Automation and then run runbooks for remediation. How many people here are familiar with OMS using it actively in your environment? Okay, so Operations Management Suite has

some really good plugins for Azure Automation. Once you get a OMS workspace set up, you can actually connect your Azure Automation runbooks, your storage accounts, stuff like that, in order to be able to better manage the

environment. Now, they've kind of moved some stuff around, so it's a little bit more difficult here to traverse, but let me go ahead and just show you what it looks like in the Azure Automation account itself. So, once you get a

workspace set up, you can actually connect it through, oh, I hope I saved that runbook, you can actually connect it through the Azure Automation account directly, so it'll just ask you what workspace you want to connect to,

and then it gets plugged directly in. Now, once you actually have this set up, you can actually get into Operations Management Suite, start setting up your own custom alerts. One of the things that I did in order to be able to start generating alerts, and I don't recommend you do this in your

production environment, but I built a virtual machine set up 3389 out to the open internet, which, believe it or not, a lot of people like to try to see if they can get into those virtual machines when they're opened up to the internet. So, I went ahead and basically went into my log search, or got

my security alerting here, and was able to start building kind of my own little query on these incoming malicious IPs. Internet's a little bit slow, let me go ahead and close out some of my doodads here. Okay, there we go. So, as we can see,

it's actually got notable issues, and, hey, look, it tells you when incoming malicious IPs are generated. The nice thing is that it actually kind of gives you a general query that it's running to be able to take note and say, hey,

you've got, you know, some stuff here that you're gonna want to look at, but we can actually take that query engine, generalize it. So, go into Notepad here, just kind of take it, and anything that's really relatively specific, you

can go ahead and generalize. So, turn People's Republic of China into Star, you know, maybe you want specific IPs, or anything like that. Now, there's just about as much, like, science, or about as much art as there is science into

building these alerts, so there's a lot of experimentation involved. What you wind up building is a generalized query that you can test in your alerting. So, I'm gonna just go ahead and scrape what I've got in my search here. And we can see here, this is a little bit more generalized, so I don't really have

any specific, you know, countries or anything like that listed, so I can go ahead and grab this alert, I can go into my log search, and pull it up once

I get back into the window. I do apologize, I had this to be a little bit more of a structured talk, but my demo environment exploded on me late last night, and I didn't catch it until this morning, so we're kind of going through

more of the thought processes and stuff. Although, I kind of feel that I'm hoping that that'll actually open this up to a little bit more of a conversative session. So, I go ahead and paste that in there, I run my query, and you can see that I'm actually getting valid data out of it. Now, once

you've got this query built up, you can actually go ahead and start building your own alerts. And it doesn't have to be security stuff, it can be service stoppages, it can be maybe you have SQL servers in the environment, and it runs a certain amount of CPU, and you want to be able to auto-remediate

that stuff. I actually like this kind of alerting, just because when I'm showing it off to C-level execs and security guys, their eyes get really wide when you see how many times a box has been attacked. But

basically, it's relatively simple to go ahead and generate these alert rules. So, you just take that query that you've built, and you throw it in here, you give it a name, obviously, in a time window, and have it check for that alert every so often, and then, based on a number of results. And,

of course, I have it set for greater than zero. So, if I get one alert or one result, it's going to go ahead and generate that alert. And then, of course, what you want to do, obviously, there's a number of different options that you can select. You can have it notify you by email,

you can have it plugged directly into ITSM. So, if you got, what's it called? Like, ServiceNow or something along those lines. A lot of the newer ITSM tools have direct connectors, so it can go ahead and generate tickets and alerts. But what we're looking at is the runbook. Now, once you've

connected to that workspace in your Azure subscription for your automation account, it will automatically show up in here. And basically, if you have multiple automation accounts, you get a little bit of a drop-down box. And then, same thing with the runbook. When you first create it, you'll get a drop-down box, and you'll be able to select exactly what you want

there. Once it's saved, all the stuff is locked in. So, if you go to the wrong runbook or anything like that, you're going to have to actually create a new alert. So, which is why you can see in here, I've got like 50 million alerts configured. So, it takes a little bit

of experimentation, but once you finally start having that process down, you're okay. Now, what do we do with the runbook? So, the runbook, essentially the way OMS notifies Azure Automation to perform an action via runbook is it uses what's called a webhook.

So, I wanted to talk to you a little bit about what a webhook at its very basic looks like, the type of information that you need in order to be able to get out of it, and then how to be able to process that information, and then run the appropriate commands to get stuff done. Is anybody here not familiar with JSON? Okay. Excellent.

So, this is basically a test JSON construct that I created in order to be able to test and validate my automation runbook code. Obviously, you need a webhook name, and then information in your

request header and request body. For the purpose of this runbook, what the runbook does is it actually is designed when it gets this malicious IP, you know, alert. The runbook that I have created will actually create a network security group to block any inbound traffic from that source IP. So, the two things that we need are

the computer name for this construct and the malicious IP. You could generalize it to update a specific network security group with additional rules. This one I just have targeting the computer. Now, from the perspective of the runbook, nope, that's work.

This is what I've got coming in. Where are my runbooks?

I can't find my runbooks today. I feel like I should be singing a song or something while I'm doing this. No, I didn't have time. Which is probably why my demo environment exploded on me this afternoon.

So, the incoming webhook data is processed as a general object. So, that's why you'll see the parameter there for object. Is anybody not created in a runbook before? Okay. So, in general, and you'll see this when you create a new Azure

automation account. If you go to the runbooks, it'll give you a couple of examples. One of the things that you'll see that it first processes is to run as the Azure run as connection. This will actually invoke the Azure run as account in order to be able to act on your behalf when it's triggering a runbook.

This basically gives the run as account, when it's created, it has the ability to create objects, process, configurations, all kinds of different things. So, when you're generating a runbook, it's going to ask you what that is. When you first create an

automation account, you have Azure run as account or run as connection. So, that's the one we're invoking here. This runbook is a little bit generalized. I just kind of wanted to put it all together to show you what, you know, what all of this looks like and how it works. So, basically, what we're doing is

we're taking that JSON object and we're converting it into something that's a little bit more usable. We're going to go ahead and parse out that webhook name, the request header and body, so we can go ahead and get a little bit of information. And then I'm using write output, just basically so when I run the runbook, I can see what

information is being processed when, and make sure that the data that's coming through is entirely valid. And then, of course, I created a commandlet to essentially configure that NSG rule for me. So, everybody pray to the demo gods, because it worked this morning. Oh, well, it's probably not going to work when

I don't give it any valid data. So, we're going to go ahead and snag this. We're going to take our test JSON

data, we're gonna put it into the parameter, and of course wait for it to stop being queued and explode because I pushed start too soon. Does anybody have any questions so far? So, what can OMS alerts do for

us? Well, actually, they, they can do a lot of things. If you have the OMS clients installed on your virtual machines, you get things like health data, health information. You know, you can get, you know, service information, anything like that. And then, you know, you can kind of

craft runbooks to be able to auto-remediate a lot of different things. Some of the, the customers that I've worked with, we've actually gone so far with it to where, if they get any type of an error from their service information, we've built runbooks for them to actually completely deploy brand spanking new VMs with DSC

configurations and everything into the environment. Because they don't even want to pay their, their operations guys to troubleshoot an issue for eight hours when they can just deploy a new VM in 30 minutes. So, it's it's really extensible and flexible as to what you can do.

Wow, it's really hanging on there. Yeah, stop. So, I'm gonna go ahead and pull the trigger on that, and hopefully we'll get some information out of it here relatively soon.

Now, one of the things that I really, really like about runbooks is regardless if you're running it in a test or production mode, you can get some really, really verbose output. So, as this is processing anything that we do, as far as like, with

write output, it's going to be generated to here, but it's also going to be thrown out into some of the output logs. So that way, if it's in the middle of the night and the runbook triggers automatically, you can actually go back and review all of the steps that it took, provided that you're actually writing that verbose output. Yay, we're running.

Everybody say yay. Yay. So, you can see here, this is where it's using that run as credential to go ahead and log into the Azure environment. I've got my JSON data here, so you can see that it's actually gone ahead and built

some of my data, and then it's going to go ahead and hit it here saying that the rule with the specified name already exists, because I've already run this with the IP address before. But I'm going to say I wanted that, just to kind of show you guys what it does when it throws a terminating error, but you can see here where it's gone

ahead and generated all of this data from that verbose output. So, we've got the webhook data, which, you know, you want to be able to kind of see that data that's incoming. I've got my right output, so it gives me all of that information. Now, I know, just looking at it, that I have valid

data coming through, and this is really important. And I'm going to start talking about some of the constructs that OMS will pass over and some of the things that you have to look for. And then you can see here, when it ran the command, this was the output that I have written into it to say this is what I'm trying to build. Now, if I go back and update

this, and we're going to just go ahead and say 2.1. I shouldn't have this rule here. As

an aside, if you want your runbooks to run

faster, hybrid workers, which are essentially dedicated VMs that are designated to process things from Azure Automation, do actually return a faster, you know, you get a faster return process out of it. Part of the reason why you'll get a lot of queuing

and kind of waiting on a runbook if you don't have one of these hybrid runbooks or hybrid workers is because on the back end, when you commit to a job, the automation account has to actually spin up like a containerized VM to actually process the runbook information. So, if you're

processing a ton of stuff, I highly, highly, highly recommend getting a hybrid worker into the environment. So you can see here, we've actually got a new output. This tells me that it actually did in fact create a new security group

rule, and it's got all of the information down in here, primarily because my function actually calls the new Azure RM network security group rule command. But we can see all of the data that's been generated here, and then we can also go ahead and take a look at that virtual machine, go

into networking, and then we can go ahead and see that that rule has been generated. So we've done it with the basic data. Let me go ahead and talk about what it looks like when OMS generates webhook data. I highly recommend, if you're

gonna do anything with webhooks, that you want to actually kind of have one of these little JSON constructs to do your testing in. Number one? Yes. Yes. Yes. Yes. Yeah. Like, I

don't see everything in OMS automatically getting shipped over all at once. So, and there's gonna probably be other use cases for webhooks going down the line as well. Yes. I don't know how much I can

speak to that. Yeah. So, yeah. Yeah. Anyway. So, you're kind of right, and I'm, I, I can't get into details. So, anyway, but OMS, to

be clear, is not going away. So, anyway, here's the little JSON object here. It, like I said, you want to test your run, but if you're using webhooks, you want to create your own test JSON object. Part of that is because it sometimes

takes a while for OMS to be able to process an alert, but you also have to wait for that alert to actually happen before, you know, it triggers. You don't want to necessarily use that as your test case, because if it explodes, then you have to wait for the next alert cycle for,

for anything to come in. So, highly recommend going ahead and creating your own JSON object to pass it through. I lost my train of thought. Oh, actual runs. So, I have another one here where I was just basically capturing it, and this is

the one that exploded. But we can go ahead and take a look into the jobs here, and you can see that I hadn't, you know, an incoming alert at 1204 AM, and it did throw a ton of errors, but this is where I wanted to kind of show you what, you know, that you can go back to these previous jobs and see what the output is. So,

this is really important for, you know, being able to go back forensically and, you know, take a look at a, you know, what went on. More importantly, I wanted to show you what the webhook data looks like incoming. Kind of a mess. So, this is where it gets to be really important to

test with your test JSON object for the webhook. So, because it, parsing sometimes can be a little bit problematic with OMS. But the other thing that you want to do is make sure that the data that you're going to be feeding into your Azure Automation runbook is actually in fact valid.

So, you may grab, if you're looking for like computer name and IP address, well, you might have to sort for some type of a sub object inside of the request data. So, if we go through here, you'll see, actually it might not even

be passing the IP address, and that might be why it blew up. Ah, here we go. So,

what you want to be able to do, and this is kind of why I craft my runbooks to be able to output that data, is to be able to see exactly how OMS is presenting it. Now, depending on the type of alert that you're using, that data may be formatted differently. So, I

always start small, you know, process the object, give me what the actual output of the object is, and then once I have a runbook with all of those references, I can go through, pick and choose the data that I want, and then have it formatted properly. Does that make sense to you guys?

All right, cool. Getting a lot of nodding. So, questions. So, basically the OMS VM agent is pretty

much the same agent as SCOM, so it can pull data. I haven't seen an agent that can actually reach in

and execute. The closest thing that I've seen is with like the PowerShell web, the Azure PowerShell console. You have things like enter Azure RMVM, but I don't think that commandlet is designed to actually process on behalf of the Azure run as account. That

would be an interesting experiment to try. So, let's try

an experiment. Anybody here not familiar with the Azure PowerShell console? OK. So, there's a PowerShell console that exists in Azure. If you click on the little

PowerShell button here. The only way I can potentially see this working, so with enter Azure RMVM, you have, it's essentially leveraging enter PS session. OK. So, and actually

I don't think it'll work in my instance because I don't have 5985 open. But, so some caveats. You know, you have to have the firewall configured to allow 5985 for HTTP PowerShell. You would have

to pass a credential and then be able to execute the run book. What I don't know is if Azure Automation can actually leverage that command. So, you'd have to experiment with it. Right. But, you could store the credential in an Azure Automation credential.

So, let me go ahead and show you guys what that is. I like this. Now, we're actually getting into like experiments and stuff. So, you have credential, so you have key vault, but you also have automation credentials. I usually tend to tell people to use the

automation credentials more than key vault. Just because with key vault, you have the ability to obfuscate the password. Whereas the credentials here, you can see that even though I'm the owner of the subscription, I can't actually peel that away. And it's actually really good to use in automation run books and also DSC configurations because

you can invoke it with get automation PS credential and then pass that credential object directly in and it never gets processed out in the open.

So, an automation account can execute anywhere inside the subscription. It's not limited to VNets or resource groups. Yes. It actually, for the most part, doesn't care. Most

of the executions happen underneath at the subscription level. Now, if you were to enter Azure RMVM working through a run book, that may. Although, yeah, I think that's something

that I might actually bring that to the Azure Automation team and say, you know what, this would be kind of cool if it doesn't work. So, that's something I'm going to have to experiment with. Yes, because your security people are 90 years old and they

think their environment's a lot more secure. Yeah. Yeah. And because your security people

are 90 years old and don't understand that the credentials exist in memory and that's a bad thing. Huh? So, and this kind of speaks back to one of the things that I've

been pushing heavily with a lot of customers too is using Azure Automation a lot more for not just automating and remediating my cloud environments or their cloud environments but also handling their on-prem stuff. A lot of this stuff with a hybrid, especially with a hybrid run book worker, sorry, I can't talk today, gives you some really,

really good management capabilities over a secure connection. 99% of the stuff operates over 443-TLS. It's private public key auth when you register a machine that exchanges

its own certificate with Azure Automation. No, you don't get a hold of it. So, if your 90-year-old security guy wants a listing of all those certificates, guess what? Tough. But yeah, it actually gives you the ability to very, very securely manage a lot of different

things. DSC configurations. Anybody here run Poll Server? Yeah. When you generate a MOF on your Poll Server, it's in plain text, right? Yeah. These are encrypted end-to-end. So the MOFs are actually encrypted in Azure Automation and then through the public-private

key auth that it uses for Poll, it'll decrypt it in memory, run the configuration, and then it scrubs it. So, and that's another thing that is insanely hard to hijack. So, your on-prem Poll Server, which is essentially a file share, can be hijacked. These really,

really can't. Actually, even being the owner of a DSC config, and I'd be more than happy to show you one except for I've got some customer ones on here, but even after, like even myself owning the subscription, when I generate the MOF file, I cannot see that

plain text MOF file. I can put credentials into it. I can, you know, I can go ahead and give it the configuration, but once the MOF is generated, I cannot take it apart. I cannot even see it. I can't download it. Nothing. And moreover, because, well,

I'm a nerd and I manage my, you know, laptop with DSC in the cloud, I can go out and do C32 configuration on that admin. Yeah, that's what it looks like. Fully encrypted,

and so you can tell your 90-year-old security guys, yeah, your Poll Server doesn't do that. So I know we kind of went a little bit off topic, but I just kind of wanted to walk you through some of the, like I said, the constructs and the mentality of OMS and

connecting it to an automation runbook. Do you have any questions so far in regards to that? Yes. Yes, they can. Actually, pretty much, I haven't seen any place where

the creds can't be leveraged. Anyone else? You got me for another 12 minutes and I'm not very good at dancing. Oh god, no. So yeah, that's pretty much all I have.

You know, the, really where it gets into the science and voodoo is crafting your alerts. The only thing that I can say is go through the visuals, you know, if you're looking

for particulars, the, you know, OMS and even like the Azure alerting and monitoring has a really great functionality in order to be able to drill down, get the particulars and it shows you the queries along the way, so that way you can go ahead and grab the information out of those. Very easy to generalize and then start testing.

Once again, when you're building your runbook, use generic JSON inputs for testing and then once you think you've got it fine-tuned to where it should be, go ahead and connect that alert up to the runbook, execute it, but also more importantly, make

sure you've got some kind of verbose output to say, okay, this is what the object looks like coming in. Like I said, it's very easy to take a look and see what the input object looks like through the input parameters so you can get that data there, but also

after you've processed it with the runbook, make sure that, you know, the data that you want is actually being formatted the way that you want it. That's pretty much it.

Any other questions? Actually, any questions in Azure Automation in general? Mostly.

And actually, I can refer you to my blog series over on powershell.org about some of the stuff that I do. Primarily, the stuff that's in the Azure Automation runbooks is where you're going to be doing PowerShell. You can actually import and export those

runbooks from your local machine into Azure Automation, but really, most of the stuff as far as like getting and formatting your queries, unless you're really, really, really good at SQL, which I'm awful, you're going to be doing it through the GUI, and I kind

of highly recommend it because you get that data back in real time to make sure you're actually getting what you want. As far as connecting Azure Automation to an OMS account, you can do it through PowerShell. It's basically just a one-liner, but you have to scrape the workspace ID and key, which, gooey. So, yeah, it's kind of amazing.

There's a mix of both. You're going to be doing a lot of stuff in the UI. OMS, in general, is really designed for the UI user. Now, the Azure Monitor stuff, you can get a lot of good data via PowerShell out of that, so it's going to be kind of that balance

of where do you want to get your data from. Ultimately, the alerting and passing it into the runbooks, the mechanisms work the same, and actually, with Azure Monitor, it's a little bit easier to build because you can invoke it directly off of that query,

but the nice thing about playing around with the webhooks, too, is you might not necessarily be using OMS. There's other things that can execute and transmit webhooks that Azure Automation can ingest, so this is a good way to kind of familiarize yourself with how do those webhooks work, how can I format that data and then use it to execute

my commands. Any SCOM people in here that could answer that?

To be clear, also, the OMS agent, because it's so similar to SCOM, it's really pretty much the same thing. It's just got the ability to add an OMS workspace key and ID.

You can actually multi-home it, so you can have it reporting to an OMS workspace as well as an on-prem SCOM server. You can kind of get that flexibility in there as well. Yeah. Yes. It depends on the rev, too, although it's been, like, three years

since they implemented that, so pretty much if you're on current branch, you should be

able to multi-home it, but if you're on SCOM 2007, eh, probably not. Any questions? Oh, so from a DSC perspective, so, basically, whenever I go to an OMS,

compile a configuration, it'll go ahead and generate a MOF in Azure Automation, and the MOF is encrypted there. Now, PowerShell 5, yes, since PowerShell 5, they've actually

been encrypting current.MOF on the machine, so that's what allows you to have that encrypted current.MOF. Where it's not encrypted is if you're using an on-prem pull server,

the MOF is actually stored in plain text, and to be true, the pull server code that has been provided is considered test code and not actually really supported by Microsoft, so it was just kind of developed as an example. If you really want that true, secure, end-to-end

capability, the general recommendation for now is to leverage Azure Automation DSC because it has that end-to-end encryption. Not necessarily. It really depends on how the

org is built, so like, for example, let's say we're dealing with something that's got

multiple storefronts, and a lot of those storefronts are connected over global internet. You might actually configure those machines to directly interact with Azure Automation and not necessarily use a hybrid worker. Now, the hybrid worker actually kind of serves

a couple of different roles. It can be used for runbook processing, just so you don't have to deal with the queue and waiting for that VM to spin up, but it can also be used to issue marching orders from Azure Automation to your on-prem boxes, and this is really

where I see it being used more often, is because, you know, for some reason security people don't like it when your servers are talking directly to the open internet, so what they'll do instead is they'll use that hybrid worker in a DMZ environment to

basically act as the pass-through and issue marching orders. Now, if the site configuration is that way, I would say that you might want to position a hybrid worker for each data center to be your front-facing point. As far as, like, limitations of clients, I

haven't seen any ceilings yet, so, you know, I've seen them being used to environments where it's like 3,000 or 4,000 nodes, and not seeing it, like, completely explode, but it also depends on how often that stuff is processing, because more often

than not, a lot of those data, like, a lot of my clients' data centers haven't fully implemented using Azure Automation for on-prem stuff. It's more experimental, so you might see only 100 or 200 nodes. It depends on what you're kicking. So patch management

usually needs a little bit of scheduling ahead of time, and there are some check-ins involved with that, so 30 minutes to an hour. I have seen, because I have done

implementations where it was like Azure Automation in Azure Cloud managing virtual machines that were in, like, 21 via net, and that can take an insanely long time, but that's primarily because of the firewall. Now, if you're talking, like, DSC configuration,

that's going to be whenever you execute up to 30 minutes for it to be able to pull its refresh. For an Azure Automation runbook, usually less than three to five minutes, and it depends on scope and scale too, because it's got to be able to issue the marching

orders across, and there's going to be some balancing in between. Once again, it depends. So as far as I know, unless the pricing's been updated, patch management is still

free across the board. For a desired state configuration, all Azure VMs are free. On-prem, it's based on compute hour, and it's prorated. So if you have typically the boxes will do a refresh every 30 minutes and check in every 15, or run a consistency every 15. That

equals about $6 per month per node. However, if you scale it back to where it's only checking in, like, once or twice a day, then that significantly drops it down. It's like 50 cents per month per node, which is generally where I see most of my clients

going for their non-critical stuff, and what they'll do is they'll actually, based on the criticality of the server, they may actually change the LCM configuration, so maybe they have something that is like PCI DSS compliance, and they want that constant auditing and reporting, so they'll go ahead and pay the $6 per month per node for that,

but this other thing is like a file and print server, and they don't necessarily care, so it's checking in once a day. So it depends on the service. Any other questions?

All right. Well, it's 1246. Thank you for suffering with me even though my demo blew up.