Hello everyone, thank you for coming to my presentation. This is metaprogramming, also known as code writing code. So while it may seem a little bit scary from the get go, it's actually really not

that hard. It's just a few things you have to consider and believe it or not, most of you are probably already experts at it. So my name is Wesley Kirkland, purposely blank here, and yeah, I do kind of like Amazon a little bit. So I have been doing professional IT about three, four years now, currently a systems

engineer, but I've done sysadmin work, Azure, AWS, just more of a generalist throughout my career so far and quite enjoyed it. One thing you'll hear me speak about and praise a lot, I'm a really freaking lazy person. So if I can have something do my job for me, not someone, but something, I'm going

to do it. Jeffrey Snover actually has a really wonderful quote on it too that I kind of love to live by. So metaprogramming. Before this talk, who actually knew what it was? Okay, so we've got a little small portion of the room.

So effectively metaprogramming is just writing computer programs that write or manipulate other programs as themselves, as their data. So if you actually go online and Google a lot of this, what you're going to get is a lot of stuff related to compiled languages like Java or C++. So I've tried to extract out or abstract out a lot of that to bring it into something

that's more usable for us with a lot of us being operations or DevOps engineers, some kind of automation, so we're going to try and get there. So to me, when I first started doing it, I didn't even know I was doing it. I just found the term later on.

But there's some actually really cool example use cases that a lot of us have already done. So number one, SQL queries. So I'm going to take a while in bed and say everyone in this room has probably written a SQL query in one language and then it just goes to SQL and then you read the results back. Another one is going to be writing simple templates to fill in.

If you really want to, you could extend this, use com objects and go straight up to Word. I don't really give a flying crap about any of that, but yeah. Jeff Hicks was actually just talking about that where people use Excel's databases and yeah, just never interact with Office programs in my opinion.

So or you could go in, write in a different language from one source language. Met somebody one time, they were using Python to write out PowerCLI and then I said, that's PowerShell, why wouldn't you just use PowerShell natively? But they were doing metaprogramming and actually writing out their code. Or you could write an entire framework or scaffold a bunch of code,

also known as templating out into the, or out from your source app. So there's some pros and some cons of metaprogramming here. So I like to divide in the good, yay, the bad. This is sad and I hate that for you. The good, more expressive.

So when you think about this, more expressive would generally mean that you're going to write a lot more code and it's going to look nice and fancy and pretty. We're not gonna do any of that. We're actually gonna do the opposite. Less code, it's less to write, less to maintain. So you can think about the expressiveness like this. So your code can drastically change based upon the environment that it's actually running in

or that, what calls it? Consistency. So I worked with a guy one time and he somehow didn't believe in variables. And so his code was shit, so.

So he would literally write the, if he wanted to touch five servers and he had 10 lines to touch five servers, he would just use find and replace to replace it with the server names. So consistency. With this, find and replace or control D, you're probably gonna screw something up right in there.

So with metaprogramming and we're actually going to make our code very consistent based upon what we're generating. The productivity of it. So your boss comes to you and he says, I want this new report. And you already have half the data and in your metaprogram, you just go in, add a few lines,

depending upon how you templatized it and built an architect of course. You can add like one thing and then you, I'm losing my track here for some reason. You can add new functionality with only a few lines of code rather than actually rewriting everything. Less code. So I have, you can have say 100 lines

generate out a couple thousand lines for you on the fly and then just go from there. The bad, boo, this is the sad part now. Complexity. As complex as can be sometimes, depending on how you architect it, you can also run into issues

with character escaping and indentation. We'll get to a lot of that later. Safety. So when you actually write your traditional programs or your scripts, you go to a code review, everyone can roughly read what you wrote, correct? So safety of this, you don't have static code

so you can't really go through the same code reviews without actually executing your code itself so it can generate its output or transform on the fly. Bugs. I wish I had like a little can of flies right now. That'd make this a lot more interesting. So small typos or errors.

You may not ever notice them until you run into one specific condition itself. A good example of this, so, actually no, get to that one in a minute. Or actually another one. So you missed a parenthesis, but that parenthesis is buried inside of an if statement somewhere that

that if statement only gets triggered under a specific condition like a SQL statement and it's a very, very specific condition so you can go months without ever noticing it and then you're like, why's my SQL query now failing out? Well, it's because it outputted a parenthesis right then and there when you weren't actually expecting it. So an extension of everything we were just talking about

at its core, metaprogramming or code writing code is actually simple. Just sounds a hell of a lot cooler when I say metaprogramming in my opinion. So it's a technique. It's not necessarily a language by itself. If you try and define it as a language, you probably shouldn't be in this talk. But so reflection and templating.

Templating, I don't even think I really need to cover. It's incredibly simple by itself. But reflection, it's kind of just like an if statement. It allows your code to adapt to itself depending on the environment that it's in. So bugs.

I really hate bugs. My personal example of this and I kind of pre-eluded to it earlier with the switch statement. It was actually on a switch statement. So I was writing out, it was a program. I was rewriting Okta's sync engine. Well, in my dev database, thank God it was dev

and not prod, I forgot to run a switch statement which populated half the SQL commands. So we have multiple domains importing into it. Forgot to run a switch statement and I dropped about half a million rows of data in a few seconds, which sucked a lot.

So don't loot your database and Grumpy Cat says so. So like I said earlier, you're already doing this. Basic SQL queries, basic code generation, Windows. On Windows, you'll see this a lot with System Center. You'll also notice that PowerShell

is literally how Microsoft builds everything now. And how many of you have actually ever seen that view script button inside System Center products? Okay, so quite a few. And for the rest of you that haven't, it's literally right there. Hey, cool, this actually works on that. So Microsoft actually does a lot of metaprogramming for you already, because they're not gonna

run a different API. Just use one freaking language and be done with it. So when should you actually do this? So if you have some repeating patterns, for example, this would be a wonderful thing. Like if you have, and I'll have this example here in a minute, SQL code.

And it only slightly transforms. You can go in and you can actually dynamically update your code with metaprogramming, so it actually transforms based upon the environment or the tables that you're actually going into with it. Tricky code. This one's actually a little bit of a weird example itself. So, actually, yeah, it still is.

So if you have some code that, or if you're breaking down into a .net method, for example, and your overload statements, they're different in different forms. You don't exactly want a human doing that in, or filling that in for you when you can have your program go ahead and fill it in for you

automatically, or if you have matching event IDs, you can structure it all out. Minor variations. Talked about this one a little bit earlier, but if you're just very slightly changing your code, just like even a for loop would be a wonderful example of this, this is a wonderful candidacy for actual metaprogramming,

or if you have, my favorite one is your boss comes to you and says, I want MVP into next week, and they'll say, yeah, it's just gonna be temporary, that's perfectly fine. We all know that's BS. So, I like the word tempermanent.

Also, automagically, it's automagically tempermanent. So if you have an anticipated change that's gonna come up, you can actually use this style of coding to help you structure towards that change out in the long run, that way you don't have to go in and actually rewrite everything that you already did.

Or if you have non-scalphing code under your anticipated change. Things to consider. All right, so this is the super fun one. You know, I really hate PowerPoint. I forgot that. So, first off, we're gonna go ahead

and just look at them here. This is the methods and demos that we're gonna cover real quick. So in the first one, character escaping. I was, I think it was Carl I was just talking to about this. Character escaping is held just normally in normal programming,

but it's even worse in metaprogramming itself. So, can everyone see that in the back, or do I need to zoom in? Okay. I also forgot to say, if I say something stupid or if you have a question, please just raise your hand. I'll do questions throughout the entire thing. That seems like it's probably good in the back. I'm assuming. Okay.

So, right here, we have our first issue itself. We want to wrap this down inside of our output code. Primarily, we want, you'll notice we have our variables in here, like binary path or binary executable, but we want those variables

to actually execute upon the output of our code. So, normally, if we didn't care about the actual output of our code here, we'd say, okay, hey, we'll just do some cool double quotes. Crap, now my code's actually gonna execute inside of my controller script. So, that's when we have to go,

if you know anything about PowerShell, single quotes versus double quotes. So, we're gonna use single quotes, but I don't really want to fix all this. So, we're just gonna go to our next pre-made demo of character escaping. So, you'll notice this is like, oh, okay, now we've got, we've escaped our single quotes everywhere. And then, when we actually output this, you know, it might help if I have pretty stupid VS code.

I forgot to pre-populate these. So, now, when we actually execute our code,

we actually get our variables printed out on the screen. And, actually, I didn't have to pre-populate this. Go figure. But, if we ran our code earlier, it's just gonna blow up spectacularly. Lots of red. My users don't like red.

They get very pissed off at me. So, indentation. Before I start this one, how many of you have actually gone in and downloaded an XML file from the internet, and it's just literally 10,000 characters on one line? Don't you hate that? Yeah, the only thing that fixes it, I found, is Notepad++, and even then, it takes a little bit of time.

So, when you actually do this, please, yes. Oh, sweet, is that in this menu here, or ah, sweet. I'm still new to VS Code.

Also, we're kind of cheating. We're using PowerShell 5.1 down here, too. No core. So, whenever you do this, if I ever see your code and you did it on one line, I'm gonna ask for your geek card back, because you've literally just made everything worse. So, when you're actually indenting with your metaprogramming,

you have to think about it a lot. So, for example, what we'll do, we'll go ahead and run this, just a simple array list. Now, this is a very, very simple indent by itself. Yeah, I don't even know why, for some reason.

Words are very hard today, for some reason, so go figure. Oh, okay, sorry. So, this would be your indentation if you just think about it, and you just roll with the code without actually doing anything. So, let's go ahead and generate our code down for our process block, and then we'll output it all out and all that fun crap.

You stupid thing. I've got an extra tab. Or, if you watch Silicon Valley, you have five extra spaces, but you're clearly wrong. So, this sucks.

It looks ugly, but if you really want to, you can just go fix it like that. But, eh, who cares about all that? So, now, if we actually go in and want to properly do our indentation, look at line 39 here, specifically. Doesn't this look a little bit funky to you all, like, in my source code?

So, it's supposed to, but what this will actually do, notice we don't have an indent or a tab right here, so it'll actually come out properly indented in the long run. That was horrible.

Sorry, before we go through that. Down here, as we normally would, we're just indenting normally. That's super easy to do. Something I have actually noticed, so down here on line 28, you'll notice that I'm storing this whole chunk of code as a string value, because when I tried to put it

just in, or use the add method on an array list, it gets incredibly pissed off, and it makes indenting a hell of a lot harder. So, it's a cool little pro tip there. All right, so now that we have that,

my surface is really hating me today. Why is that wrong? That's not supposed to be wrong. I swear I tested this beforehand. Okay, that sucks. It's supposed to look like this.

Sorry. So, yeah, I'm gonna have to figure that one out for next time. Oh, wait, no, because that was still the bad indentation. Oh, and I deleted half of it. Oh, that's why. Yeah, control-Z all the things.

Okay, so now what it looks like, well, damn. No, no, no, we have a meme for that later. We can't spoil it quite yet. My clipboard is all jacked up. That's why.

Because, sorry, I'm going off and squirreling out here. Oh, that's just horribly. Okay, I think I have something to do with void. Oh, screw that. Okay, so everyone remembers my whole, oh, wait, no, it's still on indentation here.

Okay, so here's this little cool SQL statement that we're gonna have, and we're just gonna execute it in what we would think would be source code indentation, AKA our controller script. I keep doing that.

Okay, so now that we have that, when we look at our screen, live splat. Okay, so when it actually comes out, it actually looks relatively all right, but to be fair, we didn't actually execute half these variables with it. The only thing that's jacked up is there's one extra tab here,

which I think I fixed in the next one, so let's go check that one out. So now, this is the non-expanded code, so this looks like crap inside of my VS code here, but it actually looks pretty all right when we do it. No, you suck.

Oh, or not, probably have those reversed. So, yeah, so let's go on to complete code here. Does everyone remember my little story about where I nuked my database? This is the exact code that did it.

So I just have to haunt myself with it every frickin' day of my life. So what we'll actually notice in here is, okay, so we can run this in three different ways. So the first time that we run this, we're just going to have literally no variables.

Let's actually make that cleaner. So when we run it with no variables, we literally just get a whole lot of effectively garbage itself. Specifically, it's like, oh, Target, hey, there's missing a table name. So SQL's actually just gonna straight up blow up at this part, but what we can do here is,

let's say we add in these variables from the switch statement right here. And let's rerun it. You know it might help if I specify a few of them too.

Okay, so yeah, now we've populated a little bit more of our data in here. Let me just go down to static demos too. So what this is actually supposed to show here, if you just run the merge that we generated with our SQL statement above, it will look just like this.

A lot of garbage, SQL's just gonna error out. Now, if we ran it without running the switch statement, but we populated in like our domain variables and other stuff like that, we get this. This specifically is the exact code that nuked my database for me, mainly because of the, in SQL, when not matched by source, then delete.

Kinda screwed me over a little bit. But if we fully run our code that exists beforehand, it becomes a heck of a lot more populated and then we end up with the complete code by itself. So you always want to make sure you have your complete code before you're actually executing anything by itself but methods.

So this is, when you're doing this, you're gonna have to figure out where to actually store your code by itself. Store it in RAM, store it out to a file system, store it to a database, store it to a RAM disk, whatever the heck you want to store it to. So I have a few little measurements here

and I saw Josh King's talk yesterday and I wanted to write his code into this but I didn't have time yet, which really sucks. So what we're gonna do, we're just gonna go ahead and execute all these methods and we'll go over them while they're executing. So the first output method we're actually gonna test is an array.

So in PowerShell, if you add to an array, it literally copies the array, adds it, and then puts it back on the original array. Most people know that. So it kinda sucks for our purposes here. The next method is going to be an array list. That's highlighting way too much stuff. So with an array list,

we're just gonna use the .net array list. It's effectively a vector and we're just gonna add to it so it's resizable. It's actually probably gonna be a little bit more efficient. Now with this too, a lot during this conference, we were talking about pipelines. So pipeline by itself actually adds a significant amount of overhead. So I'm doing a lot of out nulls

or you can also do content redirection to null all sorts of fun crap itself. So I actually get to see the difference between these by themselves. It's actually taking a little bit longer than I expected. Yay for demos. Next method is gonna be an array list.

So instead of actually letting the pipeline come into play, we're just going to top cast everything as void and let it, oh, that's a fine error. It's perfectly normal. No, yeah, it's close enough. So we're gonna cast everything to void so PowerShell just convene and says,

oh, hey, I have no output. I shouldn't even worry about trying to output that off the console or anything like that. And now, the last one's going to be outputting to a file. I don't know about you, but I generally don't have a pure array at my dispose or just for my own use. So outputting to a file kinda sucks. You will notice in here,

all these other examples are like 5,000. Just literally for the purpose of this demo, we're changing out to file to just 2,000 lines. And since I didn't cover this earlier, all we're doing is, sadly, we can't kill as many puppies now. I get very pissed off about that. We're literally just outputting a write host

with randomized colors. So in here, you can see, so array specifically took close to 30 seconds by duplicating everything in the array itself. Array list for this sample set, it really only matters when you're getting above about 5,000 lines of code.

At least in my test, does array list, yeah, array list versus arrays just doesn't really start to matter. I've just gotten in the habit of always using an array list collection when doing my code. Makes it so I don't have to fix the crap later. The next one was array list void, specifically right there.

So this was completely just eliminating the pipeline itself. So we don't have any of the pipelines overhead. We're just saying, hey, we don't really care about the garbage that you output. All we give a crap, all we really care about is our output file. Then we went out to the file system. There's no way that second counter's right

because that took minutes. Probably should have concatenated that. Anyways, that took a solid one to two minutes by itself. Actually gonna make a note to fix this. Yeah, so you can see just the speed difference based upon the actual methods that we're storing. I also wanna try storing to a database

mainly for historical logging to see if that would actually fix a lot of things, but you're gonna add in a lot of network latency there. And it's probably not gonna be too efficient. It would only be if people really care about what you did in the long run. So now, if we go over,

it's time for more demos. And probably should have shown that one a little bit earlier because everything's just going horribly wrong. I clearly didn't sacrifice a chicken to the demo gods. So puppy, let's go to puppy killer. So we all know PowerShell kills puppies,

and that's why the animal shelters really frickin' hate us apparently. So I used to love this demo, and then Jeffrey had to ruin all my fun in PowerShell 5.0 because it no longer kills puppies, as by this Twitter comment, but for our purposes, we're just gonna pretend this doesn't exist. So this is just incredibly simple code by itself.

It's what we were actually doing in our last demo. But what we're gonna do is we're going to extract all the colors out from right hosts, and then we're just gonna randomly print them out to the screen with the counting integer. It's ludicrously simple. So now we gotta wait for this considering it's going out to the file system,

which is dreadfully horrible. So literally the whole concept of this is inside of memory, or inside of our script here, you'll see string. So all we're doing is we're literally just using .net string replacement to fill in variables, and then we're just gonna output them straight.

We're just gonna output them to the file system. And then we get a result that looks like this, and it's horribly hard on the eyes to read. So earlier on, you heard me talking about where I'm truly an incredibly lazy person. So with this, I need to write a lot of pester tests.

So what's the module we wanna write pester tests against? Just name one. No? Okay. But thanks for being super fun, guys. So let's, I already have AWS PowerShell up here,

but it takes quite a while, so let's find another one. So let's say we wanna write a pester test for our failover cluster module in here. Okay, so what we're gonna do inside of this, we're going to find every command inside of our module that we want to, and then we're literally

just going to build a simple pester test for that module itself. So when we do this, we're going to take everything that we learned earlier about indenting, character escaping. But we're also gonna do something else. So a lot of modules, of course, have required parameters. I actually learned there's a lot less required parameters than I thought there were out there.

Just kinda sucky. So anyways, what we're gonna do is we're gonna go ahead and execute this one out here. And what it's actually going to do is it's going to generate an entire pester test framework for us, that way we get to avoid a lot of copying and pasting later on. And so it only took us 11 seconds

to build a pester test for 118 command lists in that. And we generated 99 lines of code here. So we go and look at this. I did not output that.

Sweet, there we go. Cool, so we just made a pester test framework for an entire module. And quite frankly, I don't wanna go and do that. But you'll notice something in here. So getClusterDiagnosticInfo actually has a required parameter. So in our pester test, we actually know

we're gonna have to feed that into it. So we just pre-populated everything for us. Any questions so far? All right. So this is a function that actually really got me started into this. It's called PowerShell Function from a Console App. So I don't know about the rest of you all,

but I really hate using apps like Robocopy, XCopy, AZCopy, due to their help file is horrendously awful. Like this. Who the heck wants to read this frickin' help file all day long? I want PowerShell and I want parameter-tized help. Or comment-based help. However, we need to do it.

So what we're actually gonna do is I'm gonna show the actual demo, or show the code output before we go into this. And kinda sucks is due to a VS code bug, half of this does not work in VS code. So we have to drop on over to our lovely IEC.

So let's go ahead and preload all of our function. So what we're actually gonna do, we're going to build a PowerShell function on the fly. And we're going to keep it all within memory by itself. So with this PowerShell function,

like I said, I really hate Robocopy's help system, so we're just going to build a PowerShell wrapper for it. So inside of this, we're going to say, okay, hey, Robocopy lives in system 32. Oh, yes, sorry. Probably should change.

Is that a lot better? All right. So, and then we're gonna say inside of system 32 is Robocopy, and then how we define its help argument is via this. So now, here's the really cool part. Has anyone ever played with dynamic parameters? Yes. All right, so this is actually invoking dynamic parameters under the hood, which I'll show you in a second.

So now we can literally just start going through this. Notice all these parameters over here on the right side? Those are actually Robocopy's parameters coming from it. And just to prove to you I'm not screwing with you or anything, go do xcopy.

So there's xcopy's parameters. And just to prove this even more, when we go inside of this, what we're actually doing is, well, we're specifying, hey, it lives here. Because apparently PowerShell's issues with system 32 for some stupid reason. And when we do this, we're literally just executing out

the binary itself and reading its help file. So for this, it's calling a helper function located right here. Inside this helper function it literally reads that help file, deconstructs it, puts it into actual section groups itself,

and then builds them out as a hash table with array lists. So it's just literally deconstructing all that for us. But then, so this qualifies some definition of metaprogramming that we can take another program's data and manipulate it for itself. So we're not even truly doing code generation right now.

Instead of our parameters, I literally don't have a slash MOV or MOV whatever. We're just looping through the parameters that we got back from before. And it's a concern some programs don't really like the fact that, or they don't specify like a source or a destination like Robocopy does.

Added in a couple optional parameters that you can specify. They're just string format. But now here's the really cool thing. So this actually does work, believe it or not. So if we invoke this,

actually do have this document somewhere, but I don't exactly remember where it is. So we're going to copy the contents of temp.

Oh, that's wonderful. Let's just do this then. Because I wrote this on my other laptop. So we're going to copy the contents of this PSG Suite folder over here

into our other folder. So the one way I actually built an architect of this program is if like switches. These switches by themselves, quite literally a switch.

So we're just going to use a null output. But yet it's the console wrapper for it and then it still invokes the primary function under the hood. But there's an even cooler part with this. So if we go back over to VS code, how many of you would actually like to have this as a static function that you could actually

execute and modify? All right, so I already did the work for you here. So we're just going to take, first I'll show you the function. But it literally just takes the helper function to convert from console application help from earlier. And then we're generating that as static PowerShell parameters using all of our indentation knowledge that we learned earlier.

For example here, you'll notice in our indentation we have to use the combination of actual, I mean sorry for our string, we have to use a combination of indentation and actually .NET string replacement so we can actually properly build this out. But what we're going to do here, and just to prove to you that I'm not doing anything fancy,

my clipboard's 100% empty now. So when we run this, I hope if I load it, so now what it's doing is it's reading out Robocopy and it's going to build a completely static

PowerShell function for us. So if we go over here, we just wrote 583 lines of code in a few seconds flat. Now here's the kind of sucky thing about this. Earlier on I was talking a lot about bugs. That's not the right file.

So with this, we do encounter quite a few. Specifically, Robocopy has these stupid parameters of A minus and A plus. PowerShell really frickin' hates that with a passion. So we have to go through, we have to remove them, and we have to remove them because they don't conform to its standards.

So it's not 100% valid for everything and you can write other logic for it. Same thing with Unilog down here. There might be one more that we have to fix. Oh yeah. More logs. Great. All right. It actually qualifies them as duplicate parameters specifically.

So now that we have that, we can literally go in, do the exact same thing of invoke Robocopy binary. Say hey, this is where Robocopy's living.

It helped argument derp. But now we have the same thing as, we can say okay, here's our original that we were getting. Let's clear out our destination folder real quick.

And then we can literally specify our mirror command again. Same results. So that's actually really the end of the demo and the presentation. Any questions? Sure. When you're doing it in memory, are you hated?

Invoke expressions, terrible. So that's how we, that's. As far as executing it, or. Actually, like as you're writing it out, you're building it. Once you're done building it, how are you? I generally just dump it out to a file. Or, yeah. Actually, I think I forgot one on here real quick.

No, I don't think I did. Any more questions? Nope. All right, well. Yes. Are there any more real world applications?

So I have one right now. I have, I'm actually doing this project at work and it's an email consolidation project. Something you all might have heard me talking about earlier. But with it, I have a boatload of API keys. The script actually floats around on quite a few different systems.

And I am, and same thing with the file pass, but I want to store a lot of that information in the database and encrypt what I need to in the database. And actually build out a lot of the information needed for the script to run in memory. And then, like you were saying, I can use invoke expression or invoke command

to do that as well. Specifically with G Suite, you have to have all these stupid PIM keys. Which, that's what I've been playing. Sorry, P12 keys. And I could store those at base 64 if I really want to. Just a super terrible idea. But just for fun, store it like that and then actually write that out to the file system.

Or with my API keys, store them in the database and then invoke them in memory. Any more? Sweet. Well, here's my contact information. If anyone's curious or interested. All the code is available on GitHub. Presentation will be there in probably an hour or so.

But sweet, thank you all for coming. Thank you. Thank you.