All right, I guess we'll go and get started. So today, we're going to be talking about domain-specific languages and how to write one in PowerShell.

My name's Kevin Marquette. I'm actually going to cover quite a bit of material. So some of the intro stuff, I might go through a little bit quick so we can actually get to the good stuff kind of in the middle. My slides and presentations are always up on GitHub. If you want to download it for the code samples that are presented in this presentation, feel free to do so.

So what is a DSL? A domain-specific language is used when you want to solve a problem in a specific domain space. Generally, there's like terms and language around a certain problem domain that might actually help you solve that problem better than a general language can.

So PowerShell is a jack of all trades. It can do many things. But sometimes, you need a tool to do that one specific thing really, really well. And that's kind of where DSLs kind of answer that problem. We already have several DSLs in the PowerShell community. Like if you've been working with DSC and Pester,

those are great examples. I have several up here that I've seen in the community. And I'm going to actually, we're going to walk through a couple of these real quick just to see what a DSL looks like in PowerShell. Okay, we all know and love DSC.

And as we look at this as a DSL, we actually get to describe our problem set. Let's just say we're a configuration that's configuring a node. It's setting up a Windows feature and has some like file web content. Like the language helps describe

the problems that we're solving with it. And it does a really great job of doing that. Another DSL many of us are familiar with is Pester, right? It uses beautiful syntax where we can describe what it should do. And using that terminology, we actually focus in and write better tests

using a syntax that is, that rolls off the tongue a little bit better. Couple more examples of DSLs is like Invoke. Invoke, Build. So if you use Invoke Build or Saki, right,

it's like a build automation DSL. It's built around, you have tasks that perform certain things in your build pipeline. And there's certain terms and nomenclature around that effort that is captured really well in both of those DSLs.

An example near and dear to my heart is PS Graph for a module that I wrote that is implemented to describe a node edge-based relationship of things you want to diagram visually. So with this one here, I actually describe three nodes

and I can describe how they relate to each other. And when you execute it, we can actually see the diagram based off exactly how we described it.

Then last example from the community is this Plaster Manifit. Has anybody worked with Plaster? So for those of you that haven't, it's templates that you can use to generate functions or modules or anything, anything really.

And right in the core of it is this manifest that's built in XML that describes what inputs to receive from the user and what files to transform and deploy. If you don't like working in it with XML, here is a community DSL that is specific

to defining those configurations. This one here has the parameters and the content. And if I take a quick peek at this, we should get a large, it's actually generating this XML for me to use as part of Plaster.

Like this is a great use case of a DSL. And I wanted to run through these just so you kind of get a feel of, they look kind of unique, a little bit different from the PowerShell we're normally writing. And so let's take it a little bit deeper.

Let's really deconstruct how some of these DSLs are doing the magic that they're doing. Because when we look at Pester, right, it's really just a function. That first command is just a commandlet like you would normally write. The next description that we use is a positional parameter.

And then at the end, we've got a script block that hangs off the end. Both of them are positional. So in a way, these are, this is PowerShell. But we're using it in a very different way than you would in most use cases. This pattern repeats through this DSL as well.

Where my it does something also has parameters that are positional, a script block that is also positional. And then our should statement is just another function that's taking pipeline input. And when you break it down that way, it's easier to process what's actually happening

when you're running these DSLs. So if we tear Pester down a little bit more, so I'm gonna start with our same example here. So starting with our same example,

if we look at the help, the syntax of describe, we can see that it is exactly how I described it. We've got a positional parameter name for our string. Our fixture is a parameter name for the script block. And as we take that knowledge,

we can actually expand that Pester statement into like a fully, the more verbose syntax the PowerShell tends to use. And on here we see we have the name, our fixture, and we kind of repeat this, right? So all these things actually do have name parameters.

But in the style of a DSL, we kind of don't use those. Now, if we keep going down this path, right, and tear this apart even more, imagine if this actually was more PowerShell-like, like more like the code you write

that's more procedural. You know, if we start from the bottom, could you imagine if I should write our test this way? Like if we wrote our describe block and had to pass it a script block as a parameter, right? Or then our fixture itself contained another statement that took a script block. And I don't know about you, but I probably wouldn't have adopted Pester so fast

if I had to write every one of my tests exactly this way and as I start to see how a DSL can solve certain problems, I'll let you see the answers in a different way. Because when I compare this to something like this,

I'm so glad that I can actually do my Pester tests like this. Now, I'm sure if we had to do it the other way, we would have had different solutions to the problem, but this is the problem that a DSL can address.

So, we kind of covered a lot really quickly there. There's a lot of great points when you just start deconstructing those DSLs in that way. And the first is, like I said, DSL commands are just functions. And we also don't follow community best practices.

Like, it's really weird when you start writing a DSL because I've had years of, here's the best practice to do something, and I'm fighting all those for the right reason. But it feels like I'm writing a really dirty code when I know it's really clean and solving a problem.

The names, right, we don't use the name verb so often. We name them specifically to match the problem domain that we're working in. We're always using these unnamed parameters or positional parameters. We'll often leave off the quotes in our syntax and style.

And I say we abuse script blocks. I say it that way because you normally don't hang script blocks off the end of a function unless you're doing it in a DSL this way. I think I've gotten more used to that syntax. I'll do like an invoke command now and I'll hang the script blocks. But in the earlier days of DSLs,

I don't think that was as common of a style. There's some other hacks you can do by hanging an array off the end or a hash table. I don't like it so much, but if that solves a specific solution and that syntax makes a lot of sense, it's viable to do that.

I've used it once. I've gotten away from it just because the best practice feels more like using a script block there than any other hanging structure. Then the other common pattern we see is a lot of DSLs have a invoke type function, like this helper tool that consumes the DSL a bit

and wraps it in more functionality. Pester, you can run a describe block and you get that fancy output or the beautiful output of your tests, but you miss out on other features such as the number of tests that passed or failed or status of failure.

They give you formatted XML output files, code coverage. Quite often, those helper functions take the DSLs and actually add more value to those. Okay, so we've got a good picture in our mind of what the basics are and all the little pieces.

So if we're wanting to plan a DSL, the first question I'm gonna ask is make sure a DSL is what you need, right? Just because you can do something doesn't mean that you should. And I know I've gotten excited and really enjoy working with those and I probably use DSLs way too much,

but it solves a problem in some cases. And because you break the community best practices, it does confuse your audience unless they understand that domain that you're working in. Prototype the syntax before you code it.

This is probably the most valuable tip that I've learned when writing and working with DSLs is write the code as if you're using it. Pretend like everything exists and get a feel for your interaction with it. And do a couple different versions of it. So you see the syntax that you think you can implement and that fits really, really well with the flow you want.

Yes. That's a general best practice for writing scripts or modules. We used to have teams write all their modules and then they'd come up all screwed up. And so then we adopted the practice we call demo.text.

It says before you write anything, write up the demo.text. Show us what somebody will type, what they'll get, what the flows will be, and then we review that and that's where you find all your problems. Absolutely. You get such better consistency across stuff when you do the flow that way. And save those samples and put those in your pester tests.

This test-driven development. You write code using stuff that doesn't exist yet. So you have your failing test to begin with. You write your tests so that you know what the output should be, which also fails

because you don't even have any good outputs. And then you write the code. And at that point you've already documented both the input and the output. Or what you're describing. It's the middle step. The first step, the demo.text, is really just focused in on user experience itself. And if you do the tests, it's hard to see the experience.

I'll take those prototypes and when I'm done, those feed right into your test flow. Okay. I'll always blur those too when I refer to them. Thank you.

Okay. So let's actually build a DSL. From the ground up. And for this one, if you've used RDCman, remote desktop connection manager, it actually uses an XML file to store all the information about your connections.

DSL planning. So, okay. So first let's prototype the commands that we would use. I start with the PowerShell type syntax because maybe that's gonna be my answer.

And it might actually be a better solution in this case, but we're making a DSL here. If I was gonna do functions, I think this is kind of the flow I would come up with. Where I can define here's the servers to create this file or maybe a group that I could put servers in. That'd be a good set of commandlets.

Let's think of this as a DSL. So my idea number two, is we do something with a little bit of structure to it. I define my file with a group and maybe list of servers in it. Here I'm using that hanging array. I'm not really sold on that. And the more I play with it, I don't feel like that's quite the right solution here.

So my third idea I came up with, and that's prototyping, was something more like this where I describe a group, columns from production, and that group contains my server, my RDC server, server one and server two. And what I like about this is that,

if I wanted to extend this later and add other properties to this guy, knowing that he's just a function, we can add like say a describe or a description to add more data to our DSL later.

Okay, so this is the syntax I've decided on. This is what I'm gonna commit to developing and planning for right here. As we continue on, I'll kind of expand out all the possible properties just so I know what I'm actually coding. So I sat down and made up these names here

off of my template, off my little piece of scratch code, and I'm now gonna go implement those parameters. Does everybody track with me so far? Beautiful. Okay. So the RDCman XML file,

you can think of it as this large XML document. Inside that, it contains groups. That's another XML structured, and that group contains XML servers. So I've got this outer template to this inner group to this inner object. So the first one I'm gonna write in this case

is my RDC server. And my parameters, based off of our spec, I'm just taking the computer name now. This could be expanded to more parameters later. So it's positional, mandatory. I could take it from the pipeline, but that's another discussion.

All right, so let's break down to what our process looks like. So all I'm really doing in this example is wrapping a value around, injecting it into some chunk of strings, which happens to be XML. I definitely could've actually done the XML node object and made full objects on this,

but for the simplicity of this demo, this is what we wanna do. It's much easier to follow. And all I'm really doing is creating a string and tossing on the pipe for somebody else to deal with. I'll have another function that cares what's coming down to it. So I literally just have a small chunk of valid XML for my server.

So I need to invoke this, because we're actually gonna run this here in a moment. All right, let's get that loaded. RDC group. So here's where we start getting a little bit of the fun.

So I still have my group name as a positional parameter. Then our second one is another positional parameter in the subscript block. I'm calling this one child item. I could probably pick a better name for that. And then, I'm doing the same thing before. Like, I have a small chunk of XML,

I inject my value, toss in the pipe, and then I actually go to my script block and just say, invoke the script block. Whatever's inside the script block, we're gonna run that, because there's other commands in there that need it to execute and run. And in this case, this command doesn't care what happens. It's just gonna toss on the pipe as well. So if somebody else upstream is gonna handle this.

And in a way, I've just got this giant string builder. Right, and we really break it down to it. I'm building just giant strings. So let's load this guy up as well. So now the outer object.

And while you think he might be something fancy, he's actually just as plain as our group. There's just more XML overhead. I'm gonna invoke my script block, because it's like what the collection is, the child items. And more XML footer. So at the end, all I've done is just compiled

this XML data and tossed it down the pipe. But because I did my child items in the right way, I added the structure to it. So if I look at, let me go back up to this guy slightly here. Like you'll notice, I start a group, I have properties, and I know I,

whatever I invoke in here definitely needs to give me valid XML for this to hold, for this to work. And then I close my XML object. So let's run this one as well, and see if this comes together like we expect.

Okay, so here is that first example we started with. Here's our prototype. When we take the next step, we'll save it to a file, and then we're actually gonna open it up with remote desktop connection manager. So, oh, as we do this, assuming I've loaded properly,

we execute that, and actually created my RDC connection manager profile exactly the way I described it. And that's where it gets cool is, this matches the description of what I wanted to generate.

And that's where that DSL solved that problem really beautifully. We're having all those commandments to do it, you kinda get that break from the pattern. And, because of the way I wrote it, let's try something a little bit more complicated. Let's say we get a group more like this,

where I'm defining, that's so tall. So I have got a group that contains a group that contains servers. Because the structure is valid, we were doing our XML, I can actually, I can do this.

So when we run this command right here, we get exactly what we described in RDC manager. So that's where DSLs get really cool is, that's what I told it, that's what I wanted to see.

And it aligns so perfectly with the output that I produced. I suppose, any questions at this point? We're all following, kinda what we built here? Yes? Just something I'm noticing, that DSLs really excel in hierarchical.

Absolutely. You might not get as much of a benefit in that case, but yeah, hierarchical stuff is also, that's why my mind jumps to DSLs way too fast, but that's where I find the best solutions of stuff that has a hierarchy to it.

And then, if we wanted an invoke command for this, it might be something where we save those into like an RGG file, and we could run a build RDC man where it finds all those files and saves it to an output.

I don't know if that's actually useful in this case for DSL, but for many of them, you'll see a pattern like that.

Actually, that's a great interaction, because you'd be with a client, and they could be telling you, describing it to you, and you'd be typing it out in a way that they could almost understand it.

So you can grab servers, maybe, and wrap your DSL in a loop of those services you're generating. And that's exactly it, we're using script blocks, and you can mix, that's one of the biggest discoveries I had with Pester was it truly was just PowerShell commands, and I could wrap all the PowerShell around that that I wanted, you know, for each statement,

or if logic, like I could decide, can I even run a test based off of the conditions? Now, Pester supports that with its parameters, but there's a lot of cool things you can do knowing that it's just PowerShell. So because those are script blocks, couldn't you also dynamically generate stuff nested in the hierarchical structure?

Yes. Yeah, like so that could. Concretely, what you could do is you could set like that engagement you mentioned where you say you have that written up, and that inner loop of the servers instead of hard-wiring it, you could have a query out to A, B to a specific OU,

and then generate, you just have a for loop, and boom, the power's just crazy. You lost your screen, by the way. Yeah, I'll see if it comes back up. There you go. Okay.

So like this command, okay, I'm going off script here. We'll see how this works. I will do one dot dot 10, and we'll say, I think that's what I want.

Maybe not.

Just put a comment sign before the last one. That'll just give you strings. He needs the RDC server. Oh, I see. Thank you. Deleted the wrong thing, and I couldn't see it when I was staring at it. There you go. That's what I wanted.

So this stuff can be data-driven. And I mentioned earlier, I've added pipeline support to that DSL command because that's my pattern, and I'm going off-strip on that too,

but anytime I can actually reliably add pipeline support to build the process stuff like that, I get so much value out of it downstream. I'm writing it for one thing, but if I can design it for anything, I'll use it for anything later. Okay, so that's my big demo,

but I do have some DSL tricks. I want to call them design patterns, but I don't know if they're actually design patterns or not, but we'll kind of work through that. So sometimes I have a command. For lack of a better word, I'll call it a value pass-through, where I'll actually create a DSL command name

that takes a string and just returns that string because the syntax and flow from a user might make sense to actually use that word. A good example is my PS graph. It's really just a wrapper in front of graph biz, which actually has its own DSL. And if you're an expert in graph biz

and know the syntax for something that I don't support, or maybe I do, I have an inline command, we say inline, and put the inline DSL for graph biz. But under the hoods, you could have just put a string there and it would have worked anyways. So for the sake of my DSL, I added a command for inline.

I put documentation and tell people how to use it, and it's easier for them to discover than trying to tell them, you could put random strings in your script blocks and this stuff would just work. So there's value in designing for the user. The next two, which I'll refer to as the simple template and the nested template, is exactly what we wrote with this RDCman.

The first one is just a value shoved into something that's wrapping it. We're using strings in this case, but it could be classes or objects or hash tables where you're getting the values, shoving it into something and passing it up a string. Then what I mean by a nested template is one that does the same thing

but executes the script block. And that's what my RDC group was, was something with like a nested templating pattern. Now, some other tricks I've used in DSLs, I'll call it like a hash table pass-through,

where I'll take parameters, so they can write something like this, like for like, what? Oh, changed again.

PowerPoint has been playing games with me up here. Okay, so I could have a command like this where I say a state machine, where I'm saying state started with a script block, and upstream I just want a hash table,

and I found, I actually just take all the properties and the parameters and just return the PS Brown parameters up the stack for somebody to deal with as a quick way to create hash tables that are structured with the syntax that people can discover. Then on that same note,

I've used this pattern where I want a syntax that looks just like this, which is just like DSC, but I'm not using the dynamic keyword to do this. And I'll come back to that special approach later, if I have time, but there's an alternate way to do that,

just in PowerShell, where I build my script block and actually like rebuild it in line as an ordered hash table. So I'm making some assumptions on my users that they'll use that syntax, but if I want this to come out as a hash table through my pipeline or however I'm gonna use it,

I can literally say, let's just take my script block, make it a string, wrap it in ordered, and execute it and return that object back. The value in that is that,

you can get that object the way you want to see it. So if I run this here, and I actually execute my sample up above, he's gonna give me a hash table that looks, so I got a syntax that I want,

I can get the data I want somewhere else. And if I keep going down the list here, like an object collector, I mean, so far I've done a lot of taking objects and just tossing up the pipeline and not caring about it, right? And for string builders,

sometimes your output is just everything tossed on the pipeline. There are other times though, let's say if I'm building a state machine, where I actually want to collect all those objects and do something with them. And it's actually as straightforward as you'd assume where you run your script block and you execute it and you save it to a variable,

so you have a variable full of objects that came down through your DSL, and then you walk them and process them. If they're mixed objects, you want to do your is your types, but you want to walk through this and account for what's coming up the pipe. So when I say, somebody's gonna handle that later, it might be this pattern that's handling it later

to decide what to do with those. In this example, I built a state machine. I don't know if you actually demo this really well, but my DSL up above, this guy gives me hash tables with a state and a script block,

and then I actually, in my state machine builder, I collect those in a hash table based off the key is the state, and then the action is the actual hash table object can be processed through a state machine.

Is that fun stuff?

All right, present from, current slide. All right, so maybe you want not any command to run. I'm using script blocks.

People could put anything they want in that script block, any commands. And if you have a DSL where you're writing it to disk and you're making certain assumptions about what those files are, you can actually restrict what actually gets executed. I think initially with PowerShell 1, we had the data statement, early PowerShell, we had this data statement,

that anything in there was treated as data, and you could basically whitelist the commands to execute. I think more recently, you could also use the check restricted language feature of a script block to say, the only commands, commandlets or functions to run

should be these commands that I'm defining right here. So now you can actually place control over what could arbitrarily be in that script block. Now you break the ability to do, I could get AD computer to pipe into a server list, but you do gain trust in what's in those script blocks.

Internal only commands. So maybe you don't wanna leak all your commands out into the command space, or you want them to actually execute in a certain structure. One way to do that is actually if you define

your DSL commands inside of another command, the scope holds it kind of in place. It just won't exist or live outside of that. You lose your get help and discoverability on those commands doing tricks like that. Another thing you can do, a little bit expensive,

but you could check the PS call stack to see am I being called as a child element of like say my RDC manager. Like RDC group on its own has no value, so should he even run if he's not part of the larger structure? Is that how you make sure that your it is inside of?

Probably, I don't actually know, but if I was gonna make the assumption and implement it myself, I'd probably check the call stack first, but that's on my list to dive down and see exactly their approach. The dynamic keyword.

Okay, so this is kind of like classes in PowerShell in that it was implemented specifically for DSE to be able to do what it does well, and it's kind of fuzzy when you use it for anything else. What the dynamic keyword does is it takes your commandlet and basically allows the AST to treat it as a command,

and we've been using all these dangling script blocks. Being promoted to a true command in the PowerShell AST allows that script block to not hang anymore. So just like your if statement bracket could be on the end or on the next line, creating it with the dynamic keyword

would allow you to make it look more like a native PowerShell command, and you still have the option to have a script block in there or use a hash table. The DSE resource would be a hash table dynamic keyword, and then you get things like

auto-complete and syntax validation. You can define these are the keys that are possible in there, and the AST will tell you, will parse that and throw errors if something is used that you didn't define. So we have a nice, tight structure. The dynamic keyword can kind of help you with that.

But there are gotchas, and the first one is that it has to kind of exist before the script is parsed. So if you have a script where you're using it and you try to dot source your dynamic keyword logic into it, it's gonna fail because it's gonna parse your script first before it goes to the dot source. Very similar with classes.

You kind of run into those same nuances. And you also can't currently use other function parameters. Like if you're working with Pester, you can do a describe and give it tags and have additional parameters on there, or you can skip a test or it's a pending test.

In my experimentation with dynamic keyword, I can't also use those parameters. So you kind of give up that. I think there's a third limitation because of the way dynamic keywords are implemented. And that is that your variable scoping starts to have strange behaviors.

Because the way this works is that it just takes a copy of the AST and sort of bores it into, turns it into a function call later on. I did some work. I'm on the PowerShell team. Okay. Thank you. The same word.

Prototyping the first class DSL support in PowerShell a year ago. And it took months for us to realize that there was another limitation here was in variable scoping because of the whole dynamic scope. Okay. Thank you. Now.

I'm putting this out for awareness. I don't actually use this anywhere because of limitations. As you're diving deep into DSLs, this is a thing to be aware of. And then my last detail is, I should have learned this one this week, is if you add the attributes,

debug or step through, when somebody's debugging their scripts that uses your DSL, they'll actually jump from their code to their code in the script block. And they won't actually have to debug automatically through your DSL commands to get there. This affects automatic breakpoints. Like you got a breakpoint on anytime somebody says,

write outputs globally, your functions will actually be excluded from those global breakpoint type scenarios. All right. So I suppose any questions or things we wanna touch on anymore? Yes.

What are the pain points and caveats that you've seen defining DSLs this way? So the caveats are, sometimes, well, sometimes I write really, it's like knowing best practices and doing it for so long, I feel like I'm writing a really ugly PowerShell

that's breaking a lot of best practices. That's more of a procedural process thing. I made, in my DSL, I made heavy use of positional parameters where I describe something that's a string, like say my, I describe something that has a string for like a node. And I also have like a hash table that is optional.

Like I sort of had like sliding parameters. Actually, my subgraph would be the better example. Like I have a string, a positional hash table, and a positional script block. But I made the hash table optional and the string optional. So I have like six parameter sets

to allow for this weird nuance of the user can choose a script block or the hash table and the, and that made sense in the context of that DSL. Yeah, when you're relying on those positional parameters, you're kind of enforcing a certain structure to the DSL, especially if you have two commands or two parameters that are script blocks, and you can use your loss to pick that one.

You're expecting one script block to be hanging and the other one to be passed as a parameter. People want to deviate from that, they're gonna run into issues. When you're relying on the structure being based off of this one. But it makes it cleaner to look at it. You're enforcing a certain style.

I surrounded some nuances where to be very particular about what my default parameter set was when I was doing complicated stuff like that because having sliding parameters and strings being a positional one, lots of stuff can convert to a string and where I wasn't,

there's just nuances in there. I remember I had to be very careful about my default parameters and my positional ones. Yes? Is there a script analyzer rules to assist with that layout or is it just goes and then it's mine and you start doing? I suppose script analyzer doesn't complain too much

with the exception of when, like Pester allows you to do like a before all or before each script block and then you might define a variable and then in your it statements you use that variable. Script analyzer isn't aware of processing cross script block. Like we need to understand how stuff executes

and analyzer just has no context for that. Otherwise, okay so I suppose on that note, if it does throw issues with some things, you can suppress script analyzer rules. Like there's a attribute as well to suppress specific stuff for your items.

Like you mentioned, there's a style to the DSL that's completely different than the style for command. As you say that, I think I do have those littered in my module where I'd probably like the naming, right? The noun verb. I think I had to use an attribute to say, I'm not following that. Okay, so you have to statically exclude the script rules.

Yeah, I use that too. But it might use your code to put all these attributes to shut up script analyzer in there. It'd be nice to be able to put all those into a script analyzer file and isolate all that mess with it. Yeah, if I had to do it again, I would have used those rules for script analyzer

to say this project, this one just doesn't apply. So it seems like if you were working with hierarchical data, creating something with hierarchy, you could just easily pass in a hierarchical nested custom object or something based on JSON. But I guess the advantage of going with this DSL approach

is that you're able to provide more detailed, more just I guess simplified validation of the user input. And user input is kind of the key there. Like if you're dynamically generating that hierarchical object via code,

maybe you don't necessarily need any DSL for that at all. Just programmatically build that thing. But if you're in front of it kind of writing these things out, that's who the DSL support is for the end user. So I actually worked with Kevin. My question earlier was a loaded question about whether or not you could run dynamic code

inside those script blocks. So the main benefit of this approach rather than creating, for example, nested hash tables for your hierarchical structure is you get a much cleaner dynamic, data-driven structure generation with this method.

I think you might be able to do a lot of the same stuff in a hash table initializer. But personally, I think it looks a lot less clean. And I'm not certain. I have a problem with this approach. I can't pick great examples off the top of my head.

But this really stands out and shines when you're going out to AE and you're OUs and getting machines in the OUs right and generating dynamically where everything is in line rather than having to have a bunch of variables and core loops outside and then nesting a bunch of stuff

that's now abstracted away into variables. I'm gonna take off there. I think we need to wrap it up here. Please do the speaker evaluations for all the speakers. This is actually my first time presenting here. And I'd appreciate the feedback, good or bad. I'll take it either way. And enjoy the rest of the conference. Thank you. Thank you.