Beyond Pester 101: Applying testing principles to PowerShell
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 60 | |
Author | ||
License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/37366 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Producer | ||
Production Year | 2018 |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
1
10
14
18
24
25
26
32
34
40
41
44
46
54
00:00
Statistical hypothesis testingSoftwareLevel (video gaming)Statistical hypothesis testingSpeech synthesisSlide ruleCombinational logicSoftware developerComputer animation
00:56
Statistical hypothesis testingStatistical hypothesis testingStatistical hypothesis testingSoftware bugProcess (computing)Multiplication signScripting languageInternetworkingSoftware testingComputer animation
02:33
Statistical hypothesis testingSound effectConfidence intervalBitGraph (mathematics)Statistical hypothesis testingSoftwareBridging (networking)Task (computing)TrailProcess (computing)Capability Maturity ModelSound effectComputer scienceComputer animationDiagram
03:55
Statistical hypothesis testingScripting languageCodeBoss Corporation
04:42
Statistical hypothesis testingCloningCodeForm (programming)Statistical hypothesis testingStatistical hypothesis testingType theoryPoint (geometry)Unit testingPresentation of a groupScripting languageSoftware bugSuite (music)Error messageConfidence intervalSoftware developerCodeMereologyCycle (graph theory)BitDesign by contractWebsiteCartesian coordinate systemINTEGRALForm (programming)Computer animation
07:04
Statistical hypothesis testingStatistical hypothesis testingType theoryExecution unitSingle-precision floating-point formatStack (abstract data type)DisintegrationExecution unitStatistical hypothesis testingStatistical hypothesis testingSoftware developerWorkstation <Musikinstrument>FeedbackUnit testingMedical imagingBitGroup actionVirtual machineINTEGRALComputer animation
08:42
State of matteroutputCountingStatistical hypothesis testingFunction (mathematics)Tablet computerExecution unitType theoryStatistical hypothesis testingParameter (computer programming)Expected valueGreen's functionPoint (geometry)Function (mathematics)ChatterbotoutputScripting languageVisualization (computer graphics)Functional (mathematics)Single-precision floating-point formatGoodness of fitCodeExecution unit
09:49
12 (number)Statistical hypothesis testingStatistical hypothesis testingType theoryExecution unitStatistical hypothesis testingWordUnit testingStatistical hypothesis testingBitFunction (mathematics)Computer animation
10:28
Execution unitComplex (psychology)DisintegrationStack (abstract data type)Statistical hypothesis testingStatistical hypothesis testingType theoryStatistical hypothesis testingInteractive televisionStatistical hypothesis testingBitClosed setSet (mathematics)INTEGRALUnit testingExecution unitWindows RegistryCASE <Informatik>Computer animation
11:29
Metropolitan area networkFatou-MengeAliasingInterface (computing)Statistical hypothesis testingPower (physics)Simultaneous localization and mappingEmailExecution unitDisintegrationAverageNumberStatistical hypothesis testingType theoryModule (mathematics)Configuration spaceMultiplication signModule (mathematics)BitINTEGRALUnit testingDifferent (Kate Ryan album)Statistical hypothesis testingNumberExecution unitSource codeComputer animation
12:23
Statistical hypothesis testingExecution unitStatistical hypothesis testingType theoryElectronic visual displayMathematicsStatistical hypothesis testingFunctional (mathematics)EmailSystem callService (economics)
13:02
Statistical hypothesis testingStatistical hypothesis testingExecution unitType theoryImplementationMathematicsStatistical hypothesis testingImplementationCuboidStatistical hypothesis testingPartial derivativeBlack boxSoftwareStructured analysisSoftware testingFormal grammarType theoryParameter (computer programming)Functional (mathematics)Computer animation
14:35
Service (economics)Statistical hypothesis testingType theoryStatistical hypothesis testingExecution unitNumberElectronic visual displayMathematicsStatistical hypothesis testingService (economics)CodeFunctional (mathematics)
15:26
Function (mathematics)Parameter (computer programming)Statistical hypothesis testingStatistical hypothesis testingExecution unitType theoryFunctional (mathematics)outputBlack boxBitException handlingSystem callCuboidComputer animation
16:00
outputFunction (mathematics)Parameter (computer programming)Form (programming)Statistical hypothesis testingExecution unitType theoryStatistical hypothesis testingPhysical systemCodeGroup actionSpecial unitary groupStatistical hypothesis testingBlack boxUnit testingSphereStatistical hypothesis testingBitFunction (mathematics)Latent heatNumberFunctional (mathematics)Physical systemCodeTerm (mathematics)Projective planeHuman migrationGame controllerImplementationParameter (computer programming)C sharpoutputMultiplication signCode refactoringConfidence intervalMonster groupGoodness of fitExecution unitCuboidMereologyComputer animation
20:37
Philips CD-iMonster groupCodeType theoryStatistical hypothesis testingComputer wormAreaStatistical hypothesis testingFunction (mathematics)Execution unitMaß <Mathematik>Disintegration2 (number)Statistical hypothesis testingDisk read-and-write headGreatest elementStatistical hypothesis testingCodeCuboidAuthorizationWritingType theoryBlack boxMonster groupINTEGRALComputer animation
21:17
Statistical hypothesis testingSuite (music)Multiplication signScripting languageStatistical hypothesis testingGoodness of fitCodeSuite (music)
21:45
Statistical hypothesis testingParallel portNetwork topologyStatistical hypothesis testingServer (computing)Buffer overflowMultiplication signNeuroinformatikStatistical hypothesis testingStack (abstract data type)Mobile appFreewareImplementationComputer animation
22:40
Statistical hypothesis testingParallel portComputer fileStatistical hypothesis testingProcess (computing)Statistical hypothesis testingParallel portExecution unit2 (number)Multiplication signModule (mathematics)ResultantLine (geometry)Function (mathematics)SoftwareFlow separationSpacetimeRun time (program lifecycle phase)Unit testingDiagram
24:01
Function (mathematics)Statistical hypothesis testingStatistical hypothesis testingCodeCondition numberRandom numberBefehlsprozessorMortality rateLaptopStatistical hypothesis testingRun time (program lifecycle phase)Multiplication signDescriptive statisticsComputer hardwareProcess (computing)Suite (music)StatisticsMaxima and minimaFunction (mathematics)Statistical hypothesis testingCodeWindows RegistryInformationComputer fileIdentifiabilityCondition numberResultantINTEGRALUniqueness quantificationUnit testingElectronic program guideNumberGraph (mathematics)Multi-core processorComputer animation
26:48
Pay televisionStatistical hypothesis testingStatistical hypothesis testingSoftware frameworkFunction (mathematics)DatabaseQuery languageBuildingRepresentational state transferStatistical hypothesis testingRevision controlComputer fileProcess (computing)Data modelResultant
27:41
BuildingModule (mathematics)Computer networkStatistical hypothesis testingUser profileNumberStatistical hypothesis testingDisintegrationComputer fileHydraulic jumpStatistical hypothesis testingStatistical hypothesis testingSoftwareModule (mathematics)Run time (program lifecycle phase)Confidence interval2 (number)Noise (electronics)Multiplication signUnit testingSuite (music)BuildingINTEGRALMultitier architectureFeedbackComputer fileRight angleSoftware developerMathematicsObject (grammar)Computer animation
30:24
Statistical hypothesis testingPower (physics)Statistical hypothesis testingDisintegrationParameter (computer programming)Computer fileContext awarenessError messageComputer networkModule (mathematics)Statistical hypothesis testingBlock (periodic table)NumberLine (geometry)Computer animationLecture/Conference
30:56
Statistical hypothesis testingStatistical hypothesis testingOrder (biology)Hill differential equationStatistical hypothesis testingExecution unitOrder (biology)Cellular automatonMobile appIntegrated development environmentMatrix (mathematics)Point (geometry)Insertion loss1 (number)Level (video gaming)Variable (mathematics)Statistical hypothesis testingComputer fileCycle (graph theory)BitParameter (computer programming)Scripting language
32:49
Statistical hypothesis testingSoftware maintenanceComputer networkModule (mathematics)Computer fileExecution unitStatistical hypothesis testingReduction of orderDisintegrationSoftware maintenanceGroup actionSuite (music)Disk read-and-write headStatistical hypothesis testingContext awarenessType theoryDoubling the cubeBlack boxStatistical hypothesis testingComputer fileNegative numberService (economics)NeuroinformatikModule (mathematics)Line (geometry)Maxima and minimaVideo gameFeedbackSoftwareComputer animation
34:52
Statistical hypothesis testingStatistical hypothesis testingProjective planeLevel (video gaming)Operator (mathematics)Validity (statistics)WritingInformationStatistical hypothesis testingSoftware testing
35:59
Statistical hypothesis testingEvent horizonComa BerenicesComputer animationXML
Transcript: English(auto-generated)
00:10
Hello, everyone. My name is Glenn Sarti. I'm a senior software developer at Puppet. And today we're going to be talking about Beyond Pester 101, applying some testing principles to PowerShell.
00:23
So if you have any questions today, please raise your hands. I'm happy to take questions all the way through the presentation. You do not have to keep them till the end. I will be throwing you a lot of resources today. So I've got some slides at the end of the talk that will go through all of the reference material that I've used today.
00:40
And if you want to have a chat with me after this, I'll obviously be off the stage so the next speaking can come up. Or you can catch me at the drinks tonight. Be aware that talking about testing and drinking is not a good combination. So I want to start with a story that should hopefully feel very familiar to you all.
01:01
And this is my Pester love story. So when I first started with PowerShell, I had no idea about testing except, does this work on my machine? And to be honest, that's all we really had back then. Things broke and I wasn't happy and debugging was painful. But after writing PowerShell for a while, I'd heard about this tool called Pester.
01:24
You know, I tried a few things with it, but I didn't really see the benefit. And then one day, I needed to write a PowerShell script to install some things on a TeamCity agent. And I wanted to make sure they were installed correctly. So I thought, that sounds like testing. I can write a Pester test for that.
01:41
So I wrote the test, but they kept failing. It's like, crap. That's because there was a bug in my script. It had nothing to do with my Pester test at all. And this is when I began to fall in love with testing. The more tests I wrote, the more bugs I caught, and the happier that I was. I finally understood this whole thing called testing.
02:01
But something began to happen. And I was making changes, but I had to keep changing the test all the time because they kept failing. I mean, this testing thing was supposed to be awesome, but it seemed to be making my job worse and worse by doing all this work. With a test that I wrote even good, what even makes a good test, and how do I even measure how effective my tests were?
02:22
So off to the internet I went, and I went searching, and I found this entire thing called software testing. And there was so much to learn, and much of it was even contradictory, and soon that was me. Smashing my face into a keyboard, going, I don't know how to test.
02:40
But gradually over time, I began to learn the basics of testing, and I applied them, and I actually even spoke with testers, I mean, they're not, they're actually humans that don't live under a bridge and everything. And I'm back on track to once more be taught to be loved with testing. Now I'm sure you've all had similar experiences when you're learning something new, and you may even recognize this graph as the Dunning-Kruger effect.
03:01
So as a bonus, this is a bit of psychology as well as a PowerShell testing talk. So even though my current job title is Senior Software Engineer, I spent 20 years prior to that doing desktop automation. I never went, I never got a computer science degree, I didn't learn this stuff. And my lessons about testing came hard, and to be honest, probably took a lot longer
03:23
than what they should have. And it wasn't until I started working at Puppet and doing Ruby that I really began to understand the importance of testing, and had quite a few people in my team that were mature software developers, and they taught me about how testing should really be working.
03:41
And because I just didn't know, I crested Mount Stupid, and I fell face first into the valley of despair, and today my task is to help you when you get into that valley of despair to get you back up onto the slope of enlightenment. So let's begin the ascent.
04:01
That's more a descent? Oh yeah, if I talk too fast, I know my boss says I talk funny, please feel free to put your hand up and just ask me to explain things. So who here tests? Okay, everyone's got your hand down, you're wrong. Everything is tested, every PowerShell script you write, everything you do is tested, it's
04:23
just a matter of by who, or in this case, what. So for those that did actually raise your hand, why do you test? This is an important question to answer because it's going to guide you when you need to make some compromises, and ultimately affects how you actually write your PowerShell script code in the first place.
04:42
So this is my reason why I test, to reduce the risk that our user will experience an unexpected behaviour, and there are three very important points in that sentence. The first one is reducing the risk, because you will never be able to achieve zero bugs, but you can reduce the risk that they're actually going to occur.
05:01
The second one is that ultimately your testing, your PowerShell scripts and things like that, are going to be used by a user, so your testing is to help the user. Testing is a user-centric concern. And the last one, which is bugs, issues, errors and faults, they're all behaviour that a user doesn't expect, and this raises an interesting question.
05:23
If we, the developers, think something is a bug, and the user experiences it, but they don't think it's a bug, that's normal. Is that a bug? If a bug doesn't cause the user to notice, is that a bug? So you really need to think about these things, about why you're testing.
05:41
So there's a couple of benefits of testing. By running a test suite prior to releasing code, we can gain confidence in our code, so we can get it out faster. So if you saw Gail's presentation a bit earlier today, we're talking about shifting left and developing faster, test is how you get there. One of the disciplines in producing high quality code is having good tests.
06:01
High quality code is easier to debug, it's easier to maintain, and it's easier to add features to. Ultimately, it's also cheaper, both in time and money, so it's easier to find bugs early in the development cycle than it is to actually release them to someone else. Sorry, for a user to experience them, I should say.
06:21
It also keeps us honest. It actually ensures what the thing does, actually does what we said it would do in the first place, which means this actually forms part of your documentation for your PowerShell, because it's a promise or a contract to your users that my application is going to work like this. So what kind of tests are there?
06:41
The PESTA book talks about unit testing and integration testing, but we also have smoke testing, UAT, OAT, globalization, interoperability, so on and so on. One website listed 105 types of tests, and I'm not going to go through every single one of them today, because I've only got 40 minutes.
07:00
So I'm going to go through four of them. All right, so let's go. Unit tests, and I'm pretty sure most of you would have already done these kind of tests when you're running PESTA. So one of the better ways to explain testing is to use the testing pyramid. Has anybody seen this before? One, two, three, excellent.
07:21
So this is an image that I've kind of borrowed from Martin Fowler, which is the testing pyramid. So we have the hare and the tortoise on the left-hand side, which is how fast things are, and we've got cost, which is how expensive the tests are to run on the right-hand side. So you have unit tests down the bottom in pink. They're very fast to run, and they give you a lot of fast feedback.
07:42
They're cheap, because they don't require any VMs to run, and they can just be run on any development workstation. They're also only testing a single thing, thus the word unit. So in PowerShell, generally the smallest thing that we're going to be testing is a function, so when we're doing unit tests, we're basically testing functions.
08:03
As we get higher up the pyramid, we start seeing integration tests, which unsurprisingly is how units integrate or interact with each other. And then finally, up the very top, we have full stack tests, which may be a bit hard to read, where we're setting up virtual machines and doing manual testing, so it's very expensive and slow to do.
08:21
And so notice how the size of each group of tests gets smaller and smaller as you go further up. So why would we do that? Well, think of quality, think of you having a quality budget. Unit tests are very inexpensive, so you can have lots of them. Integration tests are slightly more expensive, so you have less of them, and so on and so forth.
08:40
So let's have a quick look at some real-life unit tests. So, here's a snippet of code from Poshbot, which is a chatbot written in PowerShell, and Brandon Oldham is talking about this tomorrow. And we have a single function there called getPoshbot. We have a single input parameter called ID, which is in the green. We have a single output parameter, which is a PSCustom object, and that's in the red.
09:02
And we have two external dependencies, which are in yellow, which is a script scoped variable, and a function. So when we do our unit testing, we have to make sure we mock those. And so on the other side, we have the corresponding unit test. And we can see at the top, we're mocking our external dependencies.
09:22
About two-thirds of the way down, we're actually using our inputs to actually invoke the function. And in the red, we're asserting that the output we got from the function is what we expect. Does that make sense? Yes. Is the answer. Yes. Cool. Good. So if we want a more visual point of view, we have inputs in green at the top.
09:41
We use mocks on the right-hand side, and then we've got outputs coming at the end where we do assert that the expectation is what we expect. Now has anybody ever heard of assert act, sorry, arrange act assert? Yep. This is one of the ways we can explain how we can actually have our testing in a more, what's the word I'm looking for, ordered way.
10:02
So we arrange our test fixtures at the top. We act by calling the function, and then we assert that the output is what we expect. Does that make sense? So that's briefly unit tests, and if you want to have a bit more of that, Brian Burke is talking about that tomorrow in his session on how to love unit testing.
10:22
And there's Brian just over there. So second, integration tests. Integration tests in the middle of testing pyramids, and like I said before, they're about testing different, so how units interact with each other. They're normally slower to run and more expensive to do. So one typical case of an integration test is to actually do something on a computer,
10:44
like write a file, do registry keys and things like that. So these things have to be cleaned up and they have to be arranged, which means they take longer and they're a bit more complex to set up. But if there are these downsized integration tests, why would we even run them in the first place? And this is a perfect example why you should have integration tests as well as a unit test.
11:05
Both sets of doors in that example are passing the unit test. They behave exactly as they should. It's not until you have them interacting to each other that you get unexpected behavior, where they just constantly keep opening and closing.
11:21
And this is why they're useful. They find all these subtle interactions that you wouldn't normally get just by doing pure unit testing alone. So a brief look at an integration test. This is the networking DSC module. You'll notice down here that we're using the start DSC configuration cmdlet. So we're not using test get or set, we're using this cmdlet, which actually integrates
11:43
all three together. It's a bit more complicated and takes more time to run these tests. How much longer? So I ran all the numbers on the networking DSC module. On average, every integration test was taking four times longer than a unit test. And you can see that we've almost got a 10 to 1 ratio of unit test to integration.
12:06
So let's talk about unit tests again. This is probably the hardest lesson that I had to learn and the one that caused me the most unhappiness. Not all unit tests are the same. There are different flavors of unit tests.
12:23
So consider this example. We've got a very simple function at the top, which just gets the display name of a service, and we've got a very simple test underneath it, which just mocks the call to getWMIObject. Very simple test and function. But what would happen if you weren't given the whole function?
12:43
How would you test this if all you were given was the header at the top? You can still write a test for it, so just like that, you can just call the winrm service and make sure the display name is windows-remote-management, but knowing how a function works changes how you test that function.
13:03
Now you may have heard this referred to as black box versus white box testing. So the more formal definition is, this is quite a long one, black box testing is software testing method in which the internal structure design implementation of the item being tested is not known to the tester. White box testing is a software testing method in which the internal structure design
13:22
implementation of the item being tested is known to the tester. You may also refer to this as behavioural versus implementation testing. So in a black box scenario, you're testing how it behaves, which is what it does. In a white box scenario, you're testing how that behaviour is implemented, which
13:43
is how it does it. And in most larger software shops, you'll find that the implementation tests are written by developers, and the behavioural tests are written generally by your QA or your BA people. But let's be honest, how many people have a QA department for their partial scripts?
14:01
Yeah, no. Oh, one person, awesome. Send them there. So most of us don't have that luxury, so we need to be able to understand both and the pros and cons of using both types of techniques. So black box tests can be a lot harder to write, as we usually already know how a function works, and forgetting something is actually quite difficult when you're
14:21
doing testing, and it will force you to use parameters more often instead of using external dependencies, so you don't have to do mocks and things like that. White box tests, on the other hand, are a lot more fragile. And what do I mean by fragile? So remember my example earlier, where I had my get the service display name? So I probably shouldn't be using getWMIObject, I'd be using, say, getService.
14:46
So I changed my code, and immediately my test fails. Now, even though my code probably works, and it probably behaves the same, my test is showing that it's wrong. And this is what caused me the original unhappiness. Whenever I made a change to my code, my tests were wrong, even though I knew that the change
15:03
to my code ultimately didn't change the behavior of that function in the first place, and it doubled the number of things that went wrong when I was writing my test in my PowerShell. And even if I rewrote that test to use the correct mock, that still didn't prove the function even worked correctly, it just proved that my mock worked.
15:21
So in my gut, I knew it was true, but I still couldn't prove it with my automated tests. So how can we move from white box to black box testing? Well, your functions may need to change. So firstly, all of the input into your functions need to come through the input parameters, and so this removes the need for external calls and mocks.
15:41
But this is not always possible, even practical, to do. But what you can do, is you can wrap those external dependencies in private functions, because it's a bit easier and a bit safer to mock those. So this is a bit of an exception to the rule, where it's kind of safe to mock private functions.
16:00
And this is not even a new concept. So who's read the learn PowerShell scripting in a month of lunches? Yep, who's heard about tools and controllers? Yep, tools. Look at that. Tools accept all input only from their parameters. Tool style functions are ripe for black box testing. So this is nothing new, you've probably already done this before.
16:21
So, which should I use, which is better? And the answer is both. Both of them are useful, and neither of them are intrinsically good or bad. They just both have trade-offs. So remember way back in the beginning I talked about why I do my testing in the first place? Well, in my testing I say that it's user-centric.
16:42
So users are more worried about what it does as opposed to how it does it. So when I do my testing, I really prefer to use black box testing. But I still do use some white box testing because sometimes you do actually have to test how something actually gets done. And this leads to a million dollar question, which I'm sure some of you may be asking.
17:03
Should I test my private functions? Has anybody ever asked themselves this? Yes. So private functions are implementation details that users don't directly interact with. So they should already be tested as part of your public functions. So according to my definition of why I test, the answer is no.
17:24
But if it supports why you're testing, then yes, go and test your private functions. If you have some other behavioral tests, yes, test your private functions. If it reduces the number of your tests that you have in your test suite, but you still have the same amount of confidence in your product, then yes, test your private functions.
17:43
So that's briefly unit black box testing. So that leaves more unit tests. So we're going to talk about something called characterization tests. And yes, I'm using an S because I can spell properly back in the southern hemisphere.
18:00
So a characterization test documents the actual behavior of a system as opposed to what we specify the behavior should be. And these will probably sound a little bit familiar to black box testing. And in truth, they kind of are. But they have a different purpose. So for example, in the unit test that we looked at previously,
18:22
those are all specification tests, which is given these specific inputs, when I act on it, it's going to give me these particular outputs. That's a specification. Characterization tests don't care about that. They just say, this is what the system is looking at right now. So a really quick example of that is this function up here called step by one.
18:40
The unit test is, if you give it number one, it should step it by one, so it should have a value of two. But the actual characterization test will be, if I give it a value of one, it's going to spit out three because the function currently is actually doing something wrong. So this can seem a bit confusing, so why on earth would we use them?
19:00
Has anybody read Working Effectively with Legacy Code or even heard of this book? No? Okay, so the term characterization test was first coined in the book by Michael Feathers. This is a pretty old book back in 2004. And there are all the examples in there about Java, C Sharp and C++, but you can equally apply them to PowerShell. So he defines Legacy Code as code without tests.
19:22
How many people have Legacy Code? Excellent, that's the right answer. So I discovered this book while I was working on some old Ruby projects and the lessons in here are absolutely fantastic about how to migrate Legacy Code and actually get it under control with good unit testing.
19:41
So here we have a single function that is really horrible. It is a monster method with really bad code. So I'm going to go through how I can apply characterization tests to help me. So the first thing I do before I fix it is write the characterization test. Document how this thing currently behaves.
20:00
Then as we refactor it, we write good, clean, nice, unit tested code. Every time we make those additions, we can run our characterization test and make sure we haven't changed how the previous behavior used to run. So it gives us confidence that the new code that we're writing isn't changing any existing behavior. And at the end, we can take those characterization tests,
20:22
get rid of most of them because they're already covered, and turn some of them into unit tests. So characterization tests don't survive the entire refactoring process. They're a tool that you can use to help you as you create that nice, clean code. So here we have some of the other chapter headings from that book.
20:41
I particularly like the second bottom one, which is I need to change a monster method and I can't write tests for it. So the author goes through and explains how you can inspect your code and actually figure out ways to actually write tests for code that you haven't previously got tests for. A couple of these will reach out and slap you in the face.
21:00
When I read this book, it was like angels singing when I was writing my code. So that's really briefly four types of testing. So we've got white box testing, integration testing, black box testing, and characterization tests. So that's the first half of the slope.
21:20
Now it's time for the second half. It's not enough just to have good tests. They need to be tended for and cared for and loved for as much as your PowerShell script code. So particularly as you add more and more code, you're going to be adding more and more tests and it's good practice to periodically go through your test suite and make sure it's giving you the value that you want from it.
21:43
So how can we do that? First, you can reduce how long it takes to actually even run your tests. So has anybody seen the SQL Server DSC resource? How many tests that's got? Yep, I think it's about 3,000 tests and takes 50 minutes to run an app player. That's a really long time to be sitting there waiting
22:02
for your test to run. So we should look at ways to actually speed that up. Now you can get a bigger computer. Yay, we all like getting kit. But there might be a better way to actually speed up our tests. So I wrote invoke parallel tester. Parallel tester, sorry. So previously in Ruby, I have a little tool called parallel tests
22:22
so I just took that and converted that directly into PowerShell. Now there's been some discussions about how we can do this in Stack Overflow. There's lots of discussions. This is just one implementation of how you can actually get parallel testing if you want to try your own, feel free. But this is just the way I did it.
22:41
So previously in Pester, we had five test files. We run file one, file two, file three, file four, file five. What I did was using the invoke parallel module from Warren Frame and I run all five tests at one in separate run spaces, get all the output files from that, manage them all together and then you end up with the output of a Pester testing run
23:02
in a short amount of time. And there were some interesting results when I started running this. So this is the DSC networking module unit tests. There are about 985 unit tests. It used to take about 80 seconds to run. So that's the big red line at the top.
23:20
When I started running my parallel tests when I have one job at a time it ran in 90 seconds so it's even slower because it takes some time to set up run spaces. If I run two jobs at a time we're getting down to about 50 seconds and we bottomed out at about 42 seconds so I basically halved the run time for running unit tests for the DSC module.
23:41
What's even more interesting is the more jobs I ran at a time the performance got worse and worse. So there is a sweet spot to your testing and more is not always better when you're running parallel tests. So that's awesome, I halved the run times. There are downsides though to doing this.
24:02
The log output from PESTA was horrible. It was trying to run five tests at once. I had description blocks, I had times all over the place. This thing, you couldn't really read it. Parallel testing is really good in CI suites where we only care about the structured output files.
24:22
I didn't munge any of the code coverage. So using InvokeParallelPESTA you don't get any code coverage statistics. If you're not running that, it doesn't matter. If you are, then you can't use that technique. It also raised some subtle race conditions. So what was happening sometimes
24:40
is that if you had two tests let's say for example you had a registry test almost deleting a registry item, almost creating it. Depending on which test ran when you would get some interesting results. Sometimes it would fail, sometimes it wouldn't. So what you need to do when you're setting up your test fixtures is make sure that they're all unique. So if you're using files, you can use the test drive in PESTA.
25:01
If you're doing things like registry or things like that mark them with some unique identifiers like GUIDs or just getting random numbers. These will frustrate you for hours. Race conditions. I used to have brown hair before I did this. The other thing is you can inadvertently depend on previous test date.
25:22
So if you're running your files in 12345 you may accidentally have a test in file 4 that's depending on something that's been set up in test file 3. So please remember to always explicitly set up all your test fixtures before you do your test. Remember, arrange, act, assert. Don't arrange and then hopefully act.
25:41
It's also not suitable for everything. I couldn't run the DSC integration test in there because the LCM can only run one job at a time. So running parallel didn't make any sense. But I could run the unit test in there. And it's also dependent on the hardware. So that pretty graph where I halved the runtime that was on my old quad core laptop.
26:00
I ran this on my newer dual core laptop and it sucked. After running four jobs at a time it hated it and the performance was just getting worse and worse and worse until it was almost ridiculous. So depending on your hardware would depend how many parallel jobs you can run at a time. So remember, sometimes too much is not good.
26:22
So now we've got our tests running as fast as we can. The next best thing to run tests quickly is to not run them at all. Get rid of them. So every test you create needs to be maintained. It takes time and effort. And ideally you want to have the minimum number of tests to satisfy the risk of users getting that unexpected behaviour.
26:40
So how do we do that? Use your PES to output data. There is a wealth of information in there. So this is the output from an app via build on the DSC networking module. And you'll notice up there we've got 1075 tests. So what you can do, because you know I wish we had a tool that could query REST APIs
27:01
and put stuff into a database. So what you do is you get your PowerShell. You go, hey up there, give me every single build. And then can you give me every single build version for those builds? Can you please give me every single job for those build versions for those builds? And that will give you every single test that has ever been run for that module.
27:20
And what you'll end up is hundreds of these JSON files with all the PESTA results. You can then quickly grab that, shove that into a database with a very simple data model. So every build can have zero or more successful tests, zero or more skipped tests, or zero or more failed tests. And then we can start asking questions of the data.
27:42
So this is the build history for the DSC networking module. So we started with just over 200 tests. Now we've got just nearly 1500 tests. And I don't know what happened there, but they must have discovered the testing was really important with that huge jump that they did there.
28:00
But then we can start doing things like, when something fails, what does that look like? Well, we have 40 builds with one test, we had 25 builds with two tests, and so on and so forth, and you get the catastrophic failures of about 62 failures when we had three builds doing that. I mean, this is nice data, but it's not really going to be helpful to finding which tests you should get rid of.
28:22
So what about long-running integration tests? So this is the top 100 slowest tests in the last five builds. Can anyone see a test we should probably have a look at that's taking a while? And here's a big one right behind me up there, taking about 40 seconds. So we had one test taking 40 seconds and then a whole pile of them between 15 and 25 seconds.
28:42
These are the tests you should be looking at to see if they actually provide any value because they're taking so long to run. What about tests that have never failed? That 40 second test has been run 208 times and has never failed. That's a lot of run time.
29:01
Is that test actually needed? Is there sufficient unit test coverage that we don't need the integration test? Or can that integration test even actually fail at all? These are the questions you can start looking at your test suite to get the most out of it. You get some more advanced questions like do you have tests that always fail together?
29:21
Get rid of one of them. Done. So same test coverage, one less test. Are tests that fail too often? They're going to be giving you way too much failure noise. Are there better ways that you can do that? And then you can get really fancy. If you get all the files that are committed for each build, you can see that you now say when I touch this file,
29:41
there's a higher risk I'm going to have a failure as opposed to this file. So you can do things like add extra documentation or extra learning to help people not to make failures when they're editing those files. So now our tests are running as fast as they can and our test suite is as small as it can be. Can we reorder our tests to actually get faster feedback?
30:05
So one of the objectives is to give us, the developers of these scripts, fast feedback and confidence in our changes that we're making. And we call this fail fast. Has anyone heard that term? Yep, excellent. So one of the techniques that we use at Puppet is called test tiering. So what we do is we rank our tests
30:20
in a very simple high, medium, low value. So in our pester tests we can just put a simple tag at the end of our describe blocks that says whether it's high, medium or low. So how do you know which tests we should be marking what? Again, use your data. DSC networking module, most number of failed tests.
30:40
They seem like important things that we should mark as high. Also note that the two most failing tests have got nothing to do with code, it's all documentation and new line endings. They seem like something we should be testing for quickly. So what we can do in pester is then very quickly just put dash tag and say which level of testing
31:02
we actually want to do right now. So high, high, medium or just exclude all the low tests. Watch out for untagged tests because they will always run no matter what you put on dash tag and dash exclude tag. So if you're going to do this make sure you tag everything.
31:20
So once the tests are all ranked we can change the execution order from somewhat random. We can run the high test first then the medium and then the low. Because if the high value tests fail there's no point running the medium or low ones. Get fast feedback with no loss in test coverage. So we can quickly do this in app player for example.
31:42
So we can have a three cell matrix where we have an environment variable called test here, high, medium, low. We can then very easily change the test script so we just put in the environment variable. And then we can put in the matrix fail fast parameter so the first cell that fails it will no longer run any other cells and you'll get fail fast on your app player runs.
32:04
But we can also take this to the next level. Instead of running tests high, medium, low we can actually change how often we actually run each test here. So in Puppet we have about 28,000 unit tests that takes about half an hour to run.
32:23
So what we do in our first smoke run we only run high value tests and any test file that's been changed in the past two weeks. So what we're targeting there is anything that's high value or anything that's actually changing because that is where your greatest chance of failure is going to be. So that way again we can get fast feedback and get no loss in test coverage
32:41
because we still run the low and medium tests. We just run them a little bit later in the test cycle. And lastly some general maintenance. So this is George S. Patton style general maintenance. So say what you mean and mean what you say. This describes to military as also describes
33:02
to your test suite. The way your tests are described they shouldn't be cryptic they should be clear and concise. And if you say you're going to test something you should test something. And there are examples of this. This is the test from the DC networking module. Has anybody ever done any group policy work before?
33:21
They're aware of double and triple negatives that you get. They're really hard to understand. So should not contain files without a new line at the end. Much easier to explain it. You just go should contain files with a new line at the end. Even though these are small things when you're a newcomer coming into these test suites you can be scratching your head with the top one
33:40
trying to figure out what the hell is going on. And by newcomer I mean future you in six months when you're actually going to look at these tests again. Next one. This is an example from the PESTA book. Notice at the top we're saying WM service startup type should be set to automatic. Doesn't actually get tested.
34:02
We're only testing for the computer name there. So realistically this is what your test should look like because that is what you said you're going to test therefore you should test it. Make sense? Seems sensible? So wrapping up.
34:22
We went through four types of tests very quickly. So unit, white box, integration, black box testing and characterization testing. Then we reduced the duration of our tests we removed tests we didn't need we ordered the test to give us fast feedback and we did some general maintenance to make life easier on ourselves to make sure our tests give us the maximum value that we need.
34:41
Sorry, I can get out of it. You guys are not asking any questions today. Am I explaining this really well? I was expecting questions. Okay. Where to now? It's all well and good to have all this information. What are you going to do with it? Try things. Read, ask questions. Find a project and write tests for it.
35:03
Even a bad test is still better than no test whatsoever. Hopefully I've given you some idea of how big the world of software testing is and not everything that I've talked about is actually going to be applicable to you right now. And that's okay. But in the future you'll be able to understand and see some things and go, I remember that talk
35:21
I can try some of those things. And software testing isn't a gift or a talent it's a skill which means it can be taught and learnt and levelled up. But it means you need to practice it. And the more tests you write the more you can iterate your tests and the better you can get at it.
35:42
And the great thing about learning all of these testing principles this isn't just PowerShell. Ruby, Go, C Sharp operational level validation testing that Gayle was talking about. These testing principles still apply to these kinds of testing that you're doing. And that's it.
36:00
Thank you.