Testing VSCode extensions
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 637 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/52814 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Integral elementStatistical hypothesis testingStatistical hypothesis testingMachine codeVisual systemSoftwareSoftware developerService (economics)BuildingOpen setField extensionExecution unitDisintegrationPurchasingMachine codeComputer programmingIntegral elementStatistical hypothesis testingSoftware developerAudiovisualisierungPresentation of a groupVisualization (computer graphics)Field extensionBitMachine codeExecution unitCASE <Informatik>MereologyCross-platformSlide ruleProcedural programmingUnit testingOpen setService (economics)AngleCovering spaceTelecommunicationComputer animation
01:16
Machine codeVisual systemWeb 2.0Cross-platformVisualization (computer graphics)Software developerSinc functionCASE <Informatik>Integral elementField extensionMachine codeOpen sourceBuffer overflowStack (abstract data type)Web browserGraphical user interface19 (number)Archaeological field surveyMachine codeAudiovisualisierungShared memorySineSource code
02:12
PasswordExecution unitIntegrated development environmentField extensionStatistical hypothesis testingText editorBridging (networking)Machine codeIntegrated development environmentVisualization (computer graphics)Statistical hypothesis testingPasswordFormal verificationWordSoftware developerCASE <Informatik>Integral elementMachine codeOpen sourceImplementationData structureExecution unitSinc functionBitSoftwareInstance (computer science)Product (business)Service (economics)Module (mathematics)Field extensionBuildingPhysical systemDistribution (mathematics)MereologyNegative numberSoftware bugSoftware maintenanceMachine visionWindowState of matterUnit testingStatistical hypothesis testingUser interfaceCovering spaceBlock (periodic table)Power (physics)CuboidVirtual machineOpen setAudiovisualisierungWindows RegistryBookmark (World Wide Web)Process (computing)WebsiteMultiplication signQuicksortUsabilityStreaming mediaJava appletRaw image formatLibrary (computing)Server (computing)Lipschitz-StetigkeitSineScripting languageComputer animation
10:36
Field extensionStatistical hypothesis testingStatistical hypothesis testingExecution unitFunction (mathematics)Service (economics)Machine codeMachine codeService (economics)Integral elementMachine codeCASE <Informatik>Event horizonHoaxQuicksortStatistical hypothesis testingState of matterIntegrated development environmentVisualization (computer graphics)Functional (mathematics)MathematicsMultiplication signPoint (geometry)HookingStatistical hypothesis testingElement (mathematics)MereologyExecution unitUnit testingSet (mathematics)Field extensionSlide ruleBitModule (mathematics)ImplementationProduct (business)Instance (computer science)2 (number)Formal verificationSocial classComputer fileData storage deviceAudiovisualisierungoutputProfil (magazine)Open setProcess (computing)SpacetimeSystem callDifferent (Kate Ryan album)Endliche ModelltheorieComputer animation
19:00
Event horizonElement (mathematics)MereologyStatistical hypothesis testingIntegral elementAutomationDisintegrationText editorPersonal digital assistantRootStatistical hypothesis testingDefault (computer science)Visual systemComputer fontComputer iconThermische ZustandsgleichungCASE <Informatik>MereologyIntegrated development environmentData structureLibrary (computing)Web pageEvent horizonString (computer science)Element (mathematics)RootField extensionView (database)Network topologyVirtual machineLevel (video gaming)Multiplication signMathematicsRevision controlModule (mathematics)Computer iconStatistical hypothesis testingSinc functionVisualization (computer graphics)Integral elementMachine codeStatistical hypothesis testingSensitivity analysisRule of inferenceGame controllerSystem callSet (mathematics)Software developerSlide ruleWeb 2.0Internet service providerMachine codeVariable (mathematics)Function (mathematics)Device driverGraphical user interfaceDirectory serviceRoutingData mining1 (number)Power (physics)MultiplicationDataflowAudiovisualisierung2 (number)Structural loadChecklistProcess (computing)Forcing (mathematics)Instance (computer science)Pulse (signal processing)Physical lawComputer animation
27:24
Element (mathematics)Physical lawXMLComputer animation
Transcript: English(auto-generated)
00:06
Hello everyone, welcome to my presentation about testing Visual Studio code extensions. My name is Dan, I'm a software developer working at SUSE as part of the developer engagement program. And as you might have guessed by my participation in this track, I really like testing.
00:23
It's one of my big passions, and thus I'm giving a talk about this. In case you'd like to stalk me on social media, you can find my handles below on this slide, where I occasionally post stuff about technology. But let's take a first look at today's agenda.
00:41
I'd like to start out by talking a bit about the OpenBuild Service Connector. This is the extension that I've been working on for the past year, and it allowed me to gather some experience with respect to testing, and which approaches work, which don't. And so I'll cover, begin with that. And after that, in the main part, we are going to take a look at the actual testing procedures,
01:03
and at the different approaches that you can take, which would be unit testing, manual testing, and integration testing. And at last, we'll have a Q&A session. So with that, let's start out with the elephant in the room, and that's Visual Studio Code. So Visual Studio Code is a very, very popular cross-platform IDE,
01:24
or text editor, depending on how you want to see it. Let's build on web technology. So this thing is written completely in TypeScript, and runs in Electron, which is effectively a headless Chrome browser. It's developed by Microsoft, and fully open source.
01:43
And when I said popular, it's very popular. So it reached about 50% market share, according to the Stack Overflow developer survey, in 2019, and it grew even more in 2020. It has a very rich extension ecosystem, and also, in case you are interested in developing extensions yourself,
02:03
it has a very well-documented extension API, with quite good examples that will get you started. So, and since Visual Studio Code is fairly popular, it's also, we decided that we would like to provide an integration
02:23
between Visual Studio Code and our developer ecosystem. And we wanted to start out with the OpenBuild service. So the idea is that we will provide a bridge between Visual Studio Code and the build service itself.
02:42
The OpenBuild service, in case you've never heard of it, this is an open source build service that's tailored around, that's designed to build packages for Linux distributions, and whole Linux distributions themselves. So, for instance, the OpenSUSE distribution,
03:01
and the SUSE Linux enterprise, they are built in the OpenBuild service. And the OpenBuild service can build a whole lot more, so it can build virtual machines, it can build containers. And our vision here is that eventually, you will be able to build your packages from within VS Code, push those to the OpenBuild service.
03:22
This part is covered by the OpenBuild service connector, and is also already mostly in place. And then the OpenBuild service will build your virtual machines, it will build your containers, it will build your packages. These are automatically pushed to the OpenSUSE registry, in case you are building on build.opensuse.org.
03:42
And then you can push them from there to production to Kubernetes, and then monitor this via our toolchain. But this part is more for the future. Well, and now let's take a look at some of the roadblocks
04:05
and challenges that we encountered with testing of the OpenBuild service connector. So the first one was, well, as I said, this is an extension that connects to the OpenBuild service,
04:21
so it needs to talk to it. And that's a bit of a problem, since you don't necessarily want to actually connect to a production instance. As code has bugs, no code is perfect. And you don't want it to start issuing a whole lot of garbage requests,
04:41
or start testing of deletions. And then again, sometimes your network might have hiccups, and then your tests will just randomly fail, and so on. So we needed some way how to test the whole extension, talking to the build service,
05:01
without actually connecting to the real thing. Next thing, you need to authenticate with the build service to actually be able to do anything with it. And that means we have to store users' passwords. So the common way how this is done in Visual Studio Code
05:20
is using the Node Keytar module. And as far as I'm aware, Visual Studio Code already integrates this keytar module nowadays, but it didn't a year ago. And Node Keytar uses libsecret on Linux, which poses a bit of a challenge,
05:43
since libsecret tends to talk to DBaaS, and that also tends to pull in stuff like systemd, which makes it annoying to run this in containers. And then also, in case you want to run your tests on a developer machine,
06:01
then you don't want to modify the development environment of your developer. So in case you would start writing test passwords to the developer's keyring, that could cause all kinds of issues, and it makes stuff unreproducible. And then if they are, for instance, not running GNOME,
06:23
or they really don't use libsecret at all, then stuff will break. So that's not really optimal. Well, and then the last result, or maybe roadblock is the wrong word here,
06:40
but it turned out that my initial testing approach, which relied more or less only on unit tests, it was rather suboptimal, since the unit tests, they were checking mostly pairs of the UI, and that turned out to be relatively brittle, and they broke often, and they still required a lot of manual verification.
07:03
As I was checking, okay, does this, so how should the stuff look? Does it actually look like that? Then I took a look at the data structures that result out of that, and there was a lot of maintenance involved, and it didn't catch actual bugs
07:21
and had false positives, false negatives. So the testing approach wasn't really great. So maybe a short outlook how I managed to overcome them, at least partially. And so for talking to the OpenBuild service, that was actually relatively straightforward,
07:41
and I was able to leverage upstream's development environment. So in case you are building an integration to some kind of service, for instance, Kubernetes or something like that, take a look if upstream has some development environment. They tend to have this, especially for CIs, and you can spin these up. They tend to be not too heavy.
08:02
They have debugging enabled, and you can get these running, and they run on your local machine, and you don't need a network connection. Then for the libsecret part, this was actually, I managed to pull off a nice LD preload trick,
08:22
and I wrote a very simple libsecret implementation that would just, it just echoes out passwords that you would dump into the environment. And I just inject it into VS code via LD preload. So in case you want to, you need to use system libraries, and you don't want to test the actual system libraries,
08:42
you can use something like that. Well, and then concerning the unit tests, and I'm going to cover this in more details, unit tests are not good to test actual parts of the UI. I'll cover that extensively later on. So instead, I want to be testing actually the workflows
09:03
via manual or integration testing. So let's get more to testing actually, testing the extensions. And why is this hard? At least in my opinion, it's relatively tough. I've struggled with this a lot.
09:22
And I think that one of the big issues with testing extensions is we are testing UIs here. And UIs carry a whole lot of state. So if you have your Visual Studio Code window open, and you have your sidebar, your editor views,
09:42
and you have some bookmarks, maybe different themes, all of this is a state that influences how the UI will behave. And if you want to test this, for instance, in unit tests, you have to create this state yourself.
10:01
And that's a whole lot of work. And especially it means you'll have to create this manually. You won't let Visual Studio Code create that. And it causes all kinds of problems. So testing UIs themselves is rather hard, especially if you rely on unit tests.
10:23
And since unit tests are, at least in my experience, rather popular and especially extremely popular in the JavaScript and TypeScript community, unit tests are bad at testing workflows. And testing workflows is actually what you want to do, at least in my opinion.
10:42
So if you want to test your extension, you want to be testing the workflows. And unit tests are really not good for this. Unit tests are testing one chunk. You want to be testing a whole lot. So you want to be testing step A, step B, step C, and now did this work?
11:02
And so for that, unit tests are not a good approach. So instead, I would recommend to rely more on integration testing. And this is, as I said, rather uncommon. And at last is what I would call what you get is not what you see. And what I mean by this is essentially,
11:21
if you write unit tests, you can easily create some sort of UI state that would never happen in practice. And so you'll get something, but you will never see this. Or you'll see something different, but you'll expect that this will be displayed like this.
11:40
And so it means if you actually try to test parts of the UI, you'll still have to resort back to manual verification. Since you'll have to open the extension, you will have to take a look, okay, are my elements there? Is everything there where I want it to be? And if it isn't, then you'll have to adjust your tests again.
12:02
And that's actually not what you want to do. Your tests shouldn't rely on manual verification, unless they are manual tests, of course. So let's start out with unit tests. I think everyone has, at some point, heard about unit testing. And unit testing means you have to take one unit of your code,
12:24
let's say a function or a member function of a class, and you test one functionality of this, and preferably just one. So if you actually decide you want to touch,
12:41
you want to test the behavior of some UI elements, it means you'll have to do some extensive setup and teardown. But usually the setup is far more involved. Since, as I said, all this UI state that you want to, that needs to be present,
13:01
you will have to create this yourself, since you are not creating, letting Visual Studio Code create this, or matter of fact, any other UI. If you have external services, you will have to mock these, as your unit tests should be really just testing
13:25
the functionality of your code and not of your external service. And they should be fast, and if you want to run external stuff, that will make it rather slow. So mocking is, in this case, the preferred way.
13:41
So let's get started. And fortunately, getting started with unit testing in VS Code, that's pretty simple, since the official docs, they have an example how to get started with Mocha. And the official, or the semi-official Yeoman generator, if you use that, will also generate you already a working test setup,
14:03
which in my opinion is really great, since it encourages you to start out with unit tests. And this one will use, as I said, Mocha and the VS Code test module. And what VS Code test does, this will launch a second instance of Visual Studio Code
14:22
with a new environment. It will install the current state of your extension, and it will load all your tests, and it will run them after each other and verify that everything works. And that works rather great. Especially, the nice thing is it's pretty isolated,
14:43
so it won't mess with your existing installation. And you can also pass all kinds of custom settings. You can exclude any other installed extensions. You can install additional extensions and whatnot. Tiny caveat, if you want to collect test coverage, which at some point you probably want to do,
15:02
the setup is a bit more involved. You can't just call NYC and then VS Code test. But Connor Peet has figured out a solution for that, and it's linked on the slide. So in case you want to do that, just steal it from him
15:20
or take a look at the OpenBuild service connector. I've stolen this code and adopted a bit. Very well. So now I'd like to cover a few specific examples that I found interesting when writing unit tests for Visual Studio Code extensions. And I'd like to start off with extension settings.
15:43
So many, many, many extensions have settings, apply settings, and use settings, and some of them, hundreds. So extension settings are stored by Visual Studio Code in a JSON file, either profile-wide
16:04
or in the workspace of the user. And since I said, if you're using VS Code test, then your settings are accessible since you have a running Visual Studio Code instance. So you can just read and write them and test them,
16:23
and that all works relatively easily. So as far as I know, nothing really that you need to take care of, except one thing, and that is please clean up after yourself. So keep in mind your tests,
16:40
use the after each test hooks. If you're a test runner, if that's mocha, then that's a product. So use some after each test hook to clean up all the settings that you did, since if one of your tests fails and you are left with some sort of dirty state, you don't want this lingering around.
17:03
So just keep in mind, clean up after yourself. Then, events. So events are used fairly frequently in Visual Studio Code to signal that a certain thing happened. And as you can see, I have an example picked out here.
17:22
So this is the onDidChangeValue event that would call the function that's passed to it every time some value changes in this example, it's a quick pick. So this is, as I said, used fairly frequently
17:42
when, for instance, in UI elements, if the user provides some input. And then Visual Studio Code would call the so-called listener, which are just functions that you pass to the event. So you can use those,
18:02
and if you want to test those, keep in mind, please, Visual Studio Code will call this function, but if your function is asynchronous or if it returns a promise, it will not await this promise. It's just done like that,
18:21
whether you like it or not. The caveat with not awaiting promises is that if the code that you want to test relies on getting some events and those events invoking a listener that is passed to them,
18:41
you'll have to wait for the promise to actually resolve, but the event itself will not do that for you. So my recommendation is use some sort of fake event implementation. I've written one for the OpenBuild service connector. In case you're interested, it's really short. Just steal it.
19:01
The OBS connector is MIT licensed, so no strings attached. So use some fake events that await promises. It will make testing so much easier. Disposables. So disposables are essentially destructors that remove commands that you have registered
19:22
that unsubscribe event listeners that free up resources, and it's essentially a workaround for JavaScript not having destructors or finalizers. These tend to not really be required to be tested, but call dispose of any disposable
19:43
in some afterhook, since if you try to, for instance, register a command twice, then you'll get an error. So every time you register a command, you'll get a disposable back, and so just dispose of it after each test and do it in one of these afterhooks
20:05
as, again, your test could fail, and if you don't reach the end of the test where you deregister the command, the subsequent tests will fail, and it will not be an actual failure. So use the hooks for that. I've already talked a few times about UI elements,
20:22
and I would suggest not to check them, but if you really want to check parts of the UI, really focus only on the interesting parts. So for example, if you want to verify that you're using the correct icon, then just check the icon path.
20:43
Maybe even check only that the icon name is actually correct, but that's really your preference, yeah? And keep the part as small as possible. Since the data structures of the UI, they are not necessarily stable between the Visual Studio Code versions
21:01
since the UI changes for obvious reasons if they implement additional stuff, and so you'll have to adapt your tests every single new release, and that's just annoying, and it's a waste of your time. Very well. And with that, I'd like to cover manual testing very briefly. Well, and manual testing just means,
21:20
well, you have your workflow, and you execute it yourself. Should you do it? You can do it if your extension is, for instance, needs a very complex environment. So if you talk to multiple public clouds, and there's no staging environment, and getting a staging environment is so much work,
21:42
and you've just written this thing once, and you expect very little changes, then you can resort to manual testing. Otherwise, I would not recommend it, especially given automation is getting there. But if you go for manual testing, make yourself a checklist, make yourself all the steps that you want to execute,
22:02
and write down what should exactly happen, and verify that before each release, and you're set. Or let a machine do it, and integration testing means exactly this. So you want to run a full workflow of your extension, and you want to test the whole thing,
22:21
and you want the machine to do it. So you want some kind of test runner that will take your extension, it will provide user input, and then it will verify that what you got out, what you see, is actually what you want. The module that I have found most useful
22:42
is VS Code Extension Tester, developed by Jan Richter from Red Hat. And this leverages the fact that Visual Studio Code is built on web technology. So we can use all the nice tooling that's there to run tests on web pages. And in the case of VS Code Extension Tester,
23:02
it uses Selenium WebDriver, and it provides a pretty nice API that wraps all the calls to the DOM elements, and it gives you, as you can see in the code example, it gives you a relatively nice API to control Visual Studio Code from tests.
23:25
So what should you test? I would recommend test your main workflows. So utilize a 80-20 rule. So try to catch the most interesting workflows that will cover most of your code paths
23:42
and especially the most popular ones. And don't try to go for corner cases or minor regressions, since your integration tests are probably going to be rather slow. So you don't want to try and reach teeny tiny corner cases that are really hard to hit, especially if they're performance sensitive.
24:01
So then that's really not going to fly. How should you test? So VS Code Extension Tester is tailored around Mocha, and I'd recommend to use that, since otherwise you're going to run probably into all kinds of issues. And so just use Mocha if you can.
24:21
Use the root hooks and before and after hooks to run some setup code. If you need to spin up some development environments, do this in root hooks, do that before. Keep in mind, you can override environment variables. You can set custom home directories. You can use tricks like I talked about
24:42
with LD library path, with LD preload to modify system libraries. And that will allow you to really go into low level without modifying the actual running environment.
25:00
And then execute all your individual steps of your workflows as individual it steps. That means you can't run them in parallel, but that will not work with integration tests anyway. So what are the catches? Yeah, so these integration tests, they tend to be slow and resource demanding.
25:22
So you're running Visual Studio Code, you're running Selenium WebDriver, which includes a Chrome driver and optionally some development environment that's running there. And that can be very resource demanding. So even on my beefy development machine, if I'm running some more resource demanding
25:41
background job, then my tests will suddenly time out because the machine isn't keeping up and it got plenty of course. Also related to that, avoid sleeping for explicit amounts of time in your tests. Unfortunately, Upstream contains quite a lot of examples
26:01
that just wait 100 milliseconds for some element to pop up. This will not work on your CI. Your CI worker is much weaker than your development machine. And the 100 millisecond sleep will be 500 there, or maybe 1000, you don't know, and it will be annoying to figure out.
26:22
Another thing is certain elements in VS Code are invisible by default, unless you hover with the mouse over them. And that means if there's certain elements which only become visible, you'll have to first find the element over which you have to hover,
26:41
move the mouse there, and then it will actually show up in the DOM. If you try to find it beforehand, you'll get a test failure. So keep that in mind. It happens occasionally with tree views. And at last, forget about test coverage. So getting test coverage out of this is really, really tough.
27:02
I don't know how to do that. I've asked Upstream, they also don't know that. But I don't think you should collect test coverage of these integration tests. So, and with that, the obligatory legal slide with all the copyrights. And now, thank you for your time,
27:22
and are there any questions?