Using external dependencies in QGIS plugins
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68563 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu142 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Plug-in (computing)Physical systemControl flowSoftware developerSoftwareComponent-based software engineeringFunction (mathematics)Utility softwareAlgorithmProcess (computing)Personal digital assistantComputer configurationNetwork topologyClient (computing)Data managementUsabilityUser profileRevision controlBootingComputing platformBinary fileBinary codeDistribution (mathematics)Module (mathematics)Library (computing)Utility softwareAlgorithmCodeData structureClient (computing)CASE <Informatik>Revision controlModule (mathematics)MetadataRun time (program lifecycle phase)Binary codeLibrary (computing)Projective planeUniform resource locatorPhysical systemDifferent (Kate Ryan album)Plug-in (computing)Installation artChainData managementProfil (magazine)Structural loadSoftware developerComputer configurationOrder (biology)Game controllerInternetworkingOpen sourceFiber bundleRepository (publishing)Level (video gaming)Integrated development environmentQuery languageSource codeProduct (business)Software repositoryComputing platformMultiplication signNetwork topologyCross-platformComputing platformComplex (psychology)RootConstraint (mathematics)Lecture/ConferenceComputer animation
09:05
Plug-in (computing)CodeSinguläres IntegralModule (mathematics)SpacetimeAsynchronous Transfer ModeArtistic renderingCASE <Informatik>Software developerType theoryMultiplication signCodePhysical systemGoodness of fitRevision controlWeb pageLibrary (computing)Repository (publishing)Ocean currentInstallation artBuildingIntegrated development environmentUtility softwareRun time (program lifecycle phase)Network topologyModule (mathematics)Control flowLink (knot theory)Slide ruleAsynchronous Transfer ModeBinary codeKerberos <Kryptologie>Plug-in (computing)AuthenticationMedical imagingVirtual realityProxy serverFiber bundleComputer fileNormal (geometry)Lecture/ConferenceComputer animation
14:05
Medical imagingRevision controlLibrary (computing)CASE <Informatik>Default (computer science)Module (mathematics)Run time (program lifecycle phase)Kerberos <Kryptologie>Point (geometry)Physical systemSource codeComputer fileFiber bundleWindowDifferent (Kate Ryan album)Computing platformSoftware developerInstallation artPlug-in (computing)Integrated development environmentUser profileComputing platformNetwork topologyMaxima and minimaFinite differencePublic key certificateNormal (geometry)Keyboard shortcutComputer programmingWeb browserVirtualizationMultiplication signComputer animationLecture/Conference
19:06
Special unitary groupComputer-assisted translationComputer animation
Transcript: English(auto-generated)
00:00
Welcome. So yeah, using external dependencies in QGIS plugins is this talk's topic. Some background first. My name is Ander Okomi and I'm working as a software developer at the National Land Survey of Finland and currently working on the new topographic data production system. You might have seen the talks on Wednesday. If you haven't, you can also check those out.
00:28
About our QGIS plugin development, we do, for example, simple map tool plugins for some projects that have a template project
00:40
and the background with the layers and just some useful map tools. For example, our topographic data production system is kind of more complex where the plugin controls the whole workflow for the user as layers. And the whole system builds around the plugins.
01:01
And also we have some open source general purpose tools available as well on our GitHub. You can check those as well. So, why would you want to use QGIS plugins with dependencies? For example, if you want to share some utility functions, UI components, some useful algorithms
01:25
like structure the code better with having it in a separate repo could be useful. And also if you want to reuse existing libraries that you can find open source, that works for your use case. And also, for example, if you have a common use case for pandas for data processing,
01:44
you might want to use that as well in your plugin. But in many cases, it might be a problem to have that dependency because you have to have the library available at runtime and often you can't control the client side of things.
02:03
And then if you have a dependency, you should expect a correct version of the library. You can't work, for example, between major versions as well. And also using dependencies affects the workflow for the developer
02:26
and also in some cases might affect the tooling that you can use for the plugins. And also important is the platform dependent libraries. For example, pandas is a binary dependency, so you can't just usually include that with your code
02:45
because it might not run on the client's machine. The different options for the dependency usage could be dependency plugins, that's a QGIS built-in tool for that or those later.
03:04
And also one example could be client installed libraries that you can just request the installer, maybe even in some cases to an automatic installer. And one example could be bundled libraries with import path manipulation that you can just include the code and use, for example, some import tricks to have it available.
03:27
And one final example could be isolated imports where the dependency is included in your code with a kind of a different structure that the imports are isolated for the version.
03:41
So, the dependency plugins, it was added in some year or two ago in QGIS and in that case you just request the dependency in the metadata and the QGIS handles the dependency management. In that case there's a limitation, of course, for one version only per profile
04:05
so you have to have, like you can't use different versions for some different plugins. And of course that's limited to QGIS plugins so the dependency you are using must be a QGIS plugin and installable on the client machine.
04:20
For example, you could use a private repository if you have that use case for that. And how it looks in the code, you can often use the dependencies as is because the QGIS handles, like loads the library in order so that the imports and providers, etc. are usable directly in the code.
04:47
And, for example, on the client install libraries you can notify a user for a missing library or in some cases try to install automatically. Of course this brings some problems that the user might have network access at the time
05:03
or there might be other problems not to install the libraries automatically. And in this case as well, there's one version per environment so you can't have multiple versions available. And in this case as well, if you have like, you can't also install different versions
05:26
like if the user has like, say, Pandas 1.0, you can't have the 2.0 like request might break other workflows for the user. And in how it looks in the code, you kind of have to use some queries against the imports
05:43
where you can maybe notify the user that you are missing a library or use some fallbacks in case the library isn't available. Also, in that case you have to, like, in all code paths that you have to import you have to guard it because you can't have like top level
06:01
or your imports for that in any of the code that gets run initially. Also this brings the automatically install, maybe even the client installs bring some supply chain attack possibilities. Nowadays if you install, just request the user to install things from the internet.
06:22
So, for the bundled code where you have path manipulation you package libraries, for example with the plugin and use, for example, just path append or some import hooks to provide access to that module at runtime.
06:40
In this case as well, you only have one version available so this is also a problem if you provide a library in this way and then the user already has another version installed. So this brings problems like which one of those gets loaded first so you can't really trust the version that you have included
07:01
because it might not be that version that's available at runtime. Bundlers work with this case, but also the runtime platform has to match so you can't bundle all the platforms in case that's an issue.
07:22
Not also for the licensing, it's not often suitable to include the binaries in public repositories for the license. Another way that you can use is an isolated import.
07:40
In this case you also package a library within the plugin but you use the imports in a way that the code refers to the isolated module instead of a global top-level module. In this way you can use multiple versions, one per plugin but it works for pure Python libraries only
08:03
because in binary dependencies the module runtime path has to match the compact one. There are two different options for the bundling. For example, one could be kit-shop-module use.
08:20
This is often seen in some users' plugins. You can use kit-shop-module to pull the dependency code to the source tree and package that source tree as the plugin so you have the library available. The repo structure, in this case for the dependency,
08:42
usually must be in your control because there are some constraints on how the code is run at runtime so you must have the structure that supports good use of a submodule. For example, usually it's the package root is at the repository root
09:02
so you have the library at the correct location in the tree. Also, in this case you have to use relative imports because you don't know the runtime path that the library is at. So, in this case all the submodule imports must be relative.
09:23
In this case you can edit the dependency at the same time if you just edit the checked-out code in the tree and you can just do the commits in that tree. So, it works quite well for that. Of course, in the build system it must have access to the repository.
09:43
For example, if you are behind a proxy and you have to have access to a GitHub repository that access has to happen at the build time or when the code is checked out. Then we have another case of the isolated imports
10:01
with the build time rendering. This is kind of the approach we have chosen because this helped us to use the Python system as we normally would. We can just install dependencies as is, use the normal imports and how this happens is that we can transform both our own code and the dependency code at build time
10:23
to be in the plugin module name base. We use like a vendor subpath in the tree where we put the modules and import from there. In this way, the tooling and workflow is very well interoperable with the Python packaging ecosystem
10:42
so you can just use most poor Python libraries and use those in your code if it's not anything special in how the dependency uses its imports. Also for this, this is very useful
11:02
you can use editable installs for the development because in this case you have the dependencies as normal Python library. You can just install a repository in your local machine and do the edits there and have it also be the library that is run by QGIS the development mode.
11:24
Also for this approach, you need access to the package at build time. For example, if you use pip install for a package it also works if you have a private repository for example so it works really well with that as well. And for the packaging, this of course needs custom tooling
11:43
and for that we have built a developer tool here's a link if you don't know the slides for the last year's talk and the github page so you can check out more of that. So what it essentially is, is a development mode helper
12:03
with how it handles the for example, if you're developing a virtual environment it blocks the libraries from that automatically for QGIS and when you build the same code it takes the libraries from the current environment bundles those with the plugin.zip file
12:23
and it usually works quite well if you use very well defined modules so that the imports are not anything special. And this is kind of the approach that works best
12:43
works only for the Python libraries so for example, when we control our visualized QGIS environment we often pre-install some binary dependencies for example pandas or some Kerberos authentication libraries
13:03
the QGIS image and then we have a well defined version that we should expect to be available at runtime for those and for the rest that are Python libraries we can just use this rendering type way to include the versions.
13:21
So with this approach when we have for example a utility library and we have to have a major version bump for that and we change the API we don't have to change all the plugins that will be installed to the environment so we can just do one by one and it won't break the others.
13:41
Whereas with the previous not isolated import types for example, if you have that kind of a dependency and you update an API you have to update all the plugins that will be at that runtime QGIS session so that it will use the same API.
14:04
That's all for my talk and I think we have some good time for questions.
14:27
Hi, thanks for your talk. Did you have to deal with different operating systems or did you have plugins for both Windows and Linux and if so, how did you handle packaging up for different operating systems?
14:41
For us, the expected runtime is Windows. We have a virtualized Windows environment but for example some of our developers use Linux for development and the way we built the actual runtime images we have for example Pandas and Kerberos libraries installed
15:01
these are the versions that will be available at runtime. So the developer that uses Linux can just take that for a requirement file and the development environment should match with the versions and those will be installed like normal Python packages
15:21
and only at the build time which will happen for the target platform Windows we do the rendering so we only have that for Windows so it's not a problem but of course if you for example have both like Linux and Windows it's like you should be able to package both platforms
15:44
but we haven't had a use case yet but I think it should work.
16:06
It's okay. I had a problem with the installation of the external package through the QGIS on some users profiles but it was the situation when the user didn't have any Python
16:21
installed by his own so that was the default Python from the QGIS and what might be a reason why I couldn't install the external package by the PIP like I use normally through the QGIS
16:41
because there was an error and do you know the answer for it? We have had some problems with old versions of PIP for example that comes with QGIS so that might be one issue like for example I don't think it has happened in the recent versions but with an older version there was like PIP 10 point something
17:03
so it wasn't able to install some binary packages correctly it tried to build them from source for example and in some cases there's like the version that comes with QGIS with PIP there's a bundled request library and it also bundles the CRANIN certificates
17:22
so it might be like an old version of the certificates and you don't have access to like a newer certificate Any more questions? Any more questions? I have one more question
17:40
Do you have experience with like binding dependencies of other languages? So I'm thinking about JavaScript or Node.js ecosystem because there might be some libraries which could be interesting too and also now there's this WebAssembly coming up where programs are compiled so they can be run in a browser
18:02
I'm just wondering maybe they could also be run in QGIS or just wondering if you have any experience with other languages? No, we haven't really had experience with those Also one of the differences is of course that in Node.js you can have like a tree of dependencies
18:21
so it automatically provides a way to have like a different version if for example you use some library that wants a max version something so it bundles like a... if you do not MPN install something so it installs like one main version that works for most
18:41
and then it installs like the subtrees where it needs the different versions but because it's not the possibility with Python it's only like one module if you use plugin or something it's always the one version at the runtime So it's a kind of a different ecosystem for that
19:01
Alright, okay Then thanks again for your talk So, one applause