Custom workflows in QGIS thanks to Python - a non technical introduction
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43395 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Presentation of a groupUsabilityKey (cryptography)Lecture/Conference
00:46
Open sourceRevision controlBroadcast programmingTime evolutionFocus (optics)Keyboard shortcutPhysical systemCompilation albumGrass (card game)Plug-in (computing)Term (mathematics)Electric currentGoogolNumberMessage passingCodeMultiplication signKeyboard shortcutInheritance (object-oriented programming)Open setTouch typingFreezingMobile appProduct (business)Presentation of a groupFocus (optics)Arithmetic meanTwitterLibrary (computing)Key (cryptography)Moment (mathematics)1 (number)Process (computing)Goodness of fitRevision controlScripting languageAssociative propertyPlug-in (computing)Wave packetBitPoint (geometry)ChainSound effectTerm (mathematics)GoogolInterface (computing)Category of beingBlogPlotterField (computer science)Open sourceCausalityComputer animation
06:11
Control flowFunction (mathematics)Exception handlingKeyboard shortcutPlug-in (computing)Video game consoleScripting languageForm (programming)Macro (computer science)Regulärer Ausdruck <Textverarbeitung>StatisticsLocal GroupShape (magazine)Process (computing)Directed setComplete metric spaceCodeAreaText editorMenu (computing)Series (mathematics)Boilerplate (text)Integrated development environmentExecution unitMultiplication signCartesian coordinate systemLogical constantDefault (computer science)Process (computing)NumberSoftware testingPoint (geometry)Text editorDependent and independent variablesSoftware developerExpressionBuildingPlug-in (computing)TouchscreenOpen setGame controllerException handlingFunctional (mathematics)CodeVideo game consoleScripting languageSet (mathematics)CASE <Informatik>Library (computing)WindowUser interfaceVideo gameSoftware frameworkAlgorithmInterface (computing)Data structureSocial classKeyboard shortcutIntegerElectronic mailing listMenu (computing)Traffic reportingCuboidComputer animation
11:37
Order (biology)WebsiteScripting languageOffice suiteComputer fileLevel (video gaming)Source codeJSON
11:57
Scripting languageMaxima and minimaUniform resource nameBitScripting languagePoint (geometry)Vector spaceoutputExpressionKey (cryptography)CodeLine (geometry)GoogolUsabilityMultiplication signUser interfaceField (computer science)System callXMLProgram flowchart
13:24
Scripting languageGame theoryField (computer science)BitGeometryCodeComputer fileLine (geometry)Type theorySemiconductor memoryVirtual machineVector graphicsLevel (video gaming)ResultantAlgorithmInheritance (object-oriented programming)Control flowTable (information)Video gameCASE <Informatik>Lecture/ConferenceMeeting/InterviewXML
15:11
Regulärer Ausdruck <Textverarbeitung>Wechselseitige InformationComputer configurationCodeRootVariable (mathematics)Integrated development environmentIntegrated development environmentComputer configurationExpressionFunctional (mathematics)Projective planeMessage passingLine (geometry)CalculationKey (cryptography)Group actionField (computer science)Connected spaceInheritance (object-oriented programming)Multiplication signCodeText editorPoint (geometry)Source codeJSONXML
16:42
Computer configurationCodeRootVariable (mathematics)Integrated development environmentMaß <Mathematik>Closed setPasswordMessage passingOpen setQuicksortProjective planeMacro (computer science)Field (computer science)WordPiLecture/Conference
17:28
Line (geometry)Macro (computer science)Source codeSocial classInformationGUI widgetForm (programming)Attribute grammarObject (grammar)Field (computer science)Scale (map)User interfaceProjective planeRun time (program lifecycle phase)Form (programming)Core dumpBitKeyboard shortcutWordComplete metric spaceError messageMacro (computer science)Open setDrop (liquid)Mobile app2 (number)Greatest elementSystem callCartesian coordinate systemQueue (abstract data type)Game controllerMultiplication signRoundness (object)Computer animation
19:32
Spring (hydrology)Cartesian coordinate systemFerry CorstenMeeting/Interview
20:00
Keyboard shortcutObservational studyIndependence (probability theory)Computer fileMathematical analysisCovering spaceData managementVisualization (computer graphics)Sheaf (mathematics)Letterpress printingPlug-in (computing)View (database)Regulärer Ausdruck <Textverarbeitung>GeometryPolygon meshSpring (hydrology)Boom (sailing)System callSoftware developerBit rateÜberlastkontrolleRevision controlPoint (geometry)Inheritance (object-oriented programming)Goodness of fitComputer animation
20:28
Revision controlComputer animation
20:47
InformationOpen setPresentation of a groupTouch typingMathematicsMeeting/Interview
21:44
Lecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:07
Welcome everybody to the Bolero room, the Coral room and the Menuit room. So we're going to have a small presentation about where Python is involved in QGIS at all.
00:25
So in which kind of places in QGIS can we find Python and how can we use it to our advantage. It's going to be a non-technical introduction. It means I'm going to show you where you can do it and what kind of things you can imagine doing with it.
00:41
But I'm not going to show you how to do it. Sometimes I have some animations that will show you code and that kind of thing. But the assumption is not that you will be typing all the code, you can obviously. But you much rather get the presentation and get the code from there later on. But it's more kind of giving you ideas about what can be done at all in QGIS with Python.
01:06
I'm Marco Bernazocki, I'm Bernazocki, I'm the QGIS association co-chair. I'm the dad of QGIS on Android, it was my baby back a couple of years ago. Now it's known as Qfield, the mobile application.
01:22
And I'm also the Open GIS CEO. Open GIS is a small company in Switzerland that does, if it touches QGIS or PostGIS and it's open source, we probably know a lot about it. We don't know much about any proprietary things. To go back to QGIS.
01:45
QGIS is a product that has been growing steadily for the last 17 years. We are now at version 3.8, which has been released in June and Friday.
02:04
We have the feature freeze for 3.10, so in two months, end of October, we will have 3.10 coming out. Python has been in QGIS since 0.7.
02:23
Sorry, 2007 version 0.9, Ganymede. And as you see, the commit message then was the major focus of this race is to have Python bindings. And what I'd like to point out, beside the fact that the battery is almost gone,
02:41
is that shapefiles were already breaking things back then. So do not use shapefiles. Anyhow, we are now at 3.4, long-term release. The last long-term release used to be 2.18, and why am I mentioning this? Well, I'm mentioning this because if you had built Python scripts against the Python API of QGIS in 2.18,
03:09
well, you will have to do some work. When we change the major number of a release, so when we go from two to three for one to two and eventually from three to four, it means that we are breaking API compatibility.
03:24
So it means that you will need to go and fix your plugins, fix your scripts, fix your things. Why did we do that? It's not because we are mean, and it's not because we want you to work, it's because we needed to clean up stuff. There were lots of, you've seen the code that's been growing organically in 17 years,
03:43
and in the last three to four years, it has been growing massively, with a lot of super high quality code coming in. So it was time to get rid of our childness, how to say, well, mistakes or whatever.
04:07
So you will need to upgrade your tools. But it's most of the time, when I take a large plugin, and I have to port it from two to three,
04:22
like the biggest plugin that I ported took me maybe three days of work to change, but we are talking about very large plugins. If I take a very small plugin, maybe in half a day it's done, and there are tools to help you, but yeah, it involves a bit of working.
04:44
Somebody told me in a presentation you always need to have a training chart, and the trend has to go up, so obviously I took QGIS versus ArcGIS from 2004 to 2019, animated it, this is Google Trends, so for the ones that don't know Google Trends,
05:04
it shows you what people are looking for, and you see that we have Switzerland, we have an amazing country, which is France, I'm not from France, but all the people here from France, good job on you. They passed a long time ago, already the key moment in Switzerland is right now,
05:22
we are going over, and then we have some bizarre countries like the USA, where inexplicably ArcGIS is still very much the leader product, but yeah, that's about the trend. Now let's get into the interesting stuff, Python and QGIS.
05:40
I already mentioned the first point, you need to upgrade because we moved to Python 3, but also we moved to PyQT5, and also in the background QT5, meaning that we are getting much more up-to-date base libraries, so all the support there is much, much better. For people that don't know PyQT5, it is a library that allows us to basically build interfaces,
06:08
build code much easier, you don't have to define all the code, you have a lot of code that is pre-made for you. It's not one hour ago, but when I took the screenshot it was one hour ago,
06:23
there is now, since 3.8, there is a dedicated API documentation for Python. Up to now, if you were developing your plugin, your Python scripts, your Python custom applications, you needed to look into the C++ API, and then decide, well, if it's an integer there, it's probably going to be a number in Python as well.
06:45
Now there is a dedicated API documentation for Python as well. So there is where you find all the methods that you are allowed to use in your own tools.
07:03
Now, what can you do with Python in QGIS? Well, you can have a more granular control over the UI, you can extend functionality because you can build up your own things, basically anything that you can do with the mouse, you can do with the Python API,
07:24
and actually you can do more, because at times you can just combine things much more efficiently, and in some exceptions there is no Python API binding yet, or sometimes deliberately because it's something that should not be exposed.
07:42
Where is Python at all? Well, we have plugins, we have a Python console, we have Python scripts, there is Python in the forms, there is Python in the macros, there are custom Python expressions, and you can build your own custom application in Python. So you see that Python is just about everywhere in QGIS.
08:03
Starting from the plugins, the good news is that the plugin ecosystem is back, after when we went into 3.0, there were a lot of people that still needed to do their homework and upgrade, so the plugin ecosystem was very small, I think I managed to be the third one to have a plugin up and running,
08:24
but now you see here there is another pretty trending chart that grows a lot, but you see that by January this year we already were back at 350 plugins that are available for Python 3, so now basically any major plugin has been ported to Python 3, so you can,
08:43
it is definitely time to uninstall the 2.18 that you still have and go into 3.4 if you want to stay on a long term, or if you want to go more on bleeding edge, 3.8 or 3.10 very soon, or even help us out by testing and installing nightly and report box.
09:04
Plugins, when you are building plugins as a developer, you can use the plugin builder, which will help you out and make you kind of like the first boilerplate code, so you don't have to go and create all the default classes and all the structure that you need,
09:23
it will bring some testing infrastructure with it, which is very interesting, and the other very interesting thing that we can now do is create processing plugins, which are plugins that do not have a user interface and will just integrate into the processing framework of QGIS
09:46
and will just put an algorithm or whatever you wrote at disposal into the processing framework, which means you just deploy your own algorithms as processing plugins, they will not have any interface but they will show up in the list of processing algorithms.
10:07
Then another very interesting thing we have, the console, Python console, you can open it up in the plugin menu and then console, and there you have an interactive place where you can be typing Python and directly see the result,
10:26
so if you need to select everything that is above 500, obviously you can go and click in the select by and use an expression, but if you are a Python freak like me, you'll be doing it much quicker in Python.
10:41
The cool thing is it has auto-completion, as you can see here, so you have to go much more or less to the documentation, and on the right side it also has a small editor where you can save scripts, run scripts, I'm not saying it's an IDE, it's not an integrated development environment,
11:03
but it's a very nice environment to start off very easily. The big advantage is foremost for people on Mac or Windows, is that from here you're sure that you're going to be triggering the same Python that QGIS is triggering, so you have all the dependencies that QGIS has because in Windows and Mac,
11:24
it's not always the case that you're working exactly against the same set of libraries. Here you can save the scripts and basically rerun them. What can a script do? Well, scripts can do a lot.
11:43
This is a text file that includes all the offices of a bank in Switzerland, and I found that on their website and I wanted to have a map out of it, so I wrote a geocoder, a Google geocoder, using the Python API of QGIS,
12:07
created a script, added the script to the toolbox, here my geocoder that I wrote, it is a bit of code, it's like 40 lines maybe, but it does call to the Google API asking,
12:21
hey, I have this text, whatever, give me a point back, so it's pretty short actually for what it does. There we go, now it appears, and from here I can execute it, and once I execute it, I don't have to do anything to generate the user interface,
12:42
I just need to say my script needs to have a vector layer as an input, and it needs to have a variable and an API key and everything, so I can choose, all these are defined just as inputs to the script, and once I'm good, I choose my API key,
13:02
and I'm going to set up how is the expression that matches my fields, this is basically just because my script was built like this, you could hard code it so that it works only with Google,
13:20
obviously I wanted it to work with Google and NominaTeam, so I made it a bit more customizable, so I can say well, my fields, the question that needs to go to Google is name, which is a field in my data, comma, and down here you always see the example of what I'm creating, now it's invalid, this machine is super slow,
13:44
usually it moves a bit faster, but I also type a bit faster, so if you hire me, I'm a bit faster than that in coding. So you see here it says post finance, and now it should be clicking okay,
14:04
wow, that is slow, and now it should click run, and eventually we should get a map of, I choose Google,
14:21
you see even on this machine the algorithm is super fast, we get the result and you see well this bank, for the people that know Switzerland, this bank is pretty well distributed across Switzerland,
14:41
so it's one of the major banks in Switzerland, it's no surprise there, but basically you see with about, I think 30, 40 lines of Python, I have a reusable thing that takes a CSV file, and I can tell how the file should be scripted, and then I get a complete vectorial table,
15:03
a memory layer in this case, but I could write a geo package out of it, and I have geo data in it, so very cool thing to make more complex workflows, and I'm going to go to the next slide, then next slide, custom Python expression,
15:21
if you work with QGIS you know that we do have custom expressions everywhere, I mean expressions everywhere, we just saw before this expression editor is where I can mix up text with functions, with data, super handy and extremely used in QGIS, and at times, well there is not the expression that you want,
15:42
for example if you want an expression to get the user name out of a Postgres connection, because you want to, you know, just have the connection set up, and then you want to, in every point that you're digitizing, you want to put in who digitized it, and you can just, instead of having the person typing it, you can just reuse the connection of Postgres,
16:02
for example, it's just one example, and here I can just define with the decorator, add QGIS function, and then what arguments, and in what group it should go, and this function is going to show up in my custom group, in the field calculator, and I can reuse this function everywhere, in every project, these are not project,
16:23
these are not connected to a project, these are connected to the QGIS installation. The next thing that you can do, is basically give a code command line option, or create a startup.py file, or set a PyQGIS startup environment variable,
16:44
which points to a startup.py file, and in there, we can do things that happen whenever QGIS starts, so if I, in my startup.py, I have iface, messagebear, push message,
17:00
phosphor-g for the win, as soon as I start QGIS, I will get a message bar saying phosphor-g as a title, and phosphor-g as for the win. Super handy if you want to remember people that you do not put your own password in data fields, or sort of that kind of things.
17:21
Next thing we can do, project macros, project macros are only three, open project, save project, and close project, you'd say well that's not interesting, well, it is very interesting, because in the PyQT word, we have a thing called signal, or in the QT word there are signals, so whenever something happens,
17:42
signals are sent here and there, so a new layer created, new layer selected, new feature deleted, and when you write an open project macro, you can bind to those signals, so you can say well, whenever there is a new layer created, do something, so actually,
18:02
the open project macro allows you to just create whatever you want, connect to whatever signal you want during the whole run time of the project. The save project macro, obviously the same for whenever the save button is clicked, and the close project macro,
18:21
it's obviously for whenever you want to clean up after your project. In the feature forms, oh, sorry, something happened with the graphics there, in the feature forms, I can create custom forms, where I can create complete UI files,
18:43
I can say well I want the buttons up here, I have no idea why, I can put a drop down, so you're really creating a complete user interface for a form, a form is when you click to add a new feature, or when you click to see the feature, you can put your logos, you can put whatever you want, and this can be,
19:02
they will automatically bind to the data, and obviously here you have full control, you can add a very annoying error message, use the bar when you do error messages, but this was a bit easier to show whatever you can do.
19:21
So, yeah, plenty of things we can do there, and the last and most complex thing that you can do is build your own Python application, from QGIS core import star, and from that on, you need to set up two, three things, create a QGIS application,
19:41
call the init QGIS method on that application, and then call eventually the exit QGIS application, and in between you do whatever you want with your Python application. Last slide, the shameless plugs. If you're interested in more of this,
20:05
QGIS developer called, Bird's Eye View, Menke, Kurt Menke, created a super, it was written here, Kurt Menke created a super good book about QGIS 3x,
20:22
I'm not making any money out of it, it's a really good book. Then, as I mentioned, I was the father of QGIS and Android, it's now called Qfield, we released version 1.0 in February, super cool tool, try it out.
20:41
And the last one is maybe gonna work or maybe not, looks like not. I have two more presentations, one goes much more into technical details, about what I just presented, and on Friday afternoon at two o'clock in Boulaire,
21:00
I'm not sure where, but names change. We have a one hour and a half session called QGIS on the road, it's a very entertaining way to look at what QGIS can do for you with the story of Maya, the beekeeper that's grown her business, and with the grown QGIS needs,
21:20
we're going to start really from the beginning up to very, very detailed things that you can do. We also don't show exactly how to do it, it's more like an entertaining thing. Thanks for listening, if you have any questions, get in touch, info at openjs.ch or marco at openjs.ch,
21:40
you'll get directly to me. Thank you very much.