QGIS for Hydrological Applications
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68524 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu104 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Integral domainDigital signalOpen setWhiteboardPoint cloudPoint (geometry)Visualization (computer graphics)Process (computing)Plug-in (computing)Cartesian coordinate systemYouTubeCore dumpWater vaporFreewareWave packetBitOpen sourceType theoryPresentation of a groupLatent heatOpen setINTEGRALComputing platformLecture/ConferenceComputer animation
00:55
Internet service providerProcess (computing)Plug-in (computing)AlgorithmEndliche ModelltheorieData modelStapeldateiSocial classInternet service providerMathematical analysisSoftwareEndliche ModelltheoriePlug-in (computing)Interface (computing)AlgorithmSocial classProcess (computing)Software frameworkIntegrated development environmentStapeldateiINTEGRALFrame problemComputer animationLecture/Conference
01:48
Process (computing)Internet service providerExecution unitPlug-in (computing)BitPlug-in (computing)Internet service providerGrass (card game)Physical systemCore dumpAreaINTEGRALComputer fileProcess (computing)Programmer (hardware)Pauli exclusion principleSoftware developerSoftwareRevision controlSource code
02:55
Population densityPlug-in (computing)Independence (probability theory)Grass (card game)Process (computing)Internet service providerProcess (computing)Internet service providerLevel (video gaming)Computer animation
03:15
Visualization (computer graphics)Extension (kinesiology)SoftwareProcess modelingStochasticFunction (mathematics)Temporal logicSimulationData modelBuildingBefehlsprozessorCore dumpOperations researchMultiplicationPlug-in (computing)Integrated development environmentPrototypeScripting languageInstallation artWindowAreaOrder (biology)Conditional probabilityBoolean algebraOperator (mathematics)Finitary relationTable (information)DistanceData managementDigital signalDerivation (linguistics)Endliche ModelltheorieComputer fontStreaming mediaComputer networkoutputDiscrete element methodServer (computing)Open setEllipsoidPhotographic mosaicGamma functionPolygon meshDatabaseMathematical analysisRaster graphicsPrice indexProcess (computing)Inverse trigonometric functionsCalculationKernel (computing)Military operationCellular automatonData conversionAxonometric projectionSubsetBoundary value problemComputerRead-only memorySource codeInterpolationOvalPixelEndliche ModelltheorieIntegrated development environmentElectronic mailing listAuditory maskingPlug-in (computing)PrototypeProcess (computing)Physical systemExecution unitMathematicsImage resolutionComputer fontStreaming mediaResampling (statistics)MetreCartesian coordinate systemMappingAlgorithmReal numberLatent heatSocial classModule (mathematics)Document management systemLine (geometry)Operator (mathematics)PixelPower (physics)InternetworkingGraph coloringSpacetimeCoordinate systemLevel (video gaming)Field (computer science)Grass (card game)Different (Kate Ryan album)Core dumpAxiom of choiceSet (mathematics)Source codeTesselationCellular automatonSubsetBitBoundary value problemSoftware frameworkCalculationComputer programmingOvalAreaOpen setFunktionalanalysisProduct (business)VirtualizationPhotographic mosaicObservational studyIndependence (probability theory)HookingPoint (geometry)SoftwareDirected graphOpen sourceStochastic calculusDataflowMathematical modelInstallation artProcedural programmingMultiplication signInverse trigonometric functionsKey (cryptography)TangentTouchscreenDegree (graph theory)Direction (geometry)Condition numberWindowRaster graphicsArc (geometry)Discrete element method2 (number)DigitizingComputer animationLecture/Conference
10:11
Directed graphDiscrete element methodRadio-frequency identificationArtificial neural networkTheory of everythingDataflowDirection (geometry)Degree (graph theory)Codierung <Programmierung>Raster graphicsPointer (computer programming)Single-precision floating-point formatAreaPointer (computer programming)NumberDataflowDirection (geometry)SoftwareBitDifferent (Kate Ryan album)Codierung <Programmierung>Key (cryptography)AlgorithmMixed realityCalculationGraphics tabletArrow of timeKeyboard shortcutWater vaporSynchronizationNeuroinformatikProfil (magazine)Degree (graph theory)OvalObservational studyFile CarvingOpen sourceWindowPixelDiscrete element methodPower (physics)Raster graphicsFunktionalanalysisComputer animationLecture/Conference
13:04
AlgorithmDataflowDirection (geometry)Single-precision floating-point formatPixelError messageAddress spaceDivergencePartition (number theory)Partial derivativeSoftwareInfinityMassFluxPointer (computer programming)Core dumpComputer configurationProcess (computing)Internet service providerMathematical analysisRange (statistics)Cellular automatonSource codeInformation managementOperations researchStreaming mediaComputer networkMathematical optimizationGrass (card game)Raster graphicsScripting languageVector spaceDiscrete element methodComputer fontMilitary operationOrder (biology)outputPoint (geometry)Data modelCalculationTotal S.A.Gamma functionPlug-in (computing)Endliche ModelltheorieSoftware repositoryLocal ringOpen setVisualization (computer graphics)Maxima and minimaEmailDirection (geometry)InfinityAlgorithmDataflowOrder (biology)SatelliteOpen setMedical imagingLevel (video gaming)Axiom of choiceCuboidComputer configurationWater vaporResultantSoftwareSingle-precision floating-point formatProcedural programmingVolume (thermodynamics)Cellular automatonStreaming mediaRight angleParameter (computer programming)BitCASE <Informatik>Goodness of fitSoftware testingPixelLink (knot theory)CalculationAreaError messagePolygon meshFunktionalanalysisMusical ensembleEvent horizonMultiplication signSlide ruleEndliche ModelltheorieRevision controlGrass (card game)Internet service providerArrow of timeGame controllerWeightVirtualizationExecution unitDifferent (Kate Ryan album)Point (geometry)Pointer (computer programming)Radio-frequency identificationMultiplicationScripting languageProcess (computing)Core dumpObservational studyDirected graphPlug-in (computing)RoutingRepository (publishing)YouTubeRaster graphicsMixed realityDiscrete element methodGauge theoryCausalityComputer animation
20:01
Event horizonEvolutionarily stable strategyPurchasingPresentation of a groupCartesian coordinate system2 (number)Student's t-testLattice (order)Event horizonComputer animation
20:38
Water vaporObservational studySurfaceLevel (video gaming)Student's t-testInformationTime zoneMultiplication signRemote Access ServiceRoundness (object)Condition numberAlgebraProcedural programmingWeb 2.0AlgorithmPresentation of a groupSpecial unitary groupStreaming mediaGravitationPoint (geometry)Content (media)Raster graphicsLecture/ConferenceMeeting/Interview
23:04
Special unitary groupLeast squaresComputer-assisted translationComputer animation
Transcript: English(auto-generated)
00:00
So I'll keep my introduction also a bit short. So I work at the IHE Delta Institute for Water Education, the largest education facility in the world, specialized in everything with water. And there I teach topics on GIS. And I also have my own company, CrossGIS, to give support on open source GIS,
00:21
individual trainings, online trainings. What's very interesting is that I also have the GIS OpenCourseWare platform, where you can find a lot of free courses. And I have my YouTube channel. You might want to look there for more, related also to this presentation. But let's get started. So the nice thing about QGIS is
00:41
that it's an integrator of tools. It comes with core applications that we can use. But for hydrology, we mostly have to rely on plug-ins and a specific type of plug-in. So we're going to talk a little bit about that. So there are the so-called processing provider plug-ins. And maybe you're not so aware of what that means.
01:02
So these are different tools that are also standalone tools that you can use for your analysis in Python or in their own environment. You might see other talks at this conference specifically about these tools. But because QGIS is a nice integrator, and they are processing provider plug-ins,
01:20
they can add these algorithms from third-party software to the processing toolbox. And it will, in that way, also be integrated in the QGIS Python framework. And you can use the model builder to make your graphical models and have your workflows. You can do batch processing. It comes as Python classes. And if you develop these kind of plug-ins,
01:41
you can also have things added to the interface. That's not necessary, but you could do that. Until recently, some of these processing provider plug-ins have been packaged with QGIS, but this has changed a bit in the past years. So maybe Python programmers know PEP. There's also KEP, Q-E-P, the QGIS enhancement proposals.
02:05
And there was one in the past of removing the third-party providers from being packaged with QGIS. So we talk about Saga and Grass. You have been used to that that's packaged with QGIS. The problem with that was that the core developers who are already busy with a lot of other things
02:21
related to the core of QGIS also had to maintain the integration of these tools in QGIS. So now in recent versions, you will see that you have to install these tools and a processing provider plug-in to deal with it. And for hydrologists, this caused a bit of a problem with Saga. The upslope area tool is not working anymore.
02:42
You have to do all kinds of workarounds, install all dependencies or tweak some files on your system. And that's not what hydrologists are necessarily interested in, to tweak files related to software. So well, here you see that it finally happened that these non-native processing providers were removed.
03:03
And this Saga issue has still not been solved. And I, of course, for my work and also for courses, need to rely on robust tools. So therefore, because we had a level playing field, I started to develop during COVID,
03:21
PC Ruster tools plug-in, which relies on also very old GIS like grass. And it's also open source and it comes with specific Python modules for hydrological and environmental modeling. It's real power is dynamic modeling. So in space and in time,
03:41
stochastic modeling and data assimilation. But for QGIS, I had the idea to add those 100 Ruster processing tools to the processing toolbox, and then also have the possibility when people can hook their hydrological models that they developed in the PC Ruster framework also to QGIS. So this was developed prototype during COVID
04:03
and then with Niall Dawson, we finalized it as official plug-in, which was launched in 2021. And now it's used a lot in my classes and in the book, QGIS for Hydrological Applications. The second edition has that. The first edition is still with Saga.
04:20
And I wrote that book together with Kurt Menke, who's also here. And we teach that also together in our classes. And thanks to Jurgen Fisher, it was included in the OSGO4W installer. So people can easily install it like they can also install the missing grass or Saga packages because they're now third party tools.
04:40
And on all operating systems, it works great in the Conda environment. And this morning I heard that also grass will have a Conda installer. So you will have everything nicely in QGIS if you install it in a Conda environment. So this is then how it looks like in your processing toolbox. There are all kinds of operations, but for hydrology,
05:01
these hydrological and material transport operations are very useful. And the names of all these tools are exactly similar as their Python function that you can use in the PC Raster Python framework. So also didactically, when people get used to making their models, graphical models, or using the tools in QGIS, they can easily step to Python programming if they need to.
05:25
But back to what hydrologists often need to do is stream and catchment delineation. And independent of the software, there are certain steps you need to follow. And of course you need a digital elevation model. You need to put tiles together.
05:41
You need to fill the sinks and remove spikes, calculate the flow direction, and derive the streams, define your outlet on the derived stream because you're making a model, and then finally you can derive the catchment. And I'm going to guide you a bit through the choices that you have in QGIS for doing all this.
06:01
First of all, to get your digital elevation models, of course you have probably nice national data sets, but if you don't have that, you're relying on global open data sets. And there are two very nice plugins. There is the OpenTopography DEM Downloader plugin, which if you have an account, you can get an API key and you can use this plugin,
06:21
and as you see on the screen, it will give access to eight different global DEMs that are available. And the nice thing is it will download it just for your area of choice, so the map canvas, and you don't need to stitch the tiles together. Another plugin is the SRTM Downloader plugin, which downloads the SRTM one arc second product,
06:42
and it downloads all the tiles. And in a recent release, it creates a virtual raster that mosaics these tiles already for you together, and you can use in your further processing. So mosaicing often is skipped because the tools already do that for you. But if you need to do it, you have two choices from QGIS Core.
07:01
You can either use Merge, that will create a new data set which is quite large, just a stitched geotiff probably, or you can make a virtual raster, which doesn't duplicate the data, but defines virtually that the tiles belong together. If you have DEMs from different sources with different cell sizes,
07:21
I can recommend to look at the Resample tool from the PC Raster Tools plugin, which will fit everything to a mask raster and resample it to that resolution. And you can add as many rasters as you want. It will fill in the no data that is left over within the mask with the next raster that is in your list that's basically how it works.
07:43
There's a problem with these DEMs that you download from the internet because they often use the geographic coordinate system, and that is not useful for hydrological applications because they need to be projected to a coordinate reference system. It has to do with this problem. Maybe you know from mathematics
08:01
when you calculate a slope, let's say of this point, you draw a tangent line, and you do dz over dx with the arc tangent, you can get then degrees. The problem is, if you have the geographic coordinate system, is that your x units and your y units are in degrees, latitude, longitude,
08:21
and the z units are in meters of elevation, and that most of these algorithms will still run and will produce beautiful maps with nice colors but completely wrong. So you should always re-project your DEM to coordinate a reference system to avoid this. But of course, a raster is not a line,
08:43
so what it does, maybe you have never realized, if you calculate the slope or the flow direction, it will do that in a moving window of three by three pixels and it will assign the steepest slope to the center pixel that it encounters. And as a consequence, you will lose the boundary because there's no data.
09:01
And that's what all slope algorithms do if you apply it to a raster. Once you've downloaded, if you started with tiles, then often your study area is not nicely in the middle of a tile, but scattered over several tiles, and then you need to clip it. In raster terminology, we call that subset.
09:22
And you really need to do that because one of the next steps is quite calculation intensive, so don't keep it at a very large DEM that you might have downloaded, but clip it really to what you as an expert see as the boundaries of your study area. If you make it too small, then you will get a bit of squared catchments out of it because it stops at the boundary.
09:40
So don't make it too big and don't make it too small. What you can also have is voids. These are no data pixels and they need to be removed, otherwise you cannot do the rest of the procedure. And there's a core tool, the GDAL tools are still of course core and it will remain like that. The fill no data tool. There are other fill no data tools
10:01
that do completely different things, so you really need this fill no data tool. And what it does, if you look at the picture, it will look at the surrounding elevation values and it will interpolate that. And then you have a raster without voids. A bigger problem is sinks. These are artificial depressions in your DEM.
10:24
And here you see it, the water can get trapped into the pixels with a lower value while it need to flow to your outlet. And this has to do with the acquisition of DEM. So they always have that problem and you need to run this fill sinks algorithm to remove these artificial depressions. But it will also remove your non-artificial depressions.
10:43
So if that's important in your study area, you need to still burn it into your DEM after applying this fill sinks function. How does it work visually? So we have here with the arrow, you see the flow direction, and we look to the side, the profile of elevation, and we see that the water gets stuck.
11:03
Let's see if this works. Water gets stuck here and doesn't flow to the outlet, which is further. So there are different ways to deal with it. Either increase the level or carve that the water will flow out.
11:21
This is a calculation intensive step, needs to be done iteratively and there are many algorithms for that. We'll look at that later. But the flow direction, that is what you need to calculate that. And you can of course use a raster with directions in degrees and then we have zero to 360 degrees.
11:40
Zero and 360 are north. And all the numbers, they're readable there. We can do that. The problem is that you need more than eight bits to store it. And with eight bits we can store two to the power of eight is 256 values. Compute to store it down from zero to 255. And therefore, they came up with
12:02
that we don't need all these values. They're just eight discrete directions that we can use. And that's the D8 algorithm. So if you remember that moving window that calculated the slopes to all the directions, that's just in eight directions and not in all the 360 degrees.
12:21
So the problem there is that different tools use different encodings. And that encoding that's called a flow direction raster or the D8 flow pointer raster. You can see both in software. And here you see examples of different software, commercial and open source and they all use different encodings.
12:40
So if you wanna mix tools, it's very important that you take care of which encoding your flow direction of your previous step is using when you feed it into a next algorithm from another tool which expects a certain encoding. So PC Raster, for example, uses the encoding which is similar to the keys on your numeric pad on your keyboard which has those arrows.
13:04
So be aware of that. And then there's also the different algorithms that you can use. So the most simple one and most used one is D8 where you have these eight discrete directions. But there can also be the D infinity algorithm.
13:20
And that's uses more neighboring cells and it's good for handling ambiguous topography and it minimize the over dispersion. Then we have single flow direction and multiple flow direction. And they are also more advanced ways of doing that. But my advice would be here,
13:41
always start with the D8 and see if that already gives good results in science we normally choose for the most simple solution. And more difficult cases are when your landscapes are very flat, then things become more difficult and you might want to play with different algorithms that are mentioned here to see if the result improves.
14:03
So those were the algorithms but then which tools do you use then for calculating the flow direction? And what I found was that Whitebox tools has the most options for flow direction calculations but also for your different pointer rasters. And you can see here for D8, D infinity, the examples.
14:23
But as I said, if the workflow uses different processing provider tools, so if you want to mix grass with Whitebox, you need to take care of that you reclassify the flow direction to the right pointer grid that is needed, otherwise your answers will be wrong. And then the fill sinks algorithms from Saga and Whitebox tools,
14:42
there is Planchon and Darboo and Wang and Yu available that you can find. PC Ruster uses a very specific one which has a bit more flexibility. You can have five parameters controlling the core area and the core volume. So how much you want to fill the depressions, it can be useful if you don't want to fill
15:00
all the depressions. And with LDD create, you will create the flow direction but if you want to correct the DEM, you need to use LDD create DEM. But normally, you need it just in one step, filling the sinks and then using the flow direction for the next step that comes out of LDD create. And then grass uses the Jensen and Domingue algorithm
15:25
with the rfilldir tool that is available. If you have a river network available, you can burn it into the stream network, also for your other depressions to force the water to follow that, that's an optional step. You can also do this whole procedure by deriving the stream network from the algorithms.
15:45
For burning, there are some tools, there's from whitebox tools, the fillburn tool. For grass, there's rcarve and I have some videos on my YouTube channel if you want to know more about that but generally, these are complicated procedures,
16:00
cause a lot of errors so always be very careful and test well what works in your area. So then we have the flow direction for each pixel. When we connect all these flow directions, we can get the stream link but all these linked pixels are not your rivers. Some pixels are rivers, some are just where the rivers are not even starting but water starts collecting.
16:21
So you need to calibrate that with a reference map. You can use OpenStreetMap or a satellite image and then you have two choices. You can use Straler orders or you can use flow accumulation. Both methods are very similar, they both need calibration. So let's first have a look at Straler order. There are also other ordering tools available from whitebox for example, from PC Raster,
16:42
we use stream order. And Straler starts with giving the smallest stream value one. When two of the same order join, it increases, so it becomes two. If two of two join, it becomes three. But if a smaller order joins a higher order,
17:00
it will just stay with that higher order, et cetera. And then the user need to determine, do I consider every stream, every tributary larger than three as a river? And you calibrate that with a map. Flow accumulation works differently. I find Straler order more intuitive,
17:21
but with flow accumulation, it's a more continuous approach. So we accumulate the water over the network. We assume every cell has one unit and it uses the flow direction to accumulate then the water over the flow direction. And then you will end up with a map like that. Then you need to determine
17:41
what is the minimum accumulation that I consider as a river, and then you can delineate your river from the ruster. We can then define the outflow point of our catchment on the delineated stream. And this can be where it really gets into a larger river or into a lake or the ocean,
18:01
but it can also be where you have your discharge measuring device, your gauging. But you need to snap it to the delineated stream because you're creating a model. So if you put a point really where you see it as an open street map, big chance that it will not work. So these are tools that you can use. With Pitt, you get all the outlets automatically,
18:21
which are just the depressions that it finds, local depressions. There are some snapping tools and white box tools. There's the catchment and sub-catchment tool that you can use with PC Ruster. And there are some scripts that are provided, user-based scripts, and GRASS also has nice tools that you can use.
18:41
You can automate these things because processing provider tools can be connected in a graphical model to automate your process. And this is a tool that even only needs your study area and the point to which you want to route the water and then it will do the rest automatically. But it's still important as a specialist to know what's behind because it will not always work as you want.
19:03
With the PC Ruster tools plugin, you can add more tools and models. There's a repository and you can add it there and find more tools with the QGIS resource sharing plugin. Then there's some nice tools for visualizing the flow direction. This uses Crayfish and we can then have arrows
19:21
plotted on your Ruster which show the flow direction. It uses the mesh styling functionality of QGIS. So there you also see how things nicely integrate and at these kind of events, we cooperate with people who make this and see if we can make it happening. I still have two slides left. One of the newest things in the newest version of QGIS,
19:42
it's possible to animate rusters. So if you have rusters in a stack, each band can be assigned a start and an end time and you can use the temporal controller to animate your results of a runoff model for PC Ruster, for example. It accepts virtual rusters and NetCDF.
20:01
If you want to know more about this, so Kurt and I have already mentioned, written the book QGIS for Hydrological Applications. The first edition uses Saga, the second edition PC Ruster, recommend the second edition. And by purchasing the book, you support students from the global south to join Phosphor-G and QGIS events. Unfortunately, a visa of the one
20:21
I was supporting for this event, still waiting for the visa. And for the QGIS user conference, we have a student from IHDL from Sudan who will join, Razan Elnour. So I hope you'll meet her there. And that was my presentation. Thanks a lot.
20:42
Thanks a lot Hans and Valentine. Other questions? Thank you very much. I am Sultan from Saudi Arabia. The one we saw is a topographic watershed. What about ancient watershed under the sand?
21:05
Did you use other like radar or you use raster radar or other? I'm not specialized in that, unfortunately. And so the algorithms I presented really need gravity. It's about surface runoff. It doesn't take into account groundwater. It will not work in the Netherlands
21:21
where we invert the landscape. We pump up all the water in the rivers that are above us and not in a valley. So for your specific question that needs a study and then we need to find what works best. We had some years ago, Kurt and me, a student or participant of a short course and wanted to apply this to a desert river. But we could clearly see that all the streams there
21:43
were lower than the level of the landscape. So then you would just use map algebra and say everything lower than some value is the river. So it's not always the best solution to go through the whole workflow if you have different conditions. Yeah, for example, we are, because we have a desert and we have full of sun
22:00
and we have raining in the winter. So what we did to observe water shade, we come to when there any season and then there is many moisture content in the water shade, either big or small. And then we just subtract one year with that and then a lot of information comes. But one point is when it is go under the big sand dune.
22:25
So also this is other topic maybe. You need unsaturated zones and groundwater. Thank you very much, thank you. You have another chance.
22:41
Yeah, Seth. So it's just a question about your t-shirt. Is it the catchments of Europe? Because it's very impressive. Yes, catchments of Europe, not delineated by this procedure, but you could do that. But this comes from hydro sheds and you can buy it in my web shop. Okay, great, thanks.
23:02
Nice, okay. Then another round of applause for Hans. Thanks.