Mesh: GIS data beyond raster and vector
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43361 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Raster graphicsInfinityArray data structureInformation technology consultingMultiplication signPolygon meshPresentation of a groupCartesian coordinate systemShared memoryLecture/ConferenceMeeting/Interview
00:35
Information technology consultingArray data structureRaster graphicsPolygon meshDenial-of-service attackOpen setRepresentation (politics)Endliche ModelltheoriePolygon meshBitJSONXMLUML
01:16
Core dumpInformation technology consultingLibrary (computing)Software maintenanceSoftware developeroutputAndroid (robot)Data structurePolygon meshAbstractionArray data structureRaster graphicsPoint cloudSoftware developerPlanningoutputMeasurementLibrary (computing)NeuroinformatikDenial-of-service attackMultiplication signPolygon meshFunction (mathematics)Data structurePoint (geometry)Different (Kate Ryan album)CASE <Informatik>Projective planeCartesian coordinate systemAbstractionAndroid (robot)Core dumpArray data structureComputer animation
02:13
Array data structurePolygon meshRaster graphicsProcess modelingPresentation of a groupSource codePoint (geometry)Endliche ModelltheoriePoint cloudCellular automatonDifferent (Kate Ryan album)Similarity (geometry)Denial-of-service attackPolygon meshMeasurementXMLUML
02:57
Attribute grammarInformation technology consultingMusical ensemblePixelScalar fieldQuadrilateralTriangleRegular graphVertex (graph theory)Cellular automatonArray data structureArrow of timeData structureArithmetic meanArrow of timePolygonQuicksortAreaMultiplication signDifferent (Kate Ryan album)Order (biology)Musical ensembleMeasurementCellular automatonECosShape (magazine)Library (computing)BitDomain namePhysical systemAttribute grammarRaster graphicsScalar fieldNetwork topologyArray data structureQuadrilateralPolygon meshComputer animation
04:29
Raster graphicsArray data structureTopologyOpen setPoint (geometry)Series (mathematics)Unstrukturiertes GitterStandard deviationMetadataBitDenial-of-service attackEndliche ModelltheorieExtension (kinesiology)File formatLibrary (computing)Standard deviationSoftwarePoint (geometry)Cellular automatonArray data structureData structureNetwork topologyFocus (optics)Semiconductor memoryFlock (web browser)Multiplication signComputer animation
06:14
TopologyFrame problemPolygon meshType theoryMixed realityElement (mathematics)Information technology consultingRegular graphQuadrilateralBitPolygon meshPolygonSoftwareShape (magazine)Point (geometry)Cellular automatonFunction (mathematics)Link (knot theory)AreaReal numberNeuroinformatikDenial-of-service attackJSONXMLUMLComputer animationDiagram
07:03
Frame problemTriangleNumerical analysisFunction (mathematics)Point (geometry)Computer animation
07:29
Polygon meshUnstrukturiertes GitterFrame problemDenial-of-service attackPolygon meshLevel (video gaming)Point (geometry)PlanningSanitary sewerRight angleSubject indexingReverse engineeringReading (process)Video gameVirtual machineGreedy algorithmMeasurementArithmetic meanCollisionMetropolitan area networkComputer animation
09:29
Information technology consultingWater vaporVector graphicsScalar fieldActive contour modelArray data structureVertex (graph theory)Multiplication signArrow of timeSpeech synthesisSign (mathematics)Point (geometry)Object (grammar)JSONXMLComputer animation
10:06
EmulationData typeValue-added networkDiscrete element methodArray data structureMetropolitan area networkLocal Group1 (number)Lipschitz-StetigkeitFluxDuality (mathematics)Array data structurePolygon meshRaster graphicsMultiplication signWebsiteArrow of timeDifferent (Kate Ryan album)ResultantGreen's functionSource code
10:49
Polygon meshAbstractionLibrary (computing)Spherical capInformation technology consultingRepresentation (politics)Process modelingFile formatGeneric programmingUnstrukturiertes GitterRemote Access ServiceVariable (mathematics)Endliche ModelltheorieCellular automatonStructural loadInterpolationArtistic renderingArray data structureSystem identificationPhysical systemLibrary (computing)DialectProjective planeDenial-of-service attackPlug-in (computing)Maxima and minimaCalculationNumeral (linguistics)BenchmarkDifferent (Kate Ryan album)Group actionEndliche ModelltheorieTriangleCellular automatonPlotteroutputWrapper (data mining)File formatInterpolationRaster graphicsMultiplication signPreprocessorExtension (kinesiology)AlgorithmBlogFormal languageFlow separationTerm (mathematics)Artistic renderingFunction (mathematics)Server (computing)AreaJSONComputer animation
13:22
Information technology consultingTouchscreenCASE <Informatik>PlotterOpen setComputer animation
13:42
Open setRaster graphicsArray data structure
14:09
Information technology consultingLengthClique-widthOrder of magnitudeDevice driverPlug-in (computing)Scalar fieldPolygon meshMessage passing
14:41
Information technology consultingEndliche ModelltheorieDenial-of-service attackRaster graphicsComputer animation
15:18
Process modelingDenial-of-service attackData modelLink (knot theory)Denial-of-service attackDistanceBitTriangleSoftware developerPolygon meshDifferent (Kate Ryan album)JSONXML
15:47
Similarity (geometry)Sampling (music)Image resolutionCellular automatonPolygon meshArray data structureActive contour modelSoftware developerBitClosed setPolygon meshLevel (video gaming)Inheritance (object-oriented programming)MeasurementCalculationProjective planeMetropolitan area networkSimilarity (geometry)NumberDomain nameCellular automatonSet (mathematics)Source codeCuboidTriangleActive contour modelRaster graphicsArtistic renderingZoom lensSlide ruleJSONXMLUMLComputer animation
16:57
Polygon meshError messageInformation technology consultingPolygon meshFile formatSoftware developerMultiplication signError messageTwitterComputer fileDomain nameEmailPhysical lawXMLComputer animation
18:17
Multiplication signWeightConnectivity (graph theory)Priority queueTraffic reportingLibrary (computing)Dimensional analysisBuildingConnected spaceTrailArtistic renderingDevice driverError messageForm (programming)Cellular automatonStaff (military)Point (geometry)Polygon meshLocal ringProcess (computing)Data structureForcing (mathematics)ImplementationSemiconductor memoryDomain nameINTEGRALStandard deviationPlotterRevision controlLatent heatGeometryFile formatDirection (geometry)Server (computing)Right angleNetwork topologyAdaptive behaviorCore dumpExtension (kinesiology)Functional (mathematics)Data conversionPlug-in (computing)Front and back endsOcean currentLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:07
All right, so it's time for the final presentation in this session. So with us is Peter Patrick
00:20
from Lutela Consulting and he's going to talk about meshes and their efforts to make full citizen next to raster and vector data in in geospatial applications. So welcome to my presentation, thank you for coming. I will try to explain you about
00:46
this new stuff that's coming to QGIS and it's already there, you can use it. In this conference we talked and heard a lot about open data, Copernicus data, weather data, flood modeling and until recently it was kind of difficult to
01:05
display them in QGIS but right now the mesh data, weather data, flood data are first-class citizen and I will tell you more about it. A bit about me, I'm Lutela Consulting and I'm QGIS Core Committer and I'm mainly responsible for
01:26
this new mesh layer and support for weather and flood data in QGIS and I also did a macOS packaging project and some of the QGIS 3D and some application development for iOS, Android for with QGIS rendering engine.
01:47
Okay, what we will talk about, we will talk some differences between raster, vector and mesh and point clouds, then the structure of the data, then about the new library MDAL that's kind of input-output abstraction library for
02:02
your data and then what's implemented, some existing examples and then what are the development plans and conference plans for your next release of QGIS. Okay, so let's start. So first, this is not about point clouds. There is a lot of confusion in terminology. This is not about point
02:24
clouds, this is more about weather data and flood modelling. The difference is that pretty much we expect less data, a lot of less data. We expect that you have similar mesh than in point clouds but your mesh is small enough
02:40
that can be loaded in the memory but on the contrary, on every cell or every point of your data you have many many points. So for example you have temperature every few hours for I don't know a week or month but you don't have so much cells, so that's the difference. So let's talk a bit about
03:01
what's the difference between this mesh layer and the raster and the vector layers. Everyone knows a vector layer, you have a few features, you have attributes. For raster layer, it's long enough here, so you can imagine
03:21
pixels, you have some bands there with some data and bands, some metadata, you have some scalar values assigned to these pixels or cells. But on the contrary, the mesh is irregular cell or grid. You can have different
03:41
cells, different shape, different size all over your area, so this is not regular. You can have various types, you can have triangles, quads or whatever polygons. You can have assigned the scalar data similar to the raster but also you can have vector data and there's another confusion in
04:06
terminology. I don't mean like vector data like vector layer but I mean the arrows like X, Y and wind speed and so on. And you have a topology here, so similar to the vector layers, you have some cells that are somehow
04:23
connected in some domains. How it comes to the existing ecosystem of the libraries we have? So ogr, you have four vector layers in QGIS mainly or
04:43
QGIS. GDAL you have four raster layers, so we created a new library, it's MIT based, that is responsible similar to GDAL or ogr to handle your data to whatever solver you have, numerical, flood, weather, and get it to the same structure that is
05:04
easily recognized by QGIS or other software that will use the MDAL library. And you can then, similar to GDAL, work with API, see API, and load your data regardless of the format. For meshes, there are also a few data
05:24
standards. You have the climate and forecast metadata conventions, widely adapted for weather data mainly. And there is an extension for flood modeling mostly called U-Grid, unstructured grid, that adds a topology to this climate
05:44
and forecast metadata convention, which means that for points, you can connect the points to some faces, to some cells. So the MDAL library already supports both conventions almost fully, and we will try our best to
06:05
improve, like follow these standards and improve support. Let's talk a bit about how the data are stored. So, as I mentioned, you have a mesh that
06:27
consists of points, vertices, that are topologically connected to cells. It could be any shape, pretty much. It could be triangular, it could be mixed with any polygons, or it could be a regular grid. You can have adaptive
06:46
meshes, which means that, for example, for flood modeling, you want to have a more fine mesh near your river or area of interest, but you don't want that computation cells far from your point of interests. And here is a
07:05
real-world example from QGIS, with the output of some numerical software for the adaptive mesh and styling. So you can style the mesh and see your
07:22
data. So this is for QGIS 3.4. Okay, what's upcoming in the new QGIS releases this year, like 3.12 mainly? We would like to support, so right now
07:42
you have the standard 2D meshes, but we would like to support 1D meshes. So, for example, when you want to sewer system or just reverse as a 1D, and then this 3D layered mesh. So, for example, you have, again, river and you want to have different levels. The U-Grid convention supports this, and
08:06
the fully 3D meshes, that's not in the plan yet, but hopefully we can get to that in a later point. So, for MDAL and QGIS, for the 3.12 release, we already have a plan to implement those 1D and 3D layered
08:23
meshes. Layered mesh means that you have a 2D mesh, but you have just copy-pasted on different heights, pretty much. For now, there is some
08:40
prototyping done for meshes in QGIS 3D. So, should I run? Okay, so right now
09:02
you can visualize your topology, mesh topology, in 3D, QGIS 3D. What's coming, hopefully, next year, next calendar year, is that you will be able to visualize also your data, so you can make those nice animations of floods or all other weather data you have in 3D. Right now you can just visualize
09:25
the mesh itself. Okay, how the data are stored? So, you pretty much index your vertices or faces, and then for each vertex or face you assign for
09:44
various times some data, either scalar data or vector data like wind speed. This data can be then displayed interpolated on your grid or as arrows
10:02
if it is a vector data. So, here is again the example from QGIS 3.4, where you see some metallurgical data and displaying wind speed as arrows and temperature. You see that there is a time slider, so you can do some
10:28
visualization or see the data in different times. This is pretty much without any plug-in, so you can directly, similar to the raster layer or vector layer, you can download data from, for example, Copernicus' website
10:45
and open it as a mesh layer directly in QGIS. Here is MDAL. MDAL was started as an MIT library, so it can be used in different projects too,
11:03
not just in QGIS. It's really the wrapper around your data, so it doesn't have any algorithms or pre-processing of the data. It's C++ without any much dependencies like QT or Boost. It has C API, so if you want to do some
11:23
extension for different languages, contributions are welcome. So far it's shipped with QGIS, but we want to package it separately for future releases. We are also discussing the opportunity to move it on the OSGEO
11:41
incubator project, so hopefully you can make this happen in upcoming months too. The library itself is based on the Crayfish implementation, which came out in 2012, so it's not a new library or new code, but it was separated with its transition to QGIS 3 in 2018. It supports various formats, 2DM, NetCDF,
12:08
GRIP format, and then a lot of formats, various numerical packages. You can find your package if you are in the flood modeling or weather.
12:22
As I mentioned, we expect or we benchmark the library typically to 10 gigabyte models maximum to a few millions of cells and a few hundred of time cells for each. Various formats support lazy loading, so the loading in QGIS is very fast and data is taken on demand. As I mentioned, MDAL is for data input and
12:49
output. QGIS is responsible for data interpolation rendering, so all the data from MDAL is triangulated and then the triangles are rendered.
13:03
There is an identify tool styling calculator similar to the raster calculator, and there is still a Crayfish plugin in Python where you can do some plots or export animations, and it's most incubator for upcoming features in QGIS.
13:23
Here is a screenshot from QGIS and Crayfish for some plots on these lines. Okay, some examples. Okay, here is a wind speed from Open Data again,
13:52
Copernicus, and this trace animation is a feature that's coming in next QGIS
14:03
release 3.12. The data can be styled similarly to vector layers or raster layers. Also, now if you have some custom format, you can write a driver in
14:23
MDAL, or if you want to write some Python plugin, all this is in a QGIS public API, so you can write your own plugin and do various stuff with the mesh data. Another example with some scalar data for pollution, so you can
14:49
produce these animations with Crayfish, and again you can style it as you are
15:04
used for raster data, so nothing new there in styling. If you are QGIS user, it should be quite straightforward for you to just directly grab the data and use it. Some examples from QGIS from flood modeling. Again, we see adaptive
15:27
mesh with different sizes of triangles here, depends on the distance from a river, and then some animations produced by QGIS. Okay, let me
15:49
talk a bit about upcoming development that is confirmed. We want to do this trace animation, so on the canvas you'll be able to select the box and the
16:04
canvas will be refreshed in this way. We want to extend the mesh calculator for cell-centered datasets. We have a project for highly improving speed rendering of your mesh data by a similar technique than you have for
16:23
raster overviews or pyramids, because right now the triangular mesh that is generated is not based on your zoom level, but with this the number of triangles will be dependent on your zoom level, so it will be way faster than it is now.
16:42
There are some small enhancements like export data to contours and a huge one support for 1D and 3D meshes as I described in the previous slides. For MDOW, it goes hand by hand with QGIS support for 1D meshes. We'll also
17:08
add a support for meshes that contains more than one domain, so you can have multiple domains in one file, so you will be able to load them. Support for
17:22
3D meshes, and we want to work on various formats for adding more lazy loading for various formats and improve the error handling, so it will show you why your file is not loaded and if there is a time we want to work on the packaging,
17:42
so you can use it also outside the QGIS and you don't need to compile it yourself. Okay, so that's all I have prepared. Feel free to write me an email if you have some questions about some formats you want to support in MDOW or fetch it on GitHub.
18:06
It's all on GitHub. Feel free to look at it or sign to our Twitter account for news about upcoming development here. Thank you very much for your attention.
18:26
Any questions? Hi, thanks for the talk. It's a short question actually. Is it possible to store matched data formats inside a geo package?
18:41
Yeah, I think it's possible, but then it must follow some specifications so we can implement it as a new driver in MDOW. The structure of the drivers in MDOW is similar to the GDOW, so you have some basic structure where you want to fill the data, and if you have, for example,
19:02
these U-Grid conventions, they are all standard so you can know what's where, how you can get a topology of my data, but if you have your own format and it's spread enough, then it could be added to the drivers as a new driver and could be loaded directly.
19:26
Right now, as I mentioned, the most popular format is HDF5 because usually the mesh is small but you have many time steps for each cell, which means that you want to use some kind of
19:43
HDF5, so that's the most popular format I think in this area, but technically it's possible. Any other questions? As I understand, the functionality is available just via Crayfish plugin,
20:05
so it is not in QGIS Core? No, you can use everything without the Crayfish plugin. All you've seen is just pure QGIS, so you install QGIS 3.6 or 3.8, 3.4, you can just load your climatic weather data and
20:26
style it, show it without any plugin. Crayfish is just a Python extension to create animation or see the plots or some processing integration, but everything is in all the
20:46
rendering, all the styling, everything is in QGIS Core or MDAL, if you are talking about that. There's kind of another question to my mind. What's the adaptation or connection to QGIS Server
21:00
because there are those trail animations, for example, or a time dimension? Right now you should be able to probably just see your data in the QGIS Server natively because it's just a new layer, so it's there. Including those animations?
21:21
But for animations you probably need to integrate it in some place, that's not there, but you can render the data because it's using the QGIS Core renderer, so it should be there, but the time support is not there.
21:42
Imagine if you want to publish your mesh data, a grip or netCDF, if you were to do it with GeoServer you need to go through GDAL, but with this one you can directly use QGIS Server to render it and directly render the native mesh without the conversion. And then if you want to animate you need to have some front-end JavaScript library to be able to
22:07
fetch the WMST component to be able to browse through time.
22:24
Any other questions? We still have time for maybe one question. Maybe I have one. You mentioned at the beginning that these meshes, they are usually not that large themselves,
22:41
but they may have many time steps. So you make the assumption that the mesh definition itself always fits into memory? It's not hard coded in the library. The API has written a way that it allows you to fetch the data by some time spatial domain,
23:03
but all current drivers implementation is that it fits to the memory, but drivers could be rewritten, but the data is lazy-loaded mostly. Okay, sounds good. Thank you very much.