We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Mesh: GIS data beyond raster and vector

00:00

Formal Metadata

Title
Mesh: GIS data beyond raster and vector
Title of Series
Number of Parts
295
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Most real world features can be presented as vector or raster layers. In open source world, GDAL provides a comprehensive set of tools to interact with such datasets. But vector or raster is not always a suitable description of real world features. Data from oceanography, metrology, hydrology, etc often have multiple components at each location on an irregular structured mesh. A **mesh** can a collection of vertices, edges and faces in 2D or 3D space: - vertices - XY(Z) points (in the layer's coordinate reference system) - edges - connect pairs of vertices - faces - sets of edges forming a closed shape - typically triangles or quadrilaterals (quads), rarely polygons with higher number of vertices ![Example of mesh](https://user-images.githubusercontent.com/193367/38030812-f6c4f174-3299-11e8-91ed-30684ceae715.png) Mesh gives us information about the spatial structure. In addition to the mesh we have **datasets** that assign a value to every vertex. For example, ice cap thickness at particular moment of time. A single file may contain multiple datasets - typically multiple quantities (e.g. water depth, water flow) that may be varying in time.
Keywords
129
131
137
139
Thumbnail
28:17
Raster graphicsInfinityArray data structureInformation technology consultingMultiplication signPolygon meshPresentation of a groupCartesian coordinate systemShared memoryLecture/ConferenceMeeting/Interview
Information technology consultingArray data structureRaster graphicsPolygon meshDenial-of-service attackOpen setRepresentation (politics)Endliche ModelltheoriePolygon meshBitJSONXMLUML
Core dumpInformation technology consultingLibrary (computing)Software maintenanceSoftware developeroutputAndroid (robot)Data structurePolygon meshAbstractionArray data structureRaster graphicsPoint cloudSoftware developerPlanningoutputMeasurementLibrary (computing)NeuroinformatikDenial-of-service attackMultiplication signPolygon meshFunction (mathematics)Data structurePoint (geometry)Different (Kate Ryan album)CASE <Informatik>Projective planeCartesian coordinate systemAbstractionAndroid (robot)Core dumpArray data structureComputer animation
Array data structurePolygon meshRaster graphicsProcess modelingPresentation of a groupSource codePoint (geometry)Endliche ModelltheoriePoint cloudCellular automatonDifferent (Kate Ryan album)Similarity (geometry)Denial-of-service attackPolygon meshMeasurementXMLUML
Attribute grammarInformation technology consultingMusical ensemblePixelScalar fieldQuadrilateralTriangleRegular graphVertex (graph theory)Cellular automatonArray data structureArrow of timeData structureArithmetic meanArrow of timePolygonQuicksortAreaMultiplication signDifferent (Kate Ryan album)Order (biology)Musical ensembleMeasurementCellular automatonECosShape (magazine)Library (computing)BitDomain namePhysical systemAttribute grammarRaster graphicsScalar fieldNetwork topologyArray data structureQuadrilateralPolygon meshComputer animation
Raster graphicsArray data structureTopologyOpen setPoint (geometry)Series (mathematics)Unstrukturiertes GitterStandard deviationMetadataBitDenial-of-service attackEndliche ModelltheorieExtension (kinesiology)File formatLibrary (computing)Standard deviationSoftwarePoint (geometry)Cellular automatonArray data structureData structureNetwork topologyFocus (optics)Semiconductor memoryFlock (web browser)Multiplication signComputer animation
TopologyFrame problemPolygon meshType theoryMixed realityElement (mathematics)Information technology consultingRegular graphQuadrilateralBitPolygon meshPolygonSoftwareShape (magazine)Point (geometry)Cellular automatonFunction (mathematics)Link (knot theory)AreaReal numberNeuroinformatikDenial-of-service attackJSONXMLUMLComputer animationDiagram
Frame problemTriangleNumerical analysisFunction (mathematics)Point (geometry)Computer animation
Polygon meshUnstrukturiertes GitterFrame problemDenial-of-service attackPolygon meshLevel (video gaming)Point (geometry)PlanningSanitary sewerRight angleSubject indexingReverse engineeringReading (process)Video gameVirtual machineGreedy algorithmMeasurementArithmetic meanCollisionMetropolitan area networkComputer animation
Information technology consultingWater vaporVector graphicsScalar fieldActive contour modelArray data structureVertex (graph theory)Multiplication signArrow of timeSpeech synthesisSign (mathematics)Point (geometry)Object (grammar)JSONXMLComputer animation
EmulationData typeValue-added networkDiscrete element methodArray data structureMetropolitan area networkLocal Group1 (number)Lipschitz-StetigkeitFluxDuality (mathematics)Array data structurePolygon meshRaster graphicsMultiplication signWebsiteArrow of timeDifferent (Kate Ryan album)ResultantGreen's functionSource code
Polygon meshAbstractionLibrary (computing)Spherical capInformation technology consultingRepresentation (politics)Process modelingFile formatGeneric programmingUnstrukturiertes GitterRemote Access ServiceVariable (mathematics)Endliche ModelltheorieCellular automatonStructural loadInterpolationArtistic renderingArray data structureSystem identificationPhysical systemLibrary (computing)DialectProjective planeDenial-of-service attackPlug-in (computing)Maxima and minimaCalculationNumeral (linguistics)BenchmarkDifferent (Kate Ryan album)Group actionEndliche ModelltheorieTriangleCellular automatonPlotteroutputWrapper (data mining)File formatInterpolationRaster graphicsMultiplication signPreprocessorExtension (kinesiology)AlgorithmBlogFormal languageFlow separationTerm (mathematics)Artistic renderingFunction (mathematics)Server (computing)AreaJSONComputer animation
Information technology consultingTouchscreenCASE <Informatik>PlotterOpen setComputer animation
Open setRaster graphicsArray data structure
Information technology consultingLengthClique-widthOrder of magnitudeDevice driverPlug-in (computing)Scalar fieldPolygon meshMessage passing
Information technology consultingEndliche ModelltheorieDenial-of-service attackRaster graphicsComputer animation
Process modelingDenial-of-service attackData modelLink (knot theory)Denial-of-service attackDistanceBitTriangleSoftware developerPolygon meshDifferent (Kate Ryan album)JSONXML
Similarity (geometry)Sampling (music)Image resolutionCellular automatonPolygon meshArray data structureActive contour modelSoftware developerBitClosed setPolygon meshLevel (video gaming)Inheritance (object-oriented programming)MeasurementCalculationProjective planeMetropolitan area networkSimilarity (geometry)NumberDomain nameCellular automatonSet (mathematics)Source codeCuboidTriangleActive contour modelRaster graphicsArtistic renderingZoom lensSlide ruleJSONXMLUMLComputer animation
Polygon meshError messageInformation technology consultingPolygon meshFile formatSoftware developerMultiplication signError messageTwitterComputer fileDomain nameEmailPhysical lawXMLComputer animation
Multiplication signWeightConnectivity (graph theory)Priority queueTraffic reportingLibrary (computing)Dimensional analysisBuildingConnected spaceTrailArtistic renderingDevice driverError messageForm (programming)Cellular automatonStaff (military)Point (geometry)Polygon meshLocal ringProcess (computing)Data structureForcing (mathematics)ImplementationSemiconductor memoryDomain nameINTEGRALStandard deviationPlotterRevision controlLatent heatGeometryFile formatDirection (geometry)Server (computing)Right angleNetwork topologyAdaptive behaviorCore dumpExtension (kinesiology)Functional (mathematics)Data conversionPlug-in (computing)Front and back endsOcean currentLecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
All right, so it's time for the final presentation in this session. So with us is Peter Patrick
from Lutela Consulting and he's going to talk about meshes and their efforts to make full citizen next to raster and vector data in in geospatial applications. So welcome to my presentation, thank you for coming. I will try to explain you about
this new stuff that's coming to QGIS and it's already there, you can use it. In this conference we talked and heard a lot about open data, Copernicus data, weather data, flood modeling and until recently it was kind of difficult to
display them in QGIS but right now the mesh data, weather data, flood data are first-class citizen and I will tell you more about it. A bit about me, I'm Lutela Consulting and I'm QGIS Core Committer and I'm mainly responsible for
this new mesh layer and support for weather and flood data in QGIS and I also did a macOS packaging project and some of the QGIS 3D and some application development for iOS, Android for with QGIS rendering engine.
Okay, what we will talk about, we will talk some differences between raster, vector and mesh and point clouds, then the structure of the data, then about the new library MDAL that's kind of input-output abstraction library for
your data and then what's implemented, some existing examples and then what are the development plans and conference plans for your next release of QGIS. Okay, so let's start. So first, this is not about point clouds. There is a lot of confusion in terminology. This is not about point
clouds, this is more about weather data and flood modelling. The difference is that pretty much we expect less data, a lot of less data. We expect that you have similar mesh than in point clouds but your mesh is small enough
that can be loaded in the memory but on the contrary, on every cell or every point of your data you have many many points. So for example you have temperature every few hours for I don't know a week or month but you don't have so much cells, so that's the difference. So let's talk a bit about
what's the difference between this mesh layer and the raster and the vector layers. Everyone knows a vector layer, you have a few features, you have attributes. For raster layer, it's long enough here, so you can imagine
pixels, you have some bands there with some data and bands, some metadata, you have some scalar values assigned to these pixels or cells. But on the contrary, the mesh is irregular cell or grid. You can have different
cells, different shape, different size all over your area, so this is not regular. You can have various types, you can have triangles, quads or whatever polygons. You can have assigned the scalar data similar to the raster but also you can have vector data and there's another confusion in
terminology. I don't mean like vector data like vector layer but I mean the arrows like X, Y and wind speed and so on. And you have a topology here, so similar to the vector layers, you have some cells that are somehow
connected in some domains. How it comes to the existing ecosystem of the libraries we have? So ogr, you have four vector layers in QGIS mainly or
QGIS. GDAL you have four raster layers, so we created a new library, it's MIT based, that is responsible similar to GDAL or ogr to handle your data to whatever solver you have, numerical, flood, weather, and get it to the same structure that is
easily recognized by QGIS or other software that will use the MDAL library. And you can then, similar to GDAL, work with API, see API, and load your data regardless of the format. For meshes, there are also a few data
standards. You have the climate and forecast metadata conventions, widely adapted for weather data mainly. And there is an extension for flood modeling mostly called U-Grid, unstructured grid, that adds a topology to this climate
and forecast metadata convention, which means that for points, you can connect the points to some faces, to some cells. So the MDAL library already supports both conventions almost fully, and we will try our best to
improve, like follow these standards and improve support. Let's talk a bit about how the data are stored. So, as I mentioned, you have a mesh that
consists of points, vertices, that are topologically connected to cells. It could be any shape, pretty much. It could be triangular, it could be mixed with any polygons, or it could be a regular grid. You can have adaptive
meshes, which means that, for example, for flood modeling, you want to have a more fine mesh near your river or area of interest, but you don't want that computation cells far from your point of interests. And here is a
real-world example from QGIS, with the output of some numerical software for the adaptive mesh and styling. So you can style the mesh and see your
data. So this is for QGIS 3.4. Okay, what's upcoming in the new QGIS releases this year, like 3.12 mainly? We would like to support, so right now
you have the standard 2D meshes, but we would like to support 1D meshes. So, for example, when you want to sewer system or just reverse as a 1D, and then this 3D layered mesh. So, for example, you have, again, river and you want to have different levels. The U-Grid convention supports this, and
the fully 3D meshes, that's not in the plan yet, but hopefully we can get to that in a later point. So, for MDAL and QGIS, for the 3.12 release, we already have a plan to implement those 1D and 3D layered
meshes. Layered mesh means that you have a 2D mesh, but you have just copy-pasted on different heights, pretty much. For now, there is some
prototyping done for meshes in QGIS 3D. So, should I run? Okay, so right now
you can visualize your topology, mesh topology, in 3D, QGIS 3D. What's coming, hopefully, next year, next calendar year, is that you will be able to visualize also your data, so you can make those nice animations of floods or all other weather data you have in 3D. Right now you can just visualize
the mesh itself. Okay, how the data are stored? So, you pretty much index your vertices or faces, and then for each vertex or face you assign for
various times some data, either scalar data or vector data like wind speed. This data can be then displayed interpolated on your grid or as arrows
if it is a vector data. So, here is again the example from QGIS 3.4, where you see some metallurgical data and displaying wind speed as arrows and temperature. You see that there is a time slider, so you can do some
visualization or see the data in different times. This is pretty much without any plug-in, so you can directly, similar to the raster layer or vector layer, you can download data from, for example, Copernicus' website
and open it as a mesh layer directly in QGIS. Here is MDAL. MDAL was started as an MIT library, so it can be used in different projects too,
not just in QGIS. It's really the wrapper around your data, so it doesn't have any algorithms or pre-processing of the data. It's C++ without any much dependencies like QT or Boost. It has C API, so if you want to do some
extension for different languages, contributions are welcome. So far it's shipped with QGIS, but we want to package it separately for future releases. We are also discussing the opportunity to move it on the OSGEO
incubator project, so hopefully you can make this happen in upcoming months too. The library itself is based on the Crayfish implementation, which came out in 2012, so it's not a new library or new code, but it was separated with its transition to QGIS 3 in 2018. It supports various formats, 2DM, NetCDF,
GRIP format, and then a lot of formats, various numerical packages. You can find your package if you are in the flood modeling or weather.
As I mentioned, we expect or we benchmark the library typically to 10 gigabyte models maximum to a few millions of cells and a few hundred of time cells for each. Various formats support lazy loading, so the loading in QGIS is very fast and data is taken on demand. As I mentioned, MDAL is for data input and
output. QGIS is responsible for data interpolation rendering, so all the data from MDAL is triangulated and then the triangles are rendered.
There is an identify tool styling calculator similar to the raster calculator, and there is still a Crayfish plugin in Python where you can do some plots or export animations, and it's most incubator for upcoming features in QGIS.
Here is a screenshot from QGIS and Crayfish for some plots on these lines. Okay, some examples. Okay, here is a wind speed from Open Data again,
Copernicus, and this trace animation is a feature that's coming in next QGIS
release 3.12. The data can be styled similarly to vector layers or raster layers. Also, now if you have some custom format, you can write a driver in
MDAL, or if you want to write some Python plugin, all this is in a QGIS public API, so you can write your own plugin and do various stuff with the mesh data. Another example with some scalar data for pollution, so you can
produce these animations with Crayfish, and again you can style it as you are
used for raster data, so nothing new there in styling. If you are QGIS user, it should be quite straightforward for you to just directly grab the data and use it. Some examples from QGIS from flood modeling. Again, we see adaptive
mesh with different sizes of triangles here, depends on the distance from a river, and then some animations produced by QGIS. Okay, let me
talk a bit about upcoming development that is confirmed. We want to do this trace animation, so on the canvas you'll be able to select the box and the
canvas will be refreshed in this way. We want to extend the mesh calculator for cell-centered datasets. We have a project for highly improving speed rendering of your mesh data by a similar technique than you have for
raster overviews or pyramids, because right now the triangular mesh that is generated is not based on your zoom level, but with this the number of triangles will be dependent on your zoom level, so it will be way faster than it is now.
There are some small enhancements like export data to contours and a huge one support for 1D and 3D meshes as I described in the previous slides. For MDOW, it goes hand by hand with QGIS support for 1D meshes. We'll also
add a support for meshes that contains more than one domain, so you can have multiple domains in one file, so you will be able to load them. Support for
3D meshes, and we want to work on various formats for adding more lazy loading for various formats and improve the error handling, so it will show you why your file is not loaded and if there is a time we want to work on the packaging,
so you can use it also outside the QGIS and you don't need to compile it yourself. Okay, so that's all I have prepared. Feel free to write me an email if you have some questions about some formats you want to support in MDOW or fetch it on GitHub.
It's all on GitHub. Feel free to look at it or sign to our Twitter account for news about upcoming development here. Thank you very much for your attention.
Any questions? Hi, thanks for the talk. It's a short question actually. Is it possible to store matched data formats inside a geo package?
Yeah, I think it's possible, but then it must follow some specifications so we can implement it as a new driver in MDOW. The structure of the drivers in MDOW is similar to the GDOW, so you have some basic structure where you want to fill the data, and if you have, for example,
these U-Grid conventions, they are all standard so you can know what's where, how you can get a topology of my data, but if you have your own format and it's spread enough, then it could be added to the drivers as a new driver and could be loaded directly.
Right now, as I mentioned, the most popular format is HDF5 because usually the mesh is small but you have many time steps for each cell, which means that you want to use some kind of
HDF5, so that's the most popular format I think in this area, but technically it's possible. Any other questions? As I understand, the functionality is available just via Crayfish plugin,
so it is not in QGIS Core? No, you can use everything without the Crayfish plugin. All you've seen is just pure QGIS, so you install QGIS 3.6 or 3.8, 3.4, you can just load your climatic weather data and
style it, show it without any plugin. Crayfish is just a Python extension to create animation or see the plots or some processing integration, but everything is in all the
rendering, all the styling, everything is in QGIS Core or MDAL, if you are talking about that. There's kind of another question to my mind. What's the adaptation or connection to QGIS Server
because there are those trail animations, for example, or a time dimension? Right now you should be able to probably just see your data in the QGIS Server natively because it's just a new layer, so it's there. Including those animations?
But for animations you probably need to integrate it in some place, that's not there, but you can render the data because it's using the QGIS Core renderer, so it should be there, but the time support is not there.
Imagine if you want to publish your mesh data, a grip or netCDF, if you were to do it with GeoServer you need to go through GDAL, but with this one you can directly use QGIS Server to render it and directly render the native mesh without the conversion. And then if you want to animate you need to have some front-end JavaScript library to be able to
fetch the WMST component to be able to browse through time.
Any other questions? We still have time for maybe one question. Maybe I have one. You mentioned at the beginning that these meshes, they are usually not that large themselves,
but they may have many time steps. So you make the assumption that the mesh definition itself always fits into memory? It's not hard coded in the library. The API has written a way that it allows you to fetch the data by some time spatial domain,
but all current drivers implementation is that it fits to the memory, but drivers could be rewritten, but the data is lazy-loaded mostly. Okay, sounds good. Thank you very much.