Development of QGIS based topographic data management system
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 266 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/66377 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSS4G Prizren Kosovo 2023217 / 266
10
17
23
44
45
46
47
48
49
50
52
53
80
84
85
91
110
116
129
148
164
167
169
173
174
181
182
183
186
187
199
202
204
206
209
215
241
248
265
00:00
DatenverwaltungPhysikalisches SystemSondierungInhalt <Mathematik>ZeitbereichInternationalisierung <Programmierung>Open SourceEinfach zusammenhängender RaumProzess <Informatik>GrenzschichtablösungNichtlinearer OperatorDatenhaltungFlächeninhaltWeb SiteTextur-MappingDigitale PhotographieSpezialrechnerObjekt <Kategorie>Güte der AnpassungDatenhaltungOpen SourceSichtenkonzeptDigitale PhotographieObjekt <Kategorie>Produkt <Mathematik>Klasse <Mathematik>Bildgebendes VerfahrenPhysikalisches SystemFrequenzProzess <Informatik>SchnittmengeMathematikMapping <Computergraphik>Nichtlinearer OperatorDatenverwaltungTotal <Mathematik>MultiplikationFlächeninhaltDatenfeldDomain <Netzwerk>RückkopplungDifferenteSelbstrepräsentationOrthogonalitätXMLComputeranimation
03:16
Mengentheoretische TopologieRelation <Informatik>TypentheorieRechnernetzErhaltungssatzRandwertGebäude <Mathematik>SpeicherabzugEndliche ModelltheorieVerschlingungSondierungSoftwarewartungKonfigurationsdatenbankPhysikalisches SystemProzessautomationStandardabweichungOpen SourceProdukt <Mathematik>VorgehensmodellDatenverwaltungInhalt <Mathematik>ZeitbereichEinfach zusammenhängender RaumDatenmodellVersionsverwaltungFrequenzDatenverwaltungKonfigurationsdatenbankVersionsverwaltungOpen SourceSoftwarewartungFrequenzKonstruktor <Informatik>Endliche ModelltheorieFunktionalUmsetzung <Informatik>DifferenteSoftwareAttributierte GrammatikMathematikAppletFlächeninhaltGebäude <Mathematik>Anpassung <Mathematik>SystemverwaltungKartesische KoordinatenRandwertSchnittmengeErhaltungssatzProdukt <Mathematik>GrenzschichtablösungRelativitätstheorieDatenmodellTextur-MappingGruppenoperationStandardabweichungInformationsqualitätTopologieDienst <Informatik>Physikalisches SystemProzessautomationFront-End <Software>Objekt <Kategorie>UmwandlungsenthalpieComputeranimation
06:36
PhasenumwandlungHyperbelverfahrenTextur-MappingProdukt <Mathematik>IntegralProzess <Informatik>Mapping <Computergraphik>Physikalisches SystemPhasenumwandlungDatenverwaltungRechenschieberIterationSchnittmengeMathematikService providerSoftwaretestFunktion <Mathematik>GrenzschichtablösungDerivation <Algebra>Endliche ModelltheorieDatenhaltungOpen SourceDatenfeldDatenparallelitätDatenmodell
09:16
SpeicherabzugAppletDatenparallelitätDatenverwaltungNichtlinearer OperatorVersionsverwaltungDatenhaltungPhysikalisches SystemKontrollstrukturFlächeninhaltPlug inElement <Gruppentheorie>VollständigkeitSchlussregelImplementierungFunktion <Mathematik>Mathematische LogikWiderspruchsfreiheitFehlermeldungClientElementargeometrieAdditionAttributierte GrammatikRechnernetzMagnetbandlaufwerkVisualisierungATMComputerspielDerivation <Algebra>Gebäude <Mathematik>GeradePolygonMeterAuflösung <Mathematik>DatenmodellKonfiguration <Informatik>Notepad-ComputerSpezialrechnerDatenverwaltungLinked DataFunktionalProzess <Informatik>DatenhaltungVersionsverwaltungSchlussregelFlächeninhaltMathematikPlug inTextur-MappingProdukt <Mathematik>DatenparallelitätElement <Gruppentheorie>AdressraumElementargeometrieSchnittmengeVollständigkeitImplementierungDigitalisierungKomplex <Algebra>Attributierte GrammatikSchreib-Lese-KopfDifferenteMapping <Computergraphik>AppletMAPPhysikalisches SystemFehlermeldungCodeGüte der AnpassungFormation <Mathematik>SichtenkonzeptATMDerivation <Algebra>Endliche ModelltheorieBildgebendes VerfahrenKonfiguration <Informatik>CASE <Informatik>Dienst <Informatik>Gebäude <Mathematik>KonfigurationsraumNichtlinearer OperatorSpeicherabzugGrenzschichtablösungWiderspruchsfreiheitTopologieTypentheorieClientBitVerschlingungComputeranimation
15:57
ZeitdilatationSystemplattformDatenfeldTextur-MappingProdukt <Mathematik>DatenverwaltungInhalt <Mathematik>Endliche ModelltheoriePhysikalisches SystemVersionsverwaltungMapping <Computergraphik>PhasenumwandlungProzess <Informatik>IntegralFrequenzAnalytische FortsetzungComputeranimation
17:30
Inhalt <Mathematik>ZeitbereichInternationalisierung <Programmierung>Open SourceEinfach zusammenhängender RaumCodeGruppoidSpeicherabzugImplementierungDesintegration <Mathematik>Plug inSoftwaretestDokumentenserverSchlussregelAbfrageDienst <Informatik>DatenhaltungObjekt <Kategorie>AppletProgrammbibliothekSoftwarePlug inKontinuierliche IntegrationVerschlingungRechenschieberAnpassung <Mathematik>DatenmodellInformationsqualitätGüte der AnpassungCodeFormation <Mathematik>EntscheidungstheorieOpen SourceResultanteImplementierungSoftwaretestRelativitätstheorieSpeicherabzugIntegralSchlussregelCoxeter-GruppeObjekt <Kategorie>Prozess <Informatik>DatenhaltungSelbstrepräsentationDienst <Informatik>SchedulingTopologieEreignishorizontComputeranimation
20:39
Computeranimation
Transkript: Englisch(automatisch erzeugt)
00:08
Good afternoon everyone. My name is Erohi Atanen and as he said, my topic is development of a Guccias-based topographic data management system.
00:23
That's something we are doing in National Land Survey of Finland. First, I will tell you something about our business domain and goals of the development. Then let's take a look for the requirements and some solutions we
00:50
have provided. In the end I will tell you about our open source contributions. There are two main processes to maintain topographic data.
01:08
Periodical updating process and continuous updating process. There are about 120 operators editing the same data set in these two processes.
01:26
Periodical updating process is based on aerial photography of target area and the changes are detected with stereo mapping.
01:40
Periodical updating process includes also some on-site inspection which we called field mapping and this periodical updating process is to be repeated for the same areas in intervals of three years because our photo campaign covers almost in the entire Finland in three years. Some of the essential
02:06
themes of topographic database are updated in the continuous updating process and in that process we use multiple data sources, get the data from
02:21
the municipal policies, other governmental agencies and we get also some feedback from customers and so on. It's also quite manual process and it's done like mostly annually for different areas, important
02:46
areas. The main way to collect topographic data from aerial images is a stereo mapping. The operator gets accurate representation of objects in 3D stereo view and it allows more precise mapping than just 2D
03:05
ortho photos and those stereo desktops and stereo glasses are needed to do that work. One of the drivers for our development is new
03:22
national data models. The modeling work with the stakeholders have started already since 2015 and the new data models are based on about 10 different data teams including buildings and constructions, terrain,
03:44
hydrography, traffic, networks, special and conservation areas and administrative boundaries. Totally together there are over 100 separate feature layers with different topological relations. Here are some
04:10
expected goals from the development. Product goals enable the production of geospatial data based on new data modeling, transition from
04:24
producing map data to producing spatial data with more attribute data and so on, enable versatile data quality management, enable object specific history and change management, enable diverse data sources, automate processes and adapt
04:45
new technologies and standards and as business goals we expect high quality nation-wide topographic data, more efficient and up-to-date maintenance,
05:00
reduced application of data collection efforts, more compatible data with other registries, promote the use of datasets and enable adoption to changing needs. Next let's take a look for the requirements and
05:27
solutions. In the beginning of the development we had the current system as a reference. It's a small world based custom-made full feature topographic
05:41
data management system with topological data model which is quite different than the new one. There is building version and conflict management and it's improved over a period of more than 20 years.
06:03
And there are a lot of functionalities in it for different workflows and we also had in the beginning the idea of the new technologies we are going to use, mainly both GIS and QGIS and we very early adapted on also some
06:29
Java based backend service. So we started with analyzing the current
06:42
processes with users and process owners and we end up this kind of set of separate requirements to be solved that we can go in production and
07:07
change to use this new system. First we begin with the implementing concurrent data management. I have another slide for that. Efficient editing
07:24
of topographic data is more iterative process and it has needed a lot of iterative development and testing by our users that tools are usable.
07:43
Enabling stereo mapping is essential for us and we have created an integration from our current stereo mapping software to QGIS. We have
08:01
implemented the new national data models and also an ETL process from our current topographic database to the new system. Quality assurance and derivation of feature heights is something we are working on and I have
08:20
slides about those. We are also going to enable the processes of the various products and in the first phase we are solving it that we are going to provide same kind of output than we have from the current
08:44
system. So we will have another ETL process there but it makes it possible that we will renovate our products separate a bit later. And management of cardboard names, efficient use of the data sources is something we will need
09:03
more tools still and currently we are also planning how to implement or integrate the field mapping software to our system and processes.
09:25
Solving concurrent data management was leading to this kind of architect
09:41
solution and it's called the core of the new topographic data management system and it includes version control, history and conflict management. There is a separate Java-based service to manage primary
10:03
database and temporary work databases. And our operators use job manager plugin to control this job manager service and to create temporary work
10:23
databases from certain areas. And then all the QGIS native tools can be used to manage the data in the work database. And after the work is done, user can register the changes to the primary database. And this job manager
10:45
also includes conflict management if there are some conflicts between the changes of different people. Our quality assurance tools are very
11:01
important for us and we wanted to make the process and workflow more fluent for the users. This quality assurance tool focuses on logical consistency of the data. Other quality elements like coverage,
11:25
completeness, up-to-dateness are evaluated in separate processes. And this quality assurance is based on rules. We have three different
11:41
type of rules, geometry, attribute and topological rules. And we aim for that the production of work is guided with the severity level of these errors. And currently we have three level of errors, info,
12:03
warning and fatal. And why we did this custom implementation of the rules with database functions. QGIS provides some tools for quality checking, but custom solution provides comprehensive inspection for
12:24
the data, both in work and primary databases. And we also have only one place to manage all the rules. And we think that we got the best performance with the database. There are hundreds of those rules.
12:48
Also, advantages for the database solution is that it's a client dependent. So, for example, our ETL process from our current
13:04
production system is using these same rules, but without QGIS. For the efficient editing of topographic data, we have done some
13:23
improvements to QGIS tools. We have created a blogging to select the active layer from map. It can be used both from QGIS and from stereo mapping view. Then we have enabled editing mode and saved
13:43
for all layers. And we have created a topological segment reset tool, which is quite a handy tool. And it can also be used both in QGIS and from the stereo mapping view.
14:05
We have also provided possible to visualize all the early changes that the user has done in the same work. We have things to
14:21
do before production or use. For example, there are some custom ways of managing our road link data. Also, there are special needs for street address management. And there need to be some tools for efficient use of different imported data sets.
14:52
About the derivation of feature heights, all the maintained topographic data should be in 2.5D. And we are using
15:02
our national N2000 height system. The management of the jet coordinates of the features are mostly automated. But there are also options to set height manually using stereo mapping, for example. For certain cases, there may be
15:29
difference for the up-to-dateness of our digital terrain model or aerial image, and so on. There are also some specialties of the heights for the shoreline features and
15:47
for buildings, and it makes this implementation a bit more complex. So our current roadmap has a very simplified version.
16:06
So in the phase one, which has started from the beginning of 2021, we are aiming that we will replace the current system as the main tool to do the topographic data
16:25
management. And we are estimating that we could get it in production and use in the early 2025. And it includes these tools for periodical and continuous topographic data production. It also includes field mapping and integration
16:45
into our current product platforms. So new data and products are scheduled in the next phase. And they are needed some production to be done with the new system,
17:02
also to collect some of the new data contents defined in the new models. And also, we have on our roadmap, new management and generalization process for map data which is used as a base of our cartographic products. Okay, few
17:34
slides about open source contributions. There are open source guidelines in a national land survey. They provide
17:44
guidance whether the code should be released or not. And the main principle is that all the code should be open source, if not a specific reason. And these guidelines were published and adapted in 2022. And it has enabled us
18:06
for us the creation and adaptation of some best practices. We have developed some practices for development process for our continuous integration
18:23
pipelines. And also we are working with our cooperation practices. The core is unfortunately not published yet. Our implementation began before we got these
18:42
decisions of OS guidelines. And this, our core implementation has quite tight integration to our own infrastructure and our own data models. So it needs some work to open it. It's not scheduled on the roadmap yet,
19:03
but I hope we will get it on the roadmap sooner or later. But here are some development tools we have contributed to. First, GoodGIS plugin development and packaging tools. And another one is Pytest plugin for
19:22
testing GoodGIS Python plugins. Also we have, we have published or released some GoodGIS plugins, big layer, topological segment receptor tool and plugin for visualising quality check results. We have also
19:44
published database data quality checker, which is a service to execute quality rules for special objects in database and to produce a resulting JSON. It works together with this plugin for visualising quality
20:03
checks. There have been some other related presentations in yesterday, there was Andros and Olis presentations and also Tero had his presentation on Wednesday. There are links to these
20:21
presentations also, and these slides can be found from the event schedule. So thank you very much for the attention.