We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Modernizing the National River and Lakes Cadastre by Transition to FOSS4G

00:00

Formal Metadata

Title
Modernizing the National River and Lakes Cadastre by Transition to FOSS4G
Title of Series
Number of Parts
156
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
This presentation is intended to introduce a project completed at the end of 2023, during which the Lithuanian National River and Lake Cadastre has been modernized by transferring GIS (and not only) solutions from commercial software to open source and by extending automated GIS data processing solutions. During the presentation, we will share not only the technological solutions we have adopted, but also our experience in changing the attitude of GIS specialists with experience of working with commercial GIS software towards open source. The main technological components of the project included the development of a data management system using PostGIS and QGIS, the development of a map browser using openlayers and Vue JS (available as an open source project - https://github.com/AplinkosMinisterija/biip-maps-web), and the development of a service publishing solution based on QGIS Server. The project used Docker technology and GitHub action-based continuous deployment (CD) solutions, which should also be relevant to the audience. Many of us know that building a system from the ground up is often much easier than upgrading an existing system that has been in place for a long time but has not been updated. This was the case for the National River and Lakes Cadastre as well. This process is particularly challenging when it comes to the modernisation of national information systems and cadastres, which are often subject to quite strict legislative control. There are also a number of challenges at the technical level: 1) old and outdated software that cannot be upgraded without overhauling the system, 2) integrations with other systems, 3) old infrastructure, 4) code that is closed and unmanageable by the organization, and 5) users who are working with the data, who are challenged by the new solutions. This is exactly the same set of problems that awaited the modernisation of the Lithuanian National Cadastre of Rivers and Lakes, managed by the Environmental Protection Agency. The Lithuanian National Cadastre of Rivers and Lakes is a system for collecting, organizing and making available to other information systems and to the public data on rivers, lakes, ponds, as well as hydraulic engineering structures such as hydroelectric power plants, overflow culverts, fish ladders, fish passes, research stations, etc. The cadastre consists of three major components: the administration of GIS data, the publishing and viewing of online map services, and the provision of e-services (extracts, statistics). This cadastre is of particular importance as it is not only essential for the monitoring of the hydrographic network, but also for the protection of these features and the restriction of agricultural activities. The digital cadastre of rivers and lakes was launched almost 10 years ago, and at that time it was one of the largest GIS projects in Lithuania. The cadastre was implemented using only commercial software: ArcGIS (ArcMap, ArcGIS Server, ArcGIS JS api), Oracle, Alfresco (document management system), etc. Over time, the Environment Agency has been unable to upgrade the software of these systems due to licensing costs, and has lost support as a result. The system soon showed its weaknesses: the system's availability was unacceptable (the load time of web map was 1 minute at peak times), the incorrectly implemented database structure led to a large number of errors, as the data was administered using a programmed plug-in in the ArcMap environment, any changes were difficult to implement and in practice were difficult to support due to the old code and outdated versions of ArcMap software. We came into the modernisation project with the basic idea of migrating the core functionality to open source solutions, i.e. migrating exactly what is being used and not bloated with legacy software solutions. In this presentation, we will share these insights from the project: - Use of PostGIS for automated data filling, e.g. automated analysis of river tributaries, lengths, distances to parent river headwaters, etc. All of this is realized with the help of PostGIS functions; - Database structure and use to ensure data integrity; - Creating a map browser using OpenLayers and Vue JS as a reusable component that now serves also for other informational systems; - Use of QGIS server and Docker; - GitHub and continuous deployment to dev, staging and production environments.
Keywords
127
Workstation <Musikinstrument>Component-based software engineeringLogical constantSystem programmingLogicCalculationPlug-in (computing)Service (economics)Server (computing)Operator (mathematics)DatabaseIntegrated development environmentOracleExplosionData managementRevision controlRule of inferenceOpen sourceGeometryAttribute grammarError messageConsistencyCodeComputerPrice indexScale (map)Reading (process)Self-organizationStrategy gameMathematical optimizationDemo (music)Process (computing)Configuration spaceSanitary sewerBuildingMathematicsEuler anglesMaxima and minimaIterationMoment of inertiaFunction (mathematics)Euler anglesDatabaseWeb 2.0System programmingFront and back endsLevel (video gaming)Inheritance (object-oriented programming)MereologyCode2 (number)Server (computing)Insertion lossData managementOpen sourceSubject indexingScaling (geometry)Document management systemValidity (statistics)Plug-in (computing)Object (grammar)Functional (mathematics)Intrusion detection systemProjective planeStrategy gameDimensional analysisData structureLibrary (computing)StatisticsLogicError messageConstructor (object-oriented programming)Traffic reportingLengthEnterprise architectureField (computer science)Set (mathematics)Structural loadAreaBuildingCalculationAdditionHuman migrationGroup actionSoftwareTable (information)Rule of inferenceLink (knot theory)DebuggerArc (geometry)Lecture/ConferenceMeeting/InterviewComputer animation
Function (mathematics)Maxima and minimaConfiguration spaceServer (computing)System programmingComputer wormEuler anglesMathematicsOpen sourceMoment of inertiaFeedbackService (economics)Process (computing)Integrated development environmentComponent-based software engineeringCodeStandard deviationTable (information)Content (media)World Wide Web ConsortiumLocal ringMethodenbankElectronic visual displayStatisticsComputing platformInformationSpeciesDisintegrationStack (abstract data type)Data managementBuildingBlock (periodic table)Attribute grammarCalculationDatabaseAutomationoutputRule of inferenceField (computer science)Form (programming)Control flowFinitary relationComputer fileComputer-generated imageryPlug-in (computing)Virtual realityServer (computing)TesselationDemo (music)Plug-in (computing)Group actionSpeciesIntegrated development environmentInformationComputing platformComponent-based software engineeringMedical imagingOpen sourceBuildingLevel (video gaming)Table (information)Attribute grammarForm (programming)DatabaseMereologyProjective planeFundamental theorem of algebraSoftware developerSystem programmingGoodness of fitProduct (business)Data managementExecution unitWeb 2.0Functional (mathematics)Information systemsData integrityConnectivity (graph theory)Similarity (geometry)Computer fileLengthDifferent (Kate Ryan album)Quaternion groupMultilaterationStatisticsBitCircleRepository (publishing)Insertion lossSemiconductor memoryMetreHTTP cookieVector spaceMoving averageGeometryINTEGRALComputer animation
AutomationDatabaseAttribute grammarCalculationField (computer science)Form (programming)Stack (abstract data type)Data managementFinitary relationElectronic visual displayRule of inferenceoutputError messageProcess (computing)Function (mathematics)ComputerData modelPersonal digital assistantGame controllerSoftware maintenanceTable (information)Standard deviationAvatar (2009 film)Complex (psychology)View (database)Data integrityRäumliche StatistikService (economics)Task (computing)LogicInfinite conjugacy class propertyHuman migrationGroup actionInheritance (object-oriented programming)Connectivity (graph theory)Computer networkCodeVector spaceServer (computing)Equivalence relationLevel (video gaming)World Wide Web ConsortiumOpen sourceData managementTesselationLevel (video gaming)Multiplication signSimilarity (geometry)CalculationSuite (music)Integrated development environmentInformationTask (computing)Group actionView (database)Open sourceServer (computing)Open setFunction (mathematics)Product (business)Field (computer science)Endliche ModelltheorieStandard deviationRepository (publishing)Rule of inferenceHuman migrationMereologyINTEGRALSystem programmingProjective planeAttribute grammarScheduling (computing)Process (computing)Software developerCodeMetreDatabaseData storage deviceSystem administratorForm (programming)Parameter (computer programming)LengthData modelVirtual machineMusical ensembleProgrammer (hardware)Proxy serverComputing platformSlide ruleWebsiteEvent horizonComputer fileBitSet (mathematics)Touch typingGraphical user interfaceVector spaceConstructor (object-oriented programming)VirtualizationGoodness of fitComputer animationLecture/ConferenceMeeting/Interview
Computer-assisted translationLeast squaresComputer animation
Transcript: English(auto-generated)
So I'm always thinking that's a transition of existing system based on commercial software to open source software is more challenging than creating the new system based on the open source software because you should deal not only with the migration of data and functionality but also with
the mindset of the users that working with the system for many years. So today I will present you a project part of which one part of which was transition National River Lake Kadastro of Lithuania so open source solution and what kind of challenges and issues we face out and how we
managed to deal with this. So first of all what contains what data contains National River Kadastro in Lithuania so rivers lakes and ponds as usual and we have about 10,000 to register these objects in Lithuania and Kadastro as well we registering also engineering facilities like
hydro power plants, chlorids, dams and so on and also Kadastro contains of additional data like river basins catchment areas and so on and you know what this make really important of this Kadastro because this Kadastro also
leads to some restriction, legal restriction for example for agriculture activities or buildings or constructions near the Kadastro objects and also from the GIS part is also worth to mention that the data sets is not too big enough but actually all the objects are should be related to
each other for example hydro power plants should be linked to the river and dams should be known how the length from the dam to the river and and so on and this everything should be calculated and presented in the Kadastro.
So before we started what kind of system we have found and this yeah commercial based software and this is quite usual tech stack for Lithuanian enterprise GIS systems so on the database part you have Oracle with
ArcGIS SDI schema everyone can have opportunity to deal with ArcGIS SDI schema understands that this is quite tough because you don't understand quite well what you getting from this schema yeah and they use also alfresco for data
for document management system on the database for but actually what we have found out in the database that this database has almost any restriction on the data side so the table doesn't have relationships doesn't have rules
for automatic calculation of the data and what's it was really strange because it's Kadastro so how they are calculating this and the tricky part was that on the desktop part where the Kadastro managers are editing and creating and registering the data they have custom-made closed source
closed source plugin which works on the ArcGIS on ArcMap and all the logic of the calculations even the classifiers or some dimensions that use and calculations was hard code in this plugin so it was nice thing to find out
and also for the service and web part this was quite the usual stat so for the publishing the WMS IGS mapping services there was using IGS server and for reporting so all the cadastro should have possibilities to get
official extracts from the cadastro so they use just the report server this this was solely used only for this PDF extract and for the web part is RGS API but actually we found out that this libraries was so old that we
understand from the beginning that we are implementing new web map but not deal with the old one because it's not actually possible to update some of the API some of the libraries like RGS and then front-end libraries so about
this plugin that I understand the idea about this but actually when you have the cadastro and you put everything on the plugin you actually doesn't have validation on the database part so it's like you are creating no front-end and
back-end and all the logic and validation is doing only on the front-end but not in the back-end but you're not strict the access to the data on or edit data not from the plugins so what leads to the errors in the
database and actually we what we have found out that some of the not some but a lot of the objects have for example duplicated cadastro ID some rivers has defined its parent rivers that even doesn't exist in the cadastro and some
IDs was miscalculated and so on it and it's normally understandable because some edits probably was made not to the plugin or the plugin crashed and so on and from database parts there was no validation even the fields that are
described as a requirement required from this plugin parts was actually not defined as required in the database part and so at the end we face out that you actually cannot understand how it works and actually cannot understand
from the database part how it was made okay and another tricky part not tricky actually this is understandable when you don't know how your database work is performance issues because initial web map load was about 30 seconds 40 seconds so you understand that you open it the map and
wait for the 40 seconds just to load it but again it's understandable when you don't have any strategy for calculating spatial indexes when you have quite stuff structure of the schema on the database parts when you
don't have any actually strategy how you want to show all the data because you just put all the data directly from the database on the high scale levels and you just try to show them if on desktop it doesn't work great so on the web it will work even worse and as well there was also some kind of
interesting issues but I as far as I working also with S3 as the ishemas so it's kind of usual to find out that on one database they are putting like
a lot of systems and just separate them by schema so you actually doesn't can measure which actually data schema and so on is actually breaking up the data for example inserts and so on so let's get started and the first thing we issue actually was not the technical issues because we have some
strategy how to deal with this but actually attitude towards the open source yes from the cadastre manager because even though the cadastre work quite badly but they thought that if we try to migrate to open source and
reduce an open source probably at the end they will doesn't have any not even the badly working system and I quite understand them because in we don't have like a lot of good examples or even this is one of the
first example that national cadastre is based on open source JS and you know the rumor from vendor company that rules Lithuania is actually is also not helping here and so what we thought to do how we thought to deal with
this so we take this MVP methodology from start-ups so what this MVP so have minimal viable project a product so on one part you have minimal so bare-bones of the system so which only contains the fundamental components like JS server database and so on on the other part and you can build it
quite quickly because you can just run it from those different Dockers and set up like in one week and another party have viable project so it's final product with all the functionality the cadastre manager needs but you can build it only after the year or after eight months and so on so what
we are trying to build that the first glimpse so it was sufficient enough solution just introduced this to cadastre manager and actually after the first two months we already have the first solution so first set up and give
them to the cadastre manager and then at the first at the first demo they already saw that this working much faster at work and they don't need actually do some manual inserts because a lot of inserts was made automatically
on the database part and what was great that later on they are suggesting a feature or a request that features based on this setup not from the not from the you know memories how RGS or something else works so it was really
great introduction and great how say the circle of managing all the updates and we have also face out some general requirements for our system so that as I mentioned we use this project was the part of the bigger project so the main
idea was if you building something builds to be reusable so with the biggest our reusable component was that map also priority on open source not only use open source but if we something built open source it also so we have
already 35 this repository is open source to the community of Lithuania of course and for the rest of the world and the unified deployment so we have set up for development staging and production environment and so every
deployment is based on GitHub actions and also what is kind of usual when you're dealing with code but with you're dealing with the GIS for example publishing services it's quite I think new and was introduced also for the GIS specialists and also one part is really important for me as well because
when you're dealing with large GIS systems usually there is a tricky part how you develop locally so everything in our system was based on the Docker images so the setup of the local environment just you know the running
three or five Docker images and you can test it for example GIS server not only how it works with your local database but just changing environmental and quickly check up how deals with the production database and if it's your service is running okay or it's slow or not so it was
really a good experience and the bigger part as I mentioned so it was only one part of the project and actually the whole project calls via diverse information platform and it contains more of the 10 components that use GIS contains a little bit more component but 10 components are using
GIS and also this biodiversity platform on based on this platform is also implemented protected and invasives invasive species informational system and this lake and the river cadastre as well and it was
implemented as in-house solution so Ministry of Environment set up the in-house development team together with the spatial analysts not facial deal with the analysts the UX designers and actually GIS was quite small part of this and I was only one GIS specialist in this project so text
that was used so I think it's quite basic but let's let's review it so on database parts we are using pause GIS with Postgres well and the G
integrations and updates and we're also running GDAL or based on GitHub actions so we kind of like GitHub actions as you can see here and from for the server side and for the publishing service for WMS and WFS we are of course using not of course but we're using QGIS server Martin we
use and we introduced this technology a little bit later for publishing vector tile services for directly from postgres coil database it's usually used for example for grids for overview layers for example
statistical grids and so on and because we introduced vector tiles as the overview layers we saw that our national base map which is actually not with our needs so we built our national vector based national base map based on the national cadastral cadastral data a geo reference cadastral
data using planet Tyler and for the desktop part we use desktop for QGIS and cartography sma putnik and also cookies desktop so actually when we first open it and then show the demo for cadastral managers and they saw QGIS form they thought that we migrated this the old plug-in cookies because
actually it looks a little bit similar and the idea was to standardize the way how they insert in the data but and also so we actually use cookies forms a lot and what is great about cookies forms that
you can show relationships one to many for example if you show the lake you can show also rivers that inflows outflows of the lakes or you can show nicely edit history and so on and also you can show imagery images
related with this input or allow to download the file from the database or if external sources so and also we use some virtual attributes actually to decrease the table sizes because in all the system there was they have like
third of the one-third of the attributes dedicated only for example to show length in different units and you know the length in the meters kilometers and so on so we for that we just use virtual attributes and doesn't store them in the database because if you store in database you
should recalculate each time so why to do this and what is really nice about QGIS and PostGIS integration that you can store QGIS projects directly in the database and what does it mean so when the Cadastro administrator come to
the desktop and open it only two minutes okay so we store the QGIS projects in the database so and the in QGIS project we are storing the custom tools based on the model builder not model model designer and this model
designer calls usually custom functions from database so it's looks really simple but actually what it takes so it takes from form some parameters and just run PostgreSQL functions and we put this models inside the QGIS project QGIS project is inside database so each time they open it the
new the open it QGIS project from the database is always up to date so strict data model using standard database concept of like relationships and so on it was really strict so what is actually usual for from the database part but not usually from as the I part and and then we face out
really big problems with migration data is because when you have strict data rules and the database part and try to migrate so actually during immigration we fix a lot of data that they have broken into in the older system and for this legal this extracts we use materialize that views with
JSON fields because we have some actually quite interesting one-to-many relationship in the database and actually our API's are really lean
because everything is pre calculated in the database and also in database we are running PG Chrome to schedule the tasks to refresh materialized view each night or each hour so it depends on the views we use and yeah the concept we
for automated tasks we use functions and triggers and all these calculations are integrated in the post GIS as a functions and also everything is started as a github with the migration to the databases as CLI
process and actually for really big updates of the data for example if we should take for external data source the data and update it so we use also get have actions because we really like it and the last part so publishing actually we use also github to publish new services so for example you change
project then committed to the github repository you're creating the pull request and then you can deploy it with the github action to development staging or production system already and this is kind of similar as you
working with the code management in the it have action but actually your code is QGIS project and actually you are not putting the file on the QGIS server on the QGIS docker machine but actually you're dealing with
the github and it's actually for us it's really great I highly suggest them to check it out our national vector tile base map which is based on top of planet Tyler because it's not based on the open suite map but our national cadastro data and we map it to the open vector tiles style schema so
it's our national data sets but with the feel and touch as a used in the open vector tile maps and of course we are using PM tiles with the 300 megabytes and thanks for my colleagues and open source about is about not only
communities but and making friends so I make a few friends bodies can we take some beers after the work thanks a lot sorry that they came a little bit too much or was really quickly you can find my slide on slides on on the
website thank thank thank you very much and there was that was all good was all good so are there any questions so I have a question and you said you
know it it's with the Ministry of the Environment and did other ministries like carefully observe or were they involved or did they show interest in your work yes we already have two ministry based events for
other ministries and we set up like this competence center for the shared information about and actually in Lithuania environmental minister is really big and deals with the construction environmental and land
management and so actually it's really big and it's enough agencies inside the ministry that deal with this and actually we have already new systems that came up from the agencies on on the umbrella of ministry that are
already set tapping the open source solutions for that but because usually what they need is like really simple GIS system not too big not too small but as I mentioned here this in the middle where you can just complete
everyday tasks and if you show deeper if you look deeper you can see that these tasks are quite quite simple and not so intelligent thanks any other questions good yeah have you also calculated of how much time and money
took okay I expected this question as I mentioned here it was in-house solution so actually there was not like based on the you have what half
of million and just spend it out it was based on calculation on ours so I my part was about one and a half year and it took the one third of my so it's about 30 hours per month from me so as I mentioned I was only one GIS
specialist so and over specialists like programmers and DevOps they are working in the ministry so they take their time work time so it's not to say spent additionally to this project so because it was in-house so actually I'm only one was not who's working in the ministry but just providing the my
hours to this project so it's and it took about the whole be a diversity platform to build it took about two years and this cadastre it's about eight months