Modernizing the National River and Lakes Cadastre by Transition to FOSS4G
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 156 | |
Author | ||
Contributors | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/68500 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
| |
Keywords |
FOSS4G Europe 2024 Tartu81 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
Workstation <Musikinstrument>Component-based software engineeringLogical constantSystem programmingLogicCalculationPlug-in (computing)Service (economics)Server (computing)Operator (mathematics)DatabaseIntegrated development environmentOracleExplosionData managementRevision controlRule of inferenceOpen sourceGeometryAttribute grammarError messageConsistencyCodeComputerPrice indexScale (map)Reading (process)Self-organizationStrategy gameMathematical optimizationDemo (music)Process (computing)Configuration spaceSanitary sewerBuildingMathematicsEuler anglesMaxima and minimaIterationMoment of inertiaFunction (mathematics)Euler anglesDatabaseWeb 2.0System programmingFront and back endsLevel (video gaming)Inheritance (object-oriented programming)MereologyCode2 (number)Server (computing)Insertion lossData managementOpen sourceSubject indexingScaling (geometry)Document management systemValidity (statistics)Plug-in (computing)Object (grammar)Functional (mathematics)Intrusion detection systemProjective planeStrategy gameDimensional analysisData structureLibrary (computing)StatisticsLogicError messageConstructor (object-oriented programming)Traffic reportingLengthEnterprise architectureField (computer science)Set (mathematics)Structural loadAreaBuildingCalculationAdditionHuman migrationGroup actionSoftwareTable (information)Rule of inferenceLink (knot theory)DebuggerArc (geometry)Lecture/ConferenceMeeting/InterviewComputer animation
08:14
Function (mathematics)Maxima and minimaConfiguration spaceServer (computing)System programmingComputer wormEuler anglesMathematicsOpen sourceMoment of inertiaFeedbackService (economics)Process (computing)Integrated development environmentComponent-based software engineeringCodeStandard deviationTable (information)Content (media)World Wide Web ConsortiumLocal ringMethodenbankElectronic visual displayStatisticsComputing platformInformationSpeciesDisintegrationStack (abstract data type)Data managementBuildingBlock (periodic table)Attribute grammarCalculationDatabaseAutomationoutputRule of inferenceField (computer science)Form (programming)Control flowFinitary relationComputer fileComputer-generated imageryPlug-in (computing)Virtual realityServer (computing)TesselationDemo (music)Plug-in (computing)Group actionSpeciesIntegrated development environmentInformationComputing platformComponent-based software engineeringMedical imagingOpen sourceBuildingLevel (video gaming)Table (information)Attribute grammarForm (programming)DatabaseMereologyProjective planeFundamental theorem of algebraSoftware developerSystem programmingGoodness of fitProduct (business)Data managementExecution unitWeb 2.0Functional (mathematics)Information systemsData integrityConnectivity (graph theory)Similarity (geometry)Computer fileLengthDifferent (Kate Ryan album)Quaternion groupMultilaterationStatisticsBitCircleRepository (publishing)Insertion lossSemiconductor memoryMetreHTTP cookieVector spaceMoving averageGeometryINTEGRALComputer animation
16:29
AutomationDatabaseAttribute grammarCalculationField (computer science)Form (programming)Stack (abstract data type)Data managementFinitary relationElectronic visual displayRule of inferenceoutputError messageProcess (computing)Function (mathematics)ComputerData modelPersonal digital assistantGame controllerSoftware maintenanceTable (information)Standard deviationAvatar (2009 film)Complex (psychology)View (database)Data integrityRäumliche StatistikService (economics)Task (computing)LogicInfinite conjugacy class propertyHuman migrationGroup actionInheritance (object-oriented programming)Connectivity (graph theory)Computer networkCodeVector spaceServer (computing)Equivalence relationLevel (video gaming)World Wide Web ConsortiumOpen sourceData managementTesselationLevel (video gaming)Multiplication signSimilarity (geometry)CalculationSuite (music)Integrated development environmentInformationTask (computing)Group actionView (database)Open sourceServer (computing)Open setFunction (mathematics)Product (business)Field (computer science)Endliche ModelltheorieStandard deviationRepository (publishing)Rule of inferenceHuman migrationMereologyINTEGRALSystem programmingProjective planeAttribute grammarScheduling (computing)Process (computing)Software developerCodeMetreDatabaseData storage deviceSystem administratorForm (programming)Parameter (computer programming)LengthData modelVirtual machineMusical ensembleProgrammer (hardware)Proxy serverComputing platformSlide ruleWebsiteEvent horizonComputer fileBitSet (mathematics)Touch typingGraphical user interfaceVector spaceConstructor (object-oriented programming)VirtualizationGoodness of fitComputer animationLecture/ConferenceMeeting/Interview
24:43
Computer-assisted translationLeast squaresComputer animation
Transcript: English(auto-generated)
00:00
So I'm always thinking that's a transition of existing system based on commercial software to open source software is more challenging than creating the new system based on the open source software because you should deal not only with the migration of data and functionality but also with
00:21
the mindset of the users that working with the system for many years. So today I will present you a project part of which one part of which was transition National River Lake Kadastro of Lithuania so open source solution and what kind of challenges and issues we face out and how we
00:41
managed to deal with this. So first of all what contains what data contains National River Kadastro in Lithuania so rivers lakes and ponds as usual and we have about 10,000 to register these objects in Lithuania and Kadastro as well we registering also engineering facilities like
01:03
hydro power plants, chlorids, dams and so on and also Kadastro contains of additional data like river basins catchment areas and so on and you know what this make really important of this Kadastro because this Kadastro also
01:20
leads to some restriction, legal restriction for example for agriculture activities or buildings or constructions near the Kadastro objects and also from the GIS part is also worth to mention that the data sets is not too big enough but actually all the objects are should be related to
01:43
each other for example hydro power plants should be linked to the river and dams should be known how the length from the dam to the river and and so on and this everything should be calculated and presented in the Kadastro.
02:01
So before we started what kind of system we have found and this yeah commercial based software and this is quite usual tech stack for Lithuanian enterprise GIS systems so on the database part you have Oracle with
02:22
ArcGIS SDI schema everyone can have opportunity to deal with ArcGIS SDI schema understands that this is quite tough because you don't understand quite well what you getting from this schema yeah and they use also alfresco for data
02:43
for document management system on the database for but actually what we have found out in the database that this database has almost any restriction on the data side so the table doesn't have relationships doesn't have rules
03:02
for automatic calculation of the data and what's it was really strange because it's Kadastro so how they are calculating this and the tricky part was that on the desktop part where the Kadastro managers are editing and creating and registering the data they have custom-made closed source
03:22
closed source plugin which works on the ArcGIS on ArcMap and all the logic of the calculations even the classifiers or some dimensions that use and calculations was hard code in this plugin so it was nice thing to find out
03:44
and also for the service and web part this was quite the usual stat so for the publishing the WMS IGS mapping services there was using IGS server and for reporting so all the cadastro should have possibilities to get
04:05
official extracts from the cadastro so they use just the report server this this was solely used only for this PDF extract and for the web part is RGS API but actually we found out that this libraries was so old that we
04:26
understand from the beginning that we are implementing new web map but not deal with the old one because it's not actually possible to update some of the API some of the libraries like RGS and then front-end libraries so about
04:47
this plugin that I understand the idea about this but actually when you have the cadastro and you put everything on the plugin you actually doesn't have validation on the database part so it's like you are creating no front-end and
05:05
back-end and all the logic and validation is doing only on the front-end but not in the back-end but you're not strict the access to the data on or edit data not from the plugins so what leads to the errors in the
05:23
database and actually we what we have found out that some of the not some but a lot of the objects have for example duplicated cadastro ID some rivers has defined its parent rivers that even doesn't exist in the cadastro and some
05:44
IDs was miscalculated and so on it and it's normally understandable because some edits probably was made not to the plugin or the plugin crashed and so on and from database parts there was no validation even the fields that are
06:02
described as a requirement required from this plugin parts was actually not defined as required in the database part and so at the end we face out that you actually cannot understand how it works and actually cannot understand
06:21
from the database part how it was made okay and another tricky part not tricky actually this is understandable when you don't know how your database work is performance issues because initial web map load was about 30 seconds 40 seconds so you understand that you open it the map and
06:44
wait for the 40 seconds just to load it but again it's understandable when you don't have any strategy for calculating spatial indexes when you have quite stuff structure of the schema on the database parts when you
07:03
don't have any actually strategy how you want to show all the data because you just put all the data directly from the database on the high scale levels and you just try to show them if on desktop it doesn't work great so on the web it will work even worse and as well there was also some kind of
07:27
interesting issues but I as far as I working also with S3 as the ishemas so it's kind of usual to find out that on one database they are putting like
07:40
a lot of systems and just separate them by schema so you actually doesn't can measure which actually data schema and so on is actually breaking up the data for example inserts and so on so let's get started and the first thing we issue actually was not the technical issues because we have some
08:03
strategy how to deal with this but actually attitude towards the open source yes from the cadastre manager because even though the cadastre work quite badly but they thought that if we try to migrate to open source and
08:21
reduce an open source probably at the end they will doesn't have any not even the badly working system and I quite understand them because in we don't have like a lot of good examples or even this is one of the
08:40
first example that national cadastre is based on open source JS and you know the rumor from vendor company that rules Lithuania is actually is also not helping here and so what we thought to do how we thought to deal with
09:02
this so we take this MVP methodology from start-ups so what this MVP so have minimal viable project a product so on one part you have minimal so bare-bones of the system so which only contains the fundamental components like JS server database and so on on the other part and you can build it
09:24
quite quickly because you can just run it from those different Dockers and set up like in one week and another party have viable project so it's final product with all the functionality the cadastre manager needs but you can build it only after the year or after eight months and so on so what
09:45
we are trying to build that the first glimpse so it was sufficient enough solution just introduced this to cadastre manager and actually after the first two months we already have the first solution so first set up and give
10:03
them to the cadastre manager and then at the first at the first demo they already saw that this working much faster at work and they don't need actually do some manual inserts because a lot of inserts was made automatically
10:22
on the database part and what was great that later on they are suggesting a feature or a request that features based on this setup not from the not from the you know memories how RGS or something else works so it was really
10:41
great introduction and great how say the circle of managing all the updates and we have also face out some general requirements for our system so that as I mentioned we use this project was the part of the bigger project so the main
11:04
idea was if you building something builds to be reusable so with the biggest our reusable component was that map also priority on open source not only use open source but if we something built open source it also so we have
11:22
already 35 this repository is open source to the community of Lithuania of course and for the rest of the world and the unified deployment so we have set up for development staging and production environment and so every
11:42
deployment is based on GitHub actions and also what is kind of usual when you're dealing with code but with you're dealing with the GIS for example publishing services it's quite I think new and was introduced also for the GIS specialists and also one part is really important for me as well because
12:04
when you're dealing with large GIS systems usually there is a tricky part how you develop locally so everything in our system was based on the Docker images so the setup of the local environment just you know the running
12:22
three or five Docker images and you can test it for example GIS server not only how it works with your local database but just changing environmental and quickly check up how deals with the production database and if it's your service is running okay or it's slow or not so it was
12:44
really a good experience and the bigger part as I mentioned so it was only one part of the project and actually the whole project calls via diverse information platform and it contains more of the 10 components that use GIS contains a little bit more component but 10 components are using
13:04
GIS and also this biodiversity platform on based on this platform is also implemented protected and invasives invasive species informational system and this lake and the river cadastre as well and it was
13:21
implemented as in-house solution so Ministry of Environment set up the in-house development team together with the spatial analysts not facial deal with the analysts the UX designers and actually GIS was quite small part of this and I was only one GIS specialist in this project so text
13:45
that was used so I think it's quite basic but let's let's review it so on database parts we are using pause GIS with Postgres well and the G
14:01
integrations and updates and we're also running GDAL or based on GitHub actions so we kind of like GitHub actions as you can see here and from for the server side and for the publishing service for WMS and WFS we are of course using not of course but we're using QGIS server Martin we
14:24
use and we introduced this technology a little bit later for publishing vector tile services for directly from postgres coil database it's usually used for example for grids for overview layers for example
14:40
statistical grids and so on and because we introduced vector tiles as the overview layers we saw that our national base map which is actually not with our needs so we built our national vector based national base map based on the national cadastral cadastral data a geo reference cadastral
15:02
data using planet Tyler and for the desktop part we use desktop for QGIS and cartography sma putnik and also cookies desktop so actually when we first open it and then show the demo for cadastral managers and they saw QGIS form they thought that we migrated this the old plug-in cookies because
15:26
actually it looks a little bit similar and the idea was to standardize the way how they insert in the data but and also so we actually use cookies forms a lot and what is great about cookies forms that
15:43
you can show relationships one to many for example if you show the lake you can show also rivers that inflows outflows of the lakes or you can show nicely edit history and so on and also you can show imagery images
16:02
related with this input or allow to download the file from the database or if external sources so and also we use some virtual attributes actually to decrease the table sizes because in all the system there was they have like
16:22
third of the one-third of the attributes dedicated only for example to show length in different units and you know the length in the meters kilometers and so on so we for that we just use virtual attributes and doesn't store them in the database because if you store in database you
16:43
should recalculate each time so why to do this and what is really nice about QGIS and PostGIS integration that you can store QGIS projects directly in the database and what does it mean so when the Cadastro administrator come to
17:04
the desktop and open it only two minutes okay so we store the QGIS projects in the database so and the in QGIS project we are storing the custom tools based on the model builder not model model designer and this model
17:21
designer calls usually custom functions from database so it's looks really simple but actually what it takes so it takes from form some parameters and just run PostgreSQL functions and we put this models inside the QGIS project QGIS project is inside database so each time they open it the
17:42
new the open it QGIS project from the database is always up to date so strict data model using standard database concept of like relationships and so on it was really strict so what is actually usual for from the database part but not usually from as the I part and and then we face out
18:04
really big problems with migration data is because when you have strict data rules and the database part and try to migrate so actually during immigration we fix a lot of data that they have broken into in the older system and for this legal this extracts we use materialize that views with
18:29
JSON fields because we have some actually quite interesting one-to-many relationship in the database and actually our API's are really lean
18:40
because everything is pre calculated in the database and also in database we are running PG Chrome to schedule the tasks to refresh materialized view each night or each hour so it depends on the views we use and yeah the concept we
19:02
for automated tasks we use functions and triggers and all these calculations are integrated in the post GIS as a functions and also everything is started as a github with the migration to the databases as CLI
19:22
process and actually for really big updates of the data for example if we should take for external data source the data and update it so we use also get have actions because we really like it and the last part so publishing actually we use also github to publish new services so for example you change
19:48
project then committed to the github repository you're creating the pull request and then you can deploy it with the github action to development staging or production system already and this is kind of similar as you
20:04
working with the code management in the it have action but actually your code is QGIS project and actually you are not putting the file on the QGIS server on the QGIS docker machine but actually you're dealing with
20:22
the github and it's actually for us it's really great I highly suggest them to check it out our national vector tile base map which is based on top of planet Tyler because it's not based on the open suite map but our national cadastro data and we map it to the open vector tiles style schema so
20:47
it's our national data sets but with the feel and touch as a used in the open vector tile maps and of course we are using PM tiles with the 300 megabytes and thanks for my colleagues and open source about is about not only
21:05
communities but and making friends so I make a few friends bodies can we take some beers after the work thanks a lot sorry that they came a little bit too much or was really quickly you can find my slide on slides on on the
21:23
website thank thank thank you very much and there was that was all good was all good so are there any questions so I have a question and you said you
21:44
know it it's with the Ministry of the Environment and did other ministries like carefully observe or were they involved or did they show interest in your work yes we already have two ministry based events for
22:03
other ministries and we set up like this competence center for the shared information about and actually in Lithuania environmental minister is really big and deals with the construction environmental and land
22:26
management and so actually it's really big and it's enough agencies inside the ministry that deal with this and actually we have already new systems that came up from the agencies on on the umbrella of ministry that are
22:46
already set tapping the open source solutions for that but because usually what they need is like really simple GIS system not too big not too small but as I mentioned here this in the middle where you can just complete
23:02
everyday tasks and if you show deeper if you look deeper you can see that these tasks are quite quite simple and not so intelligent thanks any other questions good yeah have you also calculated of how much time and money
23:31
took okay I expected this question as I mentioned here it was in-house solution so actually there was not like based on the you have what half
23:43
of million and just spend it out it was based on calculation on ours so I my part was about one and a half year and it took the one third of my so it's about 30 hours per month from me so as I mentioned I was only one GIS
24:02
specialist so and over specialists like programmers and DevOps they are working in the ministry so they take their time work time so it's not to say spent additionally to this project so because it was in-house so actually I'm only one was not who's working in the ministry but just providing the my
24:27
hours to this project so it's and it took about the whole be a diversity platform to build it took about two years and this cadastre it's about eight months