An Open Source Approach to Multi-user Distributed Geospatial Data Management
Video in TIB AV-Portal:
An Open Source Approach to Multi-user Distributed Geospatial Data Management
Formal Metadata
Title |
An Open Source Approach to Multi-user Distributed Geospatial Data Management
|
Title of Series | |
Part Number |
159
|
Number of Parts |
193
|
Author |
|
License |
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. |
Identifiers |
|
Publisher |
|
Release Date |
2016
|
Language |
English
|
Content Metadata
Subject Area | |
Abstract |
Open source tools have been successful in managing geospatial data in central data stores. However, performance issues can arise from many users accessing the same table in a geospatial database at once, especially in a multi-user editing environment. The geospatial landscape also changes constantly, as a result of human activity and natural forces, this gives a need to track these changes within the geospatial database and perform change detection activities to understand changes across time, hence a need to version history. These use cases springs up the requirements to employ a data distribution across multiple geospatial databases using versioning and replication technology to integrate several desktop and mobile user applications into an adaptive geospatial communications environment connecting operations across the enterprise and throughout the organisations to improve data availability to multiple users, tracking change history within multiple table versions while increasing system performance. Several commercial geospatial applications have successfully implemented full versioning replication capabilities by leveraging middleware with the core database versioning capabilities – for example ArcSDE technology from Esri . The realization of a full solution has been far-fetched on open source geospatial applications. This presentation discusses the development of QGIS plugin and implementation of FOSS4G solutions for versioning and replication capabilities with to support multi-user access while optimizing performance using QGIS, PostgreSQL, PostGIS and SpatialLite DB technologies.
|

00:00
Goodness of fit
Computer animation
Open source
Meeting/Interview
Moving average
Self-organization
Open set
00:39
Area
Multiplication
Call centre
NP-hard
Projective plane
Time zone
Special unitary group
Flow separation
Planning
Chain
Computer animation
Different (Kate Ryan album)
Operator (mathematics)
Office suite
Summierbarkeit
Physical system
Vacuum
01:20
Computer animation
View (database)
Decision theory
Information retrieval
Projective plane
File format
Mereology
Special unitary group
Formal grammar
Asynchronous Transfer Mode
Window
01:42
Area
Mobile Web
Multiplication
Slide rule
Online help
Decision theory
Computer file
File format
Mereology
Power (physics)
Computer animation
Different (Kate Ryan album)
Window
02:22
Satellite
Mapping
Information
Multiplication sign
Decision theory
Database
Line (geometry)
02:50
Group action
Server (computing)
Sequel
Mehrplatzsystem
File format
Database
Special unitary group
Emulation
Revision control
Architecture
Geometry
Exploratory data analysis
Type theory
Customer relationship management
Different (Kate Ryan album)
Summierbarkeit
Multiplication
Physical system
Window
Slide rule
View (database)
Computer file
Projective plane
Replication (computing)
Data management
Computer animation
Software
Strategy game
Endliche Modelltheorie
Quicksort
Physical system
04:03
Metropolitan area network
Open source
View (database)
File format
Line (geometry)
Special unitary group
Pointer (computer programming)
Computer animation
Software
Hybrid computer
Summierbarkeit
Formal grammar
Physical system
Window
04:42
Server (computing)
Identifiability
Open source
Buckling
Multiplication sign
File format
Maxima and minima
Client (computing)
Special unitary group
Revision control
Architecture
Frequency
Geometry
Pointer (computer programming)
Mathematics
Customer relationship management
Hybrid computer
Operator (mathematics)
Multiplication
Posterior probability
Formal grammar
Physical system
Window
Area
View (database)
Software developer
Computer file
Open source
Sampling (statistics)
Database
Limit (category theory)
Flow separation
Hysteresekurve
Graphical user interface
Process (computing)
Computer animation
Software
Integrated development environment
Object (grammar)
Physical system
Wide area network
07:29
PowerPoint
Satellite
Open source
View (database)
Software developer
Real number
Open source
File format
8 (number)
Special unitary group
Architecture
Geometry
Computer animation
Software
Customer relationship management
Meeting/Interview
Freeware
Summierbarkeit
Formal grammar
08:25
Exploratory data analysis
Freeware
Well-formed formula
Computer animation
Software
Integrated development environment
Online help
Open source
Electronic mailing list
Special unitary group
Formal grammar
08:54
Dialect
Slide rule
View (database)
Online help
Server (computing)
Computer file
File format
Range (statistics)
Computer animation
Repository (publishing)
Website
Formal grammar
Window
09:16
Standard deviation
View (database)
File format
Database
Special unitary group
Web 2.0
Geometry
Mathematics
Mechanism design
Strategy game
Repository (publishing)
Area
Dissipation
Metropolitan area network
Enterprise architecture
View (database)
Software developer
Computer file
Open source
Replication (computing)
Data management
Repository (publishing)
Software testing
Right angle
Laptop
Filter <Stochastik>
Functional (mathematics)
Beat (acoustics)
Enterprise architecture
Image resolution
Characteristic polynomial
Branch (computer science)
Revision control
Latent heat
Customer relationship management
Integrated development environment
Data structure
Summierbarkeit
Plug-in (computing)
Mobile app
Plug-in (computing)
Slide rule
Online help
Database
Basis <Mathematik>
Limit (category theory)
Subset
Computer animation
Software
Revision control
Local ring
Window
11:46
Filter <Stochastik>
Enterprise architecture
Open source
Artificial neural network
File format
Maxima and minima
Branch (computer science)
Special unitary group
Value-added network
Revision control
Graphical user interface
Different (Kate Ryan album)
Pairwise comparison
Formal grammar
Window
User interface
Slide rule
Information
Interface (computing)
Database
Trigonometric functions
Graphical user interface
Algebra
Computer animation
Software
Revision control
Convex hull
Right angle
14:06
Plug-in (computing)
Server (computing)
Presentation of a group
Implementation
Slide rule
Open source
Online help
Software developer
Web page
Feedback
File format
Amsterdam Ordnance Datum
Special unitary group
Product (business)
Revision control
Mathematics
Computer animation
Software
Personal digital assistant
Permanent
Normal (geometry)
Right angle
15:25
Digital filter
Image resolution
Decision theory
Branch (computer science)
Instance (computer science)
Special unitary group
Revision control
Graphical user interface
Flow separation
Meeting/Interview
Conflict (process)
User interface
Sine
Interface (computing)
Attribute grammar
Line (geometry)
Connected space
Subset
Subject indexing
Uniform resource locator
Computer animation
Software
Customer relationship management
Design by contract
Right angle
Geometry
17:09
Email
Presentation of a group
Building
Group action
State of matter
Multiplication sign
Execution unit
Client (computing)
Special unitary group
Mathematics
Meeting/Interview
Different (Kate Ryan album)
Area
Collaborationism
Channel capacity
Block (periodic table)
Software developer
Closed set
Bit
Process (computing)
Self-organization
Right angle
Resultant
Filter <Stochastik>
Slide rule
Open source
Transformation (genetics)
Artificial neural network
Wave packet
Product (business)
Revision control
Frequency
Term (mathematics)
Ring (mathematics)
Data structure
Computer architecture
Self-organization
Default (computer science)
Projective plane
Coma Berenices
Core dump
Database
Multilateration
Cartesian coordinate system
Human migration
Word
Computer animation
Software
Personal digital assistant
Video game
Object (grammar)
25:59
Computer animation
00:08
and the other African talk open
00:11
source approach to multiuser distributed geospatial data and so that's the thing about taking away good afternoon everyone my name is Danny I work for the hell out of Africa layout Africa is a non-profit organization based in northern Nigeria with their quarters in Canada we
00:40
have in offices in on Guinea and Liberia I and Nigeria and we so-called public health interventions using data driven approach in in multiple African countries
00:56
and do so do different areas where about projects that focused on how delivery systems and deploying and call centers operating in emergency operations centers you know and and supporting them for their education and nutrition and severe acute non-traditional across different across different
01:25
parts of West Africa and now this defined on projects and interventions that we support required Rocky data retrieval and decision making which means we deploy multiple
01:39
that to support decision making and the different modes
01:44
of wood that what's and you to support near-real-time decision-making and require geographic information to power of D did decision making and we do a lot of we have a lot of people without bulleted if you want to do in data collection I'm in different parts of northern Nigeria especially in places with a bad natural kind how really 2 areas and to use and
02:13
require as those deploying multiple mobile devices for offline data collection and in the we
02:23
have GIS and especially is that take these different data as they are synchronized and near real-time all them off line you know editing using
02:35
satellite imagery and committing invention what did you
02:39
do that is to make different maps at the same time and feeding the information into then the dual database to support decision making now
02:51
and the main thing with this is when we started we started with its proprietary softly
03:01
rice whereby we deployed a multi-user data management system as you can see it in a sequel server believes that agrees with need to wear on top of it and and then and different versions created to to so ports more use editing you know you create your checkouts of and unto your editing comets we didn't know how this works for us and then we define the workflow but the main challenge wake and the the you know the different sort of workflows that we've be using uh with a proprietary software now the main challenge with disease it's really expensive and for projects that we work on their require us to transition and the data and the systems that we built to the government and sometimes it becomes not sustainable so we have to find a sustainable we're working
04:05
we talk about using an open source approach to to
04:10
the this and the 1st thing we did come along the line was to 1st use and so we wanted a situation whereby we can deploy proprietary software because proprietary software something is always easy to deploy and especially in emergency situations and we also wanted to have an open service and in the system and to to continue to work so the 1st thing we did was to run the hybrid system
04:42
whereby the federal the with several posterior samples grants and several not server unsynchronized did to databases by running from jobs and synchronizing them together using a unique identifier so with the proprietary software generates a GUI the ball with they're not adopted in the open source software was also use a UUID system and and credence to much each other and keep them synchronized and Bowie we then needed to take this for that so we looked into the solutions and in the month in the open source markets and 1 of the things we that with the and and solutions which we already talked about here and it's good to see and that these are really improved a lot of developments that have gone into it and so we looked at duty for example we agreed that's and the was was PG versioning and those of the president ball and all of them have their different limitations for our workflow this all we needed exactly was it
05:59
clients you know what Q GIS and being able to do regrows geoprocessing processing operations on on the text of periodicity in time synchronizing these changes back to the database so we could eventually found acute especially in so those already been deployed by all slander and we said and as you can see when we finished and said in all these both in a certain object versioning environment this is what the proprietary looked like and this is what our open-source environment looks like so whereby the user check out your data from it GIS that this into a special lights on and there these 2 do we look at editing and if you wanted to connected editing you check it out from GIS to post from the Judaizing to freeze and then you do your editing unsynchronized back into the hysteresis them so with version is there between the different pose GIS systems online on the server then you can QC of this synchronized true from the master database and then do your buckles all using the POS GIS QG areas and spatially so what if all
07:31
we can't got here and we have to do is a lot of development because of the
07:37
Auslan 2 was not in the from register I did so and 1 of the things we do we have to spend a lot of money and hired and
07:48
develop person to do this and so being in Nigeria for example in Africa you know do something you people get the perception that when you say opens it's free book books in the real sense open source is not free you have developers who are commuting yet only when it is made of a label-free and it's not actually free so we what we all slander to improve on the 2 and I'll talk question was how we got from where we will have in proprietary software to where we are in the open source software by
08:26
the 1st thing we needed to do was to look at and I walk through In the proprietary
08:32
and environments and we went through the list all the different the thing is that we normally do using the proprietary software and we did the prioritization of what we needed and for example you things like and checking out locally with very important for us we might
08:54
different priorities to be sure we developing and
08:59
so the the the other priorities only created
09:03
now I don't believe what I should of this in South Korea last year when we about start so we saw the offline there on the website we spoke to them and then you
09:18
know desired the basic characteristics of the 2 when we started you know it was created in 2014 for example and so called enterprise
09:31
and you data management you can do branch branch in and manage your branches when you check out checking and but changes accommodate different branches of then you contract the history of all the different changes I have made them to your database and right now it's not our integrated into repository and when we started it wasn't so you have to download a plug in and what we started and 1 of the challenges of which the tool when we started was the window and beats you could do a post postprocess and Bryant creation as there was some boxing right and also the dissipation light version was out of the there was no support for these and at the same time so that that means you couldn't do it you know you check out into spatial I tend to locality and also 1 of the main requirements I mean we've been talking about conflict resolution here but when you have like that then you know hundreds of people working on the view that people working on laptops creating edits it would if your software has cropped up conflict resolution strategy you need to manage your walk through to reduce the conflict so for example even if you have a conflict resolution and mechanism if you have 2 thousand conflicts you know when I'm going to finish resolving those conflicts I have to do this you know on a consistent basis so 1 of the is we normally at all to do this is the structure of the web by limited people to specific geographic areas and specific putrid so you do I attribute and geographic filters when you are creating your check out so people can only edit specific areas now these 2 also according to the right so we have to see them and down in gets out what we did develop developers to build this functionality into the into the 2 so looking at the
11:49
different problems we have these are the different fixed is that we did to add to the software so for example and we fixed the spatial light accounts and and now you can check out from more your posed serious database and check out into a spatial light and and do your editing merge branches back to depose GIS and database and when checking out you can do and geographic and filters on the on the GIS interface and create your check out into your spatial all into your post post yeah that these and then when you finish editing you can made all the different branches branches this so 1 of the other things that was also a you in which is also a very which is always a very very big challenge in the open source community is documentation right there was no proper documentation for this so and 1 of the things we did was to implement Sphinx and to do and the documentation I will shortly interface after these so on and then while we did all these works and did you I was also not very very intuitive and these are some
13:19
of the improvements we did need to and graphic user interface as you can see it and you can't see the difference differences when you create a check out and they're working on it and you can then merge these branches by and we also and improved the the GUI TQ GIS interface whereby when you want to create your own you check out to you you know you you point on the the gives you the right information of what you need to do and you know how and when doing the revision history we also did a lot of some improvement on the user interface to to make it more intuitive right and indeed
14:06
documentation so we implemented so we started from the it temporary documentation which is the the 1 we deployed on a local server and then move these over to most land to integrating the normal changes version of the
14:27
documentation so this is not a very very technical presentation but I cannot answer technical questions where I'm here to show was used to show that how there was indeed implementation community can walk with open source and developers to make improvements to open source software and you make community better because you need money right to creates tools for people to use and then you need the appropriate feedback on how people are using to to to create a you know an effective with so this a perfect example of how we work with the open source community to create products that are not very useful for the rest of the community based on use cases of how we envision open source software to to move from
15:26
an intense although enhancements and we are not there yet and destiny a lot of things and to be done right now with the software is the only thing you can do with offline editing and the spatial so if you create your and check out this week's post progressive you need to be connected to to the editing so you'll be good to be able to create a check out to my post Chris and just go line right and finished my work and manage the branches but so these need improvement also you have to create a technology spatial to a specific location when you move to location you know all of where you create general and specialized account it loses the connection and you can't answer merger branches but so these are the areas of for the improvements that we're looking into working on in the in the future that happened I mean we've seen the different presentations on on you know only in the previous president the conflict resolution user interface is very very intuitive so this is again something that needs to be improved so especially now that this is a tool that is readily integrated with you GIS that is going to be available to users to use it will be good to see it properly conflict management and you know
16:51
conflict resolution interface whereby you can't make intuitive decisions emerging and branches or doing you know choosing what geometry of edits to to speak and when the resulting your conflicts and indexing and these are the people that I
17:10
worked on the project and Eve is the development that we have from the open source community to to do this development and we were so what did the developments in and out all slander to objectives working yeah that's it thank you thank thank you so that was very rude the all 3 of
17:38
the 4 and the questions but the so it is that's going to overlap here with the Geoghegan what they're doing had and as right so things by developed to meet some of your kind of each development needs in the spirit of open source collaboration Jim is a case for merging the best that's about project sodium actually have different use cases as well and so while listening to the that's a good question while listening to and the dual presentation 1 of the block for us in adopting duties words because he was not integrated with the proper GIS clients you know like Q GIS and connecting directly to it database to manage your history so you have to use duty to do the is true which is a the which is the bottleneck because we're talking of where the network is slow for example you're going to have problems with this so and it will be good to see and and so this is again this is why I'm here right in this period of open source collaboration and it would be good to see these players work together that's what we just did is to support an existing work if we we're trying to create the product what we would have done was to for about product and then creating new product out of it but that's not what we after what we our users what we just 1 is tools that can help will solve our problem so it will be good I think there should be collaboration to answer questions and these will be be good to see this collaboration especially because you know did GIS community of adopted is to let the default within so should Judaized community facilitates slide there to work very closely with 2 and merge these together and then maybe they come up with business collaborations will support that's something we would like to see that does not say question it question other questions the and talking about the users I was wondering if now you changed for more close so fair to a more open source projects I'm wondering the users of what after any can you shows any experiences what how to do they like it's a healthy thing to think about this change into and you will probably for them but not the that would 2 questions so with every change in technology commons capacity building and and what we look at when doing this is the gains in the long run I mean so for example you pay developers to improve the tool you use money to train the people right to get then from with open source but in the long run the cost savings you get is different from paying expensive licensing costs onto the you consistently so the answer to your question is what we did during the development of the 2 was to start training all the users within the team on how to use q GIS on how to use Paul's GIS you know so that by the time you know the whole 2 was set they were comfortable to move over and if you notice what I said in the 1st place the 1st thing we did was to set up the 2 architectures together then move some people to open source while some people stay on you know the proprietary software the reason is because for every technology daddy any all right so the idea adult as the people that best is always best to choose your best people in proprietary are them all over to open source because they then crock everything that is needed within the open source so that's what we adopted and in that stands as the challenge you know for the rest of it and because they have to meet so eventually they moved over several more questions yes and no could you tell us a little bit about a timeline going and on and did it take you to do the transformation from the proprietary software to the open source solution of the good and so we started this project that are September last year right and we completed the development the dental became ready to use our own February in February March this year but very mind we had just to developers and to work this and 1 of them had the data that was working on it in at a later time while someone else was working on its own full-time and then there was a developer from was larger those working with the developers that we high so so roughly took about 6 months you know to have his work but in terms of the migration rates the migration and also to I would say about 6 months right because the 1st thing we did was to say it all and the architecture and then you know how people work on it incrementally ball right now in hindsight looking at it and I would say you can migrate and we have a fairly large database supporting units on the interventions in in different African countries so I would say you can move over i in like 3 months right both training your users having been used to solve them the the applications everything understand so the question yeah any other questions them and then give my best to connect you with the other speakers at that that think this was really a very good example of like the whole state of the art in good you can also little bit more about how often you have 2 thousand user whatever I think that you you're quite even if another use cases that they may not have to the a 3 and so on and this is brings me to the main reason why joined eHealth Africa and now because this problem even existed when we were using the proprietary software so you just create your checkouts right and the whole area and people are educated now the challenge that is someone does select all I'm just moves the features on which the book is change right and then when someone does the proper right so imagine that you now have people working in different states doing proper edits died then results in the in in conflicts right so how how do you manage that right so it would be proprietary software couldn't solve the problem when I joined the organization so what we did was to create walk-throughs right so sometimes well there may be a way to solve the technologically book I wouldn't so when I can solve something technologically then I decide you know it is not a technology problem then is a process of problems so if you can't change a technology thank you change of processes where again it would be good for you know for the technology developers to understand the processes of the users and structure that technology in such a way that it's you know makes life easier for the user or even you know what with the user closely and you know for example if you to this documentation now you can see that you should you can't geographic filters and use that to minimize conflicts right but would be good for things like this to be included even in the documentation activities and eventually did technology developed to solve the problems like this or what they can do very much group recently
