We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

An Open Source Approach to Multi-user Distributed Geospatial Data Management

00:00

Formale Metadaten

Titel
An Open Source Approach to Multi-user Distributed Geospatial Data Management
Serientitel
Teil
159
Anzahl der Teile
193
Autor
Lizenz
CC-Namensnennung 3.0 Deutschland:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Open source tools have been successful in managing geospatial data in central data stores. However, performance issues can arise from many users accessing the same table in a geospatial database at once, especially in a multi-user editing environment. The geospatial landscape also changes constantly, as a result of human activity and natural forces, this gives a need to track these changes within the geospatial database and perform change detection activities to understand changes across time, hence a need to version history. These use cases springs up the requirements to employ a data distribution across multiple geospatial databases using versioning and replication technology to integrate several desktop and mobile user applications into an adaptive geospatial communications environment connecting operations across the enterprise and throughout the organisations to improve data availability to multiple users, tracking change history within multiple table versions while increasing system performance. Several commercial geospatial applications have successfully implemented full versioning replication capabilities by leveraging middleware with the core database versioning capabilities – for example ArcSDE technology from Esri . The realization of a full solution has been far-fetched on open source geospatial applications. This presentation discusses the development of QGIS plugin and implementation of FOSS4G solutions for versioning and replication capabilities with to support multi-user access while optimizing performance using QGIS, PostgreSQL, PostGIS and SpatialLite DB technologies.
78
Vorschaubild
51:51
154
Vorschaubild
35:04
Gleitendes MittelOffene MengeDatenverwaltungOpen SourceSelbst organisierendes SystemGüte der AnpassungVorlesung/Konferenz
Spezielle unitäre GruppeSummierbarkeitZeitzoneNP-hartes ProblemAutomatische HandlungsplanungKette <Mathematik>VakuumFormale GrammatikOffice-PaketProjektive EbeneGrenzschichtablösungDifferenteFlächeninhaltPhysikalisches SystemEntscheidungstheorieNichtlinearer OperatorCall CenterMereologieMultiplikationInformation RetrievalSystemaufrufComputeranimation
Formale GrammatikSichtenkonzeptDateiformatBildschirmfensterRechenschieberElektronische PublikationEntscheidungstheorieATMLeistung <Physik>MereologieSoftwareDatenfeldDifferenteFlächeninhaltComputeranimation
RechenschieberHilfesystemDateiformatElektronische PublikationBildschirmfensterKundendatenbankArchitektur <Informatik>ElementargeometriePhysikalisches SystemSpezielle unitäre GruppeSummierbarkeitMultiplikationMobiles InternetMapping <Computergraphik>EntscheidungstheorieGeradeMultiplikationsoperatorSatellitensystemDatenbankInformationDatenverwaltungMehrplatzsystemComputeranimation
Explorative DatenanalyseRechenschieberSummierbarkeitDateiformatElementargeometrieArchitektur <Informatik>KundendatenbankElektronische PublikationPhysikalisches SystemMultiplikationEmulationSichtenkonzeptBildschirmfensterDatenreplikationStrategisches SpielDatenbankTypentheorieEndliche ModelltheorieMehrplatzsystemVersionsverwaltungServerDifferenteProjektive EbeneDatenverwaltungQuick-SortFortsetzung <Mathematik>SoftwareGruppenoperationPhysikalisches SystemHinterlegungsverfahren <Kryptologie>DatenbankMiddlewareComputeranimationFlussdiagramm
DateiformatSummierbarkeitSichtenkonzeptSpezielle unitäre GruppeFormale GrammatikZeiger <Informatik>HybridrechnerMetropolitan area networkBildschirmfensterOpen SourceGeradePhysikalisches SystemSoftwareComputeranimation
Architektur <Informatik>Open SourceHybridrechnerPhysikalisches SystemMultiplikationElektronische PublikationDateiformatBildschirmfensterWeitverkehrsnetzKundendatenbankElementargeometrieFormale GrammatikSichtenkonzeptInverser LimesSoftwareStichprobenumfangGraphische BenutzeroberflächeProzess <Informatik>SoftwareentwicklerOpen SourceDatenbankIdentifizierbarkeitA-posteriori-WahrscheinlichkeitPhysikalisches SystemGrenzschichtablösungComputeranimationFlussdiagramm
Spezielle unitäre GruppeOpen SourceKundendatenbankArchitektur <Informatik>ElementargeometrieZeiger <Informatik>Lokales MinimumSummierbarkeitPhysikalisches SystemFrequenzServerMultiplikationsoperatorKnickenObjekt <Kategorie>MathematikFlächeninhaltDatenbankClientHysteresekurveProzess <Informatik>ProgrammierumgebungNichtlinearer OperatorSoftwareentwicklerVersionsverwaltungDifferenteSicherungskopieInterface <Schaltung>Open SourceComputeranimation
Formale GrammatikArchitektur <Informatik>ElementargeometrieKundendatenbankOpen SourcePowerPointSichtenkonzeptDateiformatAchtSatellitensystemSoftwareentwicklerOpen SourceFreewareSoftwareReelle ZahlJSONComputeranimationVorlesung/Konferenz
FreewareOpen SourceSpezielle unitäre GruppeHilfesystemExplorative DatenanalyseAusdruck <Logik>Formale GrammatikSoftwareProgrammierumgebungMailing-Liste
SichtenkonzeptElektronische PublikationDateiformatBildschirmfensterHilfesystemFormale GrammatikRechenschieberDokumentenserverSpannweite <Stochastik>ServerDialektUnternehmensarchitekturElementargeometrieOpen SourceKundendatenbankApp <Programm>DatenbankVersionsverwaltungStandardabweichungDatenreplikationProgrammierumgebungUnternehmensarchitekturCharakteristisches PolynomKundendatenbankWeb SiteComputeranimationDiagramm
Spezielle unitäre GruppeDokumentenserverElementargeometrieKundendatenbankApp <Programm>Open SourceUnternehmensarchitekturDatenbankStandardabweichungDatenreplikationProgrammierumgebungSummierbarkeitSoftwaretestPlug inVersionsverwaltungTeilmengeMetropolitan area networkSichtenkonzeptFormale GrammatikPaarvergleichAlgebraisches ModellNeuronales NetzKonvexe HülleMehrwertnetzTrigonometrische FunktionLokales MinimumFunktionalVersionsverwaltungSoftwareAuflösung <Mathematik>Mechanismus-Design-TheorieVerzweigendes ProgrammDatenbankBasis <Mathematik>MathematikRechter WinkelDatenstrukturDissipationBenutzerbeteiligungFilter <Stochastik>SchwebungDokumentenserverInverser LimesUmwandlungsenthalpieFlächeninhaltBildschirmfensterNotebook-ComputerStellenringInterface <Schaltung>Plug inSichtenkonzeptDifferenteSoftwareentwicklerStrategisches SpielDatenverwaltungOpen SourceGraphische BenutzeroberflächeDatenfeldEigentliche AbbildungComputeranimation
DateiformatGraphische BenutzeroberflächeBildschirmfensterRechenschieberNichtlineares ZuordnungsproblemPlug inHilfesystemWeb-SeitePermanenteVersionsverwaltungBenutzeroberflächeMathematikInterface <Schaltung>ServerRechter WinkelDifferenteGraphische BenutzeroberflächeInformationNormalvektorVerzweigendes ProgrammComputeranimation
Web-SeiteSpezielle unitäre GruppePermanenteAttributierte GrammatikKonflikt <Informatik>Graphische BenutzeroberflächeDigitalfilterTeilmengeDesign by ContractGrenzschichtablösungInstantiierungSinusfunktionRechter WinkelVerzweigendes ProgrammRückkopplungURLCoxeter-GruppeProdukt <Mathematik>SoftwareCASE <Informatik>BenutzeroberflächeOpen SourceImplementierungEinfach zusammenhängender RaumVersionsverwaltungAuflösung <Mathematik>GeradeSoftwareentwicklerKundendatenbankFlächeninhaltInterface <Schaltung>Eigentliche AbbildungTermComputeranimation
Spezielle unitäre GruppeCOMSelbst organisierendes SystemE-MailSpeicherabzugNeuronales NetzAuflösung <Mathematik>ElementargeometrieVersionsverwaltungInterface <Schaltung>EntscheidungstheorieAutomatische IndexierungVerzweigendes ProgrammSoftwareentwicklerObjekt <Kategorie>Projektive EbeneOpen SourceSoftwareentwicklungVorlesung/KonferenzComputeranimation
Einfacher RingSoftwareentwicklerHyperbelverfahrenMathematikMultiplikationsoperatorKollaboration <Informatik>DatenbankVersionsverwaltungRechenschieberRechter WinkelDefaultAggregatzustandCoxeter-GruppeGruppenoperationResultanteSoftwareFlächeninhaltOpen SourceProjektive EbeneKanalkapazitätComputerarchitekturGebäude <Mathematik>CASE <Informatik>Abgeschlossene MengeKartesische KoordinatenProzess <Informatik>Wort <Informatik>ClientDifferenteTermSelbst organisierendes SystemMigration <Informatik>p-BlockRechenwerkBitTransformation <Mathematik>DatenstrukturProdukt <Mathematik>ComputerspielFilter <Stochastik>FrequenzWellenpaketLokales MinimumEigentliche AbbildungKundendatenbankGüte der AnpassungVorlesung/Konferenz
Computeranimation
Transkript: Englisch(automatisch erzeugt)
eHealth Africa, talking about an open source approach to multi-user distributed geospatial data management. So that seems to be the theme. Take it away.
Good afternoon, everyone. My name is Dami. I work for eHealth Africa. eHealth Africa is a nonprofit organization based in northern Nigeria with headquarters in Kano. We have
offices in Sierra Leone, Guinea, Liberia, and Nigeria, and we support public health interventions using data-driven approach in multiple African countries. So these are different areas where our projects are focused, health delivery systems, deploying call centers,
operating emergency operation centers, and supporting polio eradication, nutrition, severe acute malnutrition across different parts of West Africa.
Now, these different projects and interventions that we support require rapid data-driven decision-making, which means we deploy multiple dashboards to support decision-making. And the different
multiple dashboards, you know, to support near real-time decision-making require geographic information to power the decision-making. And we do a lot of, we have a lot of people that go to the field doing data collection in different parts of northern Nigeria, especially
in places with bad network in hard to reach areas. And this requires us deploying multiple mobile devices for offline data collection. And in the office, we have GIS specialists
that take these different data as they are synchronized near real-time or offline, you know, editing using satellite imagery and committing them to the geodatabase to make different maps at the same time, feeding the information into the geodatabase to support
decision-making. Now, the main thing with these is when we started, we started with proprietary software, whereby we deploy a multi-user data management system, as you
see here, a SQL server-based database with a middleware on top of it, and then different versions created to support multi-user editing. You know, you create your checkouts and do
your editing, commit within the database. Now, this worked for us, and we defined our workflow. But the main challenge with, and these are, you know, the different sort of workflows that we've been using with the proprietary software. Now, the main challenge
with this is it's really expensive. And for projects that we work on that require us to transition the data and the systems that we build to the government, sometimes it becomes not sustainable. So, we had to find a sustainable way of working, and we thought about
using an open source approach to doing this. The first thing we did along the line was to first use, so we wanted a situation whereby we can deploy proprietary software, because proprietary
software is always easy to deploy, especially in emergency situations. And we also wanted to have an open source, you know, system to continue to work. So, the first thing we did was
to run a hybrid system whereby we set up, we set up PostGIS and Postgres and set up MapServer and synchronized the two databases by running cron jobs and synchronizing them together using unique identifier. So, with the proprietary software, it generates a GUID, but what we've then
adopted in the open source software was to also use a UUID system and create them to match each other and keep them synchronized. But we then needed to take this further.
So, we looked into the solutions in the open source market. One of the things we looked at were the solutions which we already talked about here. It's good to see that these have really improved a lot of developments I've gone into it. So, we looked at GeoGeek, for example. We looked
at the PostPG versioning that was available. I mean, all of them have their different limitations. For our workflow, what we needed exactly was a client, you know, QGIS and being
able to do rigorous geoprocessing operations on a desktop, but at the same time synchronizing these changes back to the database. So, we eventually found a QGIS versioning tool that was already being deployed by Auslandia and we set this up and, as you can see, when we finished
setting up the versioning environment, this is what the proprietary looked like and this is what our open source environment looks like. So, whereby the user you check out
your data from a PostGIS database into a spatial light database to do local editing. And if you want to do connected editing, you check it out from PostGIS to PostGIS from the QGIS interface and then you do your editing synchronized back into the PostGIS system.
So, with the versioning setup between the different PostGIS systems online on the server, then you can do your QC and then synchronize back to the master database and then do your
backups all using PostGIS, QGIS and spatial light environments. But before we got here, we had to do a lot of development because the Auslandia tool was not in a fully ready state.
So, one of the things we did, we had to spend a lot of money and hired developers to do this. So, in Nigeria, for example, in Africa, sometimes people get the perception that when you say open source, it's free. I mean, but in the real sense, open source is not free. You have
developers who are committing their time, even if it's made available free, it's not actually free. So, we worked with Auslandia to improve on the tool. Our talk was through how we got from where we were in proprietary software to where we are
in the open source software. The first thing we needed to do was to look at our workflow in the proprietary environment. We went through the list of the different things that we normally do using the proprietary software. We did a prioritization of
what we needed. For example, things like checking out locally were very important for us. We marked different priorities to be sure we were developing. So, these are the other priorities
that we created. Now, I showed this in South Korea last year when we were about to start this project. So, we saw the Auslandia tool on their website. We spoke to them. These are the basic characteristics of the tool when we started. It was created in 2014,
for example. It supports enterprise geodata management. You can do branching and merge your branches when you check out and you are checking back your changes. You
can merge different branches and then you can track the history of the different changes that you've made to your database. Right now, it's now integrated into QGIS repository. When we started, it wasn't. So, you have to download the plugin when we started.
And one of the challenges with the tool when we started was even though you could do postgres to postgres branch creation, there were some bugs in it. And also, the spatialite version was
out of date. There was no support for this. At the same time, that means you couldn't do your checkout into spatialite and do local editing. And also, one of the main requirements, we've been talking about conflict resolution here, but when you have hundreds of people
working on the field and people working on laptops creating edits, even if your software has proper conflict resolution strategy, you need to manage your workflow to reduce the conflict. For example, even if you have conflict resolution mechanism, if you have 2,000 conflicts,
when are you going to finish resolving those conflicts? And you have to do this on a consistent basis. So, one of the approaches we normally adopt to do this is to structure our workflow whereby you limit people to specific geographic areas and specific
filtering. So, you do attribute and geographic filters when you are creating your checkout so people can only edit specific areas. Now, this tool also couldn't do this. So, we had to sit down and work with the developers to build these functionalities into the tool.
So, looking at the different problems we had with the tool, these are the different fixes that we did to the software. So, for example, we fixed the spatial light checkout and now you can
check out from your PostGIS database and check out into a spatial light and do your editing and merge the branches back to the PostGIS database. When checking out, you can do geographic
filters on the QGIS interface and create your checkout into your spatial light or into your PostGIS database. And then when you finish editing, you can merge all the different branches branches back. So, one of the other things that was also a huge challenge, which is also a
which is always a very, very big challenge in the open source community is documentation. There was no proper documentation for this tool and one of the things we did was to implement
Sphinx to do the documentation. I will show the interface after this. So, and then while we did all this work, the GUI was also not very, very intuitive and these are some of the improvements we did to the graphic user interface. As you can see, you can see the
differences when you create your checkout and you're working on it and you can then merge these branches back. We also improved the GUI on the QGIS interface whereby when you want to
create your checkout, you point on it. It gives you the right information of what you need to do and when doing the revision history, we also did a lot of improvement on the user interface
to make it more intuitive, right? And then the documentation. So, we implemented, so we started from a temporary documentation, which is the one we deployed on our local server and then moved these over to Auslandia to integrate in the normal QGIS versioning documentation.
So, this is not a very, very technical presentation, but I can answer technical questions. What I'm here to show us is to show how people in the implementation community can work with
open source developers to make improvements to open source software and, you know, make the community better because you need money, right, to create tools for people to use and then you
need appropriate feedback on how people are using to create, you know, an effective tool. So, this is a perfect example of how we work with the open source community to create products that are very useful for the rest of the community based on use cases of how we envision
open source software to move on. And in terms of enhancement, we are not there yet. There are still a lot of things to be done. Right now, with this software, the only thing you can do with offline editing is specialized. So, if you create your checkout with Postgres,
you need to be connected to do the editing. So, it will be good to be able to create a checkout to my Postgres and just go offline, right, and finish my work and merge the branches back. So, these need improvement. Also, you have to create your checkout in specialized to
a specific location. When you move the location, you know, of where you created your specialized checkout, it loses the connection and you can't merge your branches back. So, these are areas of further improvement that we are looking into working on in the future. The other thing,
I mean, we've seen the different presentations on, you know, in the previous presentations, the conflict resolution user interface is very, very intuitive. So, this is again something that needs to be improved in this tool, especially now that this is a tool that is readily
integrated with QGIS that is going to be available to users to use. It will be good to see a proper conflict management, you know, conflict resolution interface whereby you can make intuitive decisions in merging branches or doing, you know, choosing what geometry
edits to pick when you're resolving your conflicts. And that's it. These are the people that worked on the project. Eve is the developer that we hired from the open source community to do this development and we also worked with the development team at Auslandia to get this
working. Yeah, that's it. Thank you. Thanks, Amy. That was great and a real compliment to the whole program here. Questions? So, it seems like there's quite an overlap here
with GeoGig and what they're doing and it sounds like some of the things that they've developed meet some of your kind of future development needs. In the spirit of open source collaboration, do you think there's a case for merging the best bits of both projects or do you think they
actually have different use cases? Well, so while listening to the, that's a good question, while listening to the GeoGig presentation, one of the blockers for us in adopting GeoGig was because it was not integrated with a proper GIS client, you know, like QGIS and connecting,
you know, directly to a database to manage your history. So, you have to use GeoGig to do the whole history, which is a bottleneck because when you are talking of where the network is slow, for example, you are going to have problems with this. So, it will be good to see,
I mean, so this is, again, this is why I'm here, right? In the spirit of open source collaboration, it would be good to see these players work together because what we just did is to support an existing work. If we were trying to create a product, what we would have done was to fork that product and then create a new product out of it, but that's not what we are after.
What we are users, what we just want, is tools that can help us solve our problems. So, it will be good. I think there should be collaboration to answer your question, and it will be good to see this collaboration, especially because, you know, the QGIS community
have adopted this tool as a default tool within QGIS. So, should QGIS community facilitate Oslandia to work very closely with GeoGig and merge this together and then maybe they come up with business collaborations for support? That's something we would like to see.
Does that answer your question? Question? Other questions? You're talking about the users. I was wondering, now you changed from more closed software to a more open source project. So, I'm wondering, the users, can you share any experiences?
How do they like it? How do they think about this change into a new world probably for them? That's another good question. So, with every change in technology comes capacity building, and what we look at when doing this is the gains in the long run. So, for example,
you pay developers to improve the tool. You use money to train the people right, to get them familiar with open source, but on the long run, the cost savings you get is different from paying expensive licensing costs, you know, consistently. So, the answer to your question is,
what we did during the development of this tool was to start training all the users within the team on how to use QGIS, on how to use PostGIS comfortably. So, by the time the whole tool was
set up, they were comfortable to move over, and if you notice what I said in the first place, the first thing we did was to set up the two architectures together, then move some people to
open source while some people stay on, you know, the proprietary software. The reason is because for every technology, they are the early adopters, right? So, the early adopters, the people that are best is always best to choose your best people in proprietary and move them over to open source because they then crack everything that is needed within the
open source. So, that's what we adopted, and then that then stands as a challenge, you know, for the rest of the team because they have to meet up. So, eventually they moved over. Thanks. Time for one more question. Yeah.
Could you tell us a little bit about your timeline? I mean, how long did it take you to do the transformation from the proprietary software to the open source solution? Okay, good. So, we started this project around September last year, right? And we completed the development. The tool became ready to use around February March this year.
But bear in mind, we hired just two developers to work on this. One of them had a day job and was working on it, you know, at a later time while someone else was working on it full-time.
And then there was a developer from Westlandia that was working with the developers that we hired. So, roughly it took about six months, you know, to have this work. But in terms of the migration, right, the migration also took, I would say, about six months,
right? Because the first thing we did was to set up the architecture and then, you know, have people work on it incrementally. But right now, in hindsight, looking at it, I would say you can migrate. I mean, we have a fairly large database supporting, you know,
the interventions in different African countries. So, I would say you can move over in like three months, right? But training your users and having them use the applications every day. Does that answer the question? Yeah. Any other questions?
Dami, I'm going to do my best to connect you with the other speakers. I think this was really a very good example of like the whole state of the art. Can you tell us a little bit more about how often you have 2,000 users or whatever? I think that you are probably bringing
some other use cases that they may not have too. That's exciting. Okay. So, I mean, this brings me to the main reason why I joined eHealth Africa, because this problem even existed when we were using the proprietary software. So, you just create your checkouts, right? And the whole area and people are editing.
Now, the challenge with that is if someone does a select all and just moves the features and moves it back, it has changed, right? And then when someone does a proper edit, right? So, imagine then you now have people working in different states doing proper edits,
that then results in conflicts, right? So, how do you manage that, right? So, even the proprietary software couldn't solve that problem when I joined the organization. So, what we did was to create workflows, right? So, sometimes, well, there may be a way to
solve it technologically, but I couldn't. So, when I can't solve something technologically, then I decided if it's not a technology problem, then it's a process problem. So, if you can't change the technology, then you change your processes. But again, it would be good for the technology developers to understand the processes of the users
and structure the technology in such a way that it makes life easy for the users, or even work with the users closely. And for example, if you go to this documentation now, you can see that you can do geographic filters and use that to minimize conflicts, right?
But it would be good for things like this to be included even in the documentation at the list, and eventually with the technology developed to solve problems like this. All right. Well, thank you very much. Great presentation. Thank you very much.