An Open Source Approach to Multi-user Distributed Geospatial Data Management
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 159 | |
| Anzahl der Teile | 193 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/20348 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSS4G Bonn 2016159 / 193
8
20
21
27
28
34
39
51
53
56
61
64
66
68
72
75
78
79
80
84
104
108
112
120
123
126
130
131
134
135
140
142
143
145
146
153
154
157
160
165
182
183
188
190
00:00
Gleitendes MittelOffene MengeDatenverwaltungOpen SourceSelbst organisierendes SystemGüte der AnpassungVorlesung/Konferenz
00:39
Spezielle unitäre GruppeSummierbarkeitZeitzoneNP-hartes ProblemAutomatische HandlungsplanungKette <Mathematik>VakuumFormale GrammatikOffice-PaketProjektive EbeneGrenzschichtablösungDifferenteFlächeninhaltPhysikalisches SystemEntscheidungstheorieNichtlinearer OperatorCall CenterMereologieMultiplikationInformation RetrievalSystemaufrufComputeranimation
01:39
Formale GrammatikSichtenkonzeptDateiformatBildschirmfensterRechenschieberElektronische PublikationEntscheidungstheorieATMLeistung <Physik>MereologieSoftwareDatenfeldDifferenteFlächeninhaltComputeranimation
02:13
RechenschieberHilfesystemDateiformatElektronische PublikationBildschirmfensterKundendatenbankArchitektur <Informatik>ElementargeometriePhysikalisches SystemSpezielle unitäre GruppeSummierbarkeitMultiplikationMobiles InternetMapping <Computergraphik>EntscheidungstheorieGeradeMultiplikationsoperatorSatellitensystemDatenbankInformationDatenverwaltungMehrplatzsystemComputeranimation
02:57
Explorative DatenanalyseRechenschieberSummierbarkeitDateiformatElementargeometrieArchitektur <Informatik>KundendatenbankElektronische PublikationPhysikalisches SystemMultiplikationEmulationSichtenkonzeptBildschirmfensterDatenreplikationStrategisches SpielDatenbankTypentheorieEndliche ModelltheorieMehrplatzsystemVersionsverwaltungServerDifferenteProjektive EbeneDatenverwaltungQuick-SortFortsetzung <Mathematik>SoftwareGruppenoperationPhysikalisches SystemHinterlegungsverfahren <Kryptologie>DatenbankMiddlewareComputeranimationFlussdiagramm
04:03
DateiformatSummierbarkeitSichtenkonzeptSpezielle unitäre GruppeFormale GrammatikZeiger <Informatik>HybridrechnerMetropolitan area networkBildschirmfensterOpen SourceGeradePhysikalisches SystemSoftwareComputeranimation
04:42
Architektur <Informatik>Open SourceHybridrechnerPhysikalisches SystemMultiplikationElektronische PublikationDateiformatBildschirmfensterWeitverkehrsnetzKundendatenbankElementargeometrieFormale GrammatikSichtenkonzeptInverser LimesSoftwareStichprobenumfangGraphische BenutzeroberflächeProzess <Informatik>SoftwareentwicklerOpen SourceDatenbankIdentifizierbarkeitA-posteriori-WahrscheinlichkeitPhysikalisches SystemGrenzschichtablösungComputeranimationFlussdiagramm
05:57
Spezielle unitäre GruppeOpen SourceKundendatenbankArchitektur <Informatik>ElementargeometrieZeiger <Informatik>Lokales MinimumSummierbarkeitPhysikalisches SystemFrequenzServerMultiplikationsoperatorKnickenObjekt <Kategorie>MathematikFlächeninhaltDatenbankClientHysteresekurveProzess <Informatik>ProgrammierumgebungNichtlinearer OperatorSoftwareentwicklerVersionsverwaltungDifferenteSicherungskopieInterface <Schaltung>Open SourceComputeranimation
07:36
Formale GrammatikArchitektur <Informatik>ElementargeometrieKundendatenbankOpen SourcePowerPointSichtenkonzeptDateiformatAchtSatellitensystemSoftwareentwicklerOpen SourceFreewareSoftwareReelle ZahlJSONComputeranimationVorlesung/Konferenz
08:25
FreewareOpen SourceSpezielle unitäre GruppeHilfesystemExplorative DatenanalyseAusdruck <Logik>Formale GrammatikSoftwareProgrammierumgebungMailing-Liste
08:54
SichtenkonzeptElektronische PublikationDateiformatBildschirmfensterHilfesystemFormale GrammatikRechenschieberDokumentenserverSpannweite <Stochastik>ServerDialektUnternehmensarchitekturElementargeometrieOpen SourceKundendatenbankApp <Programm>DatenbankVersionsverwaltungStandardabweichungDatenreplikationProgrammierumgebungUnternehmensarchitekturCharakteristisches PolynomKundendatenbankWeb SiteComputeranimationDiagramm
09:30
Spezielle unitäre GruppeDokumentenserverElementargeometrieKundendatenbankApp <Programm>Open SourceUnternehmensarchitekturDatenbankStandardabweichungDatenreplikationProgrammierumgebungSummierbarkeitSoftwaretestPlug inVersionsverwaltungTeilmengeMetropolitan area networkSichtenkonzeptFormale GrammatikPaarvergleichAlgebraisches ModellNeuronales NetzKonvexe HülleMehrwertnetzTrigonometrische FunktionLokales MinimumFunktionalVersionsverwaltungSoftwareAuflösung <Mathematik>Mechanismus-Design-TheorieVerzweigendes ProgrammDatenbankBasis <Mathematik>MathematikRechter WinkelDatenstrukturDissipationBenutzerbeteiligungFilter <Stochastik>SchwebungDokumentenserverInverser LimesUmwandlungsenthalpieFlächeninhaltBildschirmfensterNotebook-ComputerStellenringInterface <Schaltung>Plug inSichtenkonzeptDifferenteSoftwareentwicklerStrategisches SpielDatenverwaltungOpen SourceGraphische BenutzeroberflächeDatenfeldEigentliche AbbildungComputeranimation
13:17
DateiformatGraphische BenutzeroberflächeBildschirmfensterRechenschieberNichtlineares ZuordnungsproblemPlug inHilfesystemWeb-SeitePermanenteVersionsverwaltungBenutzeroberflächeMathematikInterface <Schaltung>ServerRechter WinkelDifferenteGraphische BenutzeroberflächeInformationNormalvektorVerzweigendes ProgrammComputeranimation
14:26
Web-SeiteSpezielle unitäre GruppePermanenteAttributierte GrammatikKonflikt <Informatik>Graphische BenutzeroberflächeDigitalfilterTeilmengeDesign by ContractGrenzschichtablösungInstantiierungSinusfunktionRechter WinkelVerzweigendes ProgrammRückkopplungURLCoxeter-GruppeProdukt <Mathematik>SoftwareCASE <Informatik>BenutzeroberflächeOpen SourceImplementierungEinfach zusammenhängender RaumVersionsverwaltungAuflösung <Mathematik>GeradeSoftwareentwicklerKundendatenbankFlächeninhaltInterface <Schaltung>Eigentliche AbbildungTermComputeranimation
16:50
Spezielle unitäre GruppeCOMSelbst organisierendes SystemE-MailSpeicherabzugNeuronales NetzAuflösung <Mathematik>ElementargeometrieVersionsverwaltungInterface <Schaltung>EntscheidungstheorieAutomatische IndexierungVerzweigendes ProgrammSoftwareentwicklerObjekt <Kategorie>Projektive EbeneOpen SourceSoftwareentwicklungVorlesung/KonferenzComputeranimation
17:38
Einfacher RingSoftwareentwicklerHyperbelverfahrenMathematikMultiplikationsoperatorKollaboration <Informatik>DatenbankVersionsverwaltungRechenschieberRechter WinkelDefaultAggregatzustandCoxeter-GruppeGruppenoperationResultanteSoftwareFlächeninhaltOpen SourceProjektive EbeneKanalkapazitätComputerarchitekturGebäude <Mathematik>CASE <Informatik>Abgeschlossene MengeKartesische KoordinatenProzess <Informatik>Wort <Informatik>ClientDifferenteTermSelbst organisierendes SystemMigration <Informatik>p-BlockRechenwerkBitTransformation <Mathematik>DatenstrukturProdukt <Mathematik>ComputerspielFilter <Stochastik>FrequenzWellenpaketLokales MinimumEigentliche AbbildungKundendatenbankGüte der AnpassungVorlesung/Konferenz
25:59
Computeranimation
Transkript: Englisch(automatisch erzeugt)
00:09
eHealth Africa, talking about an open source approach to multi-user distributed geospatial data management. So that seems to be the theme. Take it away.
00:25
Good afternoon, everyone. My name is Dami. I work for eHealth Africa. eHealth Africa is a nonprofit organization based in northern Nigeria with headquarters in Kano. We have
00:42
offices in Sierra Leone, Guinea, Liberia, and Nigeria, and we support public health interventions using data-driven approach in multiple African countries. So these are different areas where our projects are focused, health delivery systems, deploying call centers,
01:07
operating emergency operation centers, and supporting polio eradication, nutrition, severe acute malnutrition across different parts of West Africa.
01:27
Now, these different projects and interventions that we support require rapid data-driven decision-making, which means we deploy multiple dashboards to support decision-making. And the different
01:43
multiple dashboards, you know, to support near real-time decision-making require geographic information to power the decision-making. And we do a lot of, we have a lot of people that go to the field doing data collection in different parts of northern Nigeria, especially
02:07
in places with bad network in hard to reach areas. And this requires us deploying multiple mobile devices for offline data collection. And in the office, we have GIS specialists
02:25
that take these different data as they are synchronized near real-time or offline, you know, editing using satellite imagery and committing them to the geodatabase to make different maps at the same time, feeding the information into the geodatabase to support
02:49
decision-making. Now, the main thing with these is when we started, we started with proprietary software, whereby we deploy a multi-user data management system, as you
03:07
see here, a SQL server-based database with a middleware on top of it, and then different versions created to support multi-user editing. You know, you create your checkouts and do
03:23
your editing, commit within the database. Now, this worked for us, and we defined our workflow. But the main challenge with, and these are, you know, the different sort of workflows that we've been using with the proprietary software. Now, the main challenge
03:44
with this is it's really expensive. And for projects that we work on that require us to transition the data and the systems that we build to the government, sometimes it becomes not sustainable. So, we had to find a sustainable way of working, and we thought about
04:07
using an open source approach to doing this. The first thing we did along the line was to first use, so we wanted a situation whereby we can deploy proprietary software, because proprietary
04:25
software is always easy to deploy, especially in emergency situations. And we also wanted to have an open source, you know, system to continue to work. So, the first thing we did was
04:40
to run a hybrid system whereby we set up, we set up PostGIS and Postgres and set up MapServer and synchronized the two databases by running cron jobs and synchronizing them together using unique identifier. So, with the proprietary software, it generates a GUID, but what we've then
05:05
adopted in the open source software was to also use a UUID system and create them to match each other and keep them synchronized. But we then needed to take this further.
05:21
So, we looked into the solutions in the open source market. One of the things we looked at were the solutions which we already talked about here. It's good to see that these have really improved a lot of developments I've gone into it. So, we looked at GeoGeek, for example. We looked
05:44
at the PostPG versioning that was available. I mean, all of them have their different limitations. For our workflow, what we needed exactly was a client, you know, QGIS and being
06:02
able to do rigorous geoprocessing operations on a desktop, but at the same time synchronizing these changes back to the database. So, we eventually found a QGIS versioning tool that was already being deployed by Auslandia and we set this up and, as you can see, when we finished
06:26
setting up the versioning environment, this is what the proprietary looked like and this is what our open source environment looks like. So, whereby the user you check out
06:44
your data from a PostGIS database into a spatial light database to do local editing. And if you want to do connected editing, you check it out from PostGIS to PostGIS from the QGIS interface and then you do your editing synchronized back into the PostGIS system.
07:08
So, with the versioning setup between the different PostGIS systems online on the server, then you can do your QC and then synchronize back to the master database and then do your
07:22
backups all using PostGIS, QGIS and spatial light environments. But before we got here, we had to do a lot of development because the Auslandia tool was not in a fully ready state.
07:42
So, one of the things we did, we had to spend a lot of money and hired developers to do this. So, in Nigeria, for example, in Africa, sometimes people get the perception that when you say open source, it's free. I mean, but in the real sense, open source is not free. You have
08:03
developers who are committing their time, even if it's made available free, it's not actually free. So, we worked with Auslandia to improve on the tool. Our talk was through how we got from where we were in proprietary software to where we are
08:23
in the open source software. The first thing we needed to do was to look at our workflow in the proprietary environment. We went through the list of the different things that we normally do using the proprietary software. We did a prioritization of
08:44
what we needed. For example, things like checking out locally were very important for us. We marked different priorities to be sure we were developing. So, these are the other priorities
09:01
that we created. Now, I showed this in South Korea last year when we were about to start this project. So, we saw the Auslandia tool on their website. We spoke to them. These are the basic characteristics of the tool when we started. It was created in 2014,
09:27
for example. It supports enterprise geodata management. You can do branching and merge your branches when you check out and you are checking back your changes. You
09:41
can merge different branches and then you can track the history of the different changes that you've made to your database. Right now, it's now integrated into QGIS repository. When we started, it wasn't. So, you have to download the plugin when we started.
10:02
And one of the challenges with the tool when we started was even though you could do postgres to postgres branch creation, there were some bugs in it. And also, the spatialite version was
10:23
out of date. There was no support for this. At the same time, that means you couldn't do your checkout into spatialite and do local editing. And also, one of the main requirements, we've been talking about conflict resolution here, but when you have hundreds of people
10:45
working on the field and people working on laptops creating edits, even if your software has proper conflict resolution strategy, you need to manage your workflow to reduce the conflict. For example, even if you have conflict resolution mechanism, if you have 2,000 conflicts,
11:03
when are you going to finish resolving those conflicts? And you have to do this on a consistent basis. So, one of the approaches we normally adopt to do this is to structure our workflow whereby you limit people to specific geographic areas and specific
11:20
filtering. So, you do attribute and geographic filters when you are creating your checkout so people can only edit specific areas. Now, this tool also couldn't do this. So, we had to sit down and work with the developers to build these functionalities into the tool.
11:46
So, looking at the different problems we had with the tool, these are the different fixes that we did to the software. So, for example, we fixed the spatial light checkout and now you can
12:04
check out from your PostGIS database and check out into a spatial light and do your editing and merge the branches back to the PostGIS database. When checking out, you can do geographic
12:22
filters on the QGIS interface and create your checkout into your spatial light or into your PostGIS database. And then when you finish editing, you can merge all the different branches branches back. So, one of the other things that was also a huge challenge, which is also a
12:47
which is always a very, very big challenge in the open source community is documentation. There was no proper documentation for this tool and one of the things we did was to implement
13:02
Sphinx to do the documentation. I will show the interface after this. So, and then while we did all this work, the GUI was also not very, very intuitive and these are some of the improvements we did to the graphic user interface. As you can see, you can see the
13:28
differences when you create your checkout and you're working on it and you can then merge these branches back. We also improved the GUI on the QGIS interface whereby when you want to
13:44
create your checkout, you point on it. It gives you the right information of what you need to do and when doing the revision history, we also did a lot of improvement on the user interface
14:02
to make it more intuitive, right? And then the documentation. So, we implemented, so we started from a temporary documentation, which is the one we deployed on our local server and then moved these over to Auslandia to integrate in the normal QGIS versioning documentation.
14:28
So, this is not a very, very technical presentation, but I can answer technical questions. What I'm here to show us is to show how people in the implementation community can work with
14:47
open source developers to make improvements to open source software and, you know, make the community better because you need money, right, to create tools for people to use and then you
15:01
need appropriate feedback on how people are using to create, you know, an effective tool. So, this is a perfect example of how we work with the open source community to create products that are very useful for the rest of the community based on use cases of how we envision
15:23
open source software to move on. And in terms of enhancement, we are not there yet. There are still a lot of things to be done. Right now, with this software, the only thing you can do with offline editing is specialized. So, if you create your checkout with Postgres,
15:46
you need to be connected to do the editing. So, it will be good to be able to create a checkout to my Postgres and just go offline, right, and finish my work and merge the branches back. So, these need improvement. Also, you have to create your checkout in specialized to
16:04
a specific location. When you move the location, you know, of where you created your specialized checkout, it loses the connection and you can't merge your branches back. So, these are areas of further improvement that we are looking into working on in the future. The other thing,
16:25
I mean, we've seen the different presentations on, you know, in the previous presentations, the conflict resolution user interface is very, very intuitive. So, this is again something that needs to be improved in this tool, especially now that this is a tool that is readily
16:41
integrated with QGIS that is going to be available to users to use. It will be good to see a proper conflict management, you know, conflict resolution interface whereby you can make intuitive decisions in merging branches or doing, you know, choosing what geometry
17:01
edits to pick when you're resolving your conflicts. And that's it. These are the people that worked on the project. Eve is the developer that we hired from the open source community to do this development and we also worked with the development team at Auslandia to get this
17:24
working. Yeah, that's it. Thank you. Thanks, Amy. That was great and a real compliment to the whole program here. Questions? So, it seems like there's quite an overlap here
17:47
with GeoGig and what they're doing and it sounds like some of the things that they've developed meet some of your kind of future development needs. In the spirit of open source collaboration, do you think there's a case for merging the best bits of both projects or do you think they
18:01
actually have different use cases? Well, so while listening to the, that's a good question, while listening to the GeoGig presentation, one of the blockers for us in adopting GeoGig was because it was not integrated with a proper GIS client, you know, like QGIS and connecting,
18:24
you know, directly to a database to manage your history. So, you have to use GeoGig to do the whole history, which is a bottleneck because when you are talking of where the network is slow, for example, you are going to have problems with this. So, it will be good to see,
18:43
I mean, so this is, again, this is why I'm here, right? In the spirit of open source collaboration, it would be good to see these players work together because what we just did is to support an existing work. If we were trying to create a product, what we would have done was to fork that product and then create a new product out of it, but that's not what we are after.
19:03
What we are users, what we just want, is tools that can help us solve our problems. So, it will be good. I think there should be collaboration to answer your question, and it will be good to see this collaboration, especially because, you know, the QGIS community
19:23
have adopted this tool as a default tool within QGIS. So, should QGIS community facilitate Oslandia to work very closely with GeoGig and merge this together and then maybe they come up with business collaborations for support? That's something we would like to see.
19:42
Does that answer your question? Question? Other questions? You're talking about the users. I was wondering, now you changed from more closed software to a more open source project. So, I'm wondering, the users, can you share any experiences?
20:03
How do they like it? How do they think about this change into a new world probably for them? That's another good question. So, with every change in technology comes capacity building, and what we look at when doing this is the gains in the long run. So, for example,
20:25
you pay developers to improve the tool. You use money to train the people right, to get them familiar with open source, but on the long run, the cost savings you get is different from paying expensive licensing costs, you know, consistently. So, the answer to your question is,
20:48
what we did during the development of this tool was to start training all the users within the team on how to use QGIS, on how to use PostGIS comfortably. So, by the time the whole tool was
21:10
set up, they were comfortable to move over, and if you notice what I said in the first place, the first thing we did was to set up the two architectures together, then move some people to
21:22
open source while some people stay on, you know, the proprietary software. The reason is because for every technology, they are the early adopters, right? So, the early adopters, the people that are best is always best to choose your best people in proprietary and move them over to open source because they then crack everything that is needed within the
21:42
open source. So, that's what we adopted, and then that then stands as a challenge, you know, for the rest of the team because they have to meet up. So, eventually they moved over. Thanks. Time for one more question. Yeah.
22:01
Could you tell us a little bit about your timeline? I mean, how long did it take you to do the transformation from the proprietary software to the open source solution? Okay, good. So, we started this project around September last year, right? And we completed the development. The tool became ready to use around February March this year.
22:31
But bear in mind, we hired just two developers to work on this. One of them had a day job and was working on it, you know, at a later time while someone else was working on it full-time.
22:43
And then there was a developer from Westlandia that was working with the developers that we hired. So, roughly it took about six months, you know, to have this work. But in terms of the migration, right, the migration also took, I would say, about six months,
23:02
right? Because the first thing we did was to set up the architecture and then, you know, have people work on it incrementally. But right now, in hindsight, looking at it, I would say you can migrate. I mean, we have a fairly large database supporting, you know,
23:22
the interventions in different African countries. So, I would say you can move over in like three months, right? But training your users and having them use the applications every day. Does that answer the question? Yeah. Any other questions?
23:46
Dami, I'm going to do my best to connect you with the other speakers. I think this was really a very good example of like the whole state of the art. Can you tell us a little bit more about how often you have 2,000 users or whatever? I think that you are probably bringing
24:03
some other use cases that they may not have too. That's exciting. Okay. So, I mean, this brings me to the main reason why I joined eHealth Africa, because this problem even existed when we were using the proprietary software. So, you just create your checkouts, right? And the whole area and people are editing.
24:24
Now, the challenge with that is if someone does a select all and just moves the features and moves it back, it has changed, right? And then when someone does a proper edit, right? So, imagine then you now have people working in different states doing proper edits,
24:41
that then results in conflicts, right? So, how do you manage that, right? So, even the proprietary software couldn't solve that problem when I joined the organization. So, what we did was to create workflows, right? So, sometimes, well, there may be a way to
25:01
solve it technologically, but I couldn't. So, when I can't solve something technologically, then I decided if it's not a technology problem, then it's a process problem. So, if you can't change the technology, then you change your processes. But again, it would be good for the technology developers to understand the processes of the users
25:22
and structure the technology in such a way that it makes life easy for the users, or even work with the users closely. And for example, if you go to this documentation now, you can see that you can do geographic filters and use that to minimize conflicts, right?
25:42
But it would be good for things like this to be included even in the documentation at the list, and eventually with the technology developed to solve problems like this. All right. Well, thank you very much. Great presentation. Thank you very much.