Mapping Green goes Open with the GIS Collective
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43366 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019217 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Green's functionTexture mappingOpen setGreen's functionComputer virusProjective planeMobile appPlanningOpen setLevel (video gaming)Game theoryPresentation of a groupComputer animation
00:43
Green's functionWebsiteNeighbourhood (graph theory)WebsiteLevel (video gaming)Field (computer science)Point (geometry)XMLComputer animation
01:10
WebsiteComputing platformServer (computing)Computer iconAttribute grammarMaxima and minimaDecimalConvex hullComa BerenicesJust-in-Time-CompilerOpen setGreen's functionSelf-organizationCombinational logicLibrary (computing)Set (mathematics)Web pageMultiplication signSoftware bugDampingMobile appComputer iconFormal languagePlug-in (computing)Computing platformAttribute grammarFreewarePoint (geometry)Level (video gaming)DebuggerQuery languageInstance (computer science)ResultantSoftware frameworkCloud computingProgramming languageOrder (biology)Differential operator1 (number)Cycle (graph theory)Right angleWebsiteData storage deviceExpert systemService (economics)Software developerShape (magazine)Labour Party (Malta)Form (programming)Server (computing)Web 2.0Computer architectureLengthView (database)Computer programmingHome pageMappingComputer animation
07:00
ArchitectureComputer fileWeb 2.0Fluid staticsRepresentational state transferTask (computing)Message passing2 (number)Computer animation
07:20
ArchitectureComputer programmingTask (computing)StatisticsCodeFluid staticsElectric generatorTask (computing)MathematicsWebsiteNumberMessage passingElectric generatorRepresentational state transferMetadataPoint (geometry)Computer programmingConsistencySpecial unitary groupProcess (computing)Ocean currentLevel (video gaming)DatabaseLecture/ConferenceComputer animation
08:47
Computer programmingElectric generatorTask (computing)Fluid staticsCodeWebsitePoint (geometry)Computer iconString (computer science)Field (computer science)AliasingTemplate (C++)Data modelAbelian categoryTelephone number mappingDependent and independent variablesRouter (computing)Social classInformationDigital filterHash functionDatabaseAuthenticationElectronic mailing listLibrary (computing)Uniqueness quantificationQuery languageSubsetCache (computing)Vector spaceType theoryRun time (program lifecycle phase)Computer programmingChord (peer-to-peer)Template (C++)Data structureCodeLine (geometry)Core dumpMetreObject (grammar)String (computer science)Functional (mathematics)Group actionMobile appOrder (biology)WebsiteCivil engineeringServer (computing)Observational studyLabour Party (Malta)Social classInstance (computer science)Programming languageRow (database)Performance appraisalMultiplication signDatabaseView (database)CASE <Informatik>Level (video gaming)Validity (statistics)Data storage deviceCuboidCategory of beingRoutingElectric generatorCompilation albumSerial portUsabilityAsynchronous Transfer ModeVector spaceEndliche ModelltheorieParameter (computer programming)TesselationAttribute grammarFluid staticsOpen setMixed realityComputer animation
12:27
Vector spaceLibrary (computing)Electric generatorLibrary (computing)Open sourcePhysical systemTemplate (C++)Order (biology)DatabaseNumberTouchscreenElectric generatorSemiconductor memoryWordLine (geometry)Query languageMetric systemRepresentational state transferCountingMobile appComputer iconComputer animation
13:26
Metric systemNumberWebsiteCountingComputer animationEngineering drawingDiagram
13:43
Mobile appMobile WebForestGame theoryPhysical systemPlanningComputer animation
14:19
Mobile WebPolygonLine (geometry)Web pageOpen setDisintegrationIntelComputer programmingData managementGamma functionINTEGRALMappingOpen sourceForm (programming)Order (biology)Medical imagingField (computer science)Server (computing)Type theoryAutomatic programmingComputer architectureInstance (computer science)Computer programmingMobile appService (economics)MathematicsTesselationAttribute grammarConsistencyData managementWeb pageWebsiteCloud computingLine (geometry)Point cloudComputer iconPolygonRight angleVirtual machinePresentation of a groupNoise (electronics)Multiplication signSoftware developerUniform resource locatorProjective planePlanningFood energyFigurate numberLevel (video gaming)Open setGame theoryComputer wormBootingProduct (business)Physical systemRoundness (object)Disk read-and-write headCodeComputer animation
Transcript: English(auto-generated)
00:09
We prepare ourselves for the next presenter, Bogdan. He will be talking about Mapping Green, goes open with the GIS Collective.
00:24
Hello. I'm Bogdan and I'm here to show you how this collective works for Open Green Map. I will talk about the project overview, some interesting technical details, and what are my future plans with this app.
00:42
But first, who didn't heard about Green Map? Yeah, I guess everyone was at the previous talk. In one phrase, Green Map is a community around the world who gather data points around sustainable living
01:00
in their neighborhoods or their country. Around five years ago, I was contacted by them to help them writing a mobile app that helps them to collect data on the field, on the site, and back then, this is the result of my work,
01:21
and back then I noticed some problems. For example, here is a screenshot with a point taken in Berlin, but I was able to put it on a map in Romania. Not very nice. You also had to fill the country every time when you added a site. This is not convenient because it's just easy
01:42
to do a query in the database and find the country. They were using this old platform, a Drupal 6 instance with a lot of plugins. From my point of view, it was very hard to maintain, and at some point, they looked into
02:03
migrating to some other platform, but none of these platforms were matching their needs. After a while, I noticed that no one will go to do this upgrade, and if someone will want to do this,
02:23
it will take a lot of time. So I thought that it's a better approach to rewrite it from scratch. And before I moved to Berlin, I talked with Ciprian and I told him that I wanted to take this challenge. And after a few months of development,
02:45
some goals started to take shape, because they started to ask me for a lot of nice features. I was thinking that this app can be used by many organizations. So my goal was to make an app that's easy to use
03:02
and to understand, because this might be used by hobbyists or maybe not GIS experts, so the visual language had to be very clear and relatable by the user. It had to be customizable in the way that you can put your own logo,
03:20
you can decide who can register for your service, maybe you want to make it public or private, or, yeah, so on. It had to be easy to deploy, because I don't expect that small organization to have a dedicated infrastructure team, so I had to make this very easy.
03:42
It has to be fast, because who doesn't like a fast app? And it also had to work on small servers, because not everyone has cloud infrastructure and so on. And I was able to transform this into this.
04:01
This is the homepage, where you already can see two nice features. For example, the legend, which it updates constantly with the icons that are visible on the current map, and you can also use it as a filter. You just click the icon that you are interested in,
04:21
and you will see only those sites. And also you can use it in combination with the search feature. But the most interesting feature is the icon sets, because you can define a set of icons with attributes, and then you can collect data
04:41
and see it on your front-end page. And you can see this contribution form, which is designed to make you focus only on the current attribute that you are adding, and it also adapts to the data that you want to add.
05:04
So if you add a bear and some steps, then you will be asked only for the attributes for these icons, and you can forget about the other ones. And even if you don't want to add these attributes, you can say, I don't want to add them now,
05:20
and you can add it later at home. And yeah, then you see your nice icon on your map. The other features are like maps. You have themes, and you can add any basemap that you want.
05:40
Now some technical details. In order to achieve this, I decided to use the D programming language and Ember.js for the front-end framework. Why D? Because it has so many nice features, and Ember.js offers
06:03
a set of convenient ways that help me, for example, to not write the data layer,
06:20
because I have it just for free. But of course, using D lang, it's not that... Using D as programming language also has some challenges, because in the beginning, there was no JS libraries that I could use.
06:42
It's hard to debug. I had to use the debugger in the command line, and I really started to love segmentation faults, and there were weird bugs in the third-party libraries that I used. As the architecture, it looks like this. I have a doc server,
07:02
which just serves some markdown files and converts them to HTML. I have the static web app, which is just an NGINX that serves the compiled Ember app, and I have the REST API. This one was designed as a monolith, but I noticed that some requests
07:21
sometimes took maybe 20 seconds, and I added a message queue, which receives messages about the hard tasks that needs to be run, and once a task runner is empty, the message, it will be passed to this runner,
07:41
and it will do this in background, so you will have a very fast API. And there is this Meselite cron job, which spawns once a day and checks if all the metadata is valid, and if there is some inconsistency,
08:01
it will trigger a task to fix it. For example, one of these tasks that I run is I keep a history of all the changes that happen in the database, or once you add a new site on a map, you want to increase the number of contributors
08:23
for that map, so this has to be in background, because you might have thousands of points for a map. Yeah, so we have this way of doing programming, which is called generative programming, and for me, it's a way to make boring tasks more fun.
08:43
So I use it to generate the REST API. You can use it for serialization or deserialization, or you can use it to generate open API docs. In order to achieve this, you use compile time function evaluation,
09:01
which means that you can run your code while the program compiles templates, code introspection, mixing statics if and static for each. You can look into the programming language documentation. You will find this very interesting.
09:23
What is code introspection? For example, I have this simplified site structure, and I can use this static for each to go through each property, and in this block, I can write the code that does the serialization, validation, and whatever.
09:42
And this happens at compile time, so there's no runtime spent for these topics. One interesting use case is that, for example, Amber just expects to have the plural for my model,
10:02
but in my code, I use the singular as a struct name, so I use this template, which just adds an S at the end of the type, and I have the string with the plural. And yeah, this can be applied to any structure in my code. Of course, not all plurals are created by this rule,
10:23
so I also can add the string attribute before my struct, and I have the custom plural. What you can also do is to introspect your functions and what attributes, what parameters they have, and I use it for adding middlewares.
10:44
Yeah, this is some code from the app. And when you put everything together, your code looks like a Lego, so only the green lines here are specific to the site model. The other classes or instances can be used
11:04
in my other models, and I don't even have to write the code that does the serialization or something for my site. It's generated automatically. Okay, yeah, and on the front end,
11:20
I have to define the model again, and with this store object, I can query the data from my server, and that's it. Another problem that I had was with serving the tiles. My initial solution was to add this custom route,
11:43
which has as an argument a view box, and I would return all the sites that are in that view box, but the solution was very slow, because sometimes I return it, maybe the whole database, and it was hard to cache, because it's unlikely that the user will have the same view box twice,
12:03
and I was also loading a lot of unused data. And then I came with the second solution where I added vector tiles, which were much easier to cache, and I was able to split the data into layers.
12:20
You can see here the debug mode for the map. I have here the sites on one layer, and on another site, I have the groups where I only have to store a counter. In order to achieve this, I wrote some libraries which are open source. You can find them on GitLab.
12:42
The first one is G-O-D, which is a GIS library. Handlebars, it's a templating library for HTML or XML, and REST API generator. But of course, my work doesn't finish when I deploy the app.
13:01
I also have to monitor it. So I use Grafana and Prometheus to get metrics from the system, and here you can see some screenshots with the memory consumption, and the hit count for a small session. I also monitor the number of queries that I do to the database,
13:21
and doing this simple monitoring, I can easily see that the icon queries are more than 1,000 for one small session, so I definitely have to optimize this. Yeah, and I count the number of data in the database,
13:41
and some more metrics. For example, right now we have 43K of sites, and I don't know, maybe 700 teams. I cannot see it. So, what's next? The mobile app, I'm working on a mobile app.
14:03
Right now it's written with Cordova, but I do plan to write native app because I really want to have an optimized system, because when you are on the field or in the forest, I don't want to drain your battery. And yeah, it would be good to have this native app.
14:22
I want to finish the WFS integration because it would be nice to work with the data in QG, so your professional tools, if you are a professionalist. I want to create host maps, to create and host tiles from OpenStreetMap, because one of my biggest challenges with this project
14:44
was, wasn't the code, was how I can find free and open tiles that also work fast. Right now, for GreenMap, we use OpenStreetMap, which are pretty slow. They are good, but yeah, it would be nice to have something more accessible.
15:03
I want to add support for lines and polygons. It would be nice to collect data from sensors because maybe you can hack an Arduino with some CO2 sensors or, I don't know, noise detectors and just put it on the GreenMap
15:21
and make this available to everyone. I want to allow discussions on the sites page because sometimes, maybe you might find some inconsistency in the data and you want to notify the one who added the site that there are some changes to do there.
15:42
And of course, I want to open source the app. As conclusions, I think using the most popular stack is not always the best solution. I was able to do this all by myself. Generative programming means productivity.
16:03
And yeah, this collective is the easiest and most affordable data management system. Yeah, this was it. I think I went too fast. So, do you have any questions?
16:29
I have one question for you Bogdan that maybe would answer things that would be interesting for people that are not currently here is how is it to deploy your solution
16:44
because you aiming it to be open sourced at many levels? How is it to be deployed by any field collector and what infrastructure they need to deploy it? I mean, we are kind of, let's say,
17:01
we know pretty much how to, what kind of server we need for common PHP, MySQL, or PHP, PostgreSQL, PostG's thing we need and there are so many resources. What about this one? Yeah, for example, the whole microservice architecture runs on the smallest virtual machine
17:23
that you can buy from Hetner but you can host it on any cloud provider. And it costs like five euros each month which is like maybe two coffees. And yeah, I create packages for Linux.
17:43
You can install the package for Ubuntu or Redhead or whatever. And I also have Docker images so if you want to do it on the cloud you can release it in the cloud. Yeah.
18:06
I have one more for you. Let's say I'm not so technical of deploying this even if it's easy, but I'm just a regular user and I want to just tap click a couple of times
18:21
and I want to see, let's say, data centric or geocentric form online and then I just want to start collecting data. So I don't want to install it anywhere. I don't want to figure out where and how. But I may be tempted to go just to a website
18:41
and I say, okay, in five minutes I want to start collecting data. Do you plan having such a service sometime in the future for, let's say, individuals or small organizations? I do plan to have this service in the future. I'm not sure when it will happen.
19:00
And yeah, deploying it, it's like any other server. But I do plan to have something like you do on Slack if you know it, where you just type your subdomain and you have the instance in five minutes. But this requires some more work and it will come in the future.
19:23
I hope this answers your question. How long would it take for a non-technical person to say a biologist that want to collect data?
19:47
Let's say he wants to build 20 fields and does he need any programming skills just to build his own form and just start collecting?
20:01
Well, the purpose of the app is that you don't need any programming skills in order to collect your data. So that's why there's the feature for, there's the feature for defining your icon sets and you can there create any attributes for your icons.
20:25
And I plan that in this mobile app to add a feature where you just add the URL to your instance and you will be able to access your data very easy. So if you want to start a project,
20:42
you can just do it overnight. And yeah, you don't need to talk to any developer that to write this mobile app that helps you to collect the data. And even if you, in the middle of the project, you discover that your attributes don't fit the project,
21:03
you can just change it and have it in maybe five minutes. If you no longer have questions, then we end the presentation and we wish you healthy and happy lunch.