We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Mapping Green goes Open with the GIS Collective

00:00

Formal Metadata

Title
Mapping Green goes Open with the GIS Collective
Title of Series
Number of Parts
295
Author
Contributors
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
The Open Green Map app has been used by map makers around the world to collect thousands of green sites over the last decade. With a new implementation, the Open Green Map has potential to truly live up to its name, and achieve a sharable, scalable, fast and easy to use application. This is what the project I will present sets out to do. Working on an open source project can sometimes make it seem like compromises around performance or user centric design are reasonable. What if you don't want to compromise in any of these 2 directions? By using an appropriate and modern tech stack, along with generative programming techniques, we hope to introduce a new approach for creating GIS applications. This talk will introduce how the new Open Green Map app works, and advantages of using DLang and Ember for a GIS app. We will also take a look at challenges that arise in relation to implementing GIS applications while having plenty of constraints around resources, and when helpful open source libraries are missing. The presentation will include an overview of the infrastructure, examples of how we generate REST APIs based on the app models and a quick look at how we efficiently display the sites on maps.
Keywords
129
131
137
139
Thumbnail
28:17
Green's functionTexture mappingOpen setGreen's functionComputer virusProjective planeMobile appPlanningOpen setLevel (video gaming)Game theoryPresentation of a groupComputer animation
Green's functionWebsiteNeighbourhood (graph theory)WebsiteLevel (video gaming)Field (computer science)Point (geometry)XMLComputer animation
WebsiteComputing platformServer (computing)Computer iconAttribute grammarMaxima and minimaDecimalConvex hullComa BerenicesJust-in-Time-CompilerOpen setGreen's functionSelf-organizationCombinational logicLibrary (computing)Set (mathematics)Web pageMultiplication signSoftware bugDampingMobile appComputer iconFormal languagePlug-in (computing)Computing platformAttribute grammarFreewarePoint (geometry)Level (video gaming)DebuggerQuery languageInstance (computer science)ResultantSoftware frameworkCloud computingProgramming languageOrder (biology)Differential operator1 (number)Cycle (graph theory)Right angleWebsiteData storage deviceExpert systemService (economics)Software developerShape (magazine)Labour Party (Malta)Form (programming)Server (computing)Web 2.0Computer architectureLengthView (database)Computer programmingHome pageMappingComputer animation
ArchitectureComputer fileWeb 2.0Fluid staticsRepresentational state transferTask (computing)Message passing2 (number)Computer animation
ArchitectureComputer programmingTask (computing)StatisticsCodeFluid staticsElectric generatorTask (computing)MathematicsWebsiteNumberMessage passingElectric generatorRepresentational state transferMetadataPoint (geometry)Computer programmingConsistencySpecial unitary groupProcess (computing)Ocean currentLevel (video gaming)DatabaseLecture/ConferenceComputer animation
Computer programmingElectric generatorTask (computing)Fluid staticsCodeWebsitePoint (geometry)Computer iconString (computer science)Field (computer science)AliasingTemplate (C++)Data modelAbelian categoryTelephone number mappingDependent and independent variablesRouter (computing)Social classInformationDigital filterHash functionDatabaseAuthenticationElectronic mailing listLibrary (computing)Uniqueness quantificationQuery languageSubsetCache (computing)Vector spaceType theoryRun time (program lifecycle phase)Computer programmingChord (peer-to-peer)Template (C++)Data structureCodeLine (geometry)Core dumpMetreObject (grammar)String (computer science)Functional (mathematics)Group actionMobile appOrder (biology)WebsiteCivil engineeringServer (computing)Observational studyLabour Party (Malta)Social classInstance (computer science)Programming languageRow (database)Performance appraisalMultiplication signDatabaseView (database)CASE <Informatik>Level (video gaming)Validity (statistics)Data storage deviceCuboidCategory of beingRoutingElectric generatorCompilation albumSerial portUsabilityAsynchronous Transfer ModeVector spaceEndliche ModelltheorieParameter (computer programming)TesselationAttribute grammarFluid staticsOpen setMixed realityComputer animation
Vector spaceLibrary (computing)Electric generatorLibrary (computing)Open sourcePhysical systemTemplate (C++)Order (biology)DatabaseNumberTouchscreenElectric generatorSemiconductor memoryWordLine (geometry)Query languageMetric systemRepresentational state transferCountingMobile appComputer iconComputer animation
Metric systemNumberWebsiteCountingComputer animationEngineering drawingDiagram
Mobile appMobile WebForestGame theoryPhysical systemPlanningComputer animation
Mobile WebPolygonLine (geometry)Web pageOpen setDisintegrationIntelComputer programmingData managementGamma functionINTEGRALMappingOpen sourceForm (programming)Order (biology)Medical imagingField (computer science)Server (computing)Type theoryAutomatic programmingComputer architectureInstance (computer science)Computer programmingMobile appService (economics)MathematicsTesselationAttribute grammarConsistencyData managementWeb pageWebsiteCloud computingLine (geometry)Point cloudComputer iconPolygonRight angleVirtual machinePresentation of a groupNoise (electronics)Multiplication signSoftware developerUniform resource locatorProjective planePlanningFood energyFigurate numberLevel (video gaming)Open setGame theoryComputer wormBootingProduct (business)Physical systemRoundness (object)Disk read-and-write headCodeComputer animation
Transcript: English(auto-generated)
We prepare ourselves for the next presenter, Bogdan. He will be talking about Mapping Green, goes open with the GIS Collective.
Hello. I'm Bogdan and I'm here to show you how this collective works for Open Green Map. I will talk about the project overview, some interesting technical details, and what are my future plans with this app.
But first, who didn't heard about Green Map? Yeah, I guess everyone was at the previous talk. In one phrase, Green Map is a community around the world who gather data points around sustainable living
in their neighborhoods or their country. Around five years ago, I was contacted by them to help them writing a mobile app that helps them to collect data on the field, on the site, and back then, this is the result of my work,
and back then I noticed some problems. For example, here is a screenshot with a point taken in Berlin, but I was able to put it on a map in Romania. Not very nice. You also had to fill the country every time when you added a site. This is not convenient because it's just easy
to do a query in the database and find the country. They were using this old platform, a Drupal 6 instance with a lot of plugins. From my point of view, it was very hard to maintain, and at some point, they looked into
migrating to some other platform, but none of these platforms were matching their needs. After a while, I noticed that no one will go to do this upgrade, and if someone will want to do this,
it will take a lot of time. So I thought that it's a better approach to rewrite it from scratch. And before I moved to Berlin, I talked with Ciprian and I told him that I wanted to take this challenge. And after a few months of development,
some goals started to take shape, because they started to ask me for a lot of nice features. I was thinking that this app can be used by many organizations. So my goal was to make an app that's easy to use
and to understand, because this might be used by hobbyists or maybe not GIS experts, so the visual language had to be very clear and relatable by the user. It had to be customizable in the way that you can put your own logo,
you can decide who can register for your service, maybe you want to make it public or private, or, yeah, so on. It had to be easy to deploy, because I don't expect that small organization to have a dedicated infrastructure team, so I had to make this very easy.
It has to be fast, because who doesn't like a fast app? And it also had to work on small servers, because not everyone has cloud infrastructure and so on. And I was able to transform this into this.
This is the homepage, where you already can see two nice features. For example, the legend, which it updates constantly with the icons that are visible on the current map, and you can also use it as a filter. You just click the icon that you are interested in,
and you will see only those sites. And also you can use it in combination with the search feature. But the most interesting feature is the icon sets, because you can define a set of icons with attributes, and then you can collect data
and see it on your front-end page. And you can see this contribution form, which is designed to make you focus only on the current attribute that you are adding, and it also adapts to the data that you want to add.
So if you add a bear and some steps, then you will be asked only for the attributes for these icons, and you can forget about the other ones. And even if you don't want to add these attributes, you can say, I don't want to add them now,
and you can add it later at home. And yeah, then you see your nice icon on your map. The other features are like maps. You have themes, and you can add any basemap that you want.
Now some technical details. In order to achieve this, I decided to use the D programming language and Ember.js for the front-end framework. Why D? Because it has so many nice features, and Ember.js offers
a set of convenient ways that help me, for example, to not write the data layer,
because I have it just for free. But of course, using D lang, it's not that... Using D as programming language also has some challenges, because in the beginning, there was no JS libraries that I could use.
It's hard to debug. I had to use the debugger in the command line, and I really started to love segmentation faults, and there were weird bugs in the third-party libraries that I used. As the architecture, it looks like this. I have a doc server,
which just serves some markdown files and converts them to HTML. I have the static web app, which is just an NGINX that serves the compiled Ember app, and I have the REST API. This one was designed as a monolith, but I noticed that some requests
sometimes took maybe 20 seconds, and I added a message queue, which receives messages about the hard tasks that needs to be run, and once a task runner is empty, the message, it will be passed to this runner,
and it will do this in background, so you will have a very fast API. And there is this Meselite cron job, which spawns once a day and checks if all the metadata is valid, and if there is some inconsistency,
it will trigger a task to fix it. For example, one of these tasks that I run is I keep a history of all the changes that happen in the database, or once you add a new site on a map, you want to increase the number of contributors
for that map, so this has to be in background, because you might have thousands of points for a map. Yeah, so we have this way of doing programming, which is called generative programming, and for me, it's a way to make boring tasks more fun.
So I use it to generate the REST API. You can use it for serialization or deserialization, or you can use it to generate open API docs. In order to achieve this, you use compile time function evaluation,
which means that you can run your code while the program compiles templates, code introspection, mixing statics if and static for each. You can look into the programming language documentation. You will find this very interesting.
What is code introspection? For example, I have this simplified site structure, and I can use this static for each to go through each property, and in this block, I can write the code that does the serialization, validation, and whatever.
And this happens at compile time, so there's no runtime spent for these topics. One interesting use case is that, for example, Amber just expects to have the plural for my model,
but in my code, I use the singular as a struct name, so I use this template, which just adds an S at the end of the type, and I have the string with the plural. And yeah, this can be applied to any structure in my code. Of course, not all plurals are created by this rule,
so I also can add the string attribute before my struct, and I have the custom plural. What you can also do is to introspect your functions and what attributes, what parameters they have, and I use it for adding middlewares.
Yeah, this is some code from the app. And when you put everything together, your code looks like a Lego, so only the green lines here are specific to the site model. The other classes or instances can be used
in my other models, and I don't even have to write the code that does the serialization or something for my site. It's generated automatically. Okay, yeah, and on the front end,
I have to define the model again, and with this store object, I can query the data from my server, and that's it. Another problem that I had was with serving the tiles. My initial solution was to add this custom route,
which has as an argument a view box, and I would return all the sites that are in that view box, but the solution was very slow, because sometimes I return it, maybe the whole database, and it was hard to cache, because it's unlikely that the user will have the same view box twice,
and I was also loading a lot of unused data. And then I came with the second solution where I added vector tiles, which were much easier to cache, and I was able to split the data into layers.
You can see here the debug mode for the map. I have here the sites on one layer, and on another site, I have the groups where I only have to store a counter. In order to achieve this, I wrote some libraries which are open source. You can find them on GitLab.
The first one is G-O-D, which is a GIS library. Handlebars, it's a templating library for HTML or XML, and REST API generator. But of course, my work doesn't finish when I deploy the app.
I also have to monitor it. So I use Grafana and Prometheus to get metrics from the system, and here you can see some screenshots with the memory consumption, and the hit count for a small session. I also monitor the number of queries that I do to the database,
and doing this simple monitoring, I can easily see that the icon queries are more than 1,000 for one small session, so I definitely have to optimize this. Yeah, and I count the number of data in the database,
and some more metrics. For example, right now we have 43K of sites, and I don't know, maybe 700 teams. I cannot see it. So, what's next? The mobile app, I'm working on a mobile app.
Right now it's written with Cordova, but I do plan to write native app because I really want to have an optimized system, because when you are on the field or in the forest, I don't want to drain your battery. And yeah, it would be good to have this native app.
I want to finish the WFS integration because it would be nice to work with the data in QG, so your professional tools, if you are a professionalist. I want to create host maps, to create and host tiles from OpenStreetMap, because one of my biggest challenges with this project
was, wasn't the code, was how I can find free and open tiles that also work fast. Right now, for GreenMap, we use OpenStreetMap, which are pretty slow. They are good, but yeah, it would be nice to have something more accessible.
I want to add support for lines and polygons. It would be nice to collect data from sensors because maybe you can hack an Arduino with some CO2 sensors or, I don't know, noise detectors and just put it on the GreenMap
and make this available to everyone. I want to allow discussions on the sites page because sometimes, maybe you might find some inconsistency in the data and you want to notify the one who added the site that there are some changes to do there.
And of course, I want to open source the app. As conclusions, I think using the most popular stack is not always the best solution. I was able to do this all by myself. Generative programming means productivity.
And yeah, this collective is the easiest and most affordable data management system. Yeah, this was it. I think I went too fast. So, do you have any questions?
I have one question for you Bogdan that maybe would answer things that would be interesting for people that are not currently here is how is it to deploy your solution
because you aiming it to be open sourced at many levels? How is it to be deployed by any field collector and what infrastructure they need to deploy it? I mean, we are kind of, let's say,
we know pretty much how to, what kind of server we need for common PHP, MySQL, or PHP, PostgreSQL, PostG's thing we need and there are so many resources. What about this one? Yeah, for example, the whole microservice architecture runs on the smallest virtual machine
that you can buy from Hetner but you can host it on any cloud provider. And it costs like five euros each month which is like maybe two coffees. And yeah, I create packages for Linux.
You can install the package for Ubuntu or Redhead or whatever. And I also have Docker images so if you want to do it on the cloud you can release it in the cloud. Yeah.
I have one more for you. Let's say I'm not so technical of deploying this even if it's easy, but I'm just a regular user and I want to just tap click a couple of times
and I want to see, let's say, data centric or geocentric form online and then I just want to start collecting data. So I don't want to install it anywhere. I don't want to figure out where and how. But I may be tempted to go just to a website
and I say, okay, in five minutes I want to start collecting data. Do you plan having such a service sometime in the future for, let's say, individuals or small organizations? I do plan to have this service in the future. I'm not sure when it will happen.
And yeah, deploying it, it's like any other server. But I do plan to have something like you do on Slack if you know it, where you just type your subdomain and you have the instance in five minutes. But this requires some more work and it will come in the future.
I hope this answers your question. How long would it take for a non-technical person to say a biologist that want to collect data?
Let's say he wants to build 20 fields and does he need any programming skills just to build his own form and just start collecting?
Well, the purpose of the app is that you don't need any programming skills in order to collect your data. So that's why there's the feature for, there's the feature for defining your icon sets and you can there create any attributes for your icons.
And I plan that in this mobile app to add a feature where you just add the URL to your instance and you will be able to access your data very easy. So if you want to start a project,
you can just do it overnight. And yeah, you don't need to talk to any developer that to write this mobile app that helps you to collect the data. And even if you, in the middle of the project, you discover that your attributes don't fit the project,
you can just change it and have it in maybe five minutes. If you no longer have questions, then we end the presentation and we wish you healthy and happy lunch.