OSM Stats: Rewarding contributors and real-time tracking of OSM
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 181 | |
| Anzahl der Teile | 193 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/20417 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
00:00
DreiFächer <Mathematik>StatistikWeb-SeiteVarianzWhiteboardMapping <Computergraphik>SinusfunktionHash-AlgorithmusMathematikMetropolitan area networkKommutativgesetzEliminationsverfahrenProgrammEreignishorizontGruppenoperationInteraktives FernsehenTotal <Mathematik>EmulationBootenVorzeichen <Mathematik>MehrwertnetzComputerunterstützte ÜbersetzungZahlenbereichVererbungshierarchieDatensatzGebäude <Mathematik>Textur-MappingGammafunktionTaskQuadratzahlInklusion <Mathematik>DatenfeldLokales MinimumPunktKonvexe HülleMetrisches SystemGebäude <Mathematik>StatistikVarietät <Mathematik>RangstatistikZahlenbereichProjektive EbeneTextur-MappingTotal <Mathematik>Mapping <Computergraphik>SchnittmengeGruppe <Mathematik>ElementargeometrieMetadatenQuick-SortWeb-SeiteMechanismus-Design-TheorieMathematikSoftwareentwicklerEreignishorizontEchtzeitsystemMultiplikationsoperatorFlächeninhaltDialektWeb SiteBitGruppenoperationBeamerTexteditorOrdnung <Mathematik>TermEinsZusammenhängender GraphAffine VarietätHomepagePhysikalisches SystemWellenpaketArithmetische FolgeDatenfeldRechter WinkelKette <Mathematik>FontPixelKonvexe MengeWhiteboardVererbungshierarchieDatenverwaltungMatchingWiderspruchsfreiheitBitrate
08:45
Immersion <Topologie>Physikalisches SystemReelle ZahlARM <Computerarchitektur>RechenbuchLambda-KalkülLokales MinimumTVD-VerfahrenSchnittmengeWurm <Informatik>StörungstheorieMetropolitan area networkSpezielle unitäre GruppeFrequenzMapping <Computergraphik>InstantiierungLambda-KalkülMultiplikationsoperatorMathematikSchnittmengeMetadatenElementargeometrieImmersion <Topologie>Differenz <Mathematik>DatenbankMetrisches SystemOpen SourceMailing-ListeZentrische StreckungRepository <Informatik>DiagrammFunktionalServerKonvexe HülleStreaming <Kommunikationstechnik>Varietät <Mathematik>Weg <Topologie>SchaltnetzHomepageTypentheorieElektronische PublikationHalbleiterspeicherStatistikBruchrechnungOrdnung <Mathematik>WarteschlangeZahlenbereichProjektive EbeneOffene MengeApp <Programm>DokumentenserverPhysikalisches SystemTextur-MappingRechenbuchInverser LimesRuhmasseStapeldateiWeb SiteMatchingComputersicherheitStrömungsrichtungRechter WinkelDienst <Informatik>LinearisierungMultiplikationZeichenketteFigurierte ZahlFlächeninhaltFormale GrammatikHash-AlgorithmusMenütechnikComputeranimation
17:13
MultiplikationsoperatorEchtzeitsystemWeb-SeiteSingularität <Mathematik>Prozess <Informatik>Textur-MappingVarietät <Mathematik>FrequenzDienst <Informatik>VarianzProjektive EbeneAbschattungMapping <Computergraphik>ZahlenbereichComputerarchitekturGebäude <Mathematik>DiagrammAdditionTwitter <Softwareplattform>Güte der AnpassungBitRechenschieberInklusion <Mathematik>HomepageVorlesung/KonferenzBesprechung/Interview
20:24
ICC-GruppeComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:08
Hello, welcome. Thanks for coming. My name is Matthew Hanson. I'm with Development Seed in Washington, D.C. I'm going to talk about a project that we did for the American Red Cross.
00:20
You're probably all familiar with the Missing Maps project. Is anyone not familiar with the Missing Maps project? Okay, so Missing Maps is a project sponsored by the Red Cross to encourage mapping of areas in need, especially after disasters. So
00:42
the missingmaps.org website We worked on the project to redesign the website. So if you go to that site now, that's a new site from last year. It's actually been up for maybe six months now or so. And our original goal was to not only redesign the website, but Red Cross wanted user pages
01:03
showing people's statistics and what they've committed into OpenStreetMap as well as statistics on those commits and some sort of reward mechanism and related to that leaderboard showing
01:22
the ranks of users and groups and they wanted this to happen in real time. So the Missing Maps project sponsors mapathons. So mapathons are where everybody they gather together for an hour, maybe two hours. It's maybe some people haven't used OpenStreetMap before and
01:46
they use the perhaps undergo a short training session and then they have a targeted area where everybody jumps on. So this is like maybe 70 users, maybe a lot fewer, maybe some more for really large ones and they map that region.
02:04
So the real-time component is to so you can show up on a projector the real-time contributions over time during these mapathons. So this comes down to tracking commits. So that's what we need to do is we need to take the commits and
02:24
and track them and a commit in OpenStreetMap is called a change set and this is made up of metadata and the data. So if you're familiar with OpenStreetMap you might go here and look at some of the details on a particular change set and this has metadata and the data
02:46
included in it. This is the geometry and this is the metadata that's published every minute. Now, I'm going to get into the details of the real-time system in a little bit because there's actually
03:01
in order to do this in real time, the geometry isn't actually available with the metadata and so it's a little bit more complicated. But if you notice in the change set we have hashtags. So hashtags are how we form communities in in missing maps or in fact other projects. When these mapathons happen or maybe outside of mapathons
03:26
people who make commits can add hashtags to their commits and then we can track those. So hashtags are spatially unbounded. They track groups and events.
03:41
The biggest one is the missing maps hashtag that the Red Cross was particularly interested in. But you could put as many hashtags as you want. So for a particular mapathon or for a particular project, you might have a hashtag and the editor that you use can be configured to just automatically add those hashtags
04:03
every time that you make a commit. So we have map time, my awesome hashtag, whatever it is that you want to add. So this brings us to leaderboards. So with these hashtags in place
04:21
we can have leaderboards where we can look at the total commits for any specific hashtag as well as the users. And so what you see here is this is interactive and so you can add any specific hashtag that you want. If you go to this page, it will default to the one on the left is the missing maps
04:45
hashtag, but I've added a hot OSM one and a map time event one and so you can see that in a mapathon, let's say, you could have two groups and one group uses one hashtag, another one uses another. They both are using maybe some common one and you could have
05:05
some sort of, you know, competition in the mapathon and see who's doing more commits, you know, left side, right side, bald people, those with hair. This is the total number of
05:22
edits that have been made since maybe in the last six or so months when we started tracking. We would like to in the future go back and add historical data so we can go back all the way to the beginning of the Missing Maps project, but right now this is, it's just since we started. And then the leaderboard show the users and
05:49
right now this is being sorted by the total number of edits, but you can sort by buildings or the kilometers of roads or or the, you know, any of the fields here. So you see we have RivW who has been on the top.
06:05
This has been, if you check every once in a while, it's usually these top five people there because they're sort of bouncing around. I don't know if they're actively trying to get on top or but right now RivW here is the top one. Now if you have
06:21
made a commit to OpenStreetMap in the last several months and you have added the Missing Maps hashtag, then you already have a user page automatically. This doesn't have to be added. It's automatically added. So those user pages, each particular user can go to a specific page, and so here we have
06:42
RivW. It shows a variety of metrics on his contributions and total edits and that sort of stuff, the hashtags that are used, as well as you see these badges. And
07:03
also, we have a contribution timeline and a map showing the regions. Now he's clearly focused right here in South Africa, but other people might be, other people actually bounce all over the place. This is a, we actually save the convex hull of a commit and then combine that with the previous convex hulls.
07:25
So we're not storing all of the geometry, just the approximate region. And also the countries will map those geometries to what countries they're in, so we can we can track the countries that are mapped as well. And
07:45
now you can look at the badges that RivW has earned, and there's a variety of them. They've been, of course, been very active. And this is all original artwork made for this for this project and with clever names. This is, the illustrations are all done by our Dylan
08:04
Moriarty at Developancy. And down below on that same page, you have your upcoming badges. So, and you can see that there's a progress there. So we've got whitewater rafting, that's mapping of waterways. There's really a quite a number of badges that you can earn.
08:26
And here's some examples, a little pixelated, but. Okay, so why rewards? Some people will be like, well, that's silly. We originally, this project was called OSM Gamification, but internally we didn't really like that term.
08:46
This, sometimes there's a negative connotation to gamification, or maybe it's a buzzword that's been perhaps used too frequently of late. Well, rewards provide a few different things. First off, there's an immersion in the mapping experience. You make commits,
09:06
and it's not just about making the commits, and going into a black hole, and then you can see them amidst all the other commits on OpenStreetMap. You can go to your stats page, and you can see exactly what you've done. So it gives a,
09:22
the statistics for what you've done, I think are very useful, and most people would be interested in that. People are after different things. Some people might not care about any of these things. Some people might be care, maybe care about a few of these. So you get a sense of achievement when you earn badges, and you strive to maybe get the next badge. So this increases retention.
09:44
It also can encourage cooperation. Like I mentioned before, you could have teams, and teams can cooperate in order to perhaps win over the other team during a mapathon. And of course the competition inherent in that. So how did we do this? We
10:06
implemented this real-time system using largely micro services, and there's a diagram here, and I'll just talk about each of the pieces. This is all implemented on AWS.
10:22
So the first thing that we need to do is we need to stream the real-time data. So OpenStreetMap makes the metadata available from planet.osm.org, and these diff files are published every minute, usually. Every minute there's a new file added, and it's all the commits that happen in the last minute.
10:43
But it doesn't include any of the geometries. So these are available. We could replicate the OpenStreetMap data ourselves, but it changes constantly. So we use the Overpass API. So Overpass
11:00
essentially replicates the OpenStreetMap database and makes the geometries available for the last minute for all the commits. Well, now these have to be matched up. So we have a node app called PlanetStream, and PlanetStream
11:21
takes in the change set metadata from OSM and the augmented, what is called the augmented diffs from the Overpass API, and has a Redis instance running and puts them in the Redis instance because sometimes these don't match up. You can't just take the change sets for one particular minute and
11:40
the geometry from the same minute because there might be a delay for a variety of reasons. So we put these in a Redis instance and have a timeout, I think, of maybe an hour or more. Maybe it's a couple hours. And we match up the metadata IDs with each other,
12:00
and so we create a final change set with the geometry. Combined change set there. Simultaneously, PlanetStream makes the map data available for the last, sorry, the the geometries available for the last hundred edits just so that if you go to the Missing Maps website, you can see a map showing the last hundred
12:24
commits made and where they are. And also it keeps track of the trending hashtags. So again, at Missing Maps, if you go in and want to add a hashtag, you can see a list of what's been popular recently.
12:42
So now we need to calculate the user metrics now that we have these combined change sets. So the repo is called omstatsworkers that we use, and we use AWS Lambda functions and Kinesis Streams, and I believe that I
13:01
Yes, here we go. So here's the diagram. You see the combined change set goes into a Amazon Kinesis Stream which is, it's just, it's a queue. You add it to the queue, and then as change sets are added to the queue, that fires off a Lambda function.
13:22
A Lambda function, if you're not familiar with it, it's a serverless, so this is all a serverless setup. So we have a node app, and that is uploaded to, as a Lambda function, and we don't have to worry about running servers or anything like that, and
13:40
they're invoked every time a change set is added to the stream. And it scales automatically, so if there's a lot in the stream, then it'll fire off a lot of Lambda functions, and it works very well. We use a RDS database to store these metrics, so the Lambda function
14:01
calculates some metrics on the change set and adds that to the database. Oh yes, and these are the types of things that are calculated, right? We have the metrics here, but also some geometry calculations to figure out what country things are in, getting the convex hull of those geometries and adding it to the user's total
14:22
contributions, geometries. Okay, so why Lambda? The Mapathons, right, are not happening all the time. They're, a lot of times there's no activity at all for a particular hashtag or
14:43
any commits to open street map, and then during a Mapathon that can spike and get really high. So we didn't want to run an EC2 instance all the time, so Lambda functions are perfect because Lambda functions, you don't pay for them, when they're not doing anything, if they're just sitting there. So you can upload a Lambda function
15:03
and it doesn't cost you anything at all. And that's very nice. It only costs for how long it runs and how many times it runs. So here you see the invocations and the number of times that the Lambda function is called over some time period here, and it can vary from
15:22
zero or maybe a couple per minute up to 100 per minute. And the Lambda functions therefore provide a very cheap way to do this. I will add that our cost for the Lambda functions for this whole project is essentially zero.
15:41
We don't, we're not running a million requests. If you, if you have something that could be a serverless setup, I would encourage you to really look at Lambda functions because they're actually very, very cheap. They cost fractions of a cent per time they're run depending on the memory usage that you can figure.
16:01
They're a very cost-effective way to do things if it makes sense. So, contribute. If you go to the missing maps page, we will, there's a list of mapathons that are coming up. So this is the, this is the current mapathons coming up.
16:20
So if you happen to be in the Czech Republic or Belgium later in the next few weeks, and there might be others in your area. Here's the overall contributions for missing maps since we started, started doing this.
16:47
And, of course, I should point out that this is all open source and you can go to the American Red Cross GitHub page here and the OSM stats repo. Now these are multiple services, so there's multiple repos, but if you go to OSM stats, the readme file
17:03
lists all the repositories that are, that we use for this. And that's it. Thank you. Okay, thank you very much Matthew. So we have time for a few questions.
17:27
I'm curious what Red Cross's reaction has been to the project. Are they getting the kind of outcomes they were looking for? Yeah, good question. Well overall, yes, there's been a variety of technical difficulties. I think that, you know, early and early on in the process
17:47
we didn't quite realize some of the, some of the technical issues that we would have in trying to do this in real time. Specifically issues with overpass, sometimes going down and that delay and dropped commits.
18:01
So we have had, since we started, we have had periods of time where we've lost commits and so we, that's why, one of the reasons why we want to do historical processing is not only to go back to the beginning of time of missing maps, but also to fill in these gaps. So, but other than these technical glitches, I think
18:23
it was, it was perhaps maybe ambitious to think that we could reliably have a 100% up all the time service that would always run. So this gap filling, periodic gap filling going back to achieve a hundred percent inclusion is what we feel is important now.
18:49
Is it working for things like retention and building the user pool? Yeah, yeah, it seems to and if you go to the leader pages you see that some people are very active and they have a lot of badges and
19:01
it's really pretty cool and especially during mapathons it's neat to have it up on a screen and watch the real-time commits coming up. Like you make a commit and then you can go to your page and it comes, you know, within a minute or two, usually sometimes longer you can see that coming in.
19:21
Another question. Yeah, thank you for your talk. A bit of a superficial one. The architectural diagrams in your slides were super pretty. I was just wondering how you generated those.
19:41
I'll have to get back to you on that. I can't remember. I can't remember if it was the I don't know, shout out a tweet to me and ask. I can't quite remember how that was done. Mark Farah is the one who did that and we've used a couple things to make those. Amazon has their own
20:02
architecture diagram, but we didn't use that. So if you give me a tweet, I will respond and let you know. Okay, will do. Anyone else? No? Okay, then thanks again, Mark.
20:20
Thank you.