mapillary2osm: OpenStreetMap Data from Computer Vision
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43459 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Bucharest 2019125 / 295
15
20
28
32
37
38
39
40
41
42
43
44
46
48
52
54
57
69
72
75
83
85
87
88
101
103
105
106
108
111
114
119
122
123
126
129
130
131
132
137
139
140
141
142
143
144
147
148
149
155
157
159
163
166
170
171
179
189
191
192
193
194
195
196
197
202
207
212
213
214
215
216
231
235
251
252
263
287
00:00
Line (geometry)Projective planeCASE <Informatik>WeightOpen setLevel (video gaming)Lecture/ConferenceMeeting/Interview
00:17
Machine visionDisintegrationText editorPersonal digital assistantProjective planeBitOpen setINTEGRALIdentifiabilityMachine visionComputational visualisticsSign (mathematics)DatabaseLevel (video gaming)Observational studyInsertion lossComputer animation
01:02
DisintegrationText editorMachine visionPersonal digital assistantProcess (computing)Machine visionIdentifiabilityPoint (geometry)Range (statistics)Computational visualisticsMultiplication signOpen setMathematicsLevel (video gaming)Context awarenessDataflowSign (mathematics)Computer animation
01:49
Projective planeContext awarenessLevel (video gaming)MassFocus (optics)Lecture/Conference
02:16
DisintegrationText editorMachine visionPersonal digital assistantPoint (geometry)Computer-generated imagerySign (mathematics)Sign (mathematics)Focus (optics)Medical imagingProjective planeBit1 (number)Lie groupLevel (video gaming)Software developerComputer animation
02:47
Text editorPoint (geometry)Computer-generated imagerySign (mathematics)1 (number)Software developerObject (grammar)Presentation of a groupText editorLevel (video gaming)Computer animation
03:17
Process (computing)Square numberProcess (computing)Projective planeLevel (video gaming)ECosGoodness of fitObject (grammar)Uniform resource locatorMedical imagingCuboidSquare numberOpen setDigital photographyMobile appType theoryComputer animation
04:21
Level (video gaming)Medical imagingAreaObject (grammar)Computer fileType theory1 (number)View (database)Branch (computer science)
04:57
Computer clusterMaxima and minimaAreaSatelliteGoogolOpen setGoogolLevel (video gaming)Process (computing)AreaSquare number
05:17
AreaLine (geometry)Medical imagingDrop (liquid)CircleLevel (video gaming)Drag (physics)DataflowEstimatorIntegrated development environmentData structure
05:52
GeometryOpen setDesign by contractLevel (video gaming)Numbering schemePosition operatorSet (mathematics)
06:08
Set (mathematics)Arithmetic meanLevel (video gaming)Multiplication signComputer animation
06:32
TrailTotal S.A.Absolute valueTrailCASE <Informatik>Computer animation
06:55
Point (geometry)Level (video gaming)Branch (computer science)GradientObservational studyOpen setAreaDigital photographyDataflow
07:28
CASE <Informatik>Object (grammar)Instance (computer science)Lecture/Conference
07:45
Level (video gaming)Medical imagingInstance (computer science)DataflowVideo gameComputer animation
08:26
Process (computing)MappingMereologyMultiplication signGroup actionLevel (video gaming)Moment (mathematics)Uniform resource locatorComputer animation
08:51
Digital photographyMotion capturePopulation densityPerturbation theoryProjective planeDigital photographyStatement (computer science)Open setLevel (video gaming)Multiplication signSimilarity (geometry)Focus (optics)Motion capturePopulation densityDegree (graph theory)Nichtlineares GleichungssystemClosed setPoint (geometry)CAPTCHAView (database)Computational visualisticsMappingMachine visionLecture/ConferenceMeeting/InterviewComputer animation
09:57
Point (geometry)View (database)MappingMachine visionUniform resource locatorComputational visualisticsProjective planeDegree (graph theory)Lecture/Conference
10:17
Motion capturePopulation densityMotion capturePopulation densityCartesian coordinate systemDegree (graph theory)Group actionNeighbourhood (graph theory)Multiplication signProjective planeLevel (video gaming)Coma Berenices
11:10
WebsiteBeer steinLevel (video gaming)Core dumpUniform resource locatorRandomizationProjective planeInformation
11:29
Source codeText editorLevel (video gaming)Open setProjective planeLecture/ConferenceMeeting/Interview
11:47
Discrete element methodPoint (geometry)Level (video gaming)Population densityMotion captureCombinational logicOpen set
12:09
Integrated development environmentComputer-generated imageryGroup actionBuildingSmartphoneTape driveWave packetInternetworkingIntegrated development environmentProjective planeObject (grammar)Computer animation
12:36
Integrated development environmentComputer-generated imageryGroup actionBuildingCASE <Informatik>Goodness of fitProjective planeMereologyMedical imagingPRINCE2Moment (mathematics)Denial-of-service attackObject (grammar)StatisticsLevel (video gaming)Computer animation
13:13
Formal verificationAlgorithmProjective planeFormal verificationMereologyObject (grammar)Electronic mailing listPosition operatorNumberWave packetShape (magazine)Level (video gaming)AlgorithmOptical disc driveAreaInformation securityMappingComputer animation
14:17
Text editorFormal verificationAlgorithmProjective planeFormal verificationLevel (video gaming)MathematicsText editorDataflowSet (mathematics)Lecture/ConferenceMeeting/InterviewProgram flowchart
14:47
Uniqueness quantificationLevel (video gaming)Multiplication signBitMappingAnalogyProjective planeDisk read-and-write headSet (mathematics)Formal verificationPoint (geometry)Open setStreaming mediaComputer animation
16:20
Projective planeComa BerenicesLevel (video gaming)Formal verificationDifferent (Kate Ryan album)Link (knot theory)Process (computing)Web pageText editorOnline helpDirection (geometry)Meeting/Interview
17:04
BuildingOverhead (computing)Absolute valueType theoryLevel (video gaming)MereologyInformationOpen setTracing (software)CASE <Informatik>Formal verificationQuicksortGoodness of fitTheory of relativityAuthorizationMultiplication signMachine visionObservational studyAreaFacebook
18:46
CASE <Informatik>Text editorObservational studyFacebookMultiplication signGoodness of fitTheory of relativityOpen setUniform resource locatorLevel (video gaming)Process (computing)Online helpMathematicsDataflowSimilarity (geometry)Meeting/InterviewLecture/Conference
Transcript: English(auto-generated)
00:07
Okay, so I want to share two projects that we worked on this year that kind of tie into what Chris was talking about earlier with open data, but specifically for OpenStreetMap. So just to gauge how familiar people are with OpenStreetMap, hands up if you've made
00:23
an edit at least once in OpenStreetMap. Okay, that's good to know in a geospatial community. So before I get into these two projects that I mentioned, I'd like to just share a bit about the history of our relationship with OpenStreetMap. So for those that don't know, we started in 2013, and very early on in 2014, in fact,
00:43
we made an integration for ID Editor, which is one of the editing tools in OpenStreetMap. And so that meant that people could see imagery while they're editing the map. And then shortly after that, in 2016, we took our computer vision capabilities where we can identify things like traffic signs automatically, and we made that available
01:01
to OpenStreetMap as well, and it complies with the OpenStreetMap database license. So it means you can take this data, you can make edits, as long as it's going to OpenStreetMap, you can do whatever you want with it. But over that time, or since that time, our computer vision capabilities have changed. So at first, we could only identify traffic signs, but now we can identify a whole range of things, things like benches, things like street lights, things like where a crosswalk
01:25
is located, but none of that has been made available to OpenStreetMap yet. So what we're focused on this year is understanding how this new data that we're able to identify can be made available to OpenStreetMap, and some of the considerations around the workflow and what that editing process might look like.
01:44
So I guess the last point to mention on this is that AI is, of course, a buzzword. You're probably all sick of hearing it. It can almost become meaningless because it's brought up so much. In the context of OpenStreetMap, there's also a lot of hype around AI and a lot of concerns. So some of the concerns is that humans have traditionally, or have always,
02:01
been responsible for making OpenStreetMap a better map and keeping it up to date. And if you have AI coming in and encouraging humans to kind of mass edit, does the quality disappear? Does that whole idea of a crowdsourced project fall apart? So these are some of the considerations that we had, and particularly this year, when there's been an increased focus on this.
02:25
So just going through the timeline quickly, we, as I said, founded in 2014. We've had the image layer merged. We had a traffic sign layer integrated, and this is happening for both ID and JOSM. But then more recently, and Chris has been a big supporter of this, actually,
02:41
we've had some really interesting side projects, projects that other people have developed that leverage street-level imagery and data. And some of the most incredible ones are Pic4Review, which we'll talk about in a little bit, Deriviste, and MapRoulette. So Chris has worked on integrating all of those, but these were actually developed by community members using some of the developer tools
03:04
that were mentioned in the previous presentation. What we're really excited about soon is the next thing, which would be map features. So all these different objects that I talked about being available in ID Editor. So the first project I want to talk about was called Mapillary to OSM.
03:22
This was a general project where we were trying to make this data available for OSM, understand what the editing process would look like, without actually writing any code, just using the existing tooling that was available. And so we thought we'd try it on a pretty experimental level. We chose five locations. We focused on 25 square kilometres in each of those locations,
03:43
and we narrowed it down to three objects that we thought were quite useful in OpenStreetMap, so benches, bicycle parking, and post boxes, all objects that we identify automatically and things that can be added to OpenStreetMap. So those five locations, Austin, Ballarup in Denmark, Kiev in Ukraine, Melbourne in Australia, and Sao Paulo in Brazil.
04:05
And the reason for these locations is that we had some really good community members there that were contributing a lot of imagery, and were also familiar with what we do in OpenStreetMap and how Mapillary is used in OpenStreetMap.
04:22
So just to walk through how people used this data, you can imagine if you go through street-level images on Mapillary, you can actually see which type of objects we've detected. You can filter by them. So in this particular example, you can see benches detected in Kiev. Someone's walked down this footpath. They've taken photos. They've uploaded them with the app,
04:40
and then these benches have automatically been extracted. And you can't really see it that well here, but there's a bench there. There's other ones on the map here. And so you can switch the map view and see how many benches have been detected in the area. So what we did is we downloaded all of these. We made them available as a GeoJSON file that they could load in,
05:01
and it looks something like this. So this is the 25-square-kilometer area in Kiev. These are all the benches we've detected. We basically said, hey, add this data to the map using OpenStreetMap. It was just on Google Drive, so nothing fancy, kind of hacky, and we just wanted to see what the process looked like. So this is what it looks like.
05:21
It's pink circles on the map. These pink circles represent a bench that we've detected and triangulated, so it's our estimation of where it's located based upon structure from motion, so a 3D reconstruction of the environment. And it's not clickable. If you try to click on that pink circle, nothing happens. So you kind of have to find a nearby image and then see whether there's a bench in that nearby image.
05:43
So it's not the most glamorous workflow, but this was step one. And that GeoJSON just loaded in OpenStreetMap drag and drop. So some of the edits that took place. This was Ukraine. They started adding these to OpenStreetMap. We used a tagging scheme, so when they put a certain tag,
06:02
Mapillary to OSM, we could track those edits, and you can see these are some of the benches that they added. The interesting thing is that the quality of the detections varied a lot. So in Ukraine, the benches seemed to work quite well as a data set. In Austin, Texas, there were a lot of benches that were false positives, meaning that we thought there was a bench there,
06:20
but it wasn't in fact there. So you can see there, you can kind of understand how AI might think that's a bench, but it's not a bench. And so that someone's wasted their time if they try and add that to the map. And so as this kind of competition went on, as this experiment went on, we'd track the edits that they made using the data. So it looked like benches were by far
06:42
the easiest for people to add. Just one thing to note that I think we found pretty interesting is that whenever you run a competition or a campaign, you need to be mindful of how people react to it, how they interpret your instructions. So we had a funny case in Ukraine where this guy was trying to maximise his score, basically.
07:03
So he saw all these benches in an open-air stadium and he thought, great, I can add these to the map. And he added, I don't know, 200 benches. So one, that's not really how you'd add them in an open street map. And two, he didn't even take photos in the area. So it wasn't even using AI.
07:21
And so we had to kind of disqualify him from this experiment, which is unfortunate. But it also shows that kind of the way you set up these experiments is really important and the right way you set up the workflow is really important because you don't want people adding crappy data. And so this kind of becomes our responsibility, like if this was added and if we'd kept it there, people could rightly be angry at us
07:41
for encouraging this kind of editing. So are there other approaches? Yes, pick for review is the one that we mentioned earlier. So this is very much focused on a case-by-case instance of where an object is. So in this case, it's telling us to look at crosswalks that have been identified and kind of judge how,
08:02
like if it's easy for a wheelchair to get up or not. And you can swap between various images in the top there and see what the quality of access is like for wheelchairs. And all you have to do is just press a button and then that makes the edit an open street map. So there it's kind of a lot slower,
08:20
but the workflow is less likely to be manipulated by someone in Ukraine or anywhere else. So takeaways, workflow and UI is really significant part of quality assurance for map editing. Instructions will always be interpreted loosely and in the favor of the participant. We've found this time and time again. So you need to be really careful
08:40
about the instructions you provide. Not all map features are suitable. So benches in Austin might not be as useful at the moment as benches in Kiev. And data varies a lot between location as well. So that was the first project. The second project is in a similar vein, but with a humanitarian focus. So we've seen a lot of this maybe
09:00
in the open street map community and more broadly, really bold statements like X technology helping end malaria or saving the world one photo at a time, droughts are a thing of the past because of this technology. Often there's that kind of desire to be hyperbolic around a new technology. And we wanted to kind of get beyond this
09:21
and see is this technology actually helping? To what degree does it help? And if it falls short, how can we kind of close that gap? So the map 2020 concept was looking mostly within the top 20 degrees. And so above the equator 20 degrees and below the equator 20 degrees. And that's where the name came from.
09:40
And also kind of trying to close this out by 2020. Understanding how Mappler is useful in humanitarian scenarios and then also encouraging dense capture over sporadic capture. So this is something that we saw time and time again was people would take one photo in a street and then move to the next street and take another photo. That's not really useful from a computer vision point of view or from a mapping point of view.
10:01
You really want someone to be focused on capturing every street. So we're trying to move that, move them towards that using some of the tools that we now have. And the two kind of most impressive projects would then present at the hot summit which is taking place next month. So these are some of the locations that participated. There's the rough 20 degrees north, 20 degrees south.
10:22
And I just wanna walk you through some of these projects. So one which was really impressive was dense capture in Dhaka. So these guys just equipped an action camera that they had from a few years ago to a rickshaw and to bicycles that wake up super early in the morning to beat the dense Dhaka traffic which is horrendous.
10:42
And then they captured one of the, I don't think this is loaded properly. There's actually even more coverage in that top left quadrant. But they captured pretty much every street and this neighborhood is particularly dense. When fires come, they often destroy a whole city block. And so they wanted to kind of get a sense
11:00
of what the quality of the infrastructure was, where street lights were located, where crosswalks were located and then add that to the map. So a great example of dense capture in difficult circumstances. One of the winning projects actually was in Kampala and these guys were mapping informal waste dumps. So people would just dump waste in random locations and that would kind of become a landfill
11:21
even though the government wasn't aware of it. So they would use street level imagery to document this, walk around and kind of see where these waste dumps are and then add them to open street map. And again, we used tags to track all this. So Henry has confirmed that he got his visa for Germany.
11:41
So if anyone's going to hot summit, you'll be able to ask him more about this next month. And the second project that actually won but won't be presenting in Germany due to visa issues is Erbil, Iraq. So again, this is a combination of dense capture and sporadic capture. But he was looking at earthquake resilience and also just trying to add a lot of shops
12:03
and kind of houses and just make a better, more detailed open street map in the city of Erbil in Kurdistan. So this is Mohammed working with the imagery that he's uploaded. He had about 20 people just using their smartphones to capture. So takeaways from that project, hacky and scrappy is sometimes best.
12:21
You can complain that you don't have the equipment or the internet's too slow but there's often a way to make it work as it evidenced in Ghana where they actually sticky taped a camera to a car. More training data is needed in these environments. So if we want better quality detections in, say, Kampala, we need to be able to annotate more objects in Kampala, which means more images in Kampala.
12:41
And just some stats, 370,000 images collected, 1,600 unique kilometers, which is a very impressive amount of data considering the circumstances. And I think we also learned that map lawyers fit for purpose in some cases. So road assets, we do really well. Urban mobility, understanding how people move around a city. Less so earthquake resilience and flooding at the moment.
13:02
That could change. Not so good for building footprints where drones would be a better use case or agriculture and kind of tracking how crops are changing. Again, not a good use case for map lawyer. One really cool thing that we're doing now and that was part of these projects was what we call verification projects.
13:21
So this is where if we have an object like that bench that I mentioned earlier in Austin that was incorrectly detected, we can create a shape around it and say, show me all the benches in Austin and then we can get the community to validate whether or not they've been identified correctly. So you can see here, this is a catch basin. People can tick yes or no or unsure
13:41
and that'll one, remove false positives. So for example, this pigeon that we thought was a CCTV camera and could be, I don't know how technology changed over the last year, maybe it's a new, I don't know, maybe this is a US government, CIA, CCTV camera pigeon or maybe not,
14:03
but yeah, false positive, you could mark that as a false positive, that will be removed from the map so when you download the list of CCTV cameras in that area, they'll all be correct. So that's really cool, that's number one and number two, it improves the algorithm once we rerun the training data, once we rerun the algorithms on the training data.
14:21
So anyone can participate in these, you can create your own as well. So just a kind of an overview of that, we have Mapillary.osm which was that first project I mentioned, Map 2020, we're using verification projects to improve the quality of the data and then very soon, we'll have the ID editor pull request which Chris and some others
14:42
have been working on where these map features will actually be a layer that you can toggle in OpenStreetMap. So going forward, I think dumping data is not very helpful. That first data set that I showed you in Kiev, remember all those benches, if 50% of them aren't accurate, is it really useful to just dump them all on the community and say, work with this?
15:02
Probably not. Workflow is really important, so maybe something like pick review is better at making sure the quality of the edits are more accurate and I think also our relationship with OSM is unique. So a lot of companies working with OpenStreetMap, they actually are reliant on the data so they might be using the data, combining it with their own proprietary map data
15:22
and then selling that to consumers. We're more reliant on the imagery that OpenStreetMap's contributors upload to Mapillary and that means that it's very important that we remain useful for OpenStreetMap because otherwise they'll stop uploading imagery. So I was thinking about different analogies for this relationship. It's kind of a symbiotic relationship.
15:41
Hopefully everyone thinks it's a mutualistic one, a bit like the honeybee pollinating different plants and not a parasitic one like this head lice which no one wants to have and is all taking and no giving. So yeah, and then the last point, community is actually one of the best ways to QA. So the verification projects I mentioned earlier,
16:02
community's been creating their own and improving data sets for things like street lights, catch basins in Turkey, so that people can then download that data and work with it. So hopefully you learned a bit about kind of our approach to providing map data for OpenStreetMap. Thanks for listening. Happy mapping. And if you have any questions, now is the time.
16:41
Yeah, I just have a question about your verification process. I understand this is all online verification. This is actually in the ID editor or? So the verification projects are actually on mapillary.com. So there's direct links to different verification projects and we have a help page where we walk through how to create those.
17:01
I'm just thinking that if, let's take the example of the benches. If it was in my area, I would like to download that data before it goes into the map and put it onto a GPS or onto a smartphone and go and verify them visually. And that would also be a way of maybe overcoming
17:20
some of the reticence on part of the community to automatically generated data. That's a good question. So you could have maybe a toggle. Maybe this is something for us to discuss, Chris, where you would have verified data and unverified data and the person in OpenStreetMap can choose which one they select. That's actually quite a good idea. But currently it's just all data
17:41
and then if you wanna verify it yourself, you can. But there's no way to know whether it's been verified unless you've actively promoted it in the community and let people know. I have a question about the building footprint stuff. So you said that's not great
18:00
and I wondered what specifically makes you say that and just clearly I can see how seeing a trace of a building from overhead from drone could be helpful, but there's also a lot of information about buildings that you can't get from overhead. So I'm curious what your thoughts are there. I specifically meant like in OpenStreetMap when you're tracing a building, just that as far as a building footprint.
18:21
But yeah, people have been using in Lesotho, for example, to get the type of roof, adding the type of roof, which they could see from street level imagery, adding the material, which they could kind of guess whether it's concrete or thatched, and that's all good for earthquake resilience. So there is a lot of building data you can get from Appler, absolutely.
18:45
More questions? So I got time for a few more. I recently saw that Facebook has some initiative to improve the ID editor. Is there some relation or? There's no relation, but I think that's a good case study in kind of what I was alluding to earlier.
19:00
So that's rapid, I think it's called rapid. And for those that don't know, what it does is Facebook's using AI to trace a lot of roads around the world. In Thailand, they started, but now they've kind of taken that and they're using it more broadly. And so you can very quickly see roads that have been traced by AI, you can go in and then the human can choose whether they take that automatically identified road
19:22
and actually add it as map data to open street map. And I think Facebook's story is similar in that the quality of those detections changes a lot. I've looked at some locations where the road that AI has traced is perfect and other locations where it's telling me to add something that is definitely not what's on the ground. And so it's really up to the human to make sure that it's being added correctly.
19:43
And I think that's why a lot of people in the open street map community quite rightly think that the human should still be very much involved in the process. But if AI can at least help us get a few steps forward and save time, I think that it's definitely something that we should do, but the workflow and all these key takeaways
20:00
that I mentioned earlier are important.