Open Data in OpenStreetMap’s RapiD Editor
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69072 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Open setSource codeOpen setFocus (optics)
00:20
Data integrityAreaVertex (graph theory)Task (computing)NumberSelf-organizationText editorDifferent (Kate Ryan album)Machine learningRevision controlFocus (optics)Web applicationElectric generatorComputer animation
00:52
BuildingElectronic program guideWikiWeb pageTexture mappingSocial classType theoryCuboidSatelliteSet (mathematics)Network topologyBitPoint (geometry)BuildingInterface (computing)Computer animation
03:16
Revision controlMachine visionComputerPattern recognitionBuildingScale (map)Image resolutionSatelliteNeumann boundary conditionData modelVariety (linguistics)Open setEntire functionComputer-generated imageryArchitectureObservational studyMathematical analysisFunction (mathematics)Bounded variationInstance (computer science)InformationSource codeError messageAddress spaceObject (grammar)Local ringWikiWeb pageContent (media)View (database)Plane (geometry)Repository (publishing)Flow separationLink (knot theory)Execution unitNetwork topologyPoint (geometry)State of matterTexture mapping1 (number)Roundness (object)Uniform resource locatorWeb pageWikiOpen setGroup actionText editorGreatest elementBuildingElectronic mailing listTesselationComputer configurationComputer animation
05:26
Open setComputer animation
05:40
Process (computing)Field (computer science)Database normalizationDisintegrationComputer configurationComputer programAreaAddress spaceScale (map)Range (statistics)Digital photographyStatisticsOpen setForm (programming)Field (computer science)Network topologyUniform resource locatorInternet service providerOpen setText editorProcess (computing)Set (mathematics)Program slicingComputer programmingINTEGRALDesign by contractContext awarenessMedical imagingPoint (geometry)Domain nameSurfaceSatelliteRight angleNormal (geometry)Address spaceTexture mappingDiagramAcoustic shadowFormal languageDifferent (Kate Ryan album)System callOnlinecommunityOnline helpShared memoryBoundary value problemLink (knot theory)Greatest elementQuicksortMappingRepository (publishing)Type theoryEmail1 (number)Computer configurationComputer animation
11:25
Medical imagingObject (grammar)Different (Kate Ryan album)Virtual machineForm (programming)Open setNeuroinformatikAutomatic differentiationPopulation densityRevision controlMachine visionTexture mappingPoint (geometry)DemosceneComputer animation
12:50
Default (computer science)Texture mappingMereologyMultiplication signDifferent (Kate Ryan album)Cycle (graph theory)Open setComputer configurationTwitterTopostheorieRoutingDemoscenePosition operatorComputing platformMedical imagingSelf-organizationDigital video recorderOpen sourceMotion captureNeuroinformatikText editorLink (knot theory)AreaSoftware testingElectronic program guidePoint (geometry)Power (physics)Derivation (linguistics)Event horizonRevision controlOcean currentGroup actionFacebookWeb crawlerQuicksortWave packetMachine visionMobile appInflection pointRepository (publishing)MetreExterior algebraReading (process)Level (video gaming)Computer animation
18:17
Open setComputer animation
Transcript: English(auto-generated)
00:02
All right, thanks for attending. So my name's Christopher and I'm gonna take you through this journey on open data in this editor. We're gonna focus not so much on the capabilities and technology as much as the ability to share data and bring in multiple sources to improve OpenStreetMap.
00:22
So first off, what is Rapid? This is an editor for OpenStreetMap. It's based on what you may also be familiar with, the ID editor, which comes from also a generation of other versions of browser-based editors for OpenStreetMap. The Rapid editor has a focus on functionality,
00:40
on machine learning and data generated that way, as well as open data with a huge number of data sets, always increasing, that are shared from different organizations like government or research. And the way I like to put it with Rapid is that you are looking to both map smarter and map faster.
01:02
One person can get a lot more mapping done using the tools this makes available to you, and you can also jump ahead as far as getting fine details and really avoid a lot of the tedious work that will only slow you down, while also mapping anywhere in the world very completely and in a lot of detail.
01:26
So when you first encounter the interface of the Rapid editor, you're gonna see this Rapid button at the top, and if you click that, it's gonna show you this aspect of it that comes from the map with AI theme.
01:40
The idea there is that there is a lot of data worldwide that can be derived from things like satellite imagery data sets. This includes the roads that we generate at META, and another one you see here that's global layer is the Microsoft buildings layer. So that's the footprints for buildings
02:01
derived also from satellite imagery. So these are just a couple examples of a lot more data that's available. So if you click the add manage data sets at the bottom, it'll take you into this ArcGIS data sets dialogue, and I'm gonna talk to you a little bit about how this comes to be there and how you can get it done yourself.
02:23
So if you scroll through this, you'll see it says there's more than 100 data sets, and a lot of these are from city governments. You have things like buildings, you have addresses, and some other types that we'll take a look at. You can filter them for a certain class you're looking for
02:42
or search for the name of a city, and again, also the drop-down box will give you some types to choose from. One of the interesting types in the drop-down box is even including trees. So Esri has crowdsourced these trees from GIS users who went out and surveyed their locations,
03:02
and it's quite a large data set. I think there's something like 120,000 as of today, and again, always increasing. These are points on the map, covering a lot in the US and I think a few other places in the world. So if you launch those, you can go to a place where they exist.
03:21
Here's Round Rock, Texas, just outside of Austin. And sure enough, the map is covered in points representing the locations of trees that the city went out and surveyed. And in OpenStreetMap as a user, you can choose to use the feature. So that would mean validating a specific one,
03:41
creating an OSM point out of it, or if it looks incorrect, ignore it. Now, we'll talk a little more later about how you would go about actually validating that since in this view, you're at the bird's eye perspective, you're looking at satellite. You may not really be sure what you're seeing, but some trees are more obvious than others.
04:04
One of the other layers here is the OpenBuildings layer from Google Research. This one is available for all of Africa inside of Rapid, and the license has been opened up especially for OpenStreetMap. So that's one option to map buildings in places where there's not a lot of other open data.
04:22
But even better, globally, is the Microsoft Building dataset. So you'll see in this map, it covers also a huge amount of Africa as well as the rest of the world. The United States and Canada are not on there because that's also a separate Microsoft dataset that in the Rapid editor is just one,
04:41
so it's all merged in. And we did cut off New Zealand from the map there, but I blame Microsoft for this. So you have the GitHub repository link there on the bottom that you can go visit and learn more about how it's generated and how it's licensed, and you can get the raw data there. But we also tile it for you into Rapid.
05:02
You can find a whole list of the ESRI provided datasets in the OSM Wiki as well. So you can see here to look up ESRI slash ArcGIS datasets, and also the link is here to that page. And this will list out a very exhaustive, long group of bullet points
05:21
that will tell you what's in there exactly. And that should keep getting updated as new ones come in. There's also a map. So here's what it looks like on ESRI's website and inside of this Wiki entry as far as the US. So quite a wide spread across the US, not just concentrated in a few places.
05:42
And the really nice thing about this open data that comes in is that ESRI helps curate it. So much of this comes from people who maybe never heard of OpenStreetMap. They're just encountering the idea of open data for the first time, but they're longtime GIS users. And ESRI has a huge resource of those types of users
06:01
that as people here at a FOSS4G conference, we may feel that we're in a smaller slice of that world but enabling these people to join the open data community is a really special opportunity here. So ESRI will assemble that data from their ArcGIS user community. Anyone who has a contract or a user account with ESRI
06:22
is able to share that way. And ESRI will do a lot of the heavy lifting for them on making sure that the license compatibility is there. They'll then also process that data. We'll take a little closer look at how here on the bottom half of the slide. And then ESRI will publish that data into ArcGIS Online, which means it also shows up in that rapid editor
06:42
as an open data set. So some of the more technical aspects of making sure that data is cleaned and prepared is really just handling the fields. So normalizing the field names is a big deal. You may have field names in different languages depending on the country.
07:01
Some may call something business underscore name, others may call it a registered underscore name. So things like that need to be merged into just name and address fields so it's all consistent across data sets. And we also wanna make sure that those are matching OSM conventions, so ESRI's handling that ahead of time
07:20
so that the user community is not left doing cleanup. And finally, you can see an example here. So street may be the name of a column in an ArcGIS, let's say CSV file, and it may have things like West abbreviated. So the normalization's gonna bring that over to the OpenStreetMap tag name
07:41
and go ahead and make sure we expand the full name of that street as well. And you can do this yourself. So again, a quick review, just in a diagram here because that's a lot better than text, I think, for most of us. So we're going to map our way through this journey. You'll start with your data. It gets prepped for OSM,
08:02
again, handled usually in the ESRI domain. And then it becomes an ArcGIS data set and emerges out into Rapid where it's available for OpenStreetMap. So one way that this happens, if you wanna go ahead and give this a try, you can share your data via ESRI's community maps program so there's a link at the bottom here
08:21
is where you can join that. And this is gonna help you get the CCBY 4.0 open data license. You'll see dialogues like this. You'll indicate where you are in the world and you'll also indicate what kind of location you're at. So you can see here that you can even upload a shapefile
08:42
as your boundary in a very ESRI way. Another way that you can do it is share via an ArcGIS hub site. So this is sort of a repository for open data. So if you have an ArcGIS hub account, these steps will get you there.
09:01
Basically, you're just gonna kind of create a repo, give it a name, upload the data there, select that you want it to be open data, and the last step is to email ESRI. So they have an email address especially for that where then they get a notification to start handling the integration and license vetting. So here's a lot more detail. So if you're watching this online
09:21
or catching this afterwards, it's great to take a deeper look into this. The first two options we already covered. The third one's even simpler really. It's just upload your data to ArcGIS online and send an email to ESRI again and ask them to go ahead and get started on it.
09:41
So jumping back to the editor process. Here we are again in Round Rock, Texas with all these trees. And before we add them to the map, we want to know that they really exist and that this open data is good data. So a nice way to do that is use another form of open data, and that's Mapillary, the street-level imagery contributed by the community.
10:02
That includes casual users. It includes corporations. It includes small contractors, city governments, nonprofits and NGOs. All of these providers are rallying around Mapillary, uploading imagery to it, it's geo-tagged,
10:21
and allows you to see a date of the imagery, the location of it, and the context of what you're looking at on the map. So for trees, it's a really perfect way to go about it, but also for many other types of data. So maybe the roads themselves, the buildings, points of interest. In this particular case,
10:40
we can launch one of these side-facing images. So it looks like the camera is probably on top of a car facing, as you can see on the map, left, right, and forward. So this side image, I went ahead and used the scroll wheel to zoom in on the tree. And luckily I could also see the shadow on the map of it.
11:01
So I circled this palm up there. There's more of a leafy tree that's right in front of the camera to the right of where I zoomed. But it's a very clear image that, okay, I have an image here. I can see it's very close to that curb that separates the kind of dirt surface from the parking lot, matches what I see in the satellite.
11:22
So I can go ahead and use that feature. It's gonna convert it to a point in OpenStreetMap, and I've validated it, I can move on to the rest. So a great way to really make sure that you know what you're adding. But with pre-suggested points. So on the Mapillary side, the computer vision is a really powerful way
11:41
of providing both this kind of map with AI theme, that's data derived using machine learning, that also becomes a form of open data once again. So the imagery itself is user contributed as we covered. And what's derived from the images is kind of the magic of Mapillary.
12:00
You can see here on the left side that we have these segmentations around different items. So things like cars are removed from the scene. It's a dynamic object. We don't really wanna know where a car is on the map, but we may wanna know what else is around it without the car blocking or interfering it. Using multiple images,
12:22
so the green you see here, a wide density of imagery, that scene is actually reconstructed in 3D. So it means we can remove things like cars, but start to assemble a version of reality showing us what's right there. So we'll find a trash bin, we'll find traffic lights, stop signs, buildings, and street lights,
12:44
all kinds of different pieces of infrastructure that we can also bring into OpenStreetMap. So in this case, back in Round Rock, Texas, we're taking a look at a bench. It's in blue. Mapillary tells us that blue bench is on the street corner
13:01
and checking the image, we can see, okay, it's about maybe two meters off, three, five, have to guess, but it looks like it's in reality in between these two gas pumps underneath the roof of that awning. So with this in mind, we're able to then create a point for that bench
13:24
and we can drag it under to where it's supposed to be and that way we've just used the Mapillary layer as a guide. So it's, especially with smaller items, not going to be perfect, but it really helps us get started on seeing what's available. The alternative being you can take the imagery and sort of crawl down the street with it,
13:42
check what you see, and manually add it to the map. That's also a great way to add businesses and POIs to the map. They will be harder to detect with computer vision. There's no logo detection or text reading going on, but you can really quickly also map the main street of any town in the world, just checking the Mapillary imagery.
14:04
There's also a nice little feature that suggests the best imagery available. So here I am looking at an area in Switzerland and you can see this red check, making sure that we know it's certified next to the Swiss Topo imagery. And so that's kind of a way of letting us know
14:21
that before we cycle through all these different imagery options to see what one is best, that's probably the first one we want to check and maybe the one that shows up by default when we're there. So it's also a handy way to save you time from guessing and checking through all of this.
14:41
So what's next? There's a lot of things that could be done with this Rapid Editor. One of the things on this data side that we want to do is make sure more open data comes into Rapid. We want to make sure that we're spreading the word, so that's part of what I'm doing here is making sure everyone knows if you have open data, you can get it in there.
15:02
You can go the Esri route if you have other questions and ideas, you can talk to us directly and I'm sure we can find a way to make it work. And the big goal there is just get as much open data as possible in front of OpenStreetMap users to improve the map. Also, Mapillary imagery is really valuable
15:21
and the more of it in any place, the better. And having better cameras, GPS quality, users who are just more familiar with how to capture high quality imagery is really valuable for improving OpenStreetMap as well. So every day we have a huge volume, I think somewhere in hundreds of thousands,
15:41
sometimes in the millions of images coming in on Mapillary's platform. And if you're able to contribute to that too, or you know an organization that's able to contribute and find value from it, encouraging that will help improve Rapid and improve the map. Mapillary data accuracy can also improve. There's a lot of ideas around things
16:02
like aligning the Mapillary imagery with satellite-based maps, with helping detect and analyze differences in drifts and GPS and suggest corrections. The 3D scene reconstruction helps to improve imagery positions. So all of this is advancing and going to make it just better and better all the time
16:21
to derive data from Mapillary. Finally, we'll have just in a few minutes here a talk about a great way to make Rapid better, which is just improving the entire engine behind it that powers it. So you'll see how it's, I like to say, stronger, faster, and better. And behind all that rallying around it
16:42
is growing the community. So the community of contributors, people who are uploading to Mapillary, sharing open data, people who are editing the map, and testing it, finding bugs, demonstrating workflows, and then people who are also contributing to the code base. So as an open source project,
17:01
it's out there for everyone to see, understand, and take part of. So getting more people involved with that as well means that we can have more ideas on how to make a great tool to edit OpenStreetMap. And finally, we encourage you to join the community. So a few links here. You can get started with Rapid at the link above.
17:22
So that's the current version one. You can join the Facebook group around Rapid. We have, I think, about 138 members today. And then we share a lot of news and events inside that group, video recordings and training. You can also get to the map, or the Rapid repository on GitHub.
17:42
And again, that's totally open source for everyone to get involved with and start understanding. And finally, there's the Mapillary app. You can check out street level imagery coverage around the world. You can download it on your mobile phone, start capturing and contributing. You can find us on Twitter at both of these Twitter
18:01
handles and yeah, tweet us what you're working on, examples of what you're editing and mapping. I'll try to retweet it and come up with something clever to comment about it and get you noticed by the rest of the community so we all know who's where and what corner of the world helping to improve the base map. That's all and I'm open for questions. Thanks a lot.