Welcome everyone to our workshop and presentation on using Rapid to map open data. I'm Jeff Underwood and presenting with me today is Yunzi Lin, Ben Clark, and Dean Kinsock. Today we're gonna talk about a few things. First, like what mapping on Facebook,

what was Facebook doing in OSM? Then into Rapid and JOSM, map with AI, our Esri ArcGIS dataset integration with Rapid, and then finally using Rapid with the hot task manager. And once we conclude, we'll transition back to the lounge and we'll have an open Q&A session for anyone that wants to ask anything,

and we'll just chat and do some mapping. All right, take it away, Yunzi. Thank you, Jeff. Hi, I'm Yunzi. I'm a QA analyst at Facebook. So why does Facebook care about maps? So the answer is simple. We have maps in a lot of our products, like marketplace, crisis response, recommendations,

check-ins, and so on. And OSM is the major data source on our map. Therefore, since 2017, we developed technology of AI-assisting road mapping and established teams to actively map on OSM. So far, our road mapping project covered Thailand,

Indonesia, Malaysia, Vietnam, India, and Tanzania. But we are not just working on adding data to OSM. We have also actively contributed to fix improper data on OSM. Especially in the last year, with the improvement of our vandalism and profanity detection technique,

we're able to fix over 360 samples of vandalism and 16 instances of profanities on feature names all over the world on OSM. Here are a couple of examples. This one, we detected vandalism of the wind name values.

As you see, the name value is improper there. And further investigated, we decided to remove the name tag and we submitted a train set with the removal. This example is similar. The name value here is improper, so remove the name tag after further investigation on this building.

And this one is related to a Pokemon Go edit, which is normally considered as vandalism. Also, with further investigation, we moved this name tab of the features on the map. And when a profanity were involved in the name, we reflect this map feature as profanity features.

With further investigation, we also find out the polygon itself is invalid on the map. So we end up removing the entire feature, including names from OSM. And for this one, profanity is in part of the names.

So remove the profanity part, but still keep the part, the legit part to align with the road surrounding in this area. And this one is very similar. We also detected the profanity named on this water body. So remove this name tag of the water body.

But we also noticed there's drag now is this on the road near the lake. So we submit the separate train sets to remove the drag node. It shows on the next slide. This is the train set, remove the drag node.

So we also enhance our detection on broken relations and repair over 4,100 relation OSM. Oftentimes, these repaired relation, they are as complicated as the example here, and it sits on live OSM for quite a while. So we're proud to do all these repairing on the relation.

And other than using improved internal technique, we apply an open source validation tools called Atlas Track to detect various data issues, including line crossing water body improperly, name gap in connected roads, missing relation type, overlapping rates, and so on.

We submitted more than 7.9K train set to repair over 122K OSM features. For example, this one is from the Atlas Track called line crossing water body. It shows that there's a highway improperly overlap with part of the water body. And further investigating,

we notice the water body is a bit out of date. So we adjust the water body to avoid this overlapping with roads. These are examples from the line crossing water body track. So we find out this highway crossed the river polygons improperly without the bridge tag.

So we added by splitting up a segment of the road and add the bridge tag information. So we can bring this highway properly across this river polygon. And this one is from the Atlas Track called road name gap.

You can see this bridge here, it was submitted and added later than the rest of the road so there's a name gap happens there. What we do is we add the name back to this bridge so we can have a continuous name on this highway.

So for Atlas, we use matrelay as the platform to systematically fix Atlas Track outputs on OSM. And we are not limited to present all this Atlas Track to our internal team. We also share a couple of ones to OSM community in early April. So if anyone is interested in supporting

these Atlas Track fixes on matrelay, please feel free to message us on Slack or email us on osmfacebook.com. This is a mapping essay for Facebook. Now I'll pass to Ben to talk about rapid and jossing. Thank you. Great. So most folks here are probably familiar with ID, which is sort of the default open street map editor

that's browser-based. And I'm just gonna take you through sort of the evolution of what Facebook has done with ID and also with our jossum plugin. But this will be a pretty quick little section, but don't worry, there'll be plenty of time afterwards for you to get your feet wet with using the rapid UI that we've created.

So go ahead and next. Yep, so we'll start with an Esri dataset. These are straight from the ArcGIS website. So this is something that you can go navigate to right now. And what's cool is that up until recently,

we have had a really good chance to sort of work with Esri and integrate all of these datasets directly within the tooling chain that Facebook has created with both a jossum plugin and with an add-on to rapid, which is a clone of ID. Go ahead and hit next and once more.

Yeah, so on the left you have ID and then after Facebook cloned ID and forked the project, we added some of our own road and building detections on top of it. These are global and they're predicted from satellite imagery. So in cases they're not perfectly aligned and there are some issues occasionally with geometry.

So we went looking for a way to get better like human approved data, so-called authoritative datasets and Esri has that kind of data in spades. So this third thing here is where we worked with Esri to add even more functionality to rapid to compliment our global datasets

that are predicted from satellite imagery. We have this third data type that is authoritative data that may come from NGOs or municipalities for example, a county building dataset or address list. Go ahead and we'll show you, this is ID. So this should be familiar to many of you

and next you will see what the same thing looks like with our predicted buildings and road layers looks like and next you will see sort of a walkthrough of what our new Esri data story looks like. So you can hit next a couple more times.

We've got this dataset thing that you can open up. You can pick from these datasets and then once you have selected a dataset, you can clearly see over in the left that we've got all the tagging and addressing that comes to us from Esri. It's just built right into the tool. So a lot less time spent dealing with tags and you can also tell that some of the geometry is like seriously excellent.

You can see that huge building over to the right which our AI predictions may not be able to be quite so accurate with. So it's really quite a nice update to our capabilities and these slides are actually a little bit dated. So we have a new UI that we actually just released about a week ago that you'll get a chance to look at later in this session.

And next we'll talk a little bit about JAWSOM. And so we have our JAWSOM plugin which we internally refer to as JUMP. And basically same deal here. You can look, this is the stock JAWSOM UI and if you load the Map with AI plugin

which is available on the JAWSOM plugin site then you get the same sort of story. You can load all of our AI predicted buildings roads and also use the JAWSOM plugins new Esri integration functionality. And as I said, we'll have plenty of time to go over this in a little bit. So I'll hand things over to Jeff.

Thanks Ben. So I wanna talk a little bit about Map with AI. Ben touched on it a little bit that we launched rapid with road predictions and UNC had mentioned that we've been mapping many of these countries using the same road predictions. So what exactly are these? In rapid, if you've never used it,

if you jump into the world anywhere that's missing, that's not completely mapped you'll see many pink roads like this. And all of these are AI predictions actually. So how do we do this exactly? Well, it starts with a global dataset for training data. We literally sample all over the world from OSM.

We use over a million one kilometer square tiles of road data evenly spread in terms of where data is. And we fed that to our machine. And so that's our prediction dataset. And that massive amount of data lets us actually have a very robust model.

And so we can basically run imagery from anywhere in the world and get a really solid result for predictions. And that allows us to have rapid available everywhere in the world. So what does this look like at a very high level? Well, we take training data and satellite imagery and we feed that to a machine

and that gives us our road predictions. Our machine learning is a type called deep learning. And we take this prediction result and it's basically a picture and needs to be turned into line data. So that's our next step. But that line data is just kind of on its own to start with. We didn't have to merge it with OSM itself.

And finally, that lets us edit it with rapid, submit it to OSM and view it on the data once people decide to upload. We of course have a quality assurance pipeline that is going on in the background of all of this. Every time we generate data, we take a look at it and we see what can do better. And we've been at this for about five years now

and we've made improvements consistently throughout time. The technology only matures. So this is just words, pretty abstract. So let's take a look at this visually. So we start at this area, I believe this is in India. Here's some satellite imagery from Maxar.

And we run this thorough machine. What we get back is something like this. This is in actuality a black and white or black and pink in this case image. And you'll see that they are of varying widths and thicknesses. Like here, you might see that there's actually a little bit of a weak spot. And if we go back,

you can see there's actually tree cover here. Trees and buildings obscuring roads are one of the reasons that our predictions can fail sometimes or be weaker because there's just nothing visually to confirm that there's a road. So going back to the black part, you'll see that there's lots of predictions that are stronger or weaker than others. And that's for lots of reasons.

Sometimes it's because it's not a road, but it looks maybe road-like. Sometimes it's because there's coverage. But what we get is a very strong prediction for much of the road. But this is ultimately just a picture and we can't map with just the picture automatically. We'd have to turn this into line data. And that's a very big task actually.

Turning these predictions into vector data has been the ongoing project of Map with AI for years. The machine learning itself matured quite a while ago, but all the improvements we make nowadays are generally from vectorization. Like how do we make better intersections? How do we make better geometry? How smooth should a road be? How many nodes should it have?

And these are really hard questions actually because you change one thing and it breaks other things. So once we've done this process though, we just have vectors. But if we take a look at OSM in this area, here's OSM, we have a road running through this area. It's already been mapped. We don't need to present that to the user.

We don't need to deal with this road. We need to conflate away our duplicates. So that's the last step. We conflate away, we add tagging. And finally, what we present to users in Rapid is this, which is a missing roads layer that will connect to this OSM roads. It has intersections with it and it makes mapping just a super easy thing.

Here's our coverage. We, again, like I said, we have coverage almost in the entire world. So it's very easy just to pick up and go with Rapid. If you want to map somewhere, just open the tool and zoom in and you will probably be able to use it. In addition, Microsoft has a AI building data set

and that's available in the US, Canada, Australia, Tanzania, and Uganda. So that's another AI data set that we have available in Rapid. But the thing we're really excited about today is our Esri data sets that have been made available in this newer version of Rapid that Dean's going to talk about now. So take it away, Dean. All right, thank you.

If you can unshare that, I can share my screen. Here we go. All right, can you see my screen okay? Yep. Great. So thanks, Jeff. Yeah, I wanted to talk a bit about some of the work

that we've been doing with Facebook to integrate ArcGIS data sets into Rapid so it can be added into OpenStreetMap. So Esri's goal in this is to help improve the quality of OpenStreetMap by providing easier access to authoritative data sets that are available from the GIS community. Most of these authoritative GIS organizations

such as a city or a county or a national mapping agency are using Esri software called ArcGIS to build and manage their data. So there's a lot of data in ArcGIS format that's accessible for use in this type of application. Many of these organizations are already openly sharing their data

through things like the ArcGIS Hub or Esri Community Maps Program. And many of these organizations would like to see their data integrated into OpenStreetMap, but they just don't have the resources to do that work themselves. So Esri is looking to help facilitate the sharing of the data from these GIS organizations,

this GIS community to the OSM community. So how can we help with doing that? Well, first we can promote the value of doing that, the value of having this type of data integrated into OpenStreetMap. We make that data available to the ArcGIS community. So many of them benefit directly that way, but they're also motivated just to see their data

used more widely in their local community. And then we can also help them enable the sharing of that data through our platform. And then once they've done that, there's some additional things we can do. One, we like to ensure that the data they're sharing is open and compatible with the OSM ODBL license.

So we can help validate the compatibility of the open data with the OSM license. And then we can help with the preparation of that data to be integrated into OpenStreetMap. Most of these organizations manage their data in their own custom data model, which is different than what OpenStreetMap expects in its data model.

So there's work to be done by somebody, could be Esri, could be others, to convert the data from its native format, native data model into the expected OSM data model. So last year at the Esri User Conference in July of 2020, we announced some of this work that we were planning to do with Facebook.

And we encouraged our users to begin sharing their data. We have a program called Esri Community Maps, where thousands of our user community members are sharing millions, actually hundreds of millions of features with us to go into our platform. And we introduced a new data sharing option whereby checking a box,

they could give us permission to share that data beyond our platform with other mapping platforms, specifically including OpenStreetMap. And then we talked about how that data would become available through an enhanced version of the Rapid Editor that would support ArcGIS. So that was about 10 months ago that we did that.

So flash forward to kind of where we were last fall, we had an OSM Connect event where we introduced some of these new tools in preview mode. And we had a sampling of data sets that were available at the time. You can see on the map about a dozen or so data sets that we had integrated as we kicked off this program.

And then if we kind of flash forward to today, you can see that there's now dozens of data sets that are available in Rapid for use within ArcGIS and within the editor. So let me go over to my screen here.

Let's take a quick look at this. So I'm now over in ArcGIS online and we're looking at an OpenStreetMap map kind of in middle America. I think this is Downingtown, Pennsylvania. And I can see on this map, I'm gonna zoom into a residential area. Looking at OSM, I can see it's lacking some data

in this area, but looking at the relief behind that, I can see that there's actually probably a development area in here. I can see kind of what looks like a residential community. If I turn on the imagery, I can confirm that and say, yeah, there's a pretty large residential area with roads and buildings that aren't being currently mapped.

If I look at the Esri map that we have for this area, I can see that there are a number of buildings that are in the Esri map that aren't in the OpenStreetMap map at this point. So I can kind of toggle between them. So several dozen features in this one little area. So this is data that's been contributed to us

through the Esri Community Maps Program. So Chester County, Pennsylvania was one of our contributors that did opt into sharing their data. So we now have access to their buildings and addresses. So if I turn on their buildings layer, you can now see that there's a large number of building features that exist

in this part of the country. And if I click on a building, I can see a little bit of attribute information. In this case, it's pretty sparse. I have a building tag with just a value of yes, and then I have the state information, but the buildings themselves didn't have a lot of information. Well, there's another data set that Chester County provides, which is address information.

And you can see that there's a bunch of address points that in most cases overlap nicely with these building footprints. And if I click on an address point, you can see there's more detailed information, the house number, the street name, the town, state, and so forth, postal code. And one minor thing that I wanted to call your attention to

if I compare, let me do it over here. If I compare the attribution, you'll notice that for this address point, turn this layer on, click this one here. You can see in our street data,

we have Westerham RD, the abbreviation, which is not the OSM convention. OSM prefers to have street types and suffixes spelled out. So as part of the conversion of the data from the native data model into the OSM data model, we're doing small things like that, converting the values of road names to be what's expected in OSM.

So this is a pretty good example of some features that could be added into OSM through ArcGIS datasets. So we do have a compiled list of all these ArcGIS datasets that are now available on the OSM Wiki. So you can kind of scroll through this page and get a sense for the different communities

that are available and then the status for them. Some of them are available for the community to review. Some have been reviewed and are available to preview, and then some have been previewed and they're now available to edit. For a little simpler version of this list, we have a live web app that you can link to, and it takes you to this application.

So this application shows you the locations of all these communities, and then it has information on their status. So all these kind of pink triangle square areas are showing the different communities that are available for editing. So I wanted to do a little editing in the Colorado area

using a couple of datasets. So one is the United States addresses dataset from the USDOT, the National Address Database, which was made available by the USDOT as a public domain dataset. And then I can see that there's several communities in Colorado that are available. The one I was gonna work on is in Aurora, Colorado.

So for any one of these communities, I can click, find out its status, its license information. I can also then click through to a link on the Wiki page where I can read more about it. And then if I actually wanna interact with the data, maybe download the data to check out in my local environment before mapping,

I can do that from here. And then what we do is take all of those layers that we've published, and we share them through this group in ArcGIS Online that has all of the different datasets organized by types like buildings or addresses. And then this group is queried by the Rapid Editor

to bring these features into Rapid to go into OSM. So let's go ahead and do that. Let's go into Rapid. So I've got a version of Rapid here that I'm interacting with. This is what Ben was showing earlier. So I'm gonna click the Rapid button. And in addition to the Facebook roads

that Jeff was describing from the ML predictions and some Microsoft buildings that Microsoft has developed from ML predictions, AI predictions, there's this new option to add and manage datasets. So if you click that, you'll see a collection of these ArcGIS datasets that are coming from this group that I showed earlier.

So this is being updated continuously. Anytime we add a new dataset, it'll immediately appear within this gallery. And you can do some filtering in this new user interface. So I can look for some datasets that are featured. I know one of the datasets I wanna use is this United States addresses and this feature. So I'm gonna add that. And then I wanna work with some buildings as well.

So I'm gonna go to buildings. I know I wanna work in Colorado. So I'll start typing Colorado and I can find this Aurora dataset. So I'm gonna add that dataset to the map. So I've added two datasets and I'm gonna work with these specific ones. So for Aurora, I'm gonna work in this area. I'm gonna center my map there.

We're gonna try to zoom in to Aurora and we're looking at some imagery. I think in this case, we're looking at the Bing imagery and I'm gonna zoom into an area where you can start to see some features

appearing on the map. So here you can see in green, these Aurora buildings that are available and then in yellow, these address points. And earlier today, I did a couple edits. You can see these three buildings have been added. They're in red. That means they've been part of OSM already

but these other ones have not been. So I'm gonna add a couple more of these buildings. So how does that work? Well, it's pretty simple. You can click on a feature and select, you can kind of preview the attributes like we looked at before. If you wanna use this feature, you can just select that option and I can select another building next to it.

You can also use the A key just to quickly select that. So I'm gonna work with these two buildings here. So I can see that the geometry looks pretty good. It's aligning quite nicely. It doesn't look like there's many extraneous points but if I wanna confirm that, I can use this square feature and if there's any extraneous points,

it'll clean that up for me. And then I'll do the same thing here. It also looks pretty good but I'm gonna square this off just to make sure it looks good. And if I wanted to, I could move this feature a little bit to relocate it more precisely if I felt like it needed to move but that actually looks pretty good. So I'm gonna leave it like that. So I've got these two buildings

but all I know now is that they're building yes. So I can select both of these. I can pretty much tell from the imagery that these are houses. So I'm gonna go change the type to house. So I've added a little extra value than what came in the dataset and I can see these address points are available here too. So I'm gonna go ahead and click that. I can preview the address information.

I'm gonna use this feature. So you can see in addition to the building information, now we have the city, the house number, the postal code, the state, the street. I can see that the street name matches the street name in OSM. So all of that looks good. And I'm gonna do the same thing with this other feature.

Use this. I can see it's got the same street name. It's got a different house number than the street next to it. All that looks good as well. So I'm gonna go ahead and use those. But in this case, what I wanna do is add the address point to the building rather than make it a separate point. I like to join the address information.

So to do that, I can select both features and then push the C button and it'll automatically conflate that. So now my house building feature has been updated to have the address information that came from the address point. And I can do the same thing for the other feature. Select those two, press C,

and it's conflated those things together. So now you can see I've got two features, two buildings with addresses now, and they're up here. So I can go ahead and save these features. A couple of buildings in Aurora, Colorado.

And I'm gonna go ahead and upload that to the map. So that is now uploading those edits that I've made to OpenStreetMap. And we have another map. Earlier, I edited a few features that I mentioned, and they're showing up in this live feature layer from ArcGIS Online. So if all things are working properly

in a minute or two, those features will also appear in these live feature layers. While that's happening, let me go back quickly to my slides. Kind of how this works. Let me explain a little bit about what's going on behind the scenes. So we have this kind of multi-step process for getting data into this ArcGIS datasets

for onboarding them. I'm gonna quickly step through it. So the first thing is identifying data. So we're looking for data that would be useful of high quality and compatible with the OSM license. So we check to see whether these features would add value to OSM. Are they missing an OSM? Are they the type of data that OSM would like to see?

Are they of good quality and accuracy? Do they come with a compatible license? All of the data shared through Community Maps has got a very much compatible license. It's explicitly designed for use in OSM. We then can obtain the latest version of the source data, analyze the source data. Maybe it's too detailed

and it needs to be generalized or thinned. We might need to transform the field names and values like I described before, converting it into OSM tags, applying the correct values. Then we publish that processed data into an ArcGIS feature layer in ArcGIS Online. We document the data on the Wiki and then we share it with the community.

The community then has an opportunity to review it. They can discover it on the Wiki. They can download the data if they'd like to work with it. They can add comments on the Wiki page if they see any issues with it. If issues are reported, we'll review those and refine the data set if necessary. And then we'll add it into Rapid in preview mode

so it can be integrated and tested that way. And then if it looks good and there are no remaining issues, we consider that community approval to proceed and we'll go ahead and share it so that anyone can access that data in the editor. And then at that point, it's available for anybody to use within Rapid

like I was using a minute ago. So in terms of opportunities to participate, there's a few ways. Pretty much at each of these steps, there are opportunities for the OSM community. One, if you have any recommendations for data sets you'd like to see included, you could help identify those. There's been several recommendations we've seen

like Johns Creek, Georgia saw a presentation like this and they reached out to me through this osm.esri.com alias and volunteer their data. So now it's available. You could also help us prepare data. So if you know of data that would be available and you'd like to help prepare it into an OSM format, you can do that. And if you'd like to help publish that data to online,

you can also reach out to us via email. We can set you up with an ArcGIS account if you don't have one for free and then you could use it to publish the data. Once the data has been published, you're more than welcome to help us review the data, make sure it's gonna be compatible for OSM from a technical perspective. And then once the data is added to OSM

and the rapid editors, I should say, you are very much welcome to start adding that data into OSM through those tools. So let me bounce back to my map. And you can see those features that we added a couple of minutes ago are now appearing in these live feature layers

in ArcGIS online. So I can click on those features and I can see all the attribute information that came through. So you kind of get the immediate gratification of having live access to this data as feature layers in ArcGIS, as well as having it appear within the OSM base maps over time.

So with that, I think I will pass it back to Jeff. Great. There was one question during that, which may make sense to address now. Is there a recommended way to get data shared? For example, Maricopa County in Arizona or cities that are in it, do you email the county city offices

concerning that concern mapping? Good question. Yeah, so I would say there's a couple of ways to do it. If you have a contact at the organization, like the county that has the data, feel free to reach out to them directly and encourage them to make the data available.

They might be making the data available as open data already. And if they point you to a place where it can be accessed, then you could follow the steps I described, reach out to esri.osm.esri.com, and we can help do that. Or if you don't have a contact at the organization, feel free to just reach out to me through that email alias, osm.esri.com.

And in most cases, we have relationships with those organizations and we can reach out to them and help facilitate that. We've done that with several dozen organizations. All right, great. So now I wanna move on to briefly just talk about using rapid with the hot tasking manager.

So rapid works great on its own, but especially for some of these big building data sets, maybe you wanna do a more systematic or organized editing approach. So how can you do that? Well, luckily there is a custom editor option nowadays in the tasking manager, which makes this very easy. You go to the edit product page

and find the custom editor panel on the sidebar. From there, you can fill in a name and description, and those can be whatever. In this case, rapid, rapid editor. And then all you need to do is stick in a URL. So in this case, this is our map with that AI slash rapid URL. But if you had your own fork of ID

or your own version of rapid or whatever, you could also stick it in there too. It's actually a great little tool that was added in. Mark it as enabled. And then when you return to the settings for your editors, there is a new field now, custom editor, rapid. Just check those boxes to enable it and save.

And when you then go to actually edit tasks in the tasking manager, you'll see an additional editor as an option, rapid. So that makes it very easy to integrate rapid into any of your projects. If you are interested in trying out a project that has this enabled, I created one for this event using one of the Esri datasets.

So it's tasks.hotosm.org slash projects slash 10938. And we'll display that again later on, but you can also search map with AI, I think, before I show up. So yeah, if you're looking for a specific thing to work on you can try that out.

So finally, we just wanna talk about what's next for rapid and map with AI. So, we're always thinking about new Esri datasets and data types. Roads and POIs are certainly things we're thinking about now.

We'd love to hear more from all of you when we move up to the lounge. Let us know what you're interested in. There's lots of data out there and what you care about will help us prioritize. We are looking at bring your own dataset options. Many people have data and they may wanna store it elsewhere or use their own systems.

And so we're considering how to make that work as well. Performance improvements is something that's been on our mind for a long time. ID and rapid as well have major performance issues when it comes to like dense urban areas. And we're starting to have lots of data sets in those sorts of areas. So we're thinking about how we can improve that.

Of course, model improvements on our AI and vectorization improvements as well as part of that. We're always looking to make things better. And that's just an ongoing process. We're considering looking at like sidewalk mapping. How can we use AI or better tooling

to make this a easier process for everyone? Pedestrian mapping is kind of like the next big thing on OSM, especially as like roads get mapped out. How can we make the map better for users to walk around with? And that concludes our presentation. We have some links here which also show in the lounge.

And if you have any questions outside of this, you can also reach us at osm.fb.com.