A Brief Technical History of Open Historical Map
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 41 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/58233 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
State of the Map US 20225 / 41
1
4
7
8
14
18
21
22
24
27
28
30
37
39
00:00
Time domainPoint cloudCodeComputer hardwareService-oriented architectureWebsiteArchitectureData structureFile formatFuzzy logicInstallation artWebsiteCASE <Informatik>SoftwareProcess (computing)Surjective functionPoint cloudCodeOpen setSystem callTexture mappingHard disk driveCuboidDomain nameRootFile formatDemo (music)Level (video gaming)QuicksortRaster graphicsCloud computingProjective planeMultiplication signState of matterTesselationOnline helpVector spaceMappingCombinational logicFront and back endsHacker (term)MathematicsBitPoint (geometry)Source codeTask (computing)Vector potentialRight angleSingle-precision floating-point formatSynchronizationExecution unitNP-hardInstance (computer science)Data managementComplex (psychology)SpacetimeUniverse (mathematics)Game theoryMetropolitan area networkLine (geometry)Natural numberFuzzy logicComputing platformDifferent (Kate Ryan album)Physical systemInformation technology consultingApproximationMultitier architectureAttribute grammarHand fanSinc functionPerspective (visual)MereologySoftware developerDomain nameWeb-DesignerServer (computing)System administratorTerm (mathematics)ArchitectureText editorBackupService-oriented architectureIntegrated development environmentFacebookPresentation of a groupPC CardWater vaporParameter (computer programming)Musical ensembleVolumenvisualisierungNormal (geometry)InformationWeb pageLibrary (computing)Electric generatorSlide ruleArtistic renderingContext awarenessDebuggerOpen sourceGreen's functionCartesian coordinate systemRange (statistics)Video gameDisk read-and-write headCycle (graph theory)Trail
Transcript: English(auto-generated)
00:05
This seems like a perfectly appropriate start for this talk. I'm being in the wrong room, and then you're sort of, yeah, chasing a technical history of open historical map.
00:20
But in my head, what technical histories are social histories, essentially. So it's going to be chasing both a technical and social history of this project that sort of meandered its way through the universe the past 13 years now. I see Richard sitting here and kind of looking up, and it's been a while since folks have been at it.
00:42
And it's followed an interesting journey. I feel grateful to be part of from the beginning and then part of some snapshots of that journey. And so I'll be sharing some of it from a personal perspective and then also cover some of the tech that's running the site and then also some
01:01
of the kind of crazy challenges we face into the future. I'm not sure we all knew what we were getting into when we started this. It just seemed like a fun thing to do. Cool, yeah, so who am I? I currently work with a consultancy firm called Development Seed. I've been with them for the past three and a half years
01:22
or so. Nature of consultancy work on a broad range of projects. And the system architect of the International Federation of the Red Cross Go platform is one of the things I do and then a bunch of other fun stuff. I was involved with the Open Historical Map
01:40
project right at the beginning. I was working for a small firm called Topomancy, along with Skylar Earl, Tim Waters, and Shekhar Krishnan, which where some of this was seeded, of course, it's a much broader community by Topomancy. They just sponsored the initial hosting
02:03
and then purchased the domain name. That's kind of where it's seeded. I worked with NutBox for a few years after that. One of the things I'm proud of that NutBox was working closely with Billy Marcel on OSM-CHA. And I developed in Seed. I'm also the lead developer for a project called OSM-Seed,
02:22
which will become relevant later in the presentation. Cool. Is this actually going to work? This is not going to work, is it? Okay, so it might work. That's great.
02:41
Okay, so video is supposed to play. So if you can just imagine me moving this time slider and seeing the history of the continental United States. So you can go to Open Historical Map. And if you go to the continental United States, you will see a time slider. And I can promise you, if you move the time slider,
03:01
you will sort of see what users have entered as data about changing time. Now, this is kind of, well, this was supposed to be a cool animating demo, which it's not, so I'll just move on. Okay, so let's do a quick kind of history recap
03:24
of how we got here. In 2009, I believe, is when the domain shows us having been registered. That's when, I believe, Sheikha Krishnan from TaubmanSeed purchased the domain. At that time, we were working with on a historical data tier with the Library of Congress
03:44
and the New York Public Library. And I think there were some ideas of, oh, there should be some sort of open software to do historical mapping, right? That sounds, we were all big fans and users of Open Street Map.
04:02
And then we ended up building the software for MITL and Library of Congress, which it worked for them. It was great, but it was a lot of complexity. It was rebuilding a lot of things.
04:21
I'll make sure I update the slides and share the code for that. But then towards the end of that project, I think on a dark, Tim Waters, who I don't know if folks here know him, he used to run a project, a great project called the Map Over. It's like, what if we just use the Open Street Map software, right?
04:42
Like, what if we just use Open Street Map and add two tags, start date and end date, right? So it'll be exactly Open Street Map. We'll add two tags, start date and end date, and then we'll be able to show features by date. I was highly skeptical.
05:01
And I remember asking, I think, at one of the State of the Map conferences in New York, asking Mikel, how is this going to work? Like, how are we going to, how is the Open Street Map software going to adapt to time? And he was like, oh, it's really easy. We just, we just want more tile and add a T parameter, right?
05:20
So we just have X, Y, Z, and T, and what tile, we'll just render tiles by time. That never happened. And now the technology was in a different space. The world was in a different space in 2009. 2013, I think, is when we had the first,
05:42
so we were like, you know what, we're not sure how we're actually going to display this time-based data, but what makes sense for now is to set up the Open Street Map software and have people enter data with a start date and end date, right, and we figure out showing it down the line. And there was an amazing community of people
06:01
who were like, okay, we don't want to use OSM. We have all this historical data that we really want to map. We really want to be open. And a kind of small, but dedicated community emerged, adding data into Open Historical Map, even though there was no spectacularly good way
06:23
to get it out. I know at some point, there was a demo of the raster tiles, which is the kind of Burning Man set up from one year to the next, rendering on the raster tiles, but that's sort of the best I remember in terms of visualizing what this could be.
06:41
But you know, a community that grew between 2013 and 16. Now, we were running this at that time on a single box that cost about 35 euro a month. As I said, they were much simpler times.
07:01
And because I was working at Topomancy, no one else wanted to do it, I ended up being kind of sysadmin on that box, even though I didn't quite know what I was doing. I had to put on that box, I would try and log in and run updates once in a while. There were folks like Tim Waters,
07:22
which is one of the people that set up kind of a lot of the software on that box. And people were plugging away at it. At that point, I started working with my box. And 2016, December, just before Christmas, I think someone was like, oh, the OHS website looks down. And I was like, oh, let me just go to the server
07:42
and see what's up, probably just needs to be rebooted. And was like, oh, one of the hard drives has failed. Well, okay, it's a rage setup, we should be able to recover with one of the hard drives. I was like, well, both the hard drives have failed. That was not nice. I spent about three days trying to run DD Rescue.
08:01
If anyone's ever tried to run DD Rescue, it's not fun. Don't ever be in a situation where you need to run DD Rescue. I tried to recover the hard drives, and it was a bad time. We lost some data. He appeared. Rob Warren had a backup of PDF files of the data.
08:25
He took it, set it up on a better hosting environment in his university. And OHS was kind of restored, and the community continued. And people continued to be dedicated editors
08:43
over the next two or three years. There was a whole team of people who set up a lot of the ancillary services, who made changes to AD to better accommodate time-based editing, who set up Tasking Manager for the historical map, which are widely set up overpass.
09:01
I remember being at SOTM in New York on the steps, kind of trying to figure it out. And Richard maintained overpass for OHM for the next many years. In parallel, there were a few things happening. At this point, we still didn't have a time slider.
09:23
This was the dream, that we were putting in the data with start and end dates, and we wanted a time slider to be able to move and see that data change. That was the point of a historical map. Jeff Meyer, who had been someone else who was involved with the project right from the beginning,
09:43
kind of just passionate about historical mapping, and started working with the team, Green Info. We have Dan here. Seth Fitzsimmons, a team of people here who were really working on generating vector tiles. It was now 2018, we had vector tiles. Vector tiles were standard across formats.
10:02
And it was, okay, if we generate vector tiles, we can actually do this filtering on the front end, right? Which was a big change, which was something that had been a blocker for a long time to try and get it into the raster tile rendering engine.
10:20
And then I somehow got involved with the project again. I was working with Seth Fitzsimmons on something else, and he's like, oh, I'm going for an OHM call. And I was like, oh, I used to have root on that box once. Let me join this call. And I could see that it was struggling to figure out the optimal architecture to kind of get this running
10:41
in a way that was sustainable, in a way that would allow us to make updates, in a way that was appropriate for 2019. So at that point, working with all these folks, sort of 2019 OHM moves to the cloud,
11:02
where we kind of deployed the new infrastructure, along with the time slider, you know, the demo that I was hoping to show earlier. And yeah, so I'll describe some of what that architecture is. I can see it's late at the end of the conference, and I think lots of eyes are gonna start
11:22
glazing over at the next slide, but I'll try and go quickly. Do you know how much time I have left? So I'll quickly run through the technical architecture, which, I'm one of those people as well, who enjoy just having a single box
11:41
and setting things up on there, and I tend to be quite suspicious of these fancy cloud-based architectures. But this is something that we were working on for some time. My colleague, Sachar Anwar, had, in 2015, given a talk at the State of the Map New York about, you know, what if we could
12:03
apt-get install OpenStreetMap, right? Like, how do we make the OpenStreetMap software more easily installable and usable in different contexts? And this is something that we had been toying around with and playing with the idea for many years,
12:20
because I think we really believe back then, as we do now, that there's lots of use cases for the OpenStreetMap software that's really useful beyond OpenStreetMap. And so, something that we had built and that we used extensively at DevelopmentSeed was something called OSMSeed,
12:42
which is essentially an easy way, easy way to deploy the OSM software onto cloud providers, right? So it deploys onto Kubernetes, it kind of creates containers for all the OSM software tools
13:00
and gives you an easy way to deploy it onto a Kubernetes cluster. I know for most people, though it's easy and Kubernetes don't really go together, I have to say that what I have enjoyed about the process is that we could do this in a way that does not depend on any proprietary cloud infrastructure.
13:20
So, you know, the code and all the dependencies are all in source and, you know, these deploy those on bare metal or other cloud providers. So, all the code for the infrastructure is open, I would very much sort of encourage people
13:42
or people who think they can help because boy, are we gonna need a lot of help to check it out and plug in. So the services that we run right now for Open Historical Map, it's a fork of the OpenStreetMap website, which has the time slider,
14:01
which is customized for time, to handle time. We use Togola to generate vector tiles. There's front-end changes, a lot of the work done by the GreenInfo team to use vector tiles, right? The OSM web site uses raster tiles,
14:21
so we realized that's not gonna work for us. Combination of being too hard and practical, plus vector tiles just seemed, you know, cooler. You could do this filtering on the front-end, it looks really nice. So there's changes to, you know, there's a fork of the front-end, which use vector tiles, an improved inspector,
14:42
a time slider. We run our own instance of Tasking Manager for OHM, and in the next few weeks, I think we have some more kind of tweaks to Tasking Manager to better support historical mapping. We run Nominatum. Right now, it's just stock Nominatum with OHM.
15:02
This is something that I love. If anyone knows anything about the internals of Nominatum, I think there's lots of potential tweaks to make it more time-aware that we haven't done yet. And we run Overpass, and we sort of now deploy all of these as a single unit using a single code base,
15:21
not worrying about kind of like four different people having to manage four different servers, you know, stay, you know, kind of have everything stay in sync. We can kind of deploy it as a single code base. And of course, we, you know, publish minutely replication,
15:41
climate dumps, all of that for Open Historical Map. We've got some really hard problems ahead. I feel like we spent 13 years and what we have is a demo, right? We've got an MVP, we have cool demos. The really hard problems of mapping history, to be completely honest, are unsolved, right?
16:01
But we have, a lot of it is a little bit of a hack. We add start date and date to OSM features. We have a cool time slider to display them. A lot of meeting a lot of historians get really angry. A lot of people who care about data formats get really angry because like what, you know, a lot of this just doesn't seem like it's the correct way to do things.
16:22
So we've got, we've got hard problems ahead. One problem that we're calling the boiling the ocean problem, coastlines, coastlines are very hard. Attracting changes to coastlines is even harder. I don't want to get into the details of this problem, but if you search in our issues for coastlines,
16:43
you're going to see some wild discussions. At some point there was a discussion about importing coastline data from the end of the last ice age so that we had a baseline and then people could edit over that, yeah. But, yeah, boiling the ocean just seems like a fun thing to come up with.
17:03
I think something that Richard has talked about in his talks, we have to get how we do the tagging schema and feature of lifecycle correct. Right now, if there's a road that changes name 10 times over the course of whatever, 100 years,
17:21
which in India that happens a lot, we just ask users to draw the same road over and over again with different attributes and different start date and dates. This is not sustainable. We need a better way to do this. I think what Richard has proposed is great, but we need to build a support into the rendering engine, into the front end, all of that. Date formats, fuzzy dates,
17:41
this is all complicated. Historians will tell you, you cannot give an exact date to the start and end of things. That's ridiculous. You need to have a notion of approximate dates or fuzzy dates, which right now we don't really because how would we deal with that? So right now, we need exact dates, but we need to start supporting fuzzy dates. Running a fork of OSM is all fun and games
18:03
in the beginning until you need to catch up with upstream. Three years later, then it gets really hard. We need to do that. We need to constantly catch up with upstream and that's really hard. We need a better process to do that. Any advice is greatly appreciated. Making things like Norman Applin and Overpass time aware is something I don't even know where to start,
18:21
but hopefully people who know these things better than I can jump in. Yeah, let's do this together. There's high problems ahead. We really hope more people jump in, helping us improve this. Yeah, open historical map on the OSM US Slack, submitting this, there's Discord. You can find all the details on the about page.
18:43
Yeah, these were again supposed to be some cool animations, but my time is up again and the animations don't work, so I encourage you to please check out openhistoricalmap.org and see these things for yourself. I think we have a minute for questions, maybe? Yeah, it is working. Oh, wow.
19:03
Okay, yeah, so you can, yeah. I'll just let that play while there's any questions. Thank you. Thank you.