Vector tiles for fast custom maps
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 188 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/31638 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produzent | ||
| Produktionsjahr | 2014 | |
| Produktionsort | Portland, Oregon, United States of America |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
FOSS4G 2014 Portland131 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
SchnittmengeVektorraumCAN-BusTesselationTermInformationMultiplikationsoperatorProgrammbibliothekZoomOffice-PaketBildgebendes VerfahrenStabMAPNummernsystemMapping <Computergraphik>ParkettierungPunktElektronische PublikationImplementierungDichte <Stochastik>Textur-MappingKomplex <Algebra>SelbstrepräsentationQuaderEndliche ModelltheorieDateiformatBrowserDemoszene <Programmierung>Vorlesung/Konferenz
01:51
Zusammenhängender GraphOpen SourceTesselationParkettierungVektorraumBildgebendes VerfahrenClientAppletFormale SpracheDifferenteElementargeometrieRenderingZentrische StreckungDateiformatKlasse <Mathematik>SkriptspracheMereologieServerWeb SiteThumbnailAttributierte GrammatikTypentheorieTextur-MappingQuick-SortProdukt <Mathematik>AuswahlaxiomSchlüsselverwaltungBitmap-GraphikRohdatenFlächeninhaltGebäude <Mathematik>Vorlesung/Konferenz
04:12
Quick-SortProgrammbibliothekElektronische PublikationOntologie <Wissensverarbeitung>Varietät <Mathematik>Kartesische KoordinatenSoftwaretestAggregatzustandEnthalpieRechter WinkelMAPRückkopplungTextur-MappingQuaderImplementierungQuadratzahlZahlenbereichMereologieKonditionszahlDatenstrukturURLOpen SourceAdressraumComputerunterstützte ÜbersetzungResultanteSpezifisches VolumenAuflösung <Mathematik>TesselationVektorraumSelbst organisierendes SystemDifferenteDateiformatMapping <Computergraphik>Lokales MinimumPunktProtokoll <Datenverarbeitungssystem>SymboltabelleEinsDemoszene <Programmierung>ZoomPuffer <Netzplantechnik>SchnittmengeNegative ZahlParkettierungDienst <Informatik>Weg <Topologie>Endliche ModelltheorieEinfache GenauigkeitSichtenkonzeptDatensichtgerätCodierungMinkowski-MetrikCodierung <Programmierung>Interface <Schaltung>FokalpunktOvalMultiplikationsoperatorSchwebungMaßerweiterungVorzeichen <Mathematik>BitElementargeometriePolygonGrenzschichtablösungCASE <Informatik>InformationsspeicherungGeradeWeb logZeichenketteParallele SchnittstelleSoftwareentwicklerGanze ZahlTopologiePartikelsystemGruppenoperationSchlüsselverwaltungFormale GrammatikFormale SpracheArray <Informatik>Grundsätze ordnungsmäßiger DatenverarbeitungWikiWeb SiteKoordinatenMultiplikationProdukt <Mathematik>RenderingKrümmungsmaßAttributierte GrammatikEindringerkennungUmwandlungsenthalpieFrequenz
12:30
ZoomAttributierte GrammatikSoundverarbeitungBildgebendes VerfahrenParkettierungElementargeometrieTesselationPuffer <Netzplantechnik>SchnittmengeDatenbankVektorraumMetadatenLokales MinimumDifferenteGlobale OptimierungBildverstehenAuflösung <Mathematik>Interface <Schaltung>Codierung <Programmierung>Ganze ZahlCASE <Informatik>Elektronische PublikationProjektive EbenePunktPolygonTermBitMAPWasserdampftafelDialektZellularer AutomatCOMp-BlockShape <Informatik>MultiplikationsoperatorEndliche ModelltheorieComputeranimation
14:38
Spezielle unitäre GruppeTextur-MappingInterface <Schaltung>ParkettierungTesselationOpen SourceBimodulVektorraumSkriptspracheElektronische PublikationQuaderBitKartesische KoordinatenMetadatenGeradeRohdatenExpertensystemEndliche ModelltheorieFreewareMultiplikationsoperatorProgramm/QuellcodeVorlesung/Konferenz
16:17
VektorraumGraphfärbungSchnittmengeDefaultClientVarietät <Mathematik>Attributierte GrammatikVektorpotenzialTesselationRenderingCOMElementargeometrie
16:54
Reelle ZahlOffene MengeMereologieVarietät <Mathematik>StichprobenumfangOpen Source
17:16
TesselationProjektive EbeneVektorraumTopostheorieHill-DifferentialgleichungDemo <Programm>ParkettierungSatellitensystemBitmap-GraphikTopologieGeradeGleichgewichtspunkt <Spieltheorie>Textur-MappingInteraktives FernsehenPhysikalische Theorie
18:16
SoundverarbeitungMultiplikationsoperatorVektorraumHill-DifferentialgleichungSatellitensystemNichtlinearer OperatorOverlay-NetzZoomParkettierungTouchscreenProtokoll <Datenverarbeitungssystem>Dienst <Informatik>Filter <Stochastik>Coxeter-GruppeRohdatenTransformation <Mathematik>Message-PassingPuffer <Netzplantechnik>Bildgebendes VerfahrenMatchingArithmetischer AusdruckQuaderDateiformatTesselationMapping <Computergraphik>ClientTextur-MappingServerCASE <Informatik>
20:01
TermTesselationSoundverarbeitungRenderingVektorraumGenerator <Informatik>BildverstehenMetropolitan area networkRechenschieberQuaderVerschiebungsoperatorVorlesung/Konferenz
20:46
VektorraumOpen SourceTextur-MappingTesselationParkettierungGeradeAbfrageSystemaufrufKartesische KoordinatenRenderingAttributierte GrammatikVorlesung/Konferenz
21:30
MAPTopologieResultanteAuflösung <Mathematik>Spannweite <Stochastik>ZoomAbfrageFontFigurierte ZahlFlächeninhaltDemo <Programm>TouchscreenProfil <Aerodynamik>SchnittmengeVektorraumGeradeBitDatensichtgerätTafelbild
22:34
VektorraumReelle ZahlTesselationElementargeometriePolygonRichtungSoftwareentwicklerKartesische KoordinatenBitCASE <Informatik>ValiditätGrenzschichtablösungGebäude <Mathematik>SicherungskopieOffene MengeDateiformatAmenable GruppeVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:00
Thanks, everyone, for coming. So I am Dane Springmeyer, and I work for Mapbox. And I'm going to be talking today about vector tiles for fast custom maps. And so Mapbox is based in Washington, DC. And now we have San Francisco offices. We're about just over 50 staff at this point.
00:23
So where are we headed today? Well, basically, I want to dive deep into what vector tiles are. My hope is that in a few years, vector tiles will be so adopted that we won't really need to talk about the details anymore. They'll just work. But since there's still a new idea or concept to a lot of folks, I want to dive deep into how they work.
00:42
And before I start, I'm going to be talking about what I'm familiar with in terms of the tools and the goals of vector tiles for me. But if you've ever worked with vector data in the browser, you've really worked with vector tiles. I mean, dating back to the early WFS implementations, it's really vector tiles. So there's a lot of precedent here.
01:01
And people have been doing this for a long time. But what I'm going to talk about is a Mapbox effort to work on a spec and a set of libraries and tools that are very high performance. OK, so what is a vector tile? Well, simply speaking, it is like an image tile, which we're all familiar with. It's easy to cache and serve rapidly. So it's divided data.
01:22
It's the same addressing scheme as image tiles as well. A zxy.png means the same thing as a zxy.vector, PDF, which is the format that we're working on. So you can address it at 000 or at higher zoom level. It's the same way to request a tile.
01:43
And vector tiles can represent many complex data layers, just like an image tile can. You can bake a lot of information into it. And where we're going with this is we want a full stack to work with vector tiles that's completely open source. So there's no proprietary components. There's no piece that you can't switch out for another piece
02:00
if it works better for you. So they should be just as common and easy and open source to work with as an image tile. But of course, vector tiles should be better. They can contain all sorts of raw vector data that you can do interesting things with. Geometries, road names, area types, building heights. They should contain highly packed data
02:22
that's efficient for rendering. But they should contain attributes that you can do interesting things with. And yeah, they're very compact. So the format that we're working on easily fits on a USB thumb drive. So it's 20 to 30 gigs. And that's all of the OpenStreetMap planet baked into vector tiles. They should be very fast to parse, both client-side
02:43
in a language like JavaScript or server-side if you're doing work with them. And so our client-side language of choice is JavaScript, of course. And we work in C++ mostly on the server-side. So they should be fast to parse. And a key to being fast to parse is you should be able to parse them lazily or incrementally. You shouldn't have to read the whole thing just
03:02
to get, say, one layer out or one feature or one geometry. You should be able to seek just right into where you need and get what you want. You should only pay for what you get, which is a slight difference from textual formats like GeoJSON or TopoJSON, which work fabulously as vector tile formats. But we've chosen a binary protobuf-based format,
03:23
largely so that we can do lazy or incremental parsing, because that scales better in our experience. So overall, vector tiles, I feel like, offer a very bright future for radically customized and efficient rendering of data as large as OpenStreetMap or larger, including raster data.
03:44
So then, how do vector tiles actually work? Well, you can put any kind of vector or raster data in. These are the formats that the Mapbox tools are prioritizing support for. But it gets really interesting, right? You can put any types of data in. You don't have a fixed schema.
04:02
So I like to think of vector tiles as ravioli, right? Because you can put anything inside, but they're really Chef's Choice ravioli. The cartographer that designs them gets to choose what ships with the product. So you can choose whatever flavor you want, right? So I imagine a near future where many people are publishing vector tiles, and they contain their own specific ingredients, right?
04:23
So if you want to publish Blueberry vector tiles, you can. If you want to publish beets, you can. If you want to throw eggs in your vector tiles, that's up to you, right? All that's common is the structure of the data, right? So the way it works is we have layers, one or many. Each layer has a feature. Features can have IDs.
04:42
Each feature has a set of attributes, key value pairs, and geometries. Pretty simple. Layers are ordered and named is how you reference them. So you might have a vector tile with parks, roads, and places, and you decide on the names. Features are dictionary coded. So what that basically means is we
05:02
don't store a key more than once in the whole tile. So if you have an attribute named name, we turn it into integer and then store that once in the tile for efficiency. And then geometries are single flat array. We found that if we compress, say, multi-geometries or multi-points into a single array, it stores much more efficiently.
05:22
So that makes it a little tricky then to unpack back into, say, nested arrays of GeoJSON for multi-geometries, for example. But we feel like it's worth it. That flat array stores very, very well. OK, so yeah, how do geometries really work? We support all geometry types, points, lines, and polygons, multi or single.
05:40
We store coordinates as integers because that's more efficient. And then those integers are delta encoded and zigzag encoded. So what does that mean? Well, delta encoding basically is storing differences rather than the actual value. So a simplified case would be if we have a line string of coordinates that looks like this, we would
06:01
store the original coordinate for the first xy. And then from then on out, we would store the differences from that first coordinate. So this is a dream case, right, where all of the rest of your coordinates would be ones, which obviously compresses really, really well if you have repeated values. But in the real world, it still compresses better delta encoding
06:21
because the numbers are smaller, takes up less space. And then zigzag encoding on top of that also helps save space. So the idea behind zigzag coding is we do want to have negative values in our vector tiles because you might have data that's buffered outside of the visible extent of vector tile. This is important often for labeling.
06:42
Or at render time, if you want to blur geometries, you need to have a little bit of a buffer. So we need negative coordinates in vector tiles. But we don't want to pay the price for that. And the trick is zigzag encoding, which is basically a method of turning signed integers all into unsigned integers.
07:00
So you only store a positive number. And then when you decode it, it's easy to go back and get the original negative number. So what can you do with vector tiles? You can overzoom them. This is the whole idea, right? You should be able to ship a tile set and not burn it all the way down to, say, zoom 17 or zoom 20. But you should be able to ship a tile set to maybe only zoom 14 and still allow a visible display of those tiles
07:24
to much deeper zoom levels. So this works around the problem of trying to pre-cache tiles for the whole world. You should be able to render zoom 14 for a fairly big data set very quickly and later on view it at deeper zoom levels. You should be able to also composite them because once you've burned vector tiles for some layers,
07:43
you'll be able to combine it with other ones. So that's easy to do with this format. Now, you might be wondering, well, how do you combine vector tiles at different zoom levels? Well, it's very easy, obviously, at the same zoom level. With protocol buffers, the format we're using, you can just concatenate them. So it's really quick. But at different zoom levels, you're going to need to clip some out and re-render them
08:03
before doing so. So you should be able to do something like this. Say you have three different vector tile sets from three different organizations or three different cartographers, parks, roads, and points of interest. They make sense to pre-bake it to a different max zoom level. Points of interest you might want to have highly resolved,
08:22
whereas parks that are just square polygons don't need to be as resolved. So the idea is you should have different tile sets that exist at different max zooms, but you should still be able to render them all together and benefit from the highest resolution of any of those tile sets. So there's also this concept of sources
08:42
that we're using to describe when you have a bunch of vector tiles together that represent one layer. So that could be a bunch of places in an MBTiles format, or it could be a URL to vector tiles that you can address by zxy, or a tile JSON file that describes somewhere else to look online.
09:03
So for more technical details, this is where you can head, github forward slash matbox, forward slash vector tile spec. This is where the emerging specification is, and it's definitely still at the emerging stage. We're looking for feedback and use cases. What we've done is come up with a solution and a format
09:23
that works for us, publicizing it, and then hoping for feedback and to learn how many other use cases it works for. We're not trying to create a spec for all use cases. We're collecting on the wiki at that site various implementations in different languages. So there's quite a number of invitations in JavaScript,
09:41
some pure JavaScript, some Node.js, some in C++, some emerging implementations in Python, and I'm sure more coming in the future. So yeah, check out the spec. We're still at 1.0. We haven't incremented past and recently, because there's been lots of work
10:02
to optimize the tools around vector tiles. The format and the structure hasn't been a bottleneck yet. We're still working on the tools around them. So you might be asking, OK, these sound interesting. How can I create them? OK, so there's really two key workflows. There's a variety of command line tools that are available.
10:22
We maintain one of them called Tile Live, which is a Node.js toolkit that can be used on the command line or you can use to build larger applications. There's Tessera, which is a module written by Seth Fitzsimmons from Stamen Design, which he is talking about right now in another room,
10:42
unfortunately. We actually thought maybe we should try to get everyone in the same room. We could just have a coup, but when I saw the size, I knew that wasn't feasible. OK, there's also a library called Avocado from MapQuest, which was released a few weeks ago, which is C++-based. And it's for creation of vector tiles, I think only.
11:03
There's obviously a variety of tools now to also render them and work with them, but I'm focusing on the creation stage. And then there's a product called Mapbox Studio, which is what I've been working on for almost two years now. It's been in development and actually was launched officially today on the Mapbox blog.
11:21
So I encourage you to check out Mapbox.com's blog. It is formerly known as Tilemill 2. So the reason we renamed it is it's just purely based off of vector tiles, which introduces new concepts and new performance abilities. So we felt like it would be reasonable to think of the tool as something separate from Tilemill.
11:41
We do plan to do one final release of Tilemill in the future for those of you that are depending on it and need some time until you can upgrade to Mapbox Studio. So the two will be supported in parallel for a period in time. But yeah, Mapbox Studio has been in development for a while, officially released today. So it has a visual UI to creating vector tiles.
12:04
It also has a styling interface to make maps from vector tiles, which you'll probably hear about. And the National Park Service gave a talk about earlier that some of you may have seen. But what I want to focus on is the slightly more hidden part of Mapbox Studio, which is how you create them.
12:21
And it allows you to create them locally with local data and create and export locally without any Mapbox.com services. So what does that look like? Well, it's a pretty simple interface because we've strove to make it without as many options as possible. So this would be an example of the interface.
12:40
The idea is you can load in multiple layers. And this would be a water polygon from OpenStreetMap. You can set the projection and the minimum zoom and the maximum zoom that you're going to render it out to. And then the buffer size. Like I mentioned before, for cases like labeling and blurring effects, buffers are very critical.
13:01
You can rename some of the metadata in terms of the attributes that are included in this, in vector tiles. Because it's imagined that you would really curate the data and then prepare it for a different audience of cartographers that are going to render it. The vision here is those that know the most about a data set and its metadata would be the producers of the vector tiles.
13:21
They would really prepare it for others. Then you can export it from Studio. And just as an example, this data set is 530 megabytes as it starts as a shapefile. I did a z to 0 to 10 export, which for the resolution of the data is more than enough to zoom levels. So at zoom 10, you can continue zooming to 12, 13, 14.
13:43
And the only point where it gets a little bit blocky is where the original data gets blocky. If I only did an export at zoom 0 to 2, then you would not be able to zoom that deep. You would start to see the quant effects of the vector tile encoding the optimizations.
14:01
But zoom 10 for this data set was more than enough to capture the original resolution of the data. It finished in eight minutes on my MacBook. And it's about one million tiles. And the resulting vector tile set inside of an SQLite database is much smaller than the original file. So that's really because of the integer encoding, delta encoding, and zigzag encoding.
14:21
Because in this case, I really feel like we're capturing the original resolution of the geometry. So what can you do with that? Well, you can obviously upload it to matbox.com, just like you could in MBTiles of image tiles, like you can in the past. And then you can see it online. And then you can bring it back into Matbox Studio
14:41
to then style. And this would be an example of the styling interface. It's very, very familiar for those of you that have used Tilemill in the past. It should be familiar. It should be exactly the same, except what's happening under the hood is your data is tiles. So it's extremely fast to access. But you can also do a headless export.
15:02
I mentioned Tile Live is the tool that we maintain that Mapbox Studio is built upon. But you can download it. It's free, 100% open source. And you can use it headless without installing Mapbox Studio GUI on the command line. And it's a Node.js application. And its dependencies are the Node.js module for Mapnik.
15:21
So if you were to run npm install, Mapnik, MBTiles, Tile Live, and Tile Live Bridge, you would have all of the dependencies in a matter of minutes to run this script. And then the way that the script works is you run the Tile Live copy command. You pass it on Mapnik XML, which is that data.xml, a bounding box. And then it dumps the tiles to an MBTiles file.
15:42
The one trick or gotcha that you'll see if you try this is that data XML is created by Mapbox Studio. And it has a little bit of metadata in it so that we know basically what attributes to put in the vector tiles and a few other things. So it's not a raw Mapnik style sheet.
16:01
It's a little bit of extra metadata that Mapbox Studio has added. But it's not too hard to author that yourself. So you could get set up for this export outside of Mapbox Studio. And yeah, then you've got an export in MBTiles. So the whole idea here is that this workflow
16:20
should scale to any large data sets that are complex and have a lot of potential layers you could represent with the potential attributes. So this is an example of the X-ray style applied to the vector tiles of OpenStreetMap. And the X-ray style is basically a default style that you get when you upload an MBTiles to Mapbox.com
16:41
and also a default style that is embedded in a variety of the client tools that can read MBTiles. Basically just renders in bright colors the various geometries and labels. So you can kind of get a sense of what's in there. So that was New York and this is Washington DC. And then the other thing that is also on GitHub
17:01
and open source, and there's a variety that were added this week as part of the launch is sample styles. So there's an OSM bright.tilemetal2 that you can use to get started styling things very quickly. And then I just wanna show you Mapbox Studio real quick and I'll show you some of the starter styles. So if I were to go to projects here, styles,
17:23
new style right now in the latest release as of last night, I have four starter styles. Oh, there we go. Thank you, Andrew. So I have four starter styles. I realize it's a little cut off. A basic OSM bright is actually built in Mapbox Outdoors
17:42
and then satellite, what is that called? Satellite afternoon. Yeah, so let's see if I can figure out how to resize this. There we go. I wanted to show you satellite afternoon. It is one of the most complex because it combines three vector tile sets,
18:01
a vector tile set of raster imagery, satellite imagery, a vector tile set of hill shades and topo lines, and then also the OSM data curated in a vector tiles that we call Mapbox Streets. So I'll just demo that real quickly. I think we should zoom into, see if I can go full screen here.
18:22
Zoom into Mount Hood to give you a sense of how cool this is. And if I get back to full screen. So the whole idea here is not pre-packaged maps but that you can change absolutely everything
18:40
about this style, including the compositing operations about how the different layers interact. And just on a technical note, you might be wondering, okay, so he just said that there's all these vector tile sets that are actually being combined. So what's happening is mapbox.com is fielding your request for what vector tile sets you want. Mapnik on the service side is concatenating
19:01
the protocol buffers and then shipping back to the client, in this case, my desktop app, one message in a protobuf format. And we can iterate through and see what layers are in there and then match up to the styles here. So basically what that means is if I misspell
19:20
one of these styles, it gracefully degrades. Basically there's just not a match between the style name and the layer name. So let's see, one of the big tricks here in the style is this image filters. We're doing HSL transformations so I can comment this out and get a look at what the raw satellite imagery looks like.
19:42
Or I could go in and see that the hill shade is being applied with an overlay compositing operation. Comment that out and see what effect it's having. So the idea is these are starter styles for anything you can imagine doing. Okay, so I'm just about up with time but I'm gonna go back to my presentation
20:02
and show you a few more slides. So the vision here is that vector tiles are gonna make possible a new generation of renders that don't have to worry as much about how big the data is that's being shipped to them and they can go crazy with styling effects. And that's kind of the whole idea is to unleash a new generation of carelessness
20:22
in terms of data size and fierceness about how beautiful and interesting and interactive maps can be made. So we're working on, sorry that that's cut off. I think I screwed up there. Well, it's only a few slides. So this is a Mapbox GL that is cut off.
20:48
I may not be able to fix this in time. Anyway, so we're working on an application called Mapbox GL which is 100% written in JavaScript and WebGL which can render vector tiles. And if you haven't heard about it, go ahead and go to Mapbox GL on mapbox.com
21:03
and see what it's about. And also vector tiles, you should think of them as about more than just rendering. This is a toolkit called VectorTile Query, also in GitHub that basically allows you to in a very quick API call say, here's a source of vector tiles
21:21
and here's a line that I'd like to query them along and here's the attributes I'd like to pull out. So what you can do with that is things like this. If you have a vector tile set of elevation and a vector tile set of temperature, you could do a line profile and get back the results.
21:40
So I'll just demo that real quick before I break. So the cool thing about this is that, I should probably put that on the screen for you guys, because you can query now at any zoom level or any and we automatically be a back data
22:02
that's at the relevant resolution. So you can do a query across the whole US or of course you could zoom in to the Portland area and say, what is the temperature and elevation range from Portland to Mount Hood? Whoops, yeah, the display obviously is a little bit
22:26
messed up with the size of the font but you get the idea. So I definitely encourage you to check that out. This is very new and experimental but has a lot of promise. Okay, real quick future work.
22:42
There's a severe lack of documentation for tinkers. I'm gonna be working on that in the future. So if you're a developer that's a little bit confused, that's an opportunity for me to answer your questions and encourage you in the right directions to tinker, hack and pull apart how this actually works so you can build backup applications on top of it. So that's a future that I wanna work on
23:00
as well as more new use cases beyond rendering and querying. We have some work to do around geometry robustness. The polygons are not, we haven't worried about them being OGC simple spec. So we have some work to do there, tools to clean up and validate. And then of course there's big tool sets out there that would be interesting to explore maybe support.
23:22
There's a rumor I heard today that maybe the post JS developers are considering support for vector tiles. Maybe in this format, maybe in another, I think either would be fantastic to entertain and maybe toolkits like OpenLayers. I know Leaflet is considering support as well. So thank you very much.