MapCache: Overview of MapServer's tile caching server
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 188 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/31711 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produzent | ||
| Produktionsjahr | 2014 | |
| Produktionsort | Portland, Oregon, United States of America |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
FOSS4G 2014 Portland58 / 188
7
10
14
15
16
23
24
25
28
29
33
37
39
40
43
45
46
48
50
56
64
65
69
72
74
82
89
91
98
102
107
111
114
118
128
131
132
135
138
141
143
147
149
150
157
158
161
164
165
166
173
174
175
179
185
00:00
Gewicht <Ausgleichsrechnung>Bildgebendes VerfahrenREST <Informatik>ParkettierungDigitale PhotographieCachingMultiplikationsoperatorQuellcodeKeller <Informatik>Regulärer Ausdruck <Textverarbeitung>Familie <Mathematik>ServerBrowserDateiformatProxy ServerZusammenhängender GraphMessage-PassingE-MailSummengleichungZeitrichtungVerkehrsinformationMAPDatenkompressionInstantiierungNeuroinformatikMethodenbankCASE <Informatik>GraphfärbungSchlüsselverwaltungDebuggingATMGeometrische QuantisierungBitExogene VariableClientAnalysisMaschinenspracheGoogolMatchingMapping <Computergraphik>t-TestProgrammbibliothekStandardabweichungProtokoll <Datenverarbeitungssystem>BenutzerbeteiligungDifferenteMeterAuflösung <Mathematik>Wrapper <Programmierung>KundendatenbankSpezialrechnerEinsSchnittmengeIntegralProjektive EbeneMaßerweiterungPixelCodierungDialektMeta-TagSkriptspracheDemo <Programm>SoftwareschnittstelleSoftwareentwicklerVersionsverwaltungCodeHeegaard-ZerlegungFormale SpracheSchlussregelTesselationCanadian Mathematical SocietyViereckVirtualisierungInterpolationElektronischer ProgrammführerLesen <Datenverarbeitung>SichtenkonzeptBitmap-GraphikDelisches ProblemVererbungshierarchieZoomStichprobenumfangHydrostatikFront-End <Software>Suite <Programmpaket>Computeranimation
08:25
ResultanteIdeal <Mathematik>DatenkompressionInverser LimesBildgebendes VerfahrenEinfügungsdämpfungSoftwareentwicklerSoftwaretestBootenVerzeichnisdienstLastCachingMinkowski-MetrikMultiplikationsoperatorQuellcodeTesselationFarbenraumParkettierungCASE <Informatik>InstantiierungDateiformatKonfiguration <Informatik>Wort <Informatik>MAPBitGruppenoperationGraphfärbungZoomARM <Computerarchitektur>Nichtlinearer OperatorDatenbankMereologieClientStrategisches SpielStandardabweichungOffene MengeSchnittmengeTypentheorieMaßerweiterungVariableAuflösung <Mathematik>Message-PassingAngewandte PhysikTermServerp-BlockDefaultDateiverwaltungÜberlagerung <Mathematik>SatellitensystemTemplateRuhmasseLeistung <Physik>Mini-DiscStatistikOverlay-NetzZahlenbereichBandmatrixDynamisches SystemFront-End <Software>Bitmap-GraphikHalbleiterspeicherEinfache GenauigkeitDifferenteHackerUmwandlungsenthalpieStützpunkt <Mathematik>CodierungAliasingComputeranimation
16:51
Arithmetischer AusdruckSicherungskopieFront-End <Software>DateiverwaltungEinfache GenauigkeitBildgebendes VerfahrenCachingServerVektorraumPolygonOffene MengeElementargeometrieMaximum-Entropie-MethodeMAPPunktATMParkettierungInstantiierungZeitstempelDatenbankBitmap-GraphikDimensionsanalyseClientSatellitensystemTesselationTeilmengeMultiplikationsoperatorFlächeninhaltInformationsspeicherungCoxeter-GruppeMaßerweiterungKundendatenbankRelationentheorieRegulärer Ausdruck <Textverarbeitung>ZoomNichtlinearer OperatorProzess <Informatik>Dichte <Physik>DateiformatMinkowski-MetrikProgrammbibliothekVersionsverwaltungKommandospracheAutorisierungBefehlsprozessorProtokoll <Datenverarbeitungssystem>Service providerFlächentheorieEin-AusgabeSteifes AnfangswertproblemCASE <Informatik>REST <Informatik>Hook <Programmierung>ZahlenbereichSchnittmengeDemoszene <Programmierung>URLInverser LimesBandmatrixMosaicing <Bildverarbeitung>TeilbarkeitMereologiep-BlockFokalpunktTaskImpulsGewicht <Ausgleichsrechnung>Umsetzung <Informatik>Lesen <Datenverarbeitung>HoaxSoftwareschnittstelleQuellcodeDreiMapping <Computergraphik>Computeranimation
25:16
Hierarchische StrukturMAPTesselationDatenkompressionQuick-SortComputeranimationVorlesung/Konferenz
25:53
CachingFehlermeldungTesselationProxy ServerRPCCodeKonfigurationsraumAbstandTypentheorieUmsetzung <Informatik>MultiplikationsoperatorBitrateVorlesung/Konferenz
26:27
ParkettierungKonfiguration <Informatik>ServerQuaderCASE <Informatik>Mixed RealitySichtenkonzeptElementargeometrieFehlermeldungTouchscreenQuick-SortVorlesung/Konferenz
27:04
MultipliziererKonfiguration <Informatik>Front-End <Software>CASE <Informatik>Vorlesung/Konferenz
27:33
ParkettierungThreadCASE <Informatik>Konfiguration <Informatik>Front-End <Software>Endliche ModelltheorieVorlesung/Konferenz
28:05
Vorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:01
Tile saving component that's been added to maps of Two years ago officially, I think as Steve said three maybe now Just a short introduction about what what's tiling server so tiles are actually pre-computed usually 256 by 256 pixel images that allow first access to
00:27
Statically static data that's been that's been cached somewhere for rapid access A tile is aligned to one grid Which is basically a subdivision of the world along a given for a given projection
00:42
For a given extent and across different resolutions Tiles contain data that has been rendered by some third-party service In usual case another WMS server or map server itself or Google data source or whatever
01:04
Once the tile has been generated by this server it's split up by the time server and stored in a cache backend So we'll be looking at all the available cache backends. We have available and then that tile can be self backed to clients over the web
01:21
respecting different Services or standards. Map cache itself is actually a tiling library more than a tiling server and it has Front ends that's tie it either to the Apache web server
01:40
as a fast CGI instance as a native nginx module or as a native node module It's versatile in the sense that it supports multiple cache backends It supports multiple client protocols and it has advanced tile management features which allow you to pre-generate
02:04
seeds beforehand so that's called seeding recompress images to gain image weight interpolate image data that's not present from lower levels it's written in native code C, which is fast and lean and
02:21
There's a tiny demo interface that allows you to quickly set up a service and then just copy paste the JavaScript code That's needed to get that use that server directly in your development and client code The code is now four years old
02:40
started in 2010 and It was officially integrated as a map server components and the map server suit in 2012 so first talk about the protocols that are implemented and how you can request tiles from a cache, so it's
03:01
uses either standard XYZ addressing KML super overviews to view tiles in Google Earth typically and it has also a WMS wrapper to create To read WMS from tiles. I'll go into more detail for that So for standard tile addressing it supports the CMS protocol, which is the
03:20
OSGO protocol for addressing tiles It supports WM TS in either restful restful or key value pair modes Which is basically very similar to CMS, but vetted by the ODC supports addressing by virtual earth quad keys map guide addressing and standard XYZ addressing
03:43
It also supports WMS get map requests, which means that it will compute Any get map requests for given size or layers and Use the tiles that it has in cash instead of using the native data itself
04:00
So in most cases, this is usually much faster than hitting a WMS server itself Because you're using pre computed image tiles as the source So it responds to untiled Requests by assembling those tiles from the caches It can either assemble them vertically in the sense that if you're asking for multiple layers
04:22
Then it will stack multiple tiles one in the other and fusion them into a single tile and also horizontally Which means that if you ask for 1,000 by 1,000 images will stack four tiles once one's next to each other to create a that image It also acts as a kind of put ODC in balance because that's the main usage
04:45
Proxy for other services. So in that case, it can be the front end to your services It will intercept the requests that are tiled or that can be served from tile caches And then you have a set of rules that can allow you to forward other requests to other servers
05:01
So you have a kind of expressing language with regular expressions. This means you can send WFS request to a WFS server You Can send WFS requests of version 1.1 to one server and then 2.0 to another server You can also serve static data from
05:22
from another server Miscellaneous features which I've just put together here It creates the caching HTTP headers So your clients don't keep requesting over and over again for the same tile If if we allow it to to live that long in the browser's browser's cache
05:45
It can expire tiles more or less automatically So basically if you define that's a tile in your cache can only be one hour old. It will Automatically delete that tile if it's requested after one hour and recreate it back from the WMS source
06:02
It will report errors either as a message either as an empty image Or just basic status codes inside there are your HTTP requests There's support for meta tiling So that's creating multiple tiles from a single large get map request to the source WMS server
06:21
You can watermark the tiles with your logo if you that's what you want and it's And it's can also up sample tiles from lower level zoom level so basically if you have 10 meter raster image, but you want to serve them up to
06:42
Up to zoom level 20 or 21 for Google Maps. There's no use caching those Low resolution tiles it will just recreate them from the low resolution data. That's that's available Once your image comes back from
07:03
WMS server it can be recompressed and optimized before being stored into the tile cache itself This means this is useful if you're using raster data and You want to avoid double JPEG
07:22
Compression decompression once I once from the WMS server and one from the tile cache So you'd request PNG from your source WMS server and then Compress the JPEG only once when you're storing to your tile cache It can also allow you to be aggressive on the compression levels. You're applying to the tiles when you're storing them
07:42
And you're preceding everything or you can also choose to use a Less aggressive compression for doing on-demand compression of image tiles So less CPU used but at the cost of a higher weight images
08:03
It supports multiple image formats that can be stored or returned to the clients so the two basic Image formats used on the web so PNG on which you can play with the compression level that's applied to the PNG data and you can also apply quantization to the data to store those tiles as 8 bits by choosing the
08:24
Correct 256 colors that best match the image. That's that's being stored For JPEG you can also play on the compression or quality level and you can choose in which color space they the JPEG compression takes place usually YCBCR gets you better compression results
08:42
There's also a kind of fun feature which allows you to choose let map cache Dynamically choose in which format it's going to store its tiles It's going to store JPEG if it sees that the image data is fully opaque and then PNG if it sees that There's a transparency in it. So basically that means if you have a satellite
09:04
raster that doesn't cover the whole world then all the past that are fully covered by the roster will be stored in JPEG to Reduce the bandwidth and then the images on the side which have no data values will be stored and returned as PNG So you can overlay that with other layers on your clients
09:23
It will also try to intelligently handle empty or uniform tiles Depending on the caching backends that are used It will use different strategies once it sees that the tile contains no data or just one single color For caching on the disk. It will use symbolic linking to
09:44
To one file It can also For other cache backends just store the color in the cache instead of storing the image style and then base Dynamically recreate a PNG image on the fly at the request time Returning a 1-bit PNG of a uniform color
10:02
So I don't remember exactly the size of that PNG file is something around the 180 bytes something like that If you are in the case where you know, your cache is already fully seeded And it's all treats a tile that's not present in the cache as a fully transparent file and return that to you
10:28
So now we talk about image formats Map cache supports multiple grids not only the Google marketer one so you can define multiple grids per
10:41
Tile set you are serving so you can serve them up in WGS 84 for overlaying on a globe and marketer It's all automatically detects. Well, you can configure it to respond to grid aliases. So the the eternal problem between the EPSG
11:00
3357 and the deprecated 913 when is handled it can handle non-standard EPS. Well non EPSG codes in that case Inside the grid you can also configure a tile set to only Be configured for a certain extent of certain resolutions of a grid
11:20
So just say I'm using the Google grid But I'm only want the first ten levels and then in one day one day if you want to change that and and Switch to a higher zoom level. You just have to change that and not have to recreate all those tiles Same for the restricted extent saying, okay. I'm now only going to cover the USA
11:42
And then switch to the whole world and not have to recreate all the times you've already created for the Cache backends, which is basically the most important part of the tile cache So a cache is a backend that's able to store tile data for a given XYZ
12:03
So that's really the basic part of it To be able to become a cache your backend has to support For basic operations, which is exists, you know if a tile is in the cache given the given XYZ Get back image data for a given XYZ store data for that one and delete data
12:25
There are some specific hacks or Features that are only available for certain cache backends. So now I'll go into more details about each cache backend This cache is stored tile as directly in the file system
12:44
I Put here mainly for development testing or small tile sets because storing millions or billions of tiles on the file system Usually very quickly get into the limits of what the file systems are designed to do So the pros about this discussion it's very simple to set up you just give it a directory in that casual
13:06
storage styles and it's it's Relatively or very fast it can support detection of blank files blank tiles and you have the option to Store tiles given different layouts so you can reuse existing caches that you have available from other servers
13:26
So it's by default it's going to use the layout used by the what tile cache created But you can also read caches created by ArcGIS Well wind or you can supply your own templates and to store given the given XYZ
13:42
And the Congress it's difficult to manage that large number of files you it's difficult to get statistics about how much Bases occupied by those files. It's difficult to copy them from one place to the other You hit file system limits depending on the file systems, of course, but in the usual case you run out of inodes
14:03
You run into too many files per directory, whatever and You may also waste storage space given the file system block size, so if you have a block size of Four twelve kilobytes and you're storing a 128 byte tiled and you still have four kilobytes that are occupied by that time
14:26
It supports SQLite caches. So this is just storing the The tile data as a blob in an SQLite database with the XY and Z columns so the pros about this Backend is that it's a single file that's going to contain all your tile cache
14:44
It's easy to copy move over to another server You can extend it to support any Schema you want well Supposing it stores data as a blob and you have XYZ columns
15:02
So If you have existing SQLite files with tiles in them, you can just plug map cache into it and it'll read the tiles from it and it's efficient in disk space in the sense that Only the space actually occupied by the tile is going to be taken on the file system
15:21
the cons about this Backend is that you may need some tweaking to do Pass advanced variables to ask you light when you're creating databases when you have more than one terabyte of data Don't remember. What's the exact limits? for that one, that's around one terabyte, I think and
15:40
Another problem is that SQLite is isn't designed to handle multiple insertions concurrently Which means if you have multiple map cache instances that are trying to push tiles inside the cache There's a lock on the database and you slow down the insertions very noticeably It supports
16:01
third-party memcache server as a backend, so memcache is For storing and transient data, it won't resist the reboots Ideal for a temporary data which for forecasts for census Whatever data is only valid for a limited amount of time or that can be easily recomputed
16:26
The Pros of this these cache backends is that you can distribute the load between your different map caches instances and memcache server and memcache will do automatic pruning for you of your caches So it's the the tiles that haven't been used for some time will get pushed out of the cache
16:44
To leave more room for the new tiles coming The cons is that as its memory based Storage that you have a limited storage available and as you have very deep pockets for lots of RAM due to caches
17:03
Cache that's used only for For raster data in JPEG. It's specialized to store satellite imagery So it's going to store the image tiles directly inside a tiff file and map cache will read inside this tiff file for the for the encoded data
17:21
You can choose to store Given number of tiles per tiff file so if you're storing two thousand by two thousand tiles and you're storing four million tiles inside one tiff file and Reducing the number of files you have to store in the file system by that factor. So It allows you to
17:40
Either create those stiff files by third-party service as long as it's aligned to the grids you're using And To determine and it's greatly reduces the number of fires you have to store in the file system The limitations of this back end is that it's
18:01
Limited to JPEG data so you can store images with the transparency or PNG data only JPEG. That's the limitation of the tiff file itself You can't write concurrently to a single tiff file. So there's a big lock around the writing operation so you know two processes right to the same tiff file and That if you're doing updates or deletions inside the tiff file
18:22
Then the tiff library won't reclaim the space that you've that you've removed from time So if you're doing lots of updates, and that's not the format to use your tiff file will continue to grow and grow This is a new cache that's added to the version that's coming out
18:41
Caches to store tiles on third parties a server that supports rest protocol. So it's simple Get put and delete on a URI By itself this probably wouldn't be very useful but It comes in handy because they are
19:00
authorization hooks To hook into popular cloud storage providers so you can if you have a cloud Credentials you can start out to s3 micro surface your and Google cloud storage for your tires Once you're storing your tiles in there, there's a cost-benefit ratio to
19:21
Take into account, of course because storage is not free on those services And you have to understand that you usually have to Run your map cache instance inside the same infrastructure as the caches themselves Given that
19:40
You have bandwidth costs to take into account. So you don't want to pay the bandwidth between your s3 storage and your server Yeah, that's it And unless Of course unless you have a fully seeded cache in which case you can just point them to the URLs of your your rest
20:04
endpoints Usually it's needed with a with a map cache instances running on EC2 basically There's a seeder that's that's shipped with map cache. So the process of seeding is just pre generating tiles
20:26
At one point so they are available. That's high performance Until then after that So the seeder is a multi-threaded or multi-process, which means you can have multiple instances running at the same time to if you to
20:41
Take advantage of multiple WMS servers or multiple CPUs on the map cache instances It works in drill down mode, which means to take advantage of the the file system caches on the WMS server So instead of just looping through the tiles by XYZ Naively it's all drilled down. So you're always staying in the same area of data on the WMS server
21:05
Usually you speed things up a lot You can seed a subset of your tile sets if that's what you want only for a given dimension you can regenerate old tiles that are older than a given date if you know that
21:26
From one dates Backwards you need to regenerate You can restrict to the zoom levels You want to seed or the extent and you can also restrict to arbitrary geometries with the OGR Expression syntax to extract the geometry. So basically you can say, okay
21:43
I want to seed level 18 on the polygons where my population density is Is bigger than a given value to pre generate where people actually are going to be using your service And there's also a pruning mode which will allow you to delete tiles. That's also match these given criterias. I
22:03
Talked about dimensions for this either just going to talk about dimensions here So basically for a tile set you can store multiple versions of the image data Typically if you think of forecast data Then you have a date for a forecast and then you want to store the tiles for a given date
22:21
For forecast actually you have two dates you have the dates when the forecast has been made and the date for which the forecast is valid So this is supported You can other typical dimension could be a client ID if you need to create time sets that are different and restricted by Extents for a given client ID or for elevation for temperature maps, whatever
22:44
Dimensions can be expressed as allowed values intervals Typically for that would be for elevation or just a regular expression that was that must match We've also added support for time dimension
23:02
In the case where you have An external database that provides you the available timestamps that you can serve You can either pass an interval So start time and end time to map cache and it will assemble the tiles that correspond to that interval
23:21
so that can either be done horizontally if you base if in the case of satellite images where you have Multiple scenes and you want to create a mosaic of all the images that have been created in 2013 Maybe for example, or it can also be What's coming next?
23:40
Is an animation so you could animate also with creating an animated gif Your tile over time For that to work you must supply a database of the available timestamps because what you get as an input is just an interval so you have to know which are the
24:01
individual values that correspond to that interval Future work that's Things that are in the pipes for upcoming versions Maybe native Google sources, so we don't have to go through WMS servers. Just read directly from Google from the raster data
24:23
Advanced cache management so that would be failover caches or redundancy if tile isn't present in a cache then look into another one or If a cache backend failed and use a backup cache Backend
24:41
Or Be able to move cache and tiles from one cache one slow cache to a faster one so move from file base caches to mem caches and Last point also that needs to be addressed rather rapidly is being able to store more than just image tiles but vector tiles open either you Jeff
25:00
grids or An upcoming vector tile spec and I'm over now. So if you have any questions I have two questions like the first one is about the duty of cash
25:22
This is this similar to the G dollar ado command where you add like overviews to to file It doesn't support overviews for now. It's only suppose so you'd have to create one if hierarchy per level presume level but it's if you create a tile tiff, so if you pass the LCO tile equals yes and
25:47
Compression equals JPEG. That's when it's going to be used. Okay second one was about the You know in the cedar You know I used a tile cache and I was also a proxy and this type of proximity problem does some of the
26:03
requesting a tile from remote WMS it didn't handle any errors and with met proxy you could have a configuration where it would redo the Request again when that failed is anything available in that cache
26:22
I think there's some code that went in to be able to Resume the ceiling after an error so you can start off from where the error happened So so it'll fail in that case and Once you fix your WMS server, it's a restart from where it stopped
26:42
Okay Thanks Seeding by geometry will use the bounding box of the geometry or actual clipping of tiles which would intersect it the actual clipping
27:00
And another one vertical assembly what kind of mixing options are available just plain old screen multiply that sort of deal Only screen. Okay. Yeah, I think I think other compositing options might make sense
27:21
The cedar is multi-threaded but Which back end do you recommend for doing multi-threaded seating since a lot of them do lock on the back end so Well, not ask you lights in any case Then cash would probably work well in that case and well
27:42
I'd like to investigate into Modifying the cedar so you can push tiles to the back end in a single thread That could be another option also any other questions. All right. Thank you